A textbook-regular user never sustains High confidence #51

New Issue

Closed

opened 2026-08-20 16:13:30 -05:00 by null · 0 comments

null commented

2026-08-20 16:13:30 -05:00

Owner

Measured by the learning-curve simulation at 93ec5b7.

What is true now. For a perfectly stable user (28 +/- 1 day), High confidence peaks at 25% of predictions at five cycles of history and decays to 10-15% by fifteen; Medium takes 85-90%. The cause is an asymmetry in PersonalPredictionEngine.confidence(): with no scored errors the accuracy factor is NEUTRAL_ACCURACY 0.92 and a stable user scores ~0.64 (High); once real errors exist, a mean error of ~0.9 days gives 1/(1 + 0.9/2.5) ~= 0.735 and she scores ~0.51 — below HIGH_THRESHOLD 0.55, permanently. Being measurably as accurate as her own cycle variability physically allows scores worse than never having been measured.

What it costs. The product's headline claim (§3, 'has the prediction engine become more accurate as it learns me?') is contradicted on screen for the exact user it should serve best: the app gets better and tells her it is less sure.

What to do. Score excess error, not raw error. For a Laplace distribution with scale b, E|X| = b — the achievable mean error of a perfectly calibrated forecast is its scale, so errors at or below scale are confirmation of calibration rather than evidence against it. Subtract the admitted error before the penalty: excess = (mean - scale * EXPECTED_ERROR_PER_SCALE).coerceAtLeast(0.0), with the constant at 1.0 as a property of the distribution the engine already chose, not a tuned number.

Traps. Do not lower HIGH_THRESHOLD instead — that is what would let a variable user read High and break §15. Under the excess formulation her accuracy factor rises to ~1.0 but her agreement term at scale ~4 is ~0.28, and High needs scale <= ~1.31 even at accuracy 1.0 — unreachable for a genuinely variable history, which is the structural argument that §15 survives. Errors must still crush confidence when they are real: they route through agreement, since ERROR_TO_SCALE already feeds scored error into the scale.

Verify: LearningCurveTest's stable-profile High-share assertion at k>=8 (red before at 0.10-0.15, green after), plus the unchanged §15 pins confidence is agreement rather than volume, volume alone never buys high confidence, and the simulation's never-High assertion over 600 variable-user predictions.

Measured by the learning-curve simulation at 93ec5b7. **What is true now.** For a perfectly stable user (28 +/- 1 day), High confidence peaks at 25% of predictions at five cycles of history and decays to 10-15% by fifteen; Medium takes 85-90%. The cause is an asymmetry in `PersonalPredictionEngine.confidence()`: with no scored errors the accuracy factor is `NEUTRAL_ACCURACY` 0.92 and a stable user scores ~0.64 (High); once real errors exist, a mean error of ~0.9 days gives `1/(1 + 0.9/2.5)` ~= 0.735 and she scores ~0.51 — below `HIGH_THRESHOLD` 0.55, permanently. Being measurably as accurate as her own cycle variability physically allows scores *worse* than never having been measured. **What it costs.** The product's headline claim (§3, 'has the prediction engine become more accurate as it learns me?') is contradicted on screen for the exact user it should serve best: the app gets better and tells her it is less sure. **What to do.** Score *excess* error, not raw error. For a Laplace distribution with scale b, E|X| = b — the achievable mean error of a perfectly calibrated forecast *is* its scale, so errors at or below scale are confirmation of calibration rather than evidence against it. Subtract the admitted error before the penalty: `excess = (mean - scale * EXPECTED_ERROR_PER_SCALE).coerceAtLeast(0.0)`, with the constant at 1.0 as a property of the distribution the engine already chose, not a tuned number. **Traps.** Do not lower `HIGH_THRESHOLD` instead — that is what would let a variable user read High and break §15. Under the excess formulation her accuracy factor rises to ~1.0 but her agreement term at scale ~4 is ~0.28, and High needs scale <= ~1.31 even at accuracy 1.0 — unreachable for a genuinely variable history, which is the structural argument that §15 survives. Errors must still crush confidence when they are real: they route through agreement, since `ERROR_TO_SCALE` already feeds scored error into the scale. Verify: `LearningCurveTest`'s stable-profile High-share assertion at k>=8 (red before at 0.10-0.15, green after), plus the unchanged §15 pins `confidence is agreement rather than volume`, `volume alone never buys high confidence`, and the simulation's never-High assertion over 600 variable-user predictions.