fix: account for not knowing her centre, not just her spread

The window promises to hold the actual start four times in five. For a
user whose cycles genuinely disagree it was delivering closer to three in
four, and the missing piece was structural rather than a mistuned knob:
the spread estimate measured how far her cycles fall from a centre and
never asked how well that centre is known. With four intervals it is not
known well.

The correction is the textbook one -- a future draw around an ESTIMATED
centre is wider than around the true centre by sqrt(1 + 1/n) -- taken over
the effective sample size, because recency weighting means eight intervals
do not contribute eight intervals' worth of evidence.

Applied only to the branch that actually estimates from a sample. The
zero- and one-interval branches are hand-tuned constants standing in for
total uncertainty when there is nothing to estimate from, and multiplying
those would have counted the same doubt twice -- measured, that mistake
widened a first-cycle window by a day and a half and bought nothing.

Not a knob: 1.22x at two intervals, 1.06x at eight, 1.03x at fifteen,
fixed by the arithmetic rather than chosen to reach a number.

The issue this closes was filed on a wrong measurement, and that is worth
recording. A 40-seed run showed mid-learning coverage collapsing to 57.5%;
at 400 seeds the same cells sit at 73-85%, and the collapse was one
unlucky cell rather than a defect. What survived the larger sample is
smaller and real, and is what this fixes: variable-user coverage 75.6% ->
76.8% late, 72.2% -> 73.3% mid-learning.

The floor in the guard stays below the promise on purpose. For an
intrinsically unpredictable cycle the last few points would cost width she
cannot afford, and the width ceiling in the same file is the other jaw of
that vice.

Fixture window returns to 4.56 days, exactly where it was before this
batch -- so across the three calibration commits the engine now predicts
the §51 fleet at MAE 0.44 instead of 0.67, with 9/9 coverage, for the same
window it always had.

closes #52

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
null 2026-08-20 16:30:00 -05:00
parent 45aefbbcb6
commit 1761bf33a0
2 changed files with 46 additions and 3 deletions

View File

@ -5,6 +5,7 @@ import kotlin.math.abs
import kotlin.math.exp import kotlin.math.exp
import kotlin.math.ln import kotlin.math.ln
import kotlin.math.roundToLong import kotlin.math.roundToLong
import kotlin.math.sqrt
/** /**
* The engine PRODUCT_PLAN.md §12 specifies. * The engine PRODUCT_PLAN.md §12 specifies.
@ -298,7 +299,30 @@ class PersonalPredictionEngine : PredictionEngine {
val meanAd = if (totalWeight <= 0.0) 0.0 else { val meanAd = if (totalWeight <= 0.0) 0.0 else {
deviations.sumOf { (d, w) -> d * w } / totalWeight * MEAN_AD_TO_SCALE deviations.sumOf { (d, w) -> d * w } / totalWeight * MEAN_AD_TO_SCALE
} }
maxOf(mad, meanAd) // Her cycles vary, AND the centre they vary around is only an
// estimate. Everything above measures the first; this is the
// second, and leaving it out is why an early window promised 80%
// and delivered less.
//
// The correction is the textbook one — a future draw around an
// ESTIMATED centre is wider than around the true centre by
// sqrt(1 + 1/n) — over the EFFECTIVE sample size, because
// recency weighting means eight intervals do not contribute
// eight intervals' worth of evidence. Kish's formula says what
// they do contribute.
//
// Applied here and not to the branches above, deliberately:
// those are hand-tuned constants standing in for total
// uncertainty when there is nothing to estimate from, so
// multiplying them would count the same doubt twice.
//
// Not a knob. 1.22x at two intervals, 1.06x at eight, 1.03x at
// fifteen — the size is fixed by the arithmetic rather than
// chosen to reach a number.
val totalWeight2 = intervals.sumOf { it.weight }
val effectiveN = totalWeight2 * totalWeight2 /
intervals.sumOf { it.weight * it.weight }
maxOf(mad, meanAd) * sqrt(1.0 + 1.0 / effectiveN)
} }
} }

View File

@ -324,14 +324,33 @@ class LearningCurveTest {
} }
} }
/** Guards the small-sample widening of the spread estimate. Was 57.5% at worst. */ /**
* Guards the estimation-uncertainty term in the spread estimate.
*
* Worth recording what this row is *not*, because the first measurement said
* otherwise. A 40-seed run showed mid-learning coverage collapsing to 57.5%
* and looked like a serious calibration fault; at 400 seeds the same cells
* sit at 73-85%, and the collapse turns out to have been one unlucky
* (profile, seed, cycle-count) cell rather than a defect. It is the reason
* every assertion in this file quantifies over a range.
*
* What survived the larger sample is smaller and real: a user whose cycles
* genuinely disagree is covered a few points under the 80% the window
* promises, because the engine knows her *centre* only approximately and the
* spread estimate did not account for that. It does now.
*
* The floor is deliberately below the promise. For an intrinsically
* unpredictable cycle, closing the last few points would cost width she
* cannot afford see the width ceiling above, which is the other jaw of
* this vice.
*/
@Test @Test
fun `the window does not tighten faster than the history earns`() { fun `the window does not tighten faster than the history earns`() {
profiles.forEach { profile -> profiles.forEach { profile ->
val c = cell(profile.name) { it in 3..6 } val c = cell(profile.name) { it in 3..6 }
assertTrue( assertTrue(
"${profile.name} covered ${"%.1f".format(c.coverage * 100)}% mid-learning", "${profile.name} covered ${"%.1f".format(c.coverage * 100)}% mid-learning",
c.coverage >= 0.55, c.coverage >= 0.65,
) )
} }
} }