docs/qa/ClaudeQACoverage.md has carried this as a standing gap for four
rounds: whether predictions measurably improve at 3, 6 and 12 confirmed
cycles cannot be reached by a QA round, because it needs a simulated
history or a year of elapsed time. So the product's headline claim was
tested only one hand-picked fixture at a time.
This walks whole histories forward the way a device does -- predict,
score against what happened, feed the error back, predict again -- for
four simulated users across 400 seeds each, and asserts on aggregates.
Both directions are pinned. Accuracy floors alone would be satisfied by
an engine that answered "some time this month", so every accuracy row is
paired with a width ceiling and a confidence-honesty row.
Landing it before the calibration work it exists to guard, at thresholds
the current engine already meets, so each fix can be watched turning its
own row from red to green. A guard never seen failing is not known to
guard anything.
Two things the sample size taught, both worth keeping:
400 seeds rather than 40 costs 1.2 seconds and changes the conclusions.
The mid-learning coverage dip that a 40-seed run showed at 57% turns out
to be sampling noise -- the aggregate sits at 76-86% -- which is why the
assertions quantify over ranges of cycle-counts and never a single cell.
The variable user reads High twice in ~6,400 predictions, so this asserts
a rate ceiling rather than zero. Those two are not the failure §15
describes: the engine judges the evidence in front of it, not the process
behind it, and six cycles that happen to land within a day of each other
are a consistent history on the evidence. Asserting zero would have been
a guard that passed at 40 seeds and failed the first time anyone looked
harder.
Part of #50
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
335d0217a8