diff --git a/docs/history/DEVELOPMENT_LOG.md b/docs/history/DEVELOPMENT_LOG.md index 96af0a7..ce136e1 100644 --- a/docs/history/DEVELOPMENT_LOG.md +++ b/docs/history/DEVELOPMENT_LOG.md @@ -32,6 +32,90 @@ written and stay true. It is exempt from review for the same reason a receipt is ## Entries +### 2026-08-20 (later) — Batch 09: the learning loop, measured + +Started from a question — *how well does this app learn a period?* — and +answered it by measuring rather than reading, which turned up more than the +question asked for. + +**The engine does learn.** A stable user reaches the engine's accuracy floor, +around 0.8 days, within two or three cycles, and an intrinsically variable one +never converges and is correctly told so. That much was already true. + +**The loop around it was losing evidence.** Four defects, none of which looked +broken because each produced plausible figures with something missing. A +forecast's `generatedAt` was restamped by every "Not yet", edit and delete, so a +period logged retroactively tripped the backfill guard and the forecast the user +was actually shown was deleted unscored — the app learned nothing from precisely +the cycle it got wrong. A null forecast skipped the retire step, leaving a +snapshot about a deleted history to be scored against an unrelated one. Scored +snapshots never followed the period they measured when it was edited or deleted. +And a retroactive confirm left later-dated "not yet" answers alive to depress the +next cycle's confidence. `generatedAt` is now a lineage origin carried across +revisions; scores re-score or retract with their ground truth; a resolving +confirm clears every observation. No schema change — all four are queries and +call order. Six of seven new tests were watched failing first. + +**Three calibration faults, all found by a harness that did not exist.** +`LearningCurveTest` walks four simulated users through fifteen cycles, 400 seeds +each, scoring and feeding back exactly as the repository does. It closes half of +the QA doc's standing long-horizon gap and immediately earned itself: + +- A textbook-regular user read High in 15.7% of forecasts. Confidence divided + raw error, so the moment real errors arrived — averaging 0.9 days against a + cycle that varies by a day — she scored worse than when she had never been + measured at all. Errors are now scored against what the window already admits + (for a Laplace, E|X| = b), and she reads High 88% of the time. +- A drifting user was tracked to within 1.2 days and told the app was losing + confidence, because spread was measured from a static centre and grew with the + history however well the trend was followed. Residuals now come off the trend + line. That also revealed the damping had been *paying for* the measurement + error: raising it 0.50 → 0.70 halved the fixture error, took fleet MAE from + 0.67 to 0.44, restored 9/9 coverage and narrowed the mean window. +- The window promised 80% and delivered less early on, because the spread + estimate never accounted for how well the *centre* was known. A √(1 + 1/n) + term over the effective sample size fixes it. + +**Two corrections worth recording.** The coverage dip that prompted #52 was a +40-seed artifact — at 400 seeds those cells sit at 73–85%, and the fix was +rewritten around what survived. And a sweep loop left `MEDIUM_THRESHOLD` at 0.44 +when 0.40 was the measured choice; both passed the suite, so only the numbers +caught it. Fixed in the following commit. + +`modelVersion` is now `personal-2`. + +**Batch 09 also wrote down what the engine assumes about bodies.** +`docs/architecture/SCIENCE.md` gives every physiology constant a verdict and a +citation, and says plainly which constants are engineering and need none. One +value moved: the population default, used only for a user with a single logged +period, goes 28 → 29, which is what the large app-data studies actually measure. +The luteal phase is filed rather than changed (#56) — 14 is above the measured +mean, but moving it shifts what the fertility screen shows every user, and that +is a product decision with its own evidence. The Insights banner also stopped +promising "predictions get noticeably better" before a single forecast had been +scored. + +**And the calendar reaches a year (#57, Batch 10).** `CycleProjection` is a +separate type from `Prediction` because a forecast is scored and a projection +never is. Uncertainty grows as √n, it declines rather than stretching past ten +days either side, and every month showing projected marks carries *"If your +cycles continue as they have, this is the forecast"* with the confidence for that +distance. The opposite of the widget library that prompted it, which stamps a +fixed 28 days twelve months forward and draws it exactly like the next period. + +Batch 10 also holds two filed-not-built items: restore-from-export (#58 — the +export is currently a one-way door, so a new phone starts the model from zero) +and a cycle-length chart for Insights (#59). + +**Next action:** decide #56 — measure a 13-day luteal phase and either move +`DEFAULT_LUTEAL_PHASE_DAYS` with its evidence or record why 14 stays. Then #58, +restore-from-export, which is the one item with a data-ownership obligation +behind it. + +**Blockers:** none in code. The unreachable half of the long-horizon QA gap +stands — a simulation exercises the engine, not a year-old install — and API 26 +still has no prediction coverage, both recorded in `docs/qa/ClaudeQACoverage.md`. + ### 2026-08-20 (final) — Discreet launcher alias landed Issue #31 is ready. The normal launcher entry is now diff --git a/docs/qa/ClaudeQACoverage.md b/docs/qa/ClaudeQACoverage.md index 5e13a57..048d9ab 100644 --- a/docs/qa/ClaudeQACoverage.md +++ b/docs/qa/ClaudeQACoverage.md @@ -234,7 +234,23 @@ existing is not a failure; it not existing while the gaps do is. - **Physical-device coverage is undecided.** Passes F and G need a real device with a lock screen and TalkBack; which device that is has not been chosen, and an emulator is not a substitute for either. -- **Long-horizon accuracy** — whether predictions measurably improve at 3, 6 and - 12 confirmed cycles — cannot be reached by a QA round at all. It needs either - a simulated history harness or real elapsed time, and until one exists the - product's headline claim is tested only at the unit level. +- **Long-horizon accuracy — half closed.** Whether predictions measurably improve + at 3, 6 and 12 confirmed cycles cannot be reached by a QA round at all: it + needs either a simulated history harness or real elapsed time. The harness now + exists. `LearningCurveTest` walks four simulated users through fifteen cycles + each, 400 seeds apiece, scoring every forecast and feeding the error back + exactly as `CycleRepository` does, and asserts on the aggregates — accuracy + floors, coverage against the window's own 80% promise, and how often each user + is told High or Low. + + It found what it was built to find: a textbook-regular user was reading High in + only 15.7% of her forecasts, and a drifting user was tracked to within 1.2 days + while being told the app was losing confidence. Both are fixed, and the guard + now holds them. + + **What is still open is the other half.** A simulation exercises the engine, + not the app: nothing here proves that a real install accumulates those cycles + correctly over a year, that the snapshots survive updates and time-zone + changes, or that what the user sees on the accuracy card matches what the + engine computed. That needs elapsed time on a device and no harness replaces + it.