Commit the learning-curve simulation as a permanent guard

null commented

2026-08-20 16:13:29 -05:00

Owner

What is true now. docs/qa/ClaudeQACoverage.md records a standing gap: whether predictions measurably improve at 3, 6 and 12 confirmed cycles cannot be reached by any QA round, so 'the product's headline claim is tested only at the unit level'. A throwaway simulation has now measured it (stable users reach a ~0.8-1.0 day MAE floor by cycle 2-3; variable users never converge and are correctly reported as uncertain), but nothing in the tree stops that regressing.

What to do. Add LearningCurveTest to domain/prediction/src/test: four profiles (stable 28, stable 35, variable 25-37, drifting +0.5 d/cycle), 40 seeds each, 15 cycles, the loop mirroring CycleRepository.forecast including fed-back absolute errors. Aggregate per cycle-count across seeds and assert on ranges — MAE ceilings, coverage floors near WINDOW_MASS, confidence-label shares. Print the per-k table as EngineComparisonTest prints its own.

Traps. Use java.util.Random(seed): its algorithm is frozen by the Java spec, whereas kotlin.random.Random is explicitly not stable across Kotlin releases and would make a committed guard flaky on a toolchain upgrade. Assert on aggregates, never a single (profile, seed, k) cell. Land the guard at thresholds the current engine already meets, then tighten each row in the calibration commit it guards, observing red-before/green-after — a guard never seen failing is not known to guard anything (docs/architecture/GUARDS.md).

Verify: ./gradlew :domain:prediction:test --tests '*LearningCurveTest*' green in under two seconds, and docs/qa/ClaudeQACoverage.md's standing-gap entry updated to say the simulated-history half is closed and on-device elapsed time remains open.

**What is true now.** docs/qa/ClaudeQACoverage.md records a standing gap: whether predictions measurably improve at 3, 6 and 12 confirmed cycles cannot be reached by any QA round, so 'the product's headline claim is tested only at the unit level'. A throwaway simulation has now measured it (stable users reach a ~0.8-1.0 day MAE floor by cycle 2-3; variable users never converge and are correctly reported as uncertain), but nothing in the tree stops that regressing. **What to do.** Add `LearningCurveTest` to domain/prediction/src/test: four profiles (stable 28, stable 35, variable 25-37, drifting +0.5 d/cycle), 40 seeds each, 15 cycles, the loop mirroring `CycleRepository.forecast` including fed-back absolute errors. Aggregate per cycle-count across seeds and assert on ranges — MAE ceilings, coverage floors near WINDOW_MASS, confidence-label shares. Print the per-k table as EngineComparisonTest prints its own. **Traps.** Use `java.util.Random(seed)`: its algorithm is frozen by the Java spec, whereas `kotlin.random.Random` is explicitly not stable across Kotlin releases and would make a committed guard flaky on a toolchain upgrade. Assert on aggregates, never a single (profile, seed, k) cell. Land the guard at thresholds the current engine already meets, then tighten each row in the calibration commit it guards, observing red-before/green-after — a guard never seen failing is not known to guard anything (docs/architecture/GUARDS.md). Verify: `./gradlew :domain:prediction:test --tests '*LearningCurveTest*'` green in under two seconds, and docs/qa/ClaudeQACoverage.md's standing-gap entry updated to say the simulated-history half is closed and on-device elapsed time remains open.

null added this to the Batch 09 — Prediction Science & Hardening milestone 2026-08-20 16:13:29 -05:00

null referenced this issue from a commit

2026-08-20 16:18:06 -05:00

test: measure whether the engine actually learns, and keep measuring it

null commented

2026-08-20 16:41:10 -05:00

Author

Owner

LearningCurveTest lands at domain/prediction/src/test/kotlin/dev/privacyllc/period/domain/prediction/LearningCurveTest.kt in 335d021: four profiles, 400 seeds each, 15 cycles, mirroring CycleRepository.forecast including fed-back errors. Runs in 1.2s. Committed at thresholds the then-current engine met, then each calibration commit tightened its own row red-before/green-after. It immediately earned itself by finding #51 and #53. Run: ./gradlew :domain:prediction:test --tests 'LearningCurveTest'

LearningCurveTest lands at domain/prediction/src/test/kotlin/dev/privacyllc/period/domain/prediction/LearningCurveTest.kt in 335d021: four profiles, 400 seeds each, 15 cycles, mirroring CycleRepository.forecast including fed-back errors. Runs in 1.2s. Committed at thresholds the then-current engine met, then each calibration commit tightened its own row red-before/green-after. It immediately earned itself by finding #51 and #53. Run: ./gradlew :domain:prediction:test --tests '*LearningCurveTest*'

null closed this issue

2026-08-20 16:41:11 -05:00

Commit the learning-curve simulation as a permanent guard #50