From eaf42c38ea6e0608b08c48d4edd7c53ee13be9c2 Mon Sep 17 00:00:00 2001 From: null Date: Mon, 17 Aug 2026 23:39:17 -0500 Subject: [PATCH] fix(guards): prove-guard rejected correct guards, and refused with the wrong code Two defects in the same script, both found by running it against a node --test suite. ## The count read one order, and the fallback is not conservative The failure count preferred the runner's own summary through a single pattern, `[0-9]+ (tests? )?failed`. That matches vitest, pytest and Gradle and nothing else. Runners that put the number on the right matched nothing: `fail 1` from node --test, `Failures: 2` from Maven and JUnit, `failures=2` from python unittest, `# fail 1` from TAP. All of them fell through to counting lines that match $PROVE_GUARD_FAIL_PATTERN. That fallback overcounts, and `[ "$COUNT" -gt 1 ]` exits 3. A guard over a status enum, mutating the string 'FAILED', matches FAIL_PATTERN three times inside one AssertionError diff -- the message, the diff line, and the actual array. So a single failing test, from a guard behaving perfectly, exited 3 with "but 3 failures" and the advice to "narrow the guard, or narrow the mutation". Followed, that advice weakens a correct guard. The script's own header records this exact false fire being tried and rejected: "a naive count calls that six coincidental failures. Tried that first; it fired on the very first run against a guard that was behaving perfectly." It was rejected as the primary strategy and left reachable as the fallback. The message compounded it, reporting "this runner printed no summary" about a runner that printed one this script could not read. GUARDS.md already claims the count "comes from the runner's own summary rather than from eyeballing red". For four common runners that was false. The code now matches the claim, so no document needed changing -- the document was right. A second pattern reads the number on the right, last match wins, before the approximate fallback. The `[:= ]` class is what reaches python unittest's `failures=2`. Two genuinely failing tests still report 2 and still exit 3. ## Refusing is not a diagnosis, and it was using the diagnosis code The mutation step refuses when the find-string is absent or ambiguous, and both used `sys.exit("message")`. That prints to stderr and exits 1 -- the code this script reserves for "the guard stayed GREEN with its target broken". So a typo in the find-string returned a verdict about the code under test, from a run that never mutated anything and never executed the guard. The two states it most matters to distinguish were indistinguishable, and the wrong one is the alarming one. TOOLS.md teaches callers to read these codes and that "two is never a pass"; every other refusal path here already exited 2, only the embedded Python did not. Both refusals now raise SystemExit(2) through a helper that still writes the message to stderr. Both codes are non-zero, so no CI run passed that should have failed. This was a wrong diagnosis, not a missed failure. ## Verified The full exit matrix against node --test: correct guard 0, guard that cannot fail 1, two genuine failures 3, bad arguments 2, absent find-string 2, ambiguous find-string 2, missing file 2. The restore trap fires on every one and the file comes back intact. vitest, pytest and Gradle summaries still resolve through the first pattern, unchanged. closes #18 closes #19 Co-Authored-By: Claude Opus 5 (1M context) --- docs/architecture/scripts/prove-guard.sh | 48 ++++++++++++++++++++---- 1 file changed, 41 insertions(+), 7 deletions(-) diff --git a/docs/architecture/scripts/prove-guard.sh b/docs/architecture/scripts/prove-guard.sh index 4914d05..ec0a3d4 100755 --- a/docs/architecture/scripts/prove-guard.sh +++ b/docs/architecture/scripts/prove-guard.sh @@ -41,11 +41,14 @@ # on the very first run against a guard that was behaving perfectly. # # So the summary line is preferred, because almost every runner prints one and it -# is the runner's own count: `1 failed` from vitest, `6 tests completed, 1 failed` -# from Gradle, `1 failed, 5 passed` from pytest. The **last** such line wins, and -# only if none is found does it fall back to counting lines matching -# `$PROVE_GUARD_FAIL_PATTERN` — saying so, because an approximate count presented -# as an exact one is the kind of thing this script exists to object to. +# is the runner's own count. Runners disagree about which side of the word the +# number goes on, so both orders are read: `1 failed` from vitest, `1 failed, 5 +# passed` from pytest, `6 tests completed, 1 failed` from Gradle — and `fail 1` +# from node --test, `Failures: 2` from Maven and JUnit, `failures=2` from python +# unittest, `# fail 1` from TAP. The **last** such line wins, and only if none is +# found does it fall back to counting lines matching `$PROVE_GUARD_FAIL_PATTERN` +# — saying so, because an approximate count presented as an exact one is the kind +# of thing this script exists to object to. # # ## Exit codes # @@ -95,15 +98,30 @@ trap restore EXIT INT TERM # Exact-string replacement, and it must be unique. A mutation that lands in two # places proves nothing about either, and a regex here would make the mutation # itself the thing to debug. +# +# Both refusals here exit 2, like every other "nothing was proven" path +# above. `sys.exit("message")` prints it and exits **1** — the code this +# script reserves for "the guard stayed GREEN with its target broken", which +# is a diagnosis about the guard, not a refusal to run. So a mistyped +# find-string accused the guard under test of being broken. `TOOLS.md` +# teaches callers to tell 1 from 2 and that "two is never a pass"; that +# distinction has to survive this block. python3 - "$FILE" "$FIND" "$REPLACE" <<'PY' import sys + + +def refuse(message: str) -> None: + print(message, file=sys.stderr) + raise SystemExit(2) + + path, find, replace = sys.argv[1], sys.argv[2], sys.argv[3] text = open(path, encoding="utf-8").read() count = text.count(find) if count == 0: - sys.exit(f"prove-guard: the string to break is not in {path}") + refuse(f"prove-guard: the string to break is not in {path}") if count > 1: - sys.exit( + refuse( f"prove-guard: {count} occurrences of that string; a mutation in " "two places proves neither. Pick a longer, unique one." ) @@ -132,7 +150,23 @@ echo # The runner's own count, from the last summary line that states one. Preferred # over counting log lines for the reason in the header: one failing test is # routinely reported on half a dozen lines. +# Two orders, because runners disagree about which side the number goes on. +# The first pattern reads `1 failed` (vitest, pytest, Gradle); the second reads +# the number on the right: `ℹ fail 1` (node --test), `Failures: 2` (Maven, +# JUnit), `failures=2` (python unittest), `# fail 1` (TAP). +# +# Matching only the first order made every node run fall through to the +# approximate line count, and that fallback is not conservative. A guard over a +# status enum — mutating the string `'FAILED'` — matches FAIL_PATTERN three +# times inside one AssertionError diff, so a single correct guard exited 3 with +# "narrow the guard, or narrow the mutation". The header says that exact false +# fire was tried once and rejected; it was still reachable through the fallback. +# The message compounded it, reporting "this runner printed no summary" about a +# runner that printed one this script could not read. COUNT="$(grep -oiE '[0-9]+ (tests? )?failed' "$LOG" | tail -1 | grep -oE '^[0-9]+' || true)" +if [ -z "$COUNT" ]; then + COUNT="$(grep -oiE '\bfail(ure)?s?[:= ]+[0-9]+' "$LOG" | tail -1 | grep -oE '[0-9]+$' || true)" +fi COUNTED_BY="the runner's summary" if [ -z "$COUNT" ]; then COUNT="$(grep -cE "$FAIL_PATTERN" "$LOG" || true)"