Two defects in the same script, both found by running it against a node --test
suite.
## The count read one order, and the fallback is not conservative
The failure count preferred the runner's own summary through a single pattern,
`[0-9]+ (tests? )?failed`. That matches vitest, pytest and Gradle and nothing
else. Runners that put the number on the right matched nothing: `fail 1` from
node --test, `Failures: 2` from Maven and JUnit, `failures=2` from python
unittest, `# fail 1` from TAP. All of them fell through to counting lines that
match $PROVE_GUARD_FAIL_PATTERN.
That fallback overcounts, and `[ "$COUNT" -gt 1 ]` exits 3. A guard over a status
enum, mutating the string 'FAILED', matches FAIL_PATTERN three times inside one
AssertionError diff -- the message, the diff line, and the actual array. So a
single failing test, from a guard behaving perfectly, exited 3 with "but 3
failures" and the advice to "narrow the guard, or narrow the mutation". Followed,
that advice weakens a correct guard.
The script's own header records this exact false fire being tried and rejected:
"a naive count calls that six coincidental failures. Tried that first; it fired
on the very first run against a guard that was behaving perfectly." It was
rejected as the primary strategy and left reachable as the fallback. The message
compounded it, reporting "this runner printed no summary" about a runner that
printed one this script could not read.
GUARDS.md already claims the count "comes from the runner's own summary rather
than from eyeballing red". For four common runners that was false. The code now
matches the claim, so no document needed changing -- the document was right.
A second pattern reads the number on the right, last match wins, before the
approximate fallback. The `[:= ]` class is what reaches python unittest's
`failures=2`. Two genuinely failing tests still report 2 and still exit 3.
## Refusing is not a diagnosis, and it was using the diagnosis code
The mutation step refuses when the find-string is absent or ambiguous, and both
used `sys.exit("message")`. That prints to stderr and exits 1 -- the code this
script reserves for "the guard stayed GREEN with its target broken".
So a typo in the find-string returned a verdict about the code under test, from
a run that never mutated anything and never executed the guard. The two states
it most matters to distinguish were indistinguishable, and the wrong one is the
alarming one. TOOLS.md teaches callers to read these codes and that "two is
never a pass"; every other refusal path here already exited 2, only the embedded
Python did not. Both refusals now raise SystemExit(2) through a helper that
still writes the message to stderr.
Both codes are non-zero, so no CI run passed that should have failed. This was a
wrong diagnosis, not a missed failure.
## Verified
The full exit matrix against node --test: correct guard 0, guard that cannot
fail 1, two genuine failures 3, bad arguments 2, absent find-string 2, ambiguous
find-string 2, missing file 2. The restore trap fires on every one and the file
comes back intact. vitest, pytest and Gradle summaries still resolve through the
first pattern, unchanged.
closes#18closes#19
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
eaf42c38ea