Two defects in the same script, both found by running it against a node --test
suite.
## The count read one order, and the fallback is not conservative
The failure count preferred the runner's own summary through a single pattern,
`[0-9]+ (tests? )?failed`. That matches vitest, pytest and Gradle and nothing
else. Runners that put the number on the right matched nothing: `fail 1` from
node --test, `Failures: 2` from Maven and JUnit, `failures=2` from python
unittest, `# fail 1` from TAP. All of them fell through to counting lines that
match $PROVE_GUARD_FAIL_PATTERN.
That fallback overcounts, and `[ "$COUNT" -gt 1 ]` exits 3. A guard over a status
enum, mutating the string 'FAILED', matches FAIL_PATTERN three times inside one
AssertionError diff -- the message, the diff line, and the actual array. So a
single failing test, from a guard behaving perfectly, exited 3 with "but 3
failures" and the advice to "narrow the guard, or narrow the mutation". Followed,
that advice weakens a correct guard.
The script's own header records this exact false fire being tried and rejected:
"a naive count calls that six coincidental failures. Tried that first; it fired
on the very first run against a guard that was behaving perfectly." It was
rejected as the primary strategy and left reachable as the fallback. The message
compounded it, reporting "this runner printed no summary" about a runner that
printed one this script could not read.
GUARDS.md already claims the count "comes from the runner's own summary rather
than from eyeballing red". For four common runners that was false. The code now
matches the claim, so no document needed changing -- the document was right.
A second pattern reads the number on the right, last match wins, before the
approximate fallback. The `[:= ]` class is what reaches python unittest's
`failures=2`. Two genuinely failing tests still report 2 and still exit 3.
## Refusing is not a diagnosis, and it was using the diagnosis code
The mutation step refuses when the find-string is absent or ambiguous, and both
used `sys.exit("message")`. That prints to stderr and exits 1 -- the code this
script reserves for "the guard stayed GREEN with its target broken".
So a typo in the find-string returned a verdict about the code under test, from
a run that never mutated anything and never executed the guard. The two states
it most matters to distinguish were indistinguishable, and the wrong one is the
alarming one. TOOLS.md teaches callers to read these codes and that "two is
never a pass"; every other refusal path here already exited 2, only the embedded
Python did not. Both refusals now raise SystemExit(2) through a helper that
still writes the message to stderr.
Both codes are non-zero, so no CI run passed that should have failed. This was a
wrong diagnosis, not a missed failure.
## Verified
The full exit matrix against node --test: correct guard 0, guard that cannot
fail 1, two genuine failures 3, bad arguments 2, absent find-string 2, ambiguous
find-string 2, missing file 2. The restore trap fires on every one and the file
comes back intact. vitest, pytest and Gradle summaries still resolve through the
first pattern, unchanged.
closes#18closes#19
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The basis for every project here was itself unversioned: no .git, no remote,
no history. Changes to it had no diff and no revert, and two of its own guards
could not run at all -- doc-claims.sh and doc-triggers.py both read git
history, so the script written to catch documentation drift could not be run
against the documents that define drift.
This is the tree as it stands, including work that until now existed only as
loose files on disk: WORK_CYCLE.md, TOOLS.md, the Portainer image-line fix in
deploy.py, the status vocabulary corrected to the four words the conformance
checker actually enforces, the Exempt: mechanism documented, and the Forgejo
instance named in README.md.
secrets.sh --tracked reports one candidate, migrate.sh:480. It is the comment
documenting the three Postgres credential shapes that script redacts, with
literal placeholders, and it is left alone deliberately: GUARDS.md section 2
is that a source-grep guard must tell code from the comment about code, and
deleting an explanation to quiet a scanner is the failure it names.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
null