prove-guard.sh reads only one order of failure summary, so node --test falls through to a count that false-fires exit 3 #18

New Issue

Closed

opened 2026-08-17 23:38:16 -05:00 by null · 0 comments

null commented

2026-08-17 23:38:16 -05:00

Owner

The failure count preferred the runner's own summary via one pattern:

COUNT="$(grep -oiE '[0-9]+ (tests? )?failed' "$LOG" | tail -1 | grep -oE '^[0-9]+' || true)"

That matches 1 failed (vitest, pytest, Gradle) and nothing else. Runners that put the number on the right print no match: ℹ fail 1 (node --test), Failures: 2 (Maven, JUnit), failures=2 (python unittest), # fail 1 (TAP). Every one of those fell through to counting lines matching $PROVE_GUARD_FAIL_PATTERN.

What it costs

The fallback is not conservative, and [ "$COUNT" -gt 1 ] exits 3. A guard over a status enum, mutating the string 'FAILED', matches FAIL_PATTERN three times inside a single AssertionError diff:

AssertionError [ERR_ASSERTION]: Expected values to be strictly deep-equal:
-   'FAILED',
    actual: [ 'PASSED', 'FAILED', 'SKIPPED' ],

End-to-end against node --test, one failing test, a guard behaving perfectly:

OLD exit: 3  — "but 3 failures, by matching log lines, approximately"
NEW exit: 0  — "the guard caught it, and only it (the runner's summary)"

Exit 3 tells the author to "narrow the guard, or narrow the mutation" — so the script's advice, followed, weakens a correct guard. The script's own header records this exact false fire being tried and rejected ("a naive count calls that six coincidental failures. Tried that first; it fired on the very first run against a guard that was behaving perfectly"), but it stayed reachable through the fallback.

The message compounded it: it printed "this runner printed no summary" about a runner that printed one the script could not read. GUARDS.md already asserts "the failure count comes from the runner's own summary rather than from eyeballing red" — for these runners that claim was false.

What to do

Keep the existing pattern first, add a second reading the number on the right (\bfail(ure)?s?[:= ]+[0-9]+, last match wins), and only then fall back to the approximate count.

Traps

tail -1 matters: a test name containing fail: earlier in the log would otherwise win over the summary.
The [:= ] class is what picks up python unittest's failures=2; [: ] alone misses it.
Do not neuter exit 3 — two genuinely failing tests must still report 2 and exit 3.
Runners already matched by the first pattern must keep using it, unchanged.

Verify: bash prove-guard.sh status.js "'FAILED'" "'MUTATED'" node --test status.test.js exits 0 and reports "by the runner's summary" where it previously exited 3; a two-failing-test command still exits 3; vitest, pytest and Gradle summaries still resolve through the first pattern.

The failure count preferred the runner's own summary via one pattern: ```sh COUNT="$(grep -oiE '[0-9]+ (tests? )?failed' "$LOG" | tail -1 | grep -oE '^[0-9]+' || true)" ``` That matches `1 failed` (vitest, pytest, Gradle) and nothing else. Runners that put the number on the **right** print no match: `ℹ fail 1` (node --test), `Failures: 2` (Maven, JUnit), `failures=2` (python unittest), `# fail 1` (TAP). Every one of those fell through to counting lines matching `$PROVE_GUARD_FAIL_PATTERN`. ## What it costs The fallback is not conservative, and `[ "$COUNT" -gt 1 ]` exits 3. A guard over a status enum, mutating the string `'FAILED'`, matches `FAIL_PATTERN` three times inside a single `AssertionError` diff: ``` AssertionError [ERR_ASSERTION]: Expected values to be strictly deep-equal: - 'FAILED', actual: [ 'PASSED', 'FAILED', 'SKIPPED' ], ``` End-to-end against `node --test`, one failing test, a guard behaving perfectly: ``` OLD exit: 3 — "but 3 failures, by matching log lines, approximately" NEW exit: 0 — "the guard caught it, and only it (the runner's summary)" ``` Exit 3 tells the author to "narrow the guard, or narrow the mutation" — so the script's advice, followed, **weakens a correct guard**. The script's own header records this exact false fire being tried and rejected ("a naive count calls that six coincidental failures. Tried that first; it fired on the very first run against a guard that was behaving perfectly"), but it stayed reachable through the fallback. The message compounded it: it printed "this runner printed no summary" about a runner that printed one the script could not read. `GUARDS.md` already asserts "the failure count comes from the runner's own summary rather than from eyeballing red" — for these runners that claim was false. ## What to do Keep the existing pattern first, add a second reading the number on the right (`\bfail(ure)?s?[:= ]+[0-9]+`, last match wins), and only then fall back to the approximate count. ## Traps - `tail -1` matters: a test *name* containing `fail:` earlier in the log would otherwise win over the summary. - The `[:= ]` class is what picks up python unittest's `failures=2`; `[: ]` alone misses it. - Do not neuter exit 3 — two genuinely failing tests must still report 2 and exit 3. - Runners already matched by the first pattern must keep using it, unchanged. Verify: `bash prove-guard.sh status.js "'FAILED'" "'MUTATED'" node --test status.test.js` exits 0 and reports "by the runner's summary" where it previously exited 3; a two-failing-test command still exits 3; vitest, pytest and Gradle summaries still resolve through the first pattern.