5.0 KiB
Claude QA Plan —
Status: Current
Owner: <who maintains this>
Last reviewed: <YYYY-MM-DD>
Governs: what a QA round consists of
Review trigger: Any new user-facing surface, or a defect class that got through
The playbook. What a round is, so two rounds are comparable and a gap is visible rather than assumed covered.
Before a round
- Build from a clean checkout at a known SHA, and record that SHA.
- Run from a detached worktree if other work is in flight, so uncommitted changes cannot contaminate what is under test.
- Note the environment: device, OS version, browser, screen size — whatever the product's behaviour actually depends on.
The passes
Each pass gets a letter, so ClaudeQACoverage.md can report per pass and a
skipped one is visible.
| Pass | What it covers |
|---|---|
| A | First run: install or load, cold start, permissions, empty states |
| B | The core flow, end to end, as a real user would do it |
| C | The core flow with things going wrong: no network, denied permission, invalid input |
| D | Persistence: quit and return, background and resume, restart |
| E | The end of the loop — the state that is hardest to reach on purpose |
| F | Accessibility: keyboard only, screen reader labels, contrast, text scaling |
| G | Performance under the load this product will actually see |
| H | Authorization and abuse: every authenticated endpoint called logged-out; a list endpoint checked for rows the caller should not see; User A requesting User B's object by id; a privileged route opened as an ordinary user; the most expensive endpoint hit repeatedly; the built bundle and localStorage inspected for anything secret |
| I | (only where money moves) Money flowing backwards: refund, chargeback, cancellation and failed renewal each take away what the purchase gave |
<Add, remove and rename to fit. A pass that never applies is noise; a pass that is always skipped is a lie.>
Why H is separate from B
Authenticated is not the same as owning, and neither is the same as permitted. Login is the front door; every room inside still needs its own lock. Pass B walks the flow as a real user, which means it walks it as a user who is entitled to everything it touches — so B cannot see the entire class of defect where the entitlement was never checked.
This has already cost this portfolio once. requireCoupleContext never verified
that the caller belonged to the couple whose data was being returned:
authentication was present, correct, and proved nothing. No pass A-G would have
looked for it, because every one of them asks a legitimate user to do legitimate
things.
Run H as somebody who is not entitled: logged out, logged in as the wrong person, and logged in as the right person asking too often.
Why I is conditional, and why it is here at all
Pass B buys. Pass I unbuys, and it is the pass most often missing: builds implement the checkout-success webhook and leave every reverse path unhandled, so access is granted once and never revoked. Refunded, charged back, cancelled, failed renewal — the money went back and the entitlement did not.
(precautionary) — this one is not learned here. It comes from people who audit these applications for a living, one of whom calls it the single thing they most often fix, and from a report of a refund defect costing a financial institution six figures a month. Borrowed evidence is still evidence, but the marker says which it is so nobody mistakes it for our own scar.
Delete this pass outright if the project takes no money. Keeping it as a permanently skipped row is the lie this file warns about two paragraphs up.
What counts as a finding
A finding needs: what was done, what happened, what should have happened, and the build SHA. Without the SHA it cannot be re-tested, and a finding that cannot be re-tested cannot be closed.
Severity
Findings are filed as issues, labelled:
- P0 — ships broken, or loses data
- P1 — materially wrong, but shippable
- P2 — cosmetic or low impact
- release-blocker — a release built today would be wrong rather than merely incomplete
Exactly these label names: the Command Center queries them by name, and a repository that spells them differently has its defects reported as not adopted rather than counted wrongly.
Severity is what it costs, not how annoying it is to fix.
After a round
File each finding as a labelled issue. Update ClaudeReport.md's run-state
block and its overall sentence, and ClaudeQACoverage.md with what each pass
actually reached. A pass that could
not be run is recorded as blocked, with what blocks it — never quietly left out,
which reads identically to "passed".
Then push, and reconcile. The verdict on the project screen at
privacyllc.dev is read out of ClaudeReport.md in the pushed repository, so a
round whose report is committed but not pushed — or pushed but not reconciled —
leaves a stakeholder reading the previous round's judgment with no indication
that a newer one exists. The rest of the cycle is in docs/WORK_CYCLE.md.