null
/
Project-Template
1
0
Fork 0
Code Issues 1 Pull Requests Packages Projects Releases Wiki Activity

chore(repo): put the template under version control 

The basis for every project here was itself unversioned: no .git, no remote,
no history. Changes to it had no diff and no revert, and two of its own guards
could not run at all -- doc-claims.sh and doc-triggers.py both read git
history, so the script written to catch documentation drift could not be run
against the documents that define drift.

This is the tree as it stands, including work that until now existed only as
loose files on disk: WORK_CYCLE.md, TOOLS.md, the Portainer image-line fix in
deploy.py, the status vocabulary corrected to the four words the conformance
checker actually enforces, the Exempt: mechanism documented, and the Forgejo
instance named in README.md.

secrets.sh --tracked reports one candidate, migrate.sh:480. It is the comment
documenting the three Postgres credential shapes that script redacts, with
literal placeholders, and it is left alone deliberately: GUARDS.md section 2
is that a source-grep guard must tell code from the comment about code, and
deleting an explanation to quiet a scanner is the failure it names.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
null 2026-08-17 22:44:26 -05:00
commit 6965915dbd
45 changed files with 12477 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

15
.gitignore vendored Normal file
View File

@ -0,0 +1,15 @@
# Python bytecode from the scripts in docs/architecture/scripts/
__pycache__/
*.py[cod]
# Credentials never belong in this tree. docs/security/SECURITY.md is the rule;
# this is the safety net, and it is not a substitute for reading that file.
.env
.env.*
# Editor and OS cruft
.DS_Store
.vscode/
.idea/
*.swp
*~

289
README.md Normal file
View File

@ -0,0 +1,289 @@
# Project Template
A working documentation skeleton, the tracker convention that goes with it, and
twenty scripts that enforce both. Copy `docs/` into a new project, replace the
angle-bracket placeholders, delete the paragraphs that do not apply — then
create the labels and the first milestone in the repository's issue tracker.
```
Status: Current
Owner: Kaspa
Last reviewed: 2026-08-12
Governs: Projects/Template/**
Review trigger: Any change to the tracker conventions or the QA run-state
grammar in the Privacy LLC Command Center's src/lib/; any
change to which project fields the agent API can write, or to
what reconcile reads; any script added to or removed from
docs/architecture/scripts/
```
## Why this exists
The Command Center at [privacyllc.dev](https://privacyllc.dev) reports on a
repository: how much is done, what the QA verdict is, and what the next
milestone entails. It can only do that for repositories that keep those things
somewhere it knows to look.
Eight repositories were surveyed before this was rewritten. Exactly one kept the
markdown batch ledger the previous version of this template asked for. Three had
invented their own dialects; one had pushed nothing but a README. Meanwhile,
with no convention at all, those same repositories had accumulated 69, 54 and
205 issues — because an issue tracker has structure built into it and a
convention has only whoever remembers it.
## The split
| Kind of truth | Lives in | Why |
| --- | --- | --- |
| Narrative documents — vision, trust map, QA playbook, QA verdict, security | git, as markdown | prose with an argument, versioned with the code, readable at any sha |
| Work items — batches, tasks, defects, blockers | the issue tracker | built for concurrent uncoordinated writers, atomic state, `closes #N` from a commit, real timestamps |
Nothing belongs in both. A to-do list in markdown and a tracker with the same
items is two records that will disagree, and neither will say which is right.
## What the Command Center actually reads
| Source | What is read |
| --- | --- |
| The repository's issue tracker | issue counts excluding pull requests, milestones with their own counters, and issues carrying the four severity labels |
| `docs/qa/ClaudeReport.md` | the three run-state lines — round, build SHA, and the overall sentence, quoted verbatim |
| `docs/data/img/` | `icon.webp`, `logo.webp`, `banner.webp` — fetched and rendered on the project page |
Everything else in this tree is here because a project needs it, not because
something parses it. Rename, split or extend those freely.
A repository that has not adopted the labels is reported as *not adopted*
rather than as zero defects, and one whose tracker is switched off is reported
as switched off rather than as an empty backlog. Absence is never rendered as a
measurement.
**The files in that table are read at a commit, and the commit is remembered.**
When the SHA they were read from falls behind the repository's newest commit,
the docs report is marked *stale* — which is the accurate description of a
document that described the code at a commit nobody is running any more. It is
also what an unpushed correction looks like from outside. Documentation edits
therefore ride in the same commit as the change that caused them, and reach the
site on the same push. See `docs/WORK_CYCLE.md`.
## What updates itself, and what does not
**The numbers on the project screen come out of this repository's tracker.** The
percentage is *defined* rather than picked: closed issues over all issues, asked
with `type=issues` so pull requests are excluded and `state=all` rather than the
API's default of open. Beside it sits a second figure counted from the
repository's **own milestones**, closed over all — this deployment keeps
milestones in the tracker and never mirrors them into the site's table, so that
is where the milestone reading comes from.
| On the project screen | Source | How it gets there |
| --- | --- | --- |
| Headline % — issues closed / all issues | the repository's tracker | automatic — on reconcile, and immediately on a webhook delivery |
| Second figure — milestones closed / all milestones | the repository's **milestones** | same read, same moment |
| Milestone coverage | how many issues sit inside a milestone at all | same read |
| QA verdict — round, build SHA, overall sentence | `docs/qa/ClaudeReport.md` | push the repository, then reconcile |
| Current summary / Next action | manual fields on the project | `PATCH /agent/projects/<slug>` — nothing else writes them |
| The Milestones card, and a typed weighted plan | the Command Center's **own** milestone table | admin UI only |
| Blockers | the Command Center's **own** blockers table | admin UI only |
So closing an issue and closing a milestone each move a number a stakeholder
can see, the same day, with no further step. That is the whole reason the
convention insists on both.
The last two rows are the ones that surprise people, and they are lists rather
than figures. The Milestones card reads the site's own table, which is empty on
this deployment by design — a repository can carry twenty-nine milestones,
have them counted in the figure above, and still show *"No milestones have been
added yet"* on that card. It is not a sync failure and reconciling will not
change it.
Blockers have a sharper version of the same split. The agent API can *narrate* a
blocker in a check-in's `blockers` field, and it has no route that creates or
resolves one in the table. A run that posts the check-in and reports "blocker
filed" has told the truth about the note and a falsehood about the record.
**An issue in no milestone is counted in the headline and invisible to the
milestone figure.** The site measures that gap on purpose — it is the reason
`Verify:` lines and milestones are per-issue requirements here rather than
housekeeping. Portfolio-wide when this was written, 272 of 650 issues belonged
to no milestone at all, including every one of the 205 in the repository that
had never adopted the convention.
## The digest is the part somebody actually reads
A worker polls the site every sixty seconds and sends a project digest to
Discord. It is where this convention stops being bookkeeping: the digest's
per-project figures come from **issues opened and closed** and from pushes, and
its next-step line comes, by rule, from **`nextAction` or the live milestone's
open issues** — never from a guess.
Three consequences, each the direct form of a step in the cycle below:
- **Work with no issue opened or closed produces a push and nothing else.** The
project reads as quiet on a day it was not.
- **A project with no `nextAction` and no open milestone gives the digest
nothing to say about what happens next**, and the digest will not invent one.
- **An opened issue labelled `P0` raises an immediate alert** — but only from a
repository whose webhook is registered. Without one, the site sees the
repository through the periodic sweep alone, and a P0 filed at nine is news at
the next reconcile rather than at nine.
## After each piece of work
`docs/WORK_CYCLE.md` is the seven-step version, with the argument for each step.
The short form:
**close what you finished, with the evidence · file what you found · close the
milestone if the batch landed · update the documents the change triggered, in
the same commit · push · log the entry, with Next action and Blockers · then
reconcile and write the summary and next action.**
The last two steps are the ones that get skipped, and the only ones nobody can
do for you later — by the time somebody notices the project screen is a week
stale, the person who knew what came next has forgotten.
Where each of those lives is deliberate and worth reading once: the next action
has a live copy (the project field) and a dated copy (the newest log entry) and
they are different kinds of fact; a blocker is an issue in the tracker first,
and a row on the site only when a human puts it there.
## The tracker convention
### Where the tracker is
**[dream.scheller.ltd](https://dream.scheller.ltd)** — a Forgejo instance, and
naming it is not a detail. The convention below is portable: milestones, issues
and labels exist on GitHub, GitLab and Jira, and every rule in it would work
there unchanged. **The reporting is not portable at all.** Every figure the
Command Center shows — the headline percentage, the milestone count, milestone
coverage, the alert an opened `P0` raises — is read from a Forgejo repository
mapped to the project. A project that follows every rule here in a different
tracker satisfies the convention completely and appears on privacyllc.dev as
having no tracker at all.
Credentials are in **`~/.openclaw/docker-registry.env`**, which carries
`FORGEJO_REGISTRY`, `FORGEJO_REGISTRY_USER` and `FORGEJO_REGISTRY_TOKEN`:
```bash
set -a; . ~/.openclaw/docker-registry.env; set +a
python3 docs/architecture/scripts/forgejo-issue.py list
```
Despite its name that token is not registry-scoped — it works across the whole
API, with push and pull. Never print it into a log or a chat.
Two things that cost an hour each otherwise. **Cloudflare fronts the instance
and 1010-blocks clients that do not look like a browser or curl** — Python's
`urllib` among them — so every request needs `User-Agent: curl/8.5.0`;
`forgejo-issue.py` already sends it and anything new must too, or the failure
arrives as a Cloudflare HTML page that looks nothing like a Forgejo error. And
**`/issues` returns pull requests too** unless `type=issues` is passed, which is
why the counts here exclude them and a hand-rolled query disagrees by exactly
the number of open PRs.
`forgejo-issue.py` takes the repository from this checkout's git remote, so it
cannot file into another project's tracker by accident. `--repo owner/name`
overrides that deliberately, and `--dry-run` prints the payloads and changes
nothing.
> This block names one instance and one path on the machine that runs it. If
> this repository will be public, replace both with your own before the first
> push.
### The rules
- **Milestone = batch.** Name it `0.2 Payments`. The description says what the
batch is for and how anybody will know it landed. A due date means a
commitment — leave it empty rather than inventing one.
- **Issue = deliverable.** One per item, in its milestone, ending with a
`Verify:` line stating the acceptance check.
- **Labels:** `P0` ships broken or loses data · `P1` materially wrong but
shippable · `P2` cosmetic or low impact · `release-blocker` means a release
built today would be wrong rather than merely incomplete. Exactly these
names — they are queried by name.
- **Defects are issues** with a severity label. The QA *verdict* stays in
`ClaudeReport.md`; the counts come from the tracker.
- **Close with `closes #N`** in the commit that does the work, so the record
comes from the thing that happened rather than from a date typed afterwards.
Where no single commit finished it, close it by hand with the evidence — a
path, a symbol, a test name, or the command that proves it. "Done" is not a
close, because a close with nothing in it cannot be reopened with confidence.
- **Close the milestone when the batch lands.** Closing the last issue under it
does not, and a milestone that is complete but open reads as a batch still in
progress. It is how a batch becomes visible as *shipped* rather than merely
finished — and it is counted: milestones closed over milestones total is one
of the two figures the project screen shows.
- **Every issue belongs to a milestone.** An issue filed outside one still
counts against the headline percentage and is invisible to the milestone
figure, so the two readings describe different amounts of work. The site
measures that gap rather than ignoring it.
## The rule that keeps the numbers honest
**Do not pad the tracker.** Every open issue is a denominator. If the real
answer is "one milestone, three issues", file exactly that. Invented future work
makes every percentage drawn from the tracker wrong, permanently and in the same
direction.
The same arithmetic runs the other way, and this half is the one that actually
happens: **an issue left open after the work is done understates the project for
as long as it stays open**, and the understatement compounds. A fortnight of
finished work with unclosed issues reads, from outside, as a fortnight of no
progress at all.
## What comes with it, besides the documents
`docs/architecture/scripts/` holds working code, not examples. Each takes its
configuration from the environment and hard-codes nothing about any deployment,
so a script copied from here cannot quietly point at the project it came from.
`docs/architecture/README.md` has the full table; the ones worth knowing about
on day one:
| | |
| --- | --- |
| `release.sh` | version bump, guards, build, verify, push. Refuses to build on a half-run test suite or a malformed public origin — both of which had shipped in every release of the project this came from. |
| `verify.sh` | every check the project has, in one command, with one table saying which passed, which failed, and which did not run |
| `check-env.sh` | which variables are set and which are missing, before anything reads them |
| `secrets.sh` | credential shapes in a staged diff, using the project's own patterns where it has written them down |
| `prove-guard.sh` | breaks what a guard protects and requires the guard to go red, then restores the file from a trap. `GUARDS.md` §1 as a command — the rule that a guard nobody has seen fail is not yet evidence. |
| `doc-claims.sh` | every file a document names must exist — and `--covers`, whether every file that exists is named. The second is the one that catches a list quietly missing rows. |
| `duplication.py` `dead-code.py` | code that exists twice; exports nothing imports and assets nothing renders |
| `forgejo-issue.py` | files and closes issues in the convention below, with every rule of the convention above as a check the script refuses to break |
| `deploy.py` | updates a running stack to a published image. Publishing and deploying are separate decisions; `release.sh` makes the first and this makes the second. |
| `release-notes.mjs` | tags the release and writes the notes: sections from the commit types the hook enforces, the image and its digest, and the batches the release finished. Set `RELEASE_DEPLOY_NOTE` to say how this project deploys, or empty where publishing and deploying are one act. |
`docs/architecture/githooks/` holds three hooks — typecheck and tests before a
commit, a conventional type in the message, and a push afterwards so a guarded
commit does not sit unpushed. They live in the repository rather than in
`.git/hooks`, which is not versioned; see their README for the one command that
points git at them.
**`docs/architecture/GUARDS.md` is worth reading before you write a check.**
Every rule in it was learned from a guard that had been green for months over
something broken, and the first — prove a guard fails before believing it
passes — is the one that costs thirty seconds and catches the rest.
## Adopting it
1. `cp -r Projects/Template/docs <your-project>/docs`
2. Fill in the status header on each file — especially **Review trigger**, which
is the line that keeps a document from going quietly stale.
3. Create the four labels, one milestone, and an issue per deliverable you can
actually see ahead of you — in the tracker named above, not whichever one the
host happens to offer.
4. Point git at the hooks: `cp docs/architecture/githooks/{pre-commit,commit-msg,post-commit} .githooks/ && chmod +x .githooks/* && git config core.hooksPath .githooks`. It is per-clone, so every checkout runs it once.
5. Run `bash docs/architecture/scripts/check-env.sh` and `bash docs/architecture/scripts/secrets.sh --tracked` — the first says what is not configured, the second is the one-time audit for what is already committed.
6. Map the repository on the project's edit screen in the Command Center, then
press **Pull**. One thing does not follow from mapping it and is silent when
missing: **register the repository's webhook** and confirm a delivery
arrives. Without one the site sees this repository only through the periodic
sweep — the counts still arrive, a day late, and an opened `P0` raises no
alert at all.
7. Skim `docs/TOOLS.md` — the fixed path every project has, saying which
scripts can stop you and where to start in a fresh clone. Then read
`docs/WORK_CYCLE.md` once, now rather than at the end of the first
session. It is what happens every time a piece of work finishes, and the two
steps it exists for — recording the next action and the blockers, and telling
the site what changed — are the two that are invisible when they are skipped.
There is also a generated prompt for doing all of this with an agent, on the
Docs template screen of the Command Center — editable before you copy it.

8
Template.code-workspace Normal file
View File

@ -0,0 +1,8 @@
{
"folders": [
{
"path": "."
}
],
"settings": {}
}

140
docs/DOC_TRUST_MAP.md Normal file
View File

@ -0,0 +1,140 @@
# Doc Trust Map — which document to believe
```
Status: Current
Owner: <who maintains this>
Last reviewed: <YYYY-MM-DD>
Governs: docs/**
Review trigger: Any doc added, deleted or moved; any change to which doc owns a subject
```
> Write this file **last**, once the rest exists, and describe what is actually
> here rather than what a template said should be. Its whole value is being
> accurate about the others.
## Work items are not in this tree
The sequence of work, the open defects, and the things blocking a release all
live in this repository's **issue tracker**, not in `docs/`. Milestones are
batches; issues are deliverables; severity labels are `P0`, `P1`, `P2` and
`release-blocker`.
This section exists to stop the next contributor starting a fresh markdown
to-do list. A list of things to do in two places is two records that will
disagree, and nothing will say which one is right.
| Question | Answer lives in |
| --- | --- |
| What are we building, and for whom? | `docs/planning/PROJECT_PLAN.md` |
| What is the sequence of work? | milestones in the tracker |
| What is left in this batch? | open issues under that milestone |
| What is broken right now? | issues labelled `P0` / `P1` / `P2` |
| What makes a release wrong? | issues labelled `release-blocker` |
| What is the next action? | the `nextAction` field on the project at privacyllc.dev — the newest entry in `docs/history/DEVELOPMENT_LOG.md` says what it was *then* |
| What is blocking us? | the tracker, for the work; the blockers table at privacyllc.dev, for the stakeholder-facing version |
| What do I do when a piece of work is finished? | `docs/WORK_CYCLE.md` |
| What happened, and when? | `docs/history/DEVELOPMENT_LOG.md` |
| Did QA pass, and what does the tester think? | `docs/qa/ClaudeReport.md` |
| What did QA actually reach? | `docs/qa/ClaudeQACoverage.md` |
| How is it built? | `docs/architecture/` |
| How do I release, back up, or check this? | `docs/architecture/scripts/` |
| Which script do I run, and can it stop me? | `docs/TOOLS.md` — the signpost; `docs/architecture/README.md` has the table |
| What runs before a commit? | `docs/architecture/githooks/` |
| How do I write a check that will actually catch something? | `docs/architecture/GUARDS.md` |
**Next action and blockers are recorded at the end of every piece of work, not
when somebody asks.** `docs/WORK_CYCLE.md` holds that procedure and the reason
each step is in it — including which of these channels updates itself and which
two only ever change because a person changed them.
## Folder layout
| Folder | Contents |
| --- | --- |
| `docs/planning/` | `PROJECT_PLAN` — the vision. Not the schedule; that is the tracker. |
| `docs/qa/` | `ClaudeQAPlan` (playbook), `ClaudeQACoverage` (what each pass reached), `ClaudeReport` (the verdict) |
| `docs/architecture/` | How it is built: module boundaries, data shapes, reference manuals |
| `docs/design/` | What it should feel like: product decisions, UI plans, copy |
| `docs/security/` | `SECURITY` — threat model, secret handling, what is out of scope |
| `docs/history/` | `DEVELOPMENT_LOG` (dated, append-only), `HISTORY` (the narrative), `BATCH_LEDGER` (archived, superseded by the tracker) |
| `docs/data/` | the three branding marks privacyllc.dev renders for this project |
| `docs/` root | this map; `WORK_CYCLE` — what to do when a piece of work is finished; `TOOLS` — where the scripts are and which can stop you |
`README.md` stays at the repository root; it is the landing page and moving it
breaks that. Everything else lives under `docs/`.
## Source-of-truth ladder
When two sources disagree, believe them in this order:
1. **Verified code behaviour** — read the source, run the tests
2. **The issue tracker** — for anything about state of work: what is open, what
closed, when, and by which commit
3. **Specialist docs** — authoritative for their own subject only
4. **History and logs** — a record of *then*, not a description of *now*
A document that describes work as done is describing what was true when it was
written. The tracker is describing now.
## The status header
Every document in this tree opens with one, immediately after its H1:
```
Status: Current | Draft | Superseded | Archived
Owner: <who maintains this>
Last reviewed: <YYYY-MM-DD>
Governs: <paths or subject this document is authoritative for>
Review trigger: <the change that should send someone back to this file>
```
**Exactly those four status words.** They are not a suggestion: a checker reads
them, and a document whose status falls outside the list is reported as having
an unknown one rather than being quietly accepted. `Historical` is the word this
convention used to have and deliberately dropped — `Archived` covers it, and a
vocabulary a script enforces beats one described in prose, which drifts the
moment somebody writes something adjacent to it. `Draft` earns its place by
naming a real state the old list had no word for: a document being written,
which is neither current nor superseded.
**Review trigger** is the line that matters. "Last reviewed" ages on its own and
a reader cannot tell a current document from an abandoned one by looking at it;
a trigger names the event that should bring somebody back, which is what stops a
document becoming confidently wrong. It is also checked: the header is read from
the first sixteen lines of the file, and one carrying `Status` without
`Review trigger` is reported as incomplete — that combination looks finished and
is not.
## Declaring a document deliberately absent
A repository may decide it will not keep one of these documents in git — QA and
security notes held outside the repository on purpose, say. Say so **here**, one
line per path, anywhere in this file:
```
Exempt: <the path> — <why, in a few words>
```
**Write the real path only when you mean it.** Outside a fenced block, an
`Exempt:` line naming a real document is not an example — it is a declaration,
and the checker will report that document as deliberately absent in every
repository that copied the file. Keep illustrations fenced, and use placeholders
anyway, as the form above does.
Fenced blocks are skipped, but only since 2026-08-12. Before that the parser
read every line, which meant that documenting this feature *used* it: the
example a writer naturally reaches for is a real required path, because an
illustration with a fake one teaches nothing. A draft of this very section
carried `Exempt: docs/security/SECURITY.md` inside a fence and would have marked
that document deliberately absent everywhere it was copied. It is recorded here
rather than quietly fixed because the shape recurs — a parser that cannot tell a
description of a thing from the thing itself.
The declaration lives in this file because the decision does. Holding the list
somewhere else would be two records of one fact, and the other one would never
hear that this repository changed its mind.
**`docs/data/` and `docs/data/img/` cannot be exempted.** A declaration naming
either is refused rather than honoured, because the Command Center renders what
is in them — an exemption would produce a project card with no icon and nothing
explaining why, which is the failure the check exists to catch.

75
docs/TOOLS.md Normal file
View File

@ -0,0 +1,75 @@
# Tools — where the scripts are, and which ones can stop you
```
Status: Current
Owner: <who maintains this>
Last reviewed: <YYYY-MM-DD>
Governs: docs/architecture/scripts/**, docs/architecture/githooks/**
Review trigger: Any script added to, removed from or repurposed in
docs/architecture/scripts/; any change to which of them gates
```
> A signpost, deliberately. Every project that adopts this template has a
> `docs/TOOLS.md`, so "read `docs/TOOLS.md` first" is an instruction that works
> without knowing anything about the project — which is the whole reason this
> file exists at a fixed path.
## The list is not here
**[`architecture/README.md`](architecture/README.md)** holds the table of what
ships and what each script is. That is the one copy.
A second table here would be two records of one fact, and the other one would
never hear that a script was renamed — the failure `DOC_TRUST_MAP.md` exists to
prevent, applied to the tooling instead of the documents. So this file answers
the questions that table does not, and points at it for everything else.
## If this project has fewer scripts than that table lists
That is the intended state, not a broken copy. `scaffold.sh` writes the
documents and deliberately leaves the scripts behind — *"an unconfigured
`release.sh` landing in every new repository is a loaded gun, not a head
start"* — so they are taken from the template one at a time, having been read.
Which makes the table a menu rather than an inventory here: it says what exists
to be copied. Once a script is in this project, it is this project's, and its
row in `architecture/README.md` should say what it does **here** if that has
drifted from the template's version.
## Which ones can stop you
Not in a table, because the honest answer lives in each script's own header and
would go stale here. The rule that matters:
**Exit code `2` is never a pass.** These scripts distinguish "the check ran and
found nothing" from "the check did not run", because those look identical from
the outside and only one of them is evidence. A CI step or a hook that treats a
`2` as success has quietly turned the check off. Each script states its codes at
the top; read them there.
The hooks are the other place work gets stopped:
[`architecture/githooks/`](architecture/githooks/README.md) has the one install
command and the table of what each hook runs.
## Where to start in a fresh clone
1. `bash docs/architecture/scripts/check-env.sh` — what is configured and what
is missing, before anything reads it.
2. `bash docs/architecture/scripts/secrets.sh --tracked` — the one-time audit of
what is already committed. The staged-diff mode is for the hook; this mode is
for the day you adopt the template.
3. [`architecture/GUARDS.md`](architecture/GUARDS.md) — how to write a check that
can actually fail, before you write one. `scripts/prove-guard.sh` performs its
first rule.
## Adding one
Put it in `docs/architecture/scripts/`, give it a header saying what it does and
**which incident motivated it**, state its exit codes, and add a row to
`architecture/README.md`'s table — this file's `Review trigger` fires on exactly
that.
The bar, from the scripts that are already here: **done by hand three times, or
once with a consequence.** A script written before either of those has no failure
to describe in its header, which is the part that stops the next person deleting
it.

262
docs/WORK_CYCLE.md Normal file
View File

@ -0,0 +1,262 @@
# The work cycle — what happens at the end of a piece of work
```
Status: Current
Owner: <who maintains this>
Last reviewed: <YYYY-MM-DD>
Governs: what must be true before a piece of work counts as finished
Review trigger: Any change to what the Command Center reads, or to which of
those channels a person rather than an agent has to write
```
> **This is a procedure, not a status board.** Nothing here records what is open,
> what is next, or what is blocked — those live in the tracker and on
> privacyllc.dev, and a copy of them in this file would be the second disagreeing
> record that `DOC_TRUST_MAP.md` exists to prevent.
## Why this file exists
A piece of work ends in more than one place. The code is committed; the issue
that asked for it is still open; the document the change contradicts still says
the old thing; and the project screen at
[privacyllc.dev](https://privacyllc.dev) still shows last week's next action to
whoever opens it.
None of those catch up on their own. Two of them cannot be caught up later by
anybody but the person who did the work, because by then nobody knows what the
next action was meant to be.
## The cycle
Run all of it, in this order, every time. It is short on purpose.
1. **Close what you finished.** `closes #N` in the commit that does the work, so
the record comes from the thing that happened rather than a date typed
afterwards. If no single commit finished it, close it by hand with the
evidence — a path, a symbol, a test name, or the command that proves it.
*"Done" is not a close.*
2. **File what you found.** A defect noticed on the way past is an issue with a
severity label and the build SHA it was seen at, not a memory. Filing it costs
a minute; the alternative is finding it again from scratch, or shipping it.
3. **Close the milestone if the batch landed.** A milestone with every issue
closed and itself still open reads as a batch still in progress — see
[Open and closed are not bookkeeping](#open-and-closed-are-not-bookkeeping).
4. **Update the documents this change triggered.** Read the `Review trigger`
lines: a new module, a changed data shape, a new migration, a new boundary
something crosses. Those edits go in **the same commit as the code**, for the
reason in [Docs travel with the push](#docs-travel-with-the-push).
5. **Push.** Nothing above is visible off this machine until you do, and step 7
reports on what was pushed.
6. **Write the log entry** in `docs/history/DEVELOPMENT_LOG.md`: what changed,
what it proved, **Next action**, and **Blockers**. Dated, append-only, newest
first.
7. **Tell the Command Center**, which is three calls: reconcile, so it re-reads
the tracker and the pushed docs; `PATCH` the project's summary and next
action, which nothing else writes; and a check-in if what changed is worth a
timestamped note. Commands are [at the bottom](#the-commands).
Steps 6 and 7 are the two that get skipped, and they are the two nobody else can
do afterwards.
## Next action — where it lives
| Copy | Where | What it is |
| --- | --- | --- |
| The current one | `nextAction` on the project, privacyllc.dev | the live answer to "what happens next", shown to whoever opens the project screen |
| The dated one | the newest entry in `DEVELOPMENT_LOG.md` | what the next action was **at that point** — history, not status |
These are not two records of the same thing, and the distinction is worth
holding on to. The field is overwritten every time and always describes now. The
log entry is never edited and describes a moment — which is what makes it safe
to keep, and why an old entry naming a next action that has since been done is
not stale, it is a receipt.
Write the next action as an **action**: the thing a person would start on
Monday, specific enough to begin without asking a question. "Continue the work"
is not one. If the honest answer is that you do not know, that is a real answer —
say what has to be decided and by whom.
`currentSummary`, `nextAction` and `description` are **write-only**: they are
deliberately absent from every API response, because free text can name a
customer or an unannounced product. Only the admin screen shows them back. Never
report them as empty because a `GET` did not return them.
## Blockers — where they live
A blocker is recorded in up to three places, and **an agent can write only the
first and the third**:
| Where | What it holds | Who writes it |
| --- | --- | --- |
| The tracker | the work itself — an issue labelled `release-blocker`, or `P0` when it ships broken | anyone, including an agent |
| The blockers table on privacyllc.dev | the business-facing blocker, with a severity, an owner and a resolution note | **a human, in the admin UI** |
| A check-in's `blockers` field | narrative: what is stuck, said in a timestamped note | anyone, including an agent |
The agent API has no route that creates, edits or resolves a blocker on the
site. Posting a check-in whose `blockers` field says something is stuck records
*narrative* — it does not touch the blockers table, and it does not clear
anything. An agent that reports "blocker filed" after a check-in has told the
truth about the note and a falsehood about the table.
So: **file the issue** — that is the copy the work is actually tracked in — and
when it belongs in front of a stakeholder, say so plainly and let a human enter
it. Before treating a site blocker as resolved, restate its title and status and
get explicit confirmation; the admin UI requires a resolution note that the API
cannot supply.
If the work is blocked and nothing is filed anywhere, the project simply looks
slow.
## Open and closed are not bookkeeping
Every open issue is a denominator, and not in the abstract: the percentage on
the project screen **is** closed issues over all issues in this tracker. Nothing
else produces it. That has three consequences worth stating in full:
- **An issue left open after the work is done** understates the project
permanently, and the understatement compounds — a fortnight of finished work
with unclosed issues reads as a fortnight of no progress.
- **An issue closed without evidence** cannot be reopened with confidence,
because nothing in it says what "fixed" meant. That is why the close comment
carries the path, symbol, test or command.
- **Invented future work** makes every percentage wrong, permanently and in one
direction. Do not pad the tracker. If the real answer is one milestone and
three issues, file exactly that.
Milestones are the same argument at batch scale, and they are counted the same
way: milestones closed over milestones total is the second figure on the project
screen. Closing the last issue under a milestone does not close the milestone,
so a tracker full of complete-but-open batches reports a project as less
finished than it is — and cannot answer "what shipped".
**File every issue into a milestone.** One filed outside still counts against
the headline percentage while being invisible to the milestone figure, which is
how two readings of the same project come to describe different amounts of work.
The site measures the gap rather than ignoring it.
One trap that costs an afternoon, and it is about a *card* rather than a figure:
the Milestones list on privacyllc.dev reads the Command Center's own table,
which only an admin can write. A repository whose milestones are being counted
in the figure above can still show *"No milestones have been added yet"* in that
list. It is not a sync failure and no amount of reconciling changes it.
## Docs travel with the push
The Command Center reads this repository's documents at a commit, and stores the
SHA it read them from. When that SHA falls behind the repository's newest
commit, the docs report is marked **stale** — not wrong, not missing, *stale*,
which is the honest description of a document that was accurate at a commit
nobody is running any more.
Two ways to produce it, and both are ordinary carelessness rather than bad luck:
- **Code pushed, documents not updated.** The report is recomputed at the new
SHA against prose describing the old behaviour. Nothing flags this; the
document is simply confidently wrong now, and its `Last reviewed` line still
looks recent.
- **Documents updated, not pushed.** The site keeps reporting the old ones. A
correction that exists only on your machine has not been made.
Hence step 4's insistence that documentation edits ride in the same commit as
the change that caused them. It is not tidiness — a doc commit that comes "after
this next thing" is the commit that does not get written, and the manual's
migration table in the project this template came from sat six migrations behind
before anybody noticed. Every reader in between trusted it.
## What the Command Center reads, and who writes it
Four independent channels, and **only two of them are automatic**:
| On the project screen | Source | How it gets there |
| --- | --- | --- |
| Headline % — issues closed / all issues | the repository's tracker | automatic — on reconcile, and immediately on a webhook delivery |
| Second figure — milestones closed / all milestones | the repository's **milestones** | same read, same moment |
| Milestone coverage — issues that sit in no milestone | the repository's tracker | same read |
| QA verdict — round, build SHA, overall sentence | `docs/qa/ClaudeReport.md` | push the repository, then reconcile |
| Current summary / Next action | manual fields on the project | `PATCH /agent/projects/<slug>` — nothing else writes them |
| The Milestones card, and a typed weighted plan | the Command Center's **own** milestone table | admin UI only |
| Blockers | the Command Center's **own** blockers table | admin UI only |
The first three are why steps 1 and 3 are steps: **closing an issue and closing
a milestone each move a figure a stakeholder can see, that day.** The last two
are lists rather than figures, and they are the ones an agent cannot write — a
repository whose milestones are counted in the second figure can still show
*"No milestones have been added yet"* on the card, because that card reads a
table only an admin fills in.
A repository that has not adopted the four label names is reported as *not
adopted* rather than as zero defects, and one with its tracker switched off is
reported as switched off rather than as an empty backlog. Absence is never
rendered as a measurement — which is exactly why a real backlog nobody filed
looks like nothing at all.
## The commands
The tracker is the Forgejo instance at
**[dream.scheller.ltd](https://dream.scheller.ltd)**, and steps 1 to 3 happen
there. Its credentials, the Cloudflare trap and the reason it has to be that
instance rather than any tracker are in the project README's *Where the tracker
is* — one copy, named once:
```bash
set -a; . ~/.openclaw/docker-registry.env; set +a
python3 docs/architecture/scripts/forgejo-issue.py list
python3 docs/architecture/scripts/forgejo-issue.py close 42 "Fixed in a1b2c3d; tests/foo.test.js covers it."
```
Step 7 talks to the Command Center instead:
```bash
# The token lives in this machine's credential store, never in the repository.
TOKEN="$PRIVACY_LLC_TOKEN"
BASE="https://privacyllc.dev/api/internal/v1"
AUTH="Authorization: Bearer $TOKEN"
```
**Reconcile** — re-read the tracker and the pushed documents. Do this after the
push, not before:
```bash
curl -sk -X POST -H "$AUTH" -H "Content-Type: application/json" \
-d '{"kind":"reconcile_all"}' "$BASE/agent/jobs"
```
**Summary and next action** — read the project first for its `version`, then
send a flat body carrying that version. A stale version is rejected rather than
silently overwriting somebody else's edit:
```bash
curl -sk -H "$AUTH" "$BASE/agent/projects/<slug>" # for version + state
curl -sk -X PATCH -H "$AUTH" -H "Content-Type: application/json" \
-d '{"version":<n>,"health":"on_track",
"currentSummary":"<where the project stands, in a sentence or two>",
"nextAction":"<the next concrete thing, specific enough to start>"}' \
"$BASE/agent/projects/<slug>"
```
**A check-in** — a timestamped "what changed", when there is something real to
report. `summary` is required and must say something; the API refuses an empty
"no change" update, deliberately:
```bash
curl -sk -X POST -H "$AUTH" -H "Content-Type: application/json" \
-d '{"projectId":"<slug>","health":"on_track",
"summary":"<what changed>","accomplishments":"<what landed>",
"blockers":"<what is stuck, or omit>","nextActions":"<what is next>"}' \
"$BASE/agent/updates"
```
Health is one of `on_track`, `caution`, `off_track`, `unknown`. Never invent
one, and never report progress the API did not return.
## What this file is not
- **Not the work list.** That is the tracker: milestones are batches, issues are
deliverables.
- **Not the release procedure.** That is `docs/architecture/scripts/release.sh`
and `docs/security/SECURITY_CHECKLIST.md`.
- **Not the QA procedure.** That is `docs/qa/ClaudeQAPlan.md`, which ends in its
own version of step 7.
- **Not a place to record status.** If you are about to add "current state" or a
list of outstanding items below this line, the tracker is where it goes.

142
docs/architecture/GUARDS.md Normal file
View File

@ -0,0 +1,142 @@
# Guards — how to write a check that actually checks
```
Status: Current
Owner: <who maintains this>
Last reviewed: <YYYY-MM-DD>
Governs: structural tests, source-grep assertions, probes, and any check whose
passing is taken as evidence
Review trigger: A guard is found to have been passing while the thing it guards
was broken; a new class of check is added to the suite.
```
A guard that cannot fail is worse than no guard, because it is trusted. Every
rule here was learned by finding one that had been green for months over
something broken.
## 1. Prove the guard fails before you believe it passes
The one discipline that matters most, and it takes thirty seconds:
```bash
cp src/lib/thing.ts /tmp/thing.bak
# break exactly the thing the test protects
sed -i 's/if (body.error)/if (false)/' src/lib/thing.ts
npx vitest run tests/thing.test.ts # expect: exactly one failure
cp /tmp/thing.bak src/lib/thing.ts
npx vitest run tests/thing.test.ts # expect: green again
```
**Exactly one** is the part people skip. If breaking the guard's target fails
three tests, two of them are coincidental and will mask a real regression later.
If it fails none, the guard is decoration — and you have just learned that for
the price of one `sed`.
`scripts/prove-guard.sh` performs exactly this, which removes the two ways it
gets skipped: the restore is a `trap`, so an interrupted run cannot leave the
code broken, and the failure count comes from the runner's own summary rather
than from eyeballing red — one failing test is routinely reported on half a
dozen lines, and counting those calls a clean result six coincidental
failures.
Do this when you write a guard, and again when you change what it guards. A
test written alongside the code it tests has never been observed failing.
## 2. A source-grep guard must tell code from the comment about code
Structural tests that assert a file does *not* contain some pattern will match
the docblock explaining why that pattern is forbidden. So the clearest possible
comment breaks the test, and the obvious fix is to delete the explanation.
Strip comments first:
```ts
const codeOf = (path: string) =>
readFileSync(path, "utf8")
.split("\n")
.filter((line) => !/^\s*(\*|\/\/|\{\/\*)/.test(line))
.join("\n");
expect(codeOf("src/lib/thing.ts")).not.toContain("dangerouslySetInnerHTML");
```
Otherwise the guard quietly punishes documenting the rule it exists to enforce —
which is exactly backwards, because the comment is how the next person learns
the rule at all.
## 3. Pin the behaviour, not the spelling
A guard should fail when the protected behaviour breaks and stay quiet
otherwise. One that asserts on a variable name fails on a rename that changed
nothing.
```ts
// Brittle: breaks when the variable is renamed, while the fallback it protects
// is untouched.
expect(route).toContain("readAsset(project.forgejoRepo");
// Pins the behaviour: the route fetches through the wrapper that tries both
// spellings, and never through the raw reader.
expect(route).toMatch(/readAsset\(\s*\w+,\s*ASSETS\[which\]\s*\)/);
expect(body).not.toContain("readFileBytes(");
```
A guard that fails on changes it does not care about is one people learn to edit
rather than heed, and the edit is usually deletion.
## 4. A negative result is only as good as the probe that produced it
"The check found nothing" and "the check did not run" are different facts, and
they look identical from the outside. Before reporting an absence, prove the
instrument worked:
```bash
# Not this alone — an unreadable file produces the same silence as an unset key
grep -c '^WANTED=' /proc/$PID/environ
# Establish the read succeeded first
tr '\0' '\n' < /proc/$PID/environ | grep -c . # 0 here means "could not read"
```
This is the confident-absence failure one level up: the same trap as a screen
rendering a failed query as a count of zero, applied to your own diagnosis.
## 5. A guard that is often wrong is worse than none
A check with a high false-positive rate trains everybody to skip its output,
including on the day it is right.
One written for this template flagged **684 of 1142** candidates on its first
run. That was not 684 findings, it was a broken heuristic — and shipping it
would have taught its readers that the check is noise. Two rounds of narrowing
brought it to 17 of 363, all of them real.
If a new guard's first run is loud, tune it until it is quiet before anybody
relies on it. Report the false-positive rate you settled at, so the next person
knows what silence is worth.
## 6. Guards belong before the artifact exists
A check that runs after publication catches the problem once it is somewhere it
cannot be taken back from: the tag is in the registry, and refusing the commit
afterwards leaves git with no record of it.
Order the gates so the expensive, irreversible step is last — preconditions,
guards, build, verify the built thing is what was asked for, publish, and record
it last of all.
## 7. When the gate finds something that invalidates the operation, stop
Printing a warning and continuing produces the worst outcome available: the bad
thing happens *and* a reassuring summary appears above it.
The question is not how bad the finding is. It is **whether it invalidates what
the operation claims**:
- A release whose test gate skipped half the suite — a release claims to be
tested. **Refuse.**
- A backup written to a group-readable directory — the backup is still a
backup. **Warn.**
Escape hatches are fine, and they have to be asked for by name, never be the
default, and say plainly what is being given up.

79
docs/architecture/README.md Normal file
View File

@ -0,0 +1,79 @@
# Architecture
```
Status: Current
Owner: <who maintains this>
Last reviewed: <YYYY-MM-DD>
Governs: docs/architecture/**
Review trigger: Any new module, any change to a module boundary or a data shape
```
## What belongs here
How the thing is built, for somebody who has to change it:
- **Module boundaries** — what each part owns, and what it is not allowed to
know about. The boundaries are the architecture; everything else is detail.
- **Data shapes** — the structures that outlive a single function, especially
anything persisted or sent over a wire.
- **Reference manuals** — the long documents that answer "how does X work"
without requiring a full read of X.
- **Decisions with consequences** — why this database, why this concurrency
model, why this dependency. Include the option that was rejected and what it
would have cost, because that is the part nobody can reconstruct later.
## Documents here
- **`GUARDS.md`** — how to write a check that actually checks. Read it before
adding a structural test or a probe; every rule in it was learned from a guard
that had been green over something broken.
## What ships in this folder
Working code, not just prose. Copy what a project needs and delete the rest —
these are a starting point with the arguments already made, not a framework.
This table is the one copy of that list. [`docs/TOOLS.md`](../TOOLS.md) is the
signpost every project is expected to have — it points here rather than
repeating it, and answers the two questions this table does not: which scripts
can stop you, and where to start in a fresh clone.
| Path | What it is |
| --- | --- |
| `scripts/release.sh` | version bump, guards, build, verify, push, prune. Refuses to build on a half-run test suite or a malformed public origin. |
| `scripts/verify.sh` | the repo's own checks, in one command |
| `scripts/check-env.sh` | which variables are set, which are missing, before anything reads them |
| `scripts/migrate.sh` | apply and report migrations, including the ones that run outside a transaction |
| `scripts/backup.sh` | a dump that is verified before it is trusted |
| `scripts/healthcheck.sh` | a liveness tick with the URL written down rather than re-derived each run |
| `scripts/status.sh` | what is deployed, and whether it matches this checkout |
| `scripts/dev.sh` | bring the local stack up |
| `scripts/scaffold.sh` | lay out a new project in this shape |
| `scripts/doc-triggers.py` | which documents a **pending** change fires, read from their `Governs:` headers. The `Review trigger` on each document names the change that should send somebody back to it; this is the check that asks before the commit rather than after |
| `scripts/prove-guard.sh` | breaks the thing a guard protects, requires the guard to go red, restores the file from a trap. `GUARDS.md` §1 written out as a command, including the count — one failing test reported on six lines is not six failures |
| `scripts/commit-mine.sh` | commits only the paths you name, by pathspec, after the secret scan. For a tree something else is also writing: what anyone else has staged is reported and left exactly as it was |
| `scripts/doc-claims.sh` | every file a document names must exist, and (`--covers`) every file that exists is named — the second is the one that catches a list missing rows |
| `scripts/duplication.py` | code that exists twice, tuned so what it reports is worth reading |
| `scripts/dead-code.py` | exports nothing imports, and assets nothing renders |
| `scripts/secrets.sh` | credential shapes in a staged diff, using the project's own patterns where it has them |
| `scripts/audit-gate.mjs` | high/critical advisories in **production** dependencies, with the allowlist npm does not have. An entry must say why the advisory cannot reach this app, what would make it reachable, and what retires the entry — three fields, so a waiver stays falsifiable. Exits 2 when nothing was checked. |
| `scripts/forgejo-issue.py` | file and close issues in the tracker convention, with every rule of it as a check |
| `scripts/deploy.py` | update the running stack to a published image. Publishing and deploying are separate; this is the second one. **The only copy** — it existed twice and drifted (#209); the `privacyllc-deploy` skill's is now a symlink to this file. Identity-free by design: it reads `DEPLOY_IMAGE`, `DEPLOY_STACK_ID`, `DEPLOY_CONTAINER` and `DEPLOY_SITE_URL` from the environment and refuses to run without them, so each project supplies its own via a wrapper. Never hard-code one here — least of all the site URL, which is frozen into the image at build time. |
| `scripts/release-notes.mjs` | tags the release and writes its notes, grouped by the commit types the message hook already enforces. Runs after `release.sh` has published, so a failure here cannot cost an image. Scrubs credential shapes out of commit subjects first — the body goes to a public repository. |
| `githooks/` | pre-commit, commit-msg, post-commit — see its README for the one install command |
Every script takes its configuration from the environment and hard-codes nothing
about any particular deployment. `check-env.sh` is the one to run first.
## What does not belong here
- Product intent — that is `docs/planning/PROJECT_PLAN.md`
- What it should feel like — that is `docs/design/`
- What happened while building it — that is a history log, not architecture
## A note on drift
Architecture docs go stale faster than any other kind, because code changes
under them silently. This is exactly what the **Review trigger** line is for:
name the change that should send somebody back here, and a reader can tell
whether the trigger has fired.

63
docs/architecture/githooks/README.md Normal file
View File

@ -0,0 +1,63 @@
# Git hooks
```
Status: Current
Owner: <who maintains this>
Last reviewed: <YYYY-MM-DD>
Governs: docs/architecture/githooks/** and the .githooks/ a project installs
Review trigger: A new guard the repository wants run before a commit; any change
to what a commit message must contain.
```
Three hooks, and the reason they live in the repository rather than in
`.git/hooks`: that directory is not versioned, so a hook living there protects
exactly one clone on exactly one machine. These are committed, and one command
points git at them.
## Install
```bash
mkdir -p .githooks
cp docs/architecture/githooks/{pre-commit,commit-msg,post-commit} .githooks/
chmod +x .githooks/*
git config core.hooksPath .githooks
```
`core.hooksPath` is per-clone, so every checkout runs that last line once. Say so
in the project README — an uninstalled hook fails silently, which is the same
class of problem the hooks exist to prevent.
## What each one does
| Hook | Guard |
| --- | --- |
| `pre-commit` | typecheck, then the test suite, when `.ts`/`.tsx` is staged |
| `commit-msg` | refuses a message with no conventional type |
| `post-commit` | pushes to the remote, so a guarded commit does not sit locally |
## Two settings worth making before you rely on them
**`pre-commit` will use a local test database if you tell it where one is.**
Set `TEST_DB_PORT` and `TEST_DB_URL` together. Leave them unset and the database
suites skip, which is the right default on a machine with no Postgres — but it
means the hook is running a fraction of the suite, so `release.sh` is
deliberately stricter and refuses rather than warns.
**`post-commit` pushes.** That is the intent — the commit that first added a
pre-commit hook to the reference project sat unpushed for a day, guarded and
invisible — but it is a surprise if you were not expecting it. Read it before
installing it on a repository with a protected branch.
It has a second consequence worth knowing about: the push is what the Command
Center reads, so **whatever documentation was not in that commit is now behind
the code by one push**. That is the mechanical reason `docs/WORK_CYCLE.md` asks
for doc edits in the same commit as the change rather than in a tidy-up
afterwards — with this hook installed, "I will document it next commit" means
the site has already published the version without it.
## Why a hook and not CI
Both. These are the guards that must run before the artifact exists: a hook that
fires after a push, or a CI job that fires after a publish, catches the problem
once it is already somewhere it cannot be taken back from. CI is the second
opinion; this is the one that runs first.

110
docs/architecture/githooks/commit-msg Executable file
View File

@ -0,0 +1,110 @@
#!/usr/bin/env bash
#
# Every commit says what kind of change it is, before it says anything else.
#
# ## Why
#
# Ninety-nine of this repository's commits already carry a conventional type —
# `feat:`, `fix:`, `docs:`, `chore:` — and ninety-six do not, including a run of
# recent ones written as bare sentences. That split is the problem: `git log
# --grep '^fix'` answers "what have we fixed" for half the history and quietly
# omits the other half, which is worse than having no convention at all, because
# the answer looks complete.
#
# So the type is required, and the vocabulary is closed. A closed list is the
# point — `feat`, `feature` and `feat!` as three spellings of one idea is how a
# convention stops being searchable.
#
# ## The vocabulary
#
# feat a new capability somebody can use
# fix a defect. The thing behaved wrongly and now does not
# ui appearance, layout, copy, or interaction, with no change in what
# the software knows or decides
# docs documentation only
# test tests only, with no change to what they test
# refactor same behaviour, different shape. If behaviour changed it is not this
# security hardening, boundaries, secret handling. Kept separate from `fix`
# on purpose: "what have we hardened" is a question worth being able
# to ask on its own, and it is the one an auditor asks first
# perf faster or lighter, same answers
# chore tooling, dependencies, releases. The bucket for work that is not
# about the product
#
# `harden`, `style` and `content` each appear once or twice in the history and
# are deliberately not here — they are `security`, `ui` and `docs` under other
# names, and a synonym is a hole in a closed list.
#
# ## Scope is optional, and lowercase
#
# fix(admin): ... the fifteen existing `admin` scopes, and `release`,
# `integrations`, `db` and the rest, all keep working.
#
# ## What it deliberately does not enforce
#
# Subject length. Thirty-five existing subjects run past 72 characters, several
# of them deliberately, and rejecting a commit for a well-written 80-character
# sentence would teach people to use `--no-verify` — which switches off the
# checks that actually matter. The type is the part a tool reads; the length is
# a matter of taste and stays that way.
#
# ## Escape hatch
#
# SKIP_GUARDS=1 git commit ... skips this too, and says so
#
# The same variable the pre-commit hook uses, because two switches for "I know
# what I am doing" is one more than anybody will remember.
set -uo pipefail
say() { printf '\033[1mcommit-msg:\033[0m %s\n' "$*" >&2; }
message_file="$1"
subject=$(head -1 "$message_file")
if [ -n "${SKIP_GUARDS:-}" ]; then
say "SKIP_GUARDS set — the commit type was NOT checked."
exit 0
fi
# Git writes these itself, or writes them on a human's behalf during a rebase.
# Rejecting them would break `git merge`, `git revert` and autosquash for a
# convention none of them ever agreed to.
case "$subject" in
"Merge "*|"Revert "*|"fixup!"*|"squash!"*|"amend!"*)
exit 0
;;
esac
# A comment-only file is an aborted commit; git handles that itself.
if [ -z "${subject// /}" ]; then
exit 0
fi
TYPES="feat|fix|ui|docs|test|refactor|security|perf|chore"
if printf '%s' "$subject" | grep -qE "^(${TYPES})(\([a-z0-9._-]+\))?!?: .+"; then
exit 0
fi
say "the subject line needs a type."
say ""
say " got: ${subject}"
say ""
say " expected: <type>: <subject> e.g. fix: stop counting milestones as records"
say " <type>(<scope>): <subject> ui(admin): mark cloud models on the picker"
say ""
say " types: feat a new capability"
say " fix a defect, now not"
say " ui appearance, layout, copy — no change to what it decides"
say " docs documentation only"
say " test tests only"
say " refactor same behaviour, different shape"
say " security hardening, boundaries, secrets"
say " perf faster or lighter, same answers"
say " chore tooling, dependencies, releases"
say ""
say " Your message is kept. Run 'git commit' again to edit it, or"
say " SKIP_GUARDS=1 git commit ... to bypass this loudly."
exit 1

91
docs/architecture/githooks/post-commit Executable file
View File

@ -0,0 +1,91 @@
#!/usr/bin/env bash
#
# Every commit goes to Forgejo, without anybody having to remember the push.
#
# ## Why a hook rather than a habit
#
# The commit that added `pre-commit` sat unpushed for a day. Nothing was wrong
# with it; it just never got the second command. That is the whole failure this
# closes — work that exists on one laptop and nowhere else is work that is one
# disk away from gone, and it is invisible to anybody reading the tracker.
#
# ## It runs after the commit, and cannot undo one
#
# git ignores this hook's exit code, which is the right shape for the job: a
# network that is down must not cost somebody a commit they already made. So a
# failed push is **reported loudly and left in place** — the commit stands, the
# branch is simply still ahead, and the next commit tries again.
#
# What it will never do is force. A rejected push means the remote has something
# this checkout has not seen, and the fix for that is a human running a pull, not
# a hook overwriting the difference.
#
# ## When it deliberately stays out of the way
#
# - mid-rebase, mid-cherry-pick, mid-am: every step of a rebase fires this
# hook, and pushing an intermediate commit publishes a history that is about
# to be rewritten. Wait for the rebase to finish.
# - detached HEAD: there is no branch to push, and guessing one is worse than
# doing nothing.
# - no `origin`: a clone with no remote is a legitimate state, not an error.
#
# SKIP_PUSH=1 git commit ... commits without publishing, loudly
#
# The counterpart to `SKIP_GUARDS` in the pre-commit hook, and loud for the same
# reason: an exception that leaves no trace becomes a habit.
set -uo pipefail
cd "$(git rev-parse --show-toplevel)" || exit 0
say() { printf '\033[1mpost-commit:\033[0m %s\n' "$*" >&2; }
if [ -n "${SKIP_PUSH:-}" ]; then
say "SKIP_PUSH set — this commit was NOT pushed."
exit 0
fi
git_dir=$(git rev-parse --git-dir)
# A rebase, cherry-pick or `git am` fires this hook once per replayed commit.
# Those commits are provisional by definition.
for marker in rebase-merge rebase-apply CHERRY_PICK_HEAD; do
if [ -e "$git_dir/$marker" ]; then
say "a rebase or cherry-pick is in progress — not pushing until it finishes."
exit 0
fi
done
branch=$(git symbolic-ref --quiet --short HEAD) || {
say "detached HEAD — no branch to push."
exit 0
}
if ! git remote get-url origin >/dev/null 2>&1; then
say "no 'origin' remote — nothing to push to."
exit 0
fi
say "pushing $branch to origin…"
# --porcelain keeps the output to one parseable line per ref rather than the
# usual banner, and the timeout is here because an unreachable SSH host
# otherwise hangs the terminal well past the point of being useful.
if timeout 60 git push --porcelain origin "$branch"; then
exit 0
fi
status=$?
if [ "$status" -eq 124 ]; then
say "push timed out after 60s. The commit is safe locally; push when the"
say " remote is reachable."
else
say "push was refused. The commit is safe locally and the branch is now ahead."
say " If this is a non-fast-forward, pull and reconcile — this hook will"
say " not force, and should not."
fi
# Deliberately zero. git ignores it either way, and returning non-zero here reads
# as though the commit failed when it did not.
exit 0

142
docs/architecture/githooks/pre-commit Executable file
View File

@ -0,0 +1,142 @@
#!/usr/bin/env bash
#
# The repo's own guards, before a commit rather than after it.
#
# ## Why this exists in the repository and not in .git/hooks
#
# `.git/hooks` is not versioned, so a hook living there protects exactly one
# checkout and silently protects nothing anywhere else. This directory is
# committed, and `core.hooksPath` points at it:
#
# git config core.hooksPath .githooks
#
# That one line is the only setup, and it is in the README beside the test
# command.
#
# ## What it checks, and why in this order
#
# Typecheck first: it takes about two seconds and catches the class of mistake
# that is most annoying to discover later — a type change whose consumers were
# never visited. Then the suite, which is where the real guarantees live: the
# conformance tests that keep registries total, the sentinel tests that keep
# secrets out of logs, and the mutation-tested guards.
#
# Both are already the rule for finishing work here. A hook is just the version
# that does not depend on somebody remembering.
#
# ## It warns about unstaged changes rather than failing on them
#
# Both commands run against the **working tree**, not against the index. So a
# clean run proves the working tree is good, which is only the same thing as the
# commit being good when nothing is left unstaged.
#
# That distinction matters in this checkout specifically: it is edited by more
# than one person at a time, and commits are staged by explicit path. A green
# hook beside three unstaged files has verified something other than what is
# about to be committed, and the honest thing is to say so rather than imply a
# guarantee that was not made.
#
# ## Escape hatch
#
# SKIP_GUARDS=1 git commit ... skips both, loudly
# git commit --no-verify ... skips the hook entirely, silently
#
# The first is preferred: it leaves a line in the terminal saying the guards did
# not run, which is the difference between a deliberate exception and a habit.
set -uo pipefail
cd "$(git rev-parse --show-toplevel)" || exit 1
say() { printf '\033[1mpre-commit:\033[0m %s\n' "$*" >&2; }
if [ -n "${SKIP_GUARDS:-}" ]; then
say "SKIP_GUARDS set — typecheck and tests did NOT run for this commit."
exit 0
fi
# Nothing staged is not this hook's problem; git will refuse on its own.
if git diff --cached --quiet; then
exit 0
fi
# Only worth running when source or tests changed. A commit that touches docs or
# migrations alone still gets the typecheck, because a migration can be
# referenced from a test, but it should not wait on the whole suite.
#
# The four files after the `|` are not source, and they are here because of what
# `tests/version.test.ts` guards: package.json, the Dockerfile, and the two
# image pins in README.md and docker-compose.example.yml must all name the same
# version. With `src|tests` alone, a commit that hand-edits only the README pin
# — precisely the drift that left those pins six versions stale — would get the
# typecheck and skip the one test that would have caught it. The guard has to
# run on the commits it exists to police.
staged=$(git diff --cached --name-only)
touches_code=$(printf '%s\n' "$staged" \
| grep -cE '^(src|tests)/.*\.(ts|tsx)$|^(package\.json|Dockerfile|README\.md|docker-compose\.example\.yml)$' || true)
say "typecheck…"
if ! npx tsc --noEmit; then
say "typecheck failed — commit refused."
exit 1
fi
# Credentials, before the commit exists.
#
# First, and cheap: it reads the staged diff only. A secret caught here costs a
# `git reset`; the same secret caught after a push costs a rotation, because
# deleting the line does not remove it from a commit that already exists.
if [ -x scripts/secrets.sh ] || [ -f scripts/secrets.sh ]; then
if ! bash scripts/secrets.sh; then
say "possible credential in the staged changes — commit refused."
say "If it is real, rotate it. If it is not, --allow the path or adjust"
say "the patterns; do not silence the check."
exit 1
fi
fi
if [ "$touches_code" -gt 0 ]; then
say "tests…"
# TEST_DATABASE_URL is picked up from the environment when it is set. Without
# one the database suites skip themselves rather than fail, which is the
# existing behaviour in tests/setup.ts — so this hook is useful on a machine
# with no Postgres and stricter on one with it.
#
# But "not set" and "not available" are different, and this used to treat them
# the same: every commit on this machine ran ~900 of 1,600 tests while a test
# database sat listening on 55432, because nothing looked. A commit hook
# should not *refuse* over a missing database — that belongs to the release
# gate, which now does — but it should use one that is there.
# Set these two together in the repository that adopts this hook — the port
# it listens on, and the URL that reaches it. Leaving them unset is fine and
# simply means the database suites skip, which is the right default on a
# machine with no Postgres.
TEST_DB_PORT="${TEST_DB_PORT:-}"
TEST_DB_URL="${TEST_DB_URL:-}"
if [ -z "${TEST_DATABASE_URL:-}" ] && [ -n "$TEST_DB_PORT" ] && [ -n "$TEST_DB_URL" ] \
&& (exec 3<>/dev/tcp/127.0.0.1/"$TEST_DB_PORT") 2>/dev/null; then
exec 3<&- 2>/dev/null || true
export TEST_DATABASE_URL="$TEST_DB_URL"
say "using the local test database on ${TEST_DB_PORT} — the full suite will run."
elif [ -z "${TEST_DATABASE_URL:-}" ]; then
say "no test database; the database suites will skip. The release gate is stricter."
fi
if ! npx vitest run --reporter=dot; then
say "tests failed — commit refused."
exit 1
fi
else
say "no .ts/.tsx staged — skipping the suite."
fi
# Said last so it is the thing still on screen when the editor opens.
if ! git diff --quiet; then
say "NOTE: unstaged changes are present. The guards ran against the working"
say " tree, so they did not verify this commit in isolation."
fi
exit 0

187
docs/architecture/scripts/audit-gate.mjs Normal file
View File

@ -0,0 +1,187 @@
#!/usr/bin/env node
/**
* Production dependency advisories, with an allowlist npm does not give you.
*
* node scripts/audit-gate.mjs # fail on unassessed high/critical
* node scripts/audit-gate.mjs --list # print what is found, exit 0 regardless
*
* 0 no unassessed high/critical advisories in production dependencies
* 1 unassessed advisories found — they are listed
* 2 NOTHING WAS CHECKED — audit did not run, or its output was not the shape
* this understands. Not a pass, and CI must not treat it as one.
*
* ## Why this exists rather than `npm audit --audit-level=high`
*
* npm audit has no allowlist. So when an advisory lands that provably cannot
* reach the application — a server-side CSRF in a router the app only uses
* declaratively, say — there are exactly two options in plain npm, and both are
* bad. Lower the threshold, which throws away a gate that has already caught
* real issues. Or let CI sit red, which for an advisory with no fixed version
* means indefinitely, and a permanently red gate is a gate everybody has
* learned to ignore. That is `GUARDS.md` rule 6 arriving through the back door.
*
* The third option is to say **which** advisories have been assessed and why,
* in a file somebody reviews. That is this.
*
* ## `--omit=dev` is the boundary, and it is a real one
*
* A build-time advisory in a test runner does not ship. The container holds
* production dependencies, so those are what this asks about. Widening it to
* dev pulls in advisories nothing can reach from outside and drowns the ones
* that can.
*
* ## Rules for adding an entry — all four, or it is not an assessment
*
* - **reason** — why it cannot reach THIS app, concretely, naming the file or
* the API that would have to be in use for it to matter. "Low risk" is not
* an assessment, it is an adjective.
* - **reachableIf** — what would make it reachable. This is the falsifiable
* half: without it, nobody can ever tell whether the waiver still holds.
* - **recheck** — the event that should retire the entry, not a date.
* - **Mirror it in the project's SECURITY document**, so an operator reading
* about the system's posture sees the same list a developer does. An
* exception that lives only in a script is one the security review misses.
*
* Delete an entry the moment a fix ships. An allowlist nobody prunes stops
* being a record of decisions and becomes a list of things nobody looks at.
*
* ===========================================================================
* TEMPLATE COPY — configure this before the first run
* ===========================================================================
*
* Copy to `scripts/audit-gate.mjs`, add `"audit": "node scripts/audit-gate.mjs"`
* to package.json, and — this is the step that matters — put it in whatever
* command the gate actually runs. A guard wired to nothing is documentation.
*
* Set SECURITY_DOC below to where this project keeps its security document, so
* the failure message names a path that exists. `docs/security/SECURITY.md` is
* the template's location; a repository that keeps it elsewhere and does not
* change this line prints a path a reader cannot open, which is exactly the
* class of stale doc-claim `doc-claims.sh` exists to catch.
*
* ALLOWED starts empty, deliberately. An allowlist inherited from another
* project waives advisories against an application that was never assessed —
* and it waives them silently, which is the worst available outcome.
*/
// ===========================================================================
// CONFIGURATION
// ===========================================================================
/** Where this project's security document lives, for the failure message. */
const SECURITY_DOC = 'docs/security/SECURITY.md';
/**
* Assessed advisories, keyed by the package name npm reports.
*
* Empty on purpose — see the header. A worked example of the shape, from the
* project this came from:
*
* 'react-router': {
* reason:
* 'Advisory is RSC Mode CSRF (action execution before a 400). This app ' +
* 'is a Vite SPA using declarative <BrowserRouter> (client/main.tsx) — ' +
* 'no RSC mode, no framework/data router, no createStaticHandler, no ' +
* '@react-router/* server package. No fixed version exists: the ' +
* 'vulnerable range extends past the newest published release, and the ' +
* 'only "fix" npm offers is a downgrade.',
* reachableIf: 'the app adopts RSC mode or a react-router server runtime',
* recheck: 'when react-router publishes a release above the vulnerable range',
* },
*
* Note what makes that one an assessment rather than a shrug: it names the
* file, the four APIs that would have to be in use, and the release that
* retires it.
*/
const ALLOWED = {};
// ===========================================================================
import { execFileSync } from 'node:child_process';
const listOnly = process.argv.includes('--list');
const die = (code, message) => {
console.error(`[audit-gate] ${message}`);
process.exit(code);
};
let raw;
try {
raw = execFileSync('npm', ['audit', '--omit=dev', '--json'], {
encoding: 'utf8',
maxBuffer: 32 * 1024 * 1024,
});
} catch (err) {
// npm audit exits non-zero whenever it finds anything at all, so a throw here
// is the normal path and the JSON is still on stdout. An empty stdout is the
// abnormal one — npm itself failed, and nothing was measured.
raw = err.stdout;
if (!raw) {
die(2, `npm audit produced no output (${err.message}) — nothing was checked`);
}
}
let report;
try {
report = JSON.parse(raw);
} catch {
die(2, 'npm audit output was not JSON — nothing was checked');
}
// Distinguishing "no vulnerabilities" from "no vulnerabilities key" is the whole
// of GUARDS.md rule 5 applied to this script's own instrument. npm's audit JSON
// has changed shape between major versions before; a run against a shape this
// does not understand must not render as a clean bill of health.
if (typeof report.vulnerabilities !== 'object' || report.vulnerabilities === null) {
die(2, 'npm audit output had no `vulnerabilities` object — nothing was checked');
}
const blocking = [];
const waived = [];
for (const [name, entry] of Object.entries(report.vulnerabilities)) {
if (entry.severity !== 'high' && entry.severity !== 'critical') continue;
(ALLOWED[name] ? waived : blocking).push({
name,
severity: entry.severity,
range: entry.range,
});
}
for (const { name, severity, range } of waived) {
console.log(`[audit-gate] WAIVED ${name} (${severity}, ${range}) — ${ALLOWED[name].reason}`);
}
if (blocking.length && !listOnly) {
console.error(`\n[audit-gate] FAILED — ${blocking.length} unassessed high/critical advisory(ies):`);
for (const { name, severity, range } of blocking) {
console.error(` ✗ ${name} (${severity}, ${range})`);
}
console.error(
`\nFix it, or — only if it provably cannot reach this app — add an assessed entry\n` +
`to this script and mirror it in ${SECURITY_DOC}. All four fields, or it is\n` +
`not an assessment.`,
);
process.exit(1);
}
if (blocking.length) {
console.log(`[audit-gate] --list: ${blocking.length} unassessed high/critical advisory(ies)`);
for (const { name, severity, range } of blocking) {
console.log(` ✗ ${name} (${severity}, ${range})`);
}
}
console.log(
`[audit-gate] OK — 0 unassessed high/critical advisories in production dependencies` +
(waived.length ? ` (${waived.length} assessed and waived)` : ''),
);

942
docs/architecture/scripts/backup.sh Executable file
View File

@ -0,0 +1,942 @@
#!/usr/bin/env bash
#
# Write one database dump, prove the file is readable before trusting it, and
# keep the newest few.
#
# bash scripts/backup.sh # write one verified dump
# bash scripts/backup.sh --dry-run # show what it would write and delete
# bash scripts/backup.sh --no-prune # write, delete nothing
# BACKUP_KEEP=30 bash scripts/backup.sh # keep thirty instead of seven
#
# 0 3 * * * cd /srv/app && bash scripts/backup.sh # from cron, see below
#
# ===========================================================================
# TEMPLATE COPY — configure this before the first run
# ===========================================================================
#
# Copy to `scripts/backup.sh`, add `"backup": "bash scripts/backup.sh"` to
# package.json if this is a Node project, and set the two values in the
# CONFIGURATION block below. The script refuses to run until they are set: it
# has no defaults, deliberately.
#
# A BACKUP_DIR or BACKUP_NAME inherited from another project is not a cosmetic
# mistake. BACKUP_NAME is the filename prefix, and the filename prefix is what
# the retention rule globs on — so the wrong name writes this project's rows
# into that project's series and then deletes that project's oldest dumps to
# make room for them. Both halves are silent, the second is irreversible, and
# the directory afterwards looks exactly like a healthy backup series. Hence:
# no defaults, and a loud failure instead.
#
# Assumes: bash, coreutils, `awk`, `sed`, and the client tools for the engine
# (`pg_dump` and `pg_restore` for the default). No jq, no Node, nothing
# language-specific — unlike the release script beside it, this does not care
# what the project is written in.
#
# awk is named because it is not coreutils, a minimal container image can be
# without it, and it is what counts the tables — so it is checked at startup
# alongside the engine's tools. Left unchecked, its absence arrives as "the
# archive holds 0 tables", which is a sentence about the database and would send
# somebody looking at the wrong thing entirely.
#
# ## Why this exists
#
# A backup nobody has restored is a hypothesis, not a backup.
#
# The failure this is built around is not "the backup did not run". That one is
# loud: the directory is empty and somebody notices. The failure is a job that
# runs every night for a year and writes a file every night for a year, and the
# file is truncated, or is a dump of the wrong database, or is 400 bytes of
# `pg_dump: error:` because the password expired in March. Nothing about the
# directory listing distinguishes that from a working backup. The size column
# is plausible. The timestamps march forward. It is discovered on the one day
# it matters, by somebody who has already lost the database.
#
# So this script does not write a file and call it a backup. It writes the file
# under a temporary name, reads the file back with `pg_restore --list`, counts
# what is in it, refuses the whole run if the archive cannot be read, holds no
# tables, or CANNOT BE COUNTED, and only then renames it into place.
#
# The third of those is not padding. The counting step is one `awk` away from
# returning nothing, and a count that did not happen is not a count of zero: if
# the result were trusted unchecked, `[ "$TABLES" -gt 0 ]` would fail with
# "integer expression expected", `if` would read that failure as false, and an
# archive nobody had counted would be renamed into place and announced as
# verified — with retention then deleting real dumps to make room for it.
#
# ## What --list proves, and what it does not
#
# It proves the file is a real archive: the header parses, the table of
# contents is intact and complete, and the dump contains the tables you can see
# named in the output. Combined with `pg_dump` exiting zero, that eliminates
# the truncated file, the zero-byte file, the error message written where a
# dump should be, and the dump taken against an empty or wrong database.
#
# It does not decompress and check every data block, and it is not a restore.
# An archive can list cleanly and still fail to load — a broken large object, a
# circular constraint order, an extension that is not installed on the machine
# you are restoring onto. Nothing short of an actual restore finds those, which
# is why "the other half" below is not optional.
#
# ## Atomic, because the failure mode is a good backup destroyed by a bad one
#
# Every write goes to `<name>.part` and is renamed only after it verifies. The
# rename is within one directory, so it is rename(2) and not a copy: the final
# name never exists holding half a file, and a reader — a sync job, a human, an
# offsite copy — cannot pick up a dump that is still being written.
#
# Writing directly to the final path would mean a dump that dies at 90% has
# already overwritten last night's, which was fine. That is worse than not
# running at all: it converts a working backup into a broken one and reports
# success while doing it.
#
# A dump that fails verification is KEPT, under its `.part` name, and the path
# is printed. It is evidence — usually the error message is inside it — and
# deleting evidence to keep the directory tidy is how the cause stays unknown.
#
# ## Retention deletes by explicit path, inside one directory, and nowhere else
#
# There is no `find -delete`, no `rm` with a glob, and no recursion. The
# candidate list is a glob of the exact filename shape this script writes, and
# every path is then checked one at a time: its parent directory, resolved with
# `pwd -P` rather than compared as text, must BE the backup directory; it must
# be a regular file and not a symlink; its name must match the pattern. Four
# checks, all of which have to pass, for each file, immediately before the `rm`.
#
# This is deliberately more than is needed for paths the script generated
# itself. The point is that no configuration value, no symlink planted in the
# directory, and no future edit to the glob can produce a deletion outside the
# backup directory — the guard does not trust the list it was handed.
#
# ## Credentials
#
# Connection details come from the environment and are never written here.
# PGHOST/PGPORT/PGUSER/PGDATABASE with a ~/.pgpass is the preferred form;
# DATABASE_URL works and is warned about, because a URL carries the password in
# it and reaches pg_dump as an argument, where `ps` shows it to every user on
# the host.
#
# Nothing prints a password or a connection URL — not to stdout, not to stderr,
# and above all not into a filename, where it would sit in the directory
# listing forever and be copied offsite with the dumps. Credentials in logs is
# the classic leak in backup scripts and it is usually introduced by an
# innocent-looking `say "dumping $DATABASE_URL"`. Do not add one.
#
# ## Changing the engine
#
# PostgreSQL is the supported default. Everything engine-specific is in the
# block marked ENGINE below — four functions and two variables — and nothing
# outside that block knows what a database is. Swapping in MySQL, SQLite or
# anything else is an edit to that block alone.
#
# One constraint carries over: `engine_verify` must READ THE FILE THAT WAS JUST
# WRITTEN. Re-querying the database, checking an exit code again, or trusting
# the file size verifies nothing about the artefact. Engines whose dump is
# plain SQL have no `--list` equivalent; for those the honest verification is a
# restore into a scratch database, and if that is too expensive to do on every
# run then the backup is unverified and the header of this script should be
# edited to stop claiming otherwise.
#
# ## What it deliberately does not do
#
# It does not copy the dump anywhere. A backup on the same disk as the database
# survives `DROP TABLE` and nothing else — not the disk, not the host, not the
# provider account. Getting these files onto different hardware is a separate,
# deliberate act (rsync, restic, object storage) for the same reason the
# release script does not deploy: two decisions, made one at a time.
#
# It does not schedule itself. BACKUP_KEEP is a count of files, not a period —
# seven of these is a week of nightly runs or seven hours of hourly ones.
#
# It does not restore. See below.
#
# ## The other half: a restore you have actually performed
#
# This script verifies the artefact. Only a restore verifies the backup, and a
# restore is also the only way to learn the number that matters during an
# incident, which is how long it takes.
#
# Do it on a schedule you write down — quarterly is a reasonable floor — into a
# scratch database, from the newest file this script produced, and record the
# duration and the command. Until that has happened once, the honest status of
# this directory is "dumps that appear to be readable", and the first restore
# will be attempted by somebody who has already lost the database.
#
# pg_restore --clean --if-exists --no-owner -d "$SCRATCH_URL" <newest file>
set -uo pipefail
# Glob expansion and `sort` follow the collation locale, and a cron job's
# locale is not the one your shell has. C collation makes the ordering of the
# candidate list byte-order and therefore the same everywhere — the ordering
# decides which files retention calls "oldest", so it deciding differently
# under cron than under test would be discovered by deleting the wrong ones.
# LC_COLLATE only: LC_ALL=C would also change LC_CTYPE, and character handling
# is not something a backup script should be quietly redefining.
unset LC_ALL
export LC_COLLATE=C
# An unmatched glob otherwise expands to the pattern itself, so an empty backup
# directory would produce one "file" literally named `myapp-[0-9][0-9]…` and
# hand it to the retention loop as a real candidate.
shopt -s nullglob
say() { printf '\033[1mbackup:\033[0m %s\n' "$*" >&2; }
die() { printf '\033[1mbackup:\033[0m %s\n' "$*" >&2; exit 1; }
# ---------------------------------------------------------------------------
# CONFIGURATION — set these two, then delete this banner.
#
# Both are empty on purpose. See the note at the top: an inherited BACKUP_NAME
# points the retention rule at another project's dumps, and an inherited
# BACKUP_DIR puts this project's rows where that project's offsite copy will
# collect them.
# ---------------------------------------------------------------------------
# Absolute path to the directory holding the dumps. Must already exist, must be
# outside the repository, and should be readable only by the user running this
# — a dump is every row in the database in one file.
BACKUP_DIR="${BACKUP_DIR:-}"
# Filename prefix identifying this project's series, e.g. `acme-orders`.
# Letters, digits, dot, underscore and hyphen only. This is what retention
# globs on; see the header for why it is not derived from the database name.
BACKUP_NAME="${BACKUP_NAME:-}"
# ---------------------------------------------------------------------------
# Tunables. These have defaults because none of them names anything belonging
# to a particular project.
# ---------------------------------------------------------------------------
# How many dumps survive, newest first. Seven is a week of nightly runs.
#
# Two is the floor enforced below. Keeping one means every run destroys the
# only copy you have in order to make a copy that has not yet been shown to be
# worth anything — and "worth something" here is more than readable, since a
# dump of an empty or wrong database verifies perfectly. Two means the newest
# dump can turn out to be worthless without that being the end of it.
KEEP="${BACKUP_KEEP:-7}"
# Refuse a dump containing fewer tables than this. One is the default because
# an archive with no tables in it is not a backup of anything, and the way you
# get one is `pg_dump` connecting somewhere you did not intend — see the
# PGDATABASE note below, where the accident produces a valid, tiny, entirely
# empty archive. Set 0 only if this project's database genuinely has no tables,
# in which case there is nothing here to protect.
MIN_TABLES="${BACKUP_MIN_TABLES:-1}"
# Warn when the new dump is smaller than this percentage of the previous one.
# A dump that halves overnight is usually a partial dump, a lost permission on
# some schema, or the wrong database — none of which fail loudly on their own.
#
# A warning and never a refusal: a legitimate large delete would otherwise
# block every backup from that moment on, which is a way of losing data by
# being careful about data.
SHRINK_PCT="${BACKUP_SHRINK_PCT:-50}"
# ===========================================================================
# ENGINE — PostgreSQL. To use another engine, replace this block and nothing
# else. See "Changing the engine" in the header for the one rule that binds.
# ===========================================================================
DUMP_SUFFIX=".dump"
# The custom format is not a preference. `pg_restore --list` only reads custom,
# directory and tar archives; a plain SQL dump (`-Fp`, the pg_dump default)
# cannot be verified without restoring it, and this whole script is built on
# being able to verify. It also compresses by default, so nothing is piped
# through gzip — a pipeline would put a second program's exit status where
# pg_dump's needs to be.
engine_tools() { printf 'pg_dump pg_restore'; }
engine_dump() {
local out="$1"
# --file rather than a `>` redirect: with a redirect the SHELL creates the
# file, so a pg_dump that never starts (missing binary, bad PATH under cron)
# still leaves a zero-byte file behind, and the writer and the exit status
# stop being the same process.
#
# -w never prompts for a password. Without it, a run whose credentials have
# gone stale blocks on a terminal that cron does not have, and the backup
# does not fail — it hangs, forever, holding the lock, which reads as "no
# output, no error" for however long it takes somebody to look.
if [ -n "${DATABASE_URL:-}" ]; then
pg_dump -w --format=custom --file="$out" -d "$DATABASE_URL"
else
pg_dump -w --format=custom --file="$out"
fi
}
# Reads the archive back and writes its table of contents to $2. Anything
# pg_restore says goes to $3, kept separately so it cannot be counted as
# archive content by the summariser below.
engine_verify() {
pg_restore --list "$1" >"$2" 2>"$3"
}
# Prints "<table definitions> <data sections>" from a TOC listing.
#
# TOC entries look like `216; 1259 16385 TABLE public users postgres`, and the
# data sections like `3021; 0 16385 TABLE DATA public users postgres`. Comment
# lines start with `;` and are excluded by the shape of field one. Counting
# both matters because a --data-only dump has no TABLE entries and a
# --schema-only dump has no TABLE DATA entries, and reporting zero for either
# without saying which kind of dump this is would be a lie by omission.
engine_summarise() {
awk '$1 ~ /^[0-9]+;$/ && $4 == "TABLE" {
if ($5 == "DATA") d++; else t++
}
END { printf "%d %d\n", t + 0, d + 0 }' "$1"
}
# ===========================================================================
# End of ENGINE block.
# ===========================================================================
DRY_RUN=""
NO_PRUNE=""
for arg in "$@"; do
case "$arg" in
--dry-run) DRY_RUN="yes" ;;
--no-prune) NO_PRUNE="yes" ;;
# No positional arguments exist, so one is a mistyped flag or a path
# somebody expected this to accept. Either way, acting on the rest of the
# command line as though it were fine is how a backup goes somewhere else.
*) die "unknown argument '$arg'. Usage: bash scripts/backup.sh [--dry-run] [--no-prune]" ;;
esac
done
# ---------------------------------------------------------------------------
# Refuse to run half-configured. Everything checkable without a side effect,
# checked before the first side effect, and named one at a time so the message
# says which value is wrong rather than "configuration error".
# ---------------------------------------------------------------------------
[ -n "$BACKUP_DIR" ] || die "set BACKUP_DIR — the directory to write dumps into. See the CONFIGURATION block."
[ -n "$BACKUP_NAME" ] || die "set BACKUP_NAME — the filename prefix for this project's dumps. See the CONFIGURATION block."
# A relative path resolves against the working directory, and cron's working
# directory is not yours. The dumps would land somewhere else, and retention —
# globbing that same relative path — would find that other directory empty,
# report "nothing to remove", and let the real series grow until the disk
# filled. Both halves look healthy in isolation.
case "$BACKUP_DIR" in
/*) : ;;
*) die "BACKUP_DIR ('$BACKUP_DIR') must be an absolute path — a relative one
points somewhere different under cron than it does in your shell." ;;
esac
# The glob and the delete guard are built from this. A name containing `*`, `?`
# or `/` would widen the candidate list past this project's dumps, which is the
# one thing the guard cannot make safe by checking paths.
case "$BACKUP_NAME" in
*[!A-Za-z0-9._-]*) die "BACKUP_NAME ('$BACKUP_NAME') may contain only letters, digits, dot,
underscore and hyphen — it is used as a filename glob." ;;
esac
case "$KEEP" in
''|*[!0-9]*) die "BACKUP_KEEP must be a whole number, got '$KEEP'." ;;
esac
# Refused rather than clamped: a caller who typed 1 meant something, and it was
# not "destroy the only copy before checking the new one is worth having".
[ "$KEEP" -ge 2 ] || die "BACKUP_KEEP must be at least 2 — see the note above KEEP in this script."
case "$MIN_TABLES" in
''|*[!0-9]*) die "BACKUP_MIN_TABLES must be a whole number, got '$MIN_TABLES'." ;;
esac
case "$SHRINK_PCT" in
''|*[!0-9]*) die "BACKUP_SHRINK_PCT must be a whole number, got '$SHRINK_PCT'." ;;
esac
for tool in $(engine_tools); do
command -v "$tool" >/dev/null 2>&1 \
|| die "$tool is not on PATH. Install the client tools for this engine
(Debian/Ubuntu: postgresql-client) and run this again."
done
# awk is checked separately because it is not part of the engine and not part of
# coreutils: it is what turns the table of contents into a number, and cron's
# PATH is not your shell's. Without this check its absence surfaces further down
# as "the archive holds 0 table(s)" — a measurement, about the database, that
# nothing measured.
command -v awk >/dev/null 2>&1 \
|| die "awk is not on PATH. It is what counts what is inside the archive, and
without it this script cannot tell a good dump from an empty one.
Under cron, PATH is not the PATH your shell has."
[ -d "$BACKUP_DIR" ] || die "BACKUP_DIR ('$BACKUP_DIR') does not exist.
Create it deliberately: mkdir -p -m 700 '$BACKUP_DIR'
It is not created here on purpose — a typo in the path is otherwise
indistinguishable from a first run, and the typo'd directory would fill
with a complete, correct-looking series that nobody restores from."
# Resolved once, here, and used for every path comparison afterwards. Comparing
# the configured string instead would let `/srv/backups/../../etc` pass a
# prefix test while being nowhere near the backup directory.
BACKUP_DIR_REAL=$(cd "$BACKUP_DIR" && pwd -P) || die "cannot resolve BACKUP_DIR ('$BACKUP_DIR')."
# Ahead of the writability check on purpose: for an unprivileged user / is not
# writable, so leaving this until later answers "permission denied" to somebody
# who is one `sudo` away from asking the same question again and being told yes.
[ "$BACKUP_DIR_REAL" != "/" ] || die "BACKUP_DIR resolves to / — retention would treat the root of the
filesystem as a directory of expendable files."
[ -w "$BACKUP_DIR" ] || die "BACKUP_DIR ('$BACKUP_DIR') is not writable by $(id -un)."
# A dump inside the checkout is one `git add -A` from being committed, and one
# push from being public, with every row in the database in it. It is also
# copied by every clone and every deploy from then on. The engine that made it
# does not put it back once that happens.
repo_root=$(git rev-parse --show-toplevel 2>/dev/null) || repo_root=""
if [ -n "$repo_root" ] && repo_real=$(cd "$repo_root" && pwd -P); then
case "$BACKUP_DIR_REAL/" in
"$repo_real"/*)
die "BACKUP_DIR ('$BACKUP_DIR_REAL') is inside the repository at $repo_real.
Put it outside the checkout — a dump in a working tree is one 'git add -A'
away from being committed and pushed." ;;
esac
fi
# Permissions are reported, not corrected. Changing the mode of a directory the
# operator created — which may be a mount, or shared with a sync agent — is a
# decision this script does not get to make on their behalf; failing to mention
# that every row in the database is world-readable would be worse.
dir_mode=$(ls -ld "$BACKUP_DIR_REAL" 2>/dev/null | cut -c5-10)
case "$dir_mode" in
------) : ;;
'') say "note: could not read the permissions of $BACKUP_DIR_REAL." ;;
*) say "WARNING: $BACKUP_DIR_REAL is accessible to group or other (mode bits"
say " '$dir_mode'). A dump is every row in the database in one file."
say " chmod 700 it unless something else is meant to read these." ;;
esac
# ---------------------------------------------------------------------------
# Connection. From the environment, never from this file.
# ---------------------------------------------------------------------------
# With neither of these set, pg_dump does not fail — it connects to a database
# named after the current user. On most hosts that database exists, is empty,
# and dumps in half a second into a perfectly valid archive. The MIN_TABLES
# check below is the second net under this; refusing here is the first.
if [ -n "${DATABASE_URL:-}" ]; then
# Describes the SOURCE of the connection, never its contents. This string is
# printed; the URL must not be.
TARGET_DESC="DATABASE_URL (contents not shown)"
say "WARNING: using DATABASE_URL. It reaches pg_dump as a command-line"
say " argument, so the password in it is visible to anyone who can"
say " run 'ps' on this host. PGHOST/PGUSER/PGDATABASE with a"
say " ~/.pgpass keeps the password out of the process table."
elif [ -n "${PGDATABASE:-}" ]; then
# PGDATABASE and PGHOST are not secrets and naming them is what makes the
# log worth reading. PGPASSWORD is never touched by anything here.
TARGET_DESC="${PGDATABASE}${PGHOST:+ on ${PGHOST}}"
else
die "no database configured: set PGDATABASE (with PGHOST/PGUSER as needed, and
a ~/.pgpass for the password) or DATABASE_URL in the environment.
Without one, pg_dump connects to a database named after the current user
and produces a valid, empty archive."
fi
# ---------------------------------------------------------------------------
# Names. Built here so the dry run and the real run cannot describe different
# files.
#
# UTC, always: local time repeats an hour every autumn, which puts two dumps
# out of order by name in the one direction that matters — retention reads that
# order to decide what is oldest.
#
# Nothing derived from the connection goes into the filename. A URL in a name
# sits in the directory listing forever and is copied offsite with the dumps.
# ---------------------------------------------------------------------------
STAMP_GLOB='[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]T[0-9][0-9][0-9][0-9][0-9][0-9]Z'
STAMP=$(date -u +%Y%m%dT%H%M%SZ) || die "cannot read the clock."
FINAL="${BACKUP_DIR_REAL}/${BACKUP_NAME}-${STAMP}${DUMP_SUFFIX}"
PART="${FINAL}.part"
# Only a second run inside the same second reaches either of these, and the
# alternative in both cases is destroying a file this script promised to keep:
# a dump that has already been verified, or the .part evidence of a run that
# failed a moment ago and is about to be looked at.
[ ! -e "$FINAL" ] || die "$FINAL already exists. Refusing to overwrite a verified dump."
[ ! -e "$PART" ] || die "$PART already exists — it is the unverified remains of a run that
failed within this same second. Inspect or remove it before running again."
human_size() {
local b="${1:-}"
# An unmeasurable file is reported as unmeasured. Printing "0 B" for a file
# whose size could not be read is the same lie as printing "nothing to
# delete" for a directory that could not be listed.
case "$b" in
''|*[!0-9]*) printf 'unknown size'; return 0 ;;
esac
# Integer arithmetic rather than `du -h`, whose output format and rounding
# differ between GNU and BSD and which reports blocks allocated rather than
# bytes written.
if [ "$b" -lt 1024 ]; then printf '%s B' "$b"
elif [ "$b" -lt 1048576 ]; then printf '%s.%s KiB' "$((b / 1024))" "$((b * 10 / 1024 % 10))"
elif [ "$b" -lt 1073741824 ]; then printf '%s.%s MiB' "$((b / 1048576))" "$((b * 10 / 1048576 % 10))"
else printf '%s.%s GiB' "$((b / 1073741824))" "$((b * 10 / 1073741824 % 10))"
fi
}
file_bytes() {
# `wc -c` is portable where `stat` is not: GNU wants -c%s and BSD wants -f%z.
# BSD pads the number with spaces, hence the tr.
#
# 2>/dev/null comes BEFORE the input redirect on purpose. Redirections are
# applied left to right, so with the other order the shell's own "no such
# file" message for a failed open is written before stderr has been silenced —
# a raw `backup.sh: line N: …` in the middle of otherwise formatted output.
# Callers read the empty result, which is what "unmeasurable" means here.
wc -c 2>/dev/null <"$1" | tr -d ' \n'
}
# Every existing dump of this series, oldest first. Pathname expansion sorts,
# and LC_COLLATE=C above makes that sort byte-order, so no external sort is
# involved and no filename has to survive a round trip through word splitting.
#
# The glob is the exact shape this script writes — prefix, UTC stamp, suffix.
# Files that merely live in the directory are not candidates for anything: a
# `.part` from a failed run, a dump somebody copied in by hand, and another
# project's series all fail to match and are never counted or deleted.
list_dumps() {
printf '%s\n' "${BACKUP_DIR_REAL}/${BACKUP_NAME}-"$STAMP_GLOB"$DUMP_SUFFIX"
}
# ---------------------------------------------------------------------------
# The delete guard.
#
# Called immediately before every `rm`, on the path about to be removed, and
# every check has to pass. It exists so that no configuration value, no symlink
# planted in the backup directory and no later edit to the glob can produce a
# deletion outside BACKUP_DIR_REAL — it does not trust the list it was given.
# ---------------------------------------------------------------------------
deletable() {
local path="$1" parent base
case "$path" in
*/*) parent="${path%/*}"; base="${path##*/}" ;;
*) return 1 ;;
esac
# Resolved, not string-compared: `.` and `..` components make a text prefix
# test agree with a path that is somewhere else entirely.
parent=$(cd "$parent" 2>/dev/null && pwd -P) || return 1
[ "$parent" = "$BACKUP_DIR_REAL" ] || return 1
# -f follows symlinks, so a link named like a dump would pass every other
# check while pointing at a file the checks were never made about.
[ -f "$path" ] || return 1
[ ! -L "$path" ] || return 1
case "$base" in
"$BACKUP_NAME"-$STAMP_GLOB"$DUMP_SUFFIX") return 0 ;;
*) return 1 ;;
esac
}
# ---------------------------------------------------------------------------
# Retention.
#
# Last, and never fatal: by the time this runs the dump has been written and
# verified, and housekeeping must not be able to report that a backup which
# succeeded failed. Everything it declines to do, it says out loud instead.
# ---------------------------------------------------------------------------
prune() {
[ -z "$NO_PRUNE" ] || { say "retention: skipped (--no-prune)."; return 0; }
if [ ! -r "$BACKUP_DIR_REAL" ] || [ ! -x "$BACKUP_DIR_REAL" ]; then
say "retention: SKIPPED — cannot list $BACKUP_DIR_REAL."
say " That is 'I could not look', which is not the same answer as"
say " 'there is nothing to delete'. Nothing was removed."
return 0
fi
local -a candidates=()
local line
while IFS= read -r line; do
[ -n "$line" ] && candidates+=("$line")
done <<EOF
$(list_dumps)
EOF
local total=${#candidates[@]}
local effective=$total
# On a dry run the dump does not exist yet, so counting only what is on disk
# would understate the total by one and under-report the deletions by one.
# Counting it in is what makes --dry-run's list the list the real run uses.
[ -z "$DRY_RUN" ] || effective=$((total + 1))
if [ "$effective" -le "$KEEP" ]; then
say "retention: ${effective} dump(s) in the series, keeping ${KEEP} — nothing to remove."
return 0
fi
local doomed=$((effective - KEEP))
local i removed=0 refused=0 failed=0 path
for (( i = 0; i < doomed; i++ )); do
path="${candidates[$i]}"
if ! deletable "$path"; then
# Loud, because a candidate that fails the guard means the directory
# holds something the glob matched and the guard did not recognise —
# a symlink, or a name shaped like ours pointing elsewhere.
say " REFUSED $path — not a plain file directly inside $BACKUP_DIR_REAL."
refused=$((refused + 1))
continue
fi
if [ -n "$DRY_RUN" ]; then
say " would delete $path ($(human_size "$(file_bytes "$path")"))"
removed=$((removed + 1))
continue
fi
if rm -f -- "$path" && [ ! -e "$path" ]; then
say " deleted $path"
removed=$((removed + 1))
else
# Said, not counted as done. A retention pass that quietly failed looks
# exactly like one that had nothing to do, right up until the disk fills.
say " FAILED to delete $path — left in place."
failed=$((failed + 1))
fi
done
if [ -n "$DRY_RUN" ]; then
say "retention: would delete ${removed}, keeping the newest ${KEEP}."
else
say "retention: deleted ${removed}, keeping the newest ${KEEP}."
fi
[ "$refused" -eq 0 ] || say "retention: ${refused} candidate(s) refused by the delete guard, above."
[ "$failed" -eq 0 ] || say "retention: ${failed} deletion(s) failed — check permissions on $BACKUP_DIR_REAL."
}
# Reported, never deleted. A .part is the wreckage of a run that failed
# verification and it is usually the only record of why; retention leaves it
# alone by construction, so somebody has to be told it is there or it stays
# forever and is eventually mistaken for a backup.
report_parts() {
local -a found=( "${BACKUP_DIR_REAL}/${BACKUP_NAME}-"*"${DUMP_SUFFIX}.part" )
local -a stale=()
local p
# Filtered before it is counted, so the number printed is the number listed.
for p in ${found[@]+"${found[@]}"}; do
[ "$p" = "$PART" ] || stale+=("$p")
done
[ "${#stale[@]}" -gt 0 ] || return 0
say "note: ${#stale[@]} unverified .part file(s) from earlier failed runs are in"
say " $BACKUP_DIR_REAL. They are never deleted by retention. Look at them,"
say " then remove them by hand:"
for p in "${stale[@]}"; do
say " $p ($(human_size "$(file_bytes "$p")"))"
done
}
# The newest verified dump before this run, for the size comparison and for
# telling the operator what they still have if this run fails.
PREV=""
PREV_BYTES=0
# Kept apart from PREV_BYTES because they answer different questions. "There is
# no previous dump" and "there is one and its size could not be read" are both
# reasons not to compare sizes, and collapsing the second into PREV_BYTES=0
# would make the script go on to say "no previous dump in this series" about a
# file that is sitting right there — the same substitution of absence for
# measurement it refuses to make about free space and table counts.
PREV_MEASURED=""
while IFS= read -r line; do
[ -n "$line" ] && PREV="$line"
done <<EOF
$(list_dumps)
EOF
if [ -n "$PREV" ]; then
PREV_BYTES=$(file_bytes "$PREV")
case "$PREV_BYTES" in
''|*[!0-9]*) PREV_BYTES=0 ;;
*) PREV_MEASURED="yes" ;;
esac
fi
# ---------------------------------------------------------------------------
# Free space, checked against something real or not claimed at all.
#
# With no previous dump there is no basis for an estimate, and printing a
# reassuring number derived from nothing is worse than printing none.
# ---------------------------------------------------------------------------
free_kb=$(df -Pk "$BACKUP_DIR_REAL" 2>/dev/null | awk 'NR == 2 { print $4 }')
case "$free_kb" in
''|*[!0-9]*)
say "note: free space on $BACKUP_DIR_REAL could not be read — not checked." ;;
*)
if [ -z "$PREV_MEASURED" ] || [ "$PREV_BYTES" -eq 0 ]; then
say "note: $(human_size $((free_kb * 1024))) free; no measured previous dump to compare"
say " against, so whether that is enough is unknown."
elif [ $((free_kb * 1024)) -lt $((PREV_BYTES * 2)) ]; then
say "WARNING: $(human_size $((free_kb * 1024))) free, and the last dump was"
say " $(human_size "$PREV_BYTES"). This run may not fit."
fi ;;
esac
# ---------------------------------------------------------------------------
# --dry-run stops here, having touched nothing.
# ---------------------------------------------------------------------------
if [ -n "$DRY_RUN" ]; then
say "--dry-run: nothing was written or deleted. It would have:"
say " dumped ${TARGET_DESC}"
say " written ${PART}"
say " verified it with: pg_restore --list ${PART}"
say " renamed it to ${FINAL}"
say ""
say "and then applied retention, shown here for real because it deletes:"
prune
report_parts
exit 0
fi
# ---------------------------------------------------------------------------
# One run at a time.
#
# mkdir is atomic on every filesystem this will meet, which flock and lock
# files are not. Two overlapping runs — a nightly job on a database that now
# takes longer than a day to dump — double the load on the server and race each
# other's retention pass.
#
# A stale lock stops backups, so this exits NON-ZERO and names the fix rather
# than skipping quietly. A skipped backup that reports success is the exact
# failure this whole script exists to prevent, and a lock is not allowed to
# reintroduce it.
# ---------------------------------------------------------------------------
LOCK_DIR="${BACKUP_DIR_REAL}/.${BACKUP_NAME}.lock"
LOCK_HELD=""
# mkdir's error is captured rather than discarded, because "the lock exists" and
# "the lock could not be created" are different failures with different fixes,
# and only the first one is about concurrency. A read-only mount, a full
# filesystem, or a missing mkdir under cron's PATH all make mkdir fail too; the
# directory is what distinguishes them, so it is looked at instead of assumed.
LOCK_ERR=""
if ! LOCK_ERR=$(mkdir "$LOCK_DIR" 2>&1); then
if [ -d "$LOCK_DIR" ]; then
say "another run holds the lock at $LOCK_DIR"
[ ! -r "$LOCK_DIR/owner" ] || say " owner: $(cat "$LOCK_DIR/owner" 2>/dev/null)"
die "if no backup is running, that lock is stale: rmdir '$LOCK_DIR'"
fi
# No lock directory, so nothing is holding anything. Saying otherwise here
# would send somebody to rmdir a path that does not exist, watch that change
# nothing, and go looking for a second backup process — while the real cause
# sits unread in mkdir's own message and every night's backup stays dead.
say "could not create the lock directory $LOCK_DIR, and no lock is present —"
say " so this is NOT another run. The directory could not be made at all."
[ -z "$LOCK_ERR" ] || say " mkdir said: $LOCK_ERR"
die "check that $BACKUP_DIR_REAL is on a writable, non-full filesystem."
fi
LOCK_HELD="yes"
WORK=""
# Armed in the same breath as the lock, not after the next command. Anything
# that exits between acquiring the lock and installing this trap leaves a stale
# lock behind, and a stale lock stops every future run — the trap tolerating an
# empty WORK is much cheaper than that. $WORK is quoted-empty-safe because rm
# is guarded on it.
trap '[ -z "$LOCK_HELD" ] || rm -rf "$LOCK_DIR"; [ -z "$WORK" ] || rm -rf "$WORK"' EXIT
printf 'pid %s on %s since %s\n' "$$" "$(hostname 2>/dev/null || echo unknown)" \
"$(date -u +%Y-%m-%dT%H:%M:%SZ)" >"$LOCK_DIR/owner" 2>/dev/null
# The .part is deliberately NOT cleaned up by that trap. It is evidence when
# verification fails, and the failure paths below print where it is.
WORK=$(mktemp -d) || die "cannot create a temporary directory"
# ---------------------------------------------------------------------------
# Dump.
# ---------------------------------------------------------------------------
say "dumping ${TARGET_DESC} -> ${PART}"
# umask in a subshell so the archive is created 0600 from its first byte. Doing
# it with a chmod afterwards leaves a window in which every row in the database
# is readable by every user on the host, and that window is the whole dump.
if ! ( umask 077; engine_dump "$PART" ); then
say "the dump failed (pg_dump's own error is above). Nothing was renamed into"
say " place and nothing was deleted."
if [ -n "$PREV" ]; then
say " The newest verified dump is still $PREV"
else
say " There is no previously verified dump in $BACKUP_DIR_REAL."
fi
say " The partial file is kept for inspection: $PART"
exit 1
fi
[ -f "$PART" ] || die "pg_dump reported success but $PART does not exist. Nothing was renamed."
BYTES=$(file_bytes "$PART")
[ -n "$BYTES" ] || die "cannot measure $PART. It has NOT been renamed into place."
# ---------------------------------------------------------------------------
# Verify, before the file is allowed to become the backup.
# ---------------------------------------------------------------------------
say "verifying ${PART}…"
LISTING="$WORK/toc"
VERR="$WORK/toc.err"
if ! engine_verify "$PART" "$LISTING" "$VERR"; then
say "VERIFICATION FAILED — the file cannot be read back as an archive."
say " $(human_size "$BYTES") written. The reader said:"
sed -n '1,10p' "$VERR" >&2 2>/dev/null
say ""
say " It has NOT been renamed into place and NOTHING was deleted."
say " Kept for inspection: $PART"
if [ -n "$PREV" ]; then
say " The newest verified dump is still $PREV"
else
say " There is no previously verified dump in $BACKUP_DIR_REAL."
fi
exit 1
fi
SUMMARY=$(engine_summarise "$LISTING") || SUMMARY=""
read -r TABLES DATA_SECTIONS <<EOF
$SUMMARY
EOF
TABLES="${TABLES:-}"
DATA_SECTIONS="${DATA_SECTIONS:-}"
# engine_summarise is the one thing in this script an engine swap is expected to
# replace, so its result is checked and not trusted. Two whole numbers or this
# run does not continue.
#
# Defaulting the empties to 0 instead — which is what this used to do — turns
# every way the count can fail to happen into the sentence "the archive holds 0
# tables", which is a claim about the database. A missing awk produces it. So
# does a replacement summariser that prints a warning, or one number, or an
# error. And a NON-numeric result is worse than misleading: `[ "$TABLES" -gt 0 ]`
# exits 2 with "integer expression expected", `if` treats that as false, the
# MIN_TABLES gate below does the same, and the run walks straight past its only
# real check into the rename — announcing a dump nobody counted as verified, and
# then deleting older dumps that were.
SUMMARY_BAD=""
case "$TABLES" in ''|*[!0-9]*) SUMMARY_BAD="yes" ;; esac
case "$DATA_SECTIONS" in ''|*[!0-9]*) SUMMARY_BAD="yes" ;; esac
if [ -n "$SUMMARY_BAD" ]; then
say "VERIFICATION INCONCLUSIVE — the archive was read back successfully, but"
say " counting what is in it did not produce two whole numbers. How much"
say " this dump contains is therefore UNKNOWN."
say ""
say " Unknown is not zero and this is not a statement about the database."
say " Look at engine_summarise, and at whether awk is on PATH — under cron"
say " PATH is not the PATH your shell has."
say " The summariser returned: '${SUMMARY%%$'\n'*}'"
say ""
say " It has NOT been renamed into place and NOTHING was deleted."
say " Kept for inspection: $PART"
if [ -n "$PREV" ]; then
say " The newest verified dump is still $PREV"
else
say " There is no previously verified dump in $BACKUP_DIR_REAL."
fi
exit 1
fi
# Which number is the real one depends on what kind of dump this is, and saying
# "0 tables" about a data-only archive that contains every row would be exactly
# the sort of unmeasured claim this script is built to avoid.
if [ "$TABLES" -gt 0 ]; then
COUNT="$TABLES"
COUNT_KIND="tables"
else
COUNT="$DATA_SECTIONS"
COUNT_KIND="table data sections (the archive has no table definitions — a data-only dump)"
fi
if [ "$COUNT" -lt "$MIN_TABLES" ]; then
say "VERIFICATION FAILED — the archive is readable but holds ${COUNT} table(s),"
say " fewer than BACKUP_MIN_TABLES=${MIN_TABLES}."
say " This is what a dump of the wrong or an empty database looks like:"
say " valid, small, and worthless. The connection was ${TARGET_DESC} —"
say " check which database that actually reaches."
say ""
say " It has NOT been renamed into place and NOTHING was deleted."
say " Kept for inspection: $PART"
exit 1
fi
# ---------------------------------------------------------------------------
# Rename. Within one directory, so it is rename(2): the final name appears
# whole or not at all, and no reader can ever see a partial dump under it.
# ---------------------------------------------------------------------------
mv -- "$PART" "$FINAL" || die "could not rename $PART to $FINAL. The verified dump is still at
$PART — move it into place by hand."
[ -f "$FINAL" ] || die "mv reported success but $FINAL is not there."
say "wrote ${FINAL}"
say " $(human_size "$BYTES"), ${COUNT} ${COUNT_KIND}, verified readable."
# ---------------------------------------------------------------------------
# Compare against the previous dump. A warning, never a refusal — see SHRINK_PCT.
# ---------------------------------------------------------------------------
# Three ways there is no comparison to make, said as three different things.
# Rule of the house: absence is reported as absence, and "I could not measure
# the file I was comparing against" is not "there was no file".
if [ -z "$PREV" ]; then
say "note: no previous dump in this series — size not compared."
elif [ -z "$PREV_MEASURED" ]; then
say "note: the previous dump could not be measured, so this dump's size was"
say " NOT compared against it. Not compared is not the same as unchanged."
say " Previous: $PREV"
elif [ "$PREV_BYTES" -eq 0 ]; then
say "note: the previous dump is zero bytes ($PREV), so there is no ratio to"
say " compare against. Whatever went wrong went wrong before this run."
else
pct=$((BYTES * 100 / PREV_BYTES))
if [ "$pct" -lt "$SHRINK_PCT" ]; then
say "WARNING: this dump is ${pct}% of the previous one"
say " ($(human_size "$BYTES") against $(human_size "$PREV_BYTES"))."
say " A dump that shrinks like that is usually a lost permission on"
say " some schema, a partial dump, or the wrong database. The"
say " previous file has been kept; compare the two table lists with"
say " pg_restore --list before relying on this one."
fi
fi
prune
report_parts
say "note: this wrote a verified dump to THIS machine. It did not copy it"
say " anywhere else, and a backup that only exists beside the database it"
say " came from does not survive the disk, the host or the account."
say " Restoring it has still never been tried; see the header."

868
docs/architecture/scripts/check-env.sh Executable file
View File

@ -0,0 +1,868 @@
#!/usr/bin/env bash
#
# Assert that the environment this project needs is present AND plausible,
# before the thing that needs it starts. Read-only: it measures and it reports.
#
# bash scripts/check-env.sh # check the process environment
# bash scripts/check-env.sh --file .env # check a file instead — only the file
# bash scripts/check-env.sh --quiet # failures only, for calling from a script
# bash scripts/check-env.sh --list # print what is declared, check nothing
#
# Exit codes, because the caller is usually another script deciding whether to
# continue:
#
# 0 every required variable is present, and everything that IS set has the
# right shape
# 1 problems found — they are listed, grouped, with the source named
# 2 NOTHING WAS CHECKED — the spec is empty, or the source could not be
# read. This is not a pass and CI must not treat it as one.
#
# All output goes to stderr, so `check-env.sh --quiet` can be called from the
# middle of another script without polluting that script's stdout.
#
# ===========================================================================
# TEMPLATE COPY — configure this before the first run
# ===========================================================================
#
# Copy to `scripts/check-env.sh`, `chmod +x` it, and fill in the SPEC block
# below — one line per variable. Nothing else needs editing.
#
# SPEC ships empty and the script exits 2 until it has an entry. An empty spec
# would examine nothing and have nothing to report, and the natural way to
# print that is "ok" — a green result from a checker that measured nothing.
# That is the worst possible output, because the whole value of this script is
# that everything downstream stops re-checking and starts trusting it.
#
# There are no default variable names, deliberately, for the reason release.sh
# gives at length: a list inherited from another project is a list of the wrong
# names. It passes — every variable it knows about really is set, in your shell,
# by that other project's tooling — while every variable this project actually
# reads goes unexamined. A checker that is confidently green about the wrong
# environment is worse than no checker.
#
# Assumes: bash, coreutils. It reads nothing from the repository, and the only
# thing it shells out to is `tr` for case folding — no docker, no git, no
# network client — so it runs unchanged inside a scratch container.
#
# ## Why this exists
#
# Absence is the cheap failure. A variable that is not set fails at the first
# line that reads it, loudly, usually before the process finishes starting.
#
# The expensive failure is a variable that is present and the wrong shape,
# because nothing refuses to start:
#
# PORT=3000/tcp a compose port mapping pasted into a port field
# DATABASE_URL=postgres://localhost/app
# right on a laptop; inside a container `localhost`
# is the container, and the database is elsewhere
# NODE_ENV=Production matches no `=== "production"` anywhere, so every
# branch takes its development arm in production
# JWT_SECRET=devsecret the eight characters someone typed to get the dev
# server up, now signing real sessions
# API_URL=https://api.example.com/
# one trailing slash; every joined path is `//v1/...`
#
# All five start. Most of them serve traffic. That is nearly the whole of
# "works on my machine, broken in the container": not a missing variable — a
# variable with the wrong shape, in an environment nobody ever compared against
# the one it was written for. So this checks SHAPE, not just presence.
#
# ## Every problem, in one pass
#
# The alternative is the loop everyone knows: start, crash on one variable, fix
# it, start, crash on the next. Each turn of that loop costs a full boot, and
# on a deploy target it costs a rollback. So nothing here short-circuits — the
# whole spec is evaluated and the report names everything that is wrong at
# once, grouped, with optional-and-absent shown separately because it is not a
# problem and must never read as one.
#
# ## Secrets are measured, never printed
#
# Anything whose name looks like a credential, and anything declared with the
# `secret-min-length` kind, is reported as "set, N characters" and never as its
# value — including inside failure messages, which is where a careless
# validator leaks: the whole point of a failure message is to show you what it
# saw. CI keeps its logs for months and shows them to everyone with read access
# to the repository, so a validator that echoes what it validated has published
# the credential to a wider audience than the breach it was guarding against.
#
# The name pattern errs toward over-matching. A false positive costs one
# unprinted value; a false negative costs a credential in a log.
#
# ## Where to call it
#
# From the container entrypoint before `exec`, from `predev`/`prestart`, and
# from CI before the deploy step. A checker nobody calls is decoration, and the
# environment is exactly the thing that differs between the place you tested
# and the place it broke.
#
# ## What it deliberately does not do
#
# It does not connect to anything. It never opens DATABASE_URL or curls
# API_URL. Reachability depends on a VPN, a network namespace and a machine
# that may not be this one, so a reachability check fails on laptops for
# reasons that are nobody's fault — and a check that cries wolf gets an
# `|| true` appended within a week, taking the shape checks down with it. This
# says the values are plausible. It does not say the services are up.
#
# It does not write, export, repair or default anything, so there is no
# --dry-run: there is no outward-facing act to preview. A checker that fixes
# what it finds is one you stop reading, and then it starts inventing the
# values that reach production.
#
# It does not merge sources. With --file it reads the file and the process
# environment is not consulted at all, which is the point: a variable that is
# set in your shell and missing from the .env is precisely the bug you came
# here to find, and merging hides it.
#
# It does not know your framework and never guesses a name. Every variable it
# checks is one you declared.
set -uo pipefail
# ---------------------------------------------------------------------------
# Snapshot the environment before this script assigns a single variable of its
# own. Nothing above this line may set a variable.
#
# `${!name}` reads a SHELL variable, and this script's own — QUIET, VALUE,
# SOURCE_LABEL, NAME_WIDTH, LIST_ONLY — live in the same namespace as the
# environment it is measuring. Read live rather than from a snapshot, the
# checker measures itself, in both directions and both are wrong:
#
# NAME_WIDTH declared in SPEC, unset in a scrubbed environment, reported
# "ok — present and the right shape: 14" and exited 0. Green for
# a variable nobody set, which is the one failure this file's
# header spends forty lines arguing is worse than no checker.
# QUIET=yes exported by the caller, reported "set, but empty" and exited 1,
# because the assignment below overwrote the value while keeping
# the export attribute. A failure nobody caused, on a variable
# that was correct.
#
# Taken here the snapshot is the environment as this process received it, and
# nothing this script does afterwards can alter what it reports. The _CE_ prefix
# is reserved from SPEC below so the snapshot cannot collide in its turn.
# ---------------------------------------------------------------------------
# `compgen` is a bash builtin, but it belongs to programmable completion and a
# bash built with --disable-progcomp does not have it. Without this the snapshot
# would come back empty and every declared variable would be reported missing:
# a confident, total, wrong answer. Refuse instead — that is a reason this
# script could not measure the environment, which is what exit 2 means. die()
# is not defined this early, so the message is written out longhand.
if ! type -t compgen >/dev/null 2>&1; then
printf '\033[1mcheck-env:\033[0m this bash has no `compgen`, so the environment cannot be\n' >&2
printf '\033[1mcheck-env:\033[0m read reliably and nothing was checked. That is not a pass.\n' >&2
exit 2
fi
_CE_ENV_NAMES=()
_CE_ENV_VALS=()
while IFS= read -r _CE_N; do
[ -n "$_CE_N" ] || continue
# `export FOO` with no value is exported-but-unset: it is not in a child's
# environment either, so it is absence rather than an empty value. `+x`
# keeps set-to-empty, which `docker run -e FOO` produces and which is a
# different bug from unset.
[ -n "${!_CE_N+x}" ] || continue
_CE_ENV_NAMES+=("$_CE_N")
_CE_ENV_VALS+=("${!_CE_N}")
done <<<"$(compgen -e)"
unset _CE_N
# ---------------------------------------------------------------------------
# SPEC — the variables this project needs. One line each, then delete this
# banner. Adding a variable later is a one-line edit and nothing else.
#
# "NAME|required|kind|what breaks when it is missing or wrong"
#
# The description is not documentation. It is the sentence printed next to the
# failure at 03:00, so write the CONSEQUENCE ("sessions cannot be signed, every
# login 500s") rather than a restatement of the name ("the JWT secret").
#
# required | optional — spelled exactly. A typo is refused rather than read as
# "optional", because silently downgrading a required
# variable is the one mistake this script cannot survive.
#
# Kinds:
# nonempty any non-empty string; for values whose shape is
# genuinely unknowable. Not a lazy default — it is the
# honest one, and it still catches unset and empty.
# int a whole number
# int:MIN..MAX a whole number within bounds, inclusive
# port 1-65535
# url scheme://host[...]; rejects a bare `host:5432`
# path a filesystem path; existence is reported only when the
# source is this machine's own environment
# one-of:a,b,c exactly one of these, compared case-sensitively
# secret-min-length:N at least N characters; the value is never printed, and
# it is checked against well-known placeholders
#
# Examples of the FORM. Delete them — they are shapes, not your variables:
#
# "DATABASE_URL|required|url|nothing can read or write; every request 500s"
# "PORT|optional|port|the server binds its own default, which the proxy is not pointed at"
# "NODE_ENV|required|one-of:development,production,test|the wrong branch of every environment check"
# "SESSION_SECRET|required|secret-min-length:32|sessions are forgeable"
# "UPLOAD_DIR|optional|path|uploads land somewhere nothing serves and nothing backs up"
# ---------------------------------------------------------------------------
SPEC=(
)
# ---------------------------------------------------------------------------
# Redaction. See "Secrets are measured, never printed" above.
#
# Matched case-insensitively against the whole variable name, so KEY catches
# API_KEY, KEYCLOAK_SECRET and MONKEY_HOST alike. The last is a false positive
# and costs nothing: its value is described rather than shown.
# ---------------------------------------------------------------------------
SECRET_NAME_PATTERN='SECRET|TOKEN|PASSWORD|PASSWD|PWD|CREDENTIAL|PRIVATE|SALT|SIGNATURE|SIGNING|AUTH|KEY|DSN|COOKIE'
# Values a `secret-min-length` variable must not be, compared in lower case and
# never echoed. These are the strings typed to make the dev server start, which
# then travel to production inside a copied .env and satisfy every length rule.
PLACEHOLDER_SECRETS='changeme change-me change_me changethis secret mysecret supersecret password passwd hunter2 test testing example placeholder todo tbd xxx xxxx admin dev devsecret development your-secret-here your_secret_here notasecret 123456 12345678 abc123'
# Redaction applies to NAMES read out of a file too, not only to values.
#
# A multi-line secret pasted into a .env — a PEM body, a wrapped base64 blob —
# has continuation lines carrying '=' padding, so `KEY=VALUE` parsing splits one
# into a "name" and a "value" and the name is a slice of the credential. Every
# message that prints a name out of a file therefore prints its LENGTH once it
# is longer than any name could plausibly be, which is the same rule values
# already follow. 40 covers the longest real variable names (the framework
# ones reach the low forties); a 64-column PEM line does not come close.
NAME_PRINT_MAX=40
say() { printf '\033[1mcheck-env:\033[0m %s\n' "$*" >&2; }
# die is 2, not the 1 its counterpart in release.sh uses: everything that calls
# it is a reason this script could not measure the environment, and a caller
# must be able to tell "your configuration is wrong" from "I never found out".
die() { printf '\033[1mcheck-env:\033[0m %s\n' "$*" >&2; exit 2; }
usage() {
say "usage: check-env.sh [--file PATH] [--quiet] [--list] [--help]"
say " --file PATH check that file instead of the process environment"
say " --quiet, -q print only failures (exit code still says everything)"
say " --list print the declared spec and check nothing"
say " --help, -h this message"
}
trim() {
local s="$1"
s="${s#"${s%%[![:space:]]*}"}"
s="${s%"${s##*[![:space:]]}"}"
printf '%s' "$s"
}
SOURCE_FILE=""
QUIET=""
LIST_ONLY=""
while [ $# -gt 0 ]; do
case "$1" in
--file) shift; [ $# -gt 0 ] || die "--file needs a path."; SOURCE_FILE="$1" ;;
--file=*) SOURCE_FILE="${1#--file=}"; [ -n "$SOURCE_FILE" ] || die "--file needs a path." ;;
--quiet|-q) QUIET="yes" ;;
--list) LIST_ONLY="yes" ;;
-h|--help) usage; exit 0 ;;
*) usage; die "unknown argument '$1'." ;;
esac
shift
done
# ---------------------------------------------------------------------------
# Read the spec, and refuse anything ambiguous.
#
# A malformed spec entry is a configuration error in this script, not a finding
# about the environment, so it exits 2 and reports nothing about the variables:
# a partial report from a spec that is half-understood is the same lie as a
# green one from an empty spec.
# ---------------------------------------------------------------------------
if [ "${#SPEC[@]}" -eq 0 ]; then
say "SPEC is empty, so nothing was checked. That is not a pass."
say ""
say "This is a fresh template copy: declare the variables this project reads"
say "in the SPEC block near the top of this file, one line each —"
say ' "DATABASE_URL|required|url|nothing can read or write; every request 500s"'
die "then run this again."
fi
SPEC_NAME=()
SPEC_REQ=()
SPEC_KIND=()
SPEC_PARAM=()
SPEC_DESC=()
NAME_WIDTH=4
entry_no=0
for entry in "${SPEC[@]}"; do
entry_no=$((entry_no + 1))
# A description may legitimately contain '|', so read takes only four fields
# and the fourth keeps the rest.
IFS='|' read -r f_name f_req f_kind f_desc <<<"$entry"
f_name=$(trim "${f_name:-}")
f_req=$(trim "${f_req:-}")
f_kind=$(trim "${f_kind:-}")
f_desc=$(trim "${f_desc:-}")
[ -n "$f_name" ] && [ -n "$f_req" ] && [ -n "$f_kind" ] && [ -n "$f_desc" ] \
|| die "SPEC entry ${entry_no} is not NAME|required|kind|description: '${entry}'"
# The shell cannot export a name it cannot parse, so a name that fails this
# could never have been set in the first place and every run would report it
# missing forever.
[[ "$f_name" =~ ^[A-Za-z_][A-Za-z0-9_]*$ ]] \
|| die "SPEC entry ${entry_no}: '${f_name}' is not a usable variable name."
# The environment snapshot has to live somewhere, and a variable it could
# collide with is one this script would measure against itself. Refused
# rather than measured wrong.
case "$f_name" in
_CE_*) die "SPEC entry ${entry_no}: '${f_name}' uses the _CE_ prefix, which this script reserves for the environment snapshot it takes before it defines anything. Rename the variable, or rename the snapshot at the top of this file." ;;
esac
case "$f_req" in
required|optional) : ;;
*) die "SPEC entry ${entry_no} (${f_name}): expected 'required' or 'optional', got '${f_req}'. Spelling matters here — see the SPEC block." ;;
esac
kind_name="${f_kind%%:*}"
kind_param=""
case "$f_kind" in *:*) kind_param="${f_kind#*:}" ;; esac
case "$kind_name" in
nonempty|url|port|path)
[ -z "$kind_param" ] || die "SPEC entry ${entry_no} (${f_name}): kind '${kind_name}' takes no ':' argument." ;;
int)
if [ -n "$kind_param" ]; then
[[ "$kind_param" =~ ^[+-]?[0-9]+\.\.[+-]?[0-9]+$ ]] \
|| die "SPEC entry ${entry_no} (${f_name}): int bounds must look like 1..100, got '${kind_param}'."
fi ;;
one-of)
[ -n "$kind_param" ] || die "SPEC entry ${entry_no} (${f_name}): one-of needs a comma-separated list, e.g. one-of:development,production." ;;
secret-min-length)
[[ "$kind_param" =~ ^[0-9]+$ ]] && [ "$kind_param" -gt 0 ] \
|| die "SPEC entry ${entry_no} (${f_name}): secret-min-length needs a positive number, e.g. secret-min-length:32." ;;
*)
die "SPEC entry ${entry_no} (${f_name}): unknown kind '${kind_name}'. Valid: nonempty, int, port, url, path, one-of:a,b, secret-min-length:N." ;;
esac
# A name declared twice gets two descriptions, and the report would print
# both — one of which is now out of date and neither of which is marked.
for existing in ${SPEC_NAME[@]+"${SPEC_NAME[@]}"}; do
[ "$existing" = "$f_name" ] && die "SPEC declares ${f_name} twice. Keep one line."
done
SPEC_NAME+=("$f_name")
SPEC_REQ+=("$f_req")
SPEC_KIND+=("$kind_name")
SPEC_PARAM+=("$kind_param")
SPEC_DESC+=("$f_desc")
[ "${#f_name}" -gt "$NAME_WIDTH" ] && NAME_WIDTH="${#f_name}"
done
if [ -n "$LIST_ONLY" ]; then
say "declared in SPEC — nothing was checked:"
for (( i = 0; i < ${#SPEC_NAME[@]}; i++ )); do
kind="${SPEC_KIND[$i]}"
[ -n "${SPEC_PARAM[$i]}" ] && kind="${kind}:${SPEC_PARAM[$i]}"
printf ' %-*s %-8s %-22s %s\n' \
"$NAME_WIDTH" "${SPEC_NAME[$i]}" "${SPEC_REQ[$i]}" "$kind" "${SPEC_DESC[$i]}" >&2
done
exit 0
fi
# ---------------------------------------------------------------------------
# Display. Every message that mentions a value goes through show(), and that is
# the only place the redaction rule is enforced — so nothing else in this
# script may interpolate a value into a message. If you add a check, use it.
# ---------------------------------------------------------------------------
is_secret() {
case "$2" in secret-min-length) return 0 ;; esac
local upper
upper=$(printf '%s' "$1" | tr '[:lower:]' '[:upper:]')
[[ "$upper" =~ $SECRET_NAME_PATTERN ]]
}
show() {
local name="$1" kind="$2" value="$3"
if is_secret "$name" "$kind"; then
printf 'set, %d characters (value withheld)' "${#value}"
return 0
fi
case "$kind" in
url)
# Scheme and host are what you need to spot the wrong environment; the
# path and query are where webhook and signed-URL secrets live, so they
# are summarised rather than shown.
local rest authority tail=""
case "$value" in
*://*)
rest="${value#*://}"
authority="${rest%%/*}"
authority="${authority%%\?*}"
case "$authority" in *@*) authority="***@${authority##*@}" ;; esac
case "$rest" in */*|*\?*) tail="/…" ;; esac
printf '%s://%s%s' "${value%%://*}" "$authority" "$tail"
return 0 ;;
esac
;;
# Bounded, enumerable and not credential-shaped by construction. `literal`
# is the mode a failure message asks for when the value IS the finding —
# a port of '3000/tcp' cannot be explained without quoting it.
int|port|one-of|path|literal) : ;;
*)
# Unknown shape, so nothing is known about what might be inside it.
printf 'set, %d characters' "${#value}"
return 0 ;;
esac
if [ "${#value}" -gt 60 ]; then
printf '%s…' "${value:0:59}"
else
printf '%s' "$value"
fi
}
# ---------------------------------------------------------------------------
# Load the source.
# ---------------------------------------------------------------------------
FILE_KEYS=()
FILE_VALS=()
FILE_LINES=()
FILE_BAD=()
FILE_NOTES=()
load_file() {
local path="$1" lineno=0 line trimmed key val
# Redirection, not `cat "$path" |`: a piped while-loop runs in a subshell and
# every array appended inside it is discarded at the `done`.
while IFS= read -r line || [ -n "$line" ]; do
lineno=$((lineno + 1))
case "$line" in
*$'\r')
line="${line%$'\r'}"
FILE_NOTES+=("line ${lineno}|has a CRLF line ending; most loaders keep the carriage return, making the value one invisible character longer than it looks") ;;
esac
trimmed=$(trim "$line")
case "$trimmed" in ''|'#'*) continue ;; esac
case "$trimmed" in
export\ *|export$'\t'*)
trimmed=$(trim "${trimmed#export}")
FILE_NOTES+=("line ${lineno}|uses 'export'; a shell sourcing this file is fine, but docker --env-file reads the name as 'export NAME'") ;;
esac
case "$trimmed" in
*=*) key="${trimmed%%=*}"; val="${trimmed#*=}" ;;
*) FILE_BAD+=("line ${lineno}|no '=', so it sets nothing at all"); continue ;;
esac
if [[ ! "$key" =~ ^[A-Za-z_][A-Za-z0-9_]*$ ]]; then
if [[ "$(trim "$key")" =~ ^[A-Za-z_][A-Za-z0-9_]*$ ]]; then
FILE_BAD+=("line ${lineno}|has spaces around '='; outside a shell those become part of the name and of the value")
elif [ "${#key}" -gt "$NAME_PRINT_MAX" ]; then
FILE_BAD+=("line ${lineno}|has ${#key} characters before its '=', so that is not a name and nothing can read it; the text is withheld in case this is one line of a wrapped multi-line secret")
else
FILE_BAD+=("line ${lineno}|'${key}' is not a usable variable name, so nothing can read it")
fi
continue
fi
# Quotes are stripped here because that is what dotenv-style loaders do —
# but docker --env-file does not, so the two see different values. Said out
# loud rather than resolved, because only the operator knows which loader
# will read this file.
case "$val" in
\"*\"|\'*\')
val="${val:1:${#val}-2}"
FILE_NOTES+=("${key}|is quoted; dotenv-style loaders strip the quotes and docker --env-file keeps them, so the two disagree about this value") ;;
esac
case "$val" in
*'${'*|*'$('*)
FILE_NOTES+=("${key}|contains a \$ expansion; dotenv-style loaders interpolate it and docker --env-file passes it through literally") ;;
esac
local j
for (( j = 0; j < ${#FILE_KEYS[@]}; j++ )); do
if [ "${FILE_KEYS[$j]}" = "$key" ]; then
FILE_NOTES+=("${key}|is set twice (lines ${FILE_LINES[$j]} and ${lineno}); loaders disagree about which one wins")
break
fi
done
FILE_KEYS+=("$key")
FILE_VALS+=("$val")
FILE_LINES+=("$lineno")
done < "$path"
}
if [ -n "$SOURCE_FILE" ]; then
# Refused rather than fallen back on. You asked about a file; answering about
# the process environment instead would be a confident pass from a source you
# did not name — see status.sh on why a tool that answers about the wrong
# thing is worse than one that answers nothing.
[ -e "$SOURCE_FILE" ] || die "--file '${SOURCE_FILE}' does not exist. Refusing to check the process environment instead."
[ -f "$SOURCE_FILE" ] || die "--file '${SOURCE_FILE}' is not a regular file."
[ -r "$SOURCE_FILE" ] || die "--file '${SOURCE_FILE}' is not readable by this user."
load_file "$SOURCE_FILE"
SOURCE_LABEL="${SOURCE_FILE}"
else
SOURCE_LABEL="the process environment"
fi
# The empty string means "not set here". Callers must not read it as "set to
# nothing" — FOUND says which.
FOUND=""
VALUE=""
# The locals are prefixed for the same reason the snapshot is: `lookup`'s own
# `want` and `j` would shadow a SPEC entry of either name, and this is the one
# function that resolves a variable name at runtime.
lookup() {
local _ce_want="$1" _ce_j
FOUND=""
VALUE=""
if [ -n "$SOURCE_FILE" ]; then
# No break: the last assignment wins, which is what a loader reading the
# file top to bottom into a map does.
for (( _ce_j = 0; _ce_j < ${#FILE_KEYS[@]}; _ce_j++ )); do
if [ "${FILE_KEYS[$_ce_j]}" = "$_ce_want" ]; then FOUND="yes"; VALUE="${FILE_VALS[$_ce_j]}"; fi
done
else
# The snapshot taken at the top, never the live variable: see the reasoning
# there. A name absent from the snapshot was absent from the environment,
# whatever this script may since have set in its own namespace.
for (( _ce_j = 0; _ce_j < ${#_CE_ENV_NAMES[@]}; _ce_j++ )); do
if [ "${_CE_ENV_NAMES[$_ce_j]}" = "$_ce_want" ]; then
FOUND="yes"; VALUE="${_CE_ENV_VALS[$_ce_j]}"; break
fi
done
fi
}
# ---------------------------------------------------------------------------
# Shape checks.
#
# check_shape prints zero or more lines, each 'fail:<reason>' or 'note:<reason>'
# — a protocol rather than array appends because it is called in a command
# substitution, and an array appended inside a subshell is gone at the closing
# paren.
# ---------------------------------------------------------------------------
check_shape() {
local name="$1" kind="$2" param="$3" value="$4"
case "$value" in
' '*|$'\t'*) printf 'fail:begins with whitespace, which is part of the value everywhere except a shell\n' ;;
esac
case "$value" in
*' '|*$'\t') printf 'fail:ends with whitespace, which is part of the value and breaks every comparison against it\n' ;;
esac
case "$kind" in
url|port|int|one-of|path)
case "$value" in
*$'\n'*) printf 'fail:contains a newline, so nothing downstream will parse it\n'; return 0 ;;
esac ;;
esac
case "$kind" in
nonempty)
# Presence and non-emptiness are the same test, already done by the
# caller. Reaching here means it passed; there is nothing else to know.
: ;;
int)
if [[ ! "$value" =~ ^[+-]?[0-9]+$ ]]; then
printf 'fail:is not a whole number: %s\n' "$(show "$name" "$kind" "$value")"
return 0
fi
local n="${value#+}"
# Bash arithmetic is 64-bit and errors past it, so an absurd number is
# rejected by length before anything tries to compare it.
if [ "${#n}" -gt 18 ]; then
printf 'fail:has %d digits; nothing downstream will read that as a number\n' "${#n}"
return 0
fi
if [ -n "$param" ]; then
local lo="${param%%..*}" hi="${param##*..}"
if [ "$n" -lt "${lo#+}" ] || [ "$n" -gt "${hi#+}" ]; then
# Through show(), not the bare value: an out-of-range number under a
# credential-shaped name — TOTP_KEY, AUTH_TOKEN_TTL — is still a
# credential, and this is a failure message, which is precisely where
# the header says a careless validator leaks. The bounds come from the
# spec, so they are always safe to print.
printf 'fail:is %s, outside the allowed %s..%s\n' \
"$(show "$name" "$kind" "$value")" "${lo#+}" "${hi#+}"
fi
fi ;;
port)
case "$value" in
*://*) printf 'fail:is a URL, not a port number: %s\n' "$(show "$name" url "$value")"; return 0 ;;
*/*) printf 'fail:looks like a compose port mapping (%s); a port field takes the number alone\n' "$(show "$name" literal "$value")"; return 0 ;;
esac
if [[ ! "$value" =~ ^[0-9]{1,5}$ ]] || [ "$value" -lt 1 ] || [ "$value" -gt 65535 ]; then
printf 'fail:is not a port; ports are 1-65535\n'
return 0
fi ;;
url)
case "$value" in
*://*) : ;;
*)
printf 'fail:has no scheme — %s is a host, not a URL; write it as scheme://host\n' "$(show "$name" literal "$value")"
return 0 ;;
esac
local scheme="${value%%://*}" rest="${value#*://}" authority
if [[ ! "$scheme" =~ ^[A-Za-z][A-Za-z0-9+.-]*$ ]]; then
printf 'fail:has a scheme that is not a scheme\n'
return 0
fi
authority="${rest%%/*}"
authority="${authority%%\?*}"
case "$authority" in *@*) authority="${authority##*@}" ;; esac
if [ -z "$authority" ]; then
printf 'fail:has no host between // and the path\n'
return 0
fi
case "$authority" in
*' '*) printf 'fail:has a space in the host\n'; return 0 ;;
esac
case "$authority" in
localhost|localhost:*|127.0.0.1|127.0.0.1:*|0.0.0.0|0.0.0.0:*|'[::1]'|'[::1]':*)
printf 'note:points at %s — correct on a laptop; inside a container that is the container itself, not the host\n' "$authority" ;;
host.docker.internal*)
printf 'note:points at host.docker.internal, which resolves on Docker Desktop and not on a Linux daemon without an extra_hosts entry\n' ;;
esac
case "$value" in
*/) printf 'note:ends with a slash, so anything joining a path onto it produces a double slash\n' ;;
esac ;;
path)
case "$value" in
'~'*) printf 'fail:starts with ~, and nothing expands a tilde inside an environment variable — a process reading this opens a directory literally named ~\n' ;;
esac
case "$value" in
/*) : ;;
*) printf 'note:is relative, so it resolves against the working directory of whatever starts the process rather than the one you typed it in\n' ;;
esac
# Existence is a fact about THIS machine. Checking it against a file that
# describes another machine would report a fault that is not one, so in
# --file mode it is not checked and the report says so rather than
# leaving a silent gap.
if [ -z "$SOURCE_FILE" ] && [ ! -e "$value" ]; then
printf 'note:does not exist on this machine (this may be correct if the volume is mounted later)\n'
fi ;;
one-of)
local rest="$param" opt matched="" ci_match=""
local lower_value lower_opt
lower_value=$(printf '%s' "$value" | tr '[:upper:]' '[:lower:]')
while [ -n "$rest" ]; do
opt="${rest%%,*}"
[ "$value" = "$opt" ] && matched="yes"
lower_opt=$(printf '%s' "$opt" | tr '[:upper:]' '[:lower:]')
[ "$lower_value" = "$lower_opt" ] && ci_match="$opt"
case "$rest" in *,*) rest="${rest#*,}" ;; *) rest="" ;; esac
done
if [ -z "$matched" ]; then
if [ -n "$ci_match" ]; then
printf 'fail:is %s, and the comparison is case-sensitive: write %s\n' "$(show "$name" one-of "$value")" "$ci_match"
else
printf 'fail:is %s; allowed: %s\n' "$(show "$name" one-of "$value")" "${param//,/, }"
fi
fi ;;
secret-min-length)
# Length is the only property reported. Nothing below prints the value,
# and the comparison below pipes it to `tr` rather than passing it as an
# argument, so it never appears in this process's command line either.
if [ "${#value}" -lt "$param" ]; then
printf 'fail:is %d characters and the spec requires %d (value withheld)\n' "${#value}" "$param"
fi
local lower placeholder
lower=$(printf '%s' "$value" | tr '[:upper:]' '[:lower:]')
for placeholder in $PLACEHOLDER_SECRETS; do
if [ "$lower" = "$placeholder" ]; then
printf 'fail:is a well-known placeholder, not a secret — anyone can guess it (value withheld)\n'
break
fi
done ;;
esac
}
# ---------------------------------------------------------------------------
# Measure everything, then report. Nothing exits early: one run must name every
# problem, or fixing them costs one restart each.
# ---------------------------------------------------------------------------
MISSING=()
MALFORMED=()
NOTES=()
OPT_ABSENT=()
PRESENT_OK=()
for (( i = 0; i < ${#SPEC_NAME[@]}; i++ )); do
name="${SPEC_NAME[$i]}"
req="${SPEC_REQ[$i]}"
kind="${SPEC_KIND[$i]}"
param="${SPEC_PARAM[$i]}"
desc="${SPEC_DESC[$i]}"
lookup "$name"
if [ -z "$FOUND" ]; then
if [ "$req" = "required" ]; then
MISSING+=("${name}|${desc}")
else
OPT_ABSENT+=("${name}|${desc}")
fi
continue
fi
if [ -z "$VALUE" ]; then
if [ "$req" = "required" ]; then
MISSING+=("${name}|set, but empty — ${desc}")
else
# Set-to-empty is not unset, and code tests it both ways: a truthiness
# check sees nothing, a key-presence check sees it. Reported so the
# ambiguity is the operator's to resolve rather than this script's.
OPT_ABSENT+=("${name}|${desc}")
NOTES+=("${name}|is set to the empty string, which is not the same as unset to code that checks whether the key is present")
fi
continue
fi
findings=$(check_shape "$name" "$kind" "$param" "$VALUE")
had_fail=""
while IFS= read -r rline; do
[ -n "$rline" ] || continue
case "$rline" in
fail:*) MALFORMED+=("${name}|${rline#fail:}"); had_fail="yes" ;;
note:*) NOTES+=("${name}|${rline#note:}") ;;
esac
done <<<"$findings"
[ -n "$had_fail" ] || PRESENT_OK+=("${name}|$(show "$name" "$kind" "$VALUE")")
done
if [ -n "$SOURCE_FILE" ]; then
# Only for a file. The process environment carries hundreds of variables from
# the shell, the init system and every tool that ever exported one, so
# "undeclared" there would be noise; in a file every line was written on
# purpose, and one that matches nothing in the spec is usually a typo in a
# name — the thing that makes people swear the value is right there.
for (( j = 0; j < ${#FILE_KEYS[@]}; j++ )); do
fkey="${FILE_KEYS[$j]}"
known=""
for name in ${SPEC_NAME[@]+"${SPEC_NAME[@]}"}; do
[ "$name" = "$fkey" ] && known="yes" && break
done
if [ -z "$known" ]; then
if [ "${#fkey}" -gt "$NAME_PRINT_MAX" ]; then
FILE_NOTES+=("line ${FILE_LINES[$j]}|sets a ${#fkey}-character name that is not declared in SPEC and is too long to be a name at all; the text is withheld in case this is one line of a wrapped multi-line secret")
else
FILE_NOTES+=("${fkey}|is set on line ${FILE_LINES[$j]} but is not declared in SPEC — a misspelled name, or a spec that has fallen behind")
fi
fi
done
fi
group() {
local heading="$1"; shift
[ $# -gt 0 ] || return 0
say ""
say "$heading"
local e
for e in "$@"; do
printf ' %-*s %s\n' "$NAME_WIDTH" "${e%%|*}" "${e#*|}" >&2
done
}
n_missing="${#MISSING[@]}"
n_malformed="${#MALFORMED[@]}"
n_badlines="${#FILE_BAD[@]}"
n_problems=$(( n_missing + n_malformed + n_badlines ))
# `${arr[@]+"${arr[@]}"}` throughout: under `set -u` a plain "${arr[@]}" on an
# empty array is an error in bash before 4.4, and macOS still ships 3.2.
if [ -z "$QUIET" ] || [ "$n_problems" -gt 0 ]; then
say "source: ${SOURCE_LABEL}"
if [ -n "$SOURCE_FILE" ]; then
say " the process environment was NOT consulted, on purpose."
fi
fi
if [ -z "$QUIET" ]; then
group "ok — present and the right shape:" ${PRESENT_OK[@]+"${PRESENT_OK[@]}"}
group "optional, not set — this is fine:" ${OPT_ABSENT[@]+"${OPT_ABSENT[@]}"}
group "worth knowing — not failures:" ${NOTES[@]+"${NOTES[@]}"}
# Kept apart from the value notes above: these are facts about how the FILE
# is written, and they apply to whichever loader reads it rather than to any
# one value being wrong.
if [ -n "$SOURCE_FILE" ]; then
group "about ${SOURCE_FILE} itself:" ${FILE_NOTES[@]+"${FILE_NOTES[@]}"}
fi
if [ -n "$SOURCE_FILE" ]; then
for (( i = 0; i < ${#SPEC_KIND[@]}; i++ )); do
if [ "${SPEC_KIND[$i]}" = "path" ]; then
say ""
say "note: path existence was not checked — a file can describe a machine"
say " that is not this one. Run without --file to check paths here."
break
fi
done
fi
fi
group "MALFORMED — set, but the wrong shape:" ${MALFORMED[@]+"${MALFORMED[@]}"}
group "UNREADABLE LINES — these set nothing:" ${FILE_BAD[@]+"${FILE_BAD[@]}"}
group "MISSING — required, not set:" ${MISSING[@]+"${MISSING[@]}"}
if [ "$n_problems" -gt 0 ]; then
say ""
say "${n_missing} missing, ${n_malformed} malformed, ${n_badlines} unreadable line(s) in ${SOURCE_LABEL}."
# Malformed counts as a failure whether the variable was required or
# optional: not setting an optional variable is a choice, and setting it to
# something unusable is a mistake that reads as a working configuration.
say "fix the entries above and run this again — nothing that reads this"
say "environment can be trusted until then."
# Exit 1, not die's 2. The difference is the whole contract: 1 means this
# measured the environment and it is wrong, 2 means it never got far enough
# to find out. A caller that cannot tell those apart will retry the wrong one.
exit 1
fi
if [ -z "$QUIET" ]; then
say ""
say "ok — ${#PRESENT_OK[@]} set and well-formed, ${#OPT_ABSENT[@]} optional and absent, ${#SPEC_NAME[@]} declared."
say "shapes only: nothing was connected to, so this says the values are"
say "plausible, not that the services behind them are up."
fi
exit 0

120
docs/architecture/scripts/commit-mine.sh Executable file
View File

@ -0,0 +1,120 @@
#!/usr/bin/env bash
#
# Commit only the paths you name, when something else is also writing the tree.
#
# ## The failure this catches
#
# Two agents, or an agent and a person, sharing one checkout share one **git
# index**. Staging is global state, and the window between staging a change and
# committing it is however long it takes to write the commit message. Anything
# that runs `git add -A` inside that window takes your files with it.
#
# Two real instances, one afternoon, one repository — both while the rule
# "stage by explicit path, and re-check the index immediately before committing"
# was being followed to the letter:
#
# - A commit about reviving a game carried two unrelated documentation edits.
# - A commit about art carried an entire renderer fix, a new class, its test
# and two documents. Its message had no `closes #N`, so the issue that work
# finished stayed open and had to be closed by hand afterwards.
#
# Nothing was lost either time. The attribution was wrong, and once the tracker
# was wrong with it. **The rule is not the fix, because the danger is the
# window** — so this closes the window instead: the message is written first and
# passed in, staging and scanning and committing happen back to back, and the
# commit itself names its paths.
#
# ## Why a pathspec commit rather than unstaging theirs
#
# `git commit -- <paths>` takes the working-tree content of exactly those paths
# and ignores the rest of the index. The other writer's staged work is neither
# swept into your commit nor removed from their index, so there is no step here
# that can break *their* commit either. What they have staged is reported, so you
# know somebody else is mid-flight before you add to the race.
#
# ## Usage
#
# bash scripts/commit-mine.sh <message-file> <path> [path…]
# bash scripts/commit-mine.sh --push <message-file> <path> [path…]
#
# ## Exit codes
#
# 0 committed (and pushed, with `--push`)
# 1 the secret scan objected, or a push left something unpushed
# 2 nothing was committed: bad arguments, or a path that does not exist.
# **Two is not a pass**
set -euo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
SECRETS="$HERE/secrets.sh"
PUSH="no"
if [ "${1:-}" = "--push" ]; then PUSH="yes"; shift; fi
if [ "$#" -lt 2 ]; then
sed -n '2,45p' "$0" >&2
exit 2
fi
MESSAGE_FILE="$1"; shift
[ -f "$MESSAGE_FILE" ] || { echo "commit-mine: no message file: $MESSAGE_FILE" >&2; exit 2; }
git rev-parse --is-inside-work-tree >/dev/null 2>&1 || {
echo "commit-mine: not inside a git repository" >&2
exit 2
}
for path in "$@"; do
[ -e "$path" ] || { echo "commit-mine: no such path: $path" >&2; exit 2; }
done
# Whatever anyone else has in flight. Not an error and not touched — but you
# should see it before committing into the same index.
OTHERS="$(git diff --cached --name-only | grep -vxF -f <(printf '%s\n' "$@") || true)"
if [ -n "$OTHERS" ]; then
echo "commit-mine: NOTE — the index also holds work that is not yours:"
printf ' %s\n' $OTHERS
echo " (left staged, and left out of this commit)"
echo
fi
# Staged only so the scanner sees exactly these paths, and only for as long as
# the scan takes. A credential is the one mistake here that cannot be undone by
# a later commit.
git add -- "$@"
if [ -x "$SECRETS" ] || [ -f "$SECRETS" ]; then
set +e
bash "$SECRETS"
SCAN=$?
set -e
# 2 means it scanned nothing, which the convention in this template treats as
# a failure rather than a pass — a scanner that did not run has not cleared
# anything.
if [ "$SCAN" -ne 0 ]; then
git restore --staged -- "$@"
echo "commit-mine: secret scan exited $SCAN; nothing committed" >&2
exit 1
fi
else
echo "commit-mine: WARNING — $SECRETS not found, committing unscanned" >&2
fi
git commit -F "$MESSAGE_FILE" -- "$@"
echo
echo "commit-mine: committed"
git show --stat --oneline HEAD | head -20
if [ "$PUSH" = "yes" ]; then
BRANCH="$(git rev-parse --abbrev-ref HEAD)"
git push origin "$BRANCH"
# Finished work is pushed work. A local commit is invisible to everything that
# reports on the project, so the push is verified rather than assumed.
UNPUSHED="$(git log --oneline "origin/$BRANCH..HEAD" 2>/dev/null || true)"
if [ -n "$UNPUSHED" ]; then
echo "commit-mine: WARNING — still unpushed after push:" >&2
echo "$UNPUSHED" >&2
exit 1
fi
echo "commit-mine: pushed; origin/$BRANCH..HEAD is empty"
fi

219
docs/architecture/scripts/dead-code.py Executable file
View File

@ -0,0 +1,219 @@
#!/usr/bin/env python3
"""Exports nothing imports, and assets nothing renders.
## Why
Dead code is not merely clutter. It is read, maintained, and trusted: somebody
eventually changes it, and nothing happens. On the repository this was written
against, two issues had already been filed by hand for exactly this — four
progress components exported and imported nowhere, and a 1.1 MB image shipping
in every container while being rendered by nothing. Both are the kind of thing a
person finds by accident and a script finds every time.
## What it checks
**Exports.** A symbol exported from a module and referenced nowhere but its own
declaration. Framework entry points are excluded by name, because a Next.js
`page.tsx` exporting `metadata` or a route exporting `GET` is *called by the
framework* and referencing it would be wrong.
**Assets.** A file under the asset roots whose name appears in no source file.
## Tuned for near-silence
`GUARDS.md` rule 5: a check that is wrong six times in ten is one people learn
to skip. So the reference scan covers **tests as well as source** — the first
draft of this scanned only `src/` and reported every test-only helper as dead,
which is the mistake that makes a tool untrustworthy on its first run.
It still cannot see a symbol referenced only by string name, by a build step, or
from outside the repository. Treat findings as candidates, and delete only what
you have confirmed.
python3 scripts/dead-code.py
python3 scripts/dead-code.py --src src --refs src tests --assets public
python3 scripts/dead-code.py --exports-only
Exit codes: 0 nothing found. 1 candidates found. 2 nothing was scanned.
"""
from __future__ import annotations
import argparse
import os
import re
import sys
from collections import Counter
SKIP_DIRS = {"node_modules", ".next", ".git", "dist", "build", "__pycache__"}
CODE_EXTS = (".ts", ".tsx", ".js", ".jsx", ".mjs")
# An asset is referenced from prose and configuration as often as from code — a
# README embeds a screenshot, a manifest names an icon, a stylesheet loads a
# font. Scanning only code reported twelve screenshots as dead when the README
# renders every one of them, which is exactly the first-run noise that ends a
# tool's credibility.
PROSE_EXTS = CODE_EXTS + (".md", ".mdx", ".json", ".yml", ".yaml", ".html", ".css", ".webmanifest")
EXPORT = re.compile(
r"^export\s+(?:async\s+)?(?:function|const|let|class|type|interface|enum)\s+(\w+)"
)
# `export default function FaqPage()` is called by the framework, and the name
# is incidental — it exists so a stack trace reads well. Matching it reported
# every page and layout in the application as dead, which is the kind of first
# run that ends a tool's credibility.
DEFAULT_EXPORT = re.compile(r"^export\s+default\b")
# Called by the framework, never imported. Reporting these would be worse than
# reporting nothing: it teaches the reader that the tool does not understand the
# project, and everything after it gets skipped too.
FRAMEWORK = {
"default", "metadata", "generateMetadata", "generateStaticParams",
"dynamic", "revalidate", "runtime", "fetchCache", "preferredRegion",
"maxDuration", "viewport", "GET", "POST", "PUT", "PATCH", "DELETE",
"HEAD", "OPTIONS", "middleware", "config", "loader", "action",
}
def walk(roots: list[str], exts: tuple[str, ...]) -> list[str]:
found: list[str] = []
for root in roots:
if os.path.isfile(root):
found.append(root)
continue
for base, dirs, names in os.walk(root):
dirs[:] = [d for d in dirs if d not in SKIP_DIRS]
found.extend(
os.path.join(base, n) for n in names if not exts or n.endswith(exts)
)
return found
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__.split("\n")[0])
parser.add_argument("--src", nargs="+", default=["src"], help="where exports live")
parser.add_argument(
"--refs", nargs="+", default=["src", "tests", "app", "scripts", "notices"],
help="where references may appear — include tests, or every helper reads as dead",
)
parser.add_argument("--assets", nargs="+", default=["public"], help="asset roots")
parser.add_argument("--exports-only", action="store_true")
parser.add_argument("--assets-only", action="store_true")
args = parser.parse_args()
sources = walk(args.src, CODE_EXTS)
references = walk([r for r in args.refs if os.path.exists(r)], CODE_EXTS)
if not sources and not args.assets_only:
print("dead-code: nothing to scan.", file=sys.stderr)
return 2
# One pass over every referencing file, counting every identifier and
# remembering the raw text for asset lookups.
counts: Counter[str] = Counter()
corpus: list[str] = []
for path in references:
try:
with open(path, encoding="utf8", errors="ignore") as handle:
body = handle.read()
except OSError:
continue
corpus.append(body)
counts.update(re.findall(r"\b\w+\b", body))
findings = 0
if not args.assets_only:
exported: dict[str, str] = {}
for path in sources:
try:
with open(path, encoding="utf8", errors="ignore") as handle:
for line in handle:
if DEFAULT_EXPORT.match(line):
continue
match = EXPORT.match(line)
if match and match.group(1) not in FRAMEWORK:
exported.setdefault(match.group(1), path)
except OSError:
continue
# Exactly one occurrence is the declaration itself and nothing else.
dead = sorted((name, path) for name, path in exported.items() if counts[name] <= 1)
if dead:
print("exported and referenced nowhere:")
for name, path in dead:
print(f" {name:<32} {path}")
findings += len(dead)
if not args.exports_only:
# A wider net than the export scan uses, for the reason above.
asset_refs = walk([r for r in args.refs if os.path.exists(r)], PROSE_EXTS)
asset_refs += [p for p in walk(["."], (".md",)) if p.count(os.sep) <= 1]
text = "\n".join(corpus)
for path in set(asset_refs):
try:
with open(path, encoding="utf8", errors="ignore") as handle:
text += "\n" + handle.read()
except OSError:
continue
assets = walk([a for a in args.assets if os.path.exists(a)], ())
orphans = []
for path in assets:
name = os.path.basename(path)
# By basename and by path — a reference is written either way, and
# missing one direction is how a false positive gets in.
if name in text or path in text or path.split(os.sep, 1)[-1] in text:
continue
try:
size = os.path.getsize(path)
except OSError:
size = 0
orphans.append((size, path))
if orphans:
print("\nasset files nothing references:" if findings else "asset files nothing references:")
for size, path in sorted(orphans, reverse=True):
print(f" {size / 1024:9.1f} KB {path}")
findings += len(orphans)
if not findings:
print(
f"dead-code: nothing unreferenced across {len(sources)} source "
f"and {len(references)} referencing file(s).",
file=sys.stderr,
)
return 0
print(
f"\ndead-code: {findings} candidate(s). A symbol referenced only by "
"string name, or from outside this repository, will appear here and is "
"not dead — confirm before deleting.",
file=sys.stderr,
)
return 1
if __name__ == "__main__":
raise SystemExit(main())

581
docs/architecture/scripts/deploy.py Executable file
View File

@ -0,0 +1,581 @@
#!/usr/bin/env python3
"""
Deploy automation: update a running stack to a newly published image.
Builds a Docker image from the project directory, tags it with an incremented
version and `latest`, pushes it to the Forgejo Docker registry, then updates the
configured Portainer stack to that image and verifies the container is running
and healthy.
## This is the only copy
It used to exist twice — here and in the `privacyllc-deploy` skill — and the two
drifted apart until neither was a subset of the other (#209). The skill copy had
the credential resolution and the two safety refusals below; this one had the
environment-driven configuration. A defect fixed in one stayed live in the
other, behind a closed issue, because nothing said which was authoritative.
The skill's copy is now a symlink to this file. Anything project-specific must
therefore arrive through the environment, never as a literal: a script copied
from another project must not silently point a deploy at that project's stack —
or, just as quietly, bake that project's canonical origin into somebody else's
image.
Credentials:
- Forgejo registry token: ~/.openclaw/docker-registry.env
- Portainer API key: PORTAINER_API_KEY, or the credential file the operator
keeps Portainer keys in — which is the file named after Portainer, not the
one named after whichever project you are deploying. Looking in the wrong
one is what made every deploy on the source project manual for weeks (#137):
the key was on the machine the whole time, and the error said it was
missing.
Usage:
export PORTAINER_API_URL="https://192.168.1.11:9443/api"
python3 deploy.py [--project-dir PATH] [--portainer-stack-id ID]
[--dry-run] [--build-only] [--skip-push] [--skip-portainer]
"""
import argparse
import json
import os
import re
import subprocess
import sys
import traceback
from pathlib import Path
import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
# ---------------------------------------------------------------------------
# Everything about a particular deployment lives here, and nowhere else.
#
# Set these before the first run. They are read from the environment so a
# checkout carries no deployment's identity — the same reason `release.sh` takes
# its image from RELEASE_IMAGE rather than hard-coding one: a script copied from
# another project must not silently point a deploy at that project's stack.
#
# DEPLOY_PROJECT_DIR the repository to build from
# DEPLOY_IMAGE fully-qualified image name, no tag
# DEPLOY_STACK_ID the Portainer stack that runs it
# DEPLOY_CONTAINER the container name, for the post-deploy check
# DEPLOY_SITE_URL public origin, baked into the image at build time
# DEPLOY_LABEL what to call this thing in output (cosmetic)
# PORTAINER_API_URL e.g. https://portainer.example.com:9443/api
# ---------------------------------------------------------------------------
PROJECT_DIR_DEFAULT = os.environ.get("DEPLOY_PROJECT_DIR", os.getcwd())
FORGEJO_IMAGE = os.environ.get("DEPLOY_IMAGE", "")
FORGEJO_REGISTRY = FORGEJO_IMAGE.split("/")[0] if FORGEJO_IMAGE else ""
PORTAINER_STACK_ID_DEFAULT = int(os.environ.get("DEPLOY_STACK_ID", "0") or 0)
CONTAINER_NAME = os.environ.get("DEPLOY_CONTAINER", "")
# The canonical origin is frozen into the image at build time — every canonical
# URL, the sitemap and robots.txt — and cannot be corrected without another
# build. Both copies of this script hard-coded one project's origin, so
# deploying any other project from it would have baked the wrong one in and
# nothing would have failed. Required, rather than defaulted, for that reason.
SITE_URL = os.environ.get("DEPLOY_SITE_URL", "")
# Cosmetic only: what the [DONE] line calls this. Nothing branches on it.
DEPLOY_LABEL = os.environ.get("DEPLOY_LABEL", "site")
def load_env_file(path: Path) -> dict:
"""Read KEY=VALUE lines, ignoring blank lines and comments."""
values = {}
if not path.exists():
return values
for line in path.read_text().splitlines():
line = line.strip()
if not line or line.startswith("#"):
continue
if "=" in line:
key, value = line.split("=", 1)
values[key.strip()] = value.strip()
return values
def load_privacyllc_credentials() -> dict:
"""Load the privacyllc credential file (KEY=VALUE format)."""
path = Path.home() / ".openclaw" / "credentials" / "privacyllc.md"
return load_env_file(path)
def load_registry_credentials() -> dict:
"""Load the docker registry credential file."""
path = Path.home() / ".openclaw" / "docker-registry.env"
return load_env_file(path)
def get_git_version(project_dir: Path) -> str:
"""Derive the next release version from the latest release commit message.
Release commits are of the form `chore(release): vX.Y.Z`. Returns the next
patch increment (vX.Y.Z+1).
"""
result = subprocess.run(
["git", "log", "--oneline", "--grep=chore(release): v", "-1"],
cwd=project_dir,
capture_output=True,
text=True,
check=True,
)
line = result.stdout.strip()
if not line:
raise RuntimeError("No release commit found; cannot auto-increment version")
match = re.search(r"v(\d+)\.(\d+)\.(\d+)", line)
if not match:
raise RuntimeError(f"Could not parse version from commit: {line}")
major, minor, patch = map(int, match.groups())
return f"v{major}.{minor}.{patch + 1}"
def run(cmd: list[str], cwd: Path | None = None, env: dict | None = None) -> subprocess.CompletedProcess:
"""Run a shell command and return the result."""
merged_env = os.environ.copy()
if env:
merged_env.update(env)
print(f"[RUN] {' '.join(cmd)}")
return subprocess.run(cmd, cwd=cwd, env=merged_env, text=True, capture_output=True)
def build_image(project_dir: Path, version: str, dry_run: bool) -> None:
"""Build the Docker image with the incremented version tag."""
full_tag = f"{FORGEJO_IMAGE}:{version}"
latest_tag = f"{FORGEJO_IMAGE}:latest"
if dry_run:
# Checked here too, so --dry-run reports a missing origin instead of
# printing a confident "would build" for an image that would be wrong.
_require_site_url()
print(f"[DRY-RUN] would build image {full_tag} and {latest_tag}")
return
_require_site_url()
# Clean build to avoid stale Next.js output layers.
print("[BUILD] Cleaning previous build artifacts...")
run(["rm", "-rf", ".next"], cwd=project_dir)
print(f"[BUILD] Building {full_tag}...")
build_result = run(
[
"docker",
"build",
"--no-cache",
"--build-arg",
f"NEXT_PUBLIC_SITE_URL={SITE_URL}",
# The Dockerfile declares ARG APP_VERSION and bakes it into
# ENV APP_VERSION and org.opencontainers.image.version. Omitting it
# here built images with an EMPTY version label, which is not
# cosmetic: buildVersion() then returns null so the admin screen
# reports no running build, and release.sh's deployed_version()
# reads that same label to decide which published tag a prune may
# not delete — an empty label makes prune fail-safe but blind.
#
# A published tag is immutable here (release.sh refuses to move
# one), so an image pushed without the label could never be
# corrected, only abandoned at the cost of a burned version number.
#
# release.sh has always passed this and additionally verifies the
# baked value with printenv before it will push. This is the same
# build arg; the verification is not duplicated because this script
# deploys an existing version rather than cutting one.
"--build-arg",
f"APP_VERSION={version}",
"-t",
full_tag,
"-t",
latest_tag,
".",
],
cwd=project_dir,
)
if build_result.returncode != 0:
print(build_result.stderr)
raise RuntimeError("Docker build failed")
print(build_result.stdout)
def push_image(version: str, dry_run: bool, skip: bool) -> None:
"""Push the version and latest tags to the Forgejo registry."""
if skip:
print("[SKIP] image push disabled by --skip-push")
return
full_tag = f"{FORGEJO_IMAGE}:{version}"
latest_tag = f"{FORGEJO_IMAGE}:latest"
if dry_run:
print(f"[DRY-RUN] would push {full_tag} and {latest_tag}")
return
creds = load_registry_credentials()
registry = creds.get("FORGEJO_REGISTRY", FORGEJO_REGISTRY)
user = creds.get("FORGEJO_REGISTRY_USER", "null")
token = creds.get("FORGEJO_REGISTRY_TOKEN")
if not token:
raise RuntimeError("FORGEJO_REGISTRY_TOKEN not found in ~/.openclaw/docker-registry.env")
print("[PUSH] Logging in to Forgejo registry...")
login = run(
["docker", "login", registry, "-u", user, "--password-stdin"],
env={"DOCKER_CONFIG": os.environ.get("DOCKER_CONFIG", "")},
)
if login.returncode != 0:
# docker login --password-stdin reads from stdin; pass token via input
login = subprocess.run(
["docker", "login", registry, "-u", user, "--password-stdin"],
input=token,
text=True,
capture_output=True,
)
if login.returncode != 0:
print(login.stderr)
raise RuntimeError("Docker login failed")
for tag in (full_tag, latest_tag):
print(f"[PUSH] Pushing {tag}...")
push = run(["docker", "push", tag])
if push.returncode != 0:
print(push.stderr)
raise RuntimeError(f"Docker push failed for {tag}")
print(push.stdout)
print("[PUSH] Logging out from Forgejo registry...")
run(["docker", "logout", registry])
def load_portainer_key(host: str = "nebula") -> str | None:
"""Read one host's Portainer API key out of the credential file.
The key lives here, not in privacyllc.md. Looking only in the latter is
what made every deploy manual for weeks: the credential was on the machine
the whole time, in the file named after the service it belongs to, while
the error said it was "not available".
Parsed rather than run through `load_env_file`, for two reasons. The file
is prose with markdown headings, not KEY=VALUE. And it documents the
variable by example — a literal `PORTAINER_API_KEY=<the-key>` line — which
a KEY=VALUE reader would happily return as the string "<the-key>", giving a
401 that looks like a revoked key rather than a parse mistake.
It also holds more than one host. Sections are `### Portainer (nebula)` and
`### Portainer (exodus)`; this takes the key under the one asked for, so a
deploy cannot authenticate against the wrong machine.
"""
path = Path.home() / ".openclaw" / "credentials" / "portainer.md"
if not path.exists():
return None
section = None
for line in path.read_text(encoding="utf-8").splitlines():
heading = re.match(r"^#{1,6}\s*Portainer\s*\(([^)]+)\)", line.strip())
if heading:
section = heading.group(1).strip().lower()
continue
if section != host.lower():
continue
found = re.search(r"`(ptr_[A-Za-z0-9+/=_-]+)`", line)
if found:
return found.group(1)
return None
def portainer_headers() -> dict:
"""Return request headers with the Portainer API key.
Four places, in the order somebody would expect them to win: an explicit
environment variable, then either credential file. Both env spellings are
honoured because the error message below names both, and promising a
variable the code never reads is how this failed silently — exporting
PORTAINER_API_KEY, the name portainer.md itself documents, did nothing.
"""
token = (
os.environ.get("PORTAINER_API_KEY")
or os.environ.get("PORTAINER_API_TOKEN")
or load_portainer_key()
or load_privacyllc_credentials().get("PORTAINER_API_KEY")
)
if not token:
raise RuntimeError(
"PORTAINER_API_KEY/PORTAINER_API_TOKEN not available. Looked in the "
"environment, ~/.openclaw/credentials/portainer.md and "
"~/.openclaw/credentials/privacyllc.md."
)
return {"X-API-Key": token, "Content-Type": "application/json"}
def portainer_api_url() -> str:
"""Return the Portainer API base URL from environment or default."""
url = os.environ.get("PORTAINER_API_URL", "https://192.168.1.11:9443/api")
return url.rstrip("/")
def update_portainer_stack(version: str, stack_id: int, dry_run: bool, skip: bool) -> None:
"""Update the Portainer stack image line and trigger a redeploy."""
if skip:
print("[SKIP] Portainer update disabled by --skip-portainer")
return
base_url = portainer_api_url()
headers = portainer_headers()
# Fetch current stack file.
file_url = f"{base_url}/stacks/{stack_id}/file"
import requests
file_resp = requests.get(file_url, headers=headers, verify=False, timeout=30)
file_resp.raise_for_status()
compose_text = file_resp.json().get("StackFileContent", "")
if not compose_text:
raise RuntimeError("Portainer stack file content is empty")
# Replace the image line's tag, keeping the line's indentation.
#
# The previous pattern matched `^\s*image:` — indentation included — and
# replaced the whole match with an unindented line. The image line landed at
# column 0, the YAML stopped parsing ("mapping values are not allowed in
# this context"), and Portainer answered 500 before its compose engine ever
# ran — which is why its log carried no entry for the failed update while a
# hand-edit through the UI, indentation intact, deployed fine. Reproduced
# locally against the real stack file and confirmed with
# `docker compose config` before this was written.
#
# `[ \t]*` rather than `\s*`, because `\s` matches newlines and could crawl
# up through blank lines above the target.
image_line = re.compile(
r"^([ \t]*)image:[ \t]*" + re.escape(FORGEJO_IMAGE) + r":(\S+)[ \t]*$",
flags=re.MULTILINE,
)
found = image_line.search(compose_text)
if not found:
raise RuntimeError(
f"Could not find image line for {FORGEJO_IMAGE} in Portainer stack file"
)
if found.group(2) == version:
# Distinguished from "not found": redeploying the version already pinned
# is a no-op somebody may have meant, not a malformed stack file.
print(f"[PORTAINER] stack already pins {version}; sending it unchanged")
updated_compose = image_line.sub(
lambda m: f"{m.group(1)}image: {FORGEJO_IMAGE}:{version}",
compose_text,
)
# The whole point of the last failure: prove the result still parses as
# YAML before it goes anywhere near the stack. `docker compose config`
# validates against the real schema; if the binary is missing the check is
# skipped with a warning rather than silently passed.
import shutil, subprocess, tempfile
if shutil.which("docker"):
with tempfile.NamedTemporaryFile("w", suffix=".yml", delete=False) as handle:
handle.write(updated_compose)
probe_path = handle.name
# --no-interpolate: the stack's variables live in Portainer's Env, not
# in this shell, and `${TURNSTILE_SECRET:?}`-style references would fail
# interpolation here while being perfectly valid there. Structure is
# what the last failure corrupted and structure is checked either way —
# the de-indented file still fails this exact command.
probe = subprocess.run(
["docker", "compose", "-f", probe_path, "config", "--no-interpolate", "-q"],
capture_output=True, text=True, timeout=60,
)
Path(probe_path).unlink(missing_ok=True)
if probe.returncode != 0:
raise RuntimeError(
"The transformed compose does not validate; refusing to send it to "
f"Portainer. docker compose says: {probe.stderr.strip()[:300]}"
)
else:
print("[WARN] docker not on PATH here; skipping compose validation of the transformed file")
# Fetch stack metadata to get endpoint ID.
stack_url = f"{base_url}/stacks/{stack_id}"
stack_resp = requests.get(stack_url, headers=headers, verify=False, timeout=30)
stack_resp.raise_for_status()
stack_data = stack_resp.json()
endpoint_id = stack_data.get("EndpointId")
if not endpoint_id:
raise RuntimeError("Portainer stack response missing EndpointId")
if dry_run:
print(f"[DRY-RUN] would update Portainer stack {stack_id} on endpoint {endpoint_id}")
print(updated_compose)
return
# Update the stack via Portainer API. Preserve existing env vars to avoid
# wiping secrets (POSTGRES_PASSWORD, OIDC_CLIENT_SECRET, etc.).
#
# A stack update REPLACES the definition, so whatever is not sent back is
# gone. This is a round trip of what the GET returned, never a rebuild.
env_vars = stack_data.get("Env") or []
# An empty list is not a stack without variables; it is far more likely a
# read that did not return them — a Portainer version that moved the field,
# a partial response, a token whose scope quietly narrowed. Sending it would
# blank every secret the stack holds, and one of them cannot be recovered by
# re-entering it: without CONTACT_ENCRYPTION_KEY every contact message,
# brainstorming note, document and business detail is permanently
# unreadable. `preflightRestore` refuses a restore for this same reason.
#
# So "I could not read them" and "there are none" are kept apart, which is
# the distinction this deployment refuses to collapse anywhere else. The
# escape hatch exists for a stack that genuinely has none, and it has to be
# asked for by name rather than being what happens by default.
if not env_vars and not os.environ.get("DEPLOY_ALLOW_EMPTY_ENV"):
raise RuntimeError(
f"Portainer stack {stack_id} returned no environment variables. Refusing to "
"update, because sending an empty list would erase every variable the stack "
"holds — including CONTACT_ENCRYPTION_KEY, which cannot be recovered by "
"typing it again. Check the stack in Portainer. If it genuinely has none, "
"re-run with DEPLOY_ALLOW_EMPTY_ENV=1."
)
payload_env = [{"name": e.get("name"), "value": e.get("value")} for e in env_vars]
# Names only. The values are secrets and this prints to a terminal and a log.
print(f"[PORTAINER] carrying {len(payload_env)} env vars: "
f"{' '.join(sorted(e['name'] for e in payload_env if e.get('name')))}")
update_url = f"{base_url}/stacks/{stack_id}?endpointId={endpoint_id}"
payload = {"stackFileContent": updated_compose, "env": payload_env, "prune": False}
update_resp = requests.put(update_url, headers=headers, json=payload, verify=False, timeout=120)
if update_resp.status_code >= 400:
# The body is where Portainer puts the reason, and discarding it is what
# turned the indentation bug into a day of guesswork. Truncated, and it
# carries Portainer's own message rather than anything from the stack's
# environment.
raise RuntimeError(
f"Portainer answered {update_resp.status_code} updating stack {stack_id}: "
f"{update_resp.text[:500]}"
)
print(f"[PORTAINER] Stack {stack_id} updated to {FORGEJO_IMAGE}:{version}")
def verify_container(endpoint_id: int, dry_run: bool) -> dict:
"""Check that the container is running and healthy via Portainer."""
if dry_run:
print(f"[DRY-RUN] would verify container {CONTAINER_NAME} on endpoint {endpoint_id}")
return {"State": "DryRun"}
base_url = portainer_api_url()
headers = portainer_headers()
import requests
url = f"{base_url}/endpoints/{endpoint_id}/docker/containers/json"
resp = requests.get(url, headers=headers, verify=False, timeout=30)
resp.raise_for_status()
containers = resp.json()
for container in containers:
names = container.get("Names", [])
if any(name.lstrip("/") == CONTAINER_NAME for name in names):
state = container.get("State", "")
status = container.get("Status", "")
print(f"[VERIFY] {CONTAINER_NAME}: state={state}, status={status}")
if state != "running":
raise RuntimeError(f"Container {CONTAINER_NAME} is not running (state={state})")
if "healthy" not in status.lower() and "(unhealthy)" in status.lower():
raise RuntimeError(f"Container {CONTAINER_NAME} is unhealthy: {status}")
return container
raise RuntimeError(f"Container {CONTAINER_NAME} not found on endpoint {endpoint_id}")
def _require_config() -> None:
"""Refuse a half-configured run.
The failure this prevents is quiet and expensive: an unset image name makes
every tag `":v1.2.3"`, and an unset stack id makes the update target stack
0. Neither raises anything obviously about configuration, and one of them
could update somebody else's stack.
DEPLOY_SITE_URL is required for the same class of reason but a worse
outcome: it is frozen into the image at build time, so an empty or wrong
value cannot be corrected without another build, and nothing fails at
deploy time to tell you. Only checked when a build will actually happen —
--skip-* runs that never build have no origin to bake.
"""
missing = [
name
for name, value in (
("DEPLOY_IMAGE", FORGEJO_IMAGE),
("DEPLOY_STACK_ID", PORTAINER_STACK_ID_DEFAULT),
("DEPLOY_CONTAINER", CONTAINER_NAME),
)
if not value
]
if missing:
raise SystemExit(
"[ERROR] not configured: " + ", ".join(missing) + "\n"
" See the header of this file; a deploy script that guesses "
"its target is one that deploys to the wrong place."
)
def _require_site_url() -> None:
"""Refuse to build without the origin that will be frozen into the image."""
if not SITE_URL:
raise SystemExit(
"[ERROR] not configured: DEPLOY_SITE_URL\n"
" It is baked into every canonical URL, the sitemap and "
"robots.txt at build time and cannot be corrected afterwards "
"without another build."
)
def main() -> int:
parser = argparse.ArgumentParser(description="Deploy a stack to a newly built image")
parser.add_argument("--project-dir", default=PROJECT_DIR_DEFAULT, help="Path to the project to build")
parser.add_argument("--portainer-stack-id", type=int, default=PORTAINER_STACK_ID_DEFAULT, help="Portainer stack ID")
parser.add_argument("--dry-run", action="store_true", help="Print actions without executing")
parser.add_argument("--build-only", action="store_true", help="Build only; do not push or update Portainer")
parser.add_argument("--skip-push", action="store_true", help="Skip pushing image to registry")
parser.add_argument("--skip-portainer", action="store_true", help="Skip Portainer stack update")
parser.add_argument("--version", help="Override the auto-incremented version tag")
args = parser.parse_args()
_require_config()
project_dir = Path(args.project_dir).expanduser().resolve()
try:
version = args.version or get_git_version(project_dir)
print(f"[DEPLOY] Target version: {version}")
build_image(project_dir, version, args.dry_run)
if args.build_only:
print("[DONE] Build-only requested; stopping after build.")
return 0
push_image(version, args.dry_run, args.skip_push)
update_portainer_stack(version, args.portainer_stack_id, args.dry_run, args.skip_portainer)
if not args.skip_portainer:
# Re-fetch endpoint ID for verification if we did not update Portainer.
import requests
base_url = portainer_api_url()
headers = portainer_headers()
stack_resp = requests.get(
f"{base_url}/stacks/{args.portainer_stack_id}", headers=headers, verify=False, timeout=30
)
stack_resp.raise_for_status()
endpoint_id = stack_resp.json().get("EndpointId")
verify_container(endpoint_id, args.dry_run)
print(f"[DONE] {DEPLOY_LABEL} deployed as {FORGEJO_IMAGE}:{version}")
return 0
except Exception as exc:
print(f"[ERROR] {exc}")
if os.environ.get("DEPLOY_DEBUG"):
traceback.print_exc()
return 1
if __name__ == "__main__":
sys.exit(main())

1282
docs/architecture/scripts/dev.sh Executable file
View File

File diff suppressed because it is too large Load Diff

274
docs/architecture/scripts/doc-claims.sh Executable file
View File

@ -0,0 +1,274 @@
#!/usr/bin/env bash
#
# Every file a document names must exist.
#
# ## The failure this catches
#
# Documentation makes claims about code, and the claims rot silently because
# nothing executes them. Three real instances, all found by hand in one
# afternoon on one repository:
#
# - A source comment stating that `tests/notice-security.test.ts` pinned a
# security rule. **That file had never existed.** The rule was real and
# enforced by nothing, and the sentence had been reassuring every reader who
# checked for two batches.
# - A reference manual whose migration table stopped at 0050 while the
# repository was at 0056. Six migrations behind, and every reader in between
# trusted it.
# - A manual promising a watchdog the code structurally could not fire.
#
# The first two are mechanically checkable and this checks them. The third is
# not — a claim about behaviour needs a person or a test — which is worth
# knowing about this script's limits: **it proves a path exists, never that the
# sentence around it is true.**
#
# ## What it looks at
#
# Anything that looks like a repository path inside backticks or a markdown
# link, in the files you point it at. A path is checked when it looks like one:
# it contains a slash or a known source extension, and it is not a URL, not a
# glob, and not obviously prose.
#
# bash scripts/doc-claims.sh # every tracked .md
# bash scripts/doc-claims.sh docs/ # one tree
# bash scripts/doc-claims.sh README.md # one file
# DOC_CLAIMS_ALSO_SRC=1 bash scripts/doc-claims.sh # also scan source comments
# DOC_CLAIMS_EXCLUDE='' bash scripts/doc-claims.sh # include forward-looking specs
#
# ## The inverse: is everything that exists written down?
#
# bash scripts/doc-claims.sh --covers src/db/migrations --in docs/MANUAL.md
#
# The check above asks whether every path a document *names* exists. This asks
# whether every file that exists is *named* — and it is the one that actually
# bit. A reference manual's migration table stopped at 0050 while the repository
# was at 0056: six rows missing, every path in the document perfectly valid, and
# no existence check can see an absent row.
#
# `docs/proposed/` is skipped by default: a specification naming the files it
# would create is not a claim that they exist.
#
# Exit codes: 0 every named path exists. 1 at least one does not. 2 nothing was
# scanned, which is not a pass — an empty run and a clean run must not look the
# same, for the same reason `verify.sh` refuses to report zero checks as green.
# ---------------------------------------------------------------------------
set -uo pipefail
# --covers <dir> --in <doc>: every file in <dir> must be mentioned in <doc>.
COVERS=""
COVERS_IN=""
ARGS=()
while [ $# -gt 0 ]; do
case "$1" in
--covers) COVERS="${2:-}"; shift 2 ;;
--in) COVERS_IN="${2:-}"; shift 2 ;;
*) ARGS+=("$1"); shift ;;
esac
done
set -- "${ARGS[@]+"${ARGS[@]}"}"
cd "$(git rev-parse --show-toplevel 2>/dev/null)" || {
printf 'doc-claims: not a git repository.\n' >&2
exit 2
}
say() { printf 'doc-claims: %s\n' "$*" >&2; }
if [ -n "$COVERS" ]; then
[ -n "$COVERS_IN" ] || { say "--covers needs --in <document>"; exit 2; }
[ -d "$COVERS" ] || { say "--covers: $COVERS is not a directory"; exit 2; }
[ -f "$COVERS_IN" ] || { say "--in: $COVERS_IN does not exist"; exit 2; }
unmentioned=0
total=0
doc_body="$(cat "$COVERS_IN")"
for entry in "$COVERS"/*; do
[ -e "$entry" ] || continue
total=$((total + 1))
name="$(basename "$entry")"
stem="${name%.*}"
prefix="${stem%%_*}"
# Three spellings, because documents legitimately use all of them and
# demanding the longest reports a perfectly correct document as broken.
# The first draft of this checked only the full name and the stem, and
# reported 57 of 58 migrations missing from a table that lists every one —
# because that table writes `0057`, not `0057_digest_recipients.sql`.
#
# The prefix is only accepted when it is distinctive: a bare `0057` is, a
# bare `route` would not be, and matching on the latter would let a document
# pass by coincidence.
case "$doc_body" in
*"$name"*|*"$stem"*) continue ;;
esac
case "$prefix" in
"$stem") ;; # no underscore; nothing new to try
[0-9][0-9][0-9]*|v[0-9]*)
case "$doc_body" in *"$prefix"*) continue ;; esac ;;
esac
printf '%s: %s is not mentioned in %s\n' "$COVERS" "$name" "$COVERS_IN"
unmentioned=$((unmentioned + 1))
done
if [ "$total" -eq 0 ]; then
say "$COVERS is empty; nothing to cover."
exit 2
fi
if [ "$unmentioned" -gt 0 ]; then
say "$unmentioned of $total entr(ies) in $COVERS are absent from $COVERS_IN."
say "A list that is missing rows reads as complete — that is the whole"
say "problem with it."
exit 1
fi
say "all $total entr(ies) in $COVERS are mentioned in $COVERS_IN."
exit 0
fi
TARGETS=("$@")
if [ ${#TARGETS[@]} -eq 0 ]; then
# Tracked files only. An untracked scratch document is not a claim this
# repository is making.
mapfile -t FILES < <(git ls-files '*.md')
else
mapfile -t FILES < <(git ls-files "${TARGETS[@]}" | grep -E '\.md$')
fi
if [ -n "${DOC_CLAIMS_ALSO_SRC:-}" ]; then
# Source comments make the same claims and rot the same way — the missing
# test file above was named in a docblock, not in a document.
mapfile -t -O "${#FILES[@]}" FILES < <(git ls-files '*.ts' '*.tsx' '*.mjs' '*.js' '*.sh')
fi
# Forward-looking documents are excluded, and this is the difference between a
# useful run and a noisy one. A specification naming the files it *would* create
# is not a rotted claim — it is the whole point of a specification. On the
# repository this was written against, every finding under `docs/proposed/` was
# of that kind and they outnumbered the real ones four to one.
# Two kinds of tree are excluded by default, and both for the same reason: the
# paths in them do not resolve against *this* repository.
#
# docs/proposed/ a specification naming the files it would create
# project-template/ a vendored copy of another project's docs, whose paths
# vendor/ resolve against whatever scaffolds from it
#
# The template copy alone accounted for 14 of 33 findings on the repository this
# was written against — every one of them a README correctly describing scripts
# that live in the template folder rather than here.
EXCLUDE="${DOC_CLAIMS_EXCLUDE:-docs/proposed/|project-template/|vendor/}"
if [ -n "$EXCLUDE" ]; then
mapfile -t FILES < <(printf '%s\n' "${FILES[@]}" | grep -vE "$EXCLUDE" || true)
fi
if [ ${#FILES[@]} -eq 0 ]; then
say "no files to scan."
exit 2
fi
# A token worth checking, and the filter matters more than the check.
#
# The first draft flagged 684 of 1142 tokens on a real repository — routes like
# `/agent/notices`, absolute paths like `~/.config/thing`, and bare filenames.
# A guard that is wrong six times in ten is one people learn to skip, so the
# rule is now deliberately narrow: **a token is only checked when its first
# segment is something that actually exists at the top of this repository.**
#
# That excludes URL routes (their first segment is empty), home-relative paths,
# and prose, and it means a genuinely missing path is reported against a
# background of near-silence.
#
# Bare filenames with a source extension — `api-handler.ts` — are resolved by
# basename anywhere in the tree, which is what a reader would do.
mapfile -t TOPLEVEL < <(git ls-tree --name-only HEAD)
is_toplevel() {
local first="${1%%/*}"
local entry
for entry in "${TOPLEVEL[@]}"; do
[ "$first" = "$entry" ] && return 0
done
return 1
}
worth_checking() {
case "$1" in
http://*|https://*|*@*) return 1 ;; # links and addresses
/*|~*|.*) return 1 ;; # routes, home paths, relative noise
*\**|*\?*|*'<'*|*'>'*|*' '*) return 1 ;; # globs, placeholders, prose
"") return 1 ;;
esac
# A path into this repository.
is_toplevel "$1" && return 0
# Or a bare source filename, resolved by basename below.
case "$1" in
*/*) return 1 ;;
*.ts|*.tsx|*.mjs|*.js|*.sh|*.py|*.sql|*.md) return 0 ;;
*) return 1 ;;
esac
}
resolves() {
local token="$1" doc="$2"
[ -e "$token" ] && return 0
[ -e "$(dirname "$doc")/$token" ] && return 0
# Bare filename: does anything in the repository carry that basename?
case "$token" in
*/*) return 1 ;;
*) git ls-files "*/$token" "$token" | grep -q . && return 0 ;;
esac
return 1
}
missing=0
checked=0
for file in "${FILES[@]}"; do
[ -f "$file" ] || continue
# Backticked spans and markdown link targets. Line numbers and anchors are
# trimmed: `src/lib/foo.ts:42` and `foo.md#heading` name a real file.
while IFS= read -r raw; do
token="${raw%%:*}"
token="${token%%#*}"
token="${token%/}"
worth_checking "$token" || continue
checked=$((checked + 1))
resolves "$token" "$file" && continue
printf '%s: names %s, which does not exist\n' "$file" "$token"
missing=$((missing + 1))
done < <(grep -oE '`[^`]+`|\]\([^)]+\)' "$file" 2>/dev/null \
| sed -E 's/^`//; s/`$//; s/^\]\(//; s/\)$//')
done
if [ "$checked" -eq 0 ]; then
say "scanned ${#FILES[@]} file(s) and found no paths to check."
exit 2
fi
if [ "$missing" -gt 0 ]; then
say "$missing claimed path(s) do not exist, of $checked checked."
say "A document naming a file that is not there is worse than one saying"
say "nothing: somebody checked, and was reassured."
exit 1
fi
say "$checked claimed path(s), all present, across ${#FILES[@]} file(s)."

211
docs/architecture/scripts/doc-triggers.py Executable file
View File

@ -0,0 +1,211 @@
#!/usr/bin/env python3
"""Which documents does a change fire, while there is still time to update them?
python3 scripts/doc-triggers.py # everything dirty right now
python3 scripts/doc-triggers.py --staged # what is staged
python3 scripts/doc-triggers.py <path> [path…] # specific paths
python3 scripts/doc-triggers.py --range HEAD~3..HEAD
## The failure this catches
Every document in this tree carries a `Review trigger:` — the change that should
send somebody back to it — and `WORK_CYCLE.md` requires the triggered documents
to be updated *in the same commit as the code*. Deciding which fired means
reading every `Governs:` line and matching globs in your head, once per commit.
It is a check with no output of its own, so it is the one that gets skipped when
the code is already green, and the cost is invisible until a reader trusts a
document that stopped being true: a reference manual six migrations behind, and
every reader in between believing it.
**This is not the doc-review check.** That one asks which baselined documents are
*overdue*, from committed history — a governed path with a commit newer than the
document's `Last reviewed` date. It is a different question and it can only be
asked after the fact. By the time a change is committed without its document, the
thing this catches has already happened.
Exit status is always 0. This is a prompt, not a gate: a trigger asks a human
whether the prose is still true, and a check that failed the build for that would
be bumped past rather than read.
## Two things it deliberately does not do
**It does not read `Exempt:` declarations.** Those mark a *required* document as
deliberately absent from a repository, and a document that does not exist cannot
govern a path. There is nothing here for them to change.
**It cannot fire a document whose `Governs:` is prose.** Several in this template
govern a subject rather than a set of paths — `GUARDS.md` governs "structural
tests, source-grep assertions, probes, and any check whose passing is taken as
evidence", which is the honest description and matches no glob. Those documents
are listed separately at the end rather than silently ignored, because a reader
who sees only the matched list would reasonably conclude the others were checked
and cleared.
"""
from __future__ import annotations
import fnmatch
import pathlib
import re
import subprocess
import sys
ROOT = pathlib.Path(__file__).resolve().parents[3]
DOCS = ROOT / "docs"
# The status header is a fenced block immediately after the H1, and a long value
# wraps onto continuation lines indented by two spaces:
#
# Governs: structural tests, source-grep assertions, probes, and any check whose
# passing is taken as evidence
#
# A regex that reads one line per field — the obvious implementation — truncates
# at the wrap and silently under-reports, which for this tool means quietly
# failing to name a document that should have been updated. So fields are
# assembled line by line instead.
FIELD_START = re.compile(r"^(Status|Owner|Last reviewed|Governs|Review trigger):\s*(.*)$")
HEADER_LINES = 16
def header_of(doc: pathlib.Path) -> dict[str, str]:
"""The status header, with wrapped values joined."""
try:
lines = doc.read_text(encoding="utf-8").splitlines()[:HEADER_LINES]
except (OSError, UnicodeDecodeError):
return {}
fields: dict[str, str] = {}
current: str | None = None
for line in lines:
match = FIELD_START.match(line)
if match:
current = match.group(1)
fields[current] = match.group(2).strip()
elif current and line.startswith((" ", "\t")) and line.strip():
fields[current] = f"{fields[current]} {line.strip()}".strip()
elif line.strip().startswith("```") and fields:
break
return fields
def looks_like_path(glob: str) -> bool:
"""Whether a `Governs:` entry is a path pattern rather than a subject.
The same test `doc-claims.sh` uses: a slash, a wildcard, or a file
extension. Prose about what a document is authoritative for will have none
of them, and must not be treated as a glob that simply never matches.
"""
glob = glob.strip()
if not glob or " " in glob and "/" not in glob and "*" not in glob:
return False
return "/" in glob or "*" in glob or re.search(r"\.\w{1,5}$", glob) is not None
def matches(path: str, glob: str) -> bool:
"""Whether `path` is governed by `glob`.
`**` means "and everything below", which `fnmatch` does not implement: its
`*` already crosses separators, so `a/**` never matches `a/b/c`. The two
forms these headers actually use are reduced to prefix tests.
"""
glob = glob.strip()
if not glob:
return False
if glob.endswith("/**"):
return path.startswith(glob[:-2]) or path == glob[:-3]
if "/**/" in glob:
head, tail = glob.split("/**/", 1)
return path.startswith(head + "/") and fnmatch.fnmatch(path, "*" + tail)
return fnmatch.fnmatch(path, glob)
def _git(*args: str) -> str:
result = subprocess.run(
["git", *args], cwd=ROOT, capture_output=True, text=True, check=False
)
return result.stdout
def changed_paths(argv: list[str]) -> tuple[list[str], str]:
if argv and argv[0] == "--staged":
return [p for p in _git("diff", "--cached", "--name-only").splitlines() if p], "staged"
if argv and argv[0] == "--range":
if len(argv) < 2:
sys.exit("doc-triggers: --range needs a revision range")
return [p for p in _git("diff", "--name-only", argv[1]).splitlines() if p], f"range {argv[1]}"
if argv:
return list(argv), "named paths"
# Untracked files are included on purpose: a brand-new module is the case
# most likely to need a document and least likely to be remembered, and it
# is invisible to `git diff`.
paths = []
for line in _git("status", "--porcelain").splitlines():
if not line.strip():
continue
path = line[3:].strip()
if " -> " in path: # a rename; the new name governs
path = path.split(" -> ", 1)[1]
paths.append(path.strip('"'))
return paths, "working tree"
def main() -> int:
if not DOCS.is_dir():
print(f"doc-triggers: no docs/ directory at {DOCS}")
return 0
paths, source = changed_paths(sys.argv[1:])
if not paths:
print(f"doc-triggers: nothing changed in the {source}.")
return 0
print(f"doc-triggers: {len(paths)} path(s) from the {source}\n")
fired: list[tuple[str, list[str], str]] = []
subject_only: list[str] = []
for doc in sorted(DOCS.rglob("*.md")):
rel = str(doc.relative_to(ROOT))
header = header_of(doc)
governs = header.get("Governs", "")
if not governs:
continue
globs = [g.strip() for g in governs.split(",") if g.strip()]
path_globs = [g for g in globs if looks_like_path(g)]
if not path_globs:
subject_only.append(rel)
continue
hits = sorted({p for p in paths for g in path_globs if matches(p, g)})
if hits:
fired.append((rel, hits, header.get("Review trigger", "(none stated)")))
for rel, hits, trigger in fired:
print(f"\033[1m{rel}\033[0m")
for hit in hits[:6]:
print(f" {hit}")
if len(hits) > 6:
print(f" … and {len(hits) - 6} more")
print(f" trigger: {trigger}\n")
if fired:
print(f"{len(fired)} document(s) govern something in this change.")
print("Read each trigger and decide — the rule is to update them in the")
print("SAME commit as the code, not afterwards.")
else:
print("No document's Governs matched these paths. Worth a second look if")
print("this change added a module, a migration, or a new boundary — an")
print("unmatched path can also mean no document claims that area yet.")
if subject_only:
print("\nNot checked here — these govern a subject rather than paths, so")
print("no change can fire them mechanically. Judge them yourself:")
for rel in subject_only:
print(f" {rel}")
return 0
if __name__ == "__main__":
raise SystemExit(main())

189
docs/architecture/scripts/duplication.py Executable file
View File

@ -0,0 +1,189 @@
#!/usr/bin/env python3
"""Find code that exists twice.
## Why this is a script and not a code review
Duplication is the thing a reader is worst at. It is invisible unless both
copies are on screen together, and by the time there are three copies nobody
remembers there were ever two. On the repository this was written against, five
separate instances were found by reading in a single afternoon — a four-part
renderer copied into two components, six inlined copies of one ternary, a path
convention written out in four places, a settings key as a literal in three, and
two functions answering the same question with different precedence.
Every one was found by accident. A twenty-line heuristic finds fifty.
## What it reports, and what it deliberately does not
A run of normalised lines that appears in more than one place. Comments, blank
lines and imports are stripped first, so a shared licence header or a block of
imports is not a finding.
It is tuned for **near-silence**, per `GUARDS.md` rule 5: a check that is wrong
six times in ten is one people learn to skip. The defaults require a long run
and a substantial amount of text, so what survives is worth looking at. Loosen
them deliberately when hunting, not by default.
**It cannot tell you whether duplication is wrong.** Two similar-looking blocks
sometimes answer different questions and should stay apart — `notice-sections`
makes that argument about a producer's shape and a wire's shape. This finds
candidates; a person decides.
python3 scripts/duplication.py # default roots
python3 scripts/duplication.py src lib # specific roots
python3 scripts/duplication.py --lines 6 --chars 250 # hunt harder
python3 scripts/duplication.py --ignore vendor/ # skip a tree
Exit codes: 0 nothing above the threshold. 1 duplication found. 2 nothing was
scanned, which is not a pass — an empty run and a clean run must not look the
same.
"""
from __future__ import annotations
import argparse
import hashlib
import os
import re
import sys
from collections import defaultdict
DEFAULT_ROOTS = ["src", "lib", "app", "scripts", "notices"]
DEFAULT_EXTS = (".ts", ".tsx", ".js", ".jsx", ".mjs", ".py", ".go", ".rs")
SKIP_DIRS = {"node_modules", ".next", ".git", "dist", "build", "__pycache__", "vendor"}
# A line that carries no logic. Stripping these is what stops a shared import
# block or a licence header being reported as duplication.
NOISE = re.compile(r"^\s*(//|#|\*|/\*|\*/|import\s|from\s+['\"]|use\s|require\()")
def normalised(path: str) -> list[tuple[int, str]]:
"""Real lines, with their original numbers, whitespace flattened."""
out: list[tuple[int, str]] = []
try:
with open(path, encoding="utf8", errors="ignore") as handle:
for number, raw in enumerate(handle, 1):
line = raw.strip()
if not line or NOISE.match(line):
continue
# Punctuation-only lines — `}`, `});`, `)` — are structure, not
# logic. Left in they make every closing brace look duplicated.
if len(line) < 4:
continue
out.append((number, re.sub(r"\s+", " ", line)))
except OSError:
return []
return out
def collect(roots: list[str], exts: tuple[str, ...], ignore: list[str]) -> list[str]:
files: list[str] = []
for root in roots:
if not os.path.isdir(root):
continue
for base, dirs, names in os.walk(root):
dirs[:] = [d for d in dirs if d not in SKIP_DIRS]
for name in names:
if not name.endswith(exts):
continue
path = os.path.join(base, name)
if any(pattern in path for pattern in ignore):
continue
files.append(path)
return files
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__.split("\n")[0])
parser.add_argument("roots", nargs="*", default=None)
parser.add_argument("--lines", type=int, default=8, help="run length (default 8)")
parser.add_argument("--chars", type=int, default=320, help="minimum text (default 320)")
parser.add_argument("--ignore", action="append", default=[], help="substring to skip")
parser.add_argument("--same-file", action="store_true", help="also report within one file")
args = parser.parse_args()
roots = args.roots or DEFAULT_ROOTS
files = collect(roots, DEFAULT_EXTS, args.ignore)
if not files:
print(f"duplication: nothing to scan in {', '.join(roots)}.", file=sys.stderr)
return 2
blocks: dict[str, list[tuple[str, int]]] = defaultdict(list)
for path in files:
lines = normalised(path)
for index in range(len(lines) - args.lines + 1):
window = lines[index : index + args.lines]
text = " ".join(text for _, text in window)
if len(text) < args.chars:
continue
digest = hashlib.sha1(text.encode("utf8")).hexdigest()
blocks[digest].append((path, window[0][0]))
findings = []
for sites in blocks.values():
distinct_files = {path for path, _ in sites}
if len(distinct_files) > 1 or (args.same_file and len(sites) > 1):
findings.append(sites)
if not findings:
print(
f"duplication: none at {args.lines}+ lines / {args.chars}+ chars "
f"across {len(files)} file(s).",
file=sys.stderr,
)
return 0
# Overlapping windows report the same duplication many times. Keep one
# finding per set of files, at its first site — otherwise a 40-line copy
# produces 33 findings and reads as a disaster.
seen: set[tuple[str, ...]] = set()
shown = 0
for sites in sorted(findings, key=lambda s: -len(s)):
key = tuple(sorted({path for path, _ in sites}))
if key in seen:
continue
seen.add(key)
shown += 1
print("duplicated block:")
for path, line in sorted(sites)[:6]:
print(f" {path}:{line}")
print(
f"\nduplication: {shown} duplicated region(s) across {len(files)} file(s), "
f"at {args.lines}+ lines and {args.chars}+ characters.",
file=sys.stderr,
)
print(
"Not every one is wrong — two blocks can answer different questions and "
"belong apart. These are candidates; you decide.",
file=sys.stderr,
)
return 1
if __name__ == "__main__":
raise SystemExit(main())

586
docs/architecture/scripts/forgejo-issue.py Executable file
View File

@ -0,0 +1,586 @@
#!/usr/bin/env python3
"""Post, list and close Forgejo issues in the convention these projects use.
Exists because every one of the rules below was learned by getting it wrong
once. The script is the enforcement; the skill is the explanation.
create file an issue, refusing one that has no `Verify:` line or that
duplicates an existing title
batch file several from a JSON file, skipping ones already there
list open issues, grouped by milestone
close close with an evidence comment — "Done" is not a close
labels / milestones — what exists, with the ids the API wants
Run any subcommand with --dry-run to see the payload and change nothing.
"""
from __future__ import annotations
import argparse
import json
import os
import re
import subprocess
import sys
import urllib.error
import urllib.parse
import urllib.request
# Where the credentials live, when they are not already in the environment.
#
# Keep this file OUTSIDE the repository. A token in a file the repo can see is
# a token one `git add -A` away from being published — the same argument
# `release.sh` makes about its registry env.
ENV_FILE = os.environ.get("FORGEJO_ENV_FILE", os.path.expanduser("~/.forgejo.env"))
# Cloudflare fronts the Forgejo instance and 1010-blocks Python's default
# urllib User-Agent (browser_signature_banned). Every call fails with a
# Cloudflare HTML body that looks nothing like a Forgejo error. Do not remove.
USER_AGENT = "curl/8.5.0"
SEVERITY = ("P0", "P1", "P2", "release-blocker")
# ── plumbing ─────────────────────────────────────────────────────────────────
def die(msg: str, code: int = 1):
print(f"error: {msg}", file=sys.stderr)
sys.exit(code)
def load_env() -> tuple[str, str]:
"""Read host + token. Never print the token; it is not registry-scoped —
it carries admin/push/pull over the whole API."""
host = os.environ.get("FORGEJO_REGISTRY")
token = os.environ.get("FORGEJO_REGISTRY_TOKEN")
if not (host and token):
try:
with open(ENV_FILE, encoding="utf-8") as fh:
for line in fh:
line = line.strip()
if not line or line.startswith("#") or "=" not in line:
continue
k, v = line.split("=", 1)
v = v.strip().strip("'\"")
if k.strip() == "FORGEJO_REGISTRY" and not host:
host = v
elif k.strip() == "FORGEJO_REGISTRY_TOKEN" and not token:
token = v
except FileNotFoundError:
pass
if not (host and token):
die(
f"FORGEJO_REGISTRY / FORGEJO_REGISTRY_TOKEN not in the environment "
f"or {ENV_FILE}.\n"
f"Set FORGEJO_ENV_FILE to point somewhere else, or export both."
)
return host, token
def detect_repo() -> str | None:
"""owner/name from the git remote of the current directory."""
try:
url = subprocess.run(
["git", "remote", "get-url", "origin"],
capture_output=True, text=True, check=True,
).stdout.strip()
except (subprocess.CalledProcessError, FileNotFoundError):
return None
m = re.search(r"[:/]([^/:]+)/([^/]+?)(?:\.git)?$", url)
return f"{m.group(1)}/{m.group(2)}" if m else None
class Api:
def __init__(self, host: str, token: str, repo: str, dry_run: bool = False):
self.base = f"https://{host}/api/v1"
self.token = token
self.repo = repo
self.dry_run = dry_run
def _call(self, method: str, path: str, body=None, params=None):
url = f"{self.base}{path}"
if params:
url += "?" + urllib.parse.urlencode(params)
data = json.dumps(body).encode() if body is not None else None
req = urllib.request.Request(url, data=data, method=method)
req.add_header("Authorization", f"token {self.token}")
req.add_header("User-Agent", USER_AGENT)
if data:
req.add_header("Content-Type", "application/json")
try:
with urllib.request.urlopen(req, timeout=30) as resp:
raw = resp.read().decode()
return json.loads(raw) if raw.strip() else None
except urllib.error.HTTPError as e:
detail = e.read().decode()[:400]
if "<html" in detail.lower():
detail = ("Cloudflare returned HTML, not a Forgejo error — the "
"User-Agent was probably rejected.")
die(f"{method} {path} → HTTP {e.code}: {detail}")
except urllib.error.URLError as e:
die(f"{method} {path} → {e.reason}")
def get(self, path, params=None):
return self._call("GET", path, params=params)
def post(self, path, body):
if self.dry_run:
print(f"[dry-run] POST {path}\n{json.dumps(body, indent=2)}")
return {"number": "?", "title": body.get("title", ""), "labels": [],
"milestone": None, "_dry": True}
return self._call("POST", path, body=body)
def patch(self, path, body):
if self.dry_run:
print(f"[dry-run] PATCH {path}\n{json.dumps(body, indent=2)}")
return {"_dry": True}
return self._call("PATCH", path, body=body)
# ── repo helpers ─────────────────────────────────────────────────────────
def issues(self, state="open", limit=50):
"""type=issues matters: without it Forgejo returns pull requests too,
and every count drawn from the result is wrong.
Stops on an EMPTY page, not a short one. The server caps a page at 50
however large a `limit` you ask for, so the natural `len(batch) < limit`
test with limit=100 is true on the very first page and the loop exits
having read 50 of 72 open issues — silently, with a plausible-looking
result. That truncation feeds duplicate detection, which is the one thing
this list is for, so a short read re-files work that already exists.
Costs one extra request; cannot truncate.
"""
out, page = [], 1
while True:
batch = self.get(
f"/repos/{self.repo}/issues",
{"type": "issues", "state": state, "limit": limit, "page": page},
) or []
if not batch:
return out
out.extend(batch)
page += 1
def labels(self) -> dict[str, int]:
return {l["name"]: l["id"]
for l in (self.get(f"/repos/{self.repo}/labels",
{"limit": 100}) or [])}
def milestones(self) -> dict[str, int]:
out = {}
for state in ("open", "closed"):
for m in (self.get(f"/repos/{self.repo}/milestones",
{"state": state, "limit": 100}) or []):
out[m["title"]] = m["id"]
return out
def milestones_full(self, state="all") -> list[dict]:
states = ("open", "closed") if state == "all" else (state,)
out = []
for s in states:
out.extend(self.get(f"/repos/{self.repo}/milestones",
{"state": s, "limit": 100}) or [])
return out
def current_milestone(self) -> dict | None:
"""The first OPEN milestone that still has open issues.
'First' is the order Forgejo returns, which is creation order — NOT a
numeric sort by title. A milestone created last therefore cannot become
current while an earlier one still has open issues, which is the lever
for filing work that must not disturb the card.
"""
openms = self.milestones_full("open")
for m in openms:
if m["open_issues"] > 0:
return m
return openms[0] if openms else None
# ── convention checks ────────────────────────────────────────────────────────
def check_verify_line(body: str, title: str) -> str:
"""Every issue ends with a Verify: line stating the acceptance check.
A deliverable nobody can re-test cannot be closed, so it must not be filed."""
lines = [l for l in body.strip().splitlines() if l.strip()]
if not any(l.strip().startswith("Verify:") for l in lines):
die(f'"{title}" has no `Verify:` line. State the acceptance check — a\n'
" finding that cannot be re-tested cannot be closed.")
if not lines[-1].strip().startswith("Verify:"):
print(f'warning: "{title}" has a Verify: line but it is not last',
file=sys.stderr)
return body
def resolve_labels(names, available: dict[str, int]) -> list[int]:
"""Names → ids, failing loudly. A typo'd severity label is reported by the
Command Center as *not adopted*, not as zero defects — silently dropping it
would hide the whole repo's defect count."""
ids = []
for n in names:
# A severity label that differs only in case is the dangerous one: it
# looks right in the UI and is invisible to a query by exact name.
if n not in available:
near = [a for a in available if a.lower() == n.lower()]
if near:
die(f"label {n!r} does not exist, but {near[0]!r} does. "
"Names are matched exactly — use that one.")
die(f"label {n!r} does not exist. "
f"Available: {', '.join(sorted(available)) or '(none)'}")
if n.upper() in {s.upper() for s in SEVERITY} and n not in SEVERITY:
die(f"severity label must be spelled exactly one of {SEVERITY}, "
f"got {n!r}")
ids.append(available[n])
return ids
def find_duplicate(title: str, existing: list) -> dict | None:
t = title.strip().lower()
for i in existing:
if i["title"].strip().lower() == t:
return i
return None
def validate_milestone_title(title: str) -> None:
"""Refuse a comma. `milestones=` takes a comma-separated list of names, so a
title containing one splits into names that do not exist, the filter DROPS,
and the query returns the newest open issue in the WHOLE repository — which
the card then presents as that milestone's next action. Percent-encoding does
not save it; Forgejo decodes before splitting. Measured on null/fruit-fall,
where `0.3.7 Logo, Icons & Branding` was renamed for exactly this reason."""
if "," in title:
die(f"milestone title contains a comma: {title!r}\n"
" That silently breaks the `milestones=` filter and makes the project\n"
" card show the wrong next action. Rename it without the comma.")
first = title.strip().split()[0] if title.strip() else ""
if first and first.replace("v", "", 1).replace(".", "").isdigit():
rest = title.strip()[len(first):].strip()
if rest:
print(f" note: the dashboard phase will show just {first!r} — a title "
f"starting with a\n version token has the rest dropped. Put a "
f"word first to keep it whole.")
_warned_current: set[str] = set()
def _warn_if_current(api: Api, milestone: str) -> None:
"""The dashboard's next action is the NEWEST open issue in the current
milestone — not the highest priority; priority labels have no influence at
all. Filing a routine item into the milestone the team is working on
therefore replaces what the card shows."""
if milestone in _warned_current:
return
_warned_current.add(milestone)
cur = api.current_milestone()
if cur and cur["title"] == milestone:
print(f" warning: {milestone!r} is the CURRENT milestone. The next action on "
"the project\n card is the NEWEST open issue in it, ignoring "
"priority — so this will\n replace whatever is shown there now. "
"To avoid that, file into a\n milestone created later; order is "
"creation order, not title order.")
# ── commands ─────────────────────────────────────────────────────────────────
def cmd_check(api: Api, args):
"""The probes that prove the card is not lying. Run before and after filing.
Everything here exists because Forgejo's filters FAIL OPEN: given a value
they cannot match they ignore the filter and return the unfiltered list, with
no error. A query returning plausible results is not evidence it filtered.
"""
print(f"tracker health — {api.repo}\n")
problems = 0
labels = api.labels()
missing = [n for n in SEVERITY if n not in labels]
if missing:
problems += 1
print(f" FAIL severity labels missing: {', '.join(missing)}")
print(" a query for a label that does not exist matches EVERYTHING")
else:
print(" ok all four severity labels defined")
openi = api.issues(state="open")
blockers = [i for i in openi
if any(l["name"] == "release-blocker" for l in i["labels"])]
if blockers:
print(f" WARN {len(blockers)} open release-blocker — takes over the whole card:")
for i in blockers[:5]:
print(f" #{i['number']} {i['title'][:58]}")
else:
print(" ok no release-blocker hijacking the card")
orphans = [i for i in openi if not i.get("milestone")]
if orphans:
problems += 1
print(f" FAIL {len(orphans)} open issue(s) with no milestone — invisible on the card:")
for i in orphans[:5]:
print(f" #{i['number']} {i['title'][:58]}")
else:
print(f" ok no orphan issues ({len(openi)} open)")
openms = api.milestones_full("open")
commas = [m["title"] for m in openms if "," in m["title"]]
if commas:
problems += 1
print(f" FAIL comma in milestone title — breaks the filter: {commas}")
else:
print(" ok no comma in any open milestone title")
empty = [m["title"] for m in openms if m["open_issues"] == 0]
if empty:
print(f" WARN {len(empty)} open milestone(s) with nothing in them — the card")
print(f" will read 'Close milestone …': {empty[:3]}")
else:
print(" ok no empty open milestones")
cur = api.current_milestone()
if cur:
first = cur["title"].strip().split()[0]
tok = first.replace("v", "", 1).replace(".", "")
print(f"\n current milestone : {cur['title']!r}")
print(f" phase shown : {(first if tok.isdigit() else cur['title'].strip())!r}")
nxt = api.get(f"/repos/{api.repo}/issues",
{"type": "issues", "state": "open", "limit": 1,
"milestones": cur["title"]}) or []
if nxt:
i = nxt[0]
in_ms = (i.get("milestone") or {}).get("title")
if in_ms != cur["title"]:
problems += 1
print(f" next action : {i['title']!r}")
print(" ^ FILTER DROPPED — that issue is in "
f"{in_ms!r}.\n The card is showing a "
"wrong next action.")
else:
print(f" next action : {i['title']!r}")
else:
print("\n no open milestones — the phase would be 'release'")
print(f"\n{'PROBLEMS: ' + str(problems) if problems else 'All checks passed.'}")
return 1 if problems else 0
def cmd_milestone(api: Api, args):
"""Create a milestone. No due date is ever set — a due date means a
commitment, and an invented one is worse than none."""
title = args.title.strip()
validate_milestone_title(title)
if title in api.milestones():
print(f"exists {title!r} — not creating a second one")
return
desc = args.description or ""
if args.description_file:
desc = open(args.description_file, encoding="utf-8").read()
if not desc.strip():
print(" note: no description. It should say what the batch is for and how "
"anybody\n will know it landed.")
m = api.post(f"/repos/{api.repo}/milestones",
{"title": title, "description": desc})
print(f"created milestone {m.get('title', title)!r} (id {m.get('id', '?')})")
def create_one(api: Api, spec: dict, labels_map, ms_map, existing,
allow_dup=False) -> dict | None:
title = spec["title"].strip()
body = check_verify_line(spec.get("body", ""), title)
dup = find_duplicate(title, existing)
if dup and not allow_dup:
print(f"skip #{dup['number']} already titled {title!r} "
f"({dup['state']}) — creates are not idempotent, so this is a skip "
f"not an error")
return None
payload = {"title": title, "body": body}
if spec.get("labels"):
payload["labels"] = resolve_labels(spec["labels"], labels_map)
if "release-blocker" in spec["labels"]:
print(" warning: release-blocker does NOT filter by milestone, so one "
"stray label\n takes over the phase and next action for "
"the entire project. It means\n *nothing else can "
"proceed* — it is not a synonym for important; that is P1.")
if spec.get("milestone"):
m = spec["milestone"]
if m not in ms_map:
die(f"milestone {m!r} does not exist. Available: "
f"{', '.join(sorted(ms_map)) or '(none)'}")
payload["milestone"] = ms_map[m]
_warn_if_current(api, m)
else:
print(f" warning: {title[:48]!r} has no milestone — it can never become the "
"next action\n and never appears anywhere on the project card.")
d = api.post(f"/repos/{api.repo}/issues", payload)
names = ",".join(l["name"] for l in d.get("labels", []))
mile = (d.get("milestone") or {}).get("title", "—")
print(f"filed #{d['number']} [{names}] {d['title']}"
+ (f" → {mile}" if mile != "—" else ""))
return d
def cmd_create(api: Api, args):
labels_map, ms_map = api.labels(), api.milestones()
existing = api.issues(state="all")
body = args.body
if args.body_file:
body = open(args.body_file, encoding="utf-8").read()
create_one(api, {"title": args.title, "body": body or "",
"labels": args.label, "milestone": args.milestone},
labels_map, ms_map, existing, args.allow_duplicate)
def cmd_batch(api: Api, args):
specs = json.load(open(args.file, encoding="utf-8"))
if isinstance(specs, dict):
specs = specs.get("issues", [])
if not isinstance(specs, list):
die("batch file must be a JSON list, or an object with an 'issues' list")
labels_map, ms_map = api.labels(), api.milestones()
existing = api.issues(state="all")
filed = 0
for spec in specs:
d = create_one(api, spec, labels_map, ms_map, existing,
args.allow_duplicate)
if d:
filed += 1
existing.append({"number": d["number"], "title": d["title"],
"state": "open"})
print(f"\n{filed} filed, {len(specs) - filed} skipped")
def cmd_list(api: Api, args):
issues = api.issues(state=args.state)
groups: dict[str, list] = {}
for i in issues:
groups.setdefault((i.get("milestone") or {}).get("title",
"(no milestone)"),
[]).append(i)
for m in sorted(groups):
print(f"\n### {m}")
for i in sorted(groups[m], key=lambda x: x["number"]):
names = ",".join(l["name"] for l in i["labels"])
print(f" #{i['number']:<4} [{names}] {i['title']}")
print(f"\n{len(issues)} {args.state} issue(s) — pull requests excluded")
def cmd_close(api: Api, args):
"""Close with the evidence that proves it: a path, a symbol, a test name,
or the command that shows it. 'Done' is not a close."""
ev = args.evidence.strip()
if len(ev) < 15:
die("evidence too thin. Give a path, a symbol, a test name, or the "
"command that proves it — 'Done' is not a close.")
api.post(f"/repos/{api.repo}/issues/{args.number}/comments", {"body": ev})
api.patch(f"/repos/{api.repo}/issues/{args.number}", {"state": "closed"})
print(f"closed #{args.number} with evidence")
print("note: prefer `closes #N` in the commit that does the work — the "
"tracker then records who and when from the thing that happened.")
def cmd_labels(api: Api, args):
labels = api.labels()
print("severity (exact names — queried by name by the Command Center):")
for s in SEVERITY:
print(f" {'✓' if s in labels else '✗ MISSING'} {s}"
+ (f" id={labels[s]}" if s in labels else ""))
print("\nother:")
for n, i in sorted(labels.items()):
if n not in SEVERITY:
print(f" {n} id={i}")
def cmd_milestones(api: Api, args):
for state in ("open", "closed"):
ms = api.get(f"/repos/{api.repo}/milestones",
{"state": state, "limit": 100}) or []
if ms:
print(f"\n{state}:")
for m in ms:
print(f" [{m['id']}] {m['title']} — open {m['open_issues']} / "
f"closed {m['closed_issues']}")
def main():
# --repo and --dry-run are accepted on BOTH sides of the subcommand. Putting
# them only on the top-level parser means `… create "T" --dry-run` — the
# natural way to type it, and the position that matters most — dies with an
# argparse usage error instead of previewing. A safety flag that is easy to
# put in the wrong place is a safety flag that gets left off.
# default=SUPPRESS is load-bearing, not tidiness. With a normal default the
# subparser re-defines the same dest and argparse writes its default over
# whatever the top-level parser already parsed — so `--repo X create …`
# silently became repo=None and `--dry-run create …` silently became False.
# A --dry-run that quietly turns itself off is the worst possible bug in a
# tool whose job is writing to a live tracker. SUPPRESS leaves the attribute
# unset unless it was actually given, so neither position clobbers the other.
common = argparse.ArgumentParser(add_help=False)
common.add_argument("--repo", default=argparse.SUPPRESS,
help="owner/name (default: from git remote)")
common.add_argument("--dry-run", action="store_true",
default=argparse.SUPPRESS,
help="print payloads, change nothing")
p = argparse.ArgumentParser(
parents=[common],
description="File Forgejo issues in the tracker convention.",
epilog="Every open issue is a denominator. Do not pad the tracker.")
sub = p.add_subparsers(dest="cmd", required=True)
c = sub.add_parser("create", parents=[common], help="file one issue")
c.add_argument("title")
c.add_argument("--body", help="issue body; must end with a Verify: line")
c.add_argument("--body-file", help="read the body from a file")
c.add_argument("--label", action="append", default=[],
help="label name, repeatable")
c.add_argument("--milestone", help="milestone title")
c.add_argument("--allow-duplicate", action="store_true")
c.set_defaults(fn=cmd_create)
b = sub.add_parser("batch", parents=[common], help="file several from a JSON file")
b.add_argument("file")
b.add_argument("--allow-duplicate", action="store_true")
b.set_defaults(fn=cmd_batch)
l = sub.add_parser("list", parents=[common], help="open issues by milestone")
l.add_argument("--state", default="open",
choices=["open", "closed", "all"])
l.set_defaults(fn=cmd_list)
x = sub.add_parser("close", parents=[common], help="close with an evidence comment")
x.add_argument("number", type=int)
x.add_argument("evidence", help="what was checked and where")
x.set_defaults(fn=cmd_close)
m = sub.add_parser("milestone", parents=[common], help="create a milestone (batch)")
m.add_argument("title")
m.add_argument("--description")
m.add_argument("--description-file")
m.set_defaults(fn=cmd_milestone)
sub.add_parser("check", parents=[common], help="health probes — run before AND after filing"
).set_defaults(fn=cmd_check)
sub.add_parser("labels", parents=[common], help="labels and their ids").set_defaults(
fn=cmd_labels)
sub.add_parser("milestones", parents=[common], help="milestones and their ids").set_defaults(
fn=cmd_milestones)
args = p.parse_args()
# getattr, because SUPPRESS means the attribute may legitimately be absent.
repo = getattr(args, "repo", None) or detect_repo()
if not repo:
die("could not detect owner/name from the git remote — pass --repo")
host, token = load_env()
rc = args.fn(Api(host, token, repo, getattr(args, "dry_run", False)), args)
# `check` returns a count so it can gate a script; the rest return None.
sys.exit(rc or 0)
if __name__ == "__main__":
main()

119
docs/architecture/scripts/healthcheck.sh Executable file
View File

@ -0,0 +1,119 @@
#!/usr/bin/env bash
#
# Liveness tick for a deployed service.
#
# ## Why this exists
#
# The five-minute healthcheck it replaced was prose handed to a model: "GET
# <origin>/healthz. Do NOT use /api/internal/v1/healthz." It duly reported a 404
# on `/api/internal/v1/health` — a third path, neither the one it was told to
# use nor the one it was told to avoid, and one that had never existed. The site
# was healthy throughout.
#
# An explicit prohibition constrained one wrong URL and left every other wrong
# URL open, because the address was being re-derived on every run rather than
# read. So it is written here once, as a string in version control, and the
# whole class of failure goes with it.
#
# That failure is the expensive kind. The job's own state read `lastRunStatus:
# ok, consecutiveErrors: 0` while a red alert went to a DM — so the monitor was
# reporting itself healthy and crying wolf at the same time, and a monitor
# nobody believes is a monitor nobody has.
#
# ## It holds no credential, and that is the point
#
# `/healthz` is unauthenticated by design — `SECURITY_CHECKLIST.md` makes it a
# checklist item, and the container's own HEALTHCHECK uses it. So unlike
# `reconcile.sh` and `analyze.sh`, this script reads no token, sources no env
# file, and has nothing to leak. At 288 runs a day that is worth more than the
# extra assurance an authenticated probe would buy.
#
# The authenticated sibling, `/api/internal/v1/agent/health`, reports more and
# needs a token. It is deliberately not used here: this asks "is the site up",
# which is a question with a public answer.
#
# ## Both halves are checked
#
# 503 is a real answer, not an outage — the route returns it when migrations
# failed, so the container is marked unhealthy while the marketing pages keep
# serving. Its body still contains `"service"`. Status alone is therefore not a
# verdict, and neither is a grep for the service name; the check asserts HTTP
# 200 *and* `"ok":true`.
#
# ## Installing it
#
# install -m 0755 healthcheck.sh ~/bin/healthcheck.sh
# # then, in the crontab — every five minutes
# */5 * * * * /home/kaspa/bin/healthcheck.sh \
# >> /home/kaspa/.healthcheck.log 2>&1
#
# On this deployment it is run by an OpenClaw cron job instead, which is the
# same thing with a scheduler that can also deliver the alert.
set -euo pipefail
# `-`, not `:-`. Unset means "no opinion, use production". Set-and-empty means
# a config is wrong, and substituting production for it would report the health
# of a site nobody asked about — quietly, and only where somebody was trying to
# point this somewhere else.
BASE_URL="${HEALTHCHECK_BASE_URL:?set HEALTHCHECK_BASE_URL to the deployed origin, e.g. https://example.com}"
TIMEOUT="${PRIVACY_TIMEOUT:-15}"
stamp() { date -Is; }
say() { printf '%s healthcheck: %s\n' "$(stamp)" "$*"; }
if [ -z "$BASE_URL" ]; then
say "FAIL no base URL. HEALTHCHECK_BASE_URL is set but empty; unset it for the default."
exit 78 # EX_CONFIG
fi
body=$(mktemp)
trap 'rm -f "$body"' EXIT
# Assigned in the `if`, not with `|| echo "000"` appended. On a connection
# failure curl *already* prints "000" via --write-out and then exits non-zero,
# so appending a fallback produces "000000", which matches no branch below and
# reports "unexpected HTTP" for the one failure this script names explicitly.
#
# No -v and no --trace, ever. This request carries no credential, but the habit
# is the rule SECURITY.md states: a request URL or header must not reach a log.
if ! status=$(
curl --silent --show-error --output "$body" --write-out '%{http_code}' \
--max-time "$TIMEOUT" \
"$BASE_URL/healthz"
); then
status="000"
fi
case "$status" in
200)
# The status got us here; the body decides. `"ok":true` is asserted rather
# than assumed because a 200 from a proxy, a cached page or a captive portal
# is still a 200, and none of them are this application answering.
if grep -q '"ok":true' "$body"; then
say "ok"
else
say "FAIL 200 but not ok. The response did not contain \"ok\":true."
exit 70 # EX_SOFTWARE
fi
;;
503)
# Named rather than left to the catch-all, because it is the one unhealthy
# answer this route is designed to give: migrations failed, the container is
# marked unhealthy, and the public pages are still being served. That is a
# different thing from the site being down and reads differently at 3am.
say "FAIL migrations are not ok (HTTP 503). The site is serving; the schema is not."
exit 70 # EX_SOFTWARE
;;
000)
say "FAIL could not reach $BASE_URL"
exit 69 # EX_UNAVAILABLE
;;
*)
# Includes 404. If this ever fires on a path this script wrote itself, the
# route moved — which is a thing to fix in one place rather than a thing for
# a caller to guess around.
say "FAIL unexpected HTTP $status from $BASE_URL/healthz"
exit 1
;;
esac

1066
docs/architecture/scripts/migrate.sh Executable file
View File

File diff suppressed because it is too large Load Diff

151
docs/architecture/scripts/prove-guard.sh Executable file
View File

@ -0,0 +1,151 @@
#!/usr/bin/env bash
#
# Prove a guard fails before you believe it passes.
#
# ## The failure this catches
#
# A guard that cannot fail is worse than no guard, because it is trusted.
# `docs/architecture/GUARDS.md` opens with that sentence and its first rule is
# this procedure, written out as a manual recipe: back the file up, break exactly
# the thing the guard protects, run the guard, expect one failure, restore.
#
# The recipe is thirty seconds and it is skipped anyway, for two reasons this
# script removes:
#
# - **Restoring is a step you can forget**, and forgetting is silent. The tests
# pass again once the mutation is undone in your head but not on disk, so the
# reverted code ships looking green. Here the restore is a `trap`, which runs
# on success, on failure, and on Ctrl-C.
# - **Counting the failures is the part people skip.** GUARDS.md §1: "If
# breaking the guard's target fails three tests, two of them are coincidental
# and will mask a real regression later." A human doing this by hand sees red
# and stops reading.
#
# ## Usage
#
# bash scripts/prove-guard.sh <file> <find> <replace> <test command…>
#
# bash scripts/prove-guard.sh src/lib/thing.ts \
# 'if (body.error)' 'if (false)' \
# npx vitest run tests/thing.test.ts
#
# Everything after the third argument is the command that runs the guard, so any
# runner works. `$PROVE_GUARD_CMD` is used when no command is given.
#
# ## Counting the failures
#
# "Exactly one" is a claim about test *cases*, and counting matching log lines
# does not measure that: Gradle reports a single failing test on six lines — the
# task, the test, its assertion, the summary, and twice more for the build — and
# a naive count calls that six coincidental failures. Tried that first; it fired
# on the very first run against a guard that was behaving perfectly.
#
# So the summary line is preferred, because almost every runner prints one and it
# is the runner's own count: `1 failed` from vitest, `6 tests completed, 1 failed`
# from Gradle, `1 failed, 5 passed` from pytest. The **last** such line wins, and
# only if none is found does it fall back to counting lines matching
# `$PROVE_GUARD_FAIL_PATTERN` — saying so, because an approximate count presented
# as an exact one is the kind of thing this script exists to object to.
#
# ## Exit codes
#
# 0 the guard caught it, and nothing else did — the outcome you want
# 1 the guard stayed GREEN with its target broken. It is not testing what you
# think it is, and you have just learned that for the price of one edit
# 2 nothing was proven: bad arguments, missing file, or a find-string that is
# absent or ambiguous. **Two is not a pass**
# 3 the guard caught it, but so did something else. Red for more than one
# reason hides the next regression behind a failure you have learned to
# expect — narrow the guard, or the mutation
#
# The file is restored in every one of those cases.
set -euo pipefail
FAIL_PATTERN="${PROVE_GUARD_FAIL_PATTERN:-(FAIL|✗|[0-9]+ (tests? )?failed|FAILED|AssertionError)}"
if [ "$#" -lt 3 ]; then
sed -n '2,30p' "$0" >&2
exit 2
fi
FILE="$1"; FIND="$2"; REPLACE="$3"; shift 3
if [ "$#" -gt 0 ]; then
CMD=("$@")
elif [ -n "${PROVE_GUARD_CMD:-}" ]; then
# shellcheck disable=SC2206
CMD=($PROVE_GUARD_CMD)
else
echo "prove-guard: no test command given and PROVE_GUARD_CMD is unset." >&2
echo "Nothing was proven, which is not the same as nothing being wrong." >&2
exit 2
fi
[ -f "$FILE" ] || { echo "prove-guard: no such file: $FILE" >&2; exit 2; }
BACKUP="$(mktemp)"
cp "$FILE" "$BACKUP"
restore() {
cp "$BACKUP" "$FILE"
rm -f "$BACKUP"
echo "prove-guard: restored $FILE"
}
trap restore EXIT INT TERM
# Exact-string replacement, and it must be unique. A mutation that lands in two
# places proves nothing about either, and a regex here would make the mutation
# itself the thing to debug.
python3 - "$FILE" "$FIND" "$REPLACE" <<'PY'
import sys
path, find, replace = sys.argv[1], sys.argv[2], sys.argv[3]
text = open(path, encoding="utf-8").read()
count = text.count(find)
if count == 0:
sys.exit(f"prove-guard: the string to break is not in {path}")
if count > 1:
sys.exit(
f"prove-guard: {count} occurrences of that string; a mutation in "
"two places proves neither. Pick a longer, unique one."
)
open(path, "w", encoding="utf-8").write(text.replace(find, replace))
PY
LOG="$(mktemp)"
trap 'restore; rm -f "$LOG"' EXIT INT TERM
echo "prove-guard: broke $FILE — expecting '${CMD[*]}' to go red"
echo
if "${CMD[@]}" >"$LOG" 2>&1; then
echo "prove-guard: FAILED — the guard stayed GREEN with its target broken." >&2
echo >&2
echo "It is not checking what you think. Either the assertion does not reach" >&2
echo "the mutated code, or it would pass without it. Log: $LOG" >&2
tail -20 "$LOG" >&2
exit 1
fi
echo "--- what failed ---"
grep -E "$FAIL_PATTERN" "$LOG" | head -12 || true
echo
# The runner's own count, from the last summary line that states one. Preferred
# over counting log lines for the reason in the header: one failing test is
# routinely reported on half a dozen lines.
COUNT="$(grep -oiE '[0-9]+ (tests? )?failed' "$LOG" | tail -1 | grep -oE '^[0-9]+' || true)"
COUNTED_BY="the runner's summary"
if [ -z "$COUNT" ]; then
COUNT="$(grep -cE "$FAIL_PATTERN" "$LOG" || true)"
COUNTED_BY="matching log lines, approximately — this runner printed no summary"
fi
if [ "$COUNT" -gt 1 ]; then
echo "prove-guard: the guard caught it — but $COUNT failures, by $COUNTED_BY."
echo
echo "GUARDS.md §1: if breaking one thing fails three tests, two are"
echo "coincidental and will mask a real regression later behind a red you have"
echo "learned to expect. Narrow the guard, or narrow the mutation."
exit 3
fi
echo "prove-guard: good — the guard caught it, and only it ($COUNTED_BY)."

605
docs/architecture/scripts/release-notes.mjs Executable file
View File

@ -0,0 +1,605 @@
#!/usr/bin/env node
/**
* Tags a release and records it on Forgejo, with notes built from the commits.
*
* ## Why this exists
*
* `release.sh` published an image and stopped. Forty-eight versions were cut
* that way, and the only record any of them existed was a registry entry and a
* `chore(release):` commit — so answering "what changed between these two
* images" meant reading the log by hand. This writes that answer down once, at
* the moment the facts are still known.
*
* ## The boundary is the previous release commit, not a tag
*
* There were no tags at all when this was written, so "since the last tag"
* had nothing to start from. Every release leaves a `chore(release): vX.Y.Z`
* commit, and the one before this release is exactly the range wanted — which
* works identically before and after tags exist, and needs no state file.
*
* ## Why the subjects are scrubbed before they are published
*
* The body goes to a **public** repository and is assembled from commit
* subjects nobody wrote with that in mind. `redact` is the same scrubber the
* daily standup uses, imported rather than reimplemented — a credential-shaped
* string in a subject would otherwise be published verbatim, and nobody would
* notice.
*
* `noDashes` is deliberately *not* applied, though `cleanQuoted` bundles it:
* that is a Discord house-style rule, and the subjects in this repository use
* em dashes on purpose.
*
* ## What it refuses to do
*
* Move a tag. `release.sh` already refuses to overwrite a published image tag —
* "a published tag is not moved" — and a git tag pointing at a different commit
* is the same fact about a different artifact. Re-running after a partial
* failure is expected and safe; rewriting history is not.
*
* scripts/release-notes.mjs v0.54.31
* scripts/release-notes.mjs v0.54.31 --dry-run
*/
import { execFileSync } from "node:child_process";
import { readFileSync } from "node:fs";
/**
* The scrubber and the escaper, carried rather than imported.
*
* This file used to `import { escapeMarkdown, redact } from "../notices/compose.mjs"`,
* which is fine here and fatal the moment the script is copied: the Template
* has no `notices/`, and a release script that cannot resolve its own imports
* is one nobody runs. `compose.mjs` earns its portability by having no imports
* at all; this now does the same.
*
* `REDACTIONS` is therefore a third copy of the list in `src/lib/log.ts` and
* `notices/compose.mjs`, and a third copy of a credential scrubber is exactly
* the kind of thing that rots quietly. `tests/release-notes.test.ts` extracts
* this block and asserts it is identical to the one the worker holds — the same
* device `tests/notice-message.test.ts` already uses for the second copy. Two
* lists that drift are the danger; lists a test refuses to let drift are a copy.
*
* This matters more here than almost anywhere: the body is assembled from
* commit subjects nobody wrote expecting publication, and it is published to a
* repository that is public.
*/
const REDACTIONS = [
[/\/\/[^/@\s:]+:[^/@\s]+@/g, "//<redacted>@"],
[/([?&](?:token|key|secret|password|access_token|api_key)=)[^&\s]+/gi, "$1<redacted>"],
[/pllc_[a-z]+_[0-9a-f]{8,}/g, "<redacted-token>"],
[/v2:[0-9a-f]{8}:[^\s"]+/g, "<redacted-envelope>"],
[/\b(Bearer|token|Basic)\s+[A-Za-z0-9._~+/=-]{12,}/gi, "$1 <redacted>"],
[/\b[0-9a-f]{40,}\b/gi, "<redacted-hex>"],
];
function redact(text) {
return REDACTIONS.reduce((out, [pattern, replacement]) => out.replace(pattern, replacement), text);
}
/**
* Newlines flattened, markup escaped — `compose.mjs`'s reasoning, unchanged.
*
* `#`, `>` and `-` are markup only at the start of a line, so escaping them
* inline would turn `My-Project` into `My\-Project` in every message.
* Flattening to one line is what actually removes the risk. `[` and `(` are the
* pair that matter most: they are how a subject becomes a working link.
*/
function escapeMarkdown(text) {
return text.replace(/\r?\n/g, " ").replace(/([\\`*_~|[\]()])/g, "\\$1");
}
/**
* The section each commit type gets, in the order they are printed.
*
* The keys are `.githooks/commit-msg`'s closed type list, which is the whole
* reason this grouping is possible: every commit is already labelled with the
* section it belongs in, enforced at the moment it is written rather than
* inferred afterwards. A type added there and not here lands in `other`, which
* is visible rather than silent.
*/
export const SECTIONS = [
["feat", "🚀 Features"],
["fix", "🐛 Fixes"],
["security", "🔒 Security"],
["perf", "⚡ Performance"],
["ui", "🎨 Interface"],
["refactor", "🧹 Refactoring"],
["docs", "📚 Documentation"],
["test", "🧪 Tests"],
["chore", "🔧 Tooling"],
["other", "📦 Other"],
];
/** `type(scope)!: subject`, the shape `.githooks/commit-msg` enforces. */
const CONVENTIONAL = /^([a-z]+)(?:\([a-z0-9._-]+\))?!?: (.+)$/;
/**
* Commits grouped by the type they declared.
*
* Takes `sha subject` lines. Two rules earn their place:
*
* `chore(release)` commits are dropped — they are the boundaries of the range,
* not content in it, and listing "chore(release): v0.54.30" under Tooling would
* be the release describing itself.
*
* Everything else that does not parse goes to `other` rather than being
* skipped. `commit-msg` exempts `Merge`, `Revert`, `fixup!`, `squash!` and
* `amend!`, so unparseable subjects genuinely occur — and a commit vanishing
* from the notes is the failure this file exists to prevent, not a tidiness
* problem.
*/
export function groupCommits(lines) {
const groups = new Map(SECTIONS.map(([key]) => [key, []]));
for (const line of lines) {
const trimmed = String(line ?? "").trim();
if (!trimmed) continue;
const split = trimmed.indexOf(" ");
const sha = split === -1 ? trimmed : trimmed.slice(0, split);
const subject = split === -1 ? "" : trimmed.slice(split + 1);
if (/^chore\(release\)/.test(subject)) continue;
const match = CONVENTIONAL.exec(subject);
const key = match && groups.has(match[1]) ? match[1] : "other";
groups.get(key).push({ sha, subject: match ? match[2] : subject });
}
return groups;
}
/**
* The release body.
*
* Every subject is scrubbed then escaped, in that order: `redact` removes what
* must not be published, `escapeMarkdown` stops what remains from becoming
* markup. `escapeMarkdown` also flattens newlines, so a subject cannot start a
* line it is not allowed to start.
*
* An empty range still produces a body. A release with no commits behind it is
* a real thing — a re-cut, a version bump — and saying so is better than an
* empty section or a missing record.
*/
export function renderBody({
tag,
image,
digest,
groups,
signoff,
milestones = [],
deployNote = DEFAULTS.deployNote,
}) {
const clean = (text) => escapeMarkdown(redact(String(text ?? "")));
const lines = [`Welcome to **${clean(tag)}**.`, ""];
const listed = SECTIONS.filter(([key]) => (groups.get(key) ?? []).length > 0);
if (listed.length === 0) {
lines.push("No commits landed between this release and the one before it.", "");
} else {
lines.push("## What's Changed", "");
for (const [key, heading] of listed) {
lines.push(`### ${heading}`);
for (const commit of groups.get(key)) {
lines.push(`- ${clean(commit.subject)} (\`${clean(commit.sha)}\`)`);
}
lines.push("");
}
}
// Named before the image, because "which batch shipped" is the question a
// release closes and the image tag is the detail underneath it.
if (milestones.length > 0) {
lines.push("## Milestones completed", "");
for (const title of milestones) lines.push(`- ${clean(title)}`);
lines.push("");
}
lines.push("## The image", "");
lines.push(` ${image}:${tag}`);
// Absent rather than invented. A digest is what makes a tag checkable, and a
// release claiming one it did not read would be worse than one that stays
// quiet about it.
if (digest) lines.push(` ${digest}`);
lines.push("");
// Publishing and deploying are separate decisions, and saying so is the one
// sentence a reader of these notes most needs. *How* they are separate is
// each project's own business, which is why it is a setting rather than a
// string: naming one project's stack in another's release notes is exactly
// the hard-coding this directory refuses.
//
// Set RELEASE_DEPLOY_NOTE to describe how this project actually deploys, or
// to an empty string where publishing and deploying are one act. A note that
// describes a workflow the project has moved past is worse than none,
// because it is read as current.
if (deployNote) {
lines.push(...deployNote.split("\n"), "");
}
// The sign-off marks a batch landing, not every patch.
//
// Forty-eight releases were cut in five days before this existed. An image on
// every one of them is wallpaper; on the release that finishes a batch it
// means something. The condition is the same fact that drives the version
// bump, so the two cannot disagree: a release either completed a milestone or
// it did not.
//
// The tag rides along as a query string, and it is not decoration.
//
// The asset ships in the image, so a release published *before* that image is
// deployed points at a URL the origin does not yet serve — and the CDN in
// front of it caches that 404 for four hours, which outlives the deploy. It
// happened on the first release cut this way: the file was in the container
// and the sibling asset served 200, while this one answered 404 from cache.
//
// A per-release URL cannot inherit another release's cached miss, and it
// re-fetches when the asset itself changes.
if (signoff && milestones.length > 0) {
const separator = signoff.includes("?") ? "&" : "?";
lines.push(`![](${signoff}${separator}v=${encodeURIComponent(tag)})`);
}
return `${lines.join("\n").trimEnd()}\n`;
}
const DEFAULTS = {
image: process.env.RELEASE_IMAGE ?? "registry.example/owner/project",
signoff:
process.env.RELEASE_SIGNOFF_URL ?? "",
repo: process.env.RELEASE_REPO ?? "owner/project",
// What separates publishing from deploying here. Overridable, and settable to
// an empty string by a project where the two are the same act.
deployNote:
process.env.RELEASE_DEPLOY_NOTE ??
"Publishing is not deploying. `deploy.py` moves a running stack to a published image.",
registryEnv:
process.env.RELEASE_REGISTRY_ENV ??
`${process.env.HOME}/.openclaw/docker-registry.env`,
};
const say = (message) => console.error(`release-notes: ${message}`);
const die = (message) => {
say(message);
process.exit(1);
};
const git = (...args) =>
execFileSync("git", args, { encoding: "utf8", timeout: 60_000 }).trim();
/**
* The same, with git's own stderr discarded.
*
* For probes where "no such thing" is the expected answer. `rev-parse` on a tag
* that does not exist prints a four-line `fatal:` block before returning
* non-zero, and a run that is working correctly should not look like one that
* broke.
*/
const gitQuiet = (...args) =>
execFileSync("git", args, {
encoding: "utf8",
timeout: 60_000,
stdio: ["ignore", "pipe", "ignore"],
}).trim();
/** When the previous release was cut, as an ISO string, or null on the first. */
/**
* The release this one follows, whether or not its own commit exists yet.
*
* `release.sh` asks twice, at two different moments, and that is the whole
* reason this is a function rather than `markers[1]`:
*
* - **before the bump**, to decide whether a batch landed and the version
* should take a minor rather than a patch (`release.sh:378`). HEAD is still
* the last ordinary commit, so the most recent `chore(release):` *is* the
* previous release.
* - **after the commit**, to write the notes (`release.sh:658`). Now the most
* recent one is this release, and the previous is the one behind it.
*
* Taking `markers[1]` unconditionally is right only in the second case. In the
* first it names the release *before* the previous one, which widened the
* window by a whole release: v0.56.0 re-announced "Batch 22", already claimed
* by v0.55.0 two minutes after that milestone closed, and took a minor bump for
* a batch that had landed in the release before it. It also made `--dry-run`
* print a commit list a release too long, so the one artifact meant to be read
* by eye before publishing did not match what would publish.
*
* `markers[0] === headSha` rather than the subject alone: a `chore(release):`
* commit somewhere behind HEAD must not make HEAD look like one.
*/
function previousReleaseCommit(run = git) {
const markers = run("log", "--grep=^chore(release):", "--format=%H", "-n", "2")
.split("\n")
.filter(Boolean);
if (markers.length === 0) {
return null;
}
const [headSha, headSubject] = run("log", "-1", "--format=%H%n%s").split("\n");
const headIsRelease = markers[0] === headSha && /^chore\(release\):/.test(headSubject ?? "");
return headIsRelease ? (markers[1] ?? null) : markers[0];
}
/** Exported for the test that pins the two moments apart. */
export const _previousReleaseCommit = previousReleaseCommit;
export function previousReleaseAt() {
const marker = previousReleaseCommit();
return marker ? git("show", "-s", "--format=%cI", marker) : null;
}
/**
* The batches this release finishes.
*
* A milestone closed since the previous release was cut. That is the unit work
* is actually planned in here — batches in the tracker — and it is what makes a
* release worth marking: forty-eight were cut in five days, and a sign-off on
* every one of them is wallpaper.
*
* Throws rather than returning `[]` when it cannot ask. "I could not check" and
* "no batch landed" are different facts and only one of them is safe to act on
* — this one decides both the version bump and the sign-off, so a silent empty
* answer would quietly downgrade a release nobody meant to downgrade.
*/
export async function closedMilestonesSince(since, env) {
const url =
`https://${env.FORGEJO_REGISTRY}/api/v1/repos/${DEFAULTS.repo}` +
"/milestones?state=closed&limit=50";
const auth = Buffer.from(
`${env.FORGEJO_REGISTRY_USER}:${env.FORGEJO_REGISTRY_TOKEN}`,
).toString("base64");
const response = await fetch(url, {
headers: { Authorization: `Basic ${auth}`, Accept: "application/json" },
signal: AbortSignal.timeout(30_000),
redirect: "manual",
});
if (!response.ok) throw new Error(`milestones answered HTTP ${response.status}`);
const all = await response.json();
if (!Array.isArray(all)) throw new Error("milestones did not answer with a list");
// No boundary means the first release ever cut this way; every closed
// milestone predates it, and claiming all of them would be a lie about what
// this release contains.
if (!since) return [];
const after = Date.parse(since);
return all
.filter((milestone) => {
const closed = Date.parse(milestone?.closed_at ?? "");
return Number.isFinite(closed) && closed > after;
})
.sort((a, b) => Date.parse(a.closed_at) - Date.parse(b.closed_at))
.map((milestone) => String(milestone.title ?? "").trim())
.filter(Boolean);
}
/** The commits since the previous release, newest first. */
export function commitsSince() {
// No previous release commit means this is the first one ever cut this way.
// The whole history is the range rather than nothing — an empty body on a
// first release would be the least useful moment to be silent.
const marker = previousReleaseCommit();
const range = marker ? `${marker}..HEAD` : "HEAD";
return git("log", range, "--format=%h %s", "--no-merges").split("\n").filter(Boolean);
}
/**
* Credentials, from the file outside the repository.
*
* The same file and the same three names `release.sh` sources. Read here rather
* than inherited, so this can be re-run by hand after a failure without the
* caller having to arrange an environment.
*/
function credentials() {
const env = { ...process.env };
try {
for (const line of readFileSync(DEFAULTS.registryEnv, "utf8").split("\n")) {
const match = /^([A-Z_]+)=(.*)$/.exec(line.trim());
if (match) env[match[1]] = match[2].replace(/^["']|["']$/g, "");
}
} catch {
// Absent is not fatal here; the check below names what is missing.
}
// All three, not two. `release.sh` guards on the host and the token and then
// dereferences the user unguarded — under `set -u` that aborts inside a
// command substitution and presents as "could not list published versions".
for (const name of ["FORGEJO_REGISTRY", "FORGEJO_REGISTRY_USER", "FORGEJO_REGISTRY_TOKEN"]) {
if (!env[name]) {
die(`${name} is not set and not in ${DEFAULTS.registryEnv} — cannot record the release.`);
}
}
return env;
}
/** The tag, created only if it does not already point somewhere else. */
function ensureTag(tag, dryRun) {
const head = git("rev-parse", "HEAD");
let existing = "";
try {
existing = gitQuiet("rev-parse", `refs/tags/${tag}`);
} catch {
existing = "";
}
if (existing && existing !== head) {
die(
`${tag} already exists and points at ${existing.slice(0, 7)}, not ${head.slice(0, 7)}. ` +
"A published tag is not moved.",
);
}
if (existing) {
say(`${tag} already tagged here — continuing.`);
} else if (dryRun) {
say(`[dry-run] git tag -a ${tag}`);
} else {
git("tag", "-a", tag, "-m", `Release ${tag}`);
say(`tagged ${tag}.`);
}
if (dryRun) {
say(`[dry-run] git push origin ${tag}`);
return;
}
try {
git("push", "origin", tag);
say(`pushed ${tag}.`);
} catch (error) {
// The shape `.githooks/post-commit` uses: say what is safe, say what to do,
// never force, and do not take down the thing that called us.
say(`could not push ${tag}: ${error.message.split("\n")[0]}`);
say(` The tag is safe locally. Push it when the remote is reachable:`);
say(` git push origin ${tag}`);
}
}
/** The release itself. A 409 means somebody already recorded it, which is done. */
async function publish(tag, body, env, dryRun) {
const url = `https://${env.FORGEJO_REGISTRY}/api/v1/repos/${DEFAULTS.repo}/releases`;
if (dryRun) {
say(`[dry-run] POST ${url}`);
process.stdout.write(`${body}\n`);
return true;
}
const auth = Buffer.from(`${env.FORGEJO_REGISTRY_USER}:${env.FORGEJO_REGISTRY_TOKEN}`).toString(
"base64",
);
let response;
try {
response = await fetch(url, {
method: "POST",
headers: {
Authorization: `Basic ${auth}`,
"Content-Type": "application/json",
Accept: "application/json",
},
body: JSON.stringify({ tag_name: tag, name: tag, body }),
signal: AbortSignal.timeout(30_000),
redirect: "manual",
});
} catch (error) {
say(`the release call failed: ${error.message}`);
return false;
}
if (response.status === 409) {
say(`a release for ${tag} already exists — nothing to do.`);
return true;
}
if (!response.ok) {
const detail = (await response.text().catch(() => "")).slice(0, 200);
// Status first, and an HTML body diagnosed rather than dumped — a Forgejo
// error body can echo the request, and Cloudflare's does not look like one
// at all.
say(
`the release call answered HTTP ${response.status}${
detail.toLowerCase().includes("<html")
? " with an HTML body — a proxy answered, not Forgejo"
: detail
? `: ${detail}`
: ""
}`,
);
if (response.status === 401 || response.status === 403) {
say(` Check FORGEJO_REGISTRY_TOKEN in ${DEFAULTS.registryEnv}.`);
}
return false;
}
const created = await response.json().catch(() => null);
say(`recorded ${tag}${created?.html_url ? ` — ${created.html_url}` : ""}.`);
return true;
}
async function main(argv) {
const dryRun = argv.includes("--dry-run");
// Asked before the version is chosen, so the bump and the sign-off are driven
// by the same fact. Prints one title per line and nothing else, because the
// caller is a shell reading it.
if (argv.includes("--closed-milestones")) {
const env = credentials();
const titles = await closedMilestonesSince(previousReleaseAt(), env);
for (const title of titles) process.stdout.write(`${title}\n`);
return;
}
const tag = argv.find((arg) => !arg.startsWith("--"));
if (!tag) die("usage: release-notes.mjs <vX.Y.Z> [--dry-run] | --closed-milestones");
if (!/^v\d+\.\d+\.\d+$/.test(tag)) die(`'${tag}' is not a vX.Y.Z tag.`);
const env = credentials();
// Passed down from `release.sh`, which already asked in order to choose the
// bump. Asking twice would risk two answers for one release — a milestone
// closed between the two calls would bump the version without earning the
// sign-off, or the reverse.
const milestones = (process.env.RELEASE_MILESTONES ?? "")
.split("\n")
.map((line) => line.trim())
.filter(Boolean);
const body = renderBody({
tag,
image: DEFAULTS.image,
digest: process.env.RELEASE_DIGEST ?? "",
groups: groupCommits(commitsSince()),
signoff: DEFAULTS.signoff,
milestones,
});
ensureTag(tag, dryRun);
if (!(await publish(tag, body, env, dryRun))) {
say(` The image IS published as ${tag} and the commit is made; only the`);
say(` release record is missing. Run: scripts/release-notes.mjs ${tag}`);
process.exit(1);
}
}
// Effects only when run, so a test can import the two pure functions above.
if (process.argv[1] && import.meta.url.endsWith(process.argv[1].split("/").pop())) {
await main(process.argv.slice(2));
}

713
docs/architecture/scripts/release.sh Executable file
View File

@ -0,0 +1,713 @@
#!/usr/bin/env bash
#
# Build the container, push it, and leave every version string in the repo
# agreeing with the tag that was published.
#
# npm run release # patch: 0.46.0 -> 0.46.1
# npm run release -- minor # 0.46.0 -> 0.47.0
# npm run release -- 1.0.0 # exact
# npm run release -- --dry-run
# npm run release -- --no-prune # keep every published version
# RELEASE_KEEP=10 npm run release # keep ten instead of two
#
# ===========================================================================
# TEMPLATE COPY — configure this before the first run
# ===========================================================================
#
# Copy to `scripts/release.sh`, add `"release": "bash scripts/release.sh"` to
# package.json, and set the five values in the CONFIGURATION block below. The
# script refuses to run until they are set: it has no defaults, deliberately.
#
# The version it was adapted from carried its origin project's image name,
# deploy host and container name as defaults. Copied into a second project and
# run once, that would have built the new project's code, pushed it OVER the
# first project's image, pruned the first project's published versions, and
# read the first project's running container to decide what was safe to delete.
# Every one of those is silent and none is reversible. Hence: no defaults, and
# a loud failure instead.
#
# Assumes: bash, git, docker, ssh, and a Node project (`npm version` does the
# semver arithmetic). For a non-Node project, replace the two `npm version`
# calls and the `node -p` reader; everything else is language-agnostic.
#
# Assumes a registry with a Forgejo-compatible package API for the prune. If
# yours differs, the prune is the only part to rewrite — it is at the bottom
# and nothing else depends on it.
#
# ## Why this exists
#
# The project this came from had no build or push command at all. `git grep
# "docker push"` found nothing, there was no CI, and the README documented a
# `docker build` line and never the push. Every release was two commands typed
# from memory, and the version bump was a third thing to remember afterwards.
#
# It was not remembered. `package.json` sat at 0.2.0 across twenty-nine image
# releases because nothing read it and nothing checked it; the README and the
# example compose file stayed pinned at v0.40.01 while production ran v0.46.0.
# The git history shows the ritual dying in stages: one `chore(release)` commit
# touched only `.dockerignore`, and two others were empty.
#
# So the version bump is not a step beside the release. It is what the release
# command does, and a test fails when the files disagree. Write that test — see
# "The guard test" at the bottom of this header.
#
# ## package.json is the source of truth
#
# The tag is always `v` + the version in package.json. Nothing else decides it,
# which is why there is no --tag flag: a flag would be a second source of truth
# and this whole script exists because there were four.
#
# Arithmetic is `npm version`, which is built in and enforces semver. That is
# load-bearing rather than incidental — it makes a tag like `v0.40.01` (leading
# zero in the patch, not valid semver, and a real tag in this registry)
# impossible to produce from now on.
#
# ## The ordering is the safety property
#
# Bump, build, push, and commit LAST. The rule is: never pass through a state
# you cannot explain to somebody reading the repository afterwards.
#
# If the build or the push fails, the edits are sitting in the working tree —
# visible, uncommitted, unpushed, one `git checkout` from gone. If the commit
# came first, a failure would leave `main` carrying a commit that announces a
# release which was never published, and `post-commit` would have already
# pushed it to everyone. One of those is recoverable by noticing; the other is
# not.
#
# ## What it deliberately does not do
#
# It does not deploy. Whatever runs the container pins a tag, and updating that
# pin stays a separate, deliberate act. Publishing an image and running it are
# two decisions and this script only makes the first one — so a bad build sits
# in a registry rather than in production.
#
# ## The guard test
#
# This script is only half the mechanism. The other half is a test asserting
# that every file in FILES/PIN_FILES names the same version, so a hand-edit or
# a half-finished release fails the suite instead of shipping. Without it,
# nothing notices the drift this script was written to end — which is exactly
# how the original got to four disagreeing version strings.
set -uo pipefail
cd "$(git rev-parse --show-toplevel)" || exit 1
# ---------------------------------------------------------------------------
# CONFIGURATION — set these five, then delete this banner.
#
# Every one is empty on purpose. See the note at the top: inherited defaults
# from another project point a release at that project's image, registry and
# running container, and every consequence of that is silent.
# ---------------------------------------------------------------------------
# Fully-qualified image name, no tag. e.g. registry.example.com/team/my-app
IMAGE="${RELEASE_IMAGE:-}"
# The public origin, when the build bakes one in.
#
# Left empty by default: a project whose artifact carries no origin should not
# have one invented for it. But when it is set it is usually frozen into the
# build — canonical URLs, Open Graph tags, robots.txt, sitemaps — and cannot be
# corrected without another build, so a wrong value ships silently and is found
# by somebody else. Validated below rather than trusted.
SITE_URL="${RELEASE_SITE_URL:-}"
# A file sourced for FORGEJO_REGISTRY / FORGEJO_REGISTRY_USER /
# FORGEJO_REGISTRY_TOKEN. Keep it OUTSIDE the repository — a token in a file
# the repo can see is a token one `git add -A` away from being published.
REGISTRY_ENV="${RELEASE_REGISTRY_ENV:-}"
# The files that carry a version string. Every one of them is checked before
# the build and staged after the push; the guard test asserts they agree.
#
# FILES is what `npm version` rewrites and must exist. PIN_FILES carry an
# `IMAGE:vX.Y.Z` pin rewritten by search-and-replace; list only the ones this
# project actually has. A project with no compose file leaves PIN_FILES holding
# just the README, or empty — an absent file is skipped with a warning rather
# than failing the release, because "you do not have that file" and "the
# rewrite silently matched nothing" need different answers.
FILES=(package.json package-lock.json)
PIN_FILES=(README.md docker-compose.example.yml)
say() { printf '\033[1mrelease:\033[0m %s\n' "$*" >&2; }
die() { printf '\033[1mrelease:\033[0m %s\n' "$*" >&2; exit 1; }
# Refused, never corrected: guessing what somebody meant by a malformed origin
# is how the wrong one gets baked in with a reassuring log line above it.
if [ -n "$SITE_URL" ]; then
case "$SITE_URL" in
https://*)
printf '%s' "$SITE_URL" \
| grep -Eq '^https://[A-Za-z0-9][A-Za-z0-9.-]*\.[A-Za-z]{2,}/?$' \
|| die "RELEASE_SITE_URL is not a bare https origin: ${SITE_URL}
Expected something like https://example.com, with no path." ;;
*)
die "RELEASE_SITE_URL must be an https origin, got: ${SITE_URL}" ;;
esac
fi
BUMP="patch"
DRY_RUN=""
NO_PRUNE=""
# How many versions survive a prune, newest first — published AND local.
#
# Two is the floor the check below enforces, and it is now also the default:
# the newest, plus one to roll back to. It is deliberately tight because the
# count that used to matter — how much of the registry we keep — turned out not
# to be the expensive one. Releases here are frequent and the deployed tag is
# spared on top of this number whatever it is, so the practical floor is three
# images, not two.
KEEP="${RELEASE_KEEP:-2}"
# Where the running stack lives, so a prune can find out what is deployed and
# refuse to delete it. See the prune section for why this is not optional.
#
# DEPLOY_HOST is an ssh destination — a Host in ~/.ssh/config, not a password
# prompt: the lookup runs with BatchMode=yes and a host that asks for input is
# read as "could not determine", which correctly stops the prune.
DEPLOY_HOST="${RELEASE_DEPLOY_HOST:-}"
DEPLOY_CONTAINER="${RELEASE_DEPLOY_CONTAINER:-}"
# ---------------------------------------------------------------------------
# Refuse to run half-configured.
#
# Checked here, before the first side effect, and named one at a time so the
# message says which value is missing rather than "configuration error".
# ---------------------------------------------------------------------------
[ -n "$IMAGE" ] || die "set IMAGE (or RELEASE_IMAGE) — the image name to build and push. See the CONFIGURATION block."
[ -n "$REGISTRY_ENV" ] || die "set REGISTRY_ENV (or RELEASE_REGISTRY_ENV) — the file holding the registry credentials."
[ -n "$DEPLOY_HOST" ] || die "set DEPLOY_HOST (or RELEASE_DEPLOY_HOST) — the ssh host running the deployed container."
[ -n "$DEPLOY_CONTAINER" ] || die "set DEPLOY_CONTAINER (or RELEASE_DEPLOY_CONTAINER) — the container name to read the deployed version from."
# Refused rather than defaulted, because the failure is a publication.
#
# Docker treats the first path component as a registry only when it contains a
# dot or a colon (or is localhost); anything else is a Docker Hub namespace. So
# `acme/my-app` is not a private registry that happens to be unreachable — it is
# Docker Hub, and a successful push there makes a private project public. That
# is not a mistake to discover from the registry's web UI.
case "${IMAGE%%/*}" in
*.*|*:*|localhost) : ;;
*) die "IMAGE ('$IMAGE') has no registry host — '${IMAGE%%/*}' is a Docker Hub
namespace, so pushing would publish this image publicly. Use a
fully-qualified name like registry.example.com/team/app." ;;
esac
for arg in "$@"; do
case "$arg" in
--dry-run) DRY_RUN="yes" ;;
--no-prune) NO_PRUNE="yes" ;;
-*) die "unknown flag $arg. Usage: npm run release -- [patch|minor|major|<version>] [--dry-run] [--no-prune]" ;;
*) BUMP="$arg" ;;
esac
done
case "$KEEP" in
''|*[!0-9]*) die "RELEASE_KEEP must be a whole number, got '$KEEP'." ;;
esac
# Keeping zero would delete the tag this run just published. Refused rather than
# clamped, because a caller who typed 0 meant something and it was not that.
[ "$KEEP" -ge 2 ] || die "RELEASE_KEEP must be at least 2 — keeping fewer leaves nothing to roll back to."
# What production is actually running, asked once and remembered.
#
# Both prunes need it and neither may delete without it, so it is a function
# rather than a line copied twice — a second copy is a second thing to keep
# correct, and the consequence of getting it wrong is deleting the image the
# running container was created from.
#
# The empty string means "could not determine", which every caller must treat
# as a reason to stop rather than as "nothing is deployed".
DEPLOYED_VERSION=""
DEPLOYED_ASKED=""
deployed_version() {
if [ -z "$DEPLOYED_ASKED" ]; then
DEPLOYED_ASKED="yes"
DEPLOYED_VERSION=$(ssh -o BatchMode=yes -o ConnectTimeout=10 "$DEPLOY_HOST" \
"docker inspect --format '{{index .Config.Labels \"org.opencontainers.image.version\"}}' $DEPLOY_CONTAINER" \
2>/dev/null | tr -d '\r\n')
fi
printf '%s' "$DEPLOYED_VERSION"
}
# ---------------------------------------------------------------------------
# The same trim, applied to this machine.
#
# `prune` below deletes published versions. Nothing ever deleted the images this
# script BUILDS, so every release since the first left another ~250 MB tag on
# the disk: seventy-four of them by the time anyone looked, inside a daemon
# holding 79 GB with 47 GB reclaimable.
#
# ## Scoped to this image by name, and to nothing else
#
# `docker image prune` is one line and would have reached every project sharing
# this daemon, including a build running in another checkout at that moment.
# Dangling layers are left alone for the same reason — an untagged layer is not
# identifiable as ours, and "probably nobody's" is not ownership. This removes
# tags of $IMAGE matching vN.N.N and nothing else.
#
# ## The deployed tag is spared here too
#
# A local image is not the copy production pulls, so deleting one is far less
# dangerous than deleting a published version — but it is the copy that makes a
# rollback instant instead of a download, and it costs nothing to keep. Same
# rule, same reason: if the deployed version cannot be read, this does nothing.
# ---------------------------------------------------------------------------
prune_local() {
[ -z "$NO_PRUNE" ] || { say "prune: local images skipped (--no-prune)."; return 0; }
local deployed
deployed=$(deployed_version)
if [ -z "$deployed" ]; then
say "prune: local images SKIPPED — could not read the deployed version."
return 0
fi
local tags total keep_from doomed
tags=$(docker images --format '{{.Tag}}' "$IMAGE" 2>/dev/null \
| grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' | sort -V)
total=$(printf '%s\n' "$tags" | grep -c .)
if [ "$total" -le "$KEEP" ]; then
say "prune: ${total} local image(s), keeping ${KEEP} — nothing to remove."
return 0
fi
keep_from=$((total - KEEP))
doomed=$(printf '%s\n' "$tags" | head -n "$keep_from" \
| grep -v "^${deployed}$" | grep -v "^${TAG}$")
if [ -z "$doomed" ]; then
say "prune: no local images to remove."
return 0
fi
if [ -n "$DRY_RUN" ]; then
say "prune: would remove $(printf '%s\n' "$doomed" | grep -c .) local image(s):"
printf ' %s\n' $doomed >&2
return 0
fi
say "prune: removing $(printf '%s\n' "$doomed" | grep -c .) local image(s), keeping"
say " the newest ${KEEP} plus ${deployed} (deployed)."
local tag
for tag in $doomed; do
# Untagging is all that is asked for. Layers shared with a kept image stay,
# and the space is reclaimed only when the last tag referencing them goes —
# which is the correct behaviour and why this reports what it untagged
# rather than claiming an amount of disk freed.
if docker rmi "${IMAGE}:${tag}" >/dev/null 2>&1; then
say " removed ${tag}"
else
# Said out loud, like its published counterpart. A container still using
# the image is the usual cause and it is not an error worth stopping for.
say " kept ${tag} (still in use, or already gone)"
fi
done
}
prune() {
[ -z "$NO_PRUNE" ] || { say "prune: skipped (--no-prune)."; return 0; }
if [ -z "${FORGEJO_REGISTRY:-}" ] || [ -z "${FORGEJO_REGISTRY_TOKEN:-}" ]; then
say "prune: skipped — no registry credentials."
return 0
fi
local owner="${IMAGE#*/}"; owner="${owner%%/*}"
local name="${IMAGE##*/}"
local api="https://${FORGEJO_REGISTRY}/api/v1/packages/${owner}/container/${name}"
# What is actually running. Not an optimisation — see the header.
local deployed
deployed=$(deployed_version)
if [ -z "$deployed" ]; then
say "prune: SKIPPED — could not read the deployed version from"
say " ${DEPLOY_HOST}/${DEPLOY_CONTAINER}. Refusing to delete anything"
say " without knowing which image production is running."
return 0
fi
local versions
versions=$(curl -sS --max-time 20 -u "${FORGEJO_REGISTRY_USER}:${FORGEJO_REGISTRY_TOKEN}" \
"https://${FORGEJO_REGISTRY}/api/v1/packages/${owner}?type=container&q=${name}" 2>/dev/null \
| PKG="$name" node -e 'let s="";process.stdin.on("data",d=>s+=d).on("end",()=>{try{
console.log(JSON.parse(s).filter(p => p.name === process.env.PKG).map(p => p.version).join("\n"))
}catch{process.exit(1)}})' 2>/dev/null)
if [ -z "$versions" ]; then
say "prune: skipped — could not list published versions."
return 0
fi
# Semver order, newest last. `latest` is a moving pointer rather than a
# release and is never a candidate; it is filtered before anything is counted
# so it cannot occupy one of the kept slots either.
local candidates
candidates=$(printf '%s\n' "$versions" | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' | sort -V)
local total keep_from doomed
total=$(printf '%s\n' "$candidates" | grep -c .)
if [ "$total" -le "$KEEP" ]; then
say "prune: ${total} published versions, keeping ${KEEP} — nothing to remove."
return 0
fi
keep_from=$((total - KEEP))
doomed=$(printf '%s\n' "$candidates" | head -n "$keep_from" | grep -v "^${deployed}$" | grep -v "^${TAG}$")
local protected
protected=$(printf '%s\n' "$candidates" | head -n "$keep_from" | grep -c "^${deployed}$" || true)
if [ "$protected" -gt 0 ]; then
say "prune: keeping ${deployed} — it is what production is running, even"
say " though it is old enough to drop."
fi
if [ -z "$doomed" ]; then
say "prune: nothing to remove."
return 0
fi
say "prune: keeping the newest ${KEEP}, plus ${deployed} (deployed) and latest."
if [ -n "$DRY_RUN" ]; then
say "prune: would remove $(printf '%s\n' "$doomed" | grep -c .) older version(s):"
printf ' %s\n' $doomed >&2
return 0
fi
say "prune: removing $(printf '%s\n' "$doomed" | grep -c .) older version(s):"
local version code
for version in $doomed; do
code=$(curl -sS --max-time 30 -o /dev/null -w '%{http_code}' \
-X DELETE -u "${FORGEJO_REGISTRY_USER}:${FORGEJO_REGISTRY_TOKEN}" \
"${api}/${version}" 2>/dev/null)
case "$code" in
2*) say " removed ${version}" ;;
# Said out loud rather than counted as done. A prune that quietly failed
# looks exactly like a prune that had nothing to do.
*) say " FAILED ${version} (HTTP ${code:-no response}) — left in place" ;;
esac
done
}
# ---------------------------------------------------------------------------
# Preconditions. Everything that can be checked without side effects, checked
# before the first side effect.
# ---------------------------------------------------------------------------
command -v docker >/dev/null 2>&1 || die "docker is not on PATH."
branch=$(git symbolic-ref --quiet --short HEAD) || die "detached HEAD — check out a branch first."
[ "$branch" = "main" ] || die "on '$branch'. Releases are cut from main."
# Only the release files are required to be clean, not the whole tree. Other
# work legitimately shares this checkout, and refusing to release because an
# unrelated file is open would make the command useless exactly when it is most
# wanted. The commit at the end stages these paths explicitly for the same
# reason: whatever else is in the tree is not part of this release.
dirty=$(git diff --name-only -- "${FILES[@]}" "${PIN_FILES[@]}" 2>/dev/null;
git diff --cached --name-only -- "${FILES[@]}" "${PIN_FILES[@]}" 2>/dev/null)
if [ -n "$dirty" ]; then
say "these release files have uncommitted changes:"
printf ' %s\n' $(printf '%s\n' "$dirty" | sort -u) >&2
die "commit or discard them first — a release must start from a known version."
fi
current=$(node -p 'require("./package.json").version') || die "cannot read package.json"
# `npm version` writes the file, so ask it what the answer would be by asking it
# somewhere disposable. Cheaper and more honest than reimplementing semver here,
# and it means the validation rules are npm's rather than a second opinion.
work=$(mktemp -d) || die "cannot create a temporary directory"
trap 'rm -rf "$work"' EXIT
printf '{"name":"v","version":"%s"}\n' "$current" > "$work/package.json"
if ! next=$(cd "$work" && npm version --no-git-tag-version "$BUMP" 2>&1 | tr -d 'v\n'); then
die "npm rejected '$BUMP': $next"
fi
[ -n "$next" ] || die "could not work out the next version from '$BUMP'."
TAG="v${next}"
say "$current -> $next (image ${IMAGE}:${TAG})"
# ---------------------------------------------------------------------------
# Refuse to move a tag that is already published.
#
# Overwriting one would silently change what a running stack pulls on its next
# recreate, and the old image would still be running with no way to tell from
# the tag. A published tag is immutable by convention here; this makes it
# immutable in practice.
# ---------------------------------------------------------------------------
if [ -r "$REGISTRY_ENV" ]; then
# shellcheck disable=SC1090
. "$REGISTRY_ENV"
fi
if [ -n "${FORGEJO_REGISTRY:-}" ] && [ -n "${FORGEJO_REGISTRY_TOKEN:-}" ]; then
repo="${IMAGE#*/}"
tags=$(curl -sS --max-time 20 -u "${FORGEJO_REGISTRY_USER}:${FORGEJO_REGISTRY_TOKEN}" \
"https://${FORGEJO_REGISTRY}/v2/${repo}/tags/list" 2>/dev/null)
if printf '%s' "$tags" | grep -q "\"${TAG}\""; then
die "${TAG} is already published. Pick a higher version — a published tag is not moved."
fi
if [ -z "$tags" ]; then
# Said out loud rather than passed over. "I could not check" and "it is not
# there" are different answers and only one of them is safe to act on.
say "WARNING: could not read the registry tag list. Proceeding without the"
say " already-published check."
fi
else
say "WARNING: no registry credentials at $REGISTRY_ENV — cannot check whether"
say " ${TAG} is already published."
fi
if [ -n "$DRY_RUN" ]; then
say "--dry-run: nothing was changed. It would have:"
# Only the ones actually present, for the reason the whole script exists:
# a "would have" line naming a file this project does not have is a claim.
present=("${FILES[@]}")
for file in "${PIN_FILES[@]}"; do [ -f "$file" ] && present+=("$file"); done
say " set version ${next} in ${present[*]}"
say " docker build --build-arg APP_VERSION=${TAG} -t ${IMAGE}:${TAG} ."
say " docker push ${IMAGE}:${TAG}"
say " git commit -m 'chore(release): ${TAG}' (post-commit then pushes)"
say ""
say "and then pruned. That part is shown for real, because it deletes:"
prune
prune_local
exit 0
fi
# ---------------------------------------------------------------------------
# Bump. Files first, so the image is built from the source that names it.
# ---------------------------------------------------------------------------
npm version --no-git-tag-version --allow-same-version "$next" >/dev/null \
|| die "npm version failed; nothing has been built or pushed."
# The pins that drifted for six releases in the original. Anchored on the image
# name so this cannot match a version string belonging to something else.
#
# A listed file that does not exist is a configuration fact, not a failure: not
# every project has a compose file. It is said out loud rather than passed over,
# because "you do not have that file" and "the rewrite matched nothing" have the
# same appearance and only one of them is fine.
BUMPED=("${FILES[@]}")
for file in "${PIN_FILES[@]}"; do
if [ ! -f "$file" ]; then
say "note: $file is listed in PIN_FILES but does not exist — skipping."
continue
fi
# Only files that already carry a pin. A README that never mentioned the
# image is not a drift risk and must not become a failure.
grep -q "${IMAGE}:v" "$file" || { say "note: $file carries no ${IMAGE} pin — skipping."; continue; }
perl -pi -e "s{\Q${IMAGE}\E:v[0-9][0-9.]*}{${IMAGE}:${TAG}}g" "$file" \
|| die "could not rewrite the pin in $file"
# Checked, not assumed. A rewrite that silently matched nothing would produce
# a release whose own guard test fails on the next commit.
grep -q "${IMAGE}:${TAG}" "$file" || die "the pin in $file did not update — refusing to build."
BUMPED+=("$file")
done
say "bumped ${BUMPED[*]}"
# ---------------------------------------------------------------------------
# Guards, before anything is built.
#
# ## Why they are here and not left to the commit hook
#
# `pre-commit` runs the typecheck and the suite, and the commit is the LAST step
# of this script — after the push. So relying on it means the gate fires once
# the image is already published and cannot be unpublished: the hook would
# refuse the commit, and the registry would be left holding a tag whose tests
# never passed, with git carrying no record of it. The first real release
# through this script did exactly that, and passed only by luck.
#
# ## A missing test database refuses the release, it does not warn about it
#
# Without TEST_DATABASE_URL the database suites skip themselves, which on a
# repository of any size is a large fraction of the tests. That is the right
# default for a laptop with no Postgres and the wrong default for cutting a
# release — and warning about it while building anyway is the worst of the
# three, because the summary above it still says the tests passed.
# ---------------------------------------------------------------------------
if [ -z "${SKIP_GUARDS:-}" ]; then
say "typecheck…"
if ! npx tsc --noEmit; then
say "typecheck failed. The bump is in your working tree; nothing was built,"
say " published or committed."
exit 1
fi
# A warning is not a gate.
#
# This printed three lines saying half the suite would skip and then cut the
# release anyway. A gate that reports "tests passed" for a run which never
# touched the database is worse than no gate, because it is trusted — and the
# project this script came from shipped every release that way for months
# before anybody added up the numbers.
#
# So: use what is set, otherwise adopt a local test database if one is
# listening, and refuse if neither. The escape hatch stays for the machine
# with no database — which is what the warning was protecting — but it has to
# be asked for by name rather than being the default.
if [ -z "${TEST_DATABASE_URL:-}" ]; then
if [ -n "${RELEASE_TEST_DB_PORT:-}" ] \
&& (exec 3<>/dev/tcp/127.0.0.1/"${RELEASE_TEST_DB_PORT}") 2>/dev/null; then
exec 3<&- 2>/dev/null || true
export TEST_DATABASE_URL="${RELEASE_TEST_DB_URL:?set RELEASE_TEST_DB_URL beside RELEASE_TEST_DB_PORT}"
say "TEST_DATABASE_URL was unset; using the local test database on ${RELEASE_TEST_DB_PORT}."
elif [ -n "${RELEASE_ALLOW_SKIPPED_TESTS:-}" ]; then
say "WARNING: no test database, and RELEASE_ALLOW_SKIPPED_TESTS is set."
say " The database suites will SKIP. You are cutting a release"
say " that has not been fully tested."
else
die "no test database, so the database suites would silently skip.
Set TEST_DATABASE_URL, or set RELEASE_TEST_DB_PORT and RELEASE_TEST_DB_URL
so this script can find the local one, or pass RELEASE_ALLOW_SKIPPED_TESTS=1
to cut a release knowing it is half-tested."
fi
fi
say "tests…"
if ! npx vitest run --reporter=dot; then
say "tests failed. The bump is in your working tree; nothing was built,"
say " published or committed."
exit 1
fi
else
say "SKIP_GUARDS set — typecheck and tests did NOT run before this release."
fi
# ---------------------------------------------------------------------------
# Build, then push. Nothing is committed until both have succeeded.
# ---------------------------------------------------------------------------
say "building ${IMAGE}:${TAG}…"
if ! docker build \
--build-arg "NEXT_PUBLIC_SITE_URL=${SITE_URL}" \
--build-arg "APP_VERSION=${TAG}" \
-t "${IMAGE}:${TAG}" . ; then
say "build failed. The version bump is in your working tree and NOTHING was"
say " published or committed. Fix the build and run this again, or"
say " 'git checkout -- ${BUMPED[*]}' to undo the bump."
exit 1
fi
# Verify before trusting it — the same move a backup script makes with `pg_restore
# --list` before renaming a dump into place. The Dockerfile threads APP_VERSION
# through two stage-scoped ARGs and an ENV; drop any one of them and the build
# still succeeds, the tests still pass, and the only symptom is the settings
# screen reading "running from source" in production. Ask the image directly
# rather than trusting that the wiring is still there.
#
# `docker run` with a command overrides CMD, so this starts no server and needs
# no database.
say "verifying the image reports ${TAG}…"
baked=$(docker run --rm --entrypoint printenv "${IMAGE}:${TAG}" APP_VERSION 2>/dev/null | tr -d '\r\n')
if [ "$baked" != "$TAG" ]; then
# Remove the local tag. A correctly-named image with the wrong contents is a
# loaded gun for a later hand-typed `docker push`.
docker rmi "${IMAGE}:${TAG}" >/dev/null 2>&1
say "the image reports '${baked:-nothing}' but should report ${TAG}."
say " Check ARG/ENV APP_VERSION in the runner stage of the Dockerfile."
say " Nothing was published or committed; the local image was removed."
exit 1
fi
say "pushing ${IMAGE}:${TAG}…"
if ! docker push "${IMAGE}:${TAG}"; then
say "push failed. The image exists locally and the bump is in your working"
say " tree, but nothing was published or committed. Check the registry"
say " login and run this again."
exit 1
fi
# ---------------------------------------------------------------------------
# Commit last, by explicit path.
# ---------------------------------------------------------------------------
git add -- "${BUMPED[@]}" || die "git add failed after a successful push — commit ${BUMPED[*]} by hand."
# pre-commit runs the typecheck, and the suite when a .ts/.tsx is staged. None
# of these paths are, so this gets the typecheck only — which is what catches a
# package.json this script has mangled.
if ! git commit -q -m "chore(release): ${TAG}"; then
say "the commit was refused (see above). The image IS published as ${TAG};"
say " only the commit is missing. Fix and commit ${BUMPED[*]} by hand."
exit 1
fi
say "released ${TAG}."
# ---------------------------------------------------------------------------
# Prune old published versions.
#
# ## Last, and deliberately so
#
# The release is complete by this point — image published, files bumped, commit
# made. Everything below is housekeeping, and housekeeping must never be able to
# fail a release that already succeeded. Nothing here exits non-zero.
#
# ## It refuses to run blind
#
# Deleting a published version is irreversible and the registry is the only copy
# — there is no `git revert` for a pushed image. The one that actually matters
# is the version PRODUCTION IS RUNNING, and that is routinely an old tag: at the
# time this was written the stack pinned v0.54.0 while v0.54.3 was the newest,
# because deploying is a separate manual step. A newest-N rule with no
# knowledge of that would eventually delete the image the running container was
# created from, and the failure surfaces later — at the next recreate, when the
# stack cannot pull the tag it pins and the site does not come back.
#
# So the deployed tag is looked up, and if it cannot be determined the prune is
# skipped entirely. "I could not check" is not "there is nothing to protect",
# and only one of those is safe to act on — the same rule the published-tag
# check above follows.
#
# ## By version name, not by digest
#
# The v2 API deletes a manifest by digest, and two tags can point at the same
# digest — `latest` here does. A digest delete would take both and the second
# one would be silent. Forgejo's package API deletes the version that was named,
# which is the only thing this ever wants to do.
# ---------------------------------------------------------------------------
prune
prune_local
say "note: this published an image. It did not deploy it — whatever runs the"
say " container still pins its own tag and that edit is a separate step."

591
docs/architecture/scripts/scaffold.sh Executable file
View File

@ -0,0 +1,591 @@
#!/usr/bin/env bash
#
# Create the documentation tree a project is expected to have, copied from the
# template's own copy of it, without ever overwriting a document that is
# already there.
#
# bash scripts/scaffold.sh # into the current directory
# bash scripts/scaffold.sh --dry-run # list everything, write nothing
# bash scripts/scaffold.sh --into ../new-app
# bash scripts/scaffold.sh --force # overwrite; needs a clean git tree
# SCAFFOLD_TEMPLATE_ROOT=/path/to/Template bash scripts/scaffold.sh
#
# ===========================================================================
# TEMPLATE COPY — configure this before the first run
# ===========================================================================
#
# Copy to `scripts/scaffold.sh` and set TEMPLATE_ROOT in the CONFIGURATION
# block below, or pass it as SCAFFOLD_TEMPLATE_ROOT. It is empty and it stays
# empty: the template sits at a different absolute path on every machine, and
# a plausible-looking default would quietly resolve to whatever checkout
# happens to be there. That is not a "file not found" — it is thirteen
# documents arriving with another project's Owner lines, another project's
# review dates and another project's branding, each one reading as this
# project's own the moment it lands.
#
# Assumes: bash and coreutils. git is used only by --force, and only to read.
#
# ## Why this exists
#
# The convention this tree encodes is cheap to state and, on the evidence,
# almost never adopted whole. Eight repositories were surveyed before the
# template was last rewritten: exactly one kept the ledger the template asked
# for, three had invented their own dialects, and one had pushed nothing but a
# README. Nobody refused the convention. They started projects, and a
# convention that has to be re-typed from memory at the start of each one is a
# convention that decays into four different ones.
#
# So the first act of a new project is a command, not a reading exercise. What
# is left afterwards — the placeholders, the tracker, the branding — is work
# that genuinely needs a person, and it is visible precisely because the
# skeleton around it is already correct.
#
# ## Never overwriting is the whole safety property
#
# Every other rule here follows from one: a scaffold that clobbers a written
# document destroys work that nothing would notice was gone. There is no test
# for "the architecture notes used to say more than this", no build failure, no
# diff to read if the file was never committed. It is silent at the moment it
# happens and silent forever afterwards, and the natural time to re-run a
# scaffold — partway into a project, to pick up a file that was missed — is
# exactly the time the tree is full of real writing.
#
# So an existing file is reported as "kept" and left untouched, always. --force
# exists for the one honest case (a tree scaffolded from a stale template, not
# yet written into) and it refuses twice over: once for the tree, which must be
# clean, and again for each file, which git must actually track. A clean tree
# alone is not enough — it says nothing about ignored files — so the worst
# --force can do is something `git checkout` undoes.
#
# ## What it deliberately does not do
#
# It does not touch git. No init, no add, no commit — creating files is enough,
# and committing them is a judgement about what belongs in the history of this
# project, made by whoever will answer for that history.
#
# It does not fill anything in. The angle-bracket placeholders are left
# standing because a document whose header says it was reviewed on a date
# nobody reviewed it is worse than one that visibly has not been filled in: the
# first is trusted at the exact moment it should not be. An unfilled header
# keeps asking; an invented one stops.
#
# It does not copy branding. See the docs/data/img note near the bottom.
#
# It does not copy the scripts that live in this folder. They are in the
# template's docs tree for the template's own reasons, and each one needs
# configuring before its first run — an unconfigured release.sh landing in
# every new repository is a loaded gun, not a head start. Copy those one at a
# time, having read them.
#
# It does not create the tracker labels, the first milestone or the project's
# README, and it does not check conformance. It reports what it did. Whether
# the project passes is a different question, asked by a different thing.
set -uo pipefail
# ---------------------------------------------------------------------------
# CONFIGURATION — set this one, then delete this banner.
#
# Empty on purpose. See the note at the top: a default here does not fail, it
# copies the wrong project's documents and says "created" thirteen times.
# ---------------------------------------------------------------------------
# Absolute path to the template checkout — the directory holding `docs/`.
TEMPLATE_ROOT="${SCAFFOLD_TEMPLATE_ROOT:-}"
# ---------------------------------------------------------------------------
# The tree, listed explicitly.
#
# A recursive copy would be shorter and would drag in whatever else is sitting
# in the template's docs tree today, including this script. An explicit list is
# also the only thing that can tell the difference between "the template does
# not have that file" and "the copy quietly produced twelve of thirteen" — the
# second looks identical to success from the outside.
# ---------------------------------------------------------------------------
DOCS=(
docs/DOC_TRUST_MAP.md
docs/WORK_CYCLE.md
docs/TOOLS.md
docs/planning/PROJECT_PLAN.md
docs/qa/ClaudeQAPlan.md
docs/qa/ClaudeQACoverage.md
docs/qa/ClaudeReport.md
docs/history/HISTORY.md
docs/history/DEVELOPMENT_LOG.md
docs/history/BATCH_LEDGER.md
docs/security/SECURITY.md
docs/security/SECURITY_CHECKLIST.md
docs/architecture/README.md
docs/design/README.md
docs/data/README.md
)
# Directories made empty, holding nothing this script is willing to invent.
DIRS=(docs/data/img)
# The three names the conformance check looks for, by name, in docs/data/img.
BRANDING=(icon.webp logo.webp banner.webp)
# The placeholders the template leaves for a person to replace. Used only to
# count what is still outstanding at the end — if the template's placeholder
# style changes this finds nothing, which is why finding nothing prints nothing
# rather than an all-clear.
PLACEHOLDER_RE='<(Project|YYYY-MM-DD|who maintains this)>'
say() { printf '\033[1mscaffold:\033[0m %s\n' "$*" >&2; }
die() { printf '\033[1mscaffold:\033[0m %s\n' "$*" >&2; exit 1; }
usage() {
cat >&2 <<'EOF'
scaffold: create the documentation tree a project is expected to have.
bash scripts/scaffold.sh into the current directory
bash scripts/scaffold.sh --into <path> into another project root
bash scripts/scaffold.sh --dry-run list what would happen, write nothing
bash scripts/scaffold.sh --force overwrite; refuses on a dirty git tree
SCAFFOLD_TEMPLATE_ROOT path to the template checkout (required, no default)
SCAFFOLD_INTO same as --into
An existing file is never overwritten without --force. Exits non-zero if any
path was skipped, including one the template itself is missing.
EOF
}
DRY_RUN=""
FORCE=""
TARGET="${SCAFFOLD_INTO:-}"
# Set only when --force resolves a repository. Declared here so the copy loop
# can test it under `set -u` whether or not --force was passed.
force_top=""
# Whether the operator named the target, as opposed to inheriting the current
# directory. The guard below is only for the inherited case.
TARGET_NAMED=""
[ -n "$TARGET" ] && TARGET_NAMED="yes"
while [ "$#" -gt 0 ]; do
case "$1" in
--dry-run) DRY_RUN="yes" ;;
--force) FORCE="yes" ;;
--into)
# A bare --into would otherwise swallow the next flag as a path and
# scaffold into a directory called "--dry-run".
[ "$#" -ge 2 ] || die "--into needs a path."
case "$2" in -*) die "--into needs a path, got '$2'." ;; esac
# An empty string is not "here". Without this it becomes "." while still
# counting as named, which silently disables the project-root guard below
# — the one case where the guard is most likely to be right.
[ -n "$2" ] || die "--into was given an empty path. Use --into . to mean the
current directory."
TARGET="$2"; TARGET_NAMED="yes"; shift ;;
--into=*)
TARGET="${1#--into=}"; TARGET_NAMED="yes"
[ -n "$TARGET" ] || die "--into needs a path." ;;
-h|--help) usage; exit 0 ;;
*) die "unknown argument '$1'. Run --help for usage." ;;
esac
shift
done
# ---------------------------------------------------------------------------
# Resolve both ends before touching either. Everything checkable is checked
# before the first file is written.
# ---------------------------------------------------------------------------
[ -n "$TEMPLATE_ROOT" ] || die "set TEMPLATE_ROOT (or SCAFFOLD_TEMPLATE_ROOT) — the path to the
template checkout holding docs/. See the CONFIGURATION block."
# A leading ~ inside a variable is a literal character, not $HOME. The failure
# is otherwise "no such directory" naming a path the operator can see exists,
# which sends them looking in the wrong place.
case "$TEMPLATE_ROOT" in
'~'*) die "TEMPLATE_ROOT is '$TEMPLATE_ROOT' — a leading ~ is not expanded inside a
variable. Write the path out, or use \"\$HOME/...\"." ;;
esac
[ -d "$TEMPLATE_ROOT" ] || die "TEMPLATE_ROOT '$TEMPLATE_ROOT' is not a directory."
TEMPLATE_ABS=$(cd "$TEMPLATE_ROOT" 2>/dev/null && pwd -P) \
|| die "cannot enter TEMPLATE_ROOT '$TEMPLATE_ROOT' — check permissions."
[ -d "$TEMPLATE_ABS/docs" ] || die "'$TEMPLATE_ABS' has no docs/ directory, so it is not the template
root. TEMPLATE_ROOT is the checkout, not the docs folder inside it."
# Same trap as TEMPLATE_ROOT above, and worse here: the bare "does not exist"
# message names a path the operator can see does exist.
case "$TARGET" in
'~'*) die "target is '$TARGET' — a leading ~ is not expanded inside a variable or
inside quotes. Write the path out, or use \"\$HOME/...\"." ;;
esac
[ -n "$TARGET" ] || TARGET="."
# Refused rather than created. `mkdir -p` on a typo succeeds, and the result is
# a complete, correct documentation tree in a directory that should not exist —
# which reads as success and is found weeks later, if at all.
[ -d "$TARGET" ] || die "target '$TARGET' does not exist. Create it first — scaffolding into a
mistyped path produces a tree that looks entirely correct."
TARGET_ABS=$(cd "$TARGET" 2>/dev/null && pwd -P) \
|| die "cannot enter target '$TARGET' — check permissions."
[ "$TARGET_ABS" != "$TEMPLATE_ABS" ] \
|| die "the target is the template root itself. There is nothing to scaffold."
# ---------------------------------------------------------------------------
# An inherited current directory is the one input nobody typed.
#
# Run from src/ or from docs/ itself — both of which are where you are when you
# think to run this — the default would build a second, wrong-place tree that
# every convention here then fails to find. Only guessed targets are checked;
# --into is an instruction and is obeyed.
# ---------------------------------------------------------------------------
if [ -z "$TARGET_NAMED" ]; then
if command -v git >/dev/null 2>&1; then
guess_top=$(git -C "$TARGET_ABS" rev-parse --show-toplevel 2>/dev/null)
if [ -n "$guess_top" ]; then
guess_top=$(cd "$guess_top" 2>/dev/null && pwd -P)
if [ -n "$guess_top" ] && [ "$guess_top" != "$TARGET_ABS" ]; then
die "you are in a subdirectory of '$guess_top'. docs/ belongs at the project
root. Re-run with --into '$guess_top', or --into . to mean here."
fi
fi
else
# Said out loud rather than passed over. Without git this check cannot run
# at all, and a silent skip reads exactly like a check that passed.
say "note: git is not installed, so the \"is this the project root?\" check did"
say " not run. '$TARGET_ABS' is being taken as the root on trust."
fi
fi
# ---------------------------------------------------------------------------
# --force may only run where the damage is undoable.
#
# Overwriting is recoverable exactly when git already holds the current
# contents. A dirty tree means it does not, and an untracked file is the worst
# case of all — scaffolded an hour ago, written into since, never committed,
# and nothing anywhere remembers what it said. So untracked counts as dirty.
#
# This is the tree-wide half of the check, and on its own it is not enough:
# `git status --porcelain` is silent about ignored files, so the per-file
# `ls-files` test in the copy loop below is what actually closes the gap.
#
# Every way of failing to answer the question is refused, not passed over.
# ---------------------------------------------------------------------------
if [ -n "$FORCE" ]; then
command -v git >/dev/null 2>&1 \
|| die "--force needs git, so an overwrite can be undone with 'git checkout'."
force_top=$(git -C "$TARGET_ABS" rev-parse --show-toplevel 2>/dev/null) \
|| force_top=""
[ -n "$force_top" ] || die "--force refused: '$TARGET_ABS' is not inside a git repository, so an
overwrite would be permanent. Commit the tree somewhere first, or drop
--force and let existing files be kept."
if ! dirty=$(git -C "$force_top" status --porcelain 2>/dev/null); then
die "--force refused: could not read git status in '$force_top'. Not knowing
whether the tree is clean is a reason to stop, not to continue."
fi
if [ -n "$dirty" ]; then
say "--force refused: '$force_top' has uncommitted or untracked changes:"
printf '%s\n' "$dirty" | head -n 10 >&2
die "commit or stash them first. --force is only safe when git already holds
what it is about to replace."
fi
fi
say "template $TEMPLATE_ABS"
say "target $TARGET_ABS"
[ -n "$DRY_RUN" ] && say "--dry-run: nothing will be written."
[ -n "$FORCE" ] && say "--force: existing files WILL be replaced (git tree is clean)."
say ""
# ---------------------------------------------------------------------------
# The copy.
# ---------------------------------------------------------------------------
CREATED=0
KEPT=0
OVERWRITTEN=0
MISSING_TEMPLATE=0
UNREADABLE=0
CONFLICTED=0
FAILED=0
HEADERLESS=0
MISSING_LIST=()
PLACEHOLDER_LIST=()
report() { printf ' %-13s %s%s\n' "$1" "$2" "${3:+ ($3)}" >&2; }
# Every document opens with an H1 and then a fenced status header. Checked on
# the template's copy rather than the result, because a header missing here
# means the template drifted and every project scaffolded from it starts
# non-conformant — a fact about the template, reported as one.
has_status_header() {
awk 'NR==1 { if ($0 !~ /^# /) { bad=1; exit } }
NR>1 && NR<=4 { if ($0 ~ /^```/) { ok=1; exit } }
NR>4 { exit }
END { exit (ok && !bad) ? 0 : 1 }' "$1" 2>/dev/null
}
for rel in "${DIRS[@]}"; do
dst="$TARGET_ABS/$rel"
if [ -d "$dst" ]; then
report "kept" "$rel/" "directory exists"
KEPT=$((KEPT + 1))
continue
fi
if [ -e "$dst" ] || [ -L "$dst" ]; then
report "SKIPPED" "$rel/" "exists and is not a directory"
CONFLICTED=$((CONFLICTED + 1))
continue
fi
if [ -n "$DRY_RUN" ]; then
report "would create" "$rel/" "empty directory"
CREATED=$((CREATED + 1))
continue
fi
if mkdir -p "$dst" 2>/dev/null; then
report "created" "$rel/" "empty directory"
CREATED=$((CREATED + 1))
else
report "FAILED" "$rel/" "could not create"
FAILED=$((FAILED + 1))
fi
done
for rel in "${DOCS[@]}"; do
src="$TEMPLATE_ABS/$rel"
dst="$TARGET_ABS/$rel"
# Reported, counted and returned in the exit status. A template missing a
# file it is supposed to carry produces a tree that is wrong in a way only
# this run can see — by the next run, "not in the template" and "already
# here" look the same from the target's side.
if [ ! -e "$src" ]; then
report "SKIPPED" "$rel" "missing from the template"
MISSING_TEMPLATE=$((MISSING_TEMPLATE + 1))
MISSING_LIST+=("$rel")
continue
fi
# Present but unreadable is a different fact with a different fix — a mode
# bit here, a missing document there. Reporting both as "missing from the
# template" sends someone to write a file that is already sitting in front
# of them.
if [ ! -f "$src" ] || [ ! -r "$src" ]; then
report "SKIPPED" "$rel" "in the template but not a readable regular file"
UNREADABLE=$((UNREADABLE + 1))
continue
fi
has_status_header "$src" || HEADERLESS=$((HEADERLESS + 1))
# A symlink is followed by cp, so "writing into docs/" could land anywhere on
# the disk, including on top of the file it points at. Never written through,
# in either direction, --force included.
if [ -L "$dst" ]; then
report "SKIPPED" "$rel" "is a symlink — never written through"
CONFLICTED=$((CONFLICTED + 1))
continue
fi
if [ -e "$dst" ] && [ ! -f "$dst" ]; then
report "SKIPPED" "$rel" "exists and is not a regular file"
CONFLICTED=$((CONFLICTED + 1))
continue
fi
if [ -f "$dst" ]; then
if [ -z "$FORCE" ]; then
report "kept" "$rel" "already here, not touched"
KEPT=$((KEPT + 1))
continue
fi
# An identical file is not an overwrite worth announcing, and calling it
# one inflates the number that is supposed to mean "work was replaced".
if cmp -s "$src" "$dst"; then
report "kept" "$rel" "identical to the template"
KEPT=$((KEPT + 1))
continue
fi
# The clean-tree check above is necessary and not sufficient. `git status
# --porcelain` says nothing about ignored files, so a repository with
# docs/qa/ClaudeReport.md in .gitignore reports clean while git holds no
# copy of it at all. Overwriting that is exactly the permanent, silent loss
# the whole script is built to refuse, announced as "(git tree is clean)".
#
# So the question is asked per file, of git, in the only form that answers
# it: does git track this path. Anything else — ignored, untracked, or an
# unreadable index — is refused and counted, never overwritten.
if ! git -C "$force_top" ls-files --error-unmatch -- "$dst" >/dev/null 2>&1; then
report "SKIPPED" "$rel" "--force: git does not track it (ignored?) — overwrite would be permanent"
CONFLICTED=$((CONFLICTED + 1))
continue
fi
if [ -n "$DRY_RUN" ]; then
report "would REPLACE" "$rel" "--force, differs from the template"
OVERWRITTEN=$((OVERWRITTEN + 1))
continue
fi
elif [ -n "$DRY_RUN" ]; then
report "would create" "$rel"
CREATED=$((CREATED + 1))
continue
fi
if ! mkdir -p "$(dirname "$dst")" 2>/dev/null; then
report "FAILED" "$rel" "could not create its directory"
FAILED=$((FAILED + 1))
continue
fi
existed=""
[ -f "$dst" ] && existed="yes"
if ! cp -- "$src" "$dst" 2>/dev/null; then
report "FAILED" "$rel" "copy failed"
FAILED=$((FAILED + 1))
continue
fi
# Verified rather than assumed. A short write on a full disk leaves a file
# that exists, reports as created, and is a truncated document — which is the
# one outcome here that is worse than not copying at all.
if ! cmp -s "$src" "$dst"; then
report "FAILED" "$rel" "copied file does not match the template — check disk space"
FAILED=$((FAILED + 1))
continue
fi
if [ -n "$existed" ]; then
report "REPLACED" "$rel" "--force"
OVERWRITTEN=$((OVERWRITTEN + 1))
else
report "created" "$rel"
CREATED=$((CREATED + 1))
fi
grep -Eq "$PLACEHOLDER_RE" "$dst" 2>/dev/null && PLACEHOLDER_LIST+=("$rel")
done
SKIPPED=$((MISSING_TEMPLATE + UNREADABLE + CONFLICTED + FAILED))
say ""
if [ -n "$DRY_RUN" ]; then
say "--dry-run: nothing was written. It would have made"
say " ${CREATED} created, ${KEPT} kept, ${OVERWRITTEN} replaced, ${SKIPPED} skipped."
else
say "${CREATED} created, ${KEPT} kept, ${OVERWRITTEN} replaced, ${SKIPPED} skipped."
fi
say " skipped breaks down as ${MISSING_TEMPLATE} missing from the template,"
say " ${UNREADABLE} present in the template but unreadable, ${CONFLICTED} conflicting"
say " with something already at that path, ${FAILED} failed."
if [ "$MISSING_TEMPLATE" -gt 0 ]; then
say ""
say "WARNING: the template at $TEMPLATE_ABS does not have:"
printf ' %s\n' ${MISSING_LIST[@]+"${MISSING_LIST[@]}"} >&2
say " This tree is incomplete and nothing downstream will say so."
say " Fix the template, or write those files by hand."
fi
if [ "$HEADERLESS" -gt 0 ]; then
say ""
say "WARNING: ${HEADERLESS} template document(s) do not open with an H1 followed by"
say " a fenced block within the first four lines — the shape a status"
say " header is written in. Only that shape was checked: the field"
say " names (Status / Owner / Last reviewed / Governs / Review"
say " trigger) were not read, so a passing document is not a"
say " conformant one. Every project scaffolded from this template"
say " inherits whatever is there. Fix it in the template."
fi
# Counted only among files this run created, and silent when there are none —
# an "all placeholders filled in" line would be a claim about documents this
# run never looked at, and about a placeholder style that may simply have
# changed underneath the pattern above.
if [ "${#PLACEHOLDER_LIST[@]}" -gt 0 ]; then
say ""
say "${#PLACEHOLDER_LIST[@]} file(s) written by this run still carry placeholders. Every status"
say "header needs a real Owner, a real Last reviewed date and a real Review"
say "trigger — that last line is what stops a document going quietly stale."
say "Find them with:"
say " grep -rnE '$PLACEHOLDER_RE' docs/"
fi
# ---------------------------------------------------------------------------
# Branding: named, measured, and never invented.
#
# The template's own docs/data/img is not a source. When this was written it
# held one zero-byte logo.webp, and copying that in would satisfy a check that
# looks for the name while producing a project card with nothing on it. The
# deeper reason is the one docs/data/README.md gives: a placeholder that looks
# deliberate outlives the issue that would have replaced it, because nobody
# files a ticket against an image that appears finished.
#
# So the directory is created empty and the three names are reported by what is
# actually there — including the difference between absent and present-but-
# empty, which the check itself distinguishes and a file listing does not.
# ---------------------------------------------------------------------------
img_dir="$TARGET_ABS/docs/data/img"
say ""
say "docs/data/img/ needs this project's own branding. The conformance check"
say "looks for exactly these three names, and reads webp by magic bytes rather"
say "than by extension — a renamed PNG does not pass:"
if [ -d "$img_dir" ]; then
for name in "${BRANDING[@]}"; do
if [ -s "$img_dir/$name" ]; then
report "present" "docs/data/img/$name"
elif [ -e "$img_dir/$name" ]; then
report "EMPTY" "docs/data/img/$name" "0 bytes — fails the magic-byte check"
else
report "absent" "docs/data/img/$name"
fi
done
else
# Not "none present". The directory does not exist yet, so nothing about
# these three files has been measured and no count would be honest.
say " not checked — $img_dir does not exist yet."
printf ' %s\n' ${BRANDING[@]+"${BRANDING[@]}"} >&2
fi
say ""
say "Nothing here can draw them. When one is absent, file an issue — title it"
say "for the asset, label it P2, and end the body with its Verify: line — rather"
say "than committing a placeholder."
say ""
say "note: no git command was run. Nothing is staged and nothing is committed;"
say " what belongs in this project's history is your call."
# Non-zero when the tree that came out is not the tree that was asked for. The
# caller — often another script, or an agent — needs that distinction to survive
# past the last line of output.
if [ "$SKIPPED" -gt 0 ]; then
say ""
die "${SKIPPED} path(s) were skipped, so this project is not fully scaffolded.
Resolve each one listed above and run this again — re-running is safe,
because everything already here is kept."
fi
exit 0

166
docs/architecture/scripts/secrets.sh Executable file
View File

@ -0,0 +1,166 @@
#!/usr/bin/env bash
#
# Credentials, before they are committed.
#
# ## Why this and not a generic scanner
#
# A general-purpose secret scanner knows about AWS keys and GitHub tokens. It
# does not know that *this* deployment issues `pllc_agent_<hex>`, or that its
# encrypted envelopes start `v2:<keyid>:`, or which of its environment variables
# hold a password. The project does know, and usually writes it down twice: once
# in whatever redacts its logs, and once in whatever redacts its outbound
# messages.
#
# So this reads the project's own patterns where they exist — point
# `SECRETS_PATTERN_FILE` at the module holding them — and falls back to a
# conservative built-in set. A scanner tuned to the shapes a project actually
# issues catches the leak a generic one misses, and stays quiet the rest of the
# time.
#
# ## What it scans
#
# By default the **staged diff**, which is the only moment a commit can still be
# stopped cheaply. `--tracked` scans every tracked file instead, which is what
# you want once, on adoption, to find what is already in the history's tip.
#
# bash scripts/secrets.sh # staged changes (use in pre-commit)
# bash scripts/secrets.sh --tracked # everything tracked, for an audit
# SECRETS_PATTERN_FILE=src/lib/log.ts bash scripts/secrets.sh
# bash scripts/secrets.sh --allow docs/examples/
#
# ## What it cannot do
#
# It reads the working tree and the index. **A secret already committed is still
# in the history** after you delete it, and this will not tell you that — the
# fix there is a rotation, not a scan. Rotate first, then clean up.
#
# Exit codes: 0 nothing found. 1 a candidate found. 2 nothing was scanned.
# ---------------------------------------------------------------------------
set -uo pipefail
cd "$(git rev-parse --show-toplevel 2>/dev/null)" || {
printf 'secrets: not a git repository.\n' >&2
exit 2
}
say() { printf 'secrets: %s\n' "$*" >&2; }
MODE="staged"
ALLOW=()
while [ $# -gt 0 ]; do
case "$1" in
--tracked) MODE="tracked"; shift ;;
--staged) MODE="staged"; shift ;;
--allow) ALLOW+=("${2:-}"); shift 2 ;;
*) say "unknown argument: $1"; exit 2 ;;
esac
done
# The built-in set. Deliberately shapes that are *structurally* credential-like
# rather than words that merely appear near credentials — `password` in a
# sentence is not a leak, and a scanner that says it is gets muted.
PATTERNS=(
'//[^/@[:space:]:]+:[^/@[:space:]]+@' # user:pass@host in a URL
'[?&](token|key|secret|password|access_token|api_key)=[^&[:space:]"]+'
'\b(Bearer|Basic)[[:space:]]+[A-Za-z0-9._~+/=-]{20,}' # an authorization header
# Anchored to the start of a line or an `export`, because unanchored it
# matched `access_token = $1` in SQL and `apiKey=` in a property list — three
# findings in src/ that were column names, not credentials.
'(^|export )[A-Z][A-Z0-9_]*(SECRET|TOKEN|PASSWORD|API_KEY|PASSWD)[A-Z0-9_]*=[^[:space:]"'"'"']{8,}'
'-----BEGIN [A-Z ]*PRIVATE KEY-----'
'\bghp_[A-Za-z0-9]{20,}' # GitHub
'\bxox[baprs]-[A-Za-z0-9-]{10,}' # Slack
'\bAKIA[0-9A-Z]{16}\b' # AWS access key id
)
# The project's own shapes, if it has written them down. A `pllc_agent_<hex>`
# token is invisible to every generic scanner and obvious to the module that
# redacts it.
if [ -n "${SECRETS_PATTERN_FILE:-}" ] && [ -f "$SECRETS_PATTERN_FILE" ]; then
loaded=0
# Anchored on the closing `/flags,` and greedy to it, rather than on "no
# commas". The first version used `[^,]+`, which cannot cross the comma inside
# a bounded quantifier — so `pllc_[a-z]+_[0-9a-f]{8,}` was silently dropped
# along with every other `{n,}` pattern: four of six on the file this was
# written against, while the script printed that it had loaded them.
while IFS= read -r found; do
[ -n "$found" ] || continue
PATTERNS+=("$found")
loaded=$((loaded + 1))
done < <(
sed -nE 's/.*\[\/(.+)\/[gimsuy]*,[[:space:]]*".*/\1/p' "$SECRETS_PATTERN_FILE" 2>/dev/null || true
)
# The count, never a bare reassurance. "Loaded project patterns" over an empty
# list is the same lie as a green test run that executed nothing.
if [ "$loaded" -gt 0 ]; then
say "loaded $loaded project pattern(s) from $SECRETS_PATTERN_FILE"
else
say "WARNING: $SECRETS_PATTERN_FILE yielded no patterns — scanning with the"
say " built-in set only. Check the file holds regex literals."
fi
fi
if [ "$MODE" = "staged" ]; then
# Added lines only. A removed line containing a token is somebody deleting
# one, which is the opposite of a leak.
CONTENT="$(git diff --cached --unified=0 --no-color | grep '^+' | grep -v '^+++' || true)"
WHAT="staged changes"
else
CONTENT=""
WHAT="tracked files"
while IFS= read -r file; do
skip=""
for allowed in ${ALLOW[@]+"${ALLOW[@]}"}; do
case "$file" in *"$allowed"*) skip="yes" ;; esac
done
[ -n "$skip" ] && continue
[ -f "$file" ] || continue
# Text only; a webp full of bytes will match anything.
file "$file" 2>/dev/null | grep -q "text" || continue
CONTENT+="$(sed "s|^|${file}: |" "$file")"$'\n'
done < <(git ls-files)
fi
if [ -z "$CONTENT" ]; then
say "nothing to scan in $WHAT."
exit 2
fi
found=0
for pattern in "${PATTERNS[@]}"; do
while IFS= read -r hit; do
[ -n "$hit" ] || continue
skip=""
for allowed in ${ALLOW[@]+"${ALLOW[@]}"}; do
case "$hit" in *"$allowed"*) skip="yes" ;; esac
done
[ -n "$skip" ] && continue
# The line is printed truncated and the match is never echoed in full — a
# scanner that prints the secret it found has published it to the terminal
# scrollback, the CI log, and wherever that log is shipped.
printf ' %.120s…\n' "$hit"
found=$((found + 1))
done < <(printf '%s\n' "$CONTENT" | grep -nEI "$pattern" 2>/dev/null | head -20)
done
if [ "$found" -gt 0 ]; then
say "$found candidate credential(s) in $WHAT."
say "If one is real: rotate it first. Deleting the line does not remove it"
say "from a commit that already exists, and the scan cannot see history."
exit 1
fi
say "no credential shapes in $WHAT."

678
docs/architecture/scripts/status.sh Executable file
View File

@ -0,0 +1,678 @@
#!/usr/bin/env bash
#
# What is actually running right now on the host this project deploys to, and
# its logs. Read-only: it inspects and it tails, and it does nothing else.
#
# bash scripts/status.sh # up? which version? how long?
# bash scripts/status.sh --logs # last 100 lines
# bash scripts/status.sh --logs 500
# bash scripts/status.sh --logs all
# bash scripts/status.sh --follow # stream until Ctrl-C
# bash scripts/status.sh --follow 20 # stream, after 20 lines of context
# bash scripts/status.sh --deployed-version # just the version, for scripts
# bash scripts/status.sh --dry-run # print the commands, contact nothing
#
# STATUS_CONTAINER=other-app bash scripts/status.sh # one-off, another target
#
# Exit codes, because the whole point is that a caller can act on the answer:
#
# 0 running — healthy, still starting, or no healthcheck defined
# 1 configuration or usage error; nothing was contacted
# 2 COULD NOT DETERMINE — host unreachable, daemon silent, or an error this
# does not recognise. This is not an outage report. Nothing is known.
# 3 the container does not exist on that host
# 4 the container exists but is not running
# 5 running, and its own healthcheck says unhealthy
#
# stdout carries only what a caller should parse — the log lines, or the one
# version string. The human report goes to stderr, so `status.sh
# --deployed-version` can be read straight into a variable without a filter.
#
# ===========================================================================
# TEMPLATE COPY — configure this before the first run
# ===========================================================================
#
# Copy to `scripts/status.sh` and set the two values in the CONFIGURATION block
# below. The script refuses to run until they are set: it has no defaults,
# deliberately, for the reason `release.sh` gives at length — a copied script
# carrying another project's host and container name answers confidently about
# the wrong machine, and an answer is exactly what you came here for. A status
# tool that lies is worse than no status tool, because you act on it.
#
# Assumes: bash, coreutils, git-less (it reads nothing from the repository), and
# docker on the target. `ssh` only when the target is not this machine.
#
# ## Why this exists
#
# Without it, "is it up?" is answered by typing docker incantations from memory,
# which in practice means `docker ps | grep`. That shows a name and a status and
# none of the four things you actually wanted: which version is deployed, how
# long it has been up, how many times it has restarted, and what its healthcheck
# thinks. So the question gets half-answered, and the half that is missing —
# restart count — is the one that distinguishes "running" from "crash-looping
# and up for nine seconds".
#
# The other half is worse. When there is no command that answers "what version
# is deployed", every script that needs the answer grows its own copy of the
# lookup. `release.sh` has one: a single `ssh … docker inspect --format` whose
# empty return value is the ONLY thing standing between the prune and deleting
# the published image production is currently running. That fact is load-bearing
# enough to deserve a name, a documented exit code, and one place to fix — not a
# line pasted into whichever script needed it that week, each copy free to
# disagree about what an empty answer means.
#
# Hence `--deployed-version`: one fact, on stdout, non-zero when it could not be
# read. A caller that treats non-zero as "stop" cannot make the prune's mistake.
#
# ## Unreachable is not stopped
#
# These are different facts and only one of them is an outage you caused:
#
# - the container is not running → something happened to the deployment
# - I could not reach the host → something happened to the network, or to
# ssh, or to the daemon, and the container
# is very probably still serving traffic
#
# Rendering the second as the first is how a 2am investigation starts by
# restarting a healthy service. So a failed connection exits 2 and says so in
# those words; only an inspect that succeeded and reported a non-running state
# exits 4. Nothing here ever prints "stopped", "0" or "ok" for something it did
# not measure — an unknown is printed as an unknown, every time.
#
# ssh runs with BatchMode=yes and a connect timeout, so a host that would prompt
# for a password or a host key is read as unreachable in seconds instead of
# hanging forever on a prompt nobody is watching.
#
# ## There is no --host flag
#
# Host and container come from the CONFIGURATION block, or from the environment
# for a one-off. A flag would make it one keystroke to read host A and file the
# answer under host B, and the environment form is self-documenting in shell
# history — which is where you will be reading it back from.
#
# ## What it deliberately does not do
#
# It never starts, stops, restarts, recreates, kills or removes anything, and it
# never pulls or prunes an image. Every docker verb it runs is `inspect`,
# `version` or `logs`. This is not an oversight to be fixed later: a status tool
# is the one command people run half-awake and without reading, so it must be
# safe to run half-awake and without reading. Fixing what it reports is a
# separate, deliberate act with a separate command.
#
# It does not compare the deployed version against this repository, and does not
# read package.json or any other file. Whether the deployed version is the RIGHT
# version is a judgement, and a judgement needs a human or a release script that
# owns the decision; this only reports what is there.
#
# It reports one configured container, not every container on the host. A tool
# that enumerates is a tool that finds another project's container and reports it
# as yours.
set -uo pipefail
# ---------------------------------------------------------------------------
# CONFIGURATION — set these two, then delete this banner.
#
# Both are empty on purpose. See the note at the top.
# ---------------------------------------------------------------------------
# An ssh destination — a Host in ~/.ssh/config, an IP, or user@host — or the
# literal word `local` to ask this machine's own docker daemon.
#
# `local` is a word you have to type rather than what an empty value means,
# because "you did not configure a host" and "the host is this machine" must not
# look the same. Defaulting to local would make an unconfigured copy of this
# script quietly report on your laptop and call it production.
HOST="${STATUS_HOST:-}"
# The container name (or id) to report on. Exactly one.
CONTAINER="${STATUS_CONTAINER:-}"
# Seconds to wait for the ssh connection. Short on purpose: this command exists
# to be run when something might be wrong, and a status tool that hangs is
# indistinguishable from the outage it was meant to describe.
SSH_TIMEOUT="${STATUS_SSH_TIMEOUT:-10}"
# Lines tailed by `--logs` with no number given.
TAIL_DEFAULT="${STATUS_TAIL:-100}"
# The image label carrying the deployed version. This one has a default because
# it is an open standard rather than a fact about your project; override it only
# if your build stamps a different label.
VERSION_LABEL="${STATUS_VERSION_LABEL:-org.opencontainers.image.version}"
# ---------------------------------------------------------------------------
E_CONFIG=1
E_UNKNOWN=2
E_MISSING=3
E_STOPPED=4
E_UNHEALTHY=5
say() { printf '\033[1mstatus:\033[0m %s\n' "$*" >&2; }
die() { printf '\033[1mstatus:\033[0m %s\n' "$*" >&2; exit "$E_CONFIG"; }
# die() with a chosen code. Every caller of this is a distinct fact a script
# downstream is expected to branch on, so the code is passed explicitly at the
# point the fact is established rather than mapped from a message afterwards.
quit() { local code="$1"; shift; printf '\033[1mstatus:\033[0m %s\n' "$*" >&2; exit "$code"; }
field() { printf '\033[1mstatus:\033[0m %-9s %s\n' "$1" "$2" >&2; }
usage() {
say "usage: bash scripts/status.sh [--logs [N|all]] [--follow [N]] [--deployed-version] [--dry-run]"
say " set STATUS_HOST and STATUS_CONTAINER, or edit the CONFIGURATION block."
}
# ---------------------------------------------------------------------------
# Arguments, before the configuration check, so that `--help` answers on a
# fresh unconfigured copy — which is the one moment somebody needs it.
# ---------------------------------------------------------------------------
MODE="report"
TAIL=""
DRY_RUN=""
# `("$@")` with no arguments is an unbound expansion under `set -u` on bash
# before 4.4, which is still what ships on macOS. Guarded rather than assumed.
if [ "$#" -gt 0 ]; then ARGS=("$@"); else ARGS=(); fi
# Refused rather than resolved last-flag-wins. `--deployed-version --logs` from a
# script would otherwise return a log stream on stdout to a caller that is about
# to compare it to a version string.
set_mode() {
[ "$MODE" = "report" ] || [ "$MODE" = "$1" ] \
|| die "--logs, --follow and --deployed-version ask different questions; pick one."
MODE="$1"
}
i=0
while [ "$i" -lt "${#ARGS[@]}" ]; do
arg="${ARGS[$i]}"
case "$arg" in
--logs|-l|--follow|-f)
case "$arg" in
--logs|-l) set_mode "logs" ;;
*) set_mode "follow" ;;
esac
# An optional operand rather than a required one, so `--logs` alone is a
# whole command. Only consumed when it looks like a count — otherwise
# `--logs --dry-run` would swallow the flag and silently mean something
# else.
if [ $((i + 1)) -lt "${#ARGS[@]}" ]; then
case "${ARGS[$((i + 1))]}" in
all|[0-9]*) TAIL="${ARGS[$((i + 1))]}"; i=$((i + 1)) ;;
esac
fi
;;
--logs=*) set_mode "logs"; TAIL="${arg#*=}" ;;
--follow=*) set_mode "follow"; TAIL="${arg#*=}" ;;
--deployed-version) set_mode "version" ;;
--dry-run) DRY_RUN="yes" ;;
-h|--help) usage; exit 0 ;;
*) usage; die "unknown argument '$arg'." ;;
esac
i=$((i + 1))
done
[ -n "$TAIL" ] || TAIL="$TAIL_DEFAULT"
case "$TAIL" in
all) : ;;
''|*[!0-9]*) die "line count must be a whole number or 'all', got '$TAIL'." ;;
esac
case "$SSH_TIMEOUT" in
''|*[!0-9]*) die "STATUS_SSH_TIMEOUT must be a whole number of seconds, got '$SSH_TIMEOUT'." ;;
esac
# ---------------------------------------------------------------------------
# Refuse to run half-configured, before anything is contacted, and name the
# missing value one at a time so the message says which one.
# ---------------------------------------------------------------------------
[ -n "$HOST" ] || die "set HOST (or STATUS_HOST) — the ssh host running the container, or the word 'local'. See the CONFIGURATION block."
[ -n "$CONTAINER" ] || die "set CONTAINER (or STATUS_CONTAINER) — the container name to report on."
# ssh takes its destination positionally, so a host beginning with a dash is
# parsed as an option and the failure that follows describes something else
# entirely. Refused here, where the message can still name the real cause.
case "$HOST" in
-*) die "HOST ('$HOST') begins with '-'; ssh would read it as an option. Use user@host, or a Host from ~/.ssh/config." ;;
esac
case "$CONTAINER" in
-*) die "CONTAINER ('$CONTAINER') begins with '-'; docker would read it as a flag." ;;
esac
# The label name is interpolated into a Go template as a quoted string below. A
# quote or backslash in it would end that string early, and the template error
# that followed would be reported as "could not inspect" — an unknown, blamed on
# the host, caused here.
case "$VERSION_LABEL" in
''|*[\"\\]*) die "STATUS_VERSION_LABEL must be a plain label name without quotes or backslashes, got '$VERSION_LABEL'." ;;
esac
# ---------------------------------------------------------------------------
# Running docker here or over there.
#
# ssh joins its command arguments with spaces and hands the result to a remote
# shell, which splits it again. Anything containing a space — every Go template
# below does — arrives as several arguments unless it is quoted for that second
# parse. Quoting is done here, once, rather than by hand at each call site.
# ---------------------------------------------------------------------------
shquote() {
local out="" a q="'" esc="'\\''"
for a in "$@"; do
a="${a//$q/$esc}"
out="${out}${out:+ }'${a}'"
done
printf '%s' "$out"
}
# ServerAlive is here for --follow: a connection that dies mid-stream otherwise
# leaves the terminal sitting quietly forever, which looks exactly like a service
# that has stopped logging. Absence of output must not be able to mean two
# things, so the connection is made to fail loudly instead.
ssh_opts() {
printf '%s' "-o BatchMode=yes -o ConnectTimeout=${SSH_TIMEOUT} -o ServerAliveInterval=15 -o ServerAliveCountMax=3"
}
SSH_WANT_TTY=""
dock() {
if [ "$HOST" = "local" ]; then
docker "$@"
return
fi
# A tty makes the remote `docker logs -f` receive a hangup when you press
# Ctrl-C. Without one it keeps running on the far side, holding the daemon's
# log stream open, and enough abandoned follows are a real resource leak on a
# box you are already worried about. Requested only when we have a tty to
# give, because ssh -t without one is a warning and no tty anyway.
if [ -n "$SSH_WANT_TTY" ] && [ -t 0 ]; then
# shellcheck disable=SC2046 # deliberate word-splitting of the option list
ssh -t $(ssh_opts) "$HOST" "docker $(shquote "$@")"
else
# shellcheck disable=SC2046
ssh $(ssh_opts) "$HOST" "docker $(shquote "$@")"
fi
}
# The remote form is printed on two lines — the connection, then the command
# string the far shell receives — because the single-line form is the same text
# quoted twice and the host and container names disappear into the punctuation.
# Those two names are the entire reason to run --dry-run.
dock_show() {
if [ "$HOST" = "local" ]; then
printf ' docker %s\n' "$(shquote "$@")" >&2
else
printf ' ssh %s %s\n' "$(ssh_opts)" "$HOST" >&2
printf ' docker %s\n' "$(shquote "$@")" >&2
fi
}
# ---------------------------------------------------------------------------
# Timestamps.
#
# Docker reports RFC3339 with nanoseconds; `date -d` is GNU and `date -j -f` is
# BSD, so both are tried and NEITHER working is reported as not knowing the
# uptime rather than as an uptime of zero.
# ---------------------------------------------------------------------------
epoch_of() {
local t="${1%Z}"
t="${t%%.*}"
local e
if e=$(date -u -d "${t}Z" +%s 2>/dev/null) && [ -n "$e" ]; then
printf '%s' "$e"
return 0
fi
if e=$(date -u -j -f '%Y-%m-%dT%H:%M:%S' "$t" +%s 2>/dev/null) && [ -n "$e" ]; then
printf '%s' "$e"
return 0
fi
return 1
}
human_duration() {
local s="$1" d h m
d=$((s / 86400)); s=$((s % 86400))
h=$((s / 3600)); s=$((s % 3600))
m=$((s / 60)); s=$((s % 60))
if [ "$d" -gt 0 ]; then printf '%dd %dh %dm' "$d" "$h" "$m"
elif [ "$h" -gt 0 ]; then printf '%dh %dm' "$h" "$m"
elif [ "$m" -gt 0 ]; then printf '%dm %ds' "$m" "$s"
else printf '%ds' "$s"
fi
}
# Elapsed time since an RFC3339 stamp, or nothing at all.
#
# The zero timestamp means the event never happened — a container created but
# never started carries 0001-01-01 in StartedAt, and subtracting it yields two
# millennia of uptime presented with a straight face.
#
# A negative result is clock skew between this machine and that one, not time
# travel, and is likewise reported as not knowing rather than as a number.
elapsed_since() {
local stamp="$1" at now diff
case "$stamp" in
''|0001-01-01*) return 1 ;;
esac
at=$(epoch_of "$stamp") || return 1
now=$(date -u +%s) || return 1
diff=$((now - at))
[ "$diff" -ge 0 ] || return 1
human_duration "$diff"
}
# ---------------------------------------------------------------------------
# One inspect, one round trip.
#
# Fields are read together rather than one call each: two calls can straddle a
# restart and produce a report that never described a single moment — an uptime
# from before the restart beside a restart count from after it.
#
# The free-text fields are LAST. `read` gives the final variable everything that
# remains, delimiters included, so a '|' inside an image name or a version label
# lands harmlessly in the last field instead of shifting every field after it.
# ---------------------------------------------------------------------------
FMT='{{.State.Status}}|{{.State.Running}}|{{.RestartCount}}|{{.State.ExitCode}}|{{if .State.Health}}{{.State.Health.Status}}{{else}}{{end}}|{{.State.StartedAt}}|{{.State.FinishedAt}}|{{.Image}}|{{.Config.Image}}|{{index .Config.Labels "'"$VERSION_LABEL"'"}}'
C_STATUS=""; C_RUNNING=""; C_RESTARTS=""; C_EXIT=""; C_HEALTH=""
C_STARTED=""; C_FINISHED=""; C_DIGEST=""; C_IMAGE=""; C_VERSION=""
# Work out WHY the inspect failed, and exit with the code for that specific
# fact. Reached only after a failure, so the extra round trips cost nothing in
# the normal case and buy the one distinction this script exists to preserve.
diagnose_and_exit() {
local err="$1"
if [ "$HOST" != "local" ]; then
# shellcheck disable=SC2046
if ! ssh $(ssh_opts) "$HOST" true >/dev/null 2>&1; then
say "cannot reach ${HOST} over ssh (BatchMode, ${SSH_TIMEOUT}s timeout)."
say " This is NOT a report that the container is down — nothing was"
say " determined about it. Check the network, the ssh key, and"
say " whether the host would have prompted for a password."
exit "$E_UNKNOWN"
fi
fi
if ! dock version --format '{{.Server.Version}}' >/dev/null 2>&1; then
if [ "$HOST" = "local" ]; then
say "this machine's docker daemon did not answer."
else
say "reached ${HOST} over ssh, but its docker daemon did not answer."
fi
say " The container's state is unknown — the daemon being down does"
say " not tell you whether the container was running before it went."
exit "$E_UNKNOWN"
fi
# Matched case-insensitively against a lowercased copy, because the wording
# is not stable across docker versions: "Error: No such object: x" on one
# daemon here, "error: no such object: x" on another, and "No such container"
# from older ones. A case-sensitive match reports a container that is simply
# absent as an unknown — which is the safe direction to be wrong in, and
# still the wrong answer.
local lower
lower=$(printf '%s' "$err" | tr '[:upper:]' '[:lower:]')
case "$lower" in
*"no such object"*|*"no such container"*)
quit "$E_MISSING" "there is no container named '${CONTAINER}' on ${HOST}." ;;
esac
# An error nobody anticipated is printed verbatim and treated as an unknown.
# Guessing at an unfamiliar message is how "I could not check" becomes "there
# is nothing there".
say "could not inspect '${CONTAINER}' on ${HOST}, and the error is not one"
say " this script recognises. Nothing is known about the container:"
printf ' %s\n' "${err:-(no error output)}" >&2
exit "$E_UNKNOWN"
}
# The one file this script creates, and the one thing it ever deletes.
#
# The path is read when the trap FIRES rather than pasted into the trap's source
# text. `trap "rm -f '$f'" EXIT` looks equivalent and is not: it builds a shell
# command out of mktemp's output, which is built out of $TMPDIR, so a directory
# whose name contains a single quote closes that string early and everything
# after it runs as shell. `TMPDIR="/tmp/x'; rm -rf ~; echo '"` is a working
# exploit against the double-quoted form. This is the only destructive verb in
# the script and it must not be constructible from the environment.
#
# The -n guard means rm is never handed an empty path, and `--` means a path
# beginning with a dash is a path rather than a bundle of options.
ERRFILE=""
# shellcheck disable=SC2329 # invoked by the EXIT trap below, not by name
cleanup() {
[ -n "$ERRFILE" ] || return 0
rm -f -- "$ERRFILE"
return 0
}
trap cleanup EXIT
inspect_container() {
local out rc
ERRFILE=$(mktemp) || die "cannot create a temporary file"
out=$(dock inspect --format "$FMT" "$CONTAINER" 2>"$ERRFILE")
rc=$?
if [ "$rc" -ne 0 ] || [ -z "$out" ]; then
diagnose_and_exit "$(tr -d '\r' < "$ERRFILE" | head -n 3)"
fi
IFS='|' read -r C_STATUS C_RUNNING C_RESTARTS C_EXIT C_HEALTH \
C_STARTED C_FINISHED C_DIGEST C_IMAGE C_VERSION \
<<EOF
$(printf '%s' "$out" | tr -d '\r' | head -n 1)
EOF
}
# ---------------------------------------------------------------------------
# --dry-run: say exactly what would be contacted, and contact nothing.
#
# Nothing here is destructive, so this is not a safety catch — it is how you
# check WHICH host and WHICH container this copy is configured for before you
# quote its answer at somebody, or before a release script trusts it.
# ---------------------------------------------------------------------------
if [ -n "$DRY_RUN" ]; then
say "--dry-run: nothing was contacted. Target: ${CONTAINER} on ${HOST}."
say "It would have run:"
dock_show inspect --format "$FMT" "$CONTAINER"
case "$MODE" in
logs) dock_show logs --tail "$TAIL" "$CONTAINER" ;;
follow) dock_show logs --tail "$TAIL" --follow "$CONTAINER" ;;
esac
exit 0
fi
if [ "$HOST" = "local" ]; then
command -v docker >/dev/null 2>&1 \
|| quit "$E_UNKNOWN" "docker is not on PATH, so nothing could be determined. Install it, or set HOST to the machine that runs the container."
else
command -v ssh >/dev/null 2>&1 \
|| quit "$E_UNKNOWN" "ssh is not on PATH, so ${HOST} could not be contacted."
fi
inspect_container
# ---------------------------------------------------------------------------
# --deployed-version: one fact on stdout, non-zero when it is not known.
#
# The contract callers depend on: a zero exit means the string on stdout was
# read from the running container's image. Anything else means stop — never
# "assume none", which is the assumption that lets a prune delete production's
# image.
# ---------------------------------------------------------------------------
if [ "$MODE" = "version" ]; then
[ -n "$C_VERSION" ] \
|| quit "$E_UNKNOWN" "'${CONTAINER}' on ${HOST} is ${C_STATUS}, but its image carries no ${VERSION_LABEL} label — the deployed version is unknown."
[ "$C_RUNNING" = "true" ] \
|| quit "$E_STOPPED" "'${CONTAINER}' on ${HOST} is ${C_STATUS}, not running. Its image says ${C_VERSION}, but nothing is serving from it."
printf '%s\n' "$C_VERSION"
exit 0
fi
# ---------------------------------------------------------------------------
# --logs / --follow.
#
# The inspect above already ran, which is what makes an empty tail meaningful:
# the container is known to exist, so no output means it has logged nothing.
# Tailing blind cannot tell that apart from a container that is not there.
# ---------------------------------------------------------------------------
if [ "$MODE" = "logs" ] || [ "$MODE" = "follow" ]; then
field container "${CONTAINER} on ${HOST}"
field state "${C_STATUS}${C_VERSION:+ (${C_VERSION})}"
# Said before the first line of output, because a wall of old log lines from a
# container that died an hour ago is indistinguishable from a live one.
if [ "$C_RUNNING" != "true" ]; then
say "note: the container is ${C_STATUS}. These are the logs it left behind,"
if [ "$MODE" = "follow" ]; then
say " not a live stream — nothing is writing, so this will return at once."
else
say " not a live stream."
fi
fi
if [ "$MODE" = "follow" ]; then
SSH_WANT_TTY="yes"
say "following ${CONTAINER} — Ctrl-C to stop."
dock logs --tail "$TAIL" --follow "$CONTAINER"
else
dock logs --tail "$TAIL" "$CONTAINER"
fi
rc=$?
# A tail that failed after a successful inspect is a new fact, not a quiet
# zero: the container exists, so an empty screen would otherwise read as a
# silent service.
[ "$rc" -eq 0 ] || quit "$E_UNKNOWN" "the log stream ended with status ${rc}; some or all of the output may be missing."
# The logs printed either way; only the code differs. Exiting 0 here would
# report the documented meaning of 0 — "running" — for a container that is
# not, which is the same mistake as printing "ok" for something unmeasured.
# The state is the fact the exit code carries, in every mode.
[ "$C_RUNNING" = "true" ] \
|| quit "$E_STOPPED" "'${CONTAINER}' exists on ${HOST} but is ${C_STATUS}; the logs above are what it left behind, not a live service."
exit 0
fi
# ---------------------------------------------------------------------------
# The report.
# ---------------------------------------------------------------------------
field container "${CONTAINER} on ${HOST}"
if [ "$C_RUNNING" = "true" ]; then
field state "running"
else
field state "${C_STATUS:-unknown} (exit code ${C_EXIT:-unknown})"
fi
field image "${C_IMAGE:-(unknown)}"
# The digest is printed beside the tag because the tag is a claim and the digest
# is the fact. A tag can be moved after a container was created, at which point
# the container's image reference names something it was never built from.
if [ -n "$C_DIGEST" ]; then
case "$C_DIGEST" in
sha256:*)
short="${C_DIGEST#sha256:}"
field digest "sha256:${short:0:12}"
;;
*)
# Whatever this is, it is not a sha256 digest, so it is not labelled as
# one. The stripping form prints "sha256:" in front of the first twelve
# characters of anything at all — podman and older daemons report a bare
# image id here, and it would have been rendered as a digest it is not.
field digest "$C_DIGEST"
;;
esac
fi
if [ -n "$C_VERSION" ]; then
field version "$C_VERSION"
else
field version "(not set — this image carries no ${VERSION_LABEL} label)"
fi
if [ "$C_RUNNING" = "true" ]; then
if up=$(elapsed_since "$C_STARTED"); then
field uptime "${up} (started ${C_STARTED})"
else
# Reported as unknown, with the raw stamp, rather than as a plausible
# number computed from a timestamp that could not be read.
field uptime "unknown — could not read the start time '${C_STARTED}'"
fi
else
if down=$(elapsed_since "$C_FINISHED"); then
field stopped "${down} ago (${C_FINISHED})"
else
field stopped "at '${C_FINISHED:-unknown}'"
fi
fi
field restarts "${C_RESTARTS:-unknown}"
# The count alone is not the signal; the count beside a short uptime is. Said
# out loud because a crash loop reads as "running" in every tool that shows only
# a state, and that is the case this command was written to catch.
if [ -n "$C_RESTARTS" ] && [ "$C_RESTARTS" != "0" ] && [ "$C_RUNNING" = "true" ]; then
say " ^ restarted ${C_RESTARTS} time(s). A climbing count with a short"
say " uptime is a crash loop, not a healthy service."
fi
if [ -n "$C_HEALTH" ]; then
field health "$C_HEALTH"
else
# Not "healthy", and not "ok". No healthcheck is defined, so nothing about
# this container's health has been measured by anyone.
field health "not configured — this image defines no HEALTHCHECK, so nothing is being checked"
fi
if [ "$C_RUNNING" != "true" ]; then
quit "$E_STOPPED" "'${CONTAINER}' exists on ${HOST} but is ${C_STATUS}."
fi
if [ "$C_HEALTH" = "unhealthy" ]; then
quit "$E_UNHEALTHY" "'${CONTAINER}' is running, but its healthcheck reports unhealthy. Try --logs."
fi
exit 0

694
docs/architecture/scripts/verify.sh Executable file
View File

@ -0,0 +1,694 @@
#!/usr/bin/env bash
#
# Run every check this project has, in one command, and print one table saying
# which passed, which failed and which did not run at all.
#
# bash scripts/verify.sh # everything
# bash scripts/verify.sh --quick # skip the test step, name it as skipped
# bash scripts/verify.sh --list # show the plan, run nothing
# bash scripts/verify.sh --only npm # only steps whose id contains 'npm'
# VERIFY_GUARD_DIR=scripts/verify.d bash scripts/verify.sh
#
# Exit codes: 0 everything that ran passed. 1 something failed, or the run
# could not start at all — a usage error, a bad VERIFY_GUARD_DIR, not a git
# repository. 2 nothing was verified: either no checks were detected, or every
# detected step skipped. Two is not a pass and CI must not treat it as one.
#
# One and two are both non-zero on purpose, so every refusal fails closed. If
# you need to tell "a check failed" apart from "it never got to run", read the
# message — a failed check always prints a table first.
#
# ===========================================================================
# TEMPLATE COPY — configure this before the first run
# ===========================================================================
#
# Copy to `scripts/verify.sh`, `chmod +x` it, and wire it in: for a Node
# project add `"verify": "bash scripts/verify.sh"` to package.json; for anything
# else call it from `pre-commit`. Then read the CONFIGURATION block below —
# two environment variables (VERIFY_GUARD_DIR, VERIFY_SLOW_STEP) and six lists
# of candidate script and target names. Every one is optional except the one
# you need, which is GUARD_DIR.
#
# Unlike its sibling release.sh this script has no host, image or container to
# get wrong, so it defaults to DETECTING rather than to refusing. The rule it
# inherits unchanged is the important half: it never claims to have checked
# something it did not check. A detector that finds nothing says so and exits
# non-zero; it does not print a green table.
#
# Assumes: bash, coreutils, git. Each detector additionally needs the toolchain
# it detects, and says so by name when that toolchain is missing instead of
# quietly dropping the step.
#
# ## Why this exists
#
# "Did I break anything" should not be a judgement call, and in a project with
# three separate check commands it always is. The commands live in different
# places — a package.json script, a Makefile target, a lint you have to
# remember — so the honest answer to "did you run everything" is usually "I ran
# the one I remembered". The suite passes, the typecheck was never run, and the
# breakage is found by the deploy.
#
# One entry point removes the judgement. It also removes the excuse: there is
# no "I ran the important one" when running all of them is the same amount of
# typing.
#
# The second reason is the summary. A chain of `&&` stops at the first failure,
# so a run tells you about one broken thing at a time and you pay the whole
# cost again to find the next. This runs every step even after one fails, so a
# single run tells you everything that is broken.
#
# ## Guard tests are the point
#
# This project family keeps getting bitten by bugs that are invisible to
# ordinary tests, because the test and the bug agree with each other:
#
# - A Date serialised across a SQL boundary came back shifted by the server's
# timezone. Every test asserted against the same shifted value, so the
# suite was green and the dates were wrong.
# - A React prop was spread over a form field and silently overwrote its
# `name`. The component rendered, the test rendered it, and the field
# submitted under the wrong key.
# - A `redirect()` was called inside a `try` block. Next.js implements
# redirect by throwing, so the `catch` swallowed it and every successful
# action reported failure.
#
# None of those is a logic error a unit test would catch. All three are SHAPES
# in the source: a shape you can grep for. The fix is a guard test — a script
# that greps the source for the shape and exits non-zero when it reappears —
# and this script is where guard tests belong, because verify.sh is the thing
# that actually gets run.
#
# Write them as small executables in GUARD_DIR. One shape per file, named for
# the bug, exiting non-zero with a message naming the file and line. They cost
# milliseconds, they run first here for exactly that reason, and they are the
# only mechanism in the repository that catches a bug the tests cannot see.
#
# ## SKIPPED is not PASS
#
# A step that did not run gets its own state in the table and its own colour,
# and it is never folded into the pass count. This matters more than it looks:
# the failure this script exists to prevent is a green table produced by
# checking nothing, and every path to that failure runs through a skip that was
# reported as a success. So a run where nothing actually executed exits 2 even
# though nothing failed, and --quick names the step it dropped rather than
# quietly shortening the table.
#
# ## What it deliberately does not do
#
# It does not fix anything. No --fix, no formatter writing to your files: this
# runs immediately before a commit, and a verify that edits the tree changes
# what you were about to commit into something you have not read.
#
# It does not touch git — no staging, no committing, no stash, no branch check.
# It answers one question about the working tree as it stands.
#
# It does not walk into workspaces or sub-packages. Detection runs at the
# repository root, once. A monorepo wants a guard script per package, or a
# Makefile target that fans out, and either is a step this will find.
#
# It has no --dry-run because it writes nothing OF ITS OWN: no file is created,
# deleted, truncated or moved anywhere in this script. That is not the same as
# "changes nothing", and the difference matters. Every step it runs is somebody
# else's program — a package.json script, a Makefile target, an executable in
# GUARD_DIR — and those inherit no restraint from here. Dropping a test
# database is the usual one.
#
# --list is the dry-run equivalent: it shows the exact command each step would
# run, and runs none of them. Read it once, on a new project, before trusting
# this in a hook. That is the only place the full list is visible before it
# executes.
set -uo pipefail
say() { printf '\033[1mverify:\033[0m %s\n' "$*" >&2; }
die() { printf '\033[1mverify:\033[0m %s\n' "$*" >&2; exit 1; }
ROOT=$(git rev-parse --show-toplevel 2>/dev/null) \
|| die "not inside a git repository. Detection runs from the repository root so every detector looks in one place; cd into the repo and run this again."
# Empty is checked separately from failed. `cd ""` succeeds and stays put, so an
# empty ROOT would not fail here — it would silently detect in whatever
# directory you happened to be in, which is the wrong answer delivered
# confidently. A bare repository is the way it happens.
[ -n "$ROOT" ] \
|| die "git reported an empty repository root — this looks like a bare repository, which has no working tree to check. Run this from a normal checkout."
cd "$ROOT" || die "cannot cd to $ROOT"
# Resolved once, with symlinks collapsed, so the guard-directory containment
# check below compares two paths of the same kind.
ROOT_ABS=$(pwd -P)
# ---------------------------------------------------------------------------
# CONFIGURATION
# ---------------------------------------------------------------------------
# Where guard scripts live, relative to the repository root. Empty by default:
# an unset path is a path that cannot point at the wrong thing, and this is the
# one value worth setting by hand — see "Guard tests are the point" above.
#
# Set and missing is a hard failure, not a skip. A guard directory that got
# renamed is precisely the case where silently running zero guards looks
# identical to running them all.
#
# IT MUST BE A DEDICATED DIRECTORY. Every executable file in it is RUN, with no
# allowlist — unlike the npm and make candidate lists below, which name the
# handful of scripts they are willing to invoke. That asymmetry is the whole
# hazard: point this at scripts/ and verify.sh runs release.sh; point it at a
# bin directory and it runs every binary there. detect_guards refuses the cases
# it can prove wrong (outside the repository, the repository root itself, the
# directory holding this script), but it cannot tell a guard from a deploy
# script that happens to sit beside one. Give guards their own directory and
# put nothing else in it.
GUARD_DIR="${VERIFY_GUARD_DIR:-}"
# Which step --quick drops, by exact id (the left column of the table). Empty
# means "every step classified as a test", which is the right guess almost
# everywhere and is stated out loud when it is used.
SLOW_STEP="${VERIFY_SLOW_STEP:-}"
# Candidate script and target names, checked against what the project actually
# has. Nothing here is run unless it exists — these are search terms, not
# defaults, and the distinction is load-bearing: `npm run lint` on a project
# without a lint script fails, and a verify that invents work to do fails for
# reasons that have nothing to do with the code.
#
# It is also an allowlist, and that is the other half of why it exists. A
# package.json contains scripts named `deploy` and `db:reset`. Running
# everything found would be a verify that publishes.
NODE_TYPECHECK_SCRIPTS="typecheck type-check tsc types"
NODE_LINT_SCRIPTS="lint lint:ci eslint format:check fmt:check"
NODE_TEST_SCRIPTS="test test:ci"
# `verify` and `all` are absent on purpose: a Makefile in a repository holding
# this script very likely has a `verify` target that calls this script, and
# that is an infinite loop rather than a failed check. The guard below catches
# it anyway, for the `check` target that turns out to do the same thing.
MAKE_LINT_TARGETS="lint fmt-check format-check"
MAKE_TYPECHECK_TARGETS="typecheck type-check types"
MAKE_TEST_TARGETS="test check"
# ---------------------------------------------------------------------------
# Refuse to be re-entered.
#
# A detected step that calls this script back — `make check` running
# scripts/verify.sh is the way it happens — recurses until the machine gives
# up, and the symptom is a hang rather than an error. Dying on the second entry
# turns that into one legible failure naming the step that did it.
# ---------------------------------------------------------------------------
if [ -n "${VERIFY_RUNNING:-}" ]; then
die "verify.sh invoked itself — a detected step calls this script back, which would recurse forever. Remove that target from the candidate lists in this file, or stop it calling verify."
fi
export VERIFY_RUNNING=1
QUICK=""
LIST=""
ONLY=""
while [ $# -gt 0 ]; do
case "$1" in
--quick) QUICK="yes" ;;
--list) LIST="yes" ;;
--only)
shift
[ $# -gt 0 ] || die "--only needs a pattern. Usage: --only <substring of a step id>"
ONLY="$1" ;;
--only=*) ONLY="${1#--only=}" ;;
*) die "unknown argument '$1'. Usage: verify.sh [--quick] [--list] [--only <pattern>]" ;;
esac
shift
done
if [ -t 2 ]; then
C_PASS=$'\033[32m'; C_FAIL=$'\033[31;1m'; C_SKIP=$'\033[33m'; C_OFF=$'\033[0m'
else
C_PASS=""; C_FAIL=""; C_SKIP=""; C_OFF=""
fi
# ---------------------------------------------------------------------------
# The plan. Built completely before anything runs, so that "nothing was
# detected" is discovered before a single command executes rather than after a
# five-minute suite.
#
# Parallel arrays indexed together, iterated with a counted loop: `${arr[@]}` on
# an empty array is an unbound-variable error under `set -u` in bash 3.2, which
# is the bash on every stock macOS, and the empty case is the one that matters
# most here.
# ---------------------------------------------------------------------------
STEP_ID=()
STEP_CLASS=()
STEP_CMD=()
STEP_SKIP=()
STEP_RESULT=()
STEP_SECS=()
add_step() { STEP_ID+=("$1"); STEP_CLASS+=("$2"); STEP_CMD+=("$3"); STEP_SKIP+=(""); }
# A step that was found but cannot run. Recorded rather than dropped, because
# "this project has a mypy config and mypy is not installed" is information,
# and a silently shorter table is not.
add_skip() { STEP_ID+=("$1"); STEP_CLASS+=("skip"); STEP_CMD+=(""); STEP_SKIP+=("$2"); }
q() { printf '%q' "$1"; }
have() { command -v "$1" >/dev/null 2>&1; }
# ---------------------------------------------------------------------------
# Guards first. They are greps: they finish before the toolchain has finished
# starting, and they are the only steps that catch the bug class described in
# the header. Putting them last would mean the cheapest answer arrives after
# the most expensive one.
# ---------------------------------------------------------------------------
detect_guards() {
[ -n "$GUARD_DIR" ] || return 0
[ -d "$GUARD_DIR" ] \
|| die "VERIFY_GUARD_DIR is set to '$GUARD_DIR' but there is no such directory. Create it, or unset the variable — a missing guard directory would otherwise run zero guards and look exactly like running all of them."
# -------------------------------------------------------------------------
# Containment. Everything below this point EXECUTES every file it finds, so a
# mis-set GUARD_DIR is not a wrong answer — it is arbitrary code with the
# developer's credentials, launched by the one command they were told is safe
# to run before every commit.
#
# The three refusals below are the cases that can be proven wrong rather than
# guessed at. Resolved with `cd`+`pwd -P` so that symlinks, `..` and relative
# paths all collapse to one comparable form before being judged.
# -------------------------------------------------------------------------
local guard_abs self_abs
guard_abs=$(cd "$GUARD_DIR" 2>/dev/null && pwd -P) \
|| die "VERIFY_GUARD_DIR is set to '$GUARD_DIR' but that directory could not be entered (permissions?)."
# Outside the repository. `..`, an absolute path and a symlink pointing out of
# the tree all land here. A guard is a check on THIS repository's source; a
# directory outside it holds someone else's programs.
case "$guard_abs/" in
"$ROOT_ABS"/*) ;;
*) die "VERIFY_GUARD_DIR '$GUARD_DIR' resolves to '$guard_abs', which is outside this repository ('$ROOT_ABS'). Every executable file in it would be RUN. Guard scripts belong in a dedicated directory inside the repository." ;;
esac
# The repository root itself. Running every executable at the root means
# running whatever release, deploy or reset script the project keeps there.
[ "$guard_abs" != "$ROOT_ABS" ] \
|| die "VERIFY_GUARD_DIR points at the repository root. Every executable file at the root would be RUN as a guard, including any release, deploy or database script. Put guards in a dedicated subdirectory — scripts/verify.d is the convention."
# The directory holding this script. This is the likely mistake, because it is
# where scripts live and it reads as the obvious answer: this template ships
# verify.sh beside release.sh, backup.sh and migrate.sh, and pointing the
# guard directory here would run all three and call the result a passing
# check. Skipped when $0 cannot be resolved, which loses nothing — the two
# refusals above still apply.
self_abs=$(cd "$(dirname -- "$0")" 2>/dev/null && pwd -P) || self_abs=""
if [ -n "$self_abs" ] && [ "$guard_abs" = "$self_abs" ]; then
die "VERIFY_GUARD_DIR points at the directory holding verify.sh itself ('$guard_abs'). Every executable file beside this script — release.sh, backup.sh, migrate.sh — would be RUN as a guard. Put guards in a dedicated subdirectory of their own."
fi
local f path found=""
while IFS= read -r f; do
[ -n "$f" ] || continue
found="yes"
# Made explicitly relative so the guard runs whether or not its directory is
# on PATH — but only when it is not already absolute, since './' in front of
# an absolute path silently resolves somewhere else entirely.
case "$f" in /*) path="$f" ;; *) path="./$f" ;; esac
# A guard that lost its executable bit never runs and nothing notices. That
# is the same failure the whole script is about, one directory down. So is a
# guard symlinked in from a shared directory whose target has since moved:
# both are recorded, neither is dropped.
if [ -x "$f" ]; then
add_step "guard:$(basename "$f")" guard "$(q "$path")"
elif [ ! -e "$f" ]; then
add_skip "guard:$(basename "$f")" "broken symlink — it points at nothing, so this guard has not run since the target moved"
else
add_skip "guard:$(basename "$f")" "not executable — chmod +x it, or it will never run again either"
fi
# -L so a symlinked guard, and a symlinked guard DIRECTORY, are both seen.
# Sharing one guard across sibling repositories by symlink is the normal way
# to do it, and plain `-type f` does not match a symlink: those guards were
# not skipped, not listed and not run, which is the silent-zero failure this
# script exists to prevent. `! -type d` rather than `-type f` so a broken
# symlink still surfaces above instead of vanishing again.
done < <(find -L "$GUARD_DIR" -maxdepth 1 ! -type d 2>/dev/null | LC_ALL=C sort)
[ -n "$found" ] || say "note: $GUARD_DIR is configured but empty. See 'Guard tests are the point' at the top of this file for what belongs there."
}
# ---------------------------------------------------------------------------
# Node. The script names come from package.json and nowhere else.
#
# Read with node rather than grepped, because a grep for '"test"' matches keys
# in devDependencies, in a nested tool config, and in lint-staged — and
# `npm run` on a name that is not a script exits non-zero, so the guess would
# surface as a failing check with no failing code behind it.
# ---------------------------------------------------------------------------
detect_node() {
[ -f package.json ] || return 0
if ! have npm || ! have node; then
add_skip "npm" "package.json is here but node/npm is not on PATH — none of its checks could run"
return 0
fi
local scripts rc
scripts=$(node -e 'try{const s=require(process.cwd()+"/package.json").scripts||{};process.stdout.write(Object.keys(s).join("\n"))}catch(e){process.exit(1)}' 2>/dev/null); rc=$?
# Kept separate from "there are no scripts" below. Both end in zero npm steps
# and they need different answers: one is a project that has no checks yet,
# the other is a file this script could not read, which is also about to break
# every npm command anyone else runs today.
if [ "$rc" -ne 0 ]; then
add_skip "npm" "package.json could not be parsed by node — fix the JSON; no npm check could be read from it"
return 0
fi
if [ -z "$scripts" ]; then
add_skip "npm" "package.json declares no scripts — this Node project was not checked at all"
return 0
fi
local class list name matched=""
for class in typecheck lint test; do
case "$class" in
typecheck) list="$NODE_TYPECHECK_SCRIPTS" ;;
lint) list="$NODE_LINT_SCRIPTS" ;;
test) list="$NODE_TEST_SCRIPTS" ;;
esac
for name in $list; do
printf '%s\n' "$scripts" | grep -qx -- "$name" || continue
add_step "npm:$name" "$class" "npm run $(q "$name")"
matched="yes"
# One per class. Two matches usually means `test` and `test:ci` are the
# same suite twice, and paying for a suite twice is how people stop
# running verify.
break
done
done
if [ -z "$matched" ]; then
say "note: package.json has scripts, but none named like a check. Looked for:"
say " $NODE_TYPECHECK_SCRIPTS $NODE_LINT_SCRIPTS $NODE_TEST_SCRIPTS"
say " Add the real names to the candidate lists near the top of this file."
fi
}
detect_make() {
local mk="" f
for f in Makefile makefile GNUmakefile; do
if [ -f "$f" ]; then mk="$f"; break; fi
done
[ -n "$mk" ] || return 0
if ! have make; then
add_skip "make" "$mk is here but make is not on PATH"
return 0
fi
local class list target matched=""
for class in typecheck lint test; do
case "$class" in
typecheck) list="$MAKE_TYPECHECK_TARGETS" ;;
lint) list="$MAKE_LINT_TARGETS" ;;
test) list="$MAKE_TEST_TARGETS" ;;
esac
for target in $list; do
# Anchored at the start of the line so this matches a rule and not a
# .PHONY declaration or a variable that happens to contain the word.
grep -qE "^${target}[[:space:]]*:" "$mk" || continue
add_step "make:$target" "$class" "make $(q "$target")"
matched="yes"
break
done
done
# Said out loud, for the same reason the npm detector says it. A Makefile
# whose target is `tests` or `ci` contributes nothing here, and in a project
# that ALSO has a package.json the run still prints a full green table — one
# that silently excludes the Makefile's suite. Contributing zero steps is
# information; contributing zero steps quietly is the failure.
if [ -z "$matched" ]; then
say "note: $mk is here, but none of its targets are named like a check. Looked for:"
say " $MAKE_TYPECHECK_TARGETS $MAKE_LINT_TARGETS $MAKE_TEST_TARGETS"
say " Add the real names to the candidate lists near the top of this file."
fi
}
detect_cargo() {
[ -f Cargo.toml ] || return 0
if ! have cargo; then
add_skip "cargo" "Cargo.toml is here but cargo is not on PATH"
return 0
fi
if cargo fmt --version >/dev/null 2>&1; then
add_step "cargo:fmt" lint "cargo fmt --all -- --check"
else
add_skip "cargo:fmt" "rustfmt is not installed (rustup component add rustfmt)"
fi
if cargo clippy --version >/dev/null 2>&1; then
# `-D warnings` is not strictness for its own sake: without it clippy prints
# its findings and exits 0, so the step passes whatever it found — a check
# that cannot fail. Loosen it here if this project has warnings it has
# decided to keep, but loosen it visibly.
add_step "cargo:clippy" lint "cargo clippy --all-targets -- -D warnings"
else
add_skip "cargo:clippy" "clippy is not installed (rustup component add clippy)"
fi
add_step "cargo:test" test "cargo test"
}
detect_python() {
[ -f pyproject.toml ] || return 0
PY_MATCHED=""
# Configured in pyproject means the project uses it. A tool that merely
# happens to be installed on this machine is not a check this project has,
# and running it would invent a standard the repository never agreed to.
py_tool() { # section-regex binary class command
grep -qE "$1" pyproject.toml || return 0
PY_MATCHED="yes"
if have "$2"; then
add_step "py:$2" "$3" "$4"
else
add_skip "py:$2" "configured in pyproject.toml but '$2' is not on PATH — activate the virtualenv, or install it"
fi
}
py_tool '^\[tool\.ruff' ruff lint "ruff check ."
py_tool '^\[tool\.black' black lint "black --check ."
py_tool '^\[tool\.mypy' mypy typecheck "mypy ."
# pytest is matched anywhere in the file, not only as a [tool.pytest] section:
# most projects configure nothing and simply depend on it.
py_tool 'pytest' pytest test "pytest"
if [ -z "$PY_MATCHED" ]; then
say "note: pyproject.toml is here, but it configures none of the tools this"
say " looks for (ruff, black, mypy, pytest). No Python check was run."
fi
}
detect_go() {
[ -f go.mod ] || return 0
if ! have go; then
add_skip "go" "go.mod is here but go is not on PATH"
return 0
fi
add_step "go:vet" lint "go vet ./..."
add_step "go:test" test "go test ./..."
}
detect_guards
detect_node
detect_make
detect_cargo
detect_python
detect_go
# ---------------------------------------------------------------------------
# Nothing detected is a failure, and the loudest one here.
#
# A verify that checks nothing and prints success is worse than no verify at
# all: it converts "I have not checked" into "it passed" for everyone
# downstream, including the hook that trusts the exit code.
# ---------------------------------------------------------------------------
if [ "${#STEP_ID[@]}" -eq 0 ]; then
say "NO CHECKS DETECTED. Nothing was verified and nothing passed."
say ""
say "Looked at the repository root for: package.json scripts, a Makefile"
say "target, Cargo.toml, pyproject.toml, go.mod, and VERIFY_GUARD_DIR"
say "(currently ${GUARD_DIR:-unset})."
say ""
say "Give it something to run: add a check script to package.json, add a"
say "target to the Makefile, or set VERIFY_GUARD_DIR to a directory of"
say "executable guard scripts. Then run this again."
exit 2
fi
if [ -n "$ONLY" ]; then
n=${#STEP_ID[@]}
kept=""
for ((i = 0; i < n; i++)); do
case "${STEP_ID[$i]}" in
*"$ONLY"*) kept="yes" ;;
# An existing reason wins. A step already skipped because its tool is not
# installed is not skipped BY --only, and overwriting the reason would
# report that it was filtered out when in fact it could not have run —
# turning a real gap in the checks into a deliberate-looking choice.
*) [ -n "${STEP_SKIP[$i]}" ] || STEP_SKIP[$i]="--only $ONLY" ;;
esac
done
[ -n "$kept" ] || die "--only '$ONLY' matched none of the detected steps: $(printf '%s ' "${STEP_ID[@]}")"
fi
# --quick, resolved against the plan rather than assumed. A SLOW_STEP naming a
# step this project does not have is a stale config that would silently skip
# nothing while reporting that it skipped, so it is refused.
if [ -n "$QUICK" ]; then
n=${#STEP_ID[@]}
hit=""
for ((i = 0; i < n; i++)); do
if [ -n "$SLOW_STEP" ]; then
[ "${STEP_ID[$i]}" = "$SLOW_STEP" ] || continue
else
[ "${STEP_CLASS[$i]}" = "test" ] || continue
fi
[ -n "${STEP_SKIP[$i]}" ] || STEP_SKIP[$i]="--quick"
hit="yes"
done
if [ -z "$hit" ]; then
if [ -n "$SLOW_STEP" ]; then
die "VERIFY_SLOW_STEP is '$SLOW_STEP' but no detected step has that id: $(printf '%s ' "${STEP_ID[@]}")"
fi
say "note: --quick had no effect — no detected step is a test step."
elif [ -z "$SLOW_STEP" ]; then
say "--quick: skipping the test steps (set VERIFY_SLOW_STEP to an id to pick a different one)."
fi
fi
n=${#STEP_ID[@]}
if [ -n "$LIST" ]; then
say "$n step(s) detected. Nothing was run:"
for ((i = 0; i < n; i++)); do
if [ -n "${STEP_SKIP[$i]}" ]; then
say " $(printf '%-18s' "${STEP_ID[$i]}") SKIPPED — ${STEP_SKIP[$i]}"
else
say " $(printf '%-18s' "${STEP_ID[$i]}") ${STEP_CMD[$i]}"
fi
done
exit 0
fi
# ---------------------------------------------------------------------------
# Run everything. No early exit on failure: the point of one entry point is one
# answer, and an answer that stops at the first broken thing makes you pay the
# whole runtime again for the second.
#
# Output is streamed rather than captured, so a step that hangs is visible
# while it hangs instead of after it is killed.
# ---------------------------------------------------------------------------
started=$SECONDS
failed=0
passed=0
skipped=0
for ((i = 0; i < n; i++)); do
if [ -n "${STEP_SKIP[$i]}" ]; then
STEP_RESULT[$i]="SKIPPED"
STEP_SECS[$i]="-"
skipped=$((skipped + 1))
continue
fi
say "── ${STEP_ID[$i]} — ${STEP_CMD[$i]}"
t0=$SECONDS
eval "${STEP_CMD[$i]}"
code=$?
STEP_SECS[$i]=$((SECONDS - t0))
if [ "$code" -eq 0 ]; then
STEP_RESULT[$i]="PASS"
passed=$((passed + 1))
else
STEP_RESULT[$i]="FAIL"
failed=$((failed + 1))
say "${STEP_ID[$i]} failed (exit $code) — continuing so this run reports everything."
fi
done
# ---------------------------------------------------------------------------
# The table. Three states, three words, three colours, and the word alone is
# enough when the colour is gone — this gets piped into files and hook output
# where nothing is a terminal.
# ---------------------------------------------------------------------------
say ""
say "───────────────────────────────────────────────────────────"
for ((i = 0; i < n; i++)); do
case "${STEP_RESULT[$i]}" in
PASS) colour="$C_PASS" ;;
FAIL) colour="$C_FAIL" ;;
*) colour="$C_SKIP" ;;
esac
line="$(printf ' %s%-8s%s %-26s %5s' \
"$colour" "${STEP_RESULT[$i]}" "$C_OFF" "${STEP_ID[$i]}" "${STEP_SECS[$i]}")"
if [ "${STEP_RESULT[$i]}" = "SKIPPED" ]; then
line="${line} ${STEP_SKIP[$i]}"
else
line="${line}s"
fi
say "$line"
done
say "───────────────────────────────────────────────────────────"
elapsed=$((SECONDS - started))
# Counts are stated only for things that were measured. "0 failed" alongside
# four skips is true and misleading, so the skip count is never omitted and the
# exit code below refuses to call an all-skipped run a pass.
summary="${passed} passed"
[ "$failed" -eq 0 ] || summary="${summary}, ${failed} FAILED"
[ "$skipped" -eq 0 ] || summary="${summary}, ${skipped} SKIPPED (not run — not verified)"
say "${summary} [${elapsed}s]"
if [ "$failed" -gt 0 ]; then
say ""
say "Fix the FAILED step(s) above. To re-run one on its own:"
for ((i = 0; i < n; i++)); do
# Both halves shell-quoted, because both routinely contain spaces: this
# family of projects lives under directories like "Privacy LLC", and a step
# id is a guard's filename. Printed unquoted, the command this suggests
# parses as extra arguments and dies on the copy-paste.
[ "${STEP_RESULT[$i]}" = "FAIL" ] \
&& say " bash $(q "$0") --only $(q "${STEP_ID[$i]}")"
done
exit 1
fi
# Nothing failed and nothing ran. That is not a pass, and the exit code is the
# only part of this output a hook or a CI job will read.
if [ "$passed" -eq 0 ]; then
say ""
say "NOTHING WAS VERIFIED — every detected step skipped. Exiting non-zero:"
say "a clean table over an empty run is the one outcome this script exists"
say "to prevent."
exit 2
fi
exit 0

80
docs/data/README.md Normal file
View File

@ -0,0 +1,80 @@
# Data — <Project>
```
Status: Current
Owner: <who maintains this>
Last reviewed: <YYYY-MM-DD>
Governs: docs/data/** — the assets privacyllc.dev renders for this project
Review trigger: A rebrand, or any change to the icon, logo or banner
```
## What goes here
Three files, in `img/`, at exactly these names:
```text
docs/data/img/icon.webp the square mark, used wherever the project is listed
docs/data/img/logo.webp the full lockup, used on the project page
docs/data/img/banner.webp the wide image, used across the project header
```
All **webp**. All **required**. Only `img/` is checked — an asset left at
`docs/data/logo.webp` is not found.
**Dimensions, weights and how to generate them are in
[`img/README.md`](img/README.md)**, beside the files they describe. This
document owns the rule; that one owns the spec. Stating both in both places is
how two copies of one convention start disagreeing.
**No placeholders ship with this template, deliberately.** It carried a 0-byte
`logo.webp` once, and an empty file is the worst of the three states: a check
that asks "does the path exist" calls it present, and anything that reads the
bytes rejects it — a consumer verifying the webp signature answers 415, which
reads as a corrupt asset rather than a missing one. Absent is honest and the
conformance check reports it as absent, which is what gets it filled in.
Extra sizes and variants are welcome beside them — `icon-512.webp`,
`logo-dark.webp` — and are not treated as clutter. Only the three exact names
are checked for.
## If an asset is missing, open an issue — do not invent one
An agent cannot draw a logo, and this is the one gap in the whole convention
that cannot be closed by writing a file.
So when an asset is absent, **file an issue** rather than producing something:
title it for the asset, label it `P2`, and end the body with its `Verify:` line
— `Verify: docs/data/img/logo.webp exists and the project card renders it.`
**Do not generate a placeholder.** A placeholder that looks deliberate outlives
the issue that would have replaced it: nobody files a ticket against an image
that appears to be finished. An obviously absent asset keeps asking.
## Why this folder is different from every other one here
The Command Center *consumes* these. Every other document in this tree is written
for a person to read; these are fetched and rendered on privacyllc.dev's project
page.
That has one consequence worth stating plainly: **this folder cannot be declared
exempt.** A repository may tell the conformance check that a required document is
deliberately absent — kept out of git on purpose, say — and the check will
believe it. It will not accept that declaration for `docs/data/`, because the
result would be a project card with nothing to show and nothing explaining why,
which is the exact failure the check exists to prevent.
## Why webp and not PNG
One format, checked by its magic bytes rather than its file extension, so the
site can serve it inline with confidence. A file whose first bytes are RIFF/WEBP
cannot be an HTML document or an SVG carrying script, which is what makes it safe
to render directly rather than forcing a download.
Renaming a PNG to `.webp` will not work, and is meant not to.
## This is not an asset library
Screenshots, mockups, reference art and exported source files do not belong here.
They belong wherever the project already keeps them. This folder holds the three
marks that identify the project elsewhere, and stays small enough that its
contents are obvious at a glance.

57
docs/data/img/README.md Normal file
View File

@ -0,0 +1,57 @@
# Project images
```
Status: Current
Owner: <who maintains this>
Last reviewed: <YYYY-MM-DD>
Governs: docs/data/img/** — the three files and their sizes
Review trigger: A rebrand; any change to a required name, dimension or ceiling;
any change to what the consumer accepts.
```
Three files, all **webp**, all **required**, at exactly these names:
| File | Dimensions | Aspect | Typical weight |
| --- | --- | --- | --- |
| `icon.webp` | **512 × 512** | 1:1 | 8–60 KB |
| `logo.webp` | **1024** on the long edge | whatever the lockup is | 20–190 KB |
| `banner.webp` | **2176 × 725** | 3:1 | 30–130 KB |
**Not `icon.ico`.** The consumer checks the file's magic bytes, not its name: a
`.ico`, or a PNG renamed to `.webp`, is refused with a 415 and the project falls
back to an initials tile. That signature check is what makes it safe to render
these inline, so it is not going to be relaxed.
**512 KB is a hard ceiling per file**, enforced in code — the size is read from
the listing before the bytes are fetched, so an oversized asset is never
downloaded and simply never appears. Nothing enforces the dimensions, which is
why they are written down.
## Making them
```bash
magick logo-source.png -resize 512x512 -quality 82 icon.webp
identify -format '%f %wx%h %b\n' *.webp # check before committing
```
Quality 80–85 suits a flat mark. If a file lands over ~200 KB it is usually a
photographic banner that wants a lower quality rather than fewer pixels.
## Why these numbers
The icon renders small — a 44 px tile in a list, 58 px on a project header — so
512 covers the densest display several times over; the reference project
deliberately halved it from 1024. The banner spans a card about 760 px wide, so
~2176 covers it at 2×, and its 3:1 shape matters more than its width because the
header crops to fill. The logo has no fixed frame, so only its long edge is
specified.
Extra sizes and variants are welcome beside these — `icon-512.webp`,
`logo-dark.webp`. Only the three exact names are checked for.
No placeholders ship with this template. An empty file is the worst of the three
states: a check that asks whether the path exists calls it present, and anything
reading the bytes rejects it. Absent is honest, and the conformance check reports
it as absent — which is what gets it filled in.
Why the requirement exists, and what reads it: [`../README.md`](../README.md).

34
docs/design/README.md Normal file
View File

@ -0,0 +1,34 @@
# Design
```
Status: Current
Owner: <who maintains this>
Last reviewed: <YYYY-MM-DD>
Governs: docs/design/**
Review trigger: Any new user-facing surface, or a change to the product's tone
```
## What belongs here
What it should feel like, and the decisions behind that:
- **Product decisions** — what the user can do, in what order, and what happens
when they get it wrong. The error states are design, not an afterthought.
- **UI plans** — screens, states, and what each one is for. Include the empty
state and the loading state; they are the two most people see first and the
two most often left undesigned.
- **Copy** — the actual words. Interface text is a design surface, and writing
it late means writing it badly.
- **Tone** — how this product talks. One paragraph is enough, and it settles a
hundred small arguments.
## What does not belong here
- How it is built — that is `docs/architecture/`
- Scope and audience — that is `docs/planning/PROJECT_PLAN.md`
## Include the rejected version
For any decision that was genuinely close, record what was not chosen and why.
Without it, the same option gets proposed every few months and re-argued from
nothing.

45
docs/history/BATCH_LEDGER.md Normal file
View File

@ -0,0 +1,45 @@
# Batch ledger — <Project>
```
Status: Archived
Owner: <who maintains this>
Last reviewed: <YYYY-MM-DD>
Governs: what the batches were, before the tracker held them
Review trigger: Nothing. Superseded by the tracker; kept for the record.
```
## Why this file is archived rather than deleted
Planning used to live in markdown as a numbered batch list. It now lives in the
tracker: **milestones are batches, issues are deliverables**, and severity is
`P0` / `P1` / `P2` / `release-blocker`.
This file is what that list *was*. It is kept because the reasoning in it is
still worth reading, and deleted files are not readable.
**It must not be updated.** A batch list beside the tracker is a second answer to
"what is open", and two records of the same thing will disagree without saying
which is right. That is precisely why the work moved. If you are tempted to add a
batch here, add a milestone instead.
Its `Status: Archived` is therefore not a nicety — it is the machine-readable
form of "do not treat this as current".
## "Ledger: missing" on the project screen is the correct answer
The Command Center's docs report still looks for a batch ledger — at
docs/planning/FUTURE.md, named here without backticks deliberately, because
`doc-claims.sh` treats a backticked path as a claim that the file exists and
this one must not — and reports it as **missing** for every repository that has
moved planning into the tracker. That is the expected state and not a gap to
close: the report's own contract is that a tracker-first project shows a missing
ledger and no batch percentages.
Creating that file to turn the line green would rebuild, in a new name, the
exact second record this one was archived for.
## The batches, as they stood
| Batch | What it covered | Outcome |
| --- | --- | --- |
| <> | <> | <> |

51
docs/history/DEVELOPMENT_LOG.md Normal file
View File

@ -0,0 +1,51 @@
# Development log — <Project>
```
Status: Current
Owner: <who maintains this>
Last reviewed: <YYYY-MM-DD>
Governs: the dated record of what happened
Review trigger: Nothing. This file is appended to, never revised.
```
## How to use this
Newest first. **One entry per work session**, written before you stop — that is
step 6 of `docs/WORK_CYCLE.md`, and the two lines it insists on are `Next
action` and `Blockers`.
Those two are not decoration. The next session starts by reading the top of this
file, and a session that ended without saying what came next hands the one after
it a re-derivation instead of a starting point — which is where drift enters.
Neither line competes with anything: the live next action is the field on the
project at privacyllc.dev and the live blockers are issues in the tracker, while
these say what both were **at this date**. A record of then never disagrees with
a record of now.
**Append-only by convention.** Correcting an old entry rewrites the record of
what was known at the time, which is the one thing this file is for. If an entry
turns out to be wrong, add a later entry saying so; do not edit the first.
Note the Review trigger above says "nothing", deliberately. A dated log cannot
rot the way a description of current state can — the entries were true when
written and stay true. It is exempt from review for the same reason a receipt is.
## Entries
### <YYYY-MM-DD> — <one line: what changed>
<What was done, what it proved, and what is still unresolved. Link the commit or
the issue rather than restating it — the value here is the reasoning that does
not fit in a commit message.>
- **Closed:** <#N, #N — or "none", which is also an answer>
- **Next action:** <the concrete thing the next session starts on. The same
sentence written to the project's nextAction field; specific enough to begin
without asking a question.>
- **Blockers:** <what is stuck and the issue it is filed as, or "none". "None"
written deliberately is worth more than a missing line, which reads as
"nobody said" rather than "nothing was".>
### <YYYY-MM-DD> — <one line>
<…>

45
docs/history/HISTORY.md Normal file
View File

@ -0,0 +1,45 @@
# History — <Project>
```
Status: Current
Owner: <who maintains this>
Last reviewed: <YYYY-MM-DD>
Governs: the narrative of how this project got to where it is
Review trigger: A decision reversed, a direction abandoned, or a rewrite
```
## What this is for
<The story, in prose: what was tried, what was abandoned, and why. A reader
arriving in six months wants to know which walls have already been walked into,
and that is not something a commit log tells them.>
**This is a record of *then*, never a description of *now*.** That distinction is
what makes it safe to leave alone as the project changes — a history document
that gets edited to stay current is not a history, it is a second and competing
description of the present.
If you find yourself updating a sentence here because the code changed, the
sentence belongs somewhere else.
## Decisions and their reasons
<One entry per decision that would otherwise look arbitrary later. The reason
matters more than the decision: "we chose X" ages badly, "we chose X because Y
was true at the time" survives Y stopping being true.>
| When | What was decided | Why, at the time |
| --- | --- | --- |
| <> | <> | <> |
## What was tried and dropped
<The most useful section, and the one most often missing. An approach that was
abandoned for a good reason will be proposed again by somebody who does not know
it was tried — including you, in a year.>
## What this file is not
- **Not the plan.** That is `docs/planning/PROJECT_PLAN.md`.
- **Not the work list.** Open work lives in the tracker as milestones and issues.
- **Not a changelog.** Dated entries go in `docs/history/DEVELOPMENT_LOG.md`.

53
docs/planning/PROJECT_PLAN.md Normal file
View File

@ -0,0 +1,53 @@
# <Project> — Project Plan
```
Status: Current
Owner: <who maintains this>
Last reviewed: <YYYY-MM-DD>
Governs: scope, audience, and what this project deliberately is not
Review trigger: Any change of scope, audience, or platform
```
> The vision. The **milestones in this repository's issue tracker** hold the
> sequence of work; this holds what the work is *for*. When the two disagree,
> this one is the argument and those are the plan — resolve it rather than
> letting both stand.
>
> Deliberately no batch list here. A schedule in a document goes stale the day
> after it is written and nothing notices; a schedule in the tracker is updated
> by the act of doing the work.
## What this is
<One paragraph. If it takes three, the scope is not settled yet.>
## Who it is for
<The actual user. "Everyone" is not an answer and produces a product with no
opinions.>
## What it is deliberately not
<The most useful section in this file. Every "not" here is a decision that stops
being re-litigated, and a feature request that can be answered in one line.>
- <not this>
- <not that>
## Stack and platform
| Concern | Choice | Why |
| --- | --- | --- |
| <language / runtime> | <choice> | <the reason, not the marketing> |
| <storage> | <choice> | <> |
| <hosting> | <choice> | <> |
## Success looks like
<Something observable. "Users are happy" is not observable; "a first-time user
completes the core flow without help" is.>
## Known risks
<What could make this fail, and what would be done about it. Written now, while
it is still cheap to be honest.>

32
docs/qa/ClaudeQACoverage.md Normal file
View File

@ -0,0 +1,32 @@
# Claude QA Coverage — <Project>
```
Status: Current
Owner: <who maintains this>
Last reviewed: <YYYY-MM-DD>
Governs: what each QA pass actually reached
Review trigger: Any QA round run
```
> Pass by pass, what was reached and what was not. The point of this file is the
> **Blocked** rows: a pass left out of a report reads exactly like a pass that
> succeeded, and that is how untested code ships believing it was tested.
## Round <N> — <YYYY-MM-DD> at `<sha>`
| Pass | Result | Notes |
| --- | --- | --- |
| A — First run | Pass | <what was observed> |
| B — Core flow | Pass | <> |
| C — Failure paths | Partial | <which failures were simulated, which were not> |
| D — Persistence | Pass | <> |
| E — End of loop | **Blocked** | <what blocks it, and the issue number> |
| F — Accessibility | Not run | <why, and when it will be> |
| G — Performance | Pass | <the numbers, not "felt fine"> |
## Standing gaps
<Things no round has ever covered, carried forward until they are. This list
existing is not a failure; it not existing while the gaps do is.>
- <gap, and what it would take to close>

74
docs/qa/ClaudeQAPlan.md Normal file
View File

@ -0,0 +1,74 @@
# Claude QA Plan — <Project>
```
Status: Current
Owner: <who maintains this>
Last reviewed: <YYYY-MM-DD>
Governs: what a QA round consists of
Review trigger: Any new user-facing surface, or a defect class that got through
```
> The playbook. What a round *is*, so two rounds are comparable and a gap is
> visible rather than assumed covered.
## Before a round
- Build from a clean checkout at a known SHA, and record that SHA.
- Run from a detached worktree if other work is in flight, so uncommitted
changes cannot contaminate what is under test.
- Note the environment: device, OS version, browser, screen size — whatever the
product's behaviour actually depends on.
## The passes
Each pass gets a letter, so `ClaudeQACoverage.md` can report per pass and a
skipped one is visible.
| Pass | What it covers |
| --- | --- |
| A | First run: install or load, cold start, permissions, empty states |
| B | The core flow, end to end, as a real user would do it |
| C | The core flow with things going wrong: no network, denied permission, invalid input |
| D | Persistence: quit and return, background and resume, restart |
| E | The end of the loop — the state that is hardest to reach on purpose |
| F | Accessibility: keyboard only, screen reader labels, contrast, text scaling |
| G | Performance under the load this product will actually see |
<Add, remove and rename to fit. A pass that never applies is noise; a pass that
is always skipped is a lie.>
## What counts as a finding
A finding needs: what was done, what happened, what should have happened, and
the build SHA. Without the SHA it cannot be re-tested, and a finding that cannot
be re-tested cannot be closed.
## Severity
Findings are filed as issues, labelled:
- **P0** — ships broken, or loses data
- **P1** — materially wrong, but shippable
- **P2** — cosmetic or low impact
- **release-blocker** — a release built today would be wrong rather than merely
incomplete
Exactly these label names: the Command Center queries them by name, and a
repository that spells them differently has its defects reported as *not
adopted* rather than counted wrongly.
Severity is what it costs, not how annoying it is to fix.
## After a round
File each finding as a labelled issue. Update `ClaudeReport.md`'s run-state
block and its overall sentence, and `ClaudeQACoverage.md` with what each pass
actually reached. A pass that could
not be run is recorded as blocked, with what blocks it — never quietly left out,
which reads identically to "passed".
Then **push, and reconcile**. The verdict on the project screen at
privacyllc.dev is read out of `ClaudeReport.md` in the pushed repository, so a
round whose report is committed but not pushed — or pushed but not reconciled —
leaves a stakeholder reading the previous round's judgment with no indication
that a newer one exists. The rest of the cycle is in `docs/WORK_CYCLE.md`.

53
docs/qa/ClaudeReport.md Normal file
View File

@ -0,0 +1,53 @@
# Claude QA Report — <Project>
```
Status: Current
Owner: <who maintains this>
Last reviewed: <YYYY-MM-DD>
Governs: the QA verdict — build SHAs, round summaries, the overall judgment
Review trigger: Any QA round run
```
> The QA verdict. Companion to `ClaudeQACoverage.md` (what each pass reached)
> and `ClaudeQAPlan.md` (the playbook).
>
> **Defects are issues, not entries here.** A defect found in a round is filed
> in the tracker with a severity label, where it can be assigned, closed by a
> commit, and counted. This file keeps the part a tracker is bad at: a human's
> judgment about whether the thing is fit to ship.
## Current run-state
- **Last QA round:** <Round N — YYYY-MM-DD>
- **Last verified build SHA:** `<sha>`
- **Last tested device / environment:** <what it ran on>
- **Overall status:** <One or two sentences, in your own words. This line is
quoted verbatim on the project screen in the Command Center, so write it as
the sentence you would want a stakeholder to read — not a status word.>
## Open defects
**Do not list them here, and do not read a defect count out of this file.** The
Command Center's docs report parses this document for open `P0` / `P1` / `P2`
counts, and under this convention they are always zero — the defects are in the
tracker, which is the whole point. A repository with seventeen open P1s reports
`openP1: 0` in that block and the real figure in its tracker counts. Anything
summarising defects reads the tracker's label counts; the zeros here mean *this
file does not hold them*, never *there are none*.
Filed as issues in this repository's tracker, labelled by what they cost:
- **P0** — ships broken, or loses data
- **P1** — materially wrong, but shippable
- **P2** — cosmetic or low impact
- **release-blocker** — a release built today would be *wrong*, not merely
incomplete
Severity is what it costs, not how annoying it is to fix. Every defect needs
the build SHA it was found at — a finding that cannot be re-tested cannot be
closed — so put it in the issue body.
## Round notes
<What each round actually did, what it could not reach, and why. A round that
skipped half the plan says so here rather than in nobody's memory.>

64
docs/security/SECURITY.md Normal file
View File

@ -0,0 +1,64 @@
# Security — <Project>
```
Status: Current
Owner: <who maintains this>
Last reviewed: <YYYY-MM-DD>
Governs: authentication, secret handling, data at rest and in transit
Review trigger: Any new secret, any new external service, any change to auth or storage
```
## What this protects, and from whom
<Name the assets and the adversary. "Security" with no threat model produces
effort spent in the wrong places — a hobby project's real adversaries are a
leaked repository and a stolen laptop, not a nation state, and the controls
differ.>
| Asset | Where it lives | What would it cost to lose |
| --- | --- | --- |
| <> | <> | <> |
## Secrets
- **Nothing secret is committed.** Not in source, not in config, not in a test
fixture, not in a screenshot.
- Secrets arrive from the environment or a secret store, and are read at the
boundary rather than passed around.
- A key stored beside the data it protects protects nothing. If data is
encrypted at rest, say here where the key lives and how it is rotated.
**Check before every release:** no private keys, keystores, `.env` files,
service-account JSON, or real credentials in the tracked tree — including in the
history, which a `git log -p` search covers and a directory listing does not.
## Authentication and authorisation
<Who can do what, and where that is enforced. If the answer is "in the UI", it
is not enforced — say so and fix it.>
## Data in transit
<TLS everywhere it applies. Certificate pinning if it applies. What is sent to
third parties, and what is deliberately not.>
## Third parties
| Service | What it receives | Why that is acceptable |
| --- | --- | --- |
| <> | <> | <> |
Every row is a decision to send someone else's data somewhere. An empty table is
a good table.
## Deliberately out of scope
<What is not defended against, and why that is a reasonable call. Writing it
down converts an unknown gap into a known one, which is the whole difference.>
## Logging
Logs are read by people and shipped to places. Nothing sensitive goes into one —
not a token, not a credential in a URL, not the contents of a user's record. An
error's *name* is almost always enough; its message, from a failed HTTP request,
is often the request URL.

43
docs/security/SECURITY_CHECKLIST.md Normal file
View File

@ -0,0 +1,43 @@
# Security checklist — <Project>
```
Status: Current
Owner: <who maintains this>
Last reviewed: <YYYY-MM-DD>
Governs: the checks run before a release, and what each one proves
Review trigger: A new class of input, a new external service, or a finding that got past this list
```
## Why this is separate from SECURITY.md
`SECURITY.md` is the threat model: what is being protected and from whom. It is
read carefully once and revisited rarely.
This is the list somebody actually works through. Keeping them apart means the
model can stay stable while the checks change, and it means a release checklist
is short enough to finish rather than a document to skim.
## Before a release
<Each entry states what it proves, not just what to do. A check whose purpose is
unstated gets skipped the first time it is inconvenient.>
- [ ] <Check> — proves <what>
- [ ] <Check> — proves <what>
## Standing checks
- [ ] No secret in the repository, in a log line, or in an error message
- [ ] Every externally reachable endpoint is either authenticated or deliberately public, and the deliberate ones are listed
- [ ] Every input that reaches a query or a filesystem path is validated at the boundary
- [ ] Dependencies audited, and any accepted advisory recorded with a reason
## What got past this list
<Add an entry whenever a real finding was not caught here, and then add the check
that would have caught it. A checklist that never grows is one nobody is honest
with.>
| When | What was missed | The check now added |
| --- | --- | --- |
| <> | <> | <> |

188
project-readme-template.md Normal file
View File

@ -0,0 +1,188 @@
# <Project Name>
```text
Status: <Current | Draft | Superseded | Archived>
Owner: <person or role>
Last reviewed: <YYYY-MM-DD>
Governs: README.md as the project-facing overview for <project or repo>
Review trigger: <event that should make an agent update this README>
```
<One clear sentence that says what the project is and who it is for.>
[Screenshots](#screenshots) | [What makes it different](#what-makes-it-different) |
[Product surface](#product-surface) | [Run locally](#run-locally) |
[QA and release](#qa-and-release) | [Project docs](#project-docs)
_{ <Platform or public surface> | <Admin or app surface> | <Backend> |
<Most important feature> | <Second important feature> | <Privacy/security posture> }_
![<Primary screenshot or feature graphic alt text>](<path/to/primary-image.webp>)
<Short product overview. Write for somebody evaluating the project, not for the
person who already knows the code. Explain the problem it solves, the shape of
the product, and the reason it exists.>
The core loop is:
```text
<first user action> -> <second user action> -> <valuable outcome>
```
## Screenshots
<Use real current screenshots. Prefer the actual product surface over diagrams
or mood images.>
<Screenshot A> | <Screenshot B> | <Screenshot C>
--- | --- | ---
![<alt text>](<path>) | ![<alt text>](<path>) | ![<alt text>](<path>)
## What Makes It Different
<Do not make this a feature dump. Name the product decisions that make the
project worth caring about.>
| Principle | What it means |
| --- | --- |
| <Differentiator> | <Specific, evidence-backed explanation.> |
| <Differentiator> | <Specific, evidence-backed explanation.> |
| <Differentiator> | <Specific, evidence-backed explanation.> |
## Product Surface
| Area | What it does | Why it matters |
| --- | --- | --- |
| <Feature area> | <User-visible behavior.> | <Outcome or reason.> |
| <Feature area> | <User-visible behavior.> | <Outcome or reason.> |
| <Feature area> | <User-visible behavior.> | <Outcome or reason.> |
## Spotlight
### <Primary Feature>
<Explain the most important product surface in plain language. Name what is
built and what the user can do with it.>
![<Primary feature screenshot alt text>](<path>)
### <Second Feature>
<Explain the next most important product surface.>
![<Second feature screenshot alt text>](<path>)
### <Third Feature>
<Explain the third most important product surface.>
![<Third feature screenshot alt text>](<path>)
## Platform Status
| Surface | Status | Notes |
| --- | --- | --- |
| <Surface> | <Built | Active | Work in progress | Planned | Deprecated> | <Evidence or limitation.> |
| <Surface> | <Built | Active | Work in progress | Planned | Deprecated> | <Evidence or limitation.> |
## Architecture
```text
<client or public surface>
|
v
<app/runtime>
|
+--> <database or storage>
+--> <external service>
+--> <worker or queue>
```
Core stack:
- <Framework/runtime>
- <Database/storage>
- <Authentication>
- <Deployment target>
- <Important integrations>
## Run Locally
Prerequisites:
- <Tool or service>
- <Tool or service>
Start the project:
```bash
<install command>
<development command>
```
Open `<local URL>`.
## Configuration
| Variable | Purpose |
| --- | --- |
| `<ENV_VAR>` | <What it controls.> |
| `<ENV_VAR>` | <What it controls.> |
Keep secrets out of screenshots, README prose, and tracked sample files.
## QA And Release
Run checks:
```bash
<typecheck command>
<test command>
```
Release or deploy:
```bash
<release command>
```
<State whether release and deployment are the same action. If they are
separate, say which tool owns each step.>
## Repository Map
```text
<path>/ <purpose>
<path>/ <purpose>
<path>/ <purpose>
docs/ product, architecture, QA, security, and history docs
```
## Project Docs
`README.md` is the landing page for someone evaluating the project. Detailed
procedures belong in docs, and open work belongs in the tracker.
| Doc | Purpose |
| --- | --- |
| [docs/DOC_TRUST_MAP.md](docs/DOC_TRUST_MAP.md) | Which document owns which answer, and which source wins when records disagree. |
| [docs/WORK_CYCLE.md](docs/WORK_CYCLE.md) | What happens at the end of a piece of work: closing issues and milestones, recording next action and blockers, and what privacyllc.dev will and will not update on its own. |
| [docs/architecture/Engineering_Reference_Manual.md](docs/architecture/Engineering_Reference_Manual.md) | Architecture, data model, and engineering rules. |
| [docs/security/SECURITY.md](docs/security/SECURITY.md) | Threat model and security posture. |
## Agent Notes
- Product truth comes from the code and the tracker before prose.
- Do not keep a work list in this README.
- Finish with `docs/WORK_CYCLE.md`, every time: close the issues you finished
with the evidence that proves it, close the milestone if the batch landed,
update the documents the change triggered **in the same commit**, then record
the next action and any blockers before you stop.
- Nothing on privacyllc.dev writes itself except the tracker counts and the
pushed docs. The summary and next action change only when they are sent; the
milestone and blocker tables there change only when a human edits them.
- Do not claim a feature is built unless you can cite the route, file,
migration, release, or screenshot that proves it.
- Keep screenshots current when the UI changes meaningfully.
- Keep the first screen product-facing. Put deep operations lower down or in
docs.