From 6965915dbdc70ee55d4e65803b43d7b15285a3ac Mon Sep 17 00:00:00 2001 From: null Date: Mon, 17 Aug 2026 22:44:26 -0500 Subject: [PATCH] chore(repo): put the template under version control The basis for every project here was itself unversioned: no .git, no remote, no history. Changes to it had no diff and no revert, and two of its own guards could not run at all -- doc-claims.sh and doc-triggers.py both read git history, so the script written to catch documentation drift could not be run against the documents that define drift. This is the tree as it stands, including work that until now existed only as loose files on disk: WORK_CYCLE.md, TOOLS.md, the Portainer image-line fix in deploy.py, the status vocabulary corrected to the four words the conformance checker actually enforces, the Exempt: mechanism documented, and the Forgejo instance named in README.md. secrets.sh --tracked reports one candidate, migrate.sh:480. It is the comment documenting the three Postgres credential shapes that script redacts, with literal placeholders, and it is left alone deliberately: GUARDS.md section 2 is that a source-grep guard must tell code from the comment about code, and deleting an explanation to quiet a scanner is the failure it names. Co-Authored-By: Claude Opus 5 (1M context) --- .gitignore | 15 + README.md | 289 +++++ Template.code-workspace | 8 + docs/DOC_TRUST_MAP.md | 140 ++ docs/TOOLS.md | 75 ++ docs/WORK_CYCLE.md | 262 ++++ docs/architecture/GUARDS.md | 142 ++ docs/architecture/README.md | 79 ++ docs/architecture/githooks/README.md | 63 + docs/architecture/githooks/commit-msg | 110 ++ docs/architecture/githooks/post-commit | 91 ++ docs/architecture/githooks/pre-commit | 142 ++ docs/architecture/scripts/audit-gate.mjs | 187 +++ docs/architecture/scripts/backup.sh | 942 ++++++++++++++ docs/architecture/scripts/check-env.sh | 868 +++++++++++++ docs/architecture/scripts/commit-mine.sh | 120 ++ docs/architecture/scripts/dead-code.py | 219 ++++ docs/architecture/scripts/deploy.py | 581 +++++++++ docs/architecture/scripts/dev.sh | 1282 +++++++++++++++++++ docs/architecture/scripts/doc-claims.sh | 274 ++++ docs/architecture/scripts/doc-triggers.py | 211 +++ docs/architecture/scripts/duplication.py | 189 +++ docs/architecture/scripts/forgejo-issue.py | 586 +++++++++ docs/architecture/scripts/healthcheck.sh | 119 ++ docs/architecture/scripts/migrate.sh | 1066 +++++++++++++++ docs/architecture/scripts/prove-guard.sh | 151 +++ docs/architecture/scripts/release-notes.mjs | 605 +++++++++ docs/architecture/scripts/release.sh | 713 +++++++++++ docs/architecture/scripts/scaffold.sh | 591 +++++++++ docs/architecture/scripts/secrets.sh | 166 +++ docs/architecture/scripts/status.sh | 678 ++++++++++ docs/architecture/scripts/verify.sh | 694 ++++++++++ docs/data/README.md | 80 ++ docs/data/img/README.md | 57 + docs/design/README.md | 34 + docs/history/BATCH_LEDGER.md | 45 + docs/history/DEVELOPMENT_LOG.md | 51 + docs/history/HISTORY.md | 45 + docs/planning/PROJECT_PLAN.md | 53 + docs/qa/ClaudeQACoverage.md | 32 + docs/qa/ClaudeQAPlan.md | 74 ++ docs/qa/ClaudeReport.md | 53 + docs/security/SECURITY.md | 64 + docs/security/SECURITY_CHECKLIST.md | 43 + project-readme-template.md | 188 +++ 45 files changed, 12477 insertions(+) create mode 100644 .gitignore create mode 100644 README.md create mode 100644 Template.code-workspace create mode 100644 docs/DOC_TRUST_MAP.md create mode 100644 docs/TOOLS.md create mode 100644 docs/WORK_CYCLE.md create mode 100644 docs/architecture/GUARDS.md create mode 100644 docs/architecture/README.md create mode 100644 docs/architecture/githooks/README.md create mode 100755 docs/architecture/githooks/commit-msg create mode 100755 docs/architecture/githooks/post-commit create mode 100755 docs/architecture/githooks/pre-commit create mode 100644 docs/architecture/scripts/audit-gate.mjs create mode 100755 docs/architecture/scripts/backup.sh create mode 100755 docs/architecture/scripts/check-env.sh create mode 100755 docs/architecture/scripts/commit-mine.sh create mode 100755 docs/architecture/scripts/dead-code.py create mode 100755 docs/architecture/scripts/deploy.py create mode 100755 docs/architecture/scripts/dev.sh create mode 100755 docs/architecture/scripts/doc-claims.sh create mode 100755 docs/architecture/scripts/doc-triggers.py create mode 100755 docs/architecture/scripts/duplication.py create mode 100755 docs/architecture/scripts/forgejo-issue.py create mode 100755 docs/architecture/scripts/healthcheck.sh create mode 100755 docs/architecture/scripts/migrate.sh create mode 100755 docs/architecture/scripts/prove-guard.sh create mode 100755 docs/architecture/scripts/release-notes.mjs create mode 100755 docs/architecture/scripts/release.sh create mode 100755 docs/architecture/scripts/scaffold.sh create mode 100755 docs/architecture/scripts/secrets.sh create mode 100755 docs/architecture/scripts/status.sh create mode 100755 docs/architecture/scripts/verify.sh create mode 100644 docs/data/README.md create mode 100644 docs/data/img/README.md create mode 100644 docs/design/README.md create mode 100644 docs/history/BATCH_LEDGER.md create mode 100644 docs/history/DEVELOPMENT_LOG.md create mode 100644 docs/history/HISTORY.md create mode 100644 docs/planning/PROJECT_PLAN.md create mode 100644 docs/qa/ClaudeQACoverage.md create mode 100644 docs/qa/ClaudeQAPlan.md create mode 100644 docs/qa/ClaudeReport.md create mode 100644 docs/security/SECURITY.md create mode 100644 docs/security/SECURITY_CHECKLIST.md create mode 100644 project-readme-template.md diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..e6776ff --- /dev/null +++ b/.gitignore @@ -0,0 +1,15 @@ +# Python bytecode from the scripts in docs/architecture/scripts/ +__pycache__/ +*.py[cod] + +# Credentials never belong in this tree. docs/security/SECURITY.md is the rule; +# this is the safety net, and it is not a substitute for reading that file. +.env +.env.* + +# Editor and OS cruft +.DS_Store +.vscode/ +.idea/ +*.swp +*~ diff --git a/README.md b/README.md new file mode 100644 index 0000000..2376e9d --- /dev/null +++ b/README.md @@ -0,0 +1,289 @@ +# Project Template + +A working documentation skeleton, the tracker convention that goes with it, and +twenty scripts that enforce both. Copy `docs/` into a new project, replace the +angle-bracket placeholders, delete the paragraphs that do not apply — then +create the labels and the first milestone in the repository's issue tracker. + +``` +Status: Current +Owner: Kaspa +Last reviewed: 2026-08-12 +Governs: Projects/Template/** +Review trigger: Any change to the tracker conventions or the QA run-state + grammar in the Privacy LLC Command Center's src/lib/; any + change to which project fields the agent API can write, or to + what reconcile reads; any script added to or removed from + docs/architecture/scripts/ +``` + +## Why this exists + +The Command Center at [privacyllc.dev](https://privacyllc.dev) reports on a +repository: how much is done, what the QA verdict is, and what the next +milestone entails. It can only do that for repositories that keep those things +somewhere it knows to look. + +Eight repositories were surveyed before this was rewritten. Exactly one kept the +markdown batch ledger the previous version of this template asked for. Three had +invented their own dialects; one had pushed nothing but a README. Meanwhile, +with no convention at all, those same repositories had accumulated 69, 54 and +205 issues — because an issue tracker has structure built into it and a +convention has only whoever remembers it. + +## The split + +| Kind of truth | Lives in | Why | +| --- | --- | --- | +| Narrative documents — vision, trust map, QA playbook, QA verdict, security | git, as markdown | prose with an argument, versioned with the code, readable at any sha | +| Work items — batches, tasks, defects, blockers | the issue tracker | built for concurrent uncoordinated writers, atomic state, `closes #N` from a commit, real timestamps | + +Nothing belongs in both. A to-do list in markdown and a tracker with the same +items is two records that will disagree, and neither will say which is right. + +## What the Command Center actually reads + +| Source | What is read | +| --- | --- | +| The repository's issue tracker | issue counts excluding pull requests, milestones with their own counters, and issues carrying the four severity labels | +| `docs/qa/ClaudeReport.md` | the three run-state lines — round, build SHA, and the overall sentence, quoted verbatim | +| `docs/data/img/` | `icon.webp`, `logo.webp`, `banner.webp` — fetched and rendered on the project page | + +Everything else in this tree is here because a project needs it, not because +something parses it. Rename, split or extend those freely. + +A repository that has not adopted the labels is reported as *not adopted* +rather than as zero defects, and one whose tracker is switched off is reported +as switched off rather than as an empty backlog. Absence is never rendered as a +measurement. + +**The files in that table are read at a commit, and the commit is remembered.** +When the SHA they were read from falls behind the repository's newest commit, +the docs report is marked *stale* — which is the accurate description of a +document that described the code at a commit nobody is running any more. It is +also what an unpushed correction looks like from outside. Documentation edits +therefore ride in the same commit as the change that caused them, and reach the +site on the same push. See `docs/WORK_CYCLE.md`. + +## What updates itself, and what does not + +**The numbers on the project screen come out of this repository's tracker.** The +percentage is *defined* rather than picked: closed issues over all issues, asked +with `type=issues` so pull requests are excluded and `state=all` rather than the +API's default of open. Beside it sits a second figure counted from the +repository's **own milestones**, closed over all — this deployment keeps +milestones in the tracker and never mirrors them into the site's table, so that +is where the milestone reading comes from. + +| On the project screen | Source | How it gets there | +| --- | --- | --- | +| Headline % — issues closed / all issues | the repository's tracker | automatic — on reconcile, and immediately on a webhook delivery | +| Second figure — milestones closed / all milestones | the repository's **milestones** | same read, same moment | +| Milestone coverage | how many issues sit inside a milestone at all | same read | +| QA verdict — round, build SHA, overall sentence | `docs/qa/ClaudeReport.md` | push the repository, then reconcile | +| Current summary / Next action | manual fields on the project | `PATCH /agent/projects/` — nothing else writes them | +| The Milestones card, and a typed weighted plan | the Command Center's **own** milestone table | admin UI only | +| Blockers | the Command Center's **own** blockers table | admin UI only | + +So closing an issue and closing a milestone each move a number a stakeholder +can see, the same day, with no further step. That is the whole reason the +convention insists on both. + +The last two rows are the ones that surprise people, and they are lists rather +than figures. The Milestones card reads the site's own table, which is empty on +this deployment by design — a repository can carry twenty-nine milestones, +have them counted in the figure above, and still show *"No milestones have been +added yet"* on that card. It is not a sync failure and reconciling will not +change it. + +Blockers have a sharper version of the same split. The agent API can *narrate* a +blocker in a check-in's `blockers` field, and it has no route that creates or +resolves one in the table. A run that posts the check-in and reports "blocker +filed" has told the truth about the note and a falsehood about the record. + +**An issue in no milestone is counted in the headline and invisible to the +milestone figure.** The site measures that gap on purpose — it is the reason +`Verify:` lines and milestones are per-issue requirements here rather than +housekeeping. Portfolio-wide when this was written, 272 of 650 issues belonged +to no milestone at all, including every one of the 205 in the repository that +had never adopted the convention. + +## The digest is the part somebody actually reads + +A worker polls the site every sixty seconds and sends a project digest to +Discord. It is where this convention stops being bookkeeping: the digest's +per-project figures come from **issues opened and closed** and from pushes, and +its next-step line comes, by rule, from **`nextAction` or the live milestone's +open issues** — never from a guess. + +Three consequences, each the direct form of a step in the cycle below: + +- **Work with no issue opened or closed produces a push and nothing else.** The + project reads as quiet on a day it was not. +- **A project with no `nextAction` and no open milestone gives the digest + nothing to say about what happens next**, and the digest will not invent one. +- **An opened issue labelled `P0` raises an immediate alert** — but only from a + repository whose webhook is registered. Without one, the site sees the + repository through the periodic sweep alone, and a P0 filed at nine is news at + the next reconcile rather than at nine. + +## After each piece of work + +`docs/WORK_CYCLE.md` is the seven-step version, with the argument for each step. +The short form: + +**close what you finished, with the evidence · file what you found · close the +milestone if the batch landed · update the documents the change triggered, in +the same commit · push · log the entry, with Next action and Blockers · then +reconcile and write the summary and next action.** + +The last two steps are the ones that get skipped, and the only ones nobody can +do for you later — by the time somebody notices the project screen is a week +stale, the person who knew what came next has forgotten. + +Where each of those lives is deliberate and worth reading once: the next action +has a live copy (the project field) and a dated copy (the newest log entry) and +they are different kinds of fact; a blocker is an issue in the tracker first, +and a row on the site only when a human puts it there. + +## The tracker convention + +### Where the tracker is + +**[dream.scheller.ltd](https://dream.scheller.ltd)** — a Forgejo instance, and +naming it is not a detail. The convention below is portable: milestones, issues +and labels exist on GitHub, GitLab and Jira, and every rule in it would work +there unchanged. **The reporting is not portable at all.** Every figure the +Command Center shows — the headline percentage, the milestone count, milestone +coverage, the alert an opened `P0` raises — is read from a Forgejo repository +mapped to the project. A project that follows every rule here in a different +tracker satisfies the convention completely and appears on privacyllc.dev as +having no tracker at all. + +Credentials are in **`~/.openclaw/docker-registry.env`**, which carries +`FORGEJO_REGISTRY`, `FORGEJO_REGISTRY_USER` and `FORGEJO_REGISTRY_TOKEN`: + +```bash +set -a; . ~/.openclaw/docker-registry.env; set +a +python3 docs/architecture/scripts/forgejo-issue.py list +``` + +Despite its name that token is not registry-scoped — it works across the whole +API, with push and pull. Never print it into a log or a chat. + +Two things that cost an hour each otherwise. **Cloudflare fronts the instance +and 1010-blocks clients that do not look like a browser or curl** — Python's +`urllib` among them — so every request needs `User-Agent: curl/8.5.0`; +`forgejo-issue.py` already sends it and anything new must too, or the failure +arrives as a Cloudflare HTML page that looks nothing like a Forgejo error. And +**`/issues` returns pull requests too** unless `type=issues` is passed, which is +why the counts here exclude them and a hand-rolled query disagrees by exactly +the number of open PRs. + +`forgejo-issue.py` takes the repository from this checkout's git remote, so it +cannot file into another project's tracker by accident. `--repo owner/name` +overrides that deliberately, and `--dry-run` prints the payloads and changes +nothing. + +> This block names one instance and one path on the machine that runs it. If +> this repository will be public, replace both with your own before the first +> push. + +### The rules + +- **Milestone = batch.** Name it `0.2 Payments`. The description says what the + batch is for and how anybody will know it landed. A due date means a + commitment — leave it empty rather than inventing one. +- **Issue = deliverable.** One per item, in its milestone, ending with a + `Verify:` line stating the acceptance check. +- **Labels:** `P0` ships broken or loses data · `P1` materially wrong but + shippable · `P2` cosmetic or low impact · `release-blocker` means a release + built today would be wrong rather than merely incomplete. Exactly these + names — they are queried by name. +- **Defects are issues** with a severity label. The QA *verdict* stays in + `ClaudeReport.md`; the counts come from the tracker. +- **Close with `closes #N`** in the commit that does the work, so the record + comes from the thing that happened rather than from a date typed afterwards. + Where no single commit finished it, close it by hand with the evidence — a + path, a symbol, a test name, or the command that proves it. "Done" is not a + close, because a close with nothing in it cannot be reopened with confidence. +- **Close the milestone when the batch lands.** Closing the last issue under it + does not, and a milestone that is complete but open reads as a batch still in + progress. It is how a batch becomes visible as *shipped* rather than merely + finished — and it is counted: milestones closed over milestones total is one + of the two figures the project screen shows. +- **Every issue belongs to a milestone.** An issue filed outside one still + counts against the headline percentage and is invisible to the milestone + figure, so the two readings describe different amounts of work. The site + measures that gap rather than ignoring it. + +## The rule that keeps the numbers honest + +**Do not pad the tracker.** Every open issue is a denominator. If the real +answer is "one milestone, three issues", file exactly that. Invented future work +makes every percentage drawn from the tracker wrong, permanently and in the same +direction. + +The same arithmetic runs the other way, and this half is the one that actually +happens: **an issue left open after the work is done understates the project for +as long as it stays open**, and the understatement compounds. A fortnight of +finished work with unclosed issues reads, from outside, as a fortnight of no +progress at all. + +## What comes with it, besides the documents + +`docs/architecture/scripts/` holds working code, not examples. Each takes its +configuration from the environment and hard-codes nothing about any deployment, +so a script copied from here cannot quietly point at the project it came from. +`docs/architecture/README.md` has the full table; the ones worth knowing about +on day one: + +| | | +| --- | --- | +| `release.sh` | version bump, guards, build, verify, push. Refuses to build on a half-run test suite or a malformed public origin — both of which had shipped in every release of the project this came from. | +| `verify.sh` | every check the project has, in one command, with one table saying which passed, which failed, and which did not run | +| `check-env.sh` | which variables are set and which are missing, before anything reads them | +| `secrets.sh` | credential shapes in a staged diff, using the project's own patterns where it has written them down | +| `prove-guard.sh` | breaks what a guard protects and requires the guard to go red, then restores the file from a trap. `GUARDS.md` §1 as a command — the rule that a guard nobody has seen fail is not yet evidence. | +| `doc-claims.sh` | every file a document names must exist — and `--covers`, whether every file that exists is named. The second is the one that catches a list quietly missing rows. | +| `duplication.py` `dead-code.py` | code that exists twice; exports nothing imports and assets nothing renders | +| `forgejo-issue.py` | files and closes issues in the convention below, with every rule of the convention above as a check the script refuses to break | +| `deploy.py` | updates a running stack to a published image. Publishing and deploying are separate decisions; `release.sh` makes the first and this makes the second. | +| `release-notes.mjs` | tags the release and writes the notes: sections from the commit types the hook enforces, the image and its digest, and the batches the release finished. Set `RELEASE_DEPLOY_NOTE` to say how this project deploys, or empty where publishing and deploying are one act. | + +`docs/architecture/githooks/` holds three hooks — typecheck and tests before a +commit, a conventional type in the message, and a push afterwards so a guarded +commit does not sit unpushed. They live in the repository rather than in +`.git/hooks`, which is not versioned; see their README for the one command that +points git at them. + +**`docs/architecture/GUARDS.md` is worth reading before you write a check.** +Every rule in it was learned from a guard that had been green for months over +something broken, and the first — prove a guard fails before believing it +passes — is the one that costs thirty seconds and catches the rest. + +## Adopting it + +1. `cp -r Projects/Template/docs /docs` +2. Fill in the status header on each file — especially **Review trigger**, which + is the line that keeps a document from going quietly stale. +3. Create the four labels, one milestone, and an issue per deliverable you can + actually see ahead of you — in the tracker named above, not whichever one the + host happens to offer. +4. Point git at the hooks: `cp docs/architecture/githooks/{pre-commit,commit-msg,post-commit} .githooks/ && chmod +x .githooks/* && git config core.hooksPath .githooks`. It is per-clone, so every checkout runs it once. +5. Run `bash docs/architecture/scripts/check-env.sh` and `bash docs/architecture/scripts/secrets.sh --tracked` — the first says what is not configured, the second is the one-time audit for what is already committed. +6. Map the repository on the project's edit screen in the Command Center, then + press **Pull**. One thing does not follow from mapping it and is silent when + missing: **register the repository's webhook** and confirm a delivery + arrives. Without one the site sees this repository only through the periodic + sweep — the counts still arrive, a day late, and an opened `P0` raises no + alert at all. +7. Skim `docs/TOOLS.md` — the fixed path every project has, saying which + scripts can stop you and where to start in a fresh clone. Then read + `docs/WORK_CYCLE.md` once, now rather than at the end of the first + session. It is what happens every time a piece of work finishes, and the two + steps it exists for — recording the next action and the blockers, and telling + the site what changed — are the two that are invisible when they are skipped. + +There is also a generated prompt for doing all of this with an agent, on the +Docs template screen of the Command Center — editable before you copy it. diff --git a/Template.code-workspace b/Template.code-workspace new file mode 100644 index 0000000..876a149 --- /dev/null +++ b/Template.code-workspace @@ -0,0 +1,8 @@ +{ + "folders": [ + { + "path": "." + } + ], + "settings": {} +} \ No newline at end of file diff --git a/docs/DOC_TRUST_MAP.md b/docs/DOC_TRUST_MAP.md new file mode 100644 index 0000000..71b8864 --- /dev/null +++ b/docs/DOC_TRUST_MAP.md @@ -0,0 +1,140 @@ +# Doc Trust Map — which document to believe + +``` +Status: Current +Owner: +Last reviewed: +Governs: docs/** +Review trigger: Any doc added, deleted or moved; any change to which doc owns a subject +``` + +> Write this file **last**, once the rest exists, and describe what is actually +> here rather than what a template said should be. Its whole value is being +> accurate about the others. + +## Work items are not in this tree + +The sequence of work, the open defects, and the things blocking a release all +live in this repository's **issue tracker**, not in `docs/`. Milestones are +batches; issues are deliverables; severity labels are `P0`, `P1`, `P2` and +`release-blocker`. + +This section exists to stop the next contributor starting a fresh markdown +to-do list. A list of things to do in two places is two records that will +disagree, and nothing will say which one is right. + +| Question | Answer lives in | +| --- | --- | +| What are we building, and for whom? | `docs/planning/PROJECT_PLAN.md` | +| What is the sequence of work? | milestones in the tracker | +| What is left in this batch? | open issues under that milestone | +| What is broken right now? | issues labelled `P0` / `P1` / `P2` | +| What makes a release wrong? | issues labelled `release-blocker` | +| What is the next action? | the `nextAction` field on the project at privacyllc.dev — the newest entry in `docs/history/DEVELOPMENT_LOG.md` says what it was *then* | +| What is blocking us? | the tracker, for the work; the blockers table at privacyllc.dev, for the stakeholder-facing version | +| What do I do when a piece of work is finished? | `docs/WORK_CYCLE.md` | +| What happened, and when? | `docs/history/DEVELOPMENT_LOG.md` | +| Did QA pass, and what does the tester think? | `docs/qa/ClaudeReport.md` | +| What did QA actually reach? | `docs/qa/ClaudeQACoverage.md` | +| How is it built? | `docs/architecture/` | +| How do I release, back up, or check this? | `docs/architecture/scripts/` | +| Which script do I run, and can it stop me? | `docs/TOOLS.md` — the signpost; `docs/architecture/README.md` has the table | +| What runs before a commit? | `docs/architecture/githooks/` | +| How do I write a check that will actually catch something? | `docs/architecture/GUARDS.md` | + +**Next action and blockers are recorded at the end of every piece of work, not +when somebody asks.** `docs/WORK_CYCLE.md` holds that procedure and the reason +each step is in it — including which of these channels updates itself and which +two only ever change because a person changed them. + +## Folder layout + +| Folder | Contents | +| --- | --- | +| `docs/planning/` | `PROJECT_PLAN` — the vision. Not the schedule; that is the tracker. | +| `docs/qa/` | `ClaudeQAPlan` (playbook), `ClaudeQACoverage` (what each pass reached), `ClaudeReport` (the verdict) | +| `docs/architecture/` | How it is built: module boundaries, data shapes, reference manuals | +| `docs/design/` | What it should feel like: product decisions, UI plans, copy | +| `docs/security/` | `SECURITY` — threat model, secret handling, what is out of scope | +| `docs/history/` | `DEVELOPMENT_LOG` (dated, append-only), `HISTORY` (the narrative), `BATCH_LEDGER` (archived, superseded by the tracker) | +| `docs/data/` | the three branding marks privacyllc.dev renders for this project | +| `docs/` root | this map; `WORK_CYCLE` — what to do when a piece of work is finished; `TOOLS` — where the scripts are and which can stop you | + +`README.md` stays at the repository root; it is the landing page and moving it +breaks that. Everything else lives under `docs/`. + +## Source-of-truth ladder + +When two sources disagree, believe them in this order: + +1. **Verified code behaviour** — read the source, run the tests +2. **The issue tracker** — for anything about state of work: what is open, what + closed, when, and by which commit +3. **Specialist docs** — authoritative for their own subject only +4. **History and logs** — a record of *then*, not a description of *now* + +A document that describes work as done is describing what was true when it was +written. The tracker is describing now. + +## The status header + +Every document in this tree opens with one, immediately after its H1: + +``` +Status: Current | Draft | Superseded | Archived +Owner: +Last reviewed: +Governs: +Review trigger: +``` + +**Exactly those four status words.** They are not a suggestion: a checker reads +them, and a document whose status falls outside the list is reported as having +an unknown one rather than being quietly accepted. `Historical` is the word this +convention used to have and deliberately dropped — `Archived` covers it, and a +vocabulary a script enforces beats one described in prose, which drifts the +moment somebody writes something adjacent to it. `Draft` earns its place by +naming a real state the old list had no word for: a document being written, +which is neither current nor superseded. + +**Review trigger** is the line that matters. "Last reviewed" ages on its own and +a reader cannot tell a current document from an abandoned one by looking at it; +a trigger names the event that should bring somebody back, which is what stops a +document becoming confidently wrong. It is also checked: the header is read from +the first sixteen lines of the file, and one carrying `Status` without +`Review trigger` is reported as incomplete — that combination looks finished and +is not. + +## Declaring a document deliberately absent + +A repository may decide it will not keep one of these documents in git — QA and +security notes held outside the repository on purpose, say. Say so **here**, one +line per path, anywhere in this file: + +``` +Exempt: — +``` + +**Write the real path only when you mean it.** Outside a fenced block, an +`Exempt:` line naming a real document is not an example — it is a declaration, +and the checker will report that document as deliberately absent in every +repository that copied the file. Keep illustrations fenced, and use placeholders +anyway, as the form above does. + +Fenced blocks are skipped, but only since 2026-08-12. Before that the parser +read every line, which meant that documenting this feature *used* it: the +example a writer naturally reaches for is a real required path, because an +illustration with a fake one teaches nothing. A draft of this very section +carried `Exempt: docs/security/SECURITY.md` inside a fence and would have marked +that document deliberately absent everywhere it was copied. It is recorded here +rather than quietly fixed because the shape recurs — a parser that cannot tell a +description of a thing from the thing itself. + +The declaration lives in this file because the decision does. Holding the list +somewhere else would be two records of one fact, and the other one would never +hear that this repository changed its mind. + +**`docs/data/` and `docs/data/img/` cannot be exempted.** A declaration naming +either is refused rather than honoured, because the Command Center renders what +is in them — an exemption would produce a project card with no icon and nothing +explaining why, which is the failure the check exists to catch. diff --git a/docs/TOOLS.md b/docs/TOOLS.md new file mode 100644 index 0000000..5cd9a30 --- /dev/null +++ b/docs/TOOLS.md @@ -0,0 +1,75 @@ +# Tools — where the scripts are, and which ones can stop you + +``` +Status: Current +Owner: +Last reviewed: +Governs: docs/architecture/scripts/**, docs/architecture/githooks/** +Review trigger: Any script added to, removed from or repurposed in + docs/architecture/scripts/; any change to which of them gates +``` + +> A signpost, deliberately. Every project that adopts this template has a +> `docs/TOOLS.md`, so "read `docs/TOOLS.md` first" is an instruction that works +> without knowing anything about the project — which is the whole reason this +> file exists at a fixed path. + +## The list is not here + +**[`architecture/README.md`](architecture/README.md)** holds the table of what +ships and what each script is. That is the one copy. + +A second table here would be two records of one fact, and the other one would +never hear that a script was renamed — the failure `DOC_TRUST_MAP.md` exists to +prevent, applied to the tooling instead of the documents. So this file answers +the questions that table does not, and points at it for everything else. + +## If this project has fewer scripts than that table lists + +That is the intended state, not a broken copy. `scaffold.sh` writes the +documents and deliberately leaves the scripts behind — *"an unconfigured +`release.sh` landing in every new repository is a loaded gun, not a head +start"* — so they are taken from the template one at a time, having been read. + +Which makes the table a menu rather than an inventory here: it says what exists +to be copied. Once a script is in this project, it is this project's, and its +row in `architecture/README.md` should say what it does **here** if that has +drifted from the template's version. + +## Which ones can stop you + +Not in a table, because the honest answer lives in each script's own header and +would go stale here. The rule that matters: + +**Exit code `2` is never a pass.** These scripts distinguish "the check ran and +found nothing" from "the check did not run", because those look identical from +the outside and only one of them is evidence. A CI step or a hook that treats a +`2` as success has quietly turned the check off. Each script states its codes at +the top; read them there. + +The hooks are the other place work gets stopped: +[`architecture/githooks/`](architecture/githooks/README.md) has the one install +command and the table of what each hook runs. + +## Where to start in a fresh clone + +1. `bash docs/architecture/scripts/check-env.sh` — what is configured and what + is missing, before anything reads it. +2. `bash docs/architecture/scripts/secrets.sh --tracked` — the one-time audit of + what is already committed. The staged-diff mode is for the hook; this mode is + for the day you adopt the template. +3. [`architecture/GUARDS.md`](architecture/GUARDS.md) — how to write a check that + can actually fail, before you write one. `scripts/prove-guard.sh` performs its + first rule. + +## Adding one + +Put it in `docs/architecture/scripts/`, give it a header saying what it does and +**which incident motivated it**, state its exit codes, and add a row to +`architecture/README.md`'s table — this file's `Review trigger` fires on exactly +that. + +The bar, from the scripts that are already here: **done by hand three times, or +once with a consequence.** A script written before either of those has no failure +to describe in its header, which is the part that stops the next person deleting +it. diff --git a/docs/WORK_CYCLE.md b/docs/WORK_CYCLE.md new file mode 100644 index 0000000..4122746 --- /dev/null +++ b/docs/WORK_CYCLE.md @@ -0,0 +1,262 @@ +# The work cycle — what happens at the end of a piece of work + +``` +Status: Current +Owner: +Last reviewed: +Governs: what must be true before a piece of work counts as finished +Review trigger: Any change to what the Command Center reads, or to which of + those channels a person rather than an agent has to write +``` + +> **This is a procedure, not a status board.** Nothing here records what is open, +> what is next, or what is blocked — those live in the tracker and on +> privacyllc.dev, and a copy of them in this file would be the second disagreeing +> record that `DOC_TRUST_MAP.md` exists to prevent. + +## Why this file exists + +A piece of work ends in more than one place. The code is committed; the issue +that asked for it is still open; the document the change contradicts still says +the old thing; and the project screen at +[privacyllc.dev](https://privacyllc.dev) still shows last week's next action to +whoever opens it. + +None of those catch up on their own. Two of them cannot be caught up later by +anybody but the person who did the work, because by then nobody knows what the +next action was meant to be. + +## The cycle + +Run all of it, in this order, every time. It is short on purpose. + +1. **Close what you finished.** `closes #N` in the commit that does the work, so + the record comes from the thing that happened rather than a date typed + afterwards. If no single commit finished it, close it by hand with the + evidence — a path, a symbol, a test name, or the command that proves it. + *"Done" is not a close.* +2. **File what you found.** A defect noticed on the way past is an issue with a + severity label and the build SHA it was seen at, not a memory. Filing it costs + a minute; the alternative is finding it again from scratch, or shipping it. +3. **Close the milestone if the batch landed.** A milestone with every issue + closed and itself still open reads as a batch still in progress — see + [Open and closed are not bookkeeping](#open-and-closed-are-not-bookkeeping). +4. **Update the documents this change triggered.** Read the `Review trigger` + lines: a new module, a changed data shape, a new migration, a new boundary + something crosses. Those edits go in **the same commit as the code**, for the + reason in [Docs travel with the push](#docs-travel-with-the-push). +5. **Push.** Nothing above is visible off this machine until you do, and step 7 + reports on what was pushed. +6. **Write the log entry** in `docs/history/DEVELOPMENT_LOG.md`: what changed, + what it proved, **Next action**, and **Blockers**. Dated, append-only, newest + first. +7. **Tell the Command Center**, which is three calls: reconcile, so it re-reads + the tracker and the pushed docs; `PATCH` the project's summary and next + action, which nothing else writes; and a check-in if what changed is worth a + timestamped note. Commands are [at the bottom](#the-commands). + +Steps 6 and 7 are the two that get skipped, and they are the two nobody else can +do afterwards. + +## Next action — where it lives + +| Copy | Where | What it is | +| --- | --- | --- | +| The current one | `nextAction` on the project, privacyllc.dev | the live answer to "what happens next", shown to whoever opens the project screen | +| The dated one | the newest entry in `DEVELOPMENT_LOG.md` | what the next action was **at that point** — history, not status | + +These are not two records of the same thing, and the distinction is worth +holding on to. The field is overwritten every time and always describes now. The +log entry is never edited and describes a moment — which is what makes it safe +to keep, and why an old entry naming a next action that has since been done is +not stale, it is a receipt. + +Write the next action as an **action**: the thing a person would start on +Monday, specific enough to begin without asking a question. "Continue the work" +is not one. If the honest answer is that you do not know, that is a real answer — +say what has to be decided and by whom. + +`currentSummary`, `nextAction` and `description` are **write-only**: they are +deliberately absent from every API response, because free text can name a +customer or an unannounced product. Only the admin screen shows them back. Never +report them as empty because a `GET` did not return them. + +## Blockers — where they live + +A blocker is recorded in up to three places, and **an agent can write only the +first and the third**: + +| Where | What it holds | Who writes it | +| --- | --- | --- | +| The tracker | the work itself — an issue labelled `release-blocker`, or `P0` when it ships broken | anyone, including an agent | +| The blockers table on privacyllc.dev | the business-facing blocker, with a severity, an owner and a resolution note | **a human, in the admin UI** | +| A check-in's `blockers` field | narrative: what is stuck, said in a timestamped note | anyone, including an agent | + +The agent API has no route that creates, edits or resolves a blocker on the +site. Posting a check-in whose `blockers` field says something is stuck records +*narrative* — it does not touch the blockers table, and it does not clear +anything. An agent that reports "blocker filed" after a check-in has told the +truth about the note and a falsehood about the table. + +So: **file the issue** — that is the copy the work is actually tracked in — and +when it belongs in front of a stakeholder, say so plainly and let a human enter +it. Before treating a site blocker as resolved, restate its title and status and +get explicit confirmation; the admin UI requires a resolution note that the API +cannot supply. + +If the work is blocked and nothing is filed anywhere, the project simply looks +slow. + +## Open and closed are not bookkeeping + +Every open issue is a denominator, and not in the abstract: the percentage on +the project screen **is** closed issues over all issues in this tracker. Nothing +else produces it. That has three consequences worth stating in full: + +- **An issue left open after the work is done** understates the project + permanently, and the understatement compounds — a fortnight of finished work + with unclosed issues reads as a fortnight of no progress. +- **An issue closed without evidence** cannot be reopened with confidence, + because nothing in it says what "fixed" meant. That is why the close comment + carries the path, symbol, test or command. +- **Invented future work** makes every percentage wrong, permanently and in one + direction. Do not pad the tracker. If the real answer is one milestone and + three issues, file exactly that. + +Milestones are the same argument at batch scale, and they are counted the same +way: milestones closed over milestones total is the second figure on the project +screen. Closing the last issue under a milestone does not close the milestone, +so a tracker full of complete-but-open batches reports a project as less +finished than it is — and cannot answer "what shipped". + +**File every issue into a milestone.** One filed outside still counts against +the headline percentage while being invisible to the milestone figure, which is +how two readings of the same project come to describe different amounts of work. +The site measures the gap rather than ignoring it. + +One trap that costs an afternoon, and it is about a *card* rather than a figure: +the Milestones list on privacyllc.dev reads the Command Center's own table, +which only an admin can write. A repository whose milestones are being counted +in the figure above can still show *"No milestones have been added yet"* in that +list. It is not a sync failure and no amount of reconciling changes it. + +## Docs travel with the push + +The Command Center reads this repository's documents at a commit, and stores the +SHA it read them from. When that SHA falls behind the repository's newest +commit, the docs report is marked **stale** — not wrong, not missing, *stale*, +which is the honest description of a document that was accurate at a commit +nobody is running any more. + +Two ways to produce it, and both are ordinary carelessness rather than bad luck: + +- **Code pushed, documents not updated.** The report is recomputed at the new + SHA against prose describing the old behaviour. Nothing flags this; the + document is simply confidently wrong now, and its `Last reviewed` line still + looks recent. +- **Documents updated, not pushed.** The site keeps reporting the old ones. A + correction that exists only on your machine has not been made. + +Hence step 4's insistence that documentation edits ride in the same commit as +the change that caused them. It is not tidiness — a doc commit that comes "after +this next thing" is the commit that does not get written, and the manual's +migration table in the project this template came from sat six migrations behind +before anybody noticed. Every reader in between trusted it. + +## What the Command Center reads, and who writes it + +Four independent channels, and **only two of them are automatic**: + +| On the project screen | Source | How it gets there | +| --- | --- | --- | +| Headline % — issues closed / all issues | the repository's tracker | automatic — on reconcile, and immediately on a webhook delivery | +| Second figure — milestones closed / all milestones | the repository's **milestones** | same read, same moment | +| Milestone coverage — issues that sit in no milestone | the repository's tracker | same read | +| QA verdict — round, build SHA, overall sentence | `docs/qa/ClaudeReport.md` | push the repository, then reconcile | +| Current summary / Next action | manual fields on the project | `PATCH /agent/projects/` — nothing else writes them | +| The Milestones card, and a typed weighted plan | the Command Center's **own** milestone table | admin UI only | +| Blockers | the Command Center's **own** blockers table | admin UI only | + +The first three are why steps 1 and 3 are steps: **closing an issue and closing +a milestone each move a figure a stakeholder can see, that day.** The last two +are lists rather than figures, and they are the ones an agent cannot write — a +repository whose milestones are counted in the second figure can still show +*"No milestones have been added yet"* on the card, because that card reads a +table only an admin fills in. + +A repository that has not adopted the four label names is reported as *not +adopted* rather than as zero defects, and one with its tracker switched off is +reported as switched off rather than as an empty backlog. Absence is never +rendered as a measurement — which is exactly why a real backlog nobody filed +looks like nothing at all. + +## The commands + +The tracker is the Forgejo instance at +**[dream.scheller.ltd](https://dream.scheller.ltd)**, and steps 1 to 3 happen +there. Its credentials, the Cloudflare trap and the reason it has to be that +instance rather than any tracker are in the project README's *Where the tracker +is* — one copy, named once: + +```bash +set -a; . ~/.openclaw/docker-registry.env; set +a +python3 docs/architecture/scripts/forgejo-issue.py list +python3 docs/architecture/scripts/forgejo-issue.py close 42 "Fixed in a1b2c3d; tests/foo.test.js covers it." +``` + +Step 7 talks to the Command Center instead: + +```bash +# The token lives in this machine's credential store, never in the repository. +TOKEN="$PRIVACY_LLC_TOKEN" +BASE="https://privacyllc.dev/api/internal/v1" +AUTH="Authorization: Bearer $TOKEN" +``` + +**Reconcile** — re-read the tracker and the pushed documents. Do this after the +push, not before: + +```bash +curl -sk -X POST -H "$AUTH" -H "Content-Type: application/json" \ + -d '{"kind":"reconcile_all"}' "$BASE/agent/jobs" +``` + +**Summary and next action** — read the project first for its `version`, then +send a flat body carrying that version. A stale version is rejected rather than +silently overwriting somebody else's edit: + +```bash +curl -sk -H "$AUTH" "$BASE/agent/projects/" # for version + state + +curl -sk -X PATCH -H "$AUTH" -H "Content-Type: application/json" \ + -d '{"version":,"health":"on_track", + "currentSummary":"", + "nextAction":""}' \ + "$BASE/agent/projects/" +``` + +**A check-in** — a timestamped "what changed", when there is something real to +report. `summary` is required and must say something; the API refuses an empty +"no change" update, deliberately: + +```bash +curl -sk -X POST -H "$AUTH" -H "Content-Type: application/json" \ + -d '{"projectId":"","health":"on_track", + "summary":"","accomplishments":"", + "blockers":"","nextActions":""}' \ + "$BASE/agent/updates" +``` + +Health is one of `on_track`, `caution`, `off_track`, `unknown`. Never invent +one, and never report progress the API did not return. + +## What this file is not + +- **Not the work list.** That is the tracker: milestones are batches, issues are + deliverables. +- **Not the release procedure.** That is `docs/architecture/scripts/release.sh` + and `docs/security/SECURITY_CHECKLIST.md`. +- **Not the QA procedure.** That is `docs/qa/ClaudeQAPlan.md`, which ends in its + own version of step 7. +- **Not a place to record status.** If you are about to add "current state" or a + list of outstanding items below this line, the tracker is where it goes. diff --git a/docs/architecture/GUARDS.md b/docs/architecture/GUARDS.md new file mode 100644 index 0000000..9b8545a --- /dev/null +++ b/docs/architecture/GUARDS.md @@ -0,0 +1,142 @@ +# Guards — how to write a check that actually checks + +``` +Status: Current +Owner: +Last reviewed: +Governs: structural tests, source-grep assertions, probes, and any check whose + passing is taken as evidence +Review trigger: A guard is found to have been passing while the thing it guards + was broken; a new class of check is added to the suite. +``` + +A guard that cannot fail is worse than no guard, because it is trusted. Every +rule here was learned by finding one that had been green for months over +something broken. + +## 1. Prove the guard fails before you believe it passes + +The one discipline that matters most, and it takes thirty seconds: + +```bash +cp src/lib/thing.ts /tmp/thing.bak +# break exactly the thing the test protects +sed -i 's/if (body.error)/if (false)/' src/lib/thing.ts +npx vitest run tests/thing.test.ts # expect: exactly one failure +cp /tmp/thing.bak src/lib/thing.ts +npx vitest run tests/thing.test.ts # expect: green again +``` + +**Exactly one** is the part people skip. If breaking the guard's target fails +three tests, two of them are coincidental and will mask a real regression later. +If it fails none, the guard is decoration — and you have just learned that for +the price of one `sed`. + +`scripts/prove-guard.sh` performs exactly this, which removes the two ways it +gets skipped: the restore is a `trap`, so an interrupted run cannot leave the +code broken, and the failure count comes from the runner's own summary rather +than from eyeballing red — one failing test is routinely reported on half a +dozen lines, and counting those calls a clean result six coincidental +failures. + +Do this when you write a guard, and again when you change what it guards. A +test written alongside the code it tests has never been observed failing. + +## 2. A source-grep guard must tell code from the comment about code + +Structural tests that assert a file does *not* contain some pattern will match +the docblock explaining why that pattern is forbidden. So the clearest possible +comment breaks the test, and the obvious fix is to delete the explanation. + +Strip comments first: + +```ts +const codeOf = (path: string) => + readFileSync(path, "utf8") + .split("\n") + .filter((line) => !/^\s*(\*|\/\/|\{\/\*)/.test(line)) + .join("\n"); + +expect(codeOf("src/lib/thing.ts")).not.toContain("dangerouslySetInnerHTML"); +``` + +Otherwise the guard quietly punishes documenting the rule it exists to enforce — +which is exactly backwards, because the comment is how the next person learns +the rule at all. + +## 3. Pin the behaviour, not the spelling + +A guard should fail when the protected behaviour breaks and stay quiet +otherwise. One that asserts on a variable name fails on a rename that changed +nothing. + +```ts +// Brittle: breaks when the variable is renamed, while the fallback it protects +// is untouched. +expect(route).toContain("readAsset(project.forgejoRepo"); + +// Pins the behaviour: the route fetches through the wrapper that tries both +// spellings, and never through the raw reader. +expect(route).toMatch(/readAsset\(\s*\w+,\s*ASSETS\[which\]\s*\)/); +expect(body).not.toContain("readFileBytes("); +``` + +A guard that fails on changes it does not care about is one people learn to edit +rather than heed, and the edit is usually deletion. + +## 4. A negative result is only as good as the probe that produced it + +"The check found nothing" and "the check did not run" are different facts, and +they look identical from the outside. Before reporting an absence, prove the +instrument worked: + +```bash +# Not this alone — an unreadable file produces the same silence as an unset key +grep -c '^WANTED=' /proc/$PID/environ + +# Establish the read succeeded first +tr '\0' '\n' < /proc/$PID/environ | grep -c . # 0 here means "could not read" +``` + +This is the confident-absence failure one level up: the same trap as a screen +rendering a failed query as a count of zero, applied to your own diagnosis. + +## 5. A guard that is often wrong is worse than none + +A check with a high false-positive rate trains everybody to skip its output, +including on the day it is right. + +One written for this template flagged **684 of 1142** candidates on its first +run. That was not 684 findings, it was a broken heuristic — and shipping it +would have taught its readers that the check is noise. Two rounds of narrowing +brought it to 17 of 363, all of them real. + +If a new guard's first run is loud, tune it until it is quiet before anybody +relies on it. Report the false-positive rate you settled at, so the next person +knows what silence is worth. + +## 6. Guards belong before the artifact exists + +A check that runs after publication catches the problem once it is somewhere it +cannot be taken back from: the tag is in the registry, and refusing the commit +afterwards leaves git with no record of it. + +Order the gates so the expensive, irreversible step is last — preconditions, +guards, build, verify the built thing is what was asked for, publish, and record +it last of all. + +## 7. When the gate finds something that invalidates the operation, stop + +Printing a warning and continuing produces the worst outcome available: the bad +thing happens *and* a reassuring summary appears above it. + +The question is not how bad the finding is. It is **whether it invalidates what +the operation claims**: + +- A release whose test gate skipped half the suite — a release claims to be + tested. **Refuse.** +- A backup written to a group-readable directory — the backup is still a + backup. **Warn.** + +Escape hatches are fine, and they have to be asked for by name, never be the +default, and say plainly what is being given up. diff --git a/docs/architecture/README.md b/docs/architecture/README.md new file mode 100644 index 0000000..0e72459 --- /dev/null +++ b/docs/architecture/README.md @@ -0,0 +1,79 @@ +# Architecture + +``` +Status: Current +Owner: +Last reviewed: +Governs: docs/architecture/** +Review trigger: Any new module, any change to a module boundary or a data shape +``` + +## What belongs here + +How the thing is built, for somebody who has to change it: + +- **Module boundaries** — what each part owns, and what it is not allowed to + know about. The boundaries are the architecture; everything else is detail. +- **Data shapes** — the structures that outlive a single function, especially + anything persisted or sent over a wire. +- **Reference manuals** — the long documents that answer "how does X work" + without requiring a full read of X. +- **Decisions with consequences** — why this database, why this concurrency + model, why this dependency. Include the option that was rejected and what it + would have cost, because that is the part nobody can reconstruct later. + +## Documents here + +- **`GUARDS.md`** — how to write a check that actually checks. Read it before + adding a structural test or a probe; every rule in it was learned from a guard + that had been green over something broken. + +## What ships in this folder + +Working code, not just prose. Copy what a project needs and delete the rest — +these are a starting point with the arguments already made, not a framework. + +This table is the one copy of that list. [`docs/TOOLS.md`](../TOOLS.md) is the +signpost every project is expected to have — it points here rather than +repeating it, and answers the two questions this table does not: which scripts +can stop you, and where to start in a fresh clone. + +| Path | What it is | +| --- | --- | +| `scripts/release.sh` | version bump, guards, build, verify, push, prune. Refuses to build on a half-run test suite or a malformed public origin. | +| `scripts/verify.sh` | the repo's own checks, in one command | +| `scripts/check-env.sh` | which variables are set, which are missing, before anything reads them | +| `scripts/migrate.sh` | apply and report migrations, including the ones that run outside a transaction | +| `scripts/backup.sh` | a dump that is verified before it is trusted | +| `scripts/healthcheck.sh` | a liveness tick with the URL written down rather than re-derived each run | +| `scripts/status.sh` | what is deployed, and whether it matches this checkout | +| `scripts/dev.sh` | bring the local stack up | +| `scripts/scaffold.sh` | lay out a new project in this shape | +| `scripts/doc-triggers.py` | which documents a **pending** change fires, read from their `Governs:` headers. The `Review trigger` on each document names the change that should send somebody back to it; this is the check that asks before the commit rather than after | +| `scripts/prove-guard.sh` | breaks the thing a guard protects, requires the guard to go red, restores the file from a trap. `GUARDS.md` §1 written out as a command, including the count — one failing test reported on six lines is not six failures | +| `scripts/commit-mine.sh` | commits only the paths you name, by pathspec, after the secret scan. For a tree something else is also writing: what anyone else has staged is reported and left exactly as it was | +| `scripts/doc-claims.sh` | every file a document names must exist, and (`--covers`) every file that exists is named — the second is the one that catches a list missing rows | +| `scripts/duplication.py` | code that exists twice, tuned so what it reports is worth reading | +| `scripts/dead-code.py` | exports nothing imports, and assets nothing renders | +| `scripts/secrets.sh` | credential shapes in a staged diff, using the project's own patterns where it has them | +| `scripts/audit-gate.mjs` | high/critical advisories in **production** dependencies, with the allowlist npm does not have. An entry must say why the advisory cannot reach this app, what would make it reachable, and what retires the entry — three fields, so a waiver stays falsifiable. Exits 2 when nothing was checked. | +| `scripts/forgejo-issue.py` | file and close issues in the tracker convention, with every rule of it as a check | +| `scripts/deploy.py` | update the running stack to a published image. Publishing and deploying are separate; this is the second one. **The only copy** — it existed twice and drifted (#209); the `privacyllc-deploy` skill's is now a symlink to this file. Identity-free by design: it reads `DEPLOY_IMAGE`, `DEPLOY_STACK_ID`, `DEPLOY_CONTAINER` and `DEPLOY_SITE_URL` from the environment and refuses to run without them, so each project supplies its own via a wrapper. Never hard-code one here — least of all the site URL, which is frozen into the image at build time. | +| `scripts/release-notes.mjs` | tags the release and writes its notes, grouped by the commit types the message hook already enforces. Runs after `release.sh` has published, so a failure here cannot cost an image. Scrubs credential shapes out of commit subjects first — the body goes to a public repository. | +| `githooks/` | pre-commit, commit-msg, post-commit — see its README for the one install command | + +Every script takes its configuration from the environment and hard-codes nothing +about any particular deployment. `check-env.sh` is the one to run first. + +## What does not belong here + +- Product intent — that is `docs/planning/PROJECT_PLAN.md` +- What it should feel like — that is `docs/design/` +- What happened while building it — that is a history log, not architecture + +## A note on drift + +Architecture docs go stale faster than any other kind, because code changes +under them silently. This is exactly what the **Review trigger** line is for: +name the change that should send somebody back here, and a reader can tell +whether the trigger has fired. diff --git a/docs/architecture/githooks/README.md b/docs/architecture/githooks/README.md new file mode 100644 index 0000000..0d9240a --- /dev/null +++ b/docs/architecture/githooks/README.md @@ -0,0 +1,63 @@ +# Git hooks + +``` +Status: Current +Owner: +Last reviewed: +Governs: docs/architecture/githooks/** and the .githooks/ a project installs +Review trigger: A new guard the repository wants run before a commit; any change + to what a commit message must contain. +``` + +Three hooks, and the reason they live in the repository rather than in +`.git/hooks`: that directory is not versioned, so a hook living there protects +exactly one clone on exactly one machine. These are committed, and one command +points git at them. + +## Install + +```bash +mkdir -p .githooks +cp docs/architecture/githooks/{pre-commit,commit-msg,post-commit} .githooks/ +chmod +x .githooks/* +git config core.hooksPath .githooks +``` + +`core.hooksPath` is per-clone, so every checkout runs that last line once. Say so +in the project README — an uninstalled hook fails silently, which is the same +class of problem the hooks exist to prevent. + +## What each one does + +| Hook | Guard | +| --- | --- | +| `pre-commit` | typecheck, then the test suite, when `.ts`/`.tsx` is staged | +| `commit-msg` | refuses a message with no conventional type | +| `post-commit` | pushes to the remote, so a guarded commit does not sit locally | + +## Two settings worth making before you rely on them + +**`pre-commit` will use a local test database if you tell it where one is.** +Set `TEST_DB_PORT` and `TEST_DB_URL` together. Leave them unset and the database +suites skip, which is the right default on a machine with no Postgres — but it +means the hook is running a fraction of the suite, so `release.sh` is +deliberately stricter and refuses rather than warns. + +**`post-commit` pushes.** That is the intent — the commit that first added a +pre-commit hook to the reference project sat unpushed for a day, guarded and +invisible — but it is a surprise if you were not expecting it. Read it before +installing it on a repository with a protected branch. + +It has a second consequence worth knowing about: the push is what the Command +Center reads, so **whatever documentation was not in that commit is now behind +the code by one push**. That is the mechanical reason `docs/WORK_CYCLE.md` asks +for doc edits in the same commit as the change rather than in a tidy-up +afterwards — with this hook installed, "I will document it next commit" means +the site has already published the version without it. + +## Why a hook and not CI + +Both. These are the guards that must run before the artifact exists: a hook that +fires after a push, or a CI job that fires after a publish, catches the problem +once it is already somewhere it cannot be taken back from. CI is the second +opinion; this is the one that runs first. diff --git a/docs/architecture/githooks/commit-msg b/docs/architecture/githooks/commit-msg new file mode 100755 index 0000000..d02ebfc --- /dev/null +++ b/docs/architecture/githooks/commit-msg @@ -0,0 +1,110 @@ +#!/usr/bin/env bash +# +# Every commit says what kind of change it is, before it says anything else. +# +# ## Why +# +# Ninety-nine of this repository's commits already carry a conventional type — +# `feat:`, `fix:`, `docs:`, `chore:` — and ninety-six do not, including a run of +# recent ones written as bare sentences. That split is the problem: `git log +# --grep '^fix'` answers "what have we fixed" for half the history and quietly +# omits the other half, which is worse than having no convention at all, because +# the answer looks complete. +# +# So the type is required, and the vocabulary is closed. A closed list is the +# point — `feat`, `feature` and `feat!` as three spellings of one idea is how a +# convention stops being searchable. +# +# ## The vocabulary +# +# feat a new capability somebody can use +# fix a defect. The thing behaved wrongly and now does not +# ui appearance, layout, copy, or interaction, with no change in what +# the software knows or decides +# docs documentation only +# test tests only, with no change to what they test +# refactor same behaviour, different shape. If behaviour changed it is not this +# security hardening, boundaries, secret handling. Kept separate from `fix` +# on purpose: "what have we hardened" is a question worth being able +# to ask on its own, and it is the one an auditor asks first +# perf faster or lighter, same answers +# chore tooling, dependencies, releases. The bucket for work that is not +# about the product +# +# `harden`, `style` and `content` each appear once or twice in the history and +# are deliberately not here — they are `security`, `ui` and `docs` under other +# names, and a synonym is a hole in a closed list. +# +# ## Scope is optional, and lowercase +# +# fix(admin): ... the fifteen existing `admin` scopes, and `release`, +# `integrations`, `db` and the rest, all keep working. +# +# ## What it deliberately does not enforce +# +# Subject length. Thirty-five existing subjects run past 72 characters, several +# of them deliberately, and rejecting a commit for a well-written 80-character +# sentence would teach people to use `--no-verify` — which switches off the +# checks that actually matter. The type is the part a tool reads; the length is +# a matter of taste and stays that way. +# +# ## Escape hatch +# +# SKIP_GUARDS=1 git commit ... skips this too, and says so +# +# The same variable the pre-commit hook uses, because two switches for "I know +# what I am doing" is one more than anybody will remember. + +set -uo pipefail + +say() { printf '\033[1mcommit-msg:\033[0m %s\n' "$*" >&2; } + +message_file="$1" +subject=$(head -1 "$message_file") + +if [ -n "${SKIP_GUARDS:-}" ]; then + say "SKIP_GUARDS set — the commit type was NOT checked." + exit 0 +fi + +# Git writes these itself, or writes them on a human's behalf during a rebase. +# Rejecting them would break `git merge`, `git revert` and autosquash for a +# convention none of them ever agreed to. +case "$subject" in + "Merge "*|"Revert "*|"fixup!"*|"squash!"*|"amend!"*) + exit 0 + ;; +esac + +# A comment-only file is an aborted commit; git handles that itself. +if [ -z "${subject// /}" ]; then + exit 0 +fi + +TYPES="feat|fix|ui|docs|test|refactor|security|perf|chore" + +if printf '%s' "$subject" | grep -qE "^(${TYPES})(\([a-z0-9._-]+\))?!?: .+"; then + exit 0 +fi + +say "the subject line needs a type." +say "" +say " got: ${subject}" +say "" +say " expected: : e.g. fix: stop counting milestones as records" +say " (): ui(admin): mark cloud models on the picker" +say "" +say " types: feat a new capability" +say " fix a defect, now not" +say " ui appearance, layout, copy — no change to what it decides" +say " docs documentation only" +say " test tests only" +say " refactor same behaviour, different shape" +say " security hardening, boundaries, secrets" +say " perf faster or lighter, same answers" +say " chore tooling, dependencies, releases" +say "" +say " Your message is kept. Run 'git commit' again to edit it, or" +say " SKIP_GUARDS=1 git commit ... to bypass this loudly." + +exit 1 diff --git a/docs/architecture/githooks/post-commit b/docs/architecture/githooks/post-commit new file mode 100755 index 0000000..a7bbaa0 --- /dev/null +++ b/docs/architecture/githooks/post-commit @@ -0,0 +1,91 @@ +#!/usr/bin/env bash +# +# Every commit goes to Forgejo, without anybody having to remember the push. +# +# ## Why a hook rather than a habit +# +# The commit that added `pre-commit` sat unpushed for a day. Nothing was wrong +# with it; it just never got the second command. That is the whole failure this +# closes — work that exists on one laptop and nowhere else is work that is one +# disk away from gone, and it is invisible to anybody reading the tracker. +# +# ## It runs after the commit, and cannot undo one +# +# git ignores this hook's exit code, which is the right shape for the job: a +# network that is down must not cost somebody a commit they already made. So a +# failed push is **reported loudly and left in place** — the commit stands, the +# branch is simply still ahead, and the next commit tries again. +# +# What it will never do is force. A rejected push means the remote has something +# this checkout has not seen, and the fix for that is a human running a pull, not +# a hook overwriting the difference. +# +# ## When it deliberately stays out of the way +# +# - mid-rebase, mid-cherry-pick, mid-am: every step of a rebase fires this +# hook, and pushing an intermediate commit publishes a history that is about +# to be rewritten. Wait for the rebase to finish. +# - detached HEAD: there is no branch to push, and guessing one is worse than +# doing nothing. +# - no `origin`: a clone with no remote is a legitimate state, not an error. +# +# SKIP_PUSH=1 git commit ... commits without publishing, loudly +# +# The counterpart to `SKIP_GUARDS` in the pre-commit hook, and loud for the same +# reason: an exception that leaves no trace becomes a habit. + +set -uo pipefail + +cd "$(git rev-parse --show-toplevel)" || exit 0 + +say() { printf '\033[1mpost-commit:\033[0m %s\n' "$*" >&2; } + +if [ -n "${SKIP_PUSH:-}" ]; then + say "SKIP_PUSH set — this commit was NOT pushed." + exit 0 +fi + +git_dir=$(git rev-parse --git-dir) + +# A rebase, cherry-pick or `git am` fires this hook once per replayed commit. +# Those commits are provisional by definition. +for marker in rebase-merge rebase-apply CHERRY_PICK_HEAD; do + if [ -e "$git_dir/$marker" ]; then + say "a rebase or cherry-pick is in progress — not pushing until it finishes." + exit 0 + fi +done + +branch=$(git symbolic-ref --quiet --short HEAD) || { + say "detached HEAD — no branch to push." + exit 0 +} + +if ! git remote get-url origin >/dev/null 2>&1; then + say "no 'origin' remote — nothing to push to." + exit 0 +fi + +say "pushing $branch to origin…" + +# --porcelain keeps the output to one parseable line per ref rather than the +# usual banner, and the timeout is here because an unreachable SSH host +# otherwise hangs the terminal well past the point of being useful. +if timeout 60 git push --porcelain origin "$branch"; then + exit 0 +fi + +status=$? + +if [ "$status" -eq 124 ]; then + say "push timed out after 60s. The commit is safe locally; push when the" + say " remote is reachable." +else + say "push was refused. The commit is safe locally and the branch is now ahead." + say " If this is a non-fast-forward, pull and reconcile — this hook will" + say " not force, and should not." +fi + +# Deliberately zero. git ignores it either way, and returning non-zero here reads +# as though the commit failed when it did not. +exit 0 diff --git a/docs/architecture/githooks/pre-commit b/docs/architecture/githooks/pre-commit new file mode 100755 index 0000000..fffc4bd --- /dev/null +++ b/docs/architecture/githooks/pre-commit @@ -0,0 +1,142 @@ +#!/usr/bin/env bash +# +# The repo's own guards, before a commit rather than after it. +# +# ## Why this exists in the repository and not in .git/hooks +# +# `.git/hooks` is not versioned, so a hook living there protects exactly one +# checkout and silently protects nothing anywhere else. This directory is +# committed, and `core.hooksPath` points at it: +# +# git config core.hooksPath .githooks +# +# That one line is the only setup, and it is in the README beside the test +# command. +# +# ## What it checks, and why in this order +# +# Typecheck first: it takes about two seconds and catches the class of mistake +# that is most annoying to discover later — a type change whose consumers were +# never visited. Then the suite, which is where the real guarantees live: the +# conformance tests that keep registries total, the sentinel tests that keep +# secrets out of logs, and the mutation-tested guards. +# +# Both are already the rule for finishing work here. A hook is just the version +# that does not depend on somebody remembering. +# +# ## It warns about unstaged changes rather than failing on them +# +# Both commands run against the **working tree**, not against the index. So a +# clean run proves the working tree is good, which is only the same thing as the +# commit being good when nothing is left unstaged. +# +# That distinction matters in this checkout specifically: it is edited by more +# than one person at a time, and commits are staged by explicit path. A green +# hook beside three unstaged files has verified something other than what is +# about to be committed, and the honest thing is to say so rather than imply a +# guarantee that was not made. +# +# ## Escape hatch +# +# SKIP_GUARDS=1 git commit ... skips both, loudly +# git commit --no-verify ... skips the hook entirely, silently +# +# The first is preferred: it leaves a line in the terminal saying the guards did +# not run, which is the difference between a deliberate exception and a habit. + +set -uo pipefail + +cd "$(git rev-parse --show-toplevel)" || exit 1 + +say() { printf '\033[1mpre-commit:\033[0m %s\n' "$*" >&2; } + +if [ -n "${SKIP_GUARDS:-}" ]; then + say "SKIP_GUARDS set — typecheck and tests did NOT run for this commit." + exit 0 +fi + +# Nothing staged is not this hook's problem; git will refuse on its own. +if git diff --cached --quiet; then + exit 0 +fi + +# Only worth running when source or tests changed. A commit that touches docs or +# migrations alone still gets the typecheck, because a migration can be +# referenced from a test, but it should not wait on the whole suite. +# +# The four files after the `|` are not source, and they are here because of what +# `tests/version.test.ts` guards: package.json, the Dockerfile, and the two +# image pins in README.md and docker-compose.example.yml must all name the same +# version. With `src|tests` alone, a commit that hand-edits only the README pin +# — precisely the drift that left those pins six versions stale — would get the +# typecheck and skip the one test that would have caught it. The guard has to +# run on the commits it exists to police. +staged=$(git diff --cached --name-only) +touches_code=$(printf '%s\n' "$staged" \ + | grep -cE '^(src|tests)/.*\.(ts|tsx)$|^(package\.json|Dockerfile|README\.md|docker-compose\.example\.yml)$' || true) + +say "typecheck…" + +if ! npx tsc --noEmit; then + say "typecheck failed — commit refused." + exit 1 +fi + +# Credentials, before the commit exists. +# +# First, and cheap: it reads the staged diff only. A secret caught here costs a +# `git reset`; the same secret caught after a push costs a rotation, because +# deleting the line does not remove it from a commit that already exists. +if [ -x scripts/secrets.sh ] || [ -f scripts/secrets.sh ]; then + if ! bash scripts/secrets.sh; then + say "possible credential in the staged changes — commit refused." + say "If it is real, rotate it. If it is not, --allow the path or adjust" + say "the patterns; do not silence the check." + exit 1 + fi +fi + +if [ "$touches_code" -gt 0 ]; then + say "tests…" + + # TEST_DATABASE_URL is picked up from the environment when it is set. Without + # one the database suites skip themselves rather than fail, which is the + # existing behaviour in tests/setup.ts — so this hook is useful on a machine + # with no Postgres and stricter on one with it. + # + # But "not set" and "not available" are different, and this used to treat them + # the same: every commit on this machine ran ~900 of 1,600 tests while a test + # database sat listening on 55432, because nothing looked. A commit hook + # should not *refuse* over a missing database — that belongs to the release + # gate, which now does — but it should use one that is there. + # Set these two together in the repository that adopts this hook — the port + # it listens on, and the URL that reaches it. Leaving them unset is fine and + # simply means the database suites skip, which is the right default on a + # machine with no Postgres. + TEST_DB_PORT="${TEST_DB_PORT:-}" + TEST_DB_URL="${TEST_DB_URL:-}" + + if [ -z "${TEST_DATABASE_URL:-}" ] && [ -n "$TEST_DB_PORT" ] && [ -n "$TEST_DB_URL" ] \ + && (exec 3<>/dev/tcp/127.0.0.1/"$TEST_DB_PORT") 2>/dev/null; then + exec 3<&- 2>/dev/null || true + export TEST_DATABASE_URL="$TEST_DB_URL" + say "using the local test database on ${TEST_DB_PORT} — the full suite will run." + elif [ -z "${TEST_DATABASE_URL:-}" ]; then + say "no test database; the database suites will skip. The release gate is stricter." + fi + + if ! npx vitest run --reporter=dot; then + say "tests failed — commit refused." + exit 1 + fi +else + say "no .ts/.tsx staged — skipping the suite." +fi + +# Said last so it is the thing still on screen when the editor opens. +if ! git diff --quiet; then + say "NOTE: unstaged changes are present. The guards ran against the working" + say " tree, so they did not verify this commit in isolation." +fi + +exit 0 diff --git a/docs/architecture/scripts/audit-gate.mjs b/docs/architecture/scripts/audit-gate.mjs new file mode 100644 index 0000000..d518be4 --- /dev/null +++ b/docs/architecture/scripts/audit-gate.mjs @@ -0,0 +1,187 @@ +#!/usr/bin/env node +/** + * Production dependency advisories, with an allowlist npm does not give you. + * + * node scripts/audit-gate.mjs # fail on unassessed high/critical + * node scripts/audit-gate.mjs --list # print what is found, exit 0 regardless + * + * 0 no unassessed high/critical advisories in production dependencies + * 1 unassessed advisories found — they are listed + * 2 NOTHING WAS CHECKED — audit did not run, or its output was not the shape + * this understands. Not a pass, and CI must not treat it as one. + * + * ## Why this exists rather than `npm audit --audit-level=high` + * + * npm audit has no allowlist. So when an advisory lands that provably cannot + * reach the application — a server-side CSRF in a router the app only uses + * declaratively, say — there are exactly two options in plain npm, and both are + * bad. Lower the threshold, which throws away a gate that has already caught + * real issues. Or let CI sit red, which for an advisory with no fixed version + * means indefinitely, and a permanently red gate is a gate everybody has + * learned to ignore. That is `GUARDS.md` rule 6 arriving through the back door. + * + * The third option is to say **which** advisories have been assessed and why, + * in a file somebody reviews. That is this. + * + * ## `--omit=dev` is the boundary, and it is a real one + * + * A build-time advisory in a test runner does not ship. The container holds + * production dependencies, so those are what this asks about. Widening it to + * dev pulls in advisories nothing can reach from outside and drowns the ones + * that can. + * + * ## Rules for adding an entry — all four, or it is not an assessment + * + * - **reason** — why it cannot reach THIS app, concretely, naming the file or + * the API that would have to be in use for it to matter. "Low risk" is not + * an assessment, it is an adjective. + * - **reachableIf** — what would make it reachable. This is the falsifiable + * half: without it, nobody can ever tell whether the waiver still holds. + * - **recheck** — the event that should retire the entry, not a date. + * - **Mirror it in the project's SECURITY document**, so an operator reading + * about the system's posture sees the same list a developer does. An + * exception that lives only in a script is one the security review misses. + * + * Delete an entry the moment a fix ships. An allowlist nobody prunes stops + * being a record of decisions and becomes a list of things nobody looks at. + * + * =========================================================================== + * TEMPLATE COPY — configure this before the first run + * =========================================================================== + * + * Copy to `scripts/audit-gate.mjs`, add `"audit": "node scripts/audit-gate.mjs"` + * to package.json, and — this is the step that matters — put it in whatever + * command the gate actually runs. A guard wired to nothing is documentation. + * + * Set SECURITY_DOC below to where this project keeps its security document, so + * the failure message names a path that exists. `docs/security/SECURITY.md` is + * the template's location; a repository that keeps it elsewhere and does not + * change this line prints a path a reader cannot open, which is exactly the + * class of stale doc-claim `doc-claims.sh` exists to catch. + * + * ALLOWED starts empty, deliberately. An allowlist inherited from another + * project waives advisories against an application that was never assessed — + * and it waives them silently, which is the worst available outcome. + */ + +// =========================================================================== +// CONFIGURATION +// =========================================================================== + +/** Where this project's security document lives, for the failure message. */ +const SECURITY_DOC = 'docs/security/SECURITY.md'; + +/** + * Assessed advisories, keyed by the package name npm reports. + * + * Empty on purpose — see the header. A worked example of the shape, from the + * project this came from: + * + * 'react-router': { + * reason: + * 'Advisory is RSC Mode CSRF (action execution before a 400). This app ' + + * 'is a Vite SPA using declarative (client/main.tsx) — ' + + * 'no RSC mode, no framework/data router, no createStaticHandler, no ' + + * '@react-router/* server package. No fixed version exists: the ' + + * 'vulnerable range extends past the newest published release, and the ' + + * 'only "fix" npm offers is a downgrade.', + * reachableIf: 'the app adopts RSC mode or a react-router server runtime', + * recheck: 'when react-router publishes a release above the vulnerable range', + * }, + * + * Note what makes that one an assessment rather than a shrug: it names the + * file, the four APIs that would have to be in use, and the release that + * retires it. + */ +const ALLOWED = {}; + +// =========================================================================== + +import { execFileSync } from 'node:child_process'; + +const listOnly = process.argv.includes('--list'); + +const die = (code, message) => { + console.error(`[audit-gate] ${message}`); + process.exit(code); +}; + +let raw; + +try { + raw = execFileSync('npm', ['audit', '--omit=dev', '--json'], { + encoding: 'utf8', + maxBuffer: 32 * 1024 * 1024, + }); +} catch (err) { + // npm audit exits non-zero whenever it finds anything at all, so a throw here + // is the normal path and the JSON is still on stdout. An empty stdout is the + // abnormal one — npm itself failed, and nothing was measured. + raw = err.stdout; + + if (!raw) { + die(2, `npm audit produced no output (${err.message}) — nothing was checked`); + } +} + +let report; + +try { + report = JSON.parse(raw); +} catch { + die(2, 'npm audit output was not JSON — nothing was checked'); +} + +// Distinguishing "no vulnerabilities" from "no vulnerabilities key" is the whole +// of GUARDS.md rule 5 applied to this script's own instrument. npm's audit JSON +// has changed shape between major versions before; a run against a shape this +// does not understand must not render as a clean bill of health. +if (typeof report.vulnerabilities !== 'object' || report.vulnerabilities === null) { + die(2, 'npm audit output had no `vulnerabilities` object — nothing was checked'); +} + +const blocking = []; +const waived = []; + +for (const [name, entry] of Object.entries(report.vulnerabilities)) { + if (entry.severity !== 'high' && entry.severity !== 'critical') continue; + + (ALLOWED[name] ? waived : blocking).push({ + name, + severity: entry.severity, + range: entry.range, + }); +} + +for (const { name, severity, range } of waived) { + console.log(`[audit-gate] WAIVED ${name} (${severity}, ${range}) — ${ALLOWED[name].reason}`); +} + +if (blocking.length && !listOnly) { + console.error(`\n[audit-gate] FAILED — ${blocking.length} unassessed high/critical advisory(ies):`); + + for (const { name, severity, range } of blocking) { + console.error(` ✗ ${name} (${severity}, ${range})`); + } + + console.error( + `\nFix it, or — only if it provably cannot reach this app — add an assessed entry\n` + + `to this script and mirror it in ${SECURITY_DOC}. All four fields, or it is\n` + + `not an assessment.`, + ); + + process.exit(1); +} + +if (blocking.length) { + console.log(`[audit-gate] --list: ${blocking.length} unassessed high/critical advisory(ies)`); + + for (const { name, severity, range } of blocking) { + console.log(` ✗ ${name} (${severity}, ${range})`); + } +} + +console.log( + `[audit-gate] OK — 0 unassessed high/critical advisories in production dependencies` + + (waived.length ? ` (${waived.length} assessed and waived)` : ''), +); diff --git a/docs/architecture/scripts/backup.sh b/docs/architecture/scripts/backup.sh new file mode 100755 index 0000000..18852ed --- /dev/null +++ b/docs/architecture/scripts/backup.sh @@ -0,0 +1,942 @@ +#!/usr/bin/env bash +# +# Write one database dump, prove the file is readable before trusting it, and +# keep the newest few. +# +# bash scripts/backup.sh # write one verified dump +# bash scripts/backup.sh --dry-run # show what it would write and delete +# bash scripts/backup.sh --no-prune # write, delete nothing +# BACKUP_KEEP=30 bash scripts/backup.sh # keep thirty instead of seven +# +# 0 3 * * * cd /srv/app && bash scripts/backup.sh # from cron, see below +# +# =========================================================================== +# TEMPLATE COPY — configure this before the first run +# =========================================================================== +# +# Copy to `scripts/backup.sh`, add `"backup": "bash scripts/backup.sh"` to +# package.json if this is a Node project, and set the two values in the +# CONFIGURATION block below. The script refuses to run until they are set: it +# has no defaults, deliberately. +# +# A BACKUP_DIR or BACKUP_NAME inherited from another project is not a cosmetic +# mistake. BACKUP_NAME is the filename prefix, and the filename prefix is what +# the retention rule globs on — so the wrong name writes this project's rows +# into that project's series and then deletes that project's oldest dumps to +# make room for them. Both halves are silent, the second is irreversible, and +# the directory afterwards looks exactly like a healthy backup series. Hence: +# no defaults, and a loud failure instead. +# +# Assumes: bash, coreutils, `awk`, `sed`, and the client tools for the engine +# (`pg_dump` and `pg_restore` for the default). No jq, no Node, nothing +# language-specific — unlike the release script beside it, this does not care +# what the project is written in. +# +# awk is named because it is not coreutils, a minimal container image can be +# without it, and it is what counts the tables — so it is checked at startup +# alongside the engine's tools. Left unchecked, its absence arrives as "the +# archive holds 0 tables", which is a sentence about the database and would send +# somebody looking at the wrong thing entirely. +# +# ## Why this exists +# +# A backup nobody has restored is a hypothesis, not a backup. +# +# The failure this is built around is not "the backup did not run". That one is +# loud: the directory is empty and somebody notices. The failure is a job that +# runs every night for a year and writes a file every night for a year, and the +# file is truncated, or is a dump of the wrong database, or is 400 bytes of +# `pg_dump: error:` because the password expired in March. Nothing about the +# directory listing distinguishes that from a working backup. The size column +# is plausible. The timestamps march forward. It is discovered on the one day +# it matters, by somebody who has already lost the database. +# +# So this script does not write a file and call it a backup. It writes the file +# under a temporary name, reads the file back with `pg_restore --list`, counts +# what is in it, refuses the whole run if the archive cannot be read, holds no +# tables, or CANNOT BE COUNTED, and only then renames it into place. +# +# The third of those is not padding. The counting step is one `awk` away from +# returning nothing, and a count that did not happen is not a count of zero: if +# the result were trusted unchecked, `[ "$TABLES" -gt 0 ]` would fail with +# "integer expression expected", `if` would read that failure as false, and an +# archive nobody had counted would be renamed into place and announced as +# verified — with retention then deleting real dumps to make room for it. +# +# ## What --list proves, and what it does not +# +# It proves the file is a real archive: the header parses, the table of +# contents is intact and complete, and the dump contains the tables you can see +# named in the output. Combined with `pg_dump` exiting zero, that eliminates +# the truncated file, the zero-byte file, the error message written where a +# dump should be, and the dump taken against an empty or wrong database. +# +# It does not decompress and check every data block, and it is not a restore. +# An archive can list cleanly and still fail to load — a broken large object, a +# circular constraint order, an extension that is not installed on the machine +# you are restoring onto. Nothing short of an actual restore finds those, which +# is why "the other half" below is not optional. +# +# ## Atomic, because the failure mode is a good backup destroyed by a bad one +# +# Every write goes to `.part` and is renamed only after it verifies. The +# rename is within one directory, so it is rename(2) and not a copy: the final +# name never exists holding half a file, and a reader — a sync job, a human, an +# offsite copy — cannot pick up a dump that is still being written. +# +# Writing directly to the final path would mean a dump that dies at 90% has +# already overwritten last night's, which was fine. That is worse than not +# running at all: it converts a working backup into a broken one and reports +# success while doing it. +# +# A dump that fails verification is KEPT, under its `.part` name, and the path +# is printed. It is evidence — usually the error message is inside it — and +# deleting evidence to keep the directory tidy is how the cause stays unknown. +# +# ## Retention deletes by explicit path, inside one directory, and nowhere else +# +# There is no `find -delete`, no `rm` with a glob, and no recursion. The +# candidate list is a glob of the exact filename shape this script writes, and +# every path is then checked one at a time: its parent directory, resolved with +# `pwd -P` rather than compared as text, must BE the backup directory; it must +# be a regular file and not a symlink; its name must match the pattern. Four +# checks, all of which have to pass, for each file, immediately before the `rm`. +# +# This is deliberately more than is needed for paths the script generated +# itself. The point is that no configuration value, no symlink planted in the +# directory, and no future edit to the glob can produce a deletion outside the +# backup directory — the guard does not trust the list it was handed. +# +# ## Credentials +# +# Connection details come from the environment and are never written here. +# PGHOST/PGPORT/PGUSER/PGDATABASE with a ~/.pgpass is the preferred form; +# DATABASE_URL works and is warned about, because a URL carries the password in +# it and reaches pg_dump as an argument, where `ps` shows it to every user on +# the host. +# +# Nothing prints a password or a connection URL — not to stdout, not to stderr, +# and above all not into a filename, where it would sit in the directory +# listing forever and be copied offsite with the dumps. Credentials in logs is +# the classic leak in backup scripts and it is usually introduced by an +# innocent-looking `say "dumping $DATABASE_URL"`. Do not add one. +# +# ## Changing the engine +# +# PostgreSQL is the supported default. Everything engine-specific is in the +# block marked ENGINE below — four functions and two variables — and nothing +# outside that block knows what a database is. Swapping in MySQL, SQLite or +# anything else is an edit to that block alone. +# +# One constraint carries over: `engine_verify` must READ THE FILE THAT WAS JUST +# WRITTEN. Re-querying the database, checking an exit code again, or trusting +# the file size verifies nothing about the artefact. Engines whose dump is +# plain SQL have no `--list` equivalent; for those the honest verification is a +# restore into a scratch database, and if that is too expensive to do on every +# run then the backup is unverified and the header of this script should be +# edited to stop claiming otherwise. +# +# ## What it deliberately does not do +# +# It does not copy the dump anywhere. A backup on the same disk as the database +# survives `DROP TABLE` and nothing else — not the disk, not the host, not the +# provider account. Getting these files onto different hardware is a separate, +# deliberate act (rsync, restic, object storage) for the same reason the +# release script does not deploy: two decisions, made one at a time. +# +# It does not schedule itself. BACKUP_KEEP is a count of files, not a period — +# seven of these is a week of nightly runs or seven hours of hourly ones. +# +# It does not restore. See below. +# +# ## The other half: a restore you have actually performed +# +# This script verifies the artefact. Only a restore verifies the backup, and a +# restore is also the only way to learn the number that matters during an +# incident, which is how long it takes. +# +# Do it on a schedule you write down — quarterly is a reasonable floor — into a +# scratch database, from the newest file this script produced, and record the +# duration and the command. Until that has happened once, the honest status of +# this directory is "dumps that appear to be readable", and the first restore +# will be attempted by somebody who has already lost the database. +# +# pg_restore --clean --if-exists --no-owner -d "$SCRATCH_URL" + +set -uo pipefail + +# Glob expansion and `sort` follow the collation locale, and a cron job's +# locale is not the one your shell has. C collation makes the ordering of the +# candidate list byte-order and therefore the same everywhere — the ordering +# decides which files retention calls "oldest", so it deciding differently +# under cron than under test would be discovered by deleting the wrong ones. +# LC_COLLATE only: LC_ALL=C would also change LC_CTYPE, and character handling +# is not something a backup script should be quietly redefining. +unset LC_ALL +export LC_COLLATE=C + +# An unmatched glob otherwise expands to the pattern itself, so an empty backup +# directory would produce one "file" literally named `myapp-[0-9][0-9]…` and +# hand it to the retention loop as a real candidate. +shopt -s nullglob + +say() { printf '\033[1mbackup:\033[0m %s\n' "$*" >&2; } +die() { printf '\033[1mbackup:\033[0m %s\n' "$*" >&2; exit 1; } + +# --------------------------------------------------------------------------- +# CONFIGURATION — set these two, then delete this banner. +# +# Both are empty on purpose. See the note at the top: an inherited BACKUP_NAME +# points the retention rule at another project's dumps, and an inherited +# BACKUP_DIR puts this project's rows where that project's offsite copy will +# collect them. +# --------------------------------------------------------------------------- + +# Absolute path to the directory holding the dumps. Must already exist, must be +# outside the repository, and should be readable only by the user running this +# — a dump is every row in the database in one file. +BACKUP_DIR="${BACKUP_DIR:-}" + +# Filename prefix identifying this project's series, e.g. `acme-orders`. +# Letters, digits, dot, underscore and hyphen only. This is what retention +# globs on; see the header for why it is not derived from the database name. +BACKUP_NAME="${BACKUP_NAME:-}" + +# --------------------------------------------------------------------------- +# Tunables. These have defaults because none of them names anything belonging +# to a particular project. +# --------------------------------------------------------------------------- + +# How many dumps survive, newest first. Seven is a week of nightly runs. +# +# Two is the floor enforced below. Keeping one means every run destroys the +# only copy you have in order to make a copy that has not yet been shown to be +# worth anything — and "worth something" here is more than readable, since a +# dump of an empty or wrong database verifies perfectly. Two means the newest +# dump can turn out to be worthless without that being the end of it. +KEEP="${BACKUP_KEEP:-7}" + +# Refuse a dump containing fewer tables than this. One is the default because +# an archive with no tables in it is not a backup of anything, and the way you +# get one is `pg_dump` connecting somewhere you did not intend — see the +# PGDATABASE note below, where the accident produces a valid, tiny, entirely +# empty archive. Set 0 only if this project's database genuinely has no tables, +# in which case there is nothing here to protect. +MIN_TABLES="${BACKUP_MIN_TABLES:-1}" + +# Warn when the new dump is smaller than this percentage of the previous one. +# A dump that halves overnight is usually a partial dump, a lost permission on +# some schema, or the wrong database — none of which fail loudly on their own. +# +# A warning and never a refusal: a legitimate large delete would otherwise +# block every backup from that moment on, which is a way of losing data by +# being careful about data. +SHRINK_PCT="${BACKUP_SHRINK_PCT:-50}" + +# =========================================================================== +# ENGINE — PostgreSQL. To use another engine, replace this block and nothing +# else. See "Changing the engine" in the header for the one rule that binds. +# =========================================================================== + +DUMP_SUFFIX=".dump" + +# The custom format is not a preference. `pg_restore --list` only reads custom, +# directory and tar archives; a plain SQL dump (`-Fp`, the pg_dump default) +# cannot be verified without restoring it, and this whole script is built on +# being able to verify. It also compresses by default, so nothing is piped +# through gzip — a pipeline would put a second program's exit status where +# pg_dump's needs to be. +engine_tools() { printf 'pg_dump pg_restore'; } + +engine_dump() { + local out="$1" + + # --file rather than a `>` redirect: with a redirect the SHELL creates the + # file, so a pg_dump that never starts (missing binary, bad PATH under cron) + # still leaves a zero-byte file behind, and the writer and the exit status + # stop being the same process. + # + # -w never prompts for a password. Without it, a run whose credentials have + # gone stale blocks on a terminal that cron does not have, and the backup + # does not fail — it hangs, forever, holding the lock, which reads as "no + # output, no error" for however long it takes somebody to look. + if [ -n "${DATABASE_URL:-}" ]; then + pg_dump -w --format=custom --file="$out" -d "$DATABASE_URL" + else + pg_dump -w --format=custom --file="$out" + fi +} + +# Reads the archive back and writes its table of contents to $2. Anything +# pg_restore says goes to $3, kept separately so it cannot be counted as +# archive content by the summariser below. +engine_verify() { + pg_restore --list "$1" >"$2" 2>"$3" +} + +# Prints "" from a TOC listing. +# +# TOC entries look like `216; 1259 16385 TABLE public users postgres`, and the +# data sections like `3021; 0 16385 TABLE DATA public users postgres`. Comment +# lines start with `;` and are excluded by the shape of field one. Counting +# both matters because a --data-only dump has no TABLE entries and a +# --schema-only dump has no TABLE DATA entries, and reporting zero for either +# without saying which kind of dump this is would be a lie by omission. +engine_summarise() { + awk '$1 ~ /^[0-9]+;$/ && $4 == "TABLE" { + if ($5 == "DATA") d++; else t++ + } + END { printf "%d %d\n", t + 0, d + 0 }' "$1" +} + +# =========================================================================== +# End of ENGINE block. +# =========================================================================== + +DRY_RUN="" +NO_PRUNE="" + +for arg in "$@"; do + case "$arg" in + --dry-run) DRY_RUN="yes" ;; + --no-prune) NO_PRUNE="yes" ;; + # No positional arguments exist, so one is a mistyped flag or a path + # somebody expected this to accept. Either way, acting on the rest of the + # command line as though it were fine is how a backup goes somewhere else. + *) die "unknown argument '$arg'. Usage: bash scripts/backup.sh [--dry-run] [--no-prune]" ;; + esac +done + +# --------------------------------------------------------------------------- +# Refuse to run half-configured. Everything checkable without a side effect, +# checked before the first side effect, and named one at a time so the message +# says which value is wrong rather than "configuration error". +# --------------------------------------------------------------------------- + +[ -n "$BACKUP_DIR" ] || die "set BACKUP_DIR — the directory to write dumps into. See the CONFIGURATION block." +[ -n "$BACKUP_NAME" ] || die "set BACKUP_NAME — the filename prefix for this project's dumps. See the CONFIGURATION block." + +# A relative path resolves against the working directory, and cron's working +# directory is not yours. The dumps would land somewhere else, and retention — +# globbing that same relative path — would find that other directory empty, +# report "nothing to remove", and let the real series grow until the disk +# filled. Both halves look healthy in isolation. +case "$BACKUP_DIR" in + /*) : ;; + *) die "BACKUP_DIR ('$BACKUP_DIR') must be an absolute path — a relative one + points somewhere different under cron than it does in your shell." ;; +esac + +# The glob and the delete guard are built from this. A name containing `*`, `?` +# or `/` would widen the candidate list past this project's dumps, which is the +# one thing the guard cannot make safe by checking paths. +case "$BACKUP_NAME" in + *[!A-Za-z0-9._-]*) die "BACKUP_NAME ('$BACKUP_NAME') may contain only letters, digits, dot, + underscore and hyphen — it is used as a filename glob." ;; +esac + +case "$KEEP" in + ''|*[!0-9]*) die "BACKUP_KEEP must be a whole number, got '$KEEP'." ;; +esac + +# Refused rather than clamped: a caller who typed 1 meant something, and it was +# not "destroy the only copy before checking the new one is worth having". +[ "$KEEP" -ge 2 ] || die "BACKUP_KEEP must be at least 2 — see the note above KEEP in this script." + +case "$MIN_TABLES" in + ''|*[!0-9]*) die "BACKUP_MIN_TABLES must be a whole number, got '$MIN_TABLES'." ;; +esac + +case "$SHRINK_PCT" in + ''|*[!0-9]*) die "BACKUP_SHRINK_PCT must be a whole number, got '$SHRINK_PCT'." ;; +esac + +for tool in $(engine_tools); do + command -v "$tool" >/dev/null 2>&1 \ + || die "$tool is not on PATH. Install the client tools for this engine + (Debian/Ubuntu: postgresql-client) and run this again." +done + +# awk is checked separately because it is not part of the engine and not part of +# coreutils: it is what turns the table of contents into a number, and cron's +# PATH is not your shell's. Without this check its absence surfaces further down +# as "the archive holds 0 table(s)" — a measurement, about the database, that +# nothing measured. +command -v awk >/dev/null 2>&1 \ + || die "awk is not on PATH. It is what counts what is inside the archive, and + without it this script cannot tell a good dump from an empty one. + Under cron, PATH is not the PATH your shell has." + +[ -d "$BACKUP_DIR" ] || die "BACKUP_DIR ('$BACKUP_DIR') does not exist. + Create it deliberately: mkdir -p -m 700 '$BACKUP_DIR' + It is not created here on purpose — a typo in the path is otherwise + indistinguishable from a first run, and the typo'd directory would fill + with a complete, correct-looking series that nobody restores from." + +# Resolved once, here, and used for every path comparison afterwards. Comparing +# the configured string instead would let `/srv/backups/../../etc` pass a +# prefix test while being nowhere near the backup directory. +BACKUP_DIR_REAL=$(cd "$BACKUP_DIR" && pwd -P) || die "cannot resolve BACKUP_DIR ('$BACKUP_DIR')." + +# Ahead of the writability check on purpose: for an unprivileged user / is not +# writable, so leaving this until later answers "permission denied" to somebody +# who is one `sudo` away from asking the same question again and being told yes. +[ "$BACKUP_DIR_REAL" != "/" ] || die "BACKUP_DIR resolves to / — retention would treat the root of the + filesystem as a directory of expendable files." + +[ -w "$BACKUP_DIR" ] || die "BACKUP_DIR ('$BACKUP_DIR') is not writable by $(id -un)." + +# A dump inside the checkout is one `git add -A` from being committed, and one +# push from being public, with every row in the database in it. It is also +# copied by every clone and every deploy from then on. The engine that made it +# does not put it back once that happens. +repo_root=$(git rev-parse --show-toplevel 2>/dev/null) || repo_root="" + +if [ -n "$repo_root" ] && repo_real=$(cd "$repo_root" && pwd -P); then + case "$BACKUP_DIR_REAL/" in + "$repo_real"/*) + die "BACKUP_DIR ('$BACKUP_DIR_REAL') is inside the repository at $repo_real. + Put it outside the checkout — a dump in a working tree is one 'git add -A' + away from being committed and pushed." ;; + esac +fi + +# Permissions are reported, not corrected. Changing the mode of a directory the +# operator created — which may be a mount, or shared with a sync agent — is a +# decision this script does not get to make on their behalf; failing to mention +# that every row in the database is world-readable would be worse. +dir_mode=$(ls -ld "$BACKUP_DIR_REAL" 2>/dev/null | cut -c5-10) +case "$dir_mode" in + ------) : ;; + '') say "note: could not read the permissions of $BACKUP_DIR_REAL." ;; + *) say "WARNING: $BACKUP_DIR_REAL is accessible to group or other (mode bits" + say " '$dir_mode'). A dump is every row in the database in one file." + say " chmod 700 it unless something else is meant to read these." ;; +esac + +# --------------------------------------------------------------------------- +# Connection. From the environment, never from this file. +# --------------------------------------------------------------------------- + +# With neither of these set, pg_dump does not fail — it connects to a database +# named after the current user. On most hosts that database exists, is empty, +# and dumps in half a second into a perfectly valid archive. The MIN_TABLES +# check below is the second net under this; refusing here is the first. +if [ -n "${DATABASE_URL:-}" ]; then + # Describes the SOURCE of the connection, never its contents. This string is + # printed; the URL must not be. + TARGET_DESC="DATABASE_URL (contents not shown)" + + say "WARNING: using DATABASE_URL. It reaches pg_dump as a command-line" + say " argument, so the password in it is visible to anyone who can" + say " run 'ps' on this host. PGHOST/PGUSER/PGDATABASE with a" + say " ~/.pgpass keeps the password out of the process table." +elif [ -n "${PGDATABASE:-}" ]; then + # PGDATABASE and PGHOST are not secrets and naming them is what makes the + # log worth reading. PGPASSWORD is never touched by anything here. + TARGET_DESC="${PGDATABASE}${PGHOST:+ on ${PGHOST}}" +else + die "no database configured: set PGDATABASE (with PGHOST/PGUSER as needed, and + a ~/.pgpass for the password) or DATABASE_URL in the environment. + Without one, pg_dump connects to a database named after the current user + and produces a valid, empty archive." +fi + +# --------------------------------------------------------------------------- +# Names. Built here so the dry run and the real run cannot describe different +# files. +# +# UTC, always: local time repeats an hour every autumn, which puts two dumps +# out of order by name in the one direction that matters — retention reads that +# order to decide what is oldest. +# +# Nothing derived from the connection goes into the filename. A URL in a name +# sits in the directory listing forever and is copied offsite with the dumps. +# --------------------------------------------------------------------------- + +STAMP_GLOB='[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]T[0-9][0-9][0-9][0-9][0-9][0-9]Z' +STAMP=$(date -u +%Y%m%dT%H%M%SZ) || die "cannot read the clock." + +FINAL="${BACKUP_DIR_REAL}/${BACKUP_NAME}-${STAMP}${DUMP_SUFFIX}" +PART="${FINAL}.part" + +# Only a second run inside the same second reaches either of these, and the +# alternative in both cases is destroying a file this script promised to keep: +# a dump that has already been verified, or the .part evidence of a run that +# failed a moment ago and is about to be looked at. +[ ! -e "$FINAL" ] || die "$FINAL already exists. Refusing to overwrite a verified dump." +[ ! -e "$PART" ] || die "$PART already exists — it is the unverified remains of a run that + failed within this same second. Inspect or remove it before running again." + +human_size() { + local b="${1:-}" + + # An unmeasurable file is reported as unmeasured. Printing "0 B" for a file + # whose size could not be read is the same lie as printing "nothing to + # delete" for a directory that could not be listed. + case "$b" in + ''|*[!0-9]*) printf 'unknown size'; return 0 ;; + esac + + # Integer arithmetic rather than `du -h`, whose output format and rounding + # differ between GNU and BSD and which reports blocks allocated rather than + # bytes written. + if [ "$b" -lt 1024 ]; then printf '%s B' "$b" + elif [ "$b" -lt 1048576 ]; then printf '%s.%s KiB' "$((b / 1024))" "$((b * 10 / 1024 % 10))" + elif [ "$b" -lt 1073741824 ]; then printf '%s.%s MiB' "$((b / 1048576))" "$((b * 10 / 1048576 % 10))" + else printf '%s.%s GiB' "$((b / 1073741824))" "$((b * 10 / 1073741824 % 10))" + fi +} + +file_bytes() { + # `wc -c` is portable where `stat` is not: GNU wants -c%s and BSD wants -f%z. + # BSD pads the number with spaces, hence the tr. + # + # 2>/dev/null comes BEFORE the input redirect on purpose. Redirections are + # applied left to right, so with the other order the shell's own "no such + # file" message for a failed open is written before stderr has been silenced — + # a raw `backup.sh: line N: …` in the middle of otherwise formatted output. + # Callers read the empty result, which is what "unmeasurable" means here. + wc -c 2>/dev/null <"$1" | tr -d ' \n' +} + +# Every existing dump of this series, oldest first. Pathname expansion sorts, +# and LC_COLLATE=C above makes that sort byte-order, so no external sort is +# involved and no filename has to survive a round trip through word splitting. +# +# The glob is the exact shape this script writes — prefix, UTC stamp, suffix. +# Files that merely live in the directory are not candidates for anything: a +# `.part` from a failed run, a dump somebody copied in by hand, and another +# project's series all fail to match and are never counted or deleted. +list_dumps() { + printf '%s\n' "${BACKUP_DIR_REAL}/${BACKUP_NAME}-"$STAMP_GLOB"$DUMP_SUFFIX" +} + +# --------------------------------------------------------------------------- +# The delete guard. +# +# Called immediately before every `rm`, on the path about to be removed, and +# every check has to pass. It exists so that no configuration value, no symlink +# planted in the backup directory and no later edit to the glob can produce a +# deletion outside BACKUP_DIR_REAL — it does not trust the list it was given. +# --------------------------------------------------------------------------- +deletable() { + local path="$1" parent base + + case "$path" in + */*) parent="${path%/*}"; base="${path##*/}" ;; + *) return 1 ;; + esac + + # Resolved, not string-compared: `.` and `..` components make a text prefix + # test agree with a path that is somewhere else entirely. + parent=$(cd "$parent" 2>/dev/null && pwd -P) || return 1 + [ "$parent" = "$BACKUP_DIR_REAL" ] || return 1 + + # -f follows symlinks, so a link named like a dump would pass every other + # check while pointing at a file the checks were never made about. + [ -f "$path" ] || return 1 + [ ! -L "$path" ] || return 1 + + case "$base" in + "$BACKUP_NAME"-$STAMP_GLOB"$DUMP_SUFFIX") return 0 ;; + *) return 1 ;; + esac +} + +# --------------------------------------------------------------------------- +# Retention. +# +# Last, and never fatal: by the time this runs the dump has been written and +# verified, and housekeeping must not be able to report that a backup which +# succeeded failed. Everything it declines to do, it says out loud instead. +# --------------------------------------------------------------------------- +prune() { + [ -z "$NO_PRUNE" ] || { say "retention: skipped (--no-prune)."; return 0; } + + if [ ! -r "$BACKUP_DIR_REAL" ] || [ ! -x "$BACKUP_DIR_REAL" ]; then + say "retention: SKIPPED — cannot list $BACKUP_DIR_REAL." + say " That is 'I could not look', which is not the same answer as" + say " 'there is nothing to delete'. Nothing was removed." + return 0 + fi + + local -a candidates=() + local line + while IFS= read -r line; do + [ -n "$line" ] && candidates+=("$line") + done </dev/null | awk 'NR == 2 { print $4 }') + +case "$free_kb" in + ''|*[!0-9]*) + say "note: free space on $BACKUP_DIR_REAL could not be read — not checked." ;; + *) + if [ -z "$PREV_MEASURED" ] || [ "$PREV_BYTES" -eq 0 ]; then + say "note: $(human_size $((free_kb * 1024))) free; no measured previous dump to compare" + say " against, so whether that is enough is unknown." + elif [ $((free_kb * 1024)) -lt $((PREV_BYTES * 2)) ]; then + say "WARNING: $(human_size $((free_kb * 1024))) free, and the last dump was" + say " $(human_size "$PREV_BYTES"). This run may not fit." + fi ;; +esac + +# --------------------------------------------------------------------------- +# --dry-run stops here, having touched nothing. +# --------------------------------------------------------------------------- + +if [ -n "$DRY_RUN" ]; then + say "--dry-run: nothing was written or deleted. It would have:" + say " dumped ${TARGET_DESC}" + say " written ${PART}" + say " verified it with: pg_restore --list ${PART}" + say " renamed it to ${FINAL}" + say "" + say "and then applied retention, shown here for real because it deletes:" + prune + report_parts + exit 0 +fi + +# --------------------------------------------------------------------------- +# One run at a time. +# +# mkdir is atomic on every filesystem this will meet, which flock and lock +# files are not. Two overlapping runs — a nightly job on a database that now +# takes longer than a day to dump — double the load on the server and race each +# other's retention pass. +# +# A stale lock stops backups, so this exits NON-ZERO and names the fix rather +# than skipping quietly. A skipped backup that reports success is the exact +# failure this whole script exists to prevent, and a lock is not allowed to +# reintroduce it. +# --------------------------------------------------------------------------- + +LOCK_DIR="${BACKUP_DIR_REAL}/.${BACKUP_NAME}.lock" +LOCK_HELD="" + +# mkdir's error is captured rather than discarded, because "the lock exists" and +# "the lock could not be created" are different failures with different fixes, +# and only the first one is about concurrency. A read-only mount, a full +# filesystem, or a missing mkdir under cron's PATH all make mkdir fail too; the +# directory is what distinguishes them, so it is looked at instead of assumed. +LOCK_ERR="" + +if ! LOCK_ERR=$(mkdir "$LOCK_DIR" 2>&1); then + if [ -d "$LOCK_DIR" ]; then + say "another run holds the lock at $LOCK_DIR" + [ ! -r "$LOCK_DIR/owner" ] || say " owner: $(cat "$LOCK_DIR/owner" 2>/dev/null)" + die "if no backup is running, that lock is stale: rmdir '$LOCK_DIR'" + fi + + # No lock directory, so nothing is holding anything. Saying otherwise here + # would send somebody to rmdir a path that does not exist, watch that change + # nothing, and go looking for a second backup process — while the real cause + # sits unread in mkdir's own message and every night's backup stays dead. + say "could not create the lock directory $LOCK_DIR, and no lock is present —" + say " so this is NOT another run. The directory could not be made at all." + [ -z "$LOCK_ERR" ] || say " mkdir said: $LOCK_ERR" + die "check that $BACKUP_DIR_REAL is on a writable, non-full filesystem." +fi + +LOCK_HELD="yes" +WORK="" + +# Armed in the same breath as the lock, not after the next command. Anything +# that exits between acquiring the lock and installing this trap leaves a stale +# lock behind, and a stale lock stops every future run — the trap tolerating an +# empty WORK is much cheaper than that. $WORK is quoted-empty-safe because rm +# is guarded on it. +trap '[ -z "$LOCK_HELD" ] || rm -rf "$LOCK_DIR"; [ -z "$WORK" ] || rm -rf "$WORK"' EXIT + +printf 'pid %s on %s since %s\n' "$$" "$(hostname 2>/dev/null || echo unknown)" \ + "$(date -u +%Y-%m-%dT%H:%M:%SZ)" >"$LOCK_DIR/owner" 2>/dev/null + +# The .part is deliberately NOT cleaned up by that trap. It is evidence when +# verification fails, and the failure paths below print where it is. +WORK=$(mktemp -d) || die "cannot create a temporary directory" + +# --------------------------------------------------------------------------- +# Dump. +# --------------------------------------------------------------------------- + +say "dumping ${TARGET_DESC} -> ${PART}" + +# umask in a subshell so the archive is created 0600 from its first byte. Doing +# it with a chmod afterwards leaves a window in which every row in the database +# is readable by every user on the host, and that window is the whole dump. +if ! ( umask 077; engine_dump "$PART" ); then + say "the dump failed (pg_dump's own error is above). Nothing was renamed into" + say " place and nothing was deleted." + if [ -n "$PREV" ]; then + say " The newest verified dump is still $PREV" + else + say " There is no previously verified dump in $BACKUP_DIR_REAL." + fi + say " The partial file is kept for inspection: $PART" + exit 1 +fi + +[ -f "$PART" ] || die "pg_dump reported success but $PART does not exist. Nothing was renamed." + +BYTES=$(file_bytes "$PART") +[ -n "$BYTES" ] || die "cannot measure $PART. It has NOT been renamed into place." + +# --------------------------------------------------------------------------- +# Verify, before the file is allowed to become the backup. +# --------------------------------------------------------------------------- + +say "verifying ${PART}…" + +LISTING="$WORK/toc" +VERR="$WORK/toc.err" + +if ! engine_verify "$PART" "$LISTING" "$VERR"; then + say "VERIFICATION FAILED — the file cannot be read back as an archive." + say " $(human_size "$BYTES") written. The reader said:" + sed -n '1,10p' "$VERR" >&2 2>/dev/null + say "" + say " It has NOT been renamed into place and NOTHING was deleted." + say " Kept for inspection: $PART" + if [ -n "$PREV" ]; then + say " The newest verified dump is still $PREV" + else + say " There is no previously verified dump in $BACKUP_DIR_REAL." + fi + exit 1 +fi + +SUMMARY=$(engine_summarise "$LISTING") || SUMMARY="" + +read -r TABLES DATA_SECTIONS </dev/null 2>&1; then + printf '\033[1mcheck-env:\033[0m this bash has no `compgen`, so the environment cannot be\n' >&2 + printf '\033[1mcheck-env:\033[0m read reliably and nothing was checked. That is not a pass.\n' >&2 + exit 2 +fi + +_CE_ENV_NAMES=() +_CE_ENV_VALS=() +while IFS= read -r _CE_N; do + [ -n "$_CE_N" ] || continue + # `export FOO` with no value is exported-but-unset: it is not in a child's + # environment either, so it is absence rather than an empty value. `+x` + # keeps set-to-empty, which `docker run -e FOO` produces and which is a + # different bug from unset. + [ -n "${!_CE_N+x}" ] || continue + _CE_ENV_NAMES+=("$_CE_N") + _CE_ENV_VALS+=("${!_CE_N}") +done <<<"$(compgen -e)" +unset _CE_N + +# --------------------------------------------------------------------------- +# SPEC — the variables this project needs. One line each, then delete this +# banner. Adding a variable later is a one-line edit and nothing else. +# +# "NAME|required|kind|what breaks when it is missing or wrong" +# +# The description is not documentation. It is the sentence printed next to the +# failure at 03:00, so write the CONSEQUENCE ("sessions cannot be signed, every +# login 500s") rather than a restatement of the name ("the JWT secret"). +# +# required | optional — spelled exactly. A typo is refused rather than read as +# "optional", because silently downgrading a required +# variable is the one mistake this script cannot survive. +# +# Kinds: +# nonempty any non-empty string; for values whose shape is +# genuinely unknowable. Not a lazy default — it is the +# honest one, and it still catches unset and empty. +# int a whole number +# int:MIN..MAX a whole number within bounds, inclusive +# port 1-65535 +# url scheme://host[...]; rejects a bare `host:5432` +# path a filesystem path; existence is reported only when the +# source is this machine's own environment +# one-of:a,b,c exactly one of these, compared case-sensitively +# secret-min-length:N at least N characters; the value is never printed, and +# it is checked against well-known placeholders +# +# Examples of the FORM. Delete them — they are shapes, not your variables: +# +# "DATABASE_URL|required|url|nothing can read or write; every request 500s" +# "PORT|optional|port|the server binds its own default, which the proxy is not pointed at" +# "NODE_ENV|required|one-of:development,production,test|the wrong branch of every environment check" +# "SESSION_SECRET|required|secret-min-length:32|sessions are forgeable" +# "UPLOAD_DIR|optional|path|uploads land somewhere nothing serves and nothing backs up" +# --------------------------------------------------------------------------- + +SPEC=( +) + +# --------------------------------------------------------------------------- +# Redaction. See "Secrets are measured, never printed" above. +# +# Matched case-insensitively against the whole variable name, so KEY catches +# API_KEY, KEYCLOAK_SECRET and MONKEY_HOST alike. The last is a false positive +# and costs nothing: its value is described rather than shown. +# --------------------------------------------------------------------------- +SECRET_NAME_PATTERN='SECRET|TOKEN|PASSWORD|PASSWD|PWD|CREDENTIAL|PRIVATE|SALT|SIGNATURE|SIGNING|AUTH|KEY|DSN|COOKIE' + +# Values a `secret-min-length` variable must not be, compared in lower case and +# never echoed. These are the strings typed to make the dev server start, which +# then travel to production inside a copied .env and satisfy every length rule. +PLACEHOLDER_SECRETS='changeme change-me change_me changethis secret mysecret supersecret password passwd hunter2 test testing example placeholder todo tbd xxx xxxx admin dev devsecret development your-secret-here your_secret_here notasecret 123456 12345678 abc123' + +# Redaction applies to NAMES read out of a file too, not only to values. +# +# A multi-line secret pasted into a .env — a PEM body, a wrapped base64 blob — +# has continuation lines carrying '=' padding, so `KEY=VALUE` parsing splits one +# into a "name" and a "value" and the name is a slice of the credential. Every +# message that prints a name out of a file therefore prints its LENGTH once it +# is longer than any name could plausibly be, which is the same rule values +# already follow. 40 covers the longest real variable names (the framework +# ones reach the low forties); a 64-column PEM line does not come close. +NAME_PRINT_MAX=40 + +say() { printf '\033[1mcheck-env:\033[0m %s\n' "$*" >&2; } + +# die is 2, not the 1 its counterpart in release.sh uses: everything that calls +# it is a reason this script could not measure the environment, and a caller +# must be able to tell "your configuration is wrong" from "I never found out". +die() { printf '\033[1mcheck-env:\033[0m %s\n' "$*" >&2; exit 2; } + +usage() { + say "usage: check-env.sh [--file PATH] [--quiet] [--list] [--help]" + say " --file PATH check that file instead of the process environment" + say " --quiet, -q print only failures (exit code still says everything)" + say " --list print the declared spec and check nothing" + say " --help, -h this message" +} + +trim() { + local s="$1" + s="${s#"${s%%[![:space:]]*}"}" + s="${s%"${s##*[![:space:]]}"}" + printf '%s' "$s" +} + +SOURCE_FILE="" +QUIET="" +LIST_ONLY="" + +while [ $# -gt 0 ]; do + case "$1" in + --file) shift; [ $# -gt 0 ] || die "--file needs a path."; SOURCE_FILE="$1" ;; + --file=*) SOURCE_FILE="${1#--file=}"; [ -n "$SOURCE_FILE" ] || die "--file needs a path." ;; + --quiet|-q) QUIET="yes" ;; + --list) LIST_ONLY="yes" ;; + -h|--help) usage; exit 0 ;; + *) usage; die "unknown argument '$1'." ;; + esac + shift +done + +# --------------------------------------------------------------------------- +# Read the spec, and refuse anything ambiguous. +# +# A malformed spec entry is a configuration error in this script, not a finding +# about the environment, so it exits 2 and reports nothing about the variables: +# a partial report from a spec that is half-understood is the same lie as a +# green one from an empty spec. +# --------------------------------------------------------------------------- + +if [ "${#SPEC[@]}" -eq 0 ]; then + say "SPEC is empty, so nothing was checked. That is not a pass." + say "" + say "This is a fresh template copy: declare the variables this project reads" + say "in the SPEC block near the top of this file, one line each —" + say ' "DATABASE_URL|required|url|nothing can read or write; every request 500s"' + die "then run this again." +fi + +SPEC_NAME=() +SPEC_REQ=() +SPEC_KIND=() +SPEC_PARAM=() +SPEC_DESC=() +NAME_WIDTH=4 + +entry_no=0 +for entry in "${SPEC[@]}"; do + entry_no=$((entry_no + 1)) + + # A description may legitimately contain '|', so read takes only four fields + # and the fourth keeps the rest. + IFS='|' read -r f_name f_req f_kind f_desc <<<"$entry" + + f_name=$(trim "${f_name:-}") + f_req=$(trim "${f_req:-}") + f_kind=$(trim "${f_kind:-}") + f_desc=$(trim "${f_desc:-}") + + [ -n "$f_name" ] && [ -n "$f_req" ] && [ -n "$f_kind" ] && [ -n "$f_desc" ] \ + || die "SPEC entry ${entry_no} is not NAME|required|kind|description: '${entry}'" + + # The shell cannot export a name it cannot parse, so a name that fails this + # could never have been set in the first place and every run would report it + # missing forever. + [[ "$f_name" =~ ^[A-Za-z_][A-Za-z0-9_]*$ ]] \ + || die "SPEC entry ${entry_no}: '${f_name}' is not a usable variable name." + + # The environment snapshot has to live somewhere, and a variable it could + # collide with is one this script would measure against itself. Refused + # rather than measured wrong. + case "$f_name" in + _CE_*) die "SPEC entry ${entry_no}: '${f_name}' uses the _CE_ prefix, which this script reserves for the environment snapshot it takes before it defines anything. Rename the variable, or rename the snapshot at the top of this file." ;; + esac + + case "$f_req" in + required|optional) : ;; + *) die "SPEC entry ${entry_no} (${f_name}): expected 'required' or 'optional', got '${f_req}'. Spelling matters here — see the SPEC block." ;; + esac + + kind_name="${f_kind%%:*}" + kind_param="" + case "$f_kind" in *:*) kind_param="${f_kind#*:}" ;; esac + + case "$kind_name" in + nonempty|url|port|path) + [ -z "$kind_param" ] || die "SPEC entry ${entry_no} (${f_name}): kind '${kind_name}' takes no ':' argument." ;; + int) + if [ -n "$kind_param" ]; then + [[ "$kind_param" =~ ^[+-]?[0-9]+\.\.[+-]?[0-9]+$ ]] \ + || die "SPEC entry ${entry_no} (${f_name}): int bounds must look like 1..100, got '${kind_param}'." + fi ;; + one-of) + [ -n "$kind_param" ] || die "SPEC entry ${entry_no} (${f_name}): one-of needs a comma-separated list, e.g. one-of:development,production." ;; + secret-min-length) + [[ "$kind_param" =~ ^[0-9]+$ ]] && [ "$kind_param" -gt 0 ] \ + || die "SPEC entry ${entry_no} (${f_name}): secret-min-length needs a positive number, e.g. secret-min-length:32." ;; + *) + die "SPEC entry ${entry_no} (${f_name}): unknown kind '${kind_name}'. Valid: nonempty, int, port, url, path, one-of:a,b, secret-min-length:N." ;; + esac + + # A name declared twice gets two descriptions, and the report would print + # both — one of which is now out of date and neither of which is marked. + for existing in ${SPEC_NAME[@]+"${SPEC_NAME[@]}"}; do + [ "$existing" = "$f_name" ] && die "SPEC declares ${f_name} twice. Keep one line." + done + + SPEC_NAME+=("$f_name") + SPEC_REQ+=("$f_req") + SPEC_KIND+=("$kind_name") + SPEC_PARAM+=("$kind_param") + SPEC_DESC+=("$f_desc") + + [ "${#f_name}" -gt "$NAME_WIDTH" ] && NAME_WIDTH="${#f_name}" +done + +if [ -n "$LIST_ONLY" ]; then + say "declared in SPEC — nothing was checked:" + for (( i = 0; i < ${#SPEC_NAME[@]}; i++ )); do + kind="${SPEC_KIND[$i]}" + [ -n "${SPEC_PARAM[$i]}" ] && kind="${kind}:${SPEC_PARAM[$i]}" + printf ' %-*s %-8s %-22s %s\n' \ + "$NAME_WIDTH" "${SPEC_NAME[$i]}" "${SPEC_REQ[$i]}" "$kind" "${SPEC_DESC[$i]}" >&2 + done + exit 0 +fi + +# --------------------------------------------------------------------------- +# Display. Every message that mentions a value goes through show(), and that is +# the only place the redaction rule is enforced — so nothing else in this +# script may interpolate a value into a message. If you add a check, use it. +# --------------------------------------------------------------------------- + +is_secret() { + case "$2" in secret-min-length) return 0 ;; esac + local upper + upper=$(printf '%s' "$1" | tr '[:lower:]' '[:upper:]') + [[ "$upper" =~ $SECRET_NAME_PATTERN ]] +} + +show() { + local name="$1" kind="$2" value="$3" + + if is_secret "$name" "$kind"; then + printf 'set, %d characters (value withheld)' "${#value}" + return 0 + fi + + case "$kind" in + url) + # Scheme and host are what you need to spot the wrong environment; the + # path and query are where webhook and signed-URL secrets live, so they + # are summarised rather than shown. + local rest authority tail="" + case "$value" in + *://*) + rest="${value#*://}" + authority="${rest%%/*}" + authority="${authority%%\?*}" + case "$authority" in *@*) authority="***@${authority##*@}" ;; esac + case "$rest" in */*|*\?*) tail="/…" ;; esac + printf '%s://%s%s' "${value%%://*}" "$authority" "$tail" + return 0 ;; + esac + ;; + # Bounded, enumerable and not credential-shaped by construction. `literal` + # is the mode a failure message asks for when the value IS the finding — + # a port of '3000/tcp' cannot be explained without quoting it. + int|port|one-of|path|literal) : ;; + *) + # Unknown shape, so nothing is known about what might be inside it. + printf 'set, %d characters' "${#value}" + return 0 ;; + esac + + if [ "${#value}" -gt 60 ]; then + printf '%s…' "${value:0:59}" + else + printf '%s' "$value" + fi +} + +# --------------------------------------------------------------------------- +# Load the source. +# --------------------------------------------------------------------------- + +FILE_KEYS=() +FILE_VALS=() +FILE_LINES=() +FILE_BAD=() +FILE_NOTES=() + +load_file() { + local path="$1" lineno=0 line trimmed key val + + # Redirection, not `cat "$path" |`: a piped while-loop runs in a subshell and + # every array appended inside it is discarded at the `done`. + while IFS= read -r line || [ -n "$line" ]; do + lineno=$((lineno + 1)) + + case "$line" in + *$'\r') + line="${line%$'\r'}" + FILE_NOTES+=("line ${lineno}|has a CRLF line ending; most loaders keep the carriage return, making the value one invisible character longer than it looks") ;; + esac + + trimmed=$(trim "$line") + case "$trimmed" in ''|'#'*) continue ;; esac + + case "$trimmed" in + export\ *|export$'\t'*) + trimmed=$(trim "${trimmed#export}") + FILE_NOTES+=("line ${lineno}|uses 'export'; a shell sourcing this file is fine, but docker --env-file reads the name as 'export NAME'") ;; + esac + + case "$trimmed" in + *=*) key="${trimmed%%=*}"; val="${trimmed#*=}" ;; + *) FILE_BAD+=("line ${lineno}|no '=', so it sets nothing at all"); continue ;; + esac + + if [[ ! "$key" =~ ^[A-Za-z_][A-Za-z0-9_]*$ ]]; then + if [[ "$(trim "$key")" =~ ^[A-Za-z_][A-Za-z0-9_]*$ ]]; then + FILE_BAD+=("line ${lineno}|has spaces around '='; outside a shell those become part of the name and of the value") + elif [ "${#key}" -gt "$NAME_PRINT_MAX" ]; then + FILE_BAD+=("line ${lineno}|has ${#key} characters before its '=', so that is not a name and nothing can read it; the text is withheld in case this is one line of a wrapped multi-line secret") + else + FILE_BAD+=("line ${lineno}|'${key}' is not a usable variable name, so nothing can read it") + fi + continue + fi + + # Quotes are stripped here because that is what dotenv-style loaders do — + # but docker --env-file does not, so the two see different values. Said out + # loud rather than resolved, because only the operator knows which loader + # will read this file. + case "$val" in + \"*\"|\'*\') + val="${val:1:${#val}-2}" + FILE_NOTES+=("${key}|is quoted; dotenv-style loaders strip the quotes and docker --env-file keeps them, so the two disagree about this value") ;; + esac + + case "$val" in + *'${'*|*'$('*) + FILE_NOTES+=("${key}|contains a \$ expansion; dotenv-style loaders interpolate it and docker --env-file passes it through literally") ;; + esac + + local j + for (( j = 0; j < ${#FILE_KEYS[@]}; j++ )); do + if [ "${FILE_KEYS[$j]}" = "$key" ]; then + FILE_NOTES+=("${key}|is set twice (lines ${FILE_LINES[$j]} and ${lineno}); loaders disagree about which one wins") + break + fi + done + + FILE_KEYS+=("$key") + FILE_VALS+=("$val") + FILE_LINES+=("$lineno") + done < "$path" +} + +if [ -n "$SOURCE_FILE" ]; then + # Refused rather than fallen back on. You asked about a file; answering about + # the process environment instead would be a confident pass from a source you + # did not name — see status.sh on why a tool that answers about the wrong + # thing is worse than one that answers nothing. + [ -e "$SOURCE_FILE" ] || die "--file '${SOURCE_FILE}' does not exist. Refusing to check the process environment instead." + [ -f "$SOURCE_FILE" ] || die "--file '${SOURCE_FILE}' is not a regular file." + [ -r "$SOURCE_FILE" ] || die "--file '${SOURCE_FILE}' is not readable by this user." + load_file "$SOURCE_FILE" + SOURCE_LABEL="${SOURCE_FILE}" +else + SOURCE_LABEL="the process environment" +fi + +# The empty string means "not set here". Callers must not read it as "set to +# nothing" — FOUND says which. +FOUND="" +VALUE="" + +# The locals are prefixed for the same reason the snapshot is: `lookup`'s own +# `want` and `j` would shadow a SPEC entry of either name, and this is the one +# function that resolves a variable name at runtime. +lookup() { + local _ce_want="$1" _ce_j + FOUND="" + VALUE="" + + if [ -n "$SOURCE_FILE" ]; then + # No break: the last assignment wins, which is what a loader reading the + # file top to bottom into a map does. + for (( _ce_j = 0; _ce_j < ${#FILE_KEYS[@]}; _ce_j++ )); do + if [ "${FILE_KEYS[$_ce_j]}" = "$_ce_want" ]; then FOUND="yes"; VALUE="${FILE_VALS[$_ce_j]}"; fi + done + else + # The snapshot taken at the top, never the live variable: see the reasoning + # there. A name absent from the snapshot was absent from the environment, + # whatever this script may since have set in its own namespace. + for (( _ce_j = 0; _ce_j < ${#_CE_ENV_NAMES[@]}; _ce_j++ )); do + if [ "${_CE_ENV_NAMES[$_ce_j]}" = "$_ce_want" ]; then + FOUND="yes"; VALUE="${_CE_ENV_VALS[$_ce_j]}"; break + fi + done + fi +} + +# --------------------------------------------------------------------------- +# Shape checks. +# +# check_shape prints zero or more lines, each 'fail:' or 'note:' +# — a protocol rather than array appends because it is called in a command +# substitution, and an array appended inside a subshell is gone at the closing +# paren. +# --------------------------------------------------------------------------- + +check_shape() { + local name="$1" kind="$2" param="$3" value="$4" + + case "$value" in + ' '*|$'\t'*) printf 'fail:begins with whitespace, which is part of the value everywhere except a shell\n' ;; + esac + case "$value" in + *' '|*$'\t') printf 'fail:ends with whitespace, which is part of the value and breaks every comparison against it\n' ;; + esac + + case "$kind" in + url|port|int|one-of|path) + case "$value" in + *$'\n'*) printf 'fail:contains a newline, so nothing downstream will parse it\n'; return 0 ;; + esac ;; + esac + + case "$kind" in + nonempty) + # Presence and non-emptiness are the same test, already done by the + # caller. Reaching here means it passed; there is nothing else to know. + : ;; + + int) + if [[ ! "$value" =~ ^[+-]?[0-9]+$ ]]; then + printf 'fail:is not a whole number: %s\n' "$(show "$name" "$kind" "$value")" + return 0 + fi + local n="${value#+}" + # Bash arithmetic is 64-bit and errors past it, so an absurd number is + # rejected by length before anything tries to compare it. + if [ "${#n}" -gt 18 ]; then + printf 'fail:has %d digits; nothing downstream will read that as a number\n' "${#n}" + return 0 + fi + if [ -n "$param" ]; then + local lo="${param%%..*}" hi="${param##*..}" + if [ "$n" -lt "${lo#+}" ] || [ "$n" -gt "${hi#+}" ]; then + # Through show(), not the bare value: an out-of-range number under a + # credential-shaped name — TOTP_KEY, AUTH_TOKEN_TTL — is still a + # credential, and this is a failure message, which is precisely where + # the header says a careless validator leaks. The bounds come from the + # spec, so they are always safe to print. + printf 'fail:is %s, outside the allowed %s..%s\n' \ + "$(show "$name" "$kind" "$value")" "${lo#+}" "${hi#+}" + fi + fi ;; + + port) + case "$value" in + *://*) printf 'fail:is a URL, not a port number: %s\n' "$(show "$name" url "$value")"; return 0 ;; + */*) printf 'fail:looks like a compose port mapping (%s); a port field takes the number alone\n' "$(show "$name" literal "$value")"; return 0 ;; + esac + if [[ ! "$value" =~ ^[0-9]{1,5}$ ]] || [ "$value" -lt 1 ] || [ "$value" -gt 65535 ]; then + printf 'fail:is not a port; ports are 1-65535\n' + return 0 + fi ;; + + url) + case "$value" in + *://*) : ;; + *) + printf 'fail:has no scheme — %s is a host, not a URL; write it as scheme://host\n' "$(show "$name" literal "$value")" + return 0 ;; + esac + + local scheme="${value%%://*}" rest="${value#*://}" authority + if [[ ! "$scheme" =~ ^[A-Za-z][A-Za-z0-9+.-]*$ ]]; then + printf 'fail:has a scheme that is not a scheme\n' + return 0 + fi + + authority="${rest%%/*}" + authority="${authority%%\?*}" + case "$authority" in *@*) authority="${authority##*@}" ;; esac + + if [ -z "$authority" ]; then + printf 'fail:has no host between // and the path\n' + return 0 + fi + case "$authority" in + *' '*) printf 'fail:has a space in the host\n'; return 0 ;; + esac + + case "$authority" in + localhost|localhost:*|127.0.0.1|127.0.0.1:*|0.0.0.0|0.0.0.0:*|'[::1]'|'[::1]':*) + printf 'note:points at %s — correct on a laptop; inside a container that is the container itself, not the host\n' "$authority" ;; + host.docker.internal*) + printf 'note:points at host.docker.internal, which resolves on Docker Desktop and not on a Linux daemon without an extra_hosts entry\n' ;; + esac + + case "$value" in + */) printf 'note:ends with a slash, so anything joining a path onto it produces a double slash\n' ;; + esac ;; + + path) + case "$value" in + '~'*) printf 'fail:starts with ~, and nothing expands a tilde inside an environment variable — a process reading this opens a directory literally named ~\n' ;; + esac + case "$value" in + /*) : ;; + *) printf 'note:is relative, so it resolves against the working directory of whatever starts the process rather than the one you typed it in\n' ;; + esac + # Existence is a fact about THIS machine. Checking it against a file that + # describes another machine would report a fault that is not one, so in + # --file mode it is not checked and the report says so rather than + # leaving a silent gap. + if [ -z "$SOURCE_FILE" ] && [ ! -e "$value" ]; then + printf 'note:does not exist on this machine (this may be correct if the volume is mounted later)\n' + fi ;; + + one-of) + local rest="$param" opt matched="" ci_match="" + local lower_value lower_opt + lower_value=$(printf '%s' "$value" | tr '[:upper:]' '[:lower:]') + while [ -n "$rest" ]; do + opt="${rest%%,*}" + [ "$value" = "$opt" ] && matched="yes" + lower_opt=$(printf '%s' "$opt" | tr '[:upper:]' '[:lower:]') + [ "$lower_value" = "$lower_opt" ] && ci_match="$opt" + case "$rest" in *,*) rest="${rest#*,}" ;; *) rest="" ;; esac + done + if [ -z "$matched" ]; then + if [ -n "$ci_match" ]; then + printf 'fail:is %s, and the comparison is case-sensitive: write %s\n' "$(show "$name" one-of "$value")" "$ci_match" + else + printf 'fail:is %s; allowed: %s\n' "$(show "$name" one-of "$value")" "${param//,/, }" + fi + fi ;; + + secret-min-length) + # Length is the only property reported. Nothing below prints the value, + # and the comparison below pipes it to `tr` rather than passing it as an + # argument, so it never appears in this process's command line either. + if [ "${#value}" -lt "$param" ]; then + printf 'fail:is %d characters and the spec requires %d (value withheld)\n' "${#value}" "$param" + fi + local lower placeholder + lower=$(printf '%s' "$value" | tr '[:upper:]' '[:lower:]') + for placeholder in $PLACEHOLDER_SECRETS; do + if [ "$lower" = "$placeholder" ]; then + printf 'fail:is a well-known placeholder, not a secret — anyone can guess it (value withheld)\n' + break + fi + done ;; + esac +} + +# --------------------------------------------------------------------------- +# Measure everything, then report. Nothing exits early: one run must name every +# problem, or fixing them costs one restart each. +# --------------------------------------------------------------------------- + +MISSING=() +MALFORMED=() +NOTES=() +OPT_ABSENT=() +PRESENT_OK=() + +for (( i = 0; i < ${#SPEC_NAME[@]}; i++ )); do + name="${SPEC_NAME[$i]}" + req="${SPEC_REQ[$i]}" + kind="${SPEC_KIND[$i]}" + param="${SPEC_PARAM[$i]}" + desc="${SPEC_DESC[$i]}" + + lookup "$name" + + if [ -z "$FOUND" ]; then + if [ "$req" = "required" ]; then + MISSING+=("${name}|${desc}") + else + OPT_ABSENT+=("${name}|${desc}") + fi + continue + fi + + if [ -z "$VALUE" ]; then + if [ "$req" = "required" ]; then + MISSING+=("${name}|set, but empty — ${desc}") + else + # Set-to-empty is not unset, and code tests it both ways: a truthiness + # check sees nothing, a key-presence check sees it. Reported so the + # ambiguity is the operator's to resolve rather than this script's. + OPT_ABSENT+=("${name}|${desc}") + NOTES+=("${name}|is set to the empty string, which is not the same as unset to code that checks whether the key is present") + fi + continue + fi + + findings=$(check_shape "$name" "$kind" "$param" "$VALUE") + had_fail="" + + while IFS= read -r rline; do + [ -n "$rline" ] || continue + case "$rline" in + fail:*) MALFORMED+=("${name}|${rline#fail:}"); had_fail="yes" ;; + note:*) NOTES+=("${name}|${rline#note:}") ;; + esac + done <<<"$findings" + + [ -n "$had_fail" ] || PRESENT_OK+=("${name}|$(show "$name" "$kind" "$VALUE")") +done + +if [ -n "$SOURCE_FILE" ]; then + # Only for a file. The process environment carries hundreds of variables from + # the shell, the init system and every tool that ever exported one, so + # "undeclared" there would be noise; in a file every line was written on + # purpose, and one that matches nothing in the spec is usually a typo in a + # name — the thing that makes people swear the value is right there. + for (( j = 0; j < ${#FILE_KEYS[@]}; j++ )); do + fkey="${FILE_KEYS[$j]}" + known="" + for name in ${SPEC_NAME[@]+"${SPEC_NAME[@]}"}; do + [ "$name" = "$fkey" ] && known="yes" && break + done + if [ -z "$known" ]; then + if [ "${#fkey}" -gt "$NAME_PRINT_MAX" ]; then + FILE_NOTES+=("line ${FILE_LINES[$j]}|sets a ${#fkey}-character name that is not declared in SPEC and is too long to be a name at all; the text is withheld in case this is one line of a wrapped multi-line secret") + else + FILE_NOTES+=("${fkey}|is set on line ${FILE_LINES[$j]} but is not declared in SPEC — a misspelled name, or a spec that has fallen behind") + fi + fi + done +fi + +group() { + local heading="$1"; shift + [ $# -gt 0 ] || return 0 + say "" + say "$heading" + local e + for e in "$@"; do + printf ' %-*s %s\n' "$NAME_WIDTH" "${e%%|*}" "${e#*|}" >&2 + done +} + +n_missing="${#MISSING[@]}" +n_malformed="${#MALFORMED[@]}" +n_badlines="${#FILE_BAD[@]}" +n_problems=$(( n_missing + n_malformed + n_badlines )) + +# `${arr[@]+"${arr[@]}"}` throughout: under `set -u` a plain "${arr[@]}" on an +# empty array is an error in bash before 4.4, and macOS still ships 3.2. +if [ -z "$QUIET" ] || [ "$n_problems" -gt 0 ]; then + say "source: ${SOURCE_LABEL}" + if [ -n "$SOURCE_FILE" ]; then + say " the process environment was NOT consulted, on purpose." + fi +fi + +if [ -z "$QUIET" ]; then + group "ok — present and the right shape:" ${PRESENT_OK[@]+"${PRESENT_OK[@]}"} + group "optional, not set — this is fine:" ${OPT_ABSENT[@]+"${OPT_ABSENT[@]}"} + group "worth knowing — not failures:" ${NOTES[@]+"${NOTES[@]}"} + + # Kept apart from the value notes above: these are facts about how the FILE + # is written, and they apply to whichever loader reads it rather than to any + # one value being wrong. + if [ -n "$SOURCE_FILE" ]; then + group "about ${SOURCE_FILE} itself:" ${FILE_NOTES[@]+"${FILE_NOTES[@]}"} + fi + + if [ -n "$SOURCE_FILE" ]; then + for (( i = 0; i < ${#SPEC_KIND[@]}; i++ )); do + if [ "${SPEC_KIND[$i]}" = "path" ]; then + say "" + say "note: path existence was not checked — a file can describe a machine" + say " that is not this one. Run without --file to check paths here." + break + fi + done + fi +fi + +group "MALFORMED — set, but the wrong shape:" ${MALFORMED[@]+"${MALFORMED[@]}"} +group "UNREADABLE LINES — these set nothing:" ${FILE_BAD[@]+"${FILE_BAD[@]}"} +group "MISSING — required, not set:" ${MISSING[@]+"${MISSING[@]}"} + +if [ "$n_problems" -gt 0 ]; then + say "" + say "${n_missing} missing, ${n_malformed} malformed, ${n_badlines} unreadable line(s) in ${SOURCE_LABEL}." + # Malformed counts as a failure whether the variable was required or + # optional: not setting an optional variable is a choice, and setting it to + # something unusable is a mistake that reads as a working configuration. + say "fix the entries above and run this again — nothing that reads this" + say "environment can be trusted until then." + # Exit 1, not die's 2. The difference is the whole contract: 1 means this + # measured the environment and it is wrong, 2 means it never got far enough + # to find out. A caller that cannot tell those apart will retry the wrong one. + exit 1 +fi + +if [ -z "$QUIET" ]; then + say "" + say "ok — ${#PRESENT_OK[@]} set and well-formed, ${#OPT_ABSENT[@]} optional and absent, ${#SPEC_NAME[@]} declared." + say "shapes only: nothing was connected to, so this says the values are" + say "plausible, not that the services behind them are up." +fi + +exit 0 diff --git a/docs/architecture/scripts/commit-mine.sh b/docs/architecture/scripts/commit-mine.sh new file mode 100755 index 0000000..8ed727c --- /dev/null +++ b/docs/architecture/scripts/commit-mine.sh @@ -0,0 +1,120 @@ +#!/usr/bin/env bash +# +# Commit only the paths you name, when something else is also writing the tree. +# +# ## The failure this catches +# +# Two agents, or an agent and a person, sharing one checkout share one **git +# index**. Staging is global state, and the window between staging a change and +# committing it is however long it takes to write the commit message. Anything +# that runs `git add -A` inside that window takes your files with it. +# +# Two real instances, one afternoon, one repository — both while the rule +# "stage by explicit path, and re-check the index immediately before committing" +# was being followed to the letter: +# +# - A commit about reviving a game carried two unrelated documentation edits. +# - A commit about art carried an entire renderer fix, a new class, its test +# and two documents. Its message had no `closes #N`, so the issue that work +# finished stayed open and had to be closed by hand afterwards. +# +# Nothing was lost either time. The attribution was wrong, and once the tracker +# was wrong with it. **The rule is not the fix, because the danger is the +# window** — so this closes the window instead: the message is written first and +# passed in, staging and scanning and committing happen back to back, and the +# commit itself names its paths. +# +# ## Why a pathspec commit rather than unstaging theirs +# +# `git commit -- ` takes the working-tree content of exactly those paths +# and ignores the rest of the index. The other writer's staged work is neither +# swept into your commit nor removed from their index, so there is no step here +# that can break *their* commit either. What they have staged is reported, so you +# know somebody else is mid-flight before you add to the race. +# +# ## Usage +# +# bash scripts/commit-mine.sh [path…] +# bash scripts/commit-mine.sh --push [path…] +# +# ## Exit codes +# +# 0 committed (and pushed, with `--push`) +# 1 the secret scan objected, or a push left something unpushed +# 2 nothing was committed: bad arguments, or a path that does not exist. +# **Two is not a pass** +set -euo pipefail + +HERE="$(cd "$(dirname "$0")" && pwd)" +SECRETS="$HERE/secrets.sh" + +PUSH="no" +if [ "${1:-}" = "--push" ]; then PUSH="yes"; shift; fi + +if [ "$#" -lt 2 ]; then + sed -n '2,45p' "$0" >&2 + exit 2 +fi + +MESSAGE_FILE="$1"; shift +[ -f "$MESSAGE_FILE" ] || { echo "commit-mine: no message file: $MESSAGE_FILE" >&2; exit 2; } + +git rev-parse --is-inside-work-tree >/dev/null 2>&1 || { + echo "commit-mine: not inside a git repository" >&2 + exit 2 +} + +for path in "$@"; do + [ -e "$path" ] || { echo "commit-mine: no such path: $path" >&2; exit 2; } +done + +# Whatever anyone else has in flight. Not an error and not touched — but you +# should see it before committing into the same index. +OTHERS="$(git diff --cached --name-only | grep -vxF -f <(printf '%s\n' "$@") || true)" +if [ -n "$OTHERS" ]; then + echo "commit-mine: NOTE — the index also holds work that is not yours:" + printf ' %s\n' $OTHERS + echo " (left staged, and left out of this commit)" + echo +fi + +# Staged only so the scanner sees exactly these paths, and only for as long as +# the scan takes. A credential is the one mistake here that cannot be undone by +# a later commit. +git add -- "$@" +if [ -x "$SECRETS" ] || [ -f "$SECRETS" ]; then + set +e + bash "$SECRETS" + SCAN=$? + set -e + # 2 means it scanned nothing, which the convention in this template treats as + # a failure rather than a pass — a scanner that did not run has not cleared + # anything. + if [ "$SCAN" -ne 0 ]; then + git restore --staged -- "$@" + echo "commit-mine: secret scan exited $SCAN; nothing committed" >&2 + exit 1 + fi +else + echo "commit-mine: WARNING — $SECRETS not found, committing unscanned" >&2 +fi + +git commit -F "$MESSAGE_FILE" -- "$@" + +echo +echo "commit-mine: committed" +git show --stat --oneline HEAD | head -20 + +if [ "$PUSH" = "yes" ]; then + BRANCH="$(git rev-parse --abbrev-ref HEAD)" + git push origin "$BRANCH" + # Finished work is pushed work. A local commit is invisible to everything that + # reports on the project, so the push is verified rather than assumed. + UNPUSHED="$(git log --oneline "origin/$BRANCH..HEAD" 2>/dev/null || true)" + if [ -n "$UNPUSHED" ]; then + echo "commit-mine: WARNING — still unpushed after push:" >&2 + echo "$UNPUSHED" >&2 + exit 1 + fi + echo "commit-mine: pushed; origin/$BRANCH..HEAD is empty" +fi diff --git a/docs/architecture/scripts/dead-code.py b/docs/architecture/scripts/dead-code.py new file mode 100755 index 0000000..7708172 --- /dev/null +++ b/docs/architecture/scripts/dead-code.py @@ -0,0 +1,219 @@ +#!/usr/bin/env python3 +"""Exports nothing imports, and assets nothing renders. + +## Why + +Dead code is not merely clutter. It is read, maintained, and trusted: somebody +eventually changes it, and nothing happens. On the repository this was written +against, two issues had already been filed by hand for exactly this — four +progress components exported and imported nowhere, and a 1.1 MB image shipping +in every container while being rendered by nothing. Both are the kind of thing a +person finds by accident and a script finds every time. + +## What it checks + +**Exports.** A symbol exported from a module and referenced nowhere but its own +declaration. Framework entry points are excluded by name, because a Next.js +`page.tsx` exporting `metadata` or a route exporting `GET` is *called by the +framework* and referencing it would be wrong. + +**Assets.** A file under the asset roots whose name appears in no source file. + +## Tuned for near-silence + +`GUARDS.md` rule 5: a check that is wrong six times in ten is one people learn +to skip. So the reference scan covers **tests as well as source** — the first +draft of this scanned only `src/` and reported every test-only helper as dead, +which is the mistake that makes a tool untrustworthy on its first run. + +It still cannot see a symbol referenced only by string name, by a build step, or +from outside the repository. Treat findings as candidates, and delete only what +you have confirmed. + + python3 scripts/dead-code.py + python3 scripts/dead-code.py --src src --refs src tests --assets public + python3 scripts/dead-code.py --exports-only + +Exit codes: 0 nothing found. 1 candidates found. 2 nothing was scanned. +""" + +from __future__ import annotations + +import argparse +import os +import re +import sys +from collections import Counter + +SKIP_DIRS = {"node_modules", ".next", ".git", "dist", "build", "__pycache__"} +CODE_EXTS = (".ts", ".tsx", ".js", ".jsx", ".mjs") + +# An asset is referenced from prose and configuration as often as from code — a +# README embeds a screenshot, a manifest names an icon, a stylesheet loads a +# font. Scanning only code reported twelve screenshots as dead when the README +# renders every one of them, which is exactly the first-run noise that ends a +# tool's credibility. +PROSE_EXTS = CODE_EXTS + (".md", ".mdx", ".json", ".yml", ".yaml", ".html", ".css", ".webmanifest") + +EXPORT = re.compile( + r"^export\s+(?:async\s+)?(?:function|const|let|class|type|interface|enum)\s+(\w+)" +) + +# `export default function FaqPage()` is called by the framework, and the name +# is incidental — it exists so a stack trace reads well. Matching it reported +# every page and layout in the application as dead, which is the kind of first +# run that ends a tool's credibility. +DEFAULT_EXPORT = re.compile(r"^export\s+default\b") + +# Called by the framework, never imported. Reporting these would be worse than +# reporting nothing: it teaches the reader that the tool does not understand the +# project, and everything after it gets skipped too. +FRAMEWORK = { + "default", "metadata", "generateMetadata", "generateStaticParams", + "dynamic", "revalidate", "runtime", "fetchCache", "preferredRegion", + "maxDuration", "viewport", "GET", "POST", "PUT", "PATCH", "DELETE", + "HEAD", "OPTIONS", "middleware", "config", "loader", "action", +} + + +def walk(roots: list[str], exts: tuple[str, ...]) -> list[str]: + found: list[str] = [] + + for root in roots: + if os.path.isfile(root): + found.append(root) + continue + + for base, dirs, names in os.walk(root): + dirs[:] = [d for d in dirs if d not in SKIP_DIRS] + found.extend( + os.path.join(base, n) for n in names if not exts or n.endswith(exts) + ) + + return found + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__.split("\n")[0]) + parser.add_argument("--src", nargs="+", default=["src"], help="where exports live") + parser.add_argument( + "--refs", nargs="+", default=["src", "tests", "app", "scripts", "notices"], + help="where references may appear — include tests, or every helper reads as dead", + ) + parser.add_argument("--assets", nargs="+", default=["public"], help="asset roots") + parser.add_argument("--exports-only", action="store_true") + parser.add_argument("--assets-only", action="store_true") + args = parser.parse_args() + + sources = walk(args.src, CODE_EXTS) + references = walk([r for r in args.refs if os.path.exists(r)], CODE_EXTS) + + if not sources and not args.assets_only: + print("dead-code: nothing to scan.", file=sys.stderr) + return 2 + + # One pass over every referencing file, counting every identifier and + # remembering the raw text for asset lookups. + counts: Counter[str] = Counter() + corpus: list[str] = [] + + for path in references: + try: + with open(path, encoding="utf8", errors="ignore") as handle: + body = handle.read() + except OSError: + continue + + corpus.append(body) + counts.update(re.findall(r"\b\w+\b", body)) + + findings = 0 + + if not args.assets_only: + exported: dict[str, str] = {} + + for path in sources: + try: + with open(path, encoding="utf8", errors="ignore") as handle: + for line in handle: + if DEFAULT_EXPORT.match(line): + continue + + match = EXPORT.match(line) + + if match and match.group(1) not in FRAMEWORK: + exported.setdefault(match.group(1), path) + except OSError: + continue + + # Exactly one occurrence is the declaration itself and nothing else. + dead = sorted((name, path) for name, path in exported.items() if counts[name] <= 1) + + if dead: + print("exported and referenced nowhere:") + + for name, path in dead: + print(f" {name:<32} {path}") + + findings += len(dead) + + if not args.exports_only: + # A wider net than the export scan uses, for the reason above. + asset_refs = walk([r for r in args.refs if os.path.exists(r)], PROSE_EXTS) + asset_refs += [p for p in walk(["."], (".md",)) if p.count(os.sep) <= 1] + + text = "\n".join(corpus) + + for path in set(asset_refs): + try: + with open(path, encoding="utf8", errors="ignore") as handle: + text += "\n" + handle.read() + except OSError: + continue + + assets = walk([a for a in args.assets if os.path.exists(a)], ()) + orphans = [] + + for path in assets: + name = os.path.basename(path) + + # By basename and by path — a reference is written either way, and + # missing one direction is how a false positive gets in. + if name in text or path in text or path.split(os.sep, 1)[-1] in text: + continue + + try: + size = os.path.getsize(path) + except OSError: + size = 0 + + orphans.append((size, path)) + + if orphans: + print("\nasset files nothing references:" if findings else "asset files nothing references:") + + for size, path in sorted(orphans, reverse=True): + print(f" {size / 1024:9.1f} KB {path}") + + findings += len(orphans) + + if not findings: + print( + f"dead-code: nothing unreferenced across {len(sources)} source " + f"and {len(references)} referencing file(s).", + file=sys.stderr, + ) + return 0 + + print( + f"\ndead-code: {findings} candidate(s). A symbol referenced only by " + "string name, or from outside this repository, will appear here and is " + "not dead — confirm before deleting.", + file=sys.stderr, + ) + + return 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/docs/architecture/scripts/deploy.py b/docs/architecture/scripts/deploy.py new file mode 100755 index 0000000..60f3829 --- /dev/null +++ b/docs/architecture/scripts/deploy.py @@ -0,0 +1,581 @@ +#!/usr/bin/env python3 +""" +Deploy automation: update a running stack to a newly published image. + +Builds a Docker image from the project directory, tags it with an incremented +version and `latest`, pushes it to the Forgejo Docker registry, then updates the +configured Portainer stack to that image and verifies the container is running +and healthy. + +## This is the only copy + +It used to exist twice — here and in the `privacyllc-deploy` skill — and the two +drifted apart until neither was a subset of the other (#209). The skill copy had +the credential resolution and the two safety refusals below; this one had the +environment-driven configuration. A defect fixed in one stayed live in the +other, behind a closed issue, because nothing said which was authoritative. + +The skill's copy is now a symlink to this file. Anything project-specific must +therefore arrive through the environment, never as a literal: a script copied +from another project must not silently point a deploy at that project's stack — +or, just as quietly, bake that project's canonical origin into somebody else's +image. + +Credentials: + - Forgejo registry token: ~/.openclaw/docker-registry.env + - Portainer API key: PORTAINER_API_KEY, or the credential file the operator + keeps Portainer keys in — which is the file named after Portainer, not the + one named after whichever project you are deploying. Looking in the wrong + one is what made every deploy on the source project manual for weeks (#137): + the key was on the machine the whole time, and the error said it was + missing. + +Usage: + export PORTAINER_API_URL="https://192.168.1.11:9443/api" + python3 deploy.py [--project-dir PATH] [--portainer-stack-id ID] + [--dry-run] [--build-only] [--skip-push] [--skip-portainer] +""" + +import argparse +import json +import os +import re +import subprocess +import sys +import traceback +from pathlib import Path + +import urllib3 + +urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) + +# --------------------------------------------------------------------------- +# Everything about a particular deployment lives here, and nowhere else. +# +# Set these before the first run. They are read from the environment so a +# checkout carries no deployment's identity — the same reason `release.sh` takes +# its image from RELEASE_IMAGE rather than hard-coding one: a script copied from +# another project must not silently point a deploy at that project's stack. +# +# DEPLOY_PROJECT_DIR the repository to build from +# DEPLOY_IMAGE fully-qualified image name, no tag +# DEPLOY_STACK_ID the Portainer stack that runs it +# DEPLOY_CONTAINER the container name, for the post-deploy check +# DEPLOY_SITE_URL public origin, baked into the image at build time +# DEPLOY_LABEL what to call this thing in output (cosmetic) +# PORTAINER_API_URL e.g. https://portainer.example.com:9443/api +# --------------------------------------------------------------------------- +PROJECT_DIR_DEFAULT = os.environ.get("DEPLOY_PROJECT_DIR", os.getcwd()) +FORGEJO_IMAGE = os.environ.get("DEPLOY_IMAGE", "") +FORGEJO_REGISTRY = FORGEJO_IMAGE.split("/")[0] if FORGEJO_IMAGE else "" +PORTAINER_STACK_ID_DEFAULT = int(os.environ.get("DEPLOY_STACK_ID", "0") or 0) +CONTAINER_NAME = os.environ.get("DEPLOY_CONTAINER", "") + +# The canonical origin is frozen into the image at build time — every canonical +# URL, the sitemap and robots.txt — and cannot be corrected without another +# build. Both copies of this script hard-coded one project's origin, so +# deploying any other project from it would have baked the wrong one in and +# nothing would have failed. Required, rather than defaulted, for that reason. +SITE_URL = os.environ.get("DEPLOY_SITE_URL", "") + +# Cosmetic only: what the [DONE] line calls this. Nothing branches on it. +DEPLOY_LABEL = os.environ.get("DEPLOY_LABEL", "site") + + +def load_env_file(path: Path) -> dict: + """Read KEY=VALUE lines, ignoring blank lines and comments.""" + values = {} + if not path.exists(): + return values + for line in path.read_text().splitlines(): + line = line.strip() + if not line or line.startswith("#"): + continue + if "=" in line: + key, value = line.split("=", 1) + values[key.strip()] = value.strip() + return values + + +def load_privacyllc_credentials() -> dict: + """Load the privacyllc credential file (KEY=VALUE format).""" + path = Path.home() / ".openclaw" / "credentials" / "privacyllc.md" + return load_env_file(path) + + +def load_registry_credentials() -> dict: + """Load the docker registry credential file.""" + path = Path.home() / ".openclaw" / "docker-registry.env" + return load_env_file(path) + + +def get_git_version(project_dir: Path) -> str: + """Derive the next release version from the latest release commit message. + + Release commits are of the form `chore(release): vX.Y.Z`. Returns the next + patch increment (vX.Y.Z+1). + """ + result = subprocess.run( + ["git", "log", "--oneline", "--grep=chore(release): v", "-1"], + cwd=project_dir, + capture_output=True, + text=True, + check=True, + ) + line = result.stdout.strip() + if not line: + raise RuntimeError("No release commit found; cannot auto-increment version") + match = re.search(r"v(\d+)\.(\d+)\.(\d+)", line) + if not match: + raise RuntimeError(f"Could not parse version from commit: {line}") + major, minor, patch = map(int, match.groups()) + return f"v{major}.{minor}.{patch + 1}" + + +def run(cmd: list[str], cwd: Path | None = None, env: dict | None = None) -> subprocess.CompletedProcess: + """Run a shell command and return the result.""" + merged_env = os.environ.copy() + if env: + merged_env.update(env) + print(f"[RUN] {' '.join(cmd)}") + return subprocess.run(cmd, cwd=cwd, env=merged_env, text=True, capture_output=True) + + +def build_image(project_dir: Path, version: str, dry_run: bool) -> None: + """Build the Docker image with the incremented version tag.""" + full_tag = f"{FORGEJO_IMAGE}:{version}" + latest_tag = f"{FORGEJO_IMAGE}:latest" + + if dry_run: + # Checked here too, so --dry-run reports a missing origin instead of + # printing a confident "would build" for an image that would be wrong. + _require_site_url() + print(f"[DRY-RUN] would build image {full_tag} and {latest_tag}") + return + + _require_site_url() + + # Clean build to avoid stale Next.js output layers. + print("[BUILD] Cleaning previous build artifacts...") + run(["rm", "-rf", ".next"], cwd=project_dir) + + print(f"[BUILD] Building {full_tag}...") + build_result = run( + [ + "docker", + "build", + "--no-cache", + "--build-arg", + f"NEXT_PUBLIC_SITE_URL={SITE_URL}", + # The Dockerfile declares ARG APP_VERSION and bakes it into + # ENV APP_VERSION and org.opencontainers.image.version. Omitting it + # here built images with an EMPTY version label, which is not + # cosmetic: buildVersion() then returns null so the admin screen + # reports no running build, and release.sh's deployed_version() + # reads that same label to decide which published tag a prune may + # not delete — an empty label makes prune fail-safe but blind. + # + # A published tag is immutable here (release.sh refuses to move + # one), so an image pushed without the label could never be + # corrected, only abandoned at the cost of a burned version number. + # + # release.sh has always passed this and additionally verifies the + # baked value with printenv before it will push. This is the same + # build arg; the verification is not duplicated because this script + # deploys an existing version rather than cutting one. + "--build-arg", + f"APP_VERSION={version}", + "-t", + full_tag, + "-t", + latest_tag, + ".", + ], + cwd=project_dir, + ) + if build_result.returncode != 0: + print(build_result.stderr) + raise RuntimeError("Docker build failed") + print(build_result.stdout) + + +def push_image(version: str, dry_run: bool, skip: bool) -> None: + """Push the version and latest tags to the Forgejo registry.""" + if skip: + print("[SKIP] image push disabled by --skip-push") + return + + full_tag = f"{FORGEJO_IMAGE}:{version}" + latest_tag = f"{FORGEJO_IMAGE}:latest" + + if dry_run: + print(f"[DRY-RUN] would push {full_tag} and {latest_tag}") + return + + creds = load_registry_credentials() + registry = creds.get("FORGEJO_REGISTRY", FORGEJO_REGISTRY) + user = creds.get("FORGEJO_REGISTRY_USER", "null") + token = creds.get("FORGEJO_REGISTRY_TOKEN") + if not token: + raise RuntimeError("FORGEJO_REGISTRY_TOKEN not found in ~/.openclaw/docker-registry.env") + + print("[PUSH] Logging in to Forgejo registry...") + login = run( + ["docker", "login", registry, "-u", user, "--password-stdin"], + env={"DOCKER_CONFIG": os.environ.get("DOCKER_CONFIG", "")}, + ) + if login.returncode != 0: + # docker login --password-stdin reads from stdin; pass token via input + login = subprocess.run( + ["docker", "login", registry, "-u", user, "--password-stdin"], + input=token, + text=True, + capture_output=True, + ) + if login.returncode != 0: + print(login.stderr) + raise RuntimeError("Docker login failed") + + for tag in (full_tag, latest_tag): + print(f"[PUSH] Pushing {tag}...") + push = run(["docker", "push", tag]) + if push.returncode != 0: + print(push.stderr) + raise RuntimeError(f"Docker push failed for {tag}") + print(push.stdout) + + print("[PUSH] Logging out from Forgejo registry...") + run(["docker", "logout", registry]) + + +def load_portainer_key(host: str = "nebula") -> str | None: + """Read one host's Portainer API key out of the credential file. + + The key lives here, not in privacyllc.md. Looking only in the latter is + what made every deploy manual for weeks: the credential was on the machine + the whole time, in the file named after the service it belongs to, while + the error said it was "not available". + + Parsed rather than run through `load_env_file`, for two reasons. The file + is prose with markdown headings, not KEY=VALUE. And it documents the + variable by example — a literal `PORTAINER_API_KEY=` line — which + a KEY=VALUE reader would happily return as the string "", giving a + 401 that looks like a revoked key rather than a parse mistake. + + It also holds more than one host. Sections are `### Portainer (nebula)` and + `### Portainer (exodus)`; this takes the key under the one asked for, so a + deploy cannot authenticate against the wrong machine. + """ + path = Path.home() / ".openclaw" / "credentials" / "portainer.md" + if not path.exists(): + return None + + section = None + for line in path.read_text(encoding="utf-8").splitlines(): + heading = re.match(r"^#{1,6}\s*Portainer\s*\(([^)]+)\)", line.strip()) + if heading: + section = heading.group(1).strip().lower() + continue + if section != host.lower(): + continue + found = re.search(r"`(ptr_[A-Za-z0-9+/=_-]+)`", line) + if found: + return found.group(1) + + return None + + +def portainer_headers() -> dict: + """Return request headers with the Portainer API key. + + Four places, in the order somebody would expect them to win: an explicit + environment variable, then either credential file. Both env spellings are + honoured because the error message below names both, and promising a + variable the code never reads is how this failed silently — exporting + PORTAINER_API_KEY, the name portainer.md itself documents, did nothing. + """ + token = ( + os.environ.get("PORTAINER_API_KEY") + or os.environ.get("PORTAINER_API_TOKEN") + or load_portainer_key() + or load_privacyllc_credentials().get("PORTAINER_API_KEY") + ) + if not token: + raise RuntimeError( + "PORTAINER_API_KEY/PORTAINER_API_TOKEN not available. Looked in the " + "environment, ~/.openclaw/credentials/portainer.md and " + "~/.openclaw/credentials/privacyllc.md." + ) + return {"X-API-Key": token, "Content-Type": "application/json"} + + +def portainer_api_url() -> str: + """Return the Portainer API base URL from environment or default.""" + url = os.environ.get("PORTAINER_API_URL", "https://192.168.1.11:9443/api") + return url.rstrip("/") + + +def update_portainer_stack(version: str, stack_id: int, dry_run: bool, skip: bool) -> None: + """Update the Portainer stack image line and trigger a redeploy.""" + if skip: + print("[SKIP] Portainer update disabled by --skip-portainer") + return + + base_url = portainer_api_url() + headers = portainer_headers() + + # Fetch current stack file. + file_url = f"{base_url}/stacks/{stack_id}/file" + import requests + + file_resp = requests.get(file_url, headers=headers, verify=False, timeout=30) + file_resp.raise_for_status() + compose_text = file_resp.json().get("StackFileContent", "") + if not compose_text: + raise RuntimeError("Portainer stack file content is empty") + + # Replace the image line's tag, keeping the line's indentation. + # + # The previous pattern matched `^\s*image:` — indentation included — and + # replaced the whole match with an unindented line. The image line landed at + # column 0, the YAML stopped parsing ("mapping values are not allowed in + # this context"), and Portainer answered 500 before its compose engine ever + # ran — which is why its log carried no entry for the failed update while a + # hand-edit through the UI, indentation intact, deployed fine. Reproduced + # locally against the real stack file and confirmed with + # `docker compose config` before this was written. + # + # `[ \t]*` rather than `\s*`, because `\s` matches newlines and could crawl + # up through blank lines above the target. + image_line = re.compile( + r"^([ \t]*)image:[ \t]*" + re.escape(FORGEJO_IMAGE) + r":(\S+)[ \t]*$", + flags=re.MULTILINE, + ) + found = image_line.search(compose_text) + if not found: + raise RuntimeError( + f"Could not find image line for {FORGEJO_IMAGE} in Portainer stack file" + ) + if found.group(2) == version: + # Distinguished from "not found": redeploying the version already pinned + # is a no-op somebody may have meant, not a malformed stack file. + print(f"[PORTAINER] stack already pins {version}; sending it unchanged") + + updated_compose = image_line.sub( + lambda m: f"{m.group(1)}image: {FORGEJO_IMAGE}:{version}", + compose_text, + ) + + # The whole point of the last failure: prove the result still parses as + # YAML before it goes anywhere near the stack. `docker compose config` + # validates against the real schema; if the binary is missing the check is + # skipped with a warning rather than silently passed. + import shutil, subprocess, tempfile + if shutil.which("docker"): + with tempfile.NamedTemporaryFile("w", suffix=".yml", delete=False) as handle: + handle.write(updated_compose) + probe_path = handle.name + # --no-interpolate: the stack's variables live in Portainer's Env, not + # in this shell, and `${TURNSTILE_SECRET:?}`-style references would fail + # interpolation here while being perfectly valid there. Structure is + # what the last failure corrupted and structure is checked either way — + # the de-indented file still fails this exact command. + probe = subprocess.run( + ["docker", "compose", "-f", probe_path, "config", "--no-interpolate", "-q"], + capture_output=True, text=True, timeout=60, + ) + Path(probe_path).unlink(missing_ok=True) + if probe.returncode != 0: + raise RuntimeError( + "The transformed compose does not validate; refusing to send it to " + f"Portainer. docker compose says: {probe.stderr.strip()[:300]}" + ) + else: + print("[WARN] docker not on PATH here; skipping compose validation of the transformed file") + + # Fetch stack metadata to get endpoint ID. + stack_url = f"{base_url}/stacks/{stack_id}" + stack_resp = requests.get(stack_url, headers=headers, verify=False, timeout=30) + stack_resp.raise_for_status() + stack_data = stack_resp.json() + endpoint_id = stack_data.get("EndpointId") + if not endpoint_id: + raise RuntimeError("Portainer stack response missing EndpointId") + + if dry_run: + print(f"[DRY-RUN] would update Portainer stack {stack_id} on endpoint {endpoint_id}") + print(updated_compose) + return + + # Update the stack via Portainer API. Preserve existing env vars to avoid + # wiping secrets (POSTGRES_PASSWORD, OIDC_CLIENT_SECRET, etc.). + # + # A stack update REPLACES the definition, so whatever is not sent back is + # gone. This is a round trip of what the GET returned, never a rebuild. + env_vars = stack_data.get("Env") or [] + + # An empty list is not a stack without variables; it is far more likely a + # read that did not return them — a Portainer version that moved the field, + # a partial response, a token whose scope quietly narrowed. Sending it would + # blank every secret the stack holds, and one of them cannot be recovered by + # re-entering it: without CONTACT_ENCRYPTION_KEY every contact message, + # brainstorming note, document and business detail is permanently + # unreadable. `preflightRestore` refuses a restore for this same reason. + # + # So "I could not read them" and "there are none" are kept apart, which is + # the distinction this deployment refuses to collapse anywhere else. The + # escape hatch exists for a stack that genuinely has none, and it has to be + # asked for by name rather than being what happens by default. + if not env_vars and not os.environ.get("DEPLOY_ALLOW_EMPTY_ENV"): + raise RuntimeError( + f"Portainer stack {stack_id} returned no environment variables. Refusing to " + "update, because sending an empty list would erase every variable the stack " + "holds — including CONTACT_ENCRYPTION_KEY, which cannot be recovered by " + "typing it again. Check the stack in Portainer. If it genuinely has none, " + "re-run with DEPLOY_ALLOW_EMPTY_ENV=1." + ) + + payload_env = [{"name": e.get("name"), "value": e.get("value")} for e in env_vars] + + # Names only. The values are secrets and this prints to a terminal and a log. + print(f"[PORTAINER] carrying {len(payload_env)} env vars: " + f"{' '.join(sorted(e['name'] for e in payload_env if e.get('name')))}") + update_url = f"{base_url}/stacks/{stack_id}?endpointId={endpoint_id}" + payload = {"stackFileContent": updated_compose, "env": payload_env, "prune": False} + update_resp = requests.put(update_url, headers=headers, json=payload, verify=False, timeout=120) + if update_resp.status_code >= 400: + # The body is where Portainer puts the reason, and discarding it is what + # turned the indentation bug into a day of guesswork. Truncated, and it + # carries Portainer's own message rather than anything from the stack's + # environment. + raise RuntimeError( + f"Portainer answered {update_resp.status_code} updating stack {stack_id}: " + f"{update_resp.text[:500]}" + ) + print(f"[PORTAINER] Stack {stack_id} updated to {FORGEJO_IMAGE}:{version}") + + +def verify_container(endpoint_id: int, dry_run: bool) -> dict: + """Check that the container is running and healthy via Portainer.""" + if dry_run: + print(f"[DRY-RUN] would verify container {CONTAINER_NAME} on endpoint {endpoint_id}") + return {"State": "DryRun"} + + base_url = portainer_api_url() + headers = portainer_headers() + import requests + + url = f"{base_url}/endpoints/{endpoint_id}/docker/containers/json" + resp = requests.get(url, headers=headers, verify=False, timeout=30) + resp.raise_for_status() + containers = resp.json() + + for container in containers: + names = container.get("Names", []) + if any(name.lstrip("/") == CONTAINER_NAME for name in names): + state = container.get("State", "") + status = container.get("Status", "") + print(f"[VERIFY] {CONTAINER_NAME}: state={state}, status={status}") + if state != "running": + raise RuntimeError(f"Container {CONTAINER_NAME} is not running (state={state})") + if "healthy" not in status.lower() and "(unhealthy)" in status.lower(): + raise RuntimeError(f"Container {CONTAINER_NAME} is unhealthy: {status}") + return container + + raise RuntimeError(f"Container {CONTAINER_NAME} not found on endpoint {endpoint_id}") + + +def _require_config() -> None: + """Refuse a half-configured run. + + The failure this prevents is quiet and expensive: an unset image name makes + every tag `":v1.2.3"`, and an unset stack id makes the update target stack + 0. Neither raises anything obviously about configuration, and one of them + could update somebody else's stack. + + DEPLOY_SITE_URL is required for the same class of reason but a worse + outcome: it is frozen into the image at build time, so an empty or wrong + value cannot be corrected without another build, and nothing fails at + deploy time to tell you. Only checked when a build will actually happen — + --skip-* runs that never build have no origin to bake. + """ + missing = [ + name + for name, value in ( + ("DEPLOY_IMAGE", FORGEJO_IMAGE), + ("DEPLOY_STACK_ID", PORTAINER_STACK_ID_DEFAULT), + ("DEPLOY_CONTAINER", CONTAINER_NAME), + ) + if not value + ] + + if missing: + raise SystemExit( + "[ERROR] not configured: " + ", ".join(missing) + "\n" + " See the header of this file; a deploy script that guesses " + "its target is one that deploys to the wrong place." + ) + + +def _require_site_url() -> None: + """Refuse to build without the origin that will be frozen into the image.""" + if not SITE_URL: + raise SystemExit( + "[ERROR] not configured: DEPLOY_SITE_URL\n" + " It is baked into every canonical URL, the sitemap and " + "robots.txt at build time and cannot be corrected afterwards " + "without another build." + ) + + +def main() -> int: + parser = argparse.ArgumentParser(description="Deploy a stack to a newly built image") + parser.add_argument("--project-dir", default=PROJECT_DIR_DEFAULT, help="Path to the project to build") + parser.add_argument("--portainer-stack-id", type=int, default=PORTAINER_STACK_ID_DEFAULT, help="Portainer stack ID") + parser.add_argument("--dry-run", action="store_true", help="Print actions without executing") + parser.add_argument("--build-only", action="store_true", help="Build only; do not push or update Portainer") + parser.add_argument("--skip-push", action="store_true", help="Skip pushing image to registry") + parser.add_argument("--skip-portainer", action="store_true", help="Skip Portainer stack update") + parser.add_argument("--version", help="Override the auto-incremented version tag") + args = parser.parse_args() + _require_config() + + project_dir = Path(args.project_dir).expanduser().resolve() + + try: + version = args.version or get_git_version(project_dir) + print(f"[DEPLOY] Target version: {version}") + + build_image(project_dir, version, args.dry_run) + + if args.build_only: + print("[DONE] Build-only requested; stopping after build.") + return 0 + + push_image(version, args.dry_run, args.skip_push) + update_portainer_stack(version, args.portainer_stack_id, args.dry_run, args.skip_portainer) + + if not args.skip_portainer: + # Re-fetch endpoint ID for verification if we did not update Portainer. + import requests + + base_url = portainer_api_url() + headers = portainer_headers() + stack_resp = requests.get( + f"{base_url}/stacks/{args.portainer_stack_id}", headers=headers, verify=False, timeout=30 + ) + stack_resp.raise_for_status() + endpoint_id = stack_resp.json().get("EndpointId") + verify_container(endpoint_id, args.dry_run) + + print(f"[DONE] {DEPLOY_LABEL} deployed as {FORGEJO_IMAGE}:{version}") + return 0 + except Exception as exc: + print(f"[ERROR] {exc}") + if os.environ.get("DEPLOY_DEBUG"): + traceback.print_exc() + return 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/docs/architecture/scripts/dev.sh b/docs/architecture/scripts/dev.sh new file mode 100755 index 0000000..fa4dd7a --- /dev/null +++ b/docs/architecture/scripts/dev.sh @@ -0,0 +1,1282 @@ +#!/usr/bin/env bash +# +# Take a fresh clone to a running dev server in one command — and when it cannot, +# say every reason at once instead of dying one prerequisite at a time. +# +# bash scripts/dev.sh # deps, database, migrations, server +# bash scripts/dev.sh --check # audit prerequisites only; changes nothing +# bash scripts/dev.sh --dry-run # print every command it would run +# bash scripts/dev.sh --skip-deps --skip-db --skip-migrate --skip-server +# DEV_DB_SERVICE=postgres bash scripts/dev.sh +# +# exit 0 ready exit 1 something is missing exit 2 could not check +# +# =========================================================================== +# TEMPLATE COPY — configure this before the first run +# =========================================================================== +# +# Copy to `scripts/dev.sh`, `chmod +x` it, and fill in the CONFIGURATION block +# below. Every value there is empty on purpose and every one of them names +# something that belongs to exactly one project: a compose service, a connection +# variable, a migrations directory, a command line. +# +# `scripts/release.sh` in this same directory explains what inherited defaults +# cost when they were carried between projects. The version of that script this +# one is modelled on shipped with its origin project's image name, deploy host +# and container name as defaults; one run in a second project would have pushed +# over the first project's image and pruned its published versions. The same +# shape of mistake here is quieter and still expensive: a DB_SERVICE guessed from +# another project starts a container that is not this project's database, and the +# first symptom is a migration running against the wrong data. +# +# Do NOT add `"dev": "bash scripts/dev.sh"` to package.json. This script starts +# the dev server by running the `dev` script, so that entry is an infinite loop. +# `"setup"` or `"bootstrap"` is the name you want. It refuses to run the loop if +# you do it anyway — twice: by reading the package.json script before running it, +# and by an exported marker that catches the spellings reading cannot see, such +# as a `dev:` target in the Makefile that calls this file. Both are guards, not +# a plan. (`DEV_SH_ALLOW_REENTRY=1` turns the second one off if you ever have a +# reason to nest this inside itself deliberately.) +# +# Assumes: bash 4.4 or newer, git, coreutils. Everything else is detected, and +# its absence is reported rather than assumed. The bash floor is checked on the +# first line that can check it, because macOS still ships 3.2 and the symptom +# there is an "unbound variable" from an unrelated line. +# +# ## Why this exists +# +# A README's setup section is a claim about the past, written by somebody who +# already had the database running and the environment exported, and it is never +# wrong for the person who wrote it. The person it is wrong for is a new clone on +# an empty machine — a new hire on their first morning, or an agent with no +# memory of the last checkout — which is the person with the least context to +# work out what the missing piece was. +# +# The specific failure this ends is the bootstrap that dies one prerequisite at a +# time. Install the runtime, run it again; install docker, run it again; export +# the connection string, run it again. Each run teaches exactly one fact and +# costs a full cycle to learn it, and none of the runs tell you how many are +# left. So the audit is the product here and the four steps are the easy part: +# EVERY prerequisite is checked before ANY of them runs, and they are reported +# together, each with how to get it. +# +# ## It is the setup documentation +# +# The CONFIGURATION block is the list of what this project needs to run, in the +# one form that cannot quietly go stale: a README paragraph that has drifted +# still renders, while a wrong value here fails on the next bootstrap and names +# itself. Read this file to find out what the project requires. Change this file +# when the requirement changes — and if the answer is not written here, then the +# project does not have one written down anywhere. +# +# ## Absence is not zero +# +# Nothing is reported as satisfied unless it was measured. A port that could not +# be probed is "could not check", never "not running"; a project with no +# REQUIRED_ENV is "nothing configured to check", never "environment ok". The one +# that bites hardest is the TCP probe: a bash built without /dev/tcp fails +# exactly like a refused connection, so a naive check sends somebody to debug a +# database that was fine all along. +# +# ## What it deliberately does not do +# +# It does not create, seed, reset or drop a database, and it never runs +# `docker compose down`, `rm` or `-v`. Nothing here deletes data. Stopping the +# stack stays a thing you type on purpose (`docker compose stop`). +# +# The only thing it writes on its own is one empty stamp file under the git +# directory, recording that migrations ran. Everything else that touches the +# disk is the project's own installer or migration command — `npm ci` replaces +# node_modules, because that is what `npm ci` does, and step 1 names the command +# before running it so that is visible rather than inferred. +# +# It never runs `docker compose up` with no service named. That starts every +# service in the file, including an app container that will take the port the dev +# server is about to want, and on a shared daemon it starts things belonging to +# whatever else that compose file describes. Only DB_SERVICE is ever started. +# +# It does not write or source `.env`. Most dev servers load their own env files, +# and two loaders with different rules disagree in ways that take an afternoon to +# find. It reads such a file only if you name it in ENV_FILE, and otherwise says +# that it saw one and left it alone. +# +# It does not install tools. It names each missing one and where to get it — +# installing a language runtime on somebody's behalf is how a machine ends up +# with three of them and no record of which is on PATH. +# +# It does not run tests, build for production, or deploy. Those are separate +# commands so that a broken build fails in the place that owns it. + +set -uo pipefail + +say() { printf '\033[1mdev:\033[0m %s\n' "$*" >&2; } +die() { printf '\033[1mdev:\033[0m %s\n' "$*" >&2; exit 1; } + +# --------------------------------------------------------------------------- +# The shell this needs, said before anything that depends on it. +# +# Before bash 4.4, `set -u` treats an empty array and a zero-length "$@" as +# unbound variables. This script has both — the findings arrays, and the flag +# loop on a run with no flags — so on macOS's stock bash 3.2 it would die with +# "COMPOSE: unbound variable" from a line that has nothing to do with the +# problem. That is exactly the one-fact-per-run failure this file exists to end, +# so the version is checked first and named. +# --------------------------------------------------------------------------- + +case "${BASH_VERSION:-}" in + '') die "this needs bash, and it was started by another shell. Run: bash ${0}" ;; + [0-3].*|4.[0-3]|4.[0-3].*) + die "this needs bash 4.4 or newer (this is $BASH_VERSION). macOS ships 3.2: 'brew install bash', then run this with that one." ;; +esac + +# --------------------------------------------------------------------------- +# The fork bomb, guarded at the door. +# +# Step 4 runs whatever this project calls its dev server, and if that command +# leads back here the two of them fork until the machine gives out. The +# package.json check further down catches the common spelling of that mistake, +# but it can only read package.json: a Makefile target, a renamed copy of this +# script, or a shell wrapper all reach step 4 unseen. An exported marker catches +# every one of them, at the cost of one variable. +# --------------------------------------------------------------------------- + +if [ -n "${DEV_SH_ACTIVE:-}" ] && [ -z "${DEV_SH_ALLOW_REENTRY:-}" ]; then + say "this script is already running further up this process tree, which means the" + say " command it started leads back here — an infinite loop, and one that reads" + say " as a hang rather than as a cycle." + say " Rename whatever entry runs it (\"setup\" is the conventional name for a" + say " bootstrap), or set DEV_SERVER_CMD to the command that actually starts the" + say " server." + die "refusing to re-enter. Set DEV_SH_ALLOW_REENTRY=1 if this nesting is deliberate." +fi +export DEV_SH_ACTIVE=1 + +# --------------------------------------------------------------------------- +# Where "the project" is. +# +# The copy of this script lives inside the project it bootstraps. The current +# directory does not, and neither does the git root: run this file by path from +# inside another checkout and `git rev-parse --show-toplevel` names that other +# checkout, which is how a bootstrap installs one project's dependencies into +# another project's tree. So the script's own location decides, and a git root +# that disagrees is said out loud rather than silently preferred. +# --------------------------------------------------------------------------- + +SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" 2>/dev/null && pwd) +[ -n "$SCRIPT_DIR" ] || die "cannot resolve the directory holding this script." + +# This copy's own filename. Used where the script has to talk about itself: the +# usage line, and the loop guard that looks for a script entry pointing back +# here — both of which are wrong the moment somebody renames the copy, which the +# instructions above invite ("setup" or "bootstrap"). +SELF_NAME=$(basename -- "$0") +[ -n "$SELF_NAME" ] || SELF_NAME="dev.sh" + +ROOT=$(CDPATH= cd -- "$SCRIPT_DIR/.." 2>/dev/null && pwd) +[ -n "$ROOT" ] || die "cannot resolve the project root above $SCRIPT_DIR." + +cd "$ROOT" || die "cannot enter the project root $ROOT." + +GIT_ROOT=$(git rev-parse --show-toplevel 2>/dev/null) + +if [ -n "$GIT_ROOT" ] && [ "$GIT_ROOT" != "$ROOT" ]; then + say "note: this script lives under $ROOT but the surrounding git checkout is" + say " $GIT_ROOT. Using $ROOT — the project this copy belongs to." +fi + +# --------------------------------------------------------------------------- +# Flags +# --------------------------------------------------------------------------- + +CHECK_ONLY="" +DRY_RUN="" +SKIP_DEPS="" +SKIP_DB="" +SKIP_MIGRATE="" +SKIP_SERVER="" + +usage() { + say "usage: bash $SELF_NAME [--check] [--dry-run]" + say " [--skip-deps] [--skip-db] [--skip-migrate] [--skip-server]" + say "exit: 0 ready · 1 something is missing · 2 something could not be checked" +} + +for arg in "$@"; do + case "$arg" in + --check) CHECK_ONLY="yes" ;; + --dry-run) DRY_RUN="yes" ;; + --skip-deps) SKIP_DEPS="yes" ;; + --skip-db) SKIP_DB="yes" ;; + --skip-migrate) SKIP_MIGRATE="yes" ;; + --skip-server) SKIP_SERVER="yes" ;; + -h|--help) usage; exit 0 ;; + *) usage; die "unknown argument '$arg'." ;; + esac +done + +# --------------------------------------------------------------------------- +# CONFIGURATION — fill these in, then delete this banner. +# +# Empty means "not configured", which is never treated as "not needed": where +# the difference matters there is an explicit way to say "this project has none". +# Each may be overridden for one run by the DEV_* variable named beside it. +# --------------------------------------------------------------------------- + +# The compose service that IS this project's database. Set it to the literal +# string "none" if this project has no database — an empty value means nobody +# has decided yet, and this script will not guess which of your services holds +# your data. +DB_SERVICE="${DEV_DB_SERVICE:-}" + +# Compose file. Empty means: look for compose.yaml / compose.yml / +# docker-compose.yml / docker-compose.yaml in the project root. +COMPOSE_FILE="${DEV_COMPOSE_FILE:-}" + +# The NAME of the environment variable holding the database connection URL — +# e.g. DATABASE_URL, not the URL itself. Its value is read to find the host and +# port to probe, and this file never needs to contain a credential. +DB_URL_ENV="${DEV_DB_URL_ENV:-}" + +# Environment variables without which this project does not run. Listing them is +# the whole point of the audit: an unset one is worth more as a line in the +# missing-prerequisites report than as a stack trace forty seconds into a boot. +REQUIRED_ENV=() + +# A file to source before checking (with `set -a`). Leave empty unless the +# project genuinely expects a shell to load it — see the header on why sourcing +# .env behind the dev server's back is usually the wrong answer. +ENV_FILE="${DEV_ENV_FILE:-}" + +# Command lines, run with `bash -c`. Leave empty to have them detected from the +# lockfiles, package.json scripts and Makefile targets that are actually here; +# set one when detection is wrong or when the project's stack is not one of the +# four this knows. Detection never invents a name — it reads them. +# +# These strings are PRINTED: in the plan, by --dry-run, and again as each one +# runs. Anything inline in them ends up in whatever captured that output — a CI +# log, a scrollback, a pasted bug report. Put credentials in an environment +# variable the command reads, never in the command line itself. +DEPS_CMD="${DEV_DEPS_CMD:-}" +MIGRATE_CMD="${DEV_MIGRATE_CMD:-}" +SERVER_CMD="${DEV_SERVER_CMD:-}" + +# Directory holding migration files. Only used to skip a migrate run when +# nothing in it has changed since the last successful one. Empty is fine and +# means migrations run every time. +MIGRATIONS_DIR="${DEV_MIGRATIONS_DIR:-}" + +# Seconds to wait for the database port to answer after starting it. +DB_WAIT="${DEV_DB_WAIT:-60}" + +# --------------------------------------------------------------------------- + +case "$DB_WAIT" in + ''|*[!0-9]*) die "DEV_DB_WAIT must be a whole number of seconds, got '$DB_WAIT'." ;; +esac + +if [ -n "${DEV_REQUIRED_ENV:-}" ]; then + # Word-split on purpose: DEV_REQUIRED_ENV is a space-separated list of names. + # shellcheck disable=SC2206 + REQUIRED_ENV+=($DEV_REQUIRED_ENV) +fi + +if [ -n "$ENV_FILE" ]; then + [ -r "$ENV_FILE" ] || die "ENV_FILE '$ENV_FILE' is set but not readable. Fix the path or clear it." + set -a + # shellcheck disable=SC1090 + . "$ENV_FILE" || die "sourcing '$ENV_FILE' failed — nothing was started." + set +a + say "sourced $ENV_FILE" +fi + +# --------------------------------------------------------------------------- +# Findings. Collected, never printed as they are found, because the entire value +# of this script is that you learn all of them in one run. +# +# problem — a step cannot run. Fixable, and the fix is stated. +# unknown — could not be determined. Not a pass and not a failure. +# note — true and worth knowing; blocks nothing. +# --------------------------------------------------------------------------- + +PROBLEMS=() +UNKNOWNS=() +NOTES=() +SKIPPED=() + +problem() { PROBLEMS+=("$1"$'\t'"$2"$'\t'"$3"); } +unknown() { UNKNOWNS+=("$1"$'\t'"$2"); } +note() { NOTES+=("$1"); } +skipped() { SKIPPED+=("$1"$'\t'"$2"); } + +have() { command -v "$1" >/dev/null 2>&1; } + +# Free-standing so a tool needed by two steps is demanded once, with one fix. +need_tool() { + local tool=$1 why=$2 how=$3 + have "$tool" && return 0 + problem "$tool" "$why" "$how" + return 1 +} + +has_word() { + case " $2 " in *" $1 "*) return 0 ;; esac + return 1 +} + +# Whole-line membership in newline-separated output, plus the same output on one +# line for a message. Both exist because `docker compose ps --services` prints a +# single empty line when nothing is running: joined into a string that is a lone +# space, which is not empty, which made "nothing is running" indistinguishable +# from "the list could not be read". Emptiness never decides anything here — the +# exit status of the command that produced the list does. +# +# A here-string rather than a pipe into `grep -q`: grep exits at the first match, +# and under `pipefail` the SIGPIPE that kills the writer becomes the pipeline's +# status — turning a found line into "not found" for a list long enough to fill +# a pipe buffer. The answer would be "no such service in your compose file" +# about a service that is right there. +has_line() { grep -qxF -- "$2" <<<"$1"; } +one_line() { printf '%s' "$1" | tr '\n' ' ' | sed 's/ */ /g; s/^ //; s/ $//'; } + +# --------------------------------------------------------------------------- +# Detect the stack. Four markers, each read from the filesystem, none assumed. +# --------------------------------------------------------------------------- + +STACKS="" +MAKEFILE="" + +[ -f package.json ] && STACKS="$STACKS node" +[ -f Cargo.toml ] && STACKS="$STACKS rust" +[ -f pyproject.toml ] && STACKS="$STACKS python" + +if [ -f Makefile ]; then MAKEFILE="Makefile"; elif [ -f makefile ]; then MAKEFILE="makefile"; fi +[ -n "$MAKEFILE" ] && STACKS="$STACKS make" + +STACKS="${STACKS# }" + +if [ -z "$STACKS" ] && [ -z "$DEPS_CMD$SERVER_CMD" ]; then + say "no package.json, Cargo.toml, pyproject.toml or Makefile in $ROOT." + say " Either this is not the project root, or the stack is one this script" + say " cannot detect." + die "set DEV_DEPS_CMD and DEV_SERVER_CMD (or the CONFIGURATION block) to teach it." +fi + +# package.json is JSON and this script has no JSON parser — deliberately, because +# the alternative is a regex that is wrong for exactly the files that matter. So +# node reads it, and when node is missing the scripts are reported as UNREADABLE +# rather than guessed at. That is honest: without node those scripts cannot be +# run either. +PKG_SCRIPTS="" +PKG_SCRIPTS_READ="" +PKG_SCRIPTS_WHY="" + +if has_word node "$STACKS"; then + if have node; then + # An empty scripts block and an unparseable package.json are different answers, + # so the read is what sets the flag, not the emptiness of the result. + if PKG_SCRIPTS=$(node -e 'const s=require("./package.json").scripts||{};process.stdout.write(Object.keys(s).join(" "))' 2>/dev/null); then + PKG_SCRIPTS_READ="yes" + else + # Why it could not be read is a separate fact from that it could not be + # read, and there are two reasons. Reporting the wrong one sends somebody + # to install a runtime they already have while a trailing comma in their + # package.json goes unmentioned. + PKG_SCRIPTS_WHY="node is installed but could not parse ./package.json — check it is valid JSON: node -e 'require(\"./package.json\")'" + fi + else + PKG_SCRIPTS_WHY="node is not on PATH" + fi +fi + +pkg_script_body() { + PKG_SCRIPT_NAME="$1" node -e 'const s=require("./package.json").scripts||{};process.stdout.write(s[process.env.PKG_SCRIPT_NAME]||"")' 2>/dev/null +} + +MAKE_TARGETS="" +if [ -n "$MAKEFILE" ]; then + # Plain grep rather than `make -qp`, which is the accurate way to list targets + # and evaluates the makefile to do it — running every `$(shell …)` in it before + # this script has checked a single prerequisite. Reading the file is worth the + # occasional missed target; a bootstrap that executes something during its own + # audit is not. + # + # The second grep drops `VAR:=x` and `VAR::=x`, which the first one reads as a + # target called VAR. That matters here because these names are not just + # printed: a variable named `dev` would become "make dev" and be run as this + # project's dev server. + MAKE_TARGETS=$(grep -hE '^[a-zA-Z0-9][a-zA-Z0-9_.-]*:' "$MAKEFILE" 2>/dev/null \ + | grep -vE '^[a-zA-Z0-9][a-zA-Z0-9_.-]*::?=' \ + | sed 's/:.*//' | sort -u | tr '\n' ' ') + MAKE_TARGETS="${MAKE_TARGETS% }" +fi + +# --------------------------------------------------------------------------- +# Plan: work out the command for each step before running any of them, so the +# audit knows which tools actually matter. A missing tool for a step that will +# not run is not a missing prerequisite. +# --------------------------------------------------------------------------- + +DEPS_ARGV=() +DEPS_DESC="" +DEPS_WHY="" +DEPS_FRESH_IF="" # file whose mtime, if older than the install marker, means skip +DEPS_MARKER="" + +plan_deps() { + if [ -n "$DEPS_CMD" ]; then + DEPS_ARGV=(bash -c "$DEPS_CMD") + DEPS_DESC="$DEPS_CMD" + DEPS_WHY="DEPS_CMD is configured" + return 0 + fi + + if has_word node "$STACKS"; then + DEPS_MARKER="node_modules" + if [ -f package-lock.json ]; then + DEPS_ARGV=(npm ci); DEPS_DESC="npm ci"; DEPS_FRESH_IF="package-lock.json" + DEPS_WHY="package-lock.json is here" + elif [ -f pnpm-lock.yaml ]; then + DEPS_ARGV=(pnpm install); DEPS_DESC="pnpm install"; DEPS_FRESH_IF="pnpm-lock.yaml" + DEPS_WHY="pnpm-lock.yaml is here" + elif [ -f yarn.lock ]; then + DEPS_ARGV=(yarn install); DEPS_DESC="yarn install"; DEPS_FRESH_IF="yarn.lock" + DEPS_WHY="yarn.lock is here" + elif [ -f bun.lockb ] || [ -f bun.lock ]; then + DEPS_ARGV=(bun install); DEPS_DESC="bun install" + if [ -f bun.lockb ]; then DEPS_FRESH_IF="bun.lockb"; else DEPS_FRESH_IF="bun.lock"; fi + DEPS_WHY="a bun lockfile is here" + else + DEPS_ARGV=(npm install); DEPS_DESC="npm install"; DEPS_FRESH_IF="package.json" + DEPS_WHY="package.json is here and no lockfile is" + fi + return 0 + fi + + if has_word python "$STACKS"; then + if [ -f uv.lock ]; then + DEPS_ARGV=(uv sync); DEPS_DESC="uv sync"; DEPS_MARKER=".venv"; DEPS_FRESH_IF="uv.lock" + DEPS_WHY="uv.lock is here" + elif [ -f poetry.lock ]; then + DEPS_ARGV=(poetry install); DEPS_DESC="poetry install"; DEPS_MARKER=".venv"; DEPS_FRESH_IF="poetry.lock" + DEPS_WHY="poetry.lock is here" + elif [ -f requirements.txt ]; then + DEPS_ARGV=(bash -c 'python3 -m venv .venv && .venv/bin/pip install -r requirements.txt') + DEPS_DESC="python3 -m venv .venv && .venv/bin/pip install -r requirements.txt" + DEPS_MARKER=".venv"; DEPS_FRESH_IF="requirements.txt" + DEPS_WHY="requirements.txt is here" + fi + [ -n "$DEPS_DESC" ] && return 0 + fi + + if has_word rust "$STACKS"; then + # No freshness check: `cargo fetch` is already a no-op when the lockfile is + # satisfied, and inventing a marker file to guess that would be a second, + # worse copy of state cargo already keeps correctly. + DEPS_ARGV=(cargo fetch); DEPS_DESC="cargo fetch"; DEPS_WHY="Cargo.toml is here" + return 0 + fi + + if [ -n "$MAKE_TARGETS" ]; then + local target + for target in deps setup install bootstrap; do + if has_word "$target" "$MAKE_TARGETS"; then + DEPS_ARGV=(make "$target"); DEPS_DESC="make $target" + DEPS_WHY="$MAKEFILE has a '$target' target" + return 0 + fi + done + fi + + return 1 +} + +MIGRATE_ARGV=() +MIGRATE_DESC="" +MIGRATE_WHY="" +MIGRATE_LOOKED="" + +plan_migrate() { + if [ -n "$MIGRATE_CMD" ]; then + MIGRATE_ARGV=(bash -c "$MIGRATE_CMD"); MIGRATE_DESC="$MIGRATE_CMD" + MIGRATE_WHY="MIGRATE_CMD is configured" + return 0 + fi + + local name + if [ -n "$PKG_SCRIPTS_READ" ]; then + for name in migrate db:migrate migrate:dev db:push; do + MIGRATE_LOOKED="$MIGRATE_LOOKED package.json:$name" + if has_word "$name" "$PKG_SCRIPTS"; then + MIGRATE_ARGV=(npm run "$name"); MIGRATE_DESC="npm run $name" + MIGRATE_WHY="package.json defines a '$name' script" + return 0 + fi + done + fi + + if [ -n "$MAKE_TARGETS" ]; then + for name in migrate db-migrate db.migrate; do + MIGRATE_LOOKED="$MIGRATE_LOOKED $MAKEFILE:$name" + if has_word "$name" "$MAKE_TARGETS"; then + MIGRATE_ARGV=(make "$name"); MIGRATE_DESC="make $name" + MIGRATE_WHY="$MAKEFILE has a '$name' target" + return 0 + fi + done + fi + + MIGRATE_LOOKED="${MIGRATE_LOOKED# }" + return 1 +} + +SERVER_ARGV=() +SERVER_DESC="" +SERVER_WHY="" +SERVER_LOOKED="" +SERVER_LOOP="" + +plan_server() { + if [ -n "$SERVER_CMD" ]; then + SERVER_ARGV=(bash -c "$SERVER_CMD"); SERVER_DESC="$SERVER_CMD" + SERVER_WHY="SERVER_CMD is configured" + return 0 + fi + + local name body + if [ -n "$PKG_SCRIPTS_READ" ]; then + for name in dev start serve; do + SERVER_LOOKED="$SERVER_LOOKED package.json:$name" + has_word "$name" "$PKG_SCRIPTS" || continue + + # The loop this template invites: somebody adds "dev": "bash scripts/dev.sh" + # to package.json, and step 4 then runs the bootstrap that runs step 4. It + # forks until something on the machine gives out, and the output is four + # identical banners a second, which reads as a hang rather than as a cycle. + # + # Matched against this copy's own filename as well as the template's, + # because the instructions at the top invite renaming it and a guard that + # only knows the name it used to have is not a guard. What this still + # cannot see — a Makefile target, a wrapper script — is caught at startup + # by the DEV_SH_ACTIVE marker instead. + body=$(pkg_script_body "$name") + case "$body" in + *"$SELF_NAME"*|*dev.sh*) + SERVER_LOOP="yes" + problem "package.json's '$name' script" \ + "it runs '$body', which re-enters this script — an infinite loop, and one that reads as a hang rather than as a cycle" \ + "rename it (\"setup\" is the conventional name for a bootstrap), or set DEV_SERVER_CMD to the command that actually starts the server" + return 1 ;; + esac + + SERVER_ARGV=(npm run "$name"); SERVER_DESC="npm run $name" + SERVER_WHY="package.json defines a '$name' script" + return 0 + done + fi + + if [ -n "$MAKE_TARGETS" ]; then + for name in dev run serve start; do + SERVER_LOOKED="$SERVER_LOOKED $MAKEFILE:$name" + if has_word "$name" "$MAKE_TARGETS"; then + SERVER_ARGV=(make "$name"); SERVER_DESC="make $name" + SERVER_WHY="$MAKEFILE has a '$name' target" + return 0 + fi + done + fi + + if has_word rust "$STACKS"; then + SERVER_LOOKED="$SERVER_LOOKED Cargo.toml" + SERVER_ARGV=(cargo run); SERVER_DESC="cargo run"; SERVER_WHY="Cargo.toml is here" + return 0 + fi + + SERVER_LOOKED="${SERVER_LOOKED# }" + return 1 +} + +HAVE_DEPS=""; plan_deps && HAVE_DEPS="yes" +HAVE_MIGRATE=""; plan_migrate && HAVE_MIGRATE="yes" +HAVE_SERVER=""; plan_server && HAVE_SERVER="yes" + +# --------------------------------------------------------------------------- +# Compose and the database. +# --------------------------------------------------------------------------- + +if [ -z "$COMPOSE_FILE" ]; then + for candidate in compose.yaml compose.yml docker-compose.yml docker-compose.yaml; do + if [ -f "$candidate" ]; then COMPOSE_FILE="$candidate"; break; fi + done +elif [ ! -f "$COMPOSE_FILE" ]; then + die "COMPOSE_FILE '$COMPOSE_FILE' does not exist. Fix the path or clear it to autodetect." +fi + +COMPOSE=() +if have docker && docker compose version >/dev/null 2>&1; then + COMPOSE=(docker compose) +elif have docker-compose; then + COMPOSE=(docker-compose) +fi + +compose_services() { + [ ${#COMPOSE[@]} -gt 0 ] || return 1 + "${COMPOSE[@]}" -f "$COMPOSE_FILE" config --services 2>/dev/null +} + +compose_running() { + [ ${#COMPOSE[@]} -gt 0 ] || return 1 + # Two spellings across compose generations. Neither is asked to be present: + # a failure here only costs the "already running" message, because `up -d` on + # a running service is a no-op anyway. + "${COMPOSE[@]}" -f "$COMPOSE_FILE" ps --services --status running 2>/dev/null \ + || "${COMPOSE[@]}" -f "$COMPOSE_FILE" ps --services --filter status=running 2>/dev/null +} + +# 0 open · 1 closed · 2 could not determine. +# +# The last one exists because of a specific trap: a bash built without network +# redirections reports /dev/tcp as a missing file, which is indistinguishable +# from a refused connection unless the message is read. Reporting that as "the +# database is down" sends somebody to debug a database that is fine. +probe_tcp() { + local host=$1 port=$2 out rc + + if have nc; then + # &1 /dev/tcp/"$1"/"$2"' dev-probe "$host" "$port" 2>&1); rc=$? + + # The status decides, not the emptiness of the output: a connection killed by + # `timeout` exits 124 having printed nothing at all, which is the same output + # a successful connection produces. + [ "$rc" -eq 0 ] && return 0 + [ "$rc" -eq 124 ] && return 1 + + case "$out" in + *"Connection refused"*|*"onnection timed out"*|*"No route to host"*|\ + *"Name or service not known"*|*"nodename nor servname"*) return 1 ;; + *) return 2 ;; + esac + fi + + # Without `timeout`, a /dev/tcp attempt against a filtered host hangs for the + # kernel's connect timeout, and a bootstrap that appears to freeze teaches less + # than one that says it could not look. + return 2 +} + +DB_HOST="" +DB_PORT="" +DB_ENDPOINT_WHY="" + +plan_db_endpoint() { + [ -n "$DB_URL_ENV" ] || { DB_ENDPOINT_WHY="DB_URL_ENV is not configured"; return 1; } + + # DB_URL_ENV holds a variable NAME. Put the URL there instead — the mistake the + # CONFIGURATION comment above warns about — and the indirect expansion below is + # bash's to complain about, not this script's: it prints `: + # invalid variable name` on stderr, and that value is a connection string with + # a password in it. So the name is checked first, and nothing about the value + # is ever echoed. + case "$DB_URL_ENV" in + *[!A-Za-z0-9_]*|[!A-Za-z_]*) + DB_ENDPOINT_WHY="DB_URL_ENV must be the NAME of a variable (DATABASE_URL), not its value. What it holds is not shown here, because a connection URL usually contains a password" + return 1 ;; + esac + + local url=${!DB_URL_ENV-} + if [ -z "$url" ]; then + DB_ENDPOINT_WHY="\$$DB_URL_ENV is unset" + return 1 + fi + + case "$url" in + *://*) : ;; + *) DB_ENDPOINT_WHY="\$$DB_URL_ENV is not a URL" ; return 1 ;; + esac + + local scheme rest hostport pathish + scheme=${url%%://*} + rest=${url#*://} + hostport=${rest%%/*} + hostport=${hostport%%\?*} + hostport=${hostport%%#*} + + case "$hostport" in + *@*) hostport=${hostport##*@} ;; # credentials, if any + *) + # No '@' before the first '/' — but if there is one after it, the cut above + # landed INSIDE the credentials, which is what an unencoded '/' in a + # password does. Base64 passwords are full of them. Left alone, every + # character now in $hostport is password: it gets printed in the report, + # and `appuser:99` out of `appuser:99/xY9zQ@dbhost` even parses as a + # host and a port, so the script resolves it — putting a fragment of the + # password into DNS. Neither probed nor printed. + pathish=${rest#*/} + pathish=${pathish%%\?*} + pathish=${pathish%%#*} + case "$pathish" in + *@*) + DB_ENDPOINT_WHY="\$$DB_URL_ENV could not be parsed: an '@' appears after the first '/', so where the credentials end cannot be known without guessing. Percent-encode any '/' in the password (%2F). Nothing from the URL is printed here, in case that guess would have been part of it" + return 1 ;; + esac ;; + esac + + case "$hostport" in + \[*\]*) # IPv6 literal + DB_HOST=${hostport#\[}; DB_HOST=${DB_HOST%%\]*} + case "$hostport" in *\]:*) DB_PORT=${hostport##*\]:} ;; esac ;; + *:*) + DB_HOST=${hostport%%:*}; DB_PORT=${hostport##*:} ;; + *) + DB_HOST=$hostport ;; + esac + + if [ -z "$DB_HOST" ]; then + # A unix-socket URL has no host, and probing a TCP port for one would be an + # answer about something that is not this connection. + DB_ENDPOINT_WHY="\$$DB_URL_ENV names no TCP host (a socket connection?)" + return 1 + fi + + # What a hostname or an IP literal can contain, and nothing else. This is the + # last gate before $DB_HOST becomes an argument to a network probe, and a host + # holding characters that cannot be in one means the parse went somewhere it + # should not have — most likely into the credentials. Refused, and not echoed. + case "$DB_HOST" in + *[!A-Za-z0-9.:_-]*) + DB_HOST=""; DB_PORT="" + DB_ENDPOINT_WHY="\$$DB_URL_ENV did not yield a usable hostname — what was found contains characters a host cannot. It is not printed here, in case it is part of a password" + return 1 ;; + esac + + if [ -z "$DB_PORT" ]; then + case "$scheme" in + postgres|postgresql) DB_PORT=5432 ;; + mysql|mariadb) DB_PORT=3306 ;; + redis|rediss) DB_PORT=6379 ;; + mongodb|mongodb+srv) DB_PORT=27017 ;; + *) DB_ENDPOINT_WHY="\$$DB_URL_ENV has no port and '$scheme' has no port this script knows" + return 1 ;; + esac + fi + + # Not quoted into the message: the commonest way this fails is a password with + # an unencoded ':' or '/' in it, which puts a piece of that password here. + case "$DB_PORT" in + ''|*[!0-9]*) + DB_HOST=""; DB_PORT="" + DB_ENDPOINT_WHY="\$$DB_URL_ENV did not yield a numeric port. If the password contains an unencoded ':' or '/', percent-encode it. The value found is not printed here, in case it is part of that password" + return 1 ;; + esac + + return 0 +} + +HAVE_DB_ENDPOINT="" +plan_db_endpoint && HAVE_DB_ENDPOINT="yes" + +# An unset DB_SERVICE is not a database this script may start. It is a question +# nobody has answered yet, and `up -d ""` would be this script guessing. +WANT_DB="" +if [ -z "$SKIP_DB" ] && [ -n "$DB_SERVICE" ] && [ "$DB_SERVICE" != "none" ] && [ -n "$COMPOSE_FILE" ]; then + WANT_DB="yes" +fi + +# --------------------------------------------------------------------------- +# The audit. Everything, before anything. +# --------------------------------------------------------------------------- + +audit() { + have git || note "git is not on PATH. Nothing here needs it, but the project's own tooling probably does." + + # --- step 1 tools ------------------------------------------------------- + if [ -z "$SKIP_DEPS" ]; then + if [ -n "$HAVE_DEPS" ]; then + case "${DEPS_ARGV[0]}" in + npm|npx) need_tool npm "installing dependencies ($DEPS_DESC)" "Node.js — https://nodejs.org, or nvm/fnm/asdf" ;; + pnpm) need_tool pnpm "installing dependencies ($DEPS_DESC)" "corepack enable pnpm — https://pnpm.io/installation" ;; + yarn) need_tool yarn "installing dependencies ($DEPS_DESC)" "corepack enable yarn — https://yarnpkg.com/getting-started/install" ;; + bun) need_tool bun "installing dependencies ($DEPS_DESC)" "https://bun.sh" ;; + cargo) need_tool cargo "fetching crates ($DEPS_DESC)" "https://rustup.rs" ;; + uv) need_tool uv "installing dependencies ($DEPS_DESC)" "https://docs.astral.sh/uv/getting-started/installation/" ;; + poetry) need_tool poetry "installing dependencies ($DEPS_DESC)" "https://python-poetry.org/docs/#installation" ;; + make) need_tool make "installing dependencies ($DEPS_DESC)" "build-essential (apt) or the Xcode command line tools" ;; + bash) : ;; + esac + # node is what runs npm's scripts, and a package.json project needs it even + # when the package manager is not npm. + if has_word node "$STACKS"; then + need_tool node "running this project at all (package.json is here)" "Node.js — https://nodejs.org, or nvm/fnm/asdf" + fi + if has_word python "$STACKS" && [ "${DEPS_ARGV[0]}" = "bash" ]; then + need_tool python3 "creating .venv ($DEPS_DESC)" "your distribution's python3 package, or pyenv" + fi + else + note "no dependency install step found for this project — skipping step 1. Set DEV_DEPS_CMD if there is one." + fi + fi + + # --- step 2: database --------------------------------------------------- + if [ -z "$SKIP_DB" ]; then + if [ "$DB_SERVICE" = "none" ]; then + note "DB_SERVICE is \"none\" — this project is declared to have no database." + elif [ -z "$DB_SERVICE" ] && [ -n "$COMPOSE_FILE" ]; then + local services="" + services=$(one_line "$(compose_services)") + problem "DB_SERVICE" \ + "$COMPOSE_FILE is here but nothing says which of its services is the database${services:+ (services: $services)}" \ + "set DB_SERVICE (or DEV_DB_SERVICE) to that service, or to \"none\" if this project has no database" + elif [ -z "$DB_SERVICE" ]; then + note "no compose file here and DB_SERVICE is unset — step 2 will do nothing. Set DB_SERVICE=\"none\" to record that on purpose." + elif [ -z "$COMPOSE_FILE" ]; then + problem "a compose file" \ + "DB_SERVICE is '$DB_SERVICE' but there is no compose file to start it from" \ + "add one, set DEV_COMPOSE_FILE, or set DB_SERVICE=\"none\" if the database runs elsewhere" + else + if need_tool docker "starting the '$DB_SERVICE' database service" "https://docs.docker.com/engine/install/"; then + if [ ${#COMPOSE[@]} -eq 0 ]; then + problem "docker compose" \ + "docker is installed but neither 'docker compose' nor 'docker-compose' works" \ + "install the Compose plugin — https://docs.docker.com/compose/install/" + else + local services + if services=$(compose_services); then + if ! has_line "$services" "$DB_SERVICE"; then + problem "$DB_SERVICE" \ + "no such service in $COMPOSE_FILE (it defines: $(one_line "$services"))" \ + "set DB_SERVICE to one of those, or to \"none\"" + fi + else + # A blocker rather than a note, because this step is going to run: an + # unparseable compose file and a stopped daemon look identical from + # here, and neither makes `up $DB_SERVICE` mean what it should. + problem "$COMPOSE_FILE" \ + "its services could not be listed, so nothing can confirm that '$DB_SERVICE' is in it" \ + "run '${COMPOSE[*]} -f $COMPOSE_FILE config --services' to see why — usually the docker daemon is not running" + fi + fi + fi + fi + fi + + # --- step 3 tools ------------------------------------------------------- + if [ -z "$SKIP_MIGRATE" ]; then + if [ -n "$HAVE_MIGRATE" ]; then + case "${MIGRATE_ARGV[0]}" in + npm) need_tool npm "running migrations ($MIGRATE_DESC)" "Node.js — https://nodejs.org" ;; + make) need_tool make "running migrations ($MIGRATE_DESC)" "build-essential (apt) or the Xcode command line tools" ;; + esac + else + note "no migrate step found — looked for ${MIGRATE_LOOKED:-nothing, because neither package.json nor a Makefile could be read}. Set DEV_MIGRATE_CMD if this project has one." + fi + fi + + # --- step 4 tools ------------------------------------------------------- + if [ -z "$SKIP_SERVER" ]; then + if [ -n "$HAVE_SERVER" ]; then + case "${SERVER_ARGV[0]}" in + npm) need_tool npm "starting the dev server ($SERVER_DESC)" "Node.js — https://nodejs.org" ;; + make) need_tool make "starting the dev server ($SERVER_DESC)" "build-essential (apt) or the Xcode command line tools" ;; + cargo) need_tool cargo "starting the dev server ($SERVER_DESC)" "https://rustup.rs" ;; + esac + elif [ -z "$SERVER_LOOP" ]; then + # Not raised when the loop guard already fired: that is the same fault, and + # a report that says one problem twice is a report you learn to skim. + problem "a way to start this project" \ + "nothing was found. Looked for: ${SERVER_LOOKED:-Makefile targets, Cargo.toml${PKG_SCRIPTS_WHY:+, and package.json scripts — which could not be read: $PKG_SCRIPTS_WHY}}" \ + "set DEV_SERVER_CMD (or SERVER_CMD in the CONFIGURATION block) to the command that starts it" + fi + + if has_word node "$STACKS" && [ -z "$PKG_SCRIPTS_READ" ]; then + unknown "package.json" "its scripts could not be read: ${PKG_SCRIPTS_WHY:-no reason was recorded} — fix that and run --check again to find out what this project actually defines" + fi + fi + + # --- environment -------------------------------------------------------- + local seen_env_file="" + if [ -n "$ENV_FILE" ]; then + seen_env_file="$ENV_FILE" + elif [ -f .env ]; then + seen_env_file=".env" + note ".env is here and this script did not source it (ENV_FILE is unset). Most dev servers load it themselves; see the header." + fi + + if [ ${#REQUIRED_ENV[@]} -eq 0 ]; then + note "REQUIRED_ENV is empty — no environment variables were checked, which is not the same as none being needed. List them to make this audit worth running." + else + local name value + for name in ${REQUIRED_ENV[@]+"${REQUIRED_ENV[@]}"}; do + value=${!name-} + [ -n "$value" ] && continue + + if [ -n "$seen_env_file" ] && grep -qE "^[[:space:]]*(export[[:space:]]+)?$name=" "$seen_env_file" 2>/dev/null; then + note "\$$name is unset in this shell but present in $seen_env_file. Fine if the dev server loads that file; not fine if it does not." + else + problem "\$$name" "listed in REQUIRED_ENV and unset" "export it, or add it to the env file this project loads" + fi + done + fi + + # --- database reachability --------------------------------------------- + # + # Only meaningful once there is somewhere to probe. When step 2 is going to + # start the database, an unreachable port is expected rather than wrong — the + # audit says so instead of reporting a failure it is about to fix. + if [ -n "$HAVE_DB_ENDPOINT" ]; then + probe_tcp "$DB_HOST" "$DB_PORT" + case $? in + 0) note "$DB_HOST:$DB_PORT answers (a port that answers, not a database that is ready — the migrate step is what proves that)." ;; + 1) if [ -n "$WANT_DB" ]; then + note "$DB_HOST:$DB_PORT is not answering yet — step 2 starts '$DB_SERVICE' and waits up to ${DB_WAIT}s for it." + else + problem "$DB_HOST:$DB_PORT" \ + "nothing is listening there, and no compose service is configured to start it" \ + "start the database yourself, or set DB_SERVICE so this script can" + fi ;; + *) unknown "$DB_HOST:$DB_PORT" "could not be probed: no nc that answered and no usable bash /dev/tcp — install netcat-openbsd (busybox's nc has no -z and cannot answer this) for a definite one" ;; + esac + else + note "database reachability was NOT checked: ${DB_ENDPOINT_WHY:-no endpoint could be worked out}." + fi +} + +audit + +# --------------------------------------------------------------------------- +# Report. All of it, once. +# --------------------------------------------------------------------------- + +# What each step will do, said before it is done — so --check and a real run +# print the same four lines and a wrong plan is caught by reading rather than by +# watching it happen. +plan_line() { + local number=$1 name=$2 skip_flag=$3 flag_name=$4 what=$5 + if [ -n "$skip_flag" ]; then + say " $number $name SKIPPED ($flag_name)" + else + say " $number $name ${what:-none found}" + fi +} + +db_plan="" +if [ "$DB_SERVICE" = "none" ]; then + db_plan="none — DB_SERVICE is \"none\"" +elif [ -z "$DB_SERVICE" ]; then + db_plan="undecided — DB_SERVICE is unset" +elif [ -n "$WANT_DB" ] && [ ${#COMPOSE[@]} -gt 0 ]; then + db_plan="${COMPOSE[*]} -f $COMPOSE_FILE up -d $DB_SERVICE" +elif [ -n "$WANT_DB" ]; then + db_plan="start '$DB_SERVICE' from $COMPOSE_FILE" +fi + +say "project: $ROOT" +say "stack: ${STACKS:-none detected}${MAKE_TARGETS:+ ($MAKEFILE targets: $MAKE_TARGETS)}" + +if has_word node "$STACKS"; then + if [ -n "$PKG_SCRIPTS_READ" ]; then + say "scripts: ${PKG_SCRIPTS:-package.json defines none}" + else + say "scripts: package.json could not be read — ${PKG_SCRIPTS_WHY:-no reason was recorded}" + fi +fi + +say "plan:" +plan_line 1 "deps " "$SKIP_DEPS" "--skip-deps" "$DEPS_DESC" +plan_line 2 "database" "$SKIP_DB" "--skip-db" "$db_plan" +plan_line 3 "migrate " "$SKIP_MIGRATE" "--skip-migrate" "$MIGRATE_DESC" +plan_line 4 "server " "$SKIP_SERVER" "--skip-server" "$SERVER_DESC" + +if [ ${#NOTES[@]} -gt 0 ]; then + say "" + for entry in ${NOTES[@]+"${NOTES[@]}"}; do + say "note: $entry" + done +fi + +if [ ${#UNKNOWNS[@]} -gt 0 ]; then + say "" + say "COULD NOT CHECK (${#UNKNOWNS[@]}) — not the same as ok:" + for entry in ${UNKNOWNS[@]+"${UNKNOWNS[@]}"}; do + IFS=$'\t' read -r what why <<<"$entry" + say " $what" + say " $why" + done +fi + +if [ ${#PROBLEMS[@]} -gt 0 ]; then + say "" + say "MISSING (${#PROBLEMS[@]}). All of them, so that fixing them is one pass:" + for entry in ${PROBLEMS[@]+"${PROBLEMS[@]}"}; do + IFS=$'\t' read -r what why how <<<"$entry" + say " $what" + say " needed for: $why" + say " fix: $how" + done + say "" + die "nothing was started. Fix the ${#PROBLEMS[@]} above and run this again." +fi + +if [ -n "$CHECK_ONLY" ]; then + if [ ${#UNKNOWNS[@]} -gt 0 ]; then + say "--check: no missing prerequisites, but ${#UNKNOWNS[@]} could not be checked." + exit 2 + fi + say "--check: everything this script knows to check is in place. Nothing was changed." + exit 0 +fi + +[ -n "$DRY_RUN" ] && say "--dry-run: nothing below is executed." + +# `run` is the single place a side effect happens, which is what makes --dry-run +# a property of the script rather than a promise repeated at four call sites. +run() { + # An empty argv is not a command that succeeded. `"$@"` with nothing in it + # runs nothing and returns 0, so a step whose plan came out empty would report + # itself done — the loudest possible version of absence dressed as a result. + [ "$#" -gt 0 ] || die "internal error: a step reached the point of running with an empty command line. Nothing was run; this is a bug in this script, not in your project." + + if [ -n "$DRY_RUN" ]; then + say " would run: $*" + return 0 + fi + say " $*" + "$@" +} + +# --------------------------------------------------------------------------- +# Step 1 — dependencies +# --------------------------------------------------------------------------- + +say "" +say "step 1/4: dependencies" + +if [ -n "$SKIP_DEPS" ]; then + say " skipped (--skip-deps)." + skipped "deps" "--skip-deps" +elif [ -z "$HAVE_DEPS" ]; then + say " nothing to do — no install step was found for this project." + skipped "deps" "no install step found" +elif [ -n "$DEPS_MARKER" ] && [ -n "$DEPS_FRESH_IF" ] && \ + [ -e "$DEPS_MARKER" ] && [ "$DEPS_MARKER" -nt "$DEPS_FRESH_IF" ]; then + # mtime, not a hash: the question is only "was this installed after the + # manifest last changed", and the answer being conservative in the wrong + # direction costs one redundant install rather than a wrong tree. + say " skipped — $DEPS_MARKER is newer than $DEPS_FRESH_IF, so it is assumed current." + say " That is an mtime, not a verified install: an interrupted one leaves" + say " a directory that is both newer and incomplete." + say " Delete $DEPS_MARKER to force a reinstall." + skipped "deps" "$DEPS_MARKER newer than $DEPS_FRESH_IF" +else + say " $DEPS_DESC ($DEPS_WHY)" + if ! run "${DEPS_ARGV[@]}"; then + die "dependency install failed. Nothing was started; fix the error above and run this again." + fi +fi + +# --------------------------------------------------------------------------- +# Step 2 — database +# +# Only the one named service is ever touched, and it is only ever started. See +# the header for why `up` with no argument and `down` are both absent. +# --------------------------------------------------------------------------- + +say "" +say "step 2/4: database" + +if [ -n "$SKIP_DB" ]; then + say " skipped (--skip-db)." + skipped "database" "--skip-db" +elif [ "$DB_SERVICE" = "none" ]; then + say " nothing to do — DB_SERVICE is \"none\"." + skipped "database" "DB_SERVICE=none" +elif [ -z "$WANT_DB" ]; then + say " nothing to do — no compose file here, and DB_SERVICE is unset." + skipped "database" "no compose file and no DB_SERVICE" +else + # The status decides whether this knows anything; the output decides what it + # knows. A successful listing with nothing in it means nothing is running, + # which is knowledge — only a failed listing is ignorance. + running="" + running_read="" + if running=$(compose_running); then running_read="yes"; fi + + if [ -n "$running_read" ] && has_line "$running" "$DB_SERVICE"; then + say " skipped — '$DB_SERVICE' is already running." + skipped "database" "'$DB_SERVICE' already running" + else + # `up -d` reconciles a service that is already up, so an unreadable list is + # not a reason to stop — but it is a reason not to claim the service was down. + [ -n "$running_read" ] || \ + say " could not tell whether '$DB_SERVICE' is already running; 'up -d' is a no-op if it is." + + if ! run "${COMPOSE[@]}" -f "$COMPOSE_FILE" up -d "$DB_SERVICE"; then + die "starting '$DB_SERVICE' failed. Nothing was migrated and no server was started." + fi + + if [ -z "$DRY_RUN" ] && [ -n "$HAVE_DB_ENDPOINT" ]; then + say " waiting for $DB_HOST:$DB_PORT (up to ${DB_WAIT}s)…" + waited=0 + state=2 + + # Probes before testing the clock, so DB_WAIT=0 still reports what it found + # rather than the "never looked" answer the initial value carries. + while :; do + probe_tcp "$DB_HOST" "$DB_PORT" + state=$? + [ "$state" -ne 1 ] && break + [ "$waited" -ge "$DB_WAIT" ] && break + sleep 1 + waited=$((waited + 1)) + done + + case "$state" in + 0) say " $DB_HOST:$DB_PORT answers after ${waited}s." ;; + 2) say " WARNING: the port could not be probed, so this waited for nothing" + say " definite. Continuing; the migrate step is the real test." ;; + *) say " $DB_HOST:$DB_PORT did not answer within ${DB_WAIT}s." + say " The container may still be starting — check" + say " '${COMPOSE[*]} -f $COMPOSE_FILE logs $DB_SERVICE'." + die "refusing to migrate against a database that has not answered." ;; + esac + elif [ -z "$DRY_RUN" ]; then + say " started. Readiness was NOT verified: ${DB_ENDPOINT_WHY:-no endpoint to probe}." + fi + fi +fi + +# --------------------------------------------------------------------------- +# Step 3 — migrations +# --------------------------------------------------------------------------- + +say "" +say "step 3/4: migrations" + +# The stamp lives inside the git directory: a marker file in the working tree is +# one `git add -A` from being committed, and a gitignore entry is a second thing +# to keep correct in every project this template lands in. +# +# Only when that git directory belongs to THIS project. Run from inside another +# checkout, `--git-dir` names that checkout, and writing there would record this +# project's migration state in somebody else's repository — the same confusion +# the project-root block at the top refuses to make. +STATE_DIR="" +if [ -n "$GIT_ROOT" ] && [ "$GIT_ROOT" = "$ROOT" ]; then + GIT_DIR=$(git rev-parse --git-dir 2>/dev/null) + [ -n "$GIT_DIR" ] && [ -d "$GIT_DIR" ] && STATE_DIR="$GIT_DIR/dev.sh" +fi +MIGRATE_STAMP="${STATE_DIR:+$STATE_DIR/migrated}" + +if [ -n "$SKIP_MIGRATE" ]; then + say " skipped (--skip-migrate)." + skipped "migrations" "--skip-migrate" +elif [ -z "$HAVE_MIGRATE" ]; then + say " nothing to do — no migrate step found (looked for ${MIGRATE_LOOKED:-nothing readable})." + skipped "migrations" "no migrate step found" +else + fresh="" + if [ -n "$MIGRATIONS_DIR" ] && [ -n "$MIGRATE_STAMP" ] && [ -f "$MIGRATE_STAMP" ]; then + if [ ! -d "$MIGRATIONS_DIR" ]; then + say " note: MIGRATIONS_DIR '$MIGRATIONS_DIR' does not exist, so nothing can be compared against the last run." + # The status again, not the emptiness: a find that cannot read the directory + # prints nothing, which is the same output as "nothing has changed" — and the + # cost of confusing them is a schema change that silently never runs. Not + # piped into `head` for the same reason: SIGPIPE would make success and + # failure return the same thing under `pipefail`. + elif newer=$(find "$MIGRATIONS_DIR" -type f -newer "$MIGRATE_STAMP" 2>/dev/null); then + [ -z "$newer" ] && fresh="yes" + else + say " note: could not list $MIGRATIONS_DIR, so this migrates rather than assuming nothing changed." + fi + fi + + if [ -n "$fresh" ]; then + say " skipped — nothing in $MIGRATIONS_DIR has changed since the last successful run." + say " Delete $MIGRATE_STAMP to force one." + skipped "migrations" "no new files in $MIGRATIONS_DIR" + else + if [ -z "$MIGRATIONS_DIR" ]; then + # Honest about why it cannot skip: the migration tool's own ledger lives in + # the database, and this script will not open a connection to read it. + say " running every time — MIGRATIONS_DIR is unset, so there is nothing to compare." + fi + + say " $MIGRATE_DESC ($MIGRATE_WHY)" + if ! run "${MIGRATE_ARGV[@]}"; then + die "migrations failed. No server was started." + fi + + if [ -z "$DRY_RUN" ] && [ -n "$MIGRATE_STAMP" ]; then + mkdir -p "$STATE_DIR" 2>/dev/null && : > "$MIGRATE_STAMP" 2>/dev/null \ + || say " note: could not write $MIGRATE_STAMP, so the next run will migrate again." + fi + fi +fi + +# --------------------------------------------------------------------------- +# Step 4 — the dev server +# --------------------------------------------------------------------------- + +# Recapped before the server takes the terminal, because that is the last moment +# anybody reads this output: a step that skipped itself for a good reason and a +# step that skipped itself because nothing was configured produce the same +# silence otherwise. +if [ ${#SKIPPED[@]} -gt 0 ]; then + say "" + say "steps skipped this run:" + for entry in ${SKIPPED[@]+"${SKIPPED[@]}"}; do + IFS=$'\t' read -r what why <<<"$entry" + say " $what — $why" + done +fi + +say "" +say "step 4/4: dev server" + +if [ -n "$SKIP_SERVER" ]; then + say " not started (--skip-server). Everything else is in place." + exit 0 +fi + +if [ -n "$DRY_RUN" ]; then + say " would exec: $SERVER_DESC ($SERVER_WHY)" + say "" + say "--dry-run: nothing above was executed." + exit 0 +fi + +say " $SERVER_DESC ($SERVER_WHY)" +say "" + +# `exec` with no arguments is not an error: it applies the (absent) redirections +# to this shell and carries on to the end of the file, which exits 0. A dev +# server that was never started would be reported as one that was. +[ ${#SERVER_ARGV[@]} -gt 0 ] || die "internal error: reached step 4 with no server command. Nothing was started; set DEV_SERVER_CMD." + +# exec, so Ctrl-C reaches the server rather than this script, and so nothing here +# outlives it holding a terminal the server thinks it owns. +exec "${SERVER_ARGV[@]}" diff --git a/docs/architecture/scripts/doc-claims.sh b/docs/architecture/scripts/doc-claims.sh new file mode 100755 index 0000000..3c8be57 --- /dev/null +++ b/docs/architecture/scripts/doc-claims.sh @@ -0,0 +1,274 @@ +#!/usr/bin/env bash +# +# Every file a document names must exist. +# +# ## The failure this catches +# +# Documentation makes claims about code, and the claims rot silently because +# nothing executes them. Three real instances, all found by hand in one +# afternoon on one repository: +# +# - A source comment stating that `tests/notice-security.test.ts` pinned a +# security rule. **That file had never existed.** The rule was real and +# enforced by nothing, and the sentence had been reassuring every reader who +# checked for two batches. +# - A reference manual whose migration table stopped at 0050 while the +# repository was at 0056. Six migrations behind, and every reader in between +# trusted it. +# - A manual promising a watchdog the code structurally could not fire. +# +# The first two are mechanically checkable and this checks them. The third is +# not — a claim about behaviour needs a person or a test — which is worth +# knowing about this script's limits: **it proves a path exists, never that the +# sentence around it is true.** +# +# ## What it looks at +# +# Anything that looks like a repository path inside backticks or a markdown +# link, in the files you point it at. A path is checked when it looks like one: +# it contains a slash or a known source extension, and it is not a URL, not a +# glob, and not obviously prose. +# +# bash scripts/doc-claims.sh # every tracked .md +# bash scripts/doc-claims.sh docs/ # one tree +# bash scripts/doc-claims.sh README.md # one file +# DOC_CLAIMS_ALSO_SRC=1 bash scripts/doc-claims.sh # also scan source comments +# DOC_CLAIMS_EXCLUDE='' bash scripts/doc-claims.sh # include forward-looking specs +# +# ## The inverse: is everything that exists written down? +# +# bash scripts/doc-claims.sh --covers src/db/migrations --in docs/MANUAL.md +# +# The check above asks whether every path a document *names* exists. This asks +# whether every file that exists is *named* — and it is the one that actually +# bit. A reference manual's migration table stopped at 0050 while the repository +# was at 0056: six rows missing, every path in the document perfectly valid, and +# no existence check can see an absent row. +# +# `docs/proposed/` is skipped by default: a specification naming the files it +# would create is not a claim that they exist. +# +# Exit codes: 0 every named path exists. 1 at least one does not. 2 nothing was +# scanned, which is not a pass — an empty run and a clean run must not look the +# same, for the same reason `verify.sh` refuses to report zero checks as green. +# --------------------------------------------------------------------------- + +set -uo pipefail + +# --covers --in : every file in must be mentioned in . +COVERS="" +COVERS_IN="" +ARGS=() + +while [ $# -gt 0 ]; do + case "$1" in + --covers) COVERS="${2:-}"; shift 2 ;; + --in) COVERS_IN="${2:-}"; shift 2 ;; + *) ARGS+=("$1"); shift ;; + esac +done + +set -- "${ARGS[@]+"${ARGS[@]}"}" + +cd "$(git rev-parse --show-toplevel 2>/dev/null)" || { + printf 'doc-claims: not a git repository.\n' >&2 + exit 2 +} + +say() { printf 'doc-claims: %s\n' "$*" >&2; } + +if [ -n "$COVERS" ]; then + [ -n "$COVERS_IN" ] || { say "--covers needs --in "; exit 2; } + [ -d "$COVERS" ] || { say "--covers: $COVERS is not a directory"; exit 2; } + [ -f "$COVERS_IN" ] || { say "--in: $COVERS_IN does not exist"; exit 2; } + + unmentioned=0 + total=0 + doc_body="$(cat "$COVERS_IN")" + + for entry in "$COVERS"/*; do + [ -e "$entry" ] || continue + + total=$((total + 1)) + name="$(basename "$entry")" + stem="${name%.*}" + prefix="${stem%%_*}" + + # Three spellings, because documents legitimately use all of them and + # demanding the longest reports a perfectly correct document as broken. + # The first draft of this checked only the full name and the stem, and + # reported 57 of 58 migrations missing from a table that lists every one — + # because that table writes `0057`, not `0057_digest_recipients.sql`. + # + # The prefix is only accepted when it is distinctive: a bare `0057` is, a + # bare `route` would not be, and matching on the latter would let a document + # pass by coincidence. + case "$doc_body" in + *"$name"*|*"$stem"*) continue ;; + esac + + case "$prefix" in + "$stem") ;; # no underscore; nothing new to try + [0-9][0-9][0-9]*|v[0-9]*) + case "$doc_body" in *"$prefix"*) continue ;; esac ;; + esac + + printf '%s: %s is not mentioned in %s\n' "$COVERS" "$name" "$COVERS_IN" + unmentioned=$((unmentioned + 1)) + done + + if [ "$total" -eq 0 ]; then + say "$COVERS is empty; nothing to cover." + exit 2 + fi + + if [ "$unmentioned" -gt 0 ]; then + say "$unmentioned of $total entr(ies) in $COVERS are absent from $COVERS_IN." + say "A list that is missing rows reads as complete — that is the whole" + say "problem with it." + exit 1 + fi + + say "all $total entr(ies) in $COVERS are mentioned in $COVERS_IN." + exit 0 +fi + +TARGETS=("$@") + +if [ ${#TARGETS[@]} -eq 0 ]; then + # Tracked files only. An untracked scratch document is not a claim this + # repository is making. + mapfile -t FILES < <(git ls-files '*.md') +else + mapfile -t FILES < <(git ls-files "${TARGETS[@]}" | grep -E '\.md$') +fi + +if [ -n "${DOC_CLAIMS_ALSO_SRC:-}" ]; then + # Source comments make the same claims and rot the same way — the missing + # test file above was named in a docblock, not in a document. + mapfile -t -O "${#FILES[@]}" FILES < <(git ls-files '*.ts' '*.tsx' '*.mjs' '*.js' '*.sh') +fi + +# Forward-looking documents are excluded, and this is the difference between a +# useful run and a noisy one. A specification naming the files it *would* create +# is not a rotted claim — it is the whole point of a specification. On the +# repository this was written against, every finding under `docs/proposed/` was +# of that kind and they outnumbered the real ones four to one. +# Two kinds of tree are excluded by default, and both for the same reason: the +# paths in them do not resolve against *this* repository. +# +# docs/proposed/ a specification naming the files it would create +# project-template/ a vendored copy of another project's docs, whose paths +# vendor/ resolve against whatever scaffolds from it +# +# The template copy alone accounted for 14 of 33 findings on the repository this +# was written against — every one of them a README correctly describing scripts +# that live in the template folder rather than here. +EXCLUDE="${DOC_CLAIMS_EXCLUDE:-docs/proposed/|project-template/|vendor/}" + +if [ -n "$EXCLUDE" ]; then + mapfile -t FILES < <(printf '%s\n' "${FILES[@]}" | grep -vE "$EXCLUDE" || true) +fi + +if [ ${#FILES[@]} -eq 0 ]; then + say "no files to scan." + exit 2 +fi + +# A token worth checking, and the filter matters more than the check. +# +# The first draft flagged 684 of 1142 tokens on a real repository — routes like +# `/agent/notices`, absolute paths like `~/.config/thing`, and bare filenames. +# A guard that is wrong six times in ten is one people learn to skip, so the +# rule is now deliberately narrow: **a token is only checked when its first +# segment is something that actually exists at the top of this repository.** +# +# That excludes URL routes (their first segment is empty), home-relative paths, +# and prose, and it means a genuinely missing path is reported against a +# background of near-silence. +# +# Bare filenames with a source extension — `api-handler.ts` — are resolved by +# basename anywhere in the tree, which is what a reader would do. +mapfile -t TOPLEVEL < <(git ls-tree --name-only HEAD) + +is_toplevel() { + local first="${1%%/*}" + local entry + for entry in "${TOPLEVEL[@]}"; do + [ "$first" = "$entry" ] && return 0 + done + return 1 +} + +worth_checking() { + case "$1" in + http://*|https://*|*@*) return 1 ;; # links and addresses + /*|~*|.*) return 1 ;; # routes, home paths, relative noise + *\**|*\?*|*'<'*|*'>'*|*' '*) return 1 ;; # globs, placeholders, prose + "") return 1 ;; + esac + + # A path into this repository. + is_toplevel "$1" && return 0 + + # Or a bare source filename, resolved by basename below. + case "$1" in + */*) return 1 ;; + *.ts|*.tsx|*.mjs|*.js|*.sh|*.py|*.sql|*.md) return 0 ;; + *) return 1 ;; + esac +} + +resolves() { + local token="$1" doc="$2" + + [ -e "$token" ] && return 0 + [ -e "$(dirname "$doc")/$token" ] && return 0 + + # Bare filename: does anything in the repository carry that basename? + case "$token" in + */*) return 1 ;; + *) git ls-files "*/$token" "$token" | grep -q . && return 0 ;; + esac + + return 1 +} + +missing=0 +checked=0 + +for file in "${FILES[@]}"; do + [ -f "$file" ] || continue + + # Backticked spans and markdown link targets. Line numbers and anchors are + # trimmed: `src/lib/foo.ts:42` and `foo.md#heading` name a real file. + while IFS= read -r raw; do + token="${raw%%:*}" + token="${token%%#*}" + token="${token%/}" + + worth_checking "$token" || continue + + checked=$((checked + 1)) + + resolves "$token" "$file" && continue + + printf '%s: names %s, which does not exist\n' "$file" "$token" + missing=$((missing + 1)) + done < <(grep -oE '`[^`]+`|\]\([^)]+\)' "$file" 2>/dev/null \ + | sed -E 's/^`//; s/`$//; s/^\]\(//; s/\)$//') +done + +if [ "$checked" -eq 0 ]; then + say "scanned ${#FILES[@]} file(s) and found no paths to check." + exit 2 +fi + +if [ "$missing" -gt 0 ]; then + say "$missing claimed path(s) do not exist, of $checked checked." + say "A document naming a file that is not there is worse than one saying" + say "nothing: somebody checked, and was reassured." + exit 1 +fi + +say "$checked claimed path(s), all present, across ${#FILES[@]} file(s)." diff --git a/docs/architecture/scripts/doc-triggers.py b/docs/architecture/scripts/doc-triggers.py new file mode 100755 index 0000000..81b1a55 --- /dev/null +++ b/docs/architecture/scripts/doc-triggers.py @@ -0,0 +1,211 @@ +#!/usr/bin/env python3 +"""Which documents does a change fire, while there is still time to update them? + + python3 scripts/doc-triggers.py # everything dirty right now + python3 scripts/doc-triggers.py --staged # what is staged + python3 scripts/doc-triggers.py [path…] # specific paths + python3 scripts/doc-triggers.py --range HEAD~3..HEAD + +## The failure this catches + +Every document in this tree carries a `Review trigger:` — the change that should +send somebody back to it — and `WORK_CYCLE.md` requires the triggered documents +to be updated *in the same commit as the code*. Deciding which fired means +reading every `Governs:` line and matching globs in your head, once per commit. +It is a check with no output of its own, so it is the one that gets skipped when +the code is already green, and the cost is invisible until a reader trusts a +document that stopped being true: a reference manual six migrations behind, and +every reader in between believing it. + +**This is not the doc-review check.** That one asks which baselined documents are +*overdue*, from committed history — a governed path with a commit newer than the +document's `Last reviewed` date. It is a different question and it can only be +asked after the fact. By the time a change is committed without its document, the +thing this catches has already happened. + +Exit status is always 0. This is a prompt, not a gate: a trigger asks a human +whether the prose is still true, and a check that failed the build for that would +be bumped past rather than read. + +## Two things it deliberately does not do + +**It does not read `Exempt:` declarations.** Those mark a *required* document as +deliberately absent from a repository, and a document that does not exist cannot +govern a path. There is nothing here for them to change. + +**It cannot fire a document whose `Governs:` is prose.** Several in this template +govern a subject rather than a set of paths — `GUARDS.md` governs "structural +tests, source-grep assertions, probes, and any check whose passing is taken as +evidence", which is the honest description and matches no glob. Those documents +are listed separately at the end rather than silently ignored, because a reader +who sees only the matched list would reasonably conclude the others were checked +and cleared. +""" +from __future__ import annotations + +import fnmatch +import pathlib +import re +import subprocess +import sys + +ROOT = pathlib.Path(__file__).resolve().parents[3] +DOCS = ROOT / "docs" + +# The status header is a fenced block immediately after the H1, and a long value +# wraps onto continuation lines indented by two spaces: +# +# Governs: structural tests, source-grep assertions, probes, and any check whose +# passing is taken as evidence +# +# A regex that reads one line per field — the obvious implementation — truncates +# at the wrap and silently under-reports, which for this tool means quietly +# failing to name a document that should have been updated. So fields are +# assembled line by line instead. +FIELD_START = re.compile(r"^(Status|Owner|Last reviewed|Governs|Review trigger):\s*(.*)$") +HEADER_LINES = 16 + + +def header_of(doc: pathlib.Path) -> dict[str, str]: + """The status header, with wrapped values joined.""" + try: + lines = doc.read_text(encoding="utf-8").splitlines()[:HEADER_LINES] + except (OSError, UnicodeDecodeError): + return {} + + fields: dict[str, str] = {} + current: str | None = None + for line in lines: + match = FIELD_START.match(line) + if match: + current = match.group(1) + fields[current] = match.group(2).strip() + elif current and line.startswith((" ", "\t")) and line.strip(): + fields[current] = f"{fields[current]} {line.strip()}".strip() + elif line.strip().startswith("```") and fields: + break + return fields + + +def looks_like_path(glob: str) -> bool: + """Whether a `Governs:` entry is a path pattern rather than a subject. + + The same test `doc-claims.sh` uses: a slash, a wildcard, or a file + extension. Prose about what a document is authoritative for will have none + of them, and must not be treated as a glob that simply never matches. + """ + glob = glob.strip() + if not glob or " " in glob and "/" not in glob and "*" not in glob: + return False + return "/" in glob or "*" in glob or re.search(r"\.\w{1,5}$", glob) is not None + + +def matches(path: str, glob: str) -> bool: + """Whether `path` is governed by `glob`. + + `**` means "and everything below", which `fnmatch` does not implement: its + `*` already crosses separators, so `a/**` never matches `a/b/c`. The two + forms these headers actually use are reduced to prefix tests. + """ + glob = glob.strip() + if not glob: + return False + if glob.endswith("/**"): + return path.startswith(glob[:-2]) or path == glob[:-3] + if "/**/" in glob: + head, tail = glob.split("/**/", 1) + return path.startswith(head + "/") and fnmatch.fnmatch(path, "*" + tail) + return fnmatch.fnmatch(path, glob) + + +def _git(*args: str) -> str: + result = subprocess.run( + ["git", *args], cwd=ROOT, capture_output=True, text=True, check=False + ) + return result.stdout + + +def changed_paths(argv: list[str]) -> tuple[list[str], str]: + if argv and argv[0] == "--staged": + return [p for p in _git("diff", "--cached", "--name-only").splitlines() if p], "staged" + if argv and argv[0] == "--range": + if len(argv) < 2: + sys.exit("doc-triggers: --range needs a revision range") + return [p for p in _git("diff", "--name-only", argv[1]).splitlines() if p], f"range {argv[1]}" + if argv: + return list(argv), "named paths" + + # Untracked files are included on purpose: a brand-new module is the case + # most likely to need a document and least likely to be remembered, and it + # is invisible to `git diff`. + paths = [] + for line in _git("status", "--porcelain").splitlines(): + if not line.strip(): + continue + path = line[3:].strip() + if " -> " in path: # a rename; the new name governs + path = path.split(" -> ", 1)[1] + paths.append(path.strip('"')) + return paths, "working tree" + + +def main() -> int: + if not DOCS.is_dir(): + print(f"doc-triggers: no docs/ directory at {DOCS}") + return 0 + + paths, source = changed_paths(sys.argv[1:]) + if not paths: + print(f"doc-triggers: nothing changed in the {source}.") + return 0 + + print(f"doc-triggers: {len(paths)} path(s) from the {source}\n") + + fired: list[tuple[str, list[str], str]] = [] + subject_only: list[str] = [] + + for doc in sorted(DOCS.rglob("*.md")): + rel = str(doc.relative_to(ROOT)) + header = header_of(doc) + governs = header.get("Governs", "") + if not governs: + continue + + globs = [g.strip() for g in governs.split(",") if g.strip()] + path_globs = [g for g in globs if looks_like_path(g)] + if not path_globs: + subject_only.append(rel) + continue + + hits = sorted({p for p in paths for g in path_globs if matches(p, g)}) + if hits: + fired.append((rel, hits, header.get("Review trigger", "(none stated)"))) + + for rel, hits, trigger in fired: + print(f"\033[1m{rel}\033[0m") + for hit in hits[:6]: + print(f" {hit}") + if len(hits) > 6: + print(f" … and {len(hits) - 6} more") + print(f" trigger: {trigger}\n") + + if fired: + print(f"{len(fired)} document(s) govern something in this change.") + print("Read each trigger and decide — the rule is to update them in the") + print("SAME commit as the code, not afterwards.") + else: + print("No document's Governs matched these paths. Worth a second look if") + print("this change added a module, a migration, or a new boundary — an") + print("unmatched path can also mean no document claims that area yet.") + + if subject_only: + print("\nNot checked here — these govern a subject rather than paths, so") + print("no change can fire them mechanically. Judge them yourself:") + for rel in subject_only: + print(f" {rel}") + + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/docs/architecture/scripts/duplication.py b/docs/architecture/scripts/duplication.py new file mode 100755 index 0000000..7db8c96 --- /dev/null +++ b/docs/architecture/scripts/duplication.py @@ -0,0 +1,189 @@ +#!/usr/bin/env python3 +"""Find code that exists twice. + +## Why this is a script and not a code review + +Duplication is the thing a reader is worst at. It is invisible unless both +copies are on screen together, and by the time there are three copies nobody +remembers there were ever two. On the repository this was written against, five +separate instances were found by reading in a single afternoon — a four-part +renderer copied into two components, six inlined copies of one ternary, a path +convention written out in four places, a settings key as a literal in three, and +two functions answering the same question with different precedence. + +Every one was found by accident. A twenty-line heuristic finds fifty. + +## What it reports, and what it deliberately does not + +A run of normalised lines that appears in more than one place. Comments, blank +lines and imports are stripped first, so a shared licence header or a block of +imports is not a finding. + +It is tuned for **near-silence**, per `GUARDS.md` rule 5: a check that is wrong +six times in ten is one people learn to skip. The defaults require a long run +and a substantial amount of text, so what survives is worth looking at. Loosen +them deliberately when hunting, not by default. + +**It cannot tell you whether duplication is wrong.** Two similar-looking blocks +sometimes answer different questions and should stay apart — `notice-sections` +makes that argument about a producer's shape and a wire's shape. This finds +candidates; a person decides. + + python3 scripts/duplication.py # default roots + python3 scripts/duplication.py src lib # specific roots + python3 scripts/duplication.py --lines 6 --chars 250 # hunt harder + python3 scripts/duplication.py --ignore vendor/ # skip a tree + +Exit codes: 0 nothing above the threshold. 1 duplication found. 2 nothing was +scanned, which is not a pass — an empty run and a clean run must not look the +same. +""" + +from __future__ import annotations + +import argparse +import hashlib +import os +import re +import sys +from collections import defaultdict + +DEFAULT_ROOTS = ["src", "lib", "app", "scripts", "notices"] +DEFAULT_EXTS = (".ts", ".tsx", ".js", ".jsx", ".mjs", ".py", ".go", ".rs") +SKIP_DIRS = {"node_modules", ".next", ".git", "dist", "build", "__pycache__", "vendor"} + +# A line that carries no logic. Stripping these is what stops a shared import +# block or a licence header being reported as duplication. +NOISE = re.compile(r"^\s*(//|#|\*|/\*|\*/|import\s|from\s+['\"]|use\s|require\()") + + +def normalised(path: str) -> list[tuple[int, str]]: + """Real lines, with their original numbers, whitespace flattened.""" + out: list[tuple[int, str]] = [] + + try: + with open(path, encoding="utf8", errors="ignore") as handle: + for number, raw in enumerate(handle, 1): + line = raw.strip() + + if not line or NOISE.match(line): + continue + + # Punctuation-only lines — `}`, `});`, `)` — are structure, not + # logic. Left in they make every closing brace look duplicated. + if len(line) < 4: + continue + + out.append((number, re.sub(r"\s+", " ", line))) + except OSError: + return [] + + return out + + +def collect(roots: list[str], exts: tuple[str, ...], ignore: list[str]) -> list[str]: + files: list[str] = [] + + for root in roots: + if not os.path.isdir(root): + continue + + for base, dirs, names in os.walk(root): + dirs[:] = [d for d in dirs if d not in SKIP_DIRS] + + for name in names: + if not name.endswith(exts): + continue + + path = os.path.join(base, name) + + if any(pattern in path for pattern in ignore): + continue + + files.append(path) + + return files + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__.split("\n")[0]) + parser.add_argument("roots", nargs="*", default=None) + parser.add_argument("--lines", type=int, default=8, help="run length (default 8)") + parser.add_argument("--chars", type=int, default=320, help="minimum text (default 320)") + parser.add_argument("--ignore", action="append", default=[], help="substring to skip") + parser.add_argument("--same-file", action="store_true", help="also report within one file") + args = parser.parse_args() + + roots = args.roots or DEFAULT_ROOTS + files = collect(roots, DEFAULT_EXTS, args.ignore) + + if not files: + print(f"duplication: nothing to scan in {', '.join(roots)}.", file=sys.stderr) + return 2 + + blocks: dict[str, list[tuple[str, int]]] = defaultdict(list) + + for path in files: + lines = normalised(path) + + for index in range(len(lines) - args.lines + 1): + window = lines[index : index + args.lines] + text = " ".join(text for _, text in window) + + if len(text) < args.chars: + continue + + digest = hashlib.sha1(text.encode("utf8")).hexdigest() + blocks[digest].append((path, window[0][0])) + + findings = [] + + for sites in blocks.values(): + distinct_files = {path for path, _ in sites} + + if len(distinct_files) > 1 or (args.same_file and len(sites) > 1): + findings.append(sites) + + if not findings: + print( + f"duplication: none at {args.lines}+ lines / {args.chars}+ chars " + f"across {len(files)} file(s).", + file=sys.stderr, + ) + return 0 + + # Overlapping windows report the same duplication many times. Keep one + # finding per set of files, at its first site — otherwise a 40-line copy + # produces 33 findings and reads as a disaster. + seen: set[tuple[str, ...]] = set() + shown = 0 + + for sites in sorted(findings, key=lambda s: -len(s)): + key = tuple(sorted({path for path, _ in sites})) + + if key in seen: + continue + + seen.add(key) + shown += 1 + print("duplicated block:") + + for path, line in sorted(sites)[:6]: + print(f" {path}:{line}") + + print( + f"\nduplication: {shown} duplicated region(s) across {len(files)} file(s), " + f"at {args.lines}+ lines and {args.chars}+ characters.", + file=sys.stderr, + ) + print( + "Not every one is wrong — two blocks can answer different questions and " + "belong apart. These are candidates; you decide.", + file=sys.stderr, + ) + + return 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/docs/architecture/scripts/forgejo-issue.py b/docs/architecture/scripts/forgejo-issue.py new file mode 100755 index 0000000..d3d8dfc --- /dev/null +++ b/docs/architecture/scripts/forgejo-issue.py @@ -0,0 +1,586 @@ +#!/usr/bin/env python3 +"""Post, list and close Forgejo issues in the convention these projects use. + +Exists because every one of the rules below was learned by getting it wrong +once. The script is the enforcement; the skill is the explanation. + + create file an issue, refusing one that has no `Verify:` line or that + duplicates an existing title + batch file several from a JSON file, skipping ones already there + list open issues, grouped by milestone + close close with an evidence comment — "Done" is not a close + labels / milestones — what exists, with the ids the API wants + +Run any subcommand with --dry-run to see the payload and change nothing. +""" + +from __future__ import annotations + +import argparse +import json +import os +import re +import subprocess +import sys +import urllib.error +import urllib.parse +import urllib.request + +# Where the credentials live, when they are not already in the environment. +# +# Keep this file OUTSIDE the repository. A token in a file the repo can see is +# a token one `git add -A` away from being published — the same argument +# `release.sh` makes about its registry env. +ENV_FILE = os.environ.get("FORGEJO_ENV_FILE", os.path.expanduser("~/.forgejo.env")) + +# Cloudflare fronts the Forgejo instance and 1010-blocks Python's default +# urllib User-Agent (browser_signature_banned). Every call fails with a +# Cloudflare HTML body that looks nothing like a Forgejo error. Do not remove. +USER_AGENT = "curl/8.5.0" + +SEVERITY = ("P0", "P1", "P2", "release-blocker") + + +# ── plumbing ───────────────────────────────────────────────────────────────── + + +def die(msg: str, code: int = 1): + print(f"error: {msg}", file=sys.stderr) + sys.exit(code) + + +def load_env() -> tuple[str, str]: + """Read host + token. Never print the token; it is not registry-scoped — + it carries admin/push/pull over the whole API.""" + host = os.environ.get("FORGEJO_REGISTRY") + token = os.environ.get("FORGEJO_REGISTRY_TOKEN") + if not (host and token): + try: + with open(ENV_FILE, encoding="utf-8") as fh: + for line in fh: + line = line.strip() + if not line or line.startswith("#") or "=" not in line: + continue + k, v = line.split("=", 1) + v = v.strip().strip("'\"") + if k.strip() == "FORGEJO_REGISTRY" and not host: + host = v + elif k.strip() == "FORGEJO_REGISTRY_TOKEN" and not token: + token = v + except FileNotFoundError: + pass + if not (host and token): + die( + f"FORGEJO_REGISTRY / FORGEJO_REGISTRY_TOKEN not in the environment " + f"or {ENV_FILE}.\n" + f"Set FORGEJO_ENV_FILE to point somewhere else, or export both." + ) + return host, token + + +def detect_repo() -> str | None: + """owner/name from the git remote of the current directory.""" + try: + url = subprocess.run( + ["git", "remote", "get-url", "origin"], + capture_output=True, text=True, check=True, + ).stdout.strip() + except (subprocess.CalledProcessError, FileNotFoundError): + return None + m = re.search(r"[:/]([^/:]+)/([^/]+?)(?:\.git)?$", url) + return f"{m.group(1)}/{m.group(2)}" if m else None + + +class Api: + def __init__(self, host: str, token: str, repo: str, dry_run: bool = False): + self.base = f"https://{host}/api/v1" + self.token = token + self.repo = repo + self.dry_run = dry_run + + def _call(self, method: str, path: str, body=None, params=None): + url = f"{self.base}{path}" + if params: + url += "?" + urllib.parse.urlencode(params) + data = json.dumps(body).encode() if body is not None else None + req = urllib.request.Request(url, data=data, method=method) + req.add_header("Authorization", f"token {self.token}") + req.add_header("User-Agent", USER_AGENT) + if data: + req.add_header("Content-Type", "application/json") + try: + with urllib.request.urlopen(req, timeout=30) as resp: + raw = resp.read().decode() + return json.loads(raw) if raw.strip() else None + except urllib.error.HTTPError as e: + detail = e.read().decode()[:400] + if " dict[str, int]: + return {l["name"]: l["id"] + for l in (self.get(f"/repos/{self.repo}/labels", + {"limit": 100}) or [])} + + def milestones(self) -> dict[str, int]: + out = {} + for state in ("open", "closed"): + for m in (self.get(f"/repos/{self.repo}/milestones", + {"state": state, "limit": 100}) or []): + out[m["title"]] = m["id"] + return out + + def milestones_full(self, state="all") -> list[dict]: + states = ("open", "closed") if state == "all" else (state,) + out = [] + for s in states: + out.extend(self.get(f"/repos/{self.repo}/milestones", + {"state": s, "limit": 100}) or []) + return out + + def current_milestone(self) -> dict | None: + """The first OPEN milestone that still has open issues. + + 'First' is the order Forgejo returns, which is creation order — NOT a + numeric sort by title. A milestone created last therefore cannot become + current while an earlier one still has open issues, which is the lever + for filing work that must not disturb the card. + """ + openms = self.milestones_full("open") + for m in openms: + if m["open_issues"] > 0: + return m + return openms[0] if openms else None + + +# ── convention checks ──────────────────────────────────────────────────────── + + +def check_verify_line(body: str, title: str) -> str: + """Every issue ends with a Verify: line stating the acceptance check. + A deliverable nobody can re-test cannot be closed, so it must not be filed.""" + lines = [l for l in body.strip().splitlines() if l.strip()] + if not any(l.strip().startswith("Verify:") for l in lines): + die(f'"{title}" has no `Verify:` line. State the acceptance check — a\n' + " finding that cannot be re-tested cannot be closed.") + if not lines[-1].strip().startswith("Verify:"): + print(f'warning: "{title}" has a Verify: line but it is not last', + file=sys.stderr) + return body + + +def resolve_labels(names, available: dict[str, int]) -> list[int]: + """Names → ids, failing loudly. A typo'd severity label is reported by the + Command Center as *not adopted*, not as zero defects — silently dropping it + would hide the whole repo's defect count.""" + ids = [] + for n in names: + # A severity label that differs only in case is the dangerous one: it + # looks right in the UI and is invisible to a query by exact name. + if n not in available: + near = [a for a in available if a.lower() == n.lower()] + if near: + die(f"label {n!r} does not exist, but {near[0]!r} does. " + "Names are matched exactly — use that one.") + die(f"label {n!r} does not exist. " + f"Available: {', '.join(sorted(available)) or '(none)'}") + if n.upper() in {s.upper() for s in SEVERITY} and n not in SEVERITY: + die(f"severity label must be spelled exactly one of {SEVERITY}, " + f"got {n!r}") + ids.append(available[n]) + return ids + + +def find_duplicate(title: str, existing: list) -> dict | None: + t = title.strip().lower() + for i in existing: + if i["title"].strip().lower() == t: + return i + return None + + +def validate_milestone_title(title: str) -> None: + """Refuse a comma. `milestones=` takes a comma-separated list of names, so a + title containing one splits into names that do not exist, the filter DROPS, + and the query returns the newest open issue in the WHOLE repository — which + the card then presents as that milestone's next action. Percent-encoding does + not save it; Forgejo decodes before splitting. Measured on null/fruit-fall, + where `0.3.7 Logo, Icons & Branding` was renamed for exactly this reason.""" + if "," in title: + die(f"milestone title contains a comma: {title!r}\n" + " That silently breaks the `milestones=` filter and makes the project\n" + " card show the wrong next action. Rename it without the comma.") + first = title.strip().split()[0] if title.strip() else "" + if first and first.replace("v", "", 1).replace(".", "").isdigit(): + rest = title.strip()[len(first):].strip() + if rest: + print(f" note: the dashboard phase will show just {first!r} — a title " + f"starting with a\n version token has the rest dropped. Put a " + f"word first to keep it whole.") + + +_warned_current: set[str] = set() + + +def _warn_if_current(api: Api, milestone: str) -> None: + """The dashboard's next action is the NEWEST open issue in the current + milestone — not the highest priority; priority labels have no influence at + all. Filing a routine item into the milestone the team is working on + therefore replaces what the card shows.""" + if milestone in _warned_current: + return + _warned_current.add(milestone) + cur = api.current_milestone() + if cur and cur["title"] == milestone: + print(f" warning: {milestone!r} is the CURRENT milestone. The next action on " + "the project\n card is the NEWEST open issue in it, ignoring " + "priority — so this will\n replace whatever is shown there now. " + "To avoid that, file into a\n milestone created later; order is " + "creation order, not title order.") + + +# ── commands ───────────────────────────────────────────────────────────────── + + +def cmd_check(api: Api, args): + """The probes that prove the card is not lying. Run before and after filing. + + Everything here exists because Forgejo's filters FAIL OPEN: given a value + they cannot match they ignore the filter and return the unfiltered list, with + no error. A query returning plausible results is not evidence it filtered. + """ + print(f"tracker health — {api.repo}\n") + problems = 0 + + labels = api.labels() + missing = [n for n in SEVERITY if n not in labels] + if missing: + problems += 1 + print(f" FAIL severity labels missing: {', '.join(missing)}") + print(" a query for a label that does not exist matches EVERYTHING") + else: + print(" ok all four severity labels defined") + + openi = api.issues(state="open") + blockers = [i for i in openi + if any(l["name"] == "release-blocker" for l in i["labels"])] + if blockers: + print(f" WARN {len(blockers)} open release-blocker — takes over the whole card:") + for i in blockers[:5]: + print(f" #{i['number']} {i['title'][:58]}") + else: + print(" ok no release-blocker hijacking the card") + + orphans = [i for i in openi if not i.get("milestone")] + if orphans: + problems += 1 + print(f" FAIL {len(orphans)} open issue(s) with no milestone — invisible on the card:") + for i in orphans[:5]: + print(f" #{i['number']} {i['title'][:58]}") + else: + print(f" ok no orphan issues ({len(openi)} open)") + + openms = api.milestones_full("open") + commas = [m["title"] for m in openms if "," in m["title"]] + if commas: + problems += 1 + print(f" FAIL comma in milestone title — breaks the filter: {commas}") + else: + print(" ok no comma in any open milestone title") + + empty = [m["title"] for m in openms if m["open_issues"] == 0] + if empty: + print(f" WARN {len(empty)} open milestone(s) with nothing in them — the card") + print(f" will read 'Close milestone …': {empty[:3]}") + else: + print(" ok no empty open milestones") + + cur = api.current_milestone() + if cur: + first = cur["title"].strip().split()[0] + tok = first.replace("v", "", 1).replace(".", "") + print(f"\n current milestone : {cur['title']!r}") + print(f" phase shown : {(first if tok.isdigit() else cur['title'].strip())!r}") + nxt = api.get(f"/repos/{api.repo}/issues", + {"type": "issues", "state": "open", "limit": 1, + "milestones": cur["title"]}) or [] + if nxt: + i = nxt[0] + in_ms = (i.get("milestone") or {}).get("title") + if in_ms != cur["title"]: + problems += 1 + print(f" next action : {i['title']!r}") + print(" ^ FILTER DROPPED — that issue is in " + f"{in_ms!r}.\n The card is showing a " + "wrong next action.") + else: + print(f" next action : {i['title']!r}") + else: + print("\n no open milestones — the phase would be 'release'") + + print(f"\n{'PROBLEMS: ' + str(problems) if problems else 'All checks passed.'}") + return 1 if problems else 0 + + +def cmd_milestone(api: Api, args): + """Create a milestone. No due date is ever set — a due date means a + commitment, and an invented one is worse than none.""" + title = args.title.strip() + validate_milestone_title(title) + if title in api.milestones(): + print(f"exists {title!r} — not creating a second one") + return + desc = args.description or "" + if args.description_file: + desc = open(args.description_file, encoding="utf-8").read() + if not desc.strip(): + print(" note: no description. It should say what the batch is for and how " + "anybody\n will know it landed.") + m = api.post(f"/repos/{api.repo}/milestones", + {"title": title, "description": desc}) + print(f"created milestone {m.get('title', title)!r} (id {m.get('id', '?')})") + + +def create_one(api: Api, spec: dict, labels_map, ms_map, existing, + allow_dup=False) -> dict | None: + title = spec["title"].strip() + body = check_verify_line(spec.get("body", ""), title) + + dup = find_duplicate(title, existing) + if dup and not allow_dup: + print(f"skip #{dup['number']} already titled {title!r} " + f"({dup['state']}) — creates are not idempotent, so this is a skip " + f"not an error") + return None + + payload = {"title": title, "body": body} + if spec.get("labels"): + payload["labels"] = resolve_labels(spec["labels"], labels_map) + if "release-blocker" in spec["labels"]: + print(" warning: release-blocker does NOT filter by milestone, so one " + "stray label\n takes over the phase and next action for " + "the entire project. It means\n *nothing else can " + "proceed* — it is not a synonym for important; that is P1.") + if spec.get("milestone"): + m = spec["milestone"] + if m not in ms_map: + die(f"milestone {m!r} does not exist. Available: " + f"{', '.join(sorted(ms_map)) or '(none)'}") + payload["milestone"] = ms_map[m] + _warn_if_current(api, m) + else: + print(f" warning: {title[:48]!r} has no milestone — it can never become the " + "next action\n and never appears anywhere on the project card.") + + d = api.post(f"/repos/{api.repo}/issues", payload) + names = ",".join(l["name"] for l in d.get("labels", [])) + mile = (d.get("milestone") or {}).get("title", "—") + print(f"filed #{d['number']} [{names}] {d['title']}" + + (f" → {mile}" if mile != "—" else "")) + return d + + +def cmd_create(api: Api, args): + labels_map, ms_map = api.labels(), api.milestones() + existing = api.issues(state="all") + body = args.body + if args.body_file: + body = open(args.body_file, encoding="utf-8").read() + create_one(api, {"title": args.title, "body": body or "", + "labels": args.label, "milestone": args.milestone}, + labels_map, ms_map, existing, args.allow_duplicate) + + +def cmd_batch(api: Api, args): + specs = json.load(open(args.file, encoding="utf-8")) + if isinstance(specs, dict): + specs = specs.get("issues", []) + if not isinstance(specs, list): + die("batch file must be a JSON list, or an object with an 'issues' list") + labels_map, ms_map = api.labels(), api.milestones() + existing = api.issues(state="all") + filed = 0 + for spec in specs: + d = create_one(api, spec, labels_map, ms_map, existing, + args.allow_duplicate) + if d: + filed += 1 + existing.append({"number": d["number"], "title": d["title"], + "state": "open"}) + print(f"\n{filed} filed, {len(specs) - filed} skipped") + + +def cmd_list(api: Api, args): + issues = api.issues(state=args.state) + groups: dict[str, list] = {} + for i in issues: + groups.setdefault((i.get("milestone") or {}).get("title", + "(no milestone)"), + []).append(i) + for m in sorted(groups): + print(f"\n### {m}") + for i in sorted(groups[m], key=lambda x: x["number"]): + names = ",".join(l["name"] for l in i["labels"]) + print(f" #{i['number']:<4} [{names}] {i['title']}") + print(f"\n{len(issues)} {args.state} issue(s) — pull requests excluded") + + +def cmd_close(api: Api, args): + """Close with the evidence that proves it: a path, a symbol, a test name, + or the command that shows it. 'Done' is not a close.""" + ev = args.evidence.strip() + if len(ev) < 15: + die("evidence too thin. Give a path, a symbol, a test name, or the " + "command that proves it — 'Done' is not a close.") + api.post(f"/repos/{api.repo}/issues/{args.number}/comments", {"body": ev}) + api.patch(f"/repos/{api.repo}/issues/{args.number}", {"state": "closed"}) + print(f"closed #{args.number} with evidence") + print("note: prefer `closes #N` in the commit that does the work — the " + "tracker then records who and when from the thing that happened.") + + +def cmd_labels(api: Api, args): + labels = api.labels() + print("severity (exact names — queried by name by the Command Center):") + for s in SEVERITY: + print(f" {'✓' if s in labels else '✗ MISSING'} {s}" + + (f" id={labels[s]}" if s in labels else "")) + print("\nother:") + for n, i in sorted(labels.items()): + if n not in SEVERITY: + print(f" {n} id={i}") + + +def cmd_milestones(api: Api, args): + for state in ("open", "closed"): + ms = api.get(f"/repos/{api.repo}/milestones", + {"state": state, "limit": 100}) or [] + if ms: + print(f"\n{state}:") + for m in ms: + print(f" [{m['id']}] {m['title']} — open {m['open_issues']} / " + f"closed {m['closed_issues']}") + + +def main(): + # --repo and --dry-run are accepted on BOTH sides of the subcommand. Putting + # them only on the top-level parser means `… create "T" --dry-run` — the + # natural way to type it, and the position that matters most — dies with an + # argparse usage error instead of previewing. A safety flag that is easy to + # put in the wrong place is a safety flag that gets left off. + # default=SUPPRESS is load-bearing, not tidiness. With a normal default the + # subparser re-defines the same dest and argparse writes its default over + # whatever the top-level parser already parsed — so `--repo X create …` + # silently became repo=None and `--dry-run create …` silently became False. + # A --dry-run that quietly turns itself off is the worst possible bug in a + # tool whose job is writing to a live tracker. SUPPRESS leaves the attribute + # unset unless it was actually given, so neither position clobbers the other. + common = argparse.ArgumentParser(add_help=False) + common.add_argument("--repo", default=argparse.SUPPRESS, + help="owner/name (default: from git remote)") + common.add_argument("--dry-run", action="store_true", + default=argparse.SUPPRESS, + help="print payloads, change nothing") + + p = argparse.ArgumentParser( + parents=[common], + description="File Forgejo issues in the tracker convention.", + epilog="Every open issue is a denominator. Do not pad the tracker.") + sub = p.add_subparsers(dest="cmd", required=True) + + c = sub.add_parser("create", parents=[common], help="file one issue") + c.add_argument("title") + c.add_argument("--body", help="issue body; must end with a Verify: line") + c.add_argument("--body-file", help="read the body from a file") + c.add_argument("--label", action="append", default=[], + help="label name, repeatable") + c.add_argument("--milestone", help="milestone title") + c.add_argument("--allow-duplicate", action="store_true") + c.set_defaults(fn=cmd_create) + + b = sub.add_parser("batch", parents=[common], help="file several from a JSON file") + b.add_argument("file") + b.add_argument("--allow-duplicate", action="store_true") + b.set_defaults(fn=cmd_batch) + + l = sub.add_parser("list", parents=[common], help="open issues by milestone") + l.add_argument("--state", default="open", + choices=["open", "closed", "all"]) + l.set_defaults(fn=cmd_list) + + x = sub.add_parser("close", parents=[common], help="close with an evidence comment") + x.add_argument("number", type=int) + x.add_argument("evidence", help="what was checked and where") + x.set_defaults(fn=cmd_close) + + m = sub.add_parser("milestone", parents=[common], help="create a milestone (batch)") + m.add_argument("title") + m.add_argument("--description") + m.add_argument("--description-file") + m.set_defaults(fn=cmd_milestone) + + sub.add_parser("check", parents=[common], help="health probes — run before AND after filing" + ).set_defaults(fn=cmd_check) + sub.add_parser("labels", parents=[common], help="labels and their ids").set_defaults( + fn=cmd_labels) + sub.add_parser("milestones", parents=[common], help="milestones and their ids").set_defaults( + fn=cmd_milestones) + + args = p.parse_args() + # getattr, because SUPPRESS means the attribute may legitimately be absent. + repo = getattr(args, "repo", None) or detect_repo() + if not repo: + die("could not detect owner/name from the git remote — pass --repo") + host, token = load_env() + rc = args.fn(Api(host, token, repo, getattr(args, "dry_run", False)), args) + # `check` returns a count so it can gate a script; the rest return None. + sys.exit(rc or 0) + + +if __name__ == "__main__": + main() diff --git a/docs/architecture/scripts/healthcheck.sh b/docs/architecture/scripts/healthcheck.sh new file mode 100755 index 0000000..6c01012 --- /dev/null +++ b/docs/architecture/scripts/healthcheck.sh @@ -0,0 +1,119 @@ +#!/usr/bin/env bash +# +# Liveness tick for a deployed service. +# +# ## Why this exists +# +# The five-minute healthcheck it replaced was prose handed to a model: "GET +# /healthz. Do NOT use /api/internal/v1/healthz." It duly reported a 404 +# on `/api/internal/v1/health` — a third path, neither the one it was told to +# use nor the one it was told to avoid, and one that had never existed. The site +# was healthy throughout. +# +# An explicit prohibition constrained one wrong URL and left every other wrong +# URL open, because the address was being re-derived on every run rather than +# read. So it is written here once, as a string in version control, and the +# whole class of failure goes with it. +# +# That failure is the expensive kind. The job's own state read `lastRunStatus: +# ok, consecutiveErrors: 0` while a red alert went to a DM — so the monitor was +# reporting itself healthy and crying wolf at the same time, and a monitor +# nobody believes is a monitor nobody has. +# +# ## It holds no credential, and that is the point +# +# `/healthz` is unauthenticated by design — `SECURITY_CHECKLIST.md` makes it a +# checklist item, and the container's own HEALTHCHECK uses it. So unlike +# `reconcile.sh` and `analyze.sh`, this script reads no token, sources no env +# file, and has nothing to leak. At 288 runs a day that is worth more than the +# extra assurance an authenticated probe would buy. +# +# The authenticated sibling, `/api/internal/v1/agent/health`, reports more and +# needs a token. It is deliberately not used here: this asks "is the site up", +# which is a question with a public answer. +# +# ## Both halves are checked +# +# 503 is a real answer, not an outage — the route returns it when migrations +# failed, so the container is marked unhealthy while the marketing pages keep +# serving. Its body still contains `"service"`. Status alone is therefore not a +# verdict, and neither is a grep for the service name; the check asserts HTTP +# 200 *and* `"ok":true`. +# +# ## Installing it +# +# install -m 0755 healthcheck.sh ~/bin/healthcheck.sh +# # then, in the crontab — every five minutes +# */5 * * * * /home/kaspa/bin/healthcheck.sh \ +# >> /home/kaspa/.healthcheck.log 2>&1 +# +# On this deployment it is run by an OpenClaw cron job instead, which is the +# same thing with a scheduler that can also deliver the alert. + +set -euo pipefail + +# `-`, not `:-`. Unset means "no opinion, use production". Set-and-empty means +# a config is wrong, and substituting production for it would report the health +# of a site nobody asked about — quietly, and only where somebody was trying to +# point this somewhere else. +BASE_URL="${HEALTHCHECK_BASE_URL:?set HEALTHCHECK_BASE_URL to the deployed origin, e.g. https://example.com}" +TIMEOUT="${PRIVACY_TIMEOUT:-15}" + +stamp() { date -Is; } +say() { printf '%s healthcheck: %s\n' "$(stamp)" "$*"; } + +if [ -z "$BASE_URL" ]; then + say "FAIL no base URL. HEALTHCHECK_BASE_URL is set but empty; unset it for the default." + exit 78 # EX_CONFIG +fi + +body=$(mktemp) +trap 'rm -f "$body"' EXIT + +# Assigned in the `if`, not with `|| echo "000"` appended. On a connection +# failure curl *already* prints "000" via --write-out and then exits non-zero, +# so appending a fallback produces "000000", which matches no branch below and +# reports "unexpected HTTP" for the one failure this script names explicitly. +# +# No -v and no --trace, ever. This request carries no credential, but the habit +# is the rule SECURITY.md states: a request URL or header must not reach a log. +if ! status=$( + curl --silent --show-error --output "$body" --write-out '%{http_code}' \ + --max-time "$TIMEOUT" \ + "$BASE_URL/healthz" +); then + status="000" +fi + +case "$status" in + 200) + # The status got us here; the body decides. `"ok":true` is asserted rather + # than assumed because a 200 from a proxy, a cached page or a captive portal + # is still a 200, and none of them are this application answering. + if grep -q '"ok":true' "$body"; then + say "ok" + else + say "FAIL 200 but not ok. The response did not contain \"ok\":true." + exit 70 # EX_SOFTWARE + fi + ;; + 503) + # Named rather than left to the catch-all, because it is the one unhealthy + # answer this route is designed to give: migrations failed, the container is + # marked unhealthy, and the public pages are still being served. That is a + # different thing from the site being down and reads differently at 3am. + say "FAIL migrations are not ok (HTTP 503). The site is serving; the schema is not." + exit 70 # EX_SOFTWARE + ;; + 000) + say "FAIL could not reach $BASE_URL" + exit 69 # EX_UNAVAILABLE + ;; + *) + # Includes 404. If this ever fires on a path this script wrote itself, the + # route moved — which is a thing to fix in one place rather than a thing for + # a caller to guess around. + say "FAIL unexpected HTTP $status from $BASE_URL/healthz" + exit 1 + ;; +esac diff --git a/docs/architecture/scripts/migrate.sh b/docs/architecture/scripts/migrate.sh new file mode 100755 index 0000000..ff61dc8 --- /dev/null +++ b/docs/architecture/scripts/migrate.sh @@ -0,0 +1,1066 @@ +#!/usr/bin/env bash +# +# Apply numbered SQL migrations to PostgreSQL, in numeric order, exactly once +# each — and refuse to touch the database at all if a migration that was already +# applied has since been edited. +# +# ./scripts/migrate.sh # apply everything pending +# ./scripts/migrate.sh --status # what is applied, what is pending, exit +# ./scripts/migrate.sh --dry-run # exactly what would run, in order +# ./scripts/migrate.sh --help +# +# MIGRATE_LOCK_WAIT=120 ./scripts/migrate.sh # wait longer for a concurrent run +# MIGRATE_ALLOW_MISSING=1 ./scripts/migrate.sh # after a deliberate squash +# +# Exit codes, so CI can tell these apart: +# 0 nothing pending, or everything pending was applied +# 1 misconfigured, or a check could not be performed +# 2 refused — an edited migration, a duplicate number, an unusable file +# 3 another migration run holds the lock +# 4 a migration failed while running +# +# =========================================================================== +# TEMPLATE COPY — configure this before the first run +# =========================================================================== +# +# Copy to `scripts/migrate.sh` and set the two values in the CONFIGURATION +# block below. The script refuses to run until they are set: it has no +# defaults, deliberately. +# +# `migrations/` looks like a safe default for the directory and +# `schema_migrations` looks like a safe default for the tracking table, and +# both are traps. A directory default means a copy of this script run from the +# wrong checkout applies another project's DDL to this project's database. A +# tracking-table default means two projects sharing one database share one +# ledger: project B reads project A's rows, concludes its own 0001 through 0009 +# are already applied, and never applies them — and nothing anywhere reports an +# error. Neither failure is visible until something reads a column that was +# never created. +# +# Assumes: bash, coreutils, git (optional), and `psql` on PATH. No jq, no +# language runtime, no migration framework. +# +# ## Why this exists +# +# Between `psql -f` typed by hand and a framework's migrator there is a gap +# that most projects live in, and the gap is where databases quietly diverge. +# Applying files by hand means the record of what ran lives in somebody's shell +# history; the second environment gets a different subset and nobody finds out +# until a query fails in production against a column that exists on the laptop. +# +# What closes that gap is not automation, it is a ledger the database itself +# carries: a row per migration, with the checksum of the file that produced it. +# Everything else here — the ordering, the lock, the transactions — exists to +# keep that ledger true. +# +# ## Immutability is the whole point +# +# A migration that has been applied is history. It is not a source file any +# more, and editing it is editing the past of every database that already ran +# it. So the checksum of every applied file is compared against the recorded +# one on every run, and a mismatch stops the run and names the file. +# +# This is the sharpest failure this script exists to prevent, because the +# unguarded version of it is silent in both directions. A runner that tracks +# only filenames will not re-apply the edited file — so the database that ran +# it yesterday keeps the old definition, the database created tomorrow gets the +# new one, and the two are now different databases with identical ledgers. A +# runner that re-applies it instead will run the edit against a schema that +# already has the original, which usually errors, and occasionally does not. +# +# The fix for a migration that was wrong is never an edit. It is a new +# migration with a higher number. +# +# ## One transaction per migration, including the tracking row +# +# Each migration and the INSERT that claims it was applied commit together. A +# failure therefore leaves neither the change nor the claim — no half-applied +# schema wearing a row that says it is done, which is the state that needs a +# human with a psql prompt to untangle. +# +# Some statements cannot run inside a transaction: CREATE INDEX CONCURRENTLY +# and DROP INDEX CONCURRENTLY are the ones that come up, and ALTER TYPE ... ADD +# VALUE on older servers. Mark such a file by putting this line anywhere in it: +# +# -- migrate: no-transaction +# +# and it runs with autocommit, with its tracking row inserted afterwards as a +# separate statement. That is strictly weaker and the script says so out loud +# each time: if the migration succeeds and the INSERT does not, the next run +# will try to apply it again. Use the marker only for statements that require +# it, and put nothing else in the file. +# +# ## What it deliberately does not do +# +# It has no `down` migration and it never rolls back a migration that already +# committed. Both are the same refusal. A down migration is written at the +# moment it is not needed, reviewed less carefully than the up, and then run +# for the first time on the one day everything is already on fire — which is +# also the first time anyone finds out it does not work, or that it drops a +# column holding data written since. Reversal that has never been executed is +# not a safety net, it is a second untested migration wearing one. +# +# Recovery here is a new forward migration, or the backup. Both are things that +# get tested. +# +# It also does not create the database, does not manage roles or extensions +# beyond whatever the migrations themselves do, and does not deploy anything. +# It reads a directory and writes to one database, and that is all. + +set -uo pipefail + +# Run from the repository root when there is one, so that a relative +# MIGRATIONS_DIR means the same directory whether this was invoked from the +# root or from a subdirectory. Not required: this also runs inside a deployed +# container where there is no checkout at all, so a missing git is not fatal. +if root=$(git rev-parse --show-toplevel 2>/dev/null) && [ -n "$root" ]; then + cd "$root" || exit 1 +fi + +# --------------------------------------------------------------------------- +# CONFIGURATION — set these two, then delete this banner. +# +# Both are empty on purpose. See the note at the top: a directory default +# points this at another project's SQL, and a tracking-table default makes two +# projects sharing a database share one ledger. Both failures are silent. +# --------------------------------------------------------------------------- + +# Directory holding the numbered migrations, e.g. db/migrations +MIGRATIONS_DIR="${MIGRATE_DIR:-}" + +# Table that records what has been applied. Schema-qualify it if this project +# does not own the search_path, e.g. myapp.schema_migrations +TRACKING_TABLE="${MIGRATE_TABLE:-}" + +say() { printf '\033[1mmigrate:\033[0m %s\n' "$*" >&2; } +die() { printf '\033[1mmigrate:\033[0m %s\n' "$*" >&2; exit 1; } + +# Exit 2 rather than 1, and kept apart from die() for the whole file: "this is +# not configured" and "this is configured and I am refusing" are different +# answers, and CI that retries the first must not retry the second. +refuse() { printf '\033[1mmigrate:\033[0m %s\n' "$*" >&2; exit 2; } + +MODE="apply" + +# Arguments are read BEFORE the configuration is checked, so that --help works +# on an unconfigured copy. This file ships unconfigured on purpose, so the +# state somebody most needs to read the help in is the state it arrives in, and +# a --help that answers "set MIGRATIONS_DIR first" answers a question nobody +# asked. It also means an unknown argument is reported as an unknown argument. +# +# ${1+"$@"} rather than "$@": with no positional parameters and `set -u`, bash +# before 4.4 treats "$@" as unset and aborts — and bash 3.2 is what /bin/bash +# is on macOS, which the shasum fallback above exists to support. The ${1+...} +# form expands to nothing when there are no arguments, and to exactly "$@" +# when there are. +for arg in ${1+"$@"}; do + case "$arg" in + --status) MODE="status" ;; + --dry-run) MODE="dry-run" ;; + --help|-h) + printf '%s\n' \ + "usage: migrate.sh [--status | --dry-run]" \ + "" \ + " (no flags) apply every pending migration, in numeric order" \ + " --status print applied and pending migrations, then exit" \ + " --dry-run print exactly what would be applied, and change nothing" \ + "" \ + "Required, and deliberately without defaults — see the CONFIGURATION" \ + "block in this file for why:" \ + "" \ + " MIGRATE_DIR= directory holding the numbered .sql files" \ + " MIGRATE_TABLE= table recording what has been applied" \ + "" \ + "Connection comes from the environment: DATABASE_URL, or the standard" \ + "libpq variables (PGHOST, PGPORT, PGDATABASE, PGUSER, PGPASSWORD)." \ + "" \ + "Prefer the libpq variables on a shared machine. DATABASE_URL is passed" \ + "to psql as an argument, and arguments are world-readable in ps and" \ + "/proc for the life of the run — so the password is visible to any" \ + "local user while a migration is applying. PGPASSWORD is not." \ + "" \ + " MIGRATE_LOCK_WAIT= how long to wait for a concurrent run (30)" \ + " MIGRATE_PG_LOCK_TIMEOUT= server lock_timeout, e.g. '5s' (unset)" \ + " MIGRATE_ALLOW_MISSING=1 proceed when an applied migration has no file" \ + " MIGRATE_ALLOW_OUT_OF_ORDER=1 proceed when a pending number is below the" \ + " highest applied" >&2 + exit 0 ;; + *) die "unknown argument '$arg'. Usage: migrate.sh [--status | --dry-run]" ;; + esac +done + +# Refused rather than defaulted, and named one at a time so the message says +# which value is missing rather than "configuration error". +[ -n "$MIGRATIONS_DIR" ] || die "set MIGRATIONS_DIR (or MIGRATE_DIR) — the directory holding the numbered .sql files. See the CONFIGURATION block." +[ -n "$TRACKING_TABLE" ] || die "set TRACKING_TABLE (or MIGRATE_TABLE) — the table recording what has been applied. See the CONFIGURATION block." + +# --status is a read-only diagnostic, and an edited migration, a missing file +# and an out-of-order pending number are precisely the three states somebody +# runs it to understand. Exiting before the report is refusing to answer the +# question that was asked — and worse, the failure text at the bottom of this +# script sends operators to --status by name to find out which migrations +# already succeeded, which is unreachable if drift makes --status refuse. +# +# So in status mode a refusal is recorded instead of taken, the report is still +# printed, and the exit code is still 2 afterwards — CI cannot read it as +# success, and a human gets the picture they asked for. +STATUS_EXIT=0 + +refuse_or_defer() { + if [ "$MODE" = "status" ]; then + STATUS_EXIT=2 + say "(--status continues so it can still show you the state below; a real run stops here.)" + return 0 + fi + exit 2 +} + +# --------------------------------------------------------------------------- +# Tools, checked before anything is read or connected to. +# --------------------------------------------------------------------------- + +command -v psql >/dev/null 2>&1 || die "psql is not on PATH. Install the PostgreSQL client, or run this from an image that has one." + +# \if and \gset are how each migration re-checks the ledger after the lock is +# held (see "the plan is re-checked" below). psql gained \if in 10; on anything +# older the conditional blocks are ignored rather than obeyed, which would run +# every migration in the plan unguarded. Refused rather than degraded. +psql_major=$(psql --version 2>/dev/null | sed -E 's/[^0-9]*([0-9]+).*/\1/') + +case "$psql_major" in + ''|*[!0-9]*) die "could not read the psql version ('$(psql --version 2>&1 | head -n 1)'). This needs psql 10 or newer for \\if." ;; +esac + +[ "$psql_major" -ge 10 ] || die "psql ${psql_major} is too old: \\if arrived in psql 10, and without it the re-check that stops a concurrent run re-applying a migration is silently ignored." + +# sha256sum is coreutils; shasum ships on macOS where coreutils does not. The +# checksum column is the load-bearing part of this script, so which tool +# produced it is decided once, here, rather than per call — two call sites +# picking different tools would produce two different hashes of one file and +# every migration would look edited. +if command -v sha256sum >/dev/null 2>&1; then + sha256_stream() { sha256sum | cut -d' ' -f1; } +elif command -v shasum >/dev/null 2>&1; then + sha256_stream() { shasum -a 256 | cut -d' ' -f1; } +else + die "no sha256 tool found (looked for sha256sum and shasum). Migrations cannot be checksummed, so this cannot verify that applied files are unchanged." +fi + +# Hashing stdin rather than a named file keeps the filename out of the digest, +# which is what makes a rename a rename instead of a new migration. +sha256_of() { sha256_stream < "$1"; } + +work=$(mktemp -d) || die "cannot create a temporary directory" + +# The generated SQL is kept when a migration fails, and only then. psql reports +# errors as "run.sql:LINE", and deleting the file it is pointing at turns the +# one piece of evidence about what actually ran into a dangling path. +KEEP_WORK="" + +# The only rm in this script, and the belt-and-braces on it is deliberate: +# $work can only ever be what mktemp -d printed (the assignment above dies +# otherwise, and this trap is not installed until after it), but `rm -rf` is +# not a place to reason from "can only ever". -d re-confirms it is a directory +# that exists, so an empty or clobbered $work removes nothing. +trap '[ -n "$KEEP_WORK" ] || { [ -n "$work" ] && [ -d "$work" ] && rm -rf "$work"; }' EXIT + +TAB=$(printf '\t') + +# --------------------------------------------------------------------------- +# Validate the configured names before they reach SQL. +# +# The table name is interpolated into statements unquoted, so it is checked +# against what a bare identifier may contain and refused otherwise. This is not +# hostile-input defence — it is configuration — but a name needing quoting +# would break at a different place each time it was used, and the failure would +# look like a syntax error inside somebody's migration. +# --------------------------------------------------------------------------- + +printf '%s' "$TRACKING_TABLE" \ + | grep -Eq '^[a-zA-Z_][a-zA-Z0-9_]*(\.[a-zA-Z_][a-zA-Z0-9_]*)?$' \ + || die "TRACKING_TABLE ('$TRACKING_TABLE') is not a plain identifier. Use name or schema.name, letters, digits and underscore only." + +[ -d "$MIGRATIONS_DIR" ] || die "MIGRATIONS_DIR ('$MIGRATIONS_DIR') is not a directory. Create it, or point MIGRATE_DIR at the right one — this will not guess." + +MIGRATIONS_ABS=$(cd "$MIGRATIONS_DIR" && pwd) || die "cannot resolve MIGRATIONS_DIR ('$MIGRATIONS_DIR')." + +# psql's \i takes a single-quoted path, and its quoting rules are not the +# shell's: inside single quotes psql itself expands \t, \n, \r, \b, \f, \NNN +# and \xNN before the path ever reaches the filesystem. So all three of these +# produce a valid-looking script that addresses the wrong file, and all three +# are refused rather than guessed at. +# +# The backslash case is the one that looks harmless and is not. A directory +# named `tab\test` is handed to psql as `tab\test` and opened as `tabest` +# — which usually does not exist, and the run fails claiming the MIGRATION is +# broken when the path is. It does not have to not-exist, either: a path that +# escapes into another real directory includes another project's SQL, silently. +# +# Spaces are fine and stay fine — the single quotes handle them, and the +# directories this lands in have spaces in their names. +case "$MIGRATIONS_ABS" in + *"'"*|*'"'*) + die "the migrations path ('$MIGRATIONS_ABS') contains a quote character. psql's \\i cannot address it safely; move the directory." ;; + *\\*) + die "the migrations path ('$MIGRATIONS_ABS') contains a backslash. psql expands backslash escapes inside \\i's quoted argument, so this would open a different path than the one on disk; move the directory." ;; + *[[:cntrl:]]*) + die "the migrations path contains a control character (a newline or a tab). psql's \\i cannot address it; move the directory." ;; +esac + +# --------------------------------------------------------------------------- +# Read the directory. +# +# Everything here is checkable without a database and is checked before one is +# opened, so a badly-named file is a failure that costs nothing rather than one +# discovered halfway through a deploy. +# --------------------------------------------------------------------------- + +FILES_TSV="$work/files.tsv" # version filename checksum transactional +: > "$FILES_TSV" + +for path in "$MIGRATIONS_ABS"/*; do + [ -f "$path" ] || continue + + base=${path##*/} + + case "$base" in + *.sql) ;; + *) + # Said out loud rather than passed over. Only *.sql is a migration, and a + # file that is being ignored should say so on the run that ignores it — + # "0007_x.sql.bak" and "0007_x.sql" look identical in a code review. + say "note: ignoring $base — only *.sql files are migrations." + continue ;; + esac + + # Refused, not skipped. A .sql file this script cannot number is a file that + # will never be applied and will never be mentioned again, which is the exact + # failure the ledger exists to prevent. + case "$base" in + [0-9]*_*) ;; + *) refuse "'$base' is not named _.sql, so it would never be applied. Rename it, or move it out of $MIGRATIONS_DIR." ;; + esac + + case "$base" in + *[!A-Za-z0-9._-]*) refuse "'$base' contains a character outside A-Z a-z 0-9 . _ - . psql's \\i cannot address it reliably; rename it." ;; + esac + + num=${base%%_*} + case "$num" in + *[!0-9]*) refuse "'$base' has a non-numeric prefix ('$num'). The prefix decides the order, so it must be digits only." ;; + esac + + # 10# forces base ten. Without it, $((0008)) is an invalid octal literal and + # the eighth migration of any project stops the run with an arithmetic error. + version=$((10#$num)) + + checksum=$(sha256_of "$path") || die "could not checksum $base." + [ -n "$checksum" ] || die "empty checksum for $base — refusing to record a migration it cannot identify." + + if grep -Eq '^[[:space:]]*--[[:space:]]*migrate:[[:space:]]*no-transaction[[:space:]]*$' "$path"; then + transactional="no" + else + transactional="yes" + fi + + # A file whose last statement has no terminating semicolon is refused, + # because psql discards an unterminated buffer at end of file without + # complaining: the migration would run everything up to the last semicolon, + # succeed, and be recorded as applied with its final statement never sent. + # Trailing blank lines and trailing comment lines are ignored. + last=$(grep -vE '^[[:space:]]*(--.*)?$' "$path" | tail -n 1) + + if [ -z "$last" ]; then + refuse "'$base' contains no SQL. Delete it, or give it a body — an empty migration would be recorded as applied and could never be filled in." + fi + + printf '%s' "$last" | grep -Eq ';[[:space:]]*(--.*)?$' \ + || refuse "the last statement in '$base' has no terminating semicolon. psql drops an unterminated statement at end of file without an error, so this would be recorded as applied while its final statement never ran." + + # Only for transactional files: this script opens the transaction, and a + # BEGIN or COMMIT inside the file would close it early — committing part of + # the migration and leaving the tracking INSERT in a transaction of its own. + # A file that genuinely manages its own transactions declares that with the + # no-transaction marker, so it is exempt. + # + # The semicolon in the pattern is load-bearing: plpgsql bodies open with a + # bare `BEGIN` and close with `END;`, and matching those would refuse every + # function definition in the project. `END` is left out of the pattern + # entirely for that reason. + if [ "$transactional" = "yes" ] \ + && grep -Eqi '^[[:space:]]*(BEGIN|COMMIT|ROLLBACK)[[:space:]]*;' "$path"; then + refuse "'$base' contains its own BEGIN;, COMMIT; or ROLLBACK;. This runner wraps each migration in a transaction together with its tracking row, and a nested one would commit half of it. Remove them — or, if this is a procedure body that must COMMIT, add '-- migrate: no-transaction' and take responsibility for the transaction yourself." + fi + + printf '%s\t%s\t%s\t%s\n' "$version" "$base" "$checksum" "$transactional" >> "$FILES_TSV" +done + +sort -t "$TAB" -k1,1n "$FILES_TSV" -o "$FILES_TSV" || die "could not sort the migration list." + +TOTAL=$(grep -c . "$FILES_TSV") + +if [ "$TOTAL" -eq 0 ]; then + say "no migrations in $MIGRATIONS_DIR — nothing to do." + exit 0 +fi + +# --------------------------------------------------------------------------- +# Duplicate numbers are refused; gaps are reported. +# +# Two files sharing a number is not a style problem: the ledger is keyed on the +# number, so whichever ran first would make the other look applied for ever +# after. Both are named, because the fix is deciding which one moves. +# +# A gap is only ever a warning. It is normal after a branch is abandoned and +# suspicious after a merge, and this script cannot tell those apart — but a +# database that is missing 0006 while it holds 0005 and 0007 should hear about +# it here rather than from a foreign key three releases later. +# --------------------------------------------------------------------------- + +prev_version="" +prev_base="" +gaps="" + +while IFS="$TAB" read -r version base _checksum _transactional; do + if [ -n "$prev_version" ]; then + if [ "$version" -eq "$prev_version" ]; then + refuse "'$prev_base' and '$base' share the number $version. The ledger is keyed on the number, so applying one would mark the other as applied. Renumber one of them." + fi + + if [ "$version" -eq $((prev_version + 2)) ]; then + gaps="${gaps}${gaps:+, }$((prev_version + 1))" + elif [ "$version" -gt $((prev_version + 1)) ]; then + gaps="${gaps}${gaps:+, }$((prev_version + 1))-$((version - 1))" + fi + fi + + prev_version="$version" + prev_base="$base" +done < "$FILES_TSV" + +if [ -n "$gaps" ]; then + say "WARNING: the numbering skips ${gaps}. That is fine after an abandoned" + say " branch and is a missing file after a merge — check which." +fi + +# --------------------------------------------------------------------------- +# Connection. From the environment, never from this file. +# +# ## DATABASE_URL puts the password somewhere the redaction below cannot reach +# +# The redaction that follows keeps credentials out of THIS script's output. It +# does nothing about `ps`: a URL passed as an argument is world-readable in +# /proc//cmdline for as long as psql runs, so on a shared machine any +# local user can read the password off a migration in progress. +# +# It is not parsed apart here, because doing that correctly means percent- +# decoding the userinfo exactly as libpq does, and a decoder that is subtly +# wrong breaks the connection — a worse outcome than the exposure it fixes. +# +# The fix is to not use a URL: set PGHOST/PGPORT/PGDATABASE/PGUSER/PGPASSWORD +# instead, which libpq reads from the environment and never places in argv. +# --help says so too. +# --------------------------------------------------------------------------- + +DB_URL="${DATABASE_URL:-}" + +PSQL=(psql -X -q -v ON_ERROR_STOP=1) + +# -X is not cosmetic. A ~/.psqlrc setting AUTOCOMMIT off, ON_ERROR_ROLLBACK on, +# or a default search_path changes what these migrations mean, and the effect +# would follow one operator's laptop and no one else's. + +if [ -n "$DB_URL" ]; then + PSQL+=(-d "$DB_URL") + # Credentials are redacted here because this string is printed on every run, + # into terminals and CI logs that are kept. psql accepts three shapes and + # only the first is a URI, so all three have to be covered or the redaction + # is theatre: + # + # postgres://user:pw@host/db URI userinfo + # host=h dbname=d password=pw keyword/value conninfo + # postgres://user@host/db?sslpassword=pw URI query parameter + # + # The userinfo pattern stops at the first '/', '?' or '#', so it can only + # ever redact inside the authority — and it takes the LAST '@' in there + # rather than the first, so a password containing a literal '@' does not + # leave its own tail sitting in the log. + TARGET=$(printf '%s' "$DB_URL" | sed -E \ + -e 's%(//[^:/@]*):[^/?#]*@%\1:***@%' \ + -e 's%(^|[[:space:]&?])(password|sslpassword)=[^[:space:]&]*%\1\2=***%g') +elif [ -n "${PGDATABASE:-}" ]; then + TARGET="${PGUSER:-$(id -un)}@${PGHOST:-local socket}${PGPORT:+:$PGPORT}/${PGDATABASE}" +else + # libpq's own default is a database named after the current unix user on the + # local socket. That is a real connection to a real database, so leaving it + # implicit means a mistyped environment migrates something at random instead + # of failing. Named values only. + die "no connection configured. Set DATABASE_URL, or at least PGDATABASE (with PGHOST/PGPORT/PGUSER as needed) — with neither, psql would connect to a database named after \$USER on the local socket, which is not a target anyone chose." +fi + +db_query() { "${PSQL[@]}" -A -t -F "$TAB" -c "$1"; } + +conn=$(db_query "SELECT current_database(), current_user, current_setting('server_version')" 2>"$work/connerr") + +if [ -z "$conn" ]; then + say "could not open the database. psql said:" + sed 's/^/ /' "$work/connerr" >&2 + die "target was ${TARGET}. Fix the connection and run this again; nothing was read or written." +fi + +conn_db=$(printf '%s' "$conn" | cut -f1) +conn_user=$(printf '%s' "$conn" | cut -f2) +conn_ver=$(printf '%s' "$conn" | cut -f3) + +say "target ${TARGET}" +say " database '${conn_db}' as '${conn_user}', PostgreSQL ${conn_ver}" + +# --------------------------------------------------------------------------- +# Read the ledger. +# +# The table's absence is reported as absence, never as an empty ledger. "No +# migration has ever been applied here" and "the migrations were applied under +# a different table name, or into a schema this role cannot see" produce the +# same empty list and want opposite reactions from the operator. +# --------------------------------------------------------------------------- + +TABLE_EXISTS=$(db_query "SELECT to_regclass('${TRACKING_TABLE}') IS NOT NULL" 2>"$work/regerr") + +case "$TABLE_EXISTS" in + t|f) ;; + *) + say "could not determine whether ${TRACKING_TABLE} exists. psql said:" + sed 's/^/ /' "$work/regerr" >&2 + die "refusing to continue without knowing what has already been applied." ;; +esac + +APPLIED_TSV="$work/applied.tsv" # version checksum filename applied_at +: > "$APPLIED_TSV" + +if [ "$TABLE_EXISTS" = "t" ]; then + if ! db_query "SELECT version, checksum, filename, applied_at FROM ${TRACKING_TABLE}" > "$APPLIED_TSV" 2>"$work/readerr"; then + say "could not read ${TRACKING_TABLE}. psql said:" + sed 's/^/ /' "$work/readerr" >&2 + die "refusing to continue without the list of applied migrations." + fi + # Checked, like the sort of the file list is. `sort -o` writing over its own + # input can leave that input truncated if it fails, and a truncated ledger + # snapshot reads as "fewer migrations are applied than really are" — which + # plans already-applied migrations for re-application. The re-check inside + # the lock would catch it, but a guard that is only load-bearing when another + # guard fails is not one to lean on. + sort -t "$TAB" -k1,1n "$APPLIED_TSV" -o "$APPLIED_TSV" \ + || die "could not sort the list of applied migrations read from ${TRACKING_TABLE}." +fi + +APPLIED_COUNT=$(grep -c . "$APPLIED_TSV") + +applied_row() { + grep -m1 "^$1$TAB" "$APPLIED_TSV" 2>/dev/null +} + +# --------------------------------------------------------------------------- +# The immutability check. Everything above this exists to make it possible. +# --------------------------------------------------------------------------- + +DRIFT="" + +while IFS="$TAB" read -r version base checksum _transactional; do + row=$(applied_row "$version") || true + [ -n "$row" ] || continue + + was_checksum=$(printf '%s' "$row" | cut -f2) + was_file=$(printf '%s' "$row" | cut -f3) + + if [ "$was_checksum" != "$checksum" ]; then + DRIFT="yes" + say "EDITED: ${base}" + say " recorded ${was_checksum}" + say " on disk ${checksum}" + continue + fi + + # Same content, different name. Harmless to the database and worth saying: + # it is how a ledger and a directory start describing each other in different + # words, and the next person to grep for the recorded filename finds nothing. + if [ "$was_file" != "$base" ]; then + say "note: migration ${version} was recorded as '${was_file}' and is now '${base}' — same contents." + fi +done < "$FILES_TSV" + +if [ -n "$DRIFT" ]; then + say "" + say "refusing to run: the file(s) above were applied to this database and have" + say "changed since. Editing an applied migration changes the past of every" + say "database that already ran it, and the databases that have not run it yet" + say "will get something different — with both ledgers claiming to agree." + say "" + say "Restore the original contents (git show is usually enough) and put the" + say "change in a NEW migration with a higher number. If the edit was only" + say "cosmetic, it still has to be reverted: the checksum is the only evidence" + say "these databases are the same, and it cannot tell cosmetic from not." + refuse_or_defer +fi + +# A row with no file. Usually the wrong MIGRATIONS_DIR, sometimes a deliberate +# squash, occasionally a deleted migration that other databases still run. +# Refused by default because two of those three are wrong and all three are +# invisible. +ORPHANS="" + +while IFS="$TAB" read -r version _checksum filename _applied_at; do + [ -n "$version" ] || continue + if ! cut -f1 "$FILES_TSV" | grep -qx "$version"; then + ORPHANS="${ORPHANS} ${version} ${filename}"$'\n' + fi +done < "$APPLIED_TSV" + +if [ -n "$ORPHANS" ]; then + say "these migrations are recorded as applied but have no file in ${MIGRATIONS_DIR}:" + printf '%s' "$ORPHANS" >&2 + + if [ -z "${MIGRATE_ALLOW_MISSING:-}" ]; then + say "refusing to run. Either this is pointed at the wrong directory, or" + say "migrations were deleted that other databases have not applied yet." + say "If they were squashed deliberately, re-run with MIGRATE_ALLOW_MISSING=1." + refuse_or_defer + else + say "MIGRATE_ALLOW_MISSING is set — continuing without them." + fi +fi + +# --------------------------------------------------------------------------- +# Pending, in order. +# --------------------------------------------------------------------------- + +PENDING_TSV="$work/pending.tsv" +: > "$PENDING_TSV" + +HIGHEST_APPLIED="" +if [ "$APPLIED_COUNT" -gt 0 ]; then + HIGHEST_APPLIED=$(cut -f1 "$APPLIED_TSV" | sort -n | tail -n 1) +fi + +BEHIND="" + +while IFS="$TAB" read -r version base checksum transactional; do + applied_row "$version" >/dev/null && continue + + if [ -n "$HIGHEST_APPLIED" ] && [ "$version" -lt "$HIGHEST_APPLIED" ]; then + BEHIND="${BEHIND} ${base} (this database is already at ${HIGHEST_APPLIED})"$'\n' + fi + + printf '%s\t%s\t%s\t%s\n' "$version" "$base" "$checksum" "$transactional" >> "$PENDING_TSV" +done < "$FILES_TSV" + +PENDING_COUNT=$(grep -c . "$PENDING_TSV") + +# A pending migration numbered below one already applied arrives when two +# branches merge. It is refused rather than applied because the order it would +# run in here is not the order it ran in anywhere that merged the other way +# first, and "same migrations, different order" is the same divergence the +# checksum guard exists to catch — just harder to see. +if [ -n "$BEHIND" ]; then + say "these pending migrations are numbered below what this database has already applied:" + printf '%s' "$BEHIND" >&2 + + if [ -z "${MIGRATE_ALLOW_OUT_OF_ORDER:-}" ]; then + say "refusing to run. Renumber them above ${HIGHEST_APPLIED} so every database" + say "applies them in the same order, or re-run with" + say "MIGRATE_ALLOW_OUT_OF_ORDER=1 if you have confirmed the order does not" + say "matter for these particular statements." + refuse_or_defer + else + say "MIGRATE_ALLOW_OUT_OF_ORDER is set — applying them anyway." + fi +fi + +# --------------------------------------------------------------------------- +# --status +# --------------------------------------------------------------------------- + +if [ "$MODE" = "status" ]; then + if [ "$TABLE_EXISTS" = "f" ]; then + say "${TRACKING_TABLE} does not exist. Nothing has been applied THROUGH THIS" + say "SCRIPT — which is not the same as an empty database. Check the table" + say "name before concluding the schema is unmigrated." + else + say "${APPLIED_COUNT} applied:" + while IFS="$TAB" read -r version _checksum filename applied_at; do + [ -n "$version" ] || continue + printf ' %-6s %-44s %s\n' "$version" "$filename" "$applied_at" >&2 + done < "$APPLIED_TSV" + fi + + if [ "$PENDING_COUNT" -eq 0 ]; then + say "0 pending — every file in ${MIGRATIONS_DIR} has been applied." + else + say "${PENDING_COUNT} pending:" + while IFS="$TAB" read -r version base _checksum transactional; do + [ -n "$version" ] || continue + if [ "$transactional" = "no" ]; then + printf ' %-6s %-44s %s\n' "$version" "$base" "(no-transaction)" >&2 + else + printf ' %-6s %s\n' "$version" "$base" >&2 + fi + done < "$PENDING_TSV" + fi + + # 0 normally; 2 if something above would have stopped a real run. The report + # is printed either way, which is the whole point of --status. + exit "$STATUS_EXIT" +fi + +if [ "$PENDING_COUNT" -eq 0 ]; then + say "${APPLIED_COUNT} applied, 0 pending — nothing to do." + exit 0 +fi + +# --------------------------------------------------------------------------- +# --dry-run +# --------------------------------------------------------------------------- + +if [ "$MODE" = "dry-run" ]; then + say "--dry-run: nothing was changed. It would apply ${PENDING_COUNT}, in this order:" + + if [ "$TABLE_EXISTS" = "f" ]; then + say " create ${TRACKING_TABLE} (it does not exist yet)" + fi + + while IFS="$TAB" read -r version base _checksum transactional; do + [ -n "$version" ] || continue + if [ "$transactional" = "no" ]; then + printf ' %-6s %-44s %s\n' "$version" "$base" "NOT in a transaction" >&2 + else + printf ' %-6s %s\n' "$version" "$base" >&2 + fi + done < "$PENDING_TSV" + + say "against ${TARGET} (database '${conn_db}')." + exit 0 +fi + +# --------------------------------------------------------------------------- +# The advisory lock, and why the whole run is a single psql session. +# +# A session-level advisory lock is released when its SESSION ends, so it can +# only span the run if the run is one session. That is the reason this builds +# one SQL program and feeds it to psql once, rather than calling psql per +# migration: per-migration calls would drop the lock between every file, and +# two deploys landing together would interleave — each seeing the other's +# half-finished ledger and both concluding the same migration was theirs to +# apply. +# +# The key is derived from the tracking table name, so two projects sharing one +# database do not block each other, and two runs of the same project always +# collide. That collision is the entire feature. +# --------------------------------------------------------------------------- + +LOCK_WAIT="${MIGRATE_LOCK_WAIT:-30}" +case "$LOCK_WAIT" in + ''|*[!0-9]*) die "MIGRATE_LOCK_WAIT must be a whole number of seconds, got '$LOCK_WAIT'." ;; +esac + +# Fifteen hex digits: the largest slice that always fits in a signed 64-bit +# integer, which is what pg_advisory_lock takes and what bash arithmetic holds. +lock_hex=$(printf '%s' "migrate:${TRACKING_TABLE}" | sha256_stream | cut -c1-15) +[ -n "$lock_hex" ] || die "could not derive the advisory lock key." +LOCK_KEY=$((16#$lock_hex)) + +sql_quote() { printf "'%s'" "$(printf '%s' "$1" | sed "s/'/''/g")"; } + +PROGRAM="$work/run.sql" +LOCK_FAILED_MARKER="MIGRATE_LOCK_UNAVAILABLE" + +# Printed by psql itself, one line before each migration starts. It is progress +# for the operator and, afterwards, the record of how far the session got — a +# migration that hangs is otherwise indistinguishable from a psql that never +# connected. +STARTED_PREFIX="migrate: applying " +SKIPPED_PREFIX="migrate: skipping " + +{ + printf '\\set ON_ERROR_STOP on\n' + printf '\\timing off\n' + + # The one place a server-side timeout is worth having: DDL waiting behind a + # long-running reader takes the whole deploy with it, and it does so while + # holding the advisory lock. Optional and unset by default, because a value + # that aborts migrations on a busy table is not something to inherit. + if [ -n "${MIGRATE_PG_LOCK_TIMEOUT:-}" ]; then + printf "SET lock_timeout = %s;\n" "$(sql_quote "$MIGRATE_PG_LOCK_TIMEOUT")" + fi + + # Retry rather than block for ever: a deploy that hangs silently on a stale + # session is worse than one that fails and says why. The lock is session + # level, so it survives the end of this DO block's own transaction. + cat <= ${LOCK_WAIT} THEN + RAISE EXCEPTION '${LOCK_FAILED_MARKER}: another migration run has held the lock for % seconds', waited; + END IF; + PERFORM pg_sleep(1); + waited := waited + 1; + END LOOP; +END +\$migrate_lock\$; +SQL + + # Created inside the lock, so two first-ever runs cannot race here. The + # notice from IF NOT EXISTS is suppressed for this statement only — it is + # printed on every run after the first and trains people to ignore output + # that on some other run says something that matters. + printf 'SET client_min_messages = warning;\n' + cat < "$PROGRAM" + +say "applying ${PENDING_COUNT} migration(s) to '${conn_db}'…" + +nontx=$(cut -f4 "$PENDING_TSV" | grep -c '^no$') +if [ "$nontx" -gt 0 ]; then + say "WARNING: ${nontx} of them run OUTSIDE a transaction (-- migrate:" + say " no-transaction). If one fails partway, its tracking row is not" + say " written and this run will report it — but the database keeps" + say " whatever it managed to do. A failed CREATE INDEX CONCURRENTLY" + say " leaves an INVALID index behind: DROP it before re-running." +fi + +LOG="$work/psql.log" + +"${PSQL[@]}" -f "$PROGRAM" 2>&1 | tee "$LOG" >&2 +psql_status=${PIPESTATUS[0]} + +# Checked first, and before the ledger is re-read: on a first-ever run the +# tracking table does not exist yet, so a lock loss there would otherwise be +# reported as "could not read the ledger" — which is true, and not the reason. +if grep -q "$LOCK_FAILED_MARKER" "$LOG"; then + say "another migration run holds the lock for ${TRACKING_TABLE} and did not" + say "release it within ${LOCK_WAIT}s. Nothing was applied by this run. Wait for" + say "the other deploy to finish, or raise MIGRATE_LOCK_WAIT. If nothing is" + say "running, look for a stale session:" + say " SELECT * FROM pg_locks WHERE locktype = 'advisory';" + exit 3 +fi + +# --------------------------------------------------------------------------- +# What was applied is read back out of the ledger, not counted from this +# script's own output. +# +# The two can disagree, and when they do the ledger is right: a no-transaction +# migration can run to completion and still fail to record itself, and a +# session killed between COMMIT and the next statement leaves output claiming +# less than the database holds. Reporting the plan, or the echoes, would be +# reporting an intention as a fact. +# +# If the ledger cannot be re-read, that is reported as not knowing — never as +# nothing having happened. +# --------------------------------------------------------------------------- + +CONFIRMED="$work/confirmed.tsv" +confirmed_ok="yes" + +if ! db_query "SELECT version FROM ${TRACKING_TABLE}" > "$CONFIRMED" 2>"$work/confirmerr"; then + confirmed_ok="no" + : > "$CONFIRMED" +fi + +applied_now=0 +skipped_now=0 +applied_list="" +skipped_list="" +missing_list="" + +while IFS="$TAB" read -r version base _checksum _transactional; do + [ -n "$version" ] || continue + + if ! grep -qx "$version" "$CONFIRMED"; then + missing_list="${missing_list} ${version} ${base}"$'\n' + elif grep -qF "${SKIPPED_PREFIX}${base}" "$LOG"; then + # In the ledger, but this run did not put it there. Counted separately + # rather than folded into the total: "we applied it" and "we found it + # already applied and stood down" are different events, and only the second + # one means another deploy was running against this database. + skipped_now=$((skipped_now + 1)) + skipped_list="${skipped_list} ${version} ${base}"$'\n' + else + applied_now=$((applied_now + 1)) + applied_list="${applied_list} ${version} ${base}"$'\n' + fi +done < "$PENDING_TSV" + +if [ "$confirmed_ok" = "no" ]; then + say "" + + if [ "$psql_status" -eq 0 ]; then + # psql was happy and the ledger has become unreadable between one statement + # and the next. Nothing can be said about what is applied, so nothing is. + say "the run reported success but ${TRACKING_TABLE} could not be re-read, so" + say "what is applied is UNKNOWN. psql said:" + sed 's/^/ /' "$work/confirmerr" >&2 + say "Run --status once the database is reachable, before running this again." + exit 1 + fi + + # The far more common shape: the run failed early — a missing schema, a role + # without CREATE — so the table never existed to be read. The error printed + # above is the real one, and calling this "unknown" would send the operator + # looking for a database problem instead of reading it. + KEEP_WORK="yes" + say "the run failed (psql exit ${psql_status}) and ${TRACKING_TABLE} does not" + say "exist or cannot be read, so nothing was applied. Read the error above —" + say "a missing schema or a role without CREATE is the usual cause." + say "the SQL that was sent is kept at ${work}/run.sql." + exit 4 +fi + +if [ "$skipped_now" -gt 0 ]; then + say "another run applied ${skipped_now} of the ${PENDING_COUNT} planned while this one" + say "waited for the lock. They were skipped, not repeated:" + printf '%s' "$skipped_list" >&2 +fi + +if [ "$psql_status" -eq 0 ]; then + say "applied ${applied_now} of ${PENDING_COUNT}:" + + if [ "$applied_now" -eq 0 ]; then + say " (none)" + else + printf '%s' "$applied_list" >&2 + fi + + # psql exited clean while the ledger disagrees with the plan. Rare, and + # exactly the state that must not be reported as success: something ran + # without recording itself, or something recorded itself twice. + if [ $((applied_now + skipped_now)) -ne "$PENDING_COUNT" ]; then + # Kept, like the other failure paths keep it. This branch tells the + # operator to check the database by hand, and the generated SQL is the + # only record of what was actually sent — deleting it on the way out would + # take the evidence with it. + KEEP_WORK="yes" + say "" + say "psql reported success but these are NOT in ${TRACKING_TABLE}:" + printf '%s' "$missing_list" >&2 + say "Do not re-run until you know why — check the database by hand." + say "the SQL that was sent is kept at ${work}/run.sql." + exit 4 + fi + + exit 0 +fi + +# psql stops at the first error, so the last migration it announced is the one +# that failed — unless the ledger says that one committed, in which case the +# error came after it and blaming it would be a claim rather than a reading. +failed=$(grep "^${STARTED_PREFIX}" "$LOG" | tail -n 1 | sed "s/^${STARTED_PREFIX}//") +failed_version="" +[ -n "$failed" ] && failed_version=$(grep -m1 "${TAB}${failed}${TAB}" "$PENDING_TSV" | cut -f1) + +KEEP_WORK="yes" + +say "" +say "a migration failed (psql exit ${psql_status}). See the error above." +say "the SQL that was sent is kept at ${work}/run.sql — the line numbers in" +say "psql's error refer to it. Delete it when you are done." + +if [ -z "$failed" ]; then + # Nothing was announced, so nothing was reached: the failure is the lock, the + # CREATE TABLE, or the connection. Said as such rather than blamed on a + # migration that never started. + say "no migration was reached — it failed on the lock, on creating" + say "${TRACKING_TABLE}, or on the connection itself." +elif [ -n "$failed_version" ] && grep -qx "$failed_version" "$CONFIRMED"; then + say "'${failed}' is the last one announced and it IS recorded as applied, so" + say "the failure came after it finished. Read the error above before re-running." +else + say "the failing migration is '${failed}'." + + # A transactional failure needs no explanation beyond "nothing happened". A + # no-transaction failure does, and it is the one case where somebody has to + # look at the database before this is run again. + if grep -q "${TAB}${failed}${TAB}.*${TAB}no$" "$PENDING_TSV"; then + say "" + say "It ran OUTSIDE a transaction, so whatever it completed before failing is" + say "still there and is NOT recorded. Inspect the database before re-running:" + say "a half-built CREATE INDEX CONCURRENTLY leaves an INVALID index that has" + say "to be dropped, and any statement that did succeed will run a second time." + fi +fi + +say "" +say "$((applied_now + skipped_now)) of ${PENDING_COUNT} are recorded in ${TRACKING_TABLE} and will" +say "not run again:" + +if [ "$applied_now" -eq 0 ]; then + say " (none by this run)" +else + printf '%s' "$applied_list" >&2 +fi +say "" +say "Fix the failing migration IN PLACE — it is not recorded, so its checksum is" +say "not history yet — then run this again. Never edit one that already" +say "succeeded; --status lists which those are." +exit 4 diff --git a/docs/architecture/scripts/prove-guard.sh b/docs/architecture/scripts/prove-guard.sh new file mode 100755 index 0000000..4914d05 --- /dev/null +++ b/docs/architecture/scripts/prove-guard.sh @@ -0,0 +1,151 @@ +#!/usr/bin/env bash +# +# Prove a guard fails before you believe it passes. +# +# ## The failure this catches +# +# A guard that cannot fail is worse than no guard, because it is trusted. +# `docs/architecture/GUARDS.md` opens with that sentence and its first rule is +# this procedure, written out as a manual recipe: back the file up, break exactly +# the thing the guard protects, run the guard, expect one failure, restore. +# +# The recipe is thirty seconds and it is skipped anyway, for two reasons this +# script removes: +# +# - **Restoring is a step you can forget**, and forgetting is silent. The tests +# pass again once the mutation is undone in your head but not on disk, so the +# reverted code ships looking green. Here the restore is a `trap`, which runs +# on success, on failure, and on Ctrl-C. +# - **Counting the failures is the part people skip.** GUARDS.md §1: "If +# breaking the guard's target fails three tests, two of them are coincidental +# and will mask a real regression later." A human doing this by hand sees red +# and stops reading. +# +# ## Usage +# +# bash scripts/prove-guard.sh +# +# bash scripts/prove-guard.sh src/lib/thing.ts \ +# 'if (body.error)' 'if (false)' \ +# npx vitest run tests/thing.test.ts +# +# Everything after the third argument is the command that runs the guard, so any +# runner works. `$PROVE_GUARD_CMD` is used when no command is given. +# +# ## Counting the failures +# +# "Exactly one" is a claim about test *cases*, and counting matching log lines +# does not measure that: Gradle reports a single failing test on six lines — the +# task, the test, its assertion, the summary, and twice more for the build — and +# a naive count calls that six coincidental failures. Tried that first; it fired +# on the very first run against a guard that was behaving perfectly. +# +# So the summary line is preferred, because almost every runner prints one and it +# is the runner's own count: `1 failed` from vitest, `6 tests completed, 1 failed` +# from Gradle, `1 failed, 5 passed` from pytest. The **last** such line wins, and +# only if none is found does it fall back to counting lines matching +# `$PROVE_GUARD_FAIL_PATTERN` — saying so, because an approximate count presented +# as an exact one is the kind of thing this script exists to object to. +# +# ## Exit codes +# +# 0 the guard caught it, and nothing else did — the outcome you want +# 1 the guard stayed GREEN with its target broken. It is not testing what you +# think it is, and you have just learned that for the price of one edit +# 2 nothing was proven: bad arguments, missing file, or a find-string that is +# absent or ambiguous. **Two is not a pass** +# 3 the guard caught it, but so did something else. Red for more than one +# reason hides the next regression behind a failure you have learned to +# expect — narrow the guard, or the mutation +# +# The file is restored in every one of those cases. +set -euo pipefail + +FAIL_PATTERN="${PROVE_GUARD_FAIL_PATTERN:-(FAIL|✗|[0-9]+ (tests? )?failed|FAILED|AssertionError)}" + +if [ "$#" -lt 3 ]; then + sed -n '2,30p' "$0" >&2 + exit 2 +fi + +FILE="$1"; FIND="$2"; REPLACE="$3"; shift 3 + +if [ "$#" -gt 0 ]; then + CMD=("$@") +elif [ -n "${PROVE_GUARD_CMD:-}" ]; then + # shellcheck disable=SC2206 + CMD=($PROVE_GUARD_CMD) +else + echo "prove-guard: no test command given and PROVE_GUARD_CMD is unset." >&2 + echo "Nothing was proven, which is not the same as nothing being wrong." >&2 + exit 2 +fi + +[ -f "$FILE" ] || { echo "prove-guard: no such file: $FILE" >&2; exit 2; } + +BACKUP="$(mktemp)" +cp "$FILE" "$BACKUP" +restore() { + cp "$BACKUP" "$FILE" + rm -f "$BACKUP" + echo "prove-guard: restored $FILE" +} +trap restore EXIT INT TERM + +# Exact-string replacement, and it must be unique. A mutation that lands in two +# places proves nothing about either, and a regex here would make the mutation +# itself the thing to debug. +python3 - "$FILE" "$FIND" "$REPLACE" <<'PY' +import sys +path, find, replace = sys.argv[1], sys.argv[2], sys.argv[3] +text = open(path, encoding="utf-8").read() +count = text.count(find) +if count == 0: + sys.exit(f"prove-guard: the string to break is not in {path}") +if count > 1: + sys.exit( + f"prove-guard: {count} occurrences of that string; a mutation in " + "two places proves neither. Pick a longer, unique one." + ) +open(path, "w", encoding="utf-8").write(text.replace(find, replace)) +PY + +LOG="$(mktemp)" +trap 'restore; rm -f "$LOG"' EXIT INT TERM + +echo "prove-guard: broke $FILE — expecting '${CMD[*]}' to go red" +echo + +if "${CMD[@]}" >"$LOG" 2>&1; then + echo "prove-guard: FAILED — the guard stayed GREEN with its target broken." >&2 + echo >&2 + echo "It is not checking what you think. Either the assertion does not reach" >&2 + echo "the mutated code, or it would pass without it. Log: $LOG" >&2 + tail -20 "$LOG" >&2 + exit 1 +fi + +echo "--- what failed ---" +grep -E "$FAIL_PATTERN" "$LOG" | head -12 || true +echo + +# The runner's own count, from the last summary line that states one. Preferred +# over counting log lines for the reason in the header: one failing test is +# routinely reported on half a dozen lines. +COUNT="$(grep -oiE '[0-9]+ (tests? )?failed' "$LOG" | tail -1 | grep -oE '^[0-9]+' || true)" +COUNTED_BY="the runner's summary" +if [ -z "$COUNT" ]; then + COUNT="$(grep -cE "$FAIL_PATTERN" "$LOG" || true)" + COUNTED_BY="matching log lines, approximately — this runner printed no summary" +fi + +if [ "$COUNT" -gt 1 ]; then + echo "prove-guard: the guard caught it — but $COUNT failures, by $COUNTED_BY." + echo + echo "GUARDS.md §1: if breaking one thing fails three tests, two are" + echo "coincidental and will mask a real regression later behind a red you have" + echo "learned to expect. Narrow the guard, or narrow the mutation." + exit 3 +fi + +echo "prove-guard: good — the guard caught it, and only it ($COUNTED_BY)." diff --git a/docs/architecture/scripts/release-notes.mjs b/docs/architecture/scripts/release-notes.mjs new file mode 100755 index 0000000..8822510 --- /dev/null +++ b/docs/architecture/scripts/release-notes.mjs @@ -0,0 +1,605 @@ +#!/usr/bin/env node +/** + * Tags a release and records it on Forgejo, with notes built from the commits. + * + * ## Why this exists + * + * `release.sh` published an image and stopped. Forty-eight versions were cut + * that way, and the only record any of them existed was a registry entry and a + * `chore(release):` commit — so answering "what changed between these two + * images" meant reading the log by hand. This writes that answer down once, at + * the moment the facts are still known. + * + * ## The boundary is the previous release commit, not a tag + * + * There were no tags at all when this was written, so "since the last tag" + * had nothing to start from. Every release leaves a `chore(release): vX.Y.Z` + * commit, and the one before this release is exactly the range wanted — which + * works identically before and after tags exist, and needs no state file. + * + * ## Why the subjects are scrubbed before they are published + * + * The body goes to a **public** repository and is assembled from commit + * subjects nobody wrote with that in mind. `redact` is the same scrubber the + * daily standup uses, imported rather than reimplemented — a credential-shaped + * string in a subject would otherwise be published verbatim, and nobody would + * notice. + * + * `noDashes` is deliberately *not* applied, though `cleanQuoted` bundles it: + * that is a Discord house-style rule, and the subjects in this repository use + * em dashes on purpose. + * + * ## What it refuses to do + * + * Move a tag. `release.sh` already refuses to overwrite a published image tag — + * "a published tag is not moved" — and a git tag pointing at a different commit + * is the same fact about a different artifact. Re-running after a partial + * failure is expected and safe; rewriting history is not. + * + * scripts/release-notes.mjs v0.54.31 + * scripts/release-notes.mjs v0.54.31 --dry-run + */ +import { execFileSync } from "node:child_process"; +import { readFileSync } from "node:fs"; + +/** + * The scrubber and the escaper, carried rather than imported. + * + * This file used to `import { escapeMarkdown, redact } from "../notices/compose.mjs"`, + * which is fine here and fatal the moment the script is copied: the Template + * has no `notices/`, and a release script that cannot resolve its own imports + * is one nobody runs. `compose.mjs` earns its portability by having no imports + * at all; this now does the same. + * + * `REDACTIONS` is therefore a third copy of the list in `src/lib/log.ts` and + * `notices/compose.mjs`, and a third copy of a credential scrubber is exactly + * the kind of thing that rots quietly. `tests/release-notes.test.ts` extracts + * this block and asserts it is identical to the one the worker holds — the same + * device `tests/notice-message.test.ts` already uses for the second copy. Two + * lists that drift are the danger; lists a test refuses to let drift are a copy. + * + * This matters more here than almost anywhere: the body is assembled from + * commit subjects nobody wrote expecting publication, and it is published to a + * repository that is public. + */ +const REDACTIONS = [ + [/\/\/[^/@\s:]+:[^/@\s]+@/g, "//@"], + [/([?&](?:token|key|secret|password|access_token|api_key)=)[^&\s]+/gi, "$1"], + [/pllc_[a-z]+_[0-9a-f]{8,}/g, ""], + [/v2:[0-9a-f]{8}:[^\s"]+/g, ""], + [/\b(Bearer|token|Basic)\s+[A-Za-z0-9._~+/=-]{12,}/gi, "$1 "], + [/\b[0-9a-f]{40,}\b/gi, ""], +]; + +function redact(text) { + return REDACTIONS.reduce((out, [pattern, replacement]) => out.replace(pattern, replacement), text); +} + +/** + * Newlines flattened, markup escaped — `compose.mjs`'s reasoning, unchanged. + * + * `#`, `>` and `-` are markup only at the start of a line, so escaping them + * inline would turn `My-Project` into `My\-Project` in every message. + * Flattening to one line is what actually removes the risk. `[` and `(` are the + * pair that matter most: they are how a subject becomes a working link. + */ +function escapeMarkdown(text) { + return text.replace(/\r?\n/g, " ").replace(/([\\`*_~|[\]()])/g, "\\$1"); +} + +/** + * The section each commit type gets, in the order they are printed. + * + * The keys are `.githooks/commit-msg`'s closed type list, which is the whole + * reason this grouping is possible: every commit is already labelled with the + * section it belongs in, enforced at the moment it is written rather than + * inferred afterwards. A type added there and not here lands in `other`, which + * is visible rather than silent. + */ +export const SECTIONS = [ + ["feat", "🚀 Features"], + ["fix", "🐛 Fixes"], + ["security", "🔒 Security"], + ["perf", "⚡ Performance"], + ["ui", "🎨 Interface"], + ["refactor", "🧹 Refactoring"], + ["docs", "📚 Documentation"], + ["test", "🧪 Tests"], + ["chore", "🔧 Tooling"], + ["other", "📦 Other"], +]; + +/** `type(scope)!: subject`, the shape `.githooks/commit-msg` enforces. */ +const CONVENTIONAL = /^([a-z]+)(?:\([a-z0-9._-]+\))?!?: (.+)$/; + +/** + * Commits grouped by the type they declared. + * + * Takes `sha subject` lines. Two rules earn their place: + * + * `chore(release)` commits are dropped — they are the boundaries of the range, + * not content in it, and listing "chore(release): v0.54.30" under Tooling would + * be the release describing itself. + * + * Everything else that does not parse goes to `other` rather than being + * skipped. `commit-msg` exempts `Merge`, `Revert`, `fixup!`, `squash!` and + * `amend!`, so unparseable subjects genuinely occur — and a commit vanishing + * from the notes is the failure this file exists to prevent, not a tidiness + * problem. + */ +export function groupCommits(lines) { + const groups = new Map(SECTIONS.map(([key]) => [key, []])); + + for (const line of lines) { + const trimmed = String(line ?? "").trim(); + + if (!trimmed) continue; + + const split = trimmed.indexOf(" "); + const sha = split === -1 ? trimmed : trimmed.slice(0, split); + const subject = split === -1 ? "" : trimmed.slice(split + 1); + + if (/^chore\(release\)/.test(subject)) continue; + + const match = CONVENTIONAL.exec(subject); + const key = match && groups.has(match[1]) ? match[1] : "other"; + + groups.get(key).push({ sha, subject: match ? match[2] : subject }); + } + + return groups; +} + +/** + * The release body. + * + * Every subject is scrubbed then escaped, in that order: `redact` removes what + * must not be published, `escapeMarkdown` stops what remains from becoming + * markup. `escapeMarkdown` also flattens newlines, so a subject cannot start a + * line it is not allowed to start. + * + * An empty range still produces a body. A release with no commits behind it is + * a real thing — a re-cut, a version bump — and saying so is better than an + * empty section or a missing record. + */ +export function renderBody({ + tag, + image, + digest, + groups, + signoff, + milestones = [], + deployNote = DEFAULTS.deployNote, +}) { + const clean = (text) => escapeMarkdown(redact(String(text ?? ""))); + const lines = [`Welcome to **${clean(tag)}**.`, ""]; + const listed = SECTIONS.filter(([key]) => (groups.get(key) ?? []).length > 0); + + if (listed.length === 0) { + lines.push("No commits landed between this release and the one before it.", ""); + } else { + lines.push("## What's Changed", ""); + + for (const [key, heading] of listed) { + lines.push(`### ${heading}`); + + for (const commit of groups.get(key)) { + lines.push(`- ${clean(commit.subject)} (\`${clean(commit.sha)}\`)`); + } + + lines.push(""); + } + } + + // Named before the image, because "which batch shipped" is the question a + // release closes and the image tag is the detail underneath it. + if (milestones.length > 0) { + lines.push("## Milestones completed", ""); + + for (const title of milestones) lines.push(`- ${clean(title)}`); + + lines.push(""); + } + + lines.push("## The image", ""); + lines.push(` ${image}:${tag}`); + + // Absent rather than invented. A digest is what makes a tag checkable, and a + // release claiming one it did not read would be worse than one that stays + // quiet about it. + if (digest) lines.push(` ${digest}`); + + lines.push(""); + + // Publishing and deploying are separate decisions, and saying so is the one + // sentence a reader of these notes most needs. *How* they are separate is + // each project's own business, which is why it is a setting rather than a + // string: naming one project's stack in another's release notes is exactly + // the hard-coding this directory refuses. + // + // Set RELEASE_DEPLOY_NOTE to describe how this project actually deploys, or + // to an empty string where publishing and deploying are one act. A note that + // describes a workflow the project has moved past is worse than none, + // because it is read as current. + if (deployNote) { + lines.push(...deployNote.split("\n"), ""); + } + + // The sign-off marks a batch landing, not every patch. + // + // Forty-eight releases were cut in five days before this existed. An image on + // every one of them is wallpaper; on the release that finishes a batch it + // means something. The condition is the same fact that drives the version + // bump, so the two cannot disagree: a release either completed a milestone or + // it did not. + // + // The tag rides along as a query string, and it is not decoration. + // + // The asset ships in the image, so a release published *before* that image is + // deployed points at a URL the origin does not yet serve — and the CDN in + // front of it caches that 404 for four hours, which outlives the deploy. It + // happened on the first release cut this way: the file was in the container + // and the sibling asset served 200, while this one answered 404 from cache. + // + // A per-release URL cannot inherit another release's cached miss, and it + // re-fetches when the asset itself changes. + if (signoff && milestones.length > 0) { + const separator = signoff.includes("?") ? "&" : "?"; + + lines.push(`![](${signoff}${separator}v=${encodeURIComponent(tag)})`); + } + + return `${lines.join("\n").trimEnd()}\n`; +} + +const DEFAULTS = { + image: process.env.RELEASE_IMAGE ?? "registry.example/owner/project", + signoff: + process.env.RELEASE_SIGNOFF_URL ?? "", + repo: process.env.RELEASE_REPO ?? "owner/project", + // What separates publishing from deploying here. Overridable, and settable to + // an empty string by a project where the two are the same act. + deployNote: + process.env.RELEASE_DEPLOY_NOTE ?? + "Publishing is not deploying. `deploy.py` moves a running stack to a published image.", + registryEnv: + process.env.RELEASE_REGISTRY_ENV ?? + `${process.env.HOME}/.openclaw/docker-registry.env`, +}; + +const say = (message) => console.error(`release-notes: ${message}`); +const die = (message) => { + say(message); + process.exit(1); +}; + +const git = (...args) => + execFileSync("git", args, { encoding: "utf8", timeout: 60_000 }).trim(); + +/** + * The same, with git's own stderr discarded. + * + * For probes where "no such thing" is the expected answer. `rev-parse` on a tag + * that does not exist prints a four-line `fatal:` block before returning + * non-zero, and a run that is working correctly should not look like one that + * broke. + */ +const gitQuiet = (...args) => + execFileSync("git", args, { + encoding: "utf8", + timeout: 60_000, + stdio: ["ignore", "pipe", "ignore"], + }).trim(); + +/** When the previous release was cut, as an ISO string, or null on the first. */ +/** + * The release this one follows, whether or not its own commit exists yet. + * + * `release.sh` asks twice, at two different moments, and that is the whole + * reason this is a function rather than `markers[1]`: + * + * - **before the bump**, to decide whether a batch landed and the version + * should take a minor rather than a patch (`release.sh:378`). HEAD is still + * the last ordinary commit, so the most recent `chore(release):` *is* the + * previous release. + * - **after the commit**, to write the notes (`release.sh:658`). Now the most + * recent one is this release, and the previous is the one behind it. + * + * Taking `markers[1]` unconditionally is right only in the second case. In the + * first it names the release *before* the previous one, which widened the + * window by a whole release: v0.56.0 re-announced "Batch 22", already claimed + * by v0.55.0 two minutes after that milestone closed, and took a minor bump for + * a batch that had landed in the release before it. It also made `--dry-run` + * print a commit list a release too long, so the one artifact meant to be read + * by eye before publishing did not match what would publish. + * + * `markers[0] === headSha` rather than the subject alone: a `chore(release):` + * commit somewhere behind HEAD must not make HEAD look like one. + */ +function previousReleaseCommit(run = git) { + const markers = run("log", "--grep=^chore(release):", "--format=%H", "-n", "2") + .split("\n") + .filter(Boolean); + + if (markers.length === 0) { + return null; + } + + const [headSha, headSubject] = run("log", "-1", "--format=%H%n%s").split("\n"); + const headIsRelease = markers[0] === headSha && /^chore\(release\):/.test(headSubject ?? ""); + + return headIsRelease ? (markers[1] ?? null) : markers[0]; +} + +/** Exported for the test that pins the two moments apart. */ +export const _previousReleaseCommit = previousReleaseCommit; + +export function previousReleaseAt() { + const marker = previousReleaseCommit(); + + return marker ? git("show", "-s", "--format=%cI", marker) : null; +} + +/** + * The batches this release finishes. + * + * A milestone closed since the previous release was cut. That is the unit work + * is actually planned in here — batches in the tracker — and it is what makes a + * release worth marking: forty-eight were cut in five days, and a sign-off on + * every one of them is wallpaper. + * + * Throws rather than returning `[]` when it cannot ask. "I could not check" and + * "no batch landed" are different facts and only one of them is safe to act on + * — this one decides both the version bump and the sign-off, so a silent empty + * answer would quietly downgrade a release nobody meant to downgrade. + */ +export async function closedMilestonesSince(since, env) { + const url = + `https://${env.FORGEJO_REGISTRY}/api/v1/repos/${DEFAULTS.repo}` + + "/milestones?state=closed&limit=50"; + const auth = Buffer.from( + `${env.FORGEJO_REGISTRY_USER}:${env.FORGEJO_REGISTRY_TOKEN}`, + ).toString("base64"); + + const response = await fetch(url, { + headers: { Authorization: `Basic ${auth}`, Accept: "application/json" }, + signal: AbortSignal.timeout(30_000), + redirect: "manual", + }); + + if (!response.ok) throw new Error(`milestones answered HTTP ${response.status}`); + + const all = await response.json(); + + if (!Array.isArray(all)) throw new Error("milestones did not answer with a list"); + + // No boundary means the first release ever cut this way; every closed + // milestone predates it, and claiming all of them would be a lie about what + // this release contains. + if (!since) return []; + + const after = Date.parse(since); + + return all + .filter((milestone) => { + const closed = Date.parse(milestone?.closed_at ?? ""); + + return Number.isFinite(closed) && closed > after; + }) + .sort((a, b) => Date.parse(a.closed_at) - Date.parse(b.closed_at)) + .map((milestone) => String(milestone.title ?? "").trim()) + .filter(Boolean); +} + +/** The commits since the previous release, newest first. */ +export function commitsSince() { + // No previous release commit means this is the first one ever cut this way. + // The whole history is the range rather than nothing — an empty body on a + // first release would be the least useful moment to be silent. + const marker = previousReleaseCommit(); + const range = marker ? `${marker}..HEAD` : "HEAD"; + + return git("log", range, "--format=%h %s", "--no-merges").split("\n").filter(Boolean); +} + +/** + * Credentials, from the file outside the repository. + * + * The same file and the same three names `release.sh` sources. Read here rather + * than inherited, so this can be re-run by hand after a failure without the + * caller having to arrange an environment. + */ +function credentials() { + const env = { ...process.env }; + + try { + for (const line of readFileSync(DEFAULTS.registryEnv, "utf8").split("\n")) { + const match = /^([A-Z_]+)=(.*)$/.exec(line.trim()); + + if (match) env[match[1]] = match[2].replace(/^["']|["']$/g, ""); + } + } catch { + // Absent is not fatal here; the check below names what is missing. + } + + // All three, not two. `release.sh` guards on the host and the token and then + // dereferences the user unguarded — under `set -u` that aborts inside a + // command substitution and presents as "could not list published versions". + for (const name of ["FORGEJO_REGISTRY", "FORGEJO_REGISTRY_USER", "FORGEJO_REGISTRY_TOKEN"]) { + if (!env[name]) { + die(`${name} is not set and not in ${DEFAULTS.registryEnv} — cannot record the release.`); + } + } + + return env; +} + +/** The tag, created only if it does not already point somewhere else. */ +function ensureTag(tag, dryRun) { + const head = git("rev-parse", "HEAD"); + let existing = ""; + + try { + existing = gitQuiet("rev-parse", `refs/tags/${tag}`); + } catch { + existing = ""; + } + + if (existing && existing !== head) { + die( + `${tag} already exists and points at ${existing.slice(0, 7)}, not ${head.slice(0, 7)}. ` + + "A published tag is not moved.", + ); + } + + if (existing) { + say(`${tag} already tagged here — continuing.`); + } else if (dryRun) { + say(`[dry-run] git tag -a ${tag}`); + } else { + git("tag", "-a", tag, "-m", `Release ${tag}`); + say(`tagged ${tag}.`); + } + + if (dryRun) { + say(`[dry-run] git push origin ${tag}`); + + return; + } + + try { + git("push", "origin", tag); + say(`pushed ${tag}.`); + } catch (error) { + // The shape `.githooks/post-commit` uses: say what is safe, say what to do, + // never force, and do not take down the thing that called us. + say(`could not push ${tag}: ${error.message.split("\n")[0]}`); + say(` The tag is safe locally. Push it when the remote is reachable:`); + say(` git push origin ${tag}`); + } +} + +/** The release itself. A 409 means somebody already recorded it, which is done. */ +async function publish(tag, body, env, dryRun) { + const url = `https://${env.FORGEJO_REGISTRY}/api/v1/repos/${DEFAULTS.repo}/releases`; + + if (dryRun) { + say(`[dry-run] POST ${url}`); + process.stdout.write(`${body}\n`); + + return true; + } + + const auth = Buffer.from(`${env.FORGEJO_REGISTRY_USER}:${env.FORGEJO_REGISTRY_TOKEN}`).toString( + "base64", + ); + + let response; + + try { + response = await fetch(url, { + method: "POST", + headers: { + Authorization: `Basic ${auth}`, + "Content-Type": "application/json", + Accept: "application/json", + }, + body: JSON.stringify({ tag_name: tag, name: tag, body }), + signal: AbortSignal.timeout(30_000), + redirect: "manual", + }); + } catch (error) { + say(`the release call failed: ${error.message}`); + + return false; + } + + if (response.status === 409) { + say(`a release for ${tag} already exists — nothing to do.`); + + return true; + } + + if (!response.ok) { + const detail = (await response.text().catch(() => "")).slice(0, 200); + + // Status first, and an HTML body diagnosed rather than dumped — a Forgejo + // error body can echo the request, and Cloudflare's does not look like one + // at all. + say( + `the release call answered HTTP ${response.status}${ + detail.toLowerCase().includes(" null); + + say(`recorded ${tag}${created?.html_url ? ` — ${created.html_url}` : ""}.`); + + return true; +} + +async function main(argv) { + const dryRun = argv.includes("--dry-run"); + + // Asked before the version is chosen, so the bump and the sign-off are driven + // by the same fact. Prints one title per line and nothing else, because the + // caller is a shell reading it. + if (argv.includes("--closed-milestones")) { + const env = credentials(); + const titles = await closedMilestonesSince(previousReleaseAt(), env); + + for (const title of titles) process.stdout.write(`${title}\n`); + + return; + } + + const tag = argv.find((arg) => !arg.startsWith("--")); + + if (!tag) die("usage: release-notes.mjs [--dry-run] | --closed-milestones"); + if (!/^v\d+\.\d+\.\d+$/.test(tag)) die(`'${tag}' is not a vX.Y.Z tag.`); + + const env = credentials(); + + // Passed down from `release.sh`, which already asked in order to choose the + // bump. Asking twice would risk two answers for one release — a milestone + // closed between the two calls would bump the version without earning the + // sign-off, or the reverse. + const milestones = (process.env.RELEASE_MILESTONES ?? "") + .split("\n") + .map((line) => line.trim()) + .filter(Boolean); + + const body = renderBody({ + tag, + image: DEFAULTS.image, + digest: process.env.RELEASE_DIGEST ?? "", + groups: groupCommits(commitsSince()), + signoff: DEFAULTS.signoff, + milestones, + }); + + ensureTag(tag, dryRun); + + if (!(await publish(tag, body, env, dryRun))) { + say(` The image IS published as ${tag} and the commit is made; only the`); + say(` release record is missing. Run: scripts/release-notes.mjs ${tag}`); + process.exit(1); + } +} + +// Effects only when run, so a test can import the two pure functions above. +if (process.argv[1] && import.meta.url.endsWith(process.argv[1].split("/").pop())) { + await main(process.argv.slice(2)); +} diff --git a/docs/architecture/scripts/release.sh b/docs/architecture/scripts/release.sh new file mode 100755 index 0000000..21dbafc --- /dev/null +++ b/docs/architecture/scripts/release.sh @@ -0,0 +1,713 @@ +#!/usr/bin/env bash +# +# Build the container, push it, and leave every version string in the repo +# agreeing with the tag that was published. +# +# npm run release # patch: 0.46.0 -> 0.46.1 +# npm run release -- minor # 0.46.0 -> 0.47.0 +# npm run release -- 1.0.0 # exact +# npm run release -- --dry-run +# npm run release -- --no-prune # keep every published version +# RELEASE_KEEP=10 npm run release # keep ten instead of two +# +# =========================================================================== +# TEMPLATE COPY — configure this before the first run +# =========================================================================== +# +# Copy to `scripts/release.sh`, add `"release": "bash scripts/release.sh"` to +# package.json, and set the five values in the CONFIGURATION block below. The +# script refuses to run until they are set: it has no defaults, deliberately. +# +# The version it was adapted from carried its origin project's image name, +# deploy host and container name as defaults. Copied into a second project and +# run once, that would have built the new project's code, pushed it OVER the +# first project's image, pruned the first project's published versions, and +# read the first project's running container to decide what was safe to delete. +# Every one of those is silent and none is reversible. Hence: no defaults, and +# a loud failure instead. +# +# Assumes: bash, git, docker, ssh, and a Node project (`npm version` does the +# semver arithmetic). For a non-Node project, replace the two `npm version` +# calls and the `node -p` reader; everything else is language-agnostic. +# +# Assumes a registry with a Forgejo-compatible package API for the prune. If +# yours differs, the prune is the only part to rewrite — it is at the bottom +# and nothing else depends on it. +# +# ## Why this exists +# +# The project this came from had no build or push command at all. `git grep +# "docker push"` found nothing, there was no CI, and the README documented a +# `docker build` line and never the push. Every release was two commands typed +# from memory, and the version bump was a third thing to remember afterwards. +# +# It was not remembered. `package.json` sat at 0.2.0 across twenty-nine image +# releases because nothing read it and nothing checked it; the README and the +# example compose file stayed pinned at v0.40.01 while production ran v0.46.0. +# The git history shows the ritual dying in stages: one `chore(release)` commit +# touched only `.dockerignore`, and two others were empty. +# +# So the version bump is not a step beside the release. It is what the release +# command does, and a test fails when the files disagree. Write that test — see +# "The guard test" at the bottom of this header. +# +# ## package.json is the source of truth +# +# The tag is always `v` + the version in package.json. Nothing else decides it, +# which is why there is no --tag flag: a flag would be a second source of truth +# and this whole script exists because there were four. +# +# Arithmetic is `npm version`, which is built in and enforces semver. That is +# load-bearing rather than incidental — it makes a tag like `v0.40.01` (leading +# zero in the patch, not valid semver, and a real tag in this registry) +# impossible to produce from now on. +# +# ## The ordering is the safety property +# +# Bump, build, push, and commit LAST. The rule is: never pass through a state +# you cannot explain to somebody reading the repository afterwards. +# +# If the build or the push fails, the edits are sitting in the working tree — +# visible, uncommitted, unpushed, one `git checkout` from gone. If the commit +# came first, a failure would leave `main` carrying a commit that announces a +# release which was never published, and `post-commit` would have already +# pushed it to everyone. One of those is recoverable by noticing; the other is +# not. +# +# ## What it deliberately does not do +# +# It does not deploy. Whatever runs the container pins a tag, and updating that +# pin stays a separate, deliberate act. Publishing an image and running it are +# two decisions and this script only makes the first one — so a bad build sits +# in a registry rather than in production. +# +# ## The guard test +# +# This script is only half the mechanism. The other half is a test asserting +# that every file in FILES/PIN_FILES names the same version, so a hand-edit or +# a half-finished release fails the suite instead of shipping. Without it, +# nothing notices the drift this script was written to end — which is exactly +# how the original got to four disagreeing version strings. + +set -uo pipefail + +cd "$(git rev-parse --show-toplevel)" || exit 1 + +# --------------------------------------------------------------------------- +# CONFIGURATION — set these five, then delete this banner. +# +# Every one is empty on purpose. See the note at the top: inherited defaults +# from another project point a release at that project's image, registry and +# running container, and every consequence of that is silent. +# --------------------------------------------------------------------------- + +# Fully-qualified image name, no tag. e.g. registry.example.com/team/my-app +IMAGE="${RELEASE_IMAGE:-}" + +# The public origin, when the build bakes one in. +# +# Left empty by default: a project whose artifact carries no origin should not +# have one invented for it. But when it is set it is usually frozen into the +# build — canonical URLs, Open Graph tags, robots.txt, sitemaps — and cannot be +# corrected without another build, so a wrong value ships silently and is found +# by somebody else. Validated below rather than trusted. +SITE_URL="${RELEASE_SITE_URL:-}" + +# A file sourced for FORGEJO_REGISTRY / FORGEJO_REGISTRY_USER / +# FORGEJO_REGISTRY_TOKEN. Keep it OUTSIDE the repository — a token in a file +# the repo can see is a token one `git add -A` away from being published. +REGISTRY_ENV="${RELEASE_REGISTRY_ENV:-}" + +# The files that carry a version string. Every one of them is checked before +# the build and staged after the push; the guard test asserts they agree. +# +# FILES is what `npm version` rewrites and must exist. PIN_FILES carry an +# `IMAGE:vX.Y.Z` pin rewritten by search-and-replace; list only the ones this +# project actually has. A project with no compose file leaves PIN_FILES holding +# just the README, or empty — an absent file is skipped with a warning rather +# than failing the release, because "you do not have that file" and "the +# rewrite silently matched nothing" need different answers. +FILES=(package.json package-lock.json) +PIN_FILES=(README.md docker-compose.example.yml) + +say() { printf '\033[1mrelease:\033[0m %s\n' "$*" >&2; } +die() { printf '\033[1mrelease:\033[0m %s\n' "$*" >&2; exit 1; } + +# Refused, never corrected: guessing what somebody meant by a malformed origin +# is how the wrong one gets baked in with a reassuring log line above it. +if [ -n "$SITE_URL" ]; then + case "$SITE_URL" in + https://*) + printf '%s' "$SITE_URL" \ + | grep -Eq '^https://[A-Za-z0-9][A-Za-z0-9.-]*\.[A-Za-z]{2,}/?$' \ + || die "RELEASE_SITE_URL is not a bare https origin: ${SITE_URL} + Expected something like https://example.com, with no path." ;; + *) + die "RELEASE_SITE_URL must be an https origin, got: ${SITE_URL}" ;; + esac +fi + +BUMP="patch" +DRY_RUN="" +NO_PRUNE="" + +# How many versions survive a prune, newest first — published AND local. +# +# Two is the floor the check below enforces, and it is now also the default: +# the newest, plus one to roll back to. It is deliberately tight because the +# count that used to matter — how much of the registry we keep — turned out not +# to be the expensive one. Releases here are frequent and the deployed tag is +# spared on top of this number whatever it is, so the practical floor is three +# images, not two. +KEEP="${RELEASE_KEEP:-2}" + +# Where the running stack lives, so a prune can find out what is deployed and +# refuse to delete it. See the prune section for why this is not optional. +# +# DEPLOY_HOST is an ssh destination — a Host in ~/.ssh/config, not a password +# prompt: the lookup runs with BatchMode=yes and a host that asks for input is +# read as "could not determine", which correctly stops the prune. +DEPLOY_HOST="${RELEASE_DEPLOY_HOST:-}" +DEPLOY_CONTAINER="${RELEASE_DEPLOY_CONTAINER:-}" + +# --------------------------------------------------------------------------- +# Refuse to run half-configured. +# +# Checked here, before the first side effect, and named one at a time so the +# message says which value is missing rather than "configuration error". +# --------------------------------------------------------------------------- +[ -n "$IMAGE" ] || die "set IMAGE (or RELEASE_IMAGE) — the image name to build and push. See the CONFIGURATION block." +[ -n "$REGISTRY_ENV" ] || die "set REGISTRY_ENV (or RELEASE_REGISTRY_ENV) — the file holding the registry credentials." +[ -n "$DEPLOY_HOST" ] || die "set DEPLOY_HOST (or RELEASE_DEPLOY_HOST) — the ssh host running the deployed container." +[ -n "$DEPLOY_CONTAINER" ] || die "set DEPLOY_CONTAINER (or RELEASE_DEPLOY_CONTAINER) — the container name to read the deployed version from." + +# Refused rather than defaulted, because the failure is a publication. +# +# Docker treats the first path component as a registry only when it contains a +# dot or a colon (or is localhost); anything else is a Docker Hub namespace. So +# `acme/my-app` is not a private registry that happens to be unreachable — it is +# Docker Hub, and a successful push there makes a private project public. That +# is not a mistake to discover from the registry's web UI. +case "${IMAGE%%/*}" in + *.*|*:*|localhost) : ;; + *) die "IMAGE ('$IMAGE') has no registry host — '${IMAGE%%/*}' is a Docker Hub + namespace, so pushing would publish this image publicly. Use a + fully-qualified name like registry.example.com/team/app." ;; +esac + +for arg in "$@"; do + case "$arg" in + --dry-run) DRY_RUN="yes" ;; + --no-prune) NO_PRUNE="yes" ;; + -*) die "unknown flag $arg. Usage: npm run release -- [patch|minor|major|] [--dry-run] [--no-prune]" ;; + *) BUMP="$arg" ;; + esac +done + +case "$KEEP" in + ''|*[!0-9]*) die "RELEASE_KEEP must be a whole number, got '$KEEP'." ;; +esac + +# Keeping zero would delete the tag this run just published. Refused rather than +# clamped, because a caller who typed 0 meant something and it was not that. +[ "$KEEP" -ge 2 ] || die "RELEASE_KEEP must be at least 2 — keeping fewer leaves nothing to roll back to." + +# What production is actually running, asked once and remembered. +# +# Both prunes need it and neither may delete without it, so it is a function +# rather than a line copied twice — a second copy is a second thing to keep +# correct, and the consequence of getting it wrong is deleting the image the +# running container was created from. +# +# The empty string means "could not determine", which every caller must treat +# as a reason to stop rather than as "nothing is deployed". +DEPLOYED_VERSION="" +DEPLOYED_ASKED="" + +deployed_version() { + if [ -z "$DEPLOYED_ASKED" ]; then + DEPLOYED_ASKED="yes" + DEPLOYED_VERSION=$(ssh -o BatchMode=yes -o ConnectTimeout=10 "$DEPLOY_HOST" \ + "docker inspect --format '{{index .Config.Labels \"org.opencontainers.image.version\"}}' $DEPLOY_CONTAINER" \ + 2>/dev/null | tr -d '\r\n') + fi + + printf '%s' "$DEPLOYED_VERSION" +} + +# --------------------------------------------------------------------------- +# The same trim, applied to this machine. +# +# `prune` below deletes published versions. Nothing ever deleted the images this +# script BUILDS, so every release since the first left another ~250 MB tag on +# the disk: seventy-four of them by the time anyone looked, inside a daemon +# holding 79 GB with 47 GB reclaimable. +# +# ## Scoped to this image by name, and to nothing else +# +# `docker image prune` is one line and would have reached every project sharing +# this daemon, including a build running in another checkout at that moment. +# Dangling layers are left alone for the same reason — an untagged layer is not +# identifiable as ours, and "probably nobody's" is not ownership. This removes +# tags of $IMAGE matching vN.N.N and nothing else. +# +# ## The deployed tag is spared here too +# +# A local image is not the copy production pulls, so deleting one is far less +# dangerous than deleting a published version — but it is the copy that makes a +# rollback instant instead of a download, and it costs nothing to keep. Same +# rule, same reason: if the deployed version cannot be read, this does nothing. +# --------------------------------------------------------------------------- +prune_local() { + [ -z "$NO_PRUNE" ] || { say "prune: local images skipped (--no-prune)."; return 0; } + + local deployed + deployed=$(deployed_version) + + if [ -z "$deployed" ]; then + say "prune: local images SKIPPED — could not read the deployed version." + return 0 + fi + + local tags total keep_from doomed + tags=$(docker images --format '{{.Tag}}' "$IMAGE" 2>/dev/null \ + | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' | sort -V) + + total=$(printf '%s\n' "$tags" | grep -c .) + + if [ "$total" -le "$KEEP" ]; then + say "prune: ${total} local image(s), keeping ${KEEP} — nothing to remove." + return 0 + fi + + keep_from=$((total - KEEP)) + doomed=$(printf '%s\n' "$tags" | head -n "$keep_from" \ + | grep -v "^${deployed}$" | grep -v "^${TAG}$") + + if [ -z "$doomed" ]; then + say "prune: no local images to remove." + return 0 + fi + + if [ -n "$DRY_RUN" ]; then + say "prune: would remove $(printf '%s\n' "$doomed" | grep -c .) local image(s):" + printf ' %s\n' $doomed >&2 + return 0 + fi + + say "prune: removing $(printf '%s\n' "$doomed" | grep -c .) local image(s), keeping" + say " the newest ${KEEP} plus ${deployed} (deployed)." + + local tag + for tag in $doomed; do + # Untagging is all that is asked for. Layers shared with a kept image stay, + # and the space is reclaimed only when the last tag referencing them goes — + # which is the correct behaviour and why this reports what it untagged + # rather than claiming an amount of disk freed. + if docker rmi "${IMAGE}:${tag}" >/dev/null 2>&1; then + say " removed ${tag}" + else + # Said out loud, like its published counterpart. A container still using + # the image is the usual cause and it is not an error worth stopping for. + say " kept ${tag} (still in use, or already gone)" + fi + done +} + +prune() { + [ -z "$NO_PRUNE" ] || { say "prune: skipped (--no-prune)."; return 0; } + + if [ -z "${FORGEJO_REGISTRY:-}" ] || [ -z "${FORGEJO_REGISTRY_TOKEN:-}" ]; then + say "prune: skipped — no registry credentials." + return 0 + fi + + local owner="${IMAGE#*/}"; owner="${owner%%/*}" + local name="${IMAGE##*/}" + local api="https://${FORGEJO_REGISTRY}/api/v1/packages/${owner}/container/${name}" + + # What is actually running. Not an optimisation — see the header. + local deployed + deployed=$(deployed_version) + + if [ -z "$deployed" ]; then + say "prune: SKIPPED — could not read the deployed version from" + say " ${DEPLOY_HOST}/${DEPLOY_CONTAINER}. Refusing to delete anything" + say " without knowing which image production is running." + return 0 + fi + + local versions + versions=$(curl -sS --max-time 20 -u "${FORGEJO_REGISTRY_USER}:${FORGEJO_REGISTRY_TOKEN}" \ + "https://${FORGEJO_REGISTRY}/api/v1/packages/${owner}?type=container&q=${name}" 2>/dev/null \ + | PKG="$name" node -e 'let s="";process.stdin.on("data",d=>s+=d).on("end",()=>{try{ + console.log(JSON.parse(s).filter(p => p.name === process.env.PKG).map(p => p.version).join("\n")) + }catch{process.exit(1)}})' 2>/dev/null) + + if [ -z "$versions" ]; then + say "prune: skipped — could not list published versions." + return 0 + fi + + # Semver order, newest last. `latest` is a moving pointer rather than a + # release and is never a candidate; it is filtered before anything is counted + # so it cannot occupy one of the kept slots either. + local candidates + candidates=$(printf '%s\n' "$versions" | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' | sort -V) + + local total keep_from doomed + total=$(printf '%s\n' "$candidates" | grep -c .) + + if [ "$total" -le "$KEEP" ]; then + say "prune: ${total} published versions, keeping ${KEEP} — nothing to remove." + return 0 + fi + + keep_from=$((total - KEEP)) + doomed=$(printf '%s\n' "$candidates" | head -n "$keep_from" | grep -v "^${deployed}$" | grep -v "^${TAG}$") + + local protected + protected=$(printf '%s\n' "$candidates" | head -n "$keep_from" | grep -c "^${deployed}$" || true) + + if [ "$protected" -gt 0 ]; then + say "prune: keeping ${deployed} — it is what production is running, even" + say " though it is old enough to drop." + fi + + if [ -z "$doomed" ]; then + say "prune: nothing to remove." + return 0 + fi + + say "prune: keeping the newest ${KEEP}, plus ${deployed} (deployed) and latest." + + if [ -n "$DRY_RUN" ]; then + say "prune: would remove $(printf '%s\n' "$doomed" | grep -c .) older version(s):" + printf ' %s\n' $doomed >&2 + return 0 + fi + + say "prune: removing $(printf '%s\n' "$doomed" | grep -c .) older version(s):" + + local version code + for version in $doomed; do + code=$(curl -sS --max-time 30 -o /dev/null -w '%{http_code}' \ + -X DELETE -u "${FORGEJO_REGISTRY_USER}:${FORGEJO_REGISTRY_TOKEN}" \ + "${api}/${version}" 2>/dev/null) + + case "$code" in + 2*) say " removed ${version}" ;; + # Said out loud rather than counted as done. A prune that quietly failed + # looks exactly like a prune that had nothing to do. + *) say " FAILED ${version} (HTTP ${code:-no response}) — left in place" ;; + esac + done +} + +# --------------------------------------------------------------------------- +# Preconditions. Everything that can be checked without side effects, checked +# before the first side effect. +# --------------------------------------------------------------------------- + +command -v docker >/dev/null 2>&1 || die "docker is not on PATH." + +branch=$(git symbolic-ref --quiet --short HEAD) || die "detached HEAD — check out a branch first." +[ "$branch" = "main" ] || die "on '$branch'. Releases are cut from main." + +# Only the release files are required to be clean, not the whole tree. Other +# work legitimately shares this checkout, and refusing to release because an +# unrelated file is open would make the command useless exactly when it is most +# wanted. The commit at the end stages these paths explicitly for the same +# reason: whatever else is in the tree is not part of this release. +dirty=$(git diff --name-only -- "${FILES[@]}" "${PIN_FILES[@]}" 2>/dev/null; + git diff --cached --name-only -- "${FILES[@]}" "${PIN_FILES[@]}" 2>/dev/null) + +if [ -n "$dirty" ]; then + say "these release files have uncommitted changes:" + printf ' %s\n' $(printf '%s\n' "$dirty" | sort -u) >&2 + die "commit or discard them first — a release must start from a known version." +fi + +current=$(node -p 'require("./package.json").version') || die "cannot read package.json" + +# `npm version` writes the file, so ask it what the answer would be by asking it +# somewhere disposable. Cheaper and more honest than reimplementing semver here, +# and it means the validation rules are npm's rather than a second opinion. +work=$(mktemp -d) || die "cannot create a temporary directory" +trap 'rm -rf "$work"' EXIT +printf '{"name":"v","version":"%s"}\n' "$current" > "$work/package.json" + +if ! next=$(cd "$work" && npm version --no-git-tag-version "$BUMP" 2>&1 | tr -d 'v\n'); then + die "npm rejected '$BUMP': $next" +fi + +[ -n "$next" ] || die "could not work out the next version from '$BUMP'." + +TAG="v${next}" + +say "$current -> $next (image ${IMAGE}:${TAG})" + +# --------------------------------------------------------------------------- +# Refuse to move a tag that is already published. +# +# Overwriting one would silently change what a running stack pulls on its next +# recreate, and the old image would still be running with no way to tell from +# the tag. A published tag is immutable by convention here; this makes it +# immutable in practice. +# --------------------------------------------------------------------------- + +if [ -r "$REGISTRY_ENV" ]; then + # shellcheck disable=SC1090 + . "$REGISTRY_ENV" +fi + +if [ -n "${FORGEJO_REGISTRY:-}" ] && [ -n "${FORGEJO_REGISTRY_TOKEN:-}" ]; then + repo="${IMAGE#*/}" + tags=$(curl -sS --max-time 20 -u "${FORGEJO_REGISTRY_USER}:${FORGEJO_REGISTRY_TOKEN}" \ + "https://${FORGEJO_REGISTRY}/v2/${repo}/tags/list" 2>/dev/null) + + if printf '%s' "$tags" | grep -q "\"${TAG}\""; then + die "${TAG} is already published. Pick a higher version — a published tag is not moved." + fi + + if [ -z "$tags" ]; then + # Said out loud rather than passed over. "I could not check" and "it is not + # there" are different answers and only one of them is safe to act on. + say "WARNING: could not read the registry tag list. Proceeding without the" + say " already-published check." + fi +else + say "WARNING: no registry credentials at $REGISTRY_ENV — cannot check whether" + say " ${TAG} is already published." +fi + +if [ -n "$DRY_RUN" ]; then + say "--dry-run: nothing was changed. It would have:" + # Only the ones actually present, for the reason the whole script exists: + # a "would have" line naming a file this project does not have is a claim. + present=("${FILES[@]}") + for file in "${PIN_FILES[@]}"; do [ -f "$file" ] && present+=("$file"); done + say " set version ${next} in ${present[*]}" + say " docker build --build-arg APP_VERSION=${TAG} -t ${IMAGE}:${TAG} ." + say " docker push ${IMAGE}:${TAG}" + say " git commit -m 'chore(release): ${TAG}' (post-commit then pushes)" + say "" + say "and then pruned. That part is shown for real, because it deletes:" + prune + prune_local + exit 0 +fi + +# --------------------------------------------------------------------------- +# Bump. Files first, so the image is built from the source that names it. +# --------------------------------------------------------------------------- + +npm version --no-git-tag-version --allow-same-version "$next" >/dev/null \ + || die "npm version failed; nothing has been built or pushed." + +# The pins that drifted for six releases in the original. Anchored on the image +# name so this cannot match a version string belonging to something else. +# +# A listed file that does not exist is a configuration fact, not a failure: not +# every project has a compose file. It is said out loud rather than passed over, +# because "you do not have that file" and "the rewrite matched nothing" have the +# same appearance and only one of them is fine. +BUMPED=("${FILES[@]}") + +for file in "${PIN_FILES[@]}"; do + if [ ! -f "$file" ]; then + say "note: $file is listed in PIN_FILES but does not exist — skipping." + continue + fi + + # Only files that already carry a pin. A README that never mentioned the + # image is not a drift risk and must not become a failure. + grep -q "${IMAGE}:v" "$file" || { say "note: $file carries no ${IMAGE} pin — skipping."; continue; } + + perl -pi -e "s{\Q${IMAGE}\E:v[0-9][0-9.]*}{${IMAGE}:${TAG}}g" "$file" \ + || die "could not rewrite the pin in $file" + + # Checked, not assumed. A rewrite that silently matched nothing would produce + # a release whose own guard test fails on the next commit. + grep -q "${IMAGE}:${TAG}" "$file" || die "the pin in $file did not update — refusing to build." + + BUMPED+=("$file") +done + +say "bumped ${BUMPED[*]}" + +# --------------------------------------------------------------------------- +# Guards, before anything is built. +# +# ## Why they are here and not left to the commit hook +# +# `pre-commit` runs the typecheck and the suite, and the commit is the LAST step +# of this script — after the push. So relying on it means the gate fires once +# the image is already published and cannot be unpublished: the hook would +# refuse the commit, and the registry would be left holding a tag whose tests +# never passed, with git carrying no record of it. The first real release +# through this script did exactly that, and passed only by luck. +# +# ## A missing test database refuses the release, it does not warn about it +# +# Without TEST_DATABASE_URL the database suites skip themselves, which on a +# repository of any size is a large fraction of the tests. That is the right +# default for a laptop with no Postgres and the wrong default for cutting a +# release — and warning about it while building anyway is the worst of the +# three, because the summary above it still says the tests passed. +# --------------------------------------------------------------------------- + +if [ -z "${SKIP_GUARDS:-}" ]; then + say "typecheck…" + + if ! npx tsc --noEmit; then + say "typecheck failed. The bump is in your working tree; nothing was built," + say " published or committed." + exit 1 + fi + + # A warning is not a gate. + # + # This printed three lines saying half the suite would skip and then cut the + # release anyway. A gate that reports "tests passed" for a run which never + # touched the database is worse than no gate, because it is trusted — and the + # project this script came from shipped every release that way for months + # before anybody added up the numbers. + # + # So: use what is set, otherwise adopt a local test database if one is + # listening, and refuse if neither. The escape hatch stays for the machine + # with no database — which is what the warning was protecting — but it has to + # be asked for by name rather than being the default. + if [ -z "${TEST_DATABASE_URL:-}" ]; then + if [ -n "${RELEASE_TEST_DB_PORT:-}" ] \ + && (exec 3<>/dev/tcp/127.0.0.1/"${RELEASE_TEST_DB_PORT}") 2>/dev/null; then + exec 3<&- 2>/dev/null || true + export TEST_DATABASE_URL="${RELEASE_TEST_DB_URL:?set RELEASE_TEST_DB_URL beside RELEASE_TEST_DB_PORT}" + say "TEST_DATABASE_URL was unset; using the local test database on ${RELEASE_TEST_DB_PORT}." + elif [ -n "${RELEASE_ALLOW_SKIPPED_TESTS:-}" ]; then + say "WARNING: no test database, and RELEASE_ALLOW_SKIPPED_TESTS is set." + say " The database suites will SKIP. You are cutting a release" + say " that has not been fully tested." + else + die "no test database, so the database suites would silently skip. + + Set TEST_DATABASE_URL, or set RELEASE_TEST_DB_PORT and RELEASE_TEST_DB_URL + so this script can find the local one, or pass RELEASE_ALLOW_SKIPPED_TESTS=1 + to cut a release knowing it is half-tested." + fi + fi + + say "tests…" + + if ! npx vitest run --reporter=dot; then + say "tests failed. The bump is in your working tree; nothing was built," + say " published or committed." + exit 1 + fi +else + say "SKIP_GUARDS set — typecheck and tests did NOT run before this release." +fi + +# --------------------------------------------------------------------------- +# Build, then push. Nothing is committed until both have succeeded. +# --------------------------------------------------------------------------- + +say "building ${IMAGE}:${TAG}…" + +if ! docker build \ + --build-arg "NEXT_PUBLIC_SITE_URL=${SITE_URL}" \ + --build-arg "APP_VERSION=${TAG}" \ + -t "${IMAGE}:${TAG}" . ; then + say "build failed. The version bump is in your working tree and NOTHING was" + say " published or committed. Fix the build and run this again, or" + say " 'git checkout -- ${BUMPED[*]}' to undo the bump." + exit 1 +fi + +# Verify before trusting it — the same move a backup script makes with `pg_restore +# --list` before renaming a dump into place. The Dockerfile threads APP_VERSION +# through two stage-scoped ARGs and an ENV; drop any one of them and the build +# still succeeds, the tests still pass, and the only symptom is the settings +# screen reading "running from source" in production. Ask the image directly +# rather than trusting that the wiring is still there. +# +# `docker run` with a command overrides CMD, so this starts no server and needs +# no database. +say "verifying the image reports ${TAG}…" + +baked=$(docker run --rm --entrypoint printenv "${IMAGE}:${TAG}" APP_VERSION 2>/dev/null | tr -d '\r\n') + +if [ "$baked" != "$TAG" ]; then + # Remove the local tag. A correctly-named image with the wrong contents is a + # loaded gun for a later hand-typed `docker push`. + docker rmi "${IMAGE}:${TAG}" >/dev/null 2>&1 + say "the image reports '${baked:-nothing}' but should report ${TAG}." + say " Check ARG/ENV APP_VERSION in the runner stage of the Dockerfile." + say " Nothing was published or committed; the local image was removed." + exit 1 +fi + +say "pushing ${IMAGE}:${TAG}…" + +if ! docker push "${IMAGE}:${TAG}"; then + say "push failed. The image exists locally and the bump is in your working" + say " tree, but nothing was published or committed. Check the registry" + say " login and run this again." + exit 1 +fi + +# --------------------------------------------------------------------------- +# Commit last, by explicit path. +# --------------------------------------------------------------------------- + +git add -- "${BUMPED[@]}" || die "git add failed after a successful push — commit ${BUMPED[*]} by hand." + +# pre-commit runs the typecheck, and the suite when a .ts/.tsx is staged. None +# of these paths are, so this gets the typecheck only — which is what catches a +# package.json this script has mangled. +if ! git commit -q -m "chore(release): ${TAG}"; then + say "the commit was refused (see above). The image IS published as ${TAG};" + say " only the commit is missing. Fix and commit ${BUMPED[*]} by hand." + exit 1 +fi + +say "released ${TAG}." + +# --------------------------------------------------------------------------- +# Prune old published versions. +# +# ## Last, and deliberately so +# +# The release is complete by this point — image published, files bumped, commit +# made. Everything below is housekeeping, and housekeeping must never be able to +# fail a release that already succeeded. Nothing here exits non-zero. +# +# ## It refuses to run blind +# +# Deleting a published version is irreversible and the registry is the only copy +# — there is no `git revert` for a pushed image. The one that actually matters +# is the version PRODUCTION IS RUNNING, and that is routinely an old tag: at the +# time this was written the stack pinned v0.54.0 while v0.54.3 was the newest, +# because deploying is a separate manual step. A newest-N rule with no +# knowledge of that would eventually delete the image the running container was +# created from, and the failure surfaces later — at the next recreate, when the +# stack cannot pull the tag it pins and the site does not come back. +# +# So the deployed tag is looked up, and if it cannot be determined the prune is +# skipped entirely. "I could not check" is not "there is nothing to protect", +# and only one of those is safe to act on — the same rule the published-tag +# check above follows. +# +# ## By version name, not by digest +# +# The v2 API deletes a manifest by digest, and two tags can point at the same +# digest — `latest` here does. A digest delete would take both and the second +# one would be silent. Forgejo's package API deletes the version that was named, +# which is the only thing this ever wants to do. +# --------------------------------------------------------------------------- + +prune +prune_local + +say "note: this published an image. It did not deploy it — whatever runs the" +say " container still pins its own tag and that edit is a separate step." diff --git a/docs/architecture/scripts/scaffold.sh b/docs/architecture/scripts/scaffold.sh new file mode 100755 index 0000000..9335dd1 --- /dev/null +++ b/docs/architecture/scripts/scaffold.sh @@ -0,0 +1,591 @@ +#!/usr/bin/env bash +# +# Create the documentation tree a project is expected to have, copied from the +# template's own copy of it, without ever overwriting a document that is +# already there. +# +# bash scripts/scaffold.sh # into the current directory +# bash scripts/scaffold.sh --dry-run # list everything, write nothing +# bash scripts/scaffold.sh --into ../new-app +# bash scripts/scaffold.sh --force # overwrite; needs a clean git tree +# SCAFFOLD_TEMPLATE_ROOT=/path/to/Template bash scripts/scaffold.sh +# +# =========================================================================== +# TEMPLATE COPY — configure this before the first run +# =========================================================================== +# +# Copy to `scripts/scaffold.sh` and set TEMPLATE_ROOT in the CONFIGURATION +# block below, or pass it as SCAFFOLD_TEMPLATE_ROOT. It is empty and it stays +# empty: the template sits at a different absolute path on every machine, and +# a plausible-looking default would quietly resolve to whatever checkout +# happens to be there. That is not a "file not found" — it is thirteen +# documents arriving with another project's Owner lines, another project's +# review dates and another project's branding, each one reading as this +# project's own the moment it lands. +# +# Assumes: bash and coreutils. git is used only by --force, and only to read. +# +# ## Why this exists +# +# The convention this tree encodes is cheap to state and, on the evidence, +# almost never adopted whole. Eight repositories were surveyed before the +# template was last rewritten: exactly one kept the ledger the template asked +# for, three had invented their own dialects, and one had pushed nothing but a +# README. Nobody refused the convention. They started projects, and a +# convention that has to be re-typed from memory at the start of each one is a +# convention that decays into four different ones. +# +# So the first act of a new project is a command, not a reading exercise. What +# is left afterwards — the placeholders, the tracker, the branding — is work +# that genuinely needs a person, and it is visible precisely because the +# skeleton around it is already correct. +# +# ## Never overwriting is the whole safety property +# +# Every other rule here follows from one: a scaffold that clobbers a written +# document destroys work that nothing would notice was gone. There is no test +# for "the architecture notes used to say more than this", no build failure, no +# diff to read if the file was never committed. It is silent at the moment it +# happens and silent forever afterwards, and the natural time to re-run a +# scaffold — partway into a project, to pick up a file that was missed — is +# exactly the time the tree is full of real writing. +# +# So an existing file is reported as "kept" and left untouched, always. --force +# exists for the one honest case (a tree scaffolded from a stale template, not +# yet written into) and it refuses twice over: once for the tree, which must be +# clean, and again for each file, which git must actually track. A clean tree +# alone is not enough — it says nothing about ignored files — so the worst +# --force can do is something `git checkout` undoes. +# +# ## What it deliberately does not do +# +# It does not touch git. No init, no add, no commit — creating files is enough, +# and committing them is a judgement about what belongs in the history of this +# project, made by whoever will answer for that history. +# +# It does not fill anything in. The angle-bracket placeholders are left +# standing because a document whose header says it was reviewed on a date +# nobody reviewed it is worse than one that visibly has not been filled in: the +# first is trusted at the exact moment it should not be. An unfilled header +# keeps asking; an invented one stops. +# +# It does not copy branding. See the docs/data/img note near the bottom. +# +# It does not copy the scripts that live in this folder. They are in the +# template's docs tree for the template's own reasons, and each one needs +# configuring before its first run — an unconfigured release.sh landing in +# every new repository is a loaded gun, not a head start. Copy those one at a +# time, having read them. +# +# It does not create the tracker labels, the first milestone or the project's +# README, and it does not check conformance. It reports what it did. Whether +# the project passes is a different question, asked by a different thing. + +set -uo pipefail + +# --------------------------------------------------------------------------- +# CONFIGURATION — set this one, then delete this banner. +# +# Empty on purpose. See the note at the top: a default here does not fail, it +# copies the wrong project's documents and says "created" thirteen times. +# --------------------------------------------------------------------------- + +# Absolute path to the template checkout — the directory holding `docs/`. +TEMPLATE_ROOT="${SCAFFOLD_TEMPLATE_ROOT:-}" + +# --------------------------------------------------------------------------- +# The tree, listed explicitly. +# +# A recursive copy would be shorter and would drag in whatever else is sitting +# in the template's docs tree today, including this script. An explicit list is +# also the only thing that can tell the difference between "the template does +# not have that file" and "the copy quietly produced twelve of thirteen" — the +# second looks identical to success from the outside. +# --------------------------------------------------------------------------- +DOCS=( + docs/DOC_TRUST_MAP.md + docs/WORK_CYCLE.md + docs/TOOLS.md + docs/planning/PROJECT_PLAN.md + docs/qa/ClaudeQAPlan.md + docs/qa/ClaudeQACoverage.md + docs/qa/ClaudeReport.md + docs/history/HISTORY.md + docs/history/DEVELOPMENT_LOG.md + docs/history/BATCH_LEDGER.md + docs/security/SECURITY.md + docs/security/SECURITY_CHECKLIST.md + docs/architecture/README.md + docs/design/README.md + docs/data/README.md +) + +# Directories made empty, holding nothing this script is willing to invent. +DIRS=(docs/data/img) + +# The three names the conformance check looks for, by name, in docs/data/img. +BRANDING=(icon.webp logo.webp banner.webp) + +# The placeholders the template leaves for a person to replace. Used only to +# count what is still outstanding at the end — if the template's placeholder +# style changes this finds nothing, which is why finding nothing prints nothing +# rather than an all-clear. +PLACEHOLDER_RE='<(Project|YYYY-MM-DD|who maintains this)>' + +say() { printf '\033[1mscaffold:\033[0m %s\n' "$*" >&2; } +die() { printf '\033[1mscaffold:\033[0m %s\n' "$*" >&2; exit 1; } + +usage() { + cat >&2 <<'EOF' +scaffold: create the documentation tree a project is expected to have. + + bash scripts/scaffold.sh into the current directory + bash scripts/scaffold.sh --into into another project root + bash scripts/scaffold.sh --dry-run list what would happen, write nothing + bash scripts/scaffold.sh --force overwrite; refuses on a dirty git tree + + SCAFFOLD_TEMPLATE_ROOT path to the template checkout (required, no default) + SCAFFOLD_INTO same as --into + +An existing file is never overwritten without --force. Exits non-zero if any +path was skipped, including one the template itself is missing. +EOF +} + +DRY_RUN="" +FORCE="" +TARGET="${SCAFFOLD_INTO:-}" + +# Set only when --force resolves a repository. Declared here so the copy loop +# can test it under `set -u` whether or not --force was passed. +force_top="" + +# Whether the operator named the target, as opposed to inheriting the current +# directory. The guard below is only for the inherited case. +TARGET_NAMED="" +[ -n "$TARGET" ] && TARGET_NAMED="yes" + +while [ "$#" -gt 0 ]; do + case "$1" in + --dry-run) DRY_RUN="yes" ;; + --force) FORCE="yes" ;; + --into) + # A bare --into would otherwise swallow the next flag as a path and + # scaffold into a directory called "--dry-run". + [ "$#" -ge 2 ] || die "--into needs a path." + case "$2" in -*) die "--into needs a path, got '$2'." ;; esac + # An empty string is not "here". Without this it becomes "." while still + # counting as named, which silently disables the project-root guard below + # — the one case where the guard is most likely to be right. + [ -n "$2" ] || die "--into was given an empty path. Use --into . to mean the + current directory." + TARGET="$2"; TARGET_NAMED="yes"; shift ;; + --into=*) + TARGET="${1#--into=}"; TARGET_NAMED="yes" + [ -n "$TARGET" ] || die "--into needs a path." ;; + -h|--help) usage; exit 0 ;; + *) die "unknown argument '$1'. Run --help for usage." ;; + esac + shift +done + +# --------------------------------------------------------------------------- +# Resolve both ends before touching either. Everything checkable is checked +# before the first file is written. +# --------------------------------------------------------------------------- + +[ -n "$TEMPLATE_ROOT" ] || die "set TEMPLATE_ROOT (or SCAFFOLD_TEMPLATE_ROOT) — the path to the + template checkout holding docs/. See the CONFIGURATION block." + +# A leading ~ inside a variable is a literal character, not $HOME. The failure +# is otherwise "no such directory" naming a path the operator can see exists, +# which sends them looking in the wrong place. +case "$TEMPLATE_ROOT" in + '~'*) die "TEMPLATE_ROOT is '$TEMPLATE_ROOT' — a leading ~ is not expanded inside a + variable. Write the path out, or use \"\$HOME/...\"." ;; +esac + +[ -d "$TEMPLATE_ROOT" ] || die "TEMPLATE_ROOT '$TEMPLATE_ROOT' is not a directory." + +TEMPLATE_ABS=$(cd "$TEMPLATE_ROOT" 2>/dev/null && pwd -P) \ + || die "cannot enter TEMPLATE_ROOT '$TEMPLATE_ROOT' — check permissions." + +[ -d "$TEMPLATE_ABS/docs" ] || die "'$TEMPLATE_ABS' has no docs/ directory, so it is not the template + root. TEMPLATE_ROOT is the checkout, not the docs folder inside it." + +# Same trap as TEMPLATE_ROOT above, and worse here: the bare "does not exist" +# message names a path the operator can see does exist. +case "$TARGET" in + '~'*) die "target is '$TARGET' — a leading ~ is not expanded inside a variable or + inside quotes. Write the path out, or use \"\$HOME/...\"." ;; +esac + +[ -n "$TARGET" ] || TARGET="." + +# Refused rather than created. `mkdir -p` on a typo succeeds, and the result is +# a complete, correct documentation tree in a directory that should not exist — +# which reads as success and is found weeks later, if at all. +[ -d "$TARGET" ] || die "target '$TARGET' does not exist. Create it first — scaffolding into a + mistyped path produces a tree that looks entirely correct." + +TARGET_ABS=$(cd "$TARGET" 2>/dev/null && pwd -P) \ + || die "cannot enter target '$TARGET' — check permissions." + +[ "$TARGET_ABS" != "$TEMPLATE_ABS" ] \ + || die "the target is the template root itself. There is nothing to scaffold." + +# --------------------------------------------------------------------------- +# An inherited current directory is the one input nobody typed. +# +# Run from src/ or from docs/ itself — both of which are where you are when you +# think to run this — the default would build a second, wrong-place tree that +# every convention here then fails to find. Only guessed targets are checked; +# --into is an instruction and is obeyed. +# --------------------------------------------------------------------------- +if [ -z "$TARGET_NAMED" ]; then + if command -v git >/dev/null 2>&1; then + guess_top=$(git -C "$TARGET_ABS" rev-parse --show-toplevel 2>/dev/null) + + if [ -n "$guess_top" ]; then + guess_top=$(cd "$guess_top" 2>/dev/null && pwd -P) + + if [ -n "$guess_top" ] && [ "$guess_top" != "$TARGET_ABS" ]; then + die "you are in a subdirectory of '$guess_top'. docs/ belongs at the project + root. Re-run with --into '$guess_top', or --into . to mean here." + fi + fi + else + # Said out loud rather than passed over. Without git this check cannot run + # at all, and a silent skip reads exactly like a check that passed. + say "note: git is not installed, so the \"is this the project root?\" check did" + say " not run. '$TARGET_ABS' is being taken as the root on trust." + fi +fi + +# --------------------------------------------------------------------------- +# --force may only run where the damage is undoable. +# +# Overwriting is recoverable exactly when git already holds the current +# contents. A dirty tree means it does not, and an untracked file is the worst +# case of all — scaffolded an hour ago, written into since, never committed, +# and nothing anywhere remembers what it said. So untracked counts as dirty. +# +# This is the tree-wide half of the check, and on its own it is not enough: +# `git status --porcelain` is silent about ignored files, so the per-file +# `ls-files` test in the copy loop below is what actually closes the gap. +# +# Every way of failing to answer the question is refused, not passed over. +# --------------------------------------------------------------------------- +if [ -n "$FORCE" ]; then + command -v git >/dev/null 2>&1 \ + || die "--force needs git, so an overwrite can be undone with 'git checkout'." + + force_top=$(git -C "$TARGET_ABS" rev-parse --show-toplevel 2>/dev/null) \ + || force_top="" + + [ -n "$force_top" ] || die "--force refused: '$TARGET_ABS' is not inside a git repository, so an + overwrite would be permanent. Commit the tree somewhere first, or drop + --force and let existing files be kept." + + if ! dirty=$(git -C "$force_top" status --porcelain 2>/dev/null); then + die "--force refused: could not read git status in '$force_top'. Not knowing + whether the tree is clean is a reason to stop, not to continue." + fi + + if [ -n "$dirty" ]; then + say "--force refused: '$force_top' has uncommitted or untracked changes:" + printf '%s\n' "$dirty" | head -n 10 >&2 + die "commit or stash them first. --force is only safe when git already holds + what it is about to replace." + fi +fi + +say "template $TEMPLATE_ABS" +say "target $TARGET_ABS" +[ -n "$DRY_RUN" ] && say "--dry-run: nothing will be written." +[ -n "$FORCE" ] && say "--force: existing files WILL be replaced (git tree is clean)." +say "" + +# --------------------------------------------------------------------------- +# The copy. +# --------------------------------------------------------------------------- + +CREATED=0 +KEPT=0 +OVERWRITTEN=0 +MISSING_TEMPLATE=0 +UNREADABLE=0 +CONFLICTED=0 +FAILED=0 +HEADERLESS=0 + +MISSING_LIST=() +PLACEHOLDER_LIST=() + +report() { printf ' %-13s %s%s\n' "$1" "$2" "${3:+ ($3)}" >&2; } + +# Every document opens with an H1 and then a fenced status header. Checked on +# the template's copy rather than the result, because a header missing here +# means the template drifted and every project scaffolded from it starts +# non-conformant — a fact about the template, reported as one. +has_status_header() { + awk 'NR==1 { if ($0 !~ /^# /) { bad=1; exit } } + NR>1 && NR<=4 { if ($0 ~ /^```/) { ok=1; exit } } + NR>4 { exit } + END { exit (ok && !bad) ? 0 : 1 }' "$1" 2>/dev/null +} + +for rel in "${DIRS[@]}"; do + dst="$TARGET_ABS/$rel" + + if [ -d "$dst" ]; then + report "kept" "$rel/" "directory exists" + KEPT=$((KEPT + 1)) + continue + fi + + if [ -e "$dst" ] || [ -L "$dst" ]; then + report "SKIPPED" "$rel/" "exists and is not a directory" + CONFLICTED=$((CONFLICTED + 1)) + continue + fi + + if [ -n "$DRY_RUN" ]; then + report "would create" "$rel/" "empty directory" + CREATED=$((CREATED + 1)) + continue + fi + + if mkdir -p "$dst" 2>/dev/null; then + report "created" "$rel/" "empty directory" + CREATED=$((CREATED + 1)) + else + report "FAILED" "$rel/" "could not create" + FAILED=$((FAILED + 1)) + fi +done + +for rel in "${DOCS[@]}"; do + src="$TEMPLATE_ABS/$rel" + dst="$TARGET_ABS/$rel" + + # Reported, counted and returned in the exit status. A template missing a + # file it is supposed to carry produces a tree that is wrong in a way only + # this run can see — by the next run, "not in the template" and "already + # here" look the same from the target's side. + if [ ! -e "$src" ]; then + report "SKIPPED" "$rel" "missing from the template" + MISSING_TEMPLATE=$((MISSING_TEMPLATE + 1)) + MISSING_LIST+=("$rel") + continue + fi + + # Present but unreadable is a different fact with a different fix — a mode + # bit here, a missing document there. Reporting both as "missing from the + # template" sends someone to write a file that is already sitting in front + # of them. + if [ ! -f "$src" ] || [ ! -r "$src" ]; then + report "SKIPPED" "$rel" "in the template but not a readable regular file" + UNREADABLE=$((UNREADABLE + 1)) + continue + fi + + has_status_header "$src" || HEADERLESS=$((HEADERLESS + 1)) + + # A symlink is followed by cp, so "writing into docs/" could land anywhere on + # the disk, including on top of the file it points at. Never written through, + # in either direction, --force included. + if [ -L "$dst" ]; then + report "SKIPPED" "$rel" "is a symlink — never written through" + CONFLICTED=$((CONFLICTED + 1)) + continue + fi + + if [ -e "$dst" ] && [ ! -f "$dst" ]; then + report "SKIPPED" "$rel" "exists and is not a regular file" + CONFLICTED=$((CONFLICTED + 1)) + continue + fi + + if [ -f "$dst" ]; then + if [ -z "$FORCE" ]; then + report "kept" "$rel" "already here, not touched" + KEPT=$((KEPT + 1)) + continue + fi + + # An identical file is not an overwrite worth announcing, and calling it + # one inflates the number that is supposed to mean "work was replaced". + if cmp -s "$src" "$dst"; then + report "kept" "$rel" "identical to the template" + KEPT=$((KEPT + 1)) + continue + fi + + # The clean-tree check above is necessary and not sufficient. `git status + # --porcelain` says nothing about ignored files, so a repository with + # docs/qa/ClaudeReport.md in .gitignore reports clean while git holds no + # copy of it at all. Overwriting that is exactly the permanent, silent loss + # the whole script is built to refuse, announced as "(git tree is clean)". + # + # So the question is asked per file, of git, in the only form that answers + # it: does git track this path. Anything else — ignored, untracked, or an + # unreadable index — is refused and counted, never overwritten. + if ! git -C "$force_top" ls-files --error-unmatch -- "$dst" >/dev/null 2>&1; then + report "SKIPPED" "$rel" "--force: git does not track it (ignored?) — overwrite would be permanent" + CONFLICTED=$((CONFLICTED + 1)) + continue + fi + + if [ -n "$DRY_RUN" ]; then + report "would REPLACE" "$rel" "--force, differs from the template" + OVERWRITTEN=$((OVERWRITTEN + 1)) + continue + fi + elif [ -n "$DRY_RUN" ]; then + report "would create" "$rel" + CREATED=$((CREATED + 1)) + continue + fi + + if ! mkdir -p "$(dirname "$dst")" 2>/dev/null; then + report "FAILED" "$rel" "could not create its directory" + FAILED=$((FAILED + 1)) + continue + fi + + existed="" + [ -f "$dst" ] && existed="yes" + + if ! cp -- "$src" "$dst" 2>/dev/null; then + report "FAILED" "$rel" "copy failed" + FAILED=$((FAILED + 1)) + continue + fi + + # Verified rather than assumed. A short write on a full disk leaves a file + # that exists, reports as created, and is a truncated document — which is the + # one outcome here that is worse than not copying at all. + if ! cmp -s "$src" "$dst"; then + report "FAILED" "$rel" "copied file does not match the template — check disk space" + FAILED=$((FAILED + 1)) + continue + fi + + if [ -n "$existed" ]; then + report "REPLACED" "$rel" "--force" + OVERWRITTEN=$((OVERWRITTEN + 1)) + else + report "created" "$rel" + CREATED=$((CREATED + 1)) + fi + + grep -Eq "$PLACEHOLDER_RE" "$dst" 2>/dev/null && PLACEHOLDER_LIST+=("$rel") +done + +SKIPPED=$((MISSING_TEMPLATE + UNREADABLE + CONFLICTED + FAILED)) + +say "" + +if [ -n "$DRY_RUN" ]; then + say "--dry-run: nothing was written. It would have made" + say " ${CREATED} created, ${KEPT} kept, ${OVERWRITTEN} replaced, ${SKIPPED} skipped." +else + say "${CREATED} created, ${KEPT} kept, ${OVERWRITTEN} replaced, ${SKIPPED} skipped." +fi + +say " skipped breaks down as ${MISSING_TEMPLATE} missing from the template," +say " ${UNREADABLE} present in the template but unreadable, ${CONFLICTED} conflicting" +say " with something already at that path, ${FAILED} failed." + +if [ "$MISSING_TEMPLATE" -gt 0 ]; then + say "" + say "WARNING: the template at $TEMPLATE_ABS does not have:" + printf ' %s\n' ${MISSING_LIST[@]+"${MISSING_LIST[@]}"} >&2 + say " This tree is incomplete and nothing downstream will say so." + say " Fix the template, or write those files by hand." +fi + +if [ "$HEADERLESS" -gt 0 ]; then + say "" + say "WARNING: ${HEADERLESS} template document(s) do not open with an H1 followed by" + say " a fenced block within the first four lines — the shape a status" + say " header is written in. Only that shape was checked: the field" + say " names (Status / Owner / Last reviewed / Governs / Review" + say " trigger) were not read, so a passing document is not a" + say " conformant one. Every project scaffolded from this template" + say " inherits whatever is there. Fix it in the template." +fi + +# Counted only among files this run created, and silent when there are none — +# an "all placeholders filled in" line would be a claim about documents this +# run never looked at, and about a placeholder style that may simply have +# changed underneath the pattern above. +if [ "${#PLACEHOLDER_LIST[@]}" -gt 0 ]; then + say "" + say "${#PLACEHOLDER_LIST[@]} file(s) written by this run still carry placeholders. Every status" + say "header needs a real Owner, a real Last reviewed date and a real Review" + say "trigger — that last line is what stops a document going quietly stale." + say "Find them with:" + say " grep -rnE '$PLACEHOLDER_RE' docs/" +fi + +# --------------------------------------------------------------------------- +# Branding: named, measured, and never invented. +# +# The template's own docs/data/img is not a source. When this was written it +# held one zero-byte logo.webp, and copying that in would satisfy a check that +# looks for the name while producing a project card with nothing on it. The +# deeper reason is the one docs/data/README.md gives: a placeholder that looks +# deliberate outlives the issue that would have replaced it, because nobody +# files a ticket against an image that appears finished. +# +# So the directory is created empty and the three names are reported by what is +# actually there — including the difference between absent and present-but- +# empty, which the check itself distinguishes and a file listing does not. +# --------------------------------------------------------------------------- + +img_dir="$TARGET_ABS/docs/data/img" + +say "" +say "docs/data/img/ needs this project's own branding. The conformance check" +say "looks for exactly these three names, and reads webp by magic bytes rather" +say "than by extension — a renamed PNG does not pass:" + +if [ -d "$img_dir" ]; then + for name in "${BRANDING[@]}"; do + if [ -s "$img_dir/$name" ]; then + report "present" "docs/data/img/$name" + elif [ -e "$img_dir/$name" ]; then + report "EMPTY" "docs/data/img/$name" "0 bytes — fails the magic-byte check" + else + report "absent" "docs/data/img/$name" + fi + done +else + # Not "none present". The directory does not exist yet, so nothing about + # these three files has been measured and no count would be honest. + say " not checked — $img_dir does not exist yet." + printf ' %s\n' ${BRANDING[@]+"${BRANDING[@]}"} >&2 +fi + +say "" +say "Nothing here can draw them. When one is absent, file an issue — title it" +say "for the asset, label it P2, and end the body with its Verify: line — rather" +say "than committing a placeholder." + +say "" +say "note: no git command was run. Nothing is staged and nothing is committed;" +say " what belongs in this project's history is your call." + +# Non-zero when the tree that came out is not the tree that was asked for. The +# caller — often another script, or an agent — needs that distinction to survive +# past the last line of output. +if [ "$SKIPPED" -gt 0 ]; then + say "" + die "${SKIPPED} path(s) were skipped, so this project is not fully scaffolded. + Resolve each one listed above and run this again — re-running is safe, + because everything already here is kept." +fi + +exit 0 diff --git a/docs/architecture/scripts/secrets.sh b/docs/architecture/scripts/secrets.sh new file mode 100755 index 0000000..e733c8f --- /dev/null +++ b/docs/architecture/scripts/secrets.sh @@ -0,0 +1,166 @@ +#!/usr/bin/env bash +# +# Credentials, before they are committed. +# +# ## Why this and not a generic scanner +# +# A general-purpose secret scanner knows about AWS keys and GitHub tokens. It +# does not know that *this* deployment issues `pllc_agent_`, or that its +# encrypted envelopes start `v2::`, or which of its environment variables +# hold a password. The project does know, and usually writes it down twice: once +# in whatever redacts its logs, and once in whatever redacts its outbound +# messages. +# +# So this reads the project's own patterns where they exist — point +# `SECRETS_PATTERN_FILE` at the module holding them — and falls back to a +# conservative built-in set. A scanner tuned to the shapes a project actually +# issues catches the leak a generic one misses, and stays quiet the rest of the +# time. +# +# ## What it scans +# +# By default the **staged diff**, which is the only moment a commit can still be +# stopped cheaply. `--tracked` scans every tracked file instead, which is what +# you want once, on adoption, to find what is already in the history's tip. +# +# bash scripts/secrets.sh # staged changes (use in pre-commit) +# bash scripts/secrets.sh --tracked # everything tracked, for an audit +# SECRETS_PATTERN_FILE=src/lib/log.ts bash scripts/secrets.sh +# bash scripts/secrets.sh --allow docs/examples/ +# +# ## What it cannot do +# +# It reads the working tree and the index. **A secret already committed is still +# in the history** after you delete it, and this will not tell you that — the +# fix there is a rotation, not a scan. Rotate first, then clean up. +# +# Exit codes: 0 nothing found. 1 a candidate found. 2 nothing was scanned. +# --------------------------------------------------------------------------- + +set -uo pipefail + +cd "$(git rev-parse --show-toplevel 2>/dev/null)" || { + printf 'secrets: not a git repository.\n' >&2 + exit 2 +} + +say() { printf 'secrets: %s\n' "$*" >&2; } + +MODE="staged" +ALLOW=() + +while [ $# -gt 0 ]; do + case "$1" in + --tracked) MODE="tracked"; shift ;; + --staged) MODE="staged"; shift ;; + --allow) ALLOW+=("${2:-}"); shift 2 ;; + *) say "unknown argument: $1"; exit 2 ;; + esac +done + +# The built-in set. Deliberately shapes that are *structurally* credential-like +# rather than words that merely appear near credentials — `password` in a +# sentence is not a leak, and a scanner that says it is gets muted. +PATTERNS=( + '//[^/@[:space:]:]+:[^/@[:space:]]+@' # user:pass@host in a URL + '[?&](token|key|secret|password|access_token|api_key)=[^&[:space:]"]+' + '\b(Bearer|Basic)[[:space:]]+[A-Za-z0-9._~+/=-]{20,}' # an authorization header + # Anchored to the start of a line or an `export`, because unanchored it + # matched `access_token = $1` in SQL and `apiKey=` in a property list — three + # findings in src/ that were column names, not credentials. + '(^|export )[A-Z][A-Z0-9_]*(SECRET|TOKEN|PASSWORD|API_KEY|PASSWD)[A-Z0-9_]*=[^[:space:]"'"'"']{8,}' + '-----BEGIN [A-Z ]*PRIVATE KEY-----' + '\bghp_[A-Za-z0-9]{20,}' # GitHub + '\bxox[baprs]-[A-Za-z0-9-]{10,}' # Slack + '\bAKIA[0-9A-Z]{16}\b' # AWS access key id +) + +# The project's own shapes, if it has written them down. A `pllc_agent_` +# token is invisible to every generic scanner and obvious to the module that +# redacts it. +if [ -n "${SECRETS_PATTERN_FILE:-}" ] && [ -f "$SECRETS_PATTERN_FILE" ]; then + loaded=0 + + # Anchored on the closing `/flags,` and greedy to it, rather than on "no + # commas". The first version used `[^,]+`, which cannot cross the comma inside + # a bounded quantifier — so `pllc_[a-z]+_[0-9a-f]{8,}` was silently dropped + # along with every other `{n,}` pattern: four of six on the file this was + # written against, while the script printed that it had loaded them. + while IFS= read -r found; do + [ -n "$found" ] || continue + + PATTERNS+=("$found") + loaded=$((loaded + 1)) + done < <( + sed -nE 's/.*\[\/(.+)\/[gimsuy]*,[[:space:]]*".*/\1/p' "$SECRETS_PATTERN_FILE" 2>/dev/null || true + ) + + # The count, never a bare reassurance. "Loaded project patterns" over an empty + # list is the same lie as a green test run that executed nothing. + if [ "$loaded" -gt 0 ]; then + say "loaded $loaded project pattern(s) from $SECRETS_PATTERN_FILE" + else + say "WARNING: $SECRETS_PATTERN_FILE yielded no patterns — scanning with the" + say " built-in set only. Check the file holds regex literals." + fi +fi + +if [ "$MODE" = "staged" ]; then + # Added lines only. A removed line containing a token is somebody deleting + # one, which is the opposite of a leak. + CONTENT="$(git diff --cached --unified=0 --no-color | grep '^+' | grep -v '^+++' || true)" + WHAT="staged changes" +else + CONTENT="" + WHAT="tracked files" + + while IFS= read -r file; do + skip="" + + for allowed in ${ALLOW[@]+"${ALLOW[@]}"}; do + case "$file" in *"$allowed"*) skip="yes" ;; esac + done + + [ -n "$skip" ] && continue + [ -f "$file" ] || continue + + # Text only; a webp full of bytes will match anything. + file "$file" 2>/dev/null | grep -q "text" || continue + + CONTENT+="$(sed "s|^|${file}: |" "$file")"$'\n' + done < <(git ls-files) +fi + +if [ -z "$CONTENT" ]; then + say "nothing to scan in $WHAT." + exit 2 +fi + +found=0 + +for pattern in "${PATTERNS[@]}"; do + while IFS= read -r hit; do + [ -n "$hit" ] || continue + + skip="" + for allowed in ${ALLOW[@]+"${ALLOW[@]}"}; do + case "$hit" in *"$allowed"*) skip="yes" ;; esac + done + [ -n "$skip" ] && continue + + # The line is printed truncated and the match is never echoed in full — a + # scanner that prints the secret it found has published it to the terminal + # scrollback, the CI log, and wherever that log is shipped. + printf ' %.120s…\n' "$hit" + found=$((found + 1)) + done < <(printf '%s\n' "$CONTENT" | grep -nEI "$pattern" 2>/dev/null | head -20) +done + +if [ "$found" -gt 0 ]; then + say "$found candidate credential(s) in $WHAT." + say "If one is real: rotate it first. Deleting the line does not remove it" + say "from a commit that already exists, and the scan cannot see history." + exit 1 +fi + +say "no credential shapes in $WHAT." diff --git a/docs/architecture/scripts/status.sh b/docs/architecture/scripts/status.sh new file mode 100755 index 0000000..600f71d --- /dev/null +++ b/docs/architecture/scripts/status.sh @@ -0,0 +1,678 @@ +#!/usr/bin/env bash +# +# What is actually running right now on the host this project deploys to, and +# its logs. Read-only: it inspects and it tails, and it does nothing else. +# +# bash scripts/status.sh # up? which version? how long? +# bash scripts/status.sh --logs # last 100 lines +# bash scripts/status.sh --logs 500 +# bash scripts/status.sh --logs all +# bash scripts/status.sh --follow # stream until Ctrl-C +# bash scripts/status.sh --follow 20 # stream, after 20 lines of context +# bash scripts/status.sh --deployed-version # just the version, for scripts +# bash scripts/status.sh --dry-run # print the commands, contact nothing +# +# STATUS_CONTAINER=other-app bash scripts/status.sh # one-off, another target +# +# Exit codes, because the whole point is that a caller can act on the answer: +# +# 0 running — healthy, still starting, or no healthcheck defined +# 1 configuration or usage error; nothing was contacted +# 2 COULD NOT DETERMINE — host unreachable, daemon silent, or an error this +# does not recognise. This is not an outage report. Nothing is known. +# 3 the container does not exist on that host +# 4 the container exists but is not running +# 5 running, and its own healthcheck says unhealthy +# +# stdout carries only what a caller should parse — the log lines, or the one +# version string. The human report goes to stderr, so `status.sh +# --deployed-version` can be read straight into a variable without a filter. +# +# =========================================================================== +# TEMPLATE COPY — configure this before the first run +# =========================================================================== +# +# Copy to `scripts/status.sh` and set the two values in the CONFIGURATION block +# below. The script refuses to run until they are set: it has no defaults, +# deliberately, for the reason `release.sh` gives at length — a copied script +# carrying another project's host and container name answers confidently about +# the wrong machine, and an answer is exactly what you came here for. A status +# tool that lies is worse than no status tool, because you act on it. +# +# Assumes: bash, coreutils, git-less (it reads nothing from the repository), and +# docker on the target. `ssh` only when the target is not this machine. +# +# ## Why this exists +# +# Without it, "is it up?" is answered by typing docker incantations from memory, +# which in practice means `docker ps | grep`. That shows a name and a status and +# none of the four things you actually wanted: which version is deployed, how +# long it has been up, how many times it has restarted, and what its healthcheck +# thinks. So the question gets half-answered, and the half that is missing — +# restart count — is the one that distinguishes "running" from "crash-looping +# and up for nine seconds". +# +# The other half is worse. When there is no command that answers "what version +# is deployed", every script that needs the answer grows its own copy of the +# lookup. `release.sh` has one: a single `ssh … docker inspect --format` whose +# empty return value is the ONLY thing standing between the prune and deleting +# the published image production is currently running. That fact is load-bearing +# enough to deserve a name, a documented exit code, and one place to fix — not a +# line pasted into whichever script needed it that week, each copy free to +# disagree about what an empty answer means. +# +# Hence `--deployed-version`: one fact, on stdout, non-zero when it could not be +# read. A caller that treats non-zero as "stop" cannot make the prune's mistake. +# +# ## Unreachable is not stopped +# +# These are different facts and only one of them is an outage you caused: +# +# - the container is not running → something happened to the deployment +# - I could not reach the host → something happened to the network, or to +# ssh, or to the daemon, and the container +# is very probably still serving traffic +# +# Rendering the second as the first is how a 2am investigation starts by +# restarting a healthy service. So a failed connection exits 2 and says so in +# those words; only an inspect that succeeded and reported a non-running state +# exits 4. Nothing here ever prints "stopped", "0" or "ok" for something it did +# not measure — an unknown is printed as an unknown, every time. +# +# ssh runs with BatchMode=yes and a connect timeout, so a host that would prompt +# for a password or a host key is read as unreachable in seconds instead of +# hanging forever on a prompt nobody is watching. +# +# ## There is no --host flag +# +# Host and container come from the CONFIGURATION block, or from the environment +# for a one-off. A flag would make it one keystroke to read host A and file the +# answer under host B, and the environment form is self-documenting in shell +# history — which is where you will be reading it back from. +# +# ## What it deliberately does not do +# +# It never starts, stops, restarts, recreates, kills or removes anything, and it +# never pulls or prunes an image. Every docker verb it runs is `inspect`, +# `version` or `logs`. This is not an oversight to be fixed later: a status tool +# is the one command people run half-awake and without reading, so it must be +# safe to run half-awake and without reading. Fixing what it reports is a +# separate, deliberate act with a separate command. +# +# It does not compare the deployed version against this repository, and does not +# read package.json or any other file. Whether the deployed version is the RIGHT +# version is a judgement, and a judgement needs a human or a release script that +# owns the decision; this only reports what is there. +# +# It reports one configured container, not every container on the host. A tool +# that enumerates is a tool that finds another project's container and reports it +# as yours. + +set -uo pipefail + +# --------------------------------------------------------------------------- +# CONFIGURATION — set these two, then delete this banner. +# +# Both are empty on purpose. See the note at the top. +# --------------------------------------------------------------------------- + +# An ssh destination — a Host in ~/.ssh/config, an IP, or user@host — or the +# literal word `local` to ask this machine's own docker daemon. +# +# `local` is a word you have to type rather than what an empty value means, +# because "you did not configure a host" and "the host is this machine" must not +# look the same. Defaulting to local would make an unconfigured copy of this +# script quietly report on your laptop and call it production. +HOST="${STATUS_HOST:-}" + +# The container name (or id) to report on. Exactly one. +CONTAINER="${STATUS_CONTAINER:-}" + +# Seconds to wait for the ssh connection. Short on purpose: this command exists +# to be run when something might be wrong, and a status tool that hangs is +# indistinguishable from the outage it was meant to describe. +SSH_TIMEOUT="${STATUS_SSH_TIMEOUT:-10}" + +# Lines tailed by `--logs` with no number given. +TAIL_DEFAULT="${STATUS_TAIL:-100}" + +# The image label carrying the deployed version. This one has a default because +# it is an open standard rather than a fact about your project; override it only +# if your build stamps a different label. +VERSION_LABEL="${STATUS_VERSION_LABEL:-org.opencontainers.image.version}" + +# --------------------------------------------------------------------------- + +E_CONFIG=1 +E_UNKNOWN=2 +E_MISSING=3 +E_STOPPED=4 +E_UNHEALTHY=5 + +say() { printf '\033[1mstatus:\033[0m %s\n' "$*" >&2; } +die() { printf '\033[1mstatus:\033[0m %s\n' "$*" >&2; exit "$E_CONFIG"; } + +# die() with a chosen code. Every caller of this is a distinct fact a script +# downstream is expected to branch on, so the code is passed explicitly at the +# point the fact is established rather than mapped from a message afterwards. +quit() { local code="$1"; shift; printf '\033[1mstatus:\033[0m %s\n' "$*" >&2; exit "$code"; } + +field() { printf '\033[1mstatus:\033[0m %-9s %s\n' "$1" "$2" >&2; } + +usage() { + say "usage: bash scripts/status.sh [--logs [N|all]] [--follow [N]] [--deployed-version] [--dry-run]" + say " set STATUS_HOST and STATUS_CONTAINER, or edit the CONFIGURATION block." +} + +# --------------------------------------------------------------------------- +# Arguments, before the configuration check, so that `--help` answers on a +# fresh unconfigured copy — which is the one moment somebody needs it. +# --------------------------------------------------------------------------- + +MODE="report" +TAIL="" +DRY_RUN="" + +# `("$@")` with no arguments is an unbound expansion under `set -u` on bash +# before 4.4, which is still what ships on macOS. Guarded rather than assumed. +if [ "$#" -gt 0 ]; then ARGS=("$@"); else ARGS=(); fi + +# Refused rather than resolved last-flag-wins. `--deployed-version --logs` from a +# script would otherwise return a log stream on stdout to a caller that is about +# to compare it to a version string. +set_mode() { + [ "$MODE" = "report" ] || [ "$MODE" = "$1" ] \ + || die "--logs, --follow and --deployed-version ask different questions; pick one." + MODE="$1" +} + +i=0 +while [ "$i" -lt "${#ARGS[@]}" ]; do + arg="${ARGS[$i]}" + + case "$arg" in + --logs|-l|--follow|-f) + case "$arg" in + --logs|-l) set_mode "logs" ;; + *) set_mode "follow" ;; + esac + + # An optional operand rather than a required one, so `--logs` alone is a + # whole command. Only consumed when it looks like a count — otherwise + # `--logs --dry-run` would swallow the flag and silently mean something + # else. + if [ $((i + 1)) -lt "${#ARGS[@]}" ]; then + case "${ARGS[$((i + 1))]}" in + all|[0-9]*) TAIL="${ARGS[$((i + 1))]}"; i=$((i + 1)) ;; + esac + fi + ;; + --logs=*) set_mode "logs"; TAIL="${arg#*=}" ;; + --follow=*) set_mode "follow"; TAIL="${arg#*=}" ;; + --deployed-version) set_mode "version" ;; + --dry-run) DRY_RUN="yes" ;; + -h|--help) usage; exit 0 ;; + *) usage; die "unknown argument '$arg'." ;; + esac + + i=$((i + 1)) +done + +[ -n "$TAIL" ] || TAIL="$TAIL_DEFAULT" + +case "$TAIL" in + all) : ;; + ''|*[!0-9]*) die "line count must be a whole number or 'all', got '$TAIL'." ;; +esac + +case "$SSH_TIMEOUT" in + ''|*[!0-9]*) die "STATUS_SSH_TIMEOUT must be a whole number of seconds, got '$SSH_TIMEOUT'." ;; +esac + +# --------------------------------------------------------------------------- +# Refuse to run half-configured, before anything is contacted, and name the +# missing value one at a time so the message says which one. +# --------------------------------------------------------------------------- + +[ -n "$HOST" ] || die "set HOST (or STATUS_HOST) — the ssh host running the container, or the word 'local'. See the CONFIGURATION block." +[ -n "$CONTAINER" ] || die "set CONTAINER (or STATUS_CONTAINER) — the container name to report on." + +# ssh takes its destination positionally, so a host beginning with a dash is +# parsed as an option and the failure that follows describes something else +# entirely. Refused here, where the message can still name the real cause. +case "$HOST" in + -*) die "HOST ('$HOST') begins with '-'; ssh would read it as an option. Use user@host, or a Host from ~/.ssh/config." ;; +esac + +case "$CONTAINER" in + -*) die "CONTAINER ('$CONTAINER') begins with '-'; docker would read it as a flag." ;; +esac + +# The label name is interpolated into a Go template as a quoted string below. A +# quote or backslash in it would end that string early, and the template error +# that followed would be reported as "could not inspect" — an unknown, blamed on +# the host, caused here. +case "$VERSION_LABEL" in + ''|*[\"\\]*) die "STATUS_VERSION_LABEL must be a plain label name without quotes or backslashes, got '$VERSION_LABEL'." ;; +esac + +# --------------------------------------------------------------------------- +# Running docker here or over there. +# +# ssh joins its command arguments with spaces and hands the result to a remote +# shell, which splits it again. Anything containing a space — every Go template +# below does — arrives as several arguments unless it is quoted for that second +# parse. Quoting is done here, once, rather than by hand at each call site. +# --------------------------------------------------------------------------- + +shquote() { + local out="" a q="'" esc="'\\''" + + for a in "$@"; do + a="${a//$q/$esc}" + out="${out}${out:+ }'${a}'" + done + + printf '%s' "$out" +} + +# ServerAlive is here for --follow: a connection that dies mid-stream otherwise +# leaves the terminal sitting quietly forever, which looks exactly like a service +# that has stopped logging. Absence of output must not be able to mean two +# things, so the connection is made to fail loudly instead. +ssh_opts() { + printf '%s' "-o BatchMode=yes -o ConnectTimeout=${SSH_TIMEOUT} -o ServerAliveInterval=15 -o ServerAliveCountMax=3" +} + +SSH_WANT_TTY="" + +dock() { + if [ "$HOST" = "local" ]; then + docker "$@" + return + fi + + # A tty makes the remote `docker logs -f` receive a hangup when you press + # Ctrl-C. Without one it keeps running on the far side, holding the daemon's + # log stream open, and enough abandoned follows are a real resource leak on a + # box you are already worried about. Requested only when we have a tty to + # give, because ssh -t without one is a warning and no tty anyway. + if [ -n "$SSH_WANT_TTY" ] && [ -t 0 ]; then + # shellcheck disable=SC2046 # deliberate word-splitting of the option list + ssh -t $(ssh_opts) "$HOST" "docker $(shquote "$@")" + else + # shellcheck disable=SC2046 + ssh $(ssh_opts) "$HOST" "docker $(shquote "$@")" + fi +} + +# The remote form is printed on two lines — the connection, then the command +# string the far shell receives — because the single-line form is the same text +# quoted twice and the host and container names disappear into the punctuation. +# Those two names are the entire reason to run --dry-run. +dock_show() { + if [ "$HOST" = "local" ]; then + printf ' docker %s\n' "$(shquote "$@")" >&2 + else + printf ' ssh %s %s\n' "$(ssh_opts)" "$HOST" >&2 + printf ' docker %s\n' "$(shquote "$@")" >&2 + fi +} + +# --------------------------------------------------------------------------- +# Timestamps. +# +# Docker reports RFC3339 with nanoseconds; `date -d` is GNU and `date -j -f` is +# BSD, so both are tried and NEITHER working is reported as not knowing the +# uptime rather than as an uptime of zero. +# --------------------------------------------------------------------------- + +epoch_of() { + local t="${1%Z}" + t="${t%%.*}" + + local e + if e=$(date -u -d "${t}Z" +%s 2>/dev/null) && [ -n "$e" ]; then + printf '%s' "$e" + return 0 + fi + + if e=$(date -u -j -f '%Y-%m-%dT%H:%M:%S' "$t" +%s 2>/dev/null) && [ -n "$e" ]; then + printf '%s' "$e" + return 0 + fi + + return 1 +} + +human_duration() { + local s="$1" d h m + + d=$((s / 86400)); s=$((s % 86400)) + h=$((s / 3600)); s=$((s % 3600)) + m=$((s / 60)); s=$((s % 60)) + + if [ "$d" -gt 0 ]; then printf '%dd %dh %dm' "$d" "$h" "$m" + elif [ "$h" -gt 0 ]; then printf '%dh %dm' "$h" "$m" + elif [ "$m" -gt 0 ]; then printf '%dm %ds' "$m" "$s" + else printf '%ds' "$s" + fi +} + +# Elapsed time since an RFC3339 stamp, or nothing at all. +# +# The zero timestamp means the event never happened — a container created but +# never started carries 0001-01-01 in StartedAt, and subtracting it yields two +# millennia of uptime presented with a straight face. +# +# A negative result is clock skew between this machine and that one, not time +# travel, and is likewise reported as not knowing rather than as a number. +elapsed_since() { + local stamp="$1" at now diff + + case "$stamp" in + ''|0001-01-01*) return 1 ;; + esac + + at=$(epoch_of "$stamp") || return 1 + now=$(date -u +%s) || return 1 + + diff=$((now - at)) + [ "$diff" -ge 0 ] || return 1 + + human_duration "$diff" +} + +# --------------------------------------------------------------------------- +# One inspect, one round trip. +# +# Fields are read together rather than one call each: two calls can straddle a +# restart and produce a report that never described a single moment — an uptime +# from before the restart beside a restart count from after it. +# +# The free-text fields are LAST. `read` gives the final variable everything that +# remains, delimiters included, so a '|' inside an image name or a version label +# lands harmlessly in the last field instead of shifting every field after it. +# --------------------------------------------------------------------------- + +FMT='{{.State.Status}}|{{.State.Running}}|{{.RestartCount}}|{{.State.ExitCode}}|{{if .State.Health}}{{.State.Health.Status}}{{else}}{{end}}|{{.State.StartedAt}}|{{.State.FinishedAt}}|{{.Image}}|{{.Config.Image}}|{{index .Config.Labels "'"$VERSION_LABEL"'"}}' + +C_STATUS=""; C_RUNNING=""; C_RESTARTS=""; C_EXIT=""; C_HEALTH="" +C_STARTED=""; C_FINISHED=""; C_DIGEST=""; C_IMAGE=""; C_VERSION="" + +# Work out WHY the inspect failed, and exit with the code for that specific +# fact. Reached only after a failure, so the extra round trips cost nothing in +# the normal case and buy the one distinction this script exists to preserve. +diagnose_and_exit() { + local err="$1" + + if [ "$HOST" != "local" ]; then + # shellcheck disable=SC2046 + if ! ssh $(ssh_opts) "$HOST" true >/dev/null 2>&1; then + say "cannot reach ${HOST} over ssh (BatchMode, ${SSH_TIMEOUT}s timeout)." + say " This is NOT a report that the container is down — nothing was" + say " determined about it. Check the network, the ssh key, and" + say " whether the host would have prompted for a password." + exit "$E_UNKNOWN" + fi + fi + + if ! dock version --format '{{.Server.Version}}' >/dev/null 2>&1; then + if [ "$HOST" = "local" ]; then + say "this machine's docker daemon did not answer." + else + say "reached ${HOST} over ssh, but its docker daemon did not answer." + fi + + say " The container's state is unknown — the daemon being down does" + say " not tell you whether the container was running before it went." + exit "$E_UNKNOWN" + fi + + # Matched case-insensitively against a lowercased copy, because the wording + # is not stable across docker versions: "Error: No such object: x" on one + # daemon here, "error: no such object: x" on another, and "No such container" + # from older ones. A case-sensitive match reports a container that is simply + # absent as an unknown — which is the safe direction to be wrong in, and + # still the wrong answer. + local lower + lower=$(printf '%s' "$err" | tr '[:upper:]' '[:lower:]') + + case "$lower" in + *"no such object"*|*"no such container"*) + quit "$E_MISSING" "there is no container named '${CONTAINER}' on ${HOST}." ;; + esac + + # An error nobody anticipated is printed verbatim and treated as an unknown. + # Guessing at an unfamiliar message is how "I could not check" becomes "there + # is nothing there". + say "could not inspect '${CONTAINER}' on ${HOST}, and the error is not one" + say " this script recognises. Nothing is known about the container:" + printf ' %s\n' "${err:-(no error output)}" >&2 + exit "$E_UNKNOWN" +} + +# The one file this script creates, and the one thing it ever deletes. +# +# The path is read when the trap FIRES rather than pasted into the trap's source +# text. `trap "rm -f '$f'" EXIT` looks equivalent and is not: it builds a shell +# command out of mktemp's output, which is built out of $TMPDIR, so a directory +# whose name contains a single quote closes that string early and everything +# after it runs as shell. `TMPDIR="/tmp/x'; rm -rf ~; echo '"` is a working +# exploit against the double-quoted form. This is the only destructive verb in +# the script and it must not be constructible from the environment. +# +# The -n guard means rm is never handed an empty path, and `--` means a path +# beginning with a dash is a path rather than a bundle of options. +ERRFILE="" + +# shellcheck disable=SC2329 # invoked by the EXIT trap below, not by name +cleanup() { + [ -n "$ERRFILE" ] || return 0 + rm -f -- "$ERRFILE" + return 0 +} + +trap cleanup EXIT + +inspect_container() { + local out rc + + ERRFILE=$(mktemp) || die "cannot create a temporary file" + + out=$(dock inspect --format "$FMT" "$CONTAINER" 2>"$ERRFILE") + rc=$? + + if [ "$rc" -ne 0 ] || [ -z "$out" ]; then + diagnose_and_exit "$(tr -d '\r' < "$ERRFILE" | head -n 3)" + fi + + IFS='|' read -r C_STATUS C_RUNNING C_RESTARTS C_EXIT C_HEALTH \ + C_STARTED C_FINISHED C_DIGEST C_IMAGE C_VERSION \ + </dev/null 2>&1 \ + || quit "$E_UNKNOWN" "docker is not on PATH, so nothing could be determined. Install it, or set HOST to the machine that runs the container." +else + command -v ssh >/dev/null 2>&1 \ + || quit "$E_UNKNOWN" "ssh is not on PATH, so ${HOST} could not be contacted." +fi + +inspect_container + +# --------------------------------------------------------------------------- +# --deployed-version: one fact on stdout, non-zero when it is not known. +# +# The contract callers depend on: a zero exit means the string on stdout was +# read from the running container's image. Anything else means stop — never +# "assume none", which is the assumption that lets a prune delete production's +# image. +# --------------------------------------------------------------------------- + +if [ "$MODE" = "version" ]; then + [ -n "$C_VERSION" ] \ + || quit "$E_UNKNOWN" "'${CONTAINER}' on ${HOST} is ${C_STATUS}, but its image carries no ${VERSION_LABEL} label — the deployed version is unknown." + + [ "$C_RUNNING" = "true" ] \ + || quit "$E_STOPPED" "'${CONTAINER}' on ${HOST} is ${C_STATUS}, not running. Its image says ${C_VERSION}, but nothing is serving from it." + + printf '%s\n' "$C_VERSION" + exit 0 +fi + +# --------------------------------------------------------------------------- +# --logs / --follow. +# +# The inspect above already ran, which is what makes an empty tail meaningful: +# the container is known to exist, so no output means it has logged nothing. +# Tailing blind cannot tell that apart from a container that is not there. +# --------------------------------------------------------------------------- + +if [ "$MODE" = "logs" ] || [ "$MODE" = "follow" ]; then + field container "${CONTAINER} on ${HOST}" + field state "${C_STATUS}${C_VERSION:+ (${C_VERSION})}" + + # Said before the first line of output, because a wall of old log lines from a + # container that died an hour ago is indistinguishable from a live one. + if [ "$C_RUNNING" != "true" ]; then + say "note: the container is ${C_STATUS}. These are the logs it left behind," + + if [ "$MODE" = "follow" ]; then + say " not a live stream — nothing is writing, so this will return at once." + else + say " not a live stream." + fi + fi + + if [ "$MODE" = "follow" ]; then + SSH_WANT_TTY="yes" + say "following ${CONTAINER} — Ctrl-C to stop." + dock logs --tail "$TAIL" --follow "$CONTAINER" + else + dock logs --tail "$TAIL" "$CONTAINER" + fi + + rc=$? + + # A tail that failed after a successful inspect is a new fact, not a quiet + # zero: the container exists, so an empty screen would otherwise read as a + # silent service. + [ "$rc" -eq 0 ] || quit "$E_UNKNOWN" "the log stream ended with status ${rc}; some or all of the output may be missing." + + # The logs printed either way; only the code differs. Exiting 0 here would + # report the documented meaning of 0 — "running" — for a container that is + # not, which is the same mistake as printing "ok" for something unmeasured. + # The state is the fact the exit code carries, in every mode. + [ "$C_RUNNING" = "true" ] \ + || quit "$E_STOPPED" "'${CONTAINER}' exists on ${HOST} but is ${C_STATUS}; the logs above are what it left behind, not a live service." + + exit 0 +fi + +# --------------------------------------------------------------------------- +# The report. +# --------------------------------------------------------------------------- + +field container "${CONTAINER} on ${HOST}" + +if [ "$C_RUNNING" = "true" ]; then + field state "running" +else + field state "${C_STATUS:-unknown} (exit code ${C_EXIT:-unknown})" +fi + +field image "${C_IMAGE:-(unknown)}" + +# The digest is printed beside the tag because the tag is a claim and the digest +# is the fact. A tag can be moved after a container was created, at which point +# the container's image reference names something it was never built from. +if [ -n "$C_DIGEST" ]; then + case "$C_DIGEST" in + sha256:*) + short="${C_DIGEST#sha256:}" + field digest "sha256:${short:0:12}" + ;; + *) + # Whatever this is, it is not a sha256 digest, so it is not labelled as + # one. The stripping form prints "sha256:" in front of the first twelve + # characters of anything at all — podman and older daemons report a bare + # image id here, and it would have been rendered as a digest it is not. + field digest "$C_DIGEST" + ;; + esac +fi + +if [ -n "$C_VERSION" ]; then + field version "$C_VERSION" +else + field version "(not set — this image carries no ${VERSION_LABEL} label)" +fi + +if [ "$C_RUNNING" = "true" ]; then + if up=$(elapsed_since "$C_STARTED"); then + field uptime "${up} (started ${C_STARTED})" + else + # Reported as unknown, with the raw stamp, rather than as a plausible + # number computed from a timestamp that could not be read. + field uptime "unknown — could not read the start time '${C_STARTED}'" + fi +else + if down=$(elapsed_since "$C_FINISHED"); then + field stopped "${down} ago (${C_FINISHED})" + else + field stopped "at '${C_FINISHED:-unknown}'" + fi +fi + +field restarts "${C_RESTARTS:-unknown}" + +# The count alone is not the signal; the count beside a short uptime is. Said +# out loud because a crash loop reads as "running" in every tool that shows only +# a state, and that is the case this command was written to catch. +if [ -n "$C_RESTARTS" ] && [ "$C_RESTARTS" != "0" ] && [ "$C_RUNNING" = "true" ]; then + say " ^ restarted ${C_RESTARTS} time(s). A climbing count with a short" + say " uptime is a crash loop, not a healthy service." +fi + +if [ -n "$C_HEALTH" ]; then + field health "$C_HEALTH" +else + # Not "healthy", and not "ok". No healthcheck is defined, so nothing about + # this container's health has been measured by anyone. + field health "not configured — this image defines no HEALTHCHECK, so nothing is being checked" +fi + +if [ "$C_RUNNING" != "true" ]; then + quit "$E_STOPPED" "'${CONTAINER}' exists on ${HOST} but is ${C_STATUS}." +fi + +if [ "$C_HEALTH" = "unhealthy" ]; then + quit "$E_UNHEALTHY" "'${CONTAINER}' is running, but its healthcheck reports unhealthy. Try --logs." +fi + +exit 0 diff --git a/docs/architecture/scripts/verify.sh b/docs/architecture/scripts/verify.sh new file mode 100755 index 0000000..2d1afdf --- /dev/null +++ b/docs/architecture/scripts/verify.sh @@ -0,0 +1,694 @@ +#!/usr/bin/env bash +# +# Run every check this project has, in one command, and print one table saying +# which passed, which failed and which did not run at all. +# +# bash scripts/verify.sh # everything +# bash scripts/verify.sh --quick # skip the test step, name it as skipped +# bash scripts/verify.sh --list # show the plan, run nothing +# bash scripts/verify.sh --only npm # only steps whose id contains 'npm' +# VERIFY_GUARD_DIR=scripts/verify.d bash scripts/verify.sh +# +# Exit codes: 0 everything that ran passed. 1 something failed, or the run +# could not start at all — a usage error, a bad VERIFY_GUARD_DIR, not a git +# repository. 2 nothing was verified: either no checks were detected, or every +# detected step skipped. Two is not a pass and CI must not treat it as one. +# +# One and two are both non-zero on purpose, so every refusal fails closed. If +# you need to tell "a check failed" apart from "it never got to run", read the +# message — a failed check always prints a table first. +# +# =========================================================================== +# TEMPLATE COPY — configure this before the first run +# =========================================================================== +# +# Copy to `scripts/verify.sh`, `chmod +x` it, and wire it in: for a Node +# project add `"verify": "bash scripts/verify.sh"` to package.json; for anything +# else call it from `pre-commit`. Then read the CONFIGURATION block below — +# two environment variables (VERIFY_GUARD_DIR, VERIFY_SLOW_STEP) and six lists +# of candidate script and target names. Every one is optional except the one +# you need, which is GUARD_DIR. +# +# Unlike its sibling release.sh this script has no host, image or container to +# get wrong, so it defaults to DETECTING rather than to refusing. The rule it +# inherits unchanged is the important half: it never claims to have checked +# something it did not check. A detector that finds nothing says so and exits +# non-zero; it does not print a green table. +# +# Assumes: bash, coreutils, git. Each detector additionally needs the toolchain +# it detects, and says so by name when that toolchain is missing instead of +# quietly dropping the step. +# +# ## Why this exists +# +# "Did I break anything" should not be a judgement call, and in a project with +# three separate check commands it always is. The commands live in different +# places — a package.json script, a Makefile target, a lint you have to +# remember — so the honest answer to "did you run everything" is usually "I ran +# the one I remembered". The suite passes, the typecheck was never run, and the +# breakage is found by the deploy. +# +# One entry point removes the judgement. It also removes the excuse: there is +# no "I ran the important one" when running all of them is the same amount of +# typing. +# +# The second reason is the summary. A chain of `&&` stops at the first failure, +# so a run tells you about one broken thing at a time and you pay the whole +# cost again to find the next. This runs every step even after one fails, so a +# single run tells you everything that is broken. +# +# ## Guard tests are the point +# +# This project family keeps getting bitten by bugs that are invisible to +# ordinary tests, because the test and the bug agree with each other: +# +# - A Date serialised across a SQL boundary came back shifted by the server's +# timezone. Every test asserted against the same shifted value, so the +# suite was green and the dates were wrong. +# - A React prop was spread over a form field and silently overwrote its +# `name`. The component rendered, the test rendered it, and the field +# submitted under the wrong key. +# - A `redirect()` was called inside a `try` block. Next.js implements +# redirect by throwing, so the `catch` swallowed it and every successful +# action reported failure. +# +# None of those is a logic error a unit test would catch. All three are SHAPES +# in the source: a shape you can grep for. The fix is a guard test — a script +# that greps the source for the shape and exits non-zero when it reappears — +# and this script is where guard tests belong, because verify.sh is the thing +# that actually gets run. +# +# Write them as small executables in GUARD_DIR. One shape per file, named for +# the bug, exiting non-zero with a message naming the file and line. They cost +# milliseconds, they run first here for exactly that reason, and they are the +# only mechanism in the repository that catches a bug the tests cannot see. +# +# ## SKIPPED is not PASS +# +# A step that did not run gets its own state in the table and its own colour, +# and it is never folded into the pass count. This matters more than it looks: +# the failure this script exists to prevent is a green table produced by +# checking nothing, and every path to that failure runs through a skip that was +# reported as a success. So a run where nothing actually executed exits 2 even +# though nothing failed, and --quick names the step it dropped rather than +# quietly shortening the table. +# +# ## What it deliberately does not do +# +# It does not fix anything. No --fix, no formatter writing to your files: this +# runs immediately before a commit, and a verify that edits the tree changes +# what you were about to commit into something you have not read. +# +# It does not touch git — no staging, no committing, no stash, no branch check. +# It answers one question about the working tree as it stands. +# +# It does not walk into workspaces or sub-packages. Detection runs at the +# repository root, once. A monorepo wants a guard script per package, or a +# Makefile target that fans out, and either is a step this will find. +# +# It has no --dry-run because it writes nothing OF ITS OWN: no file is created, +# deleted, truncated or moved anywhere in this script. That is not the same as +# "changes nothing", and the difference matters. Every step it runs is somebody +# else's program — a package.json script, a Makefile target, an executable in +# GUARD_DIR — and those inherit no restraint from here. Dropping a test +# database is the usual one. +# +# --list is the dry-run equivalent: it shows the exact command each step would +# run, and runs none of them. Read it once, on a new project, before trusting +# this in a hook. That is the only place the full list is visible before it +# executes. + +set -uo pipefail + +say() { printf '\033[1mverify:\033[0m %s\n' "$*" >&2; } +die() { printf '\033[1mverify:\033[0m %s\n' "$*" >&2; exit 1; } + +ROOT=$(git rev-parse --show-toplevel 2>/dev/null) \ + || die "not inside a git repository. Detection runs from the repository root so every detector looks in one place; cd into the repo and run this again." + +# Empty is checked separately from failed. `cd ""` succeeds and stays put, so an +# empty ROOT would not fail here — it would silently detect in whatever +# directory you happened to be in, which is the wrong answer delivered +# confidently. A bare repository is the way it happens. +[ -n "$ROOT" ] \ + || die "git reported an empty repository root — this looks like a bare repository, which has no working tree to check. Run this from a normal checkout." + +cd "$ROOT" || die "cannot cd to $ROOT" + +# Resolved once, with symlinks collapsed, so the guard-directory containment +# check below compares two paths of the same kind. +ROOT_ABS=$(pwd -P) + +# --------------------------------------------------------------------------- +# CONFIGURATION +# --------------------------------------------------------------------------- + +# Where guard scripts live, relative to the repository root. Empty by default: +# an unset path is a path that cannot point at the wrong thing, and this is the +# one value worth setting by hand — see "Guard tests are the point" above. +# +# Set and missing is a hard failure, not a skip. A guard directory that got +# renamed is precisely the case where silently running zero guards looks +# identical to running them all. +# +# IT MUST BE A DEDICATED DIRECTORY. Every executable file in it is RUN, with no +# allowlist — unlike the npm and make candidate lists below, which name the +# handful of scripts they are willing to invoke. That asymmetry is the whole +# hazard: point this at scripts/ and verify.sh runs release.sh; point it at a +# bin directory and it runs every binary there. detect_guards refuses the cases +# it can prove wrong (outside the repository, the repository root itself, the +# directory holding this script), but it cannot tell a guard from a deploy +# script that happens to sit beside one. Give guards their own directory and +# put nothing else in it. +GUARD_DIR="${VERIFY_GUARD_DIR:-}" + +# Which step --quick drops, by exact id (the left column of the table). Empty +# means "every step classified as a test", which is the right guess almost +# everywhere and is stated out loud when it is used. +SLOW_STEP="${VERIFY_SLOW_STEP:-}" + +# Candidate script and target names, checked against what the project actually +# has. Nothing here is run unless it exists — these are search terms, not +# defaults, and the distinction is load-bearing: `npm run lint` on a project +# without a lint script fails, and a verify that invents work to do fails for +# reasons that have nothing to do with the code. +# +# It is also an allowlist, and that is the other half of why it exists. A +# package.json contains scripts named `deploy` and `db:reset`. Running +# everything found would be a verify that publishes. +NODE_TYPECHECK_SCRIPTS="typecheck type-check tsc types" +NODE_LINT_SCRIPTS="lint lint:ci eslint format:check fmt:check" +NODE_TEST_SCRIPTS="test test:ci" + +# `verify` and `all` are absent on purpose: a Makefile in a repository holding +# this script very likely has a `verify` target that calls this script, and +# that is an infinite loop rather than a failed check. The guard below catches +# it anyway, for the `check` target that turns out to do the same thing. +MAKE_LINT_TARGETS="lint fmt-check format-check" +MAKE_TYPECHECK_TARGETS="typecheck type-check types" +MAKE_TEST_TARGETS="test check" + +# --------------------------------------------------------------------------- +# Refuse to be re-entered. +# +# A detected step that calls this script back — `make check` running +# scripts/verify.sh is the way it happens — recurses until the machine gives +# up, and the symptom is a hang rather than an error. Dying on the second entry +# turns that into one legible failure naming the step that did it. +# --------------------------------------------------------------------------- +if [ -n "${VERIFY_RUNNING:-}" ]; then + die "verify.sh invoked itself — a detected step calls this script back, which would recurse forever. Remove that target from the candidate lists in this file, or stop it calling verify." +fi +export VERIFY_RUNNING=1 + +QUICK="" +LIST="" +ONLY="" + +while [ $# -gt 0 ]; do + case "$1" in + --quick) QUICK="yes" ;; + --list) LIST="yes" ;; + --only) + shift + [ $# -gt 0 ] || die "--only needs a pattern. Usage: --only " + ONLY="$1" ;; + --only=*) ONLY="${1#--only=}" ;; + *) die "unknown argument '$1'. Usage: verify.sh [--quick] [--list] [--only ]" ;; + esac + shift +done + +if [ -t 2 ]; then + C_PASS=$'\033[32m'; C_FAIL=$'\033[31;1m'; C_SKIP=$'\033[33m'; C_OFF=$'\033[0m' +else + C_PASS=""; C_FAIL=""; C_SKIP=""; C_OFF="" +fi + +# --------------------------------------------------------------------------- +# The plan. Built completely before anything runs, so that "nothing was +# detected" is discovered before a single command executes rather than after a +# five-minute suite. +# +# Parallel arrays indexed together, iterated with a counted loop: `${arr[@]}` on +# an empty array is an unbound-variable error under `set -u` in bash 3.2, which +# is the bash on every stock macOS, and the empty case is the one that matters +# most here. +# --------------------------------------------------------------------------- +STEP_ID=() +STEP_CLASS=() +STEP_CMD=() +STEP_SKIP=() +STEP_RESULT=() +STEP_SECS=() + +add_step() { STEP_ID+=("$1"); STEP_CLASS+=("$2"); STEP_CMD+=("$3"); STEP_SKIP+=(""); } + +# A step that was found but cannot run. Recorded rather than dropped, because +# "this project has a mypy config and mypy is not installed" is information, +# and a silently shorter table is not. +add_skip() { STEP_ID+=("$1"); STEP_CLASS+=("skip"); STEP_CMD+=(""); STEP_SKIP+=("$2"); } + +q() { printf '%q' "$1"; } + +have() { command -v "$1" >/dev/null 2>&1; } + +# --------------------------------------------------------------------------- +# Guards first. They are greps: they finish before the toolchain has finished +# starting, and they are the only steps that catch the bug class described in +# the header. Putting them last would mean the cheapest answer arrives after +# the most expensive one. +# --------------------------------------------------------------------------- +detect_guards() { + [ -n "$GUARD_DIR" ] || return 0 + + [ -d "$GUARD_DIR" ] \ + || die "VERIFY_GUARD_DIR is set to '$GUARD_DIR' but there is no such directory. Create it, or unset the variable — a missing guard directory would otherwise run zero guards and look exactly like running all of them." + + # ------------------------------------------------------------------------- + # Containment. Everything below this point EXECUTES every file it finds, so a + # mis-set GUARD_DIR is not a wrong answer — it is arbitrary code with the + # developer's credentials, launched by the one command they were told is safe + # to run before every commit. + # + # The three refusals below are the cases that can be proven wrong rather than + # guessed at. Resolved with `cd`+`pwd -P` so that symlinks, `..` and relative + # paths all collapse to one comparable form before being judged. + # ------------------------------------------------------------------------- + local guard_abs self_abs + guard_abs=$(cd "$GUARD_DIR" 2>/dev/null && pwd -P) \ + || die "VERIFY_GUARD_DIR is set to '$GUARD_DIR' but that directory could not be entered (permissions?)." + + # Outside the repository. `..`, an absolute path and a symlink pointing out of + # the tree all land here. A guard is a check on THIS repository's source; a + # directory outside it holds someone else's programs. + case "$guard_abs/" in + "$ROOT_ABS"/*) ;; + *) die "VERIFY_GUARD_DIR '$GUARD_DIR' resolves to '$guard_abs', which is outside this repository ('$ROOT_ABS'). Every executable file in it would be RUN. Guard scripts belong in a dedicated directory inside the repository." ;; + esac + + # The repository root itself. Running every executable at the root means + # running whatever release, deploy or reset script the project keeps there. + [ "$guard_abs" != "$ROOT_ABS" ] \ + || die "VERIFY_GUARD_DIR points at the repository root. Every executable file at the root would be RUN as a guard, including any release, deploy or database script. Put guards in a dedicated subdirectory — scripts/verify.d is the convention." + + # The directory holding this script. This is the likely mistake, because it is + # where scripts live and it reads as the obvious answer: this template ships + # verify.sh beside release.sh, backup.sh and migrate.sh, and pointing the + # guard directory here would run all three and call the result a passing + # check. Skipped when $0 cannot be resolved, which loses nothing — the two + # refusals above still apply. + self_abs=$(cd "$(dirname -- "$0")" 2>/dev/null && pwd -P) || self_abs="" + if [ -n "$self_abs" ] && [ "$guard_abs" = "$self_abs" ]; then + die "VERIFY_GUARD_DIR points at the directory holding verify.sh itself ('$guard_abs'). Every executable file beside this script — release.sh, backup.sh, migrate.sh — would be RUN as a guard. Put guards in a dedicated subdirectory of their own." + fi + + local f path found="" + while IFS= read -r f; do + [ -n "$f" ] || continue + found="yes" + + # Made explicitly relative so the guard runs whether or not its directory is + # on PATH — but only when it is not already absolute, since './' in front of + # an absolute path silently resolves somewhere else entirely. + case "$f" in /*) path="$f" ;; *) path="./$f" ;; esac + + # A guard that lost its executable bit never runs and nothing notices. That + # is the same failure the whole script is about, one directory down. So is a + # guard symlinked in from a shared directory whose target has since moved: + # both are recorded, neither is dropped. + if [ -x "$f" ]; then + add_step "guard:$(basename "$f")" guard "$(q "$path")" + elif [ ! -e "$f" ]; then + add_skip "guard:$(basename "$f")" "broken symlink — it points at nothing, so this guard has not run since the target moved" + else + add_skip "guard:$(basename "$f")" "not executable — chmod +x it, or it will never run again either" + fi + + # -L so a symlinked guard, and a symlinked guard DIRECTORY, are both seen. + # Sharing one guard across sibling repositories by symlink is the normal way + # to do it, and plain `-type f` does not match a symlink: those guards were + # not skipped, not listed and not run, which is the silent-zero failure this + # script exists to prevent. `! -type d` rather than `-type f` so a broken + # symlink still surfaces above instead of vanishing again. + done < <(find -L "$GUARD_DIR" -maxdepth 1 ! -type d 2>/dev/null | LC_ALL=C sort) + + [ -n "$found" ] || say "note: $GUARD_DIR is configured but empty. See 'Guard tests are the point' at the top of this file for what belongs there." +} + +# --------------------------------------------------------------------------- +# Node. The script names come from package.json and nowhere else. +# +# Read with node rather than grepped, because a grep for '"test"' matches keys +# in devDependencies, in a nested tool config, and in lint-staged — and +# `npm run` on a name that is not a script exits non-zero, so the guess would +# surface as a failing check with no failing code behind it. +# --------------------------------------------------------------------------- +detect_node() { + [ -f package.json ] || return 0 + + if ! have npm || ! have node; then + add_skip "npm" "package.json is here but node/npm is not on PATH — none of its checks could run" + return 0 + fi + + local scripts rc + scripts=$(node -e 'try{const s=require(process.cwd()+"/package.json").scripts||{};process.stdout.write(Object.keys(s).join("\n"))}catch(e){process.exit(1)}' 2>/dev/null); rc=$? + + # Kept separate from "there are no scripts" below. Both end in zero npm steps + # and they need different answers: one is a project that has no checks yet, + # the other is a file this script could not read, which is also about to break + # every npm command anyone else runs today. + if [ "$rc" -ne 0 ]; then + add_skip "npm" "package.json could not be parsed by node — fix the JSON; no npm check could be read from it" + return 0 + fi + + if [ -z "$scripts" ]; then + add_skip "npm" "package.json declares no scripts — this Node project was not checked at all" + return 0 + fi + + local class list name matched="" + for class in typecheck lint test; do + case "$class" in + typecheck) list="$NODE_TYPECHECK_SCRIPTS" ;; + lint) list="$NODE_LINT_SCRIPTS" ;; + test) list="$NODE_TEST_SCRIPTS" ;; + esac + + for name in $list; do + printf '%s\n' "$scripts" | grep -qx -- "$name" || continue + add_step "npm:$name" "$class" "npm run $(q "$name")" + matched="yes" + # One per class. Two matches usually means `test` and `test:ci` are the + # same suite twice, and paying for a suite twice is how people stop + # running verify. + break + done + done + + if [ -z "$matched" ]; then + say "note: package.json has scripts, but none named like a check. Looked for:" + say " $NODE_TYPECHECK_SCRIPTS $NODE_LINT_SCRIPTS $NODE_TEST_SCRIPTS" + say " Add the real names to the candidate lists near the top of this file." + fi +} + +detect_make() { + local mk="" f + for f in Makefile makefile GNUmakefile; do + if [ -f "$f" ]; then mk="$f"; break; fi + done + [ -n "$mk" ] || return 0 + + if ! have make; then + add_skip "make" "$mk is here but make is not on PATH" + return 0 + fi + + local class list target matched="" + for class in typecheck lint test; do + case "$class" in + typecheck) list="$MAKE_TYPECHECK_TARGETS" ;; + lint) list="$MAKE_LINT_TARGETS" ;; + test) list="$MAKE_TEST_TARGETS" ;; + esac + + for target in $list; do + # Anchored at the start of the line so this matches a rule and not a + # .PHONY declaration or a variable that happens to contain the word. + grep -qE "^${target}[[:space:]]*:" "$mk" || continue + add_step "make:$target" "$class" "make $(q "$target")" + matched="yes" + break + done + done + + # Said out loud, for the same reason the npm detector says it. A Makefile + # whose target is `tests` or `ci` contributes nothing here, and in a project + # that ALSO has a package.json the run still prints a full green table — one + # that silently excludes the Makefile's suite. Contributing zero steps is + # information; contributing zero steps quietly is the failure. + if [ -z "$matched" ]; then + say "note: $mk is here, but none of its targets are named like a check. Looked for:" + say " $MAKE_TYPECHECK_TARGETS $MAKE_LINT_TARGETS $MAKE_TEST_TARGETS" + say " Add the real names to the candidate lists near the top of this file." + fi +} + +detect_cargo() { + [ -f Cargo.toml ] || return 0 + + if ! have cargo; then + add_skip "cargo" "Cargo.toml is here but cargo is not on PATH" + return 0 + fi + + if cargo fmt --version >/dev/null 2>&1; then + add_step "cargo:fmt" lint "cargo fmt --all -- --check" + else + add_skip "cargo:fmt" "rustfmt is not installed (rustup component add rustfmt)" + fi + + if cargo clippy --version >/dev/null 2>&1; then + # `-D warnings` is not strictness for its own sake: without it clippy prints + # its findings and exits 0, so the step passes whatever it found — a check + # that cannot fail. Loosen it here if this project has warnings it has + # decided to keep, but loosen it visibly. + add_step "cargo:clippy" lint "cargo clippy --all-targets -- -D warnings" + else + add_skip "cargo:clippy" "clippy is not installed (rustup component add clippy)" + fi + + add_step "cargo:test" test "cargo test" +} + +detect_python() { + [ -f pyproject.toml ] || return 0 + + PY_MATCHED="" + + # Configured in pyproject means the project uses it. A tool that merely + # happens to be installed on this machine is not a check this project has, + # and running it would invent a standard the repository never agreed to. + py_tool() { # section-regex binary class command + grep -qE "$1" pyproject.toml || return 0 + PY_MATCHED="yes" + if have "$2"; then + add_step "py:$2" "$3" "$4" + else + add_skip "py:$2" "configured in pyproject.toml but '$2' is not on PATH — activate the virtualenv, or install it" + fi + } + + py_tool '^\[tool\.ruff' ruff lint "ruff check ." + py_tool '^\[tool\.black' black lint "black --check ." + py_tool '^\[tool\.mypy' mypy typecheck "mypy ." + # pytest is matched anywhere in the file, not only as a [tool.pytest] section: + # most projects configure nothing and simply depend on it. + py_tool 'pytest' pytest test "pytest" + + if [ -z "$PY_MATCHED" ]; then + say "note: pyproject.toml is here, but it configures none of the tools this" + say " looks for (ruff, black, mypy, pytest). No Python check was run." + fi +} + +detect_go() { + [ -f go.mod ] || return 0 + + if ! have go; then + add_skip "go" "go.mod is here but go is not on PATH" + return 0 + fi + + add_step "go:vet" lint "go vet ./..." + add_step "go:test" test "go test ./..." +} + +detect_guards +detect_node +detect_make +detect_cargo +detect_python +detect_go + +# --------------------------------------------------------------------------- +# Nothing detected is a failure, and the loudest one here. +# +# A verify that checks nothing and prints success is worse than no verify at +# all: it converts "I have not checked" into "it passed" for everyone +# downstream, including the hook that trusts the exit code. +# --------------------------------------------------------------------------- +if [ "${#STEP_ID[@]}" -eq 0 ]; then + say "NO CHECKS DETECTED. Nothing was verified and nothing passed." + say "" + say "Looked at the repository root for: package.json scripts, a Makefile" + say "target, Cargo.toml, pyproject.toml, go.mod, and VERIFY_GUARD_DIR" + say "(currently ${GUARD_DIR:-unset})." + say "" + say "Give it something to run: add a check script to package.json, add a" + say "target to the Makefile, or set VERIFY_GUARD_DIR to a directory of" + say "executable guard scripts. Then run this again." + exit 2 +fi + +if [ -n "$ONLY" ]; then + n=${#STEP_ID[@]} + kept="" + for ((i = 0; i < n; i++)); do + case "${STEP_ID[$i]}" in + *"$ONLY"*) kept="yes" ;; + # An existing reason wins. A step already skipped because its tool is not + # installed is not skipped BY --only, and overwriting the reason would + # report that it was filtered out when in fact it could not have run — + # turning a real gap in the checks into a deliberate-looking choice. + *) [ -n "${STEP_SKIP[$i]}" ] || STEP_SKIP[$i]="--only $ONLY" ;; + esac + done + [ -n "$kept" ] || die "--only '$ONLY' matched none of the detected steps: $(printf '%s ' "${STEP_ID[@]}")" +fi + +# --quick, resolved against the plan rather than assumed. A SLOW_STEP naming a +# step this project does not have is a stale config that would silently skip +# nothing while reporting that it skipped, so it is refused. +if [ -n "$QUICK" ]; then + n=${#STEP_ID[@]} + hit="" + for ((i = 0; i < n; i++)); do + if [ -n "$SLOW_STEP" ]; then + [ "${STEP_ID[$i]}" = "$SLOW_STEP" ] || continue + else + [ "${STEP_CLASS[$i]}" = "test" ] || continue + fi + [ -n "${STEP_SKIP[$i]}" ] || STEP_SKIP[$i]="--quick" + hit="yes" + done + + if [ -z "$hit" ]; then + if [ -n "$SLOW_STEP" ]; then + die "VERIFY_SLOW_STEP is '$SLOW_STEP' but no detected step has that id: $(printf '%s ' "${STEP_ID[@]}")" + fi + say "note: --quick had no effect — no detected step is a test step." + elif [ -z "$SLOW_STEP" ]; then + say "--quick: skipping the test steps (set VERIFY_SLOW_STEP to an id to pick a different one)." + fi +fi + +n=${#STEP_ID[@]} + +if [ -n "$LIST" ]; then + say "$n step(s) detected. Nothing was run:" + for ((i = 0; i < n; i++)); do + if [ -n "${STEP_SKIP[$i]}" ]; then + say " $(printf '%-18s' "${STEP_ID[$i]}") SKIPPED — ${STEP_SKIP[$i]}" + else + say " $(printf '%-18s' "${STEP_ID[$i]}") ${STEP_CMD[$i]}" + fi + done + exit 0 +fi + +# --------------------------------------------------------------------------- +# Run everything. No early exit on failure: the point of one entry point is one +# answer, and an answer that stops at the first broken thing makes you pay the +# whole runtime again for the second. +# +# Output is streamed rather than captured, so a step that hangs is visible +# while it hangs instead of after it is killed. +# --------------------------------------------------------------------------- +started=$SECONDS +failed=0 +passed=0 +skipped=0 + +for ((i = 0; i < n; i++)); do + if [ -n "${STEP_SKIP[$i]}" ]; then + STEP_RESULT[$i]="SKIPPED" + STEP_SECS[$i]="-" + skipped=$((skipped + 1)) + continue + fi + + say "── ${STEP_ID[$i]} — ${STEP_CMD[$i]}" + t0=$SECONDS + eval "${STEP_CMD[$i]}" + code=$? + STEP_SECS[$i]=$((SECONDS - t0)) + + if [ "$code" -eq 0 ]; then + STEP_RESULT[$i]="PASS" + passed=$((passed + 1)) + else + STEP_RESULT[$i]="FAIL" + failed=$((failed + 1)) + say "${STEP_ID[$i]} failed (exit $code) — continuing so this run reports everything." + fi +done + +# --------------------------------------------------------------------------- +# The table. Three states, three words, three colours, and the word alone is +# enough when the colour is gone — this gets piped into files and hook output +# where nothing is a terminal. +# --------------------------------------------------------------------------- +say "" +say "───────────────────────────────────────────────────────────" + +for ((i = 0; i < n; i++)); do + case "${STEP_RESULT[$i]}" in + PASS) colour="$C_PASS" ;; + FAIL) colour="$C_FAIL" ;; + *) colour="$C_SKIP" ;; + esac + + line="$(printf ' %s%-8s%s %-26s %5s' \ + "$colour" "${STEP_RESULT[$i]}" "$C_OFF" "${STEP_ID[$i]}" "${STEP_SECS[$i]}")" + + if [ "${STEP_RESULT[$i]}" = "SKIPPED" ]; then + line="${line} ${STEP_SKIP[$i]}" + else + line="${line}s" + fi + + say "$line" +done + +say "───────────────────────────────────────────────────────────" + +elapsed=$((SECONDS - started)) + +# Counts are stated only for things that were measured. "0 failed" alongside +# four skips is true and misleading, so the skip count is never omitted and the +# exit code below refuses to call an all-skipped run a pass. +summary="${passed} passed" +[ "$failed" -eq 0 ] || summary="${summary}, ${failed} FAILED" +[ "$skipped" -eq 0 ] || summary="${summary}, ${skipped} SKIPPED (not run — not verified)" + +say "${summary} [${elapsed}s]" + +if [ "$failed" -gt 0 ]; then + say "" + say "Fix the FAILED step(s) above. To re-run one on its own:" + for ((i = 0; i < n; i++)); do + # Both halves shell-quoted, because both routinely contain spaces: this + # family of projects lives under directories like "Privacy LLC", and a step + # id is a guard's filename. Printed unquoted, the command this suggests + # parses as extra arguments and dies on the copy-paste. + [ "${STEP_RESULT[$i]}" = "FAIL" ] \ + && say " bash $(q "$0") --only $(q "${STEP_ID[$i]}")" + done + exit 1 +fi + +# Nothing failed and nothing ran. That is not a pass, and the exit code is the +# only part of this output a hook or a CI job will read. +if [ "$passed" -eq 0 ]; then + say "" + say "NOTHING WAS VERIFIED — every detected step skipped. Exiting non-zero:" + say "a clean table over an empty run is the one outcome this script exists" + say "to prevent." + exit 2 +fi + +exit 0 diff --git a/docs/data/README.md b/docs/data/README.md new file mode 100644 index 0000000..b183607 --- /dev/null +++ b/docs/data/README.md @@ -0,0 +1,80 @@ +# Data — + +``` +Status: Current +Owner: +Last reviewed: +Governs: docs/data/** — the assets privacyllc.dev renders for this project +Review trigger: A rebrand, or any change to the icon, logo or banner +``` + +## What goes here + +Three files, in `img/`, at exactly these names: + +```text +docs/data/img/icon.webp the square mark, used wherever the project is listed +docs/data/img/logo.webp the full lockup, used on the project page +docs/data/img/banner.webp the wide image, used across the project header +``` + +All **webp**. All **required**. Only `img/` is checked — an asset left at +`docs/data/logo.webp` is not found. + +**Dimensions, weights and how to generate them are in +[`img/README.md`](img/README.md)**, beside the files they describe. This +document owns the rule; that one owns the spec. Stating both in both places is +how two copies of one convention start disagreeing. + +**No placeholders ship with this template, deliberately.** It carried a 0-byte +`logo.webp` once, and an empty file is the worst of the three states: a check +that asks "does the path exist" calls it present, and anything that reads the +bytes rejects it — a consumer verifying the webp signature answers 415, which +reads as a corrupt asset rather than a missing one. Absent is honest and the +conformance check reports it as absent, which is what gets it filled in. + +Extra sizes and variants are welcome beside them — `icon-512.webp`, +`logo-dark.webp` — and are not treated as clutter. Only the three exact names +are checked for. + +## If an asset is missing, open an issue — do not invent one + +An agent cannot draw a logo, and this is the one gap in the whole convention +that cannot be closed by writing a file. + +So when an asset is absent, **file an issue** rather than producing something: +title it for the asset, label it `P2`, and end the body with its `Verify:` line +— `Verify: docs/data/img/logo.webp exists and the project card renders it.` + +**Do not generate a placeholder.** A placeholder that looks deliberate outlives +the issue that would have replaced it: nobody files a ticket against an image +that appears to be finished. An obviously absent asset keeps asking. + +## Why this folder is different from every other one here + +The Command Center *consumes* these. Every other document in this tree is written +for a person to read; these are fetched and rendered on privacyllc.dev's project +page. + +That has one consequence worth stating plainly: **this folder cannot be declared +exempt.** A repository may tell the conformance check that a required document is +deliberately absent — kept out of git on purpose, say — and the check will +believe it. It will not accept that declaration for `docs/data/`, because the +result would be a project card with nothing to show and nothing explaining why, +which is the exact failure the check exists to prevent. + +## Why webp and not PNG + +One format, checked by its magic bytes rather than its file extension, so the +site can serve it inline with confidence. A file whose first bytes are RIFF/WEBP +cannot be an HTML document or an SVG carrying script, which is what makes it safe +to render directly rather than forcing a download. + +Renaming a PNG to `.webp` will not work, and is meant not to. + +## This is not an asset library + +Screenshots, mockups, reference art and exported source files do not belong here. +They belong wherever the project already keeps them. This folder holds the three +marks that identify the project elsewhere, and stays small enough that its +contents are obvious at a glance. diff --git a/docs/data/img/README.md b/docs/data/img/README.md new file mode 100644 index 0000000..2b4f914 --- /dev/null +++ b/docs/data/img/README.md @@ -0,0 +1,57 @@ +# Project images + +``` +Status: Current +Owner: +Last reviewed: +Governs: docs/data/img/** — the three files and their sizes +Review trigger: A rebrand; any change to a required name, dimension or ceiling; + any change to what the consumer accepts. +``` + +Three files, all **webp**, all **required**, at exactly these names: + +| File | Dimensions | Aspect | Typical weight | +| --- | --- | --- | --- | +| `icon.webp` | **512 × 512** | 1:1 | 8–60 KB | +| `logo.webp` | **1024** on the long edge | whatever the lockup is | 20–190 KB | +| `banner.webp` | **2176 × 725** | 3:1 | 30–130 KB | + +**Not `icon.ico`.** The consumer checks the file's magic bytes, not its name: a +`.ico`, or a PNG renamed to `.webp`, is refused with a 415 and the project falls +back to an initials tile. That signature check is what makes it safe to render +these inline, so it is not going to be relaxed. + +**512 KB is a hard ceiling per file**, enforced in code — the size is read from +the listing before the bytes are fetched, so an oversized asset is never +downloaded and simply never appears. Nothing enforces the dimensions, which is +why they are written down. + +## Making them + +```bash +magick logo-source.png -resize 512x512 -quality 82 icon.webp +identify -format '%f %wx%h %b\n' *.webp # check before committing +``` + +Quality 80–85 suits a flat mark. If a file lands over ~200 KB it is usually a +photographic banner that wants a lower quality rather than fewer pixels. + +## Why these numbers + +The icon renders small — a 44 px tile in a list, 58 px on a project header — so +512 covers the densest display several times over; the reference project +deliberately halved it from 1024. The banner spans a card about 760 px wide, so +~2176 covers it at 2×, and its 3:1 shape matters more than its width because the +header crops to fill. The logo has no fixed frame, so only its long edge is +specified. + +Extra sizes and variants are welcome beside these — `icon-512.webp`, +`logo-dark.webp`. Only the three exact names are checked for. + +No placeholders ship with this template. An empty file is the worst of the three +states: a check that asks whether the path exists calls it present, and anything +reading the bytes rejects it. Absent is honest, and the conformance check reports +it as absent — which is what gets it filled in. + +Why the requirement exists, and what reads it: [`../README.md`](../README.md). diff --git a/docs/design/README.md b/docs/design/README.md new file mode 100644 index 0000000..5739353 --- /dev/null +++ b/docs/design/README.md @@ -0,0 +1,34 @@ +# Design + +``` +Status: Current +Owner: +Last reviewed: +Governs: docs/design/** +Review trigger: Any new user-facing surface, or a change to the product's tone +``` + +## What belongs here + +What it should feel like, and the decisions behind that: + +- **Product decisions** — what the user can do, in what order, and what happens + when they get it wrong. The error states are design, not an afterthought. +- **UI plans** — screens, states, and what each one is for. Include the empty + state and the loading state; they are the two most people see first and the + two most often left undesigned. +- **Copy** — the actual words. Interface text is a design surface, and writing + it late means writing it badly. +- **Tone** — how this product talks. One paragraph is enough, and it settles a + hundred small arguments. + +## What does not belong here + +- How it is built — that is `docs/architecture/` +- Scope and audience — that is `docs/planning/PROJECT_PLAN.md` + +## Include the rejected version + +For any decision that was genuinely close, record what was not chosen and why. +Without it, the same option gets proposed every few months and re-argued from +nothing. diff --git a/docs/history/BATCH_LEDGER.md b/docs/history/BATCH_LEDGER.md new file mode 100644 index 0000000..c372473 --- /dev/null +++ b/docs/history/BATCH_LEDGER.md @@ -0,0 +1,45 @@ +# Batch ledger — + +``` +Status: Archived +Owner: +Last reviewed: +Governs: what the batches were, before the tracker held them +Review trigger: Nothing. Superseded by the tracker; kept for the record. +``` + +## Why this file is archived rather than deleted + +Planning used to live in markdown as a numbered batch list. It now lives in the +tracker: **milestones are batches, issues are deliverables**, and severity is +`P0` / `P1` / `P2` / `release-blocker`. + +This file is what that list *was*. It is kept because the reasoning in it is +still worth reading, and deleted files are not readable. + +**It must not be updated.** A batch list beside the tracker is a second answer to +"what is open", and two records of the same thing will disagree without saying +which is right. That is precisely why the work moved. If you are tempted to add a +batch here, add a milestone instead. + +Its `Status: Archived` is therefore not a nicety — it is the machine-readable +form of "do not treat this as current". + +## "Ledger: missing" on the project screen is the correct answer + +The Command Center's docs report still looks for a batch ledger — at +docs/planning/FUTURE.md, named here without backticks deliberately, because +`doc-claims.sh` treats a backticked path as a claim that the file exists and +this one must not — and reports it as **missing** for every repository that has +moved planning into the tracker. That is the expected state and not a gap to +close: the report's own contract is that a tracker-first project shows a missing +ledger and no batch percentages. + +Creating that file to turn the line green would rebuild, in a new name, the +exact second record this one was archived for. + +## The batches, as they stood + +| Batch | What it covered | Outcome | +| --- | --- | --- | +| <> | <> | <> | diff --git a/docs/history/DEVELOPMENT_LOG.md b/docs/history/DEVELOPMENT_LOG.md new file mode 100644 index 0000000..c903459 --- /dev/null +++ b/docs/history/DEVELOPMENT_LOG.md @@ -0,0 +1,51 @@ +# Development log — + +``` +Status: Current +Owner: +Last reviewed: +Governs: the dated record of what happened +Review trigger: Nothing. This file is appended to, never revised. +``` + +## How to use this + +Newest first. **One entry per work session**, written before you stop — that is +step 6 of `docs/WORK_CYCLE.md`, and the two lines it insists on are `Next +action` and `Blockers`. + +Those two are not decoration. The next session starts by reading the top of this +file, and a session that ended without saying what came next hands the one after +it a re-derivation instead of a starting point — which is where drift enters. +Neither line competes with anything: the live next action is the field on the +project at privacyllc.dev and the live blockers are issues in the tracker, while +these say what both were **at this date**. A record of then never disagrees with +a record of now. + +**Append-only by convention.** Correcting an old entry rewrites the record of +what was known at the time, which is the one thing this file is for. If an entry +turns out to be wrong, add a later entry saying so; do not edit the first. + +Note the Review trigger above says "nothing", deliberately. A dated log cannot +rot the way a description of current state can — the entries were true when +written and stay true. It is exempt from review for the same reason a receipt is. + +## Entries + +### — + + + +- **Closed:** <#N, #N — or "none", which is also an answer> +- **Next action:** +- **Blockers:** + +### — + +<…> diff --git a/docs/history/HISTORY.md b/docs/history/HISTORY.md new file mode 100644 index 0000000..50dda60 --- /dev/null +++ b/docs/history/HISTORY.md @@ -0,0 +1,45 @@ +# History — + +``` +Status: Current +Owner: +Last reviewed: +Governs: the narrative of how this project got to where it is +Review trigger: A decision reversed, a direction abandoned, or a rewrite +``` + +## What this is for + + + +**This is a record of *then*, never a description of *now*.** That distinction is +what makes it safe to leave alone as the project changes — a history document +that gets edited to stay current is not a history, it is a second and competing +description of the present. + +If you find yourself updating a sentence here because the code changed, the +sentence belongs somewhere else. + +## Decisions and their reasons + + + +| When | What was decided | Why, at the time | +| --- | --- | --- | +| <> | <> | <> | + +## What was tried and dropped + + + +## What this file is not + +- **Not the plan.** That is `docs/planning/PROJECT_PLAN.md`. +- **Not the work list.** Open work lives in the tracker as milestones and issues. +- **Not a changelog.** Dated entries go in `docs/history/DEVELOPMENT_LOG.md`. diff --git a/docs/planning/PROJECT_PLAN.md b/docs/planning/PROJECT_PLAN.md new file mode 100644 index 0000000..9f6ca22 --- /dev/null +++ b/docs/planning/PROJECT_PLAN.md @@ -0,0 +1,53 @@ +# — Project Plan + +``` +Status: Current +Owner: +Last reviewed: +Governs: scope, audience, and what this project deliberately is not +Review trigger: Any change of scope, audience, or platform +``` + +> The vision. The **milestones in this repository's issue tracker** hold the +> sequence of work; this holds what the work is *for*. When the two disagree, +> this one is the argument and those are the plan — resolve it rather than +> letting both stand. +> +> Deliberately no batch list here. A schedule in a document goes stale the day +> after it is written and nothing notices; a schedule in the tracker is updated +> by the act of doing the work. + +## What this is + + + +## Who it is for + + + +## What it is deliberately not + + + +- +- + +## Stack and platform + +| Concern | Choice | Why | +| --- | --- | --- | +| | | | +| | | <> | +| | | <> | + +## Success looks like + + + +## Known risks + + diff --git a/docs/qa/ClaudeQACoverage.md b/docs/qa/ClaudeQACoverage.md new file mode 100644 index 0000000..c8d9c76 --- /dev/null +++ b/docs/qa/ClaudeQACoverage.md @@ -0,0 +1,32 @@ +# Claude QA Coverage — + +``` +Status: Current +Owner: +Last reviewed: +Governs: what each QA pass actually reached +Review trigger: Any QA round run +``` + +> Pass by pass, what was reached and what was not. The point of this file is the +> **Blocked** rows: a pass left out of a report reads exactly like a pass that +> succeeded, and that is how untested code ships believing it was tested. + +## Round — at `` + +| Pass | Result | Notes | +| --- | --- | --- | +| A — First run | Pass | | +| B — Core flow | Pass | <> | +| C — Failure paths | Partial | | +| D — Persistence | Pass | <> | +| E — End of loop | **Blocked** | | +| F — Accessibility | Not run | | +| G — Performance | Pass | | + +## Standing gaps + + + +- diff --git a/docs/qa/ClaudeQAPlan.md b/docs/qa/ClaudeQAPlan.md new file mode 100644 index 0000000..c6fb459 --- /dev/null +++ b/docs/qa/ClaudeQAPlan.md @@ -0,0 +1,74 @@ +# Claude QA Plan — + +``` +Status: Current +Owner: +Last reviewed: +Governs: what a QA round consists of +Review trigger: Any new user-facing surface, or a defect class that got through +``` + +> The playbook. What a round *is*, so two rounds are comparable and a gap is +> visible rather than assumed covered. + +## Before a round + +- Build from a clean checkout at a known SHA, and record that SHA. +- Run from a detached worktree if other work is in flight, so uncommitted + changes cannot contaminate what is under test. +- Note the environment: device, OS version, browser, screen size — whatever the + product's behaviour actually depends on. + +## The passes + +Each pass gets a letter, so `ClaudeQACoverage.md` can report per pass and a +skipped one is visible. + +| Pass | What it covers | +| --- | --- | +| A | First run: install or load, cold start, permissions, empty states | +| B | The core flow, end to end, as a real user would do it | +| C | The core flow with things going wrong: no network, denied permission, invalid input | +| D | Persistence: quit and return, background and resume, restart | +| E | The end of the loop — the state that is hardest to reach on purpose | +| F | Accessibility: keyboard only, screen reader labels, contrast, text scaling | +| G | Performance under the load this product will actually see | + + + +## What counts as a finding + +A finding needs: what was done, what happened, what should have happened, and +the build SHA. Without the SHA it cannot be re-tested, and a finding that cannot +be re-tested cannot be closed. + +## Severity + +Findings are filed as issues, labelled: + +- **P0** — ships broken, or loses data +- **P1** — materially wrong, but shippable +- **P2** — cosmetic or low impact +- **release-blocker** — a release built today would be wrong rather than merely + incomplete + +Exactly these label names: the Command Center queries them by name, and a +repository that spells them differently has its defects reported as *not +adopted* rather than counted wrongly. + +Severity is what it costs, not how annoying it is to fix. + +## After a round + +File each finding as a labelled issue. Update `ClaudeReport.md`'s run-state +block and its overall sentence, and `ClaudeQACoverage.md` with what each pass +actually reached. A pass that could +not be run is recorded as blocked, with what blocks it — never quietly left out, +which reads identically to "passed". + +Then **push, and reconcile**. The verdict on the project screen at +privacyllc.dev is read out of `ClaudeReport.md` in the pushed repository, so a +round whose report is committed but not pushed — or pushed but not reconciled — +leaves a stakeholder reading the previous round's judgment with no indication +that a newer one exists. The rest of the cycle is in `docs/WORK_CYCLE.md`. diff --git a/docs/qa/ClaudeReport.md b/docs/qa/ClaudeReport.md new file mode 100644 index 0000000..66dc9ff --- /dev/null +++ b/docs/qa/ClaudeReport.md @@ -0,0 +1,53 @@ +# Claude QA Report — + +``` +Status: Current +Owner: +Last reviewed: +Governs: the QA verdict — build SHAs, round summaries, the overall judgment +Review trigger: Any QA round run +``` + +> The QA verdict. Companion to `ClaudeQACoverage.md` (what each pass reached) +> and `ClaudeQAPlan.md` (the playbook). +> +> **Defects are issues, not entries here.** A defect found in a round is filed +> in the tracker with a severity label, where it can be assigned, closed by a +> commit, and counted. This file keeps the part a tracker is bad at: a human's +> judgment about whether the thing is fit to ship. + +## Current run-state + +- **Last QA round:** +- **Last verified build SHA:** `` +- **Last tested device / environment:** +- **Overall status:** + +## Open defects + +**Do not list them here, and do not read a defect count out of this file.** The +Command Center's docs report parses this document for open `P0` / `P1` / `P2` +counts, and under this convention they are always zero — the defects are in the +tracker, which is the whole point. A repository with seventeen open P1s reports +`openP1: 0` in that block and the real figure in its tracker counts. Anything +summarising defects reads the tracker's label counts; the zeros here mean *this +file does not hold them*, never *there are none*. + +Filed as issues in this repository's tracker, labelled by what they cost: + +- **P0** — ships broken, or loses data +- **P1** — materially wrong, but shippable +- **P2** — cosmetic or low impact +- **release-blocker** — a release built today would be *wrong*, not merely + incomplete + +Severity is what it costs, not how annoying it is to fix. Every defect needs +the build SHA it was found at — a finding that cannot be re-tested cannot be +closed — so put it in the issue body. + +## Round notes + + diff --git a/docs/security/SECURITY.md b/docs/security/SECURITY.md new file mode 100644 index 0000000..591a5bc --- /dev/null +++ b/docs/security/SECURITY.md @@ -0,0 +1,64 @@ +# Security — + +``` +Status: Current +Owner: +Last reviewed: +Governs: authentication, secret handling, data at rest and in transit +Review trigger: Any new secret, any new external service, any change to auth or storage +``` + +## What this protects, and from whom + + + +| Asset | Where it lives | What would it cost to lose | +| --- | --- | --- | +| <> | <> | <> | + +## Secrets + +- **Nothing secret is committed.** Not in source, not in config, not in a test + fixture, not in a screenshot. +- Secrets arrive from the environment or a secret store, and are read at the + boundary rather than passed around. +- A key stored beside the data it protects protects nothing. If data is + encrypted at rest, say here where the key lives and how it is rotated. + +**Check before every release:** no private keys, keystores, `.env` files, +service-account JSON, or real credentials in the tracked tree — including in the +history, which a `git log -p` search covers and a directory listing does not. + +## Authentication and authorisation + + + +## Data in transit + + + +## Third parties + +| Service | What it receives | Why that is acceptable | +| --- | --- | --- | +| <> | <> | <> | + +Every row is a decision to send someone else's data somewhere. An empty table is +a good table. + +## Deliberately out of scope + + + +## Logging + +Logs are read by people and shipped to places. Nothing sensitive goes into one — +not a token, not a credential in a URL, not the contents of a user's record. An +error's *name* is almost always enough; its message, from a failed HTTP request, +is often the request URL. diff --git a/docs/security/SECURITY_CHECKLIST.md b/docs/security/SECURITY_CHECKLIST.md new file mode 100644 index 0000000..bc80dc2 --- /dev/null +++ b/docs/security/SECURITY_CHECKLIST.md @@ -0,0 +1,43 @@ +# Security checklist — + +``` +Status: Current +Owner: +Last reviewed: +Governs: the checks run before a release, and what each one proves +Review trigger: A new class of input, a new external service, or a finding that got past this list +``` + +## Why this is separate from SECURITY.md + +`SECURITY.md` is the threat model: what is being protected and from whom. It is +read carefully once and revisited rarely. + +This is the list somebody actually works through. Keeping them apart means the +model can stay stable while the checks change, and it means a release checklist +is short enough to finish rather than a document to skim. + +## Before a release + + + +- [ ] — proves +- [ ] — proves + +## Standing checks + +- [ ] No secret in the repository, in a log line, or in an error message +- [ ] Every externally reachable endpoint is either authenticated or deliberately public, and the deliberate ones are listed +- [ ] Every input that reaches a query or a filesystem path is validated at the boundary +- [ ] Dependencies audited, and any accepted advisory recorded with a reason + +## What got past this list + + + +| When | What was missed | The check now added | +| --- | --- | --- | +| <> | <> | <> | diff --git a/project-readme-template.md b/project-readme-template.md new file mode 100644 index 0000000..8168fa8 --- /dev/null +++ b/project-readme-template.md @@ -0,0 +1,188 @@ +# + +```text +Status: +Owner: +Last reviewed: +Governs: README.md as the project-facing overview for +Review trigger: +``` + + + +[Screenshots](#screenshots) | [What makes it different](#what-makes-it-different) | +[Product surface](#product-surface) | [Run locally](#run-locally) | +[QA and release](#qa-and-release) | [Project docs](#project-docs) + +_{ | | | + | | }_ + +![]() + + + +The core loop is: + +```text + -> -> +``` + +## Screenshots + + + + | | +--- | --- | --- +![]() | ![]() | ![]() + +## What Makes It Different + + + +| Principle | What it means | +| --- | --- | +| | | +| | | +| | | + +## Product Surface + +| Area | What it does | Why it matters | +| --- | --- | --- | +| | | | +| | | | +| | | | + +## Spotlight + +### + + + +![]() + +### + + + +![]() + +### + + + +![]() + +## Platform Status + +| Surface | Status | Notes | +| --- | --- | --- | +| | | | +| | | | + +## Architecture + +```text + + | + v + + | + +--> + +--> + +--> +``` + +Core stack: + +- +- +- +- +- + +## Run Locally + +Prerequisites: + +- +- + +Start the project: + +```bash + + +``` + +Open ``. + +## Configuration + +| Variable | Purpose | +| --- | --- | +| `` | | +| `` | | + +Keep secrets out of screenshots, README prose, and tracked sample files. + +## QA And Release + +Run checks: + +```bash + + +``` + +Release or deploy: + +```bash + +``` + + + +## Repository Map + +```text +/ +/ +/ +docs/ product, architecture, QA, security, and history docs +``` + +## Project Docs + +`README.md` is the landing page for someone evaluating the project. Detailed +procedures belong in docs, and open work belongs in the tracker. + +| Doc | Purpose | +| --- | --- | +| [docs/DOC_TRUST_MAP.md](docs/DOC_TRUST_MAP.md) | Which document owns which answer, and which source wins when records disagree. | +| [docs/WORK_CYCLE.md](docs/WORK_CYCLE.md) | What happens at the end of a piece of work: closing issues and milestones, recording next action and blockers, and what privacyllc.dev will and will not update on its own. | +| [docs/architecture/Engineering_Reference_Manual.md](docs/architecture/Engineering_Reference_Manual.md) | Architecture, data model, and engineering rules. | +| [docs/security/SECURITY.md](docs/security/SECURITY.md) | Threat model and security posture. | + +## Agent Notes + +- Product truth comes from the code and the tracker before prose. +- Do not keep a work list in this README. +- Finish with `docs/WORK_CYCLE.md`, every time: close the issues you finished + with the evidence that proves it, close the milestone if the batch landed, + update the documents the change triggered **in the same commit**, then record + the next action and any blockers before you stop. +- Nothing on privacyllc.dev writes itself except the tracker counts and the + pushed docs. The summary and next action change only when they are sent; the + milestone and blocker tables there change only when a human edits them. +- Do not claim a feature is built unless you can cite the route, file, + migration, release, or screenshot that proves it. +- Keep screenshots current when the UI changes meaningfully. +- Keep the first screen product-facing. Put deep operations lower down or in + docs.