feat(docs): OPERATIONS.md, the runbook that did not exist

The template shipped scripts to back up, deploy, check health and read the
deployed version, and no document saying where errors go, what alerts, who
receives it, or what to run first when it is down. Grep across docs/** found
zero mentions of error tracking, observability or database restore.

Five sections. Where errors go, with the distinction that matters at 3am --
healthcheck.sh answers "is it up", error tracking answers "is it working", and a
service returning 500 to everything is up. What alerts and to whom, naming a
person rather than a channel nobody owns. Backups, whose last row is the date of
the last verified restore, because a backup nobody has restored is a guess.
Rate limits and cost ceilings, *(precautionary)*. And an ordered "it is down,
what now" where every step is a command that changes nothing.

Marked *(only for a deployed service)*, with the instruction to delete rather
than keep the headings unanswered: an empty runbook reads as one nobody wrote,
which is worse than one that never applied.

scaffold.sh now lays it down (17 files, 0 skipped) and DOC_TRUST_MAP.md points
at it from both tables. Two prose counts in scaffold.sh's header said
"thirteen documents" and were already stale; they no longer carry a number,
since a count in prose beside a list in code drifts the moment the list grows --
which is what just happened.

closes #6

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
null 2026-08-17 22:57:20 -05:00
parent 063b715c33
commit 2dca89e635
3 changed files with 120 additions and 6 deletions

View File

@ -33,6 +33,7 @@ disagree, and nothing will say which one is right.
| What is the next action? | the `nextAction` field on the project at privacyllc.dev — the newest entry in `docs/history/DEVELOPMENT_LOG.md` says what it was *then* | | What is the next action? | the `nextAction` field on the project at privacyllc.dev — the newest entry in `docs/history/DEVELOPMENT_LOG.md` says what it was *then* |
| What is blocking us? | the tracker, for the work; the blockers table at privacyllc.dev, for the stakeholder-facing version | | What is blocking us? | the tracker, for the work; the blockers table at privacyllc.dev, for the stakeholder-facing version |
| What do I do when a piece of work is finished? | `docs/WORK_CYCLE.md` | | What do I do when a piece of work is finished? | `docs/WORK_CYCLE.md` |
| What watches this in production, and what do I do when it breaks? | `docs/OPERATIONS.md` |
| What happened, and when? | `docs/history/DEVELOPMENT_LOG.md` | | What happened, and when? | `docs/history/DEVELOPMENT_LOG.md` |
| Did QA pass, and what does the tester think? | `docs/qa/ClaudeReport.md` | | Did QA pass, and what does the tester think? | `docs/qa/ClaudeReport.md` |
| What did QA actually reach? | `docs/qa/ClaudeQACoverage.md` | | What did QA actually reach? | `docs/qa/ClaudeQACoverage.md` |
@ -58,7 +59,7 @@ two only ever change because a person changed them.
| `docs/security/` | `SECURITY` — threat model, secret handling, what is out of scope | | `docs/security/` | `SECURITY` — threat model, secret handling, what is out of scope |
| `docs/history/` | `DEVELOPMENT_LOG` (dated, append-only), `HISTORY` (the narrative), `BATCH_LEDGER` (archived, superseded by the tracker) | | `docs/history/` | `DEVELOPMENT_LOG` (dated, append-only), `HISTORY` (the narrative), `BATCH_LEDGER` (archived, superseded by the tracker) |
| `docs/data/` | the three branding marks privacyllc.dev renders for this project | | `docs/data/` | the three branding marks privacyllc.dev renders for this project |
| `docs/` root | this map; `WORK_CYCLE` — what to do when a piece of work is finished; `TOOLS` — where the scripts are and which can stop you | | `docs/` root | this map; `WORK_CYCLE` — what to do when a piece of work is finished; `TOOLS` — where the scripts are and which can stop you; `OPERATIONS` — what watches production, *(only for a deployed service)* |
`README.md` stays at the repository root; it is the landing page and moving it `README.md` stays at the repository root; it is the landing page and moving it
breaks that. Everything else lives under `docs/`. breaks that. Everything else lives under `docs/`.

112
docs/OPERATIONS.md Normal file
View File

@ -0,0 +1,112 @@
# Operations — <Project>
```
Status: Current
Owner: <who maintains this>
Last reviewed: <YYYY-MM-DD>
Governs: what watches this in production, and what to do when it stops
Review trigger: A new alert or a new place errors are sent; a change to the
backup schedule or the restore procedure; any incident whose first ten minutes
were spent working out what to run
```
> *(only for a deployed service.)* A library, a local tool or a static site with
> no backend deletes this file. Do not keep it with the headings unanswered — an
> empty runbook reads as one nobody wrote, which is worse than one that never
> applied.
## Why this exists
Every other document here describes what the project *is*. This one describes
what happens when it stops being that at three in the morning, and it is the
only document written for somebody who is not thinking clearly.
The failure it prevents is specific: **the first ten minutes of an incident
spent working out which host, which container, and which command**, while the
thing everybody actually wants to know — is it down, or is it just unreachable
from here — goes unanswered.
## Where errors go
<The service, the project or DSN, and who can see it. Name it, do not describe
it: "Sentry, project <name>, alerts to <where>" is useful at 3am and "we have
error tracking" is not.>
**A liveness tick is not this.** `scripts/healthcheck.sh` answers *is it up*.
Error tracking answers *is it working*, and the two disagree constantly: a
service returning 500 to every request is up. If this project has no error
tracking, say so here in one line rather than leaving the section blank — a gap
somebody chose is a different thing from a gap nobody noticed.
## What alerts, and to whom
| Signal | Where it goes | Who acts on it |
| --- | --- | --- |
| <> | <> | <> |
<Name a person or a rota, not a channel nobody owns.>
**An alert nobody acts on trains everybody to ignore the ones that matter.** If
a signal here has fired more than twice without anyone doing anything, it is
either wrong or it is not an alert — fix the threshold or move it to a
dashboard, and say which happened.
## Backups
| | |
| --- | --- |
| Schedule | <when, and by what cron, systemd timer, platform feature> |
| What is captured | <database, uploads, secrets and what is deliberately not> |
| Where it lands | <path or bucket, and who else can read it> |
| Retention | <how many, how old, what prunes them> |
| **Last verified restore** | **<YYYY-MM-DD>, by <who>** |
`scripts/backup.sh` proves a dump is readable before trusting it.
`scripts/restore-check.sh` proves it can be restored, which is a different
claim: a file that parses is not a database you can get back.
**A backup nobody has restored is a guess.** The date above is the only line in
this table that says otherwise, and it is a fact with an owner — if it is empty
or a year old, this project does not have backups, it has files.
## Rate limits and cost ceilings
*(precautionary — no incident here has yet come from this.)*
| Endpoint or job | What it costs per call | What bounds it |
| --- | --- | --- |
| <> | <> | <> |
<Anything that calls a paid API, sends mail, or runs a model belongs in this
table. So does authentication, which costs nothing per call and is the one an
attacker will hit hardest.>
**Where does the spend alert go, and at what number?** <> A ceiling nobody is
told about is a bill, not a limit.
## It is down — what now
In this order. Each step is a command that answers one question, and none of
them change anything.
1. **Is it actually down, or unreachable from here?**
`bash scripts/status.sh` — it distinguishes those two, and reports the
restart count. A climbing restart count with a short uptime is a crash loop,
not a healthy service, and reads as "running" in every tool that shows only a
state.
2. **What is it saying?** `bash scripts/status.sh --logs 200`.
3. **Is the deployed version the one you think?**
`bash scripts/status.sh --deployed-version`. Non-zero means it could not be
read — stop, rather than assuming.
4. **Did something change recently?** The tracker and `git log` — a deploy, a
migration, a config edit.
5. **Roll forward or back:** `python3 scripts/deploy.py` to move the stack to a
published image. Publishing and deploying are separate acts; this is the
second one.
<Add the steps this project actually needs a queue to drain, a cache to warm,
a partner to notify — and delete any of the above it does not have.>
**Write the incident down afterwards**, in `docs/history/DEVELOPMENT_LOG.md`
with the date, and file what broke as an issue with a severity label. An
incident nobody recorded happens again with the same surprise.

View File

@ -18,10 +18,10 @@
# block below, or pass it as SCAFFOLD_TEMPLATE_ROOT. It is empty and it stays # block below, or pass it as SCAFFOLD_TEMPLATE_ROOT. It is empty and it stays
# empty: the template sits at a different absolute path on every machine, and # empty: the template sits at a different absolute path on every machine, and
# a plausible-looking default would quietly resolve to whatever checkout # a plausible-looking default would quietly resolve to whatever checkout
# happens to be there. That is not a "file not found" — it is thirteen # happens to be there. That is not a "file not found" — it is a whole
# documents arriving with another project's Owner lines, another project's # documentation tree arriving with another project's Owner lines, another
# review dates and another project's branding, each one reading as this # project's review dates and another project's branding, each document reading
# project's own the moment it lands. # as this project's own the moment it lands.
# #
# Assumes: bash and coreutils. git is used only by --force, and only to read. # Assumes: bash and coreutils. git is used only by --force, and only to read.
# #
@ -87,7 +87,7 @@ set -uo pipefail
# CONFIGURATION — set this one, then delete this banner. # CONFIGURATION — set this one, then delete this banner.
# #
# Empty on purpose. See the note at the top: a default here does not fail, it # Empty on purpose. See the note at the top: a default here does not fail, it
# copies the wrong project's documents and says "created" thirteen times. # copies the wrong project's documents and says "created" for every one.
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Absolute path to the template checkout — the directory holding `docs/`. # Absolute path to the template checkout — the directory holding `docs/`.
@ -105,6 +105,7 @@ TEMPLATE_ROOT="${SCAFFOLD_TEMPLATE_ROOT:-}"
DOCS=( DOCS=(
docs/DOC_TRUST_MAP.md docs/DOC_TRUST_MAP.md
docs/WORK_CYCLE.md docs/WORK_CYCLE.md
docs/OPERATIONS.md
docs/TOOLS.md docs/TOOLS.md
docs/planning/PROJECT_PLAN.md docs/planning/PROJECT_PLAN.md
docs/qa/ClaudeQAPlan.md docs/qa/ClaudeQAPlan.md