feat(docs): OPERATIONS.md, the runbook that did not exist
The template shipped scripts to back up, deploy, check health and read the deployed version, and no document saying where errors go, what alerts, who receives it, or what to run first when it is down. Grep across docs/** found zero mentions of error tracking, observability or database restore. Five sections. Where errors go, with the distinction that matters at 3am -- healthcheck.sh answers "is it up", error tracking answers "is it working", and a service returning 500 to everything is up. What alerts and to whom, naming a person rather than a channel nobody owns. Backups, whose last row is the date of the last verified restore, because a backup nobody has restored is a guess. Rate limits and cost ceilings, *(precautionary)*. And an ordered "it is down, what now" where every step is a command that changes nothing. Marked *(only for a deployed service)*, with the instruction to delete rather than keep the headings unanswered: an empty runbook reads as one nobody wrote, which is worse than one that never applied. scaffold.sh now lays it down (17 files, 0 skipped) and DOC_TRUST_MAP.md points at it from both tables. Two prose counts in scaffold.sh's header said "thirteen documents" and were already stale; they no longer carry a number, since a count in prose beside a list in code drifts the moment the list grows -- which is what just happened. closes #6 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
063b715c33
commit
2dca89e635
|
|
@ -33,6 +33,7 @@ disagree, and nothing will say which one is right.
|
||||||
| What is the next action? | the `nextAction` field on the project at privacyllc.dev — the newest entry in `docs/history/DEVELOPMENT_LOG.md` says what it was *then* |
|
| What is the next action? | the `nextAction` field on the project at privacyllc.dev — the newest entry in `docs/history/DEVELOPMENT_LOG.md` says what it was *then* |
|
||||||
| What is blocking us? | the tracker, for the work; the blockers table at privacyllc.dev, for the stakeholder-facing version |
|
| What is blocking us? | the tracker, for the work; the blockers table at privacyllc.dev, for the stakeholder-facing version |
|
||||||
| What do I do when a piece of work is finished? | `docs/WORK_CYCLE.md` |
|
| What do I do when a piece of work is finished? | `docs/WORK_CYCLE.md` |
|
||||||
|
| What watches this in production, and what do I do when it breaks? | `docs/OPERATIONS.md` |
|
||||||
| What happened, and when? | `docs/history/DEVELOPMENT_LOG.md` |
|
| What happened, and when? | `docs/history/DEVELOPMENT_LOG.md` |
|
||||||
| Did QA pass, and what does the tester think? | `docs/qa/ClaudeReport.md` |
|
| Did QA pass, and what does the tester think? | `docs/qa/ClaudeReport.md` |
|
||||||
| What did QA actually reach? | `docs/qa/ClaudeQACoverage.md` |
|
| What did QA actually reach? | `docs/qa/ClaudeQACoverage.md` |
|
||||||
|
|
@ -58,7 +59,7 @@ two only ever change because a person changed them.
|
||||||
| `docs/security/` | `SECURITY` — threat model, secret handling, what is out of scope |
|
| `docs/security/` | `SECURITY` — threat model, secret handling, what is out of scope |
|
||||||
| `docs/history/` | `DEVELOPMENT_LOG` (dated, append-only), `HISTORY` (the narrative), `BATCH_LEDGER` (archived, superseded by the tracker) |
|
| `docs/history/` | `DEVELOPMENT_LOG` (dated, append-only), `HISTORY` (the narrative), `BATCH_LEDGER` (archived, superseded by the tracker) |
|
||||||
| `docs/data/` | the three branding marks privacyllc.dev renders for this project |
|
| `docs/data/` | the three branding marks privacyllc.dev renders for this project |
|
||||||
| `docs/` root | this map; `WORK_CYCLE` — what to do when a piece of work is finished; `TOOLS` — where the scripts are and which can stop you |
|
| `docs/` root | this map; `WORK_CYCLE` — what to do when a piece of work is finished; `TOOLS` — where the scripts are and which can stop you; `OPERATIONS` — what watches production, *(only for a deployed service)* |
|
||||||
|
|
||||||
`README.md` stays at the repository root; it is the landing page and moving it
|
`README.md` stays at the repository root; it is the landing page and moving it
|
||||||
breaks that. Everything else lives under `docs/`.
|
breaks that. Everything else lives under `docs/`.
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,112 @@
|
||||||
|
# Operations — <Project>
|
||||||
|
|
||||||
|
```
|
||||||
|
Status: Current
|
||||||
|
Owner: <who maintains this>
|
||||||
|
Last reviewed: <YYYY-MM-DD>
|
||||||
|
Governs: what watches this in production, and what to do when it stops
|
||||||
|
Review trigger: A new alert or a new place errors are sent; a change to the
|
||||||
|
backup schedule or the restore procedure; any incident whose first ten minutes
|
||||||
|
were spent working out what to run
|
||||||
|
```
|
||||||
|
|
||||||
|
> *(only for a deployed service.)* A library, a local tool or a static site with
|
||||||
|
> no backend deletes this file. Do not keep it with the headings unanswered — an
|
||||||
|
> empty runbook reads as one nobody wrote, which is worse than one that never
|
||||||
|
> applied.
|
||||||
|
|
||||||
|
## Why this exists
|
||||||
|
|
||||||
|
Every other document here describes what the project *is*. This one describes
|
||||||
|
what happens when it stops being that at three in the morning, and it is the
|
||||||
|
only document written for somebody who is not thinking clearly.
|
||||||
|
|
||||||
|
The failure it prevents is specific: **the first ten minutes of an incident
|
||||||
|
spent working out which host, which container, and which command**, while the
|
||||||
|
thing everybody actually wants to know — is it down, or is it just unreachable
|
||||||
|
from here — goes unanswered.
|
||||||
|
|
||||||
|
## Where errors go
|
||||||
|
|
||||||
|
<The service, the project or DSN, and who can see it. Name it, do not describe
|
||||||
|
it: "Sentry, project <name>, alerts to <where>" is useful at 3am and "we have
|
||||||
|
error tracking" is not.>
|
||||||
|
|
||||||
|
**A liveness tick is not this.** `scripts/healthcheck.sh` answers *is it up*.
|
||||||
|
Error tracking answers *is it working*, and the two disagree constantly: a
|
||||||
|
service returning 500 to every request is up. If this project has no error
|
||||||
|
tracking, say so here in one line rather than leaving the section blank — a gap
|
||||||
|
somebody chose is a different thing from a gap nobody noticed.
|
||||||
|
|
||||||
|
## What alerts, and to whom
|
||||||
|
|
||||||
|
| Signal | Where it goes | Who acts on it |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| <> | <> | <> |
|
||||||
|
|
||||||
|
<Name a person or a rota, not a channel nobody owns.>
|
||||||
|
|
||||||
|
**An alert nobody acts on trains everybody to ignore the ones that matter.** If
|
||||||
|
a signal here has fired more than twice without anyone doing anything, it is
|
||||||
|
either wrong or it is not an alert — fix the threshold or move it to a
|
||||||
|
dashboard, and say which happened.
|
||||||
|
|
||||||
|
## Backups
|
||||||
|
|
||||||
|
| | |
|
||||||
|
| --- | --- |
|
||||||
|
| Schedule | <when, and by what — cron, systemd timer, platform feature> |
|
||||||
|
| What is captured | <database, uploads, secrets — and what is deliberately not> |
|
||||||
|
| Where it lands | <path or bucket, and who else can read it> |
|
||||||
|
| Retention | <how many, how old, what prunes them> |
|
||||||
|
| **Last verified restore** | **<YYYY-MM-DD>, by <who>** |
|
||||||
|
|
||||||
|
`scripts/backup.sh` proves a dump is readable before trusting it.
|
||||||
|
`scripts/restore-check.sh` proves it can be restored, which is a different
|
||||||
|
claim: a file that parses is not a database you can get back.
|
||||||
|
|
||||||
|
**A backup nobody has restored is a guess.** The date above is the only line in
|
||||||
|
this table that says otherwise, and it is a fact with an owner — if it is empty
|
||||||
|
or a year old, this project does not have backups, it has files.
|
||||||
|
|
||||||
|
## Rate limits and cost ceilings
|
||||||
|
|
||||||
|
*(precautionary — no incident here has yet come from this.)*
|
||||||
|
|
||||||
|
| Endpoint or job | What it costs per call | What bounds it |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| <> | <> | <> |
|
||||||
|
|
||||||
|
<Anything that calls a paid API, sends mail, or runs a model belongs in this
|
||||||
|
table. So does authentication, which costs nothing per call and is the one an
|
||||||
|
attacker will hit hardest.>
|
||||||
|
|
||||||
|
**Where does the spend alert go, and at what number?** <> A ceiling nobody is
|
||||||
|
told about is a bill, not a limit.
|
||||||
|
|
||||||
|
## It is down — what now
|
||||||
|
|
||||||
|
In this order. Each step is a command that answers one question, and none of
|
||||||
|
them change anything.
|
||||||
|
|
||||||
|
1. **Is it actually down, or unreachable from here?**
|
||||||
|
`bash scripts/status.sh` — it distinguishes those two, and reports the
|
||||||
|
restart count. A climbing restart count with a short uptime is a crash loop,
|
||||||
|
not a healthy service, and reads as "running" in every tool that shows only a
|
||||||
|
state.
|
||||||
|
2. **What is it saying?** `bash scripts/status.sh --logs 200`.
|
||||||
|
3. **Is the deployed version the one you think?**
|
||||||
|
`bash scripts/status.sh --deployed-version`. Non-zero means it could not be
|
||||||
|
read — stop, rather than assuming.
|
||||||
|
4. **Did something change recently?** The tracker and `git log` — a deploy, a
|
||||||
|
migration, a config edit.
|
||||||
|
5. **Roll forward or back:** `python3 scripts/deploy.py` to move the stack to a
|
||||||
|
published image. Publishing and deploying are separate acts; this is the
|
||||||
|
second one.
|
||||||
|
|
||||||
|
<Add the steps this project actually needs — a queue to drain, a cache to warm,
|
||||||
|
a partner to notify — and delete any of the above it does not have.>
|
||||||
|
|
||||||
|
**Write the incident down afterwards**, in `docs/history/DEVELOPMENT_LOG.md`
|
||||||
|
with the date, and file what broke as an issue with a severity label. An
|
||||||
|
incident nobody recorded happens again with the same surprise.
|
||||||
|
|
@ -18,10 +18,10 @@
|
||||||
# block below, or pass it as SCAFFOLD_TEMPLATE_ROOT. It is empty and it stays
|
# block below, or pass it as SCAFFOLD_TEMPLATE_ROOT. It is empty and it stays
|
||||||
# empty: the template sits at a different absolute path on every machine, and
|
# empty: the template sits at a different absolute path on every machine, and
|
||||||
# a plausible-looking default would quietly resolve to whatever checkout
|
# a plausible-looking default would quietly resolve to whatever checkout
|
||||||
# happens to be there. That is not a "file not found" — it is thirteen
|
# happens to be there. That is not a "file not found" — it is a whole
|
||||||
# documents arriving with another project's Owner lines, another project's
|
# documentation tree arriving with another project's Owner lines, another
|
||||||
# review dates and another project's branding, each one reading as this
|
# project's review dates and another project's branding, each document reading
|
||||||
# project's own the moment it lands.
|
# as this project's own the moment it lands.
|
||||||
#
|
#
|
||||||
# Assumes: bash and coreutils. git is used only by --force, and only to read.
|
# Assumes: bash and coreutils. git is used only by --force, and only to read.
|
||||||
#
|
#
|
||||||
|
|
@ -87,7 +87,7 @@ set -uo pipefail
|
||||||
# CONFIGURATION — set this one, then delete this banner.
|
# CONFIGURATION — set this one, then delete this banner.
|
||||||
#
|
#
|
||||||
# Empty on purpose. See the note at the top: a default here does not fail, it
|
# Empty on purpose. See the note at the top: a default here does not fail, it
|
||||||
# copies the wrong project's documents and says "created" thirteen times.
|
# copies the wrong project's documents and says "created" for every one.
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
# Absolute path to the template checkout — the directory holding `docs/`.
|
# Absolute path to the template checkout — the directory holding `docs/`.
|
||||||
|
|
@ -105,6 +105,7 @@ TEMPLATE_ROOT="${SCAFFOLD_TEMPLATE_ROOT:-}"
|
||||||
DOCS=(
|
DOCS=(
|
||||||
docs/DOC_TRUST_MAP.md
|
docs/DOC_TRUST_MAP.md
|
||||||
docs/WORK_CYCLE.md
|
docs/WORK_CYCLE.md
|
||||||
|
docs/OPERATIONS.md
|
||||||
docs/TOOLS.md
|
docs/TOOLS.md
|
||||||
docs/planning/PROJECT_PLAN.md
|
docs/planning/PROJECT_PLAN.md
|
||||||
docs/qa/ClaudeQAPlan.md
|
docs/qa/ClaudeQAPlan.md
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue