diff --git a/docs/DOC_TRUST_MAP.md b/docs/DOC_TRUST_MAP.md index bee1c16..4bcf952 100644 --- a/docs/DOC_TRUST_MAP.md +++ b/docs/DOC_TRUST_MAP.md @@ -33,6 +33,7 @@ disagree, and nothing will say which one is right. | What is the next action? | the `nextAction` field on the project at privacyllc.dev — the newest entry in `docs/history/DEVELOPMENT_LOG.md` says what it was *then* | | What is blocking us? | the tracker, for the work; the blockers table at privacyllc.dev, for the stakeholder-facing version | | What do I do when a piece of work is finished? | `docs/WORK_CYCLE.md` | +| What watches this in production, and what do I do when it breaks? | `docs/OPERATIONS.md` | | What happened, and when? | `docs/history/DEVELOPMENT_LOG.md` | | Did QA pass, and what does the tester think? | `docs/qa/ClaudeReport.md` | | What did QA actually reach? | `docs/qa/ClaudeQACoverage.md` | @@ -58,7 +59,7 @@ two only ever change because a person changed them. | `docs/security/` | `SECURITY` — threat model, secret handling, what is out of scope | | `docs/history/` | `DEVELOPMENT_LOG` (dated, append-only), `HISTORY` (the narrative), `BATCH_LEDGER` (archived, superseded by the tracker) | | `docs/data/` | the three branding marks privacyllc.dev renders for this project | -| `docs/` root | this map; `WORK_CYCLE` — what to do when a piece of work is finished; `TOOLS` — where the scripts are and which can stop you | +| `docs/` root | this map; `WORK_CYCLE` — what to do when a piece of work is finished; `TOOLS` — where the scripts are and which can stop you; `OPERATIONS` — what watches production, *(only for a deployed service)* | `README.md` stays at the repository root; it is the landing page and moving it breaks that. Everything else lives under `docs/`. diff --git a/docs/OPERATIONS.md b/docs/OPERATIONS.md new file mode 100644 index 0000000..fd5c127 --- /dev/null +++ b/docs/OPERATIONS.md @@ -0,0 +1,112 @@ +# Operations — + +``` +Status: Current +Owner: +Last reviewed: +Governs: what watches this in production, and what to do when it stops +Review trigger: A new alert or a new place errors are sent; a change to the + backup schedule or the restore procedure; any incident whose first ten minutes + were spent working out what to run +``` + +> *(only for a deployed service.)* A library, a local tool or a static site with +> no backend deletes this file. Do not keep it with the headings unanswered — an +> empty runbook reads as one nobody wrote, which is worse than one that never +> applied. + +## Why this exists + +Every other document here describes what the project *is*. This one describes +what happens when it stops being that at three in the morning, and it is the +only document written for somebody who is not thinking clearly. + +The failure it prevents is specific: **the first ten minutes of an incident +spent working out which host, which container, and which command**, while the +thing everybody actually wants to know — is it down, or is it just unreachable +from here — goes unanswered. + +## Where errors go + +, alerts to " is useful at 3am and "we have +error tracking" is not.> + +**A liveness tick is not this.** `scripts/healthcheck.sh` answers *is it up*. +Error tracking answers *is it working*, and the two disagree constantly: a +service returning 500 to every request is up. If this project has no error +tracking, say so here in one line rather than leaving the section blank — a gap +somebody chose is a different thing from a gap nobody noticed. + +## What alerts, and to whom + +| Signal | Where it goes | Who acts on it | +| --- | --- | --- | +| <> | <> | <> | + + + +**An alert nobody acts on trains everybody to ignore the ones that matter.** If +a signal here has fired more than twice without anyone doing anything, it is +either wrong or it is not an alert — fix the threshold or move it to a +dashboard, and say which happened. + +## Backups + +| | | +| --- | --- | +| Schedule | | +| What is captured | | +| Where it lands | | +| Retention | | +| **Last verified restore** | **, by ** | + +`scripts/backup.sh` proves a dump is readable before trusting it. +`scripts/restore-check.sh` proves it can be restored, which is a different +claim: a file that parses is not a database you can get back. + +**A backup nobody has restored is a guess.** The date above is the only line in +this table that says otherwise, and it is a fact with an owner — if it is empty +or a year old, this project does not have backups, it has files. + +## Rate limits and cost ceilings + +*(precautionary — no incident here has yet come from this.)* + +| Endpoint or job | What it costs per call | What bounds it | +| --- | --- | --- | +| <> | <> | <> | + + + +**Where does the spend alert go, and at what number?** <> A ceiling nobody is +told about is a bill, not a limit. + +## It is down — what now + +In this order. Each step is a command that answers one question, and none of +them change anything. + +1. **Is it actually down, or unreachable from here?** + `bash scripts/status.sh` — it distinguishes those two, and reports the + restart count. A climbing restart count with a short uptime is a crash loop, + not a healthy service, and reads as "running" in every tool that shows only a + state. +2. **What is it saying?** `bash scripts/status.sh --logs 200`. +3. **Is the deployed version the one you think?** + `bash scripts/status.sh --deployed-version`. Non-zero means it could not be + read — stop, rather than assuming. +4. **Did something change recently?** The tracker and `git log` — a deploy, a + migration, a config edit. +5. **Roll forward or back:** `python3 scripts/deploy.py` to move the stack to a + published image. Publishing and deploying are separate acts; this is the + second one. + + + +**Write the incident down afterwards**, in `docs/history/DEVELOPMENT_LOG.md` +with the date, and file what broke as an issue with a severity label. An +incident nobody recorded happens again with the same surprise. diff --git a/docs/architecture/scripts/scaffold.sh b/docs/architecture/scripts/scaffold.sh index 9335dd1..a36c999 100755 --- a/docs/architecture/scripts/scaffold.sh +++ b/docs/architecture/scripts/scaffold.sh @@ -18,10 +18,10 @@ # block below, or pass it as SCAFFOLD_TEMPLATE_ROOT. It is empty and it stays # empty: the template sits at a different absolute path on every machine, and # a plausible-looking default would quietly resolve to whatever checkout -# happens to be there. That is not a "file not found" — it is thirteen -# documents arriving with another project's Owner lines, another project's -# review dates and another project's branding, each one reading as this -# project's own the moment it lands. +# happens to be there. That is not a "file not found" — it is a whole +# documentation tree arriving with another project's Owner lines, another +# project's review dates and another project's branding, each document reading +# as this project's own the moment it lands. # # Assumes: bash and coreutils. git is used only by --force, and only to read. # @@ -87,7 +87,7 @@ set -uo pipefail # CONFIGURATION — set this one, then delete this banner. # # Empty on purpose. See the note at the top: a default here does not fail, it -# copies the wrong project's documents and says "created" thirteen times. +# copies the wrong project's documents and says "created" for every one. # --------------------------------------------------------------------------- # Absolute path to the template checkout — the directory holding `docs/`. @@ -105,6 +105,7 @@ TEMPLATE_ROOT="${SCAFFOLD_TEMPLATE_ROOT:-}" DOCS=( docs/DOC_TRUST_MAP.md docs/WORK_CYCLE.md + docs/OPERATIONS.md docs/TOOLS.md docs/planning/PROJECT_PLAN.md docs/qa/ClaudeQAPlan.md