307 lines
15 KiB
Markdown
307 lines
15 KiB
Markdown
# Operations — Queue North Website
|
||
|
||
```
|
||
Status: Current
|
||
Owner: _null
|
||
Last reviewed: 2026-08-18
|
||
Governs: Dockerfile, docker-compose.yml, scripts/status.sh,
|
||
scripts/healthcheck.sh, scripts/backup.sh, scripts/restore-check.sh —
|
||
what watches this in production, and what to do when it stops
|
||
Review trigger: A new alert or a new place errors are sent; a change to the
|
||
backup schedule or the restore procedure; any change to the
|
||
deploy host, container or image; any incident whose first ten
|
||
minutes were spent working out what to run
|
||
```
|
||
|
||
## Why this exists
|
||
|
||
Every other document here describes what the project *is*. This one describes
|
||
what happens when it stops being that at three in the morning, and it is the
|
||
only document written for somebody who is not thinking clearly.
|
||
|
||
The failure it prevents is specific: **the first ten minutes of an incident
|
||
spent working out which host, which container, and which command**, while the
|
||
thing everybody actually wants to know — is it down, or is it just unreachable
|
||
from here — goes unanswered.
|
||
|
||
## What is deployed, and where
|
||
|
||
| | |
|
||
| --- | --- |
|
||
| **Public origin** | **`https://queuenorth.com`** — the production front door, and the one customers use |
|
||
| Second front door | `https://qn.isnull.dev`, the same container reached through Cloudflare |
|
||
| `www` | 301s to `https://queuenorth.com/` |
|
||
| Host | **nebula** (`192.168.1.11`) — the same box as the Forgejo instance |
|
||
| Container | `qn-website-dev` |
|
||
| Image | `dream.scheller.ltd/null/queue-north-website:dev` |
|
||
| Port | 3001 in the container; the reverse proxy in front owns the public one |
|
||
| Database | `/app/db/queuenorth.db`, in the named volume `qn-website-dev_queuenorth-db` |
|
||
| Logs volume | `qn-website-dev_queuenorth-logs` → `/app/logs` |
|
||
| Runs as | UID 1001, non-root |
|
||
|
||
### Two front doors, one container
|
||
|
||
**Both hostnames are this application, and there is only one of it.** Verified
|
||
2026-08-18: both served byte-identical bundle filenames — index-pTFwovIx.js at
|
||
the time, named without backticks because it is a build artefact whose hash
|
||
changes with every build and never a path in this repository — and both answer
|
||
`/api/health` with this server's exact `{"status":"ok","db":"ok"}` shape.
|
||
|
||
```text
|
||
queuenorth.com ──► 24.41.108.95 (this network's own public IP)
|
||
└─► nginx-proxy-manager on thor/exodus (192.168.1.14)
|
||
└─┐
|
||
├──► qn-website-dev on nebula (192.168.1.11:3001)
|
||
qn.isnull.dev ──► Cloudflare ─┘
|
||
```
|
||
|
||
`www.queuenorth.com` 301s to the apex. Both origins carry HSTS and redirect
|
||
plain HTTP.
|
||
|
||
**This matters during an incident in two directions.** If only one hostname is
|
||
failing it is the ingress, not the app — compare the two before touching the
|
||
container. And if you restart that container you are restarting *production*,
|
||
not a dev alias, whatever the `-dev` in its name and its `:dev` image tag
|
||
suggest.
|
||
|
||
> **An earlier version of this document said the opposite** — that
|
||
> `queuenorth.com` "is not this deployment ... do not diagnose against it". That
|
||
> was written on 2026-08-18 from a DNS lookup and an assumption, and it was
|
||
> wrong. It is corrected here rather than quietly, because a runbook that sends
|
||
> somebody away from the failing host is worse than one that says nothing.
|
||
|
||
**There is no non-production environment.** These are two doors to the same
|
||
room. `npm run docker:test` runs the image locally, which is the closest thing
|
||
that exists.
|
||
|
||
## Where errors go
|
||
|
||
**Nowhere. This project has no error tracking.**
|
||
|
||
Said in one line rather than left blank, because a gap somebody chose is a
|
||
different thing from a gap nobody noticed. There is no Sentry project, no DSN and
|
||
no alert destination. An unhandled exception is logged to the container's stdout
|
||
and takes the process down: `server/index.js` handles `uncaughtException` and
|
||
`unhandledRejection` by logging and exiting 1, and Docker's `restart:
|
||
unless-stopped` brings it back. **That means a crash loop looks like a running
|
||
service to anything that only reads a state.** `scripts/status.sh` reports the
|
||
restart count for exactly this reason.
|
||
|
||
**A liveness tick is not error tracking.** `scripts/healthcheck.sh` answers *is
|
||
it up*. Nothing here answers *is it working* — a server returning 500 to every
|
||
form submission is up, healthy, and losing every lead.
|
||
|
||
## What alerts, and to whom
|
||
|
||
| Signal | Where it goes | Who acts on it |
|
||
| --- | --- | --- |
|
||
| Container `HEALTHCHECK` fails 3× at 30 s | Docker marks the container unhealthy on nebula | **Nobody, automatically.** It is visible to anyone who looks and notifies no one |
|
||
| `scripts/healthcheck.sh` | whoever runs it | not scheduled |
|
||
| Everything else | — | — |
|
||
|
||
**This table is nearly empty and that is the honest state.** Nothing on this
|
||
project pages anybody. The site could be down for a day and the first report
|
||
would come from the customer.
|
||
|
||
Two things would change that and neither is filed as work yet: scheduling
|
||
`healthcheck.sh` somewhere that can shout, and giving the process somewhere to
|
||
send an exception. Recorded here rather than invented into the tracker.
|
||
|
||
## Backups
|
||
|
||
| | |
|
||
| --- | --- |
|
||
| Schedule | **None yet.** `scripts/backup.sh` works and nothing runs it on a timer |
|
||
| What is captured | `/app/db/queuenorth.db` — the `leads` and `support_requests` tables. Deliberately not captured: the logs volume, and the environment, which holds the secrets |
|
||
| Where it lands | `$HOME/backups/queue-north-website` on the operator's machine, mode 700. **Still one machine** — see below |
|
||
| Retention | `BACKUP_KEEP`, default 7 |
|
||
| **Last verified restore** | **2026-08-18, by `_null` — 2 tables, 3 rows, restored in under 1 s** |
|
||
|
||
`scripts/backup.sh` proves a dump is readable before trusting it — it runs
|
||
better-sqlite3's online `.backup()` inside the container, copies the result out,
|
||
and refuses the run unless `PRAGMA integrity_check` returns `ok` and the table
|
||
count is plausible, only then renaming it into place. `scripts/restore-check.sh`
|
||
proves it can be restored, which is a different claim: a file that parses is not
|
||
a database you can get back.
|
||
|
||
**A backup nobody has restored is a guess.** The date above is the only line in
|
||
this table that says otherwise, and as of 2026-08-18 it says something: a
|
||
snapshot was taken from the running container, replayed into a scratch database
|
||
from SQL, and counted. It came back.
|
||
|
||
**Two things that are still true anyway**, and both are tracked in `Batch 15`:
|
||
|
||
1. **Nothing runs on a schedule.** One backup taken by hand is not a backup
|
||
regime. Until a timer exists, the newest dump is as old as the last time
|
||
somebody remembered.
|
||
2. **The dump lives on one machine, beside nothing.** `backup.sh` says this
|
||
itself on every run: *a backup that only exists beside the database it came
|
||
from does not survive the disk, the host or the account.* It is currently on
|
||
the operator's workstation and nowhere else.
|
||
|
||
The exposure is worth stating plainly: `/app/db/queuenorth.db` is the only live
|
||
copy of every lead and support request the site has ever taken, it lives in one
|
||
Docker volume on one host, and it took writes today.
|
||
|
||
**A number worth knowing before you need it:** the restore took under a second,
|
||
because the database is 28 KB and holds three leads. That will stay true for a
|
||
long time at this volume — this is a contact form, not a transactional system —
|
||
so during an incident, restoring is cheap and there is no reason to hesitate over
|
||
it.
|
||
|
||
`RESTORE_MIN_ROWS` is currently 0, which means the check cannot yet catch the
|
||
snapshot-of-an-empty-volume case. Set it to something below the real lead count
|
||
once that count is meaningful, and it starts catching the one failure a
|
||
structural check never can.
|
||
|
||
## Rate limits and cost ceilings
|
||
|
||
*(precautionary — no incident here has yet come from this.)*
|
||
|
||
| Endpoint or job | What it costs per call | What bounds it |
|
||
| --- | --- | --- |
|
||
| `POST /api/leads` | one SQLite insert, one reCAPTCHA verify, one Zoho form post | `RATE_LIMIT_PER_MINUTE`, default 5, per IP across all of `/api`. Body capped at 1 MB, request at 30 s |
|
||
| `POST /api/support` | the same, plus a Zoho Case when enabled | same limiter |
|
||
| Google reCAPTCHA | free at this volume | the same limiter, upstream of it |
|
||
| Zoho CRM | free at this volume; the API path has per-org daily credits | fire-and-forget, so exhausting them degrades CRM sync and never the site |
|
||
|
||
**Where does the spend alert go, and at what number?** Nowhere, and there is no
|
||
number. Nothing here bills per call at this volume, which is why that is
|
||
tolerable rather than an oversight — but if an LLM, a mail sender or a paid API
|
||
is ever added, this table is the first thing that has to change.
|
||
|
||
## It is down — what now
|
||
|
||
In this order. Each step is a command that answers one question, and none of them
|
||
change anything.
|
||
|
||
1. **Is it actually down, or unreachable from here?**
|
||
`bash scripts/healthcheck.sh` from anywhere, then
|
||
`bash scripts/status.sh`. The second distinguishes those two and reports the
|
||
restart count. **A climbing restart count with a short uptime is a crash loop**,
|
||
not a healthy service, and reads as "running" in every tool that shows only a
|
||
state.
|
||
2. **What is it saying?** `bash scripts/status.sh --logs 200`.
|
||
3. **Is the deployed version the one you think?**
|
||
|
||
```bash
|
||
bash scripts/status.sh --deployed-version
|
||
bash scripts/status.sh | grep digest
|
||
```
|
||
|
||
**The image running today carries no `org.opencontainers.image.version`
|
||
label**, so the first command correctly reports the version as unknown rather
|
||
than guessing, and the digest is the only way to tell one deploy from
|
||
another. The label was added to the `Dockerfile` on 2026-08-18 and takes
|
||
effect on the next build — until an image built after that date is deployed,
|
||
use the digest.
|
||
|
||
Non-zero from `--deployed-version` means it could not be read at all. Stop
|
||
there rather than assuming.
|
||
4. **Is it the database?** `/api/health` returns 503 with `db: error` when the
|
||
`SELECT 1` fails. That is a real answer and not an outage of the whole site:
|
||
the marketing pages keep serving while every form submission is failing.
|
||
5. **Did something change recently?** The tracker and `git log` — a deploy, a
|
||
config edit, a Cloudflare rule.
|
||
6. **Is it the ingress rather than the app?** The two front doors take
|
||
completely different paths — `queuenorth.com` through nginx-proxy-manager on
|
||
thor/exodus, `qn.isnull.dev` through Cloudflare. If one answers and the other
|
||
does not, the container is fine and the problem is in front of it:
|
||
|
||
```bash
|
||
curl -s https://queuenorth.com/api/health
|
||
curl -s https://qn.isnull.dev/api/health
|
||
```
|
||
|
||
Both failing while `scripts/status.sh` says healthy points at the LAN path or
|
||
the host itself.
|
||
|
||
### Releasing and deploying — two commands, deliberately
|
||
|
||
**Publishing an image and running it are separate decisions.** Nothing about a
|
||
published image affects the site until a deploy moves the stack to it.
|
||
|
||
```bash
|
||
npm run release # publish: patch bump, guards, build, push, tag
|
||
npm run deploy # deploy: move stack 58 to what :dev now points at
|
||
```
|
||
|
||
Both take `--dry-run`, and both refuse rather than guess. Run the dry runs first;
|
||
they print exactly what would change.
|
||
|
||
| | |
|
||
| --- | --- |
|
||
| Portainer | `https://192.168.1.11:9443` (nebula), API key in `~/.openclaw/credentials/portainer.md` |
|
||
| Stack | **id 58**, name `qn-website-dev`, endpoint 3, compose |
|
||
| Stack file on disk | `/data/compose/58/docker-compose.yml` on nebula |
|
||
|
||
**The stack file is NOT this repository's `docker-compose.yml`.** They are two
|
||
separate files that happen to look alike, and they have already drifted — the
|
||
repository commits `CORS_ORIGIN=https://queuenorth.com` and line 21 of the stack
|
||
file says `https://queuenorth.com/`, with a trailing slash. Editing the one in
|
||
git changes nothing about production. That is the single most important sentence
|
||
in this section, and `deploy.sh` does not paper over it: it reads the deployed
|
||
file, changes only the `image:` line, and sends the rest back untouched.
|
||
|
||
**What a deploy costs.** It recreates the container, so both public front doors
|
||
go down together for the few seconds it takes — they reach the same instance.
|
||
There is no non-production environment to rehearse against. `deploy.sh` takes a
|
||
verified backup before the irreversible part for that reason, and refuses to
|
||
continue if the backup fails.
|
||
|
||
### Production always runs a numbered version
|
||
|
||
**Policy, set 2026-08-18.** The stack pins `:vX.Y.Z`. Never `:dev`, never
|
||
`:latest`, never any other pointer.
|
||
|
||
The reason is what a pointer cannot do. A stack file naming `:dev` says `dev`
|
||
today and will still say `dev` after the image underneath it changes — so the
|
||
file records a preference rather than a fact, and "what is running?" has no
|
||
answer that survives the next push. Rollback is worse: there is nothing to roll
|
||
back *to*, because a pointer only ever names now.
|
||
|
||
`deploy.sh` enforces it. With no `--tag` it resolves the **newest published
|
||
numbered version** from the registry and says which it picked; given a floating
|
||
tag it refuses:
|
||
|
||
```bash
|
||
npm run deploy # newest published vX.Y.Z
|
||
npm run deploy -- --tag v0.9.4 --dry-run # a specific version, previewed
|
||
npm run deploy -- --tag v0.9.4 # deploy it
|
||
npm run deploy -- --tag v0.8.3 # roll back to a known image
|
||
```
|
||
|
||
`--allow-floating` exists for the one case that is not a mistake — reproducing a
|
||
fault on whatever a pointer currently is — and warns every time.
|
||
|
||
It also refuses a tag the registry does not hold. A missing tag would otherwise
|
||
become an outage: Portainer accepts the file, fails to pull, and leaves the
|
||
container stopped.
|
||
|
||
**Stack 58 pins `:dev` as this is written**, which is the state the policy
|
||
corrects. The first deploy made under it repoints the stack at a version, and
|
||
after that the stack file answers the question on its own.
|
||
|
||
`:dev` and `:latest` remain in the registry, frozen. They are not deleted
|
||
because the running container was created from `:dev`, and removing the tag an
|
||
existing deployment names is how a recreate fails to pull. Nothing publishes to
|
||
them any more — `release.sh` stopped moving `:dev` when this policy was set, and
|
||
scripts/docker-push.sh — named without backticks because it is gone — was
|
||
deleted, its only job having been to push an unnumbered `:dev` with no version
|
||
bump and no guards.
|
||
|
||
**What `deploy.sh` preserves, and why that is the dangerous part.** Portainer
|
||
treats a stack `PUT` as the whole desired state. Sending it without the stack's
|
||
`Env` array would strip twelve variables — including the reCAPTCHA secret and
|
||
the Zoho form tokens — and the container would come back **healthy** while
|
||
quietly failing to capture a single lead. The script reads them, counts them,
|
||
sends them back, and refuses outright if the stack reports none.
|
||
|
||
Afterwards it waits for the container's own healthcheck, then checks
|
||
`/api/health` on **both** origins, and prints the image digest and version label
|
||
from before and after.
|
||
|
||
**Write the incident down afterwards**, in `docs/history/DEVELOPMENT_LOG.md` with
|
||
the date, and file what broke as an issue with a severity label. An incident
|
||
nobody recorded happens again with the same surprise.
|