Queue-North-Website/docs/OPERATIONS.md

# Operations — Queue North Website

```
Status: Current
Owner: _null
Last reviewed: 2026-08-18
Governs: Dockerfile, docker-compose.yml, scripts/status.sh,
         scripts/healthcheck.sh, scripts/backup.sh, scripts/restore-check.sh —
         what watches this in production, and what to do when it stops
Review trigger: A new alert or a new place errors are sent; a change to the
                backup schedule or the restore procedure; any change to the
                deploy host, container or image; any incident whose first ten
                minutes were spent working out what to run
```

## Why this exists

Every other document here describes what the project *is*. This one describes
what happens when it stops being that at three in the morning, and it is the
only document written for somebody who is not thinking clearly.

The failure it prevents is specific: **the first ten minutes of an incident
spent working out which host, which container, and which command**, while the
thing everybody actually wants to know — is it down, or is it just unreachable
from here — goes unanswered.

## What is deployed, and where

| | |
| --- | --- |
| **Public origin** | **`https://queuenorth.com`** — the production front door, and the one customers use |
| Second front door | `https://qn.isnull.dev`, the same container reached through Cloudflare |
| `www` | 301s to `https://queuenorth.com/` |
| Host | **nebula** (`192.168.1.11`) — the same box as the Forgejo instance |
| Container | `qn-website-dev` |
| Image | `dream.scheller.ltd/null/queue-north-website:dev` |
| Port | 3001 in the container; the reverse proxy in front owns the public one |
| Database | `/app/db/queuenorth.db`, in the named volume `qn-website-dev_queuenorth-db` |
| Logs volume | `qn-website-dev_queuenorth-logs` → `/app/logs` |
| Runs as | UID 1001, non-root |

### Two front doors, one container

**Both hostnames are this application, and there is only one of it.** Verified
2026-08-18: both served byte-identical bundle filenames — index-pTFwovIx.js at
the time, named without backticks because it is a build artefact whose hash
changes with every build and never a path in this repository — and both answer
`/api/health` with this server's exact `{"status":"ok","db":"ok"}` shape.

```text
queuenorth.com ──► 24.41.108.95 (this network's own public IP)
                     └─► nginx-proxy-manager on thor/exodus (192.168.1.14)
                           └─┐
                             ├──► qn-website-dev on nebula (192.168.1.11:3001)
qn.isnull.dev ──► Cloudflare ─┘
```

`www.queuenorth.com` 301s to the apex. Both origins carry HSTS and redirect
plain HTTP.

**This matters during an incident in two directions.** If only one hostname is
failing it is the ingress, not the app — compare the two before touching the
container. And if you restart that container you are restarting *production*,
not a dev alias, whatever the `-dev` in its name and its `:dev` image tag
suggest.

> **An earlier version of this document said the opposite** — that
> `queuenorth.com` "is not this deployment ... do not diagnose against it". That
> was written on 2026-08-18 from a DNS lookup and an assumption, and it was
> wrong. It is corrected here rather than quietly, because a runbook that sends
> somebody away from the failing host is worse than one that says nothing.

**There is no non-production environment.** These are two doors to the same
room. `npm run docker:test` runs the image locally, which is the closest thing
that exists.

## Where errors go

**Nowhere. This project has no error tracking.**

Said in one line rather than left blank, because a gap somebody chose is a
different thing from a gap nobody noticed. There is no Sentry project, no DSN and
no alert destination. An unhandled exception is logged to the container's stdout
and takes the process down: `server/index.js` handles `uncaughtException` and
`unhandledRejection` by logging and exiting 1, and Docker's `restart:
unless-stopped` brings it back. **That means a crash loop looks like a running
service to anything that only reads a state.** `scripts/status.sh` reports the
restart count for exactly this reason.

**A liveness tick is not error tracking.** `scripts/healthcheck.sh` answers *is
it up*. Nothing here answers *is it working* — a server returning 500 to every
form submission is up, healthy, and losing every lead.

## What alerts, and to whom

| Signal | Where it goes | Who acts on it |
| --- | --- | --- |
| Container `HEALTHCHECK` fails 3× at 30 s | Docker marks the container unhealthy on nebula | **Nobody, automatically.** It is visible to anyone who looks and notifies no one |
| `scripts/healthcheck.sh` | whoever runs it | not scheduled |
| Everything else | — | — |

**This table is nearly empty and that is the honest state.** Nothing on this
project pages anybody. The site could be down for a day and the first report
would come from the customer.

Two things would change that and neither is filed as work yet: scheduling
`healthcheck.sh` somewhere that can shout, and giving the process somewhere to
send an exception. Recorded here rather than invented into the tracker.

## Backups

| | |
| --- | --- |
| Schedule | **None yet.** `scripts/backup.sh` works and nothing runs it on a timer |
| What is captured | `/app/db/queuenorth.db` — the `leads` and `support_requests` tables. Deliberately not captured: the logs volume, and the environment, which holds the secrets |
| Where it lands | `$HOME/backups/queue-north-website` on the operator's machine, mode 700. **Still one machine** — see below |
| Retention | `BACKUP_KEEP`, default 7 |
| **Last verified restore** | **2026-08-18, by `_null` — 2 tables, 3 rows, restored in under 1 s** |

`scripts/backup.sh` proves a dump is readable before trusting it — it runs
better-sqlite3's online `.backup()` inside the container, copies the result out,
and refuses the run unless `PRAGMA integrity_check` returns `ok` and the table
count is plausible, only then renaming it into place. `scripts/restore-check.sh`
proves it can be restored, which is a different claim: a file that parses is not
a database you can get back.

**A backup nobody has restored is a guess.** The date above is the only line in
this table that says otherwise, and as of 2026-08-18 it says something: a
snapshot was taken from the running container, replayed into a scratch database
from SQL, and counted. It came back.

**Two things that are still true anyway**, and both are tracked in `Batch 15`:

1. **Nothing runs on a schedule.** One backup taken by hand is not a backup
   regime. Until a timer exists, the newest dump is as old as the last time
   somebody remembered.
2. **The dump lives on one machine, beside nothing.** `backup.sh` says this
   itself on every run: *a backup that only exists beside the database it came
   from does not survive the disk, the host or the account.* It is currently on
   the operator's workstation and nowhere else.

The exposure is worth stating plainly: `/app/db/queuenorth.db` is the only live
copy of every lead and support request the site has ever taken, it lives in one
Docker volume on one host, and it took writes today.

**A number worth knowing before you need it:** the restore took under a second,
because the database is 28 KB and holds three leads. That will stay true for a
long time at this volume — this is a contact form, not a transactional system —
so during an incident, restoring is cheap and there is no reason to hesitate over
it.

`RESTORE_MIN_ROWS` is currently 0, which means the check cannot yet catch the
snapshot-of-an-empty-volume case. Set it to something below the real lead count
once that count is meaningful, and it starts catching the one failure a
structural check never can.

## Rate limits and cost ceilings

*(precautionary — no incident here has yet come from this.)*

| Endpoint or job | What it costs per call | What bounds it |
| --- | --- | --- |
| `POST /api/leads` | one SQLite insert, one reCAPTCHA verify, one Zoho form post | `RATE_LIMIT_PER_MINUTE`, default 5, per IP across all of `/api`. Body capped at 1 MB, request at 30 s |
| `POST /api/support` | the same, plus a Zoho Case when enabled | same limiter |
| Google reCAPTCHA | free at this volume | the same limiter, upstream of it |
| Zoho CRM | free at this volume; the API path has per-org daily credits | fire-and-forget, so exhausting them degrades CRM sync and never the site |

**Where does the spend alert go, and at what number?** Nowhere, and there is no
number. Nothing here bills per call at this volume, which is why that is
tolerable rather than an oversight — but if an LLM, a mail sender or a paid API
is ever added, this table is the first thing that has to change.

## It is down — what now

In this order. Each step is a command that answers one question, and none of them
change anything.

1. **Is it actually down, or unreachable from here?**
   `bash scripts/healthcheck.sh` from anywhere, then
   `bash scripts/status.sh`. The second distinguishes those two and reports the
   restart count. **A climbing restart count with a short uptime is a crash loop**,
   not a healthy service, and reads as "running" in every tool that shows only a
   state.
2. **What is it saying?** `bash scripts/status.sh --logs 200`.
3. **Is the deployed version the one you think?**

   ```bash
   bash scripts/status.sh --deployed-version
   bash scripts/status.sh | grep digest
   ```

   **The image running today carries no `org.opencontainers.image.version`
   label**, so the first command correctly reports the version as unknown rather
   than guessing, and the digest is the only way to tell one deploy from
   another. The label was added to the `Dockerfile` on 2026-08-18 and takes
   effect on the next build — until an image built after that date is deployed,
   use the digest.

   Non-zero from `--deployed-version` means it could not be read at all. Stop
   there rather than assuming.
4. **Is it the database?** `/api/health` returns 503 with `db: error` when the
   `SELECT 1` fails. That is a real answer and not an outage of the whole site:
   the marketing pages keep serving while every form submission is failing.
5. **Did something change recently?** The tracker and `git log` — a deploy, a
   config edit, a Cloudflare rule.
6. **Is it the ingress rather than the app?** The two front doors take
   completely different paths — `queuenorth.com` through nginx-proxy-manager on
   thor/exodus, `qn.isnull.dev` through Cloudflare. If one answers and the other
   does not, the container is fine and the problem is in front of it:

   ```bash
   curl -s https://queuenorth.com/api/health
   curl -s https://qn.isnull.dev/api/health
   ```

   Both failing while `scripts/status.sh` says healthy points at the LAN path or
   the host itself.

### Releasing and deploying — two commands, deliberately

**Publishing an image and running it are separate decisions.** Nothing about a
published image affects the site until a deploy moves the stack to it.

```bash
npm run release              # publish:  patch bump, guards, build, push, tag
npm run deploy               # deploy:   move stack 58 to what :dev now points at
```

Both take `--dry-run`, and both refuse rather than guess. Run the dry runs first;
they print exactly what would change.

| | |
| --- | --- |
| Portainer | `https://192.168.1.11:9443` (nebula), API key in `~/.openclaw/credentials/portainer.md` |
| Stack | **id 58**, name `qn-website-dev`, endpoint 3, compose |
| Stack file on disk | `/data/compose/58/docker-compose.yml` on nebula |

**The stack file is NOT this repository's `docker-compose.yml`.** They are two
separate files that happen to look alike, and they have already drifted — the
repository commits `CORS_ORIGIN=https://queuenorth.com` and line 21 of the stack
file says `https://queuenorth.com/`, with a trailing slash. Editing the one in
git changes nothing about production. That is the single most important sentence
in this section, and `deploy.sh` does not paper over it: it reads the deployed
file, changes only the `image:` line, and sends the rest back untouched.

**What a deploy costs.** It recreates the container, so both public front doors
go down together for the few seconds it takes — they reach the same instance.
There is no non-production environment to rehearse against. `deploy.sh` takes a
verified backup before the irreversible part for that reason, and refuses to
continue if the backup fails.

### Production always runs a numbered version

**Policy, set 2026-08-18.** The stack pins `:vX.Y.Z`. Never `:dev`, never
`:latest`, never any other pointer.

The reason is what a pointer cannot do. A stack file naming `:dev` says `dev`
today and will still say `dev` after the image underneath it changes — so the
file records a preference rather than a fact, and "what is running?" has no
answer that survives the next push. Rollback is worse: there is nothing to roll
back *to*, because a pointer only ever names now.

`deploy.sh` enforces it. With no `--tag` it resolves the **newest published
numbered version** from the registry and says which it picked; given a floating
tag it refuses:

```bash
npm run deploy                                  # newest published vX.Y.Z
npm run deploy -- --tag v0.9.4 --dry-run        # a specific version, previewed
npm run deploy -- --tag v0.9.4                  # deploy it
npm run deploy -- --tag v0.8.3                  # roll back to a known image
```

`--allow-floating` exists for the one case that is not a mistake — reproducing a
fault on whatever a pointer currently is — and warns every time.

It also refuses a tag the registry does not hold. A missing tag would otherwise
become an outage: Portainer accepts the file, fails to pull, and leaves the
container stopped.

**Stack 58 pins `:dev` as this is written**, which is the state the policy
corrects. The first deploy made under it repoints the stack at a version, and
after that the stack file answers the question on its own.

`:dev` and `:latest` remain in the registry, frozen. They are not deleted
because the running container was created from `:dev`, and removing the tag an
existing deployment names is how a recreate fails to pull. Nothing publishes to
them any more — `release.sh` stopped moving `:dev` when this policy was set, and
scripts/docker-push.sh — named without backticks because it is gone — was
deleted, its only job having been to push an unnumbered `:dev` with no version
bump and no guards.

**What `deploy.sh` preserves, and why that is the dangerous part.** Portainer
treats a stack `PUT` as the whole desired state. Sending it without the stack's
`Env` array would strip twelve variables — including the reCAPTCHA secret and
the Zoho form tokens — and the container would come back **healthy** while
quietly failing to capture a single lead. The script reads them, counts them,
sends them back, and refuses outright if the stack reports none.

Afterwards it waits for the container's own healthcheck, then checks
`/api/health` on **both** origins, and prints the image digest and version label
from before and after.

**Write the incident down afterwards**, in `docs/history/DEVELOPMENT_LOG.md` with
the date, and file what broke as an issue with a severity label. An incident
nobody recorded happens again with the same surprise.