null/Queue-North-Website
null
/
Queue-North-Website
1
0
Fork 0
Code Issues 20 Pull Requests Packages Projects Releases Wiki Activity
Queue-North-Website/docs/OPERATIONS.md

13 KiB
Raw Blame History

Operations — Queue North Website

Status: Current
Owner: _null
Last reviewed: 2026-08-18
Governs: Dockerfile, docker-compose.yml, scripts/status.sh,
         scripts/healthcheck.sh, scripts/backup.sh, scripts/restore-check.sh —
         what watches this in production, and what to do when it stops
Review trigger: A new alert or a new place errors are sent; a change to the
                backup schedule or the restore procedure; any change to the
                deploy host, container or image; any incident whose first ten
                minutes were spent working out what to run

Why this exists

Every other document here describes what the project is. This one describes what happens when it stops being that at three in the morning, and it is the only document written for somebody who is not thinking clearly.

The failure it prevents is specific: the first ten minutes of an incident spent working out which host, which container, and which command, while the thing everybody actually wants to know — is it down, or is it just unreachable from here — goes unanswered.

What is deployed, and where
Public origin https://queuenorth.com — the production front door, and the one customers use
Second front door https://qn.isnull.dev, the same container reached through Cloudflare
www 301s to https://queuenorth.com/
Host nebula (192.168.1.11) — the same box as the Forgejo instance
Container qn-website-dev
Image dream.scheller.ltd/null/queue-north-website:dev
Port 3001 in the container; the reverse proxy in front owns the public one
Database /app/db/queuenorth.db, in the named volume qn-website-dev_queuenorth-db
Logs volume qn-website-dev_queuenorth-logs → /app/logs
Runs as UID 1001, non-root

Two front doors, one container

Both hostnames are this application, and there is only one of it. Verified 2026-08-18: both serve the identical bundle (assets/index-pTFwovIx.js) and both answer /api/health with this server's exact {"status":"ok","db":"ok"} shape.

queuenorth.com ──► 24.41.108.95 (this network's own public IP)
                     └─► nginx-proxy-manager on thor/exodus (192.168.1.14)
                           └─┐
                             ├──► qn-website-dev on nebula (192.168.1.11:3001)
qn.isnull.dev ──► Cloudflare ─┘

www.queuenorth.com 301s to the apex. Both origins carry HSTS and redirect plain HTTP.

This matters during an incident in two directions. If only one hostname is failing it is the ingress, not the app — compare the two before touching the container. And if you restart that container you are restarting production, not a dev alias, whatever the -dev in its name and its :dev image tag suggest.

An earlier version of this document said the opposite — that queuenorth.com "is not this deployment ... do not diagnose against it". That was written on 2026-08-18 from a DNS lookup and an assumption, and it was wrong. It is corrected here rather than quietly, because a runbook that sends somebody away from the failing host is worse than one that says nothing.

There is no non-production environment. These are two doors to the same room. npm run docker:test runs the image locally, which is the closest thing that exists.

Where errors go

Nowhere. This project has no error tracking.

Said in one line rather than left blank, because a gap somebody chose is a different thing from a gap nobody noticed. There is no Sentry project, no DSN and no alert destination. An unhandled exception is logged to the container's stdout and takes the process down: server/index.js handles uncaughtException and unhandledRejection by logging and exiting 1, and Docker's restart: unless-stopped brings it back. That means a crash loop looks like a running service to anything that only reads a state. scripts/status.sh reports the restart count for exactly this reason.

A liveness tick is not error tracking. scripts/healthcheck.sh answers is it up. Nothing here answers is it working — a server returning 500 to every form submission is up, healthy, and losing every lead.

What alerts, and to whom

Signal Where it goes Who acts on it
Container HEALTHCHECK fails 3× at 30 s Docker marks the container unhealthy on nebula Nobody, automatically. It is visible to anyone who looks and notifies no one
scripts/healthcheck.sh whoever runs it not scheduled
Everything else — —

This table is nearly empty and that is the honest state. Nothing on this project pages anybody. The site could be down for a day and the first report would come from the customer.

Two things would change that and neither is filed as work yet: scheduling healthcheck.sh somewhere that can shout, and giving the process somewhere to send an exception. Recorded here rather than invented into the tracker.

Backups
Schedule None yet. scripts/backup.sh works and nothing runs it on a timer
What is captured /app/db/queuenorth.db — the leads and support_requests tables. Deliberately not captured: the logs volume, and the environment, which holds the secrets
Where it lands $HOME/backups/queue-north-website on the operator's machine, mode 700. Still one machine — see below
Retention BACKUP_KEEP, default 7
Last verified restore 2026-08-18, by _null — 2 tables, 3 rows, restored in under 1 s

scripts/backup.sh proves a dump is readable before trusting it — it runs better-sqlite3's online .backup() inside the container, copies the result out, and refuses the run unless PRAGMA integrity_check returns ok and the table count is plausible, only then renaming it into place. scripts/restore-check.sh proves it can be restored, which is a different claim: a file that parses is not a database you can get back.

A backup nobody has restored is a guess. The date above is the only line in this table that says otherwise, and as of 2026-08-18 it says something: a snapshot was taken from the running container, replayed into a scratch database from SQL, and counted. It came back.

Two things that are still true anyway, and both are tracked in Batch 15:

  1. Nothing runs on a schedule. One backup taken by hand is not a backup regime. Until a timer exists, the newest dump is as old as the last time somebody remembered.
  2. The dump lives on one machine, beside nothing. backup.sh says this itself on every run: a backup that only exists beside the database it came from does not survive the disk, the host or the account. It is currently on the operator's workstation and nowhere else.

The exposure is worth stating plainly: /app/db/queuenorth.db is the only live copy of every lead and support request the site has ever taken, it lives in one Docker volume on one host, and it took writes today.

A number worth knowing before you need it: the restore took under a second, because the database is 28 KB and holds three leads. That will stay true for a long time at this volume — this is a contact form, not a transactional system — so during an incident, restoring is cheap and there is no reason to hesitate over it.

RESTORE_MIN_ROWS is currently 0, which means the check cannot yet catch the snapshot-of-an-empty-volume case. Set it to something below the real lead count once that count is meaningful, and it starts catching the one failure a structural check never can.

Rate limits and cost ceilings

(precautionary — no incident here has yet come from this.)

Endpoint or job What it costs per call What bounds it
POST /api/leads one SQLite insert, one reCAPTCHA verify, one Zoho form post RATE_LIMIT_PER_MINUTE, default 5, per IP across all of /api. Body capped at 1 MB, request at 30 s
POST /api/support the same, plus a Zoho Case when enabled same limiter
Google reCAPTCHA free at this volume the same limiter, upstream of it
Zoho CRM free at this volume; the API path has per-org daily credits fire-and-forget, so exhausting them degrades CRM sync and never the site

Where does the spend alert go, and at what number? Nowhere, and there is no number. Nothing here bills per call at this volume, which is why that is tolerable rather than an oversight — but if an LLM, a mail sender or a paid API is ever added, this table is the first thing that has to change.

It is down — what now

In this order. Each step is a command that answers one question, and none of them change anything.

  1. Is it actually down, or unreachable from here? bash scripts/healthcheck.sh from anywhere, then bash scripts/status.sh. The second distinguishes those two and reports the restart count. A climbing restart count with a short uptime is a crash loop, not a healthy service, and reads as "running" in every tool that shows only a state.

  2. What is it saying? bash scripts/status.sh --logs 200.

  3. Is the deployed version the one you think? bash scripts/status.sh prints the image digest. --deployed-version cannot answer here — the image carries no org.opencontainers.image.version label, so the script correctly reports the version as unknown rather than guessing. Compare the digest against the registry instead:

    bash scripts/status.sh | grep digest

    Adding that label at build time would make this step a one-liner and is worth doing next time the Dockerfile is touched.

  4. Is it the database? /api/health returns 503 with db: error when the SELECT 1 fails. That is a real answer and not an outage of the whole site: the marketing pages keep serving while every form submission is failing.

  5. Did something change recently? The tracker and git log — a deploy, a config edit, a Cloudflare rule.

  6. Is it the ingress rather than the app? The two front doors take completely different paths — queuenorth.com through nginx-proxy-manager on thor/exodus, qn.isnull.dev through Cloudflare. If one answers and the other does not, the container is fine and the problem is in front of it:

    curl -s https://queuenorth.com/api/health
curl -s https://qn.isnull.dev/api/health

    Both failing while scripts/status.sh says healthy points at the LAN path or the host itself.

Rolling forward or back

Publishing and deploying are two acts. npm run docker:push builds the image and pushes queue-north-website:dev to the Forgejo registry. That changes nothing about what is running.

Deploying is a Portainer stack redeploy. Established 2026-08-18 by reading the container's own compose labels, which is why it is written down here rather than re-derived:
Portainer https://192.168.1.11:9443 (nebula), API key in ~/.openclaw/credentials/portainer.md
Stack id 58, name qn-website-dev, endpoint 3, compose
Stack file on disk /data/compose/58/docker-compose.yml on nebula

The stack file is NOT this repository's docker-compose.yml. They are two separate files that happen to look alike, and they have already drifted: the repository commits CORS_ORIGIN=https://queuenorth.com and line 21 of the stack file says https://queuenorth.com/, with a trailing slash. Editing the one in git changes nothing about production. That is the single most important sentence in this section.

Read the deployed file, and the environment the container actually got:

KEY=$(grep -m1 'API Key' ~/.openclaw/credentials/portainer.md | grep -oE 'ptr_[^`]+')
curl -sk -H "X-API-Key: $KEY" https://192.168.1.11:9443/api/stacks/58/file
bash scripts/status.sh          # what is running, and its image digest

To move to a newly published image, or to change an environment value, redeploy the stack — through the Portainer UI, or its API with PUT /api/stacks/58 carrying the full stack file and pullImage: true.

Redeploying recreates the container, which is a restart of the customer's live site. Both public front doors go down together for the few seconds it takes, because they reach the same instance. There is no non-production environment to rehearse against; npm run docker:test running the image locally is the closest thing that exists. Take the backup first — bash scripts/backup.sh — because the database is in a named volume that a careless down -v would remove.

The template's scripts/deploy.py is built for exactly this shape and reads DEPLOY_STACK_ID, DEPLOY_IMAGE, DEPLOY_CONTAINER and DEPLOY_SITE_URL from the environment. It was declined on adoption because none of the above was known then. It is adoptable now, and worth taking the next time a deploy is needed rather than doing it by hand twice more.

Write the incident down afterwards, in docs/history/DEVELOPMENT_LOG.md with the date, and file what broke as an issue with a severity label. An incident nobody recorded happens again with the same surprise.