268 lines
13 KiB
Markdown
268 lines
13 KiB
Markdown
# Operations — Queue North Website
|
||
|
||
```
|
||
Status: Current
|
||
Owner: _null
|
||
Last reviewed: 2026-08-18
|
||
Governs: Dockerfile, docker-compose.yml, scripts/status.sh,
|
||
scripts/healthcheck.sh, scripts/backup.sh, scripts/restore-check.sh —
|
||
what watches this in production, and what to do when it stops
|
||
Review trigger: A new alert or a new place errors are sent; a change to the
|
||
backup schedule or the restore procedure; any change to the
|
||
deploy host, container or image; any incident whose first ten
|
||
minutes were spent working out what to run
|
||
```
|
||
|
||
## Why this exists
|
||
|
||
Every other document here describes what the project *is*. This one describes
|
||
what happens when it stops being that at three in the morning, and it is the
|
||
only document written for somebody who is not thinking clearly.
|
||
|
||
The failure it prevents is specific: **the first ten minutes of an incident
|
||
spent working out which host, which container, and which command**, while the
|
||
thing everybody actually wants to know — is it down, or is it just unreachable
|
||
from here — goes unanswered.
|
||
|
||
## What is deployed, and where
|
||
|
||
| | |
|
||
| --- | --- |
|
||
| **Public origin** | **`https://queuenorth.com`** — the production front door, and the one customers use |
|
||
| Second front door | `https://qn.isnull.dev`, the same container reached through Cloudflare |
|
||
| `www` | 301s to `https://queuenorth.com/` |
|
||
| Host | **nebula** (`192.168.1.11`) — the same box as the Forgejo instance |
|
||
| Container | `qn-website-dev` |
|
||
| Image | `dream.scheller.ltd/null/queue-north-website:dev` |
|
||
| Port | 3001 in the container; the reverse proxy in front owns the public one |
|
||
| Database | `/app/db/queuenorth.db`, in the named volume `qn-website-dev_queuenorth-db` |
|
||
| Logs volume | `qn-website-dev_queuenorth-logs` → `/app/logs` |
|
||
| Runs as | UID 1001, non-root |
|
||
|
||
### Two front doors, one container
|
||
|
||
**Both hostnames are this application, and there is only one of it.** Verified
|
||
2026-08-18: both serve the identical bundle (`assets/index-pTFwovIx.js`) and both
|
||
answer `/api/health` with this server's exact `{"status":"ok","db":"ok"}` shape.
|
||
|
||
```text
|
||
queuenorth.com ──► 24.41.108.95 (this network's own public IP)
|
||
└─► nginx-proxy-manager on thor/exodus (192.168.1.14)
|
||
└─┐
|
||
├──► qn-website-dev on nebula (192.168.1.11:3001)
|
||
qn.isnull.dev ──► Cloudflare ─┘
|
||
```
|
||
|
||
`www.queuenorth.com` 301s to the apex. Both origins carry HSTS and redirect
|
||
plain HTTP.
|
||
|
||
**This matters during an incident in two directions.** If only one hostname is
|
||
failing it is the ingress, not the app — compare the two before touching the
|
||
container. And if you restart that container you are restarting *production*,
|
||
not a dev alias, whatever the `-dev` in its name and its `:dev` image tag
|
||
suggest.
|
||
|
||
> **An earlier version of this document said the opposite** — that
|
||
> `queuenorth.com` "is not this deployment ... do not diagnose against it". That
|
||
> was written on 2026-08-18 from a DNS lookup and an assumption, and it was
|
||
> wrong. It is corrected here rather than quietly, because a runbook that sends
|
||
> somebody away from the failing host is worse than one that says nothing.
|
||
|
||
**There is no non-production environment.** These are two doors to the same
|
||
room. `npm run docker:test` runs the image locally, which is the closest thing
|
||
that exists.
|
||
|
||
## Where errors go
|
||
|
||
**Nowhere. This project has no error tracking.**
|
||
|
||
Said in one line rather than left blank, because a gap somebody chose is a
|
||
different thing from a gap nobody noticed. There is no Sentry project, no DSN and
|
||
no alert destination. An unhandled exception is logged to the container's stdout
|
||
and takes the process down: `server/index.js` handles `uncaughtException` and
|
||
`unhandledRejection` by logging and exiting 1, and Docker's `restart:
|
||
unless-stopped` brings it back. **That means a crash loop looks like a running
|
||
service to anything that only reads a state.** `scripts/status.sh` reports the
|
||
restart count for exactly this reason.
|
||
|
||
**A liveness tick is not error tracking.** `scripts/healthcheck.sh` answers *is
|
||
it up*. Nothing here answers *is it working* — a server returning 500 to every
|
||
form submission is up, healthy, and losing every lead.
|
||
|
||
## What alerts, and to whom
|
||
|
||
| Signal | Where it goes | Who acts on it |
|
||
| --- | --- | --- |
|
||
| Container `HEALTHCHECK` fails 3× at 30 s | Docker marks the container unhealthy on nebula | **Nobody, automatically.** It is visible to anyone who looks and notifies no one |
|
||
| `scripts/healthcheck.sh` | whoever runs it | not scheduled |
|
||
| Everything else | — | — |
|
||
|
||
**This table is nearly empty and that is the honest state.** Nothing on this
|
||
project pages anybody. The site could be down for a day and the first report
|
||
would come from the customer.
|
||
|
||
Two things would change that and neither is filed as work yet: scheduling
|
||
`healthcheck.sh` somewhere that can shout, and giving the process somewhere to
|
||
send an exception. Recorded here rather than invented into the tracker.
|
||
|
||
## Backups
|
||
|
||
| | |
|
||
| --- | --- |
|
||
| Schedule | **None yet.** `scripts/backup.sh` works and nothing runs it on a timer |
|
||
| What is captured | `/app/db/queuenorth.db` — the `leads` and `support_requests` tables. Deliberately not captured: the logs volume, and the environment, which holds the secrets |
|
||
| Where it lands | `$HOME/backups/queue-north-website` on the operator's machine, mode 700. **Still one machine** — see below |
|
||
| Retention | `BACKUP_KEEP`, default 7 |
|
||
| **Last verified restore** | **2026-08-18, by `_null` — 2 tables, 3 rows, restored in under 1 s** |
|
||
|
||
`scripts/backup.sh` proves a dump is readable before trusting it — it runs
|
||
better-sqlite3's online `.backup()` inside the container, copies the result out,
|
||
and refuses the run unless `PRAGMA integrity_check` returns `ok` and the table
|
||
count is plausible, only then renaming it into place. `scripts/restore-check.sh`
|
||
proves it can be restored, which is a different claim: a file that parses is not
|
||
a database you can get back.
|
||
|
||
**A backup nobody has restored is a guess.** The date above is the only line in
|
||
this table that says otherwise, and as of 2026-08-18 it says something: a
|
||
snapshot was taken from the running container, replayed into a scratch database
|
||
from SQL, and counted. It came back.
|
||
|
||
**Two things that are still true anyway**, and both are tracked in `Batch 15`:
|
||
|
||
1. **Nothing runs on a schedule.** One backup taken by hand is not a backup
|
||
regime. Until a timer exists, the newest dump is as old as the last time
|
||
somebody remembered.
|
||
2. **The dump lives on one machine, beside nothing.** `backup.sh` says this
|
||
itself on every run: *a backup that only exists beside the database it came
|
||
from does not survive the disk, the host or the account.* It is currently on
|
||
the operator's workstation and nowhere else.
|
||
|
||
The exposure is worth stating plainly: `/app/db/queuenorth.db` is the only live
|
||
copy of every lead and support request the site has ever taken, it lives in one
|
||
Docker volume on one host, and it took writes today.
|
||
|
||
**A number worth knowing before you need it:** the restore took under a second,
|
||
because the database is 28 KB and holds three leads. That will stay true for a
|
||
long time at this volume — this is a contact form, not a transactional system —
|
||
so during an incident, restoring is cheap and there is no reason to hesitate over
|
||
it.
|
||
|
||
`RESTORE_MIN_ROWS` is currently 0, which means the check cannot yet catch the
|
||
snapshot-of-an-empty-volume case. Set it to something below the real lead count
|
||
once that count is meaningful, and it starts catching the one failure a
|
||
structural check never can.
|
||
|
||
## Rate limits and cost ceilings
|
||
|
||
*(precautionary — no incident here has yet come from this.)*
|
||
|
||
| Endpoint or job | What it costs per call | What bounds it |
|
||
| --- | --- | --- |
|
||
| `POST /api/leads` | one SQLite insert, one reCAPTCHA verify, one Zoho form post | `RATE_LIMIT_PER_MINUTE`, default 5, per IP across all of `/api`. Body capped at 1 MB, request at 30 s |
|
||
| `POST /api/support` | the same, plus a Zoho Case when enabled | same limiter |
|
||
| Google reCAPTCHA | free at this volume | the same limiter, upstream of it |
|
||
| Zoho CRM | free at this volume; the API path has per-org daily credits | fire-and-forget, so exhausting them degrades CRM sync and never the site |
|
||
|
||
**Where does the spend alert go, and at what number?** Nowhere, and there is no
|
||
number. Nothing here bills per call at this volume, which is why that is
|
||
tolerable rather than an oversight — but if an LLM, a mail sender or a paid API
|
||
is ever added, this table is the first thing that has to change.
|
||
|
||
## It is down — what now
|
||
|
||
In this order. Each step is a command that answers one question, and none of them
|
||
change anything.
|
||
|
||
1. **Is it actually down, or unreachable from here?**
|
||
`bash scripts/healthcheck.sh` from anywhere, then
|
||
`bash scripts/status.sh`. The second distinguishes those two and reports the
|
||
restart count. **A climbing restart count with a short uptime is a crash loop**,
|
||
not a healthy service, and reads as "running" in every tool that shows only a
|
||
state.
|
||
2. **What is it saying?** `bash scripts/status.sh --logs 200`.
|
||
3. **Is the deployed version the one you think?**
|
||
|
||
```bash
|
||
bash scripts/status.sh --deployed-version
|
||
bash scripts/status.sh | grep digest
|
||
```
|
||
|
||
**The image running today carries no `org.opencontainers.image.version`
|
||
label**, so the first command correctly reports the version as unknown rather
|
||
than guessing, and the digest is the only way to tell one deploy from
|
||
another. The label was added to the `Dockerfile` on 2026-08-18 and takes
|
||
effect on the next build — until an image built after that date is deployed,
|
||
use the digest.
|
||
|
||
Non-zero from `--deployed-version` means it could not be read at all. Stop
|
||
there rather than assuming.
|
||
4. **Is it the database?** `/api/health` returns 503 with `db: error` when the
|
||
`SELECT 1` fails. That is a real answer and not an outage of the whole site:
|
||
the marketing pages keep serving while every form submission is failing.
|
||
5. **Did something change recently?** The tracker and `git log` — a deploy, a
|
||
config edit, a Cloudflare rule.
|
||
6. **Is it the ingress rather than the app?** The two front doors take
|
||
completely different paths — `queuenorth.com` through nginx-proxy-manager on
|
||
thor/exodus, `qn.isnull.dev` through Cloudflare. If one answers and the other
|
||
does not, the container is fine and the problem is in front of it:
|
||
|
||
```bash
|
||
curl -s https://queuenorth.com/api/health
|
||
curl -s https://qn.isnull.dev/api/health
|
||
```
|
||
|
||
Both failing while `scripts/status.sh` says healthy points at the LAN path or
|
||
the host itself.
|
||
|
||
### Rolling forward or back
|
||
|
||
**Publishing and deploying are two acts.** `npm run docker:push` builds the
|
||
image and pushes `queue-north-website:dev` to the Forgejo registry. That changes
|
||
nothing about what is running.
|
||
|
||
**Deploying is a Portainer stack redeploy.** Established 2026-08-18 by reading
|
||
the container's own compose labels, which is why it is written down here rather
|
||
than re-derived:
|
||
|
||
| | |
|
||
| --- | --- |
|
||
| Portainer | `https://192.168.1.11:9443` (nebula), API key in `~/.openclaw/credentials/portainer.md` |
|
||
| Stack | **id 58**, name `qn-website-dev`, endpoint 3, compose |
|
||
| Stack file on disk | `/data/compose/58/docker-compose.yml` on nebula |
|
||
|
||
**The stack file is NOT this repository's `docker-compose.yml`.** They are two
|
||
separate files that happen to look alike, and they have already drifted: the
|
||
repository commits `CORS_ORIGIN=https://queuenorth.com` and line 21 of the stack
|
||
file says `https://queuenorth.com/`, with a trailing slash. Editing the one in
|
||
git changes nothing about production. That is the single most important sentence
|
||
in this section.
|
||
|
||
Read the deployed file, and the environment the container actually got:
|
||
|
||
```bash
|
||
KEY=$(grep -m1 'API Key' ~/.openclaw/credentials/portainer.md | grep -oE 'ptr_[^`]+')
|
||
curl -sk -H "X-API-Key: $KEY" https://192.168.1.11:9443/api/stacks/58/file
|
||
bash scripts/status.sh # what is running, and its image digest
|
||
```
|
||
|
||
To move to a newly published image, or to change an environment value, redeploy
|
||
the stack — through the Portainer UI, or its API with `PUT /api/stacks/58`
|
||
carrying the full stack file and `pullImage: true`.
|
||
|
||
**Redeploying recreates the container, which is a restart of the customer's live
|
||
site.** Both public front doors go down together for the few seconds it takes,
|
||
because they reach the same instance. There is no non-production environment to
|
||
rehearse against; `npm run docker:test` running the image locally is the closest
|
||
thing that exists. Take the backup first — `bash scripts/backup.sh` — because
|
||
the database is in a named volume that a careless `down -v` would remove.
|
||
|
||
The template's `scripts/deploy.py` is built for exactly this shape and reads
|
||
`DEPLOY_STACK_ID`, `DEPLOY_IMAGE`, `DEPLOY_CONTAINER` and `DEPLOY_SITE_URL` from
|
||
the environment. **It was declined on adoption** because none of the above was
|
||
known then. It is adoptable now, and worth taking the next time a deploy is
|
||
needed rather than doing it by hand twice more.
|
||
|
||
**Write the incident down afterwards**, in `docs/history/DEVELOPMENT_LOG.md` with
|
||
the date, and file what broke as an issue with a severity label. An incident
|
||
nobody recorded happens again with the same surprise.
|