diff --git a/docs/OPERATIONS.md b/docs/OPERATIONS.md index c362301..852b995 100644 --- a/docs/OPERATIONS.md +++ b/docs/OPERATIONS.md @@ -74,6 +74,23 @@ suggest. room. `npm run docker:test` runs the image locally, which is the closest thing that exists. +## Watchtower is no longer in the picture + +Stack 39 runs Watchtower nightly at 02:00 with no label filter and no scope, so +it watches **every** container on this host. Until 2026-08-18 `qn-website-dev` +had no opt-out label and followed the moving `:dev` tag, which means the site +could — and almost certainly did — update itself overnight whenever anything +pushed that tag. That was the undocumented deploy path nobody could find. + +Two things now prevent it: the stack pins an immutable version, which Watchtower +can never find an update for, and the service carries +`com.centurylinklabs.watchtower.enable=false`, which excludes it explicitly even +if somebody later puts a floating tag back. + +**The consequence is deliberate and worth knowing: nothing updates this site +automatically any more.** Every change reaches production through +`npm run release` and `npm run deploy`, or it does not reach production. + ## Where errors go **Nowhere. This project has no error tracking.** @@ -236,12 +253,20 @@ they print exactly what would change. | Stack file on disk | `/data/compose/58/docker-compose.yml` on nebula | **The stack file is NOT this repository's `docker-compose.yml`.** They are two -separate files that happen to look alike, and they have already drifted — the -repository commits `CORS_ORIGIN=https://queuenorth.com` and line 21 of the stack -file says `https://queuenorth.com/`, with a trailing slash. Editing the one in -git changes nothing about production. That is the single most important sentence -in this section, and `deploy.sh` does not paper over it: it reads the deployed -file, changes only the `image:` line, and sends the rest back untouched. +separate files that happen to look alike, and they drift. Editing the one in git +changes nothing about production. That is the single most important sentence in +this section. + +They drifted once already: the stack carried `CORS_ORIGIN=https://queuenorth.com/` +with a trailing slash while the repository committed it without one. Corrected on +2026-08-18 by `deploy.sh --fix-cors`, which closed #212. + +`deploy.sh` does not paper over the split — it reads the deployed file, changes +only the lines you ask for, prints the diff before sending, and asserts the +environment-line count is unchanged. Four differences remain and all four are +correct: the stack uses `image:` where the repo has `build:`, names the container +`qn-website-dev`, publishes `3032:3001` rather than `3001:3001`, and carries the +Watchtower opt-out label. **What a deploy costs.** It recreates the container, so both public front doors go down together for the few seconds it takes — they reach the same instance. @@ -278,9 +303,11 @@ It also refuses a tag the registry does not hold. A missing tag would otherwise become an outage: Portainer accepts the file, fails to pull, and leaves the container stopped. -**Stack 58 pins `:dev` as this is written**, which is the state the policy -corrects. The first deploy made under it repoints the stack at a version, and -after that the stack file answers the question on its own. +**Stack 58 pins `:v0.9.4` as of 2026-08-18.** It pinned `:dev` until then; the +first deploy made under this policy repointed it, and the stack file now answers +"what is running" on its own. `scripts/status.sh --deployed-version` answers too, +because the image carries `org.opencontainers.image.version` from the same +release. `:dev` and `:latest` remain in the registry, frozen. They are not deleted because the running container was created from `:dev`, and removing the tag an diff --git a/docs/history/DEVELOPMENT_LOG.md b/docs/history/DEVELOPMENT_LOG.md index 7ad7228..8c6a799 100644 --- a/docs/history/DEVELOPMENT_LOG.md +++ b/docs/history/DEVELOPMENT_LOG.md @@ -46,6 +46,78 @@ sequence would have implied more. ## Entries +### 2026-08-18 — v0.9.4 released and deployed. Production is on a number + +The first release and the first deploy this project has ever made through a +script. Both worked; the site did not go down beyond the recreate. + +**Asked three times whether it would break anything, and it would have — twice.** +Both were found by checking rather than by running, which is the only reason +this entry is not an incident report. + +*`release.sh` would have shipped a site whose contact form could not be +submitted.* `.dockerignore` excludes `.env` from the build context, and the +script passed `--build-arg VITE_RECAPTCHA_SITE_KEY=${VITE_RECAPTCHA_SITE_KEY:-}` +without ever loading `.env` — and that variable is unset in every shell. The +`:-` made empty silently acceptable. An empty key makes +`RecaptchaPlaceholder.jsx` render *"Security verification is not configured."* +and never produce a token, and the server has `RECAPTCHA_ENABLED=true`, so every +submission is rejected. **Lead capture would have stopped entirely** — the whole +purpose of the site — and nothing about a successful build would have said so. + +*`deploy.sh`'s default would have rolled production back two months.* The newest +published **numbered** tag was `v0.8.3`, built 2026-05-28; the running `:dev` +image was built 2026-08-01. Inspecting both images showed `v0.8.3` has no +`privacy-policy/` and **no prerendered route directories at all**. The default +path would have removed the privacy policy — which exists specifically because +Meta's crawler does not run JavaScript — and every prerendered route. + +Both are now guards that refuse, not cautions that document. The release one +also greps the **built image's** bundle for the key before pushing, because a +guard that trusts the wiring is the wiring. + +**What was checked before touching production**, since the question was asked +directly: the registry pulls anonymously (making the repo private did not affect +packages — they are owned separately); `docker compose config` validates the +rewritten stack file; the env-line count is asserted, and `compose config` +showed exactly why — without the `Env` array the `${VAR:-false}` defaults +resolve reCAPTCHA and Zoho WebToLead to **false**, so a stripped env would come +back healthy and quietly capture nothing; only one of the fourteen commits since +the running image touches anything a visitor sees. + +**Watchtower was the missing piece of the story.** Stack 39 runs nightly at +02:00 with no label filter, watching every container, and `qn-website-dev` had +no opt-out. Following the moving `:dev` tag, the site was updating itself +overnight — the undocumented deploy path nobody could find. It now pins a +version *and* carries the opt-out label, so nothing changes without a deliberate +deploy. + +**The deploy itself:** one PUT, three lines changed — the image, the CORS +trailing slash, and the Watchtower label — with a verified backup taken first +and automatic rollback armed. It did not need to roll back. + +``` +digest …3253b10c -> …1d283944 +version none -> 0.9.4 +CORS https://queuenorth.com/ -> https://queuenorth.com +leads 3 before, 3 after +``` + +`status.sh --deployed-version` answers `0.9.4` — the first time this project has +been able to say what is running without reading a digest. + +- **Closed:** **#212** (CORS origin), with the live header as evidence. Batch 10 + is now not only fixed but *shipped* — `aria-expanded` and the dual-tone focus + ring are live, and the six Label-in-Name failures are gone from the served + HTML. +- **Next action:** Batch 11 — UI defects on narrow viewports (#195–#198). Verify + each claim by rendering at the stated width first: four of Batch 10's six + issues misstated their own evidence, and these are all viewport-dependent. +- **Blockers:** none on the work. Six issues remain blocked on the site owner + (#68, #110, #162, #213 in Batch 13; #69, #70 in Batch 14). The backup is still + a single copy on one workstation with no schedule — #211, and it is the one + thing here that would actually hurt. + ### 2026-08-18 — Policy: production always runs a numbered version `_null`: *always stay on a num version.* Recorded here because it changed three