fix(seo): the production sitemap had no dates, and the build context had secrets
Three things, all in the path between this repository and the running image. Closes #225, #224 and #223. 1. THE PRODUCTION SITEMAP CARRIED NO LASTMOD AT ALL. Dates come from git history, and the image build cannot see git: .dockerignore excludes .git and node:alpine has no git binary. prerender.js read the failure into an empty catch commented "git unavailable or file untracked", so all 18 URLs came out undated while the build printed a success line. Local builds looked perfect, which is why nobody caught it. release.sh now computes the map where git exists, passes it as the SITEMAP_LASTMOD build arg, and then asks the built image whether its sitemap has dates, refusing to publish one that does not. prerender prints the count on every run, so "18 URLs, 0 dated" can never again read as success. The route-to-source map moved into scripts/lib/routes.js, where a service page now also counts its own content file, so editing one page's copy moves that page's date and no other. Proven: an image built with the arg carries 18 lastmod entries; a build with git deliberately unreadable and no arg reports "18 URLs, 0 carrying a lastmod" and warns. 2. THE DOCKER BUILD CONTEXT CARRIED CLIENT MATERIAL AND LIVE SECRETS. .drop/, zoho.md (the reCAPTCHA secret and the Zoho tokens), Levi.md and two 30 MB zips were all sent to the daemon on every build, along with four agent workspaces. The final image copies only built output, so none of it ever shipped, but one careless COPY would have changed that. Proven by listing the context from inside a throwaway image: before, all of it; after, none of it. 3. UNTRACKED FILES PASSED SILENTLY. docker build packs the working tree, so an untracked module the code imports produces an image that works and a tag that cannot rebuild it. release.sh now refuses while untracked files are present, and pre-commit's note counts them too. #223 also claimed post-commit hides a refused push. It does not: it printed "push was refused. The commit is safe locally and the branch is now ahead." during this batch. The issue was corrected on the tracker rather than acted on. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
null
parent
7cd556c4b5
commit
2f1e24892a
|
|
@ -26,6 +26,27 @@ db
|
|||
logs
|
||||
*.log
|
||||
|
||||
# Client material and operator credentials. None of it belongs in the build
|
||||
# context, which is sent to the Docker daemon in full: .drop/ holds the original
|
||||
# site drop and the owner's copy sheets, zoho.md holds the live reCAPTCHA secret
|
||||
# and the Zoho tokens, and the zips are 30 MB each. The final image copies only
|
||||
# built output, so none of this has ever shipped, but one careless COPY would
|
||||
# change that, and every build sends it across for no reason.
|
||||
.drop
|
||||
zoho.md
|
||||
Levi.md
|
||||
*.zip
|
||||
*.eml
|
||||
*.pdf
|
||||
|
||||
# Agent workspaces and local editor state. Gitignored, and no more use to a
|
||||
# build than they are to the image.
|
||||
.claude
|
||||
.codex
|
||||
.agents
|
||||
.learnings
|
||||
*.code-workspace
|
||||
|
||||
# Private docs (ignored per requirements)
|
||||
DEVELOPMENT_LOG.md
|
||||
FUTURE.md
|
||||
|
|
|
|||
|
|
@ -132,9 +132,17 @@ else
|
|||
fi
|
||||
|
||||
# Said last so it is the thing still on screen when the editor opens.
|
||||
if ! git diff --quiet; then
|
||||
say "NOTE: unstaged changes are present. The guards ran against the working"
|
||||
say " tree, so they did not verify this commit in isolation."
|
||||
#
|
||||
# Untracked files count here too, and used not to. The build above runs against
|
||||
# the WORKING TREE: a new module that is imported but never added makes the
|
||||
# build pass and the commit itself unbuildable, and `docker build .` would ship
|
||||
# the file anyway while the tagged commit could not produce it.
|
||||
untracked=$(git ls-files --others --exclude-standard | wc -l | tr -d ' ')
|
||||
if ! git diff --quiet || [ "$untracked" != "0" ]; then
|
||||
say "NOTE: the guards ran against the working tree, so they did not verify"
|
||||
say " this commit in isolation."
|
||||
git diff --quiet || say " unstaged changes to tracked files are present."
|
||||
[ "$untracked" = "0" ] || say " ${untracked} untracked file(s) are present, and the build saw them."
|
||||
fi
|
||||
|
||||
exit 0
|
||||
|
|
|
|||
|
|
@ -19,6 +19,13 @@ COPY . .
|
|||
ARG VITE_RECAPTCHA_SITE_KEY=
|
||||
ENV VITE_RECAPTCHA_SITE_KEY=$VITE_RECAPTCHA_SITE_KEY
|
||||
|
||||
# Sitemap dates, computed by scripts/release.sh where git exists. This build
|
||||
# has no git: .dockerignore excludes .git and this image ships no git binary.
|
||||
# Without this the sitemap goes out with no lastmod at all, which is what
|
||||
# production served for months.
|
||||
ARG SITEMAP_LASTMOD=
|
||||
ENV SITEMAP_LASTMOD=$SITEMAP_LASTMOD
|
||||
|
||||
# Build the frontend
|
||||
RUN npm run build
|
||||
|
||||
|
|
|
|||
|
|
@ -246,6 +246,21 @@ npm run deploy # deploy: move stack 58 to what :dev now points a
|
|||
Both take `--dry-run`, and both refuse rather than guess. Run the dry runs first;
|
||||
they print exactly what would change.
|
||||
|
||||
**The image build cannot see git, and the sitemap needs it.** `.dockerignore`
|
||||
excludes `.git` and the build image has no git binary, so the `lastmod` dates
|
||||
that come from commit history silently came out empty: **the production sitemap
|
||||
carried no dates at all** until 2026-09-10. `release.sh` now computes the dates
|
||||
where git exists and passes them in as the `SITEMAP_LASTMOD` build arg, which the
|
||||
builder stage of the `Dockerfile` reads. It then asks the built image whether its
|
||||
sitemap has dates, and refuses to publish one that does not. Search engines
|
||||
schedule recrawls partly on `lastmod`, and Bing's index feeds Copilot and ChatGPT
|
||||
search, so an undated sitemap costs visibility on exactly the surfaces this site
|
||||
was rewritten for.
|
||||
|
||||
`release.sh` also refuses to run while **untracked** files are present, not only
|
||||
uncommitted ones. `docker build` packs the working tree, so an untracked file
|
||||
produces an image that works and a tag that cannot rebuild it.
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| Portainer | `https://192.168.1.11:9443` (nebula), API key in `~/.openclaw/credentials/portainer.md` |
|
||||
|
|
|
|||
|
|
@ -110,6 +110,13 @@ when `src/`, `server/`, `index.html`, `vite.config.js` or `package.json` is
|
|||
staged. **That is a build, not a test.** It catches a broken import and will not
|
||||
catch a broken behaviour.
|
||||
|
||||
It builds the **working tree**, not the commit, so it says so when they differ,
|
||||
counting untracked files as well as unstaged edits. An untracked module that the
|
||||
staged code imports makes the build pass and the commit itself unbuildable, and
|
||||
`docker build` would ship the file anyway while the tag could not rebuild it.
|
||||
`npm run release` refuses outright while untracked files are present, for the
|
||||
same reason.
|
||||
|
||||
`post-commit` pushes, and that is the intent — but it has a consequence worth
|
||||
holding on to: whatever documentation was not in that commit is now behind the
|
||||
code by one push. That is the mechanical reason `docs/WORK_CYCLE.md` asks for doc
|
||||
|
|
|
|||
|
|
@ -109,6 +109,14 @@ tr '\0' '\n' < /proc/$PID/environ | grep -c . # 0 here means "could not read"
|
|||
This is the confident-absence failure one level up: the same trap as a screen
|
||||
rendering a failed query as a count of zero, applied to your own diagnosis.
|
||||
|
||||
**The same trap inside a data source.** `scripts/prerender.js` read sitemap dates
|
||||
from `git log` inside a `try` whose `catch` was empty and commented *"git
|
||||
unavailable or file untracked"*. Inside the image build git is always
|
||||
unavailable, so every date came out empty and the sitemap shipped with none at
|
||||
all, for months, while the build printed a cheerful success line. The fix is two
|
||||
parts and both matter: pass the data in from where it exists, and **say the
|
||||
count out loud on every run**, so "18 URLs, 0 dated" cannot read as success.
|
||||
|
||||
**The same trap inside a scanner.** A matcher that discards its errors turns
|
||||
"could not run" into "found nothing". `grep "$pattern" 2>/dev/null` answers a
|
||||
pattern it cannot read with exit 2 and no output, and a loop reading its matches
|
||||
|
|
|
|||
|
|
@ -9,9 +9,14 @@
|
|||
//
|
||||
// Plain JavaScript with no side effects, because prerender imports it in Node
|
||||
// before Vite exists, and so does the validator runner.
|
||||
import { execFileSync } from 'child_process'
|
||||
import path from 'path'
|
||||
import { fileURLToPath } from 'url'
|
||||
import { services } from '../../src/data/services.js'
|
||||
import { industries } from '../../src/data/industries.js'
|
||||
|
||||
const repoRoot = path.join(path.dirname(fileURLToPath(import.meta.url)), '../..')
|
||||
|
||||
export const STATIC_ROUTES = [
|
||||
'/',
|
||||
'/about',
|
||||
|
|
@ -46,7 +51,82 @@ export const routerPaths = (table, base = '') =>
|
|||
export const routeDrift = (table) => [
|
||||
...new Set(
|
||||
routerPaths(table)
|
||||
.filter((path) => !path.includes(':') && !path.includes('*'))
|
||||
.filter((path) => !ROUTES.includes(path)),
|
||||
.filter((route) => !route.includes(':') && !route.includes('*'))
|
||||
.filter((route) => !ROUTES.includes(route)),
|
||||
),
|
||||
]
|
||||
|
||||
// --- sitemap dates -----------------------------------------------------------
|
||||
//
|
||||
// The files that produce each page. A page's `lastmod` is the newest commit
|
||||
// date among them, never the build timestamp: a sitemap that marks every page
|
||||
// as changed on every deploy is one search engines learn to ignore.
|
||||
|
||||
const ROUTE_SOURCES = {
|
||||
'/': ['src/pages/Home.jsx'],
|
||||
'/about': ['src/pages/About.jsx'],
|
||||
'/services': ['src/pages/Services.jsx', 'src/data/services.js'],
|
||||
'/industries': ['src/pages/Industries.jsx', 'src/data/industries.js'],
|
||||
'/contact': ['src/pages/Contact.jsx'],
|
||||
'/support': ['src/pages/Support.jsx'],
|
||||
'/privacy-policy': ['src/pages/PrivacyPolicy.jsx', 'src/data/privacyPolicy.js'],
|
||||
}
|
||||
|
||||
export const sourcesFor = (url) => {
|
||||
if (ROUTE_SOURCES[url]) return ROUTE_SOURCES[url]
|
||||
if (url.startsWith('/services/')) {
|
||||
// A page with owner-approved copy has its own content file, so editing that
|
||||
// copy moves that page's date and no other.
|
||||
const slug = url.slice('/services/'.length)
|
||||
return ['src/pages/ServiceDetail.jsx', 'src/data/services.js', `src/data/serviceContent/${slug}.js`]
|
||||
}
|
||||
return ['src/pages/IndustryDetail.jsx', 'src/data/industries.js']
|
||||
}
|
||||
|
||||
const gitLastModified = (files) => {
|
||||
let newest = null
|
||||
for (const file of files) {
|
||||
try {
|
||||
const iso = execFileSync('git', ['log', '-1', '--format=%cI', '--', file], {
|
||||
cwd: repoRoot,
|
||||
encoding: 'utf8',
|
||||
stdio: ['ignore', 'pipe', 'ignore'],
|
||||
}).trim()
|
||||
if (iso && (!newest || iso > newest)) newest = iso
|
||||
} catch {
|
||||
// git is unavailable, or the file is untracked. Either way, no date from
|
||||
// this file. Whether that leaves the ROUTE undated is the caller's
|
||||
// problem to report, and it must not pass silently: the production
|
||||
// sitemap carried no dates at all for months because this catch was the
|
||||
// end of the story.
|
||||
}
|
||||
}
|
||||
return newest ? newest.slice(0, 10) : null
|
||||
}
|
||||
|
||||
/**
|
||||
* Route to YYYY-MM-DD, for the sitemap.
|
||||
*
|
||||
* The image build has no git: `.dockerignore` excludes `.git` and node:alpine
|
||||
* ships no git binary. So a map computed where git DOES exist can be injected
|
||||
* through SITEMAP_LASTMOD, which is what scripts/release.sh does.
|
||||
*/
|
||||
export const lastModByRoute = () => {
|
||||
const injected = process.env.SITEMAP_LASTMOD
|
||||
if (injected) {
|
||||
try {
|
||||
const parsed = JSON.parse(injected)
|
||||
if (parsed && typeof parsed === 'object') return parsed
|
||||
console.warn('routes: SITEMAP_LASTMOD is not an object, so falling back to git.')
|
||||
} catch (error) {
|
||||
console.warn(`routes: SITEMAP_LASTMOD is not valid JSON (${error.message}), so falling back to git.`)
|
||||
}
|
||||
}
|
||||
|
||||
const dates = {}
|
||||
for (const route of ROUTES) {
|
||||
const date = gitLastModified(sourcesFor(route))
|
||||
if (date) dates[route] = date
|
||||
}
|
||||
return dates
|
||||
}
|
||||
|
|
|
|||
|
|
@ -13,7 +13,6 @@
|
|||
//
|
||||
// Run automatically as part of `npm run build`.
|
||||
|
||||
import { execFileSync } from 'child_process'
|
||||
import { mkdirSync, readFileSync, writeFileSync } from 'fs'
|
||||
import path from 'path'
|
||||
import { fileURLToPath } from 'url'
|
||||
|
|
@ -21,7 +20,7 @@ import { fileURLToPath } from 'url'
|
|||
import { render, routes as routerTable } from '../dist-ssr/entry-server.js'
|
||||
import { services } from '../src/data/services.js'
|
||||
import { industries } from '../src/data/industries.js'
|
||||
import { ROUTES as routes, STATIC_ROUTES, routeDrift } from './lib/routes.js'
|
||||
import { ROUTES as routes, lastModByRoute, routeDrift } from './lib/routes.js'
|
||||
import { validateContent } from './lib/content.js'
|
||||
|
||||
const __dirname = path.dirname(fileURLToPath(import.meta.url))
|
||||
|
|
@ -210,36 +209,6 @@ for (const [url, size] of written) {
|
|||
|
||||
const SITE_URL = 'https://queuenorth.com'
|
||||
|
||||
// lastmod comes from git history for the files that produce each route. A build
|
||||
// timestamp would mark every page as changed on every deploy, which search engines
|
||||
// learn to distrust.
|
||||
const lastModifiedFor = (files) => {
|
||||
let newest = null
|
||||
for (const file of files) {
|
||||
try {
|
||||
const iso = execFileSync('git', ['log', '-1', '--format=%cI', '--', file], {
|
||||
cwd: path.join(__dirname, '..'),
|
||||
encoding: 'utf8',
|
||||
stdio: ['ignore', 'pipe', 'ignore'],
|
||||
}).trim()
|
||||
if (iso && (!newest || iso > newest)) newest = iso
|
||||
} catch {
|
||||
// git unavailable or file untracked — fall through
|
||||
}
|
||||
}
|
||||
return newest ? newest.slice(0, 10) : null
|
||||
}
|
||||
|
||||
const ROUTE_SOURCES = {
|
||||
'/': ['src/pages/Home.jsx'],
|
||||
'/about': ['src/pages/About.jsx'],
|
||||
'/services': ['src/pages/Services.jsx', 'src/data/services.js'],
|
||||
'/industries': ['src/pages/Industries.jsx', 'src/data/industries.js'],
|
||||
'/contact': ['src/pages/Contact.jsx'],
|
||||
'/support': ['src/pages/Support.jsx'],
|
||||
'/privacy-policy': ['src/pages/PrivacyPolicy.jsx', 'src/data/privacyPolicy.js'],
|
||||
}
|
||||
|
||||
const PRIORITY = {
|
||||
'/': '1.0',
|
||||
'/services': '0.9',
|
||||
|
|
@ -252,15 +221,13 @@ const PRIORITY = {
|
|||
|
||||
const CHANGEFREQ = { '/': 'weekly', '/privacy-policy': 'yearly' }
|
||||
|
||||
const sourcesFor = (url) => {
|
||||
if (ROUTE_SOURCES[url]) return ROUTE_SOURCES[url]
|
||||
if (url.startsWith('/services/')) return ['src/pages/ServiceDetail.jsx', 'src/data/services.js']
|
||||
return ['src/pages/IndustryDetail.jsx', 'src/data/industries.js']
|
||||
}
|
||||
// Dates come from git where git exists, and from the map release.sh injects
|
||||
// where it does not. See scripts/lib/routes.js.
|
||||
const lastmodByRoute = lastModByRoute()
|
||||
|
||||
const entries = routes.map((url) => {
|
||||
const loc = url === '/' ? SITE_URL : `${SITE_URL}${url}`
|
||||
const lastmod = lastModifiedFor(sourcesFor(url))
|
||||
const lastmod = lastmodByRoute[url] ?? null
|
||||
return [
|
||||
' <url>',
|
||||
` <loc>${loc}</loc>`,
|
||||
|
|
@ -282,4 +249,15 @@ const sitemap = [
|
|||
].join('\n')
|
||||
|
||||
writeFileSync(path.join(distDir, 'sitemap.xml'), sitemap)
|
||||
console.log(`\nGenerated sitemap.xml with ${routes.length} URLs`)
|
||||
|
||||
// Say how many pages carry a date, every time. The production sitemap carried
|
||||
// none at all for months: the image build has no git, and the failure to read
|
||||
// it was swallowed. Silence is what let that run.
|
||||
const dated = routes.filter((url) => lastmodByRoute[url]).length
|
||||
console.log(`\nGenerated sitemap.xml with ${routes.length} URLs, ${dated} carrying a lastmod`)
|
||||
if (dated < routes.length) {
|
||||
console.warn(
|
||||
`prerender: ${routes.length - dated} URL(s) have no lastmod. git history is not readable here, which is ` +
|
||||
'normal inside the image build. Pass SITEMAP_LASTMOD, as scripts/release.sh does.',
|
||||
)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -215,6 +215,18 @@ if ! git diff --quiet || ! git diff --cached --quiet; then
|
|||
cannot tell your work in progress from a release."
|
||||
fi
|
||||
|
||||
# Untracked files are checked separately, and they matter more than they look.
|
||||
# `docker build .` packs the WORKING TREE, not the commit: an untracked module
|
||||
# that the code imports produces an image that works and a tagged commit that
|
||||
# cannot be rebuilt. The tag is supposed to be the record of what shipped.
|
||||
untracked=$(git ls-files --others --exclude-standard)
|
||||
if [ -n "$untracked" ]; then
|
||||
say "the working tree has untracked files:"
|
||||
printf '%s\n' "$untracked" | sed 's/^/ /' >&2
|
||||
die "add or remove them first. docker build packs the working tree, so these
|
||||
would go into the image while the tag could not rebuild it."
|
||||
fi
|
||||
|
||||
current=$(node -p "require('./package.json').version" 2>/dev/null) \
|
||||
|| stop "could not read the version from package.json."
|
||||
|
||||
|
|
@ -265,8 +277,9 @@ if [ -n "$DRY_RUN" ]; then
|
|||
say "--dry-run: nothing was changed. It would have:"
|
||||
say " set version ${next} in ${FILES[*]}"
|
||||
say " bash scripts/verify.sh"
|
||||
say " docker build --build-arg APP_VERSION=${next} -t ${IMAGE}:${TAG} ."
|
||||
say " docker build --build-arg APP_VERSION=${next} --build-arg SITEMAP_LASTMOD=<dates> -t ${IMAGE}:${TAG} ."
|
||||
say " verify the image's org.opencontainers.image.version label reads ${next}"
|
||||
say " verify the image's sitemap carries dates"
|
||||
say " verify the built bundle contains the reCAPTCHA site key"
|
||||
say " docker push ${IMAGE}:${TAG}"
|
||||
say " git commit -m 'chore(release): ${TAG}' (post-commit then pushes)"
|
||||
|
|
@ -327,9 +340,21 @@ say " loaded a page in a browser."
|
|||
# ---------------------------------------------------------------------------
|
||||
say "building ${IMAGE}:${TAG}…"
|
||||
|
||||
# Sitemap dates come from git history, and the image build cannot see it:
|
||||
# .dockerignore excludes .git and node:alpine ships no git binary. Computing the
|
||||
# map here, where git exists, and passing it in is what puts dates in the
|
||||
# production sitemap. Without it every URL goes out undated, which is exactly
|
||||
# what production served until 2026-09-10.
|
||||
sitemap_lastmod=$(node -e "import('./scripts/lib/routes.js').then(m => console.log(JSON.stringify(m.lastModByRoute())))" 2>/dev/null)
|
||||
if [ -z "$sitemap_lastmod" ] || [ "$sitemap_lastmod" = "{}" ]; then
|
||||
die "could not compute sitemap dates from git history, so the image would ship an
|
||||
undated sitemap. Run this from a full clone, not an export."
|
||||
fi
|
||||
|
||||
if ! docker build \
|
||||
--build-arg "APP_VERSION=${next}" \
|
||||
--build-arg "VITE_RECAPTCHA_SITE_KEY=${VITE_RECAPTCHA_SITE_KEY:-}" \
|
||||
--build-arg "SITEMAP_LASTMOD=${sitemap_lastmod}" \
|
||||
-t "${IMAGE}:${TAG}" . ; then
|
||||
say "build failed. The version bump is in your working tree and NOTHING was"
|
||||
say " published or committed. 'git checkout -- ${FILES[*]}' undoes it."
|
||||
|
|
@ -360,6 +385,25 @@ fi
|
|||
# makes, for the value that actually stops the product working. This survives
|
||||
# somebody editing the Dockerfile's ARG/ENV pair or adding dist to
|
||||
# .dockerignore, neither of which would fail the build.
|
||||
say "verifying the image's sitemap carries dates…"
|
||||
|
||||
# Ask the artifact, not the wiring. A missing build arg, a typo in the
|
||||
# Dockerfile's ARG/ENV pair, or a .dockerignore change would each leave the
|
||||
# sitemap undated while the build still succeeds.
|
||||
dated=$(docker run --rm --entrypoint sh "${IMAGE}:${TAG}" -c \
|
||||
"grep -c '<lastmod>' /app/dist/sitemap.xml" 2>/dev/null | tr -d '\r\n')
|
||||
|
||||
if [ "${dated:-0}" -lt 1 ]; then
|
||||
docker rmi "${IMAGE}:${TAG}" >/dev/null 2>&1
|
||||
say "the image's sitemap carries no lastmod at all. Search engines schedule"
|
||||
say " recrawls on it, and Bing's index feeds Copilot and ChatGPT search."
|
||||
say " Check ARG/ENV SITEMAP_LASTMOD in the builder stage of the"
|
||||
say " Dockerfile. Nothing was published or committed; the local image"
|
||||
say " was removed."
|
||||
exit 1
|
||||
fi
|
||||
say " ${dated} URL(s) dated."
|
||||
|
||||
say "verifying the bundle carries the reCAPTCHA site key…"
|
||||
|
||||
if ! docker run --rm --entrypoint sh "${IMAGE}:${TAG}" -c \
|
||||
|
|
|
|||
Loading…
Reference in New Issue