Queue-North-Website/scripts/backup.sh

#!/usr/bin/env bash
#
# Write one database dump, prove the file is readable before trusting it, and
# keep the newest few.
#
#   bash scripts/backup.sh                  # write one verified dump
#   bash scripts/backup.sh --dry-run        # show what it would write and delete
#   bash scripts/backup.sh --no-prune       # write, delete nothing
#   BACKUP_KEEP=30 bash scripts/backup.sh   # keep thirty instead of seven
#
#   0 3 * * *  cd /srv/app && bash scripts/backup.sh   # from cron, see below
#
# ===========================================================================
# TEMPLATE COPY — configure this before the first run
# ===========================================================================
#
# Copy to `scripts/backup.sh`, add `"backup": "bash scripts/backup.sh"` to
# package.json if this is a Node project, and set the two values in the
# CONFIGURATION block below. The script refuses to run until they are set: it
# has no defaults, deliberately.
#
# A BACKUP_DIR or BACKUP_NAME inherited from another project is not a cosmetic
# mistake. BACKUP_NAME is the filename prefix, and the filename prefix is what
# the retention rule globs on — so the wrong name writes this project's rows
# into that project's series and then deletes that project's oldest dumps to
# make room for them. Both halves are silent, the second is irreversible, and
# the directory afterwards looks exactly like a healthy backup series. Hence:
# no defaults, and a loud failure instead.
#
# Assumes: bash, coreutils, `awk`, `sed`, and the client tools for the engine
# (`pg_dump` and `pg_restore` for the default). No jq, no Node, nothing
# language-specific — unlike the release script beside it, this does not care
# what the project is written in.
#
# awk is named because it is not coreutils, a minimal container image can be
# without it, and it is what counts the tables — so it is checked at startup
# alongside the engine's tools. Left unchecked, its absence arrives as "the
# archive holds 0 tables", which is a sentence about the database and would send
# somebody looking at the wrong thing entirely.
#
# ## Why this exists
#
# A backup nobody has restored is a hypothesis, not a backup.
#
# The failure this is built around is not "the backup did not run". That one is
# loud: the directory is empty and somebody notices. The failure is a job that
# runs every night for a year and writes a file every night for a year, and the
# file is truncated, or is a dump of the wrong database, or is 400 bytes of
# `pg_dump: error:` because the password expired in March. Nothing about the
# directory listing distinguishes that from a working backup. The size column
# is plausible. The timestamps march forward. It is discovered on the one day
# it matters, by somebody who has already lost the database.
#
# So this script does not write a file and call it a backup. It writes the file
# under a temporary name, reads the file back with `pg_restore --list`, counts
# what is in it, refuses the whole run if the archive cannot be read, holds no
# tables, or CANNOT BE COUNTED, and only then renames it into place.
#
# The third of those is not padding. The counting step is one `awk` away from
# returning nothing, and a count that did not happen is not a count of zero: if
# the result were trusted unchecked, `[ "$TABLES" -gt 0 ]` would fail with
# "integer expression expected", `if` would read that failure as false, and an
# archive nobody had counted would be renamed into place and announced as
# verified — with retention then deleting real dumps to make room for it.
#
# ## What --list proves, and what it does not
#
# It proves the file is a real archive: the header parses, the table of
# contents is intact and complete, and the dump contains the tables you can see
# named in the output. Combined with `pg_dump` exiting zero, that eliminates
# the truncated file, the zero-byte file, the error message written where a
# dump should be, and the dump taken against an empty or wrong database.
#
# It does not decompress and check every data block, and it is not a restore.
# An archive can list cleanly and still fail to load — a broken large object, a
# circular constraint order, an extension that is not installed on the machine
# you are restoring onto. Nothing short of an actual restore finds those, which
# is why "the other half" below is not optional.
#
# ## Atomic, because the failure mode is a good backup destroyed by a bad one
#
# Every write goes to `<name>.part` and is renamed only after it verifies. The
# rename is within one directory, so it is rename(2) and not a copy: the final
# name never exists holding half a file, and a reader — a sync job, a human, an
# offsite copy — cannot pick up a dump that is still being written.
#
# Writing directly to the final path would mean a dump that dies at 90% has
# already overwritten last night's, which was fine. That is worse than not
# running at all: it converts a working backup into a broken one and reports
# success while doing it.
#
# A dump that fails verification is KEPT, under its `.part` name, and the path
# is printed. It is evidence — usually the error message is inside it — and
# deleting evidence to keep the directory tidy is how the cause stays unknown.
#
# ## Retention deletes by explicit path, inside one directory, and nowhere else
#
# There is no `find -delete`, no `rm` with a glob, and no recursion. The
# candidate list is a glob of the exact filename shape this script writes, and
# every path is then checked one at a time: its parent directory, resolved with
# `pwd -P` rather than compared as text, must BE the backup directory; it must
# be a regular file and not a symlink; its name must match the pattern. Four
# checks, all of which have to pass, for each file, immediately before the `rm`.
#
# This is deliberately more than is needed for paths the script generated
# itself. The point is that no configuration value, no symlink planted in the
# directory, and no future edit to the glob can produce a deletion outside the
# backup directory — the guard does not trust the list it was handed.
#
# ## Credentials
#
# Connection details come from the environment and are never written here.
# PGHOST/PGPORT/PGUSER/PGDATABASE with a ~/.pgpass is the preferred form;
# DATABASE_URL works and is warned about, because a URL carries the password in
# it and reaches pg_dump as an argument, where `ps` shows it to every user on
# the host.
#
# Nothing prints a password or a connection URL — not to stdout, not to stderr,
# and above all not into a filename, where it would sit in the directory
# listing forever and be copied offsite with the dumps. Credentials in logs is
# the classic leak in backup scripts and it is usually introduced by an
# innocent-looking `say "dumping $DATABASE_URL"`. Do not add one.
#
# ## Changing the engine
#
# PostgreSQL is the supported default. Everything engine-specific is in the
# block marked ENGINE below — four functions and two variables — and nothing
# outside that block knows what a database is. Swapping in MySQL, SQLite or
# anything else is an edit to that block alone.
#
# One constraint carries over: `engine_verify` must READ THE FILE THAT WAS JUST
# WRITTEN. Re-querying the database, checking an exit code again, or trusting
# the file size verifies nothing about the artefact. Engines whose dump is
# plain SQL have no `--list` equivalent; for those the honest verification is a
# restore into a scratch database, and if that is too expensive to do on every
# run then the backup is unverified and the header of this script should be
# edited to stop claiming otherwise.
#
# ## What it deliberately does not do
#
# It does not copy the dump anywhere. A backup on the same disk as the database
# survives `DROP TABLE` and nothing else — not the disk, not the host, not the
# provider account. Getting these files onto different hardware is a separate,
# deliberate act (rsync, restic, object storage) for the same reason the
# release script does not deploy: two decisions, made one at a time.
#
# It does not schedule itself. BACKUP_KEEP is a count of files, not a period —
# seven of these is a week of nightly runs or seven hours of hourly ones.
#
# It does not restore. See below.
#
# ## The other half: a restore you have actually performed
#
# This script verifies the artefact. Only a restore verifies the backup, and a
# restore is also the only way to learn the number that matters during an
# incident, which is how long it takes.
#
# Do it on a schedule you write down — quarterly is a reasonable floor — into a
# scratch database, from the newest file this script produced, and record the
# duration and the command. Until that has happened once, the honest status of
# this directory is "dumps that appear to be readable", and the first restore
# will be attempted by somebody who has already lost the database.
#
#   pg_restore --clean --if-exists --no-owner -d "$SCRATCH_URL" <newest file>

set -uo pipefail

# Glob expansion and `sort` follow the collation locale, and a cron job's
# locale is not the one your shell has. C collation makes the ordering of the
# candidate list byte-order and therefore the same everywhere — the ordering
# decides which files retention calls "oldest", so it deciding differently
# under cron than under test would be discovered by deleting the wrong ones.
# LC_COLLATE only: LC_ALL=C would also change LC_CTYPE, and character handling
# is not something a backup script should be quietly redefining.
unset LC_ALL
export LC_COLLATE=C

# An unmatched glob otherwise expands to the pattern itself, so an empty backup
# directory would produce one "file" literally named `myapp-[0-9][0-9]…` and
# hand it to the retention loop as a real candidate.
shopt -s nullglob

say() { printf '\033[1mbackup:\033[0m %s\n' "$*" >&2; }
die() { printf '\033[1mbackup:\033[0m %s\n' "$*" >&2; exit 1; }

# ---------------------------------------------------------------------------
# CONFIGURATION — set these two, then delete this banner.
#
# Both are empty on purpose. See the note at the top: an inherited BACKUP_NAME
# points the retention rule at another project's dumps, and an inherited
# BACKUP_DIR puts this project's rows where that project's offsite copy will
# collect them.
# ---------------------------------------------------------------------------

# Absolute path to the directory holding the dumps. Must already exist, must be
# outside the repository, and should be readable only by the user running this
# — a dump is every row in the database in one file.
BACKUP_DIR="${BACKUP_DIR:-$HOME/backups/queue-north-website}"

# Filename prefix identifying this project's series, e.g. `acme-orders`.
# Letters, digits, dot, underscore and hyphen only. This is what retention
# globs on; see the header for why it is not derived from the database name.
BACKUP_NAME="${BACKUP_NAME:-queuenorth-leads}"

# ---------------------------------------------------------------------------
# Tunables. These have defaults because none of them names anything belonging
# to a particular project.
# ---------------------------------------------------------------------------

# How many dumps survive, newest first. Seven is a week of nightly runs.
#
# Two is the floor enforced below. Keeping one means every run destroys the
# only copy you have in order to make a copy that has not yet been shown to be
# worth anything — and "worth something" here is more than readable, since a
# dump of an empty or wrong database verifies perfectly. Two means the newest
# dump can turn out to be worthless without that being the end of it.
KEEP="${BACKUP_KEEP:-7}"

# Refuse a dump containing fewer tables than this. One is the default because
# an archive with no tables in it is not a backup of anything, and the way you
# get one is `pg_dump` connecting somewhere you did not intend — see the
# PGDATABASE note below, where the accident produces a valid, tiny, entirely
# empty archive. Set 0 only if this project's database genuinely has no tables,
# in which case there is nothing here to protect.
MIN_TABLES="${BACKUP_MIN_TABLES:-1}"

# Warn when the new dump is smaller than this percentage of the previous one.
# A dump that halves overnight is usually a partial dump, a lost permission on
# some schema, or the wrong database — none of which fail loudly on their own.
#
# A warning and never a refusal: a legitimate large delete would otherwise
# block every backup from that moment on, which is a way of losing data by
# being careful about data.
SHRINK_PCT="${BACKUP_SHRINK_PCT:-50}"

# ===========================================================================
# ENGINE — SQLite, inside the running container on the deploy host.
#
# The template ships this block as PostgreSQL and says to replace this and
# nothing else. This is that replacement.
#
# ## Where the database actually is
#
# Not on this machine. `/app/db/queuenorth.db` lives in the Docker named volume
# `qn-website-dev_queuenorth-db` on nebula, and the only copy of every lead and
# support request the site has ever taken is in it.
#
# ## Why `.backup()` and not `cp`
#
# SQLite is a file, which makes copying it look trivial and makes doing it
# wrong silent. A plain `cp` of a live database can capture a torn page mid
# write, and the result opens, reads, and is corrupt in a way nothing announces
# until the row you need is the missing one. The online backup API takes a
# consistent snapshot of a database that is being written to, which is exactly
# the situation here — the site is live and took writes today.
#
# There is no `sqlite3` binary in the container (node:20-alpine), so the
# snapshot is taken by the `better-sqlite3` the application itself already
# depends on. Its `.backup()` is the online API. `sqlite3` IS present on nebula
# and on this machine, and that is what verifies the result afterwards.
#
# ## The rule this block must not break
#
# The script's contract is that nothing is renamed into place until it has been
# read back and counted. That is kept: `engine_dump` writes a temporary name,
# `engine_verify` runs `PRAGMA integrity_check` and lists the tables, and
# `engine_summarise` counts them. A snapshot that cannot be read is refused,
# and so is one with no tables in it.
# ===========================================================================

DUMP_SUFFIX=".sqlite"

# Overridable so this can be pointed at a staging container, or at a local file
# for a rehearsal, without editing the script.
BACKUP_SSH_HOST="${BACKUP_SSH_HOST:-nebula}"
BACKUP_CONTAINER="${BACKUP_CONTAINER:-qn-website-dev}"
BACKUP_DB_PATH="${BACKUP_DB_PATH:-/app/db/queuenorth.db}"

# `docker` is needed on the REMOTE host, not here, so it cannot be checked by
# the tool test — which only looks at this machine's PATH. ssh and sqlite3 can
# be, and are: sqlite3 is what verifies, and its absence would otherwise arrive
# as "the snapshot holds 0 tables", a sentence about the database that would
# send somebody looking at entirely the wrong thing.
engine_tools() { printf 'ssh sqlite3'; }

engine_dump() {
  local out="$1"

  # Written to a temporary path INSIDE the container first, then copied out.
  # `docker cp` of a live database file would have exactly the torn-page
  # problem the online API exists to avoid, so the order matters: snapshot
  # first, copy the snapshot second.
  local remote_tmp="/tmp/qn-backup-$$.sqlite"

  # The node one-liner is passed through ssh and docker exec, so it is
  # single-quoted here and double-quoted there. It opens the live database
  # read-only and asks SQLite itself to produce the snapshot.
  if ! ssh -o BatchMode=yes "$BACKUP_SSH_HOST" \
        "docker exec '$BACKUP_CONTAINER' node -e \"
           const Database = require('better-sqlite3');
           const db = new Database('$BACKUP_DB_PATH', { readonly: true });
           db.backup('$remote_tmp')
             .then(() => { db.close(); process.exit(0); })
             .catch(e => { console.error(e.message); process.exit(1); });
         \"" >&2; then
    ssh -o BatchMode=yes "$BACKUP_SSH_HOST" "docker exec '$BACKUP_CONTAINER' rm -f '$remote_tmp'" >/dev/null 2>&1
    return 1
  fi

  # Out of the container, onto the host, then down to here. Two hops because
  # `docker cp` cannot write to a remote path and `scp` cannot read from inside
  # a container.
  local host_tmp="/tmp/qn-backup-$$.sqlite"
  if ! ssh -o BatchMode=yes "$BACKUP_SSH_HOST" \
        "docker cp '$BACKUP_CONTAINER:$remote_tmp' '$host_tmp'" >&2; then
    ssh -o BatchMode=yes "$BACKUP_SSH_HOST" "docker exec '$BACKUP_CONTAINER' rm -f '$remote_tmp'" >/dev/null 2>&1
    return 1
  fi

  # `cat` over ssh rather than scp: the caller passes an exact output path and
  # this keeps the writer and the exit status in one pipeline we control. The
  # redirect is on this side, so a failed transfer leaves a short file — which
  # is precisely what engine_verify below is for.
  if ! ssh -o BatchMode=yes "$BACKUP_SSH_HOST" "cat '$host_tmp'" >"$out"; then
    ssh -o BatchMode=yes "$BACKUP_SSH_HOST" \
      "rm -f '$host_tmp'; docker exec '$BACKUP_CONTAINER' rm -f '$remote_tmp'" >/dev/null 2>&1
    return 1
  fi

  # Both temporaries, on both sides, whatever happened above. A snapshot of the
  # whole lead table left in /tmp on a shared host is the kind of tidy-up that
  # only looks optional.
  ssh -o BatchMode=yes "$BACKUP_SSH_HOST" \
    "rm -f '$host_tmp'; docker exec '$BACKUP_CONTAINER' rm -f '$remote_tmp'" >/dev/null 2>&1

  return 0
}

# Reads the snapshot back and writes what it found to $2; anything sqlite3
# complains about goes to $3, kept separate so it cannot be counted as content.
#
# integrity_check FIRST, and its result is what decides. A truncated SQLite
# file will happily answer some queries — the header and the first pages are
# intact — so "the table list came back" is not evidence. `integrity_check`
# walks every page and says `ok` or says why not.
engine_verify() {
  local dump="$1" toc="$2" err="$3"
  local integrity

  integrity=$(sqlite3 "file:$dump?mode=ro" 'PRAGMA integrity_check;' 2>"$err") || return 1
  if [ "$integrity" != "ok" ]; then
    printf 'integrity_check did not return ok:\n%s\n' "$integrity" >>"$err"
    return 1
  fi

  # One line per table, plus its row count, so the summariser can count tables
  # and a human reading the log can see the lead count move. Internal
  # sqlite_% tables are excluded — they are not this project's data.
  sqlite3 "file:$dump?mode=ro" \
    "SELECT 'TABLE ' || name FROM sqlite_master
      WHERE type='table' AND name NOT LIKE 'sqlite\_%' ESCAPE '\'
      ORDER BY name;" >"$toc" 2>>"$err" || return 1

  return 0
}

# Prints "<table definitions> <data sections>".
#
# SQLite has no schema-only/data-only split, so a table and its data are the
# same object and both columns are the same number. Printing it twice rather
# than printing a zero is deliberate: the caller treats a zero in either column
# as "this dump has no data", which would be false here and would refuse every
# backup this script ever takes.
engine_summarise() {
  awk '$1 == "TABLE" { t++ } END { printf "%d %d\n", t + 0, t + 0 }' "$1"
}

# ===========================================================================
# End of ENGINE block.
# ===========================================================================

DRY_RUN=""
NO_PRUNE=""

for arg in "$@"; do
  case "$arg" in
    --dry-run)  DRY_RUN="yes" ;;
    --no-prune) NO_PRUNE="yes" ;;
    # No positional arguments exist, so one is a mistyped flag or a path
    # somebody expected this to accept. Either way, acting on the rest of the
    # command line as though it were fine is how a backup goes somewhere else.
    *) die "unknown argument '$arg'. Usage: bash scripts/backup.sh [--dry-run] [--no-prune]" ;;
  esac
done

# ---------------------------------------------------------------------------
# Refuse to run half-configured. Everything checkable without a side effect,
# checked before the first side effect, and named one at a time so the message
# says which value is wrong rather than "configuration error".
# ---------------------------------------------------------------------------

[ -n "$BACKUP_DIR" ]  || die "set BACKUP_DIR — the directory to write dumps into. See the CONFIGURATION block."
[ -n "$BACKUP_NAME" ] || die "set BACKUP_NAME — the filename prefix for this project's dumps. See the CONFIGURATION block."

# A relative path resolves against the working directory, and cron's working
# directory is not yours. The dumps would land somewhere else, and retention —
# globbing that same relative path — would find that other directory empty,
# report "nothing to remove", and let the real series grow until the disk
# filled. Both halves look healthy in isolation.
case "$BACKUP_DIR" in
  /*) : ;;
  *)  die "BACKUP_DIR ('$BACKUP_DIR') must be an absolute path — a relative one
      points somewhere different under cron than it does in your shell." ;;
esac

# The glob and the delete guard are built from this. A name containing `*`, `?`
# or `/` would widen the candidate list past this project's dumps, which is the
# one thing the guard cannot make safe by checking paths.
case "$BACKUP_NAME" in
  *[!A-Za-z0-9._-]*) die "BACKUP_NAME ('$BACKUP_NAME') may contain only letters, digits, dot,
      underscore and hyphen — it is used as a filename glob." ;;
esac

case "$KEEP" in
  ''|*[!0-9]*) die "BACKUP_KEEP must be a whole number, got '$KEEP'." ;;
esac

# Refused rather than clamped: a caller who typed 1 meant something, and it was
# not "destroy the only copy before checking the new one is worth having".
[ "$KEEP" -ge 2 ] || die "BACKUP_KEEP must be at least 2 — see the note above KEEP in this script."

case "$MIN_TABLES" in
  ''|*[!0-9]*) die "BACKUP_MIN_TABLES must be a whole number, got '$MIN_TABLES'." ;;
esac

case "$SHRINK_PCT" in
  ''|*[!0-9]*) die "BACKUP_SHRINK_PCT must be a whole number, got '$SHRINK_PCT'." ;;
esac

for tool in $(engine_tools); do
  command -v "$tool" >/dev/null 2>&1 \
    || die "$tool is not on PATH. Install the client tools for this engine
      (Debian/Ubuntu: postgresql-client) and run this again."
done

# awk is checked separately because it is not part of the engine and not part of
# coreutils: it is what turns the table of contents into a number, and cron's
# PATH is not your shell's. Without this check its absence surfaces further down
# as "the archive holds 0 table(s)" — a measurement, about the database, that
# nothing measured.
command -v awk >/dev/null 2>&1 \
  || die "awk is not on PATH. It is what counts what is inside the archive, and
      without it this script cannot tell a good dump from an empty one.
      Under cron, PATH is not the PATH your shell has."

[ -d "$BACKUP_DIR" ] || die "BACKUP_DIR ('$BACKUP_DIR') does not exist.
      Create it deliberately:  mkdir -p -m 700 '$BACKUP_DIR'
      It is not created here on purpose — a typo in the path is otherwise
      indistinguishable from a first run, and the typo'd directory would fill
      with a complete, correct-looking series that nobody restores from."

# Resolved once, here, and used for every path comparison afterwards. Comparing
# the configured string instead would let `/srv/backups/../../etc` pass a
# prefix test while being nowhere near the backup directory.
BACKUP_DIR_REAL=$(cd "$BACKUP_DIR" && pwd -P) || die "cannot resolve BACKUP_DIR ('$BACKUP_DIR')."

# Ahead of the writability check on purpose: for an unprivileged user / is not
# writable, so leaving this until later answers "permission denied" to somebody
# who is one `sudo` away from asking the same question again and being told yes.
[ "$BACKUP_DIR_REAL" != "/" ] || die "BACKUP_DIR resolves to / — retention would treat the root of the
      filesystem as a directory of expendable files."

[ -w "$BACKUP_DIR" ] || die "BACKUP_DIR ('$BACKUP_DIR') is not writable by $(id -un)."

# A dump inside the checkout is one `git add -A` from being committed, and one
# push from being public, with every row in the database in it. It is also
# copied by every clone and every deploy from then on. The engine that made it
# does not put it back once that happens.
repo_root=$(git rev-parse --show-toplevel 2>/dev/null) || repo_root=""

if [ -n "$repo_root" ] && repo_real=$(cd "$repo_root" && pwd -P); then
  case "$BACKUP_DIR_REAL/" in
    "$repo_real"/*)
      die "BACKUP_DIR ('$BACKUP_DIR_REAL') is inside the repository at $repo_real.
      Put it outside the checkout — a dump in a working tree is one 'git add -A'
      away from being committed and pushed." ;;
  esac
fi

# Permissions are reported, not corrected. Changing the mode of a directory the
# operator created — which may be a mount, or shared with a sync agent — is a
# decision this script does not get to make on their behalf; failing to mention
# that every row in the database is world-readable would be worse.
dir_mode=$(ls -ld "$BACKUP_DIR_REAL" 2>/dev/null | cut -c5-10)
case "$dir_mode" in
  ------) : ;;
  '')     say "note: could not read the permissions of $BACKUP_DIR_REAL." ;;
  *)      say "WARNING: $BACKUP_DIR_REAL is accessible to group or other (mode bits"
          say "         '$dir_mode'). A dump is every row in the database in one file."
          say "         chmod 700 it unless something else is meant to read these." ;;
esac

# ---------------------------------------------------------------------------
# Connection.
#
# ENGINE-SPECIFIC, and sitting outside the ENGINE block above because the
# template's own copy did. Its header says everything engine-specific lives in
# that one block; this precondition check is the exception, and it is noted here
# rather than quietly worked around so the next person changing engines knows
# there are two places, not one.
#
# The Postgres version of this refused to run when neither PGDATABASE nor
# DATABASE_URL was set, because pg_dump with no target connects to a database
# named after the current user and produces a valid, empty archive. The SQLite
# equivalent of that accident is different but not better: an `ssh` to the wrong
# host, or a container name that no longer exists, and this script's job is to
# tell those apart from an empty database before it writes anything.
#
# So all three coordinates are asserted, and the container is asked whether it
# is actually running. MIN_TABLES below is the second net; this is the first.
# ---------------------------------------------------------------------------

[ -n "$BACKUP_SSH_HOST" ]   || die "BACKUP_SSH_HOST is empty. It names the host the database lives on — nebula, unless you are rehearsing against something else."
[ -n "$BACKUP_CONTAINER" ]  || die "BACKUP_CONTAINER is empty. It names the container holding the database."
[ -n "$BACKUP_DB_PATH" ]    || die "BACKUP_DB_PATH is empty. It is the path to the database file INSIDE the container."

# Asked, not assumed. A stopped container answers this differently from a
# missing one, and both answer differently from a host that will not accept the
# connection — three problems with three fixes, which a single "backup failed"
# would flatten into one.
if ! remote_state=$(ssh -o BatchMode=yes -o ConnectTimeout=10 "$BACKUP_SSH_HOST" \
      "docker inspect -f '{{.State.Status}}' '$BACKUP_CONTAINER' 2>/dev/null" 2>/dev/null); then
  die "could not reach '$BACKUP_SSH_HOST' over ssh, or docker there would not answer.
      Nothing was backed up, and nothing is known about the database — this is
      not 'no changes to back up'."
fi

case "$remote_state" in
  running) ;;
  "")      die "no container named '$BACKUP_CONTAINER' on '$BACKUP_SSH_HOST'. Check the name
      before assuming the worst — a renamed container and a deleted one look
      identical from here." ;;
  *)       die "container '$BACKUP_CONTAINER' on '$BACKUP_SSH_HOST' is '$remote_state', not running.
      The online backup API needs the process alive. Start it, or take the
      snapshot from the volume directly while nothing is writing to it." ;;
esac

TARGET_DESC="${BACKUP_DB_PATH} in ${BACKUP_CONTAINER} on ${BACKUP_SSH_HOST}"

# ---------------------------------------------------------------------------
# Names. Built here so the dry run and the real run cannot describe different
# files.
#
# UTC, always: local time repeats an hour every autumn, which puts two dumps
# out of order by name in the one direction that matters — retention reads that
# order to decide what is oldest.
#
# Nothing derived from the connection goes into the filename. A URL in a name
# sits in the directory listing forever and is copied offsite with the dumps.
# ---------------------------------------------------------------------------

STAMP_GLOB='[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]T[0-9][0-9][0-9][0-9][0-9][0-9]Z'
STAMP=$(date -u +%Y%m%dT%H%M%SZ) || die "cannot read the clock."

FINAL="${BACKUP_DIR_REAL}/${BACKUP_NAME}-${STAMP}${DUMP_SUFFIX}"
PART="${FINAL}.part"

# Only a second run inside the same second reaches either of these, and the
# alternative in both cases is destroying a file this script promised to keep:
# a dump that has already been verified, or the .part evidence of a run that
# failed a moment ago and is about to be looked at.
[ ! -e "$FINAL" ] || die "$FINAL already exists. Refusing to overwrite a verified dump."
[ ! -e "$PART" ]  || die "$PART already exists — it is the unverified remains of a run that
      failed within this same second. Inspect or remove it before running again."

human_size() {
  local b="${1:-}"

  # An unmeasurable file is reported as unmeasured. Printing "0 B" for a file
  # whose size could not be read is the same lie as printing "nothing to
  # delete" for a directory that could not be listed.
  case "$b" in
    ''|*[!0-9]*) printf 'unknown size'; return 0 ;;
  esac

  # Integer arithmetic rather than `du -h`, whose output format and rounding
  # differ between GNU and BSD and which reports blocks allocated rather than
  # bytes written.
  if   [ "$b" -lt 1024 ];       then printf '%s B' "$b"
  elif [ "$b" -lt 1048576 ];    then printf '%s.%s KiB' "$((b / 1024))" "$((b * 10 / 1024 % 10))"
  elif [ "$b" -lt 1073741824 ]; then printf '%s.%s MiB' "$((b / 1048576))" "$((b * 10 / 1048576 % 10))"
  else                               printf '%s.%s GiB' "$((b / 1073741824))" "$((b * 10 / 1073741824 % 10))"
  fi
}

file_bytes() {
  # `wc -c` is portable where `stat` is not: GNU wants -c%s and BSD wants -f%z.
  # BSD pads the number with spaces, hence the tr.
  #
  # 2>/dev/null comes BEFORE the input redirect on purpose. Redirections are
  # applied left to right, so with the other order the shell's own "no such
  # file" message for a failed open is written before stderr has been silenced —
  # a raw `backup.sh: line N: …` in the middle of otherwise formatted output.
  # Callers read the empty result, which is what "unmeasurable" means here.
  wc -c 2>/dev/null <"$1" | tr -d ' \n'
}

# Every existing dump of this series, oldest first. Pathname expansion sorts,
# and LC_COLLATE=C above makes that sort byte-order, so no external sort is
# involved and no filename has to survive a round trip through word splitting.
#
# The glob is the exact shape this script writes — prefix, UTC stamp, suffix.
# Files that merely live in the directory are not candidates for anything: a
# `.part` from a failed run, a dump somebody copied in by hand, and another
# project's series all fail to match and are never counted or deleted.
list_dumps() {
  printf '%s\n' "${BACKUP_DIR_REAL}/${BACKUP_NAME}-"$STAMP_GLOB"$DUMP_SUFFIX"
}

# ---------------------------------------------------------------------------
# The delete guard.
#
# Called immediately before every `rm`, on the path about to be removed, and
# every check has to pass. It exists so that no configuration value, no symlink
# planted in the backup directory and no later edit to the glob can produce a
# deletion outside BACKUP_DIR_REAL — it does not trust the list it was given.
# ---------------------------------------------------------------------------
deletable() {
  local path="$1" parent base

  case "$path" in
    */*) parent="${path%/*}"; base="${path##*/}" ;;
    *)   return 1 ;;
  esac

  # Resolved, not string-compared: `.` and `..` components make a text prefix
  # test agree with a path that is somewhere else entirely.
  parent=$(cd "$parent" 2>/dev/null && pwd -P) || return 1
  [ "$parent" = "$BACKUP_DIR_REAL" ] || return 1

  # -f follows symlinks, so a link named like a dump would pass every other
  # check while pointing at a file the checks were never made about.
  [ -f "$path" ] || return 1
  [ ! -L "$path" ] || return 1

  case "$base" in
    "$BACKUP_NAME"-$STAMP_GLOB"$DUMP_SUFFIX") return 0 ;;
    *) return 1 ;;
  esac
}

# ---------------------------------------------------------------------------
# Retention.
#
# Last, and never fatal: by the time this runs the dump has been written and
# verified, and housekeeping must not be able to report that a backup which
# succeeded failed. Everything it declines to do, it says out loud instead.
# ---------------------------------------------------------------------------
prune() {
  [ -z "$NO_PRUNE" ] || { say "retention: skipped (--no-prune)."; return 0; }

  if [ ! -r "$BACKUP_DIR_REAL" ] || [ ! -x "$BACKUP_DIR_REAL" ]; then
    say "retention: SKIPPED — cannot list $BACKUP_DIR_REAL."
    say "           That is 'I could not look', which is not the same answer as"
    say "           'there is nothing to delete'. Nothing was removed."
    return 0
  fi

  local -a candidates=()
  local line
  while IFS= read -r line; do
    [ -n "$line" ] && candidates+=("$line")
  done <<EOF
$(list_dumps)
EOF

  local total=${#candidates[@]}
  local effective=$total

  # On a dry run the dump does not exist yet, so counting only what is on disk
  # would understate the total by one and under-report the deletions by one.
  # Counting it in is what makes --dry-run's list the list the real run uses.
  [ -z "$DRY_RUN" ] || effective=$((total + 1))

  if [ "$effective" -le "$KEEP" ]; then
    say "retention: ${effective} dump(s) in the series, keeping ${KEEP} — nothing to remove."
    return 0
  fi

  local doomed=$((effective - KEEP))
  local i removed=0 refused=0 failed=0 path

  for (( i = 0; i < doomed; i++ )); do
    path="${candidates[$i]}"

    if ! deletable "$path"; then
      # Loud, because a candidate that fails the guard means the directory
      # holds something the glob matched and the guard did not recognise —
      # a symlink, or a name shaped like ours pointing elsewhere.
      say "  REFUSED $path — not a plain file directly inside $BACKUP_DIR_REAL."
      refused=$((refused + 1))
      continue
    fi

    if [ -n "$DRY_RUN" ]; then
      say "  would delete $path ($(human_size "$(file_bytes "$path")"))"
      removed=$((removed + 1))
      continue
    fi

    if rm -f -- "$path" && [ ! -e "$path" ]; then
      say "  deleted $path"
      removed=$((removed + 1))
    else
      # Said, not counted as done. A retention pass that quietly failed looks
      # exactly like one that had nothing to do, right up until the disk fills.
      say "  FAILED to delete $path — left in place."
      failed=$((failed + 1))
    fi
  done

  if [ -n "$DRY_RUN" ]; then
    say "retention: would delete ${removed}, keeping the newest ${KEEP}."
  else
    say "retention: deleted ${removed}, keeping the newest ${KEEP}."
  fi

  [ "$refused" -eq 0 ] || say "retention: ${refused} candidate(s) refused by the delete guard, above."
  [ "$failed"  -eq 0 ] || say "retention: ${failed} deletion(s) failed — check permissions on $BACKUP_DIR_REAL."
}

# Reported, never deleted. A .part is the wreckage of a run that failed
# verification and it is usually the only record of why; retention leaves it
# alone by construction, so somebody has to be told it is there or it stays
# forever and is eventually mistaken for a backup.
report_parts() {
  local -a found=( "${BACKUP_DIR_REAL}/${BACKUP_NAME}-"*"${DUMP_SUFFIX}.part" )
  local -a stale=()
  local p

  # Filtered before it is counted, so the number printed is the number listed.
  for p in ${found[@]+"${found[@]}"}; do
    [ "$p" = "$PART" ] || stale+=("$p")
  done

  [ "${#stale[@]}" -gt 0 ] || return 0

  say "note: ${#stale[@]} unverified .part file(s) from earlier failed runs are in"
  say "      $BACKUP_DIR_REAL. They are never deleted by retention. Look at them,"
  say "      then remove them by hand:"

  for p in "${stale[@]}"; do
    say "        $p ($(human_size "$(file_bytes "$p")"))"
  done
}

# The newest verified dump before this run, for the size comparison and for
# telling the operator what they still have if this run fails.
PREV=""
PREV_BYTES=0

# Kept apart from PREV_BYTES because they answer different questions. "There is
# no previous dump" and "there is one and its size could not be read" are both
# reasons not to compare sizes, and collapsing the second into PREV_BYTES=0
# would make the script go on to say "no previous dump in this series" about a
# file that is sitting right there — the same substitution of absence for
# measurement it refuses to make about free space and table counts.
PREV_MEASURED=""

while IFS= read -r line; do
  [ -n "$line" ] && PREV="$line"
done <<EOF
$(list_dumps)
EOF

if [ -n "$PREV" ]; then
  PREV_BYTES=$(file_bytes "$PREV")
  case "$PREV_BYTES" in
    ''|*[!0-9]*) PREV_BYTES=0 ;;
    *)           PREV_MEASURED="yes" ;;
  esac
fi

# ---------------------------------------------------------------------------
# Free space, checked against something real or not claimed at all.
#
# With no previous dump there is no basis for an estimate, and printing a
# reassuring number derived from nothing is worse than printing none.
# ---------------------------------------------------------------------------
free_kb=$(df -Pk "$BACKUP_DIR_REAL" 2>/dev/null | awk 'NR == 2 { print $4 }')

case "$free_kb" in
  ''|*[!0-9]*)
    say "note: free space on $BACKUP_DIR_REAL could not be read — not checked." ;;
  *)
    if [ -z "$PREV_MEASURED" ] || [ "$PREV_BYTES" -eq 0 ]; then
      say "note: $(human_size $((free_kb * 1024))) free; no measured previous dump to compare"
      say "      against, so whether that is enough is unknown."
    elif [ $((free_kb * 1024)) -lt $((PREV_BYTES * 2)) ]; then
      say "WARNING: $(human_size $((free_kb * 1024))) free, and the last dump was"
      say "         $(human_size "$PREV_BYTES"). This run may not fit."
    fi ;;
esac

# ---------------------------------------------------------------------------
# --dry-run stops here, having touched nothing.
# ---------------------------------------------------------------------------

if [ -n "$DRY_RUN" ]; then
  say "--dry-run: nothing was written or deleted. It would have:"
  say "  dumped ${TARGET_DESC}"
  say "  written ${PART}"
  say "  verified it with: sqlite3 ${PART} 'PRAGMA integrity_check' + a table list"
  say "  renamed it to ${FINAL}"
  say ""
  say "and then applied retention, shown here for real because it deletes:"
  prune
  report_parts
  exit 0
fi

# ---------------------------------------------------------------------------
# One run at a time.
#
# mkdir is atomic on every filesystem this will meet, which flock and lock
# files are not. Two overlapping runs — a nightly job on a database that now
# takes longer than a day to dump — double the load on the server and race each
# other's retention pass.
#
# A stale lock stops backups, so this exits NON-ZERO and names the fix rather
# than skipping quietly. A skipped backup that reports success is the exact
# failure this whole script exists to prevent, and a lock is not allowed to
# reintroduce it.
# ---------------------------------------------------------------------------

LOCK_DIR="${BACKUP_DIR_REAL}/.${BACKUP_NAME}.lock"
LOCK_HELD=""

# mkdir's error is captured rather than discarded, because "the lock exists" and
# "the lock could not be created" are different failures with different fixes,
# and only the first one is about concurrency. A read-only mount, a full
# filesystem, or a missing mkdir under cron's PATH all make mkdir fail too; the
# directory is what distinguishes them, so it is looked at instead of assumed.
LOCK_ERR=""

if ! LOCK_ERR=$(mkdir "$LOCK_DIR" 2>&1); then
  if [ -d "$LOCK_DIR" ]; then
    say "another run holds the lock at $LOCK_DIR"
    [ ! -r "$LOCK_DIR/owner" ] || say "  owner: $(cat "$LOCK_DIR/owner" 2>/dev/null)"
    die "if no backup is running, that lock is stale: rmdir '$LOCK_DIR'"
  fi

  # No lock directory, so nothing is holding anything. Saying otherwise here
  # would send somebody to rmdir a path that does not exist, watch that change
  # nothing, and go looking for a second backup process — while the real cause
  # sits unread in mkdir's own message and every night's backup stays dead.
  say "could not create the lock directory $LOCK_DIR, and no lock is present —"
  say "      so this is NOT another run. The directory could not be made at all."
  [ -z "$LOCK_ERR" ] || say "      mkdir said: $LOCK_ERR"
  die "check that $BACKUP_DIR_REAL is on a writable, non-full filesystem."
fi

LOCK_HELD="yes"
WORK=""

# Armed in the same breath as the lock, not after the next command. Anything
# that exits between acquiring the lock and installing this trap leaves a stale
# lock behind, and a stale lock stops every future run — the trap tolerating an
# empty WORK is much cheaper than that. $WORK is quoted-empty-safe because rm
# is guarded on it.
trap '[ -z "$LOCK_HELD" ] || rm -rf "$LOCK_DIR"; [ -z "$WORK" ] || rm -rf "$WORK"' EXIT

printf 'pid %s on %s since %s\n' "$$" "$(hostname 2>/dev/null || echo unknown)" \
  "$(date -u +%Y-%m-%dT%H:%M:%SZ)" >"$LOCK_DIR/owner" 2>/dev/null

# The .part is deliberately NOT cleaned up by that trap. It is evidence when
# verification fails, and the failure paths below print where it is.
WORK=$(mktemp -d) || die "cannot create a temporary directory"

# ---------------------------------------------------------------------------
# Dump.
# ---------------------------------------------------------------------------

say "dumping ${TARGET_DESC} -> ${PART}"

# umask in a subshell so the archive is created 0600 from its first byte. Doing
# it with a chmod afterwards leaves a window in which every row in the database
# is readable by every user on the host, and that window is the whole dump.
if ! ( umask 077; engine_dump "$PART" ); then
  say "the snapshot failed (the error from ssh/docker/better-sqlite3 is above)."
  say "Nothing was renamed into"
  say "      place and nothing was deleted."
  if [ -n "$PREV" ]; then
    say "      The newest verified dump is still $PREV"
  else
    say "      There is no previously verified dump in $BACKUP_DIR_REAL."
  fi
  say "      The partial file is kept for inspection: $PART"
  exit 1
fi

[ -f "$PART" ] || die "the snapshot reported success but $PART does not exist. Nothing was renamed."

BYTES=$(file_bytes "$PART")
[ -n "$BYTES" ] || die "cannot measure $PART. It has NOT been renamed into place."

# ---------------------------------------------------------------------------
# Verify, before the file is allowed to become the backup.
# ---------------------------------------------------------------------------

say "verifying ${PART}…"

LISTING="$WORK/toc"
VERR="$WORK/toc.err"

if ! engine_verify "$PART" "$LISTING" "$VERR"; then
  say "VERIFICATION FAILED — the file cannot be read back as an archive."
  say "      $(human_size "$BYTES") written. The reader said:"
  sed -n '1,10p' "$VERR" >&2 2>/dev/null
  say ""
  say "      It has NOT been renamed into place and NOTHING was deleted."
  say "      Kept for inspection: $PART"
  if [ -n "$PREV" ]; then
    say "      The newest verified dump is still $PREV"
  else
    say "      There is no previously verified dump in $BACKUP_DIR_REAL."
  fi
  exit 1
fi

SUMMARY=$(engine_summarise "$LISTING") || SUMMARY=""

read -r TABLES DATA_SECTIONS <<EOF
$SUMMARY
EOF

TABLES="${TABLES:-}"
DATA_SECTIONS="${DATA_SECTIONS:-}"

# engine_summarise is the one thing in this script an engine swap is expected to
# replace, so its result is checked and not trusted. Two whole numbers or this
# run does not continue.
#
# Defaulting the empties to 0 instead — which is what this used to do — turns
# every way the count can fail to happen into the sentence "the archive holds 0
# tables", which is a claim about the database. A missing awk produces it. So
# does a replacement summariser that prints a warning, or one number, or an
# error. And a NON-numeric result is worse than misleading: `[ "$TABLES" -gt 0 ]`
# exits 2 with "integer expression expected", `if` treats that as false, the
# MIN_TABLES gate below does the same, and the run walks straight past its only
# real check into the rename — announcing a dump nobody counted as verified, and
# then deleting older dumps that were.
SUMMARY_BAD=""
case "$TABLES"        in ''|*[!0-9]*) SUMMARY_BAD="yes" ;; esac
case "$DATA_SECTIONS" in ''|*[!0-9]*) SUMMARY_BAD="yes" ;; esac

if [ -n "$SUMMARY_BAD" ]; then
  say "VERIFICATION INCONCLUSIVE — the archive was read back successfully, but"
  say "      counting what is in it did not produce two whole numbers. How much"
  say "      this dump contains is therefore UNKNOWN."
  say ""
  say "      Unknown is not zero and this is not a statement about the database."
  say "      Look at engine_summarise, and at whether awk is on PATH — under cron"
  say "      PATH is not the PATH your shell has."
  say "      The summariser returned: '${SUMMARY%%$'\n'*}'"
  say ""
  say "      It has NOT been renamed into place and NOTHING was deleted."
  say "      Kept for inspection: $PART"
  if [ -n "$PREV" ]; then
    say "      The newest verified dump is still $PREV"
  else
    say "      There is no previously verified dump in $BACKUP_DIR_REAL."
  fi
  exit 1
fi

# Which number is the real one depends on what kind of dump this is, and saying
# "0 tables" about a data-only archive that contains every row would be exactly
# the sort of unmeasured claim this script is built to avoid.
if [ "$TABLES" -gt 0 ]; then
  COUNT="$TABLES"
  COUNT_KIND="tables"
else
  COUNT="$DATA_SECTIONS"
  COUNT_KIND="table data sections (the archive has no table definitions — a data-only dump)"
fi

if [ "$COUNT" -lt "$MIN_TABLES" ]; then
  say "VERIFICATION FAILED — the archive is readable but holds ${COUNT} table(s),"
  say "      fewer than BACKUP_MIN_TABLES=${MIN_TABLES}."
  say "      This is what a dump of the wrong or an empty database looks like:"
  say "      valid, small, and worthless. The connection was ${TARGET_DESC} —"
  say "      check which database that actually reaches."
  say ""
  say "      It has NOT been renamed into place and NOTHING was deleted."
  say "      Kept for inspection: $PART"
  exit 1
fi

# ---------------------------------------------------------------------------
# Rename. Within one directory, so it is rename(2): the final name appears
# whole or not at all, and no reader can ever see a partial dump under it.
# ---------------------------------------------------------------------------

mv -- "$PART" "$FINAL" || die "could not rename $PART to $FINAL. The verified dump is still at
      $PART — move it into place by hand."

[ -f "$FINAL" ] || die "mv reported success but $FINAL is not there."

say "wrote ${FINAL}"
say "      $(human_size "$BYTES"), ${COUNT} ${COUNT_KIND}, verified readable."

# ---------------------------------------------------------------------------
# Compare against the previous dump. A warning, never a refusal — see SHRINK_PCT.
# ---------------------------------------------------------------------------

# Three ways there is no comparison to make, said as three different things.
# Rule of the house: absence is reported as absence, and "I could not measure
# the file I was comparing against" is not "there was no file".
if [ -z "$PREV" ]; then
  say "note: no previous dump in this series — size not compared."
elif [ -z "$PREV_MEASURED" ]; then
  say "note: the previous dump could not be measured, so this dump's size was"
  say "      NOT compared against it. Not compared is not the same as unchanged."
  say "      Previous: $PREV"
elif [ "$PREV_BYTES" -eq 0 ]; then
  say "note: the previous dump is zero bytes ($PREV), so there is no ratio to"
  say "      compare against. Whatever went wrong went wrong before this run."
else
  pct=$((BYTES * 100 / PREV_BYTES))

  if [ "$pct" -lt "$SHRINK_PCT" ]; then
    say "WARNING: this dump is ${pct}% of the previous one"
    say "         ($(human_size "$BYTES") against $(human_size "$PREV_BYTES"))."
    say "         A dump that shrinks like that is usually a lost permission on"
    say "         some schema, a partial dump, or the wrong database. The"
    say "         previous file has been kept; compare the two table lists with"
    say "         sqlite3 'PRAGMA integrity_check' before relying on this one."
  fi
fi

prune
report_parts

say "note: this wrote a verified dump to THIS machine. It did not copy it"
say "      anywhere else, and a backup that only exists beside the database it"
say "      came from does not survive the disk, the host or the account."
say "      Restoring it has still never been tried; see the header."