1067 lines
48 KiB
Bash
Executable File
1067 lines
48 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
#
|
|
# Apply numbered SQL migrations to PostgreSQL, in numeric order, exactly once
|
|
# each — and refuse to touch the database at all if a migration that was already
|
|
# applied has since been edited.
|
|
#
|
|
# ./scripts/migrate.sh # apply everything pending
|
|
# ./scripts/migrate.sh --status # what is applied, what is pending, exit
|
|
# ./scripts/migrate.sh --dry-run # exactly what would run, in order
|
|
# ./scripts/migrate.sh --help
|
|
#
|
|
# MIGRATE_LOCK_WAIT=120 ./scripts/migrate.sh # wait longer for a concurrent run
|
|
# MIGRATE_ALLOW_MISSING=1 ./scripts/migrate.sh # after a deliberate squash
|
|
#
|
|
# Exit codes, so CI can tell these apart:
|
|
# 0 nothing pending, or everything pending was applied
|
|
# 1 misconfigured, or a check could not be performed
|
|
# 2 refused — an edited migration, a duplicate number, an unusable file
|
|
# 3 another migration run holds the lock
|
|
# 4 a migration failed while running
|
|
#
|
|
# ===========================================================================
|
|
# TEMPLATE COPY — configure this before the first run
|
|
# ===========================================================================
|
|
#
|
|
# Copy to `scripts/migrate.sh` and set the two values in the CONFIGURATION
|
|
# block below. The script refuses to run until they are set: it has no
|
|
# defaults, deliberately.
|
|
#
|
|
# `migrations/` looks like a safe default for the directory and
|
|
# `schema_migrations` looks like a safe default for the tracking table, and
|
|
# both are traps. A directory default means a copy of this script run from the
|
|
# wrong checkout applies another project's DDL to this project's database. A
|
|
# tracking-table default means two projects sharing one database share one
|
|
# ledger: project B reads project A's rows, concludes its own 0001 through 0009
|
|
# are already applied, and never applies them — and nothing anywhere reports an
|
|
# error. Neither failure is visible until something reads a column that was
|
|
# never created.
|
|
#
|
|
# Assumes: bash, coreutils, git (optional), and `psql` on PATH. No jq, no
|
|
# language runtime, no migration framework.
|
|
#
|
|
# ## Why this exists
|
|
#
|
|
# Between `psql -f` typed by hand and a framework's migrator there is a gap
|
|
# that most projects live in, and the gap is where databases quietly diverge.
|
|
# Applying files by hand means the record of what ran lives in somebody's shell
|
|
# history; the second environment gets a different subset and nobody finds out
|
|
# until a query fails in production against a column that exists on the laptop.
|
|
#
|
|
# What closes that gap is not automation, it is a ledger the database itself
|
|
# carries: a row per migration, with the checksum of the file that produced it.
|
|
# Everything else here — the ordering, the lock, the transactions — exists to
|
|
# keep that ledger true.
|
|
#
|
|
# ## Immutability is the whole point
|
|
#
|
|
# A migration that has been applied is history. It is not a source file any
|
|
# more, and editing it is editing the past of every database that already ran
|
|
# it. So the checksum of every applied file is compared against the recorded
|
|
# one on every run, and a mismatch stops the run and names the file.
|
|
#
|
|
# This is the sharpest failure this script exists to prevent, because the
|
|
# unguarded version of it is silent in both directions. A runner that tracks
|
|
# only filenames will not re-apply the edited file — so the database that ran
|
|
# it yesterday keeps the old definition, the database created tomorrow gets the
|
|
# new one, and the two are now different databases with identical ledgers. A
|
|
# runner that re-applies it instead will run the edit against a schema that
|
|
# already has the original, which usually errors, and occasionally does not.
|
|
#
|
|
# The fix for a migration that was wrong is never an edit. It is a new
|
|
# migration with a higher number.
|
|
#
|
|
# ## One transaction per migration, including the tracking row
|
|
#
|
|
# Each migration and the INSERT that claims it was applied commit together. A
|
|
# failure therefore leaves neither the change nor the claim — no half-applied
|
|
# schema wearing a row that says it is done, which is the state that needs a
|
|
# human with a psql prompt to untangle.
|
|
#
|
|
# Some statements cannot run inside a transaction: CREATE INDEX CONCURRENTLY
|
|
# and DROP INDEX CONCURRENTLY are the ones that come up, and ALTER TYPE ... ADD
|
|
# VALUE on older servers. Mark such a file by putting this line anywhere in it:
|
|
#
|
|
# -- migrate: no-transaction
|
|
#
|
|
# and it runs with autocommit, with its tracking row inserted afterwards as a
|
|
# separate statement. That is strictly weaker and the script says so out loud
|
|
# each time: if the migration succeeds and the INSERT does not, the next run
|
|
# will try to apply it again. Use the marker only for statements that require
|
|
# it, and put nothing else in the file.
|
|
#
|
|
# ## What it deliberately does not do
|
|
#
|
|
# It has no `down` migration and it never rolls back a migration that already
|
|
# committed. Both are the same refusal. A down migration is written at the
|
|
# moment it is not needed, reviewed less carefully than the up, and then run
|
|
# for the first time on the one day everything is already on fire — which is
|
|
# also the first time anyone finds out it does not work, or that it drops a
|
|
# column holding data written since. Reversal that has never been executed is
|
|
# not a safety net, it is a second untested migration wearing one.
|
|
#
|
|
# Recovery here is a new forward migration, or the backup. Both are things that
|
|
# get tested.
|
|
#
|
|
# It also does not create the database, does not manage roles or extensions
|
|
# beyond whatever the migrations themselves do, and does not deploy anything.
|
|
# It reads a directory and writes to one database, and that is all.
|
|
|
|
set -uo pipefail
|
|
|
|
# Run from the repository root when there is one, so that a relative
|
|
# MIGRATIONS_DIR means the same directory whether this was invoked from the
|
|
# root or from a subdirectory. Not required: this also runs inside a deployed
|
|
# container where there is no checkout at all, so a missing git is not fatal.
|
|
if root=$(git rev-parse --show-toplevel 2>/dev/null) && [ -n "$root" ]; then
|
|
cd "$root" || exit 1
|
|
fi
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# CONFIGURATION — set these two, then delete this banner.
|
|
#
|
|
# Both are empty on purpose. See the note at the top: a directory default
|
|
# points this at another project's SQL, and a tracking-table default makes two
|
|
# projects sharing a database share one ledger. Both failures are silent.
|
|
# ---------------------------------------------------------------------------
|
|
|
|
# Directory holding the numbered migrations, e.g. db/migrations
|
|
MIGRATIONS_DIR="${MIGRATE_DIR:-}"
|
|
|
|
# Table that records what has been applied. Schema-qualify it if this project
|
|
# does not own the search_path, e.g. myapp.schema_migrations
|
|
TRACKING_TABLE="${MIGRATE_TABLE:-}"
|
|
|
|
say() { printf '\033[1mmigrate:\033[0m %s\n' "$*" >&2; }
|
|
die() { printf '\033[1mmigrate:\033[0m %s\n' "$*" >&2; exit 1; }
|
|
|
|
# Exit 2 rather than 1, and kept apart from die() for the whole file: "this is
|
|
# not configured" and "this is configured and I am refusing" are different
|
|
# answers, and CI that retries the first must not retry the second.
|
|
refuse() { printf '\033[1mmigrate:\033[0m %s\n' "$*" >&2; exit 2; }
|
|
|
|
MODE="apply"
|
|
|
|
# Arguments are read BEFORE the configuration is checked, so that --help works
|
|
# on an unconfigured copy. This file ships unconfigured on purpose, so the
|
|
# state somebody most needs to read the help in is the state it arrives in, and
|
|
# a --help that answers "set MIGRATIONS_DIR first" answers a question nobody
|
|
# asked. It also means an unknown argument is reported as an unknown argument.
|
|
#
|
|
# ${1+"$@"} rather than "$@": with no positional parameters and `set -u`, bash
|
|
# before 4.4 treats "$@" as unset and aborts — and bash 3.2 is what /bin/bash
|
|
# is on macOS, which the shasum fallback above exists to support. The ${1+...}
|
|
# form expands to nothing when there are no arguments, and to exactly "$@"
|
|
# when there are.
|
|
for arg in ${1+"$@"}; do
|
|
case "$arg" in
|
|
--status) MODE="status" ;;
|
|
--dry-run) MODE="dry-run" ;;
|
|
--help|-h)
|
|
printf '%s\n' \
|
|
"usage: migrate.sh [--status | --dry-run]" \
|
|
"" \
|
|
" (no flags) apply every pending migration, in numeric order" \
|
|
" --status print applied and pending migrations, then exit" \
|
|
" --dry-run print exactly what would be applied, and change nothing" \
|
|
"" \
|
|
"Required, and deliberately without defaults — see the CONFIGURATION" \
|
|
"block in this file for why:" \
|
|
"" \
|
|
" MIGRATE_DIR=<path> directory holding the numbered .sql files" \
|
|
" MIGRATE_TABLE=<name> table recording what has been applied" \
|
|
"" \
|
|
"Connection comes from the environment: DATABASE_URL, or the standard" \
|
|
"libpq variables (PGHOST, PGPORT, PGDATABASE, PGUSER, PGPASSWORD)." \
|
|
"" \
|
|
"Prefer the libpq variables on a shared machine. DATABASE_URL is passed" \
|
|
"to psql as an argument, and arguments are world-readable in ps and" \
|
|
"/proc for the life of the run — so the password is visible to any" \
|
|
"local user while a migration is applying. PGPASSWORD is not." \
|
|
"" \
|
|
" MIGRATE_LOCK_WAIT=<seconds> how long to wait for a concurrent run (30)" \
|
|
" MIGRATE_PG_LOCK_TIMEOUT=<value> server lock_timeout, e.g. '5s' (unset)" \
|
|
" MIGRATE_ALLOW_MISSING=1 proceed when an applied migration has no file" \
|
|
" MIGRATE_ALLOW_OUT_OF_ORDER=1 proceed when a pending number is below the" \
|
|
" highest applied" >&2
|
|
exit 0 ;;
|
|
*) die "unknown argument '$arg'. Usage: migrate.sh [--status | --dry-run]" ;;
|
|
esac
|
|
done
|
|
|
|
# Refused rather than defaulted, and named one at a time so the message says
|
|
# which value is missing rather than "configuration error".
|
|
[ -n "$MIGRATIONS_DIR" ] || die "set MIGRATIONS_DIR (or MIGRATE_DIR) — the directory holding the numbered .sql files. See the CONFIGURATION block."
|
|
[ -n "$TRACKING_TABLE" ] || die "set TRACKING_TABLE (or MIGRATE_TABLE) — the table recording what has been applied. See the CONFIGURATION block."
|
|
|
|
# --status is a read-only diagnostic, and an edited migration, a missing file
|
|
# and an out-of-order pending number are precisely the three states somebody
|
|
# runs it to understand. Exiting before the report is refusing to answer the
|
|
# question that was asked — and worse, the failure text at the bottom of this
|
|
# script sends operators to --status by name to find out which migrations
|
|
# already succeeded, which is unreachable if drift makes --status refuse.
|
|
#
|
|
# So in status mode a refusal is recorded instead of taken, the report is still
|
|
# printed, and the exit code is still 2 afterwards — CI cannot read it as
|
|
# success, and a human gets the picture they asked for.
|
|
STATUS_EXIT=0
|
|
|
|
refuse_or_defer() {
|
|
if [ "$MODE" = "status" ]; then
|
|
STATUS_EXIT=2
|
|
say "(--status continues so it can still show you the state below; a real run stops here.)"
|
|
return 0
|
|
fi
|
|
exit 2
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Tools, checked before anything is read or connected to.
|
|
# ---------------------------------------------------------------------------
|
|
|
|
command -v psql >/dev/null 2>&1 || die "psql is not on PATH. Install the PostgreSQL client, or run this from an image that has one."
|
|
|
|
# \if and \gset are how each migration re-checks the ledger after the lock is
|
|
# held (see "the plan is re-checked" below). psql gained \if in 10; on anything
|
|
# older the conditional blocks are ignored rather than obeyed, which would run
|
|
# every migration in the plan unguarded. Refused rather than degraded.
|
|
psql_major=$(psql --version 2>/dev/null | sed -E 's/[^0-9]*([0-9]+).*/\1/')
|
|
|
|
case "$psql_major" in
|
|
''|*[!0-9]*) die "could not read the psql version ('$(psql --version 2>&1 | head -n 1)'). This needs psql 10 or newer for \\if." ;;
|
|
esac
|
|
|
|
[ "$psql_major" -ge 10 ] || die "psql ${psql_major} is too old: \\if arrived in psql 10, and without it the re-check that stops a concurrent run re-applying a migration is silently ignored."
|
|
|
|
# sha256sum is coreutils; shasum ships on macOS where coreutils does not. The
|
|
# checksum column is the load-bearing part of this script, so which tool
|
|
# produced it is decided once, here, rather than per call — two call sites
|
|
# picking different tools would produce two different hashes of one file and
|
|
# every migration would look edited.
|
|
if command -v sha256sum >/dev/null 2>&1; then
|
|
sha256_stream() { sha256sum | cut -d' ' -f1; }
|
|
elif command -v shasum >/dev/null 2>&1; then
|
|
sha256_stream() { shasum -a 256 | cut -d' ' -f1; }
|
|
else
|
|
die "no sha256 tool found (looked for sha256sum and shasum). Migrations cannot be checksummed, so this cannot verify that applied files are unchanged."
|
|
fi
|
|
|
|
# Hashing stdin rather than a named file keeps the filename out of the digest,
|
|
# which is what makes a rename a rename instead of a new migration.
|
|
sha256_of() { sha256_stream < "$1"; }
|
|
|
|
work=$(mktemp -d) || die "cannot create a temporary directory"
|
|
|
|
# The generated SQL is kept when a migration fails, and only then. psql reports
|
|
# errors as "run.sql:LINE", and deleting the file it is pointing at turns the
|
|
# one piece of evidence about what actually ran into a dangling path.
|
|
KEEP_WORK=""
|
|
|
|
# The only rm in this script, and the belt-and-braces on it is deliberate:
|
|
# $work can only ever be what mktemp -d printed (the assignment above dies
|
|
# otherwise, and this trap is not installed until after it), but `rm -rf` is
|
|
# not a place to reason from "can only ever". -d re-confirms it is a directory
|
|
# that exists, so an empty or clobbered $work removes nothing.
|
|
trap '[ -n "$KEEP_WORK" ] || { [ -n "$work" ] && [ -d "$work" ] && rm -rf "$work"; }' EXIT
|
|
|
|
TAB=$(printf '\t')
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Validate the configured names before they reach SQL.
|
|
#
|
|
# The table name is interpolated into statements unquoted, so it is checked
|
|
# against what a bare identifier may contain and refused otherwise. This is not
|
|
# hostile-input defence — it is configuration — but a name needing quoting
|
|
# would break at a different place each time it was used, and the failure would
|
|
# look like a syntax error inside somebody's migration.
|
|
# ---------------------------------------------------------------------------
|
|
|
|
printf '%s' "$TRACKING_TABLE" \
|
|
| grep -Eq '^[a-zA-Z_][a-zA-Z0-9_]*(\.[a-zA-Z_][a-zA-Z0-9_]*)?$' \
|
|
|| die "TRACKING_TABLE ('$TRACKING_TABLE') is not a plain identifier. Use name or schema.name, letters, digits and underscore only."
|
|
|
|
[ -d "$MIGRATIONS_DIR" ] || die "MIGRATIONS_DIR ('$MIGRATIONS_DIR') is not a directory. Create it, or point MIGRATE_DIR at the right one — this will not guess."
|
|
|
|
MIGRATIONS_ABS=$(cd "$MIGRATIONS_DIR" && pwd) || die "cannot resolve MIGRATIONS_DIR ('$MIGRATIONS_DIR')."
|
|
|
|
# psql's \i takes a single-quoted path, and its quoting rules are not the
|
|
# shell's: inside single quotes psql itself expands \t, \n, \r, \b, \f, \NNN
|
|
# and \xNN before the path ever reaches the filesystem. So all three of these
|
|
# produce a valid-looking script that addresses the wrong file, and all three
|
|
# are refused rather than guessed at.
|
|
#
|
|
# The backslash case is the one that looks harmless and is not. A directory
|
|
# named `tab\test` is handed to psql as `tab\test` and opened as `tab<TAB>est`
|
|
# — which usually does not exist, and the run fails claiming the MIGRATION is
|
|
# broken when the path is. It does not have to not-exist, either: a path that
|
|
# escapes into another real directory includes another project's SQL, silently.
|
|
#
|
|
# Spaces are fine and stay fine — the single quotes handle them, and the
|
|
# directories this lands in have spaces in their names.
|
|
case "$MIGRATIONS_ABS" in
|
|
*"'"*|*'"'*)
|
|
die "the migrations path ('$MIGRATIONS_ABS') contains a quote character. psql's \\i cannot address it safely; move the directory." ;;
|
|
*\\*)
|
|
die "the migrations path ('$MIGRATIONS_ABS') contains a backslash. psql expands backslash escapes inside \\i's quoted argument, so this would open a different path than the one on disk; move the directory." ;;
|
|
*[[:cntrl:]]*)
|
|
die "the migrations path contains a control character (a newline or a tab). psql's \\i cannot address it; move the directory." ;;
|
|
esac
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Read the directory.
|
|
#
|
|
# Everything here is checkable without a database and is checked before one is
|
|
# opened, so a badly-named file is a failure that costs nothing rather than one
|
|
# discovered halfway through a deploy.
|
|
# ---------------------------------------------------------------------------
|
|
|
|
FILES_TSV="$work/files.tsv" # version <TAB> filename <TAB> checksum <TAB> transactional
|
|
: > "$FILES_TSV"
|
|
|
|
for path in "$MIGRATIONS_ABS"/*; do
|
|
[ -f "$path" ] || continue
|
|
|
|
base=${path##*/}
|
|
|
|
case "$base" in
|
|
*.sql) ;;
|
|
*)
|
|
# Said out loud rather than passed over. Only *.sql is a migration, and a
|
|
# file that is being ignored should say so on the run that ignores it —
|
|
# "0007_x.sql.bak" and "0007_x.sql" look identical in a code review.
|
|
say "note: ignoring $base — only *.sql files are migrations."
|
|
continue ;;
|
|
esac
|
|
|
|
# Refused, not skipped. A .sql file this script cannot number is a file that
|
|
# will never be applied and will never be mentioned again, which is the exact
|
|
# failure the ledger exists to prevent.
|
|
case "$base" in
|
|
[0-9]*_*) ;;
|
|
*) refuse "'$base' is not named <number>_<name>.sql, so it would never be applied. Rename it, or move it out of $MIGRATIONS_DIR." ;;
|
|
esac
|
|
|
|
case "$base" in
|
|
*[!A-Za-z0-9._-]*) refuse "'$base' contains a character outside A-Z a-z 0-9 . _ - . psql's \\i cannot address it reliably; rename it." ;;
|
|
esac
|
|
|
|
num=${base%%_*}
|
|
case "$num" in
|
|
*[!0-9]*) refuse "'$base' has a non-numeric prefix ('$num'). The prefix decides the order, so it must be digits only." ;;
|
|
esac
|
|
|
|
# 10# forces base ten. Without it, $((0008)) is an invalid octal literal and
|
|
# the eighth migration of any project stops the run with an arithmetic error.
|
|
version=$((10#$num))
|
|
|
|
checksum=$(sha256_of "$path") || die "could not checksum $base."
|
|
[ -n "$checksum" ] || die "empty checksum for $base — refusing to record a migration it cannot identify."
|
|
|
|
if grep -Eq '^[[:space:]]*--[[:space:]]*migrate:[[:space:]]*no-transaction[[:space:]]*$' "$path"; then
|
|
transactional="no"
|
|
else
|
|
transactional="yes"
|
|
fi
|
|
|
|
# A file whose last statement has no terminating semicolon is refused,
|
|
# because psql discards an unterminated buffer at end of file without
|
|
# complaining: the migration would run everything up to the last semicolon,
|
|
# succeed, and be recorded as applied with its final statement never sent.
|
|
# Trailing blank lines and trailing comment lines are ignored.
|
|
last=$(grep -vE '^[[:space:]]*(--.*)?$' "$path" | tail -n 1)
|
|
|
|
if [ -z "$last" ]; then
|
|
refuse "'$base' contains no SQL. Delete it, or give it a body — an empty migration would be recorded as applied and could never be filled in."
|
|
fi
|
|
|
|
printf '%s' "$last" | grep -Eq ';[[:space:]]*(--.*)?$' \
|
|
|| refuse "the last statement in '$base' has no terminating semicolon. psql drops an unterminated statement at end of file without an error, so this would be recorded as applied while its final statement never ran."
|
|
|
|
# Only for transactional files: this script opens the transaction, and a
|
|
# BEGIN or COMMIT inside the file would close it early — committing part of
|
|
# the migration and leaving the tracking INSERT in a transaction of its own.
|
|
# A file that genuinely manages its own transactions declares that with the
|
|
# no-transaction marker, so it is exempt.
|
|
#
|
|
# The semicolon in the pattern is load-bearing: plpgsql bodies open with a
|
|
# bare `BEGIN` and close with `END;`, and matching those would refuse every
|
|
# function definition in the project. `END` is left out of the pattern
|
|
# entirely for that reason.
|
|
if [ "$transactional" = "yes" ] \
|
|
&& grep -Eqi '^[[:space:]]*(BEGIN|COMMIT|ROLLBACK)[[:space:]]*;' "$path"; then
|
|
refuse "'$base' contains its own BEGIN;, COMMIT; or ROLLBACK;. This runner wraps each migration in a transaction together with its tracking row, and a nested one would commit half of it. Remove them — or, if this is a procedure body that must COMMIT, add '-- migrate: no-transaction' and take responsibility for the transaction yourself."
|
|
fi
|
|
|
|
printf '%s\t%s\t%s\t%s\n' "$version" "$base" "$checksum" "$transactional" >> "$FILES_TSV"
|
|
done
|
|
|
|
sort -t "$TAB" -k1,1n "$FILES_TSV" -o "$FILES_TSV" || die "could not sort the migration list."
|
|
|
|
TOTAL=$(grep -c . "$FILES_TSV")
|
|
|
|
if [ "$TOTAL" -eq 0 ]; then
|
|
say "no migrations in $MIGRATIONS_DIR — nothing to do."
|
|
exit 0
|
|
fi
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Duplicate numbers are refused; gaps are reported.
|
|
#
|
|
# Two files sharing a number is not a style problem: the ledger is keyed on the
|
|
# number, so whichever ran first would make the other look applied for ever
|
|
# after. Both are named, because the fix is deciding which one moves.
|
|
#
|
|
# A gap is only ever a warning. It is normal after a branch is abandoned and
|
|
# suspicious after a merge, and this script cannot tell those apart — but a
|
|
# database that is missing 0006 while it holds 0005 and 0007 should hear about
|
|
# it here rather than from a foreign key three releases later.
|
|
# ---------------------------------------------------------------------------
|
|
|
|
prev_version=""
|
|
prev_base=""
|
|
gaps=""
|
|
|
|
while IFS="$TAB" read -r version base _checksum _transactional; do
|
|
if [ -n "$prev_version" ]; then
|
|
if [ "$version" -eq "$prev_version" ]; then
|
|
refuse "'$prev_base' and '$base' share the number $version. The ledger is keyed on the number, so applying one would mark the other as applied. Renumber one of them."
|
|
fi
|
|
|
|
if [ "$version" -eq $((prev_version + 2)) ]; then
|
|
gaps="${gaps}${gaps:+, }$((prev_version + 1))"
|
|
elif [ "$version" -gt $((prev_version + 1)) ]; then
|
|
gaps="${gaps}${gaps:+, }$((prev_version + 1))-$((version - 1))"
|
|
fi
|
|
fi
|
|
|
|
prev_version="$version"
|
|
prev_base="$base"
|
|
done < "$FILES_TSV"
|
|
|
|
if [ -n "$gaps" ]; then
|
|
say "WARNING: the numbering skips ${gaps}. That is fine after an abandoned"
|
|
say " branch and is a missing file after a merge — check which."
|
|
fi
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Connection. From the environment, never from this file.
|
|
#
|
|
# ## DATABASE_URL puts the password somewhere the redaction below cannot reach
|
|
#
|
|
# The redaction that follows keeps credentials out of THIS script's output. It
|
|
# does nothing about `ps`: a URL passed as an argument is world-readable in
|
|
# /proc/<pid>/cmdline for as long as psql runs, so on a shared machine any
|
|
# local user can read the password off a migration in progress.
|
|
#
|
|
# It is not parsed apart here, because doing that correctly means percent-
|
|
# decoding the userinfo exactly as libpq does, and a decoder that is subtly
|
|
# wrong breaks the connection — a worse outcome than the exposure it fixes.
|
|
#
|
|
# The fix is to not use a URL: set PGHOST/PGPORT/PGDATABASE/PGUSER/PGPASSWORD
|
|
# instead, which libpq reads from the environment and never places in argv.
|
|
# --help says so too.
|
|
# ---------------------------------------------------------------------------
|
|
|
|
DB_URL="${DATABASE_URL:-}"
|
|
|
|
PSQL=(psql -X -q -v ON_ERROR_STOP=1)
|
|
|
|
# -X is not cosmetic. A ~/.psqlrc setting AUTOCOMMIT off, ON_ERROR_ROLLBACK on,
|
|
# or a default search_path changes what these migrations mean, and the effect
|
|
# would follow one operator's laptop and no one else's.
|
|
|
|
if [ -n "$DB_URL" ]; then
|
|
PSQL+=(-d "$DB_URL")
|
|
# Credentials are redacted here because this string is printed on every run,
|
|
# into terminals and CI logs that are kept. psql accepts three shapes and
|
|
# only the first is a URI, so all three have to be covered or the redaction
|
|
# is theatre:
|
|
#
|
|
# postgres://user:pw@host/db URI userinfo
|
|
# host=h dbname=d password=pw keyword/value conninfo
|
|
# postgres://user@host/db?sslpassword=pw URI query parameter
|
|
#
|
|
# The userinfo pattern stops at the first '/', '?' or '#', so it can only
|
|
# ever redact inside the authority — and it takes the LAST '@' in there
|
|
# rather than the first, so a password containing a literal '@' does not
|
|
# leave its own tail sitting in the log.
|
|
TARGET=$(printf '%s' "$DB_URL" | sed -E \
|
|
-e 's%(//[^:/@]*):[^/?#]*@%\1:***@%' \
|
|
-e 's%(^|[[:space:]&?])(password|sslpassword)=[^[:space:]&]*%\1\2=***%g')
|
|
elif [ -n "${PGDATABASE:-}" ]; then
|
|
TARGET="${PGUSER:-$(id -un)}@${PGHOST:-local socket}${PGPORT:+:$PGPORT}/${PGDATABASE}"
|
|
else
|
|
# libpq's own default is a database named after the current unix user on the
|
|
# local socket. That is a real connection to a real database, so leaving it
|
|
# implicit means a mistyped environment migrates something at random instead
|
|
# of failing. Named values only.
|
|
die "no connection configured. Set DATABASE_URL, or at least PGDATABASE (with PGHOST/PGPORT/PGUSER as needed) — with neither, psql would connect to a database named after \$USER on the local socket, which is not a target anyone chose."
|
|
fi
|
|
|
|
db_query() { "${PSQL[@]}" -A -t -F "$TAB" -c "$1"; }
|
|
|
|
conn=$(db_query "SELECT current_database(), current_user, current_setting('server_version')" 2>"$work/connerr")
|
|
|
|
if [ -z "$conn" ]; then
|
|
say "could not open the database. psql said:"
|
|
sed 's/^/ /' "$work/connerr" >&2
|
|
die "target was ${TARGET}. Fix the connection and run this again; nothing was read or written."
|
|
fi
|
|
|
|
conn_db=$(printf '%s' "$conn" | cut -f1)
|
|
conn_user=$(printf '%s' "$conn" | cut -f2)
|
|
conn_ver=$(printf '%s' "$conn" | cut -f3)
|
|
|
|
say "target ${TARGET}"
|
|
say " database '${conn_db}' as '${conn_user}', PostgreSQL ${conn_ver}"
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Read the ledger.
|
|
#
|
|
# The table's absence is reported as absence, never as an empty ledger. "No
|
|
# migration has ever been applied here" and "the migrations were applied under
|
|
# a different table name, or into a schema this role cannot see" produce the
|
|
# same empty list and want opposite reactions from the operator.
|
|
# ---------------------------------------------------------------------------
|
|
|
|
TABLE_EXISTS=$(db_query "SELECT to_regclass('${TRACKING_TABLE}') IS NOT NULL" 2>"$work/regerr")
|
|
|
|
case "$TABLE_EXISTS" in
|
|
t|f) ;;
|
|
*)
|
|
say "could not determine whether ${TRACKING_TABLE} exists. psql said:"
|
|
sed 's/^/ /' "$work/regerr" >&2
|
|
die "refusing to continue without knowing what has already been applied." ;;
|
|
esac
|
|
|
|
APPLIED_TSV="$work/applied.tsv" # version <TAB> checksum <TAB> filename <TAB> applied_at
|
|
: > "$APPLIED_TSV"
|
|
|
|
if [ "$TABLE_EXISTS" = "t" ]; then
|
|
if ! db_query "SELECT version, checksum, filename, applied_at FROM ${TRACKING_TABLE}" > "$APPLIED_TSV" 2>"$work/readerr"; then
|
|
say "could not read ${TRACKING_TABLE}. psql said:"
|
|
sed 's/^/ /' "$work/readerr" >&2
|
|
die "refusing to continue without the list of applied migrations."
|
|
fi
|
|
# Checked, like the sort of the file list is. `sort -o` writing over its own
|
|
# input can leave that input truncated if it fails, and a truncated ledger
|
|
# snapshot reads as "fewer migrations are applied than really are" — which
|
|
# plans already-applied migrations for re-application. The re-check inside
|
|
# the lock would catch it, but a guard that is only load-bearing when another
|
|
# guard fails is not one to lean on.
|
|
sort -t "$TAB" -k1,1n "$APPLIED_TSV" -o "$APPLIED_TSV" \
|
|
|| die "could not sort the list of applied migrations read from ${TRACKING_TABLE}."
|
|
fi
|
|
|
|
APPLIED_COUNT=$(grep -c . "$APPLIED_TSV")
|
|
|
|
applied_row() {
|
|
grep -m1 "^$1$TAB" "$APPLIED_TSV" 2>/dev/null
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# The immutability check. Everything above this exists to make it possible.
|
|
# ---------------------------------------------------------------------------
|
|
|
|
DRIFT=""
|
|
|
|
while IFS="$TAB" read -r version base checksum _transactional; do
|
|
row=$(applied_row "$version") || true
|
|
[ -n "$row" ] || continue
|
|
|
|
was_checksum=$(printf '%s' "$row" | cut -f2)
|
|
was_file=$(printf '%s' "$row" | cut -f3)
|
|
|
|
if [ "$was_checksum" != "$checksum" ]; then
|
|
DRIFT="yes"
|
|
say "EDITED: ${base}"
|
|
say " recorded ${was_checksum}"
|
|
say " on disk ${checksum}"
|
|
continue
|
|
fi
|
|
|
|
# Same content, different name. Harmless to the database and worth saying:
|
|
# it is how a ledger and a directory start describing each other in different
|
|
# words, and the next person to grep for the recorded filename finds nothing.
|
|
if [ "$was_file" != "$base" ]; then
|
|
say "note: migration ${version} was recorded as '${was_file}' and is now '${base}' — same contents."
|
|
fi
|
|
done < "$FILES_TSV"
|
|
|
|
if [ -n "$DRIFT" ]; then
|
|
say ""
|
|
say "refusing to run: the file(s) above were applied to this database and have"
|
|
say "changed since. Editing an applied migration changes the past of every"
|
|
say "database that already ran it, and the databases that have not run it yet"
|
|
say "will get something different — with both ledgers claiming to agree."
|
|
say ""
|
|
say "Restore the original contents (git show is usually enough) and put the"
|
|
say "change in a NEW migration with a higher number. If the edit was only"
|
|
say "cosmetic, it still has to be reverted: the checksum is the only evidence"
|
|
say "these databases are the same, and it cannot tell cosmetic from not."
|
|
refuse_or_defer
|
|
fi
|
|
|
|
# A row with no file. Usually the wrong MIGRATIONS_DIR, sometimes a deliberate
|
|
# squash, occasionally a deleted migration that other databases still run.
|
|
# Refused by default because two of those three are wrong and all three are
|
|
# invisible.
|
|
ORPHANS=""
|
|
|
|
while IFS="$TAB" read -r version _checksum filename _applied_at; do
|
|
[ -n "$version" ] || continue
|
|
if ! cut -f1 "$FILES_TSV" | grep -qx "$version"; then
|
|
ORPHANS="${ORPHANS} ${version} ${filename}"$'\n'
|
|
fi
|
|
done < "$APPLIED_TSV"
|
|
|
|
if [ -n "$ORPHANS" ]; then
|
|
say "these migrations are recorded as applied but have no file in ${MIGRATIONS_DIR}:"
|
|
printf '%s' "$ORPHANS" >&2
|
|
|
|
if [ -z "${MIGRATE_ALLOW_MISSING:-}" ]; then
|
|
say "refusing to run. Either this is pointed at the wrong directory, or"
|
|
say "migrations were deleted that other databases have not applied yet."
|
|
say "If they were squashed deliberately, re-run with MIGRATE_ALLOW_MISSING=1."
|
|
refuse_or_defer
|
|
else
|
|
say "MIGRATE_ALLOW_MISSING is set — continuing without them."
|
|
fi
|
|
fi
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Pending, in order.
|
|
# ---------------------------------------------------------------------------
|
|
|
|
PENDING_TSV="$work/pending.tsv"
|
|
: > "$PENDING_TSV"
|
|
|
|
HIGHEST_APPLIED=""
|
|
if [ "$APPLIED_COUNT" -gt 0 ]; then
|
|
HIGHEST_APPLIED=$(cut -f1 "$APPLIED_TSV" | sort -n | tail -n 1)
|
|
fi
|
|
|
|
BEHIND=""
|
|
|
|
while IFS="$TAB" read -r version base checksum transactional; do
|
|
applied_row "$version" >/dev/null && continue
|
|
|
|
if [ -n "$HIGHEST_APPLIED" ] && [ "$version" -lt "$HIGHEST_APPLIED" ]; then
|
|
BEHIND="${BEHIND} ${base} (this database is already at ${HIGHEST_APPLIED})"$'\n'
|
|
fi
|
|
|
|
printf '%s\t%s\t%s\t%s\n' "$version" "$base" "$checksum" "$transactional" >> "$PENDING_TSV"
|
|
done < "$FILES_TSV"
|
|
|
|
PENDING_COUNT=$(grep -c . "$PENDING_TSV")
|
|
|
|
# A pending migration numbered below one already applied arrives when two
|
|
# branches merge. It is refused rather than applied because the order it would
|
|
# run in here is not the order it ran in anywhere that merged the other way
|
|
# first, and "same migrations, different order" is the same divergence the
|
|
# checksum guard exists to catch — just harder to see.
|
|
if [ -n "$BEHIND" ]; then
|
|
say "these pending migrations are numbered below what this database has already applied:"
|
|
printf '%s' "$BEHIND" >&2
|
|
|
|
if [ -z "${MIGRATE_ALLOW_OUT_OF_ORDER:-}" ]; then
|
|
say "refusing to run. Renumber them above ${HIGHEST_APPLIED} so every database"
|
|
say "applies them in the same order, or re-run with"
|
|
say "MIGRATE_ALLOW_OUT_OF_ORDER=1 if you have confirmed the order does not"
|
|
say "matter for these particular statements."
|
|
refuse_or_defer
|
|
else
|
|
say "MIGRATE_ALLOW_OUT_OF_ORDER is set — applying them anyway."
|
|
fi
|
|
fi
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# --status
|
|
# ---------------------------------------------------------------------------
|
|
|
|
if [ "$MODE" = "status" ]; then
|
|
if [ "$TABLE_EXISTS" = "f" ]; then
|
|
say "${TRACKING_TABLE} does not exist. Nothing has been applied THROUGH THIS"
|
|
say "SCRIPT — which is not the same as an empty database. Check the table"
|
|
say "name before concluding the schema is unmigrated."
|
|
else
|
|
say "${APPLIED_COUNT} applied:"
|
|
while IFS="$TAB" read -r version _checksum filename applied_at; do
|
|
[ -n "$version" ] || continue
|
|
printf ' %-6s %-44s %s\n' "$version" "$filename" "$applied_at" >&2
|
|
done < "$APPLIED_TSV"
|
|
fi
|
|
|
|
if [ "$PENDING_COUNT" -eq 0 ]; then
|
|
say "0 pending — every file in ${MIGRATIONS_DIR} has been applied."
|
|
else
|
|
say "${PENDING_COUNT} pending:"
|
|
while IFS="$TAB" read -r version base _checksum transactional; do
|
|
[ -n "$version" ] || continue
|
|
if [ "$transactional" = "no" ]; then
|
|
printf ' %-6s %-44s %s\n' "$version" "$base" "(no-transaction)" >&2
|
|
else
|
|
printf ' %-6s %s\n' "$version" "$base" >&2
|
|
fi
|
|
done < "$PENDING_TSV"
|
|
fi
|
|
|
|
# 0 normally; 2 if something above would have stopped a real run. The report
|
|
# is printed either way, which is the whole point of --status.
|
|
exit "$STATUS_EXIT"
|
|
fi
|
|
|
|
if [ "$PENDING_COUNT" -eq 0 ]; then
|
|
say "${APPLIED_COUNT} applied, 0 pending — nothing to do."
|
|
exit 0
|
|
fi
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# --dry-run
|
|
# ---------------------------------------------------------------------------
|
|
|
|
if [ "$MODE" = "dry-run" ]; then
|
|
say "--dry-run: nothing was changed. It would apply ${PENDING_COUNT}, in this order:"
|
|
|
|
if [ "$TABLE_EXISTS" = "f" ]; then
|
|
say " create ${TRACKING_TABLE} (it does not exist yet)"
|
|
fi
|
|
|
|
while IFS="$TAB" read -r version base _checksum transactional; do
|
|
[ -n "$version" ] || continue
|
|
if [ "$transactional" = "no" ]; then
|
|
printf ' %-6s %-44s %s\n' "$version" "$base" "NOT in a transaction" >&2
|
|
else
|
|
printf ' %-6s %s\n' "$version" "$base" >&2
|
|
fi
|
|
done < "$PENDING_TSV"
|
|
|
|
say "against ${TARGET} (database '${conn_db}')."
|
|
exit 0
|
|
fi
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# The advisory lock, and why the whole run is a single psql session.
|
|
#
|
|
# A session-level advisory lock is released when its SESSION ends, so it can
|
|
# only span the run if the run is one session. That is the reason this builds
|
|
# one SQL program and feeds it to psql once, rather than calling psql per
|
|
# migration: per-migration calls would drop the lock between every file, and
|
|
# two deploys landing together would interleave — each seeing the other's
|
|
# half-finished ledger and both concluding the same migration was theirs to
|
|
# apply.
|
|
#
|
|
# The key is derived from the tracking table name, so two projects sharing one
|
|
# database do not block each other, and two runs of the same project always
|
|
# collide. That collision is the entire feature.
|
|
# ---------------------------------------------------------------------------
|
|
|
|
LOCK_WAIT="${MIGRATE_LOCK_WAIT:-30}"
|
|
case "$LOCK_WAIT" in
|
|
''|*[!0-9]*) die "MIGRATE_LOCK_WAIT must be a whole number of seconds, got '$LOCK_WAIT'." ;;
|
|
esac
|
|
|
|
# Fifteen hex digits: the largest slice that always fits in a signed 64-bit
|
|
# integer, which is what pg_advisory_lock takes and what bash arithmetic holds.
|
|
lock_hex=$(printf '%s' "migrate:${TRACKING_TABLE}" | sha256_stream | cut -c1-15)
|
|
[ -n "$lock_hex" ] || die "could not derive the advisory lock key."
|
|
LOCK_KEY=$((16#$lock_hex))
|
|
|
|
sql_quote() { printf "'%s'" "$(printf '%s' "$1" | sed "s/'/''/g")"; }
|
|
|
|
PROGRAM="$work/run.sql"
|
|
LOCK_FAILED_MARKER="MIGRATE_LOCK_UNAVAILABLE"
|
|
|
|
# Printed by psql itself, one line before each migration starts. It is progress
|
|
# for the operator and, afterwards, the record of how far the session got — a
|
|
# migration that hangs is otherwise indistinguishable from a psql that never
|
|
# connected.
|
|
STARTED_PREFIX="migrate: applying "
|
|
SKIPPED_PREFIX="migrate: skipping "
|
|
|
|
{
|
|
printf '\\set ON_ERROR_STOP on\n'
|
|
printf '\\timing off\n'
|
|
|
|
# The one place a server-side timeout is worth having: DDL waiting behind a
|
|
# long-running reader takes the whole deploy with it, and it does so while
|
|
# holding the advisory lock. Optional and unset by default, because a value
|
|
# that aborts migrations on a busy table is not something to inherit.
|
|
if [ -n "${MIGRATE_PG_LOCK_TIMEOUT:-}" ]; then
|
|
printf "SET lock_timeout = %s;\n" "$(sql_quote "$MIGRATE_PG_LOCK_TIMEOUT")"
|
|
fi
|
|
|
|
# Retry rather than block for ever: a deploy that hangs silently on a stale
|
|
# session is worse than one that fails and says why. The lock is session
|
|
# level, so it survives the end of this DO block's own transaction.
|
|
cat <<SQL
|
|
DO \$migrate_lock\$
|
|
DECLARE
|
|
waited int := 0;
|
|
BEGIN
|
|
WHILE NOT pg_try_advisory_lock(${LOCK_KEY}) LOOP
|
|
IF waited >= ${LOCK_WAIT} THEN
|
|
RAISE EXCEPTION '${LOCK_FAILED_MARKER}: another migration run has held the lock for % seconds', waited;
|
|
END IF;
|
|
PERFORM pg_sleep(1);
|
|
waited := waited + 1;
|
|
END LOOP;
|
|
END
|
|
\$migrate_lock\$;
|
|
SQL
|
|
|
|
# Created inside the lock, so two first-ever runs cannot race here. The
|
|
# notice from IF NOT EXISTS is suppressed for this statement only — it is
|
|
# printed on every run after the first and trains people to ignore output
|
|
# that on some other run says something that matters.
|
|
printf 'SET client_min_messages = warning;\n'
|
|
cat <<SQL
|
|
CREATE TABLE IF NOT EXISTS ${TRACKING_TABLE} (
|
|
version bigint PRIMARY KEY,
|
|
filename text NOT NULL,
|
|
checksum text NOT NULL,
|
|
applied_at timestamptz NOT NULL DEFAULT now(),
|
|
applied_by text NOT NULL DEFAULT current_user
|
|
);
|
|
SQL
|
|
printf 'RESET client_min_messages;\n'
|
|
|
|
# The ledger is keyed on the NUMBER, not the filename: renaming
|
|
# 0007_add_users.sql to 0007_add_user_table.sql is a rename, and a
|
|
# filename-keyed ledger would read it as a new migration and run it twice.
|
|
# The filename column keeps the last name seen so the drift is visible.
|
|
#
|
|
# ## The plan is re-checked, one migration at a time, after the lock is held
|
|
#
|
|
# The list of pending migrations was computed BEFORE the lock was acquired,
|
|
# and a run that waited on the lock waited precisely because another run was
|
|
# applying things — so by the time it gets in, its plan is stale. Without
|
|
# this check the second deploy re-runs everything the first one just did:
|
|
# every statement executes twice and the run dies on the tracking table's
|
|
# primary key, reporting a migration failure that is really a race.
|
|
#
|
|
# So each migration asks the ledger, inside the lock, whether it is still
|
|
# pending, and skips itself if not. \gset and \if do that in a static script
|
|
# with no branching from the shell side.
|
|
|
|
while IFS="$TAB" read -r version base checksum transactional; do
|
|
[ -n "$version" ] || continue
|
|
|
|
# 'on'/'off' rather than a boolean: \if is documented to take those, and a
|
|
# value it cannot parse is an error rather than a guess.
|
|
printf "SELECT CASE WHEN EXISTS (SELECT 1 FROM %s WHERE version = %s) THEN 'off' ELSE 'on' END AS mig_%s \\\\gset\n" \
|
|
"$TRACKING_TABLE" "$version" "$version"
|
|
printf '\\if :mig_%s\n' "$version"
|
|
printf "\\\\echo '%s%s'\n" "$STARTED_PREFIX" "$base"
|
|
|
|
insert="INSERT INTO ${TRACKING_TABLE} (version, filename, checksum) VALUES (${version}, $(sql_quote "$base"), $(sql_quote "$checksum"));"
|
|
|
|
if [ "$transactional" = "yes" ]; then
|
|
printf 'BEGIN;\n'
|
|
printf "\\\\i '%s/%s'\n" "$MIGRATIONS_ABS" "$base"
|
|
printf '%s\n' "$insert"
|
|
printf 'COMMIT;\n'
|
|
else
|
|
printf "\\\\i '%s/%s'\n" "$MIGRATIONS_ABS" "$base"
|
|
printf '%s\n' "$insert"
|
|
fi
|
|
|
|
printf '\\else\n'
|
|
printf "\\\\echo '%s%s (another run applied it first)'\n" "$SKIPPED_PREFIX" "$base"
|
|
printf '\\endif\n'
|
|
done < "$PENDING_TSV"
|
|
|
|
# Not strictly needed — ending the session drops it — but an explicit unlock
|
|
# keeps the intent readable in the generated script, which is the thing an
|
|
# operator reads when they are trying to work out what ran. Wrapped in a DO
|
|
# block so it does not print a result row into the run's output.
|
|
printf 'DO $migrate_unlock$ BEGIN PERFORM pg_advisory_unlock(%s); END $migrate_unlock$;\n' "$LOCK_KEY"
|
|
} > "$PROGRAM"
|
|
|
|
say "applying ${PENDING_COUNT} migration(s) to '${conn_db}'…"
|
|
|
|
nontx=$(cut -f4 "$PENDING_TSV" | grep -c '^no$')
|
|
if [ "$nontx" -gt 0 ]; then
|
|
say "WARNING: ${nontx} of them run OUTSIDE a transaction (-- migrate:"
|
|
say " no-transaction). If one fails partway, its tracking row is not"
|
|
say " written and this run will report it — but the database keeps"
|
|
say " whatever it managed to do. A failed CREATE INDEX CONCURRENTLY"
|
|
say " leaves an INVALID index behind: DROP it before re-running."
|
|
fi
|
|
|
|
LOG="$work/psql.log"
|
|
|
|
"${PSQL[@]}" -f "$PROGRAM" 2>&1 | tee "$LOG" >&2
|
|
psql_status=${PIPESTATUS[0]}
|
|
|
|
# Checked first, and before the ledger is re-read: on a first-ever run the
|
|
# tracking table does not exist yet, so a lock loss there would otherwise be
|
|
# reported as "could not read the ledger" — which is true, and not the reason.
|
|
if grep -q "$LOCK_FAILED_MARKER" "$LOG"; then
|
|
say "another migration run holds the lock for ${TRACKING_TABLE} and did not"
|
|
say "release it within ${LOCK_WAIT}s. Nothing was applied by this run. Wait for"
|
|
say "the other deploy to finish, or raise MIGRATE_LOCK_WAIT. If nothing is"
|
|
say "running, look for a stale session:"
|
|
say " SELECT * FROM pg_locks WHERE locktype = 'advisory';"
|
|
exit 3
|
|
fi
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# What was applied is read back out of the ledger, not counted from this
|
|
# script's own output.
|
|
#
|
|
# The two can disagree, and when they do the ledger is right: a no-transaction
|
|
# migration can run to completion and still fail to record itself, and a
|
|
# session killed between COMMIT and the next statement leaves output claiming
|
|
# less than the database holds. Reporting the plan, or the echoes, would be
|
|
# reporting an intention as a fact.
|
|
#
|
|
# If the ledger cannot be re-read, that is reported as not knowing — never as
|
|
# nothing having happened.
|
|
# ---------------------------------------------------------------------------
|
|
|
|
CONFIRMED="$work/confirmed.tsv"
|
|
confirmed_ok="yes"
|
|
|
|
if ! db_query "SELECT version FROM ${TRACKING_TABLE}" > "$CONFIRMED" 2>"$work/confirmerr"; then
|
|
confirmed_ok="no"
|
|
: > "$CONFIRMED"
|
|
fi
|
|
|
|
applied_now=0
|
|
skipped_now=0
|
|
applied_list=""
|
|
skipped_list=""
|
|
missing_list=""
|
|
|
|
while IFS="$TAB" read -r version base _checksum _transactional; do
|
|
[ -n "$version" ] || continue
|
|
|
|
if ! grep -qx "$version" "$CONFIRMED"; then
|
|
missing_list="${missing_list} ${version} ${base}"$'\n'
|
|
elif grep -qF "${SKIPPED_PREFIX}${base}" "$LOG"; then
|
|
# In the ledger, but this run did not put it there. Counted separately
|
|
# rather than folded into the total: "we applied it" and "we found it
|
|
# already applied and stood down" are different events, and only the second
|
|
# one means another deploy was running against this database.
|
|
skipped_now=$((skipped_now + 1))
|
|
skipped_list="${skipped_list} ${version} ${base}"$'\n'
|
|
else
|
|
applied_now=$((applied_now + 1))
|
|
applied_list="${applied_list} ${version} ${base}"$'\n'
|
|
fi
|
|
done < "$PENDING_TSV"
|
|
|
|
if [ "$confirmed_ok" = "no" ]; then
|
|
say ""
|
|
|
|
if [ "$psql_status" -eq 0 ]; then
|
|
# psql was happy and the ledger has become unreadable between one statement
|
|
# and the next. Nothing can be said about what is applied, so nothing is.
|
|
say "the run reported success but ${TRACKING_TABLE} could not be re-read, so"
|
|
say "what is applied is UNKNOWN. psql said:"
|
|
sed 's/^/ /' "$work/confirmerr" >&2
|
|
say "Run --status once the database is reachable, before running this again."
|
|
exit 1
|
|
fi
|
|
|
|
# The far more common shape: the run failed early — a missing schema, a role
|
|
# without CREATE — so the table never existed to be read. The error printed
|
|
# above is the real one, and calling this "unknown" would send the operator
|
|
# looking for a database problem instead of reading it.
|
|
KEEP_WORK="yes"
|
|
say "the run failed (psql exit ${psql_status}) and ${TRACKING_TABLE} does not"
|
|
say "exist or cannot be read, so nothing was applied. Read the error above —"
|
|
say "a missing schema or a role without CREATE is the usual cause."
|
|
say "the SQL that was sent is kept at ${work}/run.sql."
|
|
exit 4
|
|
fi
|
|
|
|
if [ "$skipped_now" -gt 0 ]; then
|
|
say "another run applied ${skipped_now} of the ${PENDING_COUNT} planned while this one"
|
|
say "waited for the lock. They were skipped, not repeated:"
|
|
printf '%s' "$skipped_list" >&2
|
|
fi
|
|
|
|
if [ "$psql_status" -eq 0 ]; then
|
|
say "applied ${applied_now} of ${PENDING_COUNT}:"
|
|
|
|
if [ "$applied_now" -eq 0 ]; then
|
|
say " (none)"
|
|
else
|
|
printf '%s' "$applied_list" >&2
|
|
fi
|
|
|
|
# psql exited clean while the ledger disagrees with the plan. Rare, and
|
|
# exactly the state that must not be reported as success: something ran
|
|
# without recording itself, or something recorded itself twice.
|
|
if [ $((applied_now + skipped_now)) -ne "$PENDING_COUNT" ]; then
|
|
# Kept, like the other failure paths keep it. This branch tells the
|
|
# operator to check the database by hand, and the generated SQL is the
|
|
# only record of what was actually sent — deleting it on the way out would
|
|
# take the evidence with it.
|
|
KEEP_WORK="yes"
|
|
say ""
|
|
say "psql reported success but these are NOT in ${TRACKING_TABLE}:"
|
|
printf '%s' "$missing_list" >&2
|
|
say "Do not re-run until you know why — check the database by hand."
|
|
say "the SQL that was sent is kept at ${work}/run.sql."
|
|
exit 4
|
|
fi
|
|
|
|
exit 0
|
|
fi
|
|
|
|
# psql stops at the first error, so the last migration it announced is the one
|
|
# that failed — unless the ledger says that one committed, in which case the
|
|
# error came after it and blaming it would be a claim rather than a reading.
|
|
failed=$(grep "^${STARTED_PREFIX}" "$LOG" | tail -n 1 | sed "s/^${STARTED_PREFIX}//")
|
|
failed_version=""
|
|
[ -n "$failed" ] && failed_version=$(grep -m1 "${TAB}${failed}${TAB}" "$PENDING_TSV" | cut -f1)
|
|
|
|
KEEP_WORK="yes"
|
|
|
|
say ""
|
|
say "a migration failed (psql exit ${psql_status}). See the error above."
|
|
say "the SQL that was sent is kept at ${work}/run.sql — the line numbers in"
|
|
say "psql's error refer to it. Delete it when you are done."
|
|
|
|
if [ -z "$failed" ]; then
|
|
# Nothing was announced, so nothing was reached: the failure is the lock, the
|
|
# CREATE TABLE, or the connection. Said as such rather than blamed on a
|
|
# migration that never started.
|
|
say "no migration was reached — it failed on the lock, on creating"
|
|
say "${TRACKING_TABLE}, or on the connection itself."
|
|
elif [ -n "$failed_version" ] && grep -qx "$failed_version" "$CONFIRMED"; then
|
|
say "'${failed}' is the last one announced and it IS recorded as applied, so"
|
|
say "the failure came after it finished. Read the error above before re-running."
|
|
else
|
|
say "the failing migration is '${failed}'."
|
|
|
|
# A transactional failure needs no explanation beyond "nothing happened". A
|
|
# no-transaction failure does, and it is the one case where somebody has to
|
|
# look at the database before this is run again.
|
|
if grep -q "${TAB}${failed}${TAB}.*${TAB}no$" "$PENDING_TSV"; then
|
|
say ""
|
|
say "It ran OUTSIDE a transaction, so whatever it completed before failing is"
|
|
say "still there and is NOT recorded. Inspect the database before re-running:"
|
|
say "a half-built CREATE INDEX CONCURRENTLY leaves an INVALID index that has"
|
|
say "to be dropped, and any statement that did succeed will run a second time."
|
|
fi
|
|
fi
|
|
|
|
say ""
|
|
say "$((applied_now + skipped_now)) of ${PENDING_COUNT} are recorded in ${TRACKING_TABLE} and will"
|
|
say "not run again:"
|
|
|
|
if [ "$applied_now" -eq 0 ]; then
|
|
say " (none by this run)"
|
|
else
|
|
printf '%s' "$applied_list" >&2
|
|
fi
|
|
say ""
|
|
say "Fix the failing migration IN PLACE — it is not recorded, so its checksum is"
|
|
say "not history yet — then run this again. Never edit one that already"
|
|
say "succeeded; --status lists which those are."
|
|
exit 4
|