Queue-North-Website/scripts/doc-triggers.py

#!/usr/bin/env python3
"""Which documents does a change fire, while there is still time to update them?

    python3 scripts/doc-triggers.py                 # everything dirty right now
    python3 scripts/doc-triggers.py --staged        # what is staged
    python3 scripts/doc-triggers.py <path> [path…]  # specific paths
    python3 scripts/doc-triggers.py --range HEAD~3..HEAD

## The failure this catches

Every document in this tree carries a `Review trigger:` — the change that should
send somebody back to it — and `WORK_CYCLE.md` requires the triggered documents
to be updated *in the same commit as the code*. Deciding which fired means
reading every `Governs:` line and matching globs in your head, once per commit.
It is a check with no output of its own, so it is the one that gets skipped when
the code is already green, and the cost is invisible until a reader trusts a
document that stopped being true: a reference manual six migrations behind, and
every reader in between believing it.

**This is not the doc-review check.** That one asks which baselined documents are
*overdue*, from committed history — a governed path with a commit newer than the
document's `Last reviewed` date. It is a different question and it can only be
asked after the fact. By the time a change is committed without its document, the
thing this catches has already happened.

Exit status is always 0. This is a prompt, not a gate: a trigger asks a human
whether the prose is still true, and a check that failed the build for that would
be bumped past rather than read.

## Matching the trigger's verb, not only its glob

A document is fired when a changed path matches its `Governs:` **and** the kind
of change matches its `Fires on:`. That second field is optional and almost
never needed; it exists for the case where `Governs:` is far broader than the
trigger. `DOC_TRUST_MAP.md` is the extreme — it governs `docs/**` while its
trigger is *any doc added, deleted or moved* — so on the glob alone it fired on
every edit to every document, forever, and correctly by the only rule there was.
A prompt that always fires is one people stop reading, and this one exits 0 by
design, so nothing forces the reading.

The kinds are `added`, `deleted`, `moved` and `changed`, read from git's own
status letter. Absent, empty or unparseable means every kind, which is the
behaviour every document without the line still has.

## Two things it deliberately does not do

**It does not read `Exempt:` declarations.** Those mark a *required* document as
deliberately absent from a repository, and a document that does not exist cannot
govern a path. There is nothing here for them to change.

**It cannot fire a document whose `Governs:` is prose.** Several in this template
govern a subject rather than a set of paths — `GUARDS.md` governs "structural
tests, source-grep assertions, probes, and any check whose passing is taken as
evidence", which is the honest description and matches no glob. Those documents
are listed separately at the end rather than silently ignored, because a reader
who sees only the matched list would reasonably conclude the others were checked
and cleared.
"""
from __future__ import annotations

import fnmatch
import pathlib
import re
import subprocess
import sys

def _find_root() -> pathlib.Path:
    """The repository root, found rather than assumed.

    This was `parents[3]`, which is correct only while the script sits at
    `docs/architecture/scripts/` — its home in the template. The moment a
    project copies it to `scripts/`, as the template's own adoption
    instructions say to, `parents[3]` climbs out of the repository entirely: in
    a checkout at `~/Projects/thing/scripts/`, it resolves to `~/`, and the
    script reports "no docs/ directory" about a directory two levels above the
    project it was run in.

    So: walk up from the script looking for a directory that has both `docs/`
    and `.git`, then fall back to either alone, then to git's own answer.
    """
    here = pathlib.Path(__file__).resolve()

    for parent in here.parents:
        if (parent / "docs").is_dir() and (parent / ".git").exists():
            return parent

    for parent in here.parents:
        if (parent / "docs").is_dir():
            return parent

    result = subprocess.run(
        ["git", "rev-parse", "--show-toplevel"],
        cwd=here.parent,
        capture_output=True,
        text=True,
        check=False,
    )

    if result.returncode == 0 and result.stdout.strip():
        return pathlib.Path(result.stdout.strip())

    return here.parents[1]


ROOT = _find_root()
DOCS = ROOT / "docs"

# The status header is a fenced block immediately after the H1, and a long value
# wraps onto continuation lines indented by two spaces:
#
#   Governs: structural tests, source-grep assertions, probes, and any check whose
#     passing is taken as evidence
#
# A regex that reads one line per field — the obvious implementation — truncates
# at the wrap and silently under-reports, which for this tool means quietly
# failing to name a document that should have been updated. So fields are
# assembled line by line instead.
FIELD_START = re.compile(
    r"^(Status|Owner|Last reviewed|Governs|Review trigger|Fires on):\s*(.*)$"
)
HEADER_LINES = 16


def header_of(doc: pathlib.Path) -> dict[str, str]:
    """The status header, with wrapped values joined."""
    try:
        lines = doc.read_text(encoding="utf-8").splitlines()[:HEADER_LINES]
    except (OSError, UnicodeDecodeError):
        return {}

    fields: dict[str, str] = {}
    current: str | None = None
    for line in lines:
        match = FIELD_START.match(line)
        if match:
            current = match.group(1)
            fields[current] = match.group(2).strip()
        elif current and line.startswith(("  ", "\t")) and line.strip():
            fields[current] = f"{fields[current]} {line.strip()}".strip()
        elif line.strip().startswith("```") and fields:
            break
    return fields


# `Status` must be one of these four. `DOC_TRUST_MAP.md` states it as a rule with
# a checker behind it, and it is the one field that distinguishes a document from
# a template for one: `project-readme-template.md` carries
# `Status: <Current | Draft | Superseded | Archived>`, and its `Governs` describes
# the README of the project that copies it, not anything in this repository.
STATUS_WORDS = {"Current", "Draft", "Superseded", "Archived"}


def governing_documents() -> list[pathlib.Path]:
    """Every document that can fire, root ones included.

    The walk used to start at `docs/`, so the documents at the repository root
    were not read at all — `README.md` and the two `START-HERE-*.md` carry a full
    status header, govern real paths, and fired nothing ever, while the output
    said "No document's Governs matched these paths". True of the tool and false
    of the repository, which is the same shape as the gloss bug one level out.

    The root is read **non-recursively**: `ROOT.glob`, not `rglob`. A vendored
    copy of this template, a scratch checkout, or somebody's directory of notes
    would otherwise enrol its documents as governing this repository.
    """
    docs = sorted(ROOT.glob("*.md"))
    if DOCS.is_dir():
        docs += sorted(DOCS.rglob("*.md"))
    return docs


def looks_like_path(glob: str) -> bool:
    """Whether a `Governs:` entry is a path pattern rather than a subject.

    The same test `doc-claims.sh` uses: a slash, a wildcard, or a file
    extension. Prose about what a document is authoritative for will have none
    of them, and must not be treated as a glob that simply never matches.
    """
    glob = glob.strip()
    if not glob or " " in glob and "/" not in glob and "*" not in glob:
        return False
    return "/" in glob or "*" in glob or re.search(r"\.\w{1,5}$", glob) is not None


# A `Governs:` entry is split on commas, so an entry that explains itself after
# the glob arrives whole: `docs/data/** — the assets privacyllc.dev renders for
# this project`. Used as a glob that matches nothing, ever, and because it
# contains a slash `looks_like_path` calls it a path — so the document was
# neither fired nor listed among the ones no change can fire mechanically. It was
# simply absent, which is the one outcome a reader cannot notice.
#
# Three of the seven path-governing documents here were in that state from the
# first commit, `docs/data/img/README.md` among them: editing the branding assets
# had never once prompted the document that specifies their names and sizes.
GLOSS = re.compile(r"\s+(?:—|–|--)\s+")


def globs_in(entry: str) -> list[str]:
    """The globs inside one `Governs:` entry, with any trailing gloss removed.

    The cut requires whitespace on both sides of the dash: `source-grep` and
    `doc-claims` appear in these headers and a bare `-` would halve them. Tokens
    are taken from the left of the gloss rather than from the whole entry,
    because prose on the right can itself look like a path — `privacyllc.dev`
    passes the extension test and would become a glob that fires on a file
    nobody has.

    An entry yielding no token falls back to itself, so a shape not foreseen here
    behaves exactly as it did before.
    """
    head = GLOSS.split(entry, 1)[0]
    return [tok for tok in head.split() if looks_like_path(tok)] or [entry]


def matches(path: str, glob: str) -> bool:
    """Whether `path` is governed by `glob`.

    `**` means "and everything below", which `fnmatch` does not implement: its
    `*` already crosses separators, so `a/**` never matches `a/b/c`. The two
    forms these headers actually use are reduced to prefix tests.
    """
    glob = glob.strip()
    if not glob:
        return False
    if glob.endswith("/**"):
        return path.startswith(glob[:-2]) or path == glob[:-3]
    if glob.endswith("/"):
        # A bare directory, as `githooks/README.md` governs ".githooks/". fnmatch
        # would not match a file inside it.
        return path.startswith(glob)
    if "/**/" in glob:
        head, tail = glob.split("/**/", 1)
        return path.startswith(head + "/") and fnmatch.fnmatch(path, "*" + tail)
    return fnmatch.fnmatch(path, glob)


# `Governs:` says *where* a document is authoritative; `Fires on:` says which
# kinds of change to that place its `Review trigger` actually names. The two come
# apart badly at the extreme: `DOC_TRUST_MAP.md` governs `docs/**`, the broadest
# glob in the tree, while its trigger is one of the narrowest — *any doc added,
# deleted or moved*. Matching on the glob alone fires it on every edit to every
# document forever, and a prompt that always fires is one people stop reading,
# which takes the true positives with it.
#
# Why a declared field rather than reading the trigger prose. The obvious first
# cut — look for `added`/`deleted`/`moved` and no `changed`/`change to` — was
# tried against the seven path-governing documents here and misclassified the one
# it exists to fix. `DOC_TRUST_MAP.md`'s trigger ends "any change to which doc
# owns a subject", so it reads as a change-verb; the clause is about which
# document owns a subject, not about a file being edited. Nothing lexical
# separates it from `architecture/README.md`'s "any change to a module boundary
# or a data shape", which genuinely does mean modification. Guessing at English
# and getting it wrong here is silent in the expensive direction: the document
# stops being prompted for and goes quietly stale.
#
# So the narrowing is declared or it does not happen. Absent, unparseable, or
# empty means fire on everything, which is the old behaviour — a document is only
# ever quietened by someone writing the line deliberately.
KIND_LETTERS = {
    "added": {"A", "C"},
    "deleted": {"D"},
    "moved": {"R"},
    "changed": {"M", "T"},
}
ALL_KINDS = {letter for letters in KIND_LETTERS.values() for letter in letters}
KIND_OF = {letter: word for word, letters in KIND_LETTERS.items() for letter in letters}


def fires_on(header: dict[str, str]) -> tuple[set[str], list[str]]:
    """The status letters a document accepts, and any words not understood.

    Returns every letter when nothing is declared or the declaration cannot be
    read, so the failure mode of a typo is a document that is prompted for too
    often rather than one that is silently dropped.
    """
    raw = header.get("Fires on", "").strip()
    if not raw:
        return ALL_KINDS, []

    words = [w.strip().lower().rstrip(".") for w in re.split(r"[,;]|\band\b", raw)]
    words = [w for w in words if w]

    letters: set[str] = set()
    unknown: list[str] = []
    for word in words:
        if word in KIND_LETTERS:
            letters |= KIND_LETTERS[word]
        else:
            unknown.append(word)

    if unknown or not letters:
        return ALL_KINDS, unknown or ["(empty)"]
    return letters, []


def _git(*args: str) -> str:
    result = subprocess.run(
        ["git", *args], cwd=ROOT, capture_output=True, text=True, check=False
    )
    return result.stdout


def _name_status(*args: str) -> list[tuple[str, str]]:
    """(letter, path) from a `--name-status` listing.

    A rename arrives as `R100<TAB>old<TAB>new`, so the path is taken from the
    last field: the new name governs, as it did when only names were read.
    """
    pairs = []
    for line in _git(*args, "--name-status").splitlines():
        if not line.strip():
            continue
        fields = line.split("\t")
        if len(fields) < 2 or not fields[0].strip():
            continue
        pairs.append((fields[0].strip()[0].upper(), fields[-1].strip()))
    return pairs


def _status_of_named(path: str) -> str:
    """The kind of change a path named on the command line represents.

    There is no diff to read here, so it is inferred: gone from disk is a
    deletion, present but untracked is an addition, and anything else is a
    modification — the usual reason to ask about a path by name.
    """
    if not (ROOT / path).exists():
        return "D"
    return "M" if _git("ls-files", "--", path).strip() else "A"


def changed_paths(argv: list[str]) -> tuple[list[tuple[str, str]], str]:
    if argv and argv[0] == "--staged":
        return _name_status("diff", "--cached"), "staged"
    if argv and argv[0] == "--range":
        if len(argv) < 2:
            sys.exit("doc-triggers: --range needs a revision range")
        return _name_status("diff", argv[1]), f"range {argv[1]}"
    if argv:
        return [(_status_of_named(a), a) for a in argv], "named paths"

    # Untracked files are included on purpose: a brand-new module is the case
    # most likely to need a document and least likely to be remembered, and it
    # is invisible to `git diff`.
    pairs = []
    for line in _git("status", "--porcelain").splitlines():
        if not line.strip():
            continue
        index, worktree = line[0], line[1]
        path = line[3:].strip()
        if " -> " in path:                      # a rename; the new name governs
            path = path.split(" -> ", 1)[1]
        if "?" in (index, worktree):
            letter = "A"                        # untracked: a file that is new
        else:
            letter = (index if index != " " else worktree).upper()
        pairs.append((letter, path.strip('"')))
    return pairs, "working tree"


def main() -> int:
    if not DOCS.is_dir():
        print(f"doc-triggers: no docs/ directory at {DOCS}")
        return 0

    paths, source = changed_paths(sys.argv[1:])
    if not paths:
        print(f"doc-triggers: nothing changed in the {source}.")
        return 0

    print(f"doc-triggers: {len(paths)} path(s) from the {source}\n")

    fired: list[tuple[str, list[tuple[str, str]], str]] = []
    subject_only: list[str] = []
    wrong_kind: list[tuple[str, str]] = []
    unfilled: list[tuple[str, str]] = []

    for doc in governing_documents():
        rel = str(doc.relative_to(ROOT))
        header = header_of(doc)
        governs = header.get("Governs", "")
        if not governs:
            continue

        filled = header.get("Status", "") in STATUS_WORDS

        entries = [g.strip() for g in governs.split(",") if g.strip()]
        # Classification reads the whole entry and extraction reads inside it:
        # deciding "path or subject?" on a token would move documents between the
        # two lists as a side effect of this fix.
        path_globs = [g for e in entries if looks_like_path(e) for g in globs_in(e)]
        if not path_globs:
            if filled:
                subject_only.append(rel)
            continue

        letters, unknown = fires_on(header)
        if unknown:
            print(
                f"doc-triggers: {rel} declares 'Fires on: "
                f"{header.get('Fires on', '')}' — {', '.join(unknown)} not "
                f"understood, so it fires on everything.\n"
            )

        matched = {(s, p) for s, p in paths for g in path_globs if matches(p, g)}

        if not filled:
            # A template for a document rather than a document. Named only when it
            # would otherwise have fired: a line on every run, about a file that is
            # supposed to look like this, is the noise this tool keeps being fixed
            # for.
            if matched:
                unfilled.append((rel, header.get("Status", "") or "(none)"))
            continue

        hits = sorted({(s, p) for s, p in matched if s in letters}, key=lambda x: x[1])
        if hits:
            fired.append((rel, hits, header.get("Review trigger", "(none stated)")))
        elif matched:
            # Governed, and deliberately not prompted for: the paths changed in a
            # way this document's trigger does not name. Said out loud, because a
            # reader who saw nothing would have to guess whether it was checked.
            wrong_kind.append((rel, header.get("Fires on", "").strip()))

    for rel, hits, trigger in fired:
        print(f"\033[1m{rel}\033[0m")
        for letter, hit in hits[:6]:
            print(f"    {KIND_OF.get(letter, letter.lower()):>7}  {hit}")
        if len(hits) > 6:
            print(f"    … and {len(hits) - 6} more")
        print(f"  trigger: {trigger}\n")

    if fired:
        print(f"{len(fired)} document(s) govern something in this change.")
        print("Read each trigger and decide — the rule is to update them in the")
        print("SAME commit as the code, not afterwards.")
    elif wrong_kind or unfilled:
        # Distinct from matching nothing, and worth separating: a path here *is*
        # governed, and the reason nothing fired is a declaration somebody wrote
        # or a header nobody filled in, not an area no document claims.
        print("Nothing fired, but these paths are governed — see below for which")
        print("documents matched them and why each was not raised.")
    else:
        print("No document's Governs matched these paths. Worth a second look if")
        print("this change added a module, a migration, or a new boundary — an")
        print("unmatched path can also mean no document claims that area yet.")

    if wrong_kind:
        print("\nGovern a path in this change but do not fire on this kind of")
        print("change, by their own Fires on declaration:")
        for rel, declared in wrong_kind:
            print(f"    {rel} — fires on {declared}")

    if unfilled:
        print("\nGovern a path in this change but their status header is still a")
        print("template, so they are not treated as documents of this repository:")
        for rel, status in unfilled:
            print(f"    {rel} — Status: {status}")

    if subject_only:
        print("\nNot checked here — these govern a subject rather than paths, so")
        print("no change can fire them mechanically. Judge them yourself:")
        for rel in subject_only:
            print(f"    {rel}")

    return 0


if __name__ == "__main__":
    raise SystemExit(main())