diff --git a/docs/Engineering_Reference_Manual.md b/docs/Engineering_Reference_Manual.md index 1c2e79b..526ff3f 100644 --- a/docs/Engineering_Reference_Manual.md +++ b/docs/Engineering_Reference_Manual.md @@ -2,7 +2,7 @@ **Status:** Current code reference **Last Updated:** 2026-07-05 -**Version:** 0.40.0 +**Version:** 0.40.0 (package version; the in-flight Track A/B/C/D commits target **v0.41.0** — see HISTORY.md and the `Last Updated` note below) **Primary stack:** Node.js 22 + Express 4 (CommonJS), React 19 + Vite 5 (TypeScript, `strict` + `noUncheckedIndexedAccess`), Tailwind CSS 3 + shadcn/ui, TanStack Query 5, Sonner, SQLite via `better-sqlite3` 12, `@simplewebauthn/*` 13, `otplib` 13, `openid-client` 5 This manual reflects the current application code in `server.js`, `routes/`, `services/`, `middleware/`, `db/`, `client/`, `package.json`, `Dockerfile`, `docker-compose.yml`, `tsconfig.json`, `eslint.config.mjs`, and `vite.config.mjs`. It is written as a current-state reference, not a changelog. The previous version of this manual (v0.28.1) is fully superseded. @@ -16,6 +16,13 @@ Notable changes since the previous manual: - **New features:** SimpleFIN bank sync, WebAuthn + TOTP 2FA, calendar feed (`/api/calendar/feed.ics`), subscription catalog + recommendations, spending categories / budgets, snowball plans (lifecycle), push notifications (ntfy / Gotify / Discord / Telegram), push notification column on users, encrypted secrets at rest with HKDF-derived key, money-cents migration (in progress), category groups, banker-rule auto-attribute. - **Schema:** migrations `v0.2` → `v1.06`. Rollback map covers `v0.44` → `v1.04`. +Since the v0.40.0 manual shipped, **8 in-flight commits** targeting v0.41.0 have been added locally and are documented in this manual ahead of push: + +- **Track A (money integrity):** every payment balance-mutation now runs in a single transaction. `POST /payments` returns `409 { code: 'DUPLICATE_SUSPECTED' }` instead of silently deduping when a live payment matches `(bill_id, paid_date, amount)`; the client surfaces a sonner "Add anyway?" toast via `client/lib/paymentActions.ts → createPaymentOrConfirm` and resends with `allow_duplicate: true` to confirm. +- **Track B (tests):** `tests/paymentsRoute.test.js` (route + transactional behavior) and `tests/reconciliation.test.js` (cross-surface reconciliation harness). +- **Track C (API response typing):** high-traffic domains (`auth`, `snowball`, `subscriptions`, `spending`, `bank-ledger`) promoted page-local shapes into `client/types.ts` and wired through `get` / `post`. `User` moved from `@/hooks/useAuth` to `@/types` (re-exported for 7 importers). The single-consumer non-money casts (`adminUsers` and the one-off `version` / `about` / `privacy` / `health` / `importHistory` doc shapes) are deliberately left as-is. +- **Track D (server error-shape consistency):** the global 500 handler in `server.js` now emits `{ error, code: 'INTERNAL_ERROR' }` so an unexpected server error carries the same `{error, code}` field the client can switch on as every structured 4xx. + --- ## 1. System Overview @@ -403,8 +410,9 @@ Response conventions: - Unauthenticated: `401`. - Forbidden: `403`. - Missing resource: `404`. -- Conflict: `409`. +- Conflict: `409` (also used for `DUPLICATE_SUSPECTED` on `POST /payments` — see §5.5). - Rate-limited: `429`. +- **Server error (Track D):** the global 500 handler in `server.js` emits `{ error: 'Internal server error', code: 'INTERNAL_ERROR' }` so an unexpected server error carries the same `{error, code}` field the client can switch on as every structured 4xx. ### 5.1 Auth (`/api/auth`) @@ -467,6 +475,12 @@ User/admin tracker access. Soft delete via `deleted_at`. `payment_source` ∈ `{ Endpoints: `GET /payments?bill_id=&year=&month=`, `GET /payments/:id`, `POST /payments`, `POST /payments/quick`, `POST /payments/bulk` (max 50), `PUT /payments/:id`, `DELETE /payments/:id`, `POST /payments/:id/restore`. +**Track A money-integrity notes:** + +- `POST /payments` body now accepts an optional `allow_duplicate: true` flag. When omitted and a live payment already matches the same `(bill_id, paid_date, amount)`, the route returns `409 { error, code: 'DUPLICATE_SUSPECTED' }` instead of silently deduping — silently dropping a legitimately-identical second payment would be a money-integrity bug. The client uses `client/lib/paymentActions.ts → createPaymentOrConfirm` to surface a sonner "Add anyway?" toast; confirming resends with `allow_duplicate: true`. +- Every balance-mutation runs in a single transaction so a partial failure cannot leave `bills.current_balance` half-updated relative to the new payment. +- New tests: `tests/paymentsRoute.test.js` (route + transactional behavior) and `tests/reconciliation.test.js` (cross-surface reconciliation harness). + ### 5.6 Categories (`/api/categories`) User/admin tracker access. Categories have a `spending_enabled` flag (`v0.88`) and a `sort_order` (`v0.75`). Categories can be grouped via `category_groups` (`v1.06`). @@ -1293,27 +1307,38 @@ Routes (all rendered through `Routes` with `ErrorBoundary`, lazy + `Suspense` fo - Reads CSRF token from the `bt_csrf_token` cookie **or** fetches once from `/api/auth/csrf-token` and caches the token in memory; sends `x-csrf-token` on `POST`/`PUT`/`PATCH`/`DELETE`. - Non-OK responses throw an `Error` whose `status`/`data`/`details`/`code` match the server's structured `errorFormatter`. - `ApiError` interface carries those fields; `TogglePaidResult` and `PaymentRecord` provide typed payment responses; `QueryParams` types the query-string builder. -- Exposes grouped functions for `auth`, `admin`, `notifications`, `profile`, `tracker`, `calendar`, `summary`, `bills`, `payments`, `categories`, `settings`, `analytics`, `status`, `version`/`about`, `import`, `export`, `data-sources`, `transactions`, `matches`, `snowball`, `subscriptions`, `spending`, `monthlyStartingAmounts`, `csvImport`, `matchSuggestions`. +- Exposes grouped functions for `auth`, `admin`, `notifications`, `profile`, `tracker`, `calendar`, `summary`, `bills`, `payments`, `categories`, `settings`, `analytics`, `status`, `version`/`about`, `import`, `export`, `data-sources`, `transactions`, `matches`, `snowball`, `subscriptions`, `spending`, `monthlyStartingAmounts`, `csvImport`, `matchSuggestions`, `bankTransactionsLedger`, `matchTransaction`, `unmatchTransaction`, `ignoreTransaction`, `unignoreTransaction`, `applyTransactionMerchantMatch`, `autoCategorizeTransactions`, `categoryGroups`, `copySpendingBudgets`, `spendingIncome`, `spendingCategoryRules`, `createPaymentOrConfirm` (via `client/lib/paymentActions.ts`). - File download/upload endpoints use raw `fetch` because responses/bodies are blobs or octet streams. - **All 16 import sites** that referenced `@/api.js` were normalized to `@/api` so Vite/TS resolve the `.ts`. +- **Track C response typing** — see §7 *Domain types* for the full list. In short, the high-traffic domains (`auth`, `bills`, `payments`, `categories`, `tracker`, `snowball`, `subscriptions`, `spending`, `bank-ledger`) all return typed responses via `get` / `post`; the remaining single-consumer non-money casts (adminUsers, version/about/privacy/health/importHistory) are deliberately left as-is. +- **Server error-shape consistency (Track D).** The global 500 handler in `server.js` emits `{ error, code: 'INTERNAL_ERROR' }` so an unexpected server error carries the same `{error, code}` field the client can switch on as every structured 4xx. See §5 *Response conventions*. ### Domain types — `client/types.ts` - Branded `Dollars` and `Cents` money types (from `client/lib/money.ts`). +- `User` (moved from `@/hooks/useAuth` in Track C; re-exported from `useAuth` for the 7 importers). - `Bill`, `Payment`, `Category`, `TrackerResponse` (with `summary` / `bank_tracking` / `cashflow` / `rows`). - `DriftBill`, `AutopaySuggestion`, `AmountSuggestion`, `AutopayStats`. - `BankTransaction` (integer cents on the wire). +- `BankLedgerTransaction extends BankTransaction` (Track C: bank-ledger domain types) — preserves the `Cents` brand. +- `BankAccount`, `BankLedger`, `BankLedgerSummary`, `BankCategoryBreakdown`, `AutoCategorizePreview`, `AutoCategorizeChange`. +- `SnowballProjection` (Track C) + `SnowballProjectionDetail` / `AvalancheProjectionDetail` / `SnowballDebtProjection`. +- `SnowballSettings` (Track C). +- `Subscription`, `SubscriptionTopType`, `SubscriptionSummary`, `SubscriptionsResponse`, `CatalogMatch`, `ExistingBillMatch`, `Recommendation` (Track C: subscriptions domain types) + `RecommendationEvidence` / `RecommendationTransaction` / `SubscriptionTx`. +- `SpendingCategoryEntry`, `SpendingSummary`, `SpendingTransaction`, `SpendingTransactionsResponse`, `SpendingIncomeTx`, `SpendingIncomeResponse`, `SpendingRule`, `CategoryGroup`, `CopyBudgetsResponse` (Track C: spending domain types). - `TimelineBill` for the Safe-to-Spend SVG timeline. - Money fields are typed as `Dollars` for bill/payment/tracker/summary amounts and as `Cents` for raw bank transactions — so the unit mixup bug class is unrepresentable in typed code. - A `money.type-test.ts` never-imported guard uses `@ts-expect-error` to assert each unit mixup is a real compile error, so `tsc --noEmit` fails loudly if the branding ever regresses. +- **Single-consumer non-money casts left in place** (deliberate Track C stop): `adminUsers` with its page-local `AdminUser`, and the one-off `version` / `about` / `privacy` / `health` / `importHistory` doc shapes. Central types add ~nothing for a single reader; typing them is the low-value tail. ### Auth state `client/hooks/useAuth.tsx`: - `AuthContext` typed (a `createContext(null)` would have made `useContext` return `never`). +- `User` interface lives in `client/types.ts` (moved there in Track C; `useAuth` re-exports it for the 7 importers that previously imported it from `@/hooks/useAuth`). - Maintains `user`, `singleUserMode`, `loading`. -- Calls `api.authMode()` and `api.me()` on startup. +- Calls `api.authMode()` and `api.me()` on startup; `api.me()` is now `get` (typed). - Exposes `logout()`, `logoutAll()`, `refresh()`. - `singleUserMode` is sourced from `/auth/mode`; `RequireAuth` consults it. @@ -1373,7 +1398,7 @@ Routes (all rendered through `Routes` with `ErrorBoundary`, lazy + `Suspense` fo ### `client/lib/` - `money.ts` — branded `Cents`/`Dollars` types, `asCents`/`asDollars`/`centsToDollars`/`dollarsToCents`, formatters. A `formatUSD(dollars)` and `formatCentsUSD(cents)` split mirrors the server's `formatUSD`/`formatCentsUSD` split. Inputs are coerced defensively so null / `''` / undefined / NaN never render as `$NaN`. -- `utils.ts` — `cn`, `fmt`, date/format helpers. `fmt` inherits `formatUSD`'s branded-dollars input. +- `utils.ts` — `cn`, `fmt`, date/format helpers, `errMessage(unknown, fallback)` for narrowing `unknown` caught errors into strings. `fmt` inherits `formatUSD`'s branded-dollars input. - `trackerUtils.ts` — row/status/sort helpers, `TrackerRow` domain interface, `TrackerStatus` union. - `billDrafts.ts` — `SourceBill` / `Template` / `Category` shapes for `makeBillDraft`. - `billingSchedule.ts` — `Schedule` union and helpers. @@ -1381,6 +1406,7 @@ Routes (all rendered through `Routes` with `ErrorBoundary`, lazy + `Suspense` fo - `trackerTableColumns.ts` — typed column descriptors. - `reorder.ts` — generic `moveInArray`. - `version.ts` — typed release-notes shapes; the Vite-injected `__APP_VERSION__` is declared inline. +- `paymentActions.ts` — `createPaymentOrConfirm(payload, onSuccess)` (Track A). The deliberate manual-add path treats the server's `409 DUPLICATE_SUSPECTED` as a question rather than an error: losing a legitimately-identical second payment (same bill, date, amount) is itself a money-integrity bug, so the server returns 409 instead of silently deduping and the helper surfaces a sonner "Add anyway?" toast whose confirm action resends with `allow_duplicate: true`. `onSuccess` runs after any real creation (immediate or confirmed retry). Non-duplicate errors re-throw for the caller's existing `catch`. ### Frontend build & lint @@ -1688,4 +1714,17 @@ End-to-end checks run against the codebase: - `npm run typecheck` was green at the time of the v0.41.0 entry in `HISTORY.md` (0 errors); 48 client unit tests pass; 17/17 e2e probe (every page renders, all API paths respond). - `node --check` on every server JS file under `server.js`, `db`, `middleware`, `routes`, `services`, `utils` (`npm run check:server`). +Track A/B/C/D additions (targeting v0.41.0, un-pushed as of this update): + +- `6084896` (Track A) — `fix(money): make every payment balance-mutation atomic` +- `dd5bf92` (Track A) — `fix(money): manual payment path flags suspected dupes (409) w/o losing legit ones` +- `c223f62` (Track B) — `test(money): cross-surface reconciliation harness` +- `b267599` (Track C) — `refactor(ts): type the bank-ledger API responses` +- `8265b4a` (Track C) — `refactor(ts): type the spending API responses` +- `6bb8c63` (Track C) — `refactor(ts): type the subscriptions API responses` +- `65a477c` (Track C) — `refactor(ts): type snowball projection + settings responses` +- `1df4b1b` (Track C/D) — `refactor(ts): move User to @/types + type auth endpoints; add code to 500` + +Diff stats across the 8 commits: 19 files changed, 813 insertions(+), 415 deletions(-). Notable: `client/types.ts` grew by 240 lines; new file `client/lib/paymentActions.ts` (~40 lines, `createPaymentOrConfirm`); `client/api.ts` re-exports 9 new typed methods; `routes/payments.js` grew by ~150 lines for the transactional + duplicate-detection logic; `server.js` gained a single line (`code: 'INTERNAL_ERROR'`). + The previous manual (v0.28.1) contained stale route/page descriptions and missed major features added between v0.28 and v0.40 (subscriptions, spending, snowball plans, SimpleFIN, WebAuthn, TOTP, push notifications, category groups, calendar feed, money-cents migration in progress, React 18 → 19 + TypeScript migration, branded money types). This version replaces that with a current-state engineering reference.