Compare commits

..

No commits in common. "278521a612a722f4dc6478915c929cfc61409ff5" and "a88d5c4647376161e7832c4dbb6c649ea41b5cc2" have entirely different histories.

8 changed files with 135 additions and 196 deletions

View File

@ -10,16 +10,6 @@
- **Migration version sync assertion**`_runMigrationVersions` module-level variable is now populated by `runMigrations()` before its loop runs. `reconcileLegacyMigrations()` — which runs after `runMigrations()` on legacy-DB upgrade paths — compares its own version array against the stored list and throws a descriptive error if any version appears in one array but not the other. Catches drift between the two migration arrays at startup rather than silently misconfiguring a legacy schema.
- **Encryption key fully app-managed — no env var required**`TOKEN_ENCRYPTION_KEY` environment variable support removed entirely. The auto-generated DB key (`_auto_encryption_key` in the settings table) is now the primary mechanism, not a fallback. The `[security]` warning that fired on every startup when no env var was set is gone. On first startup a 48-byte cryptographically random key is generated and persisted to the database; subsequent restarts reuse it. All existing encrypted data (SMTP password, OIDC secret, SimpleFIN tokens, push notification tokens) continues to decrypt correctly.
- **TrackerPage crash fixed — `activeTotalExpected` temporal dead zone** — The `cashflow` block added in an earlier change referenced `activeTotalExpected` and `activePaidTowardDue` before their `const` declarations. JavaScript's temporal dead zone caused `Cannot access 'activeTotalExpected' before initialization` on every tracker load. Fixed by moving the four `active*` declarations above the cashflow block that depends on them.
- **Bank merchant rule matching — `balance_delta`, `current_balance`, and error handling**`billMerchantRuleService.js` was the only payment creation path still missing `balance_delta` computation and `current_balance` update after the #49 fix. Both `applyMerchantRules` (full-user batch matching, called on every sync) and `syncBillPaymentsFromSimplefin` (single-bill retroactive match) now read the bill fresh before each payment, compute `computeBalanceDelta`, include `balance_delta` in the INSERT, and update `bills.current_balance`. `payment_source` corrected from `'auto_match'` (not in `VALID_PAYMENT_SOURCES`) to `'provider_sync'`. DB migration `v0.82` updates all historical `auto_match` records. Both functions also gained full error handling — `txRows` queries, the fallback notes query, and `db.transaction()` blocks are all wrapped with try/catch that logs to console and returns safe zero-match defaults instead of propagating exceptions to the sync worker or API caller.
- **`applyMerchantRules` returns matched bill names** — The function now returns `{ matched, matched_bills: ['AT&T', 'Netflix'] }` instead of just `{ matched }`. Bill name is fetched via JOIN in the rules query. The name set is deduplicated via `Map` keyed by bill_id (so one bill matched by multiple transactions still appears once). The name list is bubbled through `bankSyncService.runSync` and the `POST /api/data-sources/sync-all` endpoint (using a `Set` to dedupe across multiple SimpleFIN sources). TrackerPage Sync Bank toast now reads **"Synced — AT&T, Netflix ✓"** instead of "3 payments matched", with a `(+N more)` suffix when there are more matched bills than bill names to display. Toast duration extended to 5 seconds.
- **BillModal — bank matching Sync button moved, fixed, and made reactive** — The old "SimpleFIN payment history" sync button in the Subscriptions tab was removed (it didn't call `loadLinkedTransactions()` after success and was less discoverable). A new **Sync** button lives in the Bank Matching Rules section header of the Transactions tab, visible whenever `localHasRules` is true. A local `localHasRules` state initialises from `sourceBill.has_merchant_rule` but flips to `true` immediately when `onRulesChanged` fires — so the button appears right after adding the first rule without closing and reopening the modal. After a successful sync, `loadLinkedTransactions()` is called to refresh the Linked transactions list below, and `refetch()` updates the parent tracker view.
- **Pin Due — urgent bills float to top of tracker** — A "Pin Due" toggle button in the TrackerPage header sorts overdue and due-soon bills to the top of each bucket when enabled. Priority order: `missed``late``due_soon``upcoming` → everything else; ties broken by `due_day`. The sort runs after filtering but before the bucket split, so each half-month bucket is sorted independently. The button uses `variant="default"` (solid) when active and `variant="outline"` when off so the current mode is always visible. Preference persists across sessions via `localStorage` under `tracker_pin_upcoming`. Drag reorder is automatically disabled while the toggle is on (`reorderEnabled` now also requires `!pinUpcoming`) since the two modes conflict.
- **Tracker row keyboard navigation** — Tracker rows (desktop table view) are now keyboard navigable. Each row has `tabIndex={0}`, `data-tracker-row`, `aria-rowindex`, and an `aria-label` announcing the bill name, status, and due day. A `focus-visible:ring-2 ring-primary/60 ring-inset` focus ring appears on keyboard focus only. Key bindings: `↓`/`j` focuses the next row, `↑`/`k` the previous (both cross bucket boundaries via `querySelectorAll('[data-tracker-row]')`), `Enter` opens the edit modal, `P` toggles paid/unpaid (skipped bills ignored, `Ctrl+P`/`Cmd+P` passes through to the browser), `Esc` blurs the row. The `onKeyDown` handler guards against firing on nested interactive elements with `if (e.target !== e.currentTarget) return`.

View File

@ -143,9 +143,6 @@ export default function BillModal({ bill, initialBill, categories, onClose, onSa
const [snowballInclude, setSnowballInclude] = useState(!!sourceBill?.snowball_include);
const [snowballExempt, setSnowballExempt] = useState(!!sourceBill?.snowball_exempt);
const [syncingPayments, setSyncingPayments] = useState(false);
// Track whether rules exist locally so the Sync button appears immediately
// after the first rule is added without waiting for sourceBill to refetch.
const [localHasRules, setLocalHasRules] = useState(!!sourceBill?.has_merchant_rule);
const [showDebtSection, setShowDebtSection] = useState(
() => isDebtCat(categories, sourceBill?.category_id ? String(sourceBill.category_id) : CAT_NONE)
|| !!sourceBill?.snowball_include
@ -697,7 +694,44 @@ export default function BillModal({ bill, initialBill, categories, onClose, onSa
/>
<p className="text-[10px] text-muted-foreground/70">0-30 days before renewal.</p>
</div>
{/* Bank sync button moved to the Transactions tab → Bank Matching Rules section */}
{!isNew && (sourceBill?.has_merchant_rule || ['simplefin_recommendation', 'catalog_match'].includes(sourceBill?.subscription_source)) ? (
<div className="col-span-2 border-t border-border/40 pt-3">
<div className="flex items-center justify-between gap-3">
<div>
<p className="text-xs font-medium text-muted-foreground">SimpleFIN payment history</p>
<p className="text-[10px] text-muted-foreground/70 mt-0.5">
Scan unmatched bank transactions and backfill any missing payments.
</p>
</div>
<Button
type="button"
size="sm"
variant="outline"
className="shrink-0 gap-1.5"
disabled={syncingPayments}
onClick={async () => {
setSyncingPayments(true);
try {
const result = await api.syncBillSimplefinPayments(sourceBill.id);
if (result.added > 0) {
toast.success(`Synced ${result.added} payment${result.added !== 1 ? 's' : ''} from SimpleFIN.`);
} else {
toast.info('No new payments found in SimpleFIN history.');
}
} catch (err) {
toast.error(err.message || 'Sync failed.');
} finally {
setSyncingPayments(false);
}
}}
>
{syncingPayments
? <><Loader2 className="h-3.5 w-3.5 animate-spin" />Syncing</>
: <><RefreshCw className="h-3.5 w-3.5" />Sync payments</>}
</Button>
</div>
</div>
) : null}
</div>
)}
</div>
@ -1009,50 +1043,16 @@ export default function BillModal({ bill, initialBill, categories, onClose, onSa
{/* Bank Matching Rules */}
{!isNew && (
<div className="mt-4 rounded-lg border border-border/60 bg-background/35">
<div className="flex items-center justify-between gap-3 border-b border-border/50 px-3 py-2.5">
<div>
<p className="text-xs font-semibold uppercase tracking-wider text-muted-foreground">Bank matching rules</p>
<p className="mt-0.5 text-[11px] text-muted-foreground/70">
Transactions whose description contains these patterns are automatically imported as payments.
</p>
</div>
{localHasRules && (
<Button
type="button"
size="sm"
variant="outline"
className="shrink-0 gap-1.5 text-xs"
disabled={syncingPayments}
title="Scan unmatched bank transactions and import any matching payments for this bill"
onClick={async () => {
setSyncingPayments(true);
try {
const result = await api.syncBillSimplefinPayments(sourceBill.id);
if (result.added > 0) {
toast.success(`${result.added} payment${result.added !== 1 ? 's' : ''} imported from bank history.`);
loadLinkedTransactions?.();
refetch?.();
} else {
toast.info('No new matching transactions found.');
}
} catch (err) {
toast.error(err.message || 'Sync failed.');
} finally {
setSyncingPayments(false);
}
}}
>
{syncingPayments
? <><Loader2 className="h-3.5 w-3.5 animate-spin" />Syncing</>
: <><RefreshCw className="h-3.5 w-3.5" />Sync</>}
</Button>
)}
<div className="border-b border-border/50 px-3 py-2.5">
<p className="text-xs font-semibold uppercase tracking-wider text-muted-foreground">Bank matching rules</p>
<p className="mt-0.5 text-[11px] text-muted-foreground/70">
Transactions whose description contains these patterns are automatically imported as payments.
</p>
</div>
<div className="px-3 py-3">
<BillMerchantRules
billId={sourceBill?.id}
onRulesChanged={() => {
setLocalHasRules(true);
refetch?.();
loadLinkedTransactions?.();
}}

View File

@ -101,17 +101,10 @@ export default function TrackerPage() {
setBankSyncing(true);
try {
const result = await api.syncAllSources();
const matched = result.auto_matched ?? 0;
const newTx = result.transactions_new ?? 0;
const billNames = result.matched_bills ?? [];
if (matched > 0 && billNames.length > 0) {
toast.success(
`Synced — ${billNames.join(', ')}` +
(matched > billNames.length ? ` (+${matched - billNames.length} more)` : ''),
{ duration: 5000 }
);
} else if (matched > 0) {
toast.success(`Synced — ${matched} payment${matched === 1 ? '' : 's'} matched`);
const matched = result.auto_matched ?? 0;
const newTx = result.transactions_new ?? 0;
if (matched > 0) {
toast.success(`Synced — ${matched} payment${matched === 1 ? '' : 's'} matched${newTx > matched ? `, ${newTx} new transactions` : ''}`);
} else if (newTx > 0) {
toast.success(`Synced — ${newTx} new transaction${newTx === 1 ? '' : 's'}, no automatic matches`);
} else {

View File

@ -2694,17 +2694,6 @@ function runMigrations() {
`);
console.log('[v0.81] bill_merchant_rules composite index added');
}
},
{
version: 'v0.82',
description: 'payments: normalise auto_match source to provider_sync',
dependsOn: ['v0.81'],
run: function() {
const result = db.prepare(
"UPDATE payments SET payment_source = 'provider_sync' WHERE payment_source = 'auto_match'"
).run();
console.log(`[v0.82] Normalised ${result.changes} auto_match payment(s) to provider_sync`);
}
}
];

View File

@ -249,7 +249,6 @@ router.post('/sync-all', async (req, res) => {
let transactionsNew = 0;
let transactionsSkip = 0;
let autoMatched = 0;
const matchedBillSet = new Set(); // dedupe across multiple sources
const errors = [];
for (const source of sources) {
@ -259,7 +258,6 @@ router.post('/sync-all', async (req, res) => {
transactionsNew += result.transactionsNew ?? 0;
transactionsSkip += result.transactionsSkip ?? 0;
autoMatched += result.autoMatched ?? 0;
for (const name of result.matched_bills ?? []) matchedBillSet.add(name);
} catch (err) {
errors.push(sanitizeErrorMessage(err?.message || 'Sync failed'));
}
@ -270,7 +268,6 @@ router.post('/sync-all', async (req, res) => {
transactions_new: transactionsNew,
transactions_skip: transactionsSkip,
auto_matched: autoMatched,
matched_bills: [...matchedBillSet],
errors,
});
} catch (err) {

View File

@ -130,9 +130,9 @@ async function runSync(db, userId, dataSource, { days } = {}) {
`).run(partialError, dataSource.id, userId);
// Apply any stored merchant→bill rules to newly synced transactions
const { matched: autoMatched, matched_bills: matchedBills } = applyMerchantRules(db, userId);
const { matched: autoMatched } = applyMerchantRules(db, userId);
return { accountsUpserted, transactionsNew, transactionsSkip, autoMatched, matched_bills: matchedBills || [], errlist: raw._errlistSummary || null };
return { accountsUpserted, transactionsNew, transactionsSkip, autoMatched, errlist: raw._errlistSummary || null };
}
// ─── Public API ───────────────────────────────────────────────────────────────

View File

@ -1,7 +1,6 @@
'use strict';
const { normalizeMerchant } = require('./subscriptionService');
const { computeBalanceDelta } = require('./billsService');
// Persist a merchant→bill rule so future synced transactions auto-match.
function addMerchantRule(db, userId, billId, merchant) {
@ -25,7 +24,7 @@ function applyMerchantRules(db, userId) {
let rules;
try {
rules = db.prepare(`
SELECT bmr.bill_id, bmr.merchant, b.name AS bill_name
SELECT bmr.bill_id, bmr.merchant
FROM bill_merchant_rules bmr
JOIN bills b ON b.id = bmr.bill_id AND b.user_id = bmr.user_id AND b.deleted_at IS NULL
WHERE bmr.user_id = ?
@ -34,75 +33,54 @@ function applyMerchantRules(db, userId) {
return { matched: 0 };
}
if (rules.length === 0) return { matched: 0, matched_bills: [] };
if (rules.length === 0) return { matched: 0 };
let txRows;
try {
txRows = db.prepare(`
SELECT t.id, t.amount, t.payee, t.description, t.memo, t.posted_date, t.transacted_at
FROM transactions t
LEFT JOIN financial_accounts fa ON fa.id = t.account_id AND fa.user_id = t.user_id
WHERE t.user_id = ?
AND t.match_status = 'unmatched'
AND t.ignored = 0
AND t.amount < 0
AND (t.account_id IS NULL OR fa.id IS NULL OR fa.monitored = 1)
`).all(userId);
} catch (err) {
console.error('[applyMerchantRules] Failed to fetch transactions:', err.message);
return { matched: 0, matched_bills: [] };
}
const txRows = db.prepare(`
SELECT t.id, t.amount, t.payee, t.description, t.memo, t.posted_date, t.transacted_at
FROM transactions t
LEFT JOIN financial_accounts fa ON fa.id = t.account_id AND fa.user_id = t.user_id
WHERE t.user_id = ?
AND t.match_status = 'unmatched'
AND t.ignored = 0
AND t.amount < 0
AND (t.account_id IS NULL OR fa.id IS NULL OR fa.monitored = 1)
`).all(userId);
if (txRows.length === 0) return { matched: 0, matched_bills: [] };
if (txRows.length === 0) return { matched: 0 };
const getBill = db.prepare('SELECT current_balance, interest_rate FROM bills WHERE id = ? AND deleted_at IS NULL');
const insertPayment = db.prepare(`
INSERT OR IGNORE INTO payments (bill_id, amount, paid_date, payment_source, transaction_id, balance_delta)
VALUES (?, ?, ?, 'provider_sync', ?, ?)
INSERT OR IGNORE INTO payments (bill_id, amount, paid_date, payment_source, transaction_id)
VALUES (?, ?, ?, 'auto_match', ?)
`);
const updateBalance = db.prepare("UPDATE bills SET current_balance = ?, updated_at = datetime('now') WHERE id = ?");
const updateTx = db.prepare(`
const updateTx = db.prepare(`
UPDATE transactions
SET matched_bill_id = ?, match_status = 'matched', updated_at = datetime('now')
WHERE id = ? AND user_id = ? AND match_status = 'unmatched'
`);
let matched = 0;
const matchedBills = new Map(); // bill_id → bill_name for the summary
try {
db.transaction(() => {
for (const tx of txRows) {
const txMerchant = normalizeMerchant(tx.payee || tx.description || tx.memo || '');
if (!txMerchant) continue;
db.transaction(() => {
for (const tx of txRows) {
const txMerchant = normalizeMerchant(tx.payee || tx.description || tx.memo || '');
if (!txMerchant) continue;
const rule = rules.find(r =>
txMerchant.includes(r.merchant) || r.merchant.includes(txMerchant)
);
if (!rule) continue;
const rule = rules.find(r =>
txMerchant.includes(r.merchant) || r.merchant.includes(txMerchant)
);
if (!rule) continue;
const paidDate = tx.posted_date || (tx.transacted_at ? String(tx.transacted_at).slice(0, 10) : null);
if (!paidDate) continue;
const paidDate = tx.posted_date || (tx.transacted_at ? String(tx.transacted_at).slice(0, 10) : null);
if (!paidDate) continue;
const amount = Math.round(Math.abs(tx.amount)) / 100;
const bill = getBill.get(rule.bill_id);
const balCalc = bill ? computeBalanceDelta(bill, amount) : null;
const amount = Math.round(Math.abs(tx.amount)) / 100;
insertPayment.run(rule.bill_id, amount, paidDate, tx.id);
updateTx.run(rule.bill_id, tx.id, userId);
matched++;
}
})();
const result = insertPayment.run(rule.bill_id, amount, paidDate, tx.id, balCalc?.balance_delta ?? null);
if (result.changes > 0) {
if (balCalc) updateBalance.run(balCalc.new_balance, rule.bill_id);
updateTx.run(rule.bill_id, tx.id, userId);
matched++;
matchedBills.set(rule.bill_id, rule.bill_name || `Bill #${rule.bill_id}`);
}
}
})();
} catch (err) {
console.error('[applyMerchantRules] Transaction failed, no payments recorded:', err.message);
return { matched: 0, matched_bills: [] };
}
return { matched, matched_bills: [...matchedBills.values()] };
return { matched };
}
// Sync all unmatched SimpleFIN transactions for a single bill using its stored
@ -123,79 +101,56 @@ function syncBillPaymentsFromSimplefin(db, userId, billId) {
// Fallback: extract merchant from notes "Detected from recurring merchant: X"
if (rules.length === 0) {
try {
const bill = db.prepare('SELECT notes FROM bills WHERE id = ? AND user_id = ? AND deleted_at IS NULL').get(billId, userId);
const match = bill?.notes?.match(/Detected from recurring merchant:\s*(.+)/i);
if (match) {
const extracted = normalizeMerchant(match[1].trim());
if (extracted && extracted.length >= 3) {
addMerchantRule(db, userId, billId, extracted);
rules = [extracted];
}
const bill = db.prepare('SELECT notes FROM bills WHERE id = ? AND user_id = ? AND deleted_at IS NULL').get(billId, userId);
const match = bill?.notes?.match(/Detected from recurring merchant:\s*(.+)/i);
if (match) {
const extracted = normalizeMerchant(match[1].trim());
if (extracted && extracted.length >= 3) {
addMerchantRule(db, userId, billId, extracted);
rules = [extracted];
}
} catch (err) {
console.error('[syncBillPaymentsFromSimplefin] Failed to read bill notes:', err.message);
}
if (rules.length === 0) return { added: 0 };
}
let txRows;
try {
txRows = db.prepare(`
SELECT t.id, t.amount, t.payee, t.description, t.memo, t.posted_date, t.transacted_at
FROM transactions t
LEFT JOIN financial_accounts fa ON fa.id = t.account_id AND fa.user_id = t.user_id
WHERE t.user_id = ?
AND t.match_status = 'unmatched'
AND t.ignored = 0
AND t.amount < 0
AND (t.account_id IS NULL OR fa.id IS NULL OR fa.monitored = 1)
`).all(userId);
} catch (err) {
console.error('[syncBillPaymentsFromSimplefin] Failed to fetch transactions:', err.message);
return { added: 0 };
}
const txRows = db.prepare(`
SELECT t.id, t.amount, t.payee, t.description, t.memo, t.posted_date, t.transacted_at
FROM transactions t
LEFT JOIN financial_accounts fa ON fa.id = t.account_id AND fa.user_id = t.user_id
WHERE t.user_id = ?
AND t.match_status = 'unmatched'
AND t.ignored = 0
AND t.amount < 0
AND (t.account_id IS NULL OR fa.id IS NULL OR fa.monitored = 1)
`).all(userId);
if (txRows.length === 0) return { added: 0 };
const getBill = db.prepare('SELECT current_balance, interest_rate FROM bills WHERE id = ? AND deleted_at IS NULL');
const insertPayment = db.prepare(`
INSERT OR IGNORE INTO payments (bill_id, amount, paid_date, payment_source, transaction_id, balance_delta)
VALUES (?, ?, ?, 'provider_sync', ?, ?)
INSERT OR IGNORE INTO payments (bill_id, amount, paid_date, payment_source, transaction_id)
VALUES (?, ?, ?, 'auto_match', ?)
`);
const updateBalance = db.prepare("UPDATE bills SET current_balance = ?, updated_at = datetime('now') WHERE id = ?");
const updateTx = db.prepare(`
const updateTx = db.prepare(`
UPDATE transactions
SET matched_bill_id = ?, match_status = 'matched', updated_at = datetime('now')
WHERE id = ? AND user_id = ? AND match_status = 'unmatched'
`);
let added = 0;
try {
db.transaction(() => {
for (const tx of txRows) {
const txMerchant = normalizeMerchant(tx.payee || tx.description || tx.memo || '');
if (!txMerchant) continue;
const matches = rules.some(r => txMerchant.includes(r) || r.includes(txMerchant));
if (!matches) continue;
const paidDate = tx.posted_date || (tx.transacted_at ? String(tx.transacted_at).slice(0, 10) : null);
if (!paidDate) continue;
const amount = Math.round(Math.abs(tx.amount)) / 100;
const bill = getBill.get(billId);
const balCalc = bill ? computeBalanceDelta(bill, amount) : null;
const result = insertPayment.run(billId, amount, paidDate, tx.id, balCalc?.balance_delta ?? null);
if (result.changes > 0) {
if (balCalc) updateBalance.run(balCalc.new_balance, billId);
updateTx.run(billId, tx.id, userId);
added++;
}
}
})();
} catch (err) {
console.error('[syncBillPaymentsFromSimplefin] Transaction failed, no payments recorded:', err.message);
return { added: 0 };
}
db.transaction(() => {
for (const tx of txRows) {
const txMerchant = normalizeMerchant(tx.payee || tx.description || tx.memo || '');
if (!txMerchant) continue;
const matches = rules.some(r => txMerchant.includes(r) || r.includes(txMerchant));
if (!matches) continue;
const paidDate = tx.posted_date || (tx.transacted_at ? String(tx.transacted_at).slice(0, 10) : null);
if (!paidDate) continue;
const amount = Math.round(Math.abs(tx.amount)) / 100;
insertPayment.run(billId, amount, paidDate, tx.id);
updateTx.run(billId, tx.id, userId);
added++;
}
})();
return { added };
}

View File

@ -10,11 +10,26 @@ const HKDF_INFO = 'bill-tracker-token-encryption-v1';
// Prefix that identifies ciphertext produced with HKDF key derivation.
const V2_PREFIX = 'v2:';
// Returns the raw key material (IKM) without derivation.
// The encryption key is auto-generated on first run and stored in the database
// under `_auto_encryption_key`. No environment variable required — all settings
// live in the app. The key is created once and reused across restarts.
let _warnedAutoKey = false;
// Returns the raw key material (IKM) without derivation — shared by both paths.
function getIkm() {
const envRaw = process.env.TOKEN_ENCRYPTION_KEY || '';
if (envRaw) {
const buf = Buffer.from(envRaw, 'utf8');
if (buf.length < 32) throw new Error('TOKEN_ENCRYPTION_KEY must be at least 32 bytes');
return buf;
}
if (!_warnedAutoKey) {
_warnedAutoKey = true;
console.warn(
'[security] TOKEN_ENCRYPTION_KEY is not set. Using an auto-generated key ' +
'stored in the database alongside the encrypted data. Set TOKEN_ENCRYPTION_KEY ' +
'as an environment variable to keep the key separate from the data it protects.',
);
}
// Lazy-require to avoid circular dependency at module load time
const { getSetting, setSetting } = require('../db/database');
let stored = getSetting('_auto_encryption_key');