#42 - Project-wide Data Input & Sync — Manual + CSV Import + Provider Abstraction - null/BillTracker

null commented

2026-05-16 18:04:05 -05:00

Owner

Umbrella issue for the data input & sync roadmap.

Implementation is broken into 6 phases. Each phase is a separate issue:

#43 — Phase 1: Manual Bill Payments
#44 — Phase 2: Data Sources & Shared Transactions
#45 — Phase 3: CSV Import
#46 — Phase 4: Transaction Matching
#47 — Phase 5: SimpleFIN Provider Sync
#48 — Phase 6: Background Sync & Polish

Phases must be implemented in order. Each builds on the previous.

Original Description

Add a generalized data source layer so BillTracker supports multiple data entry paths — manual, file import, and provider sync — without forcing users into one workflow. Manual-first remains the default.

Data Modes

Manual Entry (default, existing)
File Import (CSV v1, then OFX/QFX later)
Provider Sync (SimpleFIN v1)
Hybrid Mode — mix all workflows per-bill, manual data never overwritten without confirmation

Provider Interface

class DataSourceProvider {
  async connect(input) {}
  async sync(dataSource) {}
  async disconnect(dataSource) {}
  normalizeAccount(rawAccount) {}
  normalizeTransaction(rawTransaction) {}
}

Key Tables

data_sources — type, provider, name, status, config_json, encrypted_secret, last_sync_at, last_error
financial_accounts — data_source_id (FK), provider_account_id, name, org_name, balance, available_balance
transactions — data_source_id (FK), account_id (FK), provider_transaction_id, source_type, amount (cents), description, payee, matched_bill_id (FK → bills), match_status, ignored
bill_payments — bill_id (FK), transaction_id (FK nullable), payment_source, paid_at, amount, notes

Security

Never store bank usernames/passwords
Encrypt provider secrets at rest (TOKEN_ENCRYPTION_KEY)
Never send secrets to frontend or logs
Require HTTPS warning when sync enabled

Acceptance Criteria

App works in manual-only mode
User can manually add a payment to a bill
User can import transactions from CSV
Imported transactions are deduped
User can connect SimpleFIN through backend only
Provider secrets are encrypted at rest
User can manually match/unmatch transactions to bills
User can ignore/unignore transactions
Bill status updates correctly after confirmed matches
Manual bill data not overwritten by imports or sync
Data source settings in UI
Docker env and README docs updated

**Umbrella issue for the data input & sync roadmap.** Implementation is broken into 6 phases. Each phase is a separate issue: 1. **#43** — Phase 1: Manual Bill Payments 2. **#44** — Phase 2: Data Sources & Shared Transactions 3. **#45** — Phase 3: CSV Import 4. **#46** — Phase 4: Transaction Matching 5. **#47** — Phase 5: SimpleFIN Provider Sync 6. **#48** — Phase 6: Background Sync & Polish Phases must be implemented in order. Each builds on the previous. --- ## Original Description Add a generalized data source layer so BillTracker supports multiple data entry paths — manual, file import, and provider sync — without forcing users into one workflow. Manual-first remains the default. ### Data Modes 1. **Manual Entry** (default, existing) 2. **File Import** (CSV v1, then OFX/QFX later) 3. **Provider Sync** (SimpleFIN v1) 4. **Hybrid Mode** — mix all workflows per-bill, manual data never overwritten without confirmation ### Provider Interface ```js class DataSourceProvider { async connect(input) {} async sync(dataSource) {} async disconnect(dataSource) {} normalizeAccount(rawAccount) {} normalizeTransaction(rawTransaction) {} } ``` ### Key Tables - `data_sources` — type, provider, name, status, config_json, encrypted_secret, last_sync_at, last_error - `financial_accounts` — data_source_id (FK), provider_account_id, name, org_name, balance, available_balance - `transactions` — data_source_id (FK), account_id (FK), provider_transaction_id, source_type, amount (cents), description, payee, matched_bill_id (FK → bills), match_status, ignored - `bill_payments` — bill_id (FK), transaction_id (FK nullable), payment_source, paid_at, amount, notes ### Security - Never store bank usernames/passwords - Encrypt provider secrets at rest (TOKEN_ENCRYPTION_KEY) - Never send secrets to frontend or logs - Require HTTPS warning when sync enabled ### Acceptance Criteria - [ ] App works in manual-only mode - [ ] User can manually add a payment to a bill - [ ] User can import transactions from CSV - [ ] Imported transactions are deduped - [ ] User can connect SimpleFIN through backend only - [ ] Provider secrets are encrypted at rest - [ ] User can manually match/unmatch transactions to bills - [ ] User can ignore/unignore transactions - [ ] Bill status updates correctly after confirmed matches - [ ] Manual bill data not overwritten by imports or sync - [ ] Data source settings in UI - [ ] Docker env and README docs updated