Personal finance dashboard — every number traces back to a source document
A pipeline that reads account statements, investment summary reports, and transaction-confirmation emails into a single database — the dashboard is only a view, never a place a number gets typed in.
Questions that used to mean re-reading several documents became queries · every fact carries its evidence and the date it was last checked · a build gate rejects any number hardcoded into the page.
My financial facts lived in separate documents in separate shapes — monthly account statements, summary reports from investment providers, and confirmation emails arriving in batches. A simple question like how much did I spend this month meant re-reading all of it, and the answer it produced left no trace of which line of which document it came from.
One rule shapes everything else
Every number has to walk back to a source document. That single rule forces the shape of the rest of the system.
- The database is the only source. The summary page is a view of it — there is nowhere for a number to be typed in
- Every fact carries its evidence and the date it was last checked
- A build-time gate rejects numbers hardcoded into the page. The pipeline halts itself rather than waiting to be caught
The third one was the hardest to write, because it meant letting my own tooling refuse my own work.
Missing data has to count as failure, not as empty
I once ran the pipeline after forgetting to set the document unlock password. Every document that needed it was unreadable, that entire side of the data vanished, and the pipeline reported success — because every step did exactly what it was told. Nothing broke. There was simply nothing for some steps to do.
Another time the collector picked documents by a word in the filename. One provider names its files differently, so that provider's documents dropped out completely and silently. The output still looked perfectly reasonable. It was just telling a story from incomplete data.
Both incidents taught the same thing: the expensive failure in this kind of work is not an error, it is a credible-looking result standing on data that partly went missing. So the pipeline was changed to treat no data as a failure rather than an empty value that flows through.
Reconcile two routes that know nothing about each other
Recurring purchases leave a trail on two sides — the per-batch confirmation emails, and the monthly summary report. Those two have to agree every month. When they don't, one side read something wrong, and the job is to find out which one, not to believe whichever looks better.
When retiring the older parser, I used the same method on the code itself: two independently written parsers read the same documents, and their output was compared character by character. Only after everything matched did the old one get deleted. The bugs that surfaced during that comparison all belonged to the old path, and without the comparison I would never have found them.
One trap along the way I only understood after falling into it: a script written to cross-check used to overwrite the files the real pipeline reads. Run it, then run the pipeline, and the page quietly rolls back to the old path's numbers. The rule I took from it is that a verification tool always writes to its own space and never touches what production reads.
Let several agents work at once without stepping on each other
Several streams of work run in parallel here, so there is a ledger of who currently holds which file, a lock on the pipeline so two streams never run it at once, and a shared answer book for questions the owner has already answered, so nobody asks twice.
None of this makes any individual stream smarter. It only stops a stream that is working correctly from deleting another stream's work.
What it doesn't do yet
The gates guarantee only that a number on the page has a source. They do not guarantee the source is right. If a source document itself is wrong, or if I misread it back when I wrote the parser, the gates see nothing — and go on confirming that the wrong number has a clear provenance.
A few values still have to be typed by hand, because they come from email and appear in no report at all. Those are the ones the first rule hasn't reached yet, and I know where they are.
And several of the gates are newly written; they have not run against a full year of real data. A gate that has never fired in a real situation is code whose correctness is still unknown.