Budget Request Analysis
A tool for analyzing budget requests from Excel — converted to Parquet and queried with DuckDB for fast, auditable analysis.
Turns many files of requests into one repeatable command, with every answer traceable back to its source file.
Budget-request data arrives as spreadsheet exports from an upstream system, several times a year — this tool is what makes those files answerable in SQL, and answerable again every round.
Convert, then ask
The tool converts each export into Parquet — a compressed, column-oriented storage format — then asks its questions in SQL through DuckDB, which reads that store directly, taking only the columns and rows a question needs. Nothing gets lifted into memory wholesale the way the earlier approach did, and no database server has to be stood up.
A question is kept as a file and re-run whenever a fresh export lands.
The mess is upstream
These files are made for people to read, not for machines. Column names and sheet names differ between export rounds, some sheets stack their headers over several rows and mix summary rows in with the data, and some columns carry alphabetic codes and numerics side by side.
The response is to distrust the file's shape: ingest matches column names against an alias table, finds the sheet that fits best on its own, pins data types, then halts the moment a required column goes missing — without overwriting what is already there. Failing loudly beats failing quietly. Totals already known get re-checked on the way in, to catch a silent misread.
A loop you can press again
The end state is a command that does the whole run — ingest, rebuild every report, check the system's health — and any failing step stops it. The checks do not depend on particular values, so fresh data does not turn them red for the wrong reason. A new export drops in place, the run starts, and the rest rebuilds itself.
The lesson
What paid off most was keeping raw data, transformation steps, and answers strictly apart — raw data nobody edits, transformations that live in code, answers treated as always regenerable. Then, when figures shift, there is no guessing about which copy is real.