How a DAG flows — when a pipeline breaks halfway, which steps run again

A playable orchestrator built alongside the Airflow handbook. A seven-step monthly file pipeline (receive three branch files, merge and check, transform, load, notify) is drawn as two views of the same state side by side: a graph that says what waits for what, and a table that says where things stand (rows are steps, columns are runs, each cell holds a state and a try number). Tap a step in the graph to make the next run fail there transiently or permanently, set retries from 0 to 3, choose whether notify runs when its upstream succeeds or merely finishes, and choose whether load writes to the destination by appending (append) or by replacing the whole month (replace_month); then step the loop one tick at a time, start new runs to add columns, open a cell to see its tries one by one, or Clear it — after guessing which cells will be reset, and choosing whether to rerun only that cell and its downstream, as an orchestrator does, or the whole thing, as a single script does. Under the table a "warehouse rows" line counts each month's rows, using the numbers from the stock_load.py example in chapter 12. All of the logic is the model from chapter 4 of the book; no Airflow runs on the web.

First, chapter 4's model is small enough to rewrite in TypeScript in full without drift: the loop's three rules (all upstream succeeded = run it, upstream broke = upstream_failed, otherwise wait) plus the one rule that makes it agree — every cell in a tick is decided from the table as it stood at the start of the tick. The entry's test pulls the recorded output of section 4.3 straight out of the book's file and compares it line by line; all 8 ticks, the final table and both run states match, so the entry and the book are tied together by a test rather than by the author's memory. Second, the entry's default (transform fails permanently, notify set to all_done) runs to a success run while the counters read 1 failed cell and 0 failed runs — the same pair as the card on the real Grid page in chapter 7. Flipping that one switch to all_success turns the next run red. Third, chapter 12's stock_load.py example fits the state machine as a single rule (each time load runs it writes to the destination once, either the whole file or breaking after row 50), and the test matches the book's section 12.2 output lines character for character for both modes. With retries at 0 and load failing permanently, clearing load twice gives 50 → 149 → 248 and 0 → 99 → 99, as in the book, while the load cell reads success in both modes — the state table really cannot tell the two apart. Nobody even has to press Clear for the count to double: a transient load failure with retries at 1 reaches 149 in one run-until-idle, because a retry is an automatic rerun. Fourth, the orchestrator-versus-single-script pair differs most when the break comes after load has finished writing: with notify failing, clearing notify reruns one step and the rows stay at 99, while a full rerun reruns all 7 steps and append becomes 198 but replace_month stays at 99. The two switches multiply — the damage from rerunning everything comes from steps that are not safe to rerun, not from rerunning as such. Fifth, three problems only showed up in real screenshots while every test passed: the first version labelled every graph node "next run: no fault", drowning the one node with a fault — now only nodes with a fault carry a label; after "Run until idle" the log box stayed parked on the first ticks — it now scrolls to the newest line on every append; and the note under the table sat inside the table's horizontal scroll box, so on a 360-pixel screen it slid out of view once the table scrolled to a later column — it now sits outside the box.

1. LAB.07 The gate at the exit — how a green build still ships a wrong page
2. LAB.11 How a DAG flows — when a pipeline breaks halfway, which steps run again (this entry)