SystemDesign

Series Posts

Conceptual Every Record Counts: Data Engineering for Backend Engineers

Part 0: "The Field Guide" — How to Read This Series

Larkspur's quarterly review ends with three headline numbers and two answers for each. Rohan, a staff backend engineer, is asked to "just fix the queries". This field guide maps data engineering in 2026, introduces the retailer and its incident, starts the companion lab, reproduces all three disagreements with SQL, and states the design question the series teaches you to answer.

Conceptual Every Record Counts: Data Engineering for Backend Engineers

Part 14: "Design Review I" — The Storage and Compute Layer

Season 1 ends with a design review. Rohan defends Larkspur's storage, table format, catalog, partitioning and engines against finance's volumes, retention, freshness, concurrency and cost, in interview order: clarify, estimate from the lab's own files, choose, justify, trade-offs, and what would change the decision. Plus a self-host vs managed matrix, a printable checklist and a worked drill.

Conceptual Every Record Counts: Data Engineering for Backend Engineers

Part 25: "Design Review II" — The Batch Reporting Platform, End to End

Season 2 ends where it began: with Farah and a quarter to sign. Rohan defends Larkspur's batch platform end to end in interview order, from sources and ingestion through contracts, reconciliation, governance and cost: sized from the lab's own tables, with every failure mode named against the part that caught it, what changes at ten times the volume, and what still isn't finished. Then the board pack is recomputed from the lakehouse and the restatement approved. Plus a self-host vs managed matrix, a printable checklist and a clinical-trials drill.