StreamProcessing

Series Posts

Conceptual Every Record Counts: Data Engineering for Backend Engineers

Part 26: "Time Is a Lie" — Streaming Fundamentals

Season 3 opens with a delivery van that warmed up where nobody could hear it. Rohan's consumer groups readings by arrival and misses the excursion. Event time, windows, watermarks, lateness, triggers, state, exactly-once and the Dataflow model, simulated in Python and previewed in Flink SQL.

Project Every Record Counts: Data Engineering for Backend Engineers

Part 28: "Streaming SQL" — Apache Flink I

Rohan writes Flink SQL like a Postgres report: the live sales tile loses 84 late baskets and 2,338 lines carry prices no till charged. Flink's moving parts, dynamic tables and changelogs, windows, temporal joins, dedup, Top-N, and sinks that append or upsert into Kafka and Iceberg, reconciled store by store against the lakehouse.

Project Every Record Counts: Data Engineering for Backend Engineers

Part 29: "Stateful Streams in Java" — Apache Flink II

Rohan writes the stock-out detector like a Spring service, and a restart pages stores twice. Keyed state, event-time timers, side outputs, checkpoints and savepoints, exactly-once sinks, upgrades and backpressure in Flink's Java DataStream API, proven by killing a TaskManager mid-run.

Conceptual Every Record Counts: Data Engineering for Backend Engineers

Part 35: "Design Review III" — The Whole Platform, and the Interview Playbook

The series ends with a quarter close and a final design review. Rohan defends the whole Larkspur platform, batch, streaming, serving and AI, in interview order: sized from measurements, every failure mode named against the part that caught it, and what still isn't finished as the evolution path. Farah closes Q3 with zero reconciliation exceptions, recomputed from the lakehouse. Then the playbook: a fourteen-step method for data system design interviews, a printable checklist, and three fully worked prompts, including the transit question from Part 0.