Project Every Record Counts: Data Engineering for Backend Engineers
Part 30: "The Streaming Lakehouse" — Real-Time Tables and OLAP
Store managers want stock levels "live", and Rohan wants to commit every second. Ingrid runs the same stock metric at three latencies, an hourly batch, one-minute Iceberg commits and ClickHouse reading Kafka, and measures the freshness and the cost of each. Latency tiers, commit intervals and small files, compaction in the stream, Paimon and Fluss, Spark Real-Time Mode, streaming databases and real-time OLAP.