Lakehouse
6 items
Series Posts
Part 0: "The Field Guide" — How to Read This Series
Larkspur's quarterly review ends with three headline numbers and two answers for each. Rohan, a staff backend engineer, is asked to "just fix the queries". This field guide maps data engineering in 2026, introduces the retailer and its incident, starts the companion lab, reproduces all three disagreements with SQL, and states the design question the series teaches you to answer.
Part 1: "Three Houses for Data" — Warehouse, Lake, Lakehouse
Rohan wants a bigger read replica. Ingrid runs an experiment instead: six million sales lines in PostgreSQL, a folder of Parquet files and an Iceberg table, the same questions asked of each and one row corrected in each. OLTP vs OLAP, row vs column storage, and how warehouses, lakes and lakehouses differ.
Part 7: "The Table That Remembers" — Apache Iceberg in Depth
Q3 is closed, and UK till sales just moved by £3,494.82. Ingrid opens an Iceberg table file by file: a catalog pointer, JSON metadata, Avro manifests, Parquet. Snapshots, optimistic commits and a real conflict between Spark and PyIceberg, schema and partition evolution without rewrites, time travel, tags and branches, copy-on-write vs merge-on-read, and a format v3 table with VARIANT and deletion vectors.
Part 8: "Format Wars, Ceasefire" — Delta, Hudi, Paimon, DuckLake and Interoperability
Rohan builds a feature matrix to pick a table format and every row says yes. Ingrid puts one gold table in Iceberg, Delta Lake and DuckLake and runs one late basket through each: six files, four files, or two inlined rows and a snapshot record in PostgreSQL. Delta logs and checkpoints, Hudi and Paimon for upserts, DuckLake's SQL catalog and data inlining, UniForm and XTable, and why the catalog decides more than the format.
Part 10: "Where the Bytes Live" — Partitioning, File Layout and Table Maintenance
Four years of sales lines are coming, and Rohan wants partitions by day, country and store, like indexes. Ingrid builds the same 72.6 million rows three ways and measures. Partition goals and anti-goals, Iceberg transforms, event date vs ingestion date, sort orders and Z-order, the small-file problem, compaction, snapshot expiry, orphan files, and a partition-strategy playbook.
Part 14: "Design Review I" — The Storage and Compute Layer
Season 1 ends with a design review. Rohan defends Larkspur's storage, table format, catalog, partitioning and engines against finance's volumes, retention, freshness, concurrency and cost, in interview order: clarify, estimate from the lab's own files, choose, justify, trade-offs, and what would change the decision. Plus a self-host vs managed matrix, a printable checklist and a worked drill.