Conceptual Every Record Counts: Data Engineering for Backend Engineers
Part 8: "Format Wars, Ceasefire" — Delta, Hudi, Paimon, DuckLake and Interoperability
Rohan builds a feature matrix to pick a table format and every row says yes. Ingrid puts one gold table in Iceberg, Delta Lake and DuckLake and runs one late basket through each: six files, four files, or two inlined rows and a snapshot record in PostgreSQL. Delta logs and checkpoints, Hudi and Paimon for upserts, DuckLake's SQL catalog and data inlining, UniForm and XTable, and why the catalog decides more than the format.