Project Every Record Counts: Data Engineering for Backend Engineers
Part 10: "Where the Bytes Live" — Partitioning, File Layout and Table Maintenance
Four years of sales lines are coming, and Rohan wants partitions by day, country and store, like indexes. Ingrid builds the same 72.6 million rows three ways and measures. Partition goals and anti-goals, Iceberg transforms, event date vs ingestion date, sort orders and Z-order, the small-file problem, compaction, snapshot expiry, orphan files, and a partition-strategy playbook.