ApacheIceberg
13 items
Series Posts
Part 1: "Three Houses for Data" — Warehouse, Lake, Lakehouse
Rohan wants a bigger read replica. Ingrid runs an experiment instead: six million sales lines in PostgreSQL, a folder of Parquet files and an Iceberg table, the same questions asked of each and one row corrected in each. OLTP vs OLAP, row vs column storage, and how warehouses, lakes and lakehouses differ.
Part 7: "The Table That Remembers" — Apache Iceberg in Depth
Q3 is closed, and UK till sales just moved by £3,494.82. Ingrid opens an Iceberg table file by file: a catalog pointer, JSON metadata, Avro manifests, Parquet. Snapshots, optimistic commits and a real conflict between Spark and PyIceberg, schema and partition evolution without rewrites, time travel, tags and branches, copy-on-write vs merge-on-read, and a format v3 table with VARIANT and deletion vectors.
Part 9: "Who Owns the Truth" — Catalogs and the Iceberg REST Protocol
Finance wants to query gold sales, and Rohan offers them a read key for the bucket. Ingrid shows what else that key reads. The catalog as control plane: from Hive Metastore to the Iceberg REST protocol, Polaris principals, roles and grants, credential vending through STS, four engines on one table, and an analyst refused on raw by the catalog and by the storage itself.
Part 10: "Where the Bytes Live" — Partitioning, File Layout and Table Maintenance
Four years of sales lines are coming, and Rohan wants partitions by day, country and store, like indexes. Ingrid builds the same 72.6 million rows three ways and measures. Partition goals and anti-goals, Iceberg transforms, event date vs ingestion date, sort orders and Z-order, the small-file problem, compaction, snapshot expiry, orphan files, and a partition-strategy playbook.
Part 11: "You Might Not Need a Cluster" — DuckDB, Polars and Single-Node Analytics
Rohan wants a Spark cluster for four years of sales lines. Ingrid runs the same Iceberg table on a laptop with DuckDB and Polars instead, and profiles it against Spark. Vectorised execution, out-of-core processing, Arrow hand-offs, DuckDB 2.0 and Polars 2.x over a REST catalog, and the signals that say one machine is no longer enough.
Part 12: "Shuffle, Spill and Skew" — Distributed Batch with Spark 4.1
A campaign will book 30% of Larkspur's sales lines to one store ID, and Rohan's answer is more executors. Ingrid rehearses it in PySpark instead: stages and lazy plans, Catalyst, shuffles, broadcast vs sort-merge joins, AQE, skew, salting and spill, measured from the Spark UI; plus ANSI mode, Spark Connect, Declarative Pipelines and Spark 4.2.
Part 13: "Engines at Scale" — Trino, Cloud Warehouses and Real-Time OLAP
Three hundred people open the stores dashboard at 9 a.m. Trino, federation with PostgreSQL, ClickHouse, cloud warehouse billing, real-time OLAP, caches and materialised views, measured under load.
Part 14: "Design Review I" — The Storage and Compute Layer
Season 1 ends with a design review. Rohan defends Larkspur's storage, table format, catalog, partitioning and engines against finance's volumes, retention, freshness, concurrency and cost, in interview order: clarify, estimate from the lab's own files, choose, justify, trade-offs, and what would change the decision. Plus a self-host vs managed matrix, a printable checklist and a worked drill.
Part 15: "Getting Data In" — Batch Ingestion: Files, APIs and Databases
A supplier renames one column and Larkspur's scorecard says it delivered nothing for six weeks. Season 2 opens with ingestion as a contract with systems you don't control: full vs incremental extraction, the three traps of a high-water mark, landing zones and an immutable raw zone, pagination, rate limits and retries, schema drift and load metadata, built with dlt into Iceberg through Polaris.
Part 16: "Changes, Not Snapshots" — CDC into the Lakehouse
Rohan wants to append Debezium's events to the lake and call it a table. Ingrid insists on folding them into state instead: upserts, hard and soft deletes, ordering by LSN, an initial snapshot plus a stream, a history table beside a current-state table, MERGE vs equality deletes, idempotent micro-batches and compaction. Built with Debezium 3.7, Kafka 4.4 and Spark Structured Streaming into Iceberg, it finally removes the £57,064.96 of hard-deleted orders from the 0.7%.
Part 20: "Trust, but Test" — Data Quality and Data Contracts
Rohan wants to add null checks to every dashboard query. Ingrid wants the producers and consumers to write down what the data promises, and a pipeline that tests the promise where data crosses a boundary. Quality dimensions, where to test, quarantine vs fail-fast, ODCS 3.2 data contracts tested with datacontract-cli, a local CI gate for breaking changes, and Write-Audit-Publish on an Iceberg branch in Spark. The session contract settles online conversion at 6.21%.
Part 21: "Every Record Counts" — Completeness and Correctness Engineering
Farah wants the 0.7% explained to the record. Rohan proves it: the board-deck waterfall closes to the penny, every pound the payment provider settled for the UK is traced to gold revenue or to a legitimate reason, and re-sent till baskets are decided once at the door and kept out of gold. Completeness and uniqueness as engineered properties: natural dedup keys, effectively-once delivery, control and hash totals at every hop, a daily control table gating publish in Airflow, late-data policy, restatement and a replay runbook.
Part 23: "Locks on the Lake" — Security, Privacy and Compliance
A customer asks Larkspur to forget them, and Rohan reaches for one DELETE. Ingrid counts the copies first. Personal data found and tagged, masks and row filters per user through Trino and Open Policy Agent, dev locked out of production, the lab's Season 1 shortcuts closed, and one erasure carried through rows, snapshots, tags, manifests, raw files and disks until a byte-level search finds nothing, under GDPR, India's DPDP Act and Singapore's PDPA.