"The Table That Remembers" — Apache Iceberg in Depth

Skim Card

One thing to remember: an Iceberg table is a catalog pointer to metadata that lists exactly which files are the table; every change is a new version swapped in atomically.

• Commits are compare-and-swap at the catalog: appends retry themselves (20 commits, 28 attempts); a write fails validation if someone changed rows it depends on.
• 3 KB of JSON and 28 KB of Avro described 5.8 million rows; engines plan from that, not from listings.
• Rename, add column and a new partition field rewrote nothing: one snapshot, the same 123 data files.
• Tag what you publish: q3_close keeps Q3 as closed beside Q3 now, readable by name in Spark and Trino.
• One MERGE updating 159 rows: copy-on-write copied 17,887 unchanged rows; merge-on-read wrote 7 times fewer bytes and owes compaction.
• Default to format v2: v3’s VARIANT and deletion vectors worked in Spark, DuckDB and Trino, but PyIceberg 0.12 couldn’t load the table.

Use …	When …
copy-on-write	changes are rare and batched; reads dominate
merge-on-read + compaction	frequent small updates or deletes spread over many files
a tag	a number was published and must stay reproducible
a branch	a change must be tested before main sees it (Part 20)
format v3	every engine that touches the table supports it

Verified against: Apache Iceberg 1.12.0 (spec v2, v3), Spark 4.1.3, Apache Polaris 1.8.0, Trino 484, DuckDB 2.0.0, PyIceberg 0.12.0, Java 25 · vendor docs · 28 October 2026

Jump to: Concepts · Hands-On · Landscape · Design Drill

Story Opening

Farah’s message arrived at 9:05. “Q3 closed on Tuesday. UK till sales for the quarter went up £3,494.82 overnight. Which number is real?”

Rohan knew the cause: late baskets from offline tills, rung up on 30 September, arrived in the 1 October upload. The fix was the backend reflex. “We freeze the table at close. Copy the folder to q3_final, lock writes during the close.”

“A copy drifts, and your lock only stops writers who ask for it,” Ingrid said. “Data science writes the gold tables from Python, finance from Spark. An Iceberg table already remembers every version, and the catalog refuses any commit built on a version that’s no longer current. Let’s open one.”

The Quick Map

Concepts, with use cases

A table format, and the tree under every table

A table format specifies metadata that turns immutable files in object storage into a table with versions and atomic changes. In Apache Iceberg the metadata is a tree, sketched in Part 1 and opened layer by layer in step 2:

A snapshot is one committed version: a manifest list naming the manifests, and through them the files, that make up the table at that moment. New snapshots reuse unchanged manifests, so versions share files. The metadata file lists the snapshots kept and names the current one; the catalog holds one fact per table, where the current metadata file is.

What breaks without it. Part 3’s folder of Parquet is whatever a listing returns: no atomic multi-file change, no versions, slow billed listings. Larkspur: the sales-line table in step 2 is 123 files listed by one manifest. Elsewhere: an adtech impression log gains thousands of files an hour; its planner reads a few manifests instead of listing millions of keys.

Atomic commits and optimistic concurrency

A writer writes its data files, manifests, manifest list and metadata file first, all invisible. Then it asks the catalog to move the table’s pointer from the metadata file it started from to the new one, which the catalog does only if the pointer hasn’t moved. This is optimistic concurrency: your services’ UPDATE … WHERE version = ?, for a whole table. The spec: “All changes to table state create a new metadata file and replace the old metadata with an atomic swap.”

When the swap fails, the writer re-reads the table and re-checks its assumptions. An append assumes nothing and is re-applied on top. A MERGE, DELETE or overwrite depends on the rows it read, so it can fail validation instead of overwriting work it never saw. Spark picks the check with write.merge.isolation-level and its delete and update siblings: serializable (the default) fails if a concurrent commit removed or rewrote files the write read, or added data matching its filter; snapshot checks only the first, so concurrent appends pass. A query reads one committed snapshot throughout.

What breaks without it. Last writer wins, silently: two jobs rewrite the same day and one job’s rows vanish. Larkspur: finance’s Spark job and data science’s PyIceberg job write the same gold table (step 7). Elsewhere: a parcel carrier’s scan table takes Flink appends every minute while a nightly Spark job corrects mis-scanned depots.

Schema evolution by column ID

Every column has a permanent ID; data files store IDs, names live in the metadata. A rename changes a name, an added column gets a new ID (old files lack it and read NULL), a drop removes the ID from the current schema. No file is rewritten.

What breaks without it. Name-based formats mistake a rename for a drop plus an add, or need a rewrite. Larkspur: finance renames amount_minor to net_amount_minor; the tills send till_id from October. Elsewhere: a mobile game adds event attributes every release over years of files.

Hidden partitioning and partition evolution

An Iceberg partition is a transform of a real column: days(event_ts), months(business_date), bucket(16, store_id), identity(country). Manifests store each file’s partition value, so a filter on event_ts prunes to matching days without naming a partition column. The spec can change: new files use the new spec, old files keep theirs, and planning handles both.

What breaks without it. Hive-style tables expose an ingest_date that every query must filter by hand, and a new layout means rewriting the table. Larkspur: sales lines are partitioned by UTC day and, from October, by country too, so each country’s data can be maintained apart (Part 10 judges that choice). Elsewhere: an IoT fleet partitions by month while small, then by day when a million devices arrive, without touching history.

Time travel, tags and branches

Old snapshots stay until expired, so a query can read VERSION AS OF a snapshot or TIMESTAMP AS OF a time. A tag names a snapshot and keeps it through snapshot expiry until the tag is dropped (RETAIN n DAYS caps that); a branch is a named line of snapshots you can write without touching main (Part 20 builds write-audit-publish on one).

What breaks without them. “The number we reported” lives in a spreadsheet, and copies drift. Larkspur: the Q3 close becomes the tag q3_close. Elsewhere: a bank tags the snapshot behind each regulatory filing: Part 5’s “as filed” problem, solved in storage.

Row-level changes: copy-on-write and merge-on-read

Files are immutable, so changing a row needs a strategy. Copy-on-write (CoW) rewrites every file holding a changed row: costly writes, plain reads. Merge-on-read (MoR) writes the new row versions plus a delete file naming the old rows; readers apply it, and compaction (Part 10) folds it back in. In v2 a delete file lists (file, position) pairs; in v3 it’s a deletion vector, a bitmap per data file in a Puffin file. Spark picks per operation with write.delete.mode, write.update.mode and write.merge.mode (copy-on-write by default, or merge-on-read). Equality deletes (“delete rows where order_id = 42”) let streaming writers delete without reading, at a cost to every reader; the v4 draft forbids new ones.

What breaks if you pick wrong. CoW on a table corrected all day rewrites gigabytes to change kilobytes; MoR without compaction makes reads merge thousands of delete files. Larkspur: late baskets restate a few store-days daily (step 6). Elsewhere: a ride-hailing firm adjusts fares on last week’s trips, a few rows in many files: MoR plus nightly compaction.

Format versions: v2 by default, v3 on purpose, v4 on the horizon

The spec calls versions 1, 2 and 3 “complete and adopted”; version 4 “is under active development and has not been formally adopted”. v3 adds the variant, geometry, geography, unknown and nanosecond timestamp types, column defaults, multi-argument transforms, row lineage (a stable _row_id and the sequence number of each row’s last change, so an incremental job can pick out rows changed since its last run instead of diffing tables), deletion vectors in place of position-delete files, and encryption keys. Support on 28 October 2026:

So the series creates v2 tables (Iceberg’s default since 1.4.0), which every engine in the lab reads and writes; a v3 upgrade is one-way. Clickstream payloads stay a JSON string in the default tables; step 8 contrasts a v3 table with variant. v4 is a horizon: Iceberg 1.12 has partial v4 code, nothing usable, but its rules (no new equality deletes, relative paths, a data file and its deletion vector in one manifest entry) show where to steer. Larkspur: shared tables stay on v2 until Trino’s v3 loses its label; row lineage could later let gold jobs rebuild only the store-days whose lines changed. Elsewhere: a fintech’s fraud features recompute only accounts whose transactions changed since the last run.

Under the Hood

What a commit sends, and what the catalog checks

Through a REST catalog the swap is a request, not a file rename. The writer sends its metadata changes plus requirements, such as “branch main is still at snapshot S”; Polaris checks them against its record of the table (in PostgreSQL here) and commits only if nothing moved, else answers HTTP 409 (both of its messages appear in step 7). Each commit takes the next sequence number (reassigned on retry), which orders delete files against data files, so a row re-inserted later survives an earlier delete.

Java’s SnapshotProducer refreshes the table before each attempt and retries up to commit.retry.num-retries times (default 4, backing off from 100 ms). PyIceberg 0.12 commits from the metadata it loaded and refreshes only after a 409 (step 7). Because Java’s refresh would otherwise move the check’s starting point, a writer whose decision depends on what it read pins it with validateFromSnapshot (writer C in step 7). A retry rebuilds the manifest list, not the data files, so it is cheap.

How a reader plans a scan

Manifest-list partition ranges skip whole manifests; each file’s partition value and per-column bounds skip files; only then are Parquet files opened (and Part 3’s row-group skipping applies). Bounds are bytes keyed by column ID, and strings are cut to 16 characters: step 2’s upper bound SG-0100-20260931 is no real basket, just a value that sorts after every real one.

Hands-On

The code is in the every-record-counts folder of the companion-workspaces repository (tag every-record-counts/part-7). It’s the first part with Spark (4.1.3, Iceberg 1.12.0 runtime, local mode). Spark runs first and Trino alone afterwards; the README’s matrix has the memory figures. Times are from a MacBook Pro M3 Pro.

make up-spark                                                # core + Spark
uv run python scripts/duck.py --local sql/part07/stage.sql   # raw POS lines and one day of clicks as Parquet
uv run python -m scripts.part07_iceberg reset                # empty the part07 namespace
uv run python -m scripts.part07_iceberg spark 01_load.sql    # runs sql/part07/spark/01_load.sql in the container

DuckDB stages 6,080,332 POS lines (re-sends included) and 59,881 click events in data/lab/part07, which Spark’s container mounts; spark/jobs/run_sql.py prints one line per statement and a table per query.

Step 1: create the table and load Q3

sql/part07/spark/01_load.sql, trimmed:

CREATE TABLE lakehouse.part07.sales_line (
    basket_id STRING NOT NULL, line_no INT NOT NULL, ...,   -- 13 columns
    event_ts TIMESTAMP NOT NULL, business_date DATE NOT NULL, amount_minor BIGINT NOT NULL, ...
)
USING iceberg
PARTITIONED BY (days(event_ts));

INSERT INTO lakehouse.part07.sales_line
SELECT ...    -- one row per (basket_id, line_no), the first copy uploaded
FROM pos_lines WHERE upload_date <= DATE '2026-09-30';

ALTER TABLE lakehouse.part07.sales_line CREATE TAG q3_close;   -- kept until dropped

> INSERT INTO lakehouse.part07.sales_line  (18.7s)
> ALTER TABLE lakehouse.part07.sales_line CREATE TAG q3_close  (0.1s)
   rows  baskets  utc_days
-------  -------  --------
5848820  2249851       123

Step 2: open the metadata tree by hand

tree asks Polaris for the table over plain HTTP, then reads each layer from SeaweedFS as a file: JSON, Avro (with fastavro), and the Parquet footer (UUIDs shortened):

uv run python -m scripts.part07_iceberg tree

1. catalog: GET http://localhost:8181/api/catalog/v1/larkspur/namespaces/part07/tables/sales_line
   "metadata-location": "s3://larkspur-lake/warehouse/part07/sales_line/metadata/00002-febb741b-….metadata.json"
2. metadata file: 00002-febb741b-….metadata.json (3,034 bytes, JSON)
   format-version 2, last-sequence-number 1
   schema: 1=basket_id, 2=line_no, 3=store_id, 4=country, 5=currency, 6=event_ts, 7=business_date, 8=sku, 9=qty, 10=unit_price_minor, 11=discount_minor, 12=amount_minor, 13=uploaded_at
   partition spec 0: event_ts_day = day(field 6)
   refs: main (branch) -> 6842316002461484095, q3_close (tag) -> 6842316002461484095
   current snapshot 6842316002461484095: append, sequence 1
3. manifest list: snap-6842316002461484095-1-97bc1980-….avro (4,477 bytes, Avro)
   97bc1980-…-m0.avro: content 0, 123 added files, 5,848,820 rows, days 2026-05-31..2026-09-30
4. manifest: 97bc1980-…-m0.avro (24,350 bytes, Avro), 123 entries; the one for 2026-09-30:
   status 1 (added), snapshot_id 6842316002461484095
   file_path .../data/event_ts_day=2026-09-30/00000-142-1c25cf62-…-0-00017.parquet
   record_count 43,492, file_size_in_bytes 667,184
   bounds of field 6 (event_ts): 2026-09-30 00:00:09 .. 2026-09-30 23:59:29
   bounds of field 1 (basket_id): GB-0001-20260930 .. SG-0100-20260931
5. data file: Parquet, 43,492 rows; field IDs in its footer: basket_id=1, line_no=2, store_id=3, country=4, ...

3 KB of JSON and 28 KB of Avro describe 5.8 million rows. The metadata file is already 00002: creating the table, the insert and the tag each wrote one. The partition spec and the Parquet footer both refer to columns by ID.

Step 3: evolve the schema and the partitioning

Steps 3 to 5 run in one Spark session:

uv run python -m scripts.part07_iceberg spark 02_evolve.sql 03_late_lines.sql 04_time_travel.sql

sql/part07/spark/02_evolve.sql:

ALTER TABLE lakehouse.part07.sales_line RENAME COLUMN amount_minor TO net_amount_minor;
ALTER TABLE lakehouse.part07.sales_line ADD COLUMN till_id STRING COMMENT 'sent by the tills from October';
ALTER TABLE lakehouse.part07.sales_line ADD PARTITION FIELD country;

metadata_files  snapshots  data_files  rows_without_till_id
--------------  ---------  ----------  --------------------
             6          1         123               5848820

Three more metadata files, no new snapshot, the same 123 data files; old rows read till_id as NULL.

Step 4: the 1 October upload arrives

sql/part07/spark/03_late_lines.sql merges the upload’s lines on the business key, so re-sent baskets already loaded match and are skipped:

MERGE INTO lakehouse.part07.sales_line t
USING (... lines from upload_date 2026-10-01, one per (basket_id, line_no) ...) s
ON t.basket_id = s.basket_id AND t.line_no = s.line_no
WHEN NOT MATCHED THEN INSERT *;

spec_id  data_files     rows   first_day    last_day
-------  ----------  -------  ----------  ----------
      0         123  5848820  2026-05-31  2026-09-30
      1           6    46384  2026-09-30  2026-10-01

country  late_lines_30_sep
-------  -----------------
     GB                183
     IN                125
     SG                 21

New rows use spec 1, one file per (day, country); old files keep spec 0. The second query reads the three new 30 September (UTC) files: 329 lines from offline tills, the late baskets.

Step 5: Q3 as closed, Q3 now, and a branch

sql/part07/spark/04_time_travel.sql, trimmed:

SELECT 'tag q3_close' AS version, count(*) AS lines, sum(amount_minor) AS uk_sales_minor
FROM lakehouse.part07.sales_line VERSION AS OF 'q3_close'
WHERE country = 'GB' AND business_date BETWEEN DATE '2026-07-01' AND DATE '2026-09-30'
UNION ALL
SELECT 'main', count(*), sum(net_amount_minor) FROM lakehouse.part07.sales_line WHERE ...;   -- same filter

ALTER TABLE lakehouse.part07.sales_line CREATE BRANCH rohan_dev RETAIN 7 DAYS;
DELETE FROM lakehouse.part07.sales_line.branch_rohan_dev WHERE country = 'SG';

     version    lines  uk_sales_minor
------------  -------  --------------
tag q3_close  1508591      2782694870
        main  1508774      2783044352

> SELECT sum(net_amount_minor) FROM lakehouse.part07.sales_line VERSION AS OF 'q3_close'
ERROR: AnalysisException: [UNRESOLVED_COLUMN.WITH_SUGGESTION] A column, variable, or function parameter with name `net_amount_minor` cannot be resolved. ...
             ref     rows
----------------  -------
            main  5895204
branch rohan_dev  4877130
    tag q3_close  5848820

Farah’s two numbers: £27,826,948.70 as closed, £27,830,443.52 now, 183 late lines apart, both queryable by name. The tagged snapshot keeps its own schema, so the renamed column answers only to its old name there. The branch lost Singapore; main didn’t.

Step 6: one MERGE, copy-on-write vs merge-on-read

uv run python -m scripts.part07_iceberg spark 05_merge_modes.sql builds two copies of a daily store-sales table from q3_close (by months(business_date)), one with 'write.merge.mode' = 'copy-on-write', one 'merge-on-read', and runs Part 6’s recomputing MERGE of the store-days the 1 October upload touched into each. The snapshot summaries, with Spark’s MERGE counts:

         mode  updated  inserted  copied  removed_files  added_files  delete_files  bytes_written
-------------  -------  --------  ------  -------------  -----------  ------------  -------------
copy-on-write      159       612   17887              2            2          NULL          84862
merge-on-read      159       612       0           NULL            2             2          11980

From each MERGE snapshot’s manifest entries (status 1 = added, 2 = removed; content 0 = data, 1 = position deletes; month 680 = September 2026, counted from 1970):

         mode  status  content  month  record_count   bytes
-------------  ------  -------  -----  ------------  ------
copy-on-write       1        0    680         18000   79592
copy-on-write       1        0    681           658    5270
copy-on-write       2        0    680         18000  112301
copy-on-write       2        0    681            46    2010
merge-on-read       1        0    680           113    2447
merge-on-read       1        0    681           658    5796
merge-on-read       1        1    680           113    1935
merge-on-read       1        1    681            46    1802

To change 113 September store-days, copy-on-write rewrote all 18,000 September rows. Merge-on-read wrote the 113 new versions and two delete files (one per month) pointing at the old rows: 11,980 bytes against 84,862, 7 times fewer, with compaction still owed. Both tables then hold the same 73,858 store-days.

Step 7: two writers, one table

Across engines first. conflict loads the merge-on-read table in PyIceberg, lets Spark commit an UPDATE to GB-0042’s 30 September row (sql/part07/spark/conflict_update.sql), then has PyIceberg, still on its old snapshot, delete that store-day:

uv run python -m scripts.part07_iceberg conflict

PyIceberg loaded daily_store_sales_mor at snapshot 7907628158333909710
Spark commits an UPDATE to GB-0042 on 30 September in the meantime:
> UPDATE lakehouse.part07.daily_store_sales_mor  (3.0s)
main is now 5601077621839900830
PyIceberg, still on its old snapshot, deletes the same store-day:
Commit failed due to a concurrent update, retrying (1/4) in 100 ms
  ValidationException: Added data files were found matching the filter for snapshots set()!

Polaris refused the swap; PyIceberg re-read the table, found Spark’s file matching its filter, and gave up rather than delete a row it never saw (the empty set() is PyIceberg’s wording).

Then the lowest level, Java 25 and the Iceberg API (iceberg-api-demo/, Gradle Kotlin DSL, gradle/libs.versions.toml): on a small store-closure table, A commits a file, C reads the table, two threads append ten rows each at once, then C tries to replace October:

DataFile refit = write(a, "GB-0042", LocalDate.of(2026, 9, 30), "refit");   // a Parquet file nobody can see yet
a.newAppend().appendFile(refit).commit();                                   // the pointer swap

c.newOverwrite()
        .overwriteByRowFilter(october)
        .addFile(write(c, "SG-0007", LocalDate.of(2026, 10, 1), "power cut"))
        .validateFromSnapshot(base)          // what C read before deciding
        .conflictDetectionFilter(october)    // the rows C's decision depends on
        .validateNoConflictingData()         // fail if anyone added such rows since
        .commit();

./gradlew -q :iceberg-api-demo:run

A wrote business_date_month=2026-09/00001-63114069964083-14feae7b-…-00001.parquet (956 bytes, 1 row)
A committed snapshot 1465766362871967203: metadata/00000-7bd5e039-….metadata.json -> metadata/00001-2c97eca3-….metadata.json
C read snapshot 1465766362871967203
[pool-13-thread-1] WARN org.apache.iceberg.util.Tasks - Retrying task after failure: sleepTimeMs=109 Commit failed: Failed to commit Table or View part07.store_closure because it was concurrently modified
[pool-13-thread-2] WARN org.apache.iceberg.util.Tasks - Retrying task after failure: sleepTimeMs=103 Commit failed: Requirement failed: branch main has changed: expected id 8521881052476490811 != 4626865414171864428
...
2 writers x 10 appends at once: 20 commits in 28 attempts
C failed: ValidationException: Found conflicting files that can contain records matching (ref(name="business_date") >= "2026-10-01" and ref(name="business_date") < "2026-11-01") ...
history: 21 snapshots

(Six more retry warnings and the stack traces trimmed; the count varies because the race is real.) Eight swaps refused, every append committed on retry, and the overwrite that would have erased October’s new rows refused.

Step 8: a v3 table with VARIANT and a deletion vector

uv run python -m scripts.part07_iceberg spark 06_v3.sql loads one day of clicks into a default v2 table (payload STRING) and into a v3 table (payload VARIANT), trimmed:

CREATE TABLE lakehouse.part07.clicks_v3 (event_id STRING, ..., payload VARIANT) USING iceberg
PARTITIONED BY (days(event_ts))
TBLPROPERTIES ('format-version' = '3', 'write.delete.mode' = 'merge-on-read');

INSERT INTO lakehouse.part07.clicks_v3
SELECT event_id, event_type, event_ts, anonymous_id, platform, country, parse_json(payload) FROM clicks_staged;

Both give the same top products (get_json_object on v2, variant_get(payload, '$.sku', 'string') on v3; SKU-00898 leads with 26 views). Then both delete the 5,176 bot events:

format  content  file_format  deleted_rows  file_size_in_bytes  blob_bytes
------  -------  -----------  ------------  ------------------  ----------
    v2        1      PARQUET          5176                7034        NULL
    v3        1       PUFFIN          5176                 607         163

_row_id  _last_updated_sequence_number          event_id
-------  -----------------------------  ----------------
      0                              1  7aa9b39d95acad0c
      1                              1  3ce7ae3f1eba5cae
      2                              1  2689f05dcbf10c67

> ALTER TABLE lakehouse.part07.clicks_v3 ADD COLUMN consent_version INT DEFAULT 1
ERROR: UnsupportedOperationException: Cannot add column consent_version since setting default values in Spark is currently unsupported

The same deletes: a 7,034-byte position-delete file in v2, a 607-byte Puffin file holding a 163-byte bitmap in v3. Then the other engines, on the host and in Trino (after docker compose stop spark and make up-trino):

uv run python -m scripts.part07_iceberg engines
uv run python -m scripts.part06_trino query sql/part07/trino/reads.sql

• DuckDB 2.0: 54,705 events in each table (both kinds of delete applied); payload.sku reads the variant.
• Trino 484: typeof(payload) is variant, 54,705 events; it lists the q3_close tag in "sales_line$refs" and returns its 1,508,591 lines FOR VERSION AS OF 'q3_close': the tag lives in the table, not in Spark.
• PyIceberg 0.12: reads clicks_v2; clicks_v3 fails with ValidationError: Value error, Unsupported field type: 'variant'.

Step 9: the tests

uv run pytest -q tests/test_part07.py resets the namespace, runs every Spark step and checks each claim above against independent counts, plus the conflict and the Java run: 9 passed in 64 s. Trino’s test passes in a run with Trino up.

The Landscape

Iceberg support as of 28 October 2026, from each vendor’s docs.

Tips & Gotchas

Gotcha — time travel uses the old schema. Spark and Trino both read a snapshot with the schema it was written with, so a query against a tag fails on a column renamed since. Test “as filed” reports against the tag, not against main.

Gotcha — partition evolution changes what a dynamic overwrite replaces. Part 2’s overwritePartitions() replaces the partitions new rows land in: after ADD PARTITION FIELD country, a (day, country) slice, not a day. Use a keyed MERGE (step 4).

Tip — read the snapshot summary before tuning. spark.merge-into.num-target-rows-copied over …-updated is copy-on-write’s cost as one ratio; in the hundreds on a busy table, try merge-on-read plus compaction.

Gotcha — one v3 column locks a tool out of the whole table. PyIceberg 0.12 failed while loading clicks_v3, before reading any column. Upgrading a shared table to v3 for one variant column cuts off every client that can’t parse the new type.

Gotcha — retries handle races, not job logic. Iceberg retries appends; a MERGE that fails validation is your job’s to retry, so it must be safe to run again from scratch (Part 6’s recomputing MERGE).

Troubleshooting

Design Drill

The question: “A video streaming service keeps one playback_events Iceberg table, about 40 billion rows, partitioned by day. The ingestion team’s Flink job commits new events every minute. The rights team runs a nightly Spark MERGE that corrects territory codes over the last 30 days. The privacy team deletes viewers’ events from Trino. Last month two nightly runs failed on commit conflicts and one report double-counted. What can go wrong when several engines write one table, and how do you prevent it?”

Clarify first

• Do all three engines commit through the same catalog, or does anything write by path or through a second catalog?
• Which format version, and does every engine support it for writes?
• Which partitions does each writer touch? Must the MERGE see events appended while it runs?
• What did the double-count look like: duplicate events, or a correction applied twice after a retry?

The shape of a strong answer

• One catalog is the only commit point; block path-based or second-catalog writes, because the pointer swap protects only writers who use it.
• Sort writers by what they assume: Flink appends assume nothing and retry; the MERGE and deletes depend on what they read, so keep serializable validation and let them fail rather than overwrite.
• Scope conflict detection to the MERGE’s own partitions, and schedule erasure deletes outside its window.
• Find the likely cause of the failures: Flink also appends late events dated inside the 30-day window, and a long MERGE races about 60 commits an hour, so serializable sees matching data added and fails. Remedies: snapshot isolation for the MERGE when the only concurrent writer appends (late events then wait for the next night), or exclude the last N days the stream still writes.
• Make failure cheap: a MERGE that recomputes from raw and keys on event ID can simply run again; the double-count was likely a non-idempotent job retry, not a table bug.
• Merge-on-read for the MERGE and deletes (few rows across many files), with compaction, itself a writer, on cold partitions.
• Stay on v2 unless every writer supports v3 writes; one engine with experimental v3 is a risk on the largest table.
• Alert on attempts per commit and validation failures per writer.

Trade-offs to say out loud

• serializable fails more jobs; snapshot isolation fails fewer and checks less.
• Minute commits buy freshness and cost snapshots and small files that maintenance must absorb.
• Merge-on-read makes the nightly job fast and every read slower until compaction.

Common weak answers

• “Add a distributed lock”: it protects only writers that take it; the catalog’s compare-and-swap already covers every committing writer.
• “Run the jobs at different times”: correctness shouldn’t depend on a schedule.
• Forgetting that compaction and snapshot expiry are writers too, and that erasure isn’t done at DELETE (Part 23).

Further Reading

1. Apache Iceberg, “Table Spec” (format versions, snapshots, deletes, row lineage). https://iceberg.apache.org/spec/
2. Apache Iceberg, “Reliability” (optimistic concurrency and retries). https://iceberg.apache.org/docs/latest/reliability/
3. Apache Iceberg, “Spark DDL” (branches, tags, partition evolution). https://iceberg.apache.org/docs/latest/spark-ddl/
4. Trino, “Iceberg connector” (format v3 status, metadata tables). https://trino.io/docs/current/connector/iceberg.html

Story Closing

Farah got both numbers, each with a name: the tag q3_close, kept until someone drops it, and main, 183 late lines apart. Which one goes in the board pack is a policy question, Ingrid said, and Part 21 would give finance a restatement rule. Of the five promises, this part served consistency: every engine reads one committed version, and two writers can’t quietly undo each other.

“So Iceberg is the answer,” Rohan said.

“One answer. Delta Lake, Hudi, Paimon and DuckLake give others.”

In Part 8, Rohan compares the table formats, tries the interoperability layers, and finds that the catalog decides more than the format does.

This is Part 7 of “Every Record Counts: Data Engineering for Backend Engineers” — Season 1: Foundations & the Lakehouse.