"Format Wars, Ceasefire" — Delta, Hudi, Paimon, DuckLake and Interoperability

Skim Card

One thing to remember: the table formats have converged on features; choose by who writes, who reads and which catalog decides the current version.

• Iceberg, Delta, Hudi, Paimon and DuckLake all version a set of immutable files; they differ in where that list lives and who commits.
• A feature matrix says yes everywhere; the real limits are which engines can write a table and which catalog arbitrates commits.
• One late basket: Iceberg wrote six objects, Delta with deletion vectors four, DuckLake none: two inlined rows plus a snapshot record in PostgreSQL.
• UniForm needs deletion vectors off: 20 one-row updates rewrote 9.35 MB of Parquet, against 31 KB with deletion vectors.
• Interop layers translate metadata for readers: DuckDB read the Delta table as Iceberg through UniForm, which kept no Iceberg history here.
• DuckLake keeps small writes in its catalog until flushed; back up PostgreSQL with the bucket.
• Delta on S3 has one safe writer unless every writer uses the DynamoDB log store; catalog-managed tables aim to fix that (check status).

Use …	When …
Delta, with Iceberg reads through Unity Catalog	Databricks writes the tables; other engines only read
Iceberg with a REST catalog	several engines must write, or you want an exit option
Paimon (Flink-first) or Hudi (Spark-centred)	streaming upserts by key are the main workload
DuckLake	a small team, one DuckDB-centred node, many small writes

Verified against: Delta Lake 4.4.1 on Spark 4.1.3, DuckDB 2.0.0 with DuckLake 1.1, Apache Iceberg 1.12.0, Apache Polaris 1.8.0, PostgreSQL 18.6; Hudi 1.2.1, Paimon 2.1.0, XTable 0.4.0-incubating (docs) · Snowflake and Databricks docs · 28 October 2026

Jump to: Concepts · Hands-On · Landscape · Design Drill

Story Opening

The supplier team’s contractor had delivered the fill-rate tables as Delta Lake, written from a Databricks trial. Store operations ran its own marts in DuckLake on one server. Rohan spent an evening on a feature matrix: ACID commits, time travel, schema evolution, deletion vectors, row-level updates, Iceberg, Delta, Hudi, Paimon, DuckLake.

“Every row says yes,” Rohan said the next morning. “I’ll pick the one with the most ticks and convert the others nightly.”

“Every row says yes because they’ve converged,” Ingrid said. “Your matrix skips the questions that decide it: who writes the table, who reads it, and who decides which version is current. And a nightly copy is a second truth. Let’s put one table in three formats and send one late basket through each.”

The Quick Map

Concepts, with use cases

What every table format agrees on

A table format is a contract over immutable data files: a metadata record of which files make up version N, and an atomic way to publish version N+1 (Part 7 opened Iceberg’s). Iceberg, Delta Lake, Apache Hudi, Apache Paimon and DuckLake all do this, and all offer ACID commits, time travel, schema evolution and row-level changes. They differ in three ways that a feature matrix hides:

• Where the metadata lives: a tree of JSON and Avro files (Iceberg), a log of JSON commits with Parquet checkpoints (Delta), a timeline plus a metadata table (Hudi), LSM-organised files (Paimon), or tables in a SQL database (DuckLake).
• Who arbitrates a commit: a catalog’s compare-and-swap (Iceberg), “create the next log file only if absent” on storage or a catalog (Delta), a database transaction (DuckLake).
• Which engines can read and, separately, write it.

What breaks if you choose by features. The table works for the team that chose it and nobody else: one engine writes it, three others read a stale export. Larkspur: the gold tables are read by DuckDB, Trino and Spark, and written by Spark and PyIceberg, so “who can write it” rules out more than any feature does. Elsewhere: a media company whose data science runs on Databricks and whose finance team runs on Snowflake has a catalog question, not a format question.

Delta Lake: a log in a folder

A Delta Lake table is a folder of Parquet files plus _delta_log/: numbered JSON commits, each listing files added and removed with their statistics, and a Parquet checkpoint that folds the log into one file so readers don’t replay every commit (the lab’s tables wrote one every 10 commits). A commit succeeds if the writer creates the next numbered file first. As of 4.4.1 (October 2026), the core comes with:

• Deletion vectors: a small file marking deleted rows, so an UPDATE writes the changed row and a bitmap instead of rewriting the file. The Delta docs page still says “This feature is in experimental support mode”.
• Catalog-managed tables: “the catalog coordinates commit atomicity” (Unity Catalog is the example), and path-based access stops working. The docs say it is “available in Delta Lake 4.0.1 and above”, the 4.1.0 release notes said “still in preview”, and later notes state no status: check status before relying on it.
• Delta Kernel: Java and Rust libraries that let connectors read and write Delta without reimplementing the protocol.

What breaks without it. Without deletion vectors, every small update or delete rewrites whole files (step 4: 9.35 MB for 20 rows); without the log, there is no atomic commit at all. Larkspur: the contractor’s fill-rate tables. Elsewhere: an adtech firm on Databricks keeps impressions in Delta and lets deletion vectors absorb thousands of small consent deletions a day.

Hudi and Paimon: formats built around upserts

Apache Hudi (1.2.1) organises a table around record keys. Its record index, kept in Hudi’s internal metadata table, maps each key to the file holding it, so an upsert touches only the files it must. Tables are copy-on-write (“optimized for read-heavy workloads”) or merge-on-read (updates go to log files merged at query time and compacted later), and incremental queries “only return new data written to the table since an instant on the timeline”. Larkspur: the orders table replicated from PostgreSQL by CDC (Part 16 builds it on Iceberg) is keyed upserts all day. Elsewhere: a freight company upserts shipment status by tracking number and feeds downstream jobs only the changes.

Apache Paimon (2.1.0) is streaming-first: primary-key tables use “LSM-powered streaming updates with multiple merge engines and changelog producers”; its docs cover Flink first, then Spark and a Python SDK. A write lands in a small sorted run that compaction merges later, like your database’s LSM storage. Larkspur: Season 3’s live stock level per store and SKU, upserted every few seconds. Elsewhere: an IoT platform keeps the latest reading per device and emits a changelog of what changed.

What breaks without them. A format built for batch rewrites handles a stream of single-key upserts with many small files and expensive merges. Both are kept out of the hands-on here (memory); “Choosing” below says when they win.

DuckLake: the catalog is a database

DuckLake 1.1 (spec and extension released 21 October 2026, shipped with DuckDB 2.0) keeps all metadata (snapshots, schemas, file lists, statistics) in tables of a SQL database (PostgreSQL, SQLite or a DuckDB file) and only Parquet in object storage. A commit is one database transaction. Its signature feature is data inlining: an insert or delete of up to 10 rows (the default limit) is stored as rows in the catalog database, not as a file, until ducklake_flush_inlined_data writes it out.

DuckDB is the reference client; other clients (DataFusion, Spark, community Trino connectors, pg_ducklake) exist “at various levels of maturity”. On 26 August 2026 DuckLabs, the company behind DuckDB, announced that it is joining AWS; the projects “will remain the intellectual property of the DuckDB Foundation and open source under the MIT license”.

What breaks without it. File-based metadata makes every tiny write cost several objects and a catalog round trip. Larkspur: store operations’ marts take dozens of single-row corrections a day on one server. Elsewhere: a SaaS startup streams small per-tenant updates into analytics tables without a Spark cluster.

Interoperability: translation, not conversion

Two layers let one copy of the data look like another format:

• Delta UniForm writes Iceberg metadata beside the Delta log after each commit (“asynchronously after a Delta Lake write transaction completes”). Open-source requirements: column mapping, reader 2 / writer 7, a Hive Metastore as catalog, and no deletion vectors (“UniForm does not work on tables with deletion vectors enabled”). It is “read-only from an Iceberg and Hudi perspective”, and only one conversion runs at a time, so one Iceberg version can cover several Delta commits that arrive faster (here, each of 21 versions got its own). In Delta 4.4.1 the module “supports Spark 4.1 and is not compatible with Spark 4.2”. On Databricks the same idea is “Iceberg reads”, read-only, served through Unity Catalog’s Iceberg REST endpoint; there, Iceberg v3 reads support deletion vectors.
• Apache XTable (0.4.0-incubating) translates metadata between Hudi, Delta and Iceberg (Paimon and Parquet as sources only). The project lists its own limits: no Hudi or Iceberg merge-on-read tables, no Delta deletion vectors, and commit timestamps that are not synchronised.

Both translate metadata only; neither lets a second engine write. The vendor-neutral path is to agree on a catalog protocol instead: Iceberg’s REST catalog (Part 9) is what Snowflake, Trino, Spark, DuckDB and Unity Catalog all speak.

What breaks without it. Nightly copies double storage and give two answers between runs. Larkspur: the contractor’s Delta tables become readable as Iceberg by DuckDB without a copy (step 6). Elsewhere: a retailer’s Databricks team writes Delta while its Snowflake team reads the same files through Iceberg metadata.

Choosing

Under the Hood

Three ways to commit

• Iceberg asks the catalog to swap a pointer if it hasn’t moved (Part 7). Commits scale with the catalog; the metadata JSON carries the snapshot list, so it grows: 1,718 bytes at load, 25,312 after 20 more commits (step 4).
• Delta commits by creating N.json only if it doesn’t exist yet. S3 has had conditional writes since 2024 (Part 3), but Delta’s S3 log store doesn’t use them: its docs still say “S3 currently does not provide mutual exclusion”, and the request to use them was closed as not planned. So it is safe for one writer only (see Gotchas). Catalog-managed tables (check status) are designed to move the arbitration into a catalog that implements their commit protocol. Readers start at the last checkpoint and replay the JSON after it.
• DuckLake commits with an SQL transaction: insert the new snapshot, file and statistics rows, and let the database’s own concurrency control order writers. Reading the current table is a query, and paths are stored relative to DATA_PATH (path_is_relative = t in ducklake_data_file).

Where a small write goes

A one-row UPDATE is a delete plus an insert. Iceberg merge-on-read writes a one-row data file, a position-delete file, then a new metadata tree. Delta with deletion vectors writes a one-row file and a bitmap for the old file. Without deletion vectors, Delta copies every other row of the file. DuckLake inlines both halves as catalog rows. Step 3 measures each.

Hands-On

The code is in the every-record-counts folder of the companion-workspaces repository (tag every-record-counts/part-8). Spark runs Delta Lake 4.4.1 (delta-spark_4.1_2.13, delta-iceberg_2.13) in Part 7’s Spark 4.1.3 image, writing to SeaweedFS through Hadoop’s S3A; DuckDB 2.0 on the host writes Iceberg through Polaris and DuckLake with PostgreSQL as its catalog. Trino isn’t needed; the README’s matrix has the memory figures (3.2 GiB peak).

make up-spark                                                  # core + Spark (Delta jars are in the image)
uv run python scripts/duck.py --local sql/part08/stage.sql     # daily store sales as Parquet
uv run python -m scripts.part08_formats reset                  # drop part08 tables, files, DuckLake catalog
uv run python -m scripts.part08_formats load

Step 1: one table, four homes

The table is a daily store-sales gold table that DuckDB recomputes from Part 7’s staged POS lines (uploads up to 30 September, one row per basket line): 73,246 store-days from 1 June to 30 September, plus 46 store-days whose local business date is already 1 October. sum(sales_minor) is 322,704,113,319; it adds paise, pence and Singapore cents, so treat it as a checksum, not money. DuckDB creates the Iceberg and DuckLake copies (CREATE TABLE … AS FROM read_parquet(…)); DuckLake attaches with sql/part08/ducklake_attach.sql:

ATTACH 'ducklake:postgres:host=localhost port=5432 dbname=ducklake user=larkspur password=larkspur' AS dl (
    DATA_PATH 's3://larkspur-lake/ducklake/'
);

Spark creates two Delta tables (sql/part08/spark/01_delta.sql, trimmed):

CREATE TABLE part08.delta_dv (...) USING delta
LOCATION 's3a://larkspur-lake/delta/part08/delta_dv'
TBLPROPERTIES ('delta.enableDeletionVectors' = 'true');

CREATE TABLE part08.delta_uniform (...) USING delta
LOCATION 's3a://larkspur-lake/delta/part08/delta_uniform'
TBLPROPERTIES ('delta.enableIcebergCompatV2' = 'true', 'delta.universalFormat.enabledFormats' = 'iceberg');

Its SHOW TBLPROPERTIES confirms UniForm’s prerequisites were switched on by Delta itself: delta.columnMapping.mode = name, minReaderVersion 2, minWriterVersion 7. The session config (spark/delta.conf.template) adds the Delta extension and catalog, Spark’s embedded Hive metastore (UniForm needs one) and S3A credentials for SeaweedFS.

Step 2: where the metadata lives

uv run python -m scripts.part08_formats layout

         format                   object kind  objects    bytes
---------------  ----------------------------  -------  -------
        Iceberg       Iceberg manifest (Avro)        1    3,457
        Iceberg  Iceberg manifest list (Avro)        1    1,538
        Iceberg       Iceberg metadata (JSON)        1    1,718
        Iceberg                data (Parquet)        1  333,491
    Delta (DVs)         Delta checksum (.crc)        2    5,568
    Delta (DVs)           Delta commit (JSON)        2    2,174
    Delta (DVs)                data (Parquet)        1  466,089
Delta (UniForm)         Delta checksum (.crc)        2    7,597
Delta (UniForm)           Delta commit (JSON)        2    3,762
Delta (UniForm)       Iceberg manifest (Avro)        1    6,948
Delta (UniForm)  Iceberg manifest list (Avro)        1    4,458
Delta (UniForm)       Iceberg metadata (JSON)        1    2,147
Delta (UniForm)                data (Parquet)        1  467,513
       DuckLake                data (Parquet)        1  464,920

DuckLake catalog (PostgreSQL database `ducklake`): 30 tables, 12 with rows

(The 12 tables with rows are trimmed: columns, one data file, its statistics, two snapshots.) Iceberg keeps a tree of files; Delta keeps a log (two commits: create, insert); the UniForm table keeps both; DuckLake’s bucket holds data only, and its metadata is rows in PostgreSQL. The Iceberg data file is smaller because Iceberg’s default codec, write.parquet.compression-codec = zstd, applies; the other writers used Snappy.

Step 3: one late basket

late applies sql/part08/late_basket.sql to each table, one UPDATE per store-day (DuckDB for Iceberg and DuckLake, spark/jobs/part08_late_baskets.py for Delta; Spark’s timing lines are trimmed below):

UPDATE {table}
SET baskets = baskets + 1, sales_minor = sales_minor + 1999
WHERE store_id = '{store_id}' AND business_date = DATE '2026-09-30'

uv run python -m scripts.part08_formats late

One late basket (GB-0001, 30 September), one UPDATE per table; objects written:
         format                   object kind  objects    bytes
---------------  ----------------------------  -------  -------
        Iceberg       Iceberg manifest (Avro)        2    6,814
        Iceberg  Iceberg manifest list (Avro)        1    1,626
        Iceberg       Iceberg metadata (JSON)        1    2,895
        Iceberg                data (Parquet)        1      813
        Iceberg         delete file (Parquet)        1      865
    Delta (DVs)         Delta checksum (.crc)        1    3,781
    Delta (DVs)           Delta commit (JSON)        1    2,205
    Delta (DVs)                data (Parquet)        1    1,538
    Delta (DVs)        deletion vector (.bin)        1       43
Delta (UniForm)         Delta checksum (.crc)        1    4,334
Delta (UniForm)           Delta commit (JSON)        1    2,004
Delta (UniForm)       Iceberg manifest (Avro)        1    6,948
Delta (UniForm)  Iceberg manifest list (Avro)        1    4,456
Delta (UniForm)       Iceberg metadata (JSON)        1    2,147
Delta (UniForm)                data (Parquet)        1  467,512
DuckLake catalog rows added: ducklake_inlined_data_1_1 +1, ducklake_inlined_delete_1 +1, ducklake_snapshot +1, ducklake_snapshot_changes +1

Six objects for Iceberg, four for Delta with deletion vectors (its commit JSON records numCopiedRows 0 and numDeletionVectorsAdded 1), none for DuckLake: two inlined rows plus a snapshot record. The UniForm table rewrote the whole 467 KB file (numCopiedRows 73,245), then wrote Iceberg metadata for the new version.

Step 4: twenty late baskets

The same command carries on with GB-0002 to GB-0020, one commit each:

All 20 late baskets (one UPDATE commit each), per table:
         format  objects  data files  data bytes  other objects
---------------  -------  ----------  ----------  -------------
        Iceberg      120          20      16,260            100
    Delta (DVs)       83          20      30,755             63
Delta (UniForm)      123          20   9,350,266            103
       DuckLake        0           0           0              0
DuckLake catalog rows: 29 -> 109
Iceberg metadata file: 1,718 bytes at load, 25,312 bytes after the 20th commit
Delta checkpoints: 00000000000000000010.checkpoint.parquet, 00000000000000000020.checkpoint.parquet

Twenty one-row changes cost Iceberg 120 objects and the UniForm table 9.35 MB of rewritten data; DuckLake absorbed them as 80 catalog rows. Merge-on-read Iceberg and Delta with deletion vectors both owe compaction now (Part 10).

Step 5: flush the inlined rows

uv run python -m scripts.part08_formats flush

rows flushed: 20; DuckLake objects: 1 -> 3
inlined rows left: ducklake_inlined_data_1_1 0, ducklake_inlined_delete_1 0

CALL ducklake_flush_inlined_data('dl') wrote the 20 rows as one data file and the 20 deletes against the big file as one delete file.

Step 6: every reader, one answer

uv run python -m scripts.part08_formats interop

          reader (DuckDB 2.0)    rows      sales_minor
-----------------------------  ------  ---------------
          Iceberg via Polaris  73,246  322,704,153,299
                     DuckLake  73,246  322,704,153,299
      Delta (DVs), delta_scan  73,246  322,704,153,299
  Delta (UniForm), delta_scan  73,246  322,704,153,299
Delta (UniForm), iceberg_scan  73,246  322,704,153,299

UniForm Iceberg metadata files: 21 (Delta versions 1..21); the newest:
  00000-ba98bc92-8ddb-4cd4-abee-572620d16592.metadata.json
┌─────────────────────┬───────────┬─────────┐
│     snapshot_id     │ operation │ records │
├─────────────────────┼───────────┼─────────┤
│ 5501564805624811233 │ append    │ 73246   │
└─────────────────────┴───────────┴─────────┘

DuckDB UPDATE on the Delta table: BinderException: Binder Error: Can only update base table

(Type row of DuckDB’s table trimmed; sales_minor is the same mixed-currency checksum as in step 1.) All five readers agree: the staged checksum plus 20 × 1,999. DuckDB read the Delta table as Iceberg from UniForm’s metadata file, with no copy. Two limits showed up as well:

• Every UniForm metadata file is named 00000-… and holds one snapshot: in this lab, each conversion started a fresh Iceberg table, so Iceberg readers see the current version but no Iceberg history. The docs require a Hive Metastore and the lab has only Spark’s embedded one; whether a full metastore keeps the history wasn’t tested.
• DuckDB reads Delta (deletion vectors included) but its delta extension doesn’t do UPDATE, DELETE or MERGE: an engine limit, separate from any format’s.

uv run pytest -q tests/test_part08.py re-runs the sequence and checks each claim above: 8 passed in 118 s (with Spark up; make ci ran 67 passed, 8 Trino tests skipped).

The Landscape

Product names and status as of 28 October 2026.

Tips & Gotchas

Gotcha — Delta on S3 has one safe writer by default. The Delta docs: “Concurrent writes to the same Delta table on S3 storage from multiple Spark drivers can lead to data loss.” Multi-cluster writes need S3DynamoDBLogStore (experimental) in every writer. Catalog-managed tables (check status) are designed to let the catalog arbitrate commits instead, but only with a catalog that implements their protocol. The lab’s Delta tables have one writer.

Gotcha — inlined rows live only in the catalog. After step 4, DuckLake’s bucket had no trace of 20 changes. A bucket copy is not a backup, and a tool reading the Parquet under DATA_PATH directly misses them: back up the PostgreSQL database with the files, and flush before handing files to anything that isn’t a DuckLake client.

Gotcha — a table feature raises the bar for every reader. Turning on deletion vectors moved the table to reader version 3; a client without that feature can’t read it at all. List the engines that read a table before enabling deletion vectors, column mapping or Iceberg v3.

Tip — test interop with totals, not row counts. Step 6 compares a row count and a money total from every reader. Run the same per-partition check from each engine before telling a second team that a translated table is theirs to use.

Gotcha — translated history is not native history. Test the time travel your readers need through the translated metadata: here, UniForm’s Iceberg view kept one snapshot per conversion.

Troubleshooting

Design Drill

The question: “A mobile games publisher runs its data platform on Databricks: about 300 TB of player telemetry, matchmaking and live-ops tables in Delta, written by Lakeflow pipelines and Spark jobs. The finance analytics team is moving to Snowflake, the live-ops engineers want Trino for ad-hoc queries, and leadership wants the option to leave Databricks within two years. What do you choose for table format and catalog, and what will you test before committing?”

Clarify first

• Will Snowflake or Trino ever write these tables, or only read them?
• How fresh must external reads be: minutes, hours?
• Which Delta features do the big tables use today (deletion vectors, column mapping, liquid clustering)?
• Must Unity Catalog permissions apply to external readers, or is a separate grant model acceptable?
• Is “leave Databricks” about compute only, or the catalog too?

The shape of a strong answer

• Keep one writer per table and make the catalog the contract: Unity Catalog’s Iceberg REST endpoint serves both Snowflake (catalog integration, catalog-linked database) and Trino.
• Existing Delta tables that only Databricks writes: enable Iceberg reads, read-only for Iceberg clients. With Iceberg v2 reads, deletion vectors must go; Iceberg v3 reads keep them, but only if every reader supports v3 (Trino 484’s v3 is experimental).
• Tables that others must write, and all new shared tables: managed Iceberg tables in Unity Catalog, readable and writable from external Iceberg clients; this is also the exit path.
• No nightly exports: a copy is a second truth with its own lag and cost.
• Test freshness: commit a change and measure when Snowflake and Trino see it (metadata is generated after the Delta commit).
• Test correctness: per-partition row counts and money totals from all three engines, as in step 6.
• Test features: a table with deletion vectors, a renamed column, a dropped column; confirm each engine reads them, and check time travel through the translated metadata.
• Test governance: confirm Unity Catalog grants reach Snowflake and Trino through credential vending, and that no reader holds broader storage keys.
• Test the exit: point a second Iceberg REST catalog (Polaris, Glue) at the same files and read them.
• Monitor metadata generation; the trigger to resync is MSCK REPAIR TABLE … SYNC METADATA.

Trade-offs to say out loud

• Iceberg reads on Delta keep Databricks’ Delta features; managed Iceberg is portable but may lag on Databricks-only features.
• Turning deletion vectors off for Iceberg v2 reads turns small updates into file rewrites (9.35 MB vs 31 KB for 20 rows in step 4).
• Snowflake’s Delta Direct reads files from object storage, which bypasses the catalog’s governance.

Common weak answers

• “XTable solves it”: incubating, translates metadata only, no Delta deletion vectors.
• Pointing Snowflake at Unity Catalog’s Delta tables as Delta: “Snowflake doesn’t support creating Iceberg tables from Delta table definitions in other catalogs”.
• Picking a format without asking who writes.

Further Reading

1. Delta Lake, “Universal Format (UniForm)” (requirements and limits). https://docs.delta.io/latest/delta-uniform.html
2. Delta Lake, “Storage configuration” (S3 single- and multi-cluster writes). https://docs.delta.io/latest/delta-storage.html
3. DuckLake, “Data Inlining” and the 1.1 release post. https://ducklake.select/docs/stable/duckdb/advanced_features/data_inlining
4. Apache Hudi, “Table & Query Types”. https://hudi.apache.org/docs/table_types
5. Apache XTable, project site (scope and limitations). https://xtable.apache.org/
6. Databricks, “Read Delta Lake tables with Iceberg clients”. https://docs.databricks.com/aws/en/delta/uniform

Story Closing

Rohan deleted the matrix. The contractor’s tables would stay Delta with Iceberg metadata for readers, the shared gold tables would stay Iceberg, and store operations could keep DuckLake as long as nobody outside DuckDB needed to write to it. Of the five promises, the part served consistency: five readers, three formats, one total.

“So the format matters less than the thing that says which files are the table,” Rohan said.

“The catalog,” Ingrid said. “Which is why it gets its own part.”

In Part 9, Rohan runs Polaris properly: catalogs, principals and roles, four engines on one table, and a role that can read gold but not raw.

This is Part 8 of “Every Record Counts: Data Engineering for Backend Engineers” — Season 1: Foundations & the Lakehouse.