"Three Houses for Data" — Warehouse, Lake, Lakehouse

Skim Card

One thing to remember: analytical systems store data for scanning history, not for touching single rows, and warehouse, lake and lakehouse differ in who owns the files.

• A quarterly scan read 816 MiB in PostgreSQL and 9.8 MiB of Parquet in DuckDB; a one-basket lookup reversed it (40 KiB vs 1.1 MiB).
• Columnar files win scans: a query reads only its columns, compressed, and can skip blocks using min/max statistics.
• Warehouse: one system owns storage, compute and format. Lake: open files, no transactional tables. Lakehouse: open files plus a table format and a catalog.
• A folder of Parquet can’t take an UPDATE; an Iceberg UPDATE commits a new snapshot with two tiny files and keeps the old version readable.
• Don’t run analytics on an OLTP read replica: long scans get cancelled by replication or bloat the primary.
• Warehouses now read open table formats and lakehouses add warehouse features; the decision that lasts is the catalog and who can read the data.

Use …	When …
One PostgreSQL (or DuckDB)	tens of GB, a handful of analysts, one team
A managed warehouse	one main engine, SQL users, little platform staff
A lake (files only)	raw landing and archives, ML on files, no updates
A lakehouse	several engines or teams on one copy, updates and history needed

Verified against: PostgreSQL 18.6, DuckDB 2.0.0, Trino 484, Apache Polaris 1.8.0, Iceberg format v2 tables, SeaweedFS 4.51 · 28 October 2026

Jump to: Concepts · Hands-On · Landscape · Design Drill

Story Opening

Rohan’s plan fitted on one slide: a dedicated read replica for reporting, more RAM, and indexes on every column the dashboards filter by. “The queries are slow because Postgres is missing indexes. The numbers are wrong because the queries are wrong. Fix both, done.”

Ingrid didn’t argue with the slide, and opened a terminal instead. “Your replica is a copy of a database built for one job: finding a row, changing it, and committing in a millisecond while ten thousand other requests do the same. Farah’s question is the opposite job. Let’s measure it before we buy hardware.”

The experiment took an afternoon. Six million POS sales lines from the last four months, put in three places: a PostgreSQL table with a primary key, a folder of Parquet files in object storage, and an Iceberg table in a catalog. The same two questions asked of each, then one wrong line corrected in each.

By five o’clock Rohan had stopped talking about indexes.

The Quick Map

Concepts, with use cases

OLTP and OLAP: two jobs, two shapes

OLTP (online transaction processing) is what you build every day: many small, concurrent reads and writes of current state, each touching a handful of rows by key, with strict transactional guarantees. OLAP (online analytical processing) is the other job: few, large queries that scan and aggregate months or years of history, mostly reading, often touching every row of a few columns.

What breaks if you ignore the difference. You run the quarterly report on the database that takes the orders. Either the report crawls through every whole row on disk, or it competes with checkout for I/O and memory, or both. Then someone adds indexes for the report, and every write in checkout now maintains them.

Larkspur. Rohan’s payment service asks “what is payment 8813’s status?” thousands of times a second; Farah asks “what was UK net revenue in Q3, by week, after refunds?” once a day, over 30 million rows. Same data, opposite access patterns.

Where else you’ll see this. An airline’s booking system holds seats and fares one booking at a time, while revenue management analyses years of bookings to set tomorrow’s prices.

Row storage and column storage

A row store keeps all of a row’s columns next to each other. PostgreSQL writes rows into 8 KB pages; to read one column of a million rows it must read every page those rows live on, so it reads every other column too. That’s ideal when you want the whole row (load an order, update its status) and wasteful when you want one column of all rows.

A column store keeps each column’s values together. Values of one column look alike (the same few countries, dates in order, prices in a narrow range), so they compress well with dictionary and run-length encoding. A query that sums amount_minor by country reads two compressed columns and nothing else. Files also store statistics per block, such as each column’s minimum and maximum, so a reader can skip blocks that can’t match a filter. Part 3 opens up the Parquet format byte by byte.

What breaks without it. Scan cost grows with the width of the table, not the width of the question: add a 2 KB JSON column to a row store and every report slows down, including the ones that never read it.

Larkspur. The sales_line table has 13 columns; the quarterly revenue question needs four. Hands-On step 2 shows what that is worth.

Where else you’ll see this. An IoT manufacturer keeps billions of sensor readings with 40 attributes each, and the reliability team charts one attribute across every machine; telemetry platforms are columnar for that reason.

Forty years, five problems

The three houses make more sense as answers to problems, roughly in the order those problems became common.

1. “Every department has its own numbers” (1990s). The enterprise data warehouse: one integrated, historical copy of the business, apart from operational systems. Bill Inmon’s Building the Data Warehouse (1992) defined it as subject-oriented, integrated, time-variant and non-volatile; Ralph Kimball’s The Data Warehouse Toolkit (1996) gave it dimensional modelling (Part 4). Larkspur’s incident is this problem, back again.
2. “The warehouse is too slow” (1980s–2000s). MPP (massively parallel processing) databases, later sold as appliances, spread tables across nodes, each scanning its share, with storage and compute bought together.
3. “We can’t afford to keep everything, and half of it isn’t tables” (2006–2015). The data lake, first on Hadoop’s file system and later on object storage, kept raw files of any shape cheaply for any engine. James Dixon of Pentaho coined the term in 2010. Without transactions, schemas or owners, many lakes became data swamps: files nobody trusted.
4. “We pay for compute we’re not using” (2012–2020). Cloud warehouses separated storage from compute. Snowflake’s 2016 SIGMOD paper describes virtual warehouses that start, stop and resize independently of data in object storage; BigQuery grew from Google’s Dremel (2010), which scanned shared columnar storage with thousands of workers. The format was still the vendor’s.
5. “Our data is locked in one engine, and the lake still can’t do updates” (2020–). The lakehouse kept open files in your object storage and added database features with an open table format (Apache Iceberg, Delta Lake, Apache Hudi) and a catalog. The CIDR 2021 paper by Armbrust, Ghodsi, Xin and Zaharia defined it as “a data management system based on low-cost and directly-accessible storage that also provides traditional analytical DBMS management and performance features such as ACID transactions, data versioning, auditing, indexing, caching, and query optimization”.

The sixth step is happening now. Snowflake creates and reads Iceberg tables and exposes them through Horizon Catalog’s Iceberg REST API; BigQuery has Apache Iceberg managed tables; Amazon Redshift reads and writes Iceberg, including format v3 since August 2026; and lakehouse vendors ship warehouse features such as serverless SQL warehouses. The architectures are converging; what stays hard to undo is the catalog and format your data lives in.

The three houses

Warehouse. Larkspur: Farah’s finance close would run happily on one: SQL, a few dozen analysts, exactness over flexibility. Elsewhere: a regional bank’s regulatory reporting team, whose numbers must be reproducible and whose users live in SQL.

Lake. Larkspur: the larkspur-raw bucket from Part 0 is a lake: the landing zone where tills, suppliers and the PSP drop files, kept unchanged so every pipeline can be replayed. Elsewhere: a genomics lab keeps petabytes of sequencing files that only specialised tools read.

Lakehouse. Larkspur: where Ingrid is taking the platform. Finance needs exact SQL, data science wants Python on the same history, Season 3 adds Flink writing in real time, and nobody wants to copy data to switch engines. Elsewhere: a video streaming service where recommendations (Spark), the content team’s dashboards (SQL) and real-time playback analytics (Flink) share one history.

Who else lives on the street

• Single-node analytical engines such as DuckDB run columnar SQL inside your process and read Parquet and Iceberg directly. Part 11 argues that many teams never need a cluster, and pg_duckdb (1.1.1, December 2025) embeds DuckDB inside PostgreSQL itself.
• Real-time OLAP stores (ClickHouse 26.8 LTS, Apache Pinot 1.5.1, Apache Druid 38, StarRocks 4.1) serve sub-second aggregates to dashboards and to thousands of end users, over data ingested seconds ago. They usually sit beside the lakehouse, not instead of it (Parts 13 and 30). Larkspur: store managers’ live stock tiles in Season 3. Elsewhere: an adtech platform showing every advertiser campaign stats updated every few seconds.
• HTAP databases run transactions and analytics in one system: TiDB with its TiFlash columnar replicas, SingleStore’s combined rowstore and columnstore, AlloyDB for PostgreSQL’s columnar engine, MySQL HeatWave. They shine when analytics must see one application’s very latest writes, and fit less well as a company-wide platform fed by dozens of sources. Larkspur: a “today’s takings” panel inside the payment service’s own admin tool. Elsewhere: a brokerage’s intraday position and exposure view over the trading database.

These houses aren’t exclusive. Larkspur ends up with most of them: the raw lake as the landing zone, an Iceberg lakehouse as the system of record for analytics, a real-time OLAP store in Season 3, and PostgreSQL where the services live.

Under the Hood

Why PostgreSQL read 816 MiB to answer a 4-column question

PostgreSQL stores a table as a heap of 8 KB pages. Each row (tuple) carries a 23-byte header for MVCC bookkeeping, then all its columns. The sales_line heap is 104,456 pages, 816 MiB for 6,055,922 rows (about 141 bytes per row), plus a 287 MiB primary-key index on (basket_id, line_no). A query that needs country, currency, business_date and amount_minor reads every page, because those four values are scattered across every row on every page. There’s no index on business_date, and even if there were, the filter keeps about three quarters of the table, and fetching most of a table through an index is slower than reading it in order. EXPLAIN (ANALYZE, BUFFERS) counted 104,488 buffer accesses: every heap page, plus a few dozen other pages the scan touched.

The same layout makes the basket lookup cheap: descend the primary-key B-tree from root to leaf (three or four pages for six million keys), follow the pointer to the heap page holding the five lines, done. Five pages, 40 KiB.

Why DuckDB fetched 9.8 MiB of a 66 MB folder

The Parquet copy is a set of row groups (here 50 of up to 122,880 rows) and inside each row group one column chunk per column, compressed with Snappy. The file footer lists every chunk’s byte offset, size and min/max values. DuckDB read the footers, then issued ranged GETs for only the four column chunks it needed in each row group: 9.8 MiB.

It didn’t get to skip any row groups for the date filter. The data is sorted by basket_id, whose IDs start with the store, so every row group contains every date, and every row group’s min/max range covers the quarter. The basket lookup is the opposite case: basket_id is the sort key, so only row groups whose min/max range includes GB-0042-20260815-00009 can match, and DuckDB fetched 1.1 MiB. Choosing the sort order to match your filters is a design decision, and Part 10 makes it on purpose.

Why the Iceberg copy behaved differently

The Iceberg table holds the same rows in one 43 MB data file from DuckDB’s Iceberg writer: 11 row groups of up to 772,082 rows, compressed with ZSTD (the writer’s layout varies a little between runs). ZSTD made the scan cheaper, 7.9 MiB fetched. The bigger row groups made skipping coarser, so the basket lookup fetched 7.6 MiB, and Trino decoded 2.2 million rows to return five. Same format, same data, different writer settings, different trade-offs; Part 3 measures them.

What a folder of files can’t do, and what a table format adds

A folder of Parquet files has no answer to “which files make up the table right now?” except “whatever the listing returns”. Two writers can’t add files atomically, a reader can see half of a rewrite, and correcting one row means rewriting a whole file and hoping nobody reads in between.

An Iceberg table is files plus metadata about files. The catalog holds a pointer to the table’s current metadata file; that file lists snapshots; each snapshot points to a manifest list, which points to manifests, which list data files and delete files. A change writes new files first, then asks the catalog to swap the pointer from the old metadata file to the new one, which succeeds only if nobody else swapped it first. Readers follow whichever pointer they found and see a complete version, old or new, never half of each.

(Manifest lists and manifests sit between each snapshot and its files; Part 7 opens every one of them.)

The UPDATE in Hands-On step 4 produced exactly this: the 43 MB file untouched, a 2,077-byte data file with the corrected row, an 815-byte position-delete file marking the old row as deleted, and a new snapshot. Readers merge the delete file at read time (merge-on-read); a maintenance job later folds small files into big ones (compaction, Part 10). Snapshot S1 is still complete, which is why time travel can read the old value.

PostgreSQL’s MVCC is the same idea at a smaller scale: an UPDATE writes a new row version and leaves the old one for VACUUM, so in-flight transactions still see the old version. PostgreSQL does it per row inside pages it owns; Iceberg does it per file in storage anyone can read.

Hands-On

The companion code is in the every-record-counts folder of the companion-workspaces repository (tag every-record-counts/part-1). This part needs the core profile and Trino; Spark isn’t needed yet. Peak container memory was 2.4 GB.

make up-trino      # core (SeaweedFS, Polaris, PostgreSQL) + Trino
make data upload   # once, if you skipped Part 0

Step 1: one table, three homes

The table is one row per POS basket line, built by DuckDB from the raw events in the bucket (sql/part01/sales_line.sql). It drops the re-sent events Part 0 found and computes each store’s local business date. The lines that matter:

WITH baskets AS (
    SELECT DISTINCT ON (basket_id) * FROM raw_pos_events ORDER BY basket_id, uploaded_at
)
SELECT b.basket_id,
       ...
       (b.event_ts AT TIME ZONE s.timezone)::DATE   AS business_date,
       ...
       l.qty * l.unit_price_minor - l.discount_minor AS amount_minor
FROM baskets b,
     unnest(b.lines) AS t(l)
JOIN raw_stores s ON s.store_id = b.store_id

scripts/part01_load.py writes it to the three homes: into PostgreSQL through DuckDB’s postgres extension (then a primary key and ANALYZE), as Parquet files with COPY … TO 's3://larkspur-lake/parquet/sales_line', and as an Iceberg table in the catalog that sql/part01/polaris_attach.sql attaches. Then it asks each home what it holds:

uv run python scripts/part01_load.py

build sales_line from raw POS events            6.1s
rows                                         6,055,922
1. PostgreSQL table                             9.6s
   + primary key                                4.2s
   + ANALYZE                                    0.2s
2. Parquet files in s3://larkspur-lake/parquet    0.3s
3. Iceberg table lake.part01.sales_line         0.4s

PostgreSQL heap / primary-key index          816 MB / 287 MB
Parquet files row groups                     50, up to 122,880 rows each, SNAPPY
Iceberg data file row groups                 11, up to 772,082 rows each, ZSTD

DuckDB wrote the Iceberg table; Trino will read it. That’s the lakehouse promise in one line: the table belongs to the catalog, not to the engine that wrote it.

Step 2: the same question in every home

Q3 sales by country and month, sql/part01/revenue_by_month.sql. Only {sales_line} changes per engine:

SELECT country,
       currency,
       date_trunc('month', business_date) AS month,
       count(*)                           AS lines,
       sum(amount_minor)                  AS sales_minor
FROM {sales_line}
WHERE business_date BETWEEN DATE '2026-07-01' AND DATE '2026-09-30'
GROUP BY country, currency, date_trunc('month', business_date)
ORDER BY country, month

scripts/part01_compare.py runs it in PostgreSQL (counting pages with EXPLAIN (ANALYZE, BUFFERS)), in DuckDB over the Parquet folder and the Iceberg table (counting bytes fetched with EXPLAIN ANALYZE), and in Trino over the Iceberg table (reading physicalInputBytes from its REST API). It fails if any answer differs and reports the median of three runs after a warm-up. The output, with six of the nine result rows left out:

uv run python scripts/part01_compare.py

== revenue_by_month: all four engines return the same 9 rows
   GB  GBP  2026-07  505,325  942,835,173
   GB  GBP  2026-08  513,446  946,281,830
   GB  GBP  2026-09  490,003  893,927,349

engine                                      median  read
PostgreSQL 18 (row store, B-tree PK)       448.6ms  104,488 buffers of 8 KiB (816.3 MiB; 32,043 already in shared buffers)
DuckDB 2.0 over Parquet files               16.1ms  9.8 MiB fetched from 66 MB of Parquet
DuckDB 2.0 over Iceberg (Polaris)           32.1ms  7.9 MiB fetched from 43 MB of Parquet
Trino 484 over Iceberg (Polaris)           169.8ms  12.4 MiB physical input, 6,055,922 rows decoded

Three things to take from this:

1. Bytes read is the real cost. PostgreSQL read 83 times as many bytes as DuckDB to produce the same nine rows. At six million rows that’s half a second. At six billion it’s the difference between a query and an outage, and in a warehouse priced per byte scanned, it’s the bill.
2. Times on a laptop flatter everyone. Much of PostgreSQL’s 816 MiB came from memory (32,043 pages were already in its buffer cache), and DuckDB and Trino read from a SeaweedFS container a few milliseconds away. On real object storage, per-request latency makes the number of requests matter too (Part 3).
3. Trino is slower here, and that’s fine. It’s a distributed engine paying for planning, scheduling and a coordinator round trip on a 43 MB table. Its advantage appears when the table is 43 TB and fifty analysts query it at once (Part 13).

The output came from a MacBook Pro M3 Pro; your times will differ, the bytes won’t.

Step 3: the question your services ask

One basket’s lines, sql/part01/basket_lookup.sql:

SELECT line_no, sku, qty, amount_minor
FROM {sales_line}
WHERE basket_id = 'GB-0042-20260815-00009'
ORDER BY line_no

The same run continues:

== basket_lookup: all four engines return the same 5 rows
   1  SKU-01059  1  3,099
   2  SKU-00430  2  238
   3  SKU-01081  2  2,198
   4  SKU-00552  1  319
   5  SKU-00487  1  79

engine                                      median  read
PostgreSQL 18 (row store, B-tree PK)         0.9ms  5 buffers of 8 KiB (40 KiB; 5 already in shared buffers)
DuckDB 2.0 over Parquet files                4.9ms  1.1 MiB fetched from 66 MB of Parquet
DuckDB 2.0 over Iceberg (Polaris)           27.2ms  7.6 MiB fetched from 43 MB of Parquet
Trino 484 over Iceberg (Polaris)           106.7ms  8.3 MiB physical input, 2,211,840 rows decoded

The order flips: PostgreSQL is 5 to 120 times faster and reads a few kilobytes. That’s why your services shouldn’t read from the lakehouse, and your reports shouldn’t read from your services.

Step 4: correct one line in each home

A cashier missed a £1 discount on the first line of that basket. scripts/part01_update.sh applies the same correction in each home and shows what each change cost:

./scripts/part01_update.sh

PostgreSQL (sql/part01/update_postgres.sql) shows the row version’s physical address (ctid) and the transaction that wrote it (xmin) before and after:

== 1. PostgreSQL
   ctid   | xmin | amount_minor | discount_minor
----------+------+--------------+----------------
 (7605,7) |  878 |         3099 |              0
(1 row)

UPDATE 1
    ctid    | xmin | amount_minor | discount_minor
------------+------+--------------+----------------
 (104455,1) |  883 |         2999 |            100
(1 row)

The corrected row is a new version, written by a new transaction on a page near the end of the heap that had free space; the old version on page 7,605 stays until VACUUM removes it. Fast and invisible to you, which is what an OLTP database is for.

The Parquet folder (sql/part01/update_parquet.sql):

== 2. A folder of Parquet files
-- sql/part01/update_parquet.sql
ParserException: Parser Error: syntax error at or near "("

LINE 2: UPDATE read_parquet('s3://larkspur-lake/parquet/sales_line/*.parquet')
                           ^

There’s nothing to update: read_parquet lists files, it isn’t a table. Correcting the line means rewriting the whole file that contains it.

Iceberg through Trino (sql/part01/update_iceberg.sql) runs the UPDATE, then reads the table’s own metadata tables:

UPDATE lakehouse.part01.sales_line
SET amount_minor = amount_minor - 100, discount_minor = discount_minor + 100
WHERE basket_id = 'GB-0042-20260815-00009' AND line_no = 1;

SELECT snapshot_id, operation,
       element_at(summary, 'added-data-files')       AS added_data_files,
       element_at(summary, 'added-position-deletes') AS added_position_deletes,
       element_at(summary, 'total-records')          AS total_records
FROM lakehouse.part01."sales_line$snapshots"
ORDER BY committed_at;

-- content 0 = data file, 1 = position-delete file
SELECT content, record_count, file_size_in_bytes
FROM lakehouse.part01."sales_line$files";

== 3. Iceberg, through Trino
UPDATE: 1 row
     snapshot_id     | operation | added_data_files | added_position_deletes | total_records
---------------------+-----------+------------------+------------------------+---------------
 4316301866084748316 | append    | 1                | NULL                   | 6055922
 2708287873192366323 | overwrite | 1                | 1                      | 6055923
(2 rows)
 content | record_count | file_size_in_bytes
---------+--------------+--------------------
       0 |      6055922 |           43198360
       0 |            1 |               2077
       1 |            1 |                815
(3 rows)
-- time travel: the line as it was in snapshot 4316301866084748316
 amount_minor | discount_minor
--------------+----------------
         3099 |              0
(1 row)

total_records counts rows in data files, so it’s one higher than the table’s row count until compaction. The last query reads the table FOR VERSION AS OF the first snapshot, from before the correction: an auditor’s favourite feature, and one Farah will lean on in Part 21.

make test checks all of this: the four engines agree on both queries, the scan reads the whole heap while the lookup reads a few pages, the Parquet update is refused, and the Iceberg update adds exactly one data file and one delete file while time travel still returns the old amount.

The Landscape

Current product names as of 28 October 2026. A cell that couldn’t be verified says so.

Self-host or buy

This matrix is judgement, not measurement; your team’s skills and your company’s cloud contracts move every cell.

Tips & Gotchas

Gotcha — a read replica is not an analytics database. A long query on a PostgreSQL hot standby conflicts with replication: when the primary vacuums rows the query still needs, the standby either delays replay or cancels the query with “canceling statement due to conflict with recovery”. Turning on hot_standby_feedback stops the cancellations by making the primary keep those rows, which bloats the primary instead. Either way, reporting fights checkout.

Gotcha — SELECT * is the expensive habit. In step 2, four of thirteen columns cost 9.8 MiB. Every extra column in a columnar engine is another set of chunks fetched and decoded, and in BigQuery’s on-demand pricing, more bytes processed on the bill. Name your columns, especially in views other people build on.

Gotcha — dropping a table isn’t deleting its files. The first version of the load script ran DROP TABLE and re-created the Iceberg table on every run. The catalog forgot the old tables, but their data files stayed in the bucket: after repeated runs the table’s folder held 11 data files for one live table. The script now deletes the folder itself; Part 10 covers purge, snapshot expiry and orphan-file cleanup properly.

Gotcha — a catalog can override your engine’s storage endpoint. Found while setting up Spark for Part 7: Polaris returns the catalog’s configured S3 endpoint to every engine that loads a table, and engines use it instead of their own setting. With endpoint set to http://localhost:8333 for laptop clients, Spark inside Docker tried to reach SeaweedFS at localhost and failed. The lab sets only endpointInternal, for Polaris’s own writes, and each engine keeps its own endpoint.

Gotcha — “reads Iceberg” doesn’t mean “reads every Iceberg table”. Amazon Athena works only with format v2 tables, and Trino 484 creates v2 by default and calls v3 experimental. The lab creates v2 tables so every engine can read and write them; Part 7 shows v3.

Tip — the simple option is often right. If your analytics fit on one machine and one team queries them, a PostgreSQL replica with a few summary tables, or DuckDB over Parquet files, beats any platform. Move when the queries hurt the primary, when several teams or engines need the same data, or when you need history and corrections you can audit.

Design Drill

The question: “We’re a 40-person B2B SaaS company. We want dashboards over three years of product usage events and our billing data, for the product team, sales and the founders. Today everything is in our production PostgreSQL. Warehouse, lake or lakehouse? Walk me through it.”

Clarify first

• How many events per day, how large, and how far back do people really query? Is three years a requirement or a hope?
• Who queries, how many people, and in what tool: SQL, spreadsheets, or notebooks?
• How fresh must the dashboards be: hourly, daily, or live inside the product?
• Is ML or data science on the roadmap that needs raw events in files?
• Which cloud are they on, and can anyone on the team operate data infrastructure?
• Where does billing data live (their database or a billing provider’s API), and must revenue reconcile with it?

The shape of a strong answer

• Estimate first: say 20 million events a day at ~500 bytes is ~10 GB/day of raw JSON, roughly 1–1.5 GB/day as compressed Parquet, about 1.5 TB for three years. Billing is tiny by comparison. A modest analytical workload.
• Get analytics off production PostgreSQL now: the dashboards will hurt the product before anything else.
• With no platform engineers, recommend a managed warehouse or serverless SQL engine (BigQuery, Snowflake, Redshift Serverless, or Fabric on Azure): the least operations per useful query. At this size, DuckDB over Parquet is a credible, cheaper option if the users are technical.
• Keep raw events as Parquet in object storage anyway: cheap, replayable, and it keeps the door open to ML and to changing engines.
• Lay events out for the queries: partition by event date (dashboards filter by time; retention drops whole days) and cluster or sort by workspace_id (sales and support look at one customer at a time).
• Load billing incrementally (or by CDC, Season 2) and define revenue once, so finance and the dashboards reconcile.
• Defer a full lakehouse until a second engine or team needs the same data, and say what would trigger it: ML on raw events, a second warehouse, or cost.
• Name the cost model: per-byte pricing rewards partitioning and selecting columns; capacity pricing rewards steady load.

Trade-offs to say out loud

• Warehouse simplicity now vs portability later, mitigated by keeping raw data in open files you own.
• A serverless warehouse’s per-query cost vs a provisioned engine’s idle cost.
• Freshness: hourly batch is cheap; minute-level freshness doubles the moving parts.

Common weak answers

• Jumping to Kafka, Spark and a lakehouse for 1.5 TB and ten users.
• Never estimating volumes, so the choice can’t be defended.
• Ignoring that billing numbers must match the billing system; the dashboard that disagrees with finance gets abandoned.

Further Reading

1. Armbrust, Ghodsi, Xin, Zaharia, “Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics”, CIDR 2021. https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf
2. Dageville et al., “The Snowflake Elastic Data Warehouse”, SIGMOD 2016. https://doi.org/10.1145/2882903.2903741
3. Melnik et al., “Dremel: Interactive Analysis of Web-Scale Datasets”, VLDB 2010. https://research.google/pubs/pub36632/
4. Stonebraker et al., “C-Store: A Column-oriented DBMS”, VLDB 2005: the paper that made the case for column stores.
5. Kleppmann and Riccomini, Designing Data-Intensive Applications, 2nd edition (O’Reilly, 2026): the chapters on storage and retrieval.
6. Apache Iceberg table specification. https://iceberg.apache.org/spec/

Story Closing

Rohan’s slide got a new title: “Three houses, one decision”. The read replica was gone. In its place were three columns of numbers measured in the lab that afternoon, 816 MiB against 9.8 MiB, and a note: “The replica would have been fast enough for a month and wrong for a year.”

Ingrid added a line at the bottom: “The lakehouse is where we’re going. But a lakehouse full of bad pipelines is a faster way to get the same wrong numbers.” Then, tapping the incident list from Part 0: “Duplicates, deletes, refunds. None of those are storage problems. Tomorrow: how a data engineer thinks about a pipeline, and why yours have to survive running twice.”

In Part 2, Rohan builds a tiny pipeline that doubles Larkspur’s revenue when it runs twice, makes it safe to re-run, and meets the five promises behind this series’ name.

This is Part 1 of “Every Record Counts: Data Engineering for Backend Engineers” — Season 1: Foundations & the Lakehouse.