"Engines at Scale" — Trino, Cloud Warehouses and Real-Time OLAP

Skim Card

One thing to remember: pick an engine by workload, concurrency and the shape of the bill; one big scan, three hundred small lookups and a cross-system join want different engines over one copy of the truth.

• Trino answered four years of lines in 2.1 s, then killed a 28-million-key GROUP BY at its 739 MiB per-node limit.
• Federation pushes filters, dynamic filters and some aggregates into PostgreSQL; text ranges, text grouping and regexes stay in Trino and cross the wire.
• Warehouses differ in the bill: Snowflake per running second, BigQuery per TiB scanned or slot-hour, Redshift Serverless per RPU-second, Fabric per capacity unit.
• In one shared 4-vCPU VM at 32 concurrent tile queries, Trino over Iceberg managed 35–39 a second; ClickHouse’s serving table, 1,399–1,671.
• Real-time OLAP stores win by keeping data sorted and local: one store’s four weeks came from one granule of 8,980.
• Caches and materialised views are freshness contracts: stamp every tile with its “as of” time and rebuild serving copies from the lake.

Use …	When …
Trino (or a warehouse) over the lake	Ad hoc analyst SQL, big scans and joins, queries across sources
A cloud warehouse	You’d rather rent than run; choose the billing model that matches your load’s shape
A real-time OLAP store	Many concurrent sub-second queries, user-facing tiles, data seconds old
A serving table or result cache	Many people read the same aggregates, and stated staleness is acceptable

Verified against: Trino 484, ClickHouse 26.8.36.3 LTS, PostgreSQL 18.6, Apache Polaris 1.8.0, Iceberg 1.12.0 · Trino docs and source · vendor docs · 28 October 2026

Jump to: Concepts · Hands-On · Landscape · Design Drill

Story Opening

The new stores dashboard went live on a Monday. At 9:00 every store manager opened it, then the regional leads, then finance. By 9:05 the four-week sales tile took about a second, and the chat channel filled with screenshots of spinners.

“Trino handled four years of lines in two seconds last week,” Rohan said. “We give it more workers and put Redis in front.”

“Last week was one big question,” Ingrid said. “This is three hundred small ones at once. Different workload, different engine, maybe a different bill. Let’s measure before we buy anything.”

The Quick Map

Concepts, with use cases

MPP query engines: coordinator, workers, splits

A massively parallel processing (MPP) query engine such as Trino runs SQL over data it doesn’t store. The coordinator is “responsible for parsing statements, planning queries, and managing Trino worker nodes”; workers execute tasks. The plan is cut into stages (fragments) connected by exchanges that move rows between nodes; each stage runs as tasks on workers, and tasks read splits, “sections of a larger data set” (for Iceberg, files or parts of files). Rows stream through operators in memory from stage to stage; by default nothing is written to disk between them, unlike Spark’s shuffle files.

What breaks without it. One process caps a query at one machine (Part 11); a batch engine isn’t built for an analyst waiting. Larkspur: analysts’ ad hoc SQL over the governed lake, through the same Polaris catalog the pipelines write to. Part 1 promised Trino’s advantage at “43 TB and fifty analysts”: concurrent big scans scale with workers; a flood of tiny queries doesn’t (step 6). Elsewhere: a telecom’s network team querying years of call records in object storage without loading them anywhere.

Connectors, pushdown and federation

A connector “adapts Trino to a data source”; a catalog is one configured instance of it. Pushdown hands work to the source (filters, columns, limits, some aggregations) so less data crosses the wire. Federation is one query over several catalogs: Trino pushes what it can into each source and joins the results itself.

What breaks without it. Every cross-system question becomes a copy job first. Larkspur: monthly sales targets live in the store-operations service’s PostgreSQL; sales live in the lake. “Which UK stores are furthest behind target?” is one query (step 2). Elsewhere: a SaaS company joining its CRM’s PostgreSQL accounts with product events in the lake for a churn review.

Cloud warehouses: storage, compute, and the shape of the bill

The big cloud warehouses all keep columnar storage separate from compute; they differ in what you rent and how it’s billed:

Snowflake’s newer Gen2 standard warehouses: its docs disagree on whether they’re now the default, and their rates aren’t in the docs (not verified, 2026-10-28). Databricks SQL warehouses bill in DBUs.

What breaks without this view. Teams choose on a benchmark and are surprised by the bill: per-byte pricing punishes unpruned scans, per-second pricing punishes clusters left running, capacity pricing punishes the job that eats the shared capacity. Since bytes and slot-seconds are what you buy, partition pruning and clustering (Part 10) cut the bill, not only the wait. Larkspur: finance’s month-end close (a few dozen people, predictable queries) fits an auto-suspending warehouse; the 9 a.m. rush doesn’t. Elsewhere: a media company whose analysts work office hours saves most of its bill with auto-suspend.

Real-time OLAP stores

Real-time OLAP stores (ClickHouse, StarRocks, Apache Pinot, Apache Druid) store data themselves, sorted and indexed for filtered aggregations, for many concurrent sub-second queries over data that arrived seconds ago. Pinot calls itself “a real-time distributed OLAP datastore purpose-built for low-latency, high-throughput analytics”; Druid keeps columnar segments “partitioned by time”; StarRocks and ClickHouse also read Iceberg.

What breaks without them. A lake engine pays catalog calls, planning and object-storage round trips on every query: fine for one analyst, not for twelve tiles times three hundred viewers. Larkspur: the store managers’ dashboard (step 6), and Season 3’s live stock tiles (Part 30). Elsewhere: a food-delivery app showing every restaurant its orders of the last hour.

Caches and materialised views

A result cache returns a stored answer for an identical query: Snowflake’s for 24 hours (up to 31 days with reuse), BigQuery’s “approximately 24 hours”, ClickHouse’s query cache query_cache_ttl (60 s by default). A data cache keeps file pieces near compute (Snowflake’s warehouse cache; Trino’s file system cache). A materialised view stores a query’s result and keeps it current, incrementally (BigQuery, Redshift; ClickHouse’s classic MVs process each inserted block) or by refreshing on a schedule (ClickHouse’s refreshable MVs re-run the query and swap the result in atomically).

What breaks without them. Every viewer recomputes the same aggregate from raw lines. Larkspur: a daily store-sales table refreshed hourly serves every tile (step 5). Elsewhere: a news site’s “most read” box, recomputed every minute, not per page view.

Under the Hood

How Trino runs the history query, and why the basket query died

EXPLAIN (TYPE DISTRIBUTED) of Part 11’s history query has two fragments. In Fragment 1, a SOURCE stage, each task reads splits of pos_monthly and computes a partial aggregate per (country, currency, month), sent on hash-partitioned by those keys; Fragment 0 receives the partials through an exchange and computes the final aggregate. That’s how 72.67 million rows became 153 with 2.5 MiB of peak memory.

The basket query has 27,952,255 groups (Part 11), which partial aggregation barely shrinks. Every group lives in a heap hash table, and Trino kills a query when its memory on any worker passes query.max-memory-per-node, 30% of the heap by default: 739.2 MiB of the lab’s 2,464 MiB. Trino’s docs call spilling to disk “legacy functionality” and point to fault-tolerant execution (retry-policy=TASK, with an exchange manager spooling to S3-compatible storage). The usual answer is simpler: more workers. The final aggregation is hash-partitioned, so ten workers each hold a tenth of the baskets.

Why the same query reads different bytes on each engine

For Part 11’s history query, DuckDB fetched 107.5 MiB from object storage (its HTTP stats), Trino 1,175.5 MiB (physical input) and ClickHouse 1,021.6 MiB (S3 read bytes): each engine counts differently, but two read nearly the whole 1.0 GiB table. Part 10 scaled files down to 0.6–6.9 MB, and Trino reads a Parquet file whole below parquet.small-file-threshold (3 MB; 47 of the 197 files): with the threshold at 0 it read 726 MiB (measured). The rest is, by inference, Part 10’s 2 MiB row groups: the 5 needed columns of 13 sit within parquet.max-merge-distance (1 MB) of the others, so merged reads fetch the gaps too. ClickHouse’s read size wasn’t investigated. Where the bill counts bytes scanned (Athena), measure on your real files before you budget.

How ClickHouse answers one store in milliseconds

A MergeTree table stores each insert as a part, column by column, sorted by the ORDER BY key; background merges combine parts, as compaction does for Iceberg. Every 8,192 rows (a granule) the sparse primary index records the key, so a filter on store_id and business_date reads only matching granules. Parquet has row-group statistics too, but a lake reader first asks the catalog, fetches metadata and footers over HTTP and plans splits; MergeTree’s index sits on local disk, cached in memory.

Hands-On

The code is in the every-record-counts folder of the companion-workspaces repository (tag every-record-counts/part-13). Everything runs in the Docker VM (4 vCPUs, 9.8 GB, shared with unrelated containers): core services, Trino 484 (one node as coordinator and worker, 3 GB) and ClickHouse 26.8 LTS in a new olap profile (2 GB, server capped at 1.5 GiB, 4 threads and 1 GiB per query). Spark stays stopped. The data is Part 10’s part10.pos_monthly, 72,670,965 lines over four years.

make up-trino && make up-olap                          # core + Trino + ClickHouse (Spark must be stopped)
uv run python -m scripts.part13_engines reset          # drop Part 13's ClickHouse databases and PostgreSQL schema
uv run python -m scripts.part13_engines seed           # store-ops sales targets in PostgreSQL
uv run python -m scripts.part13_engines trino          # step 1
uv run python -m scripts.part13_engines federate       # step 2
uv run python -m scripts.part13_engines clickhouse     # step 3
uv run python -m scripts.part13_engines load           # step 4
uv run python -m scripts.part13_engines serve          # step 5
uv run python -m scripts.part13_engines dashboard      # step 6

Every engine runs Part 11’s SQL files unchanged and hashes answers as Part 11 did. Timings: a first run and the median of three more; read_MiB is what the engine fetched from storage, peak_MiB the query’s peak memory as the engine reports it.

Step 1: Trino over the lake

engine                    query     first_s  warm_s  rows  digest     read_MiB  peak_MiB
trino (Iceberg)           history      2.53    2.09   153  b711851d    1,175.5       2.5  cpu 2.4s
trino (Iceberg)           baskets    ERROR: Query exceeded per-node memory limit of 739.20MB [Allocated: 738.66MB, Delta: 1MB, Top Consumers: {HashAggregationOperator=723.00MB, PagePartitioner=7.25MB, LazyOutputBuffer=2.51MB}]
JVM heap 2,464.0 MiB; query memory pool 1,724.8 MiB; per query per node: 30% of the heap, 739.2 MiB

(Trimmed: the node list and the plan described in Under the Hood.) The digest is Part 11’s: Trino agrees with DuckDB, Polars and Spark. In the same VM, Part 11’s Spark (local mode, 1.5 GB heap) took 11.98 s and Trino 2.09 s: not a benchmark, but the gap an interactive engine exists for. The basket query failed fast instead of slowing down.

Step 2: federation, and what PostgreSQL was asked

seed creates store_ops.sales_targets in Larkspur’s PostgreSQL, one target per store and month for July to December 2026; with no store-ops service in the lab, a script fills it with synthetic targets (104% of the same month a year earlier). Trino reads it through a new catalog:

connector.name=postgresql
connection-url=jdbc:postgresql://postgres:5432/larkspur
connection-user=trino_reader
connection-password=${ENV:TRINO_PG_PASSWORD}

connector.name picks the connector, the URL names the database, and trino_reader may only SELECT from store_ops (the test checks an UPDATE is refused); the password comes from the environment. Trino turns pushed-down work into SQL for PostgreSQL. Two plans from sql/part13/trino/federation.sql, cut to their scans:

-- 1. GROUP BY month, WHERE currency = 'GBP'
TableScan[table = oltp:Query[SELECT "month", count(*) AS "_pfgnrtd_0", sum("target_minor") AS "_pfgnrtd_1"
  FROM "larkspur"."store_ops"."sales_targets" WHERE "currency" = ? GROUP BY "month"] ...]
-- 3. WHERE month = DATE '2026-09-01' AND regexp_like(set_by, 'gb$')
ScanFilterProject[table = oltp:store_ops.sales_targets ... constraint on [month] ..., filterPredicate = regexp_like(set_by, JoniRegExp 'gb$')]

The first ran entirely inside PostgreSQL. The third pushed the month but filtered rows in Trino, and plan 2 (trimmed) grouped by currency in Trino: per Trino 484’s docs and source, text range predicates and text grouping stay in Trino because “the remote data source may sort strings differently than Trino”. Dates, numbers, text equality, Top-N, count/sum and friends, and dynamic filters (keys collected from the join’s other side at run time) are pushed. Whatever stays in Trino crosses the network first: nothing for 3,600 targets, an incident for a 400-million-row orders table. Then the federated question, September’s UK stores furthest behind target (targets joined to a per-store sum of pos_monthly for September and country = 'GB'), under EXPLAIN ANALYZE:

├─ ScanFilter[table = oltp:store_ops.sales_targets ... constraint on [month, currency] ..., dynamicFilters = {store_id = #df_539}]
│      Input: 200 rows (4.10kB), Filtered: 0.00%, ...
│      Dynamic filters:
│          - df_539, [ SortedRangeSet[type=varchar, ranges=200, {[GB-0001], ..., [GB-0200]}] ], collection time=140.75ms
└─ ScanFilterProject[table = lakehouse:part10.pos_monthly$data@6957362534315781484 constraint on [business_date], filterPredicate = (country = varchar 'GB')]
       Input: 493587 rows (15.09MB), Filtered: 0.73%, Physical input: 5.87MB, ..., Splits: 50, ...

store_id  target_minor  sales_minor        pct_of_target
 GB-0055     2,600,000    1,992,373  76.6000000000000000
 GB-0020     2,600,000    2,045,043  78.7000000000000000
 GB-0121     4,900,000    3,968,506  81.0000000000000000
(0.17s, 493,987 rows read)

(Trimmed: other operators, two result rows.) PostgreSQL got the month and currency filters plus the 200 store IDs the lake side produced, and returned 200 rows. The lake side pruned to September’s partition, and the country filter skipped row groups by their statistics: 493,587 of September’s 1.43 million lines, 5.87 MB fetched. The join, the division and the Top-5 ran in Trino. The file’s last statement tries a Trino materialised view (the Iceberg connector documents them as a view plus an Iceberg storage table, refreshed incrementally or fully) and gets createMaterializedView is not supported for Iceberg REST catalog.

Step 3: ClickHouse reading the lake through Polaris

ClickHouse’s DataLakeCatalog database engine attaches an Iceberg REST catalog; it’s tier Beta in 26.8 (system.settings says so) and needs allow_database_iceberg = 1 in the session:

CREATE DATABASE IF NOT EXISTS lake
ENGINE = DataLakeCatalog('http://polaris:8181/api/catalog')
SETTINGS
    catalog_type = 'rest',
    warehouse = 'larkspur',
    catalog_credential = 'root:s3cr3t',
    auth_scope = 'PRINCIPAL_ROLE:ALL',
    oauth_server_uri = 'http://polaris:8181/api/catalog/v1/oauth/tokens',
    storage_endpoint = 'http://seaweedfs:8333/larkspur-lake',
    aws_access_key_id = 'lake-reader',
    aws_secret_access_key = 'lake-reader-lab-secret';

catalog_type picks the REST protocol; warehouse is Polaris’s catalog; credential, scope and token URI are the OAuth2 login Trino and DuckDB use; storage_endpoint names SeaweedFS, since Polaris returns no endpoint; the keys are DuckDB’s read-only lake key. The root principal and a static key are lab shortcuts: in production, a read-only principal with vended credentials (Part 9). Tables appear as one name: lake.`part10.pos_monthly`.

engine                    query     first_s  warm_s  rows  digest     read_MiB  peak_MiB
clickhouse (Iceberg)      history      5.80    5.06   153  b711851d    1,021.6      40.5
clickhouse (Iceberg)      baskets     17.18   16.96    35  241362c2    1,021.6     611.2  spilled 623.0 MiB

Both digests match Part 11. ClickHouse was slower than Trino on the lake scan; where Trino killed the basket query, ClickHouse spilled 623 MiB to disk at its 512 MiB threshold and finished.

Step 4: the same lines in ClickHouse’s own storage

sql/part13/clickhouse/02_load.sql copies the table into MergeTree, PARTITION BY toYYYYMM(business_date), ORDER BY (store_id, business_date), with LowCardinality(String) for the 600 store IDs and the SKUs:

   ┌─parts─┬─────rows─┬─compressed─┬─uncompressed─┐
1. │    51 │ 72670965 │ 1.71 GiB   │ 5.64 GiB     │
   └───────┴──────────┴────────────┴──────────────┘
engine                    query     first_s  warm_s  rows  digest     read_MiB  peak_MiB
clickhouse (MergeTree)    history      2.51    2.48   153  b711851d        0.0       7.6
clickhouse (MergeTree)    baskets     13.68   13.62    35  241362c2        0.0     681.9  spilled 499.8 MiB
tile for GB-0042: 38 ms, read 8,192 rows in 1 of 8,980 marks

The copy took 24.3 s and is bigger than the lake’s: 1.71 GiB with ClickHouse’s default LZ4 against 1.0 GiB of zstd Parquet. Full scans got faster (history 5.06 → 2.48 s) but still read every row. The tile is where sorted local storage pays: four weeks of one store came from one granule of 8,980. The price is a second copy to refresh, secure and reconcile.

Step 5: a serving table and the query cache

Most tiles don’t need lines at all. 03_serve.sql keeps a daily store-sales table in ClickHouse, filled from the lake by a refreshable materialised view:

CREATE MATERIALIZED VIEW serve.daily_store_sales_mv
REFRESH EVERY 1 HOUR TO serve.daily_store_sales AS
SELECT store_id, business_date, any(country) AS country, any(currency) AS currency,
       count() AS lines, sum(amount_minor) AS sales_minor, sum(discount_minor) AS discount_minor
FROM lake.`part10.pos_monthly`
GROUP BY store_id, business_date;

last_success_duration_ms: 6237
next_refresh_time:        2026-10-28 09:00:00
read_rows:                72670965 -- 72.67 million
written_rows:             907199
country tile, run 1:   20.8 ms, 3 rows, server read 907,199 rows, query cache hits 0
country tile, run 2:    2.2 ms, 3 rows, server read 3 rows, query cache hits 1
country tile, run 3:    1.4 ms, 3 rows, server read 3 rows, query cache hits 1

Each refresh re-reads the lake (72.67 million lines in 6.2 s) and swaps in 907,199 store-days; the lake stays the source of truth. The regional “yesterday by country” tile, the same for every viewer, ran with use_query_cache = 1 (and a 300 s TTL in the lab): the first run scanned the table, the next two came from the cache. The serving copy also serves consistency: the test checks its tiles against the lines’.

Step 6: the 9 a.m. test

Monday morning, rehearsed: the store tile (one store, four weeks, sql/part13/tile_lines.sql) for random stores, 256 queries per level from 1, 8 and 32 concurrent clients on the host; the last variant reads the serving table.

256 tile queries per level, random stores (seed 13); latency per query in ms
target                       clients      p50      p95      max     qps
Trino over Iceberg                 1       71      167      316    12.1
Trino over Iceberg                 8      251      390      554    30.1
Trino over Iceberg                32      883     1364     1597    34.9
ClickHouse over Iceberg            1       38       63       85    25.0
ClickHouse over Iceberg            8       55       86      121   135.5
ClickHouse over Iceberg           32      187      309      462   158.4
ClickHouse MergeTree lines         1        3        4       10   350.3
ClickHouse MergeTree lines         8        6       15       21  1149.8
ClickHouse MergeTree lines        32       30       76      126   894.9
ClickHouse serving table           1        3        4       11   355.7
ClickHouse serving table           8        5        8       14  1535.8
ClickHouse serving table          32       18       34       50  1611.1

Alone, every engine is fast enough (Trino’s median: 71 ms). Under load, Trino over the lake levelled off at 35 queries a second, an 883 ms median at 32 clients: each query pays for parsing, planning, a catalog lookup and object-storage reads, on four busy cores. Over three runs at 32 clients, Trino managed 34.9–39.3 queries a second, ClickHouse over the same Iceberg files 4.1–4.5 times that, its MergeTree copy 26–43 times (895–1,571; the noisiest row), and the serving table 38–46 times (1,399–1,671, p95 32–42 ms). The absolute numbers belong to this shared 4-vCPU VM; the ratios are the lesson. Trino is the right engine for analysts’ questions and the wrong default for three hundred people refreshing the same tiles: a sized Trino cluster with a result cache might meet 40 queries a second, but the serving store wins on p95 and cost per query as concurrency grows.

uv run pytest -q tests/test_part13.py rebuilds the PostgreSQL table and the ClickHouse databases and checks the claims above: digests, pushdown plans, the read-only role, the refused materialised view, the Beta setting, the tile’s granules, the serving table against the lines, the cache hit (11 passed in 94 s). Containers peaked at 4.8 GiB (breakdown in the README).

The Landscape

As of 28 October 2026, from each vendor’s docs.

Tips & Gotchas

Gotcha — a federated query runs on your OLTP database too. Whatever Trino can’t push, it fetches from production PostgreSQL during business hours. Point the catalog at a replica with a read-only role, look for filterPredicate on JDBC scans in EXPLAIN, and ingest anything queried daily into the lake.

Gotcha — a materialised view is a freshness contract. A refreshable MV is as old as its last refresh (the lab’s REFRESH EVERY 1 HOUR scheduled its next run for 09:00, on the hour), and a result cache adds its TTL. ClickHouse’s classic MVs see only inserted blocks, so source updates and deletes never reach them. Print the “as of” time on every tile.

Gotcha — ClickHouse’s defaults assume a dedicated server (5 GiB mark caches, a 1 GiB query cache): in a small container, cap max_server_memory_usage, shrink the caches and set max_bytes_before_external_group_by so big aggregations spill (clickhouse/config.d/lab.xml, users.d/lab.xml).

Gotcha — ClickHouse’s Iceberg support is Beta. In 26.8, reading through a catalog and inserting are tier Beta, compaction and orphan removal Experimental (system.settings.tier). Let Spark or Trino write and maintain the tables; let ClickHouse read them or serve copies.

Tip — buy for the shape of the load. Spiky load suits elastic compute (multi-cluster warehouses with auto-suspend, autoscaling slots, Redshift Serverless or concurrency scaling); steady all-day load suits reserved capacity.

Troubleshooting

Design Drill

The question: “A national mobile operator keeps network-quality and customer data in an Iceberg lake: about 400 TB, growing 1 TB a day. 300 analysts and regional managers use 2,000 dashboards; most are opened between 8:45 and 9:30 a.m., and the rest of the day sees a tenth of that load. Today everything runs on one Trino cluster and the morning is slow. Design the serving engine choice.”

Clarify first

• How many dashboards open on a typical morning, with how many tiles, and do they auto-refresh?
• What freshness do tiles need: yesterday, last hour, last minute?
• Do tiles read aggregates (per cell site, region, day) or raw events, filtered how?
• Latency target per tile; what else runs at 9 a.m. (nightly jobs, ad hoc exploration)?
• Row-level security per region; cloud contracts and budget model?

The shape of a strong answer

• Estimate first, assuming most opens land in the first 15 minutes of the window: 300 people × 2 dashboards × 12 tiles ≈ 7,200 queries, 8 a second on average, perhaps 40 at peak with refreshes. Small queries, high concurrency: a serving problem, not a bigger-scan problem.
• Split the workloads: a serving tier for known dashboards, an exploration tier for ad hoc SQL, so neither starves the other.
• Serving tier: pre-aggregated gold tables (per cell, region, hour or day) in a real-time OLAP store sorted by the dashboards’ filters, or a warehouse with result caching; refreshed when the pipeline publishes, with an “as of” time on every tile.
• The lake stays the source of truth: the serving copy is rebuilt from gold, and each refresh reconciles counts and sums with it.
• Warm the caches before 8:45 by running the top dashboards’ queries after the morning load.
• Exploration tier: Trino or a warehouse over Iceberg, autoscaled, with queues or resource groups capping ad hoc concurrency.
• Elastic capacity for the peak, reserved capacity for the baseline.

Trade-offs to say out loud

• A second store is a copy to secure, refresh and reconcile; it buys milliseconds and predictable concurrency.
• Pre-aggregation fixes the questions in advance; drill-down to raw events goes to the slower tier.
• A managed warehouse with result caching is less to run; an OLAP store is cheaper per query at high concurrency but needs operating.

Common weak answers

• “Add Trino workers” with no caching or serving tier: parallelism per query doesn’t remove per-query overhead.
• Caching without saying how stale a tile may be, or what happens when the nightly job is late.
• No load estimate: no queries per second, no bytes per query.

Further Reading

1. Trino 484 documentation: “Trino concepts”, “Fault-tolerant execution” and the PostgreSQL connector’s pushdown section. https://trino.io/docs/current/overview/concepts.html
2. R. Sethi et al., “Presto: SQL on Everything”, ICDE 2019: the design Trino continues.
3. B. Dageville et al., “The Snowflake Elastic Data Warehouse”, SIGMOD 2016; S. Melnik et al., “Dremel: Interactive Analysis of Web-Scale Datasets”, VLDB 2010.
4. R. Schulze et al., “ClickHouse — Lightning Fast Analytics for Everyone”, PVLDB 17(12), 2024; and ClickHouse’s MergeTree and refreshable materialized view docs. https://clickhouse.com/docs/engines/table-engines/mergetree-family/mergetree
5. Snowflake, “Overview of warehouses” and Google, “BigQuery pricing”: two billing models side by side. https://docs.snowflake.com/en/user-guide/warehouses-overview · https://cloud.google.com/bigquery/pricing

Story Closing

The stores dashboard now reads ClickHouse’s serving table, refreshed hourly and stamped with its refresh time; Trino stays for analysts and the targets join; no dashboard touches production PostgreSQL. Of the five promises from Part 2, this part strengthened timeliness: a number is only on time if it arrives when people look, and says how old it is.

“So which engine did we pick?” Farah asked.

“Three,” Ingrid said. “One for each workload, all fed from one copy of the truth. Next week Rohan defends every choice we’ve made so far, and you bring the numbers.”

Part 14 is the first design review: Rohan defends Larkspur’s storage, table format, catalog, partitioning and engine choices against finance’s volumes, retention, freshness and budget.

This is Part 13 of “Every Record Counts: Data Engineering for Backend Engineers” — Season 1: Foundations & the Lakehouse.