"You Might Not Need a Cluster" — DuckDB, Polars and Single-Node Analytics

Skim Card

One thing to remember: one modern machine answers far more than a backend engineer expects; reach for a cluster when a measured limit says so, not when the data sounds big.

• On a laptop, DuckDB and Polars answered four years of sales lines (72.7 million rows, 1 GiB) in 0.01–3.0 s through the shared catalog.
• With Spark’s share (4 threads, 1.5 GB), DuckDB beat Spark 4.7x on the full scan and 5.6x on 28 million baskets (not a benchmark).
• A memory limit makes a vectorised engine spill, not crash: DuckDB wrote 2.2 GiB to disk and returned the same answer.
• Memory limits cap the engine’s buffers, not the process: with 1.5 GB limits, both processes still peaked near 1.8–2.3 GiB.
• Three engines, one Iceberg table, identical digests: the table format makes the engine a cost choice, not a truth choice.
• Leave one machine for concurrency, multiple writers, fault tolerance or data beyond its disk, not for row counts that merely sound big.

Use …	When …
DuckDB	SQL over lake tables, scheduled jobs, notebooks; one writer per database file
Polars	Dataframe pipelines in Python or Rust; lazy queries over Parquet or Iceberg
DataFusion	You’re building your own engine or service in Rust
Spark, Trino or a warehouse	Many concurrent users, data larger than one disk, or jobs that must survive a lost machine

Verified against: DuckDB 2.0.0, Polars 2.1.0 (+ polars-iceberg 0.1.0), PyIceberg 0.12.0, Spark 4.1.3 + Iceberg 1.12.0, Apache Polaris 1.8.0 · vendor docs · 28 October 2026

Jump to: Concepts · Hands-On · Landscape · Design Drill

Story Opening

Part 10’s table had 72.6 million rows (72.7 million once 4 October’s micro-batches landed) and three layouts, and every measurement had come from Spark. Rohan wrote the request on the team board: a three-node Spark cluster for “the history workloads”.

Ingrid read it and opened a laptop. “Same table, same catalog, no cluster. Then you can tell me what the cluster buys.”

The Quick Map

Concepts, with use cases

Single-node engines, and why one machine goes so far

A single-node analytical engine runs a whole query inside one process on one machine, using every core: DuckDB and Polars embed in your Python process, read Parquet and Iceberg from object storage, and need no server. Your instinct says “big data needs a cluster” because a decade ago it did: Hadoop-era engines scaled out because one machine had a few cores and spinning disks. A 2026 laptop has twelve cores, tens of gigabytes of memory and an SSD measured in gigabytes per second, and columnar files mean a query reads a fraction of the bytes.

What breaks without it. Teams pay for a cluster’s network shuffles, JVM startup, autoscaling and on-call for data that fits on one disk, and every query waits for a scheduler. Larkspur: four years of sales lines are 1 GiB of Parquet in the lab and about 50 GB at full volume (Part 10). Elsewhere: a smart-thermostat maker aggregates a few gigabytes of daily device telemetry in a DuckDB job on one VM.

Vectorised execution

A vectorised engine passes batches of values between operators, one column at a time: DuckDB’s default vector is 2,048 values. The classic database instead pulls one row at a time through every operator (the iterator model you know from PostgreSQL’s EXPLAIN), paying function calls and type checks per row. Batches amortise that overhead and keep tight loops in CPU caches. Each thread takes its own slice of row groups, so parallelism needs data: DuckDB’s tuning guide says a query on k threads needs at least k × 122,880 rows.

What breaks without it. Per-row interpretation burns CPU on bookkeeping, not arithmetic. Larkspur: the four-year revenue view sums 72.7 million rows in about two seconds (step 2). Elsewhere: an ad-tech team computes campaign rollups over a day of impressions in-process instead of submitting a job.

Out-of-core processing

Out-of-core (or larger-than-memory) processing means an operator that holds state, such as a hash aggregate, join, sort or window, writes part of that state to a temporary directory when it reaches its memory limit, and reads it back later. DuckDB’s docs say all four spill, with exceptions (aggregates such as string_agg() don’t; several blocking operators in one query can still run out). Polars 2.0 turned on out-of-core processing for its streaming engine by default, with a budget of 80% of available RAM.

What breaks without it. A query that needs 10 GB on an 8 GB machine is killed, often at 3 a.m. Larkspur: counting lines per basket builds a hash table of about 28 million baskets (step 3). Elsewhere: a genomics lab joins variant calls bigger than any workstation’s memory.

Arrow as the hand-off

Part 3 introduced Apache Arrow as the in-memory columnar format. Single-node engines use it as a hand-off: DuckDB returns Arrow, Polars adopts the buffers, and DuckDB can query a Polars frame by name without converting rows.

What breaks without it. A pandas or CSV round trip between tools serialises every value and can take longer than the analysis. Larkspur: Part 3’s promise, data scientists moving sales lines from DuckDB to Polars for forecasting (step 4). Elsewhere: a quant desk passes market data from a SQL engine to a dataframe library and back in one process.

The table is the contract

The engine doesn’t own the data; the Iceberg table and its catalog do (Part 9). DuckDB attaches Polaris and reads (and writes) the same snapshot Spark wrote; Polars plans the scan with PyIceberg or its own planner and reads the files with its native Parquet reader. So “laptop or cluster” stops being a migration and becomes a per-job choice.

What breaks without it. A team that copies the lake into a local DuckDB file has two copies and two answers. Larkspur: finance’s revenue view runs on whichever engine is cheapest that day, with the same digest. Elsewhere: a media company’s analysts query in DuckDB while its nightly jobs run in Spark, over one table.

The engines

• DuckDB 2.0.0 (LTS, 21 October 2026, supported to at least October 2027): attached Iceberg REST catalogs support CREATE TABLE, INSERT, UPDATE, DELETE and MERGE INTO, merge-on-read only (positional deletes; copy-on-write “not yet supported”); Delta reads, time travel, CREATE TABLE and INSERT; DuckLake 1.1 (Part 8); asynchronous I/O for object storage; and Quack, a client-server protocol with a CONNECT statement, (beta: “the protocol is not yet stable”).
• Polars 2.x (2.0.0 on 6 October, 2.1.0 on 25 October): the lazy API (scan_*, then collect()) lets the optimiser push filters and columns into the scan; collect() now runs on the streaming engine by default (engine="in-memory" opts out); scan_iceberg reads Iceberg and sink_iceberg writes it (unstable, per its docstring).
• Apache DataFusion (55.2 Rust crate, 55.0 Python package): “an extensible query engine written in Rust that uses Apache Arrow as its in-memory format”. You build on it rather than run it: DataFusion Comet accelerates Spark with it.
• MotherDuck is DuckDB as a managed service; its docs list client versions 1.4.x to 1.5.6, so not 2.0 yet.

When one machine stops being enough

The first row is arithmetic, not a feeling. Larkspur at full volume is about 50 GB of sales lines; the step 2 rate (72.7 million rows in 2.25 s) extrapolates to under two minutes for a four-year full scan at 50 times the rows, if storage keeps up. That’s an estimate, not a measurement. The basket aggregation is where scale bites first: it peaked at 3.8 GiB in the lab, so roughly 190 GiB at 50 times the baskets (assuming memory grows with basket count), nearly all of it spilled, and far past Polars’ 64 GB default disk budget for spilling.

Under the Hood

What a vectorised plan does with a full scan

The history query (GROUP BY month, country, currency over every row) becomes three operators: an Iceberg scan, a projection and a hash aggregate. The scan asks the catalog for the snapshot, prunes files by partition and column statistics (nothing to prune here: it reads all 197 files), then reads only the projected columns’ pages, here 5 of 13 columns, 107.5 MiB of 1,027 MiB of files (step 2). Each thread decodes a row group into vectors, computes date_trunc for 2,048 values at once and folds them into a thread-local hash table of 153 groups; the tables merge at the end. Nothing is shuffled between machines because there’s only one.

How an aggregation spills

The basket query groups by basket_id, about 28 million keys. An engine that can spill partitions the hash table by a hash of the key, so each partition can be finished on its own. When the buffer manager reaches memory_limit, partitions are written to temp_directory and processed one at a time; the answer is the same, the work is the disk round trip. Spark does the same across machines, with a shuffle: hash-partition, write to disk, read on the other side (Part 12).

Why local Spark was slower here

Spark 4.1 in local mode still plans stages, writes shuffle files between them (8 partitions here), schedules tasks and runs in a JVM in a 4-vCPU Docker VM, and the host engines even pay Docker’s port forwarding to reach SeaweedFS. It isn’t a fair benchmark of Spark, and it isn’t meant to be: it’s what this team would get from the tool it was about to buy, on this data.

Hands-On

The code is in the every-record-counts folder of the companion-workspaces repository (tag every-record-counts/part-11). The machine is a MacBook Pro M3 Pro: 12 cores (6 performance, 6 efficiency), 36 GB of RAM; Docker’s VM has 4 CPUs and 9.8 GB. DuckDB defaults to 12 threads and a 28.7 GiB memory_limit (80% of RAM); Polars to 12 threads.

The data is Part 10’s pos_monthly as Part 10 left it: 72,670,965 rows (four years plus 4 October’s micro-batches) in 197 files, 1,027 MiB, partitioned by month and sorted by store. The four queries in sql/part11/queries/ run unchanged in DuckDB and Spark; scripts/part11_single_node.py holds their Polars twins.

make up-core && uv sync                                   # polars[iceberg] adds the native Iceberg planner
make up-spark                                             # Spark only for step 1; Part 10's tables must exist
uv run python -m scripts.part11_single_node spark
make stop-batch                                           # nothing else on the machine while the host engines run
uv run python -m scripts.part11_single_node profile
uv run python -m scripts.part11_single_node profile --threads 4 --memory-mb 1500

Every engine runs each query once (first_s), then three more times (warm_s, median), and digests the answer the same way: values as text, rows sorted, SHA-256. Each host measurement runs in a fresh process, so peak_mib (peak resident memory) belongs to one query. DuckDB’s external file cache is off, so every run fetches from S3. Outputs below drop separator lines; other trims are marked.

Step 1: Spark, the baseline

spark-submit: getOrCreate() took 1.7 s once the driver ran; whole job 149 s
engine  table       query       limits               ready_s first_s warm_s peak_mib  s3_mib spill_mib rows  digest
spark   pos_monthly revenue     4 cores, 1.5 GB heap       -    3.65   0.87    2,134       -         -    3  fa139210
spark   pos_monthly store_promo 4 cores, 1.5 GB heap       -    0.87   0.70    2,134       -         -   10  ff141597
spark   pos_monthly history     4 cores, 1.5 GB heap       -   14.02  11.98    2,134       -         -  153  b711851d
spark   pos_monthly baskets     4 cores, 1.5 GB heap       -   23.40  20.94    2,134       -         -   35  241362c2

Spark’s peak_mib is the container’s, from docker stats; ready_s is for the host engines. A spark-submit for revenue alone took 9 s end to end (--queries revenue).

Step 2: DuckDB and Polars, the whole laptop

engine  table       query       limits               ready_s first_s warm_s peak_mib  s3_mib spill_mib rows  digest
duckdb  pos_monthly revenue     all thr, default mem    0.13    0.18   0.08      133     1.9         0    3  fa139210
duckdb  pos_monthly store_promo all thr, default mem    0.11    0.06   0.05      116     0.7         0   10  ff141597
duckdb  pos_monthly history     all thr, default mem    0.10    2.41   2.25      208   107.5         0  153  b711851d
duckdb  pos_monthly baskets     all thr, default mem    0.10    2.40   2.17    3,930   294.8         0   35  241362c2
polars  pos_monthly revenue     all thr, default mem    0.59    0.15   0.03      245       -         -    3  fa139210
polars  pos_monthly store_promo all thr, default mem    0.36    0.04   0.01      192       -         -   10  ff141597
polars  pos_monthly history     all thr, default mem    0.34    1.28   1.25      501       -         -  153  b711851d
polars  pos_monthly baskets     all thr, default mem    0.34    3.43   2.99    9,107       -         -   35  241362c2
baskets query: 27,952,255 baskets (the inner GROUP BY's keys)

Every digest matches Spark’s. DuckDB was ready to query 0.1 s after the script started (import and ATTACH), Polars after 0.34–0.59 s (import and catalog load). The selective queries answer in milliseconds and a few hundred MiB: they’re mostly catalog and footer reads. The full scan takes 1.3–2.3 s and stays under 0.5 GiB, because a three-key aggregate holds 153 groups. The basket query is the memory one, a hash table of 27,952,255 baskets: DuckDB peaked at 3.8 GiB and Polars at 8.9 GiB, with nobody asking either to hold back. Run against Part 10’s day-partitioned pos_daily (1,511 small files), the history query took 5.04 s in DuckDB and 2.52 s in Polars: small files cost a single machine too. (Its digest, c101f245, differs from pos_monthly’s because 4 October’s micro-batches went only into pos_monthly; compare the times.)

uv run python -m scripts.part11_single_node explain prints Polars’ plan for the revenue query: the filter and four of thirteen columns went into the scan, and partition pruning left September’s one month partition, four files (... 3 other sources). Trimmed to the scan:

    Parquet SCAN [s3://larkspur-lake/warehouse/part10/pos_monthly/data/business_date_month=2026-09/00007-534-…-00022.parquet, ... 3 other sources]
    PROJECT 4/13 COLUMNS
    SELECTION: col("business_date").is_between([2026-09-01, 2026-09-30])

The revenue query in Polars, from scripts/part11_single_node.py (trimmed):

def polars_scan(table: str):                               # nothing is read yet: a LazyFrame over the snapshot
    return pl.scan_iceberg(p7.catalog().load_table(f"part10.{table}"), storage_options=STORAGE)

"revenue": lambda lf: (
    lf.filter(pl.col("business_date").is_between(date(2026, 9, 1), date(2026, 9, 30)))
    .group_by("country", "currency")
    .agg(lines=pl.len(), sales_minor=pl.col("amount_minor").sum())
),                                                         # q.collect() runs it, on the streaming engine

Step 3: a Spark-sized share of the machine

The fairer comparison gives the host engines what Spark had: 4 threads and 1.5 GB (SET threads, SET memory_limit for DuckDB; POLARS_MAX_THREADS and POLARS_OOC_MEMORY_BUDGET_MB for Polars, the latter from Polars’ source, not yet in its user guide).

engine  table       query       limits               ready_s first_s warm_s peak_mib  s3_mib spill_mib rows  digest
duckdb  pos_monthly history     4 thr, 1500 MB          0.11    2.73   2.57      175   107.5         0  153  b711851d
duckdb  pos_monthly baskets     4 thr, 1500 MB          0.11    3.70   3.75    1,851   294.8     2,225   35  241362c2
polars  pos_monthly history     4 thr, 1500 MB          0.34    1.31   1.31      302       -         -  153  b711851d
polars  pos_monthly baskets     4 thr, 1500 MB          0.34    7.25   7.08    2,383       -         -   35  241362c2

(Revenue and store promotion rows and the basket-count line trimmed: 0.01–0.11 s.) Against Spark’s 11.98 s and 20.94 s, DuckDB was 4.7 and 5.6 times faster, Polars 9.1 and 3.0 times. DuckDB’s basket query spilled 2,225 MiB to its temp directory and Polars’ resident peak fell from 8.9 to 2.3 GiB. The price was time (1.7 and 2.4 times their unlimited runs); the digests didn’t move. Note that neither process stayed under 1.5 GB (Gotchas).

Step 4: hand-offs and a gold table

uv run python -m scripts.part11_single_node arrow
uv run python -m scripts.part11_single_node publish

DuckDB -> Arrow table: 1,434,587 rows, 159.6 MiB, 0.15 s
Arrow -> Polars DataFrame: 6.5 ms
DuckDB SQL over the Polars frame `per_store`: [('IN-0123', 598184879), ('IN-0058', 598008878), ('IN-0284', 595579687)] (0.7 ms)
DuckDB: sql/part11/publish.sql (2.5 s)
Polaris: snapshot 942062381480234093, written by DuckDB, format v2

(Polars’ printout of the table read back is trimmed.)

September’s 1.43 million lines moved from DuckDB to Polars in 6.5 ms, and DuckDB queried the Polars aggregate through con.register("per_store", per_store). Then sql/part11/publish.sql computed the four-year revenue view with DuckDB and wrote it through Polaris as lake.part11.revenue_by_month, a table Spark, Trino and Polars can read like any other; Polars read it back (the test checks every row against the history digest).

Step 5: where it stops

uv run python -m scripts.part11_single_node lock

second process, read_only=False: _duckdb.IOException: IO Error: Could not set lock on file "data/lab/part11/lock.duckdb"
second process, read_only=True: _duckdb.IOException: IO Error: Could not set lock on file "data/lab/part11/lock.duckdb"

While one process writes a DuckDB file, no other process can open it, not even read-only. DuckDB’s concurrency page offers read-only mode for many readers when nobody writes, and Quack or DuckLake for several writers. The Iceberg table in step 4 never had this problem: commits go through the catalog (Part 7).

uv run pytest -q tests/test_part11.py checks all of it: 9 passed and 1 skipped with Spark down (128 s); with Spark up, make ci also ran the test that compares Spark’s digests, and it passed.

The Landscape

As of 28 October 2026, from each vendor’s docs.

Tips & Gotchas

Gotcha — Polars 2.x doesn’t keep group order. The streaming engine is now the default, and Polars lists join, group_by and unpivot (and the lab saw unique) as returning rows in whatever order threads finished. The same September query, three runs each:

maintain_order=False run 1: first three groups ['GB-0117', 'GB-0118', 'GB-0119']
maintain_order=False run 2: first three groups ['GB-0139', 'GB-0140', 'GB-0141']
maintain_order=False run 3: first three groups ['IN-0285', 'IN-0286', 'IN-0287']
maintain_order=True  run 1: first three groups ['GB-0001', 'GB-0002', 'GB-0003']

(maintain_order=True runs 2 and 3 printed the same as run 1.)

Anything that takes “the first row” without a sort changes its answer. The lab’s own generator broke this way on unique() until it used maintain_order=True. Sort explicitly, or ask for order.

Gotcha — DuckDB 1.x can’t open a file DuckDB 2.0 created. New files use the 2.0 storage format by default:

DuckDB 2.0 ATTACH 'default.duckdb' -> opened by DuckDB 1.5.6: _duckdb.IOException: IO Error: Trying to read a database file with version number 69, but we can only read versions between 64 and 68.
DuckDB 2.0 ATTACH 'v1_5.duckdb' (STORAGE_VERSION 'v1.5.0') -> opened by DuckDB 1.5.6: 1.5.6 (42,)

Pin STORAGE_VERSION (or storage_compatibility_version) while anything that opens the file is still on 1.x. Files older than 2.0 keep their version when 2.0 writes to them.

Gotcha — a memory limit isn’t a process limit. DuckDB’s memory_limit caps its buffer manager and Polars’ budget is the point where it starts spilling. With 1,500 MB each, the processes peaked at 1,851 and 2,383 MiB resident. Size containers with headroom, or the kernel’s OOM killer does what the engine avoided.

Gotcha — DuckDB’s sum(BIGINT) is a HUGEINT. Written to Iceberg, which has no 128-bit integer, it arrived as DECIMAL(38,0) in the first publish run. Cast sums to BIGINT before you publish (publish.sql does).

Gotcha — DuckDB writes Iceberg merge-on-read only. UPDATE, DELETE and MERGE write positional deletes and fail on a table whose write.update.mode or write.delete.mode says copy-on-write. Plan compaction (Part 10) for tables DuckDB updates.

Tip — give spilling a fast, roomy disk. DuckDB spills to temp_directory (<database file>.tmp by default, or .tmp in the working directory for an in-memory database) up to max_temp_directory_size, 90% of available disk by default; Polars’ out-of-core budget for disk is 64 GB unless changed. Point both at local NVMe on a VM, not a network volume, and alert on spill size: it’s the early warning that one machine is getting tight.

Troubleshooting

Design Drill

The question: “A 25-person last-mile delivery startup has 200 GB of parcel scan events (three years, growing 8 GB a month) in Parquet on S3, plus a Postgres of depots and drivers. The new data lead wants a Spark cluster for daily reporting and route-quality analysis. Make the case for and against.”

Clarify first

• 200 GB compressed or raw? Which queries scan all of it, and how often?
• How many people query concurrently, and do they need interactive answers?
• What’s the freshness target: a nightly batch or minutes?
• Who writes the tables, and how many jobs write at once?
• What’s the biggest join or aggregation (scans × drivers × days), and its memory need?
• What happens if a nightly job fails at 5 a.m.?

The shape of a strong answer

• Size it: 200 GB compressed is about the lab’s table × 200; reports filter by date, so typical scans read a month (6–8 GB) and a few columns.
• Make the data a table first: Iceberg with a REST catalog, partitioned by month of scan date, sorted by depot. The engine decision then stays reversible.
• Start single-node: DuckDB jobs on one large VM (say 16–32 cores, 128 GB, local NVMe for spilling), orchestrated like any batch job; Polars where the team prefers dataframes.
• Measure the top 10 queries: time, peak memory, bytes read; set alarms on spill size and runtime against the batch window.
• Serve dashboards from small gold tables (daily depot metrics), not from raw scans.
• For concurrency, a serverless engine over the same Iceberg tables (Athena or a warehouse) beats a hand-run Spark cluster at this size.
• Give the case for Spark its due: if the raw scans land as uncompressed JSON that must be re-parsed every day, that parsing is CPU-bound and spreads well across machines; if live parcel tracking is on next year’s roadmap, Structured Streaming keeps one engine for batch and streams.
• And if the company already runs Spark (or a managed Spark), the marginal cost of one more job is small and the team knows how to operate it; the argument against is about running a new cluster, not about Spark.
• Name the triggers for Spark: data beyond one machine’s disk, a job missing its window after layout fixes, many writers, or failures that must retry per task.

Trade-offs to say out loud

• One VM is a single point of failure; reruns must be idempotent (Part 2) because the job restarts from zero.
• Spark’s ecosystem (streaming, MLlib, team hiring) can matter more than speed; the case against is about today’s 200 GB, not forever.
• A cluster has a fixed ops cost; a big VM has a ceiling. Growth of 8 GB a month leaves years before that ceiling.

Common weak answers

• “200 GB is big data”: no arithmetic on what one query actually reads.
• Choosing DuckDB with a local database file as the shared store, then hitting one-writer locks and copies.
• Ignoring memory: one high-cardinality join spills or dies, and nobody measured it.

Further Reading

1. DuckDB, “Announcing DuckDB 2.0.0 LTS”. https://duckdb.org/2026/10/21/announcing-duckdb-200
2. DuckDB, “Tuning Workloads” (larger-than-memory processing, threads) and “Writing to Iceberg”. https://duckdb.org/docs/2.0/guides/performance/how_to_tune_workloads
3. Polars, “Pre-release of Polars 2.0” (2 September 2026: streaming engine default, maintain_order). https://pola.rs/posts/announcing-polars-2/ and the Python Polars 2.0.0 release notes (out-of-core on by default). https://github.com/pola-rs/polars/releases/tag/py-2.0.0
4. Apache DataFusion documentation. https://datafusion.apache.org/
5. P. Boncz, M. Zukowski, N. Nes, “MonetDB/X100: Hyper-Pipelining Query Execution”, CIDR 2005: the paper that introduced vectorised execution, from CWI, where DuckDB later began.

Story Closing

Rohan withdrew the request. Larkspur’s history views would run as DuckDB jobs on one VM against the governed tables, with Polars for the forecasting team, and a list of triggers that would reopen the question. Of the five promises from Part 2, this part strengthened consistency: three engines read one Iceberg table and produced the same digest for every view.

“So when do we need Spark?” Rohan asked.

“When the work has to move between machines,” Ingrid said. “And then most of your problems will be about the moving.”

In Part 12, the data gets a campaign and a store with 30% of the traffic, and Rohan learns why shuffles, spills and skew decide what a Spark job costs.

This is Part 11 of “Every Record Counts: Data Engineering for Backend Engineers” — Season 1: Foundations & the Lakehouse.