Project Every Record Counts: Data Engineering for Backend Engineers
Part 11: "You Might Not Need a Cluster" — DuckDB, Polars and Single-Node Analytics
Rohan wants a Spark cluster for four years of sales lines. Ingrid runs the same Iceberg table on a laptop with DuckDB and Polars instead, and profiles it against Spark. Vectorised execution, out-of-core processing, Arrow hand-offs, DuckDB 2.0 and Polars 2.x over a REST catalog, and the signals that say one machine is no longer enough.