DuckDB — Tags

Series Posts

Conceptual Every Record Counts: Data Engineering for Backend Engineers

Part 2: "Think in Pipelines" — The Data Engineer's Mental Model

Rohan's first daily sales job doubles a day's revenue on a retry and loses late till uploads. Rebuilding it teaches the data engineer's mental model: pipelines as functions, idempotency, partition-scoped overwrites, raw/silver/gold layers, and the five promises behind "every record counts".

Project Every Record Counts: Data Engineering for Backend Engineers

Part 6: "SQL You Will Actually Use" — Analytical SQL for Data Engineers

Rohan wants to loop over events in Java; Ingrid hands over a SQL workbook instead. Window functions, QUALIFY, gaps and islands, sessions, PIVOT, CUBE, HyperLogLog, MERGE, high-water marks and query plans in DuckDB and Trino find the 2,455 re-sent UK baskets behind £118,076.98 and explain why conversion reads 6.21% or 5.31%.

Project Every Record Counts: Data Engineering for Backend Engineers

Part 11: "You Might Not Need a Cluster" — DuckDB, Polars and Single-Node Analytics

Rohan wants a Spark cluster for four years of sales lines. Ingrid runs the same Iceberg table on a laptop with DuckDB and Polars instead, and profiles it against Spark. Vectorised execution, out-of-core processing, Arrow hand-offs, DuckDB 2.0 and Polars 2.x over a REST catalog, and the signals that say one machine is no longer enough.