Dbt
3 items
Series Posts
Part 17: "Transformations as Code" — dbt Fundamentals
Rohan wants to put the star schema's SQL in a repo and a cron job. Ingrid wants it treated like the order service: modules, a dependency graph, tests, review. Models, sources and refs, materialisations, snapshots, data and unit tests, docs, and dbt v1 vs v2. Built with dbt 1.12 and dbt-trino on Iceberg, it reproduces Part 5's corrected numbers to the penny; dbt-oss 2.0 then runs a slice of it on DuckDB.
Part 18: "dbt at Scale" — Incremental Models, CI and Environments
Rohan wants a high-water mark on event time to stop rebuilding the sales fact every night. Ingrid makes Rohan replay two weeks of till uploads first. Incremental strategies and how each treats late data, microbatch and its lookback, full-refresh policy, dev and prod targets with their own principals, state-based slim CI with deferral, contracts and model versions. On dbt 1.12 and Trino, the naive model loses 7,146 sale lines; microbatch with a two-day lookback matches a full rebuild row for row.
Part 19: "Who Runs What, When" — Orchestration with Airflow 3
Rohan wants a cron entry and a Spring service with a retry loop. Ingrid wants a record of every day the pipeline ran, a way to re-run any of them, and no business logic in the scheduler. DAGs, logical dates and data intervals, backfills, assets and asset partitioning, retries, pools, sensors and deferrable operators, Deadline Alerts and DAG versioning, with Dagster and Prefect compared. On Airflow 3.3, a month-long backfill republishes the daily revenue report and UK Q3 stays at £29,790,674.06.