Project Every Record Counts: Data Engineering for Backend Engineers
Part 18: "dbt at Scale" — Incremental Models, CI and Environments
Rohan wants a high-water mark on event time to stop rebuilding the sales fact every night. Ingrid makes Rohan replay two weeks of till uploads first. Incremental strategies and how each treats late data, microbatch and its lookback, full-refresh policy, dev and prod targets with their own principals, state-based slim CI with deferral, contracts and model versions. On dbt 1.12 and Trino, the naive model loses 7,146 sale lines; microbatch with a two-day lookback matches a full rebuild row for row.