"dbt at Scale" — Incremental Models, CI and Environments

Skim Card

One thing to remember: an incremental model is correct only if each run recomputes everything that could still have changed; size that window from measured lateness, never from the newest event you hold.

• A high-water mark on event time skips late uploads silently: replayed over 15 days, it lost 7,146 sale lines, £64,778.44 of them British.
• Microbatch’s default lookback of one batch is too short when runs start after midnight UTC: 6,179 lines lost.
• lookback=2 matched a full rebuild row for row, and the late 30 September basket landed on 30 September.
• A daily run read 21 MiB in 3.1 s; a full rebuild, 199 MiB in 16.8 s. A microbatch rebuild runs one query per day.
• dbt-trino has no insert_overwrite; microbatch, or delete+insert keyed by day, does that job on Trino.
• Slim CI: production’s manifest, state:modified+, --defer: 37 of 69 nodes in 25.5 s; dev’s principal can’t drop production (via lakehouse_pipeline it can, until Part 23).
• On Trino a contract enforces only NOT NULL; a duplicate key went straight in, so keep the grain test.

Use …	When …
microbatch with a measured lookback	Facts keyed by event time, lateness bounded (Larkspur: under a day)
merge on a high-water mark of ingestion time	Lateness unbounded, rows only added or updated
delete+insert keyed by business date	Aggregates by local day, or engines without microbatch
A plain table	It rebuilds in seconds; incremental buys nothing yet

Verified against: dbt-core 1.12.6, dbt-trino 1.11.0, Trino 484, Apache Polaris 1.8.0, Apache Iceberg format v2 · vendor docs · 29 October 2026

Jump to: Concepts · Hands-On · Landscape · Design Drill

Story Opening

Part 17 left Larkspur’s star schema in a dbt project that rebuilds every table every night, mostly rewriting rows that haven’t changed to add one day.

“Classic high-water mark,” Rohan said. “Remember the newest event_ts loaded, take everything after it. I’ve run Kafka consumers on that idea for ten years.”

“A Kafka offset only grows,” Ingrid said. “Event time doesn’t. A till that lost its signal at noon uploads at four the next morning, and Farah closes the quarter on the 30th’s numbers. Build it your way in a sandbox, replay two weeks of uploads, and count what’s missing.”

The Quick Map

Concepts, with use cases

Incremental models

An incremental model is a table dbt builds in full once and then updates: on later runs is_incremental() is true, {{ this }} names the existing table, and the SQL returns only the rows to add or replace. --full-refresh rebuilds it from scratch.

What breaks without it. A table model rewrites all of history to add a day. Larkspur: fct_sales_line holds 6.7 million lines at the lab’s 1:50 scale; at full volume a nightly rebuild rewrites a third of a billion rows to add 2.6 million. Elsewhere: a streaming service’s play-events fact, where years of history dwarf the hours that changed.

Strategies: what a run does with those rows

insert_overwrite is adapter-dependent (dbt-spark, dbt-databricks, dbt-bigquery); on Trino, microbatch or delete+insert keyed by day does its job. Larkspur: microbatch for the till facts, merge where a CDC replica only adds and updates. Elsewhere: a ride-hailing app merging late fare adjustments into its trips fact.

The high-water mark, and when it lies

A high-water mark is “load everything newer than the newest row I already have”. It’s exact on a clock that only moves forward in arrival order: a Kafka offset, or an ingestion timestamp stamped in commit order (parallel loaders commit out of order, so re-read a small overlap). It lies on event time, because events don’t arrive in event-time order. where event_ts > (select max(event_ts) from {{ this }}) skips a sale rung up last night and uploaded this morning, with no error and no visible gap.

Two honest fixes. Mark ingestion time instead: nothing is skipped whatever its lateness, but each source needs its own mark and the model only adds and updates. Or recompute a lateness window of event time each run: corrections and deletes inside it are absorbed too, anything later is missed, so the window comes from measured lateness, as Part 2’s LATE_DAYS did. Larkspur: sql/part18/lateness.sql counts 22,454 till events (20,029 baskets) uploaded on a later UTC day than they were rung up, at worst 16 h 28 min late. Elsewhere: a fitness-watch platform whose devices sync workouts when the phone is next in range.

Microbatch

microbatch (dbt 1.9+) is that window built into dbt. Configure event_time (the column), batch_size (hour, day, month or year), begin (where history starts) and lookback (batches before the current one to redo; default 1). Each run takes its batches from the clock, and dbt filters every upstream ref or source that declares an event_time to the batch. Backfills name a range with --event-time-start and --event-time-end (both or neither). All times are UTC.

What breaks without it. Hand-written windows drift between models, and a re-run after an outage needs someone to edit dates. Larkspur: the sales and basket facts, one batch per UTC day. Elsewhere: an ad platform redoing the last six hours of spend as late clicks arrive.

Full refresh as a policy

A full refresh is needed when the logic or the schema changes, and dangerous when the source no longer holds all history: someone runs --full-refresh “to be safe” and years older than raw retention are gone. dbt’s docs recommend full_refresh: false on microbatch models: history then changes only through named backfills (--event-time-start/--event-time-end). Larkspur: raw uploads are kept, so the facts can be rebuilt (step 5 times it); set the flag after the first deploy, which itself relies on --full-refresh. Elsewhere: a bank’s balances fact over a 90-day source, where changes ship only as backfills.

Environments: targets, namespaces, principals

A target in profiles.yml is one place dbt builds. generate_schema_name picks the namespace per target: production keeps each model’s (silver, gold), every other target puts all models in its own. What breaks without it. A developer’s dbt run replaces a production table mid-afternoon. Larkspur: prod (Part 17’s target, renamed from lake) and dev, a namespace written by its own principal, developer, which Polaris lets drop tables there and nowhere else: Part 17’s purge Gotcha, answered with privileges (step 1 says what Trino still allows). Elsewhere: a SaaS analytics team with one schema per pull request.

State, slim CI and deferral

Every dbt parse writes manifest.json, the graph with a checksum per node. Given production’s manifest (--state), state:modified selects what a change touches and state:modified+ adds everything downstream. Deferral (--defer) points ref() to an unselected model at production’s table; --favor-state does so even if a stale copy sits in dev. Together that’s slim CI. dbt clone instead creates relations in dev that point at production’s: on Trino, which has no zero-copy clone, select * views. What breaks without it. CI either rebuilds everything or tests nothing. Larkspur: make dbt-ci. Elsewhere: a fintech’s 2,000-model project where a pull request touching one staging model runs the 40 nodes it affects.

Contracts and versions

A model contract (contract: {enforced: true}) declares every column’s name and type; dbt compares them with the query’s output before building and refuses a mismatch. Constraints are the platform’s job: dbt-trino enforces only not_null, as a Trino NOT NULL column; primary_key, unique, foreign_key and check aren’t supported on Trino: dbt warns and leaves them out of the DDL. A model version ships a breaking change (renamed column, new type or grain) beside the old shape: ref('m') resolves to latest_version, ref('m', v=1) pins, deprecation_date ends the overlap. What breaks without them. A renamed column breaks a dashboard silently. Larkspur: rpt_daily_net_revenue, behind Farah’s revenue dashboard. Elsewhere: a logistics platform’s shipment-status model read by 30 partner integrations.

Under the Hood

Which batches a run picks

dbt takes the run’s start time, rounds it up to the end of its batch, and goes back lookback batches from the start of the current one (dbt-core’s MicrobatchBuilder). A run at 00:30 UTC on 2 October, batch_size='day':

The rule: a run after midnight needs one batch for the day that just closed, plus one per UTC midnight an upload can cross. Larkspur’s tills cross at most one, so lookback=2; if lateness ever exceeds it, Part 17’s assert_no_sale_line_lost test is the alarm.

What one batch sends to Trino

batch-sql prints what the deploy’s last batch sent, from dbt’s log (column lists and joins trimmed):

create or replace view "lakehouse_pipeline"."gold"."fct_sales_line__dbt_tmp_20261005" …
  from (select * from "lakehouse_pipeline"."silver"."int_sales_lines" where event_ts >= TIMESTAMP '2026-10-05 00:00:00+00:00' and event_ts < TIMESTAMP '2026-10-06 00:00:00+00:00') l …
delete from "lakehouse_pipeline"."gold"."fct_sales_line" where ( event_ts >= TIMESTAMP '2026-10-05 00:00:00+00:00' and event_ts < TIMESTAMP '2026-10-06 00:00:00+00:00' )
insert into "lakehouse_pipeline"."gold"."fct_sales_line" ("date_key", "store_key", … ) (select … from "…fct_sales_line__dbt_tmp_20261005")
drop view if exists "lakehouse_pipeline"."gold"."fct_sales_line__dbt_tmp_20261005"

The upstream filter is a subquery around int_sales_lines, now a view. explain asks Trino for its plan of one day of that view; the raw scans, with their filters (timestamps shortened, reference-table scans trimmed):

pos_uploads  '2026-09-15 00:00:00.000000 UTC' <= event_ts_9 AND event_ts_9 < '2026-09-16 00:00:00.000000 UTC'
pos_uploads  '2026-09-15 00:00:00.000000 UTC' <= event_ts_24 AND event_ts_24 < '2026-09-16 00:00:00.000000 UTC'
pos_uploads  '2026-09-15 00:00:00.000000 UTC' <= event_ts_1 AND event_ts_1 < '2026-09-16 00:00:00.000000 UTC'
orders       '2026-09-15 00:00:00.000000 UTC' <= created_at AND created_at < '2026-09-16 00:00:00.000000 UTC'

All three scans of the till uploads (the lines, and the two that find re-sent baskets) and the orders read one day; the dimensions aren’t filtered. The delete and the insert are two Iceberg commits: a reader in between sees that day missing for about a second. If that matters, build in a branch and publish (Part 20’s Write-Audit-Publish).

Hands-On

The code is in the every-record-counts folder of the companion-workspaces repository (tag every-record-counts/part-18): models in dbt/larkspur/, the lab’s pitfall variants in dbt/larkspur/models/lab18/, the driver in scripts/part18_dbt.py. Core and Trino only (make up-trino), on top of Part 17’s production build. The commands, in step order:

uv run python scripts/duck.py --local sql/part18/lateness.sql
uv run python -m scripts.part18_dbt setup
uv run python -m scripts.part18_dbt prod-state
uv run python -m scripts.part18_dbt replay
uv run python -m scripts.part18_dbt full
uv run python -m scripts.part18_dbt compare
uv run python -m scripts.part18_dbt late
uv run python -m scripts.part18_dbt cost
uv run python -m scripts.part18_dbt ci          # make dbt-ci, in an emptied dev namespace
uv run python -m scripts.part18_dbt contract
uv run python -m scripts.part18_dbt deploy
uv run python -m scripts.part18_dbt batch-sql
uv run python -m scripts.part18_dbt explain

Step 1: a sandbox with its own principal

setup creates the dev namespace as the platform team (Part 9’s rule) and a developer principal, reached through a new Trino catalog file, lakehouse_dev. The profile gains a target (connection keys trimmed):

    dev:                             # Part 18: a developer's or CI's builds, all in one namespace, as another principal
      type: trino
      database: lakehouse_dev        # the `developer` principal: may create and drop tables only in `dev`
      schema: dev
      timezone: UTC
      threads: 2

namespace dev; principal developer → principal role developers → catalog role dev_builder:
  catalog           NAMESPACE_LIST
  namespace dev     NAMESPACE_READ_PROPERTIES, TABLE_CREATE, TABLE_DROP, TABLE_LIST, TABLE_READ_DATA, TABLE_READ_PROPERTIES, TABLE_WRITE_DATA, TABLE_WRITE_PROPERTIES, VIEW_CREATE, VIEW_DROP, VIEW_LIST, VIEW_READ_PROPERTIES, VIEW_WRITE_PROPERTIES
  namespace raw     NAMESPACE_READ_PROPERTIES, TABLE_LIST, TABLE_READ_DATA, TABLE_READ_PROPERTIES
$ DROP TABLE lakehouse_dev.gold.dim_date
Failed to load table 'dim_date' ← Forbidden: Principal 'developer' with activated PrincipalRoles '[developers]' and activated grants via '[developers, dev_builder]' is not authorized for op LOAD_TABLE_WITH_READ_DELEGATION

generate_schema_name returns the configured namespace only when target.name == 'prod', and dev otherwise. Polaris contains only what goes through lakehouse_dev. The dev profile logs in to Trino as the same user as prod, deferred reads go through the lakehouse_pipeline catalog, and the lab’s Trino has no access control until Part 23: a dev run that names lakehouse_pipeline can still write to or drop production. prod-state writes production’s manifest to state/prod/ by parsing the Part 17 tag, the code production runs (65 nodes plus 2 unit tests).

Step 2: the incremental sales fact

models/marts/fct_sales_line.sql keeps Part 17’s SELECT, now in a macro its lab variants share, and adds uploaded_at, so event time and ingestion time sit side by side:

{{ config(
    materialized='incremental',
    incremental_strategy='microbatch',
    event_time='event_ts',
    batch_size='day',
    lookback=2,
    begin='2026-05-31',
    on_schema_change='fail'
) }}
{{ sales_line_fact() }}

fct_basket gets the same config (a basket’s lines share its event_ts). int_sales_lines becomes a view with event_time: event_ts in its YAML. Staging needed one rewrite: Part 17 ranked the keys of every upload to drop re-sent baskets, so every batch reread all of them. Now only baskets uploaded more than once are ranked, keyed by (basket_id, event_ts) (a re-send repeats the payload), which lets the batch filter reach that scan too. The pitfall, models/lab18/fct_sales_line_naive.sql, appends after Rohan’s mark:

{{ sales_line_fact() }}
{% if is_incremental() %}
where l.event_ts > (select max(event_ts) from {{ this }})
{% endif %}

A third variant is the fact with lookback=1, dbt’s default. fct_stock_daily, Part 17’s other 11.7-million-row table, stays a table: its snapshots never arrive late, and an 18-second rebuild isn’t yet worth a strategy.

Step 3: two weeks of uploads, replayed

replay re-lands raw.pos_uploads with the uploads up to 20 September, builds the variants in dev, then lands one upload day at a time and runs them at 00:30 UTC the next morning. It runs dbt in-process and monkeypatches a private dbt internal, dbt.task.run.get_invocation_started_at, to that time (lab only, not a public API; may break on upgrade), so microbatch picks its batches from the clock and lookback, as a scheduled run would; a lab variable hides online orders placed after it. A run on today’s real clock picked 26, 27 and 28 October: two back. Output (eleven daily lines trimmed):

first build, uploads to 20 Sep (2,077,951 events), run 21 Sep 00:30: microbatch 127 s in 114 batches, naive 14 s
upload 21 Sep: 17,331 events  run 22 Sep 00:30  naive  1.1s  lookback1  1.8s  lookback2  2.9s
upload 30 Sep: 16,917 events  run 01 Oct 00:30  naive  1.0s  lookback1  2.0s  lookback2  2.9s
upload 01 Oct: 17,880 events  run 02 Oct 00:30  naive  0.9s  lookback1  1.8s  lookback2  3.2s
upload 05 Oct:    181 events  run 06 Oct 00:30  naive  0.9s  lookback1  1.4s  lookback2  2.8s

The lookback=1 table starts as a copy of the lookback=2 first build.

Step 4: row for row, to the penny

full builds the same SELECT as a plain table from everything landed; compare hashes every row of each variant and of that rebuild and counts rows on one side only:

variant                          rows  missing  extra  missing money (minor units, by currency)
fct_sales_line_naive        6,651,592    7,146      0  GBP 6,477,844, INR 236,631,944, SGD 4,230,971
fct_sales_line_lookback1    6,652,559    6,179      0  GBP 6,404,476, INR 231,362,635, SGD 1,299,953
fct_sales_line              6,658,738        0      0  none
fct_sales_line_full         6,658,738        0      0  none

Rohan’s mark lost 7,146 lines in 15 days, £64,778.44 of them in the UK, and nothing failed. The default lookback lost almost as many. lookback=2 equals the rebuild exactly. late follows one basket from store GB-0114, rung up at 12:06 BST on 30 September (quarter end) and uploaded 16 hours later:

fct_sales_line_naive       not there
fct_sales_line_lookback1   not there
fct_sales_line             date_key 20260930: 5 lines, 23,053 minor, rung up 2026-09-30 11:06:29.979000 UTC, uploaded 2026-10-01 03:06:25.565000 UTC
fct_sales_line_full        date_key 20260930: 5 lines, 23,053 minor, rung up 2026-09-30 11:06:29.979000 UTC, uploaded 2026-10-01 03:06:25.565000 UTC

Step 5: what a run costs

cost takes the median of the 15 daily runs; CPU and bytes come from Trino’s own statistics for each model’s statements:

build                               median s   CPU s  read MiB  batches
fct_sales_line_naive (daily run)         1.0     0.2      23.0  None
fct_sales_line_lookback1 (daily run)       1.8     0.2      11.2  2
fct_sales_line (daily run)               3.1     0.4      21.3  3
fct_sales_line_full (rebuild)           16.8    14.6     198.6
fct_sales_line: 156 data files, 29 position-delete files, 185 snapshots
fct_sales_line_full: 3 data files, 0 position-delete files, 1 snapshots

A daily run reads a tenth of a rebuild’s bytes and under a thirtieth of its CPU, and the gap widens with history; at lab size most of its 3.1 s is dbt’s fixed cost per batch. The bill comes elsewhere: compaction (Tips) and rebuilds. deploy runs dbt build --target prod --select state:modified+ --state state/prod --full-refresh --event-time-start 2026-05-31 --event-time-end 2026-10-06: 37 nodes in 4.1 minutes, 137 s of it for fct_sales_line in 128 daily batches (the pinned range skips 24 empty days); as a table, the same history was one 17-second statement. In production, a full refresh bounded by an end date rebuilds only up to it: the days after are gone. Production then reproduced Part 5’s UK Q3 net revenue, £29,790,674.06, exactly (Part 17’s check).

Step 6: slim CI

make dbt-ci is the job a pull request would run:

dbt-ci:  ## slim CI: dbt build of state:modified+ in dev, deferring to production (Part 18)
  $(DBT) build --target dev --select state:modified+ --state state/prod --defer --favor-state \
    --event-time-start 2026-10-02 --event-time-end 2026-10-05

The event-time range keeps microbatch models to three days in CI instead of 128, and Part 17’s no-line-lost test now compares only the event times the fact holds. Against Part 17’s manifest, in an emptied dev, the build took 25.5 s (output trimmed):

state:modified+ selects 37 of 69 nodes: {'model': 15, 'test': 21, 'unit_test': 1}
  models: dim_product, dim_sale_profile, fct_basket, fct_po_line, fct_refund, fct_sales_line, fct_stock_adjustment, fct_stock_daily, int_refunds, int_sales_lines, int_unresolved_products, rpt_daily_net_revenue, stg_online_sale_lines, stg_pos_sale_lines
20:24:58  [WARNING]: The constraint type primary_key is not supported by trino,
20:24:59  Done. PASS=37 WARN=0 ERROR=0 SKIP=0 NO-OP=0 REUSED=0 TOTAL=37

Slim is relative: int_sales_lines feeds the product dimension’s placeholders (Part 4), so a change there reaches most facts. After the deploy, a one-line change to the revenue report’s version 2 selected two nodes: the model and its grain test.

Step 7: a contract and a version

The revenue report becomes version 2 (currency renamed report_currency, is_provisional added for the last three days with sales); version 1 keeps the old table name. From models/marts/_marts.yml (descriptions and the money columns trimmed):

  - name: rpt_daily_net_revenue
    latest_version: 2
    config:
      contract: {enforced: true}
    constraints:
      - type: primary_key          # not supported on Trino: dbt warns and leaves it out; the grain test is the guard
        columns: [country, business_date]
    columns:
      - name: country
        data_type: varchar
        constraints: [{type: not_null}]   # the one constraint dbt-trino enforces (a NOT NULL column)
    versions:
      - v: 1
        deprecation_date: 2026-12-31
        config:
          alias: rpt_daily_net_revenue   # version 1 keeps the table name the dashboard reads
      - v: 2                             # `currency` renamed `report_currency`; `is_provisional` added

contract renames a column in version 2’s SQL without touching the YAML, then tests the built table directly (output trimmed):

a contracted build's commits: append +0 rows (total 0); append +378 rows (total 378)
  This model has an enforced contract that failed.
  | currency_code   | VARCHAR         |               | missing in contract   |
  | report_currency |                 | VARCHAR       | missing in definition |
INSERT a NULL country: NULL value not allowed for NOT NULL column: country
INSERT a second GB row for 3 Oct: accepted
20:25:37  2 of 2 FAIL 1 dbt_utils_unique_combination_of_columns_rpt_daily_net_revenue_v2_country__business_date  [FAIL 1 in 0.06s]

dbt stopped the rename before running anything; Trino refused the NULL; the duplicate key went in, and only the grain test saw it. Containers peaked at 4.5 GiB over the sequence, Trino at its 3 GB limit and Polaris at 0.96 of its 1 GB, so the lab now gives Polaris 1.5 GB with an explicit 1 GB heap; make ci then peaked at 4.8 GiB, Polaris at 1.1 GiB (MacBook Pro M3 Pro, 9.8 GB Docker VM). tests/test_part18.py replays it all.

The Landscape

As of 29 October 2026, from each vendor’s docs:

Tips & Gotchas

Gotcha — incremental models don’t see deletes. append and merge only add and update; a row whose source vanished stays. Part 16’s cancelled orders disappear from raw.orders, so a merged order fact would keep them. Microbatch absorbs a delete only while its day is inside the lookback; older ones need a backfill of that range.

Gotcha — UTC batches, local business days. Microbatch batches are UTC days and Larkspur’s business dates are local. Line and basket facts are fine: a batch replaces whole rows. An aggregate by business date isn’t: replacing the UTC day 30 September doesn’t replace Singapore’s. rpt_daily_net_revenue stays a table, or would be delete+insert keyed by business date.

Gotcha — a contract on Trino costs the atomic swap. Part 17’s on_table_exists: replace builds a table in one CREATE OR REPLACE TABLE AS commit. With a contract, dbt-trino creates the table empty, then inserts: the two commits in step 7, with an empty table readable in between.

Tip — compaction comes with incremental. Each run deletes and rewrites three days; fifteen runs left 156 data files and 29 delete files where a rebuild leaves 3. Schedule Part 10’s compaction and snapshot expiry next to the runs (Part 19 does).

Troubleshooting

Design Drill

The question: “An agritech company runs soil-moisture probes and tractor telematics on 40,000 farms. Devices buffer readings and upload when they get signal, some after a week in a field. A nightly incremental dbt model builds a daily water-and-yield fact per field, and agronomists see dry days that weren’t dry. Your incremental model silently drops late events: find and fix it.”

Clarify first

• What does the is_incremental() filter compare: reading time, upload time, or a fixed window?
• What is the measured lateness (upload minus reading time): p99 and maximum, per device type?
• Do devices send corrections or deletions (a recalibrated probe re-sending a week)?
• When is a day final for agronomists, and for anything billed on it?
• Does the raw zone keep every upload, by arrival date, long enough to rebuild?

The shape of a strong answer

• Prove it first: per reading date, raw readings against fact rows, bucketed by lateness. Losses only in late buckets point at an event-time high-water mark.
• Raw stays immutable, partitioned by arrival, with ingested_at; the fix and the backfill both depend on it.
• Reading-grain fact: a high-water mark on ingestion time, one per source (never one mark across clocks), loaded by merge on (device_id, reading_ts): a week-late reading lands, a re-sent one doesn’t double.
• Daily aggregates: rebuild the field-days that received rows since the last run (delete+insert, found from ingested_at). A fixed lookback would redo eight days every night for a few week-late devices.
• Backfill the damage, then compare a full rebuild with the incremental table row by row for a sample month.
• Guardrails: a per-day completeness test (raw count = fact count, Part 21’s control totals), an alert when a reading arrives later than the documented window, an is_provisional flag until a day closes.

Trade-offs to say out loud

• An ingestion-time mark never skips a late reading, but merge never deletes: corrections need their own path.
• A fixed lookback is simple and absorbs corrections, costs the same every night, and still loses anything later.
• Reopening closed days keeps data right and changes numbers people used: publish restatements.

Common weak answers

• “Set lookback to 30 days”: unmeasured, expensive, still a bound.
• Fixing the filter without backfilling the days already wrong.
• Trusting device clocks to order or deduplicate: drifted clocks put readings in the wrong day.

Further Reading

1. dbt Labs, “About microbatch incremental models” (event_time, batch_size, lookback, begin, backfills). https://docs.getdbt.com/docs/build/incremental-microbatch
2. dbt Labs, “About incremental strategy” (strategies by adapter). https://docs.getdbt.com/docs/build/incremental-strategy
3. dbt Labs, “Defer” (--defer, --favor-state). https://docs.getdbt.com/reference/node-selection/defer
4. dbt Labs, “contract” and “Model versions”. https://docs.getdbt.com/reference/resource-configs/contract · https://docs.getdbt.com/docs/mesh/govern/model-versions
5. Tyler Akidau et al., “The Dataflow Model”, VLDB 2015: event time, processing time, and why completeness is a choice.

Story Closing

Rohan read the naive line aloud: 7,146 lines. “My consumers never had this problem because the offset was the arrival order. I put the mark on the wrong clock.”

Of the five promises from Part 2, this part strengthens timeliness: the sales facts refresh in seconds a night instead of rebuilding history, without giving up completeness. The late basket from 30 September sits on 30 September, and is_provisional tells Farah which days can still move.

“Now who runs it at 00:30?” Ingrid said. “Your replay set dbt’s clock by hand. Production needs a scheduler that knows the day it’s running for, retries, backfills a month when you ask, and runs compaction after.”

In Part 19, Airflow 3 takes over the clock: ingestion, dbt, quality checks and publish, as one orchestrated pipeline.

This is Part 18 of “Every Record Counts: Data Engineering for Backend Engineers” — Season 2: Pipelines in Production.