"Who Runs What, When" — Orchestration with Airflow 3

Skim Card

One thing to remember: an orchestrator owns when and in what order, and keeps a record of every run per day; what each step does stays in commands you can run without it.

• Name runs by their data: CronDataIntervalTimetable gives upload day D an interval and {{ ds }}; a bare cron string in Airflow 3 doesn’t.
• Make every task replace, never append: one upload_date partition, named microbatch days, the whole report, a moved tag. Retries and backfills then change nothing.
• Backfilling September ran 30 days in 18.5 minutes; UK Q3 stayed £29,790,674.06, and re-running 15 September left the fact and report hashes unchanged.
• Let checks gate publishing: a hand-edited closed-day FX rate failed closed_days_moved, and the published tag stayed on the last good snapshot.
• Schedule on data, not clock offsets: 30 publish events started 30 asset-triggered maintenance runs (Trino optimize, expire_snapshots, remove_orphan_files).
• Deadline Alerts (experimental) anchored on the logical date fire for every backfilled day: route the callback by run_type.

Use …	When …
Airflow 3	Many teams and workloads, backfills and per-day history matter, the widest operator ecosystem
Dagster	Teams think in tables: assets, lineage and freshness are the model
Prefect	Python-heavy, dynamic flows started by events
Event triggers, no DAG	A few steps, low latency, no backfills
cron	One job, no dependencies, re-running by hand is acceptable

Verified against: Apache Airflow 3.3.2 (Task SDK 1.3.2), dbt-core 1.12.6, dbt-trino 1.11.0, Trino 484, Apache Polaris 1.8.0, PyIceberg 0.12.0 · vendor docs · 29 October 2026

Jump to: Concepts · Hands-On · Landscape · Design Drill

Story Opening

Part 18 left the sales facts incremental, rebuilt a few days at a time, and run by a replay script that set dbt’s clock by hand.

“I’ll put it in a Spring service,” Rohan said. “@Scheduled at midnight, a retry loop, and a status table. I’ve written that five times.”

“And the sixth time you’ll write backfills, a UI, alerting, per-day history and a queue so two jobs don’t fight over one table,” Ingrid said. “Then someone puts the revenue logic in it because it’s handy. Use an orchestrator, and keep it stupid: it decides when, and in what order. What the steps do stays where you can run it without it.”

The Quick Map

Concepts, with use cases

Orchestrator, DAG, task

An orchestrator runs a graph of steps on a schedule or on events, records each run’s state, retries failures, and lets people re-run, pause and backfill. In Airflow the graph is a DAG (directed acyclic graph) of tasks; an operator is a reusable task type (BashOperator), and TaskFlow turns a Python function into a task with @task. Since Airflow 3, DAG code imports from the Task SDK, airflow.sdk.

What breaks without it. Cron knows times, not dependencies or history: when Tuesday’s dbt step fails, nothing stops the publish, and nobody can say which days ran. Larkspur: one DAG per data product; this part’s runs ingestion, dbt, checks and publish for the daily revenue report. Elsewhere: a hospital’s claims export, where billing must never run on a day whose coding step failed.

The rule that keeps an orchestrator replaceable: no business logic in the DAG. Larkspur’s DAGs call only scripts/part19_pipeline.py commands; which days to rebuild, which dbt models and which tables to maintain are decided there, so swapping Airflow for Dagster wouldn’t change a command.

Logical date and data interval

A run’s logical date names the data it processes; its data interval is the slice of time it covers. A timetable generates them. CronDataIntervalTimetable("0 0 * * *") makes the run for upload day D cover [D 00:00, D+1 00:00) UTC, start when D has closed, and render {{ ds }} as D. A bare cron string in Airflow 3 becomes a CronTriggerTimetable instead: the run is named after its trigger time and has no interval unless you pass interval= ([scheduler] create_cron_data_intervals defaults to False). catchup defaults to False, so unpausing a DAG creates the latest run, not one per day since start_date.

What breaks without it. A job that computes “yesterday” from the wall clock processes the wrong day whenever it runs late, and can’t be re-run for last Tuesday. Larkspur: the run for 15 September rebuilds UTC days 14 and 15 (Part 18’s measured window) whether it starts on the 16th or a month later. Elsewhere: an energy retailer’s half-hourly settlement, re-run for any interval the grid operator corrects.

Idempotent tasks and backfills

A backfill creates runs for past logical dates. In Airflow 3 the scheduler manages them: airflow backfill create --dag-id … --from-date … --to-date … (both inclusive) with --reprocess-behavior none|failed|completed (default none: skip days that already have a run) and its own --max-active-runs. Part 2’s rule carries the weight: every scheduler is at-least-once, so every task must be safe to run twice.

What breaks without it. An appending ingestion task, retried after a timeout, lands the day twice. Larkspur: ingest replaces one upload_date partition (one commit, two snapshots: delete, then append); the microbatch models replace their days; the report is a table rebuilt whole; publish moves a tag. Elsewhere: a ride-hailing driver-payout job re-run after a fare correction must replace that day’s payouts, never add to them.

Assets and data-aware scheduling

An asset names data (a table, a file) by URI. A task that declares it in outlets emits an asset event when it succeeds, optionally with extra metadata; a DAG with schedule=[asset] runs when the asset is updated, and events that arrive before that run starts are consumed together. Asset partitioning (since 3.2, with no experimental label in the docs; docs only, not run in the lab) attaches a partition key to events: CronPartitionTimetable on the producer, PartitionedAssetTimetable on the consumer, with mappers to translate keys (StartOfDayMapper; in 3.3 RollupMapper waits for every day of a month, FanOutMapper goes the other way).

What breaks without it. A job scheduled “two hours after” its upstream reads half-built tables on slow days and waits needlessly on fast ones. Larkspur: table maintenance runs when the revenue report is published, not at a guessed time. A month-end pack that needs every day of September is the RollupMapper case. Elsewhere: an insurer’s pricing model retrains when a week of claims is complete, whoever loaded it.

Retries, pools, sensors and deferral

Retries (retries, retry_delay) absorb transient failures; Airflow 3.3 adds retry policies (retry_policy= with RetryRules per exception type; from the source, not run in the lab) to fail fast on errors no retry will fix. A pool caps how many tasks share a resource at once. A sensor waits for a condition: in reschedule mode it frees its worker slot between pokes; a deferrable operator hands the wait to the triggerer, an asyncio service, and holds no slot at all. The docs suggest reschedule for a file appearing, deferral for external systems.

What breaks without them. Two writers commit to one Iceberg table and one fails (Part 7); a poke-mode sensor holds a worker for hours. Larkspur: pool lakehouse (one slot) serialises every lake write through the lab’s single Trino node; wait_for_uploads waits in reschedule mode; failed checks get no retries. Elsewhere: a bank’s batch window, where 300 jobs share 20 database connections through a pool.

Deadline Alerts, versions and bundles

Deadline Alerts (experimental) replace the SLAs Airflow 3 removed: a deadline is a reference time (the run’s logical date, its queue time, a fixed time or average runtime) plus an interval, and a callback runs when it passes. DAG versioning records each parsed version of a DAG; runs keep the version they started with. Whether re-runs use old code depends on the DAG bundle: a Git bundle is versioned per commit; the local folder this lab uses is not, so every run executes the latest files.

What breaks without them. A late pipeline is found by its readers, and a fix deployed mid-backfill leaves nobody able to say which days ran which code. Larkspur: Farah’s numbers are due by 06:00 UTC after each upload day: logical date + 30 hours. Elsewhere: a payroll provider’s tax filing, where “late” is a regulatory fact, not a dashboard colour.

The alternatives

• Dagster models pipelines as assets (formerly “software-defined assets”): you declare tables and their dependencies, and runs materialise assets; Declarative Automation (AutomationCondition) decides when. It wins where lineage and freshness per table matter more than task graphs.
• Prefect runs plain Python flows with dynamic structure, started by automations and deployment triggers on events: Python-heavy, irregular workloads.
• Event-driven triggers without a central DAG win for low latency and few steps, and lose backfills and per-day history unless you build them.
• The Prefect–Dagster deal (facts only): on 13 July 2026 Prefect announced it is acquiring Dagster Labs. Dagster keeps its name (“Dagster stays Dagster”) and open-source project, Dagster and Dagster+ continue, and Dagster’s FAQ says there are “no plans to merge Dagster into Prefect” and that it expects the transaction to close “in the coming weeks”, while a banner on Dagster’s site reads “Dagster is now part of Prefect”. We found no closing announcement as of 29 October 2026.

Larkspur stays on Airflow, the series’ choice since Part 0: the widest ecosystem, and assets now cover much of what made Dagster distinctive. That is judgement, not a benchmark.

Under the Hood

One run, step by step

The dag-processor re-parses DAG files all the time: anything at a file’s top level (a query, an API call) runs on every parse. The task process “never reaches the metadata database directly” (the docs): it reports through the API server’s Execution API with a short-lived token, so every component needs the same JWT secret. State is per run: a run is keyed by (DAG, logical date), which is unique, and each task instance carries a try number.

Backfills are runs like any other (run_type backfill), started at the backfill’s own max_active_runs.

What the maintenance procedures change

Part 10 ran compaction, snapshot expiry and orphan removal as Spark procedures; Trino 484 has them as ALTER TABLE … EXECUTE optimize, expire_snapshots and remove_orphan_files, so the lab, where Spark and Trino don’t fit in memory together, needs no Spark for maintenance. optimize also folds delete files into the rewritten data files (default threshold 100MB). The other two refuse a retention below iceberg.expire-snapshots.min-retention and iceberg.remove-orphan-files.min-retention (catalog properties, 7 days by default): a younger file may belong to a commit still in flight, and a younger snapshot to someone’s time travel. A session can lower them per catalog: lakehouse_pipeline.expire_snapshots_min_retention and lakehouse_pipeline.remove_orphan_files_min_retention here. The refusal message names iceberg.expire_snapshots_min_retention; that iceberg. stands for your catalog’s name. The lab’s 1-hour cutoff is safe only because the lakehouse pool serialises the DAGs’ writes and nothing else writes meanwhile (Part 10: the cutoff must exceed your longest write).

Hands-On

The code is in the every-record-counts folder of the companion-workspaces repository (tag every-record-counts/part-19): DAGs in airflow/dags/, the steps in scripts/part19_pipeline.py and sql/part19/checks.sql, the image in airflow/Dockerfile. Core, Trino and Airflow (make up-orchestrate; Spark, Kafka and ClickHouse stopped), on top of Part 18’s production build. The commands, in step order:

make up-orchestrate
uv run python -m scripts.part19_pipeline setup
make airflow CMD="variables set lake_retention 1h"   # lab only; production keeps 7d
make airflow CMD="dags unpause larkspur_lake_maintenance"
make airflow CMD="dags unpause larkspur_daily_revenue"
uv run python -m scripts.part19_pipeline fingerprint --day 2026-09-15
make airflow CMD="backfill create --dag-id larkspur_daily_revenue --from-date 2026-09-01 --to-date 2026-09-30 --max-active-runs 1"
uv run python -m scripts.part19_pipeline wait --dag-id larkspur_daily_revenue,larkspur_lake_maintenance
uv run python -m scripts.part19_pipeline runs --dag-id larkspur_daily_revenue
uv run python -m scripts.part19_pipeline runs --dag-id larkspur_lake_maintenance
uv run python -m scripts.part19_pipeline q3
make airflow CMD="tasks clear larkspur_daily_revenue -s 2026-09-15 -e 2026-09-15 -y"
uv run python -m scripts.part19_pipeline fingerprint --day 2026-09-15
uv run python -m scripts.part19_pipeline fx-correction --day 2026-09-01
make airflow CMD="tasks clear larkspur_daily_revenue -s 2026-09-20 -e 2026-09-20 -y"
uv run python -m scripts.part17_dbt dbt run --select dim_fx_rate
make airflow CMD="tasks clear larkspur_daily_revenue -s 2026-09-20 -e 2026-09-20 -y"

Step 1: Airflow in 1.4 GB

The official docker-compose.yaml for 3.3.2 runs CeleryExecutor and says it is “for local development”. The orchestrate profile is our trim of it:

setup re-lands raw.pos_uploads partitioned by upload_date (2,338,812 events, 128 files): Part 17 landed it unpartitioned, so replacing one upload day would have rewritten files holding others. Over the whole sequence below, Airflow’s three services peaked at 0.75 GiB (scheduler, with its task processes), 0.35 (dag-processor) and 0.27 (api-server); all containers together at 5.59 GiB, Trino at its 3 GB limit (M3 Pro, 9.8 GB OrbStack VM).

Step 2: the DAG

airflow/dags/larkspur_daily_revenue.py, trimmed to its decisions:

with DAG(
    dag_id="larkspur_daily_revenue",
    schedule=CronDataIntervalTimetable("0 0 * * *", timezone="UTC"),   # run for D starts at D+1 00:00
    start_date=pendulum.datetime(2026, 9, 1, tz="UTC"),
    catchup=False,
    max_active_runs=1,
    default_args={"retries": 2, "retry_delay": timedelta(minutes=1)},
    deadline=DeadlineAlert(                                           # experimental in 3.3
        reference=DeadlineReference.DAGRUN_LOGICAL_DATE,
        interval=timedelta(hours=30),                                 # 06:00 UTC on D+1
        callback=SyncCallback("larkspur_alerts.revenue_late", kwargs={"report": REPORT.name}),
    ),
) as dag:
    wait_for_uploads = BashSensor(task_id="wait_for_uploads", mode="reschedule",
        bash_command="ls /opt/lab/data/raw/pos/events/upload_date={{ ds }}/*.jsonl.gz > /dev/null 2>&1",
        poke_interval=timedelta(minutes=5), timeout=timedelta(hours=6), retries=0)
    ingest = BashOperator(task_id="ingest_pos_uploads", pool="lakehouse", outlets=[RAW_POS],
        bash_command=f"{LAB} ingest --day {{{{ ds }}}}")
    dbt_build = BashOperator(task_id="dbt_build", pool="lakehouse", outlets=[SALES_FACTS],
        bash_command=f"{LAB} build --day {{{{ ds }}}}")
    quality_checks = BashOperator(task_id="quality_checks", retries=0,
        bash_command=f"{LAB} check --day {{{{ ds }}}}")

    @task(pool="lakehouse", outlets=[REPORT])
    def publish(ds=None):
        out = subprocess.run([... "scripts.part19_pipeline", "publish", "--day", ds], ...).stdout
        info = json.loads(out.strip().splitlines()[-1])
        yield Metadata(REPORT, {"upload_date": ds, "snapshot_id": str(info["snapshot_id"])})

    wait_for_uploads >> ingest >> dbt_build >> quality_checks >> publish()

LAB is cd /opt/lab && /opt/lab-venv/bin/python -m scripts.part19_pipeline; the excerpt joins lines the file keeps apart. build --day D runs dbt build of fct_sales_line, fct_basket, fct_refund and rpt_daily_net_revenue with --event-time-start D−1 --event-time-end D+1: the microbatch days one upload day can touch (Part 18’s measured lateness), decided in the script, not the DAG. check runs three queries from sql/part19/checks.sql that look across runs and tables, which dbt’s tests (run inside dbt build) don’t: closed days unchanged (business dates before D−1 equal the published report, since a run for D rebuilds UTC days D−1 and D, local days D−1 to D+1 east of UTC), every country reported for D, and the day landed. publish moves an Iceberg tag, published, to the report snapshot that passed: main changes with every build, the tag only after checks. Readers must opt in (FOR VERSION AS OF 'published'); Part 20’s Write-Audit-Publish keeps unchecked data off main instead, with Spark, since Trino 484 can’t write to branches. larkspur_lake_maintenance is scheduled on REPORT; its one task runs maintain, which compacts, expires and removes orphans on the three tables the daily run churns, in that order, in the same pool.

Step 3: the scheduler’s own run

Unpausing created one run, for the latest closed interval:

scheduled__2026-10-29T00:00:00+00:00 scheduled 2026-10-28T00:00:00Z 2026-10-28T00:00:00Z 2026-10-29T00:00:00Z 2026-10-29T00:00:00Z
wait_for_uploads up_for_reschedule 1
ingest_pos_uploads None 0

Run ID, type, logical date, interval start and end, run after; then task, state, try (three tasks trimmed). The lab’s raw zone ends on 5 October, so the sensor pokes every five minutes, holds no slot in between, and is set to give up after six hours (timeout; not waited out in the lab); nothing downstream runs.

Step 4: a month, backfilled

backfill        2026-09-01  success  5 tasks   25.1 s  max try 1
backfill        2026-09-02  success  5 tasks   22.6 s  max try 1
…                                                         (27 runs trimmed)
backfill        2026-09-30  success  5 tasks   23.5 s  max try 1
total: 30 backfill success, task time median 23.9 s, sum 715 s

Thirty runs in 18 min 27 s, every task on its first try, while the scheduled run kept waiting: the backfill’s --max-active-runs is its own. The 30 publish events started 30 asset_triggered maintenance runs (median 10.4 s), each done before the next publish; an earlier, slower pass needed only 21 runs for 29 events. The first run’s line for the sales fact:

{"table": "gold.fct_sales_line", "before": {"data_files": 3, "delete_files": 2, "snapshots": 140}, "optimize": [["rewritten_data_files_count", 3], ["removed_delete_files_count", 2], ["added_data_files_count", 1]], "expire_snapshots": [], "remove_orphan_files": [["processed_manifests_count", 182], ["active_files_count", 700], ["scanned_files_count", 742], ["deleted_files_count", 0], ["deleted_bytes", 0]], "after": {"data_files": 1, "delete_files": 0, "snapshots": 141}, "seconds": 5.0}

Expiry removed nothing here: make ci had rebuilt the facts within the hour, and the fact gained five snapshots per run (286 by the end). The earlier pass ran on older history: one run took gold.fct_sales_line from 261 snapshots to 180, a manual run an hour later from 180 to 24 (gold.fct_basket: 307 to 25), and each time remove_orphan_files then deleted files (82, 89 and 215; 123, 139 and 230 MB). Its first run had also removed 35 orphans from gold.fct_basket. Observed; we didn’t trace which snapshots had referenced them. Expire first, then remove orphans, as maintain does.

The published report, read as the analyst through the tag:

GB     29,790,674.06  manifest exact
IN     20,642,180.07
SG     15,242,767.64

The same as before the backfill, to the penny.

Step 5: re-run a day

Clearing 15 September’s run re-ran all five tasks (max try 2). The fingerprint hashes every fact row for UTC 14–15 September and every report row:

{"raw_events": 16458, "fact_lines": 95574, "fact_minor": 4981231836, "fact_hash": "6B99EBF094911BF6", "report_rows": 378, "report_net_minor": 9115153114, "report_hash": "112ABD3B67B27D2C"}
{"raw_events": 16458, "fact_lines": 95574, "fact_minor": 4981231836, "fact_hash": "6B99EBF094911BF6", "report_rows": 378, "report_net_minor": 9115153114, "report_hash": "112ABD3B67B27D2C"}

Before the backfill, then after the re-run; the line after the backfill was the same too. The re-run’s ingest was one commit with two snapshots, a delete of 1 file (16,458 records) and an append of 1 file (16,458 records): 2,338,812 events before and after.

Step 6: a restatement the checks stop

fx-correction raises 1 September’s INR→GBP rate by 1% in gold.dim_fx_rate, as a well-meant hand fix would. Re-running 20 September rebuilt the report from it:

closed_days_moved    FAIL 1 rows: [['IN', '2026-09-01', 18998797, 19188889]]
missing_countries    pass
day_not_landed       pass

quality_checks       failed
publish              upstream_failed

The main branch now says IN earned £191,888.89 on 1 September; the published tag still points at snapshot 8996782513477743189, and the analyst’s Q3 query still returns IN £20,642,180.07. Rebuilding dim_fx_rate with dbt and clearing 20 September again: all five tasks succeeded and the tag moved. Restating a closed day on purpose is Part 21’s job, with a record of what changed and why.

The Landscape

From vendor docs, 29 October 2026; MWAA Serverless’s and Fabric’s Airflow job’s GA status not verified.

Tips & Gotchas

Gotcha — a manual run’s interval is not its logical date’s. Triggered on 29 October with --logical-date 2026-08-20, the run got the data interval 28–29 October, from the trigger time. Template on {{ ds }} or logical_date, which every run type sets; data_interval_* is right only for scheduled and backfill runs.

Gotcha — a deadline on the logical date has passed for every old day. In a first run of this sequence, the scheduled run created late on 28 October missed its 06:00 deadline the moment it was created, and the September backfill logged 30 more DEADLINE MISSED lines within 31 seconds. The callback receives the run (context["dag_run"], as the REST API shows it), so revenue_late now logs backfill runs as “not paging” and pages for the rest; the backfill above logged 30 lines like deadline passed for backfill run backfill__2026-09-02T00:00:00+00:00: not paging. Run IDs are named by run_after, the end of the interval, so backfill__2026-09-02… is the run for 1 September.

Gotcha — a folder that exists isn’t a folder that’s complete. The lab’s sensor checks that the day’s files exist, which works only because the generator writes whole days. A real producer writes a marker last (_SUCCESS, a manifest with counts); sense the marker and check the counts in the task.

Tip — keep the job’s Python out of Airflow’s. Airflow installs against its own constraint pins; dbt, dlt and PyIceberg pin theirs. The lab image installs the lab’s uv.lock into /opt/lab-venv (3.47 GB image vs 2.39 GB) and tasks call that interpreter. At scale: a container per task.

Troubleshooting

Design Drill

The question: “A food-delivery marketplace runs 2,000 data pipelines owned by 40 teams: orders and payments, courier GPS, restaurant menus, promotions, ads, fraud and finance. Today they share one Airflow 2 installation that the platform team babysits; a bad deploy from one team stalls everyone’s runs, and finance’s 07:00 close depends on six other teams’ tables. Design the orchestration for 2,000 pipelines owned by 40 teams.”

Clarify first

• Which pipelines are tier 1 (finance close, courier pay, regulatory), and what are their deadlines?
• How do pipelines depend on other teams’ outputs today: clock offsets, sensors, or nothing?
• Where does the work run: warehouse SQL, Spark, containers? Who owns their dependencies?
• What isolation is required: blast radius only, or security boundaries between teams (fraud, payments)?
• How often do teams backfill, and who approves a backfill that rewrites published numbers?

The shape of a strong answer

• Keep the orchestrator thin. Tasks are containers or warehouse jobs; the scheduler image carries no team libraries, so one team’s upgrade can’t break another’s DAG.
• Isolate by blast radius. A few Airflow 3 deployments by domain (or Astro/MWAA environments), not one per team (40 to patch) and not one for all; Airflow 3.2’s multi-team mode (experimental; docs only, not run in the lab) gives logical isolation within one deployment but shares the scheduler and metadata database.
• Per-team Git DAG bundles (versioned), with CI that parses every DAG, rejects top-level I/O and enforces owners, retries and tiers.
• Cross-team dependencies are assets, named by table, not task IDs: finance’s close is scheduled on the six tables it reads, partitioned by day; a RollupMapper (3.3; docs only, not run in the lab) waits for every hourly partition.
• Pools per shared resource (warehouse, Spark cluster, partner APIs), sized from the real limit; priority weights for tier 1.
• Every task idempotent by partition, so retries and self-service backfills (backfill create with --max-active-runs) are safe; backfills that restate published days go through an approval.
• Deadlines on tier-1 runs routed to the owning team’s on-call (experimental in 3.3; keep an external freshness check too), and lineage (Part 22) to find who’s affected.
• A paved road: a YAML-to-DAG template for the common “extract, dbt, check, publish” shape, so 40 teams don’t each invent retries and alerting.

Trade-offs to say out loud

• Several deployments isolate failures but split the asset graph: a cross-deployment dependency needs an event bridge or a shared asset API.
• Dagster’s asset model fits table-centric teams and freshness better; Airflow wins on ecosystem and hiring. Mixing both is a cost, not a strategy.
• Event-driven triggers cut latency but lose the run history and backfills finance relies on.

Common weak answers

• “One bigger Airflow”: a bigger blast radius, and parse time grows with every team’s files.
• Cross-team dependencies by clock offsets or ExternalTaskSensor on another team’s task IDs, broken by a rename.
• Business logic in Python operators, every team’s libraries in the scheduler image.

Further Reading

1. Apache Airflow 3.3 documentation: “Architecture Overview”, “Timetables”, “Backfill”, “Assets” (including asset partitions), “Deadline Alerts”, “Dag bundles”. https://airflow.apache.org/docs/apache-airflow/3.3.2/
2. Apache Airflow, the official docker-compose.yaml for 3.3.2 (the file this lab trims). https://airflow.apache.org/docs/apache-airflow/3.3.2/docker-compose.yaml
3. Trino 484, Iceberg connector: table procedures (optimize, expire_snapshots, remove_orphan_files). https://trino.io/docs/484/connector/iceberg.html
4. Dagster documentation, “Assets” and “Declarative Automation”. https://dagster.io/docs/guides/build/assets
5. Prefect, “Prefect acquires Dagster Labs” (13 July 2026) and Dagster’s customer FAQ. https://www.prefect.io/prefect-acquires-dagster · https://dagster.io/dagster-customer-faq

Story Closing

“In the order service a retry was a risk I managed,” Rohan said. “Here re-running a day is the normal way to fix anything.”

Of the five promises from Part 2, this part strengthens timeliness: the report is built when its day closes, waits for late inputs without guessing a time, and a deadline says when it’s late. Uniqueness held through a month of backfilled days and two re-runs.

Farah’s question was shorter. “Your check stopped a rate change on the 1st. Who decides which rules a number has to pass?”

“The producer and the consumer, in writing,” Ingrid said. Ingrid’s condition from Part 14, that the maintenance jobs get a scheduler, was now met. In Part 20, the three queries in checks.sql grow into data contracts and Write-Audit-Publish on an Iceberg branch.

This is Part 19 of “Every Record Counts: Data Engineering for Backend Engineers” — Season 2: Pipelines in Production.