Project Every Record Counts: Data Engineering for Backend Engineers
Part 19: "Who Runs What, When" — Orchestration with Airflow 3
Rohan wants a cron entry and a Spring service with a retry loop. Ingrid wants a record of every day the pipeline ran, a way to re-run any of them, and no business logic in the scheduler. DAGs, logical dates and data intervals, backfills, assets and asset partitioning, retries, pools, sensors and deferrable operators, Deadline Alerts and DAG versioning, with Dagster and Prefect compared. On Airflow 3.3, a month-long backfill republishes the daily revenue report and UK Q3 stays at £29,790,674.06.