ApacheAirflow

Series Posts

Project Every Record Counts: Data Engineering for Backend Engineers

Part 19: "Who Runs What, When" — Orchestration with Airflow 3

Rohan wants a cron entry and a Spring service with a retry loop. Ingrid wants a record of every day the pipeline ran, a way to re-run any of them, and no business logic in the scheduler. DAGs, logical dates and data intervals, backfills, assets and asset partitioning, retries, pools, sensors and deferrable operators, Deadline Alerts and DAG versioning, with Dagster and Prefect compared. On Airflow 3.3, a month-long backfill republishes the daily revenue report and UK Q3 stays at £29,790,674.06.

Project Every Record Counts: Data Engineering for Backend Engineers

Part 22: "Know Before They Do" — Observability, Lineage and SLAs

Part 21's gate held the revenue report back, correctly, and the first to notice was Farah. Rohan makes the data page the team instead: freshness, completeness, volume and schema as SLOs with an error budget counted in days, OpenLineage from Airflow, dbt and Spark loaded into OpenMetadata to trace the sales and stock reports to their sources, OpenTelemetry for the runs, and a "the dashboard is wrong" runbook.