Part 22 Project Complete

"Know Before They Do" — Observability, Lineage and SLAs

Part 21's gate held the revenue report back, correctly, and the first to notice was Farah. Rohan makes the data page the team instead: freshness, completeness, volume and schema as SLOs with an error budget counted in days, OpenLineage from Airflow, dbt and Spark loaded into OpenMetadata to trace the sales and stock reports to their sources, OpenTelemetry for the runs, and a "the dashboard is wrong" runbook.

October 30, 2026 22 min read GitHub

Skim Card

One thing to remember: watch the promise the data makes to its readers, not just the jobs that make it; a green pipeline can still leave the report stale or wrong.

Four signals per data product: freshness, completeness, volume, schema. Lineage is the map you use when one fires.

A daily data SLO counts days: 27 of 28 good is a one-day error budget, and one missed morning spends it.

Measure freshness on the table: does the published snapshot cover yesterday? Record the 06:00 result once; a 09:30 fix doesn’t undo it.

Volume checks catch outages, not a lost till (−3.4% looked normal); reconciliation found the missing £66.45.

OpenLineage is the wire format; what you get depends on each integration. dbt-ol gave no column lineage for microbatch models.

One table, three names (Trino in Docker, Trino on a laptop, Spark’s S3 path): the backend must map them.

Buffer lineage events (a file or a queue) so a backend outage or a backend change loses nothing.

Use … When …
OpenLineage events into a catalog (OpenMetadata, DataHub, a cloud catalog) Several engines and teams; lineage must cross them
dbt’s own DAG and docs One dbt project is the whole story
SLO checks over your control tables The number has a deadline and an owner who gets paged
A vendor observability tool Hundreds of tables and no time to write monitors

Verified against: OpenLineage 1.54.0 (Python, dbt-ol, Spark), apache-airflow-providers-openlineage 2.20.3, Apache Airflow 3.3.2, OpenMetadata 2.0.7, Elasticsearch 9.3.0, OpenTelemetry Collector 0.164.0, Apache Spark 4.1.3 (with OpenLineage: not supported upstream, lab observation), dbt-core 1.12.6 + dbt-trino 1.11.0, Trino 484 · 29 October 2026

Jump to: Concepts · Hands-On · Landscape · Design Drill

Use …	When …
OpenLineage events into a catalog (OpenMetadata, DataHub, a cloud catalog)	Several engines and teams; lineage must cross them
dbt’s own DAG and docs	One dbt project is the whole story
SLO checks over your control tables	The number has a deadline and an owner who gets paged
A vendor observability tool	Hundreds of tables and no time to write monitors

Story Opening

At 08:05 on Monday 5 October, Farah wrote: “The UK dashboard still shows Saturday. Is that right?”

It was right, in a way. Part 21’s reconciliation gate had found two missing till baskets overnight and held the report back. Every task before it was green; the failure callback had written one line to a log nobody reads at night.

“I’ll add alerts to every task,” Rohan said. “Failures, durations, retries.”

“Those tell you a job broke,” Ingrid said. “Farah cares whether Sunday’s number is there by six and whether it’s right. Measure that, page on that, and keep a map from the number back to every job that touched it.”

The Quick Map

Backend instinct	Data engineering reality	Why it matters
Monitor the service: errors, latency, saturation	Monitor the data: freshness, completeness, volume, schema	Every task can succeed while the table is stale or short
Page on a failed job	Page on a broken promise to a reader; job failures are causes	The gate failing was correct; the late report was the incident
Tracing follows a request across services	Lineage follows a table across jobs and engines	”Where does this number come from?” crosses Airflow, dbt and Spark
SLO: 99.9% of requests in a month	SLO: good days in a window	A daily report has 28 chances a month, not millions

Concepts, with use cases

Data observability: four signals and a map

Data observability is knowing from the outside whether data is fit to use. The usual pillars are freshness (is the latest data there?), volume (did the expected amount arrive?), schema (did the shape change?), distribution (are the values plausible?) and lineage (what feeds what?). The first four are signals; lineage is where you look when one fires.

What breaks without it. Readers find problems first. Larkspur: the revenue report’s four signals at 06:00 London; distribution is left to Part 21’s controls, since a sum that must equal the ledger beats “looks normal”. Elsewhere: a ride-hailing company watches trips per city per hour; a city at zero is an outage before anyone files a ticket.

SLIs, SLOs and error budgets, for data

A service level indicator (SLI) is a measurement (was the report published by 06:00?), a service level objective (SLO) a target for it over a window (27 of the last 28 days), the error budget what the target lets fail (one day), and a service level agreement (SLA) the contract with consequences on top. A request-serving system spends its budget over millions of requests; a daily data product has one event per promise per day, so one bad morning is the month’s budget. Page on every breach; let the budget decide when reliability work beats new reports.

Define them per reader. Farah’s 06:00 is London’s; a UTC-day batch starting at 00:00 UTC can’t give Singapore its numbers by 06:00 local (22:00 UTC the evening before), so Singapore needs its own run or its own deadline.

What breaks without them. “The data is late sometimes” is never a priority, or every blip is. Elsewhere: an ad network promises advertisers yesterday’s spend by 09:00 for billing.

Lineage and OpenLineage

Lineage is the graph of datasets and the jobs that read and write them. OpenLineage is an open standard for reporting it as it happens: each run sends run events (START, RUNNING, COMPLETE, FAIL) naming the job, the run, input and output datasets, and facets (typed extras: schema, SQL, column lineage, parent run). A dataset is a namespace (where it lives, such as trino://trino:8080) plus a name. Parent and root facets chain runs (Under the Hood). Integrations emit events; a backend serves the graph.

What breaks without it. Impact analysis is grep, and “which dashboards read this column?” takes a week. Larkspur: tracing the revenue and stock reports to their raw tables, with the job behind each edge. Elsewhere: banks working to the Basel Committee’s BCBS 239 principles use lineage to show an auditor where a risk number comes from.

Column-level lineage: what you actually get

Column-level lineage maps each output column to the input columns behind it, optionally with the kind of transformation (direct, or indirect through a join or filter). It turns “the report depends on 33 tables” into “this column depends on these five”. Each integration derives it differently:

Integration	How	In the lab
dbt-ol 1.54	parses each model’s compiled SQL	Table models only: none for the two microbatch facts. A CTE (`money`) appears as an input dataset, and the trace stops there. No transformation types.
Spark listener 1.54	the query’s logical plan	DIRECT (identity, transformation) per column, plus dataset-level INDIRECT (filter, join, window). An exploded array collapses to the array: `sku <- lines`.
Airflow provider 2.20	what an operator reports, else declared inlets and outlets	None from `BashOperator`

What breaks without it. A column rename passes review because nobody sees the three reports reading it. Larkspur: before Part 23 masks personal columns such as loyalty IDs, column lineage lists the reports that carry them. Elsewhere: a health insurer must list every report carrying a diagnosis code before masking it.

Incident response for data

A data incident is a broken promise to a reader, run like a production incident: detect, triage, locate, communicate, fix, restate if needed, review, from a runbook written before the incident as queries you can run (Step 6). Larkspur: the 06:00 page, then triage. Elsewhere: an airline’s pricing team hears “fares data delayed, yesterday’s used” before its 07:00 run.

OpenTelemetry for pipeline runs

OpenTelemetry is the vendor-neutral standard for metrics, traces and logs; a collector receives and exports them. For pipelines it measures the runs (durations, queues, a span per task): when freshness breaches, it says whether the run was slow, stuck or failed. Larkspur: Airflow sends both to a collector. Elsewhere: a streaming service sees its slow nightly recommendations job and a cluster-scaling delay in one trace.

Under the Hood

One event, three levels of parentage

A real dbt-ol COMPLETE event from the daily DAG, trimmed to what builds the graph:

{"eventType": "COMPLETE", "eventTime": "2026-10-29T09:15:01.255493Z",
 "run": {"runId": "01a186f1-617f-746b-8d2d-61253ee4f07a",
         "facets": {"parent": {"job": {"namespace": "larkspur", "name": "larkspur_daily_revenue_dbt"},
                               "root": {"job": {"namespace": "larkspur", "name": "larkspur_daily_revenue"},
                                        "run": {"runId": "01a10435-d800-…"}}}}},
 "job": {"namespace": "larkspur", "name": "lakehouse_pipeline.gold.larkspur.fct_refund.build.run"},
 "inputs": [{"namespace": "trino://trino:8080", "name": "lakehouse_pipeline.gold.dim_store"},
            {"namespace": "trino://trino:8080", "name": "lakehouse_pipeline.silver.int_refunds"}],
 "outputs": [{"namespace": "trino://trino:8080", "name": "lakehouse_pipeline.gold.fct_refund"}]}

The model run’s parent is the dbt command, whose own event names the Airflow task larkspur_daily_revenue.dbt_build as parent; the root is the DAG run. The IDs travel in environment variables the DAG fills from the provider’s macros.

Names decide whether the graph joins

Two events share an edge only if they name the dataset the same way. OpenMetadata resolves a name through namespaceToServiceMapping (event namespace to a registered service), or a schema.table suffix match against known tables. Spark names Iceberg tables by path (s3://larkspur-lake + warehouse/gold/fct_pos_line) and adds a symlinks facet with the catalog name gold.fct_pos_line; without it, OpenMetadata refused the path.

flowchart LR A[Airflow provider] --> F[(event files)] D[dbt-ol: Airflow and laptop] --> F S[Spark listener] --> F F -- load, Trino and Airflow stopped --> O[OpenMetadata: tables, pipelines, edges] O --> T[trace, triage] C[(gold.daily_controls)] --> L[06:00 SLO check] --> R[(gold.slo_results)] P[(report: published tag + published-through)] --> L L -- FAIL --> G[page]

Hands-On

The code is in the every-record-counts folder of the companion-workspaces repository (tag every-record-counts/part-22), on top of Parts 19–21’s DAG and control table; the README has every command, including the fix. Three phases, because the stack doesn’t fit the lab’s 9.8 GB VM at once:

make up-spark                     # 1: core + Spark
uv run python -m scripts.part22_observe spark-lineage --day 2026-09-30
make stop-batch && make up-orchestrate   # 2: core + Trino + Airflow + collector
uv run python -m scripts.part22_observe dbt-lineage
uv run python -m scripts.part22_observe controls-history --from 2026-09-06 --to 2026-09-29
make airflow CMD="dags unpause larkspur_data_slo"
make airflow CMD="backfill create --dag-id larkspur_daily_revenue --from-date 2026-10-02 --to-date 2026-10-03 --reprocess-behavior completed --max-active-runs 1"
make airflow CMD="backfill create --dag-id larkspur_data_slo --from-date 2026-10-04 --to-date 2026-10-05 --max-active-runs 1"
uv run python -m scripts.part21_reconcile inject hold --day 2026-10-04
make airflow CMD="backfill create --dag-id larkspur_daily_revenue --from-date 2026-10-04 --to-date 2026-10-04 --reprocess-behavior completed --max-active-runs 1"
make airflow CMD="backfill create --dag-id larkspur_data_slo --from-date 2026-10-05 --to-date 2026-10-06 --max-active-runs 1"
uv run python -m scripts.part22_observe logs evaluate_slos --day 2026-10-05
make stop-orchestrate stop-batch && make up-lineage   # 3: core + OpenMetadata
uv run python -m scripts.part22_observe om-setup
uv run python -m scripts.part22_observe replay
uv run python -m scripts.part22_observe trace gold.fct_stock_daily     # and again with --column on_hand_qty
uv run python -m scripts.part22_observe triage --day 2026-10-03 --country GB

Step 1: pick a backend that runs

Marquez, the OpenLineage reference backend, has had no release since 0.50.0 (October 2024), no main commit since April 2026, and amd64-only images: dormant. DataHub’s quickstart (MySQL, OpenSearch, Kafka, four services) is tested with 8 GB of RAM. OpenMetadata 2.0.7 (Apache 2.0, arm64, releases every week or two) takes OpenLineage over HTTP and needs Elasticsearch (9.3.0, 512 MB heap) and a database (the lab’s PostgreSQL). It peaked at 2.2 GiB; with phase 2’s up to 5.6 GiB and 1.4 GB of other containers that’s about 9.2 GiB on a 9.8 GB VM. So the lab’s producers write files and replay loads them later. That’s a batch, as fresh as its last load: in production, point the transports at /api/v1/openlineage/lineage or a queue, because an 08:00 incident needs a backend that already knows last night’s runs.

Step 2: emit from Airflow, dbt and Spark

The Airflow image ships the OpenLineage provider (2.20.1; the lab pins 2.20.3). It and dbt-ol each append to a file:

    AIRFLOW__OPENLINEAGE__NAMESPACE: larkspur
    AIRFLOW__OPENLINEAGE__TRANSPORT: '{"type": "file", "log_file_path": "/opt/openlineage/airflow.jsonl", "append": true}'
    LAB_DBT_OPENLINEAGE: "1"   # scripts.part17_dbt runs dbt-ol instead of dbt
    OPENLINEAGE__TRANSPORT__TYPE: file
    OPENLINEAGE__TRANSPORT__LOG_FILE_PATH: /opt/openlineage/dbt.jsonl
    OPENLINEAGE__TRANSPORT__APPEND: "true"

AIRFLOW__OPENLINEAGE__* is the provider’s [openlineage] section, OPENLINEAGE__TRANSPORT__* the Python client’s (dbt-ol’s); append keeps one file, not one per event. A BashOperator can’t say what it read, so each task declares its tables, named as dbt-ol names them, and the dbt task passes its run IDs on:

def table(name: str) -> Dataset:
    return Dataset(namespace="trino://trino:8080", name=f"lakehouse_pipeline.{name}")

OL_PARENT = {
    "OPENLINEAGE_PARENT_ID": "{{ macros.OpenLineageProviderPlugin.lineage_parent_id(task_instance) }}",
    "OPENLINEAGE_ROOT_PARENT_ID": "{{ macros.OpenLineageProviderPlugin.lineage_root_parent_id(task_instance) }}",
    "OPENLINEAGE_NAMESPACE": "larkspur",
    "OPENLINEAGE_DBT_JOB_NAME": "larkspur_daily_revenue_dbt",
}
…
    accept = BashOperator(task_id="accept_pos_events", …,
        inlets=[table("raw.pos_uploads")], outlets=[table("silver.pos_event_dispositions")])
    dbt_build = BashOperator(task_id="dbt_build", …, env=OL_PARENT, append_env=True)

The DAG builds four models; dbt-lineage covers the 23 under them with one dbt-ol run from the laptop. spark-lineage runs Part 20’s Write-Audit-Publish job with the listener (--jars openlineage-spark_2.13-1.54.0.jar, spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener, spark.openlineage.transport.type=file). OpenLineage builds Spark modules up to 4.0; on Spark 4.1.3 with Iceberg 1.12 it worked (19 events in 28 s, Spark at 2.02 GiB). After the runs below, emitted:

Airflow provider      72 events (COMPLETE 30, FAIL 6, START 36), 13 jobs, outputs 5, with columnLineage 0
dbt-ol in Airflow     78 events (COMPLETE 39, START 39), 13 jobs, outputs 5, with columnLineage 3
                     outputs without column lineage: fct_basket, fct_sales_line
dbt-ol on a laptop    48 events (COMPLETE 24, START 24), 24 jobs, outputs 23, with columnLineage 23
Spark listener        19 events (COMPLETE 6, RUNNING 7, START 6), 6 jobs, outputs 1, with columnLineage 1
dbt-ol column lineage, rpt_daily_net_revenue_v2.net_minor <- money.amount_minor, dim_fx_rate.rate
Spark column lineage, fct_pos_line.amount_minor <- pos_uploads.lines [DIRECT TRANSFORMATION]

(Namespace lines trimmed.)

Step 3: load into OpenMetadata

om-setup creates what lineage needs to land on: a database service for the lake (what OpenMetadata’s Trino connector would ingest), the raw, silver and gold schemas, a pipeline service, and the OpenLineage settings with both Trino namespaces mapped to the service. replay registers the tables the events name (columns from the Iceberg catalog, else the event’s schema facet) and the parent jobs as pipelines, then posts each file’s COMPLETE events:

registered 45 tables and 7 pipelines
Airflow provider     COMPLETE  30: lineage from  6, no resolvable table 24, failed 0; edges 14
dbt-ol in Airflow    COMPLETE  39: lineage from 15, no resolvable table 24, failed 0; edges 54
dbt-ol on a laptop   COMPLETE  24: lineage from 21, no resolvable table  3, failed 0; edges 37
Spark listener       COMPLETE   6: lineage from  1, no resolvable table  5, failed 0; edges 2
adapted for OpenMetadata 2.0.7: eventTime +00:00 -> Z (55), schema facet without types dropped (51), parent job name: dots -> underscores (3)
not resolvable: 12 dataset references in namespace file (no service, no symlink)

“No resolvable table” are events without datasets: sensors, tests, DAG runs. Registering first works around 2.0.7’s failing auto-create on PostgreSQL. The adapter line is the price of a standard implemented unevenly (each error is in Troubleshooting); the three dotted parents are the Airflow task larkspur_daily_revenue.dbt_build, since the provider names tasks dag.task.

Step 4: trace the sales and stock reports

gold.fct_stock_daily
  <- gold.dim_product  (larkspur-dbt-run-larkspur)
    …
    <- silver.int_unresolved_products  (larkspur-dbt-run-larkspur)
      <- silver.int_sales_lines  (larkspur-dbt-run-larkspur)
        <- silver.stg_pos_sale_lines  (larkspur-dbt-run-larkspur)
          <- raw.pos_uploads  (larkspur-dbt-run-larkspur)   [no upstream]
      …
      <- silver.stg_stock_snapshots  (larkspur-dbt-run-larkspur)
        <- raw.stock_snapshots  (larkspur-dbt-run-larkspur)   [no upstream]
  <- gold.dim_store  (larkspur-dbt-run-larkspur)
  <- silver.stg_stock_snapshots  (larkspur-dbt-run-larkspur)   (above)
gold.fct_stock_daily.on_hand_qty
  <- silver.stg_stock_snapshots.on_hand_qty  (larkspur-dbt-run-larkspur)
    <- raw.stock_snapshots.on_hand_qty  (larkspur-dbt-run-larkspur)

(Nineteen lines trimmed; in brackets, the pipeline, i.e. the parent job: dbt-run-larkspur is the laptop’s run.) The stock fact depends on till uploads, online orders, purchase orders, stock adjustments and the snapshots themselves: dim_product carries placeholders for every product that turns up in any of them before the catalogue knows it (Part 4’s early-arriving facts). Nobody would draw that on a whiteboard. The revenue report, built by larkspur_daily_revenue_dbt from five direct inputs, reaches 33 tables; its net_minor column trace returns only gold.dim_fx_rate.rate, because the money CTE resolved to no table. [no upstream] marks a source or a silent job: the raw tables’ inputs are files or uninstrumented scripts.

Step 5: SLOs, a breach and a page

sql/part22/slo.sql measures the upload day a 06:00 London check expects:

SLI	Measured from	Good when	Objective
freshness	the report’s `published-through` property	covers the expected upload day	27 of 28 days
completeness	`gold.daily_controls`, that day’s run (Part 21)	no control failed	27 of 28 days
volume	Part 21’s landing control: till events per country	within 20% of the same weekday last week	every day
schema	Iceberg schema at the `published` tag vs the dbt contract	identical	every day

Freshness asks the table, not the scheduler: Part 19’s publish now sets larkspur.published-through-upload-date and larkspur.published-at in the same Iceberg commit that moves the tag. In production the SLI is “covers yesterday and was committed before 06:00”; lab commits all happen today, so the check is a DAG whose logical date is its morning and asks whether the published snapshot covers the day as it runs. Days before the checks existed are judged from run history, and the output says how many.

    schedule=CronTriggerTimetable("0 6 * * *", timezone="Europe/London"),
…
    BashOperator(task_id="evaluate_slos", bash_command=f"{LAB22} slo --at {{{{ logical_date.isoformat() }}}}",
        retries=0, pool="lakehouse", on_failure_callback=larkspur_alerts.slo_breached, …)

controls-history first ran Part 21’s control for the 22 September days that predate it (about 3 s each). Upload days 2 and 3 October ran, and the 4 October check passed, 28 of 28. Then inject hold dropped one UK till’s upload from Sunday 4 October (GB-0014-T03, 9 events), the day ran, reconcile failed on 3 October (ledger_vs_gold_sales, GB, two baskets, £66.45) and publish never ran. The 5 October check:

larkspur_data_slo backfill__2026-10-05T05:00:00+00:00 evaluate_slos: failed
the 06:00 check at 2026-10-05 06:00 London: upload day 2026-10-04 due (window 2026-09-07..2026-10-04)
freshness     report    FAIL  27/28 days, budget left 0      published through 2026-10-03
completeness  controls  FAIL  27/28 days, budget left 0      1 failed of 119: ledger_vs_gold_sales 2026-10-03 GB
volume        GB        pass                                 6553 till events uploaded on 2026-10-04 vs 6782 on 2026-09-27 (-3.4%)
schema        report    pass                                 7 columns as contracted
budget: 2 days as measured by 06:00 checks, 26 from run history
2 failed; 0.7 s
SLO BREACHED: larkspur_data_slo, the 2026-10-05 06:00:00+01:00 check; see gold.slo_results

(IN and SG volume lines trimmed: −1.1%, +0.2%.) In production that page lands two hours before Farah’s message. Freshness is the symptom, completeness the cause; volume can’t see one till among thousands. Unpausing the DAG also ran today’s check (29 October), which failed for real: the lab’s uploads end on 5 October, and the scheduled run for 28 October failed in its sensor (AirflowSensorTimeout: … run duration of 21625.64631 seconds exceeds the specified timeout of 21600.0).

Step 6: the runbook, then the fix

“The dashboard is wrong”: the first 15 minutes.

What does the reader see, since when? The published snapshot and its published-through day, against main.
Stale (the tag didn’t move) or wrong (it moved, and something disagrees)?
Which control fails, for which day and country?
Where do totals first disagree? Lineage down from the report, a control on each hop.
Tell the reader what, why and when.
Fix and replay (Part 21); restate if a published number changes.
Review: which signal should have fired earlier, which lineage was missing.

triage runs steps 1, 3 and 4 at once (phase 3):

published  snapshot 493324087334385373 committed 2026-10-29 09:06 UTC: GB 2026-10-03 net 393,508.00, provisional True
main       snapshot 4188511065259714804 committed 2026-10-29 09:08 UTC: GB 2026-10-03 net 393,441.55, provisional True
published through upload day 2026-10-03, at 2026-10-29T09:06:45+00:00
controls for 2026-10-03 GB (newest run, upload day 2026-10-04): 6 pass, 1 FAIL
  FAIL ledger_vs_gold_sales: source 8552, target 8550, money difference -66.45
  33 tables upstream; 12 with no upstream edge: gold.dim_date, gold.dim_promotion, raw.catalog_products, raw.fx_rates, raw.order_lines, raw.orders, raw.po_lines, raw.pos_uploads, raw.refunds, raw.stock_adjustments, raw.stock_snapshots, raw.stores

(The report’s five direct inputs trimmed.) The reader sees Saturday’s run; main is £66.45 short. The gate turned a correctness incident into a freshness incident, the right trade. Lineage leads through fct_sales_line to raw.pos_uploads; the controls put the loss at landing. After inject restore and Part 21’s replay (tasks clear … -t ingest_pos_uploads -d), the report published. A re-check at 09:30:

a re-check at 2026-10-05 09:30 London: upload day 2026-10-04 due (window 2026-09-07..2026-10-04)
freshness     report    pass  27/28 days, budget left 0      published through 2026-10-04
completeness  controls  pass  27/28 days, budget left 0      0 failed of 119

(Volume, schema and summary lines trimmed.) Passing now, budget spent. Clearing the 06:00 run itself prints the recorded result rather than measuring again: the 06:00 check for upload day 2026-10-04 was recorded at 2026-10-29 09:09 UTC; not measured again.

Step 7: OpenTelemetry for the runs

AIRFLOW__METRICS__OTEL_ON and AIRFLOW__TRACES__OTEL_ON switch it on; the endpoint comes from the SDK’s variables (OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318, OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf, OTEL_SERVICE_NAME), since the Airflow 3.3 docs deprecate otel_host, otel_port and the rest. The collector writes to files; otel reads them:

137 metric names; by family: ti 30, pool 18, dagrun 16, dag_processing 15, dag 12, scheduler 11
64 spans; by name: worker.evaluate_slos 3, … dag_run.larkspur_daily_revenue 3, task_run.reconcile 3, …

(Lines trimmed.) A DAG run is a dag_run.<dag> span holding task_run.<task> and worker.<task> spans; timers are exponential histograms. Phase 2 peaked at 5.20 GiB (Trino 3.0, scheduler 1.10); an earlier pass put the scheduler at 1.45 of its then 1.5 GB, so it now has 2 GB.

The Landscape

Need	Open source	AWS	GCP	Azure	Snowflake / Databricks	Notes
Lineage from OpenLineage events	OpenMetadata, DataHub; Marquez (no release since 2024)	Amazon DataZone `PostLineageEvent`, OpenLineage run events only (also in SageMaker Unified Studio)	Knowledge Catalog (formerly Dataplex Universal Catalog) Data Lineage API, `ProcessOpenLineageRunEvent`	Microsoft Purview: lineage from Airflow via OpenLineage and Event Hubs (preview)	Snowflake external lineage REST endpoint (GA, Enterprise Edition, COMPLETE events only); Databricks Unity Catalog lineage system tables and External lineage API (OpenLineage not mentioned)
Freshness and volume checks	Elementary dbt package (`freshness_anomalies`, `volume_anomalies`); SQL over control tables	AWS Glue Data Quality `DataFreshness`, `RowCount`, anomaly detection	Knowledge Catalog auto data quality (freshness, volume dimensions)	Purview Data Quality freshness rule	Snowflake system DMFs `FRESHNESS`, `ROW_COUNT` (anomaly detection (preview)); Databricks data quality monitoring anomaly detection (public preview)	None of them knows your 06:00 or your ledger
Run metrics and traces	Airflow OpenTelemetry + OpenTelemetry Collector	Amazon MWAA: StatsD metrics to CloudWatch	Managed Service for Apache Airflow (Cloud Composer is “evolving to become” it) with Cloud Monitoring	Azure Monitor for Data Factory	Databricks `system.lakeflow.*` job tables	Data checks say what is late; run telemetry says why

From vendor documentation, 29 October 2026.

Tips & Gotchas

Gotcha — one table, many names. dbt-ol builds the namespace from the profile’s host, so the same Trino is trino://trino:8080 inside Docker and trino://localhost:8080 on a laptop, and Spark names the table by its S3 path. Unmapped, that’s three nodes. Map every producer’s namespace to one service.

Gotcha — lineage stops where instrumentation stops. Every [no upstream] is a gap, not a source: files the backend can’t name, a loader script, and models whose inputs dbt-ol didn’t report. dbt-ol emitted gold.dim_promotion, which reads a snapshot, with no inputs at all. Count the gaps and keep them as backlog.

Gotcha — on a Write-Audit-Publish write, the version facet is main’s (Spark 4.1.3, not supported upstream; lab observation). The listener reported the snapshot main had before the job, not the audit snapshot it wrote, and fast_forward emitted no dataset. Lineage says the job wrote the table, not which snapshot readers got.

Gotcha — retrying a deadline check pages again. The 06:00 record is kept, so the cleared run fails and pages again. Re-check at another time, or route repeats to the open incident.

Tip — buffer lineage events. A file or queue between producers and backend means an outage loses nothing and a new backend loads from history.

Troubleshooting

Symptom	Likely cause	How to confirm	Fix
OpenMetadata `400 Invalid request format`	`eventTime` ends in `+00:00` (Airflow provider)	Re-post with `Z`	Adapt on load, or a proxy
OpenMetadata `400 query param type must not be null`	dbt-ol schema fields without `type` (no `catalog.json`)	One error per dataset	Drop the facet, or `dbt docs generate` first
`no lineage edges created`; server log `null value in column "updatedat"`	2.0.7 auto-create on PostgreSQL (issue #27548)	`GET /api/v1/tables` is empty	Register tables and pipelines first
Edges without a pipeline	Parent job name with a dot (`dag.task`); 2.0.7 builds the pipeline FQN unquoted	The pipeline exists as `openlineage."larkspur-…"`	Dotless names where you choose; adapt the rest
Elasticsearch `red`, shards unassigned	Docker disk over the 90% default high watermark	`_cluster/allocation/explain`	Free disk; the lab raises the watermarks to 95/97/98%
No traces; task log `Failed parsing HTTP/2`	The default traces exporter, `otlp`, is gRPC, sent to the HTTP port	The log names gRPC	`OTEL_TRACES_EXPORTER=otlp_proto_http`

Design Drill

The question: “A property manager runs 60,000 rental homes in four regions. Every morning at 07:00 the operations warehouse publishes yesterday’s occupancy and units available to let, built from the leasing system, move-in and move-out inspections, and a maintenance system that takes units offline. At 8 a.m. the CFO says yesterday’s occupancy is wrong. Walk through your first 30 minutes, then say what you’d build so it doesn’t happen again.”

Clarify first

Wrong how: too high, too low, or unchanged from the day before?
Which figure, cut how (portfolio, region, building), on which dashboard, from which table?
What independent counts exist: the leasing system’s occupied units, the regional managers’ daily sheets?

The shape of a strong answer

0–5 minutes: acknowledge, ask what they see, promise an update time; check whether the 07:00 SLO checks passed.
5–15: stale or wrong? Compare the published snapshot and its covered day with the latest build; read the day’s controls (units per region, leasing system vs warehouse). The first failing hop is the lead.
15–25: follow lineage from that hop to its job and source (a missing inspection export, a duplicated lease, a renamed status code), and downstream to other readers.
25–30: tell the CFO the cause or the next step, hold downstream sends (investor reports), decide fix-forward or restate.
Prevention: SLIs per reader (07:00 freshness on the table, completeness against the leasing system, volume per region, schema) with budgets; publish gated on controls; OpenLineage into a live backend; a runnable runbook.

Trade-offs to say out loud

A gate turns wrong into late: right for an investor figure; a leasing team’s dashboard may prefer publishing flagged as provisional.
Volume anomaly checks are cheap and noisy; reconciliation is exact but needs an independent source.

Common weak answers

Opening the SQL before asking what the CFO is looking at.
“Monitor the jobs”: every job can succeed while one region’s export never arrived.
Fixing silently: the number may already have been forwarded.

Story Closing

On Tuesday the 06:00 check passed, and Farah’s first message of the day was about something else.

“The budget’s spent for the month,” Rohan said. “One more miss and we’re over.”

“Then the next sprint goes on the lineage gaps and the landing control, not new reports,” Ingrid said. “That’s what the budget is for.”

Of the five promises from Part 2, this part strengthens timeliness: the report’s deadline is measured, recorded and paged on, and lineage shortens the way from page to cause. In Part 23 the same map shows something else: which tables hold customers’ names and card fragments, and who can read them. Security, privacy and the right to be forgotten in an immutable lake.

This is Part 22 of “Every Record Counts: Data Engineering for Backend Engineers” — Season 2: Pipelines in Production.

Series Progress 26/26 parts

0 Part 0: "The Field Guide" — How to Read This Series Complete 1 Part 1: "Three Houses for Data" — Warehouse, Lake, Lakehouse Complete 2 Part 2: "Think in Pipelines" — The Data Engineer's Mental Model Complete 3 Part 3: "Bytes on Disk" — File Formats, Parquet and Object Storage Complete 4 Part 4: "Grain Before Glory" — Dimensional Modeling Complete 5 Part 5: "Other Shapes" — Data Vault, Wide Tables and Modeling Trade-offs Complete 6 Part 6: "SQL You Will Actually Use" — Analytical SQL for Data Engineers Complete 7 Part 7: "The Table That Remembers" — Apache Iceberg in Depth Complete 8 Part 8: "Format Wars, Ceasefire" — Delta, Hudi, Paimon, DuckLake and Interoperability Complete 9 Part 9: "Who Owns the Truth" — Catalogs and the Iceberg REST Protocol Complete 10 Part 10: "Where the Bytes Live" — Partitioning, File Layout and Table Maintenance Complete 11 Part 11: "You Might Not Need a Cluster" — DuckDB, Polars and Single-Node Analytics Complete 12 Part 12: "Shuffle, Spill and Skew" — Distributed Batch with Spark 4.1 Complete 13 Part 13: "Engines at Scale" — Trino, Cloud Warehouses and Real-Time OLAP Complete 14 Part 14: "Design Review I" — The Storage and Compute Layer Complete 15 Part 15: "Getting Data In" — Batch Ingestion: Files, APIs and Databases Complete 16 Part 16: "Changes, Not Snapshots" — CDC into the Lakehouse Complete 17 Part 17: "Transformations as Code" — dbt Fundamentals Complete 18 Part 18: "dbt at Scale" — Incremental Models, CI and Environments Complete 19 Part 19: "Who Runs What, When" — Orchestration with Airflow 3 Complete 20 Part 20: "Trust, but Test" — Data Quality and Data Contracts Complete 21 Part 21: "Every Record Counts" — Completeness and Correctness Engineering Complete 22 Part 22: "Know Before They Do" — Observability, Lineage and SLAs Complete 23 Part 23: "Locks on the Lake" — Security, Privacy and Compliance Complete 24 Part 24: "The Bill Always Comes" — Cost and Performance Engineering Complete 25 Part 25: "Design Review II" — The Batch Reporting Platform, End to End Complete