"Trust, but Test" — Data Quality and Data Contracts

Skim Card

One thing to remember: data quality is a promise a producer makes to its consumers, written down and tested where data crosses from one owner to the next, before anyone reads it.

• Put each ownership boundary in a contract: schema, meaning, rules, owner and consumers in ODCS 3.2; datacontract-cli turns it into checks.
• Test at ingestion, between layers and before publish; July’s events contract found 2,675,104 events missing page after the rename.
• “Breaking” means breaking for consumers: a removed field, a changed type, a dropped required. CI blocks them unless the major version goes up.
• Quarantine rare row defects below a ceiling, fail above it: suppliers quarantine 0.35%; at 1% the contract test fails and the load must stop.
• Settle a metric’s inputs in a contract both consumer teams sign: 30-minute sessions, bots out, retries once. Conversion is 6.21%.
• Write-Audit-Publish on an Iceberg branch: a dedup bug’s 169 duplicate lines stayed on audit; readers of main never saw them.
• Trino 484 reads Iceberg branches but can’t create or write them, so WAP runs in Spark.

Use …	When …
Quarantine the row	Rare, independent row defects, under an agreed ceiling
Fail the batch	Missing fields, duplicates, a closed day changed, quarantine over the ceiling
WAP on a branch	Readers query main and must never see an unaudited batch
A tag after checks (Part 19)	Readers opt in to the tag, or the writer can’t use branches

Verified against: ODCS 3.2.0, datacontract-cli 1.2.8, Apache Iceberg 1.12.0, Apache Spark 4.1.3, Trino 484, Apache Polaris 1.8.0, PyIceberg 0.12.0, DuckDB 2.0.0 · vendor docs · 29 October 2026

Jump to: Concepts · Hands-On · Landscape · Design Drill

Story Opening

Part 19 ended with a check stopping a hand-edited exchange rate, and Farah asking who decides which rules a number has to pass.

Rohan had an answer ready. “The app team renamed page to page_path on 15 August and told nobody. Part 6 caught it with a coalesce. I’ll put null checks in every dashboard query, the way I validate every request body in the order service.”

“Then forty dashboards each guess what the app team meant,” Ingrid said. “And product and marketing still disagree on what a session is. Write the agreement down, version it like an API, and test it where the data crosses from one owner to the next.”

The Quick Map

Concepts, with use cases

Quality dimensions

A quality dimension names one way data can be wrong. ODCS, the contract standard used below, lists seven: accuracy (matches reality), completeness (nothing missing), conformity (agreed format), consistency (agrees with other data and over time), coverage (every expected entity), timeliness and uniqueness. Checks are how Part 2’s five promises get tested.

What breaks without them. “The data looks wrong” can’t be tested; “1.1% of supplier rows quarantined, over the 1% ceiling” can. Larkspur: every rule below names its dimension. Elsewhere: a hospital’s lab results feed, where a missing unit (conformity) is a safety problem, not a reporting one.

Where to test, and what

Test at the three places data changes hands:

What breaks without layered tests. A check only at the end can say the number is wrong but not where it went wrong; a check only at ingestion misses the join that doubled rows in silver. Elsewhere: a payments switch tests card numbers at intake and settlement totals before the scheme file goes out.

Quarantine vs fail-fast

Quarantine sets aside the rows that break a rule, with the reason, and loads the rest. Fail-fast stops the batch. The rule of thumb: row-level defects that are rare and independent (a bad date in one supplier row) are quarantined; batch-level defects (a missing column, a duplicated day, a closed day that moved) fail the run.

What breaks without the distinction. Fail-fast on every bad row stops all forty suppliers’ fill rates for one typo; quarantine without a limit turns a broken feed into silently missing data. Larkspur: Part 15 quarantined 81 supplier rows (0.35%); the supplier contract now says that above 1% the feed is broken and the load stops. Elsewhere: an insurer parks one malformed claim for a human but rejects a file whose policy numbers stopped matching.

Data contracts

A data contract is a versioned, machine-readable agreement between a dataset’s producer and its consumers: schema, meaning, quality rules, owner, location, and what changing it requires. You know it as an OpenAPI spec plus an SLA, for data at rest. The Open Data Contract Standard (ODCS) from the Bitol project (LF AI & Data, graduated in 2026) is the current open format; version 3.2.0 was released on 8 September 2026. datacontract-cli 1.2.8 (MIT) lints, compares (breaking) and tests ODCS contracts.

Ownership is the point: the producer owns the contract, the consumers listed in it sign off on breaking versions. Additive changes are minor versions; anything that can break a consumer is a new major version, run next to the old one while consumers migrate.

What breaks without it. A producer’s harmless rename becomes a consumer’s outage, found weeks later. Larkspur: contracts for the supplier tables, the app team’s events and the data platform’s sessions. Elsewhere: a ride-hailing company’s trip events, consumed by pricing, fraud and finance, where a unit change from metres to kilometres would reprice every trip.

Write-Audit-Publish

Write-Audit-Publish (WAP) writes a batch somewhere readers don’t look, audits it there, and publishes it in one atomic step only if the audit passes. On Iceberg the “somewhere” is a branch, a named line of snapshots next to main, and publishing fast-forwards main to the branch head.

What breaks without it. Part 19 published by moving a tag after the checks: readers who opt in to the tag are safe, but main held the unchecked report the moment it was built. Part 18’s delete-then-insert left a second in which a day was missing. Larkspur: the POS sales fact is published by WAP below. Elsewhere: a market-data vendor publishes end-of-day prices only after cross-checking exchanges; clients’ risk systems read them the moment they appear.

Who checks what

Soda Core (SodaCL checks) moved to the Elastic License 2.0 in January 2026, so it is no longer open source; Elementary 0.26.1 (Apache 2.0) reports dbt test results and anomalies, an observability question for Part 22. The split is our judgement: one tool per layer, chosen by where the check runs. Only datacontract-cli is installed in this part.

Under the Hood

What a contract turns into

datacontract test generates checks: per property, present and type, plus one per required, unique, logicalTypeOptions (minimum, maximum, pattern) and enum; then each quality rule, a library metric (nullValues, duplicateValues, rowCount…) or SQL with an operator (mustBe, mustBeLessThan, mustBeBetween). Where they run depends on the server type. For trino it sends SQL to Trino. For iceberg it loads the table through the REST catalog with PyIceberg, scans it to Arrow and runs the checks in an in-process DuckDB: fine for six million rows, not for six billion.

breaking compares two versions. Its default consumer perspective flags what can surprise a consumer: a removed property, a type change, a property no longer required; the producer perspective, what can fail the producer’s own tests.

What WAP changes in the metadata

A branch is an entry in the table’s refs map: a name, a snapshot ID, retention settings. With write.wap.enabled = true on the table and spark.wap.branch = audit in the session, each commit moves only audit; readers resolve main, which hasn’t moved. fast_forward('t', 'main', 'audit') sets main to the head of audit: one metadata commit, no data written.

Trino 484 reads branches (FOR VERSION AS OF 'audit') but its Iceberg connector can’t create one or write to one; PR #31135, which adds branch management, was still open on 29 October 2026. So WAP stays a Spark job, and dbt-on-Trino models can’t use it yet.

Hands-On

The code is in the every-record-counts folder of the companion-workspaces repository (tag every-record-counts/part-20). Core plus Trino for the contracts, then core plus Spark for WAP (they don’t fit in memory together). The commands, in step order:

make contracts-install         # datacontract-cli 1.2.8 in .venv-contracts (its DuckDB extra pins 1.5; the lab runs 2.0)
make up-trino
uv run python -m scripts.part20_contracts land
uv run python -m scripts.part20_contracts test CONTRACT [--server trino]   # supplier, July events, events v2
uv run python -m scripts.part20_contracts sessions
uv run python -m scripts.part20_contracts conversion
uv run python -m scripts.part20_contracts test contracts/clickstream_sessions.odcs.yaml
make contract-ci ARGS="--base OLD --head NEW"    # three pairs, in step 4
make stop-batch && make up-spark
uv run python -m scripts.part20_wap init --until 2026-09-29
uv run python -m scripts.part20_wap publish --day 2026-09-30
uv run python -m scripts.part20_wap publish --day 2026-10-01 --bug skip-dedup
uv run python -m scripts.part20_wap refs
uv run python -m scripts.part20_wap publish --day 2026-10-01
make stop-batch && make up-trino
uv run python -m scripts.part20_wap trino

test runs datacontract-cli against the contract’s lakehouse server (Iceberg through Polaris) as the principal that reads the data in production (analyst for the sessions in gold, pipeline otherwise) and prints a summary plus every check that didn’t pass.

Step 1: the supplier feed’s contract

Nobody at a supplier will read YAML, so the contract covers what Larkspur publishes from their files: Part 15’s silver.supplier_deliveries and its quarantine, with each header as sent kept as businessName.

apiVersion: v3.2.0
kind: DataContract
id: larkspur-supplier-deliveries
version: 1.0.0
schema:
  - name: supplier_deliveries
    properties:
      - name: po_number
        businessName: PO Number          # the header in the supplier's file
        logicalType: string
        required: true
        primaryKey: true
        primaryKeyPosition: 1
  - name: supplier_delivery_quarantine
    quality:
      - name: quarantine_under_one_percent
        dimension: completeness
        type: sql
        query: >-
          SELECT 100.0 * count(*) / (count(*) + (SELECT count(*) FROM supplier_deliveries))
          FROM supplier_delivery_quarantine WHERE copy_no = 1
        mustBeLessThan: 1

ODCS 3.2 requires apiVersion, kind, id and version; version is semver by convention, and the CI gate in step 4 relies on it.

{"contract": "supplier_deliveries.odcs.yaml", "server": "lakehouse", "result": "passed", "checks": {"passed": 49}}  (2 s)

--server trino passes the same 49. The quarantine rule is where quarantine meets fail-fast: 81 of 23,468 distinct rows quarantined is 0.35%; at 1% the contract test fails, and the load must stop on it.

Step 2: the events contract catches the rename

land copies the clickstream files, as sent, into raw.clickstream_events (5,609,827 events, 13.2% with a user_id, partitioned by event date, 3.9 s). The app team’s contract as it would have read in July had page as a required string. Tested against today’s events:

{"contract": "clickstream_events-1.0.0.odcs.yaml", "server": "lakehouse", "result": "failed", "checks": {"passed": 34, "failed": 1}}  (1 s)
  failed  page: Check that field page has no missing values: Actual missing_count(page) was 2675104, expected = 0

2,675,104 events, every web event since 15 August and every app 5.0 event, carry page_path instead: exactly the drift the injected-defects manifest lists. Version 2.0.0, the contract the teams agreed afterwards, describes the data as it is:

version: 2.0.0
      - name: page
        logicalType: string
        deprecated: true
        description: Sent by app 4.x only. Use page_path.
      - name: page_path
        logicalType: string
    quality:
      - name: one_page_field
        dimension: conformity
        type: sql
        query: SELECT count(*) FROM events WHERE (page IS NULL) = (page_path IS NULL)
        mustBe: 0

Part 6’s coalesce(page_path, page) is now a contracted rule rather than one consumer’s workaround, and deprecated (new in ODCS 3.2) says where it’s heading. The SDK delivers at least once, so event_id isn’t unique: two rules say a repeat must be the same event again and cap retries below 1% (they’re 0.3%). Version 2.0.0 passes all 46 checks.

Step 3: the session contract settles conversion

Part 6 found two conversion rates because two teams defined a session differently. The sessions contract, owned by the data platform and listing product and marketing as consumers, decides:

description:
  usage: >-
    Conversion rate = sessions with converted = true / all sessions, grouped by the period of started_at (UTC).
    Every dashboard that shows conversion reads this table; none sessionizes events itself.
schema:
  - name: web_sessions
    dataGranularityDescription: >-
      One session: one anonymous_id's events with no gap over 30 minutes, in event time. Counts all five event types,
      each event_id once (retries are the same event); events whose user agent matches (?i)bot|crawl|spider|headless
      start no session.
    quality:
      - name: sessions_split_at_30_minutes
        type: sql
        query: >-
          SELECT count(*) FROM (SELECT started_at - lag(ended_at) OVER (PARTITION BY anonymous_id ORDER BY started_at)
          AS gap FROM web_sessions) WHERE gap <= INTERVAL 30 MINUTE
        mustBe: 0
      - name: conversion_in_expected_range
        type: sql
        query: SELECT 100.0 * count(*) FILTER (WHERE converted) / count(*) FROM web_sessions
        mustBeBetween: [3, 10]

The decisions, each with its reason:

• 30 minutes of inactivity, in event time. A UTC day splits a Mumbai morning at 05:30 local time and merges a research visit into a later purchase.
• Bots out, retries once. A crawler can’t buy; a retry is not a second visit.
• Visitors are anonymous_id. With user_id on 13.2% of events, merging signed-in devices would change the denominator for some customers only. The contract states the limitation.

sql/part20/web_sessions.sql implements those rules in DuckDB; sessions writes the result to gold.web_sessions in one commit (2.6 s), and conversion reads it as the analyst:

{"sessions": 2161516, "converted": 134224, "conversion_pct": 6.21}

The contracted number is 6.21%, the same 2,161,516 sessions as product’s definition in Part 6. Marketing’s 5.31% was a valid answer to a different question (Part 6: neither query has a bug); it moves to 6.21% in two measured steps, dropping bots (+1.23 points, to 6.54%), then 30-minute sessions instead of UTC days (−0.33). It is a decision, not a discovery: product analytics and marketing analytics, the contract’s listed consumers, both signed version 1.0.0, and changing the definition now takes a major version. The sessions contract passes its 36 checks (one bounds max_gap_seconds inside a session at 1,800); its range rule fires if the definition or the events change, not on slow drift (Part 22’s job).

Step 4: a CI gate for breaking changes

make contract-ci lints each contract in contracts/ and runs datacontract breaking against its version at CONTRACT_BASE; a breaking change fails unless the major version went up. ARGS compares one explicit pair, here the app team’s two changes of last summer. The consent block of 1 August, new optional fields:

ok   contracts/proposals/clickstream_events-add-consent.odcs.yaml: no breaking change

The rename of 15 August, proposed as version 1.2.0:

FAIL contracts/proposals/clickstream_events-rename-page.odcs.yaml: breaking change in version 1.2.0
       ERROR Removed schema.events.properties.page
make: *** [contract-ci] Error 1

Version 2.0.0 is breaking too (page stops being required) and passes as a new major version. The gate only lets it through; the merge should still require the listed consumers as reviewers. With no ARGS it compares the working tree with the merge base with origin/main (HEAD without one): what a pull request changes. The diff catches announced changes; an unannounced one, like the real 15 August rename, is caught only by testing the data, as step 2 did.

Step 5: Write-Audit-Publish on the sales fact

gold.fct_pos_line holds POS sales lines, one per basket line, written by Spark as the pipeline principal. It sits beside dbt’s gold.fct_sales_line only because dbt here runs on Trino, which can’t write branches; in production one writer owns the fact. init loads upload days up to 29 September straight into main (5,804,944 lines). Then publish runs one upload day through WAP; spark/jobs/part20_wap.py, trimmed:

    # A fresh audit branch at main's head: whatever a failed run left on it is discarded.
    spark.sql(f"ALTER TABLE {TABLE} CREATE OR REPLACE BRANCH {BRANCH}")

    # WRITE. From here on, every write to the table commits to `audit`; main doesn't move.
    spark.conf.set("spark.wap.branch", BRANCH)
    spark.sql(f"DELETE FROM {TABLE} WHERE upload_date = DATE '{day}'")  # a re-run replaces the day
    where, keep = f"upload_date = DATE '{day}'", "true"
    if bug != "skip-dedup":  # drop re-sends within the day, and baskets main already published from an earlier day
        where += " AND " + NOT_PUBLISHED.format(published=f"{TABLE} VERSION AS OF 'main'", day=day)
        keep = FIRST_COPY
    spark.sql(f"INSERT INTO {TABLE} {LINES.format(raw=RAW, where=where, keep=keep)}")
    spark.conf.unset("spark.wap.branch")
    …   # AUDIT: sql/part20/wap_checks.sql against f"{TABLE}.branch_{BRANCH}" and main; return on any failure
    # PUBLISH. Main moves to the audited snapshot in one metadata commit; no data file is written or copied.
    row = spark.sql(f"CALL pipeline.system.fast_forward('gold.fct_pos_line', 'main', '{BRANCH}')").collect()[0]

TABLE was created with TBLPROPERTIES ('write.wap.enabled' = 'true'); publish returns before the fast-forward if any check fails. The four audit queries in sql/part20/wap_checks.sql carry Part 19’s cross-run checks over: unique_line (no basket line twice), raw_basket_missing (every basket uploaded that day is in the fact), history_changed (earlier upload days’ rows and money equal on audit and main) and invalid_line (quantity above zero, amount not negative). While the job runs, the host loads the table as the analyst every half second, as a dashboard would, and records main’s snapshot. 30 September:

write: 43,876 lines for upload day 2026-09-30 on branch audit in 2 commit(s), head 2500037962558194669; main still 6300805183550150038
…                    (four passing audit lines trimmed; timing lines trimmed too)
publish: main 6300805183550150038 -> 2500037962558194669 (was 6300805183550150038)
reader of main: 50 loads, saw snapshots [6300805183550150038, 2500037962558194669]

Two commits on the branch (the delete commits a snapshot even when the day is new, then the insert), and the reader went from the old head straight to the new one: Part 18’s missing-day second can’t happen. Now 1 October, with the dedup step “simplified away”:

write: 46,553 lines for upload day 2026-10-01 on branch audit in 2 commit(s), head 8511313869872026282; main still 2500037962558194669
audit: unique_line          FAIL 169
…                    (three passing audit lines trimmed; timing lines trimmed too)
publish: refused (unique_line); main stays at snapshot 2500037962558194669
reader of main: 48 loads, saw snapshots [2500037962558194669]

refs scans both heads:

audit  {"snapshot": 8511313869872026282, "lines": 5895373, "duplicate_lines": 169}
main   {"snapshot": 2500037962558194669, "lines": 5848820, "duplicate_lines": 0}

The only main snapshot that existed during the run has no duplicate line; the 169 lines from re-sent baskets exist only on audit, until the next run replaces the branch. Readers of main never saw them. Without the bug, the day writes 46,384 lines, passes and moves main to 5,895,204 lines; a re-run leaves the same count.

Trino, asked by part20_wap trino, reads the branch (FOR VERSION AS OF 'audit': 5,895,204 lines) and refuses to create or write one (Troubleshooting).

The Landscape

From vendor documentation, 29 October 2026. Glue Data Quality is GA; the other pages show no status label unless noted.

Tips & Gotchas

Gotcha — for a consumer, “now optional” is breaking. Making page optional felt like loosening a rule; datacontract breaking flagged it as an error, because every consumer that filters or joins on page now loses rows without an error. Postel’s law is a producer’s comfort, not a consumer’s.

Gotcha — set spark.wap.branch only around the writes. The Iceberg docs say a WAP branch and a branch identifier (t.branch_audit) can’t both be specified. The job sets the config for the delete and insert, unsets it, then reads both refs by name in the audit.

Gotcha — a failed audit branch keeps its files alive. Snapshot expiry never removes what a branch still references, so a refused batch stays in storage until the branch moves. CREATE OR REPLACE BRANCH at the next run discards it here; a branch you keep for debugging needs a RETAIN period.

Tip — test the batch, not the history, on every run. The iceberg server type scans whole tables into memory; that’s 2 s at lab size. At scale, run contract tests through the engine (--server trino) and test the whole table on a schedule. --filter (documented; not run in the lab) narrows only property and library checks, and only on SQL servers; type: sql rules must carry the partition predicate themselves.

Troubleshooting

Design Drill

The question: “You run the analytics platform for a car-parts manufacturer with eleven plants. The manufacturing execution system (MES) team publishes machine, inspection and scrap events to Kafka; you turn them into OEE dashboards, scrap cost and warranty analytics that customers audit. Three MES releases this quarter renamed or retyped a field, and the dashboards were wrong for days each time. An upstream team keeps breaking your pipelines with schema changes. Fix the process and the tech.”

Clarify first

• Which consumers are contractual or regulated (customer warranty audits, ISO/IATF quality records), and which are internal dashboards?
• How do MES releases ship today: per plant, all at once, with a staging environment?
• Is there a schema registry on the topics, and with which compatibility mode?

The shape of a strong answer

• Contracts per event type, owned by the MES team, in their repository: schema, meaning, units (scrap in kilograms or pieces), quality rules, consumers listed. The platform team provides templates, not the content.
• CI gate in the MES release pipeline: breaking against the published version; breaking changes only as a new major version, run in parallel (dual-publish both versions, or both fields) for an agreed window; the listed consumers are required reviewers of a major bump.
• The registry enforces at runtime what CI enforces at review: backward-compatible schemas on each topic, so an unreviewed producer can’t publish a breaking schema at all (Part 27).
• Producer-side contract tests on the staging environment’s output in every MES release pipeline, so an undeclared rename fails before it ships.
• Contract tests at ingestion on every batch or micro-batch: missing fields, types, enums (machine states), units; failures route the batch to quarantine and page the producer, not the dashboard owner.
• Quarantine one bad event, stop one plant’s partition when a field vanishes from all its events; WAP before publish for the warranty tables customers audit.
• Versioned consumers: staging models map v1 and v2 to one shape until the old version retires on a date in the contract.

Trade-offs to say out loud

• Contracts slow every MES release a little; the alternative is the platform team absorbing every change, forever.
• Strict registry compatibility blocks real fixes too; the escape hatch is a new topic or major version, not turning the check off.
• Quarantine protects the dashboards but delays data; for OEE a few hours are fine, for line-stop alerts they are not.

Common weak answers

• “Make the pipeline tolerant”: tolerance turns breaking changes into silently wrong numbers.
• Alerts on dashboards instead of tests at the boundary: the consumer finds out last.

Further Reading

1. Bitol, Open Data Contract Standard v3.2.0 (specification and JSON schema). https://github.com/bitol-io/open-data-contract-standard
2. datacontract-cli documentation (test, breaking, server types). https://cli.datacontract.com
3. Apache Iceberg 1.12 documentation: “Branching and Tagging” (audit branch), Spark procedures (fast_forward), Spark writes (writing to branches). https://iceberg.apache.org/docs/latest/branching/

Story Closing

“In the order service, a contract was the OpenAPI file and the consumer’s tests,” Rohan said. “I didn’t think a table needed one.”

“A table has more consumers than any API you’ve shipped,” Ingrid said.

Of the five promises from Part 2, this part strengthens correctness: the rules are written down, owned and tested where data changes hands. Consistency gained too: conversion is settled at 6.21%; Part 31 serves it as one metric.

Farah read the session contract twice. “So two of the three numbers have one answer now. What about revenue?”

Four of its five causes were fixed one by one across Parts 4 to 16, and each fix was checked on its own. In Part 21, Rohan proves the whole 0.7% gap closes record by record, against the payment provider’s settlement files.

This is Part 20 of “Every Record Counts: Data Engineering for Backend Engineers” — Season 2: Pipelines in Production.