DataQuality
3 items
Series Posts
Part 20: "Trust, but Test" — Data Quality and Data Contracts
Rohan wants to add null checks to every dashboard query. Ingrid wants the producers and consumers to write down what the data promises, and a pipeline that tests the promise where data crosses a boundary. Quality dimensions, where to test, quarantine vs fail-fast, ODCS 3.2 data contracts tested with datacontract-cli, a local CI gate for breaking changes, and Write-Audit-Publish on an Iceberg branch in Spark. The session contract settles online conversion at 6.21%.
Part 21: "Every Record Counts" — Completeness and Correctness Engineering
Farah wants the 0.7% explained to the record. Rohan proves it: the board-deck waterfall closes to the penny, every pound the payment provider settled for the UK is traced to gold revenue or to a legitimate reason, and re-sent till baskets are decided once at the door and kept out of gold. Completeness and uniqueness as engineered properties: natural dedup keys, effectively-once delivery, control and hash totals at every hop, a daily control table gating publish in Airflow, late-data policy, restatement and a replay runbook.
Part 25: "Design Review II" — The Batch Reporting Platform, End to End
Season 2 ends where it began: with Farah and a quarter to sign. Rohan defends Larkspur's batch platform end to end in interview order, from sources and ingestion through contracts, reconciliation, governance and cost: sized from the lab's own tables, with every failure mode named against the part that caught it, what changes at ten times the volume, and what still isn't finished. Then the board pack is recomputed from the lakehouse and the restatement approved. Plus a self-host vs managed matrix, a printable checklist and a clinical-trials drill.