Project Every Record Counts: Data Engineering for Backend Engineers
Part 20: "Trust, but Test" — Data Quality and Data Contracts
Rohan wants to add null checks to every dashboard query. Ingrid wants the producers and consumers to write down what the data promises, and a pipeline that tests the promise where data crosses a boundary. Quality dimensions, where to test, quarantine vs fail-fast, ODCS 3.2 data contracts tested with datacontract-cli, a local CI gate for breaking changes, and Write-Audit-Publish on an Iceberg branch in Spark. The session contract settles online conversion at 6.21%.