"Every Record Counts" — Completeness and Correctness Engineering

Skim Card

One thing to remember: “every record counts” is not a hope but a property you prove every run, with keys that make repeats harmless and totals that must balance against a source you don’t control.

• “100% of records” is four checks: nothing lost, nothing twice, nothing late-but-ignored, every total traced to an independent source.
• Exactly-once is an effect: deliver at least once, make repeats harmless with a business key. Vendor guarantees cover only their own hop.
• Decide once, at ingestion, by natural key plus payload hash; record every verdict and build gold only from accepted events.
• Carry counts, sums and hash totals across each hop you control; where totals first differ is where the bug lives.
• Bridge from money you didn’t record: fees, chargebacks, timing and FX take the PSP’s £30,068,419.08 to gold’s £29,790,674.06, nothing unexplained.
• Gate publish on reconciliation: a lost till upload passed every test and check; the ledger-vs-gold control stopped it (two baskets, £106.11).
• A closed period changes only by restatement: tag what was reconciled, keep “as published” beside “as of now”.

Use …	When …
Dedup inside one consumer (Part 17’s dbt)	One reader, a stable key, and a control behind it
One disposition per record at ingestion	Many readers, re-sends crossing batches, drops that must be counted
A reconciliation gate before publish	The number has an independent source: ledger, PSP, bank, stock
A restatement and a period tag	A closed, published period changes

Verified against: Apache Iceberg 1.12.0, Apache Polaris 1.8.0, PyIceberg 0.12.0, DuckDB 2.0.0, PyArrow 26.0.0, Apache Airflow 3.3.2, dbt-core 1.12.6 + dbt-trino 1.11.0, Trino 484 · vendor docs · 29 October 2026

Jump to: Concepts · Hands-On · Landscape · Design Drill

Story Opening

Part 20 ended with conversion settled. Revenue was not: four of the revenue gap’s five causes had been fixed one part at a time. The fifth, re-sent till baskets, had only been found (Part 6) and filtered out inside dbt’s query; nothing stopped a re-send where data enters. And nobody had shown Farah the whole 0.7% at once.

“Each fix has a test,” Rohan said. “Dedup in dbt, CDC for the deletes, the refund date, the rate, the London date. They all pass. So the number is right.”

“Each test proves its own step,” Ingrid said. “Farah needs a proof that nothing else is wrong. Start from the money the payment provider actually moved, and account for every pound between that and our revenue. Anything you can’t name is a bug you haven’t found yet.”

The Quick Map

Concepts, with use cases

What “100% of records” means

Four properties, each one a check you can run, not a hope:

What breaks without the list. “Accurate” stays an adjective, and every incident starts with an argument. Larkspur: the UK quarter, against the payment provider’s settlement files. Elsewhere: a telecom’s billing run, where every call record from every switch must be rated once, and the switch’s own counters are the independent source.

Idempotency end to end: effectively-once

Records repeat for three different reasons, and each needs its own defence:

Vendor exactly-once is always scoped. Kafka’s idempotent producer “guarantees that resending will not result in duplicate entries in the log”, transactions make writes to several partitions atomic, and Kafka Streams has exactly-once; for other systems, “exactly-once delivery … generally requires cooperation with such systems”. Flink’s exactly-once means “every event will affect the state being managed by Flink exactly once”, and end to end it needs replayable sources and transactional or idempotent sinks. Spark’s foreachBatch gives “only at-least-once write guarantees”. None of them can know that two uploads with different event IDs are one basket. Effectively-once is the honest goal: deliver at least once, and make the effect of a repeat zero with a key.

What breaks without it. Retries are safe in the broker and double counted in the report. Elsewhere: a card network retries an authorisation; the issuer’s idempotency key is what stops a second charge.

Natural and synthetic dedup keys

A synthetic key is minted by a system (an event ID, a UUID per send); a natural key comes from the business event (the basket number printed on the receipt). Larkspur’s tills mint a new event_id for every send, so it is unique even across re-sends and useless for finding them; basket_id is the natural key. A key alone isn’t enough: two different baskets can share a basket_id (a till restored from backup reuses its sequence). Compare a payload hash too: same key and same payload is a re-send, same key and a different payload is a collision, which a human must see.

What breaks without it. ROW_NUMBER() … = 1 on the wrong key keeps one of two real baskets and drops the other, silently. Elsewhere: a hospital lab feed, where a reused accession number with a different result is a safety incident.

Control totals, hash totals and source-to-target reconciliation

A control total is a count and a sum carried from one hop to the next; a hash total is a sum of a hash of each record’s key and amount, so a record swapped for another of the same value still shows. Source-to-target reconciliation compares them per partition at every hop: file to raw, raw to the accepted events, ledger to gold, ledger to the PSP. When totals disagree, the hop where they first disagree is where the bug is. Audit columns (batch ID, source file and line, ingestion time) on every landed row take you from that hop to the file and line, and a replay runbook is the written order for re-running a period from raw once the cause is fixed.

What breaks without it. Every table can pass its own tests after a file was lost before the first one. Elsewhere: a stock exchange reconciles each member firm’s trade count and notional with the clearing house every night, and replays a session from its archived message log when they disagree.

Legitimate differences and bugs

A bridge (accountants say reconciliation statement) starts at one total, adds or subtracts each difference with its reason, and must land on the other total with nothing left over. For Larkspur, the PSP’s settlement files differ from net revenue for four legitimate reasons: fees (a cost of sale), chargebacks (disputes, booked as losses, not refunds), timing (sales settle two business days after the local capture date, weekend sales counting from Monday; the lab’s PSP ignores bank holidays, so the GB files include a 31 August settlement; online orders are captured at fulfilment, days after the order) and FX (the PSP converts euros at the capture date’s rate; Larkspur’s policy is the business date’s). A difference with no reason is a bug, or fraud.

What breaks without it. “Within 0.7%” is accepted as noise, and the five bugs inside it survive. Elsewhere: a ride-hailing marketplace bridges rider charges to driver payouts and the bank statement; tips and payout timing explain the gaps, anything else is a leak.

Late data: windows, reopening, restatement

Every source gets a stated lateness window: how long after a day closes its records may still arrive. Inside it, the day is provisional and the pipeline reopens it on every run (Part 18’s lookback). After it, the day is closed: a late record still lands, but the change is a restatement, recorded with its reason and approval, while the numbers as published stay readable beside the numbers as of now. Larkspur’s windows are measured or contracted, not guessed: till uploads one UTC day (Part 18), re-sends 30 minutes (a dedup window of three upload days), settlement two business days; controls re-check closed days for a week.

What breaks without it. Yesterday’s revenue keeps moving unexplained, or late records are dropped to keep it still. Elsewhere: a statistics office publishes a first estimate, revises it on a calendar, and keeps every vintage.

Under the Hood

Why the bridge sums exactly

The bridge is a full outer join of every gold revenue record (sales by basket or order, refunds by sale and refund time) with every PSP row, plus the ledger for the reason. Each joined pair lands in exactly one bucket:

Summing the buckets is algebra, not tuning, so the bridge can’t “almost” close: every record has a reason, or the bug lines are non-zero.

Where the gate sits

Hands-On

The code is in the every-record-counts folder of the companion-workspaces repository (tag every-record-counts/part-21). Steps 1, 3, 4 and 6 and the acceptance need only core (DuckDB and PyIceberg on the host, as Part 9’s pipeline principal); step 2’s rebuild and explain need Trino, and step 5 runs in Airflow (make up-orchestrate: core, Trino, Airflow). It reads Part 16’s CDC replicas and the gold tables Parts 17 to 19 built. The commands:

uv run python -m scripts.part21_reconcile land-psp          # twice: the second run lands nothing
uv run python -m scripts.part21_reconcile accept --all
uv run python -m scripts.part17_dbt dbt build --target prod --select stg_pos_sale_lines+ --event-time-start 2026-05-31 --event-time-end 2026-10-06   # (Trino)
uv run python -m scripts.part21_reconcile explain           # (Trino)
uv run python -m scripts.part21_reconcile waterfall
uv run python -m scripts.part21_reconcile bridge
make up-orchestrate
uv run python -m scripts.part21_reconcile inject resend --day 2026-09-30   # then hold, then restore (step 5)
make airflow CMD="backfill create --dag-id larkspur_daily_revenue --from-date 2026-09-30 --to-date 2026-09-30 --reprocess-behavior completed --max-active-runs 1"
uv run python -m scripts.part21_reconcile logs reconcile --day 2026-09-30
uv run python -m scripts.part21_reconcile restate

Step 1: land the settlement files, with audit columns

land-psp lands every PSP file (one CSV per country per settlement day) not yet in raw.psp_settlements, or whose bytes changed, in integer minor units with five audit columns: _batch_id (the file), _source_file, _source_offset (line number), _file_md5, _ingested_at (kept apart from transaction_ts). Then it checks each file’s line count, gross total and checksum against the table.

{"files_seen": 267, "files_landed": 267, "rows_landed": 2011995, "seconds": 1.3, "files_mismatched": []}
{"files_seen": 267, "files_landed": 0, "rows_landed": 0, "seconds": 0.0}

One commit; the second run lands nothing. Any row Farah questions points back to a line in a PSP file.

Step 2: accept each POS event once, at the door, and build gold from the accepted

accept reads one upload day of raw.pos_uploads, ranks each basket_id’s copies by upload time and looks the key up among the accepted events of the dedup window’s earlier upload days. Verdict, first_event_id and payload hash go to silver.pos_event_dispositions, partitioned by upload day and replaced whole on a re-run: a job re-run replaces the day with itself, a till re-send becomes a resend row, and raw stays as the tills sent it. A day’s verdicts depend on the days before it, so accept --day D also re-decides any earlier day whose raw count or event-ID hash total no longer matches its dispositions, and the window’s days after D. The core of sql/part21/accept.sql (r: the day’s ranked events; p: the window’s accepted events):

       CASE WHEN p.basket_id IS NULL AND r.copy_no = 1 THEN 'accepted'
            WHEN r.payload_hash = coalesce(p.payload_hash, r.first_hash_in_batch) THEN 'resend'
            ELSE 'collision' END AS disposition,

{"upload_days": 128, "seconds": 69.1}
┌─────────────┬─────────┬───────────┬───────┐
│ disposition │ events  │ uk_events │ tills │
├─────────────┼─────────┼───────────┼───────┤
│ accepted    │ 2329358 │    795834 │  3513 │
│ resend      │    9454 │      2455 │  3253 │
└─────────────┴─────────┴───────────┴───────┘
UK re-sends in Q3 (UTC dates, as the board deck cut them): 2,455 events, GBP 118,076.98
UK accepted baskets 795,834; UK ledger captured till payments 795,834; accepted without a payment 0, payment without an accepted basket 0

(Type row trimmed.) All 2,338,812 events have a verdict, none a collision; the 9,454 re-sends from 3,253 tills are exactly Part 6’s; every accepted UK basket has one ledger payment, and the other way round.

Until now, gold’s only dedup was a ROW_NUMBER() inside dbt’s stg_pos_sale_lines: Part 6 found the re-sends, Part 17 filtered them in one consumer’s query (tag every-record-counts/part-17 keeps that model). Part 21 replaces that ranking with the boundary’s decision:

with accepted as (
    select event_id, event_ts
    from {{ source('silver', 'pos_event_dispositions') }}
    where disposition = 'accepted'
)
…
from {{ source('raw', 'pos_uploads') }} b
join accepted c on c.event_id = b.event_id and c.event_ts = b.event_ts

Joining on event_ts too keeps Part 18’s microbatches cheap: explain shows one day’s event_ts range on both scans, and Part 18’s EXPLAIN test now checks both. Rebuilding production’s facts from it (9 min 24 s, 36 nodes passed, no query over Trino’s per-node limit) gives the same 6,658,738 lines and UK Q3 £29,790,674.06, exact.

So: tills send at least once; acceptance decides once and records each drop; dbt builds gold from accepted events only (its unit test fails if the filter is deleted); and accepted_vs_gold_baskets (step 5) proves each accepted basket is in gold once. Part 20’s lab-only Spark WAP job keeps its own ranking.

Step 3: the waterfall, from records

sql/part21/waterfall.sql tags every record with the fix levels in which it counts (re-sends count only at level 0, poller extracts until fix 2, the CDC replica from fix 2), then sums each level with its own date and rate rule. Step k is total(k−1) − total(k):

UK net revenue, Q3 2026                                                       GBP  manifest
board deck (no fix)                                                 29,999,232.43  exact
- Duplicate POS events (till re-sends after reconnecting)              118,076.98  exact
- Cancelled online orders hard-deleted after the poller saw them        57,064.96  exact
- Refunds dated by original purchase date instead of refund date        42,511.91  exact
- EUR orders and refunds converted at the report-day rate               -8,797.38  exact
- Quarter boundaries in UTC instead of Europe/London                      -298.10  exact
= correct (all five fixes)                                          29,790,674.06  exact
gap = sum of steps                                                     208,558.37  exact; unexplained 0.00; 1.1 s

Every figure equals docs/injected-defects.md to the penny, computed from the lakehouse: step 1 from step 2’s dispositions, the rest from Part 16’s CDC replicas and the FX table.

Step 4: the bridge, from what the PSP settled

A waterfall between two of your own numbers proves they differ by five bugs; it can’t prove the correct number is right. The PSP’s files can, because nobody at Larkspur wrote them, and because Larkspur UK has been cashless since 2024: every UK pound passes through the PSP. (In India and Singapore, cash takings would need their own outside source, such as bank deposits.) bridge runs sql/part21/bridge.sql:

UK, Q3 2026                                                                 records            GBP
PSP net settled in Q3 (settlement date)                                     670,790  30,068,419.08
+ PSP fees: a cost of sale, not less revenue                                652,386     466,606.86
+ chargebacks: disputes, booked as losses, not refunds                          821      39,516.39
timing: June business days, settled in July (settlement lag)                 26,250  -1,253,728.56
timing: June orders, captured in Q3                                           3,191    -147,214.45
timing: Q3 business days, settled in October (settlement lag)               12,790     581,598.17
timing: Q3 orders, captured after 30 September                                  731      35,334.76
FX: euros at the capture date's rate (PSP), the business date's (gold)        3,995         141.81
matched to the penny (pounds)                                               636,533           0.00
= net revenue the settlement files support                                           29,790,674.06
gold.rpt_daily_net_revenue_v2, GB, Q3                                                29,790,674.06  equal
unexplained (BUG lines): 0.00 in 0 records; 3.0 s
Q3 revenue records with no PSP row by 5 October: 205, of which the ledger shows captured after 30 September: 205
step 1: re-sent baskets                 2,455 records: one PSP sale 2,455, no PSP sale 0
step 2: polled, then deleted orders     1,176 records: one PSP sale 0, no PSP sale 1,176, ledger payment voided 1,176

Every pound has a reason. Pairs must match in the currency the customer paid; only euro pairs may then differ in pounds, by FX, and a pound pair that differed would land on a BUG line. The 205 Q3 orders with no PSP row yet were all captured in October, so they’re timing, not missing. Gold matches the settlement files with no bug line, and the last two lines are the board deck’s extra records looked up outside Larkspur: each re-sent basket was paid once; each deleted order’s payment was voided and never settled. That is what “0.7% above what the settlement files support” means, record by record.

Step 5: a daily control table, and a gate before publish

control --day D writes one row per check, day and country to gold.daily_controls: counts, money and hash totals of source and target, pending items and a status. It re-checks the last seven closed UTC days on every run, against the PSP files that had arrived by D+1; raw_vs_dispositions covers upload days D−7 to D, and raw_vs_gold_stock and files_vs_raw_clicks carry the stock snapshot and clickstream control totals. ledger_vs_gold_sales rests on an assumption the generator makes true: a card payment reaches the ledger when the terminal authorises it (authorised_at is the sale’s event_ts), even while the till’s link to head office is down. Store-and-forward terminals, which also hold authorisations while offline, would break it: the ledger would be as late as the till. Part 21 adds three tasks to Part 19’s DAG (edited in place; tag every-record-counts/part-19 keeps the original, and tests/test_part19.py now expects eight tasks):

    reconcile = BashOperator(
        task_id="reconcile",
        bash_command=f"{LAB21} control --day {{{{ ds }}}}",
        pool="lakehouse",
        retries=0,  # a difference in the books is not transient
        on_failure_callback=larkspur_alerts.reconciliation_failed,
    )
    …
    wait_for_uploads >> ingest >> accept >> dbt_build >> quality_checks >> reconcile >> publish()
    land_settlements >> reconcile

A re-send. inject resend makes a UK till re-send, at 00:07, the last basket it uploaded on 29 September (GB-0134-20260929-00010, £4.78), into the next upload day: the case a within-batch dedup misses. The 30 September run, as logs prints it (run IDs and try numbers trimmed from the header lines):

accept_pos_events: success
{"upload_date": "2026-09-30", "received": 16918, "resend": 65, "accepted": 16853}
…                    (1 to 3 October re-decided, unchanged)
reconcile: success
│ accepted_vs_gold_baskets │    21 │    21 │     0 │       0 │   NULL │
│ ledger_vs_gold_sales     │    21 │    21 │     0 │       0 │   NULL │
│ ledger_vs_psp            │    21 │    21 │     0 │     331 │   NULL │
│ resends                  │     3 │     0 │     0 │       0 │     65 │
0 failed; 3.6 s

(Seven passing checks trimmed.) The re-send is recorded (64 real ones that day, plus ours), dbt never saw it, gold holds each accepted basket once and agrees with the ledger on every closed day, and the report published. tests/test_part21.py repeats this on 2 October and asserts the new event’s first_event_id and an unchanged accepted set.

A lost upload. inject restore takes the re-send out again; then inject hold removes one UK till’s upload from 30 September’s file (GB-0164-T09, 14 events), as if it had landed under the wrong prefix, and the same backfill create command re-runs the day. Every earlier check passes: the raw table holds what the file holds, dbt’s grain tests pass, Part 19’s checks pass. Then:

reconcile: failed
│ accepted_vs_gold_baskets │    21 │    21 │     0 │       0 │   NULL │
│ ledger_vs_gold_sales     │    21 │    20 │     1 │       0 │   NULL │
FAIL ledger_vs_gold_sales 2026-09-29 GB: source 6191, target 6189, money difference -106.11
1 failed; 3.6 s
publish: upstream_failed
RECONCILIATION FAILED: larkspur_daily_revenue for upload day 2026-09-30; publish withheld. See gold.daily_controls

(Passing checks trimmed; the last line is the failure callback’s entry in the task log.) Two of the till’s baskets (one sent twice) were sales of 29 September, a closed day. Gold matches what was accepted, so dedup is fine; the ledger has two payments gold lacks, and the PSP agrees with the ledger: the loss is ours, before landing. The other eleven are 30 September’s, which tomorrow’s run checks. The published tag stayed where it was.

The replay runbook, run here as written: (1) find the first hop where totals differ in gold.daily_controls (here: before raw, so landing); (2) put the records where ingestion reads (inject restore); (3) re-run the upload day from ingestion, safe because every task replaces what it owns, and accept re-decides that day and the window’s three days after it, whose verdicts were made against its accepted set: make airflow CMD="tasks clear larkspur_daily_revenue -s 2026-09-30 -e 2026-09-30 -t ingest_pos_uploads -d -y"; (4) if the period was already published and closed, the re-run proposes a restatement (step 6); (5) close the incident on the passing control rows.

accept_pos_events: success
{"upload_date": "2026-09-30", "received": 16917, "resend": 64, "accepted": 16853}
reconcile: success
0 failed; 3.3 s
publish: success

(The three re-decided days and the check lines trimmed; all pass.) 16,917 events: the day’s file as generated, without the re-send inject restore removed before the hold, and the same 16,853 accepted baskets. The report published.

Step 6: restatement, as published vs as of now

restate tags the report snapshot the bridge reconciled (q3_2026_reconciled; unlike Part 19’s daily published, it never moves) and writes the restatement to gold.restatements with status proposed and the five steps as reasons; approval comes with Farah’s sign-off of the quarter (Part 25). “As published” here is the legacy board-deck figure, not a tag: it never lived in the lakehouse.

board deck (as published)                     29,999,232.43
tag published (Part 19, moves daily)          29,790,674.06
tag q3_2026_reconciled (never moves)          29,790,674.06
main (as of now)                              29,790,674.06
restatement proposed in gold.restatements: -208,558.37 (snapshot 4958331260561840512)

From this quarter on, the period’s tag is “as published”: a late record for a closed, published period lands in main, the tag keeps the reported figure, and the difference becomes a proposed restatement row, never a silent change.

The Landscape

From vendor documentation, 29 October 2026.

Tips & Gotchas

Gotcha — a dedup window is also a hole. The acceptance step looks back over its window, and accepted_twice checks the same window. A copy that arrives a week late is accepted as a new basket and gold counts it twice. accepted_vs_gold_baskets still agrees (both hold it); ledger_vs_gold_sales doesn’t: two sales, one payment. Size windows from measured delays, and keep an independent control behind them.

Gotcha — XOR makes a poor hash total (reasoning, not run in the lab). XOR is order-independent like a sum, but a record XOR-ed in twice cancels itself: {a, a, b} gives b, exactly what a batch that lost a gives, and a third copy brings a back. A duplicate looks like a loss, or like nothing. The controls reduce each record’s hash modulo a prime (1,000,000,007) and add the results, so every copy adds its own term.

Gotcha — “pending” is not “missing”. Thursday’s card sales settle on Monday. A control without the PSP’s two-business-day contract fails every Friday, or gets a tolerance that also hides real losses. The control computes each record’s due date and checks the contract itself (psp_settlement_lag).

Gotcha — PyIceberg’s to_arrow() reads whole data files before filtering. One day of the stock fact took 1.9 GiB (three files, 11.7 million rows) in a task whose container has 1.5 GB. The control streams each file batch by batch with PyArrow and filters as it goes: 463 MiB for the whole run. Files with delete files still go through PyIceberg, which applies them.

Troubleshooting

Design Drill

The question: “A national vaccination registry receives dose records from 3,000 clinics, some offline for days. Every dose must be counted exactly once. Design it.”

Clarify first

• What identifies a dose: a clinic record ID, a national person ID, a lot? Can a clinic and a pharmacy both record one dose?
• How late can records arrive, and what is published when (daily dashboard, weekly official figures)?
• How are errors corrected: voided doses, edited dates, merged patient records?
• What independent count exists: vials used, clinics’ daily tallies, reimbursement claims?

The shape of a strong answer

• Store-and-forward clients with a per-clinic sequence number on every record, sent at least once with an idempotency key; the registry acknowledges by sequence, so a gap is visible and resend is safe.
• An acceptance step with one disposition per record: accepted, re-send (same key and payload), correction (same key, a newer version), collision (same key, different dose: to a review queue). The transport key catches re-sends; a business key (person, vaccine, dose number, date) catches one dose entered by two systems.
• Corrections as new versions (bitemporal: administration and recorded dates), on a raw log kept as received, so any report reproduces as published.
• A daily control table: per clinic, records received against its declared count and highest sequence; doses against vials used per lot (wastage legitimate, an unexplained gap not); a silent clinic is an alert.
• Late-data policy: facts by administration date, reopened for an agreed window (say 30 days) and marked provisional; official figures tagged; later changes published as revisions with reasons.
• A gate before publish: statistics don’t publish while a control fails or a collision is unreviewed.

Trade-offs to say out loud

• Strict person matching risks merging two people (a missed dose); loose matching double counts. Collisions go to humans, and the policy says which error is worse.
• Sequence numbers need clinic software changes; without them, completeness rests on declared counts and stock.

Common weak answers

• “Use Kafka exactly-once”: the duplicates come from clinics re-sending, not from the broker.
• Deduplicating on a UUID minted per send.
• No independent count: a clinic that never uploads looks the same as a clinic that vaccinated nobody.

Further Reading

1. Apache Kafka 4.4 documentation, “Message Delivery Semantics” (idempotent producer, transactions, exactly-once). https://kafka.apache.org/44/design/design/
2. Apache Flink documentation, “Fault Tolerance via State Snapshots” (exactly once vs end-to-end exactly once). https://nightlies.apache.org/flink/flink-docs-stable/docs/learn-flink/fault_tolerance/
3. Tyler Akidau et al., “The Dataflow Model” (VLDB 2015): windows, triggers and accumulation for late data. https://research.google/pubs/the-dataflow-model-a-practical-approach-to-balancing-correctness-latency-and-cost-in-massive-scale-unbounded-out-of-order-data-processing/
4. Apache Iceberg documentation, “Branching and Tagging” (retained tags for audited, published versions). https://iceberg.apache.org/docs/latest/branching/

Story Closing

“So the 0.7% was five bugs, and nothing else,” Rohan said.

“And you can show it with somebody else’s numbers,” Ingrid said. “That’s the difference between a fix and a proof.”

Of the five promises from Part 2, this part keeps completeness and uniqueness, the two Part 14’s review left partial: every record has a verdict, each hop you control a control total, and the gate checks both against the ledger and the PSP. Correctness gained its proof: £208,558.37 explained to the penny, nothing left over.

Farah read the bridge twice. “I’ll sign the quarter when I’ve seen the whole platform end to end.” First, in Part 22, Rohan has to know a number is wrong before Farah does: observability, lineage and SLAs.

This is Part 21 of “Every Record Counts: Data Engineering for Backend Engineers” — Season 2: Pipelines in Production.