"Getting Data In" — Batch Ingestion: Files, APIs and Databases

Skim Card

One thing to remember: ingestion is a contract with a system you don’t control; design every load for the day the other side changes, fails or sends it twice.

• Land every row as text with its file, line and load ID; type, validate and deduplicate in a layer rebuilt from raw.
• Account for every row: 23,589 landed = 23,387 valid + 81 quarantined + 121 repeats, matching the manifest’s 81 malformed and 121 duplicates.
• A timestamp cursor misses three things: a slow clock’s late stamps, updates that skip updated_at, and deletes.
• Re-read a lag window below the mark and upsert by key: 15 minutes caught three slow-clock updates without doubling a row.
• Only a full extract or CDC finds silent updates and deletes: the reconciliation found one deleted seller and two stale commission rates.
• Freeze the columns where a rename would become NULLs: S017’s new header stopped the load; a declared alias fixed it.
• dlt isn’t Databricks’ DLT. dlt 1.31 writes Iceberg through PyIceberg’s REST catalog (Polaris needed a small shim); upserts commit per 1,000 rows.

Use …	When …
Full extract	The source is small, its change signal can’t be trusted, or as the periodic check on an incremental one
Incremental cursor + lag window + upsert by key	The source is large and offers a mostly reliable updated_at
CDC	You can read the database’s log (Part 16)
Append raw, deduplicate downstream	Files and events, where what arrived, and when, is evidence

Verified against: dlt 1.31.0, PyIceberg 0.12.0, Apache Polaris 1.8.0, SeaweedFS 4.51, DuckDB 2.0.0 · dlt docs, RFC 6585, RFC 9110 · vendor docs · 28 October 2026

Jump to: Concepts · Hands-On · Landscape · Design Drill

Story Opening

Season 1 left Larkspur with a lakehouse whose queries Ingrid trusted. Season 2 opened with Farah forwarding an email from supplier S017: “Your supplier scorecard says we’ve delivered nothing since 20 August. We delivered everything.”

Rohan had built a dozen integrations as a backend engineer. “A cron job lists the bucket, parses each new CSV and inserts the rows. For the marketplace sellers, GET /sellers?updated_since= the last run’s time. An afternoon.”

“That’s how the scorecard’s job was built,” Ingrid said. “S017 renamed a column on the 20th. The job didn’t know the new header, filled the quantity with NULL and carried on. Ingestion isn’t a client calling an API. It’s a contract with a system you don’t control, and that system changes when it likes.”

The Quick Map

Concepts, with use cases

Ingestion is a contract you didn’t write

Ingestion copies data from systems you don’t own into one you do, on a schedule, while they change. The other side decides file names, headers, date formats, page sizes, rate limits and what a timestamp means. So an ingestion job is a treaty more than a client call: write down what you expect, check it every run, and decide in advance what happens when it’s broken.

What breaks without it. Every unwritten assumption fails silently: a rename becomes NULLs, a deleted record lives on in your copy. Larkspur: forty suppliers and a marketplace API, none of which ask before changing. Elsewhere: a hospital network ingesting lab results from a dozen vendors’ exports.

Full vs incremental extraction

A full extraction reads everything the source has, every run: simple, catches deletes, costs the whole source each time. An incremental one reads only what changed since the last run, using a cursor and a change signal on the source (a timestamp, a version, a log): cheap, and only as correct as its signal. Databases offer the same choice: SELECT * nightly, or WHERE updated_at > :mark.

What breaks without the distinction. Incremental alone inherits its signal’s blind spots; full alone hits rate limits as the source grows. The usual answer is both: incremental often, full rarely, and a comparison of the two. Larkspur: the sellers API (step 6); also the legacy order poller, whose missed deletes are £57,064.96 of the 0.7% gap (Part 16). Elsewhere: an insurer pulls changed policies nightly and the whole book monthly.

High-water marks and their three traps

Part 6 defined the high-water mark: the largest cursor value loaded, from which the next run continues. With updated_at as the cursor, three things break it:

1. Clock skew. updated_at is stamped by whichever application server made the change, not by the database’s commit order. A node with a slow clock, or a transaction that commits after a later-stamped one, writes a value below a mark you’ve already recorded, and updated_at >= mark never returns it.
2. Changes that don’t touch updated_at: a bulk UPDATE run by hand, a code path that skips the timestamp.
3. Deletes. A hard-deleted row has no updated_at at all.

Remedies, cheapest first: re-read a lag window below the mark and load idempotently by key, so re-reads replace instead of doubling (fixes 1); reconcile against a periodic full extract (finds 2 and 3); read the database’s log with CDC where you can (fixes all three). Larkspur: the sellers API has all three (step 6). Elsewhere: a ride-hailing app’s driver-rating sync that misses ratings a nightly batch recalculated.

Landing zones, file naming and an immutable raw zone

A landing zone is where outsiders put files (a bucket prefix, an SFTP folder); its naming convention says who sent it for which day (S017_20260820.csv). The raw zone keeps every file and row as received, append-only, for as long as retention allows; parsing, typing and deduplication happen in a next layer rebuilt from raw.

What breaks without it. Fix a file in place and you can’t prove what the supplier sent or re-run with better parsing. Larkspur: files land under raw/suppliers/deliveries/<supplier>/; raw.supplier_deliveries is append-only and silver is rebuilt from it every run. Elsewhere: a bank keeps every payment file it received, byte for byte, for the regulator.

Pulling from an API: pages, limits, retries, idempotency keys

• Pagination. Offset pages (?page=7) shift when rows are inserted or reordered while you page, so you skip or repeat rows. Keyset (cursor) pages ask for “after (updated_at, id) = X” and can’t shift; a row updated mid-paging moves past the cursor and is read again, which an upsert absorbs.
• Rate limits. The API answers 429 Too Many Requests (RFC 6585), often with Retry-After in seconds or as a date (RFC 9110). Honour it; otherwise back off exponentially with jitter and cap the attempts.
• Idempotency keys. Retrying a write is safe only if the request carries a client-generated key the server uses to recognise the retry. Ingestion meets this when it must ask for data: “create an export job” is a POST, and a retried POST without a key starts two. The Idempotency-Key header is an IETF Internet-Draft (revision 08, 22 October 2026), not an RFC; payment APIs have used the pattern for years.
• Idempotent loads. The extract may repeat; the load must not double: upsert by key, or append to raw and deduplicate downstream.

What breaks without them. Offset pages lose rows in busy hours; a retried export doubles a day. Larkspur: 12 keyset pages, two 429s retried (step 5); if the sellers API made Larkspur create an export job first, that POST would carry a key. Elsewhere: an ad-tech firm pulling campaign reports from platforms with different quotas.

Schema drift

Schema drift is the source’s shape changing: a column renamed, added, removed or retyped. The policies, permissive to strict: evolve (add the column, carry on), rescue (park unknown fields in a catch-all column), discard (drop the row or value) and freeze (fail the load). Renames are the nasty case: under evolve, the new name becomes a new, mostly NULL column beside an old one that stops filling, and nothing fails.

What breaks without a policy. The scorecard’s NULL quantities: a supplier who delivered everything shows a fill rate of zero. Larkspur: S017 renames Delivered Qty to Qty Delivered on 20 August (step 3). Elsewhere: a game studio whose SDK upgrade renames level_id in half its players’ events.

Load metadata

Stamp every row with a load (batch) ID, its source file or request, its position (row, page) and ingestion time, kept apart from event time; keep one record per load.

What breaks without it. A wrong number can’t be traced to a file, and a bad load can’t be removed cleanly. Larkspur: _dlt_load_id, _source_file, _row_number and _ingested_at on every supplier row (step 7). Elsewhere: a utility tracing a billing error to one meter vendor’s upload.

Under the Hood

What a dlt run does

pipeline.run(source) is three steps. Extract calls your resource functions and writes what they yield, plus any state they changed (the sellers cursor, the list of loaded files), into a load package named by its load ID, a Unix timestamp. Normalize infers or checks the schema and applies the schema contract, writing Parquet for Iceberg tables. Load writes each table, the new state among them, and finally a record in _dlt_loads. State travels with the data it describes, so a run that dies early leaves the cursor where it was. A failed package stays pending: the next run() normalizes and loads pending packages and exits without extracting the data you passed it. One that can never succeed (a frozen contract) must be aborted with pipeline.abort_packages(), which also restores the local state.

How dlt writes Iceberg, and where it bumped into Polaris

With table_format="iceberg", dlt’s filesystem destination hands the Parquet to PyIceberg, which commits through the catalog: any PyIceberg catalog, so Polaris over REST works. The table lives at <bucket_url>/<dataset>/<table>, and the dataset name is the namespace. append becomes table.append(), one snapshot per load; replace becomes table.overwrite(); merge (upsert or insert-only) becomes PyIceberg’s upsert, one call (one commit) per 1,000 rows (step 6). dlt’s bookkeeping (_dlt_loads, _dlt_pipeline_state, _dlt_version) is not Iceberg: small JSONL files, plus an init marker, next to the tables, written with the destination’s own storage key. Credential vending covers the tables only.

One seam needed code. dlt 1.31 calls create_namespace on every load and accepts only “already exists” (409). Polaris checks the grant before existence and answers 403, and Part 9’s rule is that pipelines never create namespaces. The fix is a catalog class that asks first, plugged in through PyIceberg’s py-catalog-impl:

class PolarisRestCatalog(RestCatalog):
    def create_namespace(self, namespace, properties=EMPTY_DICT) -> None:
        if self.namespace_exists(namespace):
            raise NamespaceAlreadyExistsError(f"Namespace already exists: {namespace}")
        super().create_namespace(namespace, properties)

The alternative is granting the pipeline NAMESPACE_CREATE on the whole catalog.

Hands-On

The code is in the every-record-counts folder of the companion-workspaces repository (tag every-record-counts/part-15). It needs only the core profile: dlt runs on the host and writes its tables as Part 9’s pipeline principal, which holds no storage key; Polaris vends one per table. dlt’s bookkeeping files still use the lab’s static S3 key, one of the keys Part 14 said must go before production.

make up-core
make sellers-api &                                                         # the mock API, on :8765
uv run python -m scripts.part15_ingest reset
uv run python -m scripts.part15_ingest suppliers --through 2026-08-19     # step 1 (twice: step 2)
uv run python -m scripts.part15_ingest suppliers --through 2026-08-20 --ignore-aliases  # step 3
uv run python -m scripts.part15_ingest suppliers
uv run python -m scripts.part15_ingest checks                             # step 4
uv run python -m scripts.part15_ingest sellers --pull first               # step 5 (and --lag 0)
uv run python -m scripts.part15_ingest sellers --pull second              # step 6 (and --lag 0)
uv run python -m scripts.part15_ingest reconcile
uv run python -m scripts.part15_ingest loads                              # step 7

Step 1: land the supplier files

The landing zone holds 2,512 files from 40 suppliers: 2,155 CSV and 357 XLSX workbooks from five of them. What Larkspur knows about the senders is declared in ingest/supplier_feed.json: the file-name pattern, the canonical header, per-supplier aliases and date formats (seven suppliers write DD/MM/YYYY). The resource (trimmed) remembers each loaded file’s ETag in dlt state and lands every row as text, plus one row per file, header as sent, in raw.supplier_files:

@dlt.resource(
    name="supplier_deliveries",
    table_format="iceberg",
    write_disposition="append",  # raw is append-only: a re-sent file is a new version, never an overwrite
    schema_contract={"columns": "freeze"},  # an unknown header stops the load instead of adding a column
)
def deliveries():
    loaded = dlt.current.resource_state().setdefault("files", {})  # path -> ETag of the version loaded
    for f in landing_files():
        ...  # trimmed: skip bad names and known versions, read the file, apply aliases, one row to supplier_files
        yield [
            {**dict(zip(names, row, strict=True)), **meta, "_row_number": n, "_ingested_at": ingested_at}
            for n, row in enumerate(rows, start=2)  # line 1 is the header
        ]
        loaded[f["path"]] = f["etag"]

--through replays arrival: the first run takes files dated up to 19 August.

through 2026-08-19: load 1793184459.151907: 1,561 new files, 14,645 new rows
  raw.supplier_files           1,561 rows  1 snapshots
  raw.supplier_deliveries     14,645 rows  1 snapshots

Step 2: run it again

through 2026-08-19: load (none): 0 new files, 0 new rows
  raw.supplier_deliveries     14,645 rows  1 snapshots

(Trimmed.) No new files, no load, no snapshot. A corrected file re-sent under the same name has a new ETag and lands again beside the old version, under a new load ID (tests/test_part15.py checks it); an identical re-upload is skipped.

Step 3: 20 August, the renamed column

--ignore-aliases lands the next day as if S017’s alias weren’t declared yet:

FAILED: NormalizeJobFailed: In schema `larkspur_suppliers`: In Table: `supplier_deliveries` Column: `qty_delivered` . Contract on `columns` with `contract_mode=freeze` is violated. Can't add table column `qty_delivered` to table `supplier_deliveries` because `columns` are frozen. Offending data item: _dlt_id: A+uDOAhfV9Kxtg.

Nothing landed, and the script aborted the package, so the files aren’t marked as loaded. With the alias declared ("S017": {"Qty Delivered": "Delivered Qty"}), the last run lands everything else, including a re-sent row dated 5 October, a day after the generator’s window.

through the latest file: load 1793184473.164045: 951 new files, 8,944 new rows
  raw.supplier_files           2,512 rows  2 snapshots
  raw.supplier_deliveries     23,589 rows  2 snapshots

Step 4: account for every row

sql/part15/supplier_checks.sql (DuckDB, over the raw table read through PyIceberg) parses each value with the supplier’s declared date format, gives each row at most one issue, and numbers identical rows from the same supplier in file order:

CASE WHEN po IS NULL           THEN 'missing PO number'
     WHEN line_no IS NULL      THEN 'PO line not a number'
     WHEN qty IS NULL          THEN 'quantity not a number'
     WHEN qty < 0              THEN 'negative quantity'
     WHEN delivered_on IS NULL THEN 'date not valid in the declared format'
END                                                                             AS issue,
row_number() OVER sent_as                                                       AS copy_no,
...
WINDOW sent_as AS (PARTITION BY supplier_id, po_number, po_line, sku, delivered_qty, delivery_date, dc
                   ORDER BY file_date, _source_file, _row_number);

First copies without an issue go to silver.supplier_deliveries; every row with an issue goes to silver.supplier_delivery_quarantine with its file and line; both are rebuilt from raw each run.

│ landed in raw.supplier_deliveries                    │ 23589 │
│   first copies, valid -> silver                      │ 23387 │
│   first copies, quarantined                          │    81 │
│   repeats in the same file                           │    57 │
│   repeats in a later file                            │    64 │
│   (repeats of quarantined rows)                      │     2 │
│ valid PO lines delivered twice with different values │     0 │

│                 issue                 │ rows  │ distinct_rows │ suppliers │ example_as_sent │
│ missing PO number                     │    23 │            23 │        14 │ (empty)         │
│ negative quantity                     │    23 │            22 │        14 │ -120            │
│ quantity not a number                 │    20 │            19 │        13 │ n/a             │
│ date not valid in the declared format │    17 │            17 │        13 │ 2026-09-31      │

│                             header                             │ files │ suppliers │ first_file_date │ last_file_date │
│ PO Number | PO Line | SKU | Delivered Qty | Delivery Date | DC │  2482 │        40 │ 2026-06-04      │ 2026-10-05     │
│ PO Number | PO Line | SKU | Qty Delivered | Delivery Date | DC │    30 │         1 │ 2026-08-20      │ 2026-10-03     │

(Trimmed.) Every count matches docs/injected-defects.md: 81 malformed rows, 121 duplicates (57 + 64, two of them copies of malformed rows, so quarantine holds 83), 30 renamed-header files, and 23,387 + 81 + 121 = 23,589. Part 4 rejected the same 83 rows; now each has a reason, a file and a line to send back. No supplier contradicted itself about a PO line.

Step 5: the sellers API, first pull

scripts/part15_sellers_api.py is a small, deterministic stand-in for the marketplace’s API (datagen/sellers.py, seed 42). It pages with a keyset cursor, answers 429 with Retry-After: 1 after every five pages, and has a simulated clock the lab moves from the first pull (1 October, 06:00 UTC) to the second (2 October, 06:00 UTC). The incremental resource (excerpt; the full extract is a second resource with write_disposition="replace"):

@dlt.resource(
    name="marketplace_sellers",
    table_format="iceberg",
    primary_key="seller_id",
    write_disposition={"disposition": "merge", "strategy": "upsert"},  # a re-read row replaces, never doubles
)
def changed(
    updated_at=dlt.sources.incremental("updated_at", initial_value=FIRST_SEEN, lag=lag or None),
):
    since = updated_at.start_value  # the stored high-water mark minus the lag
    for page in client.paginate("/v1/sellers", params={"updated_since": iso(since), "limit": 100}):
        yield typed(page)

client is dlt’s RESTClient with a JSONResponseCursorPaginator; it retries 429 and 5xx answers, honouring Retry-After (five attempts by default). lag is 900 seconds; --lag 0 writes an experiment table, part15.marketplace_sellers, as the root principal (lab namespace).

pull first at 2026-10-01T06:00:00Z (lag 900 s) into raw.marketplace_sellers
  incremental: updated_since=2024-01-01T00:00:00Z
  1,200 rows in 12 pages, 2 answered 429 and retried, 3.7 s
  raw.marketplace_sellers: 1,200 rows, 2 snapshots:
    append +1,000 | append +200

Step 6: the second pull, and what the cursor missed

Between the pulls, the API’s owners created 24 sellers and updated 40; an application node with a clock 10 minutes slow updated 3 more just after the first pull; a bulk fix changed 2 commission rates without touching updated_at; and one seller was hard-deleted.

pull second at 2026-10-02T06:00:00Z (lag 900 s) into raw.marketplace_sellers
  incremental: updated_since=2026-10-01T05:43:17Z
  72 rows in 1 pages, 0 answered 429 and retried, 1.8 s
  raw.marketplace_sellers: 1,224 rows, 5 snapshots:
    append +1,000 | append +200 | overwrite -1,200 +1,152 | append +48 | append +24
  full extract: raw.marketplace_sellers_full 1,223 rows (replace)
pull second at 2026-10-02T06:00:00Z (lag 0 s) into part15.marketplace_sellers
  incremental: updated_since=2026-10-01T05:58:17Z
  64 rows in 1 pages, 0 answered 429 and retried, 1.7 s

(Trimmed.) The first pull’s mark was 05:58:17. Without lag, the pull read the 64 rows stamped since (24 new + 40 updated). With lag it read 72: those, the 3 slow-clock updates stamped 05:51–05:54, and 5 rows it already had, which the upsert replaced rather than doubled. reconcile compares each replica with the full extract by key and by a hash of every field:

== raw (lag 900 s) vs the full extract
│ gone from the API (deleted at source)      │       1 │ MS-00018           │
│ stale: changed without touching updated_at │       2 │ MS-00772, MS-00840 │
== part15 (no lag) vs the full extract
│ gone from the API (deleted at source)      │       1 │ MS-00018                     │
│ stale: changed without touching updated_at │       2 │ MS-00772, MS-00840           │
│ stale: missed a change, updated_at moved   │       3 │ MS-00442, MS-00722, MS-00815 │

(Trimmed.) The lag window fixed the clock skew; no cursor can find the other two, and only the full extract did. Refreshing the replica from it closes them; for a database Larkspur owns, CDC does it continuously.

The snapshot lists show how the upsert worked. Each 1,000-row batch is one PyIceberg upsert call, one commit: the first pull’s 1,200 rows were two (1,000 + 200), each a single append. The second pull’s 72 rows were one commit of three snapshots: an overwrite that deleted 1,200 rows and rewrote 1,152, an append of the 48 changed rows (40 + 3 + 5 re-read: a new _dlt_load_id makes even an unchanged row “changed”), and an append of the 24 new ones. Upserting 72 rows rewrote the whole table: fine at 1,224 sellers; at millions, partition the table or let a SQL engine run the MERGE.

Step 7: load metadata

│      load_id      │       ingested_at        │ files │ rows  │ first_file_date │ last_file_date │     snapshot_id     │ added_rows │
│ 1793184459.151907 │ 2026-10-28 10:47:38+00   │  1561 │ 14645 │ 2026-06-04      │ 2026-08-19     │ 4290474291969260085 │      14645 │
│ 1793184473.164045 │ 2026-10-28 10:47:52+00   │   951 │  8944 │ 2026-08-20      │ 2026-10-05     │ 2824731144148710278 │       8944 │

(Trimmed.) Each successful load is one snapshot whose ID is on every row it wrote: “remove that load” is a delete by _dlt_load_id, and “which file did this row come from” is a column. tests/test_part15.py replays the whole sequence and asserts every count against the manifest and the files on disk (14 passed in 35 s).

The Landscape

Build or buy. Ingestion code is mostly connector plumbing: pagination, auth, retries, schema inference, state. Libraries such as dlt give you the pieces inside your own code; platforms run connectors for you. Buy common connectors (a CRM, Postgres, S3); build for odd or in-house sources, as Larkspur does for forty suppliers’ spreadsheets. The contract, raw zone and reconciliation stay yours either way. As of 28 October 2026, from each vendor’s docs:

Tips & Gotchas

Gotcha — dlt is not Databricks’ DLT. dlt (data load tool) is dltHub’s open-source Python library for loading data. Databricks’ Delta Live Tables, once called DLT, is now Lakeflow pipelines (“Spark Declarative Pipelines on Lakeflow” in its docs), and its old Python module was also import dlt. Same three letters, unrelated products.

Gotcha — updated_at is an application’s clock. It’s set by whichever server handled the request, so it’s neither monotonic nor the commit order. Treat a timestamp cursor as a hint plus a lag window, never as a log position.

Gotcha — never infer DD/MM from the data. 05/06/2026 parses both ways; only a format declared per sender is safe. A date that doesn’t parse in the declared format is quarantined, not guessed.

Gotcha — a frozen contract fails the whole run. S017’s header stopped 19 other suppliers’ files dated 20 August. Where senders are independent, check headers per file before loading and quarantine just that file; keep the frozen contract as the backstop.

Tip — test against a fake of the source. A small, deterministic stand-in with the traps built in (a slow clock, a silent update, a delete, a 429) turns “the cursor might miss things” into an assertion.

Troubleshooting

Design Drill

The question: “You run a payroll platform. Every hour you pull employees, timesheets and pay adjustments from the HR systems of 300 client companies through their vendors’ REST APIs; each tenant has its own credentials, rate limits and API version. Payroll must never miss a timesheet and never pay one twice. Design the ingestion.”

Clarify first

• What change signal does each API offer: updated_since, an event feed, webhooks, nothing? Does it expose deletes?
• Are rate limits per tenant or per vendor, and how long does a full extract of the largest tenant take?
• How late can a timesheet be edited, and does an edit after payroll close change pay?
• Are record IDs stable per tenant? Unique across tenants?

The shape of a strong answer

• One job per (tenant, entity), each with its own cursor state, so one tenant’s outage or 429s don’t block 299 others; per-vendor concurrency limits in the scheduler.
• Keyset pagination where offered; honour Retry-After, back off exponentially with jitter, cap attempts, then alert.
• Cursor = updated_at minus a lag window sized from the worst skew observed, with an idempotent upsert keyed by (tenant, entity, record ID): re-reads replace, never double.
• Land every response page raw and append-only, with tenant, load ID, request window and received-at; a parsing bug is fixed by replaying raw, not by re-calling 300 APIs.
• Periodic full extracts per tenant (nightly for small ones, weekly for large), reconciled by key and row hash: deletes and silent updates found, counted and applied.
• Per-tenant schema contracts with pinned API versions; a drifted tenant is quarantined alone.
• Before payroll, per-tenant completeness checks (row counts and hash totals per pay period, against the source’s own totals where offered); payroll runs only for tenants that pass.

Trade-offs to say out loud

• A wider lag window spends API quota to buy safety against skew.
• Full extracts catch what cursors miss but take hours for large tenants; spread them across the week.
• Webhooks cut latency but arrive at least once and unordered: still deduplicate, still reconcile.

Common weak answers

• “Store the last run time and ask for everything updated since”: all three high-water-mark traps at once.
• One global loop over tenants, so tenant 17’s timeouts delay the rest.
• Exactly-once claimed from retries, with no key to deduplicate on.

Further Reading

1. dltHub, dlt documentation: “Incremental loading” (cursor and lag), “Schema and data contracts”, and the “Iceberg” table format. https://dlthub.com/docs/general-usage/incremental/cursor · https://dlthub.com/docs/general-usage/schema-contracts · https://dlthub.com/docs/dlt-ecosystem/destinations/iceberg
2. Joe Reis and Matt Housley, Fundamentals of Data Engineering (O’Reilly, 2022), chapter 7, “Ingestion”.
3. IETF, RFC 6585 §4 (429 Too Many Requests) and RFC 9110 §10.2.3 (Retry-After). https://www.rfc-editor.org/rfc/rfc6585 · https://www.rfc-editor.org/rfc/rfc9110#section-10.2.3
4. IETF httpapi working group, “The Idempotency-Key HTTP Header Field”, draft-ietf-httpapi-idempotency-key-header-08. https://datatracker.ietf.org/doc/draft-ietf-httpapi-idempotency-key-header/
5. Apache Iceberg, PyIceberg “Python API” (append, overwrite, upsert). https://py.iceberg.apache.org/api/

Story Closing

Farah forwarded the corrected scorecard to S017 with an apology, and the 81 quarantined rows to their suppliers with file names and line numbers. Of the five promises from Part 2, this part strengthened completeness: every file, row and seller change that arrived is landed, counted and explained.

“The full extract caught the deleted seller a day late,” Rohan said. “Our own order database deletes rows too. Do we poll that in full every day as well?”

“We own that one,” Ingrid said. “We don’t have to ask it what changed. We can read its log.”

In Part 16, Rohan streams Larkspur’s orders, payments and refunds out of PostgreSQL with change data capture, and the hard-deleted orders behind £57,064.96 of the 0.7% finally show up.

This is Part 15 of “Every Record Counts: Data Engineering for Backend Engineers” — Season 2: Pipelines in Production.