"The Dual-Write Trap" — CDC from First Principles

About This Mini-Series

Who it’s for: Java and Spring developers who know Kafka producers, consumers and transactions well, have heard “just use CDC”, and have never run Debezium in production.

What you’ll build: Dispatchly’s shipment pipeline. Every committed change to the shipments table reaches a tracking view and the notification service, and the application never calls kafkaTemplate.send() after save().

Three parts: the concepts and the raw log (this one), Kafka Connect and Debezium end to end (Part 2), and production practice: outbox, snapshots, slots, schemas and failure drills (Part 3).

Prerequisites:

The code is in the cdc-with-debezium folder of the companion-workspaces repository, and every output below is from a real run of it.

Story Opening

The incident review had a timeline and a number. During Tuesday’s 18:40 deploy, 312 shipments changed to OUT_FOR_DELIVERY and their customers never got the SMS. The tracking page agreed with the database. The search index, fed from Kafka, still showed yesterday’s ETAs.

Zoya Khan had written the code herself, years ago, and it looked fine:

shipmentRepository.save(shipment);              // commits
kafkaTemplate.send("shipment-events", event);   // lost if the pod dies first

Kubernetes sent SIGTERM, the graceful-shutdown window ran out, and pods died between the two lines, or just after the second. send() is asynchronous: it returns once the record is in the producer’s buffer, and a buffer dies with its pod. The rows were committed. The events were never delivered.

The second bug was worse. A retry path published its event inside the transaction, the transaction then rolled back on a version conflict, and a customer in Whitefield got “Your parcel has been delivered” for a parcel still sitting at hub BLR-07.

“Easy,” Zoya said. “Move the send into a @TransactionalEventListener. It fires after commit.”

Tomás Ferreira, from data platform, didn’t look up. “That moves the hole. It doesn’t close it.” He had once been paged at three in the morning by a PostgreSQL disk at 100%, and has respected the write-ahead log ever since. “The database already writes down every committed change, in order, before it tells you ‘committed’. Stop asking your code to remember. Read the log.”

The Quick Map

Conceptual Deep-Dive

The dual-write problem

Your instinct says: write to the database, then tell Kafka. Two systems, two writes, and no transaction that spans both. Every ordering of those writes has a window where one succeeds and the other doesn’t.

Flip the order (send first, then commit) and you get the second bug: the event is out, and then the transaction rolls back. Consumers react to a change that never happened. Call that window 3: any send before the commit is final, which is what Zoya’s retry path did.

Blocking on the send’s future shrinks window 2 and does nothing for window 1. Kafka transactions are atomic across Kafka partitions, not across Kafka and PostgreSQL, and Kafka isn’t an XA resource Spring could enlist next to your JDBC connection. Two-phase commit, where you can have it, swaps lost events for in-doubt transactions that block both systems.

@TransactionalEventListener (default phase AFTER_COMMIT) closes window 3: a rolled-back transaction never publishes. Windows 1 and 2 stay open, because the listener runs in the same process after the commit, and a crash or a failed send there has nothing left to roll back.

What does close it: make the database write the only write, and derive the event from the database’s own record of that write. That record already exists. It’s the write-ahead log.

Three ways to capture changes

Log-based capture is the only approach that sees exactly what committed, in commit order, deletes included, without adding writes to your transactions. Its cost is operational, and Part 3 spends most of its time there. The hands-on makes all three of the poller’s misses happen on demand.

CDC is not event sourcing

Both produce streams of changes, which is why they get confused. In event sourcing, the event log is the system of record and the current state is derived from it; the events are domain facts you designed (“ParcelHandedToCourier”). In CDC, the table is the system of record and the events are derived from it; they’re row diffs, shaped by your schema. Part 3’s transactional outbox combines the two, using CDC to publish designed domain events reliably. CDC alone tells you “status went from AT_HUB to OUT_FOR_DELIVERY”, not why.

What teams build on CDC

Read models (Part 2’s tracking_view), search indexes and caches that also see the batch job’s plain-SQL updates, analytics feeds instead of nightly dumps, the transactional outbox (Part 3), and, newest, vector stores for RAG: Debezium 3.7 ships an embeddings transformation (io.debezium.ai.embeddings.FieldToEmbedding) and an official, still-alpha Python package, PyDebeziumAI.

When not to use CDC

Technical Explanation

The write-ahead log

PostgreSQL never changes a data page without first describing the change in the write-ahead log (WAL): an append-only sequence of records in 16 MB segment files under pg_wal/. A COMMIT returns only after the transaction’s commit record is flushed to disk (with the default synchronous_commit = on). After a crash, PostgreSQL replays the WAL to recover; a streaming replica is the same replay, running on another server.

Every position in the WAL is a log sequence number (LSN): a 64-bit byte offset, printed as two hex halves like 0/1BB72D0. LSNs only grow, so “how far behind is this reader?” is subtraction, and pg_wal_lsn_diff() does it for you.

The WAL is written in the order work happens, and transactions interleave in it. A logical decoder reassembles each transaction’s changes and hands them out when the transaction commits, in commit order. A transaction that rolls back is never handed out (by default; PostgreSQL 14+ can stream in-progress transactions to clients that ask, and Debezium doesn’t). A long transaction that started first but committed last comes out last.

Other databases have the same idea. MySQL and MariaDB write a binlog in commit order, with GTIDs that identify each transaction across a replication topology. Oracle writes redo logs, addressed by SCN, which Debezium reads through LogMiner by default.

Logical decoding, publications and slots

The WAL describes physical changes (“these bytes on that page”). Logical decoding turns it back into row changes (“UPDATE shipments, id 1, status AT_HUB”). Three pieces cooperate:

• Output plugin: formats decoded changes. pgoutput is built in (PostgreSQL 10+) and speaks the logical replication protocol; Debezium uses it here. test_decoding is a demo plugin that prints text.
• Publication: which tables (and which operations) pgoutput sends: CREATE PUBLICATION … FOR TABLE shipments. test_decoding ignores publications.
• Replication slot: a named, crash-safe cursor into the WAL for one consumer. It remembers how far the consumer has confirmed, and it guarantees the WAL after that point stays on disk until the consumer confirms it.

That guarantee is why slots are both wonderful and dangerous. The PostgreSQL docs say slots “will prevent removal of required resources even when there is no connection using them”. That includes old catalog rows that VACUUM then can’t remove, and the same page warns that “In extreme cases this could cause the database to shut down to prevent transaction ID wraparound”.

Two distances matter. Current minus confirmed_flush_lsn is how far the consumer is behind. Current minus restart_lsn is how much disk the slot is holding. Two settings act as safety valves:

• max_slot_wal_keep_size (PostgreSQL 13+) caps how much WAL a slot may hold. Past the cap the slot goes unreserved, and once a checkpoint removes WAL it still needed, it is lost. Its consumer then has to start over with a fresh snapshot. The default, -1, means no cap.
• idle_replication_slot_timeout, new in PostgreSQL 18, invalidates slots that have been inactive for longer than the setting. The default is 0, which means off.

Both trade a full disk for a re-snapshot; Part 3 drills that trade.

wal_level must be logical for any of this. The default is replica, where creating a slot fails with ERROR: logical decoding requires "wal_level" >= "logical", and changing it needs a restart. On a managed database that’s a parameter change plus a reboot, so check it first.

Snapshot, then stream

A slot only holds changes from the moment it was created. The rows that already exist need a snapshot first, and the hand-off between snapshot and stream must leave no gap. Here is the order the Debezium PostgreSQL connector followed in the companion lab, reconstructed from docker compose logs connect:

The key log line is SET TRANSACTION SNAPSHOT '0000007E-00000004-1';, right after CREATE_REPLICATION_SLOT "part1_preview" LOGICAL pgoutput. Creating a slot exports a snapshot, “which will show exactly the state of the database after which all changes will be included in the change stream” (PostgreSQL docs). The connector reads every table through that snapshot, then streams from the slot’s start point. In this run, the logged xlogStart (0/A7AB120) is exactly the new slot’s flushed position, and streaming started there. The snapshot and the stream meet at one LSN, so there’s no gap by construction. If the connector dies before Connect has written the post-snapshot offset (mid-snapshot, or within the offset flush interval after it), the whole snapshot runs again: duplicate READ events, never missing ones.

A full snapshot of a large table can take hours and can’t pause. Incremental snapshots read a table in primary-key chunks while streaming continues, and de-duplicate the overlap with watermarks written to a signal table. They’re how you backfill a table you add later; Part 3 runs one.

The change event

Debezium turns each decoded change into a Kafka record. The pipeline arrives in Part 2, but the envelope is a concept, so here is a real one. part1/preview-event.sh (it needs the full stack: docker compose up -d --wait && ./part1/preview-event.sh) resets the seed rows, registers a throwaway connector, moves DSP-1003 from PICKED_UP to AT_HUB, prints the topic, and removes the connector, its offsets, slot, publication and topic. The output holds four snapshot READ events (their source.snapshot reads first, true, true, last) and then the update, shown here with the microsecond and nanosecond twins of each ts_ms removed:

key:   {"id":3}
value:
{
  "before": null,
  "after": {
    "id": 3,
    "tracking_code": "DSP-1003",
    "status": "AT_HUB",
    "hub_id": "BLR-12",
    "eta": "2026-10-23T14:30:00.000000Z",
    "version": 1,
    "updated_at": "2026-10-22T16:38:26.563484Z"
  },
  "source": {
    "version": "3.7.0.Final",
    "connector": "postgresql",
    "name": "preview",
    "ts_ms": 1792687106563,
    "snapshot": "false",
    "db": "dispatchly",
    "sequence": "[null,\"176185440\"]",
    "schema": "public",
    "table": "shipments",
    "txId": 800,
    "lsn": 176185440,
    "xmin": null,
    "origin": null,
    "origin_lsn": null
  },
  "transaction": null,
  "op": "u",
  "ts_ms": 1792687106901
}

• Key: the primary key, so every change to shipment 3 goes to the same partition.
• op: c create, u update, d delete, r read (snapshot), t truncate, m logical decoding message. The PostgreSQL connector skips truncates by default (skipped.operations defaults to t).
• before: null, and that’s not a bug. With the table’s default REPLICA IDENTITY, the WAL doesn’t carry the old row, so there’s nothing to put in before. Step 2 of the hands-on shows when it does.
• source: where the change came from: lsn and txId locate it in the WAL, and source.ts_ms is when it happened in the database. sequence, xmin and the origin fields are bookkeeping this series doesn’t need.
• Top-level ts_ms: when the connector processed the event, by the Connect worker’s clock. ts_ms − source.ts_ms is capture lag (338 ms here), not commit time.

When a row is deleted, Debezium sends a d event and then, by default (tombstones.on.delete=true), a tombstone: the same key with a null value, so that log compaction can eventually drop that key. Consumers must tolerate null values.

The event’s shape follows the table’s shape, so renaming a column is a breaking change for every consumer. That’s the real argument for schema registries and compatibility rules, which Part 3 adds.

Ordering

One row’s changes go to one partition in commit order. There is no order across keys once a topic has more than one partition. And “per-key order” has three fine-print clauses:

• After a connector restart, events since the last stored offset are produced again, so a consumer can see v1, v2, v3, v2, v3 for the same key. That’s why Part 2’s consumer compares version instead of trusting arrival order.
• An update that changes the primary key becomes a delete under the old key and a create under the new one, on possibly different partitions.
• Adding partitions to a topic remaps keys to partitions.

If a consumer needs transaction boundaries, Debezium can emit them (provide.transaction.metadata=true, off by default): BEGIN and END markers on a <topic.prefix>.transaction topic, plus a transaction block in each event.

Delivery: at-least-once is the contract

There are three bookmarks, and each lags for its own reason:

1. The slot’s confirmed_flush_lsn, in PostgreSQL. It stops moving when the connector is down, and on a quiet database even when it’s healthy, which is step 3’s WAL growth.
2. The connector’s offset, an LSN plus a little more, which Kafka Connect writes to a Kafka topic on a timer (offset.flush.interval.ms, 60 seconds in the lab), not after every event.
3. Your consumer group’s offset, which lags when the consumer is slow or down.

On restart, the connector resumes from its last stored offset, and everything it produced after that is produced again. Debezium’s PostgreSQL docs: “The number of duplicate events depends on the offset flush period and the volume of data changes just before the crash.” Raw slot readers like steps 1 and 4 get a second source: a slot’s position “is persisted only at checkpoint”, so after a PostgreSQL crash it can rewind.

Kafka Connect supports exactly-once for source connectors (KIP-618, since Kafka 3.3), and the Debezium PostgreSQL connector can use it. Debezium’s exactly-once page adds that “it remains unclear whether the implementation is fully correct”, citing open Kafka transaction bugs. Even if it were perfect, it would only cover writes into Kafka. Your consumer’s upsert into a read model, its call to the search index and its SMS are all outside any Kafka transaction. So consumers must be idempotent whatever the producer side promises, and that belongs in the design before production, not after the first restart.

The landscape

• Debezium on Kafka Connect: the classic deployment, used in this series. Connect stores offsets and runs transformations and converters.
• Debezium Server: a standalone application, no Kafka required, that sends changes to Kinesis, Pub/Sub, Pulsar, Redis, NATS and others.
• Debezium Engine: the connector as a library in your JVM application; awkward for a horizontally scaled service, because a slot has exactly one reader.
• Debezium Operator and Platform: the Operator runs Debezium Server on Kubernetes. The Platform adds a UI and API on top and is still incubating.
• Flink CDC: a sub-project of Apache Flink (3.7.0, October 2026) for pipelines that transform data in flight.

Step-by-Step Hands-On

You’ll read PostgreSQL’s change stream directly, so that in Part 2 you know what Debezium is doing for you.

Step 0: Start PostgreSQL with logical decoding

git clone https://github.com/phoenixtb/companion-workspaces.git
cd companion-workspaces/cdc-with-debezium
docker compose up -d --wait postgres

compose.yaml starts PostgreSQL 18.6 with -c wal_level=logical, plus max_replication_slots, max_wal_senders and a max_slot_wal_keep_size cap of 2 GB (Part 2 walks through the file). On first start, postgres/init/01-dispatchly-schema.sql creates shipments(id, tracking_code, status, hub_id, eta, version, updated_at) with four seed rows.

Step 1: Your first slot

part1/01-first-slot.sql resets the four seed shipments, then creates a slot with test_decoding and runs three kinds of transaction: one with three statements, one that rolls back, and a single autocommit statement.

SELECT * FROM pg_create_logical_replication_slot('part1_demo', 'test_decoding');

-- One transaction, three statements.
BEGIN;
INSERT INTO shipments (tracking_code, status, hub_id, eta)
VALUES ('DSP-1005', 'CREATED', 'BLR-07', '2026-10-25 10:00+05:30');
UPDATE shipments SET status = 'AT_HUB', version = version + 1, updated_at = now()
WHERE tracking_code = 'DSP-1001';
DELETE FROM shipments WHERE tracking_code = 'DSP-1002';
COMMIT;

-- A transaction that changes its mind.
BEGIN;
UPDATE shipments SET status = 'DELIVERED', version = version + 1, updated_at = now()
WHERE tracking_code = 'DSP-1003';
ROLLBACK;

-- A single statement in autocommit mode is a transaction too.
UPDATE shipments SET status = 'OUT_FOR_DELIVERY', version = version + 1, updated_at = now()
WHERE tracking_code = 'DSP-1001';

-- Peek: read the decoded changes without consuming them.
SELECT lsn, xid, data FROM pg_logical_slot_peek_changes('part1_demo', NULL, NULL);

docker compose exec -T postgres psql -U dispatchly -d dispatchly < part1/01-first-slot.sql

The peek, with each row’s column list trimmed (…) to fit:

    lsn    | xid |  data
-----------+-----+------------------------------------------------------------------------
 0/1BB6F70 | 758 | BEGIN 758
 0/1BB6F70 | 758 | table public.shipments: INSERT: id[bigint]:5 tracking_code[text]:'DSP-1005' status[text]:'CREATED' … updated_at[timestamp with time zone]:'2026-10-22 16:38:17.862286+00'
 0/1BB7068 | 758 | table public.shipments: UPDATE: id[bigint]:1 tracking_code[text]:'DSP-1001' status[text]:'AT_HUB' … updated_at[timestamp with time zone]:'2026-10-22 16:38:17.862286+00'
 0/1BB70E8 | 758 | table public.shipments: DELETE: id[bigint]:2
 0/1BB7160 | 758 | COMMIT 758
 0/1BB7210 | 760 | BEGIN 760
 0/1BB7210 | 760 | table public.shipments: UPDATE: id[bigint]:1 tracking_code[text]:'DSP-1001' status[text]:'OUT_FOR_DELIVERY' … updated_at[timestamp with time zone]:'2026-10-22 16:38:17.86497+00'
 0/1BB72D0 | 760 | COMMIT 760
(8 rows)

Three things to notice:

1. Transaction 759 is missing. That was the rollback. A rolled-back transaction never reaches this consumer, or Debezium, so the phantom-event bug can’t happen.
2. The delete carries only id. The row is gone; the WAL kept just enough to identify it. Step 2 shows how to get more.
3. The INSERT and UPDATE in transaction 758 have the same updated_at, to the microsecond, because now() is the transaction’s start time. Step 4 turns that into a lost event.

The script ends with pg_logical_slot_get_changes, which returns the same 8 rows and moves the slot’s confirmed_flush_lsn past them; a second peek returns nothing. Debezium does the equivalent over the streaming replication protocol: it receives changes, and separately tells the server how far it has safely processed.

Step 2: What the log remembers about the old row

REPLICA IDENTITY is a per-table setting that decides how much of the old row the WAL records for updates and deletes. part1/02-replica-identity.sql (run it like step 1, as with every part1/*.sql file) compares the two common settings:

-- REPLICA IDENTITY DEFAULT: the old row is identified by its primary key only.
UPDATE shipments SET hub_id = 'BLR-03', version = version + 1, updated_at = now()
WHERE tracking_code = 'DSP-1004';

-- REPLICA IDENTITY FULL: the whole old row is written to the WAL.
ALTER TABLE shipments REPLICA IDENTITY FULL;
UPDATE shipments SET hub_id = 'BLR-07', version = version + 1, updated_at = now()
WHERE tracking_code = 'DSP-1004';
DELETE FROM shipments WHERE tracking_code = 'DSP-1005';
ALTER TABLE shipments REPLICA IDENTITY DEFAULT;

SELECT data FROM pg_logical_slot_get_changes('part1_demo', NULL, NULL);

 BEGIN 761
 table public.shipments: UPDATE: id[bigint]:4 tracking_code[text]:'DSP-1004' status[text]:'CREATED' hub_id[text]:'BLR-03' …
 COMMIT 761
 BEGIN 762
 COMMIT 762
 BEGIN 763
 table public.shipments: UPDATE: old-key: id[bigint]:4 … hub_id[text]:'BLR-03' … new-tuple: id[bigint]:4 … hub_id[text]:'BLR-07' …
 COMMIT 763
 BEGIN 764
 table public.shipments: DELETE: id[bigint]:5 tracking_code[text]:'DSP-1005' status[text]:'CREATED' hub_id[text]:'BLR-07' …
 COMMIT 764
 BEGIN 765
 COMMIT 765

With DEFAULT, an update that doesn’t touch the primary key carries only the new row: that’s the "before": null in the Debezium event. With FULL, you get old-key (the entire old row), and the delete carries every column. Transactions 762 and 765 are the two ALTER TABLE statements, decoded as empty transactions: DDL is not part of the logical change stream. Debezium learns about a new column only when the next row change arrives carrying it.

FULL writes the whole old row to the WAL on every update and delete, so turn it on only for tables whose consumers need before. DEFAULT has one more consequence: a large value stored out of line (TOAST) that an UPDATE didn’t touch isn’t in the WAL either, and Debezium sends the placeholder __debezium_unavailable_value in its place. Part 2 shows it.

The second half of the script is the trap that catches teams on day one: a table without a primary key, added to a publication.

-- A table with no primary key, added to a publication (the thing pgoutput and Debezium read through).
CREATE TABLE hub_scans (tracking_code text, hub_id text, scanned_at timestamptz DEFAULT now());
INSERT INTO hub_scans (tracking_code, hub_id) VALUES ('DSP-1003', 'BLR-12');
CREATE PUBLICATION part1_pub FOR TABLE hub_scans;

-- Inserts still work. Updates and deletes are now rejected, on the application's connection.
INSERT INTO hub_scans (tracking_code, hub_id) VALUES ('DSP-1004', 'BLR-07');
UPDATE hub_scans SET hub_id = 'BLR-03' WHERE tracking_code = 'DSP-1003';

-- FULL gives the table an identity: every column of the old row.
ALTER TABLE hub_scans REPLICA IDENTITY FULL;
UPDATE hub_scans SET hub_id = 'BLR-03' WHERE tracking_code = 'DSP-1003';

Output from CREATE PUBLICATION on:

CREATE PUBLICATION
INSERT 0 1
ERROR:  cannot update table "hub_scans" because it does not have a replica identity and publishes updates
HINT:  To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
ALTER TABLE
UPDATE 1

Creating a publication made the application’s UPDATE fail.

Step 3: A slot nobody reads

This is Tomás’s 3 a.m. page in miniature. part1/03-slot-pins-wal.sql reuses the part1_demo slot from step 1, which lives in the dispatchly database. It defines two reports, then writes 300,000 rows into a table in a different database on the same server:

-- Per slot: how much WAL its consumer hasn't confirmed, how much it holds on disk, how close it is to the cap.
\set slot_report 'SELECT slot_name, wal_status, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)) AS unconfirmed, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained, pg_size_pretty(safe_wal_size) AS safe_wal_size FROM pg_replication_slots'
\set wal_dir 'SELECT count(*) AS wal_files, pg_size_pretty(sum(size)) AS pg_wal_size FROM pg_ls_waldir()'
:slot_report;
:wal_dir;

-- Busy traffic in ANOTHER database on the same server. The slot belongs to dispatchly,
-- but the WAL is shared by the whole server, so the slot holds on to this too.
CREATE DATABASE hub_archive;
\c hub_archive
CREATE TABLE scan_noise (id bigint, payload text);
INSERT INTO scan_noise SELECT g, repeat('x', 400) FROM generate_series(1, 300000) g;
CHECKPOINT;
\c dispatchly
:slot_report;
:wal_dir;

-- A consumer confirming its position is what releases WAL. Here we confirm by hand.
SELECT slot_name, end_lsn FROM pg_replication_slot_advance('part1_demo', pg_current_wal_lsn());
CHECKPOINT;
:slot_report;
:wal_dir;

Output (the CREATE/INSERT lines left out):

 slot_name  | wal_status | unconfirmed | retained | safe_wal_size
------------+------------+-------------+----------+---------------
 part1_demo | reserved   | 0 bytes     | 145 kB   | 2052 MB

 wal_files | pg_wal_size
-----------+-------------
         1 | 16 MB

 slot_name  | wal_status | unconfirmed | retained | safe_wal_size
------------+------------+-------------+----------+---------------
 part1_demo | reserved   | 140 MB      | 140 MB   | 1912 MB

 wal_files | pg_wal_size
-----------+-------------
        10 | 160 MB

 slot_name  | wal_status | unconfirmed | retained  | safe_wal_size
------------+------------+-------------+-----------+---------------
 part1_demo | reserved   | 208 bytes   | 384 bytes | 2056 MB

 wal_files | pg_wal_size
-----------+-------------
        10 | 160 MB

A slot in dispatchly held 140 MB of WAL written by hub_archive, a database it doesn’t decode and never will. WAL is per server, not per database, so a quiet captured database on a busy server can’t release anything until its consumer confirms a newer position. Part 3’s heartbeats exist for exactly this; Debezium even logs consider using heartbeat.action.query to prevent WAL growth on low-activity databases at startup. safe_wal_size, the headroom left under the 2 GB cap, fell from 2052 MB to 1912 MB. One pg_replication_slot_advance later, the slot holds 384 bytes.

pg_wal grew from 16 MB to 160 MB and didn’t shrink when the slot let go: PostgreSQL keeps released segment files around to reuse, within its min_wal_size/max_wal_size limits. So per-slot retained is the early warning that tells you which slot to blame. Disk usage is the last-line alarm.

The script then drops the slot and the hub_archive database. Tomás’s slot, left over from a proof of concept, had no cap and no DROP, and sat inactive for eleven days. The PostgreSQL docs describe how that ends: “If the disk holding the WAL files grows full, database server panic and consequent shutdown might occur.”

Step 4: The poller against the log

naive-poller is a small Java 25 program with two readers watching the same writes. UpdatedAtPoller is the code most teams write first; it starts with watermark = max(updated_at), and then:

    List<Row> poll() throws SQLException {
        var rows = new ArrayList<Row>();
        try (var query = connection.prepareStatement("""
                SELECT tracking_code, status, updated_at
                FROM shipments
                WHERE updated_at > ?
                ORDER BY updated_at""")) {
            query.setObject(1, watermark);
            try (var rs = query.executeQuery()) {
                while (rs.next()) {
                    var row = new Row(rs.getString(1), rs.getString(2), rs.getObject(3, OffsetDateTime.class));
                    rows.add(row);
                    watermark = row.updatedAt();   // the high-water mark only ever moves forward
                }
            }
        }
        return rows;
    }

SlotReader creates a test_decoding slot and condenses each change to one line. PollerVsLog runs a fixed script of writes, each one setting status, version = version + 1 and updated_at = now() like an application would:

./gradlew :naive-poller:run

writer  DSP-1001 -> AT_HUB
writer  DSP-1001 -> OUT_FOR_DELIVERY
writer  DSP-1002 deleted
poller  poll 1: [DSP-1001 OUT_FOR_DELIVERY]
writer  tx A: DSP-1003 -> AT_HUB (not committed yet)
writer  tx B: DSP-1004 -> PICKED_UP (committed)
poller  poll 2: [DSP-1004 PICKED_UP]
writer  tx A committed
poller  poll 3: []

slot    UPDATE DSP-1001 AT_HUB
slot    UPDATE DSP-1001 OUT_FOR_DELIVERY
slot    DELETE id=2
slot    UPDATE DSP-1004 PICKED_UP
slot    UPDATE DSP-1003 AT_HUB

The poller saw 2 of 5 committed changes.

Three different losses, all silent:

• Intermediate state: DSP-1001 was AT_HUB and then overwritten before the poll. If notifications hang off status changes, the “arrived at hub” SMS never goes out.
• Delete: DSP-1002 is simply absent from the next query. A poller can’t see absence.
• The slow transaction: tx A stamped DSP-1003 with its start time, tx B committed a later timestamp, and poll 2 moved the watermark past both. When tx A committed, its row sat behind the watermark for good. Polling faster doesn’t help, and an application-side clock (Instant.now()) has the same flaw plus clock skew between pods.

The usual rebuttal is to re-read an overlap window (updated_at > watermark - 5 seconds) and de-duplicate. That still misses every delete, every intermediate state, and any transaction that runs longer than the window.

The slot saw all five, and look at the last two lines: tx B before tx A, in commit order, even though tx A’s statement ran first. PollerVsLogTest asserts exactly these lists against a fresh postgres:18.6 Testcontainers database, and checks that the run leaves no slot behind:

./gradlew :naive-poller:test

Tips, Tricks & Gotchas

Gotcha — deleting a Debezium connector leaves its slot and its offsets behind. The slot keeps pinning WAL (step 3). The offsets stay in Connect’s offsets topic, and a new connector registered under the same name resumes from them. In the lab, re-registering after the slot had been dropped got the connector RUNNING and its task FAILED: “Last recorded offset is no longer available on the server.” preview-event.sh ends with the full clean-up: stop the connector, delete its offsets, delete it, drop the slot and publication. After any experiment, SELECT slot_name, active FROM pg_replication_slots; (part1/list-slots.sql) should show nothing you don’t recognise.

Gotcha — Debezium’s default publication needs superuser and captures every table. publication.autocreate.mode defaults to all_tables, which runs CREATE PUBLICATION … FOR ALL TABLES. PostgreSQL requires a superuser for that. A least-privilege connector fails at startup; a proof of concept running as postgres succeeds, and then every table without a primary key or replica identity starts rejecting UPDATEs, as step 2 showed. Find PK-less tables first with part1/pk-less-tables.sql. Part 2 uses a publication the DBA creates for just the captured tables.

Gotcha — Debezium’s plugin.name defaults to decoderbufs, not pgoutput. The stock PostgreSQL image doesn’t ship that plugin. Since the August 2026 minor releases (18.6, 17.11, 16.15, 15.19, 14.24), a new output_plugin_libraries setting also allows only pgoutput and test_decoding by default. part1/decoderbufs-refused.sql asks for it on 18.6 and gets ERROR: library "decoderbufs" may not be used as an output plugin. Always set plugin.name=pgoutput.

Tip — updated_at still has a job. It’s useful for humans and for reconciliation queries (“anything changed in the last hour that the read model doesn’t have?”). It just can’t be your change feed.

Key Takeaways

Story Closing

Zoya ran the poller demo twice to make sure she wasn’t fooling herself. Two of five. She posted the output in the incident channel without comment, and the conversation about “just tightening the poll interval” ended on its own.

Tomás looked at her step 3 terminal over her shoulder. “Drop it,” he said, pointing at part1_demo.

“It’s a demo slot.”

“So was mine.”

She dropped it, then sketched the pipeline: the log, a reader that remembers its position in Kafka, a projection that doesn’t mind hearing the same news twice. Tomás added two boxes: Kafka Connect, and a dashboard for slot lag. “Debezium’s the easy part,” he said. “Connect is what you’ll be operating.”

In Part 2, Zoya plugs Debezium into the WAL: Kafka Connect from the ground up, a least-privilege connector, real change events, and a Spring Boot projection that survives restarts and replays.

References

1. Debezium PostgreSQL connector (3.7). https://debezium.io/documentation/reference/3.7/connectors/postgresql.html
2. Debezium exactly-once delivery (3.7). https://debezium.io/documentation/reference/3.7/configuration/eos.html
3. PostgreSQL 18: Logical Decoding Concepts. https://www.postgresql.org/docs/18/logicaldecoding-explanation.html
4. PostgreSQL 18: pg_replication_slots. https://www.postgresql.org/docs/18/view-pg-replication-slots.html
5. Gunnar Morling, Mastering Postgres Replication Slots. https://morling.dev/blog/mastering-postgres-replication-slots

This is Part 1 of a 3-part series: “The Log Never Lies: Change Data Capture with Debezium.”