"Production Without Prayers" — Outbox, Snapshots, Slots and Schemas

Story Opening

Monday’s calendar invite said Game day and nothing else. Tomás had written the agenda on the whiteboard in the order things had gone wrong for him over the years:

1. Retire the dual write.
2. Backfill a table nobody captured.
3. Let a quiet database fill the disk.
4. Break a schema.
5. Kill a Connect worker.
6. Turn on exactly-once and see what it’s worth.

“Every drill has one rule,” he said. “You predict the outcome before you run it. Out loud.”

The game day found its first bug before anyone had predicted anything. Zoya’s shipment-service returned HTTP 500 on its very first outbox write: duplicate key value violates unique constraint "shipments_pkey". That had nothing to do with the outbox. It was an ordinary bug in her Part 2 code, and the game day found it in two minutes.

The Quick Map

Conceptual Deep-Dive

Raw table CDC versus a published contract

Part 2 published shipments as it is: every column, named as the table names it, with an event for every write. That’s right for a read model the same team owns. It’s wrong as a contract with another team. A column rename becomes their outage, an internal column becomes their dependency, and “status changed” has to be inferred from a row diff.

The transactional outbox keeps CDC’s guarantee and changes what’s published. The service writes its business change and an event row in one local transaction. Debezium reads the event row from the WAL and its EventRouter transformation turns it into a clean Kafka record: the event’s payload as the value, a chosen topic, the aggregate id as the key, the event id in a header. Nothing is sent from the request thread, so nothing is lost when the pod dies.

The row is deleted in the same transaction it was inserted in. The INSERT is in the WAL anyway, which is all Debezium needs, and the table never grows. The cost: once the WAL is gone, there is no second copy of the event. Keep that in mind for drill 3.

An alternative needs no table at all. pg_logical_emit_message(true, 'shipment', payload) writes a transactional message straight into the WAL. Debezium emits it as an m event (pgoutput, PostgreSQL 14+), and the DecodeLogicalDecodingMessageContent SMT turns it into a normal record. It’s lighter, but there’s no row to inspect or re-publish, and every producer has to call a PostgreSQL function. This series uses the table.

Incremental snapshots: watermarks instead of locks

Part 1’s initial snapshot reads everything in one transaction before streaming starts. For a table added later, or one that needs re-reading, that’s a long stop. An incremental snapshot (the DBLog design from Netflix) reads the table in primary-key chunks while streaming continues. Each chunk is wrapped in a window of two watermark rows that Debezium writes to a signal table:

Because the watermarks travel through the WAL like any other change, Debezium knows exactly which streamed changes happened during the chunk’s read. Any row that changed meanwhile is emitted from the stream, not from the stale snapshot read. The snapshot can be paused, resumed and stopped by more signals, and it resumes after a restart.

Retention is a budget

Every slot pins WAL from its restart_lsn. Part 1 showed the mechanics. In production, the questions are how much a slot may pin, and what happens when it hits the limit. max_slot_wal_keep_size turns “the disk fills and PostgreSQL panics” into “the slot is invalidated and its consumer needs a new snapshot”. The heartbeat and flush settings in drill 3 decide whether a healthy slot ever approaches that limit.

Technical Explanation

The outbox event router

io.debezium.transforms.outbox.EventRouter reads columns by convention:

• id becomes the id header, for deduplication.
• aggregatetype feeds route.by.field, so it picks the topic.
• aggregateid becomes the record key, for per-aggregate order.
• type is the event type.
• payload becomes the value.

The default topic is outbox.event.${routedByValue}; this series uses route.topic.replacement=dispatchly.${routedByValue}.events, so shipment events go to dispatchly.shipment.events. table.expand.json.payload=true with JsonConverter emits the jsonb payload as real JSON instead of an escaped string. Updates to outbox rows are invalid (table.op.invalid.behavior, default warn) and deletes are filtered out. Apply the router only to outbox records, with a predicate: heartbeat and transaction records would break it.

Other table.field.* and table.fields.additional.placement options rename columns or copy extra ones into headers. For tracing, a tracingspancontext column (tracing.span.context.field) carries an OpenTelemetry context from the writing request into the record’s headers, so a trace spans service, outbox and consumer.

Signals and snapshot options

The signal table must have three columns, id, type and data, in that order. The docs require it to be unique to each connector instance (another connector would act on the same signals and watermarks), and be readable from the WAL: with a DBA-owned publication it must be in the publication, and Debezium needs INSERT on it to write watermarks. It does not need to be in table.include.list. A signal is an INSERT:

INSERT INTO debezium_signal (id, type, data) VALUES ('backfill-hubs-1', 'execute-snapshot',
  '{"data-collections": ["public.hubs"], "type": "incremental"}');

That’s part3/03-backfill-hubs.sql from drill 2. The options worth knowing:

• additional-conditions: snapshot only matching rows, e.g. {"data-collection": "public.hubs", "filter": "city = 'Bengaluru'"}.
• Other actions: stop-snapshot, pause-snapshot, resume-snapshot.
• "type": "blocking": pauses streaming for the snapshot’s duration and can produce duplicates around the signal.
• The Kafka signal channel: add kafka to signal.enabled.channels (default source) and set signal.kafka.topic. The message key must equal topic.prefix.
• Read-only incremental snapshots (read.only=true, PostgreSQL 13+): don’t write watermarks to the database, so they need no writable signal table. Since 3.7 (and 3.6.2) they also work on a hot standby.
• Housekeeping tables grow. Watermark rows pile up in the signal table (and on its topic); incremental.snapshot.watermarking.strategy=insert_delete deletes them again, where the default insert_insert keeps them. An inbox like drill 1’s gains a row per event, so give it a retention job.
• Chunk-based parallel initial snapshots (snapshot.max.threads > 1): incubating in 3.7.

Keeping slots moving

Drill 3 shows the problem and both fixes. heartbeat.action.query writes a row to a table in the publication on every heartbeat, giving Debezium a change of its own to confirm. lsn.flush.mode=connector_and_driver, a Technology Preview in 3.7, lets the JDBC driver’s keep-alive confirm the server’s position when Debezium has nothing pending; with heartbeat.interval.ms=0 and no transaction metadata, it sets the heartbeat interval to 600000 ms itself. It can move the slot ahead of the stored offset, so the docs pair it with offset.mismatch.strategy (also a Technology Preview). Its trust_slot and trust_greater_lsn values resolve the difference by, in the docs’ words, skipping “replaying events between the stored offset and the slot position”.

On the database side:

• max_slot_wal_keep_size caps each slot.
• idle_replication_slot_timeout (PostgreSQL 18, off by default) invalidates slots that have been inactive too long.
• Failover slots (slot.failover=true, PostgreSQL 17+) survive a primary failover. They need synchronized_standby_slots on the primary, and on the standby sync_replication_slots=on, hot_standby_feedback=on, a primary_slot_name and a dbname in primary_conninfo.
• Logical decoding on a standby has worked since PostgreSQL 16, but those slots aren’t synchronised with the primary’s.

Schemas and registries

With a registry, the converter registers the record schema on first use and the registry enforces a compatibility rule on every new version. BACKWARD means a consumer on the new schema can read data written with the old one: adding an optional field passes, adding a required field fails, and removing a field passes. CDC changes the producer first, though, so it’s consumers on the old schema that need protecting. That’s FORWARD, and FULL_TRANSITIVE checks both directions against every earlier version. It has two limits worth knowing. Renaming a nullable column passes every mode, because a missing optional field resolves to its default null. And under FORWARD a NOT NULL column can never be dropped, since even making it nullable first is a change old readers reject. Raw table topics can’t be fully protected by a registry; contracts belong behind the outbox. Debezium’s image ships Apicurio’s converters (ENABLE_APICURIO_CONVERTERS=true, io.apicurio.registry.utils.converter.AvroConverter). Confluent’s io.confluent.connect.avro.AvroConverter (with …converter.schema.registry.url) works the same way, but its jars haven’t been in Debezium’s images since 2.0. For envelopes that cross organisational boundaries, io.debezium.converters.CloudEventsConverter wraps events in the CNCF CloudEvents format.

Remember the PostgreSQL detail from Part 1: DDL isn’t in the change stream. The new schema appears with the next DML, so run migrations in expand/contract order:

1. Add the new column as nullable.
2. Write both columns.
3. Move consumers over.
4. Stop writing the old column.
5. Drop it.

Exactly-once source delivery

Kafka Connect’s exactly-once support for source connectors (KIP-618) writes a poll’s records and the connector’s source offsets in one Kafka transaction:

• Workers: exactly.once.source.support=enabled on every worker. On an existing cluster, roll through preparing first.
• Connector: exactly.once.support=required. transaction.boundary=poll is the default, and what Debezium needs.
• Consumers: they must read with isolation.level=read_committed to skip aborted transactions.

Debezium’s own documentation is the right amount of sceptical: “it remains unclear whether the implementation is fully correct or if there are edge cases where exactly-once semantics might be violated”. EOS also stops at Kafka: an SMS, an HTTP call or a database write in your consumer is outside the transaction. And it doesn’t remove the duplicates that a restarted snapshot or a re-snapshot after a lost slot produce. Keep consumers idempotent.

Observability

• Connector lag: MilliSecondsBehindSource on debezium.postgres:type=connector-metrics,context=streaming,server=<topic.prefix>. It’s the time from the database change to the connector, and it is not the same as consumer lag, which you read from the consumer group.
• Freshness: MilliSecondsSinceLastEvent and Connected on the same MBean, plus the age of the heartbeat row (now() - beat_at): together they catch a connector that is RUNNING and delivering nothing.
• Throughput and snapshots: TotalNumberOfEventsSeen and the snapshot MBean’s TotalNumberOfReadEventsSeen (new in 3.7). Counters reset when the task restarts.
• Slots: part3/slot-report.sql. This is the only metric that predicts a full disk.
• Notifications: notification.enabled.channels (log, sink to a topic, jmx; an http channel arrived in 3.7.0.Final, in the release notes but not yet in the docs).
• Prometheus: the image’s entrypoint understands ENABLE_JMX_EXPORTER, but the exporter jar isn’t in the image. Build a derived image with the agent jar and a metrics.yaml, as Debezium’s monitoring example does.
• Lineage: openlineage.integration.enabled=true plus a config file emits OpenLineage events for data-platform lineage.

Running Kafka Connect for real

• Clusters per domain. One Connect cluster per team or blast radius, so a noisy connector can’t rebalance everyone else’s.
• Sizing. Each Debezium task buffers up to max.queue.size (8192) records and hands them over in batches of max.batch.size (2048); multiply by your widest rows (TOAST-heavy tables included) when you set worker heap.
• Resetting. Never delete connect-offsets: use PUT …/stop, then DELETE or PATCH …/offsets. Renaming a connector also resets it, but leaves orphaned offsets behind.
• Configuration. The Debezium image turns every CONNECT_* variable into a worker property (CONNECT_A_B becomes a.b). Apart from a fixed list (BOOTSTRAP_SERVERS, GROUP_ID, *_STORAGE_TOPIC and a few more), anything else, like CONFIG_STORAGE_REPLICATION_FACTOR, is silently ignored.
• Decommissioning. DELETE /connectors/{name} leaves the slot behind, pinning WAL until someone drops it or the cap invalidates it. Drop it in the same change.
• Upgrades. Read the release notes, PUT …/stop the connectors, roll the workers onto the new image or plugin directory, PUT …/resume. Offsets live in Kafka, so they carry over. 3.7’s notes list several PostgreSQL data-loss fixes; stay on current patches.

Where Debezium can run

Step-by-Step Hands-On

The game day starts from Part 2’s running lab (docker compose up -d --wait, part2/01-debezium-user.sql, the shipments-cdc connector), and every drill below ran in this order. The repository holds the end-of-day state; its README lists what to remove to replay each drill from the state before it.

Drill 1: Retire the dual write

Zoya’s prediction: “the outbox is just a second write, so it can fail like the first one.” The drill answers that below. The 500 came first, though. Seed shipments have version = 0, and Shipment declared @Version long version. With a primitive version, Spring Data treats 0 as “new”, so save() issued an INSERT. Part 2 never hit it, because its API updates only touched rows already at version 1 or more. The fix is one word, Long, which treats null as new:

// One row of the shipments table. Spring Data JDBC increments version on every save
// and refuses the update if someone else saved in between (optimistic locking).
// The version is a Long, not a long: with a primitive, Spring Data treats version 0 as "new" and
// would INSERT a seed row (version 0) instead of updating it.
@Table("shipments")
public record Shipment(
        @Id Long id,
        String trackingCode,
        ShipmentStatus status,
        String hubId,
        Instant eta,
        @Version Long version,

New shipments now start at version 0, where Part 2’s started at 1.

The outbox. part3/01-outbox.sql creates the table, a publication for it alone, and notification-service’s database:

CREATE TABLE outbox_event (
    id            uuid         PRIMARY KEY,   -- becomes the Kafka header "id": the consumer's dedupe key
    aggregatetype varchar(255) NOT NULL,      -- picks the topic: dispatchly.<aggregatetype>.events
    aggregateid   varchar(255) NOT NULL,      -- becomes the Kafka key: order per shipment
    type          varchar(255) NOT NULL,      -- the event type, sent as a header
    payload       jsonb        NOT NULL       -- the event itself
);
GRANT SELECT ON outbox_event TO debezium;

-- A separate connector reads the outbox through its own publication and slot.
CREATE PUBLICATION dbz_outbox FOR TABLE outbox_event;

shipment-service writes through a small Outbox component, in the caller’s transaction:

    void publish(String aggregateType, String aggregateId, String type, Object payload) {
        var id = UUID.randomUUID();
        jdbc.sql("""
                INSERT INTO outbox_event (id, aggregatetype, aggregateid, type, payload)
                VALUES (?, ?, ?, ?, ?::jsonb)""")
                .params(id, aggregateType, aggregateId, type, json.writeValueAsString(payload))
                .update();
        jdbc.sql("DELETE FROM outbox_event WHERE id = ?").param(id).update();
    }

    // One transaction, one database: the status change and the event commit together or not at all.
    @Transactional
    public Shipment changeStatus(String trackingCode, ShipmentStatus status, String hubId) {
        var shipment = shipmentRepository.save(find(trackingCode).moveTo(status, hubId));
        outbox.publish("shipment", trackingCode, "ShipmentStatusChanged",
                new ShipmentStatusChanged(trackingCode, status, hubId, shipment.updatedAt()));
        return shipment;
    }

KafkaTemplate is gone, and so is spring-boot-starter-kafka from the service’s build. The outbox gets its own connector, connectors/outbox.json: its own slot dispatchly_outbox and publication dbz_outbox, snapshot.mode=no_data (old outbox rows don’t exist anyway), and tombstones.on.delete=false. Its converters and router:

  "key.converter": "org.apache.kafka.connect.storage.StringConverter",
  "value.converter": "org.apache.kafka.connect.json.JsonConverter",
  "value.converter.schemas.enable": "false",
  "topic.creation.default.replication.factor": "1",
  "topic.creation.default.partitions": "3",
  "transforms": "outbox",
  "transforms.outbox.type": "io.debezium.transforms.outbox.EventRouter",
  "transforms.outbox.route.topic.replacement": "dispatchly.${routedByValue}.events",
  "transforms.outbox.table.expand.json.payload": "true",
  "transforms.outbox.table.fields.additional.placement": "type:header:eventType",
  "transforms.outbox.predicate": "isOutboxTable",
  "predicates": "isOutboxTable",
  "predicates.isOutboxTable.type": "org.apache.kafka.connect.transforms.predicates.TopicNameMatches",
  "predicates.isOutboxTable.pattern": "outbox\\.public\\.outbox_event"

StringConverter keeps the key a plain DSP-1001 instead of a JSON-quoted string. One status change through the API produced this record (__debezium.context.* headers trimmed), and left zero rows in outbox_event:

Partition:0  id:c5e70bfc-08b0-413d-b457-4aadde80e1f3,eventType:ShipmentStatusChanged  DSP-1001  {"hubId":"BLR-07","status":"OUT_FOR_DELIVERY","occurredAt":"2026-10-22T19:13:27.678160Z","trackingCode":"DSP-1001"}

That’s a contract a notifications team can rely on: their names, their shape, no internal columns. notification-service handles each event once with an inbox table:

    @Transactional
    @KafkaListener(topics = "dispatchly.shipment.events", groupId = "notification-service")
    void on(ConsumerRecord<String, String> record) {
        var eventId = UUID.fromString(header(record, "id"));
        var eventType = header(record, "eventType");
        if (!inbox.firstTime(eventId, eventType)) {
            log.info("duplicate {} {} for {} ignored", eventType, eventId, record.key());
            return;
        }
        log.info("SMS for {}: {} (event {})", record.key(), record.value(), eventId);
    }

inbox.firstTime is INSERT INTO processed_event … ON CONFLICT DO NOTHING, true when a row was inserted. The inbox row and the SMS share a transaction: if sending throws, the row rolls back and the record is retried. A crash after sending but before the commit sends the SMS twice, so the side effect itself is still at-least-once, unless the SMS provider accepts an idempotency key: pass the event id, and that window closes too. processed_event grows with every event; delete rows older than the longest replay you could ever see.

Prediction: duplicates after a Connect crash will be ignored. Three status changes for DSP-3001, then docker compose kill connect inside the offset-flush window, then start:

SMS for DSP-3001: {"hubId":"BLR-12","status":"PICKED_UP",…} (event d9a74390-dee5-4925-99cf…)
SMS for DSP-3001: {"hubId":"BLR-12","status":"AT_HUB",…} (event dfd141b8-cfae-4192-8c45-1d…)
SMS for DSP-3001: {"hubId":"BLR-12","status":"OUT_FOR_DELIVERY",…} (event 18a404c0-fb41-47…)
duplicate ShipmentStatusChanged d9a74390-dee5-4925-99cf-4dd8554d75b3 for DSP-3001 ignored
duplicate ShipmentStatusChanged dfd141b8-cfae-4192-8c45-1df7f2c1ce57 for DSP-3001 ignored
duplicate ShipmentStatusChanged 18a404c0-fb41-4782-b6df-eb87efca31e1 for DSP-3001 ignored

Correct, and the opening prediction was wrong: the outbox row isn’t a second write to a second system, it commits or rolls back with the shipment. OutboxPipelineTest asserts the duplicate handling in containers, with the real connectors/outbox.json and a deliberately redelivered event.

Drill 2: Backfill a table nobody captured

Dispatchly’s 40 hubs have lived in a hubs table for years. part3/02-hubs-and-signals.sql creates it with data, a signal table and a heartbeat table (for drill 3), and adds all three to the DBA’s publication:

-- Signals: id, type and data, exactly these names. Debezium reads signals from the WAL and
-- writes its snapshot watermarks back into the same table, so it needs INSERT.
CREATE TABLE debezium_signal (id varchar(64) PRIMARY KEY, type varchar(32) NOT NULL, data varchar(2048) NULL);
GRANT SELECT, INSERT ON debezium_signal TO debezium;

The connector gains public.hubs in table.include.list, "signal.data.collection": "public.debezium_signal", "incremental.snapshot.chunk.size": "16" (tiny, to see the chunks) and "notification.enabled.channels": "log".

Prediction from the room: the signal works whether or not the signal table is published. With debezium_signal taken out of the publication, the INSERT sat in the table, the connector stayed RUNNING/RUNNING, and nothing at all happened: no log line, no topic. Published again, the same kind of signal produced this (Debezium’s log, trimmed):

Requested 'INCREMENTAL' snapshot of data collections '[public.hubs]' with additional conditions '[]' and surrogate key 'PK of table will be used'
[Notification Service]  {id='backfill-hubs-1', aggregateType='Incremental Snapshot', type='STARTED', …}
[Notification Service]  {id='backfill-hubs-1', aggregateType='Incremental Snapshot', type='IN_PROGRESS', additionalData={chunk_index=1, last_processed_key=BLR-16, total_rows=40, …, total_chunks=3, …}}
[Notification Service]  {id='backfill-hubs-1', aggregateType='Incremental Snapshot', type='IN_PROGRESS', additionalData={chunk_index=2, last_processed_key=BLR-32, …}}
[Notification Service]  {id='backfill-hubs-1', aggregateType='Incremental Snapshot', type='IN_PROGRESS', additionalData={chunk_index=3, last_processed_key=BLR-40, …}}
[Notification Service]  {id='backfill-hubs-1', aggregateType='Incremental Snapshot', type='TABLE_SCAN_COMPLETED', additionalData={…, total_rows_scanned=40, …, status=SUCCEEDED}, …}
[Notification Service]  {id='backfill-hubs-1', aggregateType='Incremental Snapshot', type='COMPLETED', …}

Forty r events landed on dispatchly.public.hubs, each with "snapshot":"incremental" in source. The watermarks are visible in the signal table afterwards: four snapshot-window-open/-close pairs, one per chunk plus the final empty read. One surprise: because the signal table is published, Debezium also produced its own watermark rows as change events, nine records on a new dispatchly.public.debezium_signal topic. They’re harmless, but they’re there.

Drill 3: The 3 a.m. drill

part3/slot-report.sql reports every slot’s state:

SELECT slot_name, active, wal_status,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)) AS unconfirmed,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn))         AS retained,
       pg_size_pretty(safe_wal_size)                                               AS safe_wal_size
FROM pg_replication_slots
ORDER BY slot_name;

part3/wal-noise.sql writes about 140 MB of WAL in hub_archive, a database nobody captures.

Prediction: a running connector keeps its slot clean. With the main connector stopped, both slots grew to 140 MB, and the outbox slot was active the whole time, which had already answered the prediction. Then the main connector was resumed:

      slot_name       | active | wal_status | unconfirmed | retained | safe_wal_size
----------------------+--------+------------+-------------+----------+---------------
 dispatchly_outbox    | t      | reserved   | 140 MB      | 140 MB   | 1919 MB
 dispatchly_shipments | t      | reserved   | 140 MB      | 140 MB   | 1919 MB

Wrong. Both connectors were RUNNING, both slots active, both still holding 140 MB, with heartbeat.interval.ms=10000 set on both. Nothing changed in their tables, so neither had an LSN to confirm. Debezium even says so in its startup log: LSN flush mode 'connector' … consider using heartbeat.action.query to prevent WAL growth on low-activity databases.

The main connector got heartbeat.action.query, an upsert into the published heartbeat table:

  "heartbeat.action.query": "INSERT INTO debezium_heartbeat (id, beat_at) VALUES (1, now()) ON CONFLICT (id) DO UPDATE SET beat_at = EXCLUDED.beat_at",

The outbox connector got "lsn.flush.mode": "connector_and_driver" and "offset.mismatch.strategy": "trust_greater_lsn". Forty seconds later the outbox slot held 704 bytes. The main slot needed one more offset flush (it confirms with Connect’s stored offsets, Part 2), and about 100 seconds after the change:

      slot_name       | active | wal_status | unconfirmed | retained  | safe_wal_size
----------------------+--------+------------+-------------+-----------+---------------
 dispatchly_outbox    | t      | reserved   | 0 bytes     | 56 bytes  | 2063 MB
 dispatchly_shipments | t      | reserved   | 248 bytes   | 936 bytes | 2063 MB

Prediction: a 64 MB cap protects us. ALTER SYSTEM SET max_slot_wal_keep_size = '64MB' plus pg_reload_conf() did nothing: SHOW still said 2GB, because compose passes the setting as a -c command-line flag, which outranks ALTER SYSTEM. Managed databases have the same trap with parameter groups. The compose file now reads max_slot_wal_keep_size=${PG_MAX_SLOT_WAL_KEEP_SIZE:-2GB}:

PG_MAX_SLOT_WAL_KEEP_SIZE=64MB docker compose up -d --wait postgres

Both connectors rode through the database restart on their own. Then, with the main connector stopped, 135 MB of noise and a checkpoint:

      slot_name       | active | wal_status | invalidation_reason
----------------------+--------+------------+---------------------
 dispatchly_outbox    | f      | lost       | wal_removed
 dispatchly_shipments | f      | lost       | wal_removed

LOG:  invalidating obsolete replication slot "dispatchly_shipments"
LOG:  invalidating obsolete replication slot "dispatchly_outbox"

Both slots went unreserved, then lost at the checkpoint run seconds after the burst, including the outbox slot, whose connector was running but hadn’t confirmed past the burst yet. unreserved can recover; a checkpoint while a slot is past the cap makes it lost. Without the manual checkpoint the next timed one (checkpoint_timeout, five minutes by default) would have done the same, and a burst bigger than max_wal_size triggers one sooner. A slot past the cap when a checkpoint runs is lost, however healthy its consumer. The disk was safe; the slots were not. Size the cap above the WAL you write in one confirm interval (60 seconds of offset flush for the main connector here) plus your longest tolerable outage.

Resumed, the main connector showed RUNNING with no task listed, and the outbox connector RUNNING/RUNNING. Neither recovered on its own; the log said why, misleadingly, every two seconds:

WARN  Cannot obtain valid replication slot 'dispatchly_shipments' for plugin 'pgoutput' and database 'dispatchly' [during attempt 75 out of 900, concurrent tx probably blocks taking snapshot.

There was no concurrent transaction: the slot was invalid, and 900 attempts at two seconds is half an hour of RUNNING. Recovery means accepting a new snapshot:

curl -s -X PUT localhost:8083/connectors/shipments-cdc/stop
docker compose exec -T postgres psql -U dispatchly -d dispatchly \
  -c "SELECT pg_drop_replication_slot(slot_name) FROM pg_replication_slots WHERE wal_status = 'lost'"
curl -s -X DELETE localhost:8083/connectors/shipments-cdc/offsets
curl -s -X PUT localhost:8083/connectors/shipments-cdc/resume

The same went for outbox. The main connector created a new slot and snapshotted again, which the projection’s version guard absorbs. snapshot.mode=when_needed is documented to snapshot when the stored position is no longer available, but it doesn’t drop an invalidated slot: you still drop that first. The same stop, delete-offsets, resume sequence is also how you rebuild a consumer’s world after a compacted or expired topic (Part 2’s retention gotcha). The outbox connector (no_data) just started streaming from now. Had shipments changed status during the outage, those events would have existed only in the WAL that was removed. That’s the price of deleting outbox rows immediately. If you can’t afford it, keep outbox rows for a retention period and delete them on a schedule. A gap could then be backfilled by giving the outbox connector a signal table of its own in dbz_outbox and snapshotting outbox_event incrementally. That wasn’t tried in this lab; the inbox would absorb the re-published rows.

Drill 4: Break a schema

A third connector, connectors/shipments-avro.json, with its own slot, writes Avro to avro.public.shipments through Apicurio Registry 3.3.3:

  "schema.name.adjustment.mode": "avro",
  "key.converter": "io.apicurio.registry.utils.converter.AvroConverter",
  "key.converter.apicurio.registry.url": "http://apicurio:8080/apis/registry/v3",
  "key.converter.apicurio.registry.auto-register": "true",
  "key.converter.apicurio.registry.find-latest": "true",

schema.name.adjustment.mode=avro replaces characters Avro doesn’t allow in names, and auto-register lets the converter register schemas it hasn’t seen (find-latest follows Debezium’s documented example). The value converter is configured the same way. Debezium’s Avro docs still show the registry’s /apis/registry/v2 path; the image’s Apicurio 3.2.5 converter worked against the 3.3.3 server on /v3. A global rule makes the registry strict:

curl -s -X POST localhost:8081/apis/registry/v3/admin/rules -H 'Content-Type: application/json' \
  -d '{"ruleType":"COMPATIBILITY","config":"BACKWARD"}'

Expand: part3/04-add-column.sql adds a nullable customer_phone. Right after the ALTER TABLE, the value schema still had one version (["1"]). After the next status change, two (["1","2"]), with the new field as {"name":"customer_phone","type":["null","string"],"default":null}. DDL alone changes nothing downstream; the next row does.

Break: part3/05-rename-column.sql renames hub_id to current_hub_id and updates a row. Two connectors read that change:

• The Avro connector’s task went FAILED with Tolerance exceeded in error handler … io.apicurio.registry.rest.client.models.ProblemDetails. That’s opaque, but the registry’s audit log is not: Incompatible artifact: avro.public.shipments.Value [AVRO], num of incompatible diffs: {1}, list of diff types: [current_hub_id at /fields/3].
• The JSON connector carried on, and tracking-projection logged applied op=u id=4 AT_HUB v1. Its view now had hub_id = NULL for DSP-1004, silently, until the next correct write repaired it.

The registry turned a silent data bug into a loud stop, which is the trade you want for a contract. Note why it stopped: hub_id is NOT NULL, so the renamed field is required with no default. Dropping a NOT NULL column would pass BACKWARD and fail FORWARD or FULL_TRANSITIVE. Renaming a nullable column passes every mode, and old consumers silently read null: no registry catches that one, which is why contracts come from the outbox, not from the table. After undoing the rename and POST …/restart?includeTasks=true&onlyFailed=true, the Avro task returned to RUNNING; the lab didn’t check what became of the renamed UPDATE on avro.public.shipments, so look before you trust it. The Avro connector was then removed with part2/remove-connector.sh, slot and topic included.

Drill 5: Kill a Connect worker

docker compose --profile ha up -d --wait connect-2

A second worker with the same GROUP_ID joined, and the cluster spread the two remaining connectors, one task per worker. Prediction: losing a worker means a five-minute wait (scheduled.rebalance.max.delay.ms, Part 2):

Half right, from one run each. A worker that leaves cleanly hands its tasks over immediately. One that vanishes is waited for, five minutes plus failure detection. The leader row is the odd one: the previous assignment lives in the leader’s memory, so a new leader has nothing to wait for (the leader’s identity in that run is inferred; it was confirmed for the 312 s run). Rolling restarts should be graceful, and five minutes of CDC silence after a node crash is configuration, not a bug.

Drill 6: Exactly-once, measured

Every worker gets CONNECT_EXACTLY_ONCE_SOURCE_SUPPORT: enabled, and shipments-cdc gets "exactly.once.support": "required". The consumers get isolation-level: read-committed in their Spring Boot configuration. Then Part 2’s drill again: a new shipment, four status changes, docker compose kill connect, start:

part2/events-for-key.sh 7
ISOLATION=read_committed part2/events-for-key.sh 7

Both printed the same five lines, CREATED v0 through DELIVERED v4, each once. In Part 2 the same drill produced ten. With EOS, the source offsets are committed in the same Kafka transaction as the records, so a restart has nothing to replay. A kill in the middle of a transaction would leave an aborted batch that read_committed consumers skip. This drill, a single run, didn’t catch one in flight. Keep the version guard and the inbox anyway: EOS removed the offset-flush replay window, it didn’t make snapshots duplicate-free or the consumers’ side effects transactional.

Drill 7: Read the metrics

With JMXHOST/JMXPORT set on the worker, Kafka’s JmxTool reads Debezium’s MBeans from inside the container:

docker compose exec -T connect /kafka/bin/kafka-run-class.sh org.apache.kafka.tools.JmxTool \
  --jmx-url service:jmx:rmi:///jndi/rmi://localhost:1976/jmxrmi --one-time true \
  --object-name 'debezium.postgres:type=connector-metrics,context=streaming,server=dispatchly' \
  --attributes Connected,MilliSecondsBehindSource,TotalNumberOfEventsSeen,LastTransactionId

"time",…:Connected",…:LastTransactionId",…:MilliSecondsBehindSource",…:TotalNumberOfEventsSeen"
1792698206613,true,973,164,6

164 ms behind the source, six events of any kind since the task last restarted. Right after a restart, MilliSecondsBehindSource read -1 even with four update events counted: treat -1 as unknown, not zero. For freshness, MilliSecondsSinceLastEvent read 2287 alongside Connected=true, and SELECT now() - beat_at FROM debezium_heartbeat returned 0.14 s. Alert when either grows past your heartbeat interval.

Drill 8: Secrets and personal data

The worker loads Kafka’s EnvVarConfigProvider (CONNECT_CONFIG_PROVIDERS: env, CONNECT_CONFIG_PROVIDERS_ENV_CLASS: org.apache.kafka.common.config.provider.EnvVarConfigProvider) and gets the password as an environment variable. The connector references it, and masks the new phone column with a salted hash:

  "database.password": "${env:DEBEZIUM_DB_PASSWORD}",

  "column.mask.hash.v2.SHA-256.with.salt.dispatchly-cdc-2026": "public.shipments.customer_phone",

GET /connectors/shipments-cdc/config now returns ${env:DEBEZIUM_DB_PASSWORD}, not the password. After UPDATE shipments SET customer_phone = '+91 98450 12345' WHERE tracking_code = 'DSP-1001', the event carried "customer_phone":"dded01d19b21dfc43308afe7faee03a3206c632047a8c30c2e4232166b677ea7". That’s pseudonymisation, not protection: a 10-digit phone number hashed with a known salt can be brute-forced quickly, and the salt is part of the property name, which GET …/config returns. Use v2 hashing so the same value hashes identically everywhere, and exclude truly sensitive columns. Mount secrets from your platform rather than plain environment variables, and use column.exclude.list for columns nobody downstream needs. Never log full events in production, because they carry everything the table does.

The production checklist

Tips, Tricks & Gotchas

Gotcha — RUNNING is not health. In this series a connector reported RUNNING while it shared a slot (Part 2), while its signal table was unpublished, and while its slot was lost and it retried for half an hour. Alert on MilliSecondsSinceLastEvent, heartbeat age and slot retention, not task state.

Gotcha — deleted connectors leave their slots. DELETE /connectors/{name} doesn’t touch PostgreSQL. An abandoned slot is the classic 3 a.m. disk; drop it as part of decommissioning.

Gotcha — a cap protects the disk, not the slot. A burst larger than max_slot_wal_keep_size invalidated an active slot. Measure your largest WAL bursts (bulk loads, index rebuilds) before choosing the number.

Gotcha — trust_greater_lsn trades duplicates for gaps. When the slot is ahead of the stored offset, it skips the events in between. That’s fine for an outbox connector whose slot moved because of keep-alives over unrelated WAL; it’s data loss if someone advanced the slot by hand.

Tip — the outbox connector deserves its own slot and publication. Raw table capture and contract publishing then fail, scale and get reconfigured independently, and the outbox publication holds exactly one table.

Tip — the registry’s audit log explains what the connector won’t. ProblemDetails in a task trace means “go and read the registry”: the incompatibility and the field are logged there.

Debugging / Troubleshooting

part3/slot-report.sql (drill 3) and the task trace first; then the registry’s and PostgreSQL’s logs. Every row except the last was reproduced in this series’ lab.

Key Takeaways

Story Closing

At five, Tomás erased the whiteboard except for one line: predict, then run. Every wrong prediction was now a row in the checklist.

The last thing Zoya did was merge the game-day branch into shipment-service. It deleted KafkaTopics.java, removed spring-boot-starter-kafka from the build, and rewrote ShipmentService.changeStatus (straight from git diff):

// The legacy dual write from Part 1's incident. tracking-projection no longer depends on it
// (it reads the WAL through Debezium); notifications still do, until Part 3 replaces it.
// One transaction, one database: the status change and the event commit together or not at all.
@Transactional
public Shipment changeStatus(String trackingCode, ShipmentStatus status, String hubId) {
    var shipment = find(trackingCode).moveTo(status, hubId);
    var event = new ShipmentStatusChanged(trackingCode, status, hubId, shipment.updatedAt());
    shipmentRepository.save(shipment);              // commits
    kafkaTemplate.send("shipment-events", event);   // lost if the pod dies first
    return find(trackingCode);
    var shipment = shipmentRepository.save(find(trackingCode).moveTo(status, hubId));
    outbox.publish("shipment", trackingCode, "ShipmentStatusChanged",
            new ShipmentStatusChanged(trackingCode, status, hubId, shipment.updatedAt()));
    return shipment;

Tomás approved it with a comment of one word: “Finally.”

The database had been keeping a perfect record of every change all along. It had taken her a week to stop asking her code to remember.

References

1. Debezium Outbox Event Router (3.7). https://debezium.io/documentation/reference/3.7/transformations/outbox-event-router.html
2. Debezium signalling and incremental snapshots (3.7). https://debezium.io/documentation/reference/3.7/configuration/signalling.html
3. Debezium exactly-once delivery (3.7). https://debezium.io/documentation/reference/3.7/configuration/eos.html
4. Debezium Avro serialization (3.7). https://debezium.io/documentation/reference/3.7/configuration/avro.html
5. Andreakis and Papapanagiotou, DBLog: A Watermark Based Change-Data-Capture Framework. https://arxiv.org/abs/2010.12597
6. Gunnar Morling, Reliable Microservices Data Exchange With the Outbox Pattern. https://debezium.io/blog/2019/02/19/reliable-microservices-data-exchange-with-the-outbox-pattern/
7. Chris Richardson, Pattern: Transactional outbox and Pattern: Idempotent Consumer. https://microservices.io/patterns/data/transactional-outbox.html

This is Part 3 of a 3-part series: “The Log Never Lies: Change Data Capture with Debezium.”