Project Every Record Counts: Data Engineering for Backend Engineers
Part 16: "Changes, Not Snapshots" — CDC into the Lakehouse
Rohan wants to append Debezium's events to the lake and call it a table. Ingrid insists on folding them into state instead: upserts, hard and soft deletes, ordering by LSN, an initial snapshot plus a stream, a history table beside a current-state table, MERGE vs equality deletes, idempotent micro-batches and compaction. Built with Debezium 3.7, Kafka 4.4 and Spark Structured Streaming into Iceberg, it finally removes the £57,064.96 of hard-deleted orders from the 0.7%.