"Stateful Streams in Java" — Apache Flink II

Skim Card

One thing to remember: state, timers and input offsets are one snapshot in Flink; exactly-once output also needs sinks that commit with it and readers that read only committed data.

• Keep per-key state and timers in Flink, not a HashMap: after a killed TaskManager, the lake held exactly the batch answer’s 9,833 alerts.
• The sink decides exactly-once: Iceberg and a transactional topic matched batch row for row after the crash; the at-least-once topic repeated 659 alerts.
• A transactional topic is exact only for read_committed readers; a read_uncommitted reader also saw the 659 aborted alerts.
• Send late data to a side output: 1,141 late lines kept; the watermark alone caught 154 to 1,128, depending on reading order.
• Set a uid on every operator from day one: a job that adds them can’t restore a no-uid savepoint; forcing it re-sent 4,080 alerts.
• Keep state in POJO or Avro types: version 2 added a field and restored at double parallelism with the same alerts.
• Backpressure stretches aligned checkpoints: 6 s behind a slow operator, 95 ms unaligned on a keyed connection.

Use …	When …
hashmap state backend	State fits the heap comfortably
rocksdb, incremental checkpoints	State outgrows the heap, or checkpoints should upload only changes
Unaligned checkpoints	Backpressure stretches aligned ones, on keyed or rebalanced connections
Kubernetes Operator, upgradeMode: savepoint	Production upgrades that must keep state

Verified against: Apache Flink 2.2.1 (Java 17) with the Kafka connector 5.0.0-2.2, Apache Iceberg 1.12.0 (iceberg-flink-runtime-2.2), Apicurio Registry 3.3.3, Apache Kafka 4.4.0, Apache Polaris 1.8.0, DuckDB 2.0.0, Flink Kubernetes Operator 1.16.1 (docs only) · vendor docs · 30 October 2026

Jump to: Concepts · Hands-On · Landscape · Design Drill

Story Opening

Part 25’s store manager wanted to know a shelf was empty “while the customer is still standing there”. Rohan wrote the detector like any Spring service: a Kafka consumer, a ConcurrentHashMap of stock per store and SKU, and a scheduled executor that fired when an empty shelf had sold nothing for 30 minutes.

Then came a deploy. The new pod rebuilt the map from the morning file, re-read the topic from the last committed offset and paged stores twice about the same shelves; others it never paged, because their timers died with the old pod.

“You wrote a database without a log,” Ingrid said. “The map, the timers and the offsets are one piece of state. Flink snapshots them together. Then we kill it on purpose and count the pages.”

The Quick Map

Concepts, with use cases

Of Part 2’s five promises, this part strengthens uniqueness: one alert per event through a crash and an upgrade, proven against a batch answer. It builds on Part 26’s checkpoints and watermarks and Part 28’s cluster.

Keyed state and the process function

A KeyedProcessFunction is your code inside Flink: one record at a time, with keyed state scoped to that record’s key (ValueState, MapState, ListState). Keys hash into key groups (the max parallelism, 128 here), a range per subtask; a checkpoint snapshots every key group with the offsets it reflects. State can expire with a TTL, which counts processing time only.

What breaks without it. Your own map is lost on a crash, and wrong after a replay of records it already saw. Larkspur: book stock per store and SKU, 37,897 keys for one day. Elsewhere: a ride-hailing platform keeps each driver’s current trip and earnings so far.

Timers

ctx.timerService().registerEventTimeTimer(t) calls onTimer when the watermark reaches t; processing-time timers follow the wall clock. Timers are checkpointed with the state, “at most one timer per key and timestamp”.

What breaks without them. Nothing tells you an event didn’t happen: a shelf stopped selling, a van went quiet. A wall-clock scheduler fires again, or never, after a restart. Larkspur: STOCK_OUT when book stock hits zero and no sale follows within 30 minutes of event time. Elsewhere: a card issuer expires a payment authorisation that was never captured within seven days.

Side outputs for late data

An OutputTag gives a function a second output stream: ctx.output(LATE, line). Part 26 left the DataStream API’s lateness tools to this part: a window takes allowedLateness(...) to update its result for late events and sideOutputLateData(tag) for anything later still. This detector isn’t a window, so it decides what “late” means itself and routes late lines to its own tag.

What breaks without it. Late records vanish without an error, and a reconciliation can’t tell lost from late. Larkspur: sales from tills that uploaded hours late skip the live alert and land in stream.stock_late_sales for the nightly correction. Elsewhere: a smart-meter platform bills late readings in the next cycle and keeps them out of the live outage map.

Checkpoints, savepoints and state backends

A checkpoint is Flink’s own frequent recovery snapshot (Part 26’s barriers); a savepoint is one you trigger and own, for upgrades and moves, portable by default. The state backend decides where live state sits: hashmap (the default) keeps objects on the JVM heap, fast and bounded by it; rocksdb keeps serialised state on local disk, caches it in Flink’s managed memory, and can upload only new files per checkpoint (execution.checkpointing.incremental); forst keeps it on remote storage, the disaggregated design, “still in the experimental stage and is not fully available for production” (experimental).

What breaks without the distinction. Upgrades planned on checkpoints find them deleted with the job or unreadable by another backend. Larkspur: RocksDB with incremental checkpoints every 10 s; a savepoint for each code change. Elsewhere: a game studio keeps every player’s session state in RocksDB because it outgrew the heap.

Exactly-once sinks

Flink’s state is exactly-once by itself; output isn’t. A two-phase-commit sink writes during the checkpoint interval, pre-commits at the barrier and commits only when the checkpoint is complete. Iceberg’s FlinkSink does that, and so does KafkaSink with EXACTLY_ONCE, using Kafka transactions that readers must read with read_committed. AT_LEAST_ONCE flushes at each checkpoint and repeats whatever was sent after it.

What breaks without it. After a restore, everything emitted since the last checkpoint comes again: a pager rings twice. Larkspur: alerts go to Iceberg and two Kafka topics, one per guarantee, to count the difference. Elsewhere: usage-based billing, where a repeated minute is an overcharge.

Upgrades and state compatibility

A savepoint maps state to operators by operator ID: the uid you set, or a hash Flink generates from the graph’s topology, not your code. So a job that ran without uids can’t be restored by one that adds them, or that adds or removes an operator; setUidHash pins an old generated hash afterwards, “as a workaround” per its javadoc. The state’s schema must evolve compatibly: POJOs may add and remove fields (“Declared fields types cannot change”); Kryo-serialised types can’t evolve. Max parallelism fixes the key groups and must not change.

What breaks without it. The restore is refused. --allowNonRestoredState lets it run by discarding savepoint state no operator claims; operators without matching state start empty either way. Larkspur: version 2 adds a field to the state class and a LOW_STOCK alert, restored at twice the parallelism. Elsewhere: a bank adds a fraud rule to a running scoring job without forgetting each card’s history.

Backpressure

A slow operator stops granting network buffers, its input fills, and the wait “is propagated in the opposite direction, up the stream”. Each subtask reports how much of every second it was busy, idle or back-pressured (busyTimeMsPerSecond, backPressuredTimeMsPerSecond); the bottleneck is the busy task just downstream of back-pressured ones. An aligned checkpoint barrier waits behind queued records; an unaligned one overtakes them and stores the in-flight buffers instead (execution.checkpointing.unaligned.enabled).

What breaks if you ignore it. Lag grows, checkpoints slow down and time out, and restarts replay more input into the same bottleneck. Larkspur: at the evening peak, a slow price lookup or a slow Iceberg commit would hold the detector back the same way; in the lab a throttle plays that part. Elsewhere: an ad exchange whose enrichment call slows at peak and drags its bid counters behind.

Running on Kubernetes

The Flink Kubernetes Operator (1.16.1, compatible with Flink 2.3.x, 2.2.x and older) runs a job as a FlinkDeployment; spec.job.upgradeMode decides what a spec change does: stateless starts empty, last-state restarts from the latest checkpoint, savepoint stops with a savepoint and restores it. One cluster per job (application mode) also avoids a session cluster’s leaks (Troubleshooting). Explained, not run.

Under the Hood

A stock level that doesn’t depend on arrival order

Lines arrive out of order. processElement only buffers each line’s units in a MapState under its event time and registers a timer there; onTimer applies them when the watermark passes, oldest first. Deadline timers (sale + 30 minutes) share the queue, so sales and deadlines run in event-time order (a sale at the very deadline first). The same input gives the same alerts on every run, crashed or not, which lets Hands-On compare the stream with batch.

What “late” means here, and why the watermark alone wasn’t enough

Late = behind the watermark on arrival, or uploaded more than the watermark’s 5-minute bound after the sale. The watermark is the minimum over six partitions, so it depends on reading order: with Kafka’s default 1 MB per partition per fetch, only 154 of the 1,141 late lines were behind it; with 2 KB fetches (the job’s setting), 1,128. Presumably big fetches let some partitions run ahead (positions weren’t logged). The upload-delay test reads the record’s own clocks and gives 1,141 every time (on-time uploads take at most 90 s here, late ones two hours or more).

Hands-On

The code is in the every-record-counts folder of the companion-workspaces repository (tag every-record-counts/part-29). make up-flink-lake starts core, Kafka, Apicurio and Flink 2.2.1 (still no released Kafka connector for 2.3, re-checked 30 October); Spark, Trino and Connect stay stopped. Container memory peaked at 3.97 GiB over the captures and make ci (TaskManager 1.04, JobManager 0.93).

make up-flink-lake
uv run python -m scripts.part29_stock setup        # opening stock, empty tables and topics
make flink-jobs                                    # job versions 1 and 2 (Java 17)
uv run python -m scripts.part29_stock batch        # the expected answer (DuckDB + Python)
uv run python -m scripts.part29_stock checkpoints [--unaligned]
uv run python -m scripts.part29_stock run --kill-at 0.4
uv run python -m scripts.part29_stock upgrade --stop-at 0.5
uv run python -m scripts.part29_stock nouids [--same-topology] --stop-at 0.5

Step 0: a Java 17 module beside the Java 25 one

Flink 2.2 supports Java 11 and 17 (21 experimental, 25 not at all), so flink-jobs/ has a Java 17 toolchain while Part 27’s producers stay on 25. Flink, the Kafka connector and Iceberg are compileOnly (the cluster’s /opt/flink/lib has them); Apicurio’s deserializer, the Kafka client and Avro are bundled into a 38 MiB jar, submitted with flink run in the JobManager container.

Step 1: the detector

The rule, for business day 15 August: book stock = opening stock (the 14th’s closing level plus the overnight delivery, by Part 5’s snapshot rule; 121,450 store-SKUs) minus every on-time unit sold. A sale that empties the shelf starts a 30-minute event-time timer; if it fires with no further sale, STOCK_OUT. A sale while the books already say zero means the books are wrong: BOOK_ERROR, and the pending STOCK_OUT is cancelled, because the job is a stock count, not a refill. The opening stock is immutable reference data, read once in open(); the level that changes lives in keyed state.

public void processElement(SaleLine line, Context ctx, Collector<Alert> out) throws Exception {
    long watermark = ctx.timerService().currentWatermark();
    boolean behindWatermark = line.eventTs <= watermark;   // a window operator would drop it silently
    if (behindWatermark || line.uploadedAt - line.eventTs > maxUploadDelayMs) {
        line.watermark = watermark;
        ctx.output(LATE, line);
        return;
    }
    Integer units = pending.get(line.eventTs);
    pending.put(line.eventTs, (units == null ? 0 : units) + line.qty);
    ctx.timerService().registerEventTimeTimer(line.eventTs);
}

public void onTimer(long ts, OnTimerContext ctx, Collector<Alert> out) throws Exception {
    String key = ctx.getCurrentKey();
    StockState s = stock.value();
    if (s == null) {                          // first sale of this store and SKU today
        s = new StockState();
        s.opening = openingStock.getOrDefault(key, 0);
        s.level = s.opening;
    }
    Integer units = pending.get(ts);          // one timer per key and timestamp: sales first, then the deadline
    if (units != null) {
        pending.remove(ts);
        sell(key, s, units, ts, ctx, out);    // may emit BOOK_ERROR, or register ts + 30 min
    }
    if (s.outSince > 0 && !s.alerted && ts == s.outSince + confirmMs) {
        s.alerted = true;
        out.collect(Alert.of(key, Alert.STOCK_OUT, s.outSince, ts, s, Version.NUMBER));
    }
    stock.update(s);
}

Around it, StockOutJob: KafkaSource with read_committed set explicitly (Part 27), baskets split into lines, a per-event 5-minute watermark, dedup on the first copy of each basket_id (Part 28’s rule; TTL three days), the business-date filter, the detector, a uid on every operator. Alerts and late lines go to stream.stock_alerts and stream.stock_late_sales through Iceberg’s default FlinkSink (not the experimental SinkV2 IcebergSink), alerts also to two Kafka topics, EXACTLY_ONCE and AT_LEAST_ONCE. Job-level settings:

The restart strategy is Flink’s default with checkpointing, exponential-delay (1 s initial backoff, 1 minute maximum, unlimited attempts).

Step 2: the answer the stream must reproduce

batch applies the rule to the raw zone in DuckDB and Python, with the same input, dedup, business date, lateness test and event-time order:

on-time lines of 2026-08-15: 56,429 over 37,897 store-SKUs
expected alerts: 9,833 = STOCK_OUT 6,779 + BOOK_ERROR 3,054
expected late lines (side output): 1,141
BOOK_ERROR: 1,948 on store-SKUs that opened at zero or below (6,386 did), 978 after a STOCK_OUT, 128 within the confirm window

Every run below must produce exactly these. The lab’s WMS file and tills are generated independently, so the books disagree far more than a store’s would: these counts say nothing about Larkspur’s availability.

Step 3: backpressure, and what it does to checkpoints

Unthrottled, the replay takes seconds, so the job takes --throttle-ms 2: a lab-only map sleeping 2 ms per line, which stretches the day to about two and a half minutes and makes a perfect bottleneck. A quarter of the way in (Step 4’s run):

task (ms per second, max over subtasks)                    back-pressured   busy   idle
Source: POS basket events                                            1000      0      8
Throttle (lab)                                                          0    965     35
First copy of each basket -> Business date 2026-08-15                   0      0   1000
Stock-out detector -> (alerts-lake-alerts-lake -> alert...              0     33    967

Source back-pressured, throttle busy, the rest nearly idle. checkpoints runs the throttled replay twice and reads Flink’s checkpoint history (checkpoints 8 to 15 trimmed):

aligned checkpoints every 10 s, RocksDB, incremental; job b35fee88 FINISHED in 159 s
checkpoint  state (KiB) uploaded (KiB) in-flight (KiB) duration (ms)
         7        1,567            429               0         6,355
        16        2,597            140               0            53
median duration 6,008 ms over 10 checkpoints

unaligned checkpoints every 10 s, RocksDB, incremental; job b7fb9f67 FINISHED in 159 s
         7        1,822            656             268            68
        16        2,702            281               0            52
median duration 95 ms over 10 checkpoints

Aligned barriers wait about 6 s behind the records queued for the throttle (53 ms once nothing is queued); unaligned ones overtake them and store those buffers instead, about 270 KiB, in about 0.1 s. Flink skips unaligned checkpoints on forward (one-to-one) connections, so the lab keys the throttle onto a hash-partitioned one.

Step 4: kill the TaskManager

run --kill-at 0.4 waits until 40% of the lines are read and the last checkpoint is 7 s old (an earlier untimed kill repeated 252 alerts: the count depends on timing), sends SIGKILL to the only TaskManager, starts a new container five seconds later (as Kubernetes would replace a pod) and logs what the job does (exception class shortened):

+0.0 s   docker kill (SIGKILL) the TaskManager; last completed checkpoint 5, 7.3 s ago
+5.6 s   a new TaskManager container starts
+19.9 s   all tasks running, state restored from checkpoint 5
root cause: …ResourceManagerException: TaskManager with id 192.168.107.8:42341-0a7f14 is no longer reachable.
job 99732668 FINISHED after 184 s; restarts 1; checkpoints completed 16, failed 2
checkpoint folder at the kill (data/lab/flink-state/checkpoints/<job id>/):
  chk-5/         1 files       23 KiB
  shared/       10 files      863 KiB

Heartbeats noticed (heartbeat.interval 10 s; two failed RPCs mark a TaskManager unreachable). chk-5/_metadata maps operator IDs to state handles, Kafka offsets and timers included; shared/ holds RocksDB files that incremental checkpoints share. The restore loaded them into a fresh RocksDB and rewound Kafka. The check (data-file line trimmed):

expected (batch rule over the raw zone): 9,833 alerts
stream.stock_alerts: 9,833 rows (STOCK_OUT 6,779, BOOK_ERROR 3,054), duplicates 0, missing 0, unexpected 0; snapshots 15
  commits: checkpoints 1..end of input, 15 snapshots, checkpoint IDs strictly increasing: True; rows added 9,833
stream.stock_late_sales: 1,141 late lines (expected 1,141; same lines: True); behind the watermark on arrival: 1,124
larkspur.alerts.stock read_committed: 9,833 records, 9,833 distinct, duplicates 0, missing 0, unexpected 0
larkspur.alerts.stock-at-least-once read_committed: 10,492 records, 9,833 distinct, duplicates 659, missing 0, unexpected 0
larkspur.alerts.stock read_uncommitted: 10,492 records, 9,833 distinct, duplicates 659, missing 0, unexpected 0

• The lake is exact, row for row: each snapshot records its checkpoint (flink.max-committed-checkpoint-id), and after a restore the committer skips anything at or below it.
• The at-least-once topic repeated 659 alerts, those sent between checkpoint 5 and the kill.
• The exactly-once topic matches batch for read_committed readers; read_uncommitted (Kafka’s default, Part 27) also sees the aborted transaction’s 659.

Step 5: upgrade through a savepoint

Version 2 adds a field to the state POJO (lowSince) and a rule: LOW_STOCK when the level falls to one unit. upgrade stops version 1 with a savepoint halfway and restores version 2 at parallelism 2, without the throttle (commit, file and at-least-once lines trimmed):

v1 67641c1a stopped after 34,028 lines; savepoint file:/flink-state/savepoints/savepoint-67641c-0487f03447f2
v2 4a4a5ba5 restored at parallelism 2: FINISHED after stop --drain
expected (batch rule over the raw zone): 9,833 alerts
stream.stock_alerts: 9,833 rows (STOCK_OUT 6,779, BOOK_ERROR 3,054), duplicates 0, missing 0, unexpected 0; snapshots 10
  LOW_STOCK (version 2 only): 3,925; rows by job version: {2: 9147, 1: 4611}
stream.stock_late_sales: 1,141 late lines (expected 1,141; same lines: True); behind the watermark on arrival: 1,064
larkspur.alerts.stock read_committed: 13,758 records, 9,833 distinct, duplicates 0, missing 0, unexpected 0

The new field got its default, 0, for every restored key; the key groups split across two subtasks; dropping the stateless throttle needed no -n, since every stateful operator kept its uid. STOCK_OUT and BOOK_ERROR equal the batch answer; LOW_STOCK covers sales after the savepoint. flink stop --drain ends version 2: the watermark jumps to the end of time, every timer fires, the sinks commit. Two lab problems on the way are in Troubleshooting.

Step 6: the same upgrade without operator IDs

nouids runs version 1 without a single uid, so Flink generates IDs from the graph, then restores version 2, which sets uids and has no throttle (trimmed to the refusal and the two tables; path shortened):

v1 a569b56b (no uids) stopped after 29,936 lines; savepoint file:/flink-state/savepoints/savepoint-a569b5-799cce97f316
v2 with uids, no throttle: refused: Failed to rollback to checkpoint/savepoint file:/flink-state/savepoints/savepoint-a569b5-…. Cannot map checkpoint/savepoint state for operator ae87cbae2d8f896d9c88a2c7fcd3b394 to the new program, because the operator is not available in the new program. …
v2 with --allowNonRestoredState: FINISHED
stream.stock_alerts: 13,913 rows (STOCK_OUT 9,507, BOOK_ERROR 4,406), duplicates 4080, missing 0, unexpected 4080; snapshots 8
stream.stock_late_sales: 1,312 late lines (expected 1,141; same lines: False); behind the watermark on arrival: 1,295

Refused, rightly. With --allowNonRestoredState no operator found its state, the source restarted from the earliest offset, and the 4,080 alerts and 171 late lines version 1 had committed were written again (duplicates and unexpected are the same 4,080 rows). nouids --same-topology restores a no-uid version 2 that keeps the throttle, the same graph shape:

v2 without uids, same operators (throttle included), no -n: FINISHED after stop --drain
stream.stock_alerts: 9,833 rows (STOCK_OUT 6,779, BOOK_ERROR 3,054), duplicates 0, missing 0, unexpected 0; snapshots 15

Generated IDs follow topology, so the restore worked; uids keep it working when the topology changes.

The Landscape

Current product names as of 30 October 2026. A cell that couldn’t be verified says so.

Tips & Gotchas

Gotcha — state TTL counts processing time. The dedup forgets a basket three wall-clock days after seeing it, so a replay of last month remembers everything. Event-time cleanup is a timer you register.

Gotcha — size transaction.timeout.ms for EXACTLY_ONCE. The 5.0.0 builder defaults to one hour; above the broker’s 15-minute transaction.max.timeout.ms, the producer is refused with InvalidTxnTimeoutException (Kafka’s error; not run here). Too short is worse: if tolerated checkpoint failures keep a transaction open past its timeout, the broker aborts it, the commit fails, the job restarts.

Gotcha — --allowNonRestoredState turns an error into wrong numbers. It is for removing an operator on purpose; used to get past a refused restore, it re-sent 4,080 alerts (Step 6).

Tip — put a savepoint round trip in CI. Savepoint the old jar on a small topic, restore the new one, compare with a batch answer: it catches missing uids, Kryo-serialised state and changed max parallelism before production.

Troubleshooting

Design Drill

The question: “We distribute electricity to four million homes. A Flink job keeps per-meter state from smart-meter readings and ‘last gasp’ power-loss messages, and raises outage alerts per feeder. Since last month it restarts about every hour, and consumer lag grows after each restart. Diagnose and fix it.”

Clarify first

• What is each restart’s root exception, and is it the same every time?
• Exactly hourly? In line with checkpoints, deploys, credential expiry?
• How have checkpoint duration, size and restore time moved?
• Flink restarting the job, or Kubernetes restarting pods? What changed last month?

The shape of a strong answer

• Start from the exception history and pod events; lag is the symptom. Strings to look for: “Checkpoint expired before completing.” and “Exceeded checkpoint tolerable failure threshold.” (with execution.checkpointing.tolerable-failed-checkpoints at 0, one failure fails the job); “Heartbeat of TaskManager with id … timed out.” (often long GC pauses); a pod exit code 137 (OOMKilled: RocksDB’s native memory over the pod limit, when managed memory is off or slots outnumber its budget).
• The likely spiral: state grows without a TTL; behind a busy operator aligned barriers wait, a checkpoint times out, the job spends minutes restoring large state and re-reads everything since, while lag builds.
• Find the bottleneck from busy and back-pressured time per task, and per-partition lag for a hot key (one feeder with a million meters); fix that first: scale it within the max parallelism, split the key, make lookups asynchronous.
• Let checkpoints finish under pressure: unaligned on keyed edges, incremental RocksDB, a timeout and tolerated failures sized from measured durations. Bound state with a TTL or a timer that clears a silent meter.
• If it is exactly hourly, suspect a one-hour clock: a KafkaSink transaction outliving its timeout while checkpoints fail, hourly credentials, a CronJob.
• Prove it: restarts per day, checkpoint p99 against the timeout, lag recovery after a restart, a day’s alerts reconciled with batch.

Trade-offs to say out loud

• Unaligned checkpoints store in-flight data: larger, faster under backpressure, not on forward or broadcast edges.
• Tolerating failed checkpoints lengthens the replay after a real crash.
• Scaling out needs a savepoint round trip and max parallelism chosen on day one.

Common weak answers

• “More memory and TaskManagers” without naming the busy task.
• A longer checkpoint interval: bigger replays, the same spiral.
• Turning checkpointing off, and exactly-once with it.

Further Reading

1. Apache Flink 2.2 documentation, “Process Function” (keyed state, timers). https://nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/operators/process_function/
2. Apache Flink 2.2 documentation, “Savepoints” (operator IDs, restore, --allowNonRestoredState). https://nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/state/savepoints/
3. Apache Flink 2.2 documentation, “State Backends” and “Checkpointing under backpressure”. https://nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/state/state_backends/ and https://nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/state/checkpointing_under_backpressure/
4. Apache Iceberg 1.12 documentation, “Flink Writes”. https://iceberg.apache.org/docs/latest/flink-writes/
5. Flink Kubernetes Operator 1.16 documentation, “Job Management”. https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-release-1.16/docs/managing/job-management/

Story Closing

Rohan’s replay of 15 August matched the batch answer alert for alert, through a killed TaskManager and an upgrade, and every late till line sat in a table, counted.

“Some stores still got paged twice,” Farah said, reading the at-least-once topic.

“Which is why the stores will read the other one,” Ingrid said, “with read_committed.”

In Part 30, the same stock level runs at three latencies, hourly batch, one-minute Iceberg commits and sub-second OLAP, with the freshness and the cost of each measured.

This is Part 29 of “Every Record Counts: Data Engineering for Backend Engineers” — Season 3: Real Time, Serving and AI.