"Shuffle, Spill and Skew" — Distributed Batch with Spark 4.1
A campaign will book 30% of Larkspur's sales lines to one store ID, and Rohan's answer is more executors. Ingrid rehearses it in PySpark instead: stages and lazy plans, Catalyst, shuffles, broadcast vs sort-merge joins, AQE, skew, salting and spill, measured from the Spark UI; plus ANSI mode, Spark Connect, Declarative Pipelines and Spark 4.2.
Marketing’s next campaign was a national “scan and go” promotion in the app. Every app checkout would be booked to one store ID, GB-0001, the Glasgow pilot store, and the forecast said that, at peak, that ID would carry 30% of all sales lines.
“Fine,” Rohan said. “That’s what Spark is for. If the nightly job slows down, we add executors.”
Ingrid didn’t look convinced. “Executors help when the work splits. Thirty per cent of the data behind one key doesn’t split by itself. Let’s rehearse it before the campaign does.”
The Quick Map
Backend instinct
Data engineering reality
Why it matters
More machines make it faster
A stage finishes when its slowest task does
One hot key turns 200 cores into one
A join is an index lookup per row
A distributed join co-locates rows by key: shuffle both sides or broadcast the small one
The shuffle is the expensive part, and it’s yours to avoid
Code runs when you call it
Transformations build a plan; an action runs it
Cost and errors appear at the write, not on the line that caused them
Out of memory means a crash
Sorts, joins and aggregations spill to disk
Spill is the early warning; reading it is part of the job
Concepts, with use cases
Driver, executors, stages and tasks
A Spark application has one driver, which plans the work and schedules it, and executors, JVM processes on worker machines that run tasks. Your code builds a DataFrame, a description of a computation; nothing runs until an action (a write, collect(), count()). That’s lazy evaluation: the driver turns the whole chain into one plan, cuts it into stages at every point where data has to be regrouped by key, and runs each stage as one task per partition. The stages form a DAG: a stage starts when the stages it reads from have finished.
What breaks without it. Eager, step-by-step execution can’t skip columns nobody uses or push a filter into the scan. Larkspur: the gold job’s DataFrame took 0.11 s to define and read nothing (step 1). Elsewhere: a telco’s thirty-step billing job, planned as three stages.
Catalyst and Tungsten
Catalyst is Spark SQL’s optimiser: it resolves names against the catalog, rewrites the logical plan (pushing filters and column selections into the scan, folding constants), then chooses a physical plan, including the join strategy. Tungsten began in 2015 as the project to bring Spark “closer to bare metal”: explicit memory management with binary rows instead of JVM objects, cache-aware algorithms, and runtime code generation. In a plan, a * before an operator means whole-stage code generation compiled it, with its neighbours, into one Java function.
What breaks without it. Filters and column choices stay where you wrote them, so every scan reads more than it needs, and SQL, Python and Scala would each plan differently. Larkspur: the gold job’s date range reached the Iceberg scan as business_date >= 20635, business_date <= 20726 (step 1). Elsewhere: a bank’s SQL and DataFrame reports, one plan.
Narrow and wide transformations, and the shuffle
A narrow transformation (filter, projection, a map-side join against a broadcast table) computes each output partition from one input partition. A wide one (group by key, join two big tables, sort, distinct) needs every row with the same key in one place, so Spark inserts an Exchange: the shuffle. Map tasks write rows into files split by hash(key) mod N; reduce tasks fetch their piece from every map task. Spark’s own guide calls it “a complex and costly operation” involving “disk I/O, data serialization, and network I/O”.
What breaks without it. Nothing: a wide operation needs it. The work is to need fewer of them and to make them smaller. Larkspur: the gold job has two (step 1); the broadcast join has none. Elsewhere: an ad network’s impressions-to-clicks join, whose shuffle is most of its bill.
flowchart LR
subgraph map[Map stage: 12 tasks read the lines]
M1[task 1] & M2[task 2] & M3[task …12]
end
subgraph files[Shuffle files: one per map task, split by hash of store_id]
end
subgraph reduce[Reduce stage: 200 tasks]
R1[partition 0 ~0.5 MiB]
RH[partition with GB-0001 37.9 MiB]
R2[partition 199 ~0.5 MiB]
end
M1 & M2 & M3 --> files --> R1 & RH & R2
Join strategies
Broadcast hash join. The small side is collected, built into a hash table and sent to every executor; the big side is joined where it lies, with no shuffle. Spark chooses it when it estimates one side below spark.sql.autoBroadcastJoinThreshold (10 MB by default).
Sort-merge join. Both sides are shuffled by the join key and sorted; each task merges its two sorted runs. The default for two large sides; it spills rather than fails.
Shuffle hash join. Both sides are shuffled, and each task builds a hash table from its share of the smaller side, which saves the sort. Spark prefers sort-merge (spark.sql.join.preferSortMergeJoin, an internal setting, true) unless a hint or AQE says otherwise.
Hints (BROADCAST, MERGE, SHUFFLE_HASH, SHUFFLE_REPLICATE_NL, in that priority) override the choice, with “no guarantee” Spark obeys. What breaks without the choice: shuffling 25 million lines to meet 600 store rows. Larkspur: step 3 measures both. Elsewhere: an e-commerce site broadcasting its category table into clickstream jobs.
Adaptive query execution
AQE (on by default since Spark 3.2) re-plans at stage boundaries using what the finished stages actually wrote, not what the optimiser guessed. It does three things: coalesces many small shuffle partitions into fewer tasks, converts a sort-merge join to a broadcast join when one side turns out small (spark.sql.adaptive.autoBroadcastJoinThreshold), and splits skewed partitions in sort-merge and shuffled hash joins (per the config’s doc; the tuning page mentions only sort-merge). Spark 4.2 adds skew handling for broadcast hash joins too.
What breaks without it. Plans built on estimates: after three joins and a filter, the optimiser’s row counts are guesses. Larkspur: with AQE’s default coalescing, the lab’s 8 shuffle partitions became 4 tasks in the gold job (step 1). Elsewhere: a logistics firm whose nightly input swings tenfold between weekdays and peak season.
Skew and salting
Skew is uneven data per key: one partition gets far more rows than the others, so one task does that work alone while the other cores idle. AQE treats a partition as skewed when it’s larger than skewedPartitionFactor (5) times the median and larger than skewedPartitionThresholdInBytes (256 MB). Salting is the manual fix: add a salt column (0..n−1) to the hot keys’ rows, copy the matching rows on the other side n times, and join on key and salt, so the hot key spreads over n partitions.
What breaks without it. The job’s runtime becomes the hot key’s runtime, and adding executors changes nothing. Larkspur: the campaign’s store ID; also the NULL or “unknown” keys every source eventually sends. Elsewhere: a social network’s follower joins, where one celebrity has 100 million followers.
Partitions, memory and spill
Input is split into partitions (Iceberg plans splits from files) and shuffles go into spark.sql.shuffle.partitions (200 by default; the lab’s spark-defaults says 8). Executor memory is one unified region, spark.memory.fraction (0.6) of the heap minus 300 MB, shared by execution (shuffles, joins, sorts, aggregations) and storage (cached data). When a task’s operator can’t get more execution memory, it spills: writes sorted runs to local disk and merges them later. Caching (df.cache()) keeps a DataFrame in storage memory for reuse; it’s lazy, costs that memory, and is worth it only when the same intermediate is read several times.
What breaks without it. Tasks larger than memory would kill the executor. Larkspur: the hot task spilled and finished (step 3); a job that built several gold tables from one filtered silver DataFrame would cache it once. Elsewhere: a genomics pipeline caches a reference panel that twenty joins reuse.
Spark 4.x: what changed, what’s new, what the lab runs
ANSI mode, default since 4.0. Invalid casts, overflows and division by zero raise errors instead of returning NULL or wrapping; try_cast, try_add and friends opt out per expression. It protects Larkspur’s minor-unit sums (Gotcha below); a payroll system gets the same guard against silently wrapped totals.
Spark Connect (since 3.4): the client sends unresolved logical plans over gRPC and gets Arrow batches back, without Py4J, SparkContext or RDDs (pyspark-client; spark.api.mode switches an application, since 4.0). Larkspur’s notebooks could share one Spark server instead of starting drivers; a SaaS company embeds Spark queries in a web service. Not run in the lab: the image lacks the client’s Python packages (grpc, pandas).
Spark Declarative Pipelines (SDP, since 4.1.0): declare datasets and flows (@dp.materialized_view, @dp.table, SQL CREATE MATERIALIZED VIEW), and Spark works out “the execution graph, dependency ordering, parallelism, checkpoints, and retries”. The docs carry no status label; Databricks’ Lakeflow pipelines (formerly DLT) build on it. Larkspur’s silver and gold could be two declared datasets instead of a hand-ordered job (orchestration proper is Part 19); a media company could declare its hourly viewing aggregates the same way.
Spark 4.2.0 (14 July 2026) adds GEOMETRY/GEOGRAPHY types, a SQL CHANGES clause, Auto CDC in SDP “for declarative SCD Type 1 upserts” (Python and Spark Connect APIs only; no AUTO CDC SQL), Arrow Python UDFs by default and Real-Time Mode for PySpark (“This API is evolving”). The lab stays on 4.1.3: Maven Central has no iceberg-spark-runtime-4.2_2.13, and Docker Hub no images for 4.2.1 or 4.1.4 (checked 28 October 2026).
PySpark vs Scala and Java. DataFrame and SQL code performs the same in every language, because Catalyst plans it and the JVM runs it; only Python UDFs cost extra, and the typed Dataset API is Scala/Java only. By downloads (about 48 million pyspark last month on pypistats, CI included) and in our experience, most new Spark code is PySpark, so this series writes PySpark.
Under the Hood
What a shuffle writes
Each map task sorts its output by target partition and writes one data file plus an index to local disk, compressed with lz4 (spark.io.compression.codec). Each reduce task fetches its byte range from every map file: 12 × 200 = 2,400 blocks for one join here. The files outlive the map tasks, so a failed reduce task retries without rereading the source. Step 3’s lines shuffle was 140.3 MiB for 25 million rows: Catalyst had pruned 9 of 13 columns before the Exchange, and lz4 compressed the rest.
How AQE splits a skewed partition
After the map stage, the driver knows the size of every (map task, reduce partition) block. OptimizeSkewedJoin marks a reduce partition as skewed if it exceeds the threshold described above, then splits it along map-task boundaries: groups of map outputs that add up to roughly the target size (the average non-skewed partition, or advisoryPartitionSizeInBytes if that’s larger). Each piece is joined with a full copy of the other side’s matching partition. So a hot partition splits into at most as many pieces as there were map tasks: 12 here, which is why 200 tasks became 211 in step 4. Only a side whose rows may safely meet duplicated partners can be split (Gotchas).
Why the hot task spilled
The Spark UI reports a unified memory region of 741.6 MiB (step 1): (1.5 GB heap − 300 MB) × 0.6. Spark’s ExecutionMemoryPool lets each of N active tasks take between 1/2N and 1/N of the execution share before it must spill. With four tasks running, the hot task’s sort could count on roughly 90–185 MiB, and its 7.57 million rows needed more. So it sorted in chunks, wrote them to disk (400 MiB of rows in their in-memory form, 33.8 MiB serialized), and merged them. Nothing failed; the task got slower. At 50 times the volume, Larkspur’s real size, the same task would spill gigabytes while 199 others sat idle.
Hands-On
The code is in the every-record-counts folder of the companion-workspaces repository (tag every-record-counts/part-12). Spark 4.1.3 runs in local mode (4 vCPUs, 1.5 GB driver heap, 2.5 GB container limit): driver and executor share one JVM, so stages, tasks and shuffle files are real; only the network is missing. Input is Part 10’s part10.pos_monthly (four years, 72.67 million lines).
Terminal window
makeup-spark# core + Spark; Part 10's tables must exist
uvrunpython-mscripts.part12_sparkreset# drop Part 12's tables and their files
uvrunpython-mscripts.part12_sparkpipeline# step 1 (run it twice: step 2)
uvrunpython-mscripts.part12_sparkansi# the ANSI Gotcha
uvrunpython-mscripts.part12_sparkbuild# step 3: the campaign rehearsal table
uvrunpython-mscripts.part12_sparkskew# steps 3-5: five ways to run one join
Every number below comes from the Spark UI’s REST API: spark/jobs/part12_ui.py reads the final plan and each stage’s task list (task times, shuffle bytes, spill), as the Stages tab shows them.
Step 1: the nightly job in PySpark
spark/jobs/part12_pipeline.py writes three Iceberg tables through Polaris: silver.dim_store (the 600-row store file), silver.sales_line (Q3 2026’s lines, partitioned by month) and gold.daily_sales_by_format. The gold half:
unified memory (execution + storage): 741.6 MiB of a 1.5 GB heap
silver written in 8.7 s
gold DataFrame defined in 0.11 s; no data read yet
== Physical Plan ==
AdaptiveSparkPlan (14)
+- HashAggregate (13)
+- Exchange (12)
+- HashAggregate (11)
+- HashAggregate (10)
+- Exchange (9)
+- HashAggregate (8)
+- Project (7)
+- BroadcastHashJoin Inner BuildRight (6)
:- Filter (2)
: +- BatchScan lakehouse.silver.sales_line (1)
+- BroadcastExchange (5)
+- Filter (4)
+- BatchScan lakehouse.silver.dim_store (3)
(1) BatchScan lakehouse.silver.sales_line
IcebergScan(table=lakehouse.silver.sales_line, …, filters=business_date IS NOT NULL, business_date >= 20635, business_date <= 20726, store_id IS NOT NULL, …)
(Trimmed: other nodes’ details, three one-task stages.) Read the plan bottom-up. The store table is broadcast, so the 4.4 million lines never shuffle for the join. countDistinct costs two aggregations: shuffle (9) groups by the four output keys plus basket_id, shuffle (12) by the four. Stage 5 scanned the three month files in three tasks and wrote 32.9 MiB of partial aggregates; stage 7 read them in four tasks, AQE having coalesced the lab’s 8 shuffle partitions (this printout is the plan before AQE, isFinalPlan=false).
Step 2: run it again
The second run left every count unchanged and gave each table a second snapshot (append, overwrite): writeTo(...).overwritePartitions() replaced the three month partitions of silver and gold instead of appending them (Part 2’s partition-scoped overwrite), and createOrReplace() replaced the small dimension. Because whole partitions are replaced, the job only accepts whole months.
Step 3: the campaign, broadcast vs sort-merge
build makes part12.campaign_lines from the last twelve months of Part 10’s history, plus re-keyed copies of other UK stores’ baskets booked to GB-0001 until it carries 30%. That’s synthetic, and a year rather than a quarter so the hot partition is big enough to spill in this driver.
|store_id| lines|pct_of_lines|
| GB-0001|7513089| 30.01|
| GB-0185| 67721| 0.27|
| GB-0111| 67273| 0.27|
(Trimmed: the totals table, 25,039,235 lines from 600 stores, 1 October 2025 to 30 September 2026.)
The query joins the lines to silver.dim_store and sums by day, format, city and currency. Every variant runs Spark’s default 200 shuffle partitions with coalescing off and AQE’s skew join off (on by default; step 4 turns it on), so each partition is one task, as on a cluster with about 200 cores; otherwise AQE would merge them into a handful of tasks for the lab’s 4 cores. merge sets spark.sql.autoBroadcastJoinThreshold to -1: the plan you get when the small side is too big to broadcast.
variant broadcast: 2.9 s (median of 3), 18,615 rows, 25,039,235 lines, checksum 980,281,444,872
(Trimmed: the aggregation’s Exchange line, one-task stages, the repeated header.) With the broadcast, the join and a partial aggregation ran inside the 12 scan tasks and only 0.9 MiB of partial sums crossed a shuffle. Without it, all 25 million lines were shuffled by store_id (140.3 MiB), and in stage 34 one task got 7,570,057 rows (GB-0001’s lines plus the other stores that hash to the same partition): 2.86 s against a median of 0.04 s, and the only task that spilled.
Step 4: AQE’s skew join, default and scaled
variant aqe_scaled: 4.9 s (median of 3), 18,615 rows, 25,039,235 lines, checksum 980,281,444,872
aqe_default (output trimmed; see the table) switches the skew join on at its defaults, and nothing changed: the hot partition was 37.9 MiB, about 75 times the others’ 0.5 MiB average, but under 256 MB. The lab holds about a fiftieth of Larkspur’s volume, so aqe_scaled scales the size thresholds by 1/64, as Part 10 scaled file sizes: 4 MB to count as skewed, 1 MB advisory (Spark’s docs advise keeping the threshold above the advisory size). Then the plan says SortMergeJoin(skew=true), the lines’ shuffle read is skewed, the hot partition became 12 tasks (211 in all), the largest task read 650,977 rows, and nothing spilled. At full volume the hot partition would be about 1.9 GB, well past the default threshold.
Step 5: salting by hand
WITH lines AS (
SELECT l.*, CASEWHENl.store_idIN ({hot}) THEN pmod(xxhash64(l.basket_id), {salts}) ELSE0ENDAS salt
FROMpart12.campaign_lines l),
stores AS (
SELECT s.*, salt
FROMlakehouse.silver.dim_store s
LATERAL VIEW explode(sequence(0, CASEWHENs.store_idIN ({hot}) THEN {salts} -1ELSE0END)) t AS salt)
SELECTl.business_date, s.format, s.city, l.currency, count(*) AS lines, sum(l.amount_minor) AS sales_minor
FROM lines l JOIN stores s ONl.store_id=s.store_idANDl.salt=s.salt
GROUP BYl.business_date, s.format, s.city, l.currency
Python fills in {hot} ('GB-0001') and {salts} (16): only the hot store is salted, and its dimension row becomes 16. The shuffle grows (the salt travels too); the hot task shrinks.
Variant
Join
Time
Tasks doing the join
Median / max task
Largest task’s rows
Shuffle written
Spill (memory / disk)
broadcast
broadcast hash
2.9 s
12 (scan)
0.46 / 0.59 s
(no shuffle read)
0.9 MiB
0
merge
sort-merge
6.6 s
200
0.04 / 2.86 s
7,570,057
149.6 MiB
400.0 / 33.8 MiB
aqe_default
sort-merge
6.1 s
200
0.04 / 2.80 s
7,570,057
149.6 MiB
360.0 / 31.7 MiB
aqe_scaled
sort-merge, skew split
4.9 s
211
0.04 / 0.26 s
650,977
149.7 MiB
0
salted
sort-merge on key + salt
6.1 s
200
0.05 / 0.25 s
604,431
177.2 MiB
0
All five returned the same 18,615 rows and checksum. Four cores can’t run 200 tasks at once, so wall-clock gaps stay modest here; on a cluster that runs them in parallel, the stage lasts as long as its maximum task: 2.86 s against 0.25–0.26 s. Spill varies with how many tasks share memory at the moment (320–416 MiB in three runs). uv run pytest -q tests/test_part12.py asserts all of it on a fresh run (12 passed in 2 min 52 s). The Spark container peaked at 1.95 GiB of its 2.5 GB, all lab containers at 2.97 GiB.
The Landscape
As of 28 October 2026, from each vendor’s docs.
Need
Open source
AWS
GCP
Azure
Snowflake / Databricks
Notes
Spark clusters you size
Apache Spark 4.2.1 / 4.1.4 on YARN, Kubernetes or standalone
EMR: emr-spark-8.1.0 LTS (Spark 4.1.1); EMR 7.14 still Spark 3.5.8
Managed Service for Apache Spark (formerly Dataproc on Compute Engine): image 3.0, Spark 4.1.2
Check the minor version before relying on a 4.2 feature
Serverless Spark jobs
—
EMR Serverless (emr-spark-8.1.0); Glue 6.0 (Spark 4.1.1, GA 21 Aug 2026; 5.1 stays the default)
Managed Service for Apache Spark, serverless (formerly Google Cloud Serverless for Apache Spark): runtime 3.0 = Spark 4.0.1
not verified (2026-10-28)
Snowpark Connect for Apache Spark (GA; Spark 3.5.x API on Snowflake compute); Databricks serverless: not verified (2026-10-28)
Serverless hides executors, not shuffles: skew still bills you
Skew and join tuning
AQE, on by default; skew join for sort-merge and shuffled hash joins
same engine
same engine
same engine
Databricks: AQE on by default, skew join factor 5 and 256 MB
Photon (Databricks’ vectorised engine) runs the plan, it doesn’t remove the skew
Tips & Gotchas
Gotcha — the plan changes when a table grows. The broadcast decision compares an estimate with 10 MB. A dimension that grows past it, or that Spark overestimates, silently becomes a sort-merge join that shuffles the big side. Pin known-small joins with a BROADCAST hint and alert on runtime. Too large a broadcast fails instead: driver memory, or the 300 s spark.sql.broadcastTimeout.
Gotcha — AQE can’t split everything. It splits the left side of inner, left outer, left semi and left anti joins and the right side of inner and right outer joins; a full outer join gets no help, and neither do window functions partitioned by the hot key. Salting, or handling the hot keys separately and unioning, is the fix there.
Gotcha — salt with a hash, not rand(). A failed task is recomputed. rand() is non-deterministic, so a recomputed partition can hand rows different salts than the first attempt did, and Spark must treat the stage as indeterminate. pmod(xxhash64(basket_id), 16) gives the same salt every time.
Gotcha — Spark 4 is ANSI by default. Code from Spark 3 that relied on bad values turning into NULL now fails. From sql/part12/ansi.sql (trimmed):
spark.sql.ansi.enabled true
> SELECT CAST('12.40 GBP' AS BIGINT) AS amount_minor
ERROR: NumberFormatException: [CAST_INVALID_INPUT] The value '12.40 GBP' of the type "STRING" cannot be cast to "BIGINT" because it is malformed. …
> SELECT CAST(9223372036854775807 AS BIGINT) + 1 AS total_minor
ERROR: ArithmeticException: [ARITHMETIC_OVERFLOW] long overflow. Use 'try_add' to tolerate overflow and return NULL instead. …
spark.sql.ansi.enabled false
amount_minor total_minor
NULL -9223372036854775808
For money in minor units the error is the feature: the old behaviour wrapped a sum to a negative number without a word. Use try_cast where bad input is expected and counted, not to silence errors.
Tip — read the stage page, not the job’s duration. Skew is the gap between median and maximum task in a stage’s summary metrics; spill shows as “Shuffle spill (memory)” (deserialized) and “(disk)” (serialized). Look there first when a job slows without more data.
Troubleshooting
Symptom
Likely cause
How to confirm
Fix
RuntimeIOException: java.io.EOFException: Reached the end of stream with 628 bytes left to read from VectorizedParquetReader
A read from SeaweedFS cut short (the intermittent fault Part 10 met); local mode gives a task one attempt
The same job passes on a re-run
Submit with --master local[*,3] so a task can retry, as on a cluster
A date filter doesn’t prune the Iceberg scan
The filter wraps the column (date_format(business_date, 'yyyy-MM') IN …)
explain(): the IcebergScan has no business_date filter
Filter on a range of the raw column (step 1’s plan shows it pushed)
Design Drill
The question: “An ad-tech company’s nightly Spark job joins about 3 billion impressions with clicks and conversions and attributes spend to campaigns. It ran in 40 minutes for a year. The week a major advertiser launched a holiday campaign, it took 6 hours, though total volume rose only 25%. Diagnose it, fix it, and keep it from happening again.”
Clarify first
Did anything else change that week: code, cluster size, Spark version, table layout?
Which stage got slower: one task, or all of them?
What are the join keys (campaign, user, device), and can they be NULL?
How big is each join’s smaller side, and was any join broadcast before?
The shape of a strong answer
25% more data can’t explain 9× the time; suspect skew or a plan change before capacity.
Open the slow stage: max task against median, the largest task’s shuffle read and spill. One task at hours, median at seconds: skew.
Find the key: count rows per join key (one campaign ID, or the NULL user ID of unconsented traffic).
Compare plans with the last fast run: a campaign table past the broadcast threshold turns a broadcast join into a sort-merge join.
Quick fixes: re-broadcast the small side with a hint; check AQE’s skew join is on with fitting thresholds; filter out NULL keys before an inner join (they never match).
Durable fix for known hot keys: salt them deterministically, or process the top keys in a separate branch and union the results.
Prevent it: alert on max-to-median ratio and runtime against the window, profile rows per key, and rehearse big campaigns on synthetic hot-key data (step 3).
Trade-offs to say out loud
Raising the broadcast threshold moves the risk to driver and executor memory.
Salting multiplies the other side’s hot rows by the salt count and adds code; AQE costs nothing to try but has limits (full outer joins, windows).
Executors don’t help one hot task; they cost money while idle.
Common weak answers
“Scale the cluster”: the hot task still runs on one core.
Repartitioning by the skewed key: the hot key just moves to another task.
The rehearsal went into the campaign runbook: broadcast the store dimension explicitly, keep AQE’s skew join on with thresholds checked against real partition sizes, and salt GB-0001 if it ever joins a table too big to broadcast. Of the five promises from Part 2, this part strengthened timeliness: a job whose slowest task is under control finishes inside its window.
“So Spark for the heavy nightly work,” Rohan said. “And for the 9 a.m. dashboards?”
“Different problem,” Ingrid said. “Three hundred people asking small questions at once.”
In Part 13, Rohan meets Trino, the cloud warehouses and real-time OLAP, and learns to pick an engine by workload, concurrency and the shape of the bill.
This is Part 12 of “Every Record Counts: Data Engineering for Backend Engineers” — Season 1: Foundations & the Lakehouse.