Project Every Record Counts: Data Engineering for Backend Engineers
Part 12: "Shuffle, Spill and Skew" — Distributed Batch with Spark 4.1
A campaign will book 30% of Larkspur's sales lines to one store ID, and Rohan's answer is more executors. Ingrid rehearses it in PySpark instead: stages and lazy plans, Catalyst, shuffles, broadcast vs sort-merge joins, AQE, skew, salting and spill, measured from the Spark UI; plus ANSI mode, Spark Connect, Declarative Pipelines and Spark 4.2.