ApacheSpark
2 items
Series Posts
Part 7: "The Table That Remembers" — Apache Iceberg in Depth
Q3 is closed, and UK till sales just moved by £3,494.82. Ingrid opens an Iceberg table file by file: a catalog pointer, JSON metadata, Avro manifests, Parquet. Snapshots, optimistic commits and a real conflict between Spark and PyIceberg, schema and partition evolution without rewrites, time travel, tags and branches, copy-on-write vs merge-on-read, and a format v3 table with VARIANT and deletion vectors.
Part 12: "Shuffle, Spill and Skew" — Distributed Batch with Spark 4.1
A campaign will book 30% of Larkspur's sales lines to one store ID, and Rohan's answer is more executors. Ingrid rehearses it in PySpark instead: stages and lazy plans, Catalyst, shuffles, broadcast vs sort-merge joins, AQE, skew, salting and spill, measured from the Spark UI; plus ANSI mode, Spark Connect, Declarative Pipelines and Spark 4.2.