Broadcast Variables & Accumulators

Stories about how Spark shares read-only data and collects metrics across distributed tasks.

Stories

Shared State in a Distributed Job — broadcast variables, tree-based distribution, accumulators, and their limitations

How Spark Chooses a Join — broadcast hash join uses broadcast variables to distribute the small side
Bytes on the Wire: How Spark Serializes Data for Tasks and Shuffles — how broadcast data is serialized before being sent to executors
The Driver, the Executors, and How a Job Actually Runs — the driver/executor model that broadcast and accumulators sit on top of