sparklearning

UI & Metrics

Stories about observing Spark jobs — the live Spark UI and the event log for completed applications.

Reading the Spark UI: What Every Tab Is Actually Telling You — Jobs, Stages, SQL, Executors, Storage, and Environment tabs; task metrics and what they reveal
The Event Log: A Complete Record of Everything That Happened in Your Job — event log structure, enabling logging, History Server, task metrics, programmatic parsing

EXPLAIN Yourself: How to Read a Spark Physical Plan — the SQL tab in the UI shows the physical plan; this story teaches you how to read it
Taming the Shuffle: Partition Count, Spill, and the Right Shuffle for Your Job — the Stages tab’s shuffle metrics are the primary diagnostic for shuffle problems
Out of Memory: A Field Guide to Spark OOM Errors — the Executors tab and task metrics help locate which executor or task ran out of memory
When One Partition Holds Up Everyone: The Data Skew Story — task duration distribution in the Stages tab is how you spot data skew
SparkConf to Code: How Configuration Reaches the Component That Needs It — the Environment tab shows the actual merged configuration a job is running with