Execution & Scheduling

Stories about how Spark turns an action into a DAG of stages and tasks, and how the driver and executors coordinate.

Stories

The Driver, the Executors, and How a Job Actually Runs — driver role, executor model, job/stage/task hierarchy, data locality, heartbeats
Elastic Executors: How Dynamic Allocation Grows and Shrinks the Cluster — scale-up/scale-down timers, External Shuffle Service, executor decommissioning

From One Action to Many Tasks — the DAG Scheduler and Task Scheduler that turn jobs into running tasks
The Journey of a Shuffle Record — what happens between stages when tasks exchange data
The Two Lives of Spark’s Memory — how executor memory is managed once tasks are running
How Spark Survives Failure — what happens when an executor or task fails mid-job
From spark-submit to Running Tasks: The Resource Negotiation Story — how executors are provisioned before they can run any tasks