sparklearning

Cluster & Deploy

Stories about how Spark runs on cluster managers — from submitting a job to running tasks on worker nodes.

From spark-submit to Running Tasks: The Resource Negotiation Story — spark-submit launch chain, driver startup, executor acquisition, client vs cluster mode
Spark on YARN: ApplicationMaster, Containers, and the Queue — YARN resource model, ApplicationMaster, container lifecycle, queue scheduling
Spark on Kubernetes: Pods, Namespaces, and Ephemeral Executors — driver/executor pods, container images, ephemeral storage, RBAC

The Driver, the Executors, and How a Job Actually Runs — the roles of the driver and executors once they are running
Elastic Executors: How Dynamic Allocation Grows and Shrinks the Cluster — how Spark requests and releases executors at runtime via the cluster manager
SparkConf to Code: How Configuration Reaches the Component That Needs It — how configuration set at submission time propagates to executors
The Event Log: A Complete Record of Everything That Happened in Your Job — how to observe completed jobs after the cluster tears them down
The Journey of a Shuffle Record — the External Shuffle Service that makes dynamic allocation safe