Cluster & Deploy

Stories about how Spark runs on cluster managers — from submitting a job to running tasks on worker nodes.

Stories

• From spark-submit to Running Tasks: The Resource Negotiation Story — spark-submit launch chain, driver startup, executor acquisition, client vs cluster mode
• Spark on YARN: ApplicationMaster, Containers, and the Queue — YARN resource model, ApplicationMaster, container lifecycle, queue scheduling
• Spark on Kubernetes: Pods, Namespaces, and Ephemeral Executors — driver/executor pods, container images, ephemeral storage, RBAC

Related stories

• The Driver, the Executors, and How a Job Actually Runs — the roles of the driver and executors once they are running
• Elastic Executors: How Dynamic Allocation Grows and Shrinks the Cluster — how Spark requests and releases executors at runtime via the cluster manager
• SparkConf to Code: How Configuration Reaches the Component That Needs It — how configuration set at submission time propagates to executors
• The Event Log: A Complete Record of Everything That Happened in Your Job — how to observe completed jobs after the cluster tears them down
• The Journey of a Shuffle Record — the External Shuffle Service that makes dynamic allocation safe