sparklearning

Configuration

Stories about how Spark configuration is assembled, propagated, and applied.

SparkConf to Code: How Configuration Reaches the Component That Needs It — config sources and precedence, frozen vs dynamic properties, executor distribution, Environment tab

From spark-submit to Running Tasks: The Resource Negotiation Story — where command-line --conf flags fit in the submission flow
The Driver, the Executors, and How a Job Actually Runs — the components that consume configuration at runtime
Reading the Spark UI: What Every Tab Is Actually Telling You — the Environment tab shows the final merged configuration for a running job
AQE: How Spark Rewrites Plans After the Shuffle — AQE is controlled by config flags that can be set dynamically at runtime