Fault Tolerance

Stories about how Spark recovers from failures — through lineage, recomputation, checkpointing, and speculation.

Stories

How Spark Survives Failure — lineage graph, narrow vs wide dependencies, task/stage retries, checkpointing, speculative execution

The Driver, the Executors, and How a Job Actually Runs — the execution model that fault tolerance is built on top of
From One Action to Many Tasks — how the DAG Scheduler detects failures and retries stages
The Journey of a Shuffle Record — fetch failures during shuffle are a common trigger for stage retries
Batch by Batch: Inside the Structured Streaming Micro-Batch Engine — how Structured Streaming extends fault tolerance to continuous queries via checkpointing
Keeping Score: How Spark Maintains State Across Micro-Batches — how stateful streaming operations survive failures through the state store