Memory & Storage

Stories about how Spark manages memory — from the unified memory model to OOM diagnosis and caching strategy.

Stories

• The Two Lives of Spark’s Memory — unified memory manager, execution vs storage memory, BlockManager, LRU eviction
• Out of Memory: A Field Guide to Spark OOM Errors — heap vs off-heap OOMs, driver vs executor, common causes and fixes
• Cache Wisely: When Persisting Data Helps and When It Hurts — when to cache, storage levels, cache invalidation, unpersist discipline

Related stories

• Tungsten: How Spark Stopped Trusting the JVM — off-heap Tungsten memory sits outside the unified memory pool
• Taming the Shuffle: Partition Count, Spill, and the Right Shuffle for Your Job — shuffle spill is directly caused by insufficient execution memory
• Partitions: The Grain of Parallelism — partition count determines how much data each task holds in memory at once
• Garbage Collection in Spark: Why the JVM Pauses and How to Make It Stop — GC pressure is closely related to how memory is used and how data is cached
• RocksDB in Structured Streaming — RocksDB state store moves streaming state off the JVM heap to reduce GC pressure