Data Sources & I/O

Stories about reading and writing data — file formats, storage APIs, and the columnar data exchange layer.

Stories

Inside a Parquet File: Row Groups, Column Chunks, and Why Spark Loves It — row groups, column chunks, encoding, predicate/projection pushdown, page indexes, bloom filters, vectorized reader fallbacks
From DataFrame to Disk: How Spark Writes a Parquet File — output tasks and files, row groups at write time, compression, small-file problem, coalesce/repartition before save
Folders, Filters, and Skipping: How Layout Makes or Breaks Parquet Scans — Hive-style partitions, partition pruning vs row-group skipping, clustering, over-partitioning
Columns That Change: Schema, Nested Types, and Evolution in Parquet — footer schemas, nested types, additive evolution vs renames, mergeSchema cost
From Footer to Batches: How Spark Reads Parquet — planning, pruning, footers, vectorized batches and fallbacks, UI scan metrics

The Transaction Log: How Delta Lake Brings ACID to Object Storage — transaction log, protocol/features, snapshot isolation, OCC and isolation levels, time travel, checkpoints
Which Files Am I Reading? How Delta Resolves a Snapshot — version → checkpoint + JSON replay → active files, data skipping, hand-off to Parquet
Rewriting Reality: How UPDATE, DELETE, and MERGE Change a Delta Table — copy-on-write file replacement, write amplification, OCC conflicts, deletion-vector preview
Beyond Copy-on-Write: Deletion Vectors, Column Mapping, and Liquid Clustering — DVs, column IDs/renames, liquid clustering, protocol features
Housekeeping the Lake: OPTIMIZE, Z-Order, and VACUUM — compaction, clustering rewrites, data vs log retention
Delta in Motion: Streaming Reads and Exactly-Once Sinks — version-based source, CDF, per-batch commits, SetTransaction, small-file loop

The DataSource V2 API: How Spark Talks to Storage Systems — pluggable connector API, pushdown negotiation, transactional writes, streaming source support
The Columnar Fast Lane: How Apache Arrow Speeds Up PySpark — Arrow columnar format, zero-copy transfer, toPandas() and pandas UDF performance

What Is a Table to Spark? The Catalog, Metadata, and the Metastore — how catalog metadata points to the physical files these stories describe
Partitions: The Grain of Parallelism — coalesce/repartition and partition pruning as Spark-wide concepts
Bytes on the Wire: How Spark Serializes Data for Tasks and Shuffles — how data is serialized once it leaves the file reader
Cache Wisely: When Persisting Data Helps and When It Hurts — when to cache the results of expensive reads from these formats
Two Runtimes, One Job: How PySpark Bridges Python and the JVM — Arrow is the fast path for getting data from Spark into Python
Batch by Batch: Inside the Structured Streaming Micro-Batch Engine — DataSource V2’s MicroBatchStream is how streaming sources plug into Structured Streaming
Exactly Once, For Real: How Structured Streaming Guarantees No Duplicates — Delta as an exactly-once sink via transactional commits