sparklearning

Catalog & Tables

Stories about how Spark tracks tables, databases, and metadata — from the session catalog to organization-wide governance.

What Is a Table to Spark? The Catalog, Metadata, and the Metastore — session catalog, Hive Metastore, managed vs external tables, pluggable catalogs
Beyond the Session Catalog: Unity Catalog and the Governed Lakehouse — three-level namespace, fine-grained access control, data lineage, external locations

Inside a Parquet File: Row Groups, Column Chunks, and Why Spark Loves It — the physical format of files that catalog tables point to
The Transaction Log: How Delta Lake Brings ACID to Object Storage — how Delta Lake tables maintain ACID guarantees on top of the catalog
The DataSource V2 API: How Spark Talks to Storage Systems — the connector API that pluggable catalogs and external tables use
From SQL to a Running Plan: The Catalyst Story — how the catalog is consulted during analysis to resolve table names and schemas