Chapter 3: Storage Foundations and Open Table Formats

1. File Formats and Columnar Storage

Storage decisions cascade through every layer of a data platform. The bytes you commit to disk determine how much money you spend on storage, how much money you spend on scans, and how fast queries return. This partition focuses on file formats — the lowest layer of the storage stack — with special attention to why columnar Parquet has become the default for analytics.

1.1 Row-Oriented vs Columnar: The Core Trade-off

Row-oriented formats — CSV, JSON, Avro — store records the way you'd write them on a notepad: all the fields for one record together, then all the fields for the next. CSV is the lowest common denominator (every spreadsheet and scripting language reads it), JSON layers in nested types (great for API payloads), and Avro is a compact binary row format with an embedded schema designed for high-throughput streaming on Apache Kafka.

Row formats are great when you need to read or write whole records: ingesting webhook events, exporting a customer profile, replaying a Kafka topic. They are terrible for analytics because answering “average order value last quarter?” forces the engine to read every byte of every order — including customer addresses, SKUs, and shipping notes you don't care about — just to grab one column.

Parquet and ORC flip the layout. Instead of grouping rows together, they group values from the same column together. A query that touches three of fifty columns reads roughly 6% of the file rather than 100%. Apache Parquet has become the de facto standard for analytical lakes; ORC is its close cousin, more common in legacy Hive deployments.

1.2 Parquet's Hierarchical Layout

Parquet organizes a file hierarchically: the file is split into row groups of around 128 MB, each row group is sliced into per-column column chunks, and each chunk is broken into pages that hold the actual encoded values. At every level, Parquet stores rich metadata: min/max values, null counts, distinct counts, and optional bloom filters.

That metadata enables predicate pushdown — the engine consults statistics and skips data that cannot match the filter without reading the values. If a query asks WHERE shipdate ≤ '1996-09-02' and a row group's max shipdate is '1996-08-15', the engine reads it. If the row group's min shipdate is '1997-01-01', the engine skips the entire row group. On TPC-H SF20 this kind of pruning skips roughly 30% of data on selective filters.

1.3 Encodings That Compound the Win

Parquet stacks several encodings on top of compression:

The numbers are striking. On a 194 GB Allstate dataset, Parquet compressed the data to 4.7 GB — ~97% reduction — and read 3.5x less data on full scans. On TPC-H SF20, Parquet was 5x smaller than CSV (3.2 GB vs 16 GB) and 7-10x faster on joins (2 s versus 20 s). On AWS Athena, the same query scanned 117 KB of Parquet versus 48 MB of CSV, which directly shows up on the bill.

1.4 Choosing a Format by Access Pattern

The choice is rarely “Parquet always wins.” It depends on access pattern.

A common pattern is to land raw data as JSON or Avro for fidelity, then immediately convert to Parquet for analytics. The raw layer is the audit trail; the Parquet layer is what dashboards actually query.

2. Open Table Formats: Iceberg, Delta, Hudi

A Parquet file is just a Parquet file. It has no notion of “the current state of the orders table.” If two writers append concurrently, or one writer fails halfway through, you can end up with partial data, duplicates, or queries that see inconsistent snapshots. Open table formats are thin metadata layers on top of Parquet/ORC that provide ACID transactions, schema evolution, and time travel on commodity object storage.

2.1 Apache Iceberg: A Tree of Snapshots

Iceberg organizes a table as a tree of metadata files. At the root sits a metadata.json file pointing at the current snapshot. Each snapshot lists manifest files, and each manifest lists the data files (Parquet or ORC) that belong to the table at that moment. Writers append new data files and produce a new snapshot atomically; readers see whatever snapshot was current when their query started. The result is serializable isolation via optimistic concurrency control — two writers can prepare commits in parallel; the second one detects a conflict and retries.

Because every snapshot is preserved (until expired), Iceberg supports time travel:

-- Query Iceberg as of a specific time
SELECT * FROM iceberg_table
FOR SYSTEM_TIME AS OF '2026-05-07 14:00:00';

-- Or by snapshot ID
SELECT * FROM iceberg_table
FOR SYSTEM_VERSION AS OF 4538291;

Iceberg's biggest superpower is schema evolution backed by stable column IDs. Each column has an immutable numeric ID, so you can rename, reorder, drop, or add columns without rewriting any data files. Partition evolution works the same way: change the partition spec on a go-forward basis — old partitions and new partitions coexist seamlessly.

2.2 Delta Lake: A Transaction Log

Delta Lake takes a different route. Instead of a tree of manifests, it maintains a _delta_log directory containing an ordered series of JSON commit files, each describing one transaction (add file X, remove file Y, update schema, etc.). To compute the current table state, the reader replays the log from the last checkpoint forward — much like a database write-ahead log applied to object storage.

-- Delta time travel by version
SELECT * FROM delta_table VERSION AS OF 5;

-- By timestamp
SELECT * FROM delta_table TIMESTAMP AS OF '2026-05-01';

Delta's default version retention is 30 days. Schema evolution is more limited than Iceberg's: you can add columns at the end and merge schemas during writes, but native rename/drop is constrained. Where Delta shines is the Databricks ecosystem: zero-copy table cloning, deep MERGE optimizations, and tight integration with MLflow and Unity Catalog.

2.3 Apache Hudi: Streaming-First Upserts

Hudi was built at Uber to solve a problem the others didn't initially tackle: efficient row-level upserts and deletes on a lake. Hudi maintains a timeline of instants and offers two table types:

Hudi also brings record-level indexing (Bloom, Hash File, HBase-backed), so updating one row by primary key doesn't require scanning the whole partition. This makes Hudi the format of choice for CDC pipelines from transactional sources. Schema evolution is more limited than Iceberg's, and ecosystem support outside Spark and Flink is thinner.

2.4 Side by Side

An analogy: Iceberg is like Git for your lake — every commit is a snapshot you can travel back to. Delta is like a single transaction log a la PostgreSQL's WAL, replayed to derive state. Hudi is like a journaling filesystem optimized for many small updates with periodic compaction.

3. S3 Object Storage and Partitioning

This partition covers the bottom and the connective layer: where the bytes physically live (S3 storage classes, S3 Tables, lifecycle policies) and how they are arranged for fast pruning (Hive partitioning, Iceberg hidden partitioning, bucketing, clustering, Z-ordering).

3.1 S3 Storage Classes

Amazon S3 is the de facto storage backbone for AWS-based lakes. What many teams miss is that S3 is not one storage tier — it's a family of classes with very different cost and access profiles. Choosing the right class per object can cut the storage bill 50-95%.

3.2 Amazon S3 Tables: Managed Iceberg

Running Iceberg yourself is workable but operationally heavy: schedule compaction (small file problem), expire old snapshots, clean up unreferenced files, tune metadata layout. Most teams underinvest, and table performance silently degrades.

Amazon S3 Tables, announced at AWS re:Invent 2024, is a fully managed Iceberg service built on a new bucket type called table buckets. Tables are first-class AWS resources with their own ARNs, IAM policies, and dedicated endpoints.

1. Performance — up to 3x query throughput and 10x TPS versus self-managed Iceberg.
2. Automatic maintenance — background compaction, snapshot expiration, unreferenced file removal.
3. Native integrations — auto-registered in AWS Glue Data Catalog; queryable from Athena, EMR, Spark, Redshift, Kinesis Data Firehose, and QuickSight.

3.3 Lifecycle Policies and Intelligent Tiering

For data not in S3 Tables, control cost through lifecycle policies — declarative rules that transition objects between storage classes by age. A common lake lifecycle:

# Conceptual S3 lifecycle policy
- Day 0-30:     S3 Standard (hot ETL output)
- Day 30-90:    S3 Standard-IA (occasional ad-hoc queries)
- Day 90-365:   S3 Glacier Instant Retrieval (audit access)
- Day 365+:     S3 Glacier Deep Archive (compliance only)
- Day 2555:     Delete (7 years)

3.4 Hive-Style Partitioning

The classical scheme, inherited from Apache Hive, encodes partition values directly into the directory path:

s3://lake/orders/
  region=us/year=2026/month=05/day=07/part-0001.parquet
  region=us/year=2026/month=05/day=07/part-0002.parquet
  region=eu/year=2026/month=05/day=07/part-0001.parquet

A query with WHERE region='us' AND year=2026 AND month=5 only lists those directories. The query engine never sees the EU files. Downsides: users must include partition columns explicitly or pruning silently fails; partition values are tied to physical layout, so changing the scheme requires rewriting all historical data; too-fine partitioning produces millions of tiny files.

3.5 Hidden Partitioning in Iceberg

Iceberg solves the awkward parts with hidden partitioning. The partition spec lives in metadata, not the directory layout, and Iceberg automatically derives partition values from source columns using transforms:

-- Define partitioning by day, derived from a timestamp column
CREATE TABLE orders (
  order_id BIGINT,
  customer_id BIGINT,
  order_ts TIMESTAMP,
  amount DECIMAL(12,2)
) PARTITIONED BY (days(order_ts));

-- Users write natural SQL, Iceberg prunes automatically
SELECT SUM(amount)
FROM orders
WHERE order_ts BETWEEN '2026-05-01' AND '2026-05-07';

Available transforms include years, months, days, hours, bucket(N, col), and truncate(N, col); the partition spec can evolve over time without rewriting old data.

3.6 Bucketing, Clustering, and Z-Ordering

Partitioning works best for low-cardinality columns. For high-cardinality columns like customer_id or order_id, partitioning is impossible — you'd get millions of tiny partitions. The answer is bucketing, clustering, and Z-ordering.

A worked example: a 5 TB clickstream table queried mostly by event_date, occasionally by user_id, sometimes by country:

1. Partition by days(event_ts) — coarse pruning on the dominant filter.
2. Z-order within each partition by (user_id, country) — second-dimension pruning.
3. Target file size 128-512 MB to keep file count manageable.

Queries filtering on date alone hit the partition prune. Queries also filtering on user_id additionally skip most files within the day. The metadata stays small because there are at most a few thousand date partitions, not millions of user partitions.