Chapter 2: The Lakehouse Paradigm

1. Warehouse vs Lake vs Lakehouse

OLTP vs OLAP: why warehouses exist

Operational systems power transactions: each touches a few rows, completes in milliseconds, and demands ACID so two customers don't book the same airline seat. Analytics is the inverse — scanning millions of rows across a few columns. Running that scan on the OLTP database would lock tables and degrade the customer experience, so organizations built the data warehouse: a separate, OLAP-optimized store with proprietary columnar formats and star/snowflake schemas tuned for BI.

Warehouses excel at structured, repeatable reporting, but pay three costs: dollars (proprietary storage is expensive), rigidity (schema-on-write — every column must be defined before loading), and narrowness (structured tabular data only — no images, JSON blobs, or sensor logs).

Data lake origin story

Around 2010, three forces collided: storage got dramatically cheaper (S3 commoditized durable bytes), new data sources exploded (mobile clickstreams, IoT, JSON events), and ML practitioners needed raw, unsampled data. The data lake was the answer — dump everything into S3, ADLS, or GCS, and worry about structure later. This is schema-on-read: define a schema when you query, not when you write.

The economics were transformative — pennies per gigabyte versus dollars in proprietary storage. But lakes lacked ACID guarantees: two pipelines writing the same Parquet directory could produce partial files, duplicates, or corrupt partitions. There was no atomic update, no easy GDPR delete. Lakes that accumulated undocumented, inconsistent data became known as data swamps.

Analogy: a warehouse is a bonded archive where every box is labeled before entry; a lake is a self-storage unit where you can throw anything in any order — and the unit next door might leak into yours overnight.

Why lakehouses emerged

By the late 2010s, organizations were running two parallel stacks: a lake for ML and exploration, and a warehouse for BI, with brittle ETL between them. This duplicated storage and let definitions drift ("revenue" meant different things in each system).

The lakehouse keeps the cheap object-storage substrate and open Parquet format, but adds a transaction layer that gives ACID semantics, schema enforcement, time travel, and fast row-level updates. Three open table formats now dominate this layer: Delta Lake (Databricks), Apache Iceberg (Netflix; used by Snowflake and AWS), and Apache Hudi (Uber). All three provide optimistic concurrency control, snapshot isolation, and engine interoperability.

Figure 2.1: Evolution from warehouse to lake to lakehouse

Comparison across architectures

2. Lakehouse Architecture and SageMaker Lakehouse

Object storage as the foundation

Every lakehouse rests on a cloud object store: Amazon S3, Azure Data Lake Storage Gen2, or Google Cloud Storage. These provide effectively infinite, eleven-nines-durable, pay-per-byte storage that decouples completely from compute. Files are typically Apache Parquet — a columnar format that compresses well, supports predicate pushdown, and is readable by virtually every analytic engine.

Analogy: object storage is like a city's water supply. Every restaurant draws from the same pipes; each cooks its own cuisine. The utility doesn't care what you cook — it guarantees clean water at scale and low cost.

Metastore and catalog layer

Raw Parquet files in S3 are just files. The metastore turns them into tables — things with names, schemas, partitions, statistics, and access control. When you commit a write, the metastore atomically swaps the snapshot pointer; that is what gives lakehouses ACID guarantees on top of immutable object-store files.

Production lakehouses use one of: Unity Catalog (Databricks), AWS Glue Data Catalog, Apache Nessie (Git-like branching), or the older Hive Metastore. Without a central catalog, every engine maintains its own picture of the data and they drift apart.

Query engines accessing shared data

The top layer is whatever compute you bring: Apache Spark for ETL/ML, Trino/Presto for interactive SQL, Amazon Athena for serverless ad-hoc, Amazon Redshift for BI, EMR/Glue for managed Spark/Flink, Databricks Photon, and DuckDB for local analytics. The architectural shift: storage is shared, compute is plural and disposable — no copies, no ETL between engines.

Figure 2.2: Three-layer lakehouse architecture

AWS SageMaker Lakehouse

Announced at AWS re:Invent 2024 and now generally available, Amazon SageMaker Lakehouse presents a single copy of data spanning S3 lakes, Redshift Managed Storage, and federated SaaS sources (Salesforce, SAP, ServiceNow, Zendesk, DynamoDB, Snowflake). A Spark job in EMR can read a Redshift table in place — without unloading it to S3 — and a Redshift query can join that table with an S3-resident Iceberg table in the same SQL statement. Zero-ETL integrations pull from SaaS sources directly into the lakehouse.

Built on Apache Iceberg as the open table format and AWS Glue Data Catalog as the unified metastore. Fine-grained, identity-based access (row/column rules) is enforced consistently across Athena, Redshift, and EMR through SageMaker Data and AI Governance on Amazon DataZone. SageMaker Unified Studio bundles EMR Studio, Glue Studio, the Redshift Query Editor, SageMaker Studio, and Bedrock tools into one workspace.

Figure 2.3: SageMaker Lakehouse unified access

SageMaker Lakehouse to generic-layer mapping

3. Medallion Architecture

Bronze: raw landing zone

Bronze is the single source of truth for ingested data. Tables are append-only, immutable, and as close to the source format as possible. Schema is applied loosely (schema-on-read) so ingestion stays robust against upstream changes. Bronze captures the full payload, an ingestion timestamp, a source-system identifier, and optional metadata such as file path or Kafka offset.

Bronze deliberately does not deduplicate, join, or drop columns. Because everything is append-only and timestamped, you can replay any downstream pipeline against any historical snapshot. Analogy: Bronze is a hospital's intake record — you write down everything the patient says, contradictions and all. Diagnosis happens later.

Silver: cleaned and conformed

Silver is where data becomes trustworthy. Transformations include: cleansing (repair malformed rows), deduplication (one record per business event), normalization (standard units, dates, country codes), enrichment via joins with reference data, and schema enforcement (every Silver table has a strict, documented schema; non-conforming writes fail).

Crucially, Silver stays non-aggregated: each row is a real-world event or entity (an order, a session, a sensor reading). This makes Silver the natural home for data scientists and analytics engineers.

Gold: business-ready aggregates

Gold materializes business meaning. Tables are aggregated, dimensional, or feature-engineered for direct consumption by BI dashboards, executive reports, and ML models. They are optimized for query performance — partitioned, clustered, and often pre-joined.

Typical Gold tables: daily/hourly aggregates ("revenue by region by day"), star-schema dimensional models, KPIs aligned to business definitions, and ML feature tables ready for training and inference.

Worked example: e-commerce orders

• Bronze (`orders_raw`): every API payload, including duplicates from client retries, malformed addresses, mixed-case currency codes ("USD", "Usd", "usd").
• Silver (`orders`): deduplicate on order ID + timestamp; reject rows with invalid postal codes (sent to a quarantine table); upper-case all currency codes; join the customer dimension to add region; enforce a strict schema with non-nullable order ID and amount.
• Gold (`daily_revenue_by_region`): aggregate the Silver `orders` table grouped by region and order date — powers the executive dashboard, refreshes nightly, fully rebuildable from Bronze if the definition changes.

Figure 2.4: Medallion architecture flow

Layer comparison

A common deployment shape is a hub-and-spoke arrangement: a central Data Hub owns canonical Bronze/Silver/Gold tables for cross-organization concepts (customer, employee, product), and each domain (Sales, Marketing, Finance) maintains its own Bronze/Silver/Gold extensions that join to the hub. This avoids both central bottlenecks and uncoordinated domain duplication.