Lakehouse Architecture

The lakehouse architecture combines the flexibility and low-cost storage of a data lake with the ACID transactions, schema enforcement, and performance of a data warehouse. It is built on open table formats like Delta Lake, Apache Iceberg, and Apache Hudi.

Architecture Overview

A lakehouse consists of three core layers:

1. Storage Layer - Object storage (S3, ADLS, GCS) holding data in open columnar formats (Parquet, ORC)
2. Table Format Layer - A metadata layer (Delta Lake, Iceberg, Hudi) that provides ACID transactions, time travel, and schema evolution on top of object storage
3. Compute Layer - Query engines (Spark, Trino, Databricks SQL, Athena) that read and write the table formats

Delta Lake Example

-- Create a Delta table CREATE TABLE events ( event_id    STRING, user_id     INT, event_type  STRING, event_time  TIMESTAMP, payload     MAP<STRING, STRING> ) USING DELTA PARTITIONED BY (event_date DATE) LOCATION 's3://data-lake/events/'; -- ACID transaction: merge (upsert) MERGE INTO events AS target USING updates AS source ON target.event_id = source.event_id WHEN MATCHED THEN UPDATE SET target.event_type = source.event_type, target.payload = source.payload WHEN NOT MATCHED THEN INSERT *; -- Time travel: query as of a specific version SELECT * FROM events VERSION AS OF 42; -- Time travel: query as of a specific timestamp SELECT * FROM events TIMESTAMP AS OF '2025-01-15 12:00:00'; -- Optimize: compact small files OPTIMIZE events; -- Vacuum: clean up old files (retention 7 days) VACUUM events RETAIN 168 HOURS;

Apache Iceberg Example

-- Create an Iceberg table CREATE TABLE orders ( order_id    BIGINT, customer_id INT, total       DECIMAL(12,2), order_ts    TIMESTAMP ) USING ICEBERG PARTITIONED BY (days(order_ts)) LOCATION 's3://data-lake/orders/'; -- Schema evolution: add a column ALTER TABLE orders ADD COLUMN status STRING; -- Incremental read: consume only new data SELECT * FROM orders WHERE order_ts > ( SELECT COALESCE(MAX(order_ts), '1970-01-01') FROM last_processed );

Key Benefits

• Single copy of data - No need to maintain separate warehouse and lake copies
• ACID transactions - Concurrent reads and writes with serializable isolation
• Schema evolution - Add, drop, rename, and reorder columns without rewriting data
• Time travel - Query historical versions for audit, rollback, or reprocessing
• Open formats - Avoid vendor lock-in; all major engines can read the data

Platform Implementations

When to Use a Lakehouse

• You need both BI/analytics and ML/data science on the same data
• Data volumes are large (100s of TB to PB) and warehouse costs are growing
• You want open formats to avoid vendor lock-in
• You need ACID guarantees on a data lake

Resources

• Delta Lake Documentation
• Apache Iceberg Documentation
• Apache Hudi Documentation