Lakehouse

A lakehouse adds a table layer on top of data-lake storage, giving files in object storage the guarantees of a database: transactions, schemas, updates and deletes, and fast SQL queries. The same copy of the data serves BI dashboards, data engineering and machine learning.

The goal is to stop maintaining two separate systems, a lake for data science and a warehouse for reporting, with pipelines copying data between them.

Diagram

flowchart LR
    src[Sources] --> bronze[(Bronze: raw)]
    bronze --> silver[(Silver: cleaned and joined)]
    silver --> gold[(Gold: business-level tables)]
    subgraph lh [Open table format on object storage]
        bronze
        silver
        gold
    end
    gold --> bi[BI and SQL]
    silver --> ml[Data science and ML]
    gold --> ml

How it works

Open table formats. Formats such as Delta Lake, Apache Iceberg and Apache Hudi add a transaction log and metadata on top of Parquet files.
ACID transactions. Concurrent reads and writes see consistent data; failed jobs don’t leave half-written tables.
Schema enforcement and evolution. Tables have schemas that are checked on write and can change in controlled ways.
Medallion layers. Data commonly flows from bronze (raw) to silver (cleaned) to gold (business-ready).
Multiple engines, one copy. SQL engines, Spark and ML frameworks read the same tables.

Pros

One platform for BI and ML: fewer copies and pipelines.
Low-cost storage: data stays in object storage.
Open formats: less lock-in than proprietary warehouse storage.
Reliability of a database: transactions, updates, deletes and time travel.

Cons

Still maturing: tooling, performance tuning and best practices are evolving.
Operational work: compaction, file sizing and table maintenance need attention.
BI performance can trail a dedicated warehouse for highly concurrent dashboard workloads.
Format choice matters: engines differ in which table formats they support well.

When to use it

Organizations running both BI and data science on large datasets.
Teams replacing a lake-plus-warehouse setup that copies data between the two.
When open formats and avoiding lock-in are priorities.

When to avoid it

Small or simple reporting needs, where a managed cloud warehouse is less work.
Teams without data engineering capacity to run the platform.

Common pitfalls

Skipping table maintenance. Without compaction, small files pile up and queries slow down.
Treating bronze as a dumping ground with no retention, ownership or quality checks.
Mixing table formats without a plan, leaving engines unable to read each other’s tables.

Glossary: lakehouse