Definition of the lakehouse
The lakehouse is a data platform architecture that combines, on a single foundation, the properties of the data warehouse and those of the data lake. The word is a contraction of the two terms. It spread from 2020 onwards.
The definition of the lakehouse comes down to two layers. Data is stored in open-format files, on object storage, as in a lake. A table management layer then brings what characterises a warehouse: reliable transactions, controlled schemas and high-performance queries.
What a lakehouse is for
Many organisations run two systems in parallel. A lake receives raw data and serves data science. A warehouse receives part of this data, reprocessed, and serves reporting. This two-tier architecture comes at a price: copies to synchronise, discrepancies between the figures of the two worlds, delays and a double bill for storage and compute.
The lakehouse aims to serve all uses from a single copy of the data: dashboards, exploratory analysis, model training, streaming processing. The expected benefit is consistency more than technical novelty.
How a lakehouse works
Four components are stacked.
- Object storage, which keeps all files at low cost and with no practical volume limit.
- An open columnar file format, such as Apache Parquet, suited to analytical queries.
- An open table format, such as Delta Lake, Apache Iceberg or Apache Hudi. It keeps a transaction log on top of the files. It is this format that brings ACID transactions, schema evolution and the ability to read a table in an earlier state.
- A catalogue, which lists the tables, holds access rights and, depending on the tools, data lineage.
Compute is separated from storage. Several engines (SQL, distributed processing, machine learning) read and write the same tables, and each is sized independently. Data is often organised in successive layers, from raw to ready to use, following a so-called medallion pattern.
Lakehouse, data warehouse, data lake: not to be confused
| Data warehouse | Data lake | Lakehouse | |
|---|---|---|---|
| Data accepted | Structured | All types, in raw form | All types |
| Schema | Enforced on write | Applied on read | Controlled by the table format |
| Reliable transactions | Yes | No | Yes |
| Storage format | Often vendor-specific | Open | Open |
| Main uses | Reporting, BI | Data science, archiving | BI, data science, machine learning |
The boundary is blurring. Data warehouses can now read and write open table formats, and lakehouse platforms have strengthened their SQL engines. The useful question is no longer the product label. It concerns three points: the format in which your data is stored, the engines able to read it, and the layer that carries governance.
The lakehouse is, moreover, a technical architecture. It says nothing about the allocation of responsibilities for data, which is the subject of the data mesh.
The limitations of the lakehouse
- It does not solve governance. Without an up-to-date catalogue or quality control, a lakehouse becomes a disorderly data lake with an SQL engine.
- It requires maintenance. Compacting small files, purging old versions, physical organisation of tables: these operations determine performance.
- It assumes engineering skills broader than those needed for a fully managed warehouse.
- The choice of table format and catalogue is a commitment. The promised openness depends on the actual compatibility between the engines you use.
- It is oversized for a simple need. A moderate volume of structured data, intended for reporting alone, can make do with a warehouse.
A concrete example: a single source for BI and models
Take an industrial group whose financial control team works on a data warehouse, while the data science team trains its maintenance models on sensor readings stored in a lake. The two teams each calculate a machine availability rate, and the figures diverge, because they do not start from the same data and the same rules.
With a lakehouse, raw readings, cleaned data and reference tables reside in the same storage, under the same catalogue. The dashboard and the model read the same availability table. When a calculation rule changes, it changes for everyone. The version history also makes it possible to reconstruct the state of the data used to train a model, a point useful to MLOps practices.
For the choice of architecture and its implementation, see our page dedicated to the data platform.