Skip to content
Technology transformation expertise firm · ARDIANE Group Innovate, secure, succeed

Glossary

Data lake: definition, operation and limitations

Domain

Data

In brief

A data lake is a centralised storage space that keeps data of any kind (database extracts, files, logs, images) in its original format, without imposing a structure on it at entry. The structure is applied at read time, by the use that needs it.

Page contents

Definition of the data lake

A data lake is a centralised storage space that keeps data of any kind in its original format: database extracts, files, event logs, documents, images, sensor readings. Nothing is modelled at entry. The structure is applied later, at read time, by the use that needs it. This is the principle of schema on read.

The term is attributed to James Dixon, who used it in a blog post in October 2010. In it he compares the data mart to bottled water, cleaned and packaged, and the lake to a body of water in a more natural state, where anyone can come to examine, dive in or take samples.

What a data lake is for

A data warehouse keeps what was planned: structured data, modelled for questions known in advance. The data lake starts from the opposite assumption. It keeps raw data without waiting to know what it will be used for. Three uses follow from this.

  • Taking in what a warehouse is not designed to receive: texts, images, recordings, application logs.
  • Giving data science teams a detailed, non-aggregated history to explore the data and train models.
  • Serving as a landing zone from which the processes that feed warehouses and dashboards start.

Storage, zones and catalogue: the anatomy of a data lake

The foundation is high-capacity storage. The 2010 post reasoned in terms of Apache Hadoop. Object storage from a cloud provider or an in-house infrastructure can play the same role.

The data is organised in zones: a raw zone, a faithful copy of the sources, then one or more zones where it is cleaned and prepared. The name and number of these zones vary from one organisation to another.

Two elements decide what the lake becomes. The catalogue describes each dataset: origin, content, owner, load date. Access rights state who can read what. Without them, the lake slides towards the swamp described below.

Data lake, warehouse, lakehouse, data swamp: not to be confused

The data warehouse imposes its schema on write: data is checked and modelled before being loaded. The lake defers this work to read time.

The lakehouse takes the lake’s storage and adds a table management layer to it. Our entry on the lakehouse compares the three architectures.

The data swamp is not an architecture. It is the state of a lake where files accumulate with no description, no owner and no retention rule.

Finally, the data mesh distributes responsibilities for data. It says nothing about how to store it.

What a data lake does not guarantee

A lake stores files. It checks neither their content nor their structure at entry. A format change in a source application can therefore go unnoticed until the day a process fails. Two simultaneous writes to the same files do not benefit from the transactional guarantees of a database either.

Keeping data “just in case” also has a legal limit. For personal data, Article 5 of the GDPR, Regulation (EU) 2016/679, requires specified purposes and a retention period limited to what they require (paragraph 1, points b and e).

A concrete example: vehicle positions on a transport network

Take a public transport operator. Its vehicles transmit their position at short intervals, and its ticket validators record each passage. The data warehouse keeps only aggregates: ridership by line and by day, punctuality for the month.

The data lake keeps the positions and validations as they are transmitted. Later, a team wants to study how delays build up stop by stop, a question no one had asked when the warehouse was designed. It reads the raw positions and applies the structure it needs to them. The study is only possible if these files have been described in the catalogue, format changes included.

The choice between lake, warehouse and lakehouse, then building the whole, are covered in our data platform page.

FAQ

Frequently asked questions

Another question? Ask us

What is a data lake?

A data lake is a centralised storage space where an organisation keeps its data in its original format, whether structured or not: database extracts, files, logs, documents, images. No model is imposed at entry. The structure is defined at read time, by each use.

What is the difference between a data lake and a data warehouse?

The data warehouse receives structured data, conforming to a model defined before loading, and serves management reporting first. The data lake accepts any type of data in its raw state and defers structuring to read time, which suits exploration and data science. The two can coexist in the same platform, with the lake feeding the warehouse.

What does schema on read mean?

Schema on read consists of storing data as it arrives and only giving it a structure (columns, types, rules) when a process reads it. It is the opposite of the schema on write of the data warehouse, where the structure is checked before loading. With schema on read, loading becomes simpler, and control is deferred to each reader.

What is a data swamp?

A data swamp is a data lake that has become unusable: files have accumulated in it with no description, no owner and no retention rule. The data exists, but no one knows any longer what it contains or whether it is reliable. An up-to-date catalogue and owners appointed for each dataset prevent this drift.

A project to bring to completion?

A first conversation is enough to understand your challenge, identify the right people and tell you frankly what we can bring.