Skip to content
Technology transformation expertise firm · ARDIANE Group Innovate, secure, succeed

Glossary

Modern data stack: definition, layers and criticisms

Domain

Data

In brief

The modern data stack is a family of analytical data architectures built around a cloud data warehouse: data is loaded raw, then transformed in place, by specialised tools assembled layer by layer. A term of usage, the modern data stack refers to a way of assembling, which each organisation puts together with its own tools.

Page contents

Definition of the modern data stack

The expression modern data stack refers to a family of analytical data architectures. Usage attributes three traits to it: a data warehouse hosted in the cloud, data loaded raw then transformed in that warehouse (ELT, for extract, load, transform), and specialised tools, one per function, which the organisation assembles itself. Its scope varies from one source to another.

A post signed by Tristan Handy, “The Modern Data Stack: Past, Present, and Future”, published on 1 December 2020 on the dbt Labs blog, places its starting point at the release of Amazon Redshift, which it dates to October 2012 (AWS announced the limited preview on 28 November 2012). The author presents this starting point as his opinion and writes as a player in this market. He describes the previous state: data was transformed before being loaded, because the warehouse was too slow and too constrained for this work.

The layers of a modern data stack

Layer Function Example tools
Loading Copy data from source applications into the warehouse, without modifying it Fivetran, Airbyte
Warehouse Store data and run computations in SQL Snowflake, BigQuery
Transformation Produce, in the warehouse, the tables intended for analysis, through versioned code dbt, SQLMesh
Reporting Present data to users: dashboards, analyses Looker, Metabase

Around this core, depending on needs, come an orchestrator, a data catalogue, a data observability tool or synchronisation from the warehouse to applications (reverse ETL).

Two properties follow from this split. Since raw data is retained, a faulty transformation can be corrected and then replayed. The layers communicate with the warehouse in SQL, so that one tool can in principle be replaced without reworking the others.

Criticisms levelled at this family of architectures

The 2020 post already noted several weaknesses: “immature” governance, entirely batch-based operation, and data that does not flow back to operational tools.

Other criticisms relate to assembly.

  • Each tool brings its own contract, permission model and metadata. Integration, overall security and diagnosing an incident that spans several tools fall to the organisation.
  • When computing is paid for on consumption, the bill can grow without any trade-off if spending is not linked to a use, which is the purpose of FinOps.
  • The warehouse is primarily designed for structured data loaded in batches. Documents, images or continuous streams may call for other components.

Modern data stack, lakehouse, data lake, ELT: not to be confused

The lakehouse describes a way of storing and managing tables. In a modern data stack, it can take the place of the warehouse.

The data lake retains files of any format, with no imposed structure. The modern data stack also retains raw data, but in the form of tables.

ELT is an order of operations. The modern data stack is the architecture built on that order.

Example: the occupancy rate of a hotel chain

Suppose a hotel chain wants to track its occupancy rate by property. A loading tool copies the booking system’s tables into a cloud warehouse every night. Versioned SQL models derive from them a “room nights by hotel and by day” table, which the reporting tool reads.

The day the rule for counting a cancelled room night changes, the team corrects the model and replays it on the raw data, with no new extraction. In return, it administers four tools, as many contracts, and computing spend to monitor.

Whether or not to adopt this assembly is an architecture decision, made on the basis of the uses to be served: our data platform page sets out the criteria.

FAQ

Frequently asked questions

Another question? Ask us

What does the expression modern data stack cover?

The modern data stack is a family of analytical data architectures organised around a data warehouse hosted in the cloud. Data from applications is loaded raw, then transformed in the warehouse, and each function (loading, transformation, reporting) is entrusted to a specialised tool. The scope of the modern data stack varies depending on the source describing it.

Why is the modern data stack associated with ELT?

ELT (extract, load, transform) loads raw data into the warehouse before transforming it, unlike ETL, which transforms it before loading. The modern data stack relies on this order, because the computing capacity of a cloud warehouse makes it possible to run transformations in place. Business rules are then written in SQL, in versioned code.

Can a lakehouse be part of a modern data stack?

Yes. The lakehouse describes a way of storing and managing tables: open table formats laid over files. The modern data stack describes how the functions of an analytical pipeline are split between specialised tools, around a central engine. That central engine can be a data warehouse or a lakehouse.

What criticisms are levelled at the modern data stack?

A post published on 1 December 2020 on the dbt Labs blog already criticised the modern data stack for immature governance, batch-based operation and the lack of data flowing back to operational applications. Added to these are criticisms linked to assembly: as many contracts, permission models and invoices as there are tools, and integration that remains the organisation's responsibility.

A project to bring to completion?

A first conversation is enough to understand your challenge, identify the right people and tell you frankly what we can bring.