Definition of the modern data stack
The expression modern data stack refers to a family of analytical data architectures. Usage attributes three traits to it: a data warehouse hosted in the cloud, data loaded raw then transformed in that warehouse (ELT, for extract, load, transform), and specialised tools, one per function, which the organisation assembles itself. Its scope varies from one source to another.
A post signed by Tristan Handy, “The Modern Data Stack: Past, Present, and Future”, published on 1 December 2020 on the dbt Labs blog, places its starting point at the release of Amazon Redshift, which it dates to October 2012 (AWS announced the limited preview on 28 November 2012). The author presents this starting point as his opinion and writes as a player in this market. He describes the previous state: data was transformed before being loaded, because the warehouse was too slow and too constrained for this work.
The layers of a modern data stack
| Layer | Function | Example tools |
|---|---|---|
| Loading | Copy data from source applications into the warehouse, without modifying it | Fivetran, Airbyte |
| Warehouse | Store data and run computations in SQL | Snowflake, BigQuery |
| Transformation | Produce, in the warehouse, the tables intended for analysis, through versioned code | dbt, SQLMesh |
| Reporting | Present data to users: dashboards, analyses | Looker, Metabase |
Around this core, depending on needs, come an orchestrator, a data catalogue, a data observability tool or synchronisation from the warehouse to applications (reverse ETL).
Two properties follow from this split. Since raw data is retained, a faulty transformation can be corrected and then replayed. The layers communicate with the warehouse in SQL, so that one tool can in principle be replaced without reworking the others.
Criticisms levelled at this family of architectures
The 2020 post already noted several weaknesses: “immature” governance, entirely batch-based operation, and data that does not flow back to operational tools.
Other criticisms relate to assembly.
- Each tool brings its own contract, permission model and metadata. Integration, overall security and diagnosing an incident that spans several tools fall to the organisation.
- When computing is paid for on consumption, the bill can grow without any trade-off if spending is not linked to a use, which is the purpose of FinOps.
- The warehouse is primarily designed for structured data loaded in batches. Documents, images or continuous streams may call for other components.
Modern data stack, lakehouse, data lake, ELT: not to be confused
The lakehouse describes a way of storing and managing tables. In a modern data stack, it can take the place of the warehouse.
The data lake retains files of any format, with no imposed structure. The modern data stack also retains raw data, but in the form of tables.
ELT is an order of operations. The modern data stack is the architecture built on that order.
Example: the occupancy rate of a hotel chain
Suppose a hotel chain wants to track its occupancy rate by property. A loading tool copies the booking system’s tables into a cloud warehouse every night. Versioned SQL models derive from them a “room nights by hotel and by day” table, which the reporting tool reads.
The day the rule for counting a cancelled room night changes, the team corrects the model and replays it on the raw data, with no new extraction. In return, it administers four tools, as many contracts, and computing spend to monitor.
Whether or not to adopt this assembly is an architecture decision, made on the basis of the uses to be served: our data platform page sets out the criteria.