Definition of the data product
A data product is a set of data that a team makes available to other teams as it would a product: with identified users, documentation, quality commitments and an owner. The term is one of common usage. In a broader sense, it also refers to any deliverable based on data, such as a dashboard or a predictive model. This entry deals with the first sense.
Zhamak Dehghani formulated it in two articles published on martinfowler.com. “How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh”, dated 20 May 2019, asks domain teams to apply product thinking to the datasets they provide and to consider their consumers as customers. “Data Mesh Principles and Logical Architecture”, dated 3 December 2020, makes the data product the architectural unit of the data mesh. Our data mesh entry presents the overall model.
The responsibility that the data product shifts
A table made available without an owner leaves each consumer to find it, guess its meaning and discover its changes after the fact. The 2020 article names this cost: the difficulty of discovering quality data, understanding it, trusting it and using it.
Treating data as a product places responsibility for quality as close as possible to the source. The team that produces the data is accountable for what it publishes to those who use it.
The qualities and components of a data product
The 2019 article lists six qualities.
- Discoverable: the product is listed in a data catalogue, with its owner, its origin and its lineage.
- Addressable: it has a unique address, following a common convention.
- Trustworthy: its owner announces service level objectives on the fidelity of the data.
- Self-describing: its semantics and syntax are documented.
- Interoperable: it follows standards common to all domains.
- Secure: access control applies to each product.
The 2020 article describes three components: the code (processing, access interfaces, policy enforcement), the data with its metadata, and the infrastructure that runs and stores them. The data product owner, a role described in the 2019 article, decides on the product roadmap and is accountable for its lifecycle.
Data product, dataset, data contract, data mesh: not to be confused
| Concept | What it refers to | What separates it from the data product |
|---|---|---|
| Dataset | A table, a file or a stream | The product adds an owner, documentation and commitments to it |
| Data contract | By convention, the agreement that describes what a producer provides: schema, freshness, rules for change | The contract records the product’s commitments, of which it is one part |
| Data mesh | An organisational model for analytical data | The data product is its unit; the expression is also used outside this model |
| Data catalogue | The inventory of datasets and their metadata | The catalogue references the products and makes it possible to find them |
Example: the “validations by line” product of a transport network
Take a passenger transport company whose ticketing team publishes a data product “validations by line and by day”. Its catalogue entry gives an address, the definition of each field, availability every morning and the name of its owner. A published check measures the share of validations attached to a line.
When the team changes the coding of ticket types, it publishes a new version of the product and notifies its known consumers, service planning and finance. The same table, dropped into a shared space with no owner or commitment, would remain a dataset.
The platform on which a data product is built, published and monitored is the subject of our data platform page. Appointing owners and setting the rules common to all products falls under data governance.