Definition of data lineage
Data lineage refers to the description of the path a piece of data takes through the information system: where it is read, which processes it passes through, what they do with it and where it is written, from the system that produces it to the uses that consume it. The term is one of common usage, and its wording varies from one source to another.
Lineage is represented by a directed graph. Its nodes are datasets (tables, files, streams) and processes. Its edges indicate that a process reads or writes a dataset.
Two directions of reading: upstream and downstream
Upstream, reading starts from a figure and traces back its inputs: tables, calculation rules, sources. It is used to justify a metric or to look for the origin of an anomaly. Downstream, it starts from a source or a column and lists what depends on it: intermediate tables, reports, extracts, models. This impact analysis precedes a schema change, the replacement of an application or the retirement of a table.
Lineage describes a path. It says neither whether the source is reliable nor whether the value is correct.
Granularity, level of description and ways of building it
Granularity is the grain of the nodes. At system level, the graph links applications. At table level, it links datasets. At column level, it indicates which input columns are used to produce each output column. A fine granularity gives a larger graph, which depends more on tooling.
The level of description separates, by convention, technical lineage, made of tables and processes, from so-called business lineage, which links glossary concepts and metrics.
Three ways of building it are distinguished here:
- analysing the code of the transformations (SQL queries, scripts), which deduces the dependencies without running the processes;
- the processes themselves emitting events, at each run;
- manual declaration, for steps that escape tooling: re-keying, spreadsheets, file exchanges.
For the second way, the OpenLineage project publishes an open standard. According to its website, the components of a processing pipeline (schedulers, warehouses, SQL engines) can use it to send a compatible tool information about their runs, their jobs and their datasets. Its documentation describes a column-level lineage facet, which qualifies the relationship: direct when the output value is derived from the input column, indirect when the input column influences the result without the value being derived from it, as in a filter or a join.
Data lineage, provenance, catalogue, access log: not to be confused
Data provenance refers, by convention, to the origin and history of a piece of data rather than its path; usage does not always separate the two words. Our article data provenance vs data lineage compares the two concepts and places the W3C PROV model.
The data catalogue describes each dataset taken in isolation: content, owner, classification. Lineage describes the links between them.
The access log records who read or changed a piece of data, not the processes it passes through.
Example: a column to be split in the invoicing application
Take a company whose invoicing application must split the “amount” column into an amount excluding tax and a tax amount. Before go-live, the data team reads the lineage downstream. The graph, maintained at column level, points to three warehouse tables that take it over, a revenue metric calculated from one of them and a monthly extract intended for accounting.
At table level, the same reading would have pointed to all the reports fed by the invoices table, including those that do not use the amount. One step remains invisible: a spreadsheet that the financial control team fills by copying. It only appears in the graph if someone has declared it.
Deciding which data must have its lineage maintained, at what granularity and under whose responsibility falls under data governance. Producing it automatically is the job of the data platform.