Skip to content
Technology transformation expertise firm · ARDIANE Group Innovate, secure, succeed

Glossary

Data lineage: definition, granularity and how it is built

Domain

Data

In brief

Data lineage is the description of the path a piece of data takes through the information system: the datasets a process reads, the transformations it applies and those it writes, from the system that produces the data to the uses that consume it. A term of common usage, data lineage is represented by a dependency graph, read upstream or downstream.

Page contents

Definition of data lineage

Data lineage refers to the description of the path a piece of data takes through the information system: where it is read, which processes it passes through, what they do with it and where it is written, from the system that produces it to the uses that consume it. The term is one of common usage, and its wording varies from one source to another.

Lineage is represented by a directed graph. Its nodes are datasets (tables, files, streams) and processes. Its edges indicate that a process reads or writes a dataset.

Two directions of reading: upstream and downstream

Upstream, reading starts from a figure and traces back its inputs: tables, calculation rules, sources. It is used to justify a metric or to look for the origin of an anomaly. Downstream, it starts from a source or a column and lists what depends on it: intermediate tables, reports, extracts, models. This impact analysis precedes a schema change, the replacement of an application or the retirement of a table.

Lineage describes a path. It says neither whether the source is reliable nor whether the value is correct.

Granularity, level of description and ways of building it

Granularity is the grain of the nodes. At system level, the graph links applications. At table level, it links datasets. At column level, it indicates which input columns are used to produce each output column. A fine granularity gives a larger graph, which depends more on tooling.

The level of description separates, by convention, technical lineage, made of tables and processes, from so-called business lineage, which links glossary concepts and metrics.

Three ways of building it are distinguished here:

  • analysing the code of the transformations (SQL queries, scripts), which deduces the dependencies without running the processes;
  • the processes themselves emitting events, at each run;
  • manual declaration, for steps that escape tooling: re-keying, spreadsheets, file exchanges.

For the second way, the OpenLineage project publishes an open standard. According to its website, the components of a processing pipeline (schedulers, warehouses, SQL engines) can use it to send a compatible tool information about their runs, their jobs and their datasets. Its documentation describes a column-level lineage facet, which qualifies the relationship: direct when the output value is derived from the input column, indirect when the input column influences the result without the value being derived from it, as in a filter or a join.

Data lineage, provenance, catalogue, access log: not to be confused

Data provenance refers, by convention, to the origin and history of a piece of data rather than its path; usage does not always separate the two words. Our article data provenance vs data lineage compares the two concepts and places the W3C PROV model.

The data catalogue describes each dataset taken in isolation: content, owner, classification. Lineage describes the links between them.

The access log records who read or changed a piece of data, not the processes it passes through.

Example: a column to be split in the invoicing application

Take a company whose invoicing application must split the “amount” column into an amount excluding tax and a tax amount. Before go-live, the data team reads the lineage downstream. The graph, maintained at column level, points to three warehouse tables that take it over, a revenue metric calculated from one of them and a monthly extract intended for accounting.

At table level, the same reading would have pointed to all the reports fed by the invoices table, including those that do not use the amount. One step remains invisible: a spreadsheet that the financial control team fills by copying. It only appears in the graph if someone has declared it.

Deciding which data must have its lineage maintained, at what granularity and under whose responsibility falls under data governance. Producing it automatically is the job of the data platform.

FAQ

Frequently asked questions

Another question? Ask us

What is data lineage?

Data lineage describes the path a piece of data takes through the information system: the datasets read by each process, the transformations applied and the datasets written. It takes the form of a dependency graph that runs from the system that produces the data to the uses that consume it. The term is one of common usage and its wording varies from one source to another.

What is the difference between table-level and column-level lineage?

Table-level lineage links entire datasets: a given table is produced from certain others. Column-level lineage indicates, for each column produced, the input columns used to calculate it. The second answers more precise questions, such as the effect of changing a single field, but it produces a larger graph and depends more on tooling.

How is data lineage built?

Data lineage can be built in three ways, which can be combined. Analysing the code of the transformations deduces the dependencies without running the processes. Processes can also emit lineage events themselves at each run, for example following the open standard OpenLineage. Steps that escape tooling, such as a copy into a spreadsheet, are declared by hand.

Does data lineage tell you about the quality of a piece of data?

No. Data lineage describes a path: it shows which processes a piece of data has passed through, without saying whether the source is reliable or whether the value is correct. It does, however, help in looking for the origin of an anomaly, by following the graph upstream from the wrong figure.

A project to bring to completion?

A first conversation is enough to understand your challenge, identify the right people and tell you frankly what we can bring.