Skip to content
Technology transformation expertise firm · ARDIANE Group Innovate, secure, succeed

Glossary

Data observability: definition, observed signals and limitations

Domain

Data

In brief

Data observability is the continuous monitoring of datasets based on signals collected on each table: freshness, volume, schema, distribution of values and dependencies. A concept that tool vendors word in different ways, data observability is used to detect an anomaly that no written rule targeted, then to look for its origin.

Page contents

Definition of data observability

A job can finish without error and deliver a table that is empty, truncated or frozen at the day before yesterday. Data observability refers to the continuous monitoring of the datasets themselves, based on signals collected on each table: freshness, volume, schema, distribution of values, dependencies. It is used to detect an unforeseen anomaly, then to look for its origin. Tool vendors word it in different ways.

On the blog of the vendor Monte Carlo, Barr Moses writes that she coined the expression in 2019, by analogy with software observability.

Data observability, quality tests, job monitoring, application observability: not to be confused

A quality test compares a table with an expectation declared for it and returns a verdict. Data observability extends statistical detection to all the tables monitored, without requiring a written rule: each one is compared with its own history. It flags a deviation; qualifying it as an error is up to a person. Written rules and statistical detection are distinguished in our article on data quality testing methods.

Job monitoring looks at execution: did the job start, did it finish, how long did it take? The content delivered escapes it.

Application observability concerns the behaviour of software, based on traces, metrics and logs, signals described in our OpenTelemetry entry. Its object is a request that passes through services. That of data observability is a dataset that passes through jobs. What separates software observability from its monitoring is covered in our article observability vs monitoring.

The signals collected on the data

Monte Carlo presents five “pillars”: freshness, quality, volume, schema and lineage. The vendor Sifflet, for its part, lists five “technical pillars”, where distribution takes the place of quality. These vendor lists differ from one another. Here are the signals they cover.

  • Freshness: the age of the most recent data, relative to the expected update rhythm.
  • Volume: the number of rows received, compared with previous loads.
  • Schema: columns added, removed or changed in type.
  • Distribution: the rate of null values, the bounds and the distribution of a column’s values.
  • Lineage: the dependencies between tables, the subject of our data lineage entry. They are used to trace back to the origin of an anomaly and to list what it affects downstream.

The first four are collected without business knowledge, from the platform’s metadata and queries on the tables. A data contract can set their expected values; failing that, they are deduced from history. Collecting them is part of operating a data platform, whether or not it follows the modern data stack model.

A half-empty table, with no job error

Imagine a parcel carrier whose delivery events table is reloaded every night. One Monday, the job finishes on time, without error. Yet the table contains only half its usual rows: a depot has changed the format of its files, and the ingestion step silently discards them.

No written rule fails, since the rows present are compliant. The volume reading deviates from that of previous Mondays and triggers an alert. Lineage points downstream to the delays dashboard, whose distribution the team suspends, and upstream to the depot’s source.

Signals that do not tell the whole story

  • A wrong but plausible value moves no signal.
  • A threshold deduced from history also reacts to legitimate variations: public holiday, period close, sales promotion. Poorly tuned, it multiplies false alerts.
  • An alert observes. Without a designated recipient, it goes nowhere, and the cause continues to produce the defect. In the example, the discarded files still have to be reprocessed.

Qualifying and then correcting a flagged anomaly requires rules, thresholds and a handling process: this is the subject of our data quality page.

FAQ

Frequently asked questions

Another question? Ask us

What does data observability refer to?

Data observability is the continuous monitoring of a platform's datasets based on signals collected on each table: freshness, volume, schema, distribution of values and dependencies. Data observability is used to detect an anomaly that no one had anticipated, then to look for its origin. Tool vendors word it in different ways.

What is the difference between data observability and data quality tests?

A data quality test compares a table with an expectation declared for it and returns a verdict. Data observability applies statistical detection to all the tables monitored, without requiring a written rule, and flags a deviation, which a person then qualifies as an error or a legitimate variation. Data observability adds lineage to this, which is used to look for the origin of the deviation and what it affects downstream.

Is data observability a form of job monitoring?

No. Job monitoring checks that a job started, finished and how long it took. Data observability looks at the content delivered: a job that finished without error may have produced an incomplete table, which a signal collected on the data makes it possible to spot.

What are the five pillars of data observability?

The five pillars of data observability are a presentation by the vendor Monte Carlo, which lists on its blog freshness, quality, volume, schema and lineage. Another tool vendor lists five technical pillars, where distribution replaces quality: the list changes from one vendor to another.

Does data observability guarantee data quality?

Data observability detects deviations in freshness, volume, schema or distribution: it does not correct their cause and therefore does not guarantee data quality. A wrong but plausible value moves none of these signals and can go unnoticed. Correction falls to the anomaly handling process and to the owners appointed for each piece of data.

A project to bring to completion?

A first conversation is enough to understand your challenge, identify the right people and tell you frankly what we can bring.