Definition of data observability
A job can finish without error and deliver a table that is empty, truncated or frozen at the day before yesterday. Data observability refers to the continuous monitoring of the datasets themselves, based on signals collected on each table: freshness, volume, schema, distribution of values, dependencies. It is used to detect an unforeseen anomaly, then to look for its origin. Tool vendors word it in different ways.
On the blog of the vendor Monte Carlo, Barr Moses writes that she coined the expression in 2019, by analogy with software observability.
Data observability, quality tests, job monitoring, application observability: not to be confused
A quality test compares a table with an expectation declared for it and returns a verdict. Data observability extends statistical detection to all the tables monitored, without requiring a written rule: each one is compared with its own history. It flags a deviation; qualifying it as an error is up to a person. Written rules and statistical detection are distinguished in our article on data quality testing methods.
Job monitoring looks at execution: did the job start, did it finish, how long did it take? The content delivered escapes it.
Application observability concerns the behaviour of software, based on traces, metrics and logs, signals described in our OpenTelemetry entry. Its object is a request that passes through services. That of data observability is a dataset that passes through jobs. What separates software observability from its monitoring is covered in our article observability vs monitoring.
The signals collected on the data
Monte Carlo presents five “pillars”: freshness, quality, volume, schema and lineage. The vendor Sifflet, for its part, lists five “technical pillars”, where distribution takes the place of quality. These vendor lists differ from one another. Here are the signals they cover.
- Freshness: the age of the most recent data, relative to the expected update rhythm.
- Volume: the number of rows received, compared with previous loads.
- Schema: columns added, removed or changed in type.
- Distribution: the rate of null values, the bounds and the distribution of a column’s values.
- Lineage: the dependencies between tables, the subject of our data lineage entry. They are used to trace back to the origin of an anomaly and to list what it affects downstream.
The first four are collected without business knowledge, from the platform’s metadata and queries on the tables. A data contract can set their expected values; failing that, they are deduced from history. Collecting them is part of operating a data platform, whether or not it follows the modern data stack model.
A half-empty table, with no job error
Imagine a parcel carrier whose delivery events table is reloaded every night. One Monday, the job finishes on time, without error. Yet the table contains only half its usual rows: a depot has changed the format of its files, and the ingestion step silently discards them.
No written rule fails, since the rows present are compliant. The volume reading deviates from that of previous Mondays and triggers an alert. Lineage points downstream to the delays dashboard, whose distribution the team suspends, and upstream to the depot’s source.
Signals that do not tell the whole story
- A wrong but plausible value moves no signal.
- A threshold deduced from history also reacts to legitimate variations: public holiday, period close, sales promotion. Poorly tuned, it multiplies false alerts.
- An alert observes. Without a designated recipient, it goes nowhere, and the cause continues to produce the defect. In the example, the discarded files still have to be reprocessed.
Qualifying and then correcting a flagged anomaly requires rules, thresholds and a handling process: this is the subject of our data quality page.