A data quality test is an automated check that compares a data set with an explicit expectation and gives a verdict: compliant, or not. Data quality testing methods fall into families: schema, completeness, uniqueness, freshness and cross-source consistency checks, volume and distribution tests. Data contracts are added to these, setting these expectations at the boundary between the team that produces a data set and those that use it.
No family covers the others. Each detects one type of defect, at a given point in the data pipeline, and lets the rest through. A testing set-up is therefore composed defect by defect. The dimensions of quality, writing rules with the business and handling anomalies are presented in our page data quality; this article sticks to testing methods.
Written rules or statistical detection: the nature of the test
A written rule expresses an expectation known in advance: this column is never empty, this key is unique, this amount is positive. Its verdict is clear and reproducible. Its limit: it checks only what someone thought to write.
A statistical test compares the day’s batch with history: number of rows, rate of null values, mean, distribution by category. It flags an unusual deviation without any rule having been written. The trade-off is twofold. The threshold has to be tuned, otherwise seasonality or an accounting close triggers false alerts. And the alert indicates a change: whether it is an error remains to be established.
A second distinction avoids a frequent misunderstanding. The unit test of a transformation checks the code: it feeds it a small set of fixed inputs, before deployment, in the continuous integration pipeline. The data test checks the real batch, at each run.
Data quality testing methods: what each family detects and lets through
| Family of tests | What it checks | Typical defect detected | What it lets through |
|---|---|---|---|
| Schema | Presence of columns, types, formats, lists of allowed values | Column renamed or deleted at source, type changed, code outside the nomenclature | A wrong but well-formed value |
| Completeness | Mandatory fields filled in, rate of null values below a threshold | Field no longer populated, join that no longer finds a match | A default value that masks the absence: “N/A”, zero, dummy date |
| Uniqueness | A single row per declared key | Batch loaded twice, join that multiplies rows | Functional duplicates: the same entity under two identifiers |
| Freshness | Delay between the check and the most recent data | Late load, frozen source | A table reloaded on time with old content |
| Cross-source consistency | Counts, totals and keys reconciled between source and target, or between two systems | Rows lost or wrongly filtered, variance on an aggregate, key with no match | An error present on both sides |
| Volume | Number of rows in the batch compared with history | Empty, partial or doubled load | The right number of rows, with wrong content |
| Distribution | Mean, quantiles and distribution by category compared with a reference period | Change of unit or coding, outliers | A slow drift, gradually absorbed by the reference |
Several lines call for a comment.
Uniqueness tested on a key says nothing about functional duplicates. Two records for the same supplier carry two distinct identifiers and pass the test. Detecting them falls under record matching, handled with reference data management.
Freshness is measured in two ways. The time of the last load indicates that the process ran. The date of the last business event contained in the table indicates that the data is recent. Only the second detects a source that sends the same file every day.
Reconciliation between sources assumes a common cut-off date and a declared tolerance. Two systems that do not close at the same moment show legitimate variances, and a test without tolerance raises an alert every morning.
Volume and distribution tests require no business rule. They can therefore be extended to all tables, whereas written rules concentrate on critical data.
The place of each test in the data pipeline
Mapped to the checkpoints of a pipeline, from ingestion to exposure, the families of tests are distributed as follows.
At ingestion come schema, volume and freshness. These tests do not depend on business meaning, cost little to run and stop a defective batch before it feeds the following processes.
After each transformation come key uniqueness, threatened by any join, the completeness of calculated fields, integrity between tables and business rules.
Before exposure come cross-source consistency and distribution, applied to the aggregates users will read. When the data feeds a model, the distribution test joins the drift monitoring described in our page MLOps and LLMOps.
Two mechanisms give these tests their effect. The first consists in writing the batch to an area that consumers do not see, testing it, then publishing it only if it passes. The second quarantines only the faulty rows and lets the others through: the table remains available, but its totals become partial, which is decided with the business.
Tests are code. They are versioned with the transformations, run by the orchestrator and their results are kept. Tools such as Great Expectations, Soda, Deequ or the tests built into dbt make it possible to declare them rather than program them one by one. The whole runs in the layers of a data platform.
Data contracts: the test moved upstream to the producer
A data contract is a versioned, machine-readable document in which the team that produces a data set commits to those that consume it: schema, meaning of the fields, quality rules, freshness deadline, owner, notice period in the event of change. Open specifications, such as the Open Data Contract Standard from the Bitol project (Linux Foundation), propose a common structure for it.
The contract is not an additional family of tests. It carries the previous ones and changes the moment at which they run. It is checked a first time in the producer’s integration pipeline: a schema change that breaks the contract is rejected before it is deployed. It is checked a second time at each data delivery.
It thus detects what the other tests observe too late: the change made at the source, before it reaches the platform. It lets through everything it does not mention. And a contract written by the consumers alone, without commitment from the producer, remains documentation. The approach therefore requires an identified owner on the source side, which the notion of data product in the data mesh formalises.
The blind spots common to all families of tests
Accuracy largely escapes tests. An address that exists but is not the customer’s, an amount entered with a plausible error: the value is well-formed, present, unique, fresh. Only comparison with a reference source or with reality disproves it.
A change of meaning also gets through. A field that contained an amount excluding tax and now receives an amount including tax keeps its name and type. A distribution test will flag it if the deviation is marked enough, without giving the cause.
Finally, a test detects and corrects nothing. Finding the cause in the originating application or process falls under the anomaly handling circuit described in our data quality offering.
Checklist before adding a test
- The targeted defect is named, because it has already occurred or because a use fears it.
- The test is placed at the first point in the pipeline where this defect becomes detectable.
- Its threshold and tolerance have been set with the data owner.
- Its effect in the event of failure is decided: blocking the batch, quarantining the rows or a simple alert.
- The alert has a named recipient, who knows what to do on receiving it.
- The test is versioned with the transformation it protects, and its history is kept.
- What it lets through is known, and covered by another test if the risk justifies it.