Skip to content
Technology transformation expertise firm · ARDIANE Group Innovate, secure, succeed

Data

Which data quality metrics to track, and how to calculate them

In brief

Data quality metrics are calculated measures, with a constant definition, on records or on the handling of their anomalies: rate of conformity to a rule, fill rate, duplicate rate, availability lead time, age of open anomalies. Management Square specifies here, for each one, what it measures, its formula in words, its reader and what it does not say.

Page contents

A quality rate displayed without the rule checked, without the population counted and without the date of measurement cannot be verified, and cannot be compared with any other. We use six data quality metrics here. Four are calculated on records: the rate of conformity to a rule, the fill rate, the duplicate rate and the availability lead time. Two are calculated on the handling of defects: the age of open anomalies and the share of defects detected by checks before a user reports them.

For each, this article gives what it measures, its formula in words, the person it serves and what it leaves in the shadows. None of these metrics contains its threshold: the expected level is decided separately, use by use.

Rule, denominator and date of measurement define a metric

A quality metric is here a ratio, a lead time or a count. We suggest that its specification carry four items: what is counted in the numerator, the population of the denominator, the granularity (a rule, a source, a domain) and the date of the measurement.

Take the supplier master data of a furniture manufacturer, that is 48,000 active records, and the rule “every active record carries a populated SIREN number”. If 45,600 records comply with it, the conformity rate relates these 45,600 compliant records to the 48,000 records checked. These quantities, assumed for the example, illustrate the formula and set no level to be reached. Calculated only on records created during the year, or on all records, inactive ones included, the same count would give a different rate: the denominator is part of the definition.

Two neighbouring notions are distinct from it. The dimension (completeness, uniqueness, freshness) names the angle of judgement and serves to classify metrics; choosing a list is the subject of our article on data quality dimensions. The threshold is a target set on the measure, just as the objective is distinguished from the indicator in our entry SLO, SLI and SLA.

Data quality metrics calculated on records

The conformity rate relates the number of records that comply with a rule to the number of records that rule checks. Automated checks produce it at each run. It serves the data owner, who set the rule, and the steward who investigates the deviations. It ignores defects that no rule targets and, averaged over several rules, can remain stable while one of them deteriorates.

The team that enters a field and the team that uses it read the fill rate: the populated values, relative to the records in which that field is expected. The UK government’s data quality framework (“The Government Data Quality Framework”, published on 3 December 2020 on gov.uk) gives an example: a school expects an emergency contact number for 300 pupils and collects 294, which the text puts at 98%. It adds that these numbers are not necessarily correct. This rate does not separate a real value from a default value, unless the convention excludes the latter from the numerator.

The duplicate rate has two conventions. One relates the surplus records to the total number of records. The other gives its complement, uniqueness, like the example in the UK framework: the real entities relative to the records, that is 500 pupils for 501 records, which that text puts at 99.8%. Both assume that the records designating the same entity have been matched. The rate therefore depends on the matching rule and varies when it is tightened or relaxed, without the data having changed: the master data manager reads it together with that rule. The method is described in our article on data deduplication.

Between a business event and the moment the data describing it becomes usable runs the availability lead time. It is tracked as a value, or as the share of loads that arrived within the agreed time, for the user who decides at a fixed time and for the team that operates the platform. Measured on the load time, it does not see a source that sends the same content every day, a point covered in our comparison of data quality testing methods.

Accuracy has no metric as direct. It is measured by comparison with a reference source, or by checking a sample against reality; the rate obtained then holds for the sample and for its date.

Measures drawn from the anomaly handling process

The age of open anomalies counts, at the date of measurement, the anomalies not closed and distributes them by age; a variant tracks the median correction time of anomalies closed over the period. Its reader is the body that arbitrates corrections. Its definition specifies the moment at which an anomaly is opened (detection or qualification) and what closes it (correction at source, or a workaround downstream). This measure says nothing about severity, and a closure by workaround lowers the stock without the cause having disappeared.

The share of defects detected by checks relates the data incidents revealed by a check to the total of known incidents over the period, user reports included. Our data quality page sees in it the sign that sums up the ambition of a programme of work, and describes the process that produces these counts. The management that funds the arrangement looks there for the effect of the checks. The denominator contains only known incidents: a defect that no one has seen enters neither term, and the share rises if users stop reporting.

Stock, flow and granularity change the reading of a rate

An overall rate mixes old records and those just created. Measuring separately the records created or modified over the period indicates whether the source is still producing the defect; our article on the levers to improve data quality makes it the criterion for judging a correction.

A fill rate broken down by entry channel or by source application points to the place where the field gets lost.

Counting can also mislead about what is at stake. Each record carries the same weight in a rate, whereas the non-compliant records may be those that carry most of the orders. A variant weights by a volume of activity, provided this is written into the definition.

A rate taken as a target can also change the way data is entered. As soon as a team is judged on how well a field is filled, default values can multiply there, and the rate rises without the data being better known. Tracking it together with the share of default values limits this effect.

The threshold is decided by use, with the data owner

The UK framework states that perfect quality is not always achievable and that the effort aims for data “as fit for purpose as it can be”. It accepts trade-offs between dimensions: prioritising timeliness can cost completeness.

We therefore suggest entrusting the threshold to the data owner, in the sense of our data governance page. It is set in advance, together with the action to take if it is crossed: blocking a release, opening an anomaly, a simple report. When a data set is delivered under a data contract, the threshold and the expected lead time can appear in it, and checking them then provides the measure.

A first set of metrics can be limited to the rules already checked on the data for a single use. The conformity rate of each, measured separately on the records created during the month, is enough to open the first review with their owner.

FAQ

Frequently asked questions

Another question? Ask us

Which data quality metrics should you start with?

We suggest starting with the conformity rate of each rule already checked on the data for a specific use, calculated separately on recent records and on the existing stock. The fill rate and the duplicate rate are special cases of it, as soon as a completeness or uniqueness rule is written. The age of open anomalies is added when a handling process exists: it indicates whether the defects reported are being corrected.

How do you calculate a completeness rate?

A completeness rate, or fill rate, divides the number of populated values in a field by the number of records in which that field is expected. The UK government's data quality framework, published in 2020, gives as an example 294 numbers collected for 300 pupils, which it puts at 98%. The calculation convention specifies two points: records for which the field is not applicable are removed from the denominator, and default values do or do not count as populated.

What threshold should you aim for with a data quality metric?

The UK government's data quality framework, published in 2020, states that perfect quality is not always achievable and refers to the fitness of data for its purpose. We therefore suggest setting the threshold by use, with the data owner, together with the action to take if it is crossed. The fill rate of a telephone number may suit a sales follow-up without suiting an alert procedure.

How often should data quality metrics be calculated?

Rates produced by automated checks can be calculated at each run of the processing pipeline, since the check supplies them. Their review follows a different rhythm, that of the body in which data owners arbitrate corrections. Between two reviews, only the crossing of a threshold set in advance calls for a reaction.

A project to bring to completion?

A first conversation is enough to understand your challenge, identify the right people and tell you frankly what we can bring.