Suppose an audit has just delivered its conclusions on the addresses of the members of a mutual insurer. It establishes four causes: an address entered as free text at subscription, an interface that truncates the second address line, a job in the management application that restores the old address after a move, and the absence of anyone responsible for dealing with returned mail. Management will not fund everything in the same year.
Improving data quality then comes down to choosing which lever to start with: fixing the source, framing data entry, checking in the processing pipelines or assigning responsibilities. The rule we propose fits in one sentence. The first lever is the one that stops the defect from forming again, at the furthest upstream point where the organisation has control, and reworking records that are already wrong comes afterwards. It carries two reservations: someone must be responsible for handling what the lever brings to light, and the defect must be worth correcting.
Improving data quality starts from established causes
This article begins where a diagnosis ends. If the origin of the defects is not known, choosing a lever is a gamble: our article on the data quality audit describes how to establish it and what remediation plan comes out of it. The lines of such a plan fall under four levers, which do not act in the same place.
| Lever | Where it acts | Its effect | Its limit |
|---|---|---|---|
| Fix at source | The application or process that produces the data | The cause disappears: interface, business rule, screen | Records that are already wrong stay wrong |
| Data entry rule | The moment the data is created | What can be recorded is bounded: mandatory field, list of values, format, duplicate search | A well-formed value can remain inaccurate |
| Check in the pipeline | The processing between source and use | The spread of the defect is stopped or reported | The cause remains intact at the source |
| Responsibility | The organisation | A person sets the expected level, receives reports and can have the source modified | Data does not change through the appointment alone |
Rules, checks and the anomaly handling circuit are presented in our data quality page. The question addressed here is that of order.
The starting lever is chosen according to the source, the flow, the deadline and the recipient
The choice is made defect by defect, based on findings that the remediation plan makes it possible to establish.
The reach of the source comes first. An application developed in-house can be modified. A software package can be configured within the limits set by its vendor. Data bought or received from a partner is corrected through the contract. When the source is out of reach, the check in the pipeline becomes the starting lever, for want of any other.
The flow comes next. A defect inherited from an old data migration, which nothing feeds any longer, calls for a one-off correction of the stock and no permanent mechanism. A defect created every day requires action on whatever creates it before any clean-up.
The deadline of the use can change this order. A near date, for example that of a regulatory filing, can justify a check placed downstream without waiting for the source to be fixed, provided it is written down that the check is temporary.
The recipient conditions the other three findings. If nobody is responsible for anomalies, a technical lever produces reports that go nowhere. Appointing an owner then comes first, among the owners and stewards appointed by data governance.
In the starting scenario, the interface that truncates and the job that restores the old address are fixed at source. The free text calls for a data entry rule. Returned mail, for its part, already flags wrong addresses: as long as nobody deals with it, none of the other three levers can be judged on its effects, and it is with this responsibility that the plan begins.
A data entry rule can turn against the data
Making a field mandatory does not make the information appear. If the person entering it does not have it at that moment, or has neither instructions nor time to look for it, the field may be filled with a placeholder value: a fictitious postcode, a default date, the first line of a list. The blank, which could be counted, gives way to a false value that completeness checks no longer see.
Before setting the rule, the plan therefore checks that the information exists at the time of entry, that the person entering it uses it or has been instructed to do so, and that an explicit “unknown” value is provided for legitimate cases. The rule is placed at the step in the process where the information becomes available, which may come after the record is created.
The check in the pipeline, temporary or permanent
A check placed downstream stops or reports a defect. It has two uses. Permanent, it monitors what no data entry rule guarantees and what comes from sources out of reach. Temporary, it protects a use while waiting for the source to be fixed. The choice of tests is compared in our article on data quality testing methods.
The drift to watch for is silent correction: a transformation that repairs the data in the warehouse at every load. The source remains wrong for its other consumers, and the repair rule becomes code to maintain whose rationale can get lost. If such a correction is adopted, it is written down with its rationale, the cause it compensates for and the condition for its removal.
Defects it is better to leave in place
Refraining is a decision that belongs to the data owner. It is justified in several situations.
- The level reached is sufficient for the known uses of the data.
- The source application is being replaced: the correction is carried over into the migration project’s conversion rules.
- The correction would cost more than the defect, across all the identified uses.
- The correction would destroy information. Replacing a missing value with an estimated value erases the fact that it was missing. Rewriting a history prevents figures already published from being reproduced. It is then better to flag the record than to modify it.
This freedom has a legal limit. For personal data, Article 5(1)(d) of the GDPR, Regulation (EU) 2016/679, requires that “every reasonable step” be taken to ensure that data that are inaccurate, having regard to the purposes of the processing, are erased or rectified without delay. Its Article 16 gives the individual a right to rectification.
A defect left in place remains flagged to those who use the data, in the catalogue or in the report that relies on it.
The first lever is judged on new records
The effect of a lever is read on the records created or modified after it was put in place, measured separately from the stock. An overall rate mixes the two. It falls slowly even though the source has been cured, because it still carries the old records. It can also fall after a clean-up even though the source continues to produce defects.
If new records remain defective, the cause identified was not the right one, or not the only one, and that is what must be reopened. If they comply, the clean-up of the stock can be started without fear of having to redo it, and the next lever is chosen with the same criteria.