Skip to content
Technology transformation expertise firm · ARDIANE Group Innovate, secure, succeed

Data

Which lever should you pull first to improve data quality?

In brief

Improving data quality means choosing, for each defect whose cause has been established, the lever that stops it from forming again: a fix at source, a data entry rule, a check in the processing pipeline or the appointment of an owner. Management Square sets out here the criteria for this choice, the order in which to pull the levers and the cases where it is better not to correct.

Page contents

Suppose an audit has just delivered its conclusions on the addresses of the members of a mutual insurer. It establishes four causes: an address entered as free text at subscription, an interface that truncates the second address line, a job in the management application that restores the old address after a move, and the absence of anyone responsible for dealing with returned mail. Management will not fund everything in the same year.

Improving data quality then comes down to choosing which lever to start with: fixing the source, framing data entry, checking in the processing pipelines or assigning responsibilities. The rule we propose fits in one sentence. The first lever is the one that stops the defect from forming again, at the furthest upstream point where the organisation has control, and reworking records that are already wrong comes afterwards. It carries two reservations: someone must be responsible for handling what the lever brings to light, and the defect must be worth correcting.

Improving data quality starts from established causes

This article begins where a diagnosis ends. If the origin of the defects is not known, choosing a lever is a gamble: our article on the data quality audit describes how to establish it and what remediation plan comes out of it. The lines of such a plan fall under four levers, which do not act in the same place.

Lever Where it acts Its effect Its limit
Fix at source The application or process that produces the data The cause disappears: interface, business rule, screen Records that are already wrong stay wrong
Data entry rule The moment the data is created What can be recorded is bounded: mandatory field, list of values, format, duplicate search A well-formed value can remain inaccurate
Check in the pipeline The processing between source and use The spread of the defect is stopped or reported The cause remains intact at the source
Responsibility The organisation A person sets the expected level, receives reports and can have the source modified Data does not change through the appointment alone

Rules, checks and the anomaly handling circuit are presented in our data quality page. The question addressed here is that of order.

The starting lever is chosen according to the source, the flow, the deadline and the recipient

The choice is made defect by defect, based on findings that the remediation plan makes it possible to establish.

The reach of the source comes first. An application developed in-house can be modified. A software package can be configured within the limits set by its vendor. Data bought or received from a partner is corrected through the contract. When the source is out of reach, the check in the pipeline becomes the starting lever, for want of any other.

The flow comes next. A defect inherited from an old data migration, which nothing feeds any longer, calls for a one-off correction of the stock and no permanent mechanism. A defect created every day requires action on whatever creates it before any clean-up.

The deadline of the use can change this order. A near date, for example that of a regulatory filing, can justify a check placed downstream without waiting for the source to be fixed, provided it is written down that the check is temporary.

The recipient conditions the other three findings. If nobody is responsible for anomalies, a technical lever produces reports that go nowhere. Appointing an owner then comes first, among the owners and stewards appointed by data governance.

In the starting scenario, the interface that truncates and the job that restores the old address are fixed at source. The free text calls for a data entry rule. Returned mail, for its part, already flags wrong addresses: as long as nobody deals with it, none of the other three levers can be judged on its effects, and it is with this responsibility that the plan begins.

A data entry rule can turn against the data

Making a field mandatory does not make the information appear. If the person entering it does not have it at that moment, or has neither instructions nor time to look for it, the field may be filled with a placeholder value: a fictitious postcode, a default date, the first line of a list. The blank, which could be counted, gives way to a false value that completeness checks no longer see.

Before setting the rule, the plan therefore checks that the information exists at the time of entry, that the person entering it uses it or has been instructed to do so, and that an explicit “unknown” value is provided for legitimate cases. The rule is placed at the step in the process where the information becomes available, which may come after the record is created.

The check in the pipeline, temporary or permanent

A check placed downstream stops or reports a defect. It has two uses. Permanent, it monitors what no data entry rule guarantees and what comes from sources out of reach. Temporary, it protects a use while waiting for the source to be fixed. The choice of tests is compared in our article on data quality testing methods.

The drift to watch for is silent correction: a transformation that repairs the data in the warehouse at every load. The source remains wrong for its other consumers, and the repair rule becomes code to maintain whose rationale can get lost. If such a correction is adopted, it is written down with its rationale, the cause it compensates for and the condition for its removal.

Defects it is better to leave in place

Refraining is a decision that belongs to the data owner. It is justified in several situations.

  • The level reached is sufficient for the known uses of the data.
  • The source application is being replaced: the correction is carried over into the migration project’s conversion rules.
  • The correction would cost more than the defect, across all the identified uses.
  • The correction would destroy information. Replacing a missing value with an estimated value erases the fact that it was missing. Rewriting a history prevents figures already published from being reproduced. It is then better to flag the record than to modify it.

This freedom has a legal limit. For personal data, Article 5(1)(d) of the GDPR, Regulation (EU) 2016/679, requires that “every reasonable step” be taken to ensure that data that are inaccurate, having regard to the purposes of the processing, are erased or rectified without delay. Its Article 16 gives the individual a right to rectification.

A defect left in place remains flagged to those who use the data, in the catalogue or in the report that relies on it.

The first lever is judged on new records

The effect of a lever is read on the records created or modified after it was put in place, measured separately from the stock. An overall rate mixes the two. It falls slowly even though the source has been cured, because it still carries the old records. It can also fall after a clean-up even though the source continues to produce defects.

If new records remain defective, the cause identified was not the right one, or not the only one, and that is what must be reopened. If they comply, the clean-up of the stock can be started without fear of having to redo it, and the next lever is chosen with the same criteria.

FAQ

Frequently asked questions

Another question? Ask us

Where should you start to improve data quality?

With a defect whose cause has been established, and with the lever that stops it from forming again: a fix to the source application, a data entry rule, a check in the processing pipeline or the appointment of an owner. The choice depends on whether the source can be modified, whether the defect recurs, the deadline of the use and whether there is a person responsible for handling anomalies. Reworking existing records comes afterwards.

Should existing data be cleansed before the source is dealt with?

No, unless the defect no longer forms. As long as the source produces defective records, a clean-up of the existing stock will have to be redone. An exception can be discussed when a deadline requires a clean stock by a given date, for example a migration: the clean-up is then accepted as a one-off operation.

Is a data quality tool enough to improve quality?

A profiling or checking tool measures and reports. It does not modify the application that produces the data and does not appoint anyone to handle anomalies. It therefore serves the lever of checks in processing pipelines; the other three require a decision on an application, a process or a responsibility.

Can you decide not to correct inaccurate data?

Yes, when the level reached is sufficient for the uses, when the source is about to be replaced or when the correction would destroy information; the decision belongs to the data owner and is recorded. For personal data, this freedom is limited by Article 5(1)(d) of the GDPR, Regulation (EU) 2016/679, which requires every reasonable step to be taken to ensure that data that are inaccurate, having regard to the purposes, are erased or rectified without delay. Its Article 16 also gives the individual a right to rectification.

How do you check that an improvement holds over time?

By measuring separately the records created after the lever was put in place and the earlier stock. If the defect rate of new records does not fall, the cause addressed was not the right one, or not the only one. A permanent check placed in the processing pipeline then reports any regression.

A project to bring to completion?

A first conversation is enough to understand your challenge, identify the right people and tell you frankly what we can bring.