Clean-up campaigns that do not last
Almost every organisation has already run a clean-up campaign: deduplicating the customer file before a migration, making addresses reliable, putting a product master back in order. The result holds for a while, then data quality returns to its starting level. The same causes have produced the same anomalies.
Three methodological errors explain this backsliding. The data was corrected without touching the process that produces it: an entry screen without controls, an interface that truncates a field, a business rule understood differently by different teams. The aim was “clean” data in general, whereas the level required depends on the use. And checks, where they exist, are manual and one-off: they observe, they do not prevent.
The subject takes on particular significance with AI. A model trained on erroneous data learns those errors, and an assistant that relies on a document base reproduces its contradictions with confidence.
The six dimensions of data quality
Talking about quality without specifying the dimension concerned leads to endless debates. Six dimensions cover most needs.
| Dimension | Question asked | Example rule |
|---|---|---|
| Accuracy | Does the value reflect reality? | The delivery address corresponds to an existing address |
| Completeness | Is the necessary information present? | Every active contract has an effective date |
| Consistency | Are the values compatible with one another and from one system to another? | A customer’s status is identical in the sales tool and in billing |
| Validity | Does the value respect the expected format and list of values? | The country code belongs to the current nomenclature |
| Uniqueness | Does a given entity appear only once? | A supplier has only one identifier |
| Timeliness | Is the data up to date and available in time? | The previous day’s sales are loaded before the stores open |
These dimensions do not carry the same weight depending on the use. A regulatory report requires accuracy and completeness. A recommendation engine tolerates missing values, but not out-of-date data. This is why a quality rule is always written for a use and with the business concerned.
The law, moreover, recognises some of these dimensions. The GDPR, Regulation (EU) 2016/679, sets out in its Article 5 a principle of accuracy of personal data. Article 10 of the AI Act, Regulation (EU) 2024/1689, requires for high-risk AI systems training, validation and testing datasets that are relevant, sufficiently representative and, to the best extent possible, free of errors and complete in view of the intended purpose of the system (paragraph 3).
From rules to automated controls
A usable rule has four elements: a statement the business can understand, an acceptance threshold, an owner and a course of action on failure. Without the last two, the control produces alerts that nobody reads.
Controls are placed at four points in the chain.
- At data entry, in the source application: lists of values, mandatory fields, duplicate detection at creation. This is the least costly control.
- At ingestion: schema conformity, expected volumes, freshness of the load.
- During transformation: business rules, integrity between tables, reconciliations between systems.
- Before exposure: plausibility of aggregates, comparison with previous periods.
Each control is either blocking or merely flagged. A blocking control stops the release of questionable data, at the cost of a late report. This choice belongs to the business, not to the technical team. In addition to explicit rules, there is statistical detection of unusual variations in volume or distribution, useful for anomalies that nobody had anticipated.
Master data: what MDM resolves, and its four styles
Master data is the data that several applications share: customers, products, suppliers, sites, organisational structure. Nomenclatures (country codes, currencies, charts of accounts) are also included. When each application keeps its own version, no consolidation is reliable.
MDM (Master Data Management) is the discipline that maintains a reference version of these entities: rules for matching records, choice of the value retained in the event of a conflict, process for creation and modification. It is implemented in one of four styles, from the lightest to the most demanding.
| Style | How it works | Consequence |
|---|---|---|
| Registry | An index links the applications’ records without copying or modifying them | Quick to set up, corrects nothing at the source |
| Consolidation | Data is copied and matched in a hub to produce a reference version intended for analysis | Makes reporting and analytics reliable, leaves applications unchanged |
| Coexistence | The reference version is built in the hub, then synchronised back to the applications | Aligns systems, requires flows in both directions |
| Centralised | The hub becomes the single place of creation and modification | Maximum control, profound transformation of processes |
The choice depends less on the tool than on the organisation’s ability to modify its processes and applications. It determines the place of the hub in the information system, which we address with the methods of IT architecture.
Our approach: start from a use, trace back to the source
We start with a use whose value is recognised and with the data it consumes. Profiling that data gives a measured state: the discussion then turns on findings rather than impressions.
The rules are then written with the owners and stewards appointed by data governance. If these roles do not yet exist within the scope, we recommend having them appointed: a rule without an owner is not applied.
Controls are developed as code, versioned and run by the platform’s processing pipelines. Each anomaly follows an explicit process, from qualification to closure. For recurring anomalies, we trace back to the cause and propose the correction in the originating application or process. An MDM workstream is opened only if the assessment shows a problem of sharing between systems, and not a simple data entry defect.
What makes a quality initiative fail
- Cleaning without addressing the cause. A downstream correction has to be redone with every load.
- Requiring the same level everywhere. Quality has a cost. It is concentrated on critical data and proportioned to the use.
- Multiplying controls with no recipient. Alerts that are too numerous and unassigned are ignored, including those that matter.
- Launching MDM as a tool project. Without agreed matching rules or a decision process, the hub becomes one more application to reconcile.
- Forgetting documents. The unstructured content that feeds a RAG system also has its duplicates, its out-of-date versions and its contradictions.
The indicators to track
- The compliance rate by rule and by dimension on critical data, and how it changes over time.
- The number of open anomalies, their age and their correction time.
- The share of anomalies corrected at the source rather than downstream.
- The share of incidents detected by controls before being reported by a user.
- For master data: the duplicate rate and the time taken to create a new record.
The fourth indicator sums up the ambition of the initiative. As long as it is users who discover the errors, trust is not restored, whatever compliance rate is displayed.