When an application is replaced or changes hosting, its code can stop overnight; its data, however, continues to be used. Files, contracts, balances and histories entered over years are expected, accurate and complete, in the system that takes over. The data migration plan is the document that organises this move.
The method we suggest links six steps: inventory and scope, model mapping, cleansing, rehearsals, cutover, then post-cutover checks and rollback. Each delivers a document that the next one needs. The type of migration and the cutover strategy are chosen upstream, according to the criteria in our article on IT migration; here we deal only with the journey of the data.
The inventory says what migrates, what is archived and what is abandoned
The inventory lists the datasets of the source system: tables, files, attachments, configuration settings. It adds what lives alongside the application and what the cutover may break: spreadsheets fed by exports, scheduled extractions, flows to other systems. Where it is maintained, data lineage, read downstream, lists what depends on each table.
For each dataset, the business owner decides its fate: migrated, archived in a form accessible without the old application, or abandoned. They also set the depth of history taken over.
Personal data require a separate reading. The GDPR, Regulation (EU) 2016/679, names “retrieval”, “adaptation or alteration” and “storage” in its definition of processing in Article 4, point (2). Article 5(1)(e) requires data to be kept in a form which permits identification of data subjects “for no longer than is necessary for the purposes for which the personal data are processed”. We draw from this a practice, distinct from the text: asking the question of the retention period for each dataset before taking it over. Deciding that period belongs to the organisation, informed by its data protection officer where there is one.
The step delivers a scope register: one line per dataset, with its fate, its owner, its volume and its history rule.
Model mapping, then cleansing at source
The mapping dictionary links each field of the target model to its source. It gives the transformation rule, the default value and the handling of cases the rule does not cover. Let us take, as an assumption, a student residence manager replacing its rental management application.
| Target field | Source | Rule | Cases to decide |
|---|---|---|---|
| Tenant identifier | File number in the old application | Kept in an “old identifier” field; the target assigns its own | Two files for the same person |
| Date of moving in | Free text field | Conversion to the target’s date format | Unreadable or missing value |
| Lease status | One-letter code | Transcoding to the target’s statuses | Code missing from the transcoding table |
| Account balance | Value calculated by the old application | Recalculated in the target from the entries taken over, then compared | Discrepancy between the two balances |
The last column counts as much as the rule. For each case, the owner chooses between correcting in the source application, transforming by a written rule, or rejecting the record for manual take-on. The table linking old and new identifiers is kept after cutover: it serves downstream systems and later searches. When the same entities exist in several applications, reconciling them falls under master data management.
Cleansing starts from profiling the source, which measures missing, badly formatted or inconsistent values. Corrections are made in the source application, by those who know the files, before the take-on. A correction slipped into a transformation script without appearing in the dictionary can distort reconciliations. Duplicates are handled according to the method in our article on data deduplication; lasting rules and checks are the subject of our data quality page.
These two steps deliver the dictionary approved by the owner, and the list of anomalies with, for each, a decision and an owner.
Rehearsals put the data migration plan to the test
A rehearsal runs the whole chain (extraction, transformation, loading, checks) on a copy of production, in an environment comparable to the target. It measures three things: the duration of each operation, rejections rule by rule, and what separates source and target in number of rows as well as in checksums.
The measured durations size the cutover. Let us set some hypothetical input figures: 12 million rows to load, a measured throughput of 2,000 rows per second, 40 minutes of extraction and 1 hr 30 min of checks. Loading takes 12,000,000 / 2,000 = 6,000 seconds, that is 1 hr 40 min. The whole chain requires 40 + 100 + 90 = 230 minutes, that is 3 hr 50 min. In an 8-hour window, 4 hr 10 min remain for contingencies and for a possible rollback. Halving the throughput would take loading to 3 hr 20 min and the chain to 5 hr 30 min: the margin would fall to 2 hr 30 min.
The production copy used may contain personal data. Article 32 of the GDPR cites “the pseudonymisation and encryption of personal data” among the security measures to be implemented “as appropriate”; the possible techniques are compared in our article on anonymisation.
Each rehearsal delivers a report: durations, rejections, discrepancies, corrections to be made to the dictionary or the source. It also updates the timed cutover sequence.
Cutover, post-cutover checks and rollback
The cutover follows the sequence tested in rehearsal: freeze on data entry in the source, backup, extraction, loading, checks, decision to reopen. Each line bears an owner and a measured duration. The window is negotiated with the business, the only judge of the closure it can bear; our entry on RTO RPO gives the related vocabulary of acceptable interruption.
Before reopening, the checks follow one another from the most global to the most detailed. Counts verify, for each dataset, that the rows extracted are found loaded or rejected. Control totals compare sums and distributions: in the assumption chosen, the total of balances and the number of leases by status. A sample of files is reviewed by the business in the target, and a few user journeys are played out in it. The owner signs off the reconciliation; accepted discrepancies appear in it, each with its explanation.
Downstream flows are checked in the same way. When pipelines feed the data platform from the old model, they are adapted before cutover, and the indicators they produce are compared on both sides.
Rollback has its deadline, also derived from the measurements: the end of the window minus the observed duration of the restore. With the 8-hour window of the assumption, opening at 10 pm and closing at 6 am, and a restore measured at one hour, the decision is taken at 5 am at the latest. The findings that trigger it and the person who takes it are written down in advance. After reopening, rolling back means dealing with the data entered in the target, a point developed in our article on IT migration.
When the target is hosted by a cloud provider, transferring volumes over the network adds to the durations to be measured, and the plan joins the cutover file for each wave, described on our cloud migration page.
The plan closes on two dates: that of the last signed reconciliation, and that on which the source, kept until then in read-only mode, is switched off.