Skip to content
Technology transformation expertise firm · ARDIANE Group Innovate, secure, succeed

Cloud and platforms

Building a data migration plan and rehearsing it before cutover

In brief

A data migration plan organises the move of data from a source system to a target system: what migrates, under which rules, in which window and with which checks. Management Square sets out here a method in six successive steps, from inventory to post-cutover checks and rollback, each closed by a deliverable that the next one uses.

Page contents

When an application is replaced or changes hosting, its code can stop overnight; its data, however, continues to be used. Files, contracts, balances and histories entered over years are expected, accurate and complete, in the system that takes over. The data migration plan is the document that organises this move.

The method we suggest links six steps: inventory and scope, model mapping, cleansing, rehearsals, cutover, then post-cutover checks and rollback. Each delivers a document that the next one needs. The type of migration and the cutover strategy are chosen upstream, according to the criteria in our article on IT migration; here we deal only with the journey of the data.

The inventory says what migrates, what is archived and what is abandoned

The inventory lists the datasets of the source system: tables, files, attachments, configuration settings. It adds what lives alongside the application and what the cutover may break: spreadsheets fed by exports, scheduled extractions, flows to other systems. Where it is maintained, data lineage, read downstream, lists what depends on each table.

For each dataset, the business owner decides its fate: migrated, archived in a form accessible without the old application, or abandoned. They also set the depth of history taken over.

Personal data require a separate reading. The GDPR, Regulation (EU) 2016/679, names “retrieval”, “adaptation or alteration” and “storage” in its definition of processing in Article 4, point (2). Article 5(1)(e) requires data to be kept in a form which permits identification of data subjects “for no longer than is necessary for the purposes for which the personal data are processed”. We draw from this a practice, distinct from the text: asking the question of the retention period for each dataset before taking it over. Deciding that period belongs to the organisation, informed by its data protection officer where there is one.

The step delivers a scope register: one line per dataset, with its fate, its owner, its volume and its history rule.

Model mapping, then cleansing at source

The mapping dictionary links each field of the target model to its source. It gives the transformation rule, the default value and the handling of cases the rule does not cover. Let us take, as an assumption, a student residence manager replacing its rental management application.

Target field Source Rule Cases to decide
Tenant identifier File number in the old application Kept in an “old identifier” field; the target assigns its own Two files for the same person
Date of moving in Free text field Conversion to the target’s date format Unreadable or missing value
Lease status One-letter code Transcoding to the target’s statuses Code missing from the transcoding table
Account balance Value calculated by the old application Recalculated in the target from the entries taken over, then compared Discrepancy between the two balances

The last column counts as much as the rule. For each case, the owner chooses between correcting in the source application, transforming by a written rule, or rejecting the record for manual take-on. The table linking old and new identifiers is kept after cutover: it serves downstream systems and later searches. When the same entities exist in several applications, reconciling them falls under master data management.

Cleansing starts from profiling the source, which measures missing, badly formatted or inconsistent values. Corrections are made in the source application, by those who know the files, before the take-on. A correction slipped into a transformation script without appearing in the dictionary can distort reconciliations. Duplicates are handled according to the method in our article on data deduplication; lasting rules and checks are the subject of our data quality page.

These two steps deliver the dictionary approved by the owner, and the list of anomalies with, for each, a decision and an owner.

Rehearsals put the data migration plan to the test

A rehearsal runs the whole chain (extraction, transformation, loading, checks) on a copy of production, in an environment comparable to the target. It measures three things: the duration of each operation, rejections rule by rule, and what separates source and target in number of rows as well as in checksums.

The measured durations size the cutover. Let us set some hypothetical input figures: 12 million rows to load, a measured throughput of 2,000 rows per second, 40 minutes of extraction and 1 hr 30 min of checks. Loading takes 12,000,000 / 2,000 = 6,000 seconds, that is 1 hr 40 min. The whole chain requires 40 + 100 + 90 = 230 minutes, that is 3 hr 50 min. In an 8-hour window, 4 hr 10 min remain for contingencies and for a possible rollback. Halving the throughput would take loading to 3 hr 20 min and the chain to 5 hr 30 min: the margin would fall to 2 hr 30 min.

The production copy used may contain personal data. Article 32 of the GDPR cites “the pseudonymisation and encryption of personal data” among the security measures to be implemented “as appropriate”; the possible techniques are compared in our article on anonymisation.

Each rehearsal delivers a report: durations, rejections, discrepancies, corrections to be made to the dictionary or the source. It also updates the timed cutover sequence.

Cutover, post-cutover checks and rollback

The cutover follows the sequence tested in rehearsal: freeze on data entry in the source, backup, extraction, loading, checks, decision to reopen. Each line bears an owner and a measured duration. The window is negotiated with the business, the only judge of the closure it can bear; our entry on RTO RPO gives the related vocabulary of acceptable interruption.

Before reopening, the checks follow one another from the most global to the most detailed. Counts verify, for each dataset, that the rows extracted are found loaded or rejected. Control totals compare sums and distributions: in the assumption chosen, the total of balances and the number of leases by status. A sample of files is reviewed by the business in the target, and a few user journeys are played out in it. The owner signs off the reconciliation; accepted discrepancies appear in it, each with its explanation.

Downstream flows are checked in the same way. When pipelines feed the data platform from the old model, they are adapted before cutover, and the indicators they produce are compared on both sides.

Rollback has its deadline, also derived from the measurements: the end of the window minus the observed duration of the restore. With the 8-hour window of the assumption, opening at 10 pm and closing at 6 am, and a restore measured at one hour, the decision is taken at 5 am at the latest. The findings that trigger it and the person who takes it are written down in advance. After reopening, rolling back means dealing with the data entered in the target, a point developed in our article on IT migration.

When the target is hosted by a cloud provider, transferring volumes over the network adds to the durations to be measured, and the plan joins the cutover file for each wave, described on our cloud migration page.

The plan closes on two dates: that of the last signed reconciliation, and that on which the source, kept until then in read-only mode, is switched off.

FAQ

Frequently asked questions

Another question? Ask us

What does a data migration plan contain?

It brings together the scope decided for each dataset (migrated, archived or abandoned), the mapping dictionary between the source model and the target model, the list of anomalies to be corrected before the take-on, the schedule of rehearsals with their success criteria, the timed cutover sequence, the checks to be passed before reopening the service and the rollback procedure. Each document bears the name of the person who approves it.

Do data migration and data take-on mean the same thing?

In practice, the two expressions are used interchangeably. When they are distinguished, "data take-on" takes the point of view of the target system, which takes over the existing data of a replaced application, and "data migration" describes the complete journey, from extraction to checks. In a contract or a plan, it is better to write down what the chosen expression covers: extraction, transformation, loading, checks, archiving of what is not taken over.

Should the entire history be migrated?

Not necessarily. The depth of history is decided dataset by dataset, with the business owner: what the target system needs in order to function, what must remain accessible under retention obligations, what can be abandoned. What is not migrated is archived in a form that can be read without the old application, since that application will be switched off.

How many rehearsals should be planned before cutover?

The number is not set in advance: it depends on what each rehearsal reveals. We suggest a stopping criterion rather than a count: the last rehearsal covered the whole scope and real volumes, it fitted within the window with a margin, its discrepancies are all explained and the rollback was played out in it. As long as one of these conditions is missing, an extra rehearsal is justified.

Does a data migration fall under the GDPR?

Regulation (EU) 2016/679 covers data relating to natural persons. Among the operations that its Article 4, point (2), includes in the notion of processing are retrieval, adaptation or alteration and storage, three actions in a migration; its Article 32 requires a level of security appropriate to the risk. How the text applies to a given migration and which measures to adopt is a matter for the organisation, advised by its data protection officer where there is one.

A project to bring to completion?

A first conversation is enough to understand your challenge, identify the right people and tell you frankly what we can bring.