Practical Data Migration

Johny Morris

• Publisher: BCS, The Chartered Institute for IT
• Year: 2020
• ISBN:
  9781780175164
• Edition: 3
• Topic: Betriebswirtschaft
• Subtopic: Projektmanagement

This book is for executives, practitioners, and project managers who are tasked with the movement of data from old systems to a new repository. It is designed as a practical guide and uses a series of steps developed in real-life situations that will get you from an empty new system to one that is populated, working and backed by the user population.This new edition is updated to take account of changes in technology and the maturing of the market for data migration services, with two brand new chapters. It guarantees to get the dirty old data out of your legacy systems and transform it into clean new data for your new system.

SECTION 1

PRINCIPLES AND OVERVIEW

1 DATA MIGRATION: WHAT’S ALL THE FUSS?

This chapter explains what an enterprise application data migration is and looks at the key mistakes that lead to the high failure rate of data migrations across the board. It also explains the responsibility gap and why it is instrumental in most failing data migration projects.

WHAT IS DATA MIGRATION?

This book is about data migration, or more properly enterprise application migration, by which I do not mean the relocation of data centres or the regular movement of data between, say, a business system and a data warehouse. Data migration is the one-off movement of data from old systems to a new repository. It is a one-way trip with no return. More formally I define data migration as:

Data migration is the selection, preparation, extraction, transformation and permanent movement of appropriate data that are of the right quality, to the right place, at the right time, and the decommissioning of legacy data stores, to deliver the business transformation aspirations of the organisation.

Each of these clauses is important because collectively they are the key technical and business activities that deliver the changes the organisation is aiming for.

• Selection – in modern, post-client–server environments, there are often multiple potential sources of data. Practical Data Migration version 3 (PDMv3), which this book describes, acknowledges this and will show you how to make selections that balance technical, business and project needs.
• Preparation, extraction, transformation – data quality is one of the most significant challenges to any data migration. Even where data are fine in their existing setting they may not work in the new environment. PDMv3 will show you how transformation rules are generated that have business as well as technical relevance.
• Permanent – PDMv3 is for enterprise application data migration where the data are permanently to be moved from source to target, not the cyclical integration that occurs between transactions and reporting systems, for instance. This is significant because there is no going back from it.
• Movement – there has been an explosion of software tools available to transport data from source to location. This book will introduce you to the various options and explain their strengths and weaknesses.
• Right quality – in the tight timescales of modern data migration projects you do not have time to perfect data. PDMv3 has tools and techniques that will allow you to get your prioritisation decisions right from both a technical and a business perspective.
• Right place – it is imperative that the date is not just placed in the correct field but that it is consistent with other data (e.g. the right date of birth associated with the right person).
• Right time – the business driver for data migration projects usually includes some time-based criteria. PDM will introduce you to tools and techniques that will allow you to stay in control of your project.
• Legacy data stores – these are usually databases or spreadsheets but can be reports, notebooks, rolodexes etc., not just databases.
• Decommissioning – PDMv3 is for enterprise application data migration. Although the legacy data stores may persist for other purposes, the data that have been extracted, and the business processes they were supporting, will be permanently moved to the target. Implicit in a data migration project as opposed to a system integration process is that the legacy data store(s) will cease to exist (at least from the point of view of the impacted group). There will be a point of no return. This book will show you how to use this to your advantage in business engagement activities and to ensure the elegant closedown of legacy systems.
• Business transformation – you only perform enterprise application migrations for business reasons. There will always be some business transformation involved. As you will see when you get to them, the Golden Rules of PDM are predicated on this truth.

First, the bad news

As an industry we are appalling at data migration. Figures suggest a range of 40 per cent to over 50 per cent of data migration projects are over time,¹ over budget or fail entirely.

Outside the public sector, with its greater transparency, it often seems like everybody else is succeeding. This makes it so much harder to admit to having difficulties. The feeling I often encounter is ‘If everybody else is doing it so well, why can’t we?’ Well, let me reassure you, almost everybody else is doing just as badly. Those of us on the inside wince at the millions of dollars we see wasted on failed or floundering data migrations.

So to contextualise that, if you are embarking on a $5m project due to complete in 12 months, say for next March, expect it to finish next August and expect to have to go back to the board for an additional $1.5m. And that’s if you are lucky. It could be longer and much more expensive.

Still feeling good about taking on this challenge? Stick with PDM and you will have vastly increased the chance of still having a career at the end of it (I’m joking, of course – but using PDM could improve your career prospects). Various studies have shown that using a proven methodology greatly increases your chances of on-time, on-budget, zero-defect migrations.²

So, what usually goes wrong?

If the chances of getting your data migration exercise right, on your own, using an approach developed in-house, is significantly worse than using a tried-and-tested methodology like PDM, what are the areas that most often trip up the unwary?

• Techno-centricity – seeing data migration as a purely technical problem. Given the choices over data selection, data preparation, data quality, decommissioning etc., the business needs to provide guidance and understanding and take ownership of the historic data. PDMv3 is built around business ownership of the migration process.
  
  The surveys referenced above recorded that successful data migrations put business engagement and support of the migration ahead of everything else in terms of success criteria.
• Lack of specialist skills – data migration analysts need to have an eclectic mix of skills. They need to have the business-facing skills of a business analyst but also the technical understanding that allows them to interface effectively with their technical colleagues when discussing solutions and the use of migration software. They need to be able to facilitate understanding between their technical and business colleagues. They also need project leadership skills to manage a virtual team of business and technical staff. Finally, they need an understanding of a formal process, if everything they do is not to be reactive and made up on the hoof.
• Underestimating – not knowing the scale of the activities that need to be undertaken leads to underestimating. This is especially true of the unforeseen amount of data preparation activity required. PDMv3 provides a framework of activities that allows for more consistent planning and manages all data issues through a single, integrated and consistent process.
• Uncontrolled recursion – you will see, when I look at the responsibility gap next, how easy it is to fall into the vortex of uncontrolled recursion where problems get batted back and forth across an unnecessary boundary between the project and the business.

What is rarely a problem (indeed so rare that I can remember only one example where it was the case) is the migration technology itself.

This is not to say that you do not have to exercise care in specifying, writing, testing and deploying technology, but that these are activities that over the last 30 or 40 years we, as an industry, have grown good at. If you add to that the better software that is out there now to perform data migration functions, it is not surprising that you can produce good, fit-for-purpose data migration software. There are, however, a number of common confusions at the heart of data migration projects, so that when they go astray, the unwary can be led into thinking that the migration software is to blame when, as the people who wrote it will tell you, it was written, tested and performs exactly to specification. To understand why the specifications can be so often incomplete or misunderstood, you need to appreciate the responsibility gap.

This problem software was not the fault of individual software engineers but a consequence of the stop–start history of the programme. However, by using the tight iterations and system architecture described in Chapters 15, ‘Agile, Waterfall and Blended’, and 16, ‘Release and Configuration Management’, we effectively ran a trial migration every two weeks, allowing us to tune our migration software before the real cutover.

The responsibility gap

To explain the responsibility gap let us start with what I generally call the naïve or industry-standard view of a data migration.

At one end of an imaginary pipeline there are the legacy systems. At the other end there is the target. In between there are the data extraction routines, some transformations where data are enriched, combined, separated and modified to fit the target, and the load programs that will write data to the target. How hard can that be?

To understand why this is naïve let us look at a typical migration story.

There is a point in the project timeline, provided by the suppliers of the new system, that indicates when they will be ready for the first cut of your company’s data. OK, so that point has slipped a little due to some unplanned changes to tailor the solution to your particular needs. The supplier’s account manager and project manager do not appear that fazed by it, so you assume that it is a normal situation. Your project manager has assigned a lead analyst and technical resources and given them a briefing on the requirements. It is obvious where the data will come from – the system you are replacing – it is obvious where they are going. The migration team are reporting back to the steering group regularly. Load program specifications, design, build and test goes ahead to plan.

Then you come to the first test load of real data, and it all goes wrong. Half the data are miss...

Table of contents

• Front Cover
• Half-Title Page
• BCS, THE CHARTERED INSTITUTE FOR IT
• Title Page
• Copyright Page
• Dedicatoin
• Contents
• List of figures and tables
• Author
• Acknowledgements
• Abbreviations
• Glossary
• Introduction
• Section 1: Principles and Overview
• Section 2: Details
• Section 3: Rescuing Failing Data Migration Projects
• Appendices
• Index
• Back Cover