Data migration – getting it right

Data migration is necessary when moving and/or upgrading enterprise software to a new data centre. We recently completed a software migration project for a major professional services firm.

For administrative and performance reasons, they needed to move and upgrade their IBM CLM system and all its artefacts from the UK to India (where their development team is based).

Having been involved in a number of these projects, there are definitely best practices to adhere to.

This can essentially be divided into things to consider in advance, things to consider during the process, and things to consider on project completion.

Before you start

Make sure you have a plan. Sounds simple enough but one of the main reasons migrations go wrong is poor planning.

Migration plans need to be based on realistic timescales. Tried and tested elements of the plan can be accurately estimated but unknowns need time allowances to deal with any unexpected occurrences. Once completed and agreed with senior management, make sure your migration schedule is well communicated to all stakeholders.

The two main risks involved in any migration are loss of data and extended downtime, therefore good data backups are required. If you need to roll the migration back it’s essential you’ve got this version to revert to.

Many migrations also require the use of a test environment to assess whether the chosen migration method will work, especially when it’s untried or when the business importance of the data migration is so high, no outage can be tolerated.

This is more often the case if the upgrade process you’re involved with isn’t well documented because it hasn’t been carried out that often. Most enterprise software comes with its own upgrade utility – some that have been used hundreds of times don’t require a test environment as the process is well tried and should be straightforward.

For riskier upgrades, a test environment is useful because it will enable you to spot any bugs or problems that could scupper the actual migration. Upgrades of complex systems always carry risk.

Even after a trial upgrade in a test environment, the process may go wrong in live and you would have to roll back. You will have lost time and money but avoided loss of data and unplanned downtime.

If a lengthy outage is unavoidable, or can in fact be tolerated, then a period during which users have read-only access to the data is an option, being in most cases better than no access at all.

Ensuring your users are fully briefed on the period in which the read-only status applies is important, to avoid surprising them with a system that won’t take updates, or worse, allowing them to make updates that will be lost in the new system because you are migrating from an earlier copy.

During the process

Any data migration problems that occur are often to do with the software you’re migrating the data to. If you’re performing an upgrade, the new applications may well use the database differently; many upgrades involve changes to database structure, for example.

Vendor-supplied upgrade programs contain series of instructions to upgrade their database for the new version of the application. If it’s a common, well-documented migration then it’ll likely run smoothly.

If it’s not something that’s been completed too many times before, then building a test environment and experimenting with a copy of the existing data (as discussed), is likely to be a good idea.

Final steps

Consider that a data migration requires trained personnel at the other end to set up, maintain and manage it once complete – if they don’t have any experience with the new database or the software it serves, then they’ll need the right level of training to ensure long-term migration success.

Important data migrations will often be assigned ‘early life support’ which will see dedicated teams fix any problems after go live.

Things like memory leaks can occur in the first few days following a migration. Your support for this initial period will be based on the importance of the data and the applications you’re working with – if they’re business critical then more needs to be invested.

Ultimately the most important consideration during any data migration is taking the right amount of risk. Migration risks are essentially losing data, and extended, unplanned outages.

Careful planning mitigates risk. But don’t forget to balance risk with time and expense – if it’s business critical data then the process has to be very involved and risk levels need to be low.Vice versa, if it’s less important

Vice versa, if it’s less important then expenditure to support the migration will likely be lower. The lengths you go to, and the money you spend to avoid risk, must be based on how important the data and the system it resides in are to the business.

Francis Miers, director at Automation Consultants.