The Covid-19 pandemic has most businesses thinking critically about how they will be structured in the future and how the economics of the business will change. Some are considering a transition from office spaces to a wholly virtual setup; others are changing the shape of the business and hoping to survive. IT budgets previously earmarked for large infrastructure projects have been reallocated to cover the cost of transitioning the employee base to a productive work from home environment, and uncertainty about revenue forecasts has led to non-essential expenditure being put on hold and every dollar spent is closely scrutinised.
It is still unclear what a post-pandemic business world will look like; however, as companies contemplate moving to a virtual workforce, the question of how to manage data centres will come up. Regardless of whether the company wants a complete overhaul or simply wants to begin the process, hybrid technologies offer the flexibility many companies are looking for. Capable of being deployed on-premises, in a colocation or in a public cloud, hybrid technologies provide assurances that as the business morphs in the years to come, the same technology can continue to support it.
Data migration: Getting started
Migrating a data warehouse across technologies, and from different vendors, has its risks and requires careful planning. It’s never as simple as unloading data from the old system, loading it into the new system, and repointing the data feeds and business intelligence (BI) tools. Instead, a comprehensive migration plan outlining the roadmap for delivering individual applications, agreed upon success criteria, and regular milestones, is a game changer. Additionally, it’s generally best practice to start the migration process with low-hanging fruit – i.e. the simplest applications – so the migration methodology can be validated, and the team can celebrate an early success.
Challenges are often an inevitable part of a process like this, even with the best laid plans. To ensure the most successful migration possible, organisations need to set a few things in motion from the get-go. For example:
- Articulate the benefits of the migration to the user base and get their buy-in.
- Ensure the new system provides, at a minimum, the same performance and functionality as the old system. Ideally, it should out-perform and fill functional gaps relative to the old system.
- Communicate the migration roadmap, with both milestones and success criteria, to the entire team.
- Deliver a phased plan for ‘how’ and ‘when’ individual applications will be moved and show success early to counter any resistance.
- Include the business users in testing the new system and have them sign off on migrated applications.
Once the foundation has been laid, encountering and addressing challenges will be more manageable. Most importantly, know that no matter which roadblocks arise, there is always a solution.
Key challenges to be aware of when migrating data and recommended solutions
- Challenge: The lack of a migration roadmap
- Database migration projects often fail not so much because of the actual migration process, but because companies don’t map out their entire data ecosystem up front. Without this crucial step, enterprises leave themselves blind to interdependencies.
- Solution: Take time to build a roadmap based on the data needs of the business
- Businesses should map out the entire data ecosystem and use that to build a phased migration plan. This includes compiling a complete inventory of all the scripts, ETL jobs, applications, and services that feed the database or touch the data and will have to be repointed to the new system, as well as an agreed upon sequence for moving each item.
- Challenge: Not fully understanding what exactly is being moved, or how to move it
- If a business does not have a thorough understanding of what exactly is required to migrate an application, they may miss business critical pieces. The failure to migrate even a single script to the new system could result in applications not functioning against the new database, rendering the migration project dead on arrival. For example, if there’s a table containing foreign exchange rates and it needs to be refreshed hourly by an external data service, and that service is overlooked in the migration process, the business will be operating against stale, outdated exchange rates. This issue could potentially snowball until the end of the quarter when revenue is being reconciled.
- Solution: Create a phased plan for how to migrate your data, and when
- The importance of mapping the entire data ecosystem cannot be overstated. Understanding the interplay between all of the database objects used by an individual application, together with the interactions and interdependencies to migrate applications holistically, is key.
- Challenge: Trying to migrate everything all at once
- History has shown that attempts at fork-lifting data warehouses are destined to fail more often than not. When everything is moved all at once, the testing phase of a migration plan essentially goes bust. Many businesses therefore find themselves with applications that don’t work correctly, causing disruption to the everyday flow and operation of the business.
- Solution: Implement a strategy to migrate data in small batches vs. all at the same time
- The most successful database migration projects don’t take a big-bang approach, but instead migrate single applications, or logical groups of applications, one at a time. This requires a thorough understanding of the data architecture and the data ecosystem that feeds it, so the database objects and scripts that relate to an application can be migrated and tested.
- Challenge: Further Covid fallout requires additional platform changes
- As the outlook for many companies remains unclear, there is a chance that an additional platform change may be required in the future; many businesses, as they currently stand, have not planned for this.
- Solution: Deploy a hybrid data warehouse platform for maximum flexibility
- Futureproofing a business is essential and hybrid data warehouse solutions can support that. With the capability to be deployed on-premises, in a public or private cloud or across both environments, migrating to a hybrid platform is a one-shot deal. Once the migration is complete, it allows for the flexibility to move workloads to whichever environment makes sense for the business. For instance, workloads with wild peaks and troughs are best suited to the cloud, where the resources can be scaled to meet business needs in an instant; whereas sensitive data that is subject to strict, regulatory compliance requirements, might be best located in an on-premises solution.
As companies are pushed to change their IT operations in response to the global Covid-19 pandemic, data migration will become a more immediate need. At the end of the day, planning and strategy reign supreme, and separate a successful data migration project from an unsuccessful one – and will be what supports a more future-ready and future-proofed enterprise.
Emma McGrattan, SVP of engineering, Actian