Five keys to a smooth data migration

Moving data from on-premises storage to a commercial cloud provider is a monumental task under the best of circumstances. Normally, it takes a year to 18 months, depending on the amount of data and work needed to clean it up and rationalize it. However, organizations do not always have the luxury of time.

Having recently finished leading a team that moved 2.4 petabytes of data scattered among 16 physical locations into the Microsoft Azure cloud on a 90-day deadline, I have some thoughts about how to make the process smoother when an agency has to do with against a ticking clock, I’ll lay those out below.

The customer, a contractor in this case, had largely shunned the cloud because of security requirements. However, the company’s customers were demanding compliance with FedRAMP and other cloud-based standards by the end of the year, prompting the need for a rapid move.  

Data rationalization is the most time-consuming part of most migration projects. With enough time, you can cut the volume of data to be moved by half or more, in many cases. Without adequate time, you may simply have to move the data as is and work on rationalizing later. Ideally, organizations should rationalize data as they accumulate it, but don’t count on that happening.

Expect to come up against difficulties in user education. People who need access to the data have to know where it lives—both virtually (in the cloud) and often, geographically.  Be prepared to mitigate latency concerns and checksum/validation issues, as well as educating the user community as a whole.

After that comes decommissioning the hardware that once held and transported the data. Data migrations are massive undertakings and organizations rarely attempt them unless it is part of an overall digital transformation project. At our client site, we have the data where we want it, but we want to modernize the business processes. We are putting in high performance capabilities to help them take a big first step into the modern IT era.

 Five key steps to a smooth migration

Organizations who have not already moved to the cloud and want to can make the process easier (it’s never easy!) with these five key steps:

1. Inventory and rationalize your data before you begin a migration project. You will weed out a lot of obsolete data (do you really need to move Lotus 1-2-3 files from the 1980s into Azure or AWS?), and make what is left interoperable and consistent. By the time it is done, all data should be, aligned with business functions, and contain identified archival data, with duplicate or obsolete records removed.
2. Partner early with executive leadership, data owners, security and the business segment from your organization. Keeping everyone informed and looped in will ease uncertainty and address concerns. Tip: Use milestone dates to track progress and inform everyone across all stakeholder groups.
3. Prune data that can be archived directly to the “cool” tier first, rather than active data tiers. On a rapid timeline like we had, this might not be possible. When you can, keep the outmoded data out of the production data entirely.
4. Ensure that the on-premises data center has the infrastructure to facilitate a migration. You will need high bandwidth to transfer such enormous volumes to external devices such as Amazon Snowball or Azure Databox. Prior to cutover, data must be synchronized and have ample bandwidth on the production network to support daily operations as well as the sync.
5. Avoid simultaneously conducting multiple migration projects with interdependencies. Trust me on this. Spare yourself the chaos.

Data migrations are never simple and without adequate time they can be the hardest work you will ever do. I certainly do not recommend these five best practices in lieu of a phased strategy; nothing will ever beat a coordinated plan of attack with the luxury of time. But if you find yourself up against a ticking clock, then I can assure you embracing these principles will help you in a sprint and ensure your success with few complications.

For another perspective, read my colleague John Nemoto’s blog about moves to the hybrid cloud. He does a fabulous job breaking down a truly complex process into an easy-to-digest approach, correctly stating that successful cloud migration requires a holistic, end-to-end approach that must incorporate three primary phases: assess, transform and manage.