The 3 challenges to migrating file storages

Demanding users, complex environments, production downtime, data loss risks, exploding budgets… Data migration projects multiply the challenges! To meet these challenges, it is not enough to be a data storage expert. A successful migration strategy needs a powerful migration tool and smoothly organized job scheduling.

If a company is an ecosystem, then data represents its blood flow. Perhaps we shouldn't be surprised that file migrations speed up the storage manager's heart rate.

There are, of course, proven methods to reduce the risk of file migration. But even the most muscular IT systems will face some technical challenges. Under the pressure of several petabytes of data to move, file migration quickly takes on the appearance of an obstacle course!

At Atempo, we love an athletic storage set up. Our file migration coaches ensure the excellent health of all IT environments. Their agility is based on unique solutions to the three major technical challenges of file migration.

***

Challenge n°1 – Completing file tree-walking in record time

Challenge: Reduce the impact of EXCESSIVE tree walking on production during the file migration

Tree walking is an operation that consists of browsing the contents of a data structure, in this case the file system of a storage to collect the list of files and their attributes. This operation is performed by collecting from each inode (folder) the list of files they contain and their attributes. The sum of the list of files obtained is equal to the list of files in the storage.

tree-walk-en Remaining in production during file migration assumes:

- keeping the source storage in production;

- tracking changes in files made by users on the source storage in order to pass them on to the target storage;

- building the list of files to collect by browsing the file system of the storage to migrate.

All without impacting the bandwidth dedicated to production. It can be challenging because the number of files stored slows the tree walking. Our engineers frequently encounter storage for which each tree walking can last several weeks.

Unsurprisingly, tree walking operations lengthen file migration schedules. However, this is a necessary step to the smooth migration progress. So, what solutions for a media major who stores over 500 million files ?

Solution: Miria for Migration FASTSCAN FEATURE

Atempo's Miria for Migration solution separates operations into two distinct streams: tree walking flow, migration flow. The solution allows to adjust the number of streams launched simultaneously in order to find the right balance to migrate files while preserving operational access to the file tree to migrate.

The tree walking operations will use Miria's FastScan feature whenever the source storage allows it, which in many cases saves a lot of time since this part of the operation will now be measured in hours instead of weeks.

The FastScan feature has no limitation in terms of its ability to detect daily changes made to a storage. Some of our customers have less than 5% daily changes, others up to 25%. As soon as the first changes are identified the actual data transfers can begin.

The situation is complicated for older source storage facilities that do not allow to start an intelligent route via the Miria FastScan. If their low speed and the difficulty to quickly list the files will slow down the process, other features take over: intelligent multithreading manages file collection flows and data transfers from source storage to target storage. The aim is parallelize and thread as many tree walking and migration flows as possible to advance on all fronts.

In every case, Miria for Migration optimizes the speed of file migration according to the technological context, and is able to build on the target storage a tree that mirrors the source structure to place several petabytes of data without interrupting production.

***

Challenge n°2 – Juggling ACLs and network protocols

Challenge: Establish an accurate file picture, according to the network protocols

To preserve user habits and respect the software in production which can access the data after the migration, we must understand the access rights to this data in order to map it faithfully.

During the mapping phase of the migration process, the shares of each storage are identified as well as the network protocol they use. In some cases, a "strategic choice" must be made in order to favor one protocol over another.

For example, some NAS will give access to the same storage space from two different network protocols: CIFS and NFS. If the same file is indeed accessible via both paths, its ACLs are not exposed in exactly the same way by these two network protocols. Performing the migration using either protocol will capture a facet of the ACLs associated with that file. We recommend choosing the one that satisfies the users and preserves the most information!

A good migration process must deal with the subject of shares and their network protocols by anticipating the validation of their migration. Understanding how to validate full ACL file matches helps you avoid making the wrong choices and discovering that you are unable to access some files after migrating them.

Solution: Processes that ensure consistency of access after migration

migration-project-configuration

Atempo's Professional Services teams work with tried and tested workflows with the aim of never losing access to files. Workflows involve:

- Listing all shares and file shares,

- Establishing validation criteria specific to each share, defined with the business users of each share upstream of the migration.

This methodical approach makes it possible to check the coherence of the accesses and the completion of the migration at each share level.

***

Challenge n°3 - Switching from source storage to target storage without impacting file production and migration

Most file migration solutions independently maintain the source and target storage throughout the operation. The planned date for the switch between the old and the new storage crystallizes all the stakes.

- For the team that performs file migration, it's the acid test! The pressure doubles with each unforeseen event.

- Business users are concerned about the suspension of their activities in the event of an error.

However, despite the pilot tests, unforeseen events are almost inevitable. The migration progress of remains difficult to predict, as it deals with storage perimeters that are not comparable with each other.

As a result, the mere prospect of having to switch from one to the other is becoming a source of resistance to change.

Solution: ITERATIVE SYNCHRONIZATIONS

A successful cutover is prepared throughout the entire file migration process:

- When mapping, Miria for Migration builds homogeneous business data sections.

- Miria anticipates the verification phase, keeping data from production and migration separate. Data is first transferred to an empty space on the target storage pending a local transfer to an optimal location.

- Finally, file migration progresses in iterative cycles. At each cycle, Miria detects the modified source files and reproduces the changes in the target storage.

The two storages converge in a controlled and fluid way, the actual data to migrate is reduced cycle after cycle and the date of cutover is simply a way to confirm the completion of the file migration.

migration-cycles-en

***

You win!

You have arrived at the end of this journey which has combined best practices and features to help you find your way around the obstacle course that is data migration. Atempo's Miria for Migration is a highly accessible solution with a unique vendor agnostic solution to handle any migration project from 100 TB to many petabytes of file data.

***

Want to learn more about Miria ?

***