Compatibility issues between storage manufacturers, ageing source storages, large amounts of data to move... When choosing a file migration solution, the list of elements to consider complicates the decision-making process.
The file migration market is in full expansion mode. Often, high project costs and complex implementations mean many offers are primarily aimed at major corporations. Others target SME’s but are limited to small numbers or volumes of files to be transferred.
Atempo's Miria for Migration solution addresses the challenges faced by growing number of organizations as their data volumes explode : handling hundreds of thousands to millions or even billions of files, ranging from hundreds of terabytes to multiple petabytes of data in complex, heterogeneous, multi-vendor IT environments.
We have studied the question at length: to control file migrations, the tool you choose must offer 8 essential features for that single objective: agile file migration which guarantees data integrity and gives priority back to users.
Most file migration solutions first collect the data and, when this step is finally completed, transfer the data. The problem is that on many systems collecting the files to be migrated can take several days or even weeks! During this time, your migration team and your target storage are eagerly awaiting the arrival of first bytes.
With its intelligent multi-threading feature, Miria for Migration makes file migration both smoother and faster. No idle waiting: your files start moving within hours, file data collection and migration are parallelized in a smart way. The solution accesses both storages at the same time, and simultaneously performs source storage analysis and the actual transfer to the target.
The large number of files to be migrated often means complex and extensive file tree structures. The tree walk required the solution to build the list of files to be moved which lengthened migration times. The situation becomes even more complicated when it is necessary to maintain production on the source storage because users modify the tree structure throughout the migration project. All production modifications must be detected and taken into account during the project.
Miria for Migration's FastScan feature quickly detects files to be migrated from a source storage. Each migration cycle uses FastScan to identify all created, modified or deleted files since the last migration cycle. FastScan not only saves time when creating the initial list of files to be migrated, but also the overall project throughout helping to identify the "leftover" files to be migrated. This unique feature has a very strong impact on the speed of the migration process.
The requirement of maintaining production during the migration project puts users and the migration solution in competition with each other. Bandwidth becomes a confrontational issue!
Miria for Migration gives priority to production. Data flows related to migration are limited during predefined time slots in order to guarantee user service levels. Outside of these reserved time slots, Miria will speed up transfers by using a higher bandwidth to reach deadlines.
***
In areas such as Research or IoT, it is common to see exponential growth in the volumes of data managed by organizations of all sizes. Users are faced with the sudden need to migrate millions of files or even petabytes of data to higher capacity storage.
Miria for Migration uses modular data movement to make the transfer of such a large volume of data accessible. At any time, storage managers can add or remove a Miria Data Mover, depending on the power and speed required at each step of the file migration process. Combined with multi-threading and task parallelization, modular data movement provides agile data flow control.
Such flexibility in migrating large volumes of files cannot be managed in a linear way! To maintain control over data flows and progressively consolidate the target architecture, Miria for Migration proceeds by iterative migration cycles. At each step, Miria detects changes made to the source storage and replicates them to the target storage.
Thanks to this synchronization in "successive loops", the source and target storage converge seamlessly towards the final cutover. Thanks to this circular process, the cutover becomes a formality that is limited to synchronizing a delta file in an umpteenth operation of an already repeated and known process: convergence is greatly facilitated and predictable.
Petabyte-scale migration projects often stretch over several weeks. Impossible to record everything by hand, it is imperative for the migration solution to have a log of all transfers.
This is the case with Miria, which makes it possible to find the trace of all transfers in the transfer log both during and well after the file migration. Each version of a file is carefully recorded in the transfer log. If necessary, it is possible to retrieve the individual history of each file.
The switch between source and target storage crystallizes the challenges of file migration. This step often takes place under the pressure of queued production and users eager to regain access to their valuable data. Maximum pressure for storage managers, time is running out, the list of points to validate is infinite. This approach alone is a source of unintentional omissions or control deadlocks.
Miria for Migration demystifies the tipping point, it is no longer a "leap of faith" into the unknown but a continuous, methodical and controlled process: a succession of migrations and control phases. Not only does this approach allow you to reconcile production and migration, but it also allows you to reduce the final cutover to just a final migration cycle to cap off a method that has already been repeated and mastered.
Each cycle is much shorter and simpler than the previous ones. Once the last checks have been validated, the final steps of taking the source storage offline and pointing users to the new storage becomes child’s play.
The best tools are nothing if you hesitate on how to use them. Data management is one job; data transfer is another.
That's why Atempo provides its customers with file migration experts. Working alongside business teams and storage managers, they audit storage and deploy Miria for Migration, optimizing its data movement to best fit the needs of each migration. At the end of the process, our customers' teams are fully trained:
- at each step of the file migration process: preparation, user involvement, validation preparation, division into sub-projects / homogeneous zones;
- and to the handling of the Miria for Migration solution - before, during and after the operation.
Thanks to the support offered by Atempo, file migration is always coupled with a complete skill transfer.
***
Want to learn more about Miria ?
***
Further reading: