File storage data: how to make migration processes run smoothly and without risk
Even for those who've been there before, it's always the same. Migrating files between storages and NAS is always a treacherous road to travel even for the most experienced storage managers.
The operation is not without risk! The data which is migrated is the lifeblood of all organizations, the resource without which everything would collapse. And exploding data volumes simply make life more complex. More and more these organizations find themselves with several petabytes of unstructured data. Simply managing this data growth is a challenge in itself.
Some IT teams understandably postpone the file migration process. New storage servers remain in their boxes and existing servers remain unconsolidated. Meanwhile, file volumes continue their unbounded growth. Yet solutions do exist for each of these risks.
***
Risk #1 - Unmanageable projects
Starting a file migration project between two storages sometimes seems like a leap into the unknown. IT teams may well understand the technical aspects but the issue lies with involving the end users.
Identifying needs, understanding the quality and norms behind the actual data... A storage or NAS migration project may not be the ideal moment to take stock of best IT practices. But the experts who launch the project without involving their end users face these potential pitfalls:
- the migration project being put on hold;
- reproducing sub-optimal architectures in the new storage environment;
- over-duplication of files - or worse, generating false duplications which means actual data can be lost;
- a time-consuming trial and error approach;
- and, as a result, a situation which can be disappointing and not in line with user expectations even if no data is lost.
Instead of gaining time, the workloads are more complex with few or no advantages.
Solution: Get Professional help
Departmental managers understand their own requirements and how to use their professional applications. They are not data experts; and neither they, nor the storage manager, are professionals in change management. This is why Atempo accompanies storage and NAS migrations using the Miria platform and takes control of:
- the audit and definition of the entire technical perimeter;
- the preparation of the migration alongside non-IT teams (end users) and storage administrators;
- process monitoring;
- the final cutover for all migrated files to the new storage infrastructure.
Involving the end user is a way of guaranteeing long-term storage quality - one of the key reasons the storage needed to be upgraded/changed in the first place.
***
Risk #2- Losing data integrity
The nightmare scenario of file migration. Blocked on the source storage, some data gets lost in transit. Other data is impossible or very difficult to locate on the new storage. 75% of companies consulted recently by Kroll Ontrack admit to losing some data during a migration and 23 % never got this data back at all.
Solution: technologY that rises to the occasion!
Performance levels attained by the migration tool make all the difference. The volume of files to migrate rapidly saturates the capacities of off-the-shelf solutions that are less-well adapted to high volumes of data and very high numbers of files. Commonly-used tools such as rsync or robocopy hit an operational ceiling.
Atempo's Miria for Migration solution uses highly parallelized data movement technology. The actual data movement is managed independently from object detection and is launched well before the initial file tree scan is completed. At each step of the way, detailed reports show the migration convergence point approaching and list any data integrity issues that may arise. Thanks to this sophisticated process of movement and management, Miria for Migration handles -and in complete security- data volumes from 100 TB to 100 PB.
***
Risk #3 - Migrations that take too long to complete
The logistics of migrating petascale levels of data means that the whole process can take several weeks… But experienced storage managers will know that the longer the migration takes, the higher the element of risk and the higher the overall cost impact. A poorly planned and executed process can take up to 6 months!
Stretching deadlines further only leads to increased budgets, increased risks to data access and integrity and carries serious issues for end user applications that rely on the file storage systems.
Solution: rapiditY and flexibility are crucial
The success of a Storage/NAS migration is first and foremost built on its rapidity. The Miria for Migration software and associated services brings you the capacity to:
- limit the impact of file migrations on the source storage. This enables the migration process to cohabit alongside user applications and day-to-day storage modifications, deletions and additions;
- move data very quickly – you can, for example, add extra Data Movers, or even decide to saturate a 10 gbps network and use all available storage I/Os to move as many files as possible in the shortest possible time;
- restrict migration to certain times of the day and reduce the impact on the network. Your migrations will no longer monopolize your overworked I/Os and bandwidth for several weeks!
But as we know technology never completely avoids the unexpected. In the event of a data transfer slowdown or bottleneck or error, Miria for Migration remains flexible and adapts to the migration target: parallel data streams, multi-threaded processes and intelligent data flow management…
***
Risk #4 - Incompatible storage platforms
This is one of the chief difficulties when migration between storages and NAS. There is often no easy solution. Switching to a new storage platform with a different protocol and/or different hardware can give rise to genuine compatibility and file formatting issues.
Organizations find themselves suffering from vendor lock-in because changing platforms is just too complex.
One choice can be to limit growth capacity or dive head first into a risk-heavy migration with the possibility of losing data.
Solution: OPT for a vendor-agnostic migration solution
- Atempo does not believe in captive markets. For data to flow freely to its new storage location, Miria for Migration transcribes files to the format which is compatible with the target systems and protocols. This transcription fully preserves the ACLs and metadata for each file.
- Dell / EMC Isilon / ECS, Qumulo, Lustre, DDN GRIDScaler, IBM Spectrum Storage, Panasas, and also multi-node NAS, shared file system and Windows and Linux file systems…
***
Risk #5 - Production downtime
Even if all processes are perfectly calibrated, file storage migration remains intrusive. Data production is impacted across the entire organization with the associated costs of a production shutdown/slowdown.
Solution: PROCEED WITH ITERATIVE MIGRATION CYCLES
The need to suspend production is not a fatality. Miria for Migration maintains constant access to the source storage during the entire operation thanks to incremental synchronizations.
Each cycle:
- throttles network bandwidth in such a way that the migration process can give full priority to end user activity.
- starts automatically in ever-shorter intervals. At each cycle, the tool detects differences without a complete tree scan and performs an incremental synchronization of all modified and new objects until the convergence point is achieved between the source and target storages.
At this stage, we can finalize the cutover event to the new storage after a very brief and scheduled source storage production stoppage and network switch. These iterative cycles ensure the files are correctly and securely migrated with zero impact on the end users and business or organizational processes.
***
For more information about Miria - attend an upcoming webinar:
***
See also:
