HMS IT is upgrading to a new storage system designed to efficiently manage large amounts of data at high speed. The Research Data Migration project aims to streamline the storage architecture to a centralized, stable, and secure infrastructure. As of March 15, 2024, O2 Home folders and scratch have already been migrated to the new storage successfully, with several benefits and efficiencies already achieved:
- Automated deduplication has increased the total available capacity in these directories by 35%.
- Automated data compression functionality has reduced data traffic to the storage system by 60%.
- The new storage system reduced energy usage by 75% compared to energy use on Isilon.
- The storage footprint has been reduced by 66%, leading to significant savings in future data center operational costs.
Migration plan and impact
The project will run through August 2024, and data migration to the new storage system will be completed in phases to minimize disrupting researcher workflows.
Outages of specific filesystems in the existing storage system will be needed to migrate data between the old and the new environment. Outages will occur every six weeks, lasting less than 48 hours. Unless otherwise noted on the project update website, it is expected that O2 will remain fully operational during outage windows, while the filesystem being migrated will be unavailable during the specified outage window. Outages will be announced 60 days in advance via the O2 announce email list.
Review the detailed timeline that follows for additional details about scheduled upgrades and maintenance windows.
All data will be migrated by September 2024, when the HMS community can look forward to a stable, modern, and more performant storage environment.
Outages and impacts
| Start | End | Filesystem | Impact description | |
|---|---|---|---|---|
| February 13 at 5:00 PM | February 15 at 10:00 PM | O2 Home | Inaccessible | |
| February 13 at 5:00 PM | February 15 at 10:00 PM | Slurm software | Job scheduling unavailable | |
| February 13 at 5:00 PM | February 15 at 10:00 PM | O2 Apps | Inaccessible | |
| April 8 at 9:00 AM | April 9 at 10:00 AM | o2groups/htem | Read and write unavailable, viewing not affected | |
| April 13 at 9:00 AM | April 16 at 5:00 PM | /n/data1 | Inaccessible | |
| April 13 at 9:00 AM | April 16 at 5:00 PM | O2 cluster | Certain O2 tools or commands cannot be run | |
| May 3 at 5:00 PM | May 4 at 5:00 PM | CryoEMData | Inaccessible | |
| May 3 at 5:00 PM | May 4 at 5:00 PM | Cellbio_gygi_VM | Inaccessible | |
| May 27 at 5:00 PM | May 28 at 5:00 PM | L4 | Inaccessible | |
| May 29 at 9:00 AM | May 31 at 9:00 AM | /n/data2 | Inaccessible | |
| May 29 at 9:00 AM | May 31 at 9:00 AM | /n/no_backup2 | Inaccessible | |
| May 29 at 9:00 AM | May 31 at 9:00 AM | OMERO | Inaccessible | |
| June 10 | June 12 | /n/data3 | Read and write unavailable, viewing not affected | |
| July 8 at 5:00 PM | July 10 at 5:00 PM | /n/groups | Inaccessible | |
| September 2 at 5:00 PM | September 5 at 10:00 AM | Standby Standard | Inaccessible |
Consultation and support
As usual, throughout the process, support for researchers will be available through rchelp@hms.harvard.edu.