Research data storage

* Allocation amounts dependent on lab needs and available resources.

** None refers to HMS professors at affiliate hospitals without an appointment in an HMS Basic Science department. 

Additional information about quad status is available in the External Use of HMS High-Performance Computing (HPC) policy. 

Each storage offering integrates robust security measures to protect and preserve research data. Deleted data can be recovered from snapshots within 60 days of the last weekly snapshot taken before the deletion. Snapshots are taken weekly, along with daily snapshots retained for 14 days. 

• All storage offerings displayed support up to Harvard Security Level 3.  
• An active HMS account ID is required to access HMS Storage offerings.  

*Labs can inquire with Research Computing to discuss collaborative options (ex. a group storage space). 

The O2 cluster is a shared high-performance computing environment with dedicated hardware available for high-memory and GPU-intensive tasks. Active Compute comprises shared group or project folders accessible when signed in to the O2 cluster or the transfer cluster. It is intended for active research data that is frequently accessed, modified, or computed against.

• Access: Login to the O2 or transfer cluster
• Eligibility: The O2 cluster is available to researchers at HMS and its affiliate hospitals. An HMS ID is required. Use of O2 CPUs and GPUs is free for labs in HMS Basic Science departments, and fee-based for affiliates. If ineligible, please contact HMS Research Computing to discuss further options.
• Notifications: When a folder grows above the pre-established storage limit, automatic email notifications are sent to the responsible party (often the PI) and the Lab Data Manager (if the PI has designated one), so they can work with lab members to manage the lab's storage usage.

Contact

• To request storage, complete the Active Compute Storage request form. Allocation amounts are dependent on lab needs and available resources.
• Contact Research Computing Consultants: rchelp@hms.harvard.edu

Resources

• New user guide to O2 and Slurm
• O2 documentation
• Two-factor authentication on O2
• Checking user and group quota usage

Scratch space is available on O2, mounted at /n/scratch/ It is intended to be used as temporary storage (days to weeks) for data that can be easily regenerated. Should be utilized for temporary files during a single job on the HMS High-Performance Compute (HPC) Cluster.

Note – Non-temporary data should be stored elsewhere, as an automatic deletion process will take place, periodically removing files.

• Access: Create a user scratch directory
• Eligibility: An O2 account must use the scratch storage space—additional information is provided in the resources below. 25TiB is provided per user. 

Contact

Contact Research Computing Consultants: rchelp@hms.harvard.edu.

Resources

• Additional information about scratch storage is available on the Research Computing Confluence web pages. 

Active Collaborations are shared group or project folders designed for active research data that is frequently accessed or modified. Active Collaborations allows individuals to share documents and files with colleagues, both within and outside their department.

• Access – Shared group or project drives accessed from desktops and laptops. Also available on the O2 cluster as /n/files.
• Eligibility – Requires at least two co-owners who can add, edit, and remove files and grant additional user access. If you seek access to an existing collaboration, you need to have the manager or owner of the collaboration grant access.
• Notifications – When a folder grows above the pre-established limit, automatic email notifications are sent to the responsible party (often the principle investigator) and the Lab Data Manager (if the principle investigator has designated one) so they can work with lab members to manage the lab's storage usage. Storage limit notifications have been applied to all Active Collaboration folders over 1TiB in size. 

Contact

• To request storage, complete the Active Collaborations storage request form. Allocation amount depended on lab needs and available resources.
• Contact Research Data Management: rdmhelp@hms.harvard.edu

Resources

• Connect to your home and department shared folders on Windows
• Connect to your home and department shared folders from a Mac

We recommend Standby storage for infrequently accessed data that is still directly available for reference, retrieval, or analysis. It can act as an intermediary location, a space to organize and prepare research data for long-term retention. Data stored in Standby does not need to be recalled or downloaded for access; it is available immediately.

Standby is accessible via desktop or the transfer cluster on O2 and provides a similar degree of data protection as Active storage, including snapshots and off-site backup. Please note the temporary modification to the Standby storage snapshot retention schedule summarized in the “Upcoming data center relocation” news article.

• Access – Standby Storage can be provisioned in both NFS and SMB. Windows and macOS clients typically access shared storage over SMB, whereas Linux hosts access Standby Storage via NFS. HMS Storage will provide two subfolders within the lab folder called collaborations and compute, which are designed to mirror permissions set on Active storage. The compute subfolder will store data generated from computations on O2, and the collaborations subfolder will store data originating from collaboration folders (research.files). Ensure data are moved into the correct subfolder, depending on the source location; this is necessary to maintain correct permissions in Standby.
• Eligibility – HMS, HSDM, and HSPH Quad-based faculty and staff are currently eligible to use Standby storage. Other HMS affiliates may also qualify for access. Email rdmhelp@hms.harvard.edu to discuss available storage options.
• Notifications – When a folder grows above the pre-established limit, automatic email notifications are sent to the responsible party (often the principal investigator) and the Lab Data Manager (if the principal investigator has designated one) so they can work with lab members to manage the lab's storage usage.
• Storage limits – Standby storage enforces filesystem quotas to optimize storage capacity. Exceeding these limits triggers email notifications to the responsible party and the Lab Data Manager. It's important to note that storage limits are flexible and can be adapted to meet changing research needs.

Learn more about how to access and transfer data across HMS filesystems.

Contact

To request storage, complete the Standby Storage Request Form. Allocation amounts are dependent on lab needs and available resources.

If you have any questions, email Research Data Management at rdmhelp@hms.harvard.edu.

Cold storage is a low-cost data storage service intended for long-term storage of inactive research data, such as after project completion, that must be retained to meet data retention requirements. Data identified for Cold storage should not be expected to be accessed or retrieved except in rare and unexpected circumstances. Due to the slow transfer rates and costs of transfer (currently covered by HMS), only the minimum amount of data needed should be retrieved. The current Cold Storage offering does not support any sharing or distribution. If you are interested in data sharing and distribution email, contact rdmhelp@hms.harvard.edu.

 Learn more about the Cold storage offering. 

• Access – Direct access to the files in Cold storage will be limited to HMS IT. Metadata associated with the migrated files will be viewable using your group’s Starfish Zone dashboard and corresponding manifest files. HMS IT will perform the data migrations to Cold storage, as it continues to investigate options for self-service. HMS IT will scan the dataset prior to the data transfer to document the files to be moved. HMS IT will inform the lab if they encounter any unsupported files during the pre-migration screening process
• Eligibility – Cold storage is available to labs whose PIs have a primary or secondary faculty appointment in an HMS Quad department. If ineligible, HMS IT encourages labs to contact their affiliate institutions to discuss other long term storage options. There is currently no cost for eligible labs to utilize Cold Storage. HMS IT is currently assuming all costs associated with the administration of Cold storage (including storage and transfer). 

Examples for when it is appropriate to move data to Cold storage: 

• Project Completion
• Grant or funding agency data retention requirements for project datasets
• Journal stipulations for data retention of published data
• Harvard institutional policy stipulating 7-year retention of “essential research records” 
• Data retained for intellectual property purposes (e.g., patents)
• Inactive data associated with a departed lab member
• Raw data difficult to regenerate, associated with completed projects  

Contact

To request data be moved to or retrieved from Cold Storage please complete the Cold Storage Request Form. You can also contact the Research Data Management team: rdmhelp@hms.harvard.edu