HMS Storage Options
General Information:
- Research involves increasingly complex arrangements for the storage and dissemination of research data. Proper storage maintenance throughout the research data lifecycle is imperative, to ensure data remains secure and adheres to recommended safety protocols.
- In recent years, HMS has undertaken a broad effort in research data management offerings. Establishing, reviewing, and following data storage plans for research data can substantially reduce the amount of storage a lab requires – for example, by building workflows that remove excess intermediate files and organizing storage to avoid redundancy.
- The HMS Research Data Management team works directly with researchers to aid in the management of their data throughout the various stages of the data lifecycle. Early and attentive management at each step of the data lifecycle will ensure the discoverability and longevity of your research.
- HMS offers several storage options that allow users to store data in different places, each with distinct behaviors, performance, and means of access.
- Institutions across the Longwood Medical Area maintain various storage solutions, so it is important to compare and select the best storage offering for your intended research. The storage options listed here center on HMS resources, so eligibility requirements may apply.
- All of the tools and dashboards maintained by HMS IT display storage amounts in tebibytes (TiB). Standard convention is to refer to the storage amount as terabytes (TB). While the storage amounts have not changed, HMS IT will be modifying its language to more accurately reflect the true storage amounts being showcased by our tools.
- Snapshots: Deleted data can be recovered from snapshots within 60 days of the last weekly snapshot taken prior to the deletion. Snapshots are taken on a weekly basis, along with daily snapshots retained for 14 days.
- To learn more about best practices and support services for research data lifecycles, please visit the Harvard Biomedical Data Management website or contact the HMS Research Data Manager at rdmhelp@hms.harvard.edu.
-
Active: Compute
Shared high-performance computing environment, including dedicated hardware available for high-memory and GPU-intensive tasks. Intended for active research data that is frequently accessed, modified, or computed againstOverview:
The O2 cluster is a shared high-performance computing environment serving a large research community with diverse research requirements and workflows, including dedicated hardware available for high-memory and GPU-intensive tasks. Active Compute comprises shared group or project folders accessible when logged into the O2 cluster or the transfer cluster. It is intended for active research data that is frequently accessed, modified, or computed against.
Access:
How to Login to O2 from the O2 Cluster or Transfer Cluster
Request Storage:
We are currently accepting storage requests from groups with a primary appointment with an HMS Quad-based pre-clinical department. Allocation amounts dependent on lab needs and available resources. If not eligible, please contact HMS Research Computing to discuss further options.
To request storage, please complete the Active: Compute Storage Request Form.
Contact:
Contact Research Computing Consultants: rchelp@hms.harvard.edu
Resources:
-
Active: Collaborations
Share files and documents with colleagues within and outside of your department. Intended for active research data that is frequently accessed or modified.Overview:
Active Collaborations are shared group or project drives, designed for active research data that is frequently accessed or modified. Active Collaborations allows individuals to share documents and files with colleagues, both from within and outside of their department.
How to Access
- Active Collaborations are shared group or project drives, generally accessed from desktops and laptops. Also available on the O2 Cluster as '/n/files.'
Request Storage
To request storage, please complete the Active: Collaborations Storage Request Form.
Contact
Contact Research Data Management: rdmhelp@hms.harvard.edu
Resources:
-
Scratch
Scratch space available on O2 that should be utilized for transient files during a single job on the HPC Cluster. Intended to be used as a temporary storage space for data that can be easily regenerated.Overview:
A scratch space available on O2, mounted at /n/scratch3. Intended to be used as temporary storage (days to weeks) for data that can be easily regenerated. Should be utilized for transient files during a single job on the HMS High-Performance Compute (HPC) Cluster.
NOTE: Non-temporary data should be stored elsewhere, as an automatic deletion process will take place, periodically removing files.
Access
To Create a User Scratch Directory
Contact:
Contact Research Computing Consultants: rchelp@hms.harvard.edu
Resources:
-
Standby
Should be leveraged for infrequently accessed data. Can also act as an intermediary location, a space to organize and prepare research data for long-term retention.Overview:
Standby is a new storage offering, built as an alternative to Tier 2. Standby should be leveraged for infrequently accessed data that is still directly available for reference, retrieval or analysis. Standby can act as an intermediary location; a space to organize and prepare research data for long-term retention.
Standby can also be used to store large data sets, transferring portions of the data back and forth to the Active space for analysis. Data stored in Standby does not need to be recalled or downloaded for access, the data is available immediately.
How to Access
Standby is accessible via desktop or transfer nodes on O2. To access /n/standby on the O2 cluster, you must login to one of the transfer servers at transfer.rc.hms.harvard.edu; /n/standby is not available from O2 login or compute nodes.
Standby storage provides a similar degree of data protection as Active, including snapshots and off-site backup.
Request Storage:
Allocation amounts dependent on lab needs and available resources.
To request storage, please complete the Standby Storage Request Form.
Contact:
Contact Research Data Management: rdmhelp@hms.harvard.edu
Resources
For additional information, please visit the Standby storage webpage.
-
Cold
Future Storage Offering: Will be used to store rarely accessed data requiring long-term retention, for regulatory or historical purposes.Overview:
Cold Storage will be a future storage option, intended to store rarely accessed data requiring long-term retention, for regulatory or historical purposes. Additional information will be made available as the storage offering evolves. We recommend that labs continue to identify data to move to Cold Storage, to enable an easier transition once the storage offering is finalized. Data identified for Cold Storage will be moved to Standby in the interim.