/tag/storage

  • How To Tips for Storage

    Map your Value or Project Storage to your Desktop Use Globus from the Command Line Work with files in Amazon S3
  • Globus Command-Line Interface

    Install the Globus CLI The Globus CLI is available through the pip installer: If you have administrator privileges type pip install globus-cli Otherwise use pip install –user globus-cli The project is open source, so you can also download the source at https://github.com/globus/globus-cli If you would like to use the CLI from Rivanna, please follow these directions below. Authenticate using the Globus CLI Log in against your institutional authentication provider. In the case of UVA, we use NetBadge for signing in: globus login This will open a page in your web browser where you select your institution and proceed to login:
  • Transfer Files Using Amazon S3

    Setup You will need to install and configure the awscli package in order to access objects in S3. Install the AWS CLI The AWS CLI is available through the pip installer: If you have administrator privileges type pip install awscli Otherwise use pip install –user awscli The project is open source, so you can also download the source at https://github.com/aws/aws-cli Rivanna Users - Install the awscli to your user directory: pip install –user awscli Authenticate the CLI to Amazon Web Services Once the aws package is installed, you will need to configure it: aws configure You will be prompted to enter four values:
  • Drive Mapping on Windows or Mac OSX

    Value and Project storage can be mapped to your Windows or Mac computer as a remote drive. If you are off Grounds you must be running a VPN, such as the UVA Anywhere or the More Secure VPN from ITS. We recommend the More Secure VPN if that is available to you. Windows Open a File Explorer page. In Windows 10 first select This PC. This is not necessary for earlier versions of Windows. When you see the Computer tab, select that, then select Map Network Drive. When the dialog box appears, the default drive letter will be Z. You can change this to any other available letter if you wish.
  • Image Data Management with OMERO

    OMERO is a database for management of imaging data. UVA is is hosting a centralized OMERO database instance backed by centralized storage that facilitates sharing, processing and annotating images for your research group and invited collaborators. Overview With the advent of high-throughput screening, the need for efficient image management tools is greater than ever. From the microscope to publication, OMERO is a database solution that handles all your images in a secure central repository. You can view, organize, analyze and share your data from anywhere you have internet access. Work with your images from a desktop app (Windows, Mac or Linux), on UVA’s high performance computing platform (Rivanna), from the web, or through 3rd party software like Fiji and ImageJ, Python, and MATLAB.
  • Frequently Asked Questions

    General Do you have a general computing question? Read our FAQ› Rivanna High Performance Computing Platform Read our FAQ › Ivy Secure Data Computing Platform Read our FAQ › Storage Research Data Storage & Transfer Read our FAQ ›
  • How To

    General General Tips and Tricks General HowTos › Rivanna High Performance Computing Platform Rivanna HowTos › Ivy Secure Data Computing Platform Ivy HowTos › Storage Research Data Storage & Transfer Storage HowTos ›
  • Storage FAQs

    Accidental File Deletions Why Lease Storage Leased Storage Options File Transfer with Globus Help! I deleted my files accidentally! What can I do? Your home directory contains a hidden directory .snapshot which is not visible even with ls -a. Change into that directory to find timestamped versions of your home directory. Snapshots are created once per day. Project (leased) storage also has a .snapshot directory. Neither Value (leased) nor Rivanna scratch storage is backed up in any way. Why should I lease storage? Leasing storage from Research Computing means that you do not have to run your own data server or backup system.
  • Research Data Storage

    There are a variety of options for storing large-scale research data at UVA. Public and moderately sensitive data storage systems can be accessed from the Rivanna high performance computing system. Highly sensitive data can be stored and accessed within the Ivy secure computing environment. Information Security at UVA provides an overview of the data sensitivity classifications. UVA graduate and undergraduate students are not permitted to request storage accounts. This must be done by their faculty advisor[s]. Information Technology Services (ITS) also provides multiple tiers of data storage for personal and non-research storage needs. .tg {border-collapse:collapse;border-spacing:0;border-color:#ccc;} .tg td{font-family:Arial, sans-serif;font-size:14px;padding:10px 5px;border-style:solid;border-width:0px;overflow:hidden;word-break:normal;border-color:#ccc;color:#333;background-color:#fff;} .
  • Globus Data Transfer

    Globus Data Transfer Globus is a simple, reliable, and fast way to access and move your research data between systems. Globus allows you to transfer data to and from systems such as: Laptops HPC clusters (Rivanna) Lab / departmental storage Tape archives Cloud storage Off-campus resources (XSEDE, National Labs) Globus can help you share research data with colleagues and co-investigators, or to move data back and forth between a lab workstation and Rivanna or your personal computer. Are your data stored at a different institution? At a supercomputing facility? All you need is your campus login. Getting Started A Globus “collection” (also called an “endpoint”) is any computer running the Globus Connect software.
  • Research Value Storage

    Overview Research Computing offers several budget options for storing public and moderately sensitive research data. Information Security at UVA provides details about data sensitivity classifications. The Research Value Storage provides users with a solution for research data storage and collaboration. Members in the same group have access to a shared directory created by the team lead or PI. Group membership can be defined and managed through ITS MyGroups system. Value storage is mounted on the Rivanna HPC cluster and can also be accessed on a personal computer with an SMB mount, allowing for point-and-click file manipulation. If you are not a researcher, UVA ITS offers Academic Value storage for long-term storage of large scale data.
  • Public and Moderately Sensitive Data Storage

    /home /home is a free 50GB space provided to users of the Rivanna HPC system and is visible from the Rivanna login and compute nodes. /home is the default working directory when logging on to Rivanna. Users can also access their home directory at /home/$USER, where $USER is an individual’s UVa computing ID. /scratch /scratch is a Lustre high performance parallel filesystem accessible via the Rivanna login and compute nodes. All Rivanna HPC users are granted 10TB for free and can access this space within Rivanna at /scratch/$USER, where $USER is an individual’s UVa computing ID. The /scratch directory is not intended for long-term data storage.
  • Storage Request

    function setCookie(key, value, expiry) { var expires = new Date(); expires.setTime(expires.getTime() + (expiry * 60 * 60 * 1000)); document.cookie = key + ‘=’ + value + ‘;expires=’ + expires.toUTCString() + ‘;path=/'; } function getCookie(key) { var keyValue = document.cookie.match('(^|;) ?’ + key + ‘=([^;]*)(;|$)'); return keyValue ? keyValue[2] : null; } function gen_random(length) { var text = “"; var possible = “ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789”; for (var i = 0; i Name * E-mail * User ID * Classification * - Select -FacultyStaffPostdoctoral AssociateOther Affiliation * - Select - College of Arts & Sciences School of Data Science School of Engineering and Applied Sciences School of Medicine Darden School of Business UVA Health System Other Type of Request *   Create new storage share   Increase size of existing share   Decrease size of existing share   Retire existing share Space (TB) * The size of storage to be created/retired, or the amount of the increase/decrease to your storage.
  • Rivanna Storage

    There are a variety of options for storing large-scale research data at UVa. Public and moderately sensitive data storage systems can be accessed from the Rivanna high performance computing system. .tg {border-collapse:collapse;border-spacing:0;border-color:#ccc;} .tg td{font-family:Arial, sans-serif;font-size:14px;padding:10px 5px;border-style:solid;border-width:0px;overflow:hidden;word-break:normal;border-color:#ccc;color:#333;background-color:#fff;} .tg th{font-family:Arial, sans-serif;font-size:14px;font-weight:normal;padding:10px 5px;border-style:solid;border-width:0px;overflow:hidden;word-break:normal;border-color:#ccc;color:#333;background-color:#f0f0f0;} .tg .tg-hy9w{background-color:#eceeef;border-color:inherit;vertical-align:top} .tg .tg-dc35{background-color:#f9f9f9;border-color:inherit;vertical-align:top} .tg .tg-0qmj{font-weight:bold;background-color:#eceeef;border-color:inherit;vertical-align:top} Storage Directories Name Quota Price Data Protection Accessible from Best Practices /home 50GB Free 2-week snapshots Rivanna /home is best used as a working directory when using Rivanna interactively. SLURM jobs run against /home will be slower than those run against /scratch. The /home directory is a personal storage space that is not shareable with other users.
  • Rivanna

    Rivanna is the University of Virginia’s High-Performance Computing (HPC) system. As a centralized resource it has hundreds of pre-installed software packages available for computational research across many disciplines. Currently the Rivanna supercomputer has over 8,000 cores and 8PB of various storage. All UVA faculty, staff, and postdoctoral associates are eligible to use Rivanna, or students when part of faculty research. The sections below contain important information for new and existing Rivanna users. Please read each carefully. New users are invited to attend one of our free orientation sessions (“Introduction to the HPC System”) held throughout the year during office hours or by appointment.
  • Pricing

    Below is a schedule of prices for Research Computing resources. Rivanna Allocations Type SU Limits Cost SU Lifetime Standard 100,000 per application; renewable (400K SUs max per fiscal year) Free 12 months Deans’ Allocations None Free 12 months by default, negotiable Purchased None $0.015 (<1M SUs); $0.01 (=1M SUs) Forever Instructional 25,000 Free 2 weeks after last teaching session ** Non-UVA personnel are charged at a rate of $0.07/SU About Allocations Storage Name Security Cost Project Standard $60 TB/year Value Standard $45 TB/year ZFS Standard $30 TB/year Ivy Central Storage High $45 TB/year Ivy NAS Storage High $60 TB/year Storage Details Request Storage
  • SLURM Job Manager

    Overview Rivanna is a multi-user, managed environment. It is divided into frontends, which are directly accessible by users, and compute nodes, which must be accessed through the resource manager. We use the Simple Linux Utility for Resource Management (SLURM), an open-source tool that performs cluster management and job scheduling for Linux clusters. Jobs are submitted to the resource manager, which queues them until the system is ready to run them. SLURM selects which jobs to run, when to run them, and how to place them on the compute node, according to a predetermined site policy meant to balance competing user needs and to maximize efficient use of cluster resources.
  • Sensitive Data Storage - Ivy

    Overview The Ivy secure computing environment meets both HIPAA- and CUI-compliance standards and is ideal for storing highly sensitive research data. Ivy offers several storage options to fit your research computing needs. Ivy Central Storage Ivy Central Storage (ICS) is a highly sensitive data parking zone and central storage pool with a capacity greater than 1PB. This storage space is available for researchers with highly sensitive data and can be mounted on an Ivy virtual machine (VM). For added security, files stored on ICS are read & write only. Executable files can be moved from ICS to VM storage. Researchers can request space on ICS by first requesting an Ivy account using the Ivy request form.
  • Cloud Storage Solutions

    Amazon Web Services Tiered object storage Amazon S3 and Glacier offer cloud-based, affordable, unlimited capacity for storage from anywhere. Advanced features include scalability, lifecycle management, encryption, and sharing. S3 is ideal for static files that need to be retrieved from any location (PDFs, images, video, etc.). Glacier is archival storage, perfect for grant compliance that reqires data retention. How RC can help: Lower pricing - UVA has an Internet2 discount available for educational use. Contact us to create an account for you or your research project. Cost estimates - Cloud storage is not free. Consideration should be made to the size of your files and how often they will be retrieved.
  • Commercial Data Sharing and Archiving Solutions

    Box® Non-sensitive cloud storage UVA Box is a cloud-based storage and collaboration service that gives eligible members of the University community the ability to access, store, and share up to 1 TB of non-sensitive/moderately sensitive University files securely—anywhere, anytime, on any device. Read more DropBox®/Sookasa® Highly Sensitive Data (PHI/PII) storage If you plan on storing highly sensitive data such as PHI or PII, UVA Health System offers a secure encrypted storage for Health System affiliated researchers, students, and staff. “DropBox Sookasa” is a free cloud-based service hosted on Dropbox that can be accessed over the internet on any device.
  • User Guides

    High Performance Computing Standard and high security HPC to run your code, generally written in R, Python or shell scripts. Get Started › Secure Computing Secure virtual machines and interactive notebooks for processing HIPAA and other highly sensitive data. Get Started › Storage Need large, or extremely large storage offsite or on grounds? Can you count in GB, TB, or PB? Learn more about storage options and pricing. Get Started › Cloud Have an idea you’d like to test? Need an environment provisioned in short-order? We can help you build in the AWS cloud.
  • Mission

    Research Computing is a support team at the University of Virginia whose mission is to empower researchers to achieve more through the use of cutting-edge computational resources. We strive to create innovative solutions for researchers who need help solving complex optimization, parallelization, workflow, and data analysis issues. Our strategy is to build and maintain the University’s best computing platforms while educating the next generation of researchers on the power and utility of advanced computing.