Efficient and reliable data transfer is a critical component of research computing. A variety of useful tools is available for rapid data transfer, whether you are transferring data from an external site or within different computing environments at UVA.
Common Scenarios
- Transfer public or not-highly sensitive data between local workstation/laptop and UVA storage
- Transfer highly-sensitive data to Ivy storage
- Transfer data between external institutions/supercomputing centers and UVA
- Transfer between Rivanna and Cloud storage
The data transfer method you choose heavily relies on the data sensitivity classification, where the data are currently located and to where you want to transfer the data. Click on a row in the table below to learn more about the data transfer methods available for a specific scenario.
| System 1 | System 2 | Example Scenario |
|---|
Lab Workstation |
Storage for Public and Moderatively Sensitive Data | - Copy data from a lab workstation to a
/project storage share.
- Copy result files from a
/scratch directory on Rivanna.
Expand |
Local Computer | Remote System | Data Transfer Methods |
| System 1 | System 2 | Example Scenario |
|---|
Secure Lab Workstation |
Storage for Highly Sensitive Data | - Transfer HIPAA data from a Health Systems workstation to Ivy Central Storage.
Expand |
Sources - Lab Workstation
- Health Systems Workstation
| Destination | Data Transfer Methods |
| System 1 | System 2 | Example Scenario |
|---|
External Institution |
University of Virginia | - Transfer public or not-highly sensitive data collected at another institution to UVA storage.
- Transfer results from an analysis carried out on a remote supercomputer at a national lab or supercomputing center.
Expand |
Sources Any institution that uses Globus, such as:- Other universities
- Supercomputing facilities
| Destinations | Data Transfer Methods |
| System 1 | System 2 | Example Scenario |
|---|
Rivanna Storage |
Cloud Storage | - Transfer public or not-highly sensitive data from Research Project & Research Standard storage or Rivanna home & scratch directories to AWS cloud storage.
Expand |
Sources Any institution that uses Globus, such as: | Destinations - AWS S3
- AWS S3 Deep Glacier
| Data Transfer Methods |
Data Transfer Methods
Globus Connect
Large-scale research data transfer
Transferring large amounts of research data is fast and simple with Globus Connect. Globus gives researchers unified access to their data through an easy-to-use web interface, and can be used to transfer data between your laptop and storage systems mounted on the Rivanna HPC system. Globus can also be used to transfer data from other universities or supercomputing facilities.
For transferring highly sensitive data such as HIPAA or CUI data to the Ivy secure computing environment, researchers must use the secure Globus data transfer node (DTN).
Graphical SFTP Clients
Secure file transfer protocol (SFTP)
Programs such as MobaXterm, Filezilla, and Cyberduck provide a graphical user interface to transfer data between a local computer and a remote storage location that permits scp or sftp. These applications allow drag-and-drop file manipulation.
Command Line Tools
Transferring Files from a Terminal
Researchers who are comfortable with the command line can use a variety of command line tools to transfer their data between their laptops and storage systems. Programs such as scp, sftp, rsync and aws cli can be used to quickly transfer files.
Local Data Transfer with the Command Line
When using a Linux file system, users can invoke generic Linux commands to manage files and directories (mv, cp, mkdir), manage permissions (chmod) and navigate the file system (cd, ls, pwd). If you or your collaborators are unfamiliar with some of these commands, we encourage you to take time to review some of the material below:
Transfering Data to Cloud Storage
Several command line tools are available to transfer data from your UVA storage locations to the cloud. On Rivanna we provide the rsync and aws cli tools to transfer files files from Research Project, Research Standard and Rivanna home & scratch directories to AWS storage.
Learn more about the AWS CLI tools
For more help, please feel free to contact RC staff to set up a consultation or visit us during office hours.
|
userinfo
data-transfer, globus, sftp
