Data Transfer

Efficient and reliable data transfer is a critical component of research computing. A variety of useful tools is available for rapid data transfer, whether you are transferring data from an external site or within different computing environments at UVA.

Common Scenarios

  • Transfer public or internal use data between local workstation/laptop and UVA storage
  • Transfer sensitive or highly sensitive data to Ivy storage
  • Transfer data between external institutions/supercomputing centers and UVA
  • Transfer between UVA HPC and Cloud storage

The data transfer method you choose heavily relies on the data sensitivity classification, where the data are currently located and to where you want to transfer the data. Click on a row in the table below to learn more about the data transfer methods available for a specific scenario.

System 1 System 2 Example Scenario

Lab Workstation
Storage for Public and Internal Use Data
  • Copy data from a lab workstation to a /project storage share.

  • Copy result files from a /scratch directory on Rivanna or Afton.
Expand

Local Computer

  • Laptop
  • Lab Workstation

Remote System

Data Transfer Methods
System 1 System 2 Example Scenario

Secure Lab Workstation
Storage for Sensitive and Highly Sensitive Data
  • Transfer HIPAA data from a Health Systems workstation to High-Security Research Standard Storage.

 

Expand

Sources

  • Lab Workstation
  • Health Systems Workstation

Destination

Data Transfer Methods


System 1 System 2 Example Scenario

External Institution
University of Virginia
  • Transfer public or not-highly sensitive data collected at another institution to UVA storage.

  • Transfer results from an analysis carried out on a remote supercomputer at a national lab or supercomputing center.
Expand

Sources

Any institution that uses Globus, such as:
  • Other universities
  • Supercomputing facilities

Destinations

Data Transfer Methods


System 1 System 2 Example Scenario

Rivanna/Afton Storage
Cloud Storage
  • Transfer public or not-highly sensitive data from Research Project & Research Standard storage or Rivanna/Afton home & scratch directories to AWS cloud storage.
Expand

Sources

Any institution that uses Globus, such as:

Destinations

  • AWS S3
  • AWS S3 Deep Glacier

Data Transfer Methods

Data Transfer Methods

Globus Connect

Large-scale research data transfer
Globus

Transferring large amounts of research data is fast and simple with Globus Connect. Globus gives researchers unified access to their data through an easy-to-use web interface, and can be used to transfer data between your laptop and storage systems mounted on the HPC system. Globus can also be used to transfer data from other universities or supercomputing facilities.

 

For transferring highly sensitive data such as HIPAA or CUI data to the Ivy secure computing environment, researchers must use the secure Globus data transfer node (DTN).

Graphical SFTP Clients

Graphical-SFTP
Secure file transfer protocol (SFTP)

Programs such as MobaXterm, Filezilla, and Cyberduck provide a graphical user interface to transfer data between a local computer and a remote storage location that permits scp or sftp. These applications allow drag-and-drop file manipulation.

Command Line Tools

Transferring Files from a Terminal

Researchers who are comfortable with the command line can use a variety of command line tools to transfer their data between their laptops and storage systems. Programs such as scp, sftp, rsync and aws cli can be used to quickly transfer files.

Local Data Transfer with the Command Line

When using a Linux file system, users can invoke generic Linux commands to manage files and directories (mv, cp, mkdir), manage permissions (chmod) and navigate the file system (cd, ls, pwd). If you or your collaborators are unfamiliar with some of these commands, we encourage you to take time to review some of the material below:

Transfering Data to Cloud Storage

Several command line tools are available to transfer data from your UVA storage locations to the cloud. On the HPC system we provide the rsync and aws cli tools to transfer files from Research Project, Research Standard and Rivanna/Afton home & scratch directories to AWS storage.

Learn more about the AWS CLI tools

For more help, please feel free to contact RC staff to set up a consultation or visit us during office hours.

|  userinfo    , ,