Install the Globus CLI

The Globus CLI is available through the pip installer:

If you have administrator privileges type

  
pip install globus-cli

Otherwise use

  
pip install --user globus-cli

The project is open source, so you can also download the source at https://github.com/globus/globus-cli

If you would like to use the CLI from Rivanna, please follow these directions below.

Authenticate using the Globus CLI

Log in against your institutional authentication provider. In the case of UVA, we use NetBadge for signing in:

  
globus login

This will open a page in your web browser where you select your institution and proceed to login:

Globus CLI login

Select “University of Virginia” and then click Continue. You are then taken to an authorization page where you agree to allow the Globus CLI to access your Globus account. Click on Allow.

Globus CLI authentication

You may now close your browser window. Your command-line tools are now authenticated and ready to use.

Basic Commands

Issue the base command for the tools and you will see the primary set of commands:

  
globus

  
  --jmespath, --jq TEXT     A JMESPath expression to apply to json output.
                            Takes precedence over any specified '--format' and
                            forces the format to be json processed by this
                            expression
  --map-http-status TEXT    Map HTTP statuses to any of these exit codes:
                            0,1,50-99. e.g. "404=50,403=51"

Commands:
  bookmark        Manage endpoint bookmarks
  config          Manage your Globus config file. (Advanced Users)
  delete          Submit a delete task
  endpoint        Manage Globus endpoint definitions
  get-identities  Lookup Globus Auth Identities
  list-commands   List all CLI Commands
  login           Log into Globus to get credentials for the Globus CLI
  logout          Logout of the Globus CLI
  ls              List endpoint directory contents
  mkdir           Make a directory on an endpoint
  rename          Rename a file or directory on an endpoint
  task            Manage asynchronous tasks
  transfer        Submit a transfer task
  update          Update the Globus CLI to its latest version
  version         Show the version and exit
  whoami          Show the currently logged-in identity.

For a full list of commands available:

  
globus list-commands

List/Search Endpoints using the Globus CLI

Find a Globus endpoint. Here is how you might find the UVA Main DTN:

  
globus endpoint search "uva#main"

ID                                   | Owner            | Display Name
------------------------------------ | ---------------- | ------------
c4d80096-7612-11e7-8b5e-22000b9923ef | uva@globusid.org | uva#main-DTN

Or search more broadly for all UVA endpoints in Globus:

  
globus endpoint search "uva"

ID                                   | Owner                | Display Name     
------------------------------------ | -------------------- | -----------------
c4d80096-7612-11e7-8b5e-22000b9923ef | uva@globusid.org     | uva#main-DTN     
67b9cb38-301c-11e7-bcac-22000b9a448b | uva@globusid.org     | uva#portable-DTN 
e1c6b195-6d04-11e5-ba46-22000b92c6ec | uvastro@globusid.org | uvastro#almuhit  
31a68704-2422-11e6-bfeb-22000b1701d1 | uvastro@globusid.org | uvastro#scandium 
7bb92d80-6d04-11e5-ba46-22000b92c6ec | uvacse@globusid.org  | uvacse#fir       
de463ced-6d04-11e5-ba46-22000b92c6ec | uvastro@globusid.org | uvastro#tupungato
de463ce4-6d04-11e5-ba46-22000b92c6ec | uvastro@globusid.org | uvastro#helix    
a9a9ae5d-6d04-11e5-ba46-22000b92c6ec | uvacse@globusid.org  | uvacse#cooper    
df70ec7d-6d04-11e5-ba46-22000b92c6ec | uvastro@globusid.org | uvastro#cavi     
24b0ca0c-3013-11e7-bcab-22000b9a448b | ars9ac@virginia.edu  | UVA Portable DTN 

For transfers and file operations, reference endpoints by their unique ID. Names are only convenient tags to help humans differentiate between endpoints.

Traverse Directory Trees using the Globus CLI

Once you know the ID of a specific endpoint, you can list directories visible to you. Here are some paths open to users of the Rivanna HPC cluster:

  
globus ls c4d80096-7612-11e7-8b5e-22000b9923ef

home/
nv/
scratch/

To drill deeper, append directories to the endpoint ID with a colon :

  
globus ls c4d80096-7612-11e7-8b5e-22000b9923ef:home/mst3k/

directory1/
directory2/
archive1.tar.gz
file1.txt
file2.txt
my-file.txt

If we would like to transfer a file, we will need the full Globus ID and path of the source file:

  
c4d80096-7612-11e7-8b5e-22000b9923ef:home/mst3k/archive1.tar.gz

Transfer Files using the Globus CLI

To transfer a file you will also need the ID and path to a destination directory and new filename – the place to which you would like to copy the remote file from above. It should look something like this:

  
39e0bf8a-3037-11e7-bcae-22000b9a448b:/home/user1/archive1.tar.gz

With the full path to a source file and a full path to a destination, we can now request a transfer. The simplest form of a transfer request looks like this:

  
globus transfer c4d80096-7612-11e7-8b5e-22000b9923ef:home/mst3k/my-file.txt c4d80096-7612-11e7-8b5e-22000b9923ed:nv/vol179/staff/my-new-file.txt

Message: The transfer has been accepted and a task has been created and queued for execution
Task ID: 94d15980-9c94-11e7-acbc-22000a92523b

Note: If you wish to encrypt your transfer, add the –encrypt flag:

  
globus transfer --encrypt c4d80096-7612-11e7-8b5e-22000b9923ef:home/mst3k/my-file.txt c4d80096-7612-11e7-8b5e-22000b9923ed:nv/vol179/staff/my-new-file.txt

Get the Status of a Transfer

Using the Task ID returned from a request, you can get the status of a task:

  
globus task show 94d15980-9c94-11e7-acbc-22000a92523b

Label:                   None
Task ID:                 94d15980-9c94-11e7-acbc-22000a92523b
Is Paused:               False
Type:                    TRANSFER
Directories:             0
Files:                   1
Status:                  SUCCEEDED
Request Time:            2017-09-18 17:12:43+00:00
Faults:                  0
Total Subtasks:          1
Subtasks Succeeded:      1
Subtasks Pending:        0
Subtasks Retrying:       0
Subtasks Failed:         0
Subtasks Canceled:       0
Subtasks Expired:        0
Completion Time:         2017-09-18 17:12:44+00:00
Source Endpoint:         uva#main-DTN
Source Endpoint ID:      c4d80096-7612-11e7-8b5e-22000b9923ef
Destination Endpoint:    uva#main-DTN
Destination Endpoint ID: c4d80096-7612-11e7-8b5e-22000b9923ed
Bytes Transferred:       2812
Bytes Per Second:        2468

Script Transfers Against the Globus CLI

Using the commands above, automated file transfers should not be difficult if run under a user account that has already authenticated. A simple bash script run via cron should work well for automated file shipments. Each shipment will trigger an automatic success/failure email, so there is no need to set up additional notifications.

Single file transfers

Transfer a single file at a time to another DTN, via script:

Folder sync transfers

Synchronize an entire folder and all contents with another DTN, via script:

Run your script:

  
user@host$ ./sync-directories.sh 

Either operation should result in a confirmation message like this:

  
Message: The transfer has been accepted and a task has been created and queued for execution
Task ID: 5ffe3058-5543-11e8-90ce-0a6d4e044368

Automating your scripts

  • In a Unix/Linux/MacOS environment, you can set any script or application to run on any schedule using cron.
  • In Windows, we recommend writing the above into a PowerShell script, which can then be scheduled using the “Task Scheduler” tool from the Windows menu.

More Information

Use the Globus-CLI from your Rivanna Account

  1. Load the globus-cli module and its dependencies:
  
module load gcc openmpi globus_cli

  1. Authorize globus-cli

Run this command:

  
globus login

You will then be given an Oauth2 login URL. Start a Web browser, either through FastX or through an X11 server on your local computer. Copy and paste this URL into the web browser, and authorize your connection as instructed in the topic above.

Be sure to give your authorization a useful name, such as -rivanna, i.e. mst3k-rivanna. This will help you distinguish it in your list of Globus authorizations.

Return to the top of the page for information about using the Globus CLI generally.

When referencing the globus binary in scripts, you may want to issue a which globus command to find its path as that may change over time with new versions. If used with backticks this can be used to populate a variable in a script:

  
globus=`which globus`

Please note that users are not permitted to run cron jobs on Rivanna. Scheduling should be done from another system.