/category/howto

  • The Make Tool

    Overview Make is a program used primarily on Unix systems to manage compiling and linking (building) programs written in C, C++, Fortran, or other compiled languages. Make operates on targets using rules to create those targets. It has a set of built-in rules but users may write their own or override the default rules. Make scans the dependencies of each target looking for files newer than the target. If it finds them, it recreates the target. Targets may and usually do depend on other targets; make will work its way through the chain to rebuild the final target, which is typically an executable.
  • Converting a Jupyter Notebook to a PDF

    Users cannot load modules inside the OpenOnDemand App for JupyterLab. Therefore it is not possible to convert a Jupyter Notebook to a PDF directly inside the JupyterLab Interactive App on OpenOnDemand. There are 2 ways to convert a Jupyter Notebook to a PDF: Directly from the command line. ssh from your terminal and type the following: module load anaconda/2020 texlive jupyter nbconvert –to pdf you_script.ipynb If you want to use GUI, please request a desktop session. Fill out the form as you normally would for JupyterLab. After you get to a desktop, open a terminal (black box next to Firefox in the top bar) and type these commands: module load anaconda/2020 texlive jupyter notebook This will pull up JupyterLab.
  • Launching RStudio Server from a Singularity Container

    Rocker provides many software containers for R. Due to the default permission settings of our file system, launching an RStudio Server session is not straightforward. If you are interested in using their containers on Rivanna, please follow these steps. Pull container Use Singularity to pull the container. We will use geospatial in this example. module load singularity singularity pull docker://rocker/geospatial You should see geospatial_latest.sif in your current directory. One-time setup The commands in this section are to be executed as a one-time setup on the frontend. You will need to create a directory where you have write permissions, typically somewhere under $HOME.
  • Migrating Python packages

    Scenario You have installed Python packages locally in one version and now wish to use them in a different version. For example, you have been using Python 3.6 but it is obsolete and will be removed soon, so you need to set up those packages for Python 3.8. There are several ways to accomplish this, depending on the package manager. In this how-to we will discuss pip and conda. You will need to load the module for the newer Python version. For this example, module load anaconda/2020.11-py3.8 $('#copybtn683429571’).click(function(){ var $temp = $(""); $(“body”).append($temp); $temp.val($('#683429571’).text()).select(); document.
  • How To Guides for Rivanna Users

    Guides Building compiled code Using make Building and running MPI Code Bioinformatics on Rivanna Convert Jupyter Notebook to PDF Custom Jupyter kernels Loading Modules in Jupyter Docker images on Rivanna Adding packages to a container Migrate Python packages Launch RStudio Server from a Singularity container More Documentation Connecting Using SSH Using a browser Using FastX Jobs / Slurm / Queues Slurm Overview Queues Storage and File Transfer Storage overview Data transfer methods Allocations Allocations Overview
  • Transfer Files Using Amazon S3

    Setup You will need to install and configure the awscli package in order to access objects in S3. Install the AWS CLI The AWS CLI is available through the pip/pip3 installer: If you have administrator privileges type pip install awscli $('#copybtn576849213’).click(function(){ var $temp = $(""); $(“body”).append($temp); $temp.val($('#576849213’).text()).select(); document.execCommand(“copy”); $temp.remove(); var $this = $(this); $(this).fadeOut(200); $(this).delay(2000); $(this).fadeIn(200); }); Otherwise use pip install –user awscli $('#copybtn357812469’).click(function(){ var $temp = $(""); $(“body”).append($temp); $temp.val($('#357812469’).text()).select(); document.execCommand(“copy”); $temp.remove(); var $this = $(this); $(this).fadeOut(200); $(this).delay(2000); $(this).fadeIn(200); }); The project is open source, so you can also download the source at https://github.
  • Loading Module in Jupyter

    Users cannot load modules inside a JupyterLab session. If you need access to modules, please request a desktop session instead of JupyterLab. Fill out the form as you normally would for JupyterLab. After you get to a desktop, open a terminal (next to Firefox in the top bar) and type these commands: module load gcc jupyter_conda/.2020.11-py3.8 module load … # your modules here jupyter-lab This should start up Firefox shortly. If you accidentally close the window, right click on the link in the terminal and choose “open link” to restart. An example of using LaTeX inside a JupyterLab session is shown in the screenshot below.
  • How To Tips for Storage

    Map your Research Standard or Research Project Storage to your Desktop Use Globus from the Command Line Work with files in Amazon S3
  • Building Your Code on Rivanna

    Building your Application Creating an executable from source with a compiled language requires two steps, compiling and linking. The combination of these is generally called building. The output of the compiler is generally an object file, which on Unix will end in a .o suffix. Object files are machine code and are not human-readable, but they are not standalone and cannot be executed. The linker, which is usually invoked through the compiler, takes all object files, along with any external libraries, and creates the executable (also called a binary). Compilers are invoked on source files with a line such as
  • Sensitive Data Transfer with Globus

    Login In to Globus and Choose Ivy Collection Disconnect your High Security VPN – it will interfere with the high security network that Globus uses. Open https://www.globus.org/ and sign in through the University of Virginia using NetBadge. Once you are on the Transfer Files page, at the first “Collection” type ivy to search. Find the uva#ivy-DTN collection and select it. You will then be asked to authenticate. Once signed in, simply click through the names of your shares until you find the source or destination for your file transfers. Navigate to Your Personal Collection
  • Running a Bioinformatics Software Pipeline with Wdl/Cromwell

    WDL (pronounced widdle) is a workflow description language to define tasks and workflows. WDL aims to describe tasks with abstract commands that have inputs, and once defined, allows you to wire them together to form complex workflows. Learn More CROMWELL is the execution engine (written in Java) that supports running WDL scripts on three types of platforms: local machine (e.g. your laptop), a local cluster/compute farm accessed via a job scheduler (e.g. Slurm, GridEngine) or a cloud platform (e.g. Google Cloud or Amazon AWS). Learn More Introduction Pre-requisites: This tutorial assumes that you have an understanding of the basic structure of a WDL script.
  • Globus Command-Line Interface

    Install the Globus CLI The Globus CLI is available through the pip installer: If you have administrator privileges type pip install globus-cli Otherwise use pip install –user globus-cli The project is open source, so you can also download the source at https://github.com/globus/globus-cli If you would like to use the CLI from Rivanna, please follow these directions below. Authenticate using the Globus CLI Log in against your institutional authentication provider. In the case of UVA, we use NetBadge for signing in: globus login This will open a page in your web browser where you select your institution and proceed to login:
  • Docker - The Basics

    What Is Docker? “Docker is a set of platform-as-a-service (PaaS) products that use OS-level virtualization to deliver software in packages called containers. Containers are isolated from one another and bundle their own software, libraries and configuration files; they can communicate with each other through well-defined channels. All containers are run by a single operating-system kernel and are thus more lightweight than virtual machines. The service has both free and premium tiers. The software that hosts…” Continue reading on Wikipedia Click to watch on YouTube:

    Install Docker Docker is available for Windows, Mac, and Linux. Download the appropriate Docker Edition for your platform directly from Docker.


  • Custom Jupyter Kernels

    You can create custom kernels from an Anaconda environment or a Singularity container. In both cases you’ll need to install the ipykernel package. Jupyter kernel based on Anaconda environment To create a custom kernel of the Anaconda environment myenv that uses Python 3.7: module load anaconda conda create -n myenv python=3.7 ipykernel <other_packages> source activate myenv python -m ipykernel install –user –name myenv –display-name "My Env" Note: You can customize the display name for your kernel. It is shown when you hover over a tile in JupyterLab. If you do not specify a display name, the default Python [conda env:<ENV_NAME>] will be shown.
  • Bioinformatics Resources on Rivanna

    The UVA research community has access to numerous bioinformatics software installed directly or available through the bioconda Python modules. Click here for a comprehensive list of currently-installed bioinformatics software. Popular Bioinformatics Software Below are some popular tools and useful links for their documentation and usage: .tg {border-collapse:collapse;border-spacing:0;border-color:#ccc;} .tg td{font-family:Arial, sans-serif;font-size:14px;padding:10px 5px;border-style:solid;border-width:0px;overflow:hidden;word-break:normal;border-color:#ccc;color:#333;background-color:#fff;} .tg th{font-family:Arial, sans-serif;font-size:14px;font-weight:normal;padding:10px 5px;border-style:solid;border-width:0px;overflow:hidden;word-break:normal;border-color:#ccc;color:#333;background-color:#f0f0f0;} .tg .tg-hy9w{background-color:#eceeef;border-color:inherit;vertical-align:middle;} .tg .tg-dc35{background-color:#f9f9f9;border-color:inherit;vertical-align:middle;} .tg .tg-hy9w-nw{background-color:#eceeef;border-color:inherit;vertical-align:middle;white-space:nowrap;} .tg .tg-dc35-nw{background-color:#f9f9f9;border-color:inherit;vertical-align:middle;white-space:nowrap;} .tg .tg-0qmj{font-weight:bold;background-color:#eceeef;border-color:inherit;vertical-align:middle;} .scroll thead, .scroll tbody {display: block} .scroll tbody {overflow-y: auto; height: 500px;} .scroll thead tr:after {content: ‘';overflow-y: scroll; visibility: hidden; height: 0;} Tool Version Description Useful Links BEDTools 2.
  • Drive Mapping on Windows or Mac OSX

    Research Standard and Research Project storage can be mapped to your Windows or Mac computer as a remote drive. If you are off Grounds you must be running a VPN, such as the UVA Anywhere or the More Secure VPN from ITS. We recommend the More Secure VPN if that is available to you. Windows Open a File Explorer page. In Windows 10 first select This PC. This is not necessary for earlier versions of Windows. When you see the Computer tab, select that, then select Map Network Drive. When the dialog box appears, the default drive letter will be Z.
  • A Short MPI Tutorial

    Tutorials and books on MPI A helpful online tutorial is available from the Lawrence Livermore National Laboratory. The following books can be found in UVA libraries: Parallel Programming with MPI by Peter Pacheco. Using MPI : Portable Parallel Programming With the Message-Passing Interface by William Gropp, Ewing Lusk, and Anthony Skjellum. Using MPI-2: Advanced Features of the Message- Passing Interface by William Gropp, Ewing Lusk, and Rajeev Thakur. MPI: The Complete Reference : The MPI Core by Marc Snir, Steve Otto, Steven Huss-Lederman, David Walker, and Jack Dongarra. MPI: The Complete Reference : The MPI-2 Extensions by William Gropp, Steven Huss-Lederman, Andrew Lumsdaine, Ewing Lusk, Bill Nitzberg, and Marc Snir.
  • Building and Running MPI Code

    Building an MPI Code All implementations provide wrappers around the underlying compilers that simplify compilation. As it is very important to use the headers that correspond to a given library, users are urged to make use of the wrappers whenever possible. For OpenMPI and MVAPICH2 these are: mpicc (C) mpicxx (C++) mpif90 (Fortran free or fixed format) For Intel MPI these use gcc/g++/gfortran by default, which is generally not recommended; to use the Intel compilers the corresponding wrappers are: mpiicc mpiicpc mpiifort Note: At this time, we recommend MPI users build with Intel 18.0 and IntelMPI 18.
  • Docker Images on Rivanna

    Docker requires sudo privilege and therefore it is not supported on Rivanna. To use a Docker image you will need to convert it into Singularity. Convert a Docker image There are several ways to convert a Docker image: Download a remote image from Docker Hub Build from a local image cached in Docker daemon Build from a definition file (advanced) Instructions are provided in each of the following sections. Docker Hub Docker images hosted on Docker Hub can be downloaded and converted in one step via the singularity pull command: module load singularity singularity pull docker://account/image Use the exact same command as you would for docker pull.
  • General How To Guides

    User Guides Databases MySQL Redis MongoDB Docker - Basics Authentication with SSH Keys
  • How to add packages to a container?

    Strictly speaking, you cannot add packages to an existing container since it is not editable. However, you can try to install missing packages locally. Using python-pip as an example: singularity exec <container.sif> python -m pip install –user <package> Replace <container.sif> with the actual filename of the container and <package> with the package name. The Python package will be installed in your home directory under .local/lib/pythonX.Y where X.Y is the Python version in the container. If the installation results in a binary, it will often be placed in .local/bin. Remember to add this to your PATH: export PATH=~/.local/bin:$PATH You should be able to use the new package/binary in the container, as your entire home directory is mounted at runtime.
  • How To Guides for Ivy Users

    Transfer Files to or from Ivy Using Globus
  • How To

    General General tips and tricks for computational research. General HowTos › Rivanna High Performance Computing platform Rivanna HowTos › Ivy Secure Data Computing Platform Ivy HowTos › Storage Research Data Storage & Transfer Storage HowTos ›
  • Image Processing & Scientific Visualization on Rivanna

    Available Software To get an up-to-date list of the installed image processing and visualization tools, log on to Rivanna and run the following command in a terminal window: module keyword vis To get more information about a specific module version, run the module spider command, for example: module spider blender/2.78c List of Image Processing and Visualization Software Modules function searchFunction() { var input, filter, table, tr, td, i, txtValue; input = document.getElementById(“searchInput”); filter = input.value.toUpperCase(); table = document.getElementById(“moduleTable”); tr = table.getElementsByTagName(“tr”); for (i = 0; i -1) { tr[i].style.display = “"; } else { tr[i].style.display = “none”; } } } } Module Category Description atk vis ATK provides the set of accessibility interfaces that are implemented by other toolkits and applications.