MPI stands for Message Passing Interface. The MPI standard is defined by the Message Passing Interface Forum. The standard defines the interface for a set of functions that can be used to pass messages between processes on the same computer or on different computers. MPI can be used to program shared memory or distributed memory computers. There is a large number of implementations of MPI from various computer vendors and academic groups. MPI is supported on the Rivanna cluster.
MPI On Rivanna
MPI is a standard that describes the behavior of a library. It is intended to be used with compiled languages (C/C++/Fortran). Several implementations of this standard exist. Rivanna supports OpenMPI for all our compilers and IntelMPI for the Intel compiler. MPI can also be used with the interpreted languages R and Python through packages that link to an implementation; on Rivanna these languages use OpenMPI.
Selecting Compiler and Implementation
An MPI implementation must be built with a specific compiler. Consequently, only compilers for which MPI has been prepared can be used with it. All versions of the Intel compiler will have a corresponding IntelMPI. For OpenMPI run
module spider openmpi
This will respond with the versions of OpenMPI available. To see which version goes with which compiler, run
module spider openmpi/<version>
You will need to load all module(s) on any one of the lines below before the "" module is available to load. gcc/system gcc/5.4.0 gcc/7.1.0
This shows that OpenMPI version is available for the gcc system (4.8.5), 5.4.0, and 7.1.0 compilers.
Once a choice of compiler and MPI implementation have been made, the modules must be loaded. First load the compiler, then the MPI. For instance, to use OpenMPI with gcc 7.1.0, run
module load gcc/7.1.0 module load openmpi
To load the Intel compiler version 18.0 and its IntelMPI version, run
module load intel/18.0 module load intelmpi/18.0
It is also possible to combine these into one line, as long as the compiler is specified first (this can result in errors if you not using the default compiler, however)
module load gcc openmpi
For a detailed description of building and running MPI codes on Rivanna, please see our HowTo.
**Available MPI library modules**
|intelmpi||mpi||IntelMPI from Intel.|
|mvapich2||mpi||The MVAPICH2 software, based on MPI 3.1 standard, delivers the best performance, scalability and fault tolerance for high-end computing systems and servers using InfiniBand, Omni-Path, Ethernet/iWARP, and RoCE networking technologies.|
|openmpi||mpi||The Open MPI Project is an open source MPI-3 implementation.|
|platform-mpi||mpi||Platform MPI is an MPI-2 implementation from IBM.|
Example Slurm Scripts
This example is a Slurm job command file to run a parallel (MPI) job using the OpenMPI implementation:
#!/bin/bash #SBATCH --nodes=2 #SBATCH --ntasks-per-node=16 #SBATCH --time=12:00:00 #SBATCH --output=output_filename #SBATCH --partition=parallel #SBATCH -A mygroup module load gcc module load openmpi srun ./parallel_executable
In this example, the Slurm job file is requesting two nodes with sixteen tasks per node for a total of 32 processes. Both OpenMPI and IntelMPI are able to obtain the number of processes and the host list from Slurm, so these are not specified. In general, MPI jobs should use all of a node, but some codes cannot be distributed in that manner so we are showing a more general example here.
Slurm can also place the job freely if the directives specify only the number of tasks. In this case do not specify a node count. This is not generally recommended, however, as it can have a significant negative impact on performance.
#!/bin/bash #SBATCH --ntasks=8 #SBATCH --time=12:00:00 #SBATCH --output=output_filename #SBATCH --partition=parallel #SBATCH -A mygroup module load gcc module load openmpi srun ./parallel_executable
Example: MPI over an odd number of tasks
#!/bin/bash #SBATCH --ntasks=97 #SBATCH --nodes=5 #SBATCH --ntasks-per-node=20 #SBATCH --time=12:00:00 #SBATCH --output=output_filename #SBATCH --partition=parallel #SBATCH -A mygroup module load gcc module load openmpi srun ./parallel_executable
MPI with OpenMP
The following example runs a total of 32 MPI processes, 4 on each node, with each task using 5 cores for threading. The total number of cores utilized is thus 160.
#!/bin/bash #SBATCH --ntasks=32 #SBATCH --ntasks-per-node=4 #SBATCH --cpus-per-task=5 #SBATCH --time=12:00:00 #SBATCH --output=output_filename #SBATCH --partition=parallel #SBATCH -A mygroup module load mvapich2/gcc export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK srun ./hybrid_executable