Running a Multi-Node MPI Job on NCShare

This guide introduces the key concepts of MPI for cross-node parallelization through an example that demonstrates the ring topology communication pattern. It is intended for users who are new to MPI and want to verify that their environment is configured correctly for running multi-node jobs on NCShare.

The code for this example is available at: https://github.com/NCShare/examples/tree/main/MPI-on-NCShare

This example introduces the Message Passing Interface (MPI) and walks through a short Fortran program, mpi_ring_test.f90, that exercises the two communication patterns found in nearly every parallel application: point-to-point messages between neighboring processes and a collective reduction across all of them.

The program is short, but it functions as a practical diagnostic. Running it across two NCShare compute nodes confirms that the compiler wrappers, the Slurm launcher, and the inter-node fabric are all working together before allocation time is committed to a production run.

What is MPI?

Laptops and individual compute nodes support parallelization with threads, where every worker shares a single pool of memory (shared-memory parallelism). That model is bounded by the node. To use more than one machine, each process requires its own private memory and an explicit mechanism for exchanging data. MPI is the standard that defines those exchanges, an approach known as distributed-memory parallelism.

The following terminology is used throughout this guide,

Term	Meaning
Process / task	One independent copy of the program with its own memory. On NCShare, one Slurm task.
Rank	The integer ID of a process within a communicator, from `0` to `nprocs-1`.
Communicator	A group of processes that can communicate with each other. `MPI_COMM_WORLD` contains all of them.
Point-to-point	One rank sends, one rank receives (`MPI_Send`, `MPI_Recv`, `MPI_Sendrecv`).
Collective	All ranks in a communicator participate (`MPI_Barrier`, `MPI_Bcast`, `MPI_Allreduce`).

MPI programs follow the SPMD model: single program, multiple data. Every rank executes the same executable. The ranks are distinguished only by the rank number returned by MPI_Comm_rank, and the code branches on that value to assign work.

Nearly every MPI program shares the same skeleton,

call MPI_Init(ierr)                              ! start the MPI environment
call MPI_Comm_rank(MPI_COMM_WORLD, rank, ierr)   ! rank of this process
call MPI_Comm_size(MPI_COMM_WORLD, nprocs, ierr) ! total number of processes

! ... computation and communication ...

call MPI_Finalize(ierr)                          ! shut down MPI

MPI and Slurm

Slurm and MPI have distinct responsibilities. Slurm handles resource allocation and process placement, determining how many tasks are started and on which nodes. MPI defines the communication layer between those processes. The two are coupled at launch time, when mpirun reads the Slurm allocation and starts one task per allocated slot on the assigned nodes.

MPI on NCShare

NCShare provides Open MPI system-wide. There is no environment module system on the cluster and nothing to load: the wrappers and the launcher are installed in /usr/bin and are on your PATH in every shell, interactive or batch.

$ mpirun --version

mpirun (Open MPI) 4.1.6

Report bugs to http://www.open-mpi.org/community/help/

This build was configured --with-slurm, so mpirun reads the Slurm allocation directly and places one task per allocated slot without any additional flags.

The compiler wrappers invoke the underlying compiler with the include paths, library paths, and link flags required by Open MPI,

Language	Wrapper
Fortran	`mpif90` / `mpifort`
C	`mpicc`
C++	`mpicxx`

To see which compiler a wrapper actually calls, along with the include and library paths it adds, use the --showme flag,

$ mpif90 --showme

gfortran -I/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi -I/usr/lib/x86_64-linux-gnu/openmpi/lib -L/usr/lib/x86_64-linux-gnu/openmpi/lib/fortran/gfortran -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lopen-rte -lopen-pal -lhwloc -levent_core -levent_pthreads -lm -lz

The wrapper calls the system gfortran, so no compiler needs to be loaded or installed either.

Do not let a Conda environment shadow the system MPI

Installing a package that pulls in its own MPI (mpi4py, openmpi, or mpich from conda-forge) places a second mpirun and a second set of wrappers ahead of /usr/bin on your PATH. Compiling with one MPI and launching with another produces confusing failures. Run which mpirun mpif90 to confirm both resolve to /usr/bin before you build, and either deactivate the environment or keep the whole toolchain inside it consistently.

Other MPI implementations

MPICH and Intel MPI are not provided on NCShare. Codes that require them can be installed into your home directory or /work with Conda or Spack, which does not require administrator privileges. An executable built against one MPI implementation will generally not run under another, so build and launch with the same one.

A minimal MPI program

Before introducing communication, confirm that ranks start correctly and are placed as requested. The program below reports the rank, communicator size, and hostname of each process. Save it as mpi_hello.f90,

mpi_hello.f90

program mpi_hello
  use mpi
  implicit none

  integer :: ierr, rank, nprocs, name_len
  character(len=MPI_MAX_PROCESSOR_NAME) :: hostname

  call MPI_Init(ierr)
  call MPI_Comm_rank(MPI_COMM_WORLD, rank, ierr)
  call MPI_Comm_size(MPI_COMM_WORLD, nprocs, ierr)
  call MPI_Get_processor_name(hostname, name_len, ierr)

  write (*,'(A,I0,A,I0,A,A)') 'Hello from rank ', rank, ' of ', nprocs, &
                              ' on ', trim(hostname)

  call MPI_Finalize(ierr)
end program mpi_hello

Request an interactive session with four tasks, then compile and run,

srun -p interactive -n 4 --pty bash -i
mpif90 mpi_hello.f90 -o mpi_hello
mpirun -n 4 ./mpi_hello

Hello from rank 1 of 4 on compute-08
Hello from rank 3 of 4 on compute-08
Hello from rank 2 of 4 on compute-08
Hello from rank 0 of 4 on compute-08

The interactive partition has a one hour wall-time limit, which is ample for a compile-and-check cycle. Use common for anything longer.

Four processes were started and each determined its own identity, but no data was exchanged between them. Communication is what the ring test adds.

The ring topology

The ring test arranges the ranks in a logical circle. Each rank has exactly one left neighbor and one right neighbor, and rank nprocs-1 wraps around to rank 0. Every rank simultaneously sends its own rank number to the right and receives its left neighbor's number.

MPI ring topology: eight ranks arranged in a circle, each sending its rank number clockwise to its right neighbor.

The neighbor indices follow from modular arithmetic,

left_rank  = mod(rank - 1 + nprocs, nprocs)
right_rank = mod(rank + 1, nprocs)

The + nprocs term in the first expression is required. Fortran's mod takes the sign of its first argument, so mod(-1, 8) evaluates to -1 rather than 7. Adding nprocs before taking the modulus places the left neighbor of rank 0 at the top of the ring, as intended.

The code

The complete mpi_ring_test.f90 is shown below.

mpi_ring_test.f90

program mpi_ring_test
  use mpi
  implicit none

  integer :: ierr
  integer :: rank, nprocs, name_len
  integer :: left_rank, right_rank
  integer :: sendbuf, recvbuf, rank_sum
  character(len=MPI_MAX_PROCESSOR_NAME) :: hostname

  call MPI_Init(ierr)
  call MPI_Comm_rank(MPI_COMM_WORLD, rank, ierr)
  call MPI_Comm_size(MPI_COMM_WORLD, nprocs, ierr)
  call MPI_Get_processor_name(hostname, name_len, ierr)

  write (*,'(A,I0,A,I0,A,A)') 'Rank ', rank, ' of ', nprocs, ' on ', trim(hostname)
  call MPI_Barrier(MPI_COMM_WORLD, ierr)

  left_rank = mod(rank - 1 + nprocs, nprocs)
  right_rank = mod(rank + 1, nprocs)
  sendbuf = rank
  recvbuf = -1

  ! Exchange one integer with neighboring ranks to verify point-to-point traffic.
  call MPI_Sendrecv(sendbuf, 1, MPI_INTEGER, right_rank, 0, &
                    recvbuf, 1, MPI_INTEGER, left_rank, 0, &
                    MPI_COMM_WORLD, MPI_STATUS_IGNORE, ierr)

  write (*,'(A,I0,A,I0,A,I0)') 'Rank ', rank, ' received ', recvbuf, ' from rank ', left_rank

  call MPI_Allreduce(rank, rank_sum, 1, MPI_INTEGER, MPI_SUM, MPI_COMM_WORLD, ierr)
  if (rank == 0) then
    write (*,'(A,I0)') 'Global rank sum = ', rank_sum
  end if

  call MPI_Finalize(ierr)
end program mpi_ring_test

Startup and identity

use mpi

This imports the MPI module, which provides the named constants (MPI_COMM_WORLD, MPI_INTEGER, MPI_SUM, …) and the interfaces for the routines used below.

MPI_Init must be the first MPI call and MPI_Finalize the last. Between them, MPI_Comm_rank and MPI_Comm_size return the rank of the calling process and the total number of processes, while MPI_Get_processor_name returns the hostname, which confirms that the job spans two nodes.

The ierr argument

In the Fortran bindings, every MPI routine takes a trailing integer error code. By default MPI aborts the job on error, so production codes rarely inspect it, but omitting the argument is one of the most common compile-time errors for new users.

The barrier

call MPI_Barrier(MPI_COMM_WORLD, ierr)

MPI_Barrier blocks until every rank in the communicator has reached it. Here it separates the two phases of output so that the "Rank i of n" lines are not interleaved with the exchange results. It is a synchronization point, not a data transfer.

Barriers do not order output within a phase

A barrier orders phases, not individual lines. Standard output from multiple ranks is funneled through the launcher and can still arrive out of order within a phase, as the sample output below shows. Print ordering should not be used to infer execution order; use timers or rank-tagged output files instead.

The neighbor exchange

call MPI_Sendrecv(sendbuf, 1, MPI_INTEGER, right_rank, 0, &
                  recvbuf, 1, MPI_INTEGER, left_rank,  0, &
                  MPI_COMM_WORLD, MPI_STATUS_IGNORE, ierr)

The argument list has two halves,

Arguments	Role
`sendbuf, 1, MPI_INTEGER, right_rank, 0`	send 1 integer from `sendbuf` to `right_rank` with tag `0`
`recvbuf, 1, MPI_INTEGER, left_rank, 0`	receive 1 integer into `recvbuf` from `left_rank` with tag `0`
`MPI_COMM_WORLD, MPI_STATUS_IGNORE, ierr`	communicator, discard the status object, error code

The tag is a user-defined label that allows a receiver to distinguish messages that would otherwise match the same source and communicator. With a single message type in flight, 0 is sufficient.

MPI_Sendrecv is used here in preference to an MPI_Send followed by an MPI_Recv because a ring is precisely the topology in which the latter can deadlock. If every rank calls MPI_Send first and the message is too large for MPI to buffer internally, every rank blocks waiting for a matching receive that has not been posted. The program then hangs, and it does so only above a threshold message size, making the failure difficult to reproduce and diagnose. MPI_Sendrecv passes both halves of the exchange to MPI at once and allows the library to schedule them safely.

Alternatives

Non-blocking MPI_Isend / MPI_Irecv followed by MPI_Waitall are equally safe and allow communication to be overlapped with computation. MPI_Sendrecv is the simplest correct choice when there is no work available to overlap.

The collective reduction

call MPI_Allreduce(rank, rank_sum, 1, MPI_INTEGER, MPI_SUM, MPI_COMM_WORLD, ierr)

MPI_Allreduce combines one value from every rank using a specified operation, here MPI_SUM, and delivers the result to all ranks. (MPI_Reduce delivers the result to a single root rank only.) Other operators include MPI_MAX, MPI_MIN, and MPI_PROD.

The result is self-verifying. Summing $0 + 1 + \dots + (n-1)$ gives $n(n-1)/2$, so with eight ranks the expected value is 28. Only rank 0 prints the result, a common idiom that keeps the output readable.

Compiling

mpif90 invokes the underlying Fortran compiler with the MPI headers and libraries already supplied on the command line.

mpif90 mpi_ring_test.f90 -o mpi_ring_test

The job script

jobscript.sh is a Slurm batch job script that requests two nodes with four tasks each, for eight MPI ranks in total,

jobscript.sh

#!/bin/bash
#SBATCH -J mpi_ring_test      # Job name
#SBATCH -p common             # Partition name
#SBATCH -N 2                  # Total # of nodes
#SBATCH --ntasks-per-node 4   # Tasks per node

cd $SLURM_SUBMIT_DIR

# Compile
mpif90 mpi_ring_test.f90 -o mpi_ring_test

# Execute
mpirun -n $SLURM_NTASKS ./mpi_ring_test > mpi_ring_test_output.txt

Submit the job and monitor it with,

sbatch jobscript.sh
squeue -u $USER

Compile once rather than on every run

Compiling inside the job script keeps a small example such as this one self-contained, but it consumes allocated node time on a step that does not require a parallel allocation. For larger production applications, compile on the login node and have the job script launch the executable only.

Output

The program produces the following output.

Rank 0 of 8 on compute-06
Rank 1 of 8 on compute-06
Rank 2 of 8 on compute-06
Rank 3 of 8 on compute-06
Rank 4 of 8 on compute-07
Rank 5 of 8 on compute-07
Rank 6 of 8 on compute-07
Rank 7 of 8 on compute-07
Rank 0 received 7 from rank 7
Rank 3 received 2 from rank 2
Rank 4 received 3 from rank 3
Rank 7 received 6 from rank 6
Rank 1 received 0 from rank 0
Rank 2 received 1 from rank 1
Rank 6 received 5 from rank 5
Rank 5 received 4 from rank 4
Global rank sum = 28

Three points are worth noting.

The job spanned two nodes
Ranks 0–3 were placed on compute-06 and ranks 4–7 on compute-07, as requested by -N 2 --ntasks-per-node 4. Slurm assigns ranks in blocks by node, so the 3 → 4 and 7 → 0 links of the ring cross the network while the remaining links stay within a node.
Every exchange completed correctly
Each line reports Rank r received r-1 from rank r-1, and the wrap-around line, Rank 0 received 7 from rank 7, closes the circle. A rank reporting -1 would indicate that its receive never completed.
The reduction is consistent
$28 = 8 \times 7 / 2$, confirming that all eight ranks contributed to the collective.

The exchange lines are not sorted, which is expected. The eight ranks write to standard output concurrently and the launcher merges those streams in the order they arrive. That the identity lines happen to appear in rank order in this particular run is incidental and should not be relied upon. The barrier is nevertheless effective: all eight identity lines appear before any exchange line.

Checklist: what a successful run confirms

The compiler wrappers resolved to the system Open MPI and worked.
Slurm placed the requested number of ranks on the requested number of nodes.
Point-to-point messages crossed both intra-node and inter-node boundaries.
Collective communication completed across the full communicator.
The job ran to completion and called MPI_Finalize cleanly.

If the ring test passes but a production code still fails, the fault lies in the application or its input rather than in the MPI environment, which considerably narrows the search.

Troubleshooting

Error: Can't open module file 'mpi.mod'

The source was compiled with gfortran instead of mpif90. Compile with the wrapper, which supplies the MPI include and library paths.

error while loading shared libraries: libmpi...

The executable was built against an MPI that is not on the library path at run time, which on NCShare almost always means a Conda environment was active during the build but not in the job. Run which mpif90 and which mpirun in both places and make them agree.

All ranks report Rank 0 of 1

The launcher did not see the Slurm allocation and started a single independent process instead. Verify that -n $SLURM_NTASKS is passed to mpirun, and that mpirun is the system one in /usr/bin rather than a Conda copy that was built without Slurm support.

The job hangs with no output

This is the signature of a send/receive deadlock, which is the reason this example uses MPI_Sendrecv rather than a separate MPI_Send and MPI_Recv.

All ranks are placed on one node

Confirm that -N 2 is present in the job script and is not overridden by a later --nodes directive.