Single core

Programs that do not use any parallelisation, which includes most programs, are known as serial or sequential programs. They only use one CPU core at a time, so many instances can run at the same time on one of Hamilton's multi-core compute nodes.

An example job script to run a program called my_serial_program would be:

#!/bin/bash
 
# Request resources:
#SBATCH -c 1           # 1 CPU core
#SBATCH --mem=1G       # memory required, up to 250G on standard nodes.
#SBATCH --time=1:0:0   # time limit for job (format:  days-hours:minutes:seconds)
#SBATCH --gres=tmp:1G  # temporary disk space required on the compute node ($TMPDIR),
#                        up to 400G
# Run in the 'shared' queue (job may share node with other jobs)
#SBATCH -p shared
 
# Commands to be run:
module load my_module
./my_serial_program

If saved in a file called my_serial_job.sh, this could be submitted to the queue with the command:

sbatch my_serial_job.sh

Shared memory

Some programs can use more than one CPU core at a time, but are limited to a single compute node. They typically use programming techniques such as OpenMP or threading to achieve this. We call them shared memory programs, because the parallelisation requires that all CPU cores have access to the same RAM/memory.

An example job script to run a program called my_sharedmemory_program would be:

#!/bin/bash
 
# Request resources:
#SBATCH -c 2          # number of CPU cores, one per thread, up to 128
#SBATCH --mem=1G      # memory required, up to 250G on standard nodes
#SBATCH --time=1:0:0  # time limit for job (format:  days-hours:minutes:seconds)
#SBATCH --gres=tmp:1G # temporary disk space required on the compute node ($TMPDIR), 
#                       up to 400G
# Run in the 'shared' queue (job may share node with other jobs)
#SBATCH -p shared
 
# Commands to be run:
module load my_module
./my_sharedmemory_program

If saved in a file called my_shared_job.sh, this can be submitted to the queue with the command:

sbatch my_shared_job.sh

Distributed memory

Programs can be written to take advantage of CPU cores and memory spread across multiple compute nodes. They typically use the low-level library called MPI (Message Passing Interface) to allow communication between many copies of the same program, each with access to its own CPU core and memory. We call this a distributed memory programming model.

An example job script to run an MPI program called my_mpi_program would be:

#!/bin/bash
 
# Request resources:
#SBATCH -n 1           # number of MPI ranks (1 per CPU core)
#SBATCH --mem=1G       # memory required per node, in units M, G or T
#SBATCH --time=1:0:0   # time limit for job (format:  days-hours:minutes:seconds)
#SBATCH --gres=tmp:1G  # temporary disk space required on each compute node ($TMPDIR)
#SBATCH -N 1           # number of compute nodes.  
 
# Smaller jobs can run in the shared queue.  
# Larger jobs that will occupy one or more whole nodes should use the multi queue.
#SBATCH -p shared 
 
# Commands to be run.  
# Note that mpirun will automatically launch the number of ranks specified above 
module load my_module
mpirun ./my_mpi_program

If saved in a file called my_dist_job.sh, this can be submitted to the queue with the command:

sbatch my_dist_job.sh

Note that it is not normally necessary to specify the number of tasks/ranks in the mpirun command; this will be matched to the number allocated in the line #SBATCH -n.

Hybrid

Using a mixed MPI / OpenMP model can have benefits, for example, to reduce the memory and computation dedicated to halo exchanges between different processes in grid-based codes.  Codes written in this hybrid way can run on multiple CPU cores across one or more nodes.

For these codes we recommend running one MPI rank per CPU socket (two MPI ranks per compute node on Hamilton). An example job script would be:

#!/bin/bash
 
# Request resources:
#SBATCH -n 1                    # number of MPI ranks
#SBATCH -c 1                    # number of threads per rank (one thread per CPU core)
#SBATCH --ntasks-per-socket=1   # number of MPI ranks per CPU socket
#SBATCH -N 1                    # number of compute nodes. 
#SBATCH --mem=1G                # memory required per node, in units M, G or T
#SBATCH --gres=tmp:1G           # temporary disk space on each compute node ($TMPDIR)
#SBATCH -t 1:0:0                # time limit for job (format: days-hours:minutes:seconds) 
 
# Smaller jobs can run in the shared queue. 
# Larger jobs that will occupy one or more whole nodes should use hte multi queue.
#SBATCH -p shared 
 
# Commands to be run. 
# Note that mpirun will automatically launch the number of ranks specified above 
module load my_module o provide the CPUs requested)  
mpirun ./my_hybrid_program

If saved in a file called my_hybrid_job.sh, this can be submitted to the queue with the command:

sbatch my_hybrid_job.sh

High memory

Jobs that require >250GB memory (per node) should run in the bigmem queue. The nodes in this queue each have 1.95TB memory. An example job script my_bigmem_job.sh might be:

#!/bin/bash
 
# Request resources:
#SBATCH -c 1            # number of CPU cores, up to 128 for shared-memory programs
#SBATCH --mem=260G      # memory required, up to 1.95T
#SBATCH --time=1:0:0    # time limit for job (format:  days-hours:minutes:seconds)
#SBATCH --gres=tmp:1G   # temporary disk space required on the compute node ($TMPDIR),
#                         up to 400G
# Run in the bigmem queue (job may share node with other jobs)
#SBATCH -p bigmem
 
# Commands to be run:
module load my_module
./my_bigmem_program

If saved in a file called my_bigmem_job.sh, this can be submitted to the queue with the command:

sbatch my_bigmem_job.sh