Multiple concurrent programs on a single node

You can use srun  to start multiple job steps concurrently on a single node, e.g. if your job is not big enough to fill a whole node. There are a few details to follow:

• By default, the srun command gets exclusive access to all resources of the job allocation and uses all tasks
  • you therefore need to limit srun to only use part of the allocation
  • this includes implicitly granted resources, i.e. memory and GPUs
  • the --exact flag is needed.
  • if running non-mpi programs, use the -c option to denote the number of cores, each process should have access to
• srun  waits for the program to finish, so you need to start concurrent processes in the background

• Good default memory per cpu values (without hyperthreading) are usually are:

  	standard96	large96	huge96	medium40	large40/gpu
  --mem-per-cpu	3770M	7781M	15854M	4525M	19075M

Examples


#!/bin/bash
#SBATCH -p standard96
#SBATCH -t 06:00:00
#SBATCH -N 1

srun --exact -n1 -c 10 --mem-per-cpu 3770M  ./program1 &
srun --exact -n1 -c 80 --mem-per-cpu 3770M  ./program2 &
srun --exact -n1 -c 6 --mem-per-cpu 3770M  ./program3 &
wait

#!/bin/bash
#SBATCH -p gpu
#SBATCH -t 12:00:00
#SBATCH -N 1

srun --exact -n1 -c 10 -G1 --mem-per-cpu 19075M  ./single-gpu-program &
srun --exact -n1 -c 10 -G1 --mem-per-cpu 19075M  ./single-gpu-program &
srun --exact -n1 -c 10 -G1 --mem-per-cpu 19075M  ./single-gpu-program &
srun --exact -n1 -c 10 -G1 --mem-per-cpu 19075M  ./single-gpu-program &
wait