...
| Codeblock |
|---|
| language | bash |
|---|
| title | Serial job for v5 based on cavity tutorial |
|---|
| collapse | true |
|---|
|
#!/bin/bash
#SBATCH --time 1:00:00
#SBATCH --nodes 1
#SBATCH --tasks-per-node 96
#SBATCH -p standard96:test
#SBATCH --job-name=test_job
#SBATCH --output=ol-%x.%j.out
#SBATCH --error=ol-%x.%j.err
export I_MPI_FALLBACK=0
export I_MPI_DEBUG=6
export I_MPI_FABRICS=shm:ofi
export I_MPI_OFI_PROVIDER=psm2
export I_MPI_PMI_LIBRARY=libpmi.so
module load gcc/9.2.0
module load openmpi/gcc.9/3.1.5
module load openfoam/gcc.9/5
# initialize OpenFOAM environment
#---------------------
source $WM_PROJECT_DIR/etc/bashrc
source ${WM_PROJECT_DIR:?}/bin/tools/RunFunctions # provides fcts like runApplication
# set working directory
#---------------------
WORKDIR="$(pwd)"
# get and open example
#---------------------
cp -r $WM_PROJECT_DIR/tutorials/incompressible/icoFoam/cavity $WORKDIR/
cd cavity
# run script with several cases
#------------------------------
./Allrun
# run single case
#--------------------------
#cd cavity
#runApplication blockMesh
#icoFoam > icoFoam.log 2>&1 |
The next example is derived from https://develop.openfoam.com/committees/hpc/-/wikis/HPC-motorbike. The required input/case file can be downloaded here: motorbike_with_parallel_slurm_script.tar.gz.
| Codeblock |
|---|
| language | bash |
|---|
| title | Parallel job for v2112 based on motorbike tutorial |
|---|
| collapse | true |
|---|
|
#!/bin/bash
#SBATCH --time 1:00:00
#SBATCH --nodes 2
#SBATCH --tasks-per-node 96
#SBATCH --partition standard96
#SBATCH --job-name foam_test_job
#SBATCH --output ol-%x.%j.out
#SBATCH --error ol-%x.%j.err
#module load gcc/9.3.0 openmpi/4.1.2
module load gcc/9.3.0 openmpi/gcc.9/3.1.5
module load openfoam/gcc.9/v2112
. $WM_PROJECT_DIR/etc/bashrc # initialize OpenFOAM environment
. $WM_PROJECT_DIR/bin/tools/RunFunctions # source OpenFOAM helper functions (wrappers)
tasks_per_node=${SLURM_TASKS_PER_NODE%\(*}
ntasks=$(($tasks_per_node*$SLURM_JOB_NUM_NODES))
foamDictionary -entry "numberOfSubdomains" -set "$ntasks" system/decomposeParDict # number of geometry fractions after decompositon will be number of tasks provided by slurm
date "+%T"
runApplication blockMesh # create coarse master mesh (here one block)
date "+%T"
runApplication decomposePar # decompose coarse master mesh over processors
mv log.decomposePar log.decomposePar_v0
date "+%T"
runParallel snappyHexMesh -overwrite # parallel: refine mesh for each processor (slow if large np) matching surface geometry (of the motorbike)
date "+%T"
runApplication reconstructParMesh -constant # reconstruct fine master mesh 1/2 (super slow if large np)
runApplication reconstructPar -constant # reconstruct fine master mesh 2/2
date "+%T"
rm -fr processor* # delete decomposed coarse master mesh
cp -r 0.org 0 # provide start field
date "+%T"
runApplication decomposePar # parallel: decompose fine master mesh and start field over processors
date "+%T"
runParallel potentialFoam # parallel: run potentialFoam
date "+%T"
runParallel simpleFoam # parallel: run simpleFoam
date "+%T" |
Some important advice when running OpenFOAM on a supercomputer
...