an object-oriented open source CFD toolkit

Description

OpenFOAM is an extensible Computational Fluid Dynamics framework written in C++, i.e. it provides an abstraction for a programmer to build their own code for an underlying mathematical model.

Modules

To see our installed versions type:

module avail openfoam

Example Jobscripts

Serial job for v5 based on cavity tutorial

#!/bin/bash
#SBATCH --time 1:00:00
#SBATCH --nodes 1 
#SBATCH --tasks-per-node 96 
#SBATCH -p standard96:test
#SBATCH --job-name=test_job
#SBATCH --output=ol-%x.%j.out
#SBATCH --error=ol-%x.%j.err
 
export I_MPI_FALLBACK=0
export I_MPI_DEBUG=6
export I_MPI_FABRICS=shm:ofi
export I_MPI_OFI_PROVIDER=psm2
export I_MPI_PMI_LIBRARY=libpmi.so
 
module load gcc/9.2.0
module load openmpi/gcc.9/3.1.5
module load openfoam/gcc.9/5

# initialize OpenFOAM environment
#---------------------
source $WM_PROJECT_DIR/etc/bashrc
source ${WM_PROJECT_DIR:?}/bin/tools/RunFunctions # provides fcts like runApplication
 
# set working directory
#---------------------
WORKDIR="$(pwd)"

# get and open example
#---------------------
cp -r $WM_PROJECT_DIR/tutorials/incompressible/icoFoam/cavity $WORKDIR/
cd cavity
 
# run script with several cases 
#------------------------------
./Allrun

# run single case
#--------------------------
#cd cavity
#runApplication blockMesh
#icoFoam > icoFoam.log 2>&1

The next example is derived from https://develop.openfoam.com/committees/hpc/-/wikis/HPC-motorbike. It utilizes two full nodes and has collated file I/O. All required input/case files can be downloaded here: motorbike_with_parallel_slurm_script.tar.gz.

Parallel job for v2112 based on motorbike tutorial

#!/bin/bash
#SBATCH --time 1:00:00
#SBATCH --nodes 2
#SBATCH --tasks-per-node 96
#SBATCH --partition standard96
#SBATCH --job-name foam_test_job
#SBATCH --output ol-%x.%j.out
#SBATCH --error ol-%x.%j.err

module load gcc/9.3.0 openmpi/gcc.9/3.1.5
module load openfoam/gcc.9/v2112

. $WM_PROJECT_DIR/etc/bashrc                # initialize OpenFOAM environment
. $WM_PROJECT_DIR/bin/tools/RunFunctions    # source OpenFOAM helper functions (wrappers)

tasks_per_node=${SLURM_TASKS_PER_NODE%\(*}
ntasks=$(($tasks_per_node*$SLURM_JOB_NUM_NODES))
foamDictionary -entry "numberOfSubdomains" -set "$ntasks" system/decomposeParDict # number of geometry fractions after decompositon will be number of tasks provided by slurm

date "+%T"
runApplication blockMesh                    # create coarse master mesh (here one block)
date "+%T"

runApplication decomposePar                 # decompose coarse master mesh over processors
mv log.decomposePar log.decomposePar_v0
date "+%T"

runParallel snappyHexMesh -overwrite        # parallel: refine mesh for each processor (slow if large np) matching surface geometry (of the motorbike)
date "+%T"

runApplication reconstructParMesh -constant # reconstruct fine master mesh 1/2 (super slow if large np)
runApplication reconstructPar -constant     # reconstruct fine master mesh 2/2
date "+%T"

rm -fr processor*                           # delete decomposed coarse master mesh
cp -r 0.org 0                               # provide start field
date "+%T"

runApplication decomposePar                 # parallel: decompose fine master mesh and start field over processors
date "+%T"

runParallel potentialFoam                   # parallel: run potentialFoam
date "+%T"

runParallel simpleFoam                     # parallel: run simpleFoam
date "+%T"

Some important advice when running OpenFOAM on a supercomputer

Typically, OpenFOAM causes a lot of meta data operations. This default behavior jams no only your job but may slow down the shared parallel file system (=Lustre) for all other users. Also, your job is interrupted if the inode limit (number of files) of the quota system (show-quota) is exceeded.

If you can not use our local 2TB-SSDs (see Special Filesystems) #SBATCH --partition={standard,large,huge}96:ssd at $LOCAL_TMPDIR please refer to our general advices to reduce Metadata Usage on WORK (=Lustre).

To adapt/optimize your OpenFOAM job specifically for I/O operations on $WORK (=Lustre) we strongly recommend the following steps:

Always, to avoid that each processor writes in its own file please use collated file I/O.
This feature was released 2017 for all OpenFOAM versions.
[ESI www.openfoam.com/releases/openfoam-v1712/parallel.php]
[Foundation www.openfoam.org/news/parallel-io]
```
OptimisationSwitches
{
    fileHandler collated; // all processors share a file
}
```
to the /.OpenFOAM/v#/controlDict file or per-case override in the $FOAM_CASE/system/controlDict file.
Always, set
```
runTimeModifiable false;
```
to reduce I/O activity. Only set "true" (default), if it is strictly necessary to re-read dictionaries (controlDict, ...) each time step.
Possibly, do not save every time step:
[www.openfoam.com/documentation/guides/latest/doc/guide-case-system-controldict.html]
[www.cfd.direct/openfoam/user-guide/v6-controldict]
```
writeControl	timeStep;
writeInterval	100;
```
Possibly, save only the latest n time steps (overwrite older ones), such as:
```
purgeWrite	1000;
```

Typically, only a subset of variables is needed frequently (post-processing). The full set of variables can be saved less frequently (e.g., restart purposes). This can be achieved with [https://wiki.bwhpc.de/e/OpenFoam]:

writeControl    clockTime;
writeInterval   21600; // write ALL variables every 21600 seconds = 6 h

functions
{
    writeFields
    {
        type writeObjects;
        libs ("libutilityFunctionObjects.so");

        objects
        (
	    T U // specified variables
        );

        outputControl timeStep;
        writeInterval 100; // write specified variables every 100 steps
    }
}

In case your job accidentally generated thousands of small files, please pack them (at least the small-size metadata files) into a single file afterwards:
```
tar -xvzf singlefile.tar.gz -C /folder/subfolder/location/
```
Thanks a lot for your contribution making the compute systems a great place for all...

Compiling Own Code Over OpenFOAM

...

OpenFOAM Best Practices

...