Atlassian uses cookies to improve your browsing experience, perform analytics and research, and conduct advertising. Accept all cookies to indicate that you agree to our use of cookies on your device. Atlassian cookies and tracking notice, (opens new window)
User Manual

User Manual
Results will update as you type.
  • Application Guide
  • Status of System
  • Usage Guide
    • SSH Login
    • Data Transfer
    • File Systems
    • MPI, OpenMP build and run workflows
    • Slurm usage
      • Advanced options
      • Multiple programs on one node
  • Compute partitions
  • Software
  • FAQ
  • NHR Community
  • Contact

    You‘re viewing this with anonymous access, so some content might be blocked.
    /
    Advanced options

    Advanced options

    Sept. 12, 2024

    Content

    Slurm offers a lot of options for job allocation, process placement, job dependencies and arrays and much more. We cannot exhaustively cover all topics here. As mentioned at the top of the page, please consult the official slurm documentation and the man pages for an in depth description of all parameters.

    +48 Hour Jobs & beyond

    We prefer short job times and enable +48h jobs only as a last resort. Minimal walltimes have several advantages regarding efficient queuing (backfilling). If all jobs were +48h, two days before each maintenance our machine would be empty...

    Most Slurm partition CPU CLX have a maximal wall time of 12 hours. In contrast, 48 hours is offered per default by all shared partitions.

    During normal office hours, one can request the extension of the wall time of any running job (mail with your user and job ID to support@nhr.zib.de
    ). Alternatively - also per mail request (including user and project ID) - permanent access to run 48h jobs on all partitions can be granted (and be used by adding e.g. #SBATCH -q 48h). Other Quality of Service levels for even longer runtimes can also be requested, but have additional restrictions regarding job size (number of nodes).
    However, we recommend permanent access to the long running QoS only as a last resort. We do not guarantee to refund your compute time on the long running QoS if something fails. You should exploit all possibilities to parallelize/speed up your code or make it restartable (see also below), first.

    Dependent & Restartable Jobs - How to pass the wall time limit

    If your simulation is restartable, it might be handy to automatically trigger a follow-up job. Simply provide the ID of the previous job as an additional sbatch argument:

    Example of a job chain with 3 parts
    # submit a first job, extract job id
    jobid=$(sbatch --parsable job1.sbatch)
    
    # submit a second job with dependency: starts only if the previous job terminates successfully)
    jobid=$(sbatch --parsable --dependency=afterok:$jobid job2.sbatch)
    
    # submit a third job with dependency: starts only if the previous job terminates successfully)
    jobid=$(sbatch --parsable --dependency=afterok:$jobid job3.sbatch)

    squeue -l -u $USER will mark all your dependent jobs with "(Dependency)" in the column "NODELIST(REASON)".

    Please note: As soon as a follow-up jobscript (sbatch file) is submitted, you can not change its content any more. Lines starting with #SBATCH will be evaluated immediately. The remaining content of the jobscript is evaluated as soon as the dependency condition is fulfilled and compute nodes are available. Besides afterok there exist other dependency types (sbatch man page).

    Job Arrays

    Job arrays are the preferred way to submit many similar job. Jobs, for instance, if you need to run the same program on a number of input files, or with different settings or run them with a range of parameters. Arrays are created with the -a start-finish sbatch parameter. E.g. sbatch -a 0-19 will create 20 jobs indexed from 0 to 19. There are different ways to index the arrays, which are described below.

    The behavior of the jobs can then be tied to Slurm Environment variables, which tell the program, which part of the array they are.

    Job Array Indexing, Stepsize and more

    Slurm supports a number of ways to set up the indexing in job arrays.

    • Range: -a 0-5

    • Multiple values: -a 1,5,12

    • Step size: -a 0-5:2 (same as -a 0,2,4)

    • Combined: -a 0-5:2,20 (same as -a 0,2,4,20)

    Additionally, you can limit the number of simultaneously running jobs with the %x parameter in there:

    • -a 0-11%4 only four jobs at once

    • -a 0-11%1 run all jobs sequentially

    • -a 0-5:2,20%2 everything combined. Run IDs 0,2,4,20, but only two at a time.

    You can read everything on array indexing in the sbatch man page.

    Slurm Array Environment Variables

    The most used environment variable in Slurm arrays is $SLURM_ARRAY_TASK_ID. This contains the index of the job in the array and is different in every Job of the array. Other variables are:

    SLURM_ARRAY_TASK_COUNT
        Total number of tasks in a array. 
    SLURM_ARRAY_TASK_ID
        Job array ID (index) number. 
    SLURM_ARRAY_TASK_MAX
        Job array's maximum ID (index) number. 
    SLURM_ARRAY_TASK_MIN
        Job array's minimum ID (index) number. 
    SLURM_ARRAY_TASK_STEP
        Job array's index step size. 
    SLURM_ARRAY_JOB_ID
        Job array's master job ID number. 

    Example Job Array

    This is an example of a a job array, creates a job for every file ending in “.inp” in the current workding directory

    Example of an array job Quelle erweitern
    #!/bin/bash
    #SBATCH -p standard96
    #SBATCH -t 12:00:00 #one day
    #SBATCH -N 16
    #SBATCH --tasks-per-node 96
    # insert X as the number of .inp files you have -1 (since bash arrays start counting from 0)
    # ls *.inp | wc -l
    #SBATCH -a 0-X
    #SBATCH -o arrayjob-%A_%a #"%A" is replaced by the job ID and "%a" with the array index.
     
    #for safety reasons
    shopt -s nullglob
    #create a bash array with all files
    arr=(./*.inp)
     
    #put your command here. This just runs the fictional "big_computation" program with one of the files as input
    ./big_computation ${arr[$SLURM_ARRAY_TASK_ID]}
    

    In this case, you have to get the number of files beforehand (fill in the X). You can also automatically do that by removing the #SBATCH -a line and adding that information when submitting the job:

    sbatch -a 0-$(($(ls ./*.inp | wc -l)-1)) jobarray.sh

    The part in the parenthesis just uses ls to output all .inp files, counts them with wc and then subtracts 1, since bash arrays start counting at 0.


    , multiple selections available, Use left or right arrow keys to navigate selected items
    slurm
    batch
    partitions
    lsf
    moab
    scheduler
    jobscript
    job
    array
    dependency
    {"serverDuration": 32, "requestCorrelationId": "9fbac7cb9105409db0ea59d4df3e328a"}