{"type":"doc","content":[{"type":"paragraph","content":[{"text":"Content","type":"text","marks":[{"type":"strong"}]}]},{"type":"extension","attrs":{"extensionType":"com.atlassian.confluence.macro.core","extensionKey":"toc","parameters":{"macroParams":{},"macroMetadata":{"macroId":{"value":"3b7d67e6-d89a-473e-85dc-214f703cba39"},"schemaVersion":{"value":"1"},"title":"Inhalt"}}}},{"type":"heading","attrs":{"level":1},"content":[{"text":"Slurm partitions","type":"text"}]},{"type":"paragraph","content":[{"text":"To match your job requirements with the hardware, you choose among the","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Slurm partitions of the ","type":"text"},{"text":"Compute partitions","type":"text","marks":[{"type":"link","attrs":{"href":"https://nhr-zib.atlassian.net/wiki/spaces/PUB/pages/430577"}}]},{"text":" which are linked to their","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"charge rates on the page ","type":"text"},{"text":"Accounting","type":"text","marks":[{"type":"link","attrs":{"href":"https://nhr-zib.atlassian.net/wiki/spaces/PUB/pages/426811"}}]},{"text":".","type":"text"}]}]}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Important Slurm commands","type":"text"}]},{"type":"paragraph","content":[{"text":"The commands normally used for job control and management are","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Job submission: ","type":"text"},{"type":"hardBreak"},{"text":"$ sbatch ","type":"text","marks":[{"type":"code"}]},{"text":" ","type":"text"},{"type":"hardBreak"},{"text":"$ srun ","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Status of a specific job: ","type":"text"},{"type":"hardBreak"},{"text":"$ squeue -j ","type":"text","marks":[{"type":"code"}]},{"text":" ","type":"text"},{"type":"hardBreak"},{"text":"$ scontrol show job ","type":"text","marks":[{"type":"code"}]},{"text":" (for full job information, even after the job finished)","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Job cancellation: ","type":"text"},{"type":"hardBreak"},{"text":"$ scancel ","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"$ scancel -i -u $USER ","type":"text","marks":[{"type":"code"}]},{"text":"(cancel all your jobs, asking at each job)","type":"text"},{"type":"hardBreak"},{"text":"$ scancel -9 ","type":"text","marks":[{"type":"code"}]},{"text":"(send ","type":"text"},{"text":"SIGKILL","type":"text","marks":[{"type":"code"}]},{"text":" instead of ","type":"text"},{"text":"SIGTERM)","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Job overview: ","type":"text"},{"type":"hardBreak"},{"text":"$ squeue","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"$ squeue -l --me","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Estimated job start: ","type":"text"},{"type":"hardBreak"},{"text":"$ squeue --start -j ","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Workload overview of the whole system:","type":"text"},{"type":"hardBreak"},{"text":"$ sinfo","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"$ sinfo --format=\"%25C %A\"","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"$ squeue -l","type":"text","marks":[{"type":"code"}]}]}]}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Job scripts","type":"text"}]},{"type":"paragraph","content":[{"text":"A job script can be any script that contains special instruction for Slurm. Most commonly used forms are shell scripts, such as ","type":"text"},{"text":"bash","type":"text","marks":[{"type":"code"}]},{"text":" or plain ","type":"text"},{"text":"sh","type":"text","marks":[{"type":"code"}]},{"text":". But other scripting languages (e.g. Python, Perl, R) are also possible.","type":"text"}]},{"type":"heading","attrs":{"level":5},"content":[{"text":"Example Batch Script","type":"text","marks":[{"type":"strong"}]}]},{"type":"codeBlock","attrs":{"language":"shell"},"content":[{"text":"#!/bin/bash\n\n#SBATCH -p cpu-clx:test\n#SBATCH -N 4\n#SBATCH -t 06:00:00\n\nmodule load impi\nmpirun my-mpi-parallel-binary","type":"text"}]},{"type":"paragraph","content":[{"text":"Job scripts need to have a shebang line at the top, followed by the ","type":"text"},{"text":"#SBATCH","type":"text","marks":[{"type":"code"}]},{"text":" options. These ","type":"text"},{"text":"#SBATCH","type":"text","marks":[{"type":"code"}]},{"text":" comments have to be at the top, as Slurm stops scanning for them after the first non-comment non-whitespace line (e.g., a command or a declaration of a variable).","type":"text"}]},{"type":"paragraph","content":[{"text":"More examples can be found at ","type":"text"},{"text":"Examples and Recipes","type":"text","marks":[{"type":"link","attrs":{"href":"https://nhr-zib.atlassian.net/wiki/spaces/PUB/pages/427021"}}]},{"text":".","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Parameters","type":"text"}]},{"type":"table","attrs":{"layout":"align-start","width":753.0},"content":[{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1,"colwidth":[163.0]},"content":[{"type":"paragraph","content":[{"text":"Parameter","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1,"colwidth":[205.0]},"content":[{"type":"paragraph","content":[{"text":"SBATCH flag","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1,"colwidth":[385.0]},"content":[{"type":"paragraph","content":[{"text":"Comment","type":"text","marks":[{"type":"strong"}]}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[163.0]},"content":[{"type":"paragraph","content":[{"text":"# nodes","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[205.0]},"content":[{"type":"paragraph","content":[{"text":"-N <#>","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[385.0]},"content":[{"type":"paragraph"}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[163.0]},"content":[{"type":"paragraph","content":[{"text":"# tasks","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[205.0]},"content":[{"type":"paragraph","content":[{"text":"-n <#>","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[385.0]},"content":[{"type":"paragraph"}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[163.0]},"content":[{"type":"paragraph","content":[{"text":"# tasks per node","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[205.0]},"content":[{"type":"paragraph","content":[{"text":"--tasks-per-node <#>","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[385.0]},"content":[{"type":"paragraph","content":[{"text":"different defaults between mpirun and srun","type":"text"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[163.0]},"content":[{"type":"paragraph","content":[{"text":"Slurm partition","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[205.0]},"content":[{"type":"paragraph","content":[{"text":"-p ","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[385.0]},"content":[{"type":"paragraph","content":[{"text":"e.g. cpu-clx, overview: ","type":"text"},{"text":"Slurm partition CPU CLX","type":"text","marks":[{"type":"link","attrs":{"href":"https://nhr-zib.atlassian.net/wiki/spaces/PUB/pages/428826"}}]}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[163.0]},"content":[{"type":"paragraph","content":[{"text":"# CPU cores per task","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[205.0]},"content":[{"type":"paragraph","content":[{"text":"-c <#>","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[385.0]},"content":[{"type":"paragraph","content":[{"text":"interesting for OpenMP/Hybrid jobs","type":"text"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[163.0]},"content":[{"type":"paragraph","content":[{"text":"job walltime limit","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[205.0]},"content":[{"type":"paragraph","content":[{"text":"-t hh:mm:ss","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[385.0]},"content":[{"type":"paragraph","content":[{"text":"realistic estimates for best scheduling efficiency","type":"text"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[163.0]},"content":[{"type":"paragraph","content":[{"text":"e-mail notification","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[205.0]},"content":[{"type":"paragraph","content":[{"text":"--mail-type=ALL","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[385.0]},"content":[{"type":"paragraph","content":[{"text":"see sbatch manpage for notification types","type":"text"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[163.0]},"content":[{"type":"paragraph","content":[{"text":"project account","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[205.0]},"content":[{"type":"paragraph","content":[{"text":"-A ","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[385.0]},"content":[{"type":"paragraph","content":[{"text":"project account to be charged","type":"text"}]}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Job runtime limits","type":"text"}]},{"type":"paragraph","content":[{"text":"Maximum runtimes of jobs, also known as walltime limits, are defined for each partition and can be viewed either on the command line using ","type":"text"},{"text":"sinfo","type":"text","marks":[{"type":"code"}]},{"text":" , or ","type":"text"},{"text":"here","type":"text","marks":[{"type":"link","attrs":{"href":"https://nhr-zib.atlassian.net/wiki/spaces/PUB/pages/428826"}}]},{"text":". There is no minimum runtime, but a runtime of at least 1 hour is encouraged. Large amounts of small and short jobs can trigger problems in our accounting system. Occasional short jobs are fine, but if you submit large amounts of jobs that finish (or crash) quickly, we might have to intervene and temporarily suspend your account. If you have lots of smaller workloads, please consider combining them into a single job that runs for at least 1 hour.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Project account selection","type":"text"}]},{"type":"paragraph","content":[{"text":"Each job is running under the name of the user who submitted the job to the system. The user name is not to be confused with the project account which a job gets charged to. ","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"For users new to the system, the default project account is their Test Project. It reads the same as the user name.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Users with membership in one or more compute projects need to decide which project account a job will be charged to.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Decision can be taken via the sbatch flags ","type":"text"},{"text":"--account=my-project-ID ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}},{"type":"strong"}]},{"text":"or, in short, ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}}]},{"text":"-A my-project-ID","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}},{"type":"strong"}]},{"text":".","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"If no explicit decision is provided, the default applies which can be changed by the user in the ","type":"text"},{"text":"Portal","type":"text","marks":[{"type":"link","attrs":{"href":"https://portal.nhr.zib.de"}}]},{"text":" (section ","type":"text"},{"text":"User Data","type":"text","marks":[{"type":"em"}]},{"text":").","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"After job submission, Slurm checks the balance of the job's project account. When a negative balance is returned by the Slurm database, the job will not be scheduled. In this case, ","type":"text"},{"text":"squeue","type":"text","marks":[{"type":"code"}]},{"text":" displays a corresponding message (","type":"text"},{"text":"AccountOutOfComputeTime","type":"text","marks":[{"type":"code"}]},{"text":") to the user.","type":"text"}]},{"type":"heading","attrs":{"level":5},"content":[{"text":"Example: Negative balance of the project account","type":"text","marks":[{"type":"strong"}]}]},{"type":"codeBlock","content":[{"text":"$ squeue --me\nJOBID PARTITION NAME USER ACCOUNT STATE TIME NODES NODELIST(REASON)\n ... ... ... ... ... PENDING 0:00 ... (AccountOutOfComputeTime)\n","type":"text"}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Interactive jobs ","type":"text"},{"type":"inlineExtension","attrs":{"extensionType":"com.atlassian.confluence.macro.core","extensionKey":"anchor","parameters":{"macroParams":{"":{"value":"interactive_jobs"},"legacyAnchorId":{"value":"Slurmusage-interactive_jobs"}},"macroMetadata":{"macroId":{"value":"973c50be-e1a9-4ca4-9429-2263bc950cef"},"schemaVersion":{"value":"1"},"title":"Anker"}}}}]},{"type":"paragraph","content":[{"text":"To use compute resources interactively, e.g. to follow the execution of MPI programs, the following steps are required. Note that non-interactive batch jobs via job scripts (see below) are the primary way of using the compute resources.","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"A resource allocation for interactive usage has to be requested first with the ","type":"text"},{"text":"salloc","type":"text","marks":[{"type":"code"}]},{"text":" command which should also include your resource requirements.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"When ","type":"text"},{"text":"salloc","type":"text","marks":[{"type":"code"}]},{"text":" successfully allocated the requested resources, you have to issue an additional ","type":"text"},{"text":"srun","type":"text","marks":[{"type":"code"}]},{"text":" command if you want to work directly on one of the allocated nodes (see example below)","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Afterwards, ","type":"text"},{"text":"srun","type":"text","marks":[{"type":"code"}]},{"text":" or MPI launcher commands (","type":"text"},{"text":"mpirun","type":"text","marks":[{"type":"code"}]},{"text":", ","type":"text"},{"text":"mpiexec)","type":"text","marks":[{"type":"code"}]},{"text":" can be used to start parallel programs.","type":"text"}]}]}]},{"type":"codeBlock","attrs":{"language":"plaintext"},"content":[{"text":"blogin1:~ $ salloc -t 00:10:00 -p cpu-clx:test -N2 --tasks-per-node 24\nsalloc: Granted job allocation [...]\nsalloc: Waiting for resource configuration\nsalloc: Nodes bcn[1001,1003] are ready for job\n# To get a shell on one of the allocated nodes\nblogin1:~ $ srun --pty --interactive --preserve-env ${SHELL}\nbcn1001:~ $ srun hostname | sort | uniq -c\n 24 bcn1001\n 24 bcn1003\nbcn1001:~ $ exit\nblogin1:~ $ exit\nsalloc: Relinquishing job allocation [...]","type":"text"}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Using shared nodes","type":"text"}]},{"type":"paragraph","content":[{"text":"In some of our Slurm partitions, compute nodes are allocated in shared mode, so that multiple jobs of a partition requesting only parts of a node can run concurrently on the same nodes. You can explicitly request less CPU cores (implies less memory, too) or less memory per node than available, and take care that memory limits are not exceeded.","type":"text"}]},{"type":"expand","attrs":{"title":"Example for a post-processing job script using 10 cores."},"content":[{"type":"paragraph","content":[{"text":"This job's memory usage should not exceed a fraction of 10/96 of the total memory. ","type":"text"}]},{"type":"codeBlock","content":[{"text":"#!/bin/bash\n#SBATCH -p cpu-clx:huge\n#SBATCH -t 1-0 # one day\n#SBATCH -n 10\n#SBATCH -N 1\n\npython postprocessing.py","type":"text"}]}]},{"type":"paragraph","content":[{"type":"emoji","attrs":{"id":"atlassian-warning","text":":warning:","shortName":":warning:"}},{"text":" Please remember that most of Lise's compute nodes belong to Slurm partitions where nodes are allocated exclusively to a job, meaning nodes are not shared between jobs (see ","type":"text"},{"text":"Accounting","type":"text","marks":[{"type":"link","attrs":{"href":"https://nhr-zib.atlassian.net/wiki/spaces/PUB/pages/426811"}}]},{"text":").","type":"text"}]}],"version":1}