...
HLRN provides tailored WORK filesystem file system for improved IO throughput for of IO intense job workloads.
Filesystem Types
Default Lustre (WORK)
This WORK is the default shared Filesystem file system for all jobs and can be accessed using the $WORK variable. WORK is accessible for all users.
WORK consists of 8 Metadata Targets (MDT's) with NVMe SSDs and 28 Object Storage Targets (OST's) on Lise and 96 OST's on Emmy. Both using rotational classical hard drives.
Access: $WORK
Size: 8 PiB quoted
Special File System Types
Lustre with striping (WORK)
...
Access: create a directory with striping using "lfs setstripe -c <stripsize> <dir>"
Size: 8 PiB like $WORKWORK
Local SSDs
Some Compute Nodes have local SSD storage . Files up to 2 TB on Lise and 400 GB on Emmy.
| Info |
|---|
Data on local SSDs can not be shared across nodes and will be deleted after the job |
...
is finished. |
For single node unshared local IO this is the best performing filesystem file system to use.
| Lise: SSD | Lise: CAS | Emmy: SSD | |
|---|---|---|---|
| Access | via queue: using | via queue: using | via queue: using |
| Type and size | Intel NVMe SSD DC P4511 (2 TB) | Intel NVMe SSD DC P4511 (2 TB) using Intel Optane SSD DC P4801X (200 GB) as |
write-trough cache | Intel DC S4500 (400 GB) |
FastIO - Lise only
Additional WORK is extended with 4 additional OST's using NVMe SSDs integrated into WORKto accelerate heavy IO-demands. To accelerate specific IO-demands further striping for up to 4 OSTs is available.
Access: ask support@hlrn.de for for access
Size: 55 TiB - quoted
IME - Emmy only
Göttingen:
DDN Infinite Memory Engine (IME) based Burst Buffers and file system Cache - is a fast data tier between compute nodes and the Lustre WORK file system /scratch - is used as an I/O accelerator for I/O bound problems. .
This helps avoid overload on the system when a program tries to write large amount of data within a short period of time to the global parallel file system.
IME servers consist of Solid State Disk (SSD) that acts as a cache and burst buffer to improve the global file system performance. IME servers are currently available for use in EMMY.random io?!
Access: IME Burst Buffer, File System Cache
Size: 48 TiB
Finding the right
...
File System
If your jobs have a significant IO part we recommend asking your consultant via support@hlrn.de to recommend the right filesystem file system for you.
Local IO
If you have a significant amount of node-local IO which is not needed to be accessed after job end and will be smaller than 2 TB on Lise and 400 GB on Emmy we recommend using $LOCAL_TMPDIR. Depending on your IO pattern this may accelerate IO to up to 100%.
...
Especially random IO on small files will be accelerated up to 200% using FastIO on Lise or IME on Emmy
...
INTERNAL - not public - will be deleted
Recommendation Matrix:
Max performance gain on IO versus default $WORK in brackets.
| small random IO | lots of large IO per process | few large IO accessed from many nodes | unknown IO | |||||
|---|---|---|---|---|---|---|---|---|
| local IO | global IO | local IO | global IO | local IO | global IO | local IO | global IO | |
| Code e.g. | OpenFOAM? | FESOM? | ||||||
| write IO | Local SSDs (+100%) | FastIO stripe=4 (+30%) | Local SSDs (+15%) | $WORK | $WORK | FastIO stripe=4 (+200%) or $WORK stripe=4-8 (+200%) | Local SSDs (+40%) | FastIO stripe=4 (+80%) $WORK stripe=4-8 (+70%) |
| read IO | Local SSDs (+30%) | FastIO (+140%) | Local SSDs (+30%) | FastIO stripe=4 (+20%) | Local SSDs (+35%) | FastIO stripe=4 (+200%) | Local SSDs (+30%) | FastIO stripe=4 (+120%) $WORK stripe=MAX (+90%) |
| balanced IO | Local SSDs (+60%) | FastIO stripe=4 (+90%) | Local SSDs (+25%) | FastIO (+15%) | Local SSDs (+20%) | FastIO stripe=4 (+200%) or $WORK stripe=MAX (+150%) | Local SSDs (+35%) | FastIO stripe=4 (+100%) FastIO (+50%) |
Commands
Striping einrichten
mkdir <dirname>
lfs setstripe -c <count> <dirname>
DoM einrichten
mkdir <dirname>
lfs setstripe -E 64K -L mdt -E -1 -p work.rotational <dirname>
FastIO einrichten
mkdir <dirname>
lfs setstripe -p flash <dirname>
Fastio mit Striping 4 einrichten:
mkdir <dirname>
lfs setstripe -p flash -c 4 <dirname>
Prüfen
lfs getstripe <dirname>
Vorschlag Ankündigung
Subj: additional WORK filesystemsfile systems available
Dear HLRN Users.
to
To achieve betterimproved IO performance HLRN has installed additional file filesystemssystems tailored for dedicatesdedicated IO demands.
LISE has now node-local SSDs using NVMe and Optane to accelerate local IO. Furthermore we have installed a SSD based Lustre called 'FastIO'.
EMMY is upgraded with an IME.....
If your jobs have a significant amount of IO which can be accelerated visit: _________ or contact our support.
Kind Regards
HLRN-Team
-----
TODO:
- Links
- CMDs