...
Especially large shared file IO patterns will benefit from striping. Up to 28 OSTs on Lise and up to 100 OST's on Emmy can be used, recommended are up to 8 OSTs for Lise and 32 OSTs on Emmy. We have preconfigured a progressive file layout (PFL), which sets an automatic striping based on the file size.
Access: create a new directory in $WORK and set lfs setstripe -c <stripsize> <dir>
...
Some Compute Nodes are installed with local SSD storage up to 2 TB on Lise and 400 480 GB or 1TB (depending on the node) on Emmy.
...
| Lise: SSD | Lise: CAS | Emmy: SSD | |
|---|---|---|---|
| Access | via partition: using | via partition: using | via partition: using |
| Type and size | Intel NVMe SSD DC P4511 (2 TB) | Intel NVMe SSD DC P4511 (2 TB) using Intel Optane SSD DC P4801X (200 GB) as write-trough cache | Intel S-ATA SSD DC S4500 (400 480 GB) Intel NVMe SSD DC P4511 (1TB) |
FastIO
...
WORK is extended with 4 (Lise)/8 (Emmy) additional OST's using NVMe SSDs to accelerate heavy (random) IO-demands. To accelerate specific IO-demands further striping for up to 4/8 OSTs is available.
Access:
Lise: create a new directory in $WORK and set lfs setstripe -p flash <dir>
Emmy: Using either $SHARED_SSD_TMPDIR for a job specific folder (liek $LOCAL_TMPDIR) oder set the storage pool ddn_ssd for a new directory like for Lise (there the SSD pool is called flash).
Size:
Lise: 55 TiB - quoted
Emmy: 120 TiB - quoted
IME - Emmy only
DDN Infinite Memory Engine (IME) based Burst Buffers is a fast NVMe Solid State Disk (SSD) based data tier. Like Lise's FastIO this is especially helpful for random IO (especially random read) on large files.
...
IME servers are currently available for use in EMMY.
Access: ask support@hlrn.de for accessadd the option --constraint=ime to your jobscript and use $IME_TMPDIR for a job specific folder or /ime/usr/$user for more permanent usage.
Size: 48 300 TiB
Finding the right File System
...
If you have a significant amount of node-local IO which is not needed to be accessed after job end and will be smaller than 2 TB on Lise and 400 GB/1 TB (depending on the node) on Emmy we recommend using $LOCAL_TMPDIR. Depending on your IO pattern this may accelerate IO to up to 100%.
...
Global IO is defined as shared IO which will be able to be accessed from multiple nodes at the same time and will be persistent after job end.
Especially random IO on small files will be accelerated up to 200% using FastIO on Lise or IME on Emmy.
...