The NHR@ZIB Next-Generation Technology Pool of systems serves for the exploration and evaluation of new technologies for HPC and AI workloads. NHR@ZIB has a strong partnership with various vendors sharing the common goal to give experienced users insights and hands-on to future technologies.

Systems

NextSilicon Maverick-2

Hostnames: maverick[1-2]

One dataflow engine hardware accelerator card per server (accelerator cards provided by ParTec).

Open

Server Board Layout

Hardware Configuration

Software

• OS: Rocky Linux 9

Next Profiler GUI

After starting nextsystemd, the web interface is available on port 3001. Use ssh port forwarding to make it reachable from your laptop. Then, open http://127.0.0.1:3001/ in your local browser.

NVIDIA Grace CPU Superchip Servers

Hostnames: grace[1-2]

Hardware

Hosts with Intel Optane Memory

Hostnames: apass{1,2}

The two systems apass{1,2}  are equipped with Intel Optane Memory components (first generation Apache Pass): Storage Class Memory modules (SCM/NVRAM) and SSDs. The main difference between the two hosts is the memory capacity.

The Optane Memory, i.e. each of the systems, can be configured in two (three) modes:

• Memory Mode: The Optane Memory is exposed as RAM to the OS. Although the hardware technology is different to DRAM, this mode allows transparent usage of the SCM memory. The main benefits are the gained memory capacity and easy usage (no modification of applications required). The DRAM memory acts as cache for Optane Memory. In this mode, the SCM is effectively not persistent.

• AppDirect Mode: The Optane Memory is exposed as block device(s) to the OS (usually as /dev/pmemX, depending on actual configuration). On such block devices, a file system can be created which should support the direct access (DAX) option. This allows to map data on the persistent Optane Memory into an application's virtual address space while avoiding the OS' page cache. Thus, direct media access to the SCM is possible with load/store operations. Note that data in the Optane Memory is effectively cleared when the mode is changed. Therefore, data on /dev/pmemX should be considered as ephemeral.

• (Mixed/Hybrid Mode: The System can also be configured to provide a portion of the Optane capacity for the Memory Mode and another portion for AppDirect mode.)

By default: apass1 is configured in Memory Mode while apass2 is configured in AppDirect Mode. If you need a different configuration contact S. Christgau @Steffen Christgau (Unlicensed) . The mount points for the persistent memory are usually /mnt/pmemX. X often matches the NUMA domain of the socket/processor the memory is attached to. To be sure run lstopo from the hwloc environment module. Not every pmem device might be mounted or accessible if a system is in AppDirect mode because other software (DAOS, e.g.) may exclusively grab a device. Check the output of mount to find mount points of /dev/pmemX.

The login message (message of the day) displays the mode in which the system is currently running in. You can also check the CurrentVolatileMode property in the /var/run/optane/state file. As a further simple check for the given mode, you can run free -h. If the total memory capacity is around or larger than 3 TB the system is in memory mode. Further, if /dev/pmem[01] exists, the AppDirect (or Mixed/Hybrid) mode is in effect.

Hardware

Software

• OS: Rocky Linux 8

• more recent software (compilers, libraries, utilities) are available via environment modules (module avail).

Open

click to enlarge

Pic 1: Server Board Layout

NEC SX-Aurora TSUBASA A300-8

Hostname: aurora

Hardware Configuration

Software

• OS: CentOS 7.9

• VE OS: 2.4.3

• NEC Compiler Suite: NCC 3.3.1 

Documentation

• Overview (NEC website), Introduction of SX-Aurora TSUBASA for ISC21 (PDF slides)

• NEC SX-Aurora TSUBASA Documentation (2021-12-24)

• SX-Aurora Vector Engine (NEC blog: https://sx-aurora.github.io)

Open

click to enlarge

Pic 2: Aurora Server with 8 VE's

3rd Gen Intel Xeon Cooper Lake

Hostname: cpl

Cooper Lake is Intel's codename for the third-generation of their Xeon scalable processors, developed as the successor to Cascade Lake.

Improvements:

• New bfloat16 instruction

• Support for up to 12 DIMMs of DDR4 memory per CPU socket

Hardware Configuration

Software

• OS: CentOS Stream 8

Intel Xeon Ice Lake

Hostname: icl

Hardware Configuration

Software

• OS: CentOS Stream 8

Open

Pic 3: Server Board Layout

Access

To gain access to the Next-Generation Technology Pool, contact support@nhr.zib.de. Please give a short description of your intention and the system you intend to use.

Use Slurm on the NGT login node to access individual NGT systems.

The NGT login node is "login-ngt", reachable using ssh via our public login nodes "blogin.nhr.zib.de" (replace USERNAME with your NHR@ZIB account name):

Make use of the ssh-agent to avoid repeated prompts for the passphrase (ALL keys used to access the NHR@ZIB must have a passphrase). Run "ssh-agent" to start the agent and load your default key. Or, if your ssh key is in ~/.ssh/id_rsa_nhr, run:

With a suitable ssh config, you can jump to the NGT login node using one simple command:

The ssh config in ~/.ssh/config looks like this (replace USERNAME with your NHR@ZIB account name):

Unused compute nodes are shut down. Slurm will start nodes when needed. Depending on the node this takes 2..5 minutes.

Use sinfo to query the node status. In the following example, "icl" is up an running, "cpl" is powered down to save energy (indicated by the "~" mark at the end):

To start an interactive session on a compute node, use srun.

Alternatively, you can use "salloc" to start and allocate a node:

When a node is up, direct ssh access is still possible, but needs login-ngt as jump host. An example ssh-config (for node "cpl") is:

Software

For some of the systems, e.g. the Nvidia Grace nodes, additional software is available in environment modules which need to be made available first using a provided shell script.