Atlassian uses cookies to improve your browsing experience, perform analytics and research, and conduct advertising. Accept all cookies to indicate that you agree to our use of cookies on your device. Atlassian cookies and tracking notice, (opens new window)
User Manual
User Manual
Results will update as you type.
  • Application Guide
  • Status of System
  • Usage Guide
  • Compute partitions
    • CPU CLX partition
    • CPU Genoa partition
    • GPU A100 partition
    • GPU PVC partition
      • PVC AI Tools and Frameworks
      • PVC GPU Programming
      • PVC MPI Usage
      • Interactive PVC
    • Next-Gen Technology Pool
  • Software
  • FAQ
  • NHR Community
  • Contact
    You‘re viewing this with anonymous access, so some content might be blocked.
    /
    GPU PVC partition
    Updated Juni 26, 2024

    GPU PVC partition

    Content

    • 1 Hardware Overview
    • 2 Login Nodes
    • 3 File systems
    • 4 Software and environment modules
    • 5 Slurm Partitions
    • 6 Charge rates

    This partition offers access to eight nodes, each equipped with four Intel Data Center GPU Max 1550 (formerly known as Ponte Vecchio aka PVC). The following offers more details on their usage.

    More PVC related content

    • PVC AI Tools and Frameworks
    • PVC GPU Programming
    • PVC MPI Usage
    • Interactive PVC

    Hardware Overview

    Property

    Login Node

    Compute Nodes

    Property

    Login Node

    Compute Nodes

    Count

    2

    8

    CPU

    2x Intel(R) Xeon(R) Platinum 8480L (Sapphire Rapids; 56 cores; 105 MB L3 cache)

    RAM

    512 GB

    1024 GB (1 TB)

    Local Storage

    2x 1.8 TB SATA SSD

    1x 3.6 TB NVMe drive

    GPUs

    none

    4 x Intel Data Center GPU Max 1550 (128 GB HBM; Xe links with all-to-all topology between GPUs)

    2 per Socket/NUMA domain

    Fabric

    InfiniBand HDR 200 GBit/s (1 HFI per Node; on NUMA domain 0)

    Operating System

    Rocky Linux 8

    Note that a single GPU Max 1550 is comprised of two “tiles” or “stacks” which can be considered as NUMA domains. Depending on your workload you may restrict your application to a tile or make use of a full GPU.

    Login Nodes

    Login to the GPU PVC partition is possible through dedicated login nodes, reachable via SSH under bgilogin.nhr.zib.de:

    $ ssh -i $HOME/.ssh/id_rsa_nhr nhr_username@bgilogin.nhr.zib.de Enter passphrase for key '...': bgilogin1 $

    File systems

    The file systems HOME and WORK on the GPU system are the same as on the CPU system, see Quickstart. Access to compute node local SSD space is provided via the environment variable LOCAL_TMPDIR defined during a SLURM session (batch or interactive job).

    Software and environment modules

    Login and compute nodes of the PVC GPU partition are running under Rocky Linux.

    Software for the PVC GPU partition provided by NHR@ZIB can be found using the module command, see Quickstart.

    bgilogin1 ~ $ module avail ... gcc/13.2.0 intel/2024.0.0 ... bgilogin1 ~ $ module load intel Module for Intel oneAPI Compilers and Libraries (next generation compiler version 2024.0.0) loaded. bgnlogin1 ~ $ module list Currently Loaded Modulefiles: 1) HLRNenv 2) sw.pvc 3) slurm 4) intel/2024.0.0
    bgilogin1 ~ $ module avail ... gcc/13.2.0 intel/2024.0.0 ... bgilogin1 ~ $ module load intel Module for Intel oneAPI Compilers and Libraries (next generation compiler version 2024.0.0) loaded. bgnlogin1 ~ $ module list Currently Loaded Modulefiles: 1) HLRNenv 2) sw.pvc 3) slurm 4) intel/2024.0.0

     Please note the presence of the sw.pvc environment module. When loaded, environment modules are shown for software installed for the PVC GPU partition. This is the default setting on the PVC GPU login and compute nodes.

    When compiling applications for the PVC GPU partition, we recommend to use the PVC GPU login nodes or, in case of really demanding compilations and/or need for the presence of the GPU drivers, the use of a PVC GPU compute node via an interactive SLURM job session.

    Slurm Partitions

    Partition Name

    Nodes

    GPUs per Node

    GPU Hardware

    Description

    Partition Name

    Nodes

    GPUs per Node

    GPU Hardware

    Description

    gpu-pvc

    8

    4

    Intel Data Center GPU Max 1550

    full node exclusive

    Example usage of two nodes (eight GPUs in total). Note that is currently not required to request GPU resources via Slurm for using nodes/GPUs of the PVC partition.

    $ srun --nodes=2 --partition=gpu-pvc example_cmd

    Charge rates

    Charge rates for the slurm partitions you find in Accounting. The PVC partitions are currently available free of charge.

    {"serverDuration": 10, "requestCorrelationId": "8c31fb5dfe7c4e1b9e2b4ef0e38a4371"}