Atlassian uses cookies to improve your browsing experience, perform analytics and research, and conduct advertising. Accept all cookies to indicate that you agree to our use of cookies on your device. Atlassian cookies and tracking notice, (opens new window)
User Manual

User Manual
Results will update as you type.
  • Application Guide
  • Status of System
  • Usage Guide
  • Compute partitions
    • CPU CLX partition
      • Workflow CPU CLX
      • Slurm partition CPU CLX
      • Examples and Recipes
      • Fat Tree OPA network of CLX partition
      • Operating system migration from CentOS to Rocky Linux
    • CPU Genoa partition
    • GPU A100 partition
    • GPU PVC partition
    • Next-Gen Technology Pool
  • Software
  • FAQ
  • NHR Community
  • Contact

    You‘re viewing this with anonymous access, so some content might be blocked.
    /
    Fat Tree OPA network of CLX partition

    Fat Tree OPA network of CLX partition

    Feb. 10, 2025

           

    Notes:

    Each of the two symmetrically connected director switches (Intel® Omni-Path Director Class Switch 100 Series 24 Slot Base 1MM) forms a FT(96,3) network.

    Each black dot of the gray racks in the middle represents an edge switch connected to up to 24 nodes. If you want to know, where your compute nodes are attached to (rack number [1-14] and edge switch [0-3]) call: scontrol show topology

    The omitted rack pairs 2-3, 4-5, 6-7, 8-9 hold the same node types and are connected in the same manner as rack 1-2.

    The colored OPA lane numbers ##x ## encode: 1st the number of different switches connected to (each of these connections is visualized by one line), 2nd the number of ports utilized per connection (100 Gbps per OPA Host Fabric Interface Adapter 100 Series PCIe-v3 x16).

    The racks 1-13 are fully equipped with the maximum of 96 nodes; rack 14 has only 64 nodes (24 bcn and 40 bdaos).

    All 16 leafs are linked each with all their 48 "up-ports" to all 12 spines (4 OPA lanes per individual leaf-spine connection). Leafs 1-14 are connected with all their 48 "down-ports" to edge switches. In contrast, half of the "down-ports" of leafs 15-16 are unused.

    The 12 spines each have 8x4 free ports, allowing a future upgrade of 8 more leafs (maximum of 24 leafs).




    , multiple selections available,
    {"serverDuration": 9, "requestCorrelationId": "3e1b48e38d7d416c95fb9fc1d36e697d"}