Fat Tree OPA network of CLX partition

Notes:

Each of the two symmetrically connected director switches (Intel® Omni-Path Director Class Switch 100 Series 24 Slot Base 1MM) forms a FT(96,3) network.

Each black dot of the gray racks in the middle represents an edge switch connected to up to 24 nodes. If you want to know, where your compute nodes are attached to (rack number [1-14] and edge switch [0-3]) call: scontrol show topology

The omitted rack pairs 2-3, 4-5, 6-7, 8-9 hold the same node types and are connected in the same manner as rack 1-2.

The colored OPA lane numbers ##x ## encode: 1st the number of different switches connected to (each of these connections is visualized by one line), 2nd the number of ports utilized per connection (100 Gbps per OPA Host Fabric Interface Adapter 100 Series PCIe-v3 x16).

The racks 1-13 are fully equipped with the maximum of 96 nodes; rack 14 has only 64 nodes (24 bcn and 40 bdaos).

All 16 leafs are linked each with all their 48 "up-ports" to all 12 spines (4 OPA lanes per individual leaf-spine connection). Leafs 1-14 are connected with all their 48 "down-ports" to edge switches. In contrast, half of the "down-ports" of leafs 15-16 are unused.

The 12 spines each have 8x4 free ports, allowing a future upgrade of 8 more leafs (maximum of 24 leafs).