Physical Topology — infra.bea.ai

Physical topology determines what parallelism strategies are possible. NVLink cables are short (within rack), InfiniBand cables can span rows, and the physical arrangement of servers, switches, and cooling determines the cluster’s communication topology.

1. Cable Types and Constraints

Cable	Bandwidth	Max length	Use case
NVLink (copper)	900 GB/s (H100)	0.5-2m	Intra-node/intra-rack GPU↔GPU
InfiniBand NDR (copper)	400 Gbps	2-3m	Intra-rack
InfiniBand NDR (AOC)	400 Gbps	10-100m	Inter-rack
InfiniBand NDR (fiber)	400 Gbps	100m-10km	Inter-row, inter-hall
Ethernet 100G/400G (fiber)	100-400 Gbps	100m+	Management, storage
Power cables	N/A	Per PDU spec	Power distribution
Liquid cooling hoses	N/A	Per CDU spec	Coolant distribution

Critical: NVLink cable length limits GPU-to-GPU connectivity to within a rack or adjacent racks. This is why NVLink domains (like NVL72) are contained within a single rack.

2. Reference: DGX SuperPOD (H100)

DGX SuperPOD: 32 DGX H100 nodes = 256 GPUs

Physical layout (4 racks):
  Rack 1: 8× DGX H100 + 1× leaf switch
  Rack 2: 8× DGX H100 + 1× leaf switch
  Rack 3: 8× DGX H100 + 1× leaf switch
  Rack 4: 8× DGX H100 + 1× leaf switch

Networking:
  Intra-node: NVLink (900 GB/s all-to-all, 8 GPUs)
  Inter-node: 8× InfiniBand NDR400 per node
  Total: 256 GPUs, 8× IB per node = 256 IB cables to spine switches
  
  Spine switches: 2-4 IB switches (in separate switch rack)
  Topology: Rail-optimized (each GPU rank connects to same spine)

Cable count per SuperPOD:
  NVLink: 18 cables × 32 nodes = 576 NVLink cables (within nodes)
  IB: 8 cables × 32 nodes = 256 IB cables (to spine switches)
  Power: 2 × 32 = 64 power cables (redundant feeds)
  Ethernet: 32 management cables
  Liquid cooling: 32 supply + 32 return hoses
  ─────────────────────────────
  Total: ~960 cables + 64 hoses per SuperPOD

3. Reference: GB200 NVL72 Rack

Single rack: 72 GPUs + 36 CPUs in one liquid-cooled rack

Internal layout:
  ├── 18× compute trays (each: 4 GPUs + 2 Grace CPUs)
  ├── 9× NVSwitch trays (connecting all 72 GPUs via NVLink)
  ├── Liquid cooling manifold (top/bottom)
  └── Rack-level networking (CX-8 NICs to external fabric)

Key properties:
  - All 72 GPUs: NVLink connected (5th gen, 1.8 TB/s bisection)
  - Single memory domain: 72 × 192GB = 13.8 TB unified memory
  - External: 72× CX-8 IB NICs (to inter-rack fabric)
  - Power: ~120 kW per rack
  - Cooling: 100% DLC, ~70 L/min coolant flow

This entire rack is ONE NVLink domain — can run TP=72 if needed

4. Rack Layout Principles

Server Placement

Top of rack:    Leaf switches (easy cable access up)
Upper rows:     GPU servers (hot air rises, close to switches)
Middle rows:    GPU servers
Lower rows:     Storage servers (heavier, denser)
Bottom:         PDU connections, liquid cooling manifold

Cable management:
  Front: Power cables, management Ethernet
  Rear:  NVLink (short, within rack), InfiniBand (to ToR switch)
  Overhead: Inter-rack IB cables via cable trays

Row Layout

Row A (compute):
  [Rack 1: 8× DGX] [Rack 2: 8× DGX] [Rack 3: 8× DGX] [Rack 4: 8× DGX]

Row B (switches):
  [Rack 5: IB spine switches] [Rack 6: IB spine switches]

Row C (compute):
  [Rack 7: 8× DGX] [Rack 8: 8× DGX] [Rack 9: 8× DGX] [Rack 10: 8× DGX]

Row D (infrastructure):
  [Rack 11: Storage] [Rack 12: CDU] [Rack 13: Management]

5. Cable Management at Scale

At 1000+ GPUs, cable management becomes a major engineering challenge:

Scale	IB cables	Challenges
8 GPUs (1 node)	8	Trivial
256 GPUs (SuperPOD)	~260	Manageable with cable trays
1024 GPUs (4 SuperPODs)	~1,040	Requires structured cabling plant
10,000 GPUs	~10,000+	Professional cable contractor essential

Best Practices

Practice	Rationale
Color-code by rail	Rail 0 = blue, Rail 1 = green, etc. — faster debugging
Label both ends	Cable ID visible at both switch and server end
Maintain cable spreadsheet	Map: server port → cable ID → switch port
Use structured cabling trays	Overhead trays per rail, not random bundles
Leave 20% spare capacity	For future expansion without re-cabling
Test every cable at install	Verify link speed before production use

6. Topology Verification

After physical installation, verify the logical topology matches the intended design:

# Verify InfiniBand fabric topology
ibnetdiscover > /tmp/topology.txt

# Check all links are at expected speed (NDR = 400 Gbps)
ibdiagnet --speed_check

# Verify NVLink connectivity within each node
nvidia-smi topo -m
# Expected: NVLink between all GPU pairs in same node

# Check for link errors
ibdiagnet --counter_check
# Any non-zero error counters → investigate cable/port

Self-Check

Calculate total cable count for a 256-GPU SuperPOD
Explain why NVLink domains are limited to within a rack
Design a rack layout for 4 DGX H100 servers with leaf switches
Describe the GB200 NVL72 internal architecture and cooling
Set up post-install topology verification for an InfiniBand cluster

References

NVIDIA DGX SuperPOD Design Guide
NVIDIA GB200 NVL72 Architecture
For network topology concepts, see Data Center Network Design
For reading topology diagrams, see Reading Cluster Topology Diagrams