Navigation
🌐 Distributed GPU Systems· 10 min read

InfiniBand

NDR/XDR, multi-rail topology, why 8 NICs per node, rail-optimized vs full fat-tree — the inter-node backbone of AI clusters.

infinibandndrxdrmulti-railfat-treeinter-node

InfiniBand is the inter-node network for most large AI training clusters. While NVLink handles intra-node at 900+ GB/s, InfiniBand connects nodes at 50-100 GB/s per port — and the cluster topology determines how well distributed training scales.

1. InfiniBand Generations

GenerationYearPer-port BWTypical per-nodeUsed by
HDR2019200 Gb/s (25 GB/s)8 ports = 200 GB/sA100 clusters
NDR2022400 Gb/s (50 GB/s)8 ports = 400 GB/sH100 clusters (DGX)
XDR2024800 Gb/s (100 GB/s)8 ports = 800 GB/sB200 clusters

Multi-rail: Each node has 8 NICs (ConnectX-7/8), one per GPU pair. All 8 are used simultaneously for aggregate bandwidth.

2. Multi-Rail Topology

Why 8 NICs?

Multi-Rail Topology

DGX H100 has 8 GPUs and 8 NICs. Each GPU pair is PCIe-adjacent to one NIC:

GPU 0,1 → NIC 0 (ConnectX-7, 400 Gb/s)
GPU 2,3 → NIC 1
GPU 4,5 → NIC 2
GPU 6,7 → NIC 3
GPU 0,1 → NIC 4 (second rail)
GPU 2,3 → NIC 5
GPU 4,5 → NIC 6
GPU 6,7 → NIC 7

Rail-optimized traffic: GPU ii on Node A sends to GPU ii on Node B via the same NIC/rail. This ensures:

  • No PCIe cross-socket traffic
  • Dedicated bandwidth per GPU pair
  • Predictable latency

Aggregate Bandwidth

8 NICs × 50 GB/s = 400 GB/s per node (NDR). For AllReduce of 14 GB gradients:

T=2×14 GB400 GB/s≈70 ms (theoretical)T = \frac{2 \times 14 \text{ GB}}{400 \text{ GB/s}} \approx 70 \text{ ms (theoretical)}

Real-world: ~80-100 ms (protocol overhead, congestion).

3. Network Topology

InfiniBand networks use spine-leaf (Clos) topology for AI clusters. For detailed topology design, oversubscription ratios, and parallelism-to-network mapping, see Data Center Network Design.

Key point for InfiniBand: AI clusters need full bisection bandwidth (1:1 oversubscription) because AllReduce traffic is all-to-all with no locality. InfiniBand switches (Quantum-2 NDR) support 64 ports at 400 Gb/s each.

4. Rail-Optimized vs Full Fat-Tree

Rail-OptimizedFull Fat-Tree
Traffic patternGPU ii ↔ GPU ii across nodes (same rail)Any GPU ↔ any GPU
Best forData parallel AllReduce (each rail independent)Tensor parallel, expert parallel (arbitrary communication)
Switch costLower (separate network per rail)Higher (all rails share spines)
FlexibilityLow (optimized for one pattern)High (any pattern works)

DGX SuperPOD uses rail-optimized topology for DP traffic and a separate management network for control. Some clusters add a full fat-tree for EP All-to-All traffic.

5. Practical Verification

# Check IB link speed
ibstat | grep -E "Rate|State"
# Should show: Rate: 400 (NDR) or 200 (HDR)

# Check multi-rail configuration
ibv_devinfo | grep -E "hca_id|port:|active_speed"

# Measure inter-node bandwidth
ib_write_bw --size=1048576 <remote_ip>  # should see ~48 GB/s for NDR

# NCCL AllReduce benchmark across nodes
mpirun -np 16 -H node0:8,node1:8 \
  ./nccl-tests/build/all_reduce_perf -b 1M -e 1G -f 2 -g 1

For RDMA concepts (queue pairs, memory registration) and InfiniBand vs RoCE tradeoffs, see RDMA Concepts.

Self-Check

  • Name InfiniBand generations and per-port bandwidth (HDR, NDR, XDR)
  • Explain multi-rail topology and why DGX uses 8 NICs
  • Calculate aggregate inter-node bandwidth for a DGX H100 node
  • Explain spine-leaf topology and oversubscription ratio
  • Explain rail-optimized vs full fat-tree and when each is used
  • Verify InfiniBand speed with ibstat and ib_write_bw

References