Data Center Network Design

The physical network topology of an AI cluster determines the upper bound on training performance. No amount of software optimization can overcome a network that doesn’t provide enough bandwidth for your parallelism strategy.

1. Traffic Patterns in AI Training

Different parallelism strategies generate different network traffic:

Strategy	Traffic pattern	Bandwidth demand	Locality
Data Parallel	AllReduce across all nodes	High volume, bursty	All-to-all (no locality)
Tensor Parallel	AllReduce within a group	Very high, per-layer	Intra-node (NVLink)
Pipeline Parallel	Point-to-point between stages	Moderate, streaming	Adjacent nodes
Expert Parallel	All-to-All per MoE layer	High, bursty	All-to-all (no locality)
Checkpoint I/O	All nodes → storage	Burst write	Many-to-one

Key insight: DP and EP create all-to-all traffic that stresses the network uniformly. PP creates localized traffic between neighbors. This determines the minimum oversubscription ratio.

2. Spine-Leaf (Clos) Topology

Two-Tier: Leaf + Spine

Spine-Leaf Topology

     S0    S1    S2    S3     ← Spine switches (IB or Ethernet)
     │╲    │╲    │╲    │╲
     │  ╲  │  ╲  │  ╲  │  ╲
    L0   L1   L2   L3   L4   ← Leaf switches
    │    │    │    │    │
   N0-3 N4-7 N8-11 ...  ...  ← Nodes (4 per leaf)

Leaf: connects to nodes (downlinks) and spines (uplinks)
Spine: connects all leaves (no direct node connections)
Every leaf connects to every spine → any-to-any communication in 2 hops

Sizing

Leaf has	Downlinks (to nodes)	Uplinks (to spines)	Oversubscription
32 ports	16 down	16 up	1:1 (full bisection)
32 ports	20 down	12 up	1.67:1
32 ports	24 down	8 up	3:1

Full Bisection vs Oversubscribed

Full bisection (1:1): aggregate uplink bandwidth = aggregate downlink bandwidth. Any half of the cluster can talk to the other half at full speed.

Oversubscribed (e.g., 2:1): uplink bandwidth = half of downlink. Cheaper (fewer spine switches) but inter-leaf traffic is bottlenecked.

AI clusters should use 1:1 or at most 1.5:1 — AllReduce traffic is all-to-all with no locality, so any oversubscription directly reduces gradient sync throughput.

3. Mapping Parallelism to Topology

The Typical DGX SuperPOD Layout

1 SuperPOD = 32 DGX H100 nodes = 256 GPUs

Physical layout:
  Leaf group 0: Nodes 0-7   (8 nodes, 64 GPUs) ← TP + DP-local
  Leaf group 1: Nodes 8-15  (8 nodes, 64 GPUs)
  Leaf group 2: Nodes 16-23
  Leaf group 3: Nodes 24-31

Parallelism mapping for 70B model:
  TP=8 (within each node, NVLink)
  PP=4 (across 4 nodes in a leaf group, IB)
  DP=8 (across leaf groups, IB spine)

Communication hierarchy:

TP AllReduce: NVLink within node → 900 GB/s
PP P2P: InfiniBand within leaf group → 50 GB/s per NIC
DP AllReduce: InfiniBand across leaf groups (through spines) → depends on oversubscription

4. Three-Tier for Large Clusters

For >1000 nodes (>10K GPUs):

Core switches ← connects pods
  │
Spine switches ← connects leaf groups within a pod
  │
Leaf switches ← connects nodes
  │
Nodes

Meta’s 24K GPU cluster: multiple pods, each ~2000 GPUs, connected via core switches. Inter-pod traffic (DP AllReduce across pods) is the bottleneck.

5. Network Design Checklist for AI

Question	What to check
Is TP intra-node?	NVLink connectivity: `nvidia-smi topo -m`
Is inter-node BW sufficient for DP?	Per-node aggregate IB BW × oversubscription ratio
Is AllReduce latency acceptable?	nccl-tests across nodes
Can EP All-to-All cross leaf boundaries?	Spine bandwidth ≥ EP traffic volume
Is checkpoint I/O a bottleneck?	Storage network separate from training network?

For communication cost formulas and worked examples, see Communication Cost Analysis.

Self-Check

Name the traffic pattern for each parallelism strategy (DP, TP, PP, EP)
Draw a 2-tier spine-leaf topology for 32 nodes
Calculate the oversubscription ratio for a given leaf switch port allocation
Explain why AI clusters need 1:1 bisection (unlike web/cloud at 4:1)
Map TP/PP/DP to network tiers for a given cluster