Communication Overhead Estimation

Memory determines if a configuration fits. Communication determines how fast it runs. This chapter gives you the formulas and practical numbers to predict communication overhead before running a single experiment.

1. The Communication Budget Framework

Communication Timeline

Each training step has communication from multiple parallelism dimensions. The total exposed overhead depends on what can be overlapped with compute:

Communication	When it happens	Can overlap with…	Typical approach
TP AllReduce	After every layer	Nothing (on critical path)	Keep on NVLink
DP AllReduce	After backward	Backward compute (bucket by bucket)	Overlap with backward
PP P2P send/recv	Between micro-batches	Other micro-batch compute	Interleave 1F1B
EP All-to-All	Every MoE layer	Nothing (on critical path)	Keep on NVLink

Key insight: TP and EP communication is exposed (can’t be hidden), so it must be fast (NVLink). DP communication is overlappable, so it can tolerate slower networks (InfiniBand).

2. Tensor Parallel Communication

Per-Layer Cost

Each Transformer layer with TP does 2 AllReduce operations (one for attention output, one for FFN output):

T_{\text{TP per layer}} = 2 \times \frac{2 \times (b_\mu \times s \times h) \times 2\text{ bytes}}{B_{\text{NVLink}}}

The factor of $2$ inside the AllReduce formula is from the ring AllReduce algorithm ( $2 \times \frac{(n-1)}{n} \approx 2$ for large messages).

Worked Example: 70B, TP=8, NVLink 4.0

$b_\mu=1$ , $s=4096$ , $h=8192$ , $L=80$
Message size per AllReduce: $1 \times 4096 \times 8192 \times 2 = 64$ MB
Ring AllReduce time: $\frac{2 \times 64 \text{ MB}}{900 \text{ GB/s}} \approx 0.14$ ms per AllReduce
Per layer: $2 \times 0.14 = 0.28$ ms
All 80 layers: $80 \times 0.28 = 22.4$ ms total TP communication per step

What If TP Crosses Node Boundaries?

Same calculation but with InfiniBand (50 GB/s per NIC, ~100 GB/s aggregate for TP traffic):

T_{\text{cross-node TP}} = 80 \times 2 \times \frac{2 \times 64 \text{ MB}}{100 \text{ GB/s}} = 80 \times 2.56 \text{ ms} = 205 \text{ ms}

That’s 9× slower than NVLink — this is why TP must stay intra-node.

3. Data Parallel Communication

Per-Step Cost

DP AllReduce synchronizes gradients once per training step:

T_{\text{DP}} = \frac{2 \times \Psi_{\text{per-GPU}} \times 2\text{ bytes}}{B_{\text{network}}}

Where $\Psi_{\text{per-GPU}}$ is the parameter count after TP/PP sharding, and $B_{\text{network}}$ is the inter-node bandwidth.

Worked Example: 70B, TP=8, DP=32

Parameters per GPU: 70B / 8 = 8.75B
Gradient size: 8.75B × 2 bytes = 17.5 GB
Ring AllReduce on IB NDR: $\frac{2 \times 17.5 \text{ GB}}{400 \text{ GB/s}} = 87.5$ ms
Backward compute time (estimated): ~500 ms for 70B on H100

Overlap analysis: With gradient bucketing, NCCL starts AllReduce as soon as the last layer’s gradients are ready, while backward continues for earlier layers. If backward takes 500 ms and AllReduce takes 87.5 ms, the AllReduce is fully hidden — zero exposed overhead.

But if AllReduce > backward time, the excess is exposed:

T_{\text{exposed DP}} = \max(0, T_{\text{DP}} - T_{\text{backward}})

When DP Overlap Fails

Small models on many GPUs: 7B on 256 GPUs → gradients are tiny (0.25 GB per GPU), but AllReduce latency dominates
Very large DP degree: Ring AllReduce latency scales as $O(\log n)$ for tree, O(n) for ring start-up
Network congestion: If IB is oversubscribed, actual bandwidth < theoretical

4. Pipeline Parallel Communication

Per-Micro-Batch Cost

PP sends/receives activation tensors between pipeline stages:

T_{\text{PP P2P}} = \frac{b_\mu \times s \times h \times 2}{B_{\text{IB per-NIC}}}

Worked Example: 70B, PP=4

Activation tensor: $1 \times 4096 \times 8192 \times 2 = 64$ MB
IB NDR (one NIC): $\frac{64 \text{ MB}}{50 \text{ GB/s}} = 1.28$ ms per send

With 1F1B schedule and $m$ micro-batches:

Pipeline bubble: $(P-1) / m$ fraction of time wasted
For PP=4, $m=16$ : bubble = 3/16 = 18.75% overhead

PP communication itself is fast (1.28 ms), but the pipeline bubble is the real cost. Use enough micro-batches to minimize it.

5. Expert Parallel Communication

Per-MoE-Layer Cost

EP does All-to-All to route tokens to experts:

T_{\text{EP}} = \frac{b_\mu \times s \times h \times 2 \times \text{top-k}}{B_{\text{network}} \times E / (\text{EP degree})}

This is complex because the traffic pattern depends on the routing decisions. Worst case: every token goes to an expert on a different GPU.

Worked Example: DeepSeek-V3 (671B MoE)

256 experts, top-2 routing, EP=64
Per expert per token: $h \times 2 = 14336 \times 2 = 28$ KB
Tokens per step: $b_\mu \times s = 1 \times 4096 = 4096$
Total traffic: $4096 \times 2 \times 28 \text{ KB} = 224$ MB (dispersed across 64 GPUs)
On NVLink (within node): ~0.25 ms
On IB (across nodes): ~4.5 ms

This is why DeepSeek uses TP=4 (not 8) — they need to keep EP on NVLink within the 8-GPU node, using the remaining 2× for EP.

6. Summary: Communication Time Budget

For a complete training step with TP=8, PP=4, DP=8 on 256 H100 GPUs training 405B:

Communication	Time	Overlapped?	Exposed
TP AllReduce (80 layers, NVLink)	~22 ms	No	22 ms
PP P2P (16 micro-batches, IB)	~20 ms	With other micro-batches	~4 ms (bubble)
DP AllReduce (IB)	~100 ms	With backward	~0 ms (fully overlapped)
Total exposed			~26 ms
Compute per step			~500 ms
Communication fraction			~5% ✅

Target: < 10% of step time in communication. If higher, adjust parallelism.

7. Measuring Real Communication

Theory gives estimates. Validate with real measurements:

# NCCL AllReduce benchmark
mpirun -np 16 -H node0:8,node1:8 \
  ./nccl-tests/build/all_reduce_perf -b 1M -e 1G -f 2 -g 1

# Look for "busbw" column — this is the algorithm bandwidth
# H100 NVLink: expect ~850 GB/s busbw
# H100 IB NDR 8×: expect ~380 GB/s busbw

# PyTorch profiler with comm breakdown
with torch.profiler.profile(
    activities=[torch.profiler.ProfilerActivity.CUDA],
    record_shapes=True,
    profile_memory=True,
) as prof:
    train_step()

# Look for nccl:allReduce, nccl:send, nccl:recv in the trace

For profiling setup, see PyTorch Profiler.

Self-Check

Calculate TP AllReduce time per layer on NVLink for a given model
Explain why TP on InfiniBand is 9× slower and must stay intra-node
Calculate DP AllReduce time and determine if it overlaps with backward
Identify when DP overlap fails (small models, large DP degree)
Calculate pipeline bubble fraction for a given PP degree and micro-batch count
Estimate EP All-to-All time for a MoE model
Use nccl-tests to validate theoretical bandwidth