Communication Cost Analysis

The difference between “distributed training that scales” and “distributed training that wastes GPUs” is almost always communication cost. This chapter gives you the formulas to predict whether adding more GPUs will help — before you spend the compute.

1. The α-β Model

Every communication operation has two cost components:

T = \alpha + \frac{N}{B}

α (alpha): startup latency — fixed cost per message regardless of size (~5-10 μs for NVLink, ~1-5 μs for InfiniBand)
N/B: transfer time — message size $N$ divided by bandwidth $B$

Two Regimes

Regime	When	Dominates	Optimization
Latency-bound	Small messages ( $N < \alpha \times B$ )	Startup cost α	Reduce message count (batching, fusion)
Bandwidth-bound	Large messages ( $N > \alpha \times B$ )	Data volume N/B	Use bandwidth-optimal algorithms (ring)

Crossover point for NVLink ( $\alpha = 5μs$ , $B = 450 \text{ GB/s}$ ):

N_{\text{crossover}} = \alpha \times B = 5 \times 10^{-6} \times 450 \times 10^9 = 2.25 \text{ MB}

Below 2.25 MB: latency-bound (use tree). Above: bandwidth-bound (use ring).

2. Cost Per Collective

For $P$ GPUs, message size $N$ bytes, per-link bandwidth $B$ :

Collective	Latency component	Bandwidth component	Total
AllReduce (ring)	$2(P-1)\alpha$	$\frac{2N(P-1)}{PB}$	$2(P-1)\alpha + \frac{2N(P-1)}{PB}$
AllGather (ring)	$(P-1)\alpha$	$\frac{N(P-1)}{PB}$	$(P-1)\alpha + \frac{N(P-1)}{PB}$
ReduceScatter (ring)	$(P-1)\alpha$	$\frac{N(P-1)}{PB}$	Same as AllGather
All-to-All	$(P-1)\alpha$	$\frac{N(P-1)}{PB}$	Same as AllGather
Broadcast (tree)	$\log_2 P \times \alpha$	$\frac{N \log_2 P}{B}$	$\log_2 P(\alpha + \frac{N}{B})$

For Large N (Bandwidth-Dominated)

When $N \gg \alpha \times B \times P$ , the bandwidth terms dominate and simplify:

T_{\text{AllReduce}} \approx \frac{2N}{B} \qquad \text{(for large P)}

This is the key result: AllReduce time is approximately constant regardless of how many GPUs you use (for large messages). This is why data-parallel training can scale to thousands of GPUs.

3. Worked Examples

Example 1: Gradient Sync in DDP

7B model, BF16 gradients: $N = 7 \times 10^9 \times 2 = 14$ GB. 8 GPUs on NVLink ( $B = 450$ GB/s per direction):

T_{\text{AllReduce}} = 2 \times 7 \times 5\mu s + \frac{2 \times 14 \times 10^9 \times 7}{8 \times 450 \times 10^9} = 70\mu s + 54.4ms = 54.5ms

With gradient bucketing (start AllReduce during backward): effective overhead ≈ last bucket only ≈ 5-10 ms.

Example 2: FSDP Parameter Gather

70B model sharded across 8 GPUs. Before each layer, AllGather the parameters. Per-layer parameters (one of 80 layers): $70B / 80 \times 2 = 1.75$ GB.

T_{\text{AllGather}} = 7 \times 5\mu s + \frac{1.75 \times 10^9 \times 7}{8 \times 450 \times 10^9} = 35\mu s + 3.4ms = 3.4ms

Per layer: 3.4ms communication. If the layer’s compute takes 10ms, communication is 25% overhead — acceptable with overlap.

Example 3: MoE All-to-All

DeepSeek-V3: 256 experts across 32 GPUs. Each GPU sends different tokens to each other. Token batch: 4096 tokens × 7168 hidden × 2 bytes = 58.7 MB per GPU.

T_{\text{All-to-All}} = 31 \times 5\mu s + \frac{58.7 \times 10^6 \times 31}{32 \times 450 \times 10^9} = 155\mu s + 0.13ms = 0.28ms

Fast — but this happens twice per MoE layer (dispatch + combine), and there are many MoE layers. At 61 layers: $61 \times 2 \times 0.28 = 34ms$ total All-to-All per forward pass.

4. Intra-Node vs Inter-Node

Communication cost depends dramatically on whether GPUs are in the same node:

Path	Bandwidth	Latency	Technology
Intra-node (same server)	450-900 GB/s	~5 μs	NVLink 4.0/5.0
Inter-node (across servers)	25-50 GB/s	~1-5 μs	InfiniBand NDR

The bandwidth gap: NVLink is 10-36× faster than InfiniBand. This is why:

Tensor Parallelism (high communication) → must be intra-node (NVLink)
Data Parallelism (moderate communication) → can be inter-node (InfiniBand)
Pipeline Parallelism (low communication) → inter-node is fine

Example: AllReduce Intra vs Inter

14 GB gradient AllReduce:

Intra-node (8 GPUs, NVLink 450 GB/s): ~55 ms
Inter-node (8 nodes × 8 GPUs, IB 50 GB/s): ~560 ms for inter-node part

This 10× gap is why parallelism strategy must match the network topology.

5. Communication-Computation Overlap

The goal: hide communication behind computation so it’s “free.”

Gradient Bucketing (DDP)

Backward:    [Layer 32 grad] [Layer 31 grad] [Layer 30 grad] ...
AllReduce:                   [Bucket 8 AR]   [Bucket 7 AR]   ...

Overlap fraction = min(backward_time, allreduce_time) / allreduce_time. If backward takes 100ms and AllReduce takes 50ms, overlap is 100% — AllReduce is completely hidden.

FSDP Prefetching

Forward:    [AllGather L0] [Compute L0] [AllGather L1] [Compute L1] ...
Optimal:    [AG L0][Compute L0 + AG L1][Compute L1 + AG L2] ...

Prefetch the next layer’s parameters while computing the current layer. FSDP does this with forward_prefetch=True.

Pipeline Overlap

In pipeline parallelism, one stage sends activations to the next while the current stage starts the next micro-batch:

Stage 0:  [F_mb0] [F_mb1] [F_mb2] ...
Stage 1:         [recv] [F_mb0] [F_mb1] ...
                   ↑ hidden behind Stage 0's next micro-batch

6. Predicting Scaling Efficiency

Scaling efficiency = ideal time / actual time:

\eta = \frac{T_{\text{compute}}}{T_{\text{compute}} + T_{\text{comm\_exposed}}}

Where $T_{\text{comm\_exposed}}$ is the communication time not overlapped with compute.

Scenario	Compute	Comm (total)	Overlap	Exposed comm	Efficiency
DDP, large batch	100 ms	50 ms	90%	5 ms	95%
DDP, small batch	20 ms	50 ms	40%	30 ms	40%
FSDP, good prefetch	100 ms	30 ms	80%	6 ms	94%
TP across InfiniBand	100 ms	200 ms	0%	200 ms	33%

Key insight: Efficiency depends on the ratio of compute to exposed communication. Large batches help (more compute to hide behind). Bad topology kills efficiency (TP on InfiniBand).

Self-Check

Apply the α-β model to calculate AllReduce time for given N, P, B
Explain the crossover between latency-bound and bandwidth-bound regimes
Calculate gradient sync time for a 7B model across 8 GPUs
Explain why NVLink vs InfiniBand determines parallelism strategy placement
Calculate scaling efficiency given compute time and communication overlap
Explain three overlap techniques: gradient bucketing, FSDP prefetch, pipeline overlap