Tensor-Parallel Serving

Tensor parallelism for inference splits model weights across GPUs within a node to reduce per-request latency and fit models that exceed single-GPU memory. The mechanics are the same as training TP, but the goals and tradeoffs differ.

1. Why TP for Inference

TP Serving

Two reasons to use TP for serving:

Reason	Example
Model doesn’t fit	70B BF16 = 140 GB > 80 GB GPU → need TP≥2
Latency reduction	Decode step on 70B: 42ms (1 GPU) → ~5ms (TP=8)

How TP Reduces Latency

Each GPU loads only $1/\text{TP}$ of the weights. Decode is memory-bandwidth-bound:

T_{\text{decode}} = \frac{W / \text{TP}}{B_{\text{HBM}}} = \frac{140\text{GB} / 8}{3350\text{GB/s}} \approx 5.2\text{ms}

8× less weight data → 8× faster decode (minus AllReduce overhead).

2. TP in Inference vs Training

Aspect	Training TP	Inference TP
Goal	Fit model + activations	Fit model + KV cache, reduce latency
Communication per layer	2× AllReduce (forward + backward)	1× AllReduce (forward only)
Batch size	Large (hundreds of sequences)	Variable (1 to hundreds)
Key overhead	Acceptable (hidden by compute)	Critical (directly adds to TPOT)
Typical TP degree	8 (full node)	2, 4, or 8 (minimize for cost)

Inference prefers the minimum TP degree that fits the model + KV cache. More TP = lower latency per request, but fewer requests per GPU (each GPU is partially occupied by one request’s TP group).

3. AllReduce Overhead for Inference

Each Transformer layer requires 2 AllReduce operations (attention output + FFN output). For inference:

For small messages (single-token decode: 16 KB per AllReduce), the overhead is dominated by latency (~5μs per AllReduce call on NVLink), not bandwidth:

T_{\text{AllReduce/layer}} \approx 2 \times 5\mu\text{s} = 10\mu\text{s per layer}

Over 80 layers: $80 \times 10\mu\text{s} = 0.8\text{ms}$ total AllReduce overhead.

At larger batch sizes (batch=32+), the message grows to 512 KB+ and bandwidth starts to matter, but NVLink’s 900 GB/s keeps it under 1ms total.

On InfiniBand (50 GB/s, ~5μs latency), the same 160 AllReduces would take $80 \times 2 \times (5 + 0.3)\mu\text{s} \approx 0.85\text{ms}$ for latency alone — but real-world IB AllReduce for small messages takes ~20-50μs due to protocol overhead, giving $80 \times 2 \times 30\mu\text{s} = 4.8\text{ms}$ . This is 6× slower than NVLink, making cross-node TP impractical for latency-sensitive serving.

4. TP and KV Cache

With TP, each GPU stores KV cache only for its assigned heads:

\text{KV per GPU} = \frac{2 \times L \times H_{kv} \times d_h \times \text{bytes}}{\text{TP}} \times S

For GQA models with $H_{kv} = 8$ : TP=8 means each GPU stores just 1 KV head per layer. This is efficient — each GPU’s KV cache is 1/8 of the total.

But: If $H_{kv} < \text{TP}$ , some GPUs have no KV heads for certain layers. This happens with MQA ( $H_{kv}=1$ ) at TP>1 — the single KV head must be broadcast to all TP ranks, adding communication.

5. Choosing TP Degree for Serving

Model	Weights (BF16)	Min TP (80GB)	Recommended TP	Why
8B	16 GB	1	1	Fits easily, TP adds overhead
13B	26 GB	1	1	Fits, avoid overhead
70B	140 GB	2	2-4	TP=2 fits, TP=4 halves latency
70B (INT4)	35 GB	1	1	Quantization removes the need for TP
405B	810 GB	11	8 + PP=2	Doesn’t fit even at TP=8

When TP Isn’t Enough: Pipeline Parallelism

For models that don’t fit even at TP=8 (e.g., 405B = 810 GB > 8 × 80 GB), add PP:

Config	How it works	Latency impact
TP=8, PP=2	2 nodes, each with TP=8. Layers split across nodes.	~2× per-layer latency (sequential across PP stages)
TP=4, PP=4	4 groups of 4 GPUs	Higher latency but uses less NVLink per group

PP for inference has no pipeline bubble (only one request flows through at a time — no micro-batching). But each PP stage adds an inter-node P2P transfer, increasing TTFT and TPOT. Use PP only when TP alone can’t fit the model.

Key tradeoff: Higher TP = lower latency but fewer independent serving instances per node. TP=8 on 8 GPUs = 1 instance. TP=2 on 8 GPUs = 4 independent instances (higher aggregate throughput).

# vLLM with TP=4
vllm serve meta-llama/Llama-3.1-70B --tensor-parallel-size 4

# TP=2 for throughput (4 instances on 8 GPUs)
# Instance 0: GPUs 0,1
CUDA_VISIBLE_DEVICES=0,1 vllm serve ... --tensor-parallel-size 2 --port 8000
# Instance 1: GPUs 2,3
CUDA_VISIBLE_DEVICES=2,3 vllm serve ... --tensor-parallel-size 2 --port 8001

For TP mechanics (column/row splitting, AllReduce patterns), see Tensor Parallelism. For how engines implement TP workers, see vLLM Architecture.

Self-Check

Calculate decode latency reduction from TP (weight size / TP / HBM bandwidth)
Calculate AllReduce overhead per layer and total for a given model
Explain why TP must stay intra-node (NVLink vs InfiniBand latency)
Explain the TP-degree vs instance-count tradeoff for throughput
Choose TP degree for a given model size and latency/throughput goal
Explain how GQA interacts with TP for KV cache distribution