Expert Parallelism for MoE Inference

MoE models like DeepSeek-V3 (671B, 256 experts) require expert parallelism at inference time — distributing experts across GPUs and routing tokens via All-to-All communication. The inference challenge is different from training: latency matters per-request, and routing patterns are unpredictable.

1. Why EP for Inference

MoE models have many parameters but only activate a subset per token:

Model	Total params	Active params/token	Top-k	Experts
Mixtral 8×7B	47B	13B	2	8
DeepSeek-V3	671B	37B	2	256
Llama-4 Maverick	400B	17B	1	128

The total weight size may be too large for a single node. EP distributes experts across GPUs:

\text{Experts per GPU} = \frac{\text{total experts}}{\text{EP degree}}

DeepSeek-V3 with EP=64: $256/64 = 4$ experts per GPU.

2. The All-to-All Communication Pattern

EP Token Routing

Each MoE layer requires token routing:

1. Router computes expert assignments for each token
2. ALL-TO-ALL DISPATCH: send each token to its assigned expert's GPU
3. Expert compute: each GPU runs its local experts
4. ALL-TO-ALL COMBINE: send results back to originating GPU

Unlike TP’s AllReduce (same data pattern every step), All-to-All traffic is data-dependent — different prompts route to different experts, creating unpredictable communication patterns.

3. Expert Placement Strategies

Uniform Placement

Simplest: distribute experts evenly across GPUs.

64 GPUs, 256 experts:
  GPU 0: experts 0-3
  GPU 1: experts 4-7
  ...
  GPU 63: experts 252-255

Problem: Some experts receive more tokens than others (skewed routing). Hot experts become bottlenecks while cold experts’ GPUs sit idle.

Replication of Hot Experts

Replicate frequently-used experts on multiple GPUs:

Expert 0 (hot): replicated on GPUs 0, 16, 32, 48
Expert 200 (cold): only on GPU 50

Router sends tokens to least-loaded replica of the assigned expert

Tradeoff: Replication uses more memory but balances load.

Dynamic Rebalancing

Monitor expert load and migrate/replicate at runtime:

If expert_load[i] > 2× average:
    Replicate expert i to an underloaded GPU
If expert_load[j] < 0.5× average:
    Remove one replica of expert j (free memory)

4. Load Balancing for Inference

Challenge	Training solution	Inference difference
Skewed routing	Auxiliary loss	Can’t change loss at inference — must handle skew
Token dropping	Drop overflow tokens	Unacceptable — every token must be processed
Expert capacity	Fixed capacity per expert	Must handle worst-case without dropping

Inference-Specific Solutions

Expert capacity buffer: Over-provision each GPU’s compute for the expected maximum load
Token buffering: Queue excess tokens and process in the next step (adds latency)
Routing-aware scheduling: Batch requests with complementary routing patterns

5. Wide Expert Parallelism

For very large MoE models, EP spans multiple nodes:

DeepSeek-V3 on 8 nodes (64 GPUs):
  Node 0-1 (16 GPUs): Attention layers (TP=8 within each node)
  Node 2-7 (48 GPUs): Expert layers (EP=48, ~5 experts per GPU)
  
  All-to-All crosses InfiniBand between nodes — 50 GB/s per NIC

The InfiniBand bottleneck: All-to-All across 48 GPUs on IB is much slower than within a node on NVLink. llm-d’s “wide EP” approach addresses this with optimized NCCL configurations and UCCL transport.

6. EP for Inference: Practical Configuration

# vLLM doesn't directly expose EP — it's handled through TP
# For MoE models, vLLM places experts across TP ranks

# SGLang with EP support
python -m sglang.launch_server \
    --model deepseek-ai/DeepSeek-V3 \
    --tp 8 \
    --ep 8 \
    --trust-remote-code

For the training perspective on EP and All-to-All, see Expert Parallelism. For orchestrating wide EP at scale, see llm-d.

Self-Check

Calculate experts per GPU for a given model and EP degree
Describe the All-to-All dispatch/combine pattern for MoE inference
Explain why load balancing is harder at inference than training (no aux loss, no dropping)
Compare uniform placement vs hot-expert replication
Explain the IB bottleneck for wide EP across nodes