Expert Parallelism — infra.bea.ai

Expert parallelism (EP) distributes Mixture-of-Experts FFN layers across GPUs — each GPU holds a subset of experts. Tokens are dispatched to the GPU holding their selected expert via All-to-All communication.

For MoE model architecture details (routing, shared experts, load balancing), see MoE Architecture. For All-to-All communication details, see Communication Primitives.

1. Why Expert Parallelism?

MoE models have many experts (8-256) with potentially 10× more total parameters than active parameters. Even though each token only uses 2-8 experts, all expert weights must be stored somewhere.

DeepSeek-V3: 256 experts × 3 weight matrices per expert. Total FFN params: ~500B. Can’t fit on one GPU — experts must be distributed.

2. The Dispatch Pattern

EP Dispatch

Input tokens on all GPUs
     │
     ▼
Router: assigns each token to top-k experts
     │
     ▼
All-to-All #1 (dispatch): send tokens to the GPU holding their expert
     │
     ▼
Expert computation: each GPU processes tokens through its local experts
     │
     ▼
All-to-All #2 (combine): send results back to the originating GPU
     │
     ▼
Weighted sum of expert outputs

Two All-to-All per MoE layer — this is the dominant communication cost.

3. Communication Cost

Each All-to-All: each GPU sends $\sim N_{\text{tokens}} / P$ tokens to each other GPU.

Per MoE layer: $2 \times B \times S \times d \times 2 \times (P-1)/P$ bytes (two All-to-Alls).

For $B=1, S=4096, d=7168$ , BF16, 32 GPUs:

T = 2 \times \frac{4096 \times 7168 \times 2 \times 31}{32 \times 50 \times 10^9} \approx 2.3 \text{ ms (InfiniBand)}

With 61 MoE layers: $61 \times 2.3 = 140$ ms total All-to-All per step. Significant — but overlappable with non-MoE computation.

4. Load Balancing Challenges

The router may send unequal numbers of tokens to each expert (and thus each GPU):

Problem	Impact	Mitigation
Popular experts get many tokens	Some GPUs overloaded, others idle	Auxiliary load-balancing loss
Expert capacity overflow	Tokens dropped if expert is “full”	Capacity factor > 1.0
Skewed routing per batch	Variable compute time per GPU	Dynamic token redistribution

Expert capacity factor: Max tokens per expert = capacity_factor × (total_tokens / num_experts). Typical: 1.0-1.5. Higher = fewer drops, more memory.

5. EP + TP + DP Composition

Expert parallelism interacts with other dimensions:

256 GPUs organized as:
  DP=8 × TP=4 × EP=8

- DP: 8 copies of the model, each on 32 GPUs
- TP: each model copy split across 4 GPUs (within a node)
- EP: 8 expert groups, each expert group on one TP group

Within a TP group: experts are NOT split (each GPU has full expert weights)
Across EP groups: each group has different experts

All-to-All in EP happens between expert groups (potentially inter-node), while TP AllReduce happens within groups (intra-node NVLink).

6. Optimizations

Overlapping All-to-All with Compute

MoE layers alternate with attention layers. While the attention layer computes, the All-to-All for the next MoE layer can start:

[Attention L1 compute] [All-to-All MoE L2 dispatch] (overlapped)
                       [MoE L2 compute]

Expert Replication

For popular experts, replicate them across multiple GPUs to reduce load imbalance:

GPU 0: Expert 0 (replica 1), Expert 1
GPU 1: Expert 0 (replica 2), Expert 2
→ Tokens for Expert 0 can go to either GPU 0 or GPU 1

Cost: More memory. Benefit: Better load balance, less token dropping.

Self-Check

Explain why expert parallelism is needed (experts don’t fit on one GPU)
Draw the two All-to-All dispatch pattern for one MoE layer
Calculate All-to-All communication volume for given model and GPU count
Explain load balancing challenges and the capacity factor tradeoff
Explain how EP composes with TP and DP