MoE-Specific Scheduling

Mixture-of-experts models have a unique communication pattern: All-to-All dispatch and combine operations happen inside every MoE layer. This creates scheduling challenges that don’t exist in dense model training.

1. MoE Layer Communication Pattern

Each MoE layer executes:

1. Router: decide which experts each token goes to (local computation)
2. DISPATCH (All-to-All): send tokens to their assigned expert's GPU
3. Expert compute: each GPU runs its local experts on received tokens
4. COMBINE (All-to-All): send results back to the originating GPU
5. Continue to next layer

The two All-to-All operations are on the critical path — the next step depends on their completion.

2. The Scheduling Challenge

Dense model	MoE model
Communication: TP AllReduce (2 per layer)	Communication: All-to-All (2 per MoE layer) + TP AllReduce
Communication is same size every step	Communication varies with routing decisions
Overlap DP AllReduce with backward	All-to-All cannot be overlapped with expert compute

All-to-All traffic is unpredictable: if the router sends all tokens to 2 experts, those GPUs are overloaded while others sit idle. Load imbalance creates scheduling problems.

3. Overlap Strategies

Strategy 1: Overlap All-to-All with Non-MoE Compute

In a Transformer with alternating dense and MoE layers (e.g., every other layer is MoE):

Dense layer L:    Compute attention + FFN
                  ↓ (start prefetching expert inputs for layer L+1)
MoE layer L+1:   All-to-All dispatch ← overlap with dense layer L+1 attention
                  Expert compute
                  All-to-All combine  ← overlap with post-MoE operations

DeepSeek-V3 uses this: the attention sublayer of the next layer overlaps with the current MoE’s All-to-All combine.

Strategy 2: Expert Computation Pipelining

Split expert compute into chunks and overlap with All-to-All:

All-to-All (chunk 1) → Expert compute (chunk 1) → All-to-All combine (chunk 1)
All-to-All (chunk 2) ↗  Expert compute (chunk 2) ↗

This requires chunking the token batch and pipelining dispatch/compute/combine.

Strategy 3: FP8 Dispatch

Reduce All-to-All volume by dispatching tokens in FP8 instead of BF16:

Dispatch precision	Bytes per token	Volume reduction
BF16	$2h$	Baseline
FP8	$h$	2× less

DeepSeek-V3 uses FP8 dispatch to halve All-to-All traffic with negligible quality impact.

4. Load Balancing for Scheduling

Expert load imbalance creates scheduling waste — if one GPU’s experts receive 3× more tokens, it becomes the bottleneck while others wait.

Approach	How it works	Scheduling impact
Auxiliary loss	Penalty for uneven routing	Smooths load, more predictable scheduling
Expert capacity	Drop tokens exceeding capacity	Bounded compute per expert, predictable
Aux-loss-free (DeepSeek)	Bias terms learned to balance load	Best balance without quality loss
Token dropping	Skip overflow tokens	Bounded but loses information

For expert parallelism details, see Expert Parallelism.

5. MoE + PP Scheduling

Combining MoE with pipeline parallelism is complex:

Different stages may have different compute times (MoE layers are slower than dense)
Load imbalance in MoE layers creates variable stage times
Pipeline bubble calculation must account for the slowest stage

Solution: Place MoE layers evenly across stages, and use enough micro-batches to amortize variability.

Self-Check

Describe the All-to-All dispatch/combine pattern in an MoE layer
Explain why MoE All-to-All cannot overlap with expert compute (data dependency)
Describe three strategies for overlapping All-to-All with other computation
Explain how FP8 dispatch reduces communication volume
Identify the scheduling challenge when combining MoE with pipeline parallelism