MoE Architecture

Mixture of Experts (MoE) is the architecture that enables models like DeepSeek-V3 (671B total, 37B active per token) and Mixtral (46.7B total, 12.9B active). The idea: not every token needs every parameter — route each token to a subset of “expert” FFN networks, achieving larger model capacity at lower per-token compute cost.

---

1. The Core Idea: Sparse Activation

In a standard Transformer, every token passes through every parameter. In MoE, the FFN is replaced by $E$ expert FFNs, and a router selects the top-$k$ experts for each token:

$$y = \sum_{i \in \text{TopK}} g_i \cdot \text{Expert}_i(x)$$

Where $g_i$ are routing weights from a gating function.

Per-token compute: only $k$ out of $E$ experts are activated. If $k=2, E=64$, each token uses only 3% of expert parameters.

Total model capacity: much larger than the per-token compute suggests. DeepSeek-V3 has 671B parameters but activates only 37B per token — getting the quality benefits of a 671B model at the compute cost of a ~37B dense model.

---

2. The Router

Gating Function

The router is typically a simple linear layer + softmax:

$$\mathbf{g} = \text{softmax}(\mathbf{x} \mathbf{W}_{\text{gate}}), \qquad \mathbf{W}_{\text{gate}} \in \mathbb{R}^{d \times E}$$

Then select the top-$k$ experts and renormalize their weights:

def route(x, W_gate, k=2):
    scores = x @ W_gate          # (B*S, E)
    topk_scores, topk_ids = scores.topk(k, dim=-1)
    weights = softmax(topk_scores, dim=-1)
    return topk_ids, weights

Load Balancing

The biggest practical challenge: tokens may cluster on a few popular experts while others are idle — expert imbalance.

Auxiliary load balancing loss:

$$\mathcal{L}_{\text{balance}} = \alpha \cdot E \cdot \sum_{i=1}^{E} f_i \cdot p_i$$

Where $f_i$ is the fraction of tokens routed to expert $i$, and $p_i$ is the average routing probability for expert $i$. This loss penalizes imbalance and is added to the training loss.

Expert capacity and token dropping: Each expert has a maximum number of tokens per batch (the capacity factor, typically 1.0–1.5× of average). Tokens routed to a full expert are dropped — they contribute nothing to the forward pass. This causes training instability, gradient noise, and the model may learn to avoid routing to high-demand experts.

Auxiliary-loss-free balancing (DeepSeek-V3): Instead of an explicit loss, uses a per-expert bias term updated based on load statistics. Over-loaded experts get a negative bias, reducing their routing probability — avoiding the training instability of the auxiliary loss approach.

Training Instability

MoE models are harder to train than dense models:

• Routing collapse: all tokens converge to 1–2 experts, starving the rest. The auxiliary loss prevents this but adds hyperparameter sensitivity.
• Loss spikes: load imbalance causes sudden gradient magnitude changes, especially early in training.
• Expert utilization drift: some experts become permanently underutilized after initial routing patterns are established.

The Switch Transformer (Fedus et al., 2022) used top-1 routing (one expert per token) and was the first to demonstrate MoE at scale, but required careful capacity factor tuning.

---

3. Expert Design Variants

Standard MoE

Replace FFN with $E$ identical FFN experts:

Input → Router → Top-k experts → Weighted sum → Output

Each expert is a standard SwiGLU FFN. Parameters: $E \times (3 \times d \times d_{ff}^{\text{expert}})$.

Shared Experts (DeepSeek)

DeepSeek-V2/V3 adds shared experts that process every token (not routed):

Input → Shared Expert(s) + Router → Top-k routed experts → Sum all → Output

Shared experts capture common patterns; routed experts handle specialized knowledge. This improves quality and reduces the impact of routing errors.

Fine-Grained Experts (DeepSeek-V3)

Instead of $E$ large experts, use $E_{\text{fine}}$ smaller experts with higher $k$:

• DeepSeek-V3: 256 routed experts, top-8 active, plus 1 shared expert
• Each expert is smaller → more routing flexibility
• Higher $k$ with smaller experts ≈ similar compute but better coverage

---

4. Infrastructure Implications

MoE creates unique infrastructure challenges that don’t exist for dense models:

Expert Parallelism

Experts are distributed across GPUs. Each token must be routed to the correct GPU:

GPU 0: Experts 0-15     GPU 1: Experts 16-31
GPU 2: Experts 32-47    GPU 3: Experts 48-63

Token routing requires All-to-All communication:
  1. Each GPU computes routing for its tokens
  2. All-to-All: send tokens to the GPU holding their selected experts
  3. Each GPU processes tokens through its local experts
  4. All-to-All: send results back to originating GPUs

Two All-to-All per MoE layer — this is the dominant communication cost for MoE training and inference.

Memory: Total vs Active

Model	Total params	Active params/token	Memory for weights (FP16)
LLaMA-2-70B (dense)	70B	70B	140 GB
Mixtral 8×7B	46.7B	12.9B	93 GB
DeepSeek-V3	671B	37B	1.3 TB

MoE models need all expert weights loaded — even though each token only uses a fraction. The memory requirement is based on total parameters, not active parameters.

Inference Challenges

• Load imbalance: some experts receive more tokens than others, creating GPU utilization skew
• Expert caching: popular experts should be cached in faster memory; cold experts can be offloaded
• Batch efficiency: different tokens in a batch may route to different experts, creating irregular compute patterns
• KV cache: same as dense models (MoE doesn’t change attention)

---

5. Model Comparison

Model	Type	Experts	Active	Total params	Active params	Architecture
LLaMA-2-70B	Dense	—	—	70B	70B	GQA
Mixtral 8×7B	MoE	8	2	46.7B	12.9B	GQA + MoE FFN
DeepSeek-V3	MoE	256+1 shared	8+1	671B	37B	MLA + fine-grained MoE
Qwen2-57B-A14B	MoE	64	8	57B	14B	GQA + MoE FFN

---

Self-Check

• Explain the difference between total parameters and active parameters in MoE
• Explain the router (gating function) and top-k expert selection
• Explain why load balancing is critical and how auxiliary loss helps
• Explain shared experts and why DeepSeek uses them
• Explain expert parallelism and the two All-to-All communications per MoE layer
• Calculate memory requirements for MoE vs dense models of similar quality

---

References

• Mixtral of Experts (Jiang et al., 2024)
• DeepSeek-V2 (DeepSeek-AI, 2024)
• DeepSeek-V3 Technical Report (DeepSeek-AI, 2024)
• Switch Transformers (Fedus et al., 2022)