Data Mixing

Data mixing determines what the model learns to do well. The ratio of code vs math vs web text directly shapes capabilities — and getting it wrong means expensive retraining.

---

1. Why Mixing Matters

The training data distribution directly determines model capabilities:

If you over-weight…	Model gets better at…	But worse at…
Web text	General knowledge, fluency	Code, math, reasoning
Code	Programming, structured reasoning	Creative writing, general knowledge
Math	Mathematical reasoning	General fluency
Books	Long-form coherence, depth	Short-form, conversational
Instruction data	Following instructions	Raw knowledge generation

There is no universally optimal mix — it depends on the target use case. Llama-3’s mix differs from DeepSeek’s which differs from CodeLlama’s.

---

2. Mixing Strategies

Static Mixing

Fix the ratios before training and don’t change them:

# Example: Llama-style static mix
mix = {
    "common_crawl_filtered": 0.67,  # 67% web
    "github_code": 0.08,           # 8% code  
    "wikipedia": 0.05,             # 5% encyclopedia
    "books": 0.05,                 # 5% books
    "arxiv": 0.03,                 # 3% academic
    "stackexchange": 0.02,         # 2% Q&A
    "c4_realnews": 0.10,           # 10% news
}

Upsampling and Downsampling

High-quality domains are often small. Repeat them to increase their effective weight:

Domain	Raw size	Target weight	Upsampling factor
Web (Common Crawl)	5 TB	67%	1.0× (downsample!)
Code (GitHub)	500 GB	8%	1.2×
Math (proofs, textbooks)	20 GB	5%	18× (heavily upsampled)
Books	100 GB	5%	3.5×

Upsampling risk: Too much repetition → model memorizes specific examples. Rule of thumb: don’t upsample more than 5-10× for any domain.

Curriculum Learning (Dynamic Mixing)

Change the mix during training:

Phase 1 (0-50% tokens): Heavy web text (learn language)
Phase 2 (50-80%): Increase code + math (learn reasoning)
Phase 3 (80-100%): High-quality curated data (refine quality)

Annealing: Increase the proportion of high-quality data toward the end of training. The model learns broad patterns first, then refines on curated content.

---

3. How Mixing Ratios Are Determined

Method	How it works	Cost
Scaling law experiments	Train small models at different mixes, extrapolate	Medium (many small runs)
Ablation studies	Train same-size models varying one domain at a time	High (multiple full runs)
Domain weighting by loss	Weight domains inversely to their loss (DoReMi)	Medium
Copy from prior art	Use published mixes (Llama, Falcon, etc.)	Free

DoReMi: Data Reweighting by Model Inspection

DoReMi trains a small proxy model to determine optimal domain weights:

1. Train a small model on uniform mix → observe per-domain losses
2. Upweight domains where the model struggles → train again
3. Use the resulting weights for the large model training

---

4. Practical Considerations

Issue	Solution
Domain proportions don’t sum to 1.0	Normalize: weight_i / sum(weights)
Small domain exhausted before large	Repeat small domain (with shuffle), or stop sampling it
Data quality varies within domains	Quality-filter each domain separately, then mix
Can’t re-run full training to tune mix	Use scaling law proxy experiments (1B model, 10B tokens)

For how to tokenize the mixed data, see Tokenization at Scale.

---

Self-Check

• Explain how data mixing ratios affect model capabilities (with examples)
• Compare static mixing vs curriculum learning
• Calculate upsampling factors for a given domain size and target weight
• Explain the upsampling risk (memorization) and the rule of thumb
• Describe DoReMi’s approach to determining optimal domain weights

---

References

• DoReMi: Optimizing Data Mixtures (Xie et al., 2023)
• The Llama 3 Herd of Models (Grattafiori et al., 2024)