Synthetic Data Generation

When real data is insufficient — too small, missing domains, or lacking quality — synthetic data fills the gap. Models generate training data for other models, creating a data flywheel.

1. Why Synthetic Data

Problem	Synthetic solution	Example
Domain gap	Generate domain-specific examples	Medical Q&A from textbooks
Instruction data scarcity	Generate instruction-response pairs	Self-Instruct, Evol-Instruct
Reasoning traces	Generate step-by-step solutions	Math CoT, code explanations
Multilingual coverage	Translate or generate in target language	English → Chinese paraphrase
Safety training	Generate adversarial examples	Red-teaming datasets
Scale	Multiply dataset size cheaply	Paraphrase existing examples

2. Generation Approaches

Self-Instruct

Use a strong model to generate instruction-response pairs:

Seed: 175 human-written instructions
  ↓
Prompt strong model (GPT-4): "Generate a diverse instruction and response"
  ↓
Filter: remove low-quality, duplicates, too-similar-to-seed
  ↓
Output: 50K+ synthetic instruction-response pairs

Evol-Instruct (WizardLM)

Iteratively make instructions harder:

Start: "Write a Python function to sort a list"
  ↓
Evolve (depth): "Write a Python function to sort a list of custom objects
  by multiple keys, handling None values, with O(n log n) complexity"
  ↓
Evolve (breadth): "Write a Python function using a merge sort implementation..."

Rejection Sampling

Generate many candidates, keep only the best:

For each prompt:
  Generate K=16 responses
  Score each with reward model or verifier
  Keep top-1 (or top-4 for diversity)

DeepSeek-R1 used rejection sampling extensively — generating millions of reasoning traces, then filtering to keep only correct ones.

3. Quality Verification

Synthetic data can be wrong, repetitive, or degenerate. Verification is essential:

Verification method	What it catches	Cost
Automated scoring (reward model)	Quality, helpfulness, coherence	Low
Execution (code, math)	Correctness (run code, check math answer)	Medium
Dedup against training data	Contamination, memorization	Medium
Human review (sample)	Subtle errors, hallucinations	High
Diverse scoring (multiple models)	Model-specific biases	Medium

Deduplication Against Real Data

Synthetic data must not duplicate real training or evaluation data:

# Check synthetic data against training corpus
for synthetic_doc in synthetic_dataset:
    if minhash_similarity(synthetic_doc, training_index) > 0.8:
        remove(synthetic_doc)  # too similar to training data
    
    if exact_match(synthetic_doc, eval_benchmarks):
        remove(synthetic_doc)  # benchmark contamination!

4. Infrastructure for Generation

At scale, synthetic data generation is an inference workload:

Scale	Tokens to generate	GPU-hours (70B model)	Cost
Small (10K examples)	~5M tokens	~1 GPU-hour	~$4
Medium (1M examples)	~500M tokens	~100 GPU-hours	~$400
Large (100M examples)	~50B tokens	~10K GPU-hours	~$40K

Pipeline

1. Design prompt templates
2. Generate at scale (vLLM serving + batch requests)
3. Filter and verify
4. Dedup against real data + benchmarks
5. Mix with real data at appropriate ratio

5. When Synthetic Data Helps (and Hurts)

Scenario	Helps?	Why
Bootstrapping instruction tuning	✅	Cheap way to get initial SFT data
Augmenting rare domains	✅	Fill gaps in underrepresented domains
Replacing real data entirely	❌	“Model collapse” — quality degrades over generations
Pre-training at scale	⚠️	Can help if mixed carefully (<30% synthetic)
Reasoning traces	✅	Models generate correct reasoning more consistently than humans write it

Model collapse: If you train on synthetic data from model A, then use that model to generate data for model B, quality degrades. Each generation amplifies biases and reduces diversity. Always mix synthetic with real data.

For how to mix synthetic data with real data, see Data Mixing.

Self-Check

Name 3 scenarios where synthetic data is the right approach
Describe the Self-Instruct pipeline
Explain rejection sampling for synthetic data quality
Design a verification pipeline for synthetic code data
Explain model collapse and why mixing with real data is essential
Estimate GPU cost for generating 1M synthetic examples