Data Flywheel

The data flywheel turns production usage into training data — model outputs, user corrections, and human feedback loop back to improve the next model version.

---

1. The Flywheel Concept

Train model v1 → Deploy → Users interact
  ↓                              ↓
Collect: queries, outputs, feedback, corrections
  ↓
Filter: quality checks, dedup, privacy
  ↓
Label: human review, automated scoring
  ↓
Add to training data
  ↓
Train model v2 (better!) → Deploy → ...

Each iteration: Model gets better → attracts more users → generates more data → next model is even better.

---

2. Data Sources from Production

Source	What you collect	Value
User queries	Real input distribution	Better than synthetic prompts
Chosen responses	Which generation users accepted	Preference signal
Corrections	When users edit/regenerate	Direct quality signal
Thumbs up/down	Explicit feedback	Reward model training
Tool usage logs	Which tools called, results	Agent training data
Session patterns	Multi-turn conversations	Conversation quality signal

Privacy Considerations

• Anonymize: Remove PII before storing
• Consent: Terms of service must allow data collection for improvement
• Opt-out: Provide mechanism to exclude data
• Retention: Delete raw data after processing (keep only derived training data)

---

3. Human-in-the-Loop Annotation

Not all production data is labeled. Human annotators add quality:

Task	Annotator action	Scale
Preference ranking	Rank 2-4 model outputs by quality	1K-10K/day per annotator
Quality scoring	Rate response 1-5	2K-20K/day
Error correction	Fix factual errors, formatting	500-2K/day
Safety labeling	Flag harmful/unsafe content	5K-50K/day
Domain expertise	Verify medical/legal/technical accuracy	100-500/day

Annotation Platform Requirements

Requirement	Why
Calibration	Annotators must agree (inter-annotator agreement > 0.8)
Quality control	Hidden test questions to detect sloppy annotators
Throughput tracking	Monitor annotation speed and quality over time
Versioned guidelines	Track which instructions annotators followed

---

4. Closing the Loop

# Automated flywheel pipeline
def daily_flywheel():
    # 1. Collect yesterday's production data
    raw_data = collect_production_logs(yesterday)
    
    # 2. Filter and anonymize
    clean_data = anonymize(quality_filter(raw_data))
    
    # 3. Auto-label with reward model
    scored_data = reward_model.score(clean_data)
    
    # 4. Select high-quality examples
    good_examples = scored_data[scored_data.score > 0.8]
    
    # 5. Add to SFT/RLHF training pool
    training_pool.add(good_examples)
    
    # 6. Queue difficult examples for human review
    hard_examples = scored_data[scored_data.score.between(0.4, 0.6)]
    annotation_queue.add(hard_examples)

---

5. Anti-Patterns

Anti-pattern	Risk	Fix
Training on all production data	Amplifies model biases	Filter + score + human review
No diversity control	Popular queries dominate	Stratified sampling across categories
Feedback loop without monitoring	Quality degrades silently	Track data quality metrics each cycle
Ignoring negative feedback	Model repeats same mistakes	Prioritize corrections and thumbs-down

For data quality monitoring that keeps the flywheel healthy, see Data Quality Monitoring.

---

Self-Check

• Draw the data flywheel (deploy → collect → filter → label → train → deploy)
• List production data sources and their value for training
• Design an annotation pipeline with quality control
• Implement an automated daily flywheel pipeline
• Identify anti-patterns that can corrupt the flywheel

---

References

• Constitutional AI (Bai et al., 2022)
• RLHF: Training language models to follow instructions (Ouyang et al., 2022)