Data Lineage

Data lineage answers: “Which data went into which model?” This is essential for debugging quality regressions, compliance audits, and reproducing training runs.

---

1. What to Track

Item	Why	Example
Raw sources	Which crawls, datasets, APIs	“Common Crawl 2024-09, GitHub dump 2024-08”
Processing steps	Which filters, transforms, versions	“quality_filter v2.3, dedup MinHash threshold 0.8”
Mixing ratios	Domain proportions	“67% web, 8% code, 5% math”
Tokenizer	Which vocabulary was used	“Llama-3 tokenizer, 128K vocab”
Training config	Hyperparameters, parallelism	“LR 3e-4, batch 4M tokens, TP=8”
Output model	Which checkpoint	“model-v2.3-step-500000”

---

2. Lineage Record

# lineage.yaml — stored with each model checkpoint
model_id: llama-3.1-70b-v2.3
training_date: 2024-10-15
data:
  sources:
    - name: common_crawl_2024_09
      hash: sha256:abc123...
      size: 5.2TB
      filter_version: quality_v2.3
    - name: github_permissive_2024_08
      hash: sha256:def456...
      size: 480GB
  mix_ratios:
    web: 0.67
    code: 0.08
    math: 0.05
    books: 0.05
  tokenizer:
    name: llama3_128k
    hash: sha256:789abc...
  total_tokens: 15_000_000_000_000
processing:
  dedup: minhash_0.8
  pii_removal: presidio_v2.1
  quality_filter: classifier_v2.3

---

3. Debugging with Lineage

Model v2.3 shows degraded math performance.

Lineage check:
  v2.2 data: math_domain = 5%, source = "MATH + GSM8K + proofs"
  v2.3 data: math_domain = 3%, source = "MATH + GSM8K" (proofs removed!)

Root cause: proofs dataset was accidentally excluded in v2.3 pipeline.
Fix: restore proofs dataset, retrain from v2.2 checkpoint.

Without lineage, this debugging would require guessing which data change caused the regression.

For monitoring data quality over time, see Data Quality Monitoring.

---

Self-Check

• Design a lineage record for a training run
• Use lineage to debug a model quality regression
• Explain why data hashes are essential for reproducibility

---

References

• ML Metadata (Google)