Data Quality Monitoring

Data quality degrades silently — sources change, pipelines break, distributions shift. Without monitoring, bad data becomes bad models.

1. What Can Go Wrong

Problem	How it happens	Impact
Source degradation	Website redesign breaks scraper	Missing domain data
Distribution shift	New slang, events, topics appear	Model doesn’t know recent things
Pipeline bugs	Filter threshold changed accidentally	Quality drops or data lost
Label drift	Annotation guidelines changed	SFT quality inconsistent
Contamination	Benchmark data leaked into training	Inflated eval scores

2. Monitoring Dimensions

Statistical Drift

Compare new data batches against a reference distribution:

# Token distribution comparison
from scipy.stats import ks_2samp

ref_token_freq = compute_token_frequencies(reference_data)
new_token_freq = compute_token_frequencies(new_batch)

stat, pvalue = ks_2samp(ref_token_freq, new_token_freq)
if pvalue < 0.01:
    alert("Significant distribution shift detected!")

Domain Coverage

Track that all intended domains are represented:

expected_domains = {"web": 0.67, "code": 0.08, "math": 0.05, ...}
actual_domains = measure_domain_distribution(new_data)

for domain, expected in expected_domains.items():
    actual = actual_domains.get(domain, 0)
    if abs(actual - expected) > 0.02:  # >2% deviation
        alert(f"Domain {domain}: expected {expected:.0%}, got {actual:.0%}")

Quality Metrics

Metric	How to measure	Alert threshold
Average document length	Mean tokens per document	>20% change from baseline
Language mix	fastText distribution	>5% change in target language
Duplicate rate	MinHash within new batch	>10% duplicates
Perplexity distribution	Sample PPL with reference model	Mean PPL shift >15%
PII leak rate	Presidio on sample	Any detection > threshold

3. Automated Quality Gates

# CI/CD quality gate for data pipelines
quality_checks:
  - name: language_purity
    check: english_fraction > 0.95
    action: block_pipeline
    
  - name: duplicate_rate
    check: dedup_fraction < 0.10
    action: warn
    
  - name: domain_balance
    check: all_domains_within_2pct
    action: block_pipeline
    
  - name: benchmark_contamination
    check: no_eval_overlap
    action: block_pipeline  # critical!

For the feedback loop that uses quality monitoring to improve data, see Data Flywheel.

Self-Check

List 5 data quality problems and their impacts
Implement distribution drift detection with KS test
Design domain coverage monitoring for a training data pipeline
Set up automated quality gates in a CI/CD pipeline