Cleaning & Filtering

Raw collected data is noisy — duplicates, spam, PII, toxic content, and low-quality text. Cleaning and filtering transforms it into a training corpus. This stage has the highest impact on model quality of any data pipeline step.

---

1. The Cleaning Pipeline

Raw text
  → Language detection
  → Heuristic filters (length, repetition, special chars)
  → Quality scoring (perplexity, classifier)
  → Deduplication (exact → fuzzy → semantic)
  → PII removal
  → Toxicity filtering
  → Clean corpus

Each stage is a filter — data flows through and low-quality content is removed or flagged.

---

2. Language Detection

# fastText language detection (most common)
import fasttext
model = fasttext.load_model('lid.176.bin')

def detect_language(text):
    predictions = model.predict(text.replace('\n', ' ')[:512])
    lang = predictions[0][0].replace('__label__', '')
    confidence = predictions[1][0]
    return lang, confidence

# Filter: keep only target language with confidence > 0.65

Threshold matters: Too strict (>0.9) drops code-mixed text. Too loose (<0.5) includes wrong-language content.

---

3. Quality Scoring

Heuristic Filters

Filter	What it catches	Threshold
Minimum length	Empty/stub pages	> 50 words
Maximum length	Generated spam, data dumps	< 100K words
Repetition ratio	SEO spam, template pages	< 30% repeated lines
Special char ratio	Encoding errors, non-text	< 20% non-alphanumeric
Word count / line count	Lists, navigation menus	> 5 words per line average
Bullet/list ratio	Table-of-contents pages	< 50% lines starting with bullets

Perplexity-Based Scoring

Use a small language model to score text quality:

$$\text{PPL}(x) = \exp\left(-\frac{1}{N}\sum_{i=1}^{N} \log p(x_i | x_{<i})\right)$$

Low perplexity = text looks like typical language → likely high quality. High perplexity = unusual text → could be spam, encoding errors, or non-English.

Caveat: Very low perplexity can also indicate repetitive templates (e.g., product listings, boilerplate legal text, auto-generated pages). These score well on PPL because they’re predictable — but they’re low-value training data. Best practice: filter out both extremes (very high AND very low PPL), or combine PPL with a trained quality classifier.

# KenLM perplexity scoring (fast, used by CCNet)
import kenlm
model = kenlm.Model('cc_en.arpa')
ppl = model.perplexity("This is a well-written sentence.")  # low PPL = good

Classifier-Based Scoring

Train a binary classifier (fastText or small BERT) on “high quality” vs “low quality” examples:

• Positive: Wikipedia, curated textbooks, edited articles
• Negative: Random web samples, spam, boilerplate

---

4. Deduplication

Dedup is critical — duplicated data biases the model toward repeated content and wastes compute.

Exact Dedup

Hash each document, remove identical hashes:

import hashlib
seen = set()
for doc in corpus:
    h = hashlib.sha256(doc.encode()).hexdigest()
    if h not in seen:
        seen.add(h)
        yield doc  # keep unique

Fast but limited: Misses near-duplicates (same content with minor formatting differences).

Fuzzy Dedup (MinHash + LSH)

MinHash creates compact signatures for documents. Similar documents have similar signatures:

Document A: "The cat sat on the mat"
Document B: "The cat sat on a mat"  (1 word different)

MinHash signature (128 hashes): 95% of hashes match
→ Flagged as near-duplicate → remove one copy

MinHash at scale: Datasketch or Spark-based implementations process billions of documents. Typical threshold: Jaccard similarity > 0.8 → duplicate.

Semantic Dedup

Embed documents, find clusters of semantically similar content:

Document A: "Python is a programming language..."
Document B: "Python programming language overview..."

Embedding similarity: 0.95 → semantically duplicate
→ Keep the higher-quality version

Most expensive but catches paraphrased duplicates that MinHash misses.

---

5. PII Removal

PII type	Detection method	Action
Email addresses	Regex	Replace with [EMAIL]
Phone numbers	Regex	Replace with [PHONE]
Names	NER (spaCy, Presidio)	Replace with [NAME]
Addresses	NER + regex	Replace with [ADDRESS]
SSN / ID numbers	Regex	Replace with [ID]
URLs with PII	Pattern matching	Redact

from presidio_analyzer import AnalyzerEngine
from presidio_anonymizer import AnonymizerEngine

analyzer = AnalyzerEngine()
anonymizer = AnonymizerEngine()

results = analyzer.analyze(text=document, language='en')
anonymized = anonymizer.anonymize(text=document, analyzer_results=results)

---

6. Toxicity Filtering

Remove or flag content that’s harmful:

Approach	Speed	Coverage
Keyword blocklist	Very fast	Low (easy to evade)
Classifier (Perspective API, custom)	Medium	Good
LLM-as-judge	Slow	Best (understands context)

The tradeoff: Too aggressive filtering removes legitimate content about sensitive topics (medical, legal, historical). Too loose passes harmful content into training data.

For how cleaned data gets mixed into training, see Data Mixing.

---

Self-Check

• Design a cleaning pipeline with all 7 stages
• Explain perplexity-based quality scoring and when high PPL is bad vs interesting
• Compare exact vs MinHash vs semantic deduplication
• Implement MinHash near-dedup at the conceptual level
• Describe PII removal with Presidio
• Balance toxicity filtering aggressiveness vs content coverage

---

References

• RefinedWeb (Penedo et al., 2023)
• CCNet (Wenzek et al., 2020)
• Presidio PII Detection