Tokenization at Scale

Tokenization converts text into integer sequences that the model processes. At pre-training scale (trillions of tokens), tokenization must be fast, deterministic, and carefully designed — the vocabulary directly affects model quality and efficiency.

---

1. Tokenizer Training

BPE (Byte Pair Encoding)

The dominant algorithm for LLM tokenizers:

1. Start with individual bytes (or characters) as the initial vocabulary
2. Count all adjacent pair frequencies in the training corpus
3. Merge the most frequent pair into a new token
4. Repeat until vocabulary reaches target size (32K-128K tokens)

# SentencePiece BPE training
import sentencepiece as spm
spm.SentencePieceTrainer.train(
    input='training_corpus.txt',
    model_prefix='tokenizer',
    vocab_size=128000,
    model_type='bpe',
    byte_fallback=True,  # handle unknown bytes
    num_threads=64,       # parallel training
)

Vocabulary Design Decisions

Decision	Impact	Common choice
Vocab size	Larger = shorter sequences, more parameters in embedding	32K (GPT-2), 128K (Llama-3)
Byte fallback	Handle any Unicode without UNK tokens	✅ Always enable
Special tokens	BOS, EOS, PAD, tool-use markers	Model-specific
Multilingual	Include non-English scripts in training corpus	Yes for multilingual models
Code-aware	Include whitespace/indentation tokens	Yes for code models

---

2. Tokenization at Scale

The Scale Challenge

For a 15T token training run:

• Raw text: ~10-30 TB
• Tokenized output: ~30-60 TB (int32 token IDs + metadata)
• Tokenization speed needed: process 10 TB in hours, not days

Parallel Tokenization Pipeline

# Multi-process tokenization with HuggingFace
from transformers import AutoTokenizer
from multiprocessing import Pool
import pyarrow.parquet as pq

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B")

def tokenize_shard(shard_path):
    texts = pq.read_table(shard_path)['text'].to_pylist()
    tokens = tokenizer(texts, add_special_tokens=False)['input_ids']
    # Save as memory-mapped numpy array
    save_tokens(tokens, shard_path.replace('.parquet', '.bin'))

# Process 1000 shards across 64 workers
with Pool(64) as pool:
    pool.map(tokenize_shard, shard_paths)

Pre-Tokenization

Tokenize the entire corpus once before training starts. Store as binary files (numpy arrays or memory-mapped files). The training loop reads pre-tokenized data — no tokenization during training.

Pre-tokenization pipeline:
  text shards (.parquet) → tokenize (parallel) → token shards (.bin)
  → shuffle → combine into training-ready format

---

3. Tokenizer Impact on Training

Aspect	Impact
Sequence length	Longer vocab = shorter sequences = less compute per text
Embedding table	vocab_size × hidden_dim parameters (128K × 8192 = 1B params for Llama-3)
Fertility	Tokens per word — affects effective context length
Multilingual fairness	Poor tokenizers need 3× more tokens for non-English text
Determinism	Same text must produce same tokens across runs (reproducibility)

---

4. Output Format

Format	Description	Used by
Numpy .bin	Raw int32/int16 arrays, memory-mappable	Custom pipelines
Arrow/Parquet	Columnar, compressed, metadata-rich	HuggingFace datasets
MDS (Mosaic)	Streaming-optimized shards	MosaicML StreamingDataset
WebDataset .tar	Tar archives of samples	Large-scale web pipelines

For how tokenized data is loaded into training, see PyTorch DataLoader.

---

Self-Check

• Explain BPE tokenizer training (merge most frequent pairs iteratively)
• Choose vocabulary size for a given use case (multilingual, code, English-only)
• Design a parallel tokenization pipeline for 10 TB of text
• Explain pre-tokenization and why it’s done before training starts
• Evaluate tokenizer fertility for non-English languages

---

References

• SentencePiece (Kudo & Richardson, 2018)
• tiktoken (OpenAI)