Tokenization & Vocabulary

Before any computation happens, text must be converted to numbers. Tokenization is the first and last step of every LLM pipeline — and its design choices ripple through the entire infrastructure stack.

1. What Is a Token?

A token is the atomic unit of text that the model processes. It’s not a word, not a character, but something in between:

"Tokenization is fun!" → ["Token", "ization", " is", " fun", "!"]
                          [  1234,     5678,   890,   2345,  33]

Granularity	Example	Vocabulary size	Sequence length
Characters	`['T','o','k','e','n','i','z','a','t','i','o','n']`	~100	Very long
Words	`['Tokenization', 'is', 'fun']`	~100K+	Short
Subwords	`['Token', 'ization', ' is', ' fun', '!']`	~32K-128K	Balanced

Modern LLMs use subword tokenization — common words are single tokens, rare words are split into pieces. This balances vocabulary size against sequence length.

2. Byte Pair Encoding (BPE)

The dominant tokenization algorithm. BPE iteratively merges the most frequent pair of tokens:

Training

Start with individual characters (or bytes) as tokens
Count all adjacent token pairs in the corpus
Merge the most frequent pair into a new token
Repeat until vocabulary reaches target size

Corpus: "low lower lowest"

Step 0: ['l','o','w',' ','l','o','w','e','r',' ','l','o','w','e','s','t']
Step 1: Merge ('l','o') → 'lo':  ['lo','w',' ','lo','w','e','r',' ','lo','w','e','s','t']
Step 2: Merge ('lo','w') → 'low': ['low',' ','low','e','r',' ','low','e','s','t']
Step 3: Merge ('low','e') → 'lowe': ['lowe',' ','lowe','r',' ','lowe','s','t']
...

Encoding (Inference)

Apply the learned merge rules greedily to new text. The same text always produces the same tokens (deterministic).

Byte-Level BPE (GPT-2+)

Start from individual bytes (256 base tokens) instead of characters. This guarantees any text in any language can be tokenized — no “unknown token” problem.

3. Tokenizer Implementations

Tokenizer	Used by	Vocabulary	Key feature
SentencePiece	LLaMA, Qwen, Mistral	~32K	Language-agnostic, treats text as bytes, no pre-tokenization
tiktoken	GPT-3.5/4, Claude	~100K	Fast (Rust), regex-based pre-tokenization for better splits
HuggingFace Tokenizers	Many open models	Varies	Python library wrapping multiple algorithms

tiktoken vs SentencePiece

The main difference is pre-tokenization: tiktoken uses regex patterns to split text into coarse chunks before BPE, resulting in more natural token boundaries (e.g., never merging across word boundaries). SentencePiece treats the raw byte stream directly.

4. Token Embedding

After tokenization, each token ID is mapped to a dense vector via the embedding table:

\mathbf{E} \in \mathbb{R}^{V \times d}

Token ID $i$ → row $\mathbf{E}[i]$ , a $d$ -dimensional vector.

embedding = nn.Embedding(vocab_size, d_model)
x = embedding(token_ids)  # (B, S) → (B, S, d_model)

Memory cost: $V \times d \times \text{bytes}$ . For $V=128K, d=4096$ in FP16: $128K \times 4096 \times 2 = 1$ GB.

Weight Tying

Many models tie the embedding and LM head weights — the same $(V, d)$ matrix is used for both input embedding and output projection. This saves memory ( $-1$ GB for the example above) with minimal quality loss.

Special Tokens and Chat Templates

Beyond regular text tokens, LLMs use special tokens for control:

Token	Purpose	Example
`<BOS>` / `<s>`	Beginning of sequence	Marks sequence start
`<EOS>` / `</s>`	End of sequence	Signals generation should stop
`<PAD>`	Padding	Fills shorter sequences in a batch
`<UNK>`	Unknown token	Rare with byte-level BPE

Chat templates structure multi-turn conversations with role markers. Different models use different formats (ChatML, Llama-style, Mistral-style). The serving infrastructure must apply the correct template for each model — wrong templates produce garbled output.

Stopping criteria: Generation stops when the model outputs <EOS>, reaches max_tokens, or matches a stop string (e.g., "\nUser:" for chat). The serving engine must check all conditions after each token.

5. Infrastructure Impact

Token Count = Cost

LLM APIs charge per token. Efficient tokenization directly reduces cost:

GPT-4 tokenizer: “Tokenization” = 1 token
Poor tokenizer: “Token” + “ization” = 2 tokens

More efficient tokenization → shorter sequences → lower compute cost.

Vocabulary Size Tradeoffs

Larger vocabulary	Smaller vocabulary
✅ Shorter sequences (fewer tokens per text)	✅ Smaller embedding table
✅ Better representation of rare words	✅ Less memory
❌ Larger embedding table ( $V \times d$ )	❌ Longer sequences
❌ Sparser training signal per token	❌ More compute per text

Sweet spot: 32K–128K tokens. LLaMA uses 32K, GPT-4 uses ~100K, Qwen-2.5 uses 152K.

Multilingual Considerations

Tokenizers trained primarily on English data may produce 2-3× more tokens for Chinese, Japanese, or Korean text — directly increasing cost and latency for non-English users. Modern tokenizers (Qwen, GPT-4) use larger vocabularies with better multilingual coverage.

Serving

Tokenization runs on CPU (not GPU) and happens before and after inference. For high-throughput serving:

Pre-tokenize inputs in parallel (tokenization is CPU-bound)
Use fast tokenizer implementations (tiktoken, HuggingFace’s Rust-based tokenizers)
Cache tokenized prompts for repeated system prompts

Self-Check

Explain BPE: how it trains and how it encodes new text
Explain the difference between SentencePiece and tiktoken
Calculate the memory cost of an embedding table for a given V and d
Explain weight tying and its memory savings
Explain why tokenizer efficiency affects serving cost
Explain why multilingual tokenization matters for infrastructure