Embedding Models

Embedding models convert text (or images) into dense vectors for search, retrieval, RAG, and clustering. Unlike generative models, embedding inference is a single forward pass per input — the challenge is pure throughput at massive scale.

---

1. Embedding vs Generation

Dimension	Generation (LLM)	Embedding
Output	Variable-length text (autoregressive)	Fixed-size vector (one forward pass)
Steps per input	Hundreds (decode steps)	1 (single forward)
KV cache	Yes (grows with context)	No
Bottleneck	Memory bandwidth (decode)	Compute (encode)
Batch efficiency	Limited by KV cache memory	Unlimited (no per-request state)

Embedding is the simplest inference workload — high batch sizes, no KV cache, no autoregressive loop. But the scale can be enormous: indexing billions of documents requires billions of forward passes.

---

2. Throughput Optimization

Large Batch Sizes

Without KV cache, all GPU memory is available for batch data:

# Embedding with large batch
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("BAAI/bge-large-en-v1.5", device="cuda")
embeddings = model.encode(
    texts,
    batch_size=512,        # much larger than LLM serving
    show_progress_bar=True,
    normalize_embeddings=True,
)

Model	Batch size	Throughput (H100)
BGE-large (335M)	512	15,000+ texts/sec
E5-mistral (7B)	32	500+ texts/sec
CLIP ViT-L (400M)	256	8,000+ images/sec

Variable-Length Packing

Texts have different lengths. Naive padding wastes compute:

Without packing:
  "Hello" (2 tokens)     → [Hello, PAD, PAD, ..., PAD]  (512 tokens, 99% waste)
  "Long document..." (500 tokens) → [Long, document, ..., end] (512 tokens, 2% waste)

With packing:
  Pack multiple short texts into one sequence:
  [Hello | SEP | How are you | SEP | Hi there | SEP | PAD]
  → 3 texts in 1 sequence, much less waste

Sorted batching: Sort texts by length, batch similar lengths together — reduces padding waste from ~60% to ~10%.

---

3. Serving Architecture for RAG

Document ingestion:
  Documents → Chunker (512 tokens each) → Embedding model → Vector DB (Pinecone, Weaviate, etc.)

Query time:
  User query → Embedding model → query vector → Vector DB search → top-K chunks → LLM

Serving pattern:
  - Ingestion: OFFLINE, batch millions of documents (throughput-critical)
  - Query: ONLINE, single query at a time (latency-critical, <10ms)

Dual Optimization

Phase	Priority	Batch size	Optimization
Ingestion	Throughput	256-1024	FP16, torch.compile, multi-GPU data parallel
Query	Latency	1-8	CUDA graphs, pre-warmed model, cached computation

---

4. Scaling to Billions

For indexing billions of documents:

Approach	Throughput	Cost
Single H100	15K texts/sec	1×
8× H100 (data parallel)	120K texts/sec	8×
Spot instances (elastic)	Variable	0.3× per GPU
CPU inference (quantized)	1K texts/sec per node	0.05× per unit

At 15K texts/sec: 1 billion documents takes ~18.5 hours on a single H100. With 8 GPUs: ~2.3 hours.

---

Self-Check

• Compare embedding inference to LLM generation (no KV cache, no autoregression)
• Calculate throughput for a given model and batch size
• Implement sorted batching to reduce padding waste
• Design a dual-optimization serving architecture for RAG (ingestion + query)
• Estimate time to embed 1B documents on a given hardware setup

---

References

• BGE Embedding Models
• Sentence-Transformers