Reranker Serving

Vector search is fast but imprecise. It compresses an entire passage into a single vector, losing nuance. A reranker — a cross-encoder model that scores query-passage pairs jointly — recovers this lost precision, improving retrieval quality by 10-25% at the cost of 20-50ms additional latency.

---

1. Why Rerank?

The fundamental problem: Bi-encoder embeddings (used in vector search) encode query and passage independently. They can’t capture fine-grained interactions between query terms and passage terms.

Cross-encoders process the concatenated [query; passage] pair through a transformer, enabling full token-level attention between query and passage:

Bi-encoder (vector search):     Cross-encoder (reranker):
  query → [embed] → q_vec       [query + passage] → transformer → relevance_score
  passage → [embed] → p_vec     
  score = cosine(q_vec, p_vec)   Much more accurate, but 100-1000× slower

Stage	Model type	Speed	Accuracy	Purpose
Stage 1: Retrieval	Bi-encoder	1-5ms for 1M docs	Good	Narrow candidates (top-100)
Stage 2: Reranking	Cross-encoder	20-50ms for 100 docs	Excellent	Precision filter (top-10)

The two-stage approach gets the best of both: broad recall from vector search, then precise ranking from the cross-encoder.

---

2. Reranker Model Options

Model	Size	NDCG@10 (BEIR)	Latency (100 docs, A100)	Notes
bge-reranker-base	278M	52.1	15ms	Good quality/speed balance
bge-reranker-large	560M	54.3	25ms	Better quality, still fast
bge-reranker-v2-m3	568M	55.8	28ms	Multilingual support
cross-encoder/ms-marco-MiniLM-L-12	33M	49.2	5ms	Fastest, lowest quality
Cohere Rerank v3	— (API)	56.5	50-150ms	Best quality, API latency
Jina Reranker v2	278M	53.7	18ms	Good multilingual

Selection guidance:

• Latency budget >30ms: Use bge-reranker-large for best self-hosted quality
• Latency budget <15ms: Use ms-marco-MiniLM (faster but lower quality)
• Highest quality, latency tolerant: Cohere Rerank API
• Multilingual: bge-reranker-v2-m3 or Jina v2

---

3. Serving Architecture

Basic Reranker Server

from fastapi import FastAPI
from sentence_transformers import CrossEncoder
import torch

app = FastAPI()
model = CrossEncoder("BAAI/bge-reranker-large", device="cuda", max_length=512)

@app.post("/rerank")
async def rerank(query: str, passages: list[str], top_k: int = 10):
    """Score and rerank passages for a given query."""
    # Create query-passage pairs
    pairs = [[query, passage] for passage in passages]
    
    # Batch inference on GPU
    with torch.no_grad():
        scores = model.predict(pairs, batch_size=len(pairs))
    
    # Sort by score, return top-K
    ranked = sorted(
        zip(passages, scores.tolist()),
        key=lambda x: -x[1]
    )
    return {"results": [{"passage": p, "score": s} for p, s in ranked[:top_k]]}

Batch Optimization

Cross-encoders process each (query, passage) pair independently — perfect for batching:

def optimized_rerank(query: str, passages: list[str], model, batch_size=64):
    """Rerank with optimized batching."""
    pairs = [[query, p] for p in passages]
    all_scores = []
    
    for i in range(0, len(pairs), batch_size):
        batch = pairs[i : i + batch_size]
        # Pad to same length within batch for GPU efficiency
        scores = model.predict(batch, batch_size=len(batch))
        all_scores.extend(scores)
    
    return all_scores

Throughput scaling:

Configuration	Throughput (queries/sec, 100 docs each)
bge-reranker-large, 1× A100, batch=100	~40 qps
bge-reranker-large, 1× A100, batch=200 (2 queries)	~70 qps
bge-reranker-base, 1× A100, batch=100	~65 qps
ms-marco-MiniLM, 1× A100, batch=100	~200 qps
bge-reranker-large, CPU (8 cores)	~3 qps

Key insight: Reranking is compute-bound (not memory-bound), so GPU is essential for production throughput.

---

4. Latency Budget Allocation

In a RAG pipeline, every component eats into the total latency budget:

Total RAG budget: 2000ms (user-facing application)

Allocation:
  Query embedding:         10ms  (bi-encoder, cached model)
  Vector search:           20ms  (HNSW, P99)
  Reranking:              30ms  (100 docs through cross-encoder)  ← you are here
  Prompt construction:      5ms  (template + retrieved passages)
  LLM generation:       1800ms  (TTFT + decode)
  Overhead:              135ms  (network, parsing, logging)
                       --------
  Total:               2000ms

Reducing Reranker Latency

If the reranker exceeds its budget:

Technique	Latency reduction	Quality impact
Reduce candidates (100 → 50)	2× faster	Slight recall drop
Use smaller model (large → base)	1.5× faster	-2% NDCG
Truncate passages (512 → 256 tokens)	1.3× faster	Minimal for short passages
Use FP16 inference	1.2× faster	No quality loss
Use TensorRT/ONNX	1.5-2× faster	No quality loss

# ONNX optimization for reranker serving
from optimum.onnxruntime import ORTModelForSequenceClassification
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("BAAI/bge-reranker-large")
model = ORTModelForSequenceClassification.from_pretrained(
    "BAAI/bge-reranker-large",
    export=True,
    provider="CUDAExecutionProvider",
)
# ~1.5× faster than vanilla PyTorch

---

5. Caching Reranker Results

Many RAG queries are similar — cache reranker scores to avoid redundant computation:

import hashlib
from functools import lru_cache

class CachedReranker:
    def __init__(self, model, cache_ttl=3600):
        self.model = model
        self.cache = {}  # (query_hash, doc_hash) → score
        self.cache_ttl = cache_ttl
    
    def rerank(self, query: str, passages: list[str], top_k: int = 10):
        query_hash = hashlib.md5(query.encode()).hexdigest()
        
        cached_scores = {}
        uncached_pairs = []
        uncached_indices = []
        
        for i, passage in enumerate(passages):
            doc_hash = hashlib.md5(passage.encode()).hexdigest()
            cache_key = (query_hash, doc_hash)
            
            if cache_key in self.cache:
                cached_scores[i] = self.cache[cache_key]
            else:
                uncached_pairs.append([query, passage])
                uncached_indices.append(i)
        
        # Only run model on uncached pairs
        if uncached_pairs:
            new_scores = self.model.predict(uncached_pairs)
            for idx, score in zip(uncached_indices, new_scores):
                cached_scores[idx] = score
                doc_hash = hashlib.md5(passages[idx].encode()).hexdigest()
                self.cache[(query_hash, doc_hash)] = score
        
        # Rank all passages
        ranked = sorted(cached_scores.items(), key=lambda x: -x[1])
        return [(passages[i], score) for i, score in ranked[:top_k]]

Cache hit rates depend on workload:

• FAQ/support: 30-50% hit rate (repeated questions)
• General search: 5-15% hit rate
• Personalized: <5% (unique per user)

---

6. When to Skip Reranking

Reranking isn’t always worth the latency:

Scenario	Skip reranking?	Rationale
Top-1 retrieval is sufficient	Maybe	If vector search recall@1 > 90%, reranking helps little
Latency budget <100ms total	Yes	Can’t afford 30ms for reranking
Corpus <10K documents	Yes	Vector search is already precise on small corpora
Documents are very similar	No — reranking helps most here	Vector search struggles to distinguish near-duplicates
Multi-hop retrieval (3+ rounds)	No — use reranking per hop	Each hop’s precision compounds

---

Self-Check

• Explain why cross-encoders are more accurate than bi-encoders for ranking
• Select a reranker model given a latency budget constraint
• Calculate reranker throughput on GPU for a given QPS target
• Design a caching strategy for reranker scores
• Decide when reranking adds value vs when to skip it

---

References

• BGE Reranker Models
• MS MARCO Cross-Encoders (sentence-transformers)
• Cohere Rerank API
• For embedding models and pipeline, see Embedding Pipeline