Navigation
🤖 Apps & Agents· 12 min read

Model Routing

Classifying input difficulty and dispatching to small/medium/large models — saving 60-80% on inference cost without sacrificing quality.

model-routingdifficulty-classificationdispatchcost-optimizationmulti-model

Not every request needs your largest model. A “what’s the weather?” query doesn’t require 405B parameters. Model routing classifies each request’s difficulty and dispatches it to the cheapest model that can handle it well — saving 60-80% on inference cost while maintaining quality where it matters.

1. The Economics of Routing

Using a single model for all traffic is simple but wasteful:

Query type% of trafficNeeded modelCost if always use 405B
Simple factual (“capital of France?”)40%8B is fine$5.00/M tokens (overkill)
Medium reasoning (“explain photosynthesis”)35%70B is fine$5.00/M tokens (overkill)
Complex multi-step (“prove this theorem”)25%405B needed$5.00/M tokens (correct)

Always-405B cost: $5.00/M tokens.

With routing:

Routed cost=0.40×$0.22+0.35×$1.11+0.25×$5.00=$1.73/M tokens\text{Routed cost} = 0.40 \times \$0.22 + 0.35 \times \$1.11 + 0.25 \times \$5.00 = \$1.73\text{/M tokens}

65% cost reduction — and quality stays the same because each request goes to a model that can handle it.

2. Router Architectures

Model Router Architecture

Pre-Routing (Classify Then Dispatch)

A lightweight classifier examines the request before any model sees it, then dispatches to the appropriate model:

Request → Router (classifier) → small / medium / large model

The classifier runs once per request. If it’s fast (1-5ms), the overhead is negligible.

Three Classifier Approaches

Rule-based — simplest, no training needed:

def rule_router(prompt: str) -> str:
    """Route based on heuristics."""
    # Short, simple queries → small model
    if len(prompt.split()) < 20 and "?" in prompt:
        return "8b"
    
    # Code generation, math, multi-step → large model
    code_keywords = ["implement", "debug", "algorithm", "prove", "derive"]
    if any(kw in prompt.lower() for kw in code_keywords):
        return "405b"
    
    # Everything else → medium model
    return "70b"

Accuracy: ~60-70%. Cheap to run, easy to understand, but brittle — misses nuance.

Learned classifier — a small model trained on difficulty labels:

from sklearn.linear_model import LogisticRegression
from sentence_transformers import SentenceTransformer

# Embed queries with a tiny model (30M params, <1ms per query)
embedder = SentenceTransformer("all-MiniLM-L6-v2")

# Train on historical data: (query, which_model_was_needed)
# Labels come from comparing small/large model outputs
X_train = embedder.encode(training_queries)
y_train = difficulty_labels  # "easy", "medium", "hard"

classifier = LogisticRegression().fit(X_train, y_train)

def learned_router(prompt: str) -> str:
    embedding = embedder.encode([prompt])
    difficulty = classifier.predict(embedding)[0]
    return {"easy": "8b", "medium": "70b", "hard": "405b"}[difficulty]

Accuracy: ~80-85%. The embedding model adds ~1ms latency. Training data comes from offline evaluation: run the same queries through all models, label which model was the cheapest correct one.

LLM-based classifier — ask a small model to judge difficulty:

ROUTER_PROMPT = """Rate this query's difficulty: EASY, MEDIUM, or HARD.
EASY: Simple facts, greetings, short answers.
MEDIUM: Explanations, summaries, moderate reasoning.
HARD: Multi-step reasoning, code, math proofs, creative writing.

Query: {query}
Difficulty:"""

def llm_router(prompt: str, router_model) -> str:
    response = router_model.generate(ROUTER_PROMPT.format(query=prompt))
    difficulty = response.strip().upper()
    return {"EASY": "8b", "MEDIUM": "70b", "HARD": "405b"}.get(difficulty, "70b")

Accuracy: ~85-90%. But adds 50-200ms latency (a full LLM call). Only worth it if the cost savings from correct routing exceed the router model’s cost.

Which Classifier to Use?

ClassifierLatencyAccuracyCostBest for
Rule-based<1ms60-70%$0Starting point, low-stakes
Learned (embedding + sklearn)1-5ms80-85%~$0.001/M queriesProduction at scale
LLM-based50-200ms85-90%$0.05-0.20/M queriesHigh-value, latency-tolerant

3. Training Data for Learned Routers

The hardest part: generating accurate difficulty labels. Two approaches:

Offline Model Comparison

Run a sample of production queries through all candidate models:

def generate_routing_labels(queries, models):
    """Compare model outputs to find cheapest adequate model."""
    labels = []
    for query in queries:
        outputs = {name: model.generate(query) for name, model in models.items()}
        
        # Use the largest model as reference
        reference = outputs["405b"]
        
        # Find cheapest model that matches reference quality
        # (measured by LLM-as-judge or task-specific metric)
        for model_name in ["8b", "70b", "405b"]:
            if quality_score(outputs[model_name], reference) > 0.9:
                labels.append(model_name)
                break
    return labels

Online Feedback Loop

Log routing decisions and outcomes, then retrain:

Request → Router → 8B model → user feedback (👍/👎)
  If 👎 → re-run with 70B → label this query as "medium"
  If 👍 → confirm this query as "easy"

Over time, the classifier improves as it sees more labeled data.

4. Routing in Practice: Real Systems

SystemApproachModelsKey insight
MartianLearned routerAny OpenAI/Anthropic/open modelsTrains on model-specific benchmarks, routes per-query
OpenRouterUser-selected + auto100+ modelsPrice/quality tiers, fallback chains
UnifyBenchmark-basedMultiple providersRoutes based on benchmark category match
RouteLLM (open source)Trained classifierAny pair of modelsTrains on Chatbot Arena preference data

RouteLLM Example

# RouteLLM: open-source router from LMSYS
from routellm.controller import Controller

client = Controller(
    routers=["mf"],  # matrix factorization router
    strong_model="gpt-4o",
    weak_model="gpt-4o-mini",
)

# Threshold controls cost-quality tradeoff
# Lower threshold = more queries go to strong model
response = client.chat.completions.create(
    model="router-mf-0.116",  # 0.116 = threshold
    messages=[{"role": "user", "content": query}],
)

At threshold 0.116, RouteLLM achieves 95% of GPT-4 quality while routing 50% of queries to the cheaper model — halving cost.

5. Monitoring Router Quality

A router that misclassifies “hard” queries as “easy” degrades quality silently. Monitor:

# Key metrics for router health
router_metrics = {
    # Routing distribution — should be stable
    "pct_easy": count_easy / total,      # expect ~40%
    "pct_medium": count_medium / total,   # expect ~35%  
    "pct_hard": count_hard / total,       # expect ~25%
    
    # Quality per tier — catch misrouting
    "quality_easy_tier": avg_score_easy,   # should be >0.85
    "quality_medium_tier": avg_score_medium,
    "quality_hard_tier": avg_score_hard,
    
    # Escalation rate — how often users retry after routing to small model
    "escalation_rate": retries / total,    # should be <5%
    
    # Cost savings vs always-large
    "cost_savings_pct": 1 - (routed_cost / always_large_cost),
}

Red flags:

  • escalation_rate > 10% → router is too aggressive, sending hard queries to small models
  • pct_easy suddenly drops → query distribution shifted, retrain router
  • quality_easy_tier drops → small model degraded, or easy queries got harder

Self-Check

  • Calculate cost savings from routing 50% of queries to a model 10× cheaper
  • Compare rule-based, learned, and LLM-based classifiers on latency, accuracy, and cost
  • Describe how to generate training data for a learned router using offline model comparison
  • Explain why monitoring escalation rate is critical for router health
  • Design a routing strategy for a customer support chatbot with 3 model sizes

References