Cascading

Cascading is the simplest multi-model strategy: try the cheapest model first, check if the answer is good enough, and escalate to a bigger model only when needed. No separate classifier — the model’s own output tells you whether to trust it.

---

1. Core Idea

Request → 8B model → confidence check
  ├── confidence ≥ threshold → return 8B result     (70% of requests)
  └── confidence < threshold → 70B model → return    (30% of requests)

Why this works: Most requests are easy. If 70% of queries are handled by the 8B model, the average cost drops dramatically:

$$\text{Cascade cost} = 0.70 \times C_{8B} + 0.30 \times (C_{8B} + C_{70B})$$

Since $C_{8B} \ll C_{70B}$: the 30% of escalated queries pay for two model calls, but the 70% savings more than compensate.

Strategy	Avg cost/M tokens	Quality vs always-70B
Always 70B	$1.11	Baseline
Always 8B	$0.22	-15% quality
Cascade 8B → 70B (70/30 split)	$0.55	-1% quality
Cascade 8B → 70B → 405B	$0.80	+2% quality (hard queries get 405B)

---

2. Confidence Detection Methods

The key challenge: how do you know the small model’s answer is good enough?

Token Probability Thresholds

Use the model’s output log-probabilities as a confidence signal:

def check_confidence_logprobs(response, threshold=0.85):
    """Check if model is confident based on average token probability."""
    token_probs = [math.exp(lp) for lp in response.logprobs]
    
    # Average probability of generated tokens
    avg_prob = sum(token_probs) / len(token_probs)
    
    # Also check minimum — one very uncertain token is a red flag
    min_prob = min(token_probs)
    
    return avg_prob > threshold and min_prob > 0.3

Pros: Zero extra cost — logprobs come free with generation.
Cons: Models can be confidently wrong (hallucination). High confidence doesn’t guarantee correctness.

Self-Consistency Checking

Generate multiple responses and check agreement:

def check_confidence_consistency(prompt, model, n_samples=3, temperature=0.7):
    """Generate multiple responses, check if they agree."""
    responses = [model.generate(prompt, temperature=temperature) for _ in range(n_samples)]
    
    # For structured outputs (code, math): exact match
    if is_structured(responses[0]):
        return len(set(responses)) == 1  # all identical = confident
    
    # For free-text: semantic similarity
    embeddings = embed(responses)
    avg_similarity = mean_pairwise_cosine(embeddings)
    return avg_similarity > 0.9

Pros: Catches hallucination — if the model gives different answers each time, it’s uncertain.
Cons: 3× the cost of a single call. Only worth it if the escalation model is 10×+ more expensive.

Output Validation

Check if the output meets structural requirements:

def check_confidence_validation(response, expected_format):
    """Validate output structure — invalid format = low confidence."""
    checks = {
        "json": lambda r: is_valid_json(r),
        "code": lambda r: compiles_without_error(r),
        "number": lambda r: is_numeric(r),
        "regex": lambda r: re.match(expected_format, r) is not None,
    }
    return checks.get(expected_format, lambda r: True)(response)

Pros: Ground truth for structured tasks — if the code doesn’t compile, escalate.
Cons: Only works for tasks with checkable outputs.

Which Method to Use?

Method	Cost overhead	Catches hallucination?	Best for
Token logprobs	0%	No	General-purpose, fast
Self-consistency	2-3× first model	Yes	High-stakes, factual
Output validation	0%	Partially	Code, JSON, structured
Combined	0-3×	Best	Production systems

---

3. Cascading Chain Implementation

A production cascade controller:

class CascadeController:
    def __init__(self, models, thresholds):
        """
        models: ordered list of (name, client) from cheapest to most expensive
        thresholds: confidence threshold for each model (except last)
        """
        self.models = models          # [("8b", client_8b), ("70b", client_70b), ("405b", client_405b)]
        self.thresholds = thresholds  # [0.85, 0.80] — last model always used if reached
    
    def generate(self, prompt: str) -> CascadeResult:
        for i, (name, client) in enumerate(self.models):
            response = client.generate(prompt, logprobs=True)
            
            # Last model in chain — always accept
            if i == len(self.models) - 1:
                return CascadeResult(response, model=name, escalations=i)
            
            # Check confidence
            confidence = self._compute_confidence(response)
            if confidence >= self.thresholds[i]:
                return CascadeResult(response, model=name, escalations=i)
            
            # Log escalation for monitoring
            logger.info(f"Escalating from {name} (confidence={confidence:.3f})")
        
    def _compute_confidence(self, response):
        """Combine multiple signals."""
        logprob_conf = avg_token_probability(response.logprobs)
        format_conf = 1.0 if passes_validation(response.text) else 0.0
        return 0.7 * logprob_conf + 0.3 * format_conf

# Usage
cascade = CascadeController(
    models=[("8b", llm_8b), ("70b", llm_70b), ("405b", llm_405b)],
    thresholds=[0.85, 0.80],
)
result = cascade.generate("Explain the Riemann hypothesis")
# → Tries 8B, confidence=0.62 (too low) → escalates to 70B → confidence=0.91 → returns

---

4. Latency Tradeoff

Cascading’s hidden cost: failed attempts add latency.

Direct to 70B:     |--- 70B generation (800ms) ---|
                    Total: 800ms

Cascade (8B hit):   |-- 8B (200ms) --|
                    Total: 200ms ✓ (faster!)

Cascade (8B miss):  |-- 8B (200ms) --||--- 70B (800ms) ---|
                    Total: 1000ms ✗ (slower!)

Average latency depends on hit rate:

$$\text{Avg latency} = p_{hit} \times L_{small} + (1 - p_{hit}) \times (L_{small} + L_{large})$$

With 70% hit rate: $0.7 \times 200 + 0.3 \times 1000 = 440\text{ms}$ — still faster than always-70B (800ms).

Break-even: cascading improves average latency when:

$$p_{hit} > \frac{L_{small}}{L_{large}}$$

If $L_{small}/L_{large} = 200/800 = 0.25$, you need >25% hit rate for latency improvement. Below that, you’re paying latency for cost savings.

Parallel Speculation (Advanced)

Run both models simultaneously, cancel the large model if the small one is confident:

async def parallel_cascade(prompt, small_model, large_model, threshold):
    """Run both models, return cheapest confident result."""
    small_task = asyncio.create_task(small_model.generate(prompt))
    large_task = asyncio.create_task(large_model.generate(prompt))
    
    # Wait for small model first (faster)
    small_result = await small_task
    if confidence(small_result) >= threshold:
        large_task.cancel()  # Don't waste GPU time
        return small_result
    
    # Small model uncertain — wait for large model
    return await large_task

Latency: Always equal to $\max(L_{small}, L_{large}) = L_{large}$. No latency penalty for escalation.
Cost: Always pay for both models (but can cancel large model early if small finishes first with high confidence, saving some tokens).

---

5. When NOT to Cascade

Scenario	Problem	Alternative
Latency-critical (<200ms budget)	Can’t afford failed attempts	Pre-routing with fast classifier
Streaming required	Can’t check confidence until generation completes	Pre-routing or speculative with parallel
All queries are hard	Hit rate is low, cascade just adds cost	Use the large model directly
Quality is paramount	Even 1% quality drop unacceptable	Large model only

---

6. Tuning the Confidence Threshold

The threshold controls the cost-quality tradeoff:

Threshold	Effect	Hit rate	Quality	Cost
0.99 (very high)	Almost everything escalates	~10%	Near-large-model	Near-large-model
0.85 (balanced)	Good balance	~65-75%	-1-2% vs large	~50% of large
0.50 (aggressive)	Almost everything accepted	~95%	-5-10% vs large	Near-small-model

Finding the right threshold: sweep thresholds on a held-out evaluation set:

def find_optimal_threshold(eval_set, small_model, large_model, quality_floor=0.95):
    """Find threshold that minimizes cost while maintaining quality."""
    results = []
    for threshold in [0.5, 0.6, 0.7, 0.8, 0.85, 0.9, 0.95]:
        hits, misses, quality_scores = 0, 0, []
        for prompt, reference in eval_set:
            small_resp = small_model.generate(prompt, logprobs=True)
            conf = avg_token_probability(small_resp.logprobs)
            
            if conf >= threshold:
                hits += 1
                quality_scores.append(score(small_resp, reference))
            else:
                misses += 1
                large_resp = large_model.generate(prompt)
                quality_scores.append(score(large_resp, reference))
        
        avg_quality = mean(quality_scores)
        hit_rate = hits / (hits + misses)
        avg_cost = hit_rate * COST_SMALL + (1 - hit_rate) * (COST_SMALL + COST_LARGE)
        results.append((threshold, hit_rate, avg_quality, avg_cost))
    
    # Pick lowest cost above quality floor
    valid = [(t, hr, q, c) for t, hr, q, c in results if q >= quality_floor]
    return min(valid, key=lambda x: x[3])  # minimize cost

---

Self-Check

• Calculate average cost and latency for a cascade with 70% hit rate
• Compare token logprobs, self-consistency, and output validation as confidence signals
• Explain the latency break-even condition for cascading
• Implement a cascade controller with configurable threshold
• Describe when cascading is worse than pre-routing

---

References

• FrugalGPT: How to Use Large Language Models While Reducing Cost (Chen et al., 2023)
• Cascade Routing for LLMs (2024)
• Let’s Verify Step by Step — confidence estimation