Navigation
🤖 Apps & Agents· 10 min read

Cost-Quality Optimization

Pareto curves, finding the optimal model mix for a given budget and quality target, and dynamic adjustment under load.

cost-optimizationparetomodel-selectionquality-budgetmulti-model

Given a dozen model options (different sizes, quantization levels, providers), which combination minimizes cost while meeting your quality bar? Cost-quality optimization turns this from guesswork into a systematic analysis using Pareto curves and constrained optimization.

1. The Configuration Space

Every model variant has a cost and quality point:

ConfigurationQuality (MT-Bench)Cost/M tokensLatency P50GPUs
Llama-405B BF16, TP=89.1$5.001200ms8× H100
Llama-405B INT4, TP=48.9$2.80900ms4× H100
Llama-70B BF16, TP=48.4$1.50600ms4× H100
Llama-70B INT4, TP=18.2$0.55400ms1× H100
Llama-70B INT4, TP=1 (A100)8.2$0.40500ms1× A100
Llama-8B BF16, 1 GPU7.2$0.22150ms1× H100
Llama-8B INT4, 1 GPU7.0$0.12100ms1× H100
GPT-4o (API)9.0$2.50800ms— (API)
GPT-4o-mini (API)7.8$0.15300ms— (API)
Claude 3.5 Sonnet (API)8.8$3.00700ms— (API)

That’s 10 options — and it gets worse when you add caching, batching, and speculative decoding variants.

2. Pareto Frontier

Pareto Frontier

The Pareto frontier is the set of configurations where you can’t improve quality without increasing cost (or vice versa):

def compute_pareto_frontier(configs):
    """Find Pareto-optimal configurations (lower cost, higher quality)."""
    # Sort by cost ascending
    sorted_configs = sorted(configs, key=lambda c: c.cost)
    
    pareto = []
    best_quality = -float('inf')
    
    for config in sorted_configs:
        if config.quality > best_quality:
            pareto.append(config)
            best_quality = config.quality
    
    return pareto

# Example result:
# Pareto frontier: 8B-INT4 ($0.12, 7.0) → 70B-INT4 ($0.55, 8.2) → 405B-INT4 ($2.80, 8.9) → 405B-BF16 ($5.00, 9.1)
# 
# NOT on frontier: 70B-BF16 ($1.50, 8.4) — 70B-INT4 is cheaper at similar quality
# NOT on frontier: GPT-4o ($2.50, 9.0) — 405B-INT4 is cheaper at similar quality

Configurations below the Pareto curve are dominated — there exists a cheaper option at equal or better quality. Never use a dominated configuration.

3. Choosing Your Operating Point

Once you have the Pareto frontier, your operating point depends on constraints:

Given a Quality Floor → Minimize Cost

def cheapest_above_quality(pareto, quality_floor):
    """Find cheapest Pareto-optimal config above quality threshold."""
    candidates = [c for c in pareto if c.quality >= quality_floor]
    if not candidates:
        raise ValueError(f"No config meets quality floor {quality_floor}")
    return min(candidates, key=lambda c: c.cost)

# "I need MT-Bench ≥ 8.0"
# → 70B-INT4 ($0.55, 8.2) — cheapest option above 8.0

# "I need MT-Bench ≥ 8.5"  
# → 405B-INT4 ($2.80, 8.9) — must jump to 405B

Given a Budget → Maximize Quality

def best_quality_under_budget(pareto, budget_per_m_tokens):
    """Find highest-quality config within budget."""
    candidates = [c for c in pareto if c.cost <= budget_per_m_tokens]
    if not candidates:
        raise ValueError(f"No config fits budget ${budget_per_m_tokens}/M tokens")
    return max(candidates, key=lambda c: c.quality)

# Budget: $1.00/M tokens
# → 70B-INT4 ($0.55, 8.2) — best quality under $1.00

Routing as a Mixed Strategy

Routing across multiple models creates intermediate operating points between Pareto-optimal configurations:

Mixed quality=∑ipi×qi,Mixed cost=∑ipi×ci\text{Mixed quality} = \sum_i p_i \times q_i, \quad \text{Mixed cost} = \sum_i p_i \times c_i

where pip_i is the fraction of traffic to model ii.

# Example: Route 60% to 8B, 40% to 70B
mixed_quality = 0.60 * 7.0 + 0.40 * 8.2  # = 7.48
mixed_cost    = 0.60 * 0.12 + 0.40 * 0.55 # = $0.29/M tokens

# This creates a new point (7.48, $0.29) between the two Pareto configs
# Useful when you need quality between 7.0 and 8.2 at intermediate cost

4. Building the Quality Benchmark

The Pareto curve is only as good as your quality measurement. Generic benchmarks (MMLU, MT-Bench) may not reflect your workload.

Custom Evaluation Set

# Build evaluation set from real production queries
eval_set = {
    "customer_support": [
        {"prompt": "How do I reset my password?", "reference": "...", "weight": 0.40},
        {"prompt": "My order hasn't arrived after 2 weeks", "reference": "...", "weight": 0.30},
        # ...
    ],
    "technical_qa": [
        {"prompt": "Explain OAuth 2.0 PKCE flow", "reference": "...", "weight": 0.20},
        # ...
    ],
    "content_generation": [
        {"prompt": "Write a product description for...", "reference": "...", "weight": 0.10},
        # ...
    ],
}

def evaluate_config(config, eval_set):
    """Score a model configuration on your custom benchmark."""
    scores = []
    for category, items in eval_set.items():
        for item in items:
            output = config.model.generate(item["prompt"])
            score = judge_quality(output, item["reference"])  # LLM-as-judge or custom metric
            scores.append(score * item["weight"])
    return sum(scores) / sum(item["weight"] for cat in eval_set.values() for item in cat)

Evaluation Dimensions

Quality isn’t one number. Track multiple dimensions:

DimensionMetricMinimum acceptable
CorrectnessFactual accuracy (LLM judge)90%
Instruction followingIFEval-style format compliance85%
SafetyRefusal on harmful queries99%
LatencyP99 TTFT<2s
ConsistencySame question → similar answer80% agreement

A model might score high on correctness but fail on safety — quality is a vector, not a scalar.

5. Dynamic Adjustment Under Load

The optimal operating point shifts with traffic:

class DynamicCostQualityOptimizer:
    """Adjust routing based on real-time conditions."""
    
    def __init__(self, pareto_configs, quality_floor, budget_per_hour):
        self.configs = pareto_configs
        self.quality_floor = quality_floor
        self.budget_per_hour = budget_per_hour
    
    def get_routing_weights(self, current_load, current_spend_rate):
        """Adjust routing based on current conditions."""
        budget_remaining_ratio = 1 - (current_spend_rate / self.budget_per_hour)
        
        if budget_remaining_ratio < 0.1:
            # Running out of budget — shift traffic to cheaper models
            return self._budget_saving_weights()
        
        if current_load > self.peak_threshold:
            # High load — route to faster (smaller) models to reduce queue
            return self._low_latency_weights()
        
        # Normal operation — optimal cost-quality point
        return self._optimal_weights()
    
    def _budget_saving_weights(self):
        """Emergency: maximize 8B usage."""
        return {"8b": 0.90, "70b": 0.10, "405b": 0.00}
    
    def _low_latency_weights(self):
        """High load: minimize latency."""
        return {"8b": 0.70, "70b": 0.25, "405b": 0.05}
    
    def _optimal_weights(self):
        """Normal: best cost-quality balance."""
        return {"8b": 0.40, "70b": 0.35, "405b": 0.25}

Time-of-Day Optimization

Traffic patterns are predictable — adjust proactively:

TimeTrafficStrategyRationale
Business hours (9am-5pm)HighMore 8B, less 405BManage queue depth
Evening (5pm-10pm)MediumBalanced routingNormal optimization
Night (10pm-6am)LowMore 405BGPU capacity is idle anyway
Batch processing (weekend)VariableAll 70BBest throughput/quality balance

6. Worked Example: Customer Support Bot

Requirements: Quality ≥ 8.0/10, budget ≤ $500/day, peak 50 req/s.

Step 1: Evaluate all models on your support-specific benchmark:

ModelSupport qualityCost/M tokensThroughput
8B-INT47.2$0.12200 tok/s/GPU
70B-INT48.5$0.5550 tok/s/GPU
405B-INT49.0$2.8012 tok/s/GPU

Step 2: 8B alone doesn’t meet quality (7.2 < 8.0). 70B alone: 0.55×2Mtokens/day=0.55 × 2M tokens/day = 1,100/day (over budget).

Step 3: Use cascading 8B → 70B with 65% hit rate:

  • Cost: 0.65×0.65 × 0.12 + 0.35 × (0.12+0.12 + 0.55) = $0.31/M tokens
  • Daily: 0.31×2M=0.31 × 2M = 620/day — still over budget

Step 4: Add routing to reduce escalation. Train router to identify easy queries:

  • 50% → 8B (quality 7.2, but these are easy queries where 8B gets 8.5+)
  • 35% → 70B
  • 15% → 405B (complex complaints, escalations)
  • Cost: 0.50×0.50 × 0.12 + 0.35 × 0.55+0.15×0.55 + 0.15 × 2.80 = $0.67/M tokens… actually worse

Step 5: Cascade 8B → 70B only (no 405B), accept quality dip on hardest 5%:

  • Cost: 70% hit → 0.31/Mtokens→0.31/M tokens → 620/day
  • Reduce avg output length by improving prompts: 2M → 1.5M tokens/day → $465/day ✓

Result: Cascading 8B → 70B, prompt optimization to reduce token count, $465/day, quality 8.3/10.

Self-Check

  • Plot a Pareto frontier from a set of model configurations
  • Find the cheapest configuration above a quality threshold
  • Calculate the cost and quality of a mixed routing strategy
  • Explain why custom evaluation sets matter more than generic benchmarks
  • Design a dynamic optimizer that adjusts routing based on budget remaining

References