Optimization Levers

AI inference cost = GPU-hours consumed. Every optimization either reduces GPU-hours per request (efficiency) or reduces GPU cost per hour (procurement). Here are the levers, ranked by typical impact.

---

1. Impact Ranking

Lever	Typical savings	Quality impact	Implementation effort
Quantization (BF16→INT4)	2-4× fewer GPUs	-1-3% quality	Low (config change)
Batching optimization	1.5-3× throughput	None	Low-Medium
Model routing (big+small)	2-5× cost reduction	-0-2%	Medium
Caching (prefix + exact)	20-60% fewer calls	None	Medium
Right-sizing (GPU selection)	1.5-3× cost per token	None	Low
Speculative decoding	1.5-2.5× throughput	None (mathematically exact)	Medium
Spot instances	60-70% per-GPU savings	None (with checkpointing)	Medium
Prompt optimization	20-50% fewer tokens	Variable	High (requires tuning)

---

2. Quantization

The single biggest lever — halve memory, double concurrent requests:

# BF16: 70B model needs ~140 GB → 2× H100 (TP=2)
# INT4: 70B model needs ~35 GB → 1× H100 (TP=1)
# Cost per request: 2× reduction from one config change

# vLLM configuration
llm = LLM(
    model="meta-llama/Llama-3.1-70B",
    quantization="awq",     # or "gptq", "fp8"
    dtype="float16",
    tensor_parallel_size=1,  # TP=1 is possible with INT4
)

Quality check: Run your eval suite on quantized vs full-precision. Typical quality loss:

• INT8/FP8: <0.5% on most benchmarks
• INT4 (AWQ/GPTQ): 1-3% on most benchmarks
• INT4 on reasoning/math: 3-5% (more sensitive)

For detailed quantization guide, see Quantization Selection Guide.

---

3. Batching Optimization

Higher batch utilization = more tokens per GPU-second:

# vLLM settings that affect batching efficiency
engine_config = {
    "max_num_seqs": 256,          # Max concurrent sequences (higher = better batching)
    "max_num_batched_tokens": 8192, # Max tokens in one batch
    "enable_chunked_prefill": True, # Don't let long prefills block decodes
    "gpu_memory_utilization": 0.92, # More KV cache = more concurrent requests
}

# Throughput vs batch size (Llama-70B INT4, H100):
# Batch 1:   ~50 tok/s   (GPU barely utilized)
# Batch 16:  ~400 tok/s  (decent utilization)
# Batch 64:  ~1200 tok/s (good utilization)
# Batch 256: ~2500 tok/s (near-optimal)

---

4. Caching

Avoid redundant computation:

Cache type	Hit rate	Setup effort	Savings
Prefix KV cache	30-80%	Built into engine	Skip prefill for shared prefix
Exact match cache (Redis)	15-40%	Medium	Skip entire LLM call
Semantic cache	5-15%	High	Skip entire LLM call

# Enable prefix caching in vLLM
llm = LLM(model=..., enable_prefix_caching=True)

# For workloads with shared system prompts:
# 2000-token system prompt × 1000 req/s = 2M tokens/s saved in prefill
# At $1/M tokens: $172K/day saved

See Caching Layers for implementation details.

---

5. Right-Sizing GPU Selection

Don’t use H100 for workloads that run fine on cheaper hardware:

Workload	Optimal GPU	Reason
70B+ inference (latency-sensitive)	H100, H200	Need HBM bandwidth + Tensor Cores
8B inference	L40S, A100	Doesn’t need H100’s bandwidth
Embedding (batch)	L40S, T4	Compute-bound, not bandwidth-bound
Fine-tuning (LoRA)	A100, L40S	Memory matters more than peak compute
Pre-training	H100, H200	Need everything: compute + memory + interconnect

# Cost comparison: 8B model inference
# H100: $12.29/hr, serves 5000 tok/s → $0.0025/1K tokens
# L40S: $4.85/hr,  serves 2000 tok/s → $0.0024/1K tokens  
# T4:   $1.40/hr,  serves 200 tok/s  → $0.0070/1K tokens
# → L40S is cheapest per token for 8B models

---

6. Prompt Engineering for Cost

Shorter prompts = fewer input tokens = less cost:

# Before: 800 tokens system prompt
VERBOSE_PROMPT = """You are a helpful customer support assistant. 
You work for Acme Corp. Your job is to help customers with their 
questions about our products. Please be polite, professional, and 
thorough in your responses. If you don't know the answer, say so...
[400 more words of instructions]"""

# After: 200 tokens, same behavior
CONCISE_PROMPT = """You are Acme Corp's support assistant. Be helpful, 
professional, and concise. If unsure, say so. Key policies:
- Refunds: within 30 days, original payment method
- Shipping: 3-5 business days standard, 1-2 express
- Technical: escalate to engineering if unresolved in 2 messages"""

# Savings: 600 tokens × 1000 req/hr × $0.50/M tokens = $0.30/hr
# Annualized: $2,628/year per prompt template optimized

---

7. Waterfall Analysis

Show cumulative savings from stacking optimizations:

Baseline: 70B BF16, TP=4, no caching, round-robin
  Cost: $5.00/M tokens

Apply INT4 quantization (TP=4 → TP=1):
  Cost: $1.50/M tokens (-70%)

Enable prefix caching (40% hit rate):
  Cost: $1.05/M tokens (-30% of remaining)

Add model routing (50% to 8B):
  Cost: $0.65/M tokens (-38% of remaining)

Enable speculative decoding (1.8× throughput):
  Cost: $0.45/M tokens (-31% of remaining)

Optimize prompts (-30% tokens):
  Cost: $0.35/M tokens (-22% of remaining)

────────────────────────────────────
Total: 93% cost reduction ($5.00 → $0.35)

---

Self-Check

• Rank optimization levers by impact for your specific workload
• Calculate savings from INT4 quantization on a 70B model
• Estimate prefix cache savings for a workload with shared system prompts
• Right-size GPU selection for an 8B embedding workload
• Build a waterfall analysis showing cumulative savings from stacked optimizations

---

References

• For quantization details, see Quantization Selection Guide
• For caching details, see Caching Layers
• For model routing, see Model Routing
• For batching, see Continuous Batching