Caching Layers

LLM inference is expensive — $1-5 per million tokens. Caching avoids redundant computation by reusing results from identical or similar previous requests. Three cache layers stack together: exact match (cheapest), semantic (broader hits), and KV prefix (deepest, inside the engine).

---

1. Three Cache Levels

Level	What’s cached	Hit condition	Typical hit rate	Savings per hit
L1: Exact match	(prompt hash → response)	Identical prompt	15-40% (FAQ, system prompts)	Skip entire LLM call
L2: Semantic	(similar prompt → response)	Semantically equivalent prompt	5-15% (paraphrased queries)	Skip entire LLM call
L3: KV prefix	KV cache blocks for shared prefix	Common prefix tokens	30-80% (shared system prompt)	Skip prefill computation

Request → L1 exact cache → L2 semantic cache → L3 KV prefix cache → Full inference
           15-40% hit        5-15% hit           30-80% partial hit

---

2. L1: Exact Match Cache

Cache the full response for identical prompts. The simplest and most reliable layer:

class ExactMatchCache:
    def __init__(self, redis: Redis, ttl=3600):
        self.redis = redis
        self.ttl = ttl
    
    def _cache_key(self, messages: list[dict], model: str, params: dict) -> str:
        """Hash the full request — messages + model + sampling params."""
        content = json.dumps({
            "messages": messages,
            "model": model,
            "temperature": params.get("temperature", 1.0),
            "max_tokens": params.get("max_tokens"),
        }, sort_keys=True)
        return f"llm:exact:{hashlib.sha256(content.encode()).hexdigest()}"
    
    async def get(self, messages, model, params) -> str | None:
        key = self._cache_key(messages, model, params)
        cached = await self.redis.get(key)
        if cached:
            metrics.increment("cache.exact.hit")
            return json.loads(cached)
        metrics.increment("cache.exact.miss")
        return None
    
    async def put(self, messages, model, params, response: str):
        key = self._cache_key(messages, model, params)
        await self.redis.setex(key, self.ttl, json.dumps(response))

When Exact Match Works Well

• System prompts: Same system prompt across all users → cache warmed quickly
• FAQ patterns: Customer support gets the same questions repeatedly
• Structured queries: “Convert 5 USD to EUR” → deterministic answer
• Temperature=0: Deterministic output makes caching valid

When It Doesn’t Work

• Temperature > 0: Same prompt gives different outputs (cache returns stale variety)
• Personalized context: Each user has unique context → no overlap
• Long conversations: Multi-turn conversations rarely repeat exactly

---

3. L2: Semantic Cache

Match queries that are semantically equivalent even if worded differently:

class SemanticCache:
    def __init__(self, embedder, vector_db, similarity_threshold=0.95, ttl=3600):
        self.embedder = embedder    # e.g., all-MiniLM-L6-v2
        self.vector_db = vector_db  # e.g., Qdrant
        self.threshold = similarity_threshold
        self.ttl = ttl
    
    async def get(self, messages: list[dict], model: str) -> str | None:
        # Embed the user query (last user message)
        user_query = messages[-1]["content"]
        query_vector = self.embedder.encode(user_query)
        
        # Search for similar cached queries
        results = self.vector_db.search(
            query_vector,
            filter={"model": model},
            top_k=1,
        )
        
        if results and results[0].score >= self.threshold:
            metrics.increment("cache.semantic.hit")
            return results[0].payload["response"]
        
        metrics.increment("cache.semantic.miss")
        return None
    
    async def put(self, messages, model, response: str):
        user_query = messages[-1]["content"]
        query_vector = self.embedder.encode(user_query)
        
        self.vector_db.upsert(
            vector=query_vector,
            payload={
                "query": user_query,
                "model": model,
                "response": response,
                "created_at": time.time(),
            }
        )

Similarity Threshold Tuning

Threshold	Behavior	Risk
0.99	Almost exact match only	Very few hits, safe
0.95	Close paraphrases matched	Good balance
0.90	Broader matching	Some wrong cache returns
0.85	Aggressive	High risk of returning wrong answer

Danger: “How do I delete my account?” and “How do I create an account?” may have 0.92 similarity but completely different answers. The semantic cache can return catastrophically wrong results.

Mitigation: Use semantic cache only for narrow, well-defined domains where similar queries have similar answers (e.g., product documentation lookup).

---

4. L3: KV Prefix Cache

The deepest optimization — share computed KV cache blocks across requests with common prefixes. This happens inside the inference engine:

Request A: [system prompt: 2000 tokens] + [user A: 50 tokens]
Request B: [system prompt: 2000 tokens] + [user B: 80 tokens]
                                ↑
          KV cache for 2000-token prefix computed ONCE, shared

Prefill savings: 2000 tokens of KV computation saved per request
At 1000 req/s: 2M tokens/s of prefill avoided

How It Works (Engine Level)

# In vLLM/SGLang, prefix caching is automatic:
# 1. Engine hashes token blocks
# 2. On new request, check if prefix blocks already exist
# 3. If yes, skip prefill for cached portion
# 4. Only compute attention for new (uncached) tokens

# vLLM configuration
llm = LLM(
    model="meta-llama/Llama-3.1-70B",
    enable_prefix_caching=True,   # Enable automatic prefix sharing
)

# SGLang: RadixAttention (tree-based prefix caching)
# Automatically shares prefixes across all requests in a tree structure

For detailed prefix caching internals, see Prefix Caching.

---

5. Cache Invalidation

The hardest problem in caching — when should cached responses be invalidated?

Trigger	What to invalidate	Method
Model update	All responses for that model	Flush by model version tag
System prompt change	All responses with old prompt	Include prompt hash in cache key
TTL expiry	Stale entries	Automatic (Redis TTL, vector DB TTL)
Quality issue detected	Specific bad responses	Manual purge + quality monitoring
RAG index update	Responses using outdated context	Tag cache entries with index version

async def invalidate_on_model_update(model_name: str, new_version: str):
    """Flush all cache entries for a model when it's updated."""
    # Exact cache: delete by pattern
    keys = await redis.keys(f"llm:exact:*:{model_name}:*")
    if keys:
        await redis.delete(*keys)
    
    # Semantic cache: delete by metadata filter
    vector_db.delete(filter={"model": model_name})
    
    logger.info(f"Invalidated cache for {model_name} v{new_version}")

---

6. Cache Economics

# Cost-benefit analysis
def cache_savings(
    requests_per_day: int,
    avg_tokens_per_request: int,
    cost_per_m_tokens: float,
    hit_rate: float,
    cache_cost_per_day: float,  # Redis/vector DB cost
) -> dict:
    total_tokens = requests_per_day * avg_tokens_per_request
    inference_cost = total_tokens / 1_000_000 * cost_per_m_tokens
    
    saved = inference_cost * hit_rate
    net_savings = saved - cache_cost_per_day
    
    return {
        "daily_inference_cost_without_cache": inference_cost,
        "daily_savings": saved,
        "cache_cost": cache_cost_per_day,
        "net_daily_savings": net_savings,
        "roi": net_savings / cache_cost_per_day if cache_cost_per_day > 0 else float('inf'),
    }

# Example: 1M requests/day, 500 tokens avg, $1/M tokens, 30% hit rate
result = cache_savings(1_000_000, 500, 1.0, 0.30, 50.0)
# daily_inference_cost: $500
# daily_savings: $150
# cache_cost: $50 (Redis)
# net_daily_savings: $100
# ROI: 2.0× (200% return on cache investment)

---

Self-Check

• Compare exact match, semantic, and KV prefix caching on hit rate, risk, and savings
• Implement an exact match cache with model version in the cache key
• Explain the danger of semantic caching and how threshold tuning mitigates it
• Design a cache invalidation strategy for model updates and RAG index changes
• Calculate the ROI of caching for a given request volume and hit rate

---

References

• GPTCache: Semantic Caching for LLMs
• SGLang RadixAttention
• For prefix caching internals, see Prefix Caching
• For gateway integration, see API Gateway