llm-d

llm-d is a Kubernetes-native distributed inference stack for production LLM deployments. While vLLM/SGLang handle single-instance serving and Dynamo handles disaggregation, llm-d focuses on operational excellence — routing, autoscaling, multi-tenancy, and reliability at cluster scale.

---

1. What llm-d Provides

llm-d sits above model servers (vLLM, SGLang) and provides the production infrastructure layer:

┌────────────────────────────────────────────────────┐
│  External Traffic (OpenAI-compatible API)           │
├────────────────────────────────────────────────────┤
│  llm-d Gateway                                     │
│  ├── Intelligent Router                            │
│  │   Prefix-cache-aware, load-aware, SLO-aware     │
│  ├── Flow Control                                  │
│  │   Rate limiting, priority queues, multi-tenant  │
│  └── Autoscaler                                    │
│      SLO-driven scaling, predictive scaling         │
├────────────────────────────────────────────────────┤
│  KV Cache Index (global)                           │
│  Tracks which prefix KV exists on which instance   │
│  Tiered offload: GPU → CPU → disk                  │
├──────────────┬──────────────┬──────────────────────┤
│  vLLM Pod 0  │  vLLM Pod 1  │  vLLM Pod 2  │ ...  │
│  (Kubernetes) │              │              │      │
└──────────────┴──────────────┴──────────────────────┘

---

2. Intelligent Routing

Prefix-Cache-Aware Routing

The router knows which KV cache each instance has:

Request: "You are a helpful assistant... What is CUDA?"

Route decision:
  Pod 0: has "You are a helpful assistant..." cached → cache HIT
  Pod 1: different system prompt cached → cache MISS
  Pod 2: cold (just scaled up) → cache MISS

→ Route to Pod 0: skip 2K tokens of prefill

This transforms prefix caching from a per-instance optimization to a cluster-wide optimization. Even with many instances, common prefixes are efficiently served.

Load-Aware Routing

Beyond cache affinity, the router considers:

• KV cache utilization: Don’t route to a pod that’s almost out of KV blocks
• Queue depth: Prefer pods with shorter request queues
• Predicted latency: Estimate TTFT based on prompt length and pod state

SLO-Aware Routing

For multi-tenant deployments:

tenants:
  premium:
    slo:
      ttft_p99: 500ms
      tpot_p99: 50ms
    priority: high
    
  standard:
    slo:
      ttft_p99: 2000ms
      tpot_p99: 100ms
    priority: normal

Premium requests are routed preferentially to less-loaded pods. Standard requests absorb more queuing to protect premium SLOs.

---

3. Tiered KV Cache Management

llm-d provides a global KV cache index across all pods:

Global KV Index:
  Hash("sys_prompt_v2 tokens 0-500") → Pod 0, blocks [7,12,15]
  Hash("sys_prompt_v2 tokens 0-500") → Pod 3, blocks [2,8,11]  (replicated)
  Hash("few_shot_math tokens 501-1200") → Pod 0, blocks [20,21,22]

When Pod 0's GPU memory fills:
  Offload least-recently-used KV to CPU memory (Pod 0)
  Still tracked in global index → can be restored if needed
  
When CPU fills:
  Offload to local disk
  Much slower to restore, but still cheaper than re-prefill

---

4. Autoscaling

SLO-Driven Scaling

Traditional autoscaling uses GPU utilization as the signal. llm-d uses inference-aware signals:

Signal	What it means	Action
P99 TTFT rising	Too many prefill requests queued	Scale up prefill capacity
P99 TPOT rising	Too many concurrent decode requests	Scale up decode capacity
KV cache utilization >90%	Running out of memory for new requests	Scale up (more memory)
GPU compute <30%	Over-provisioned	Scale down

Batch Processing

For offline workloads (evaluation, data processing):

# llm-d batch job
apiVersion: batch.llm-d.ai/v1
kind: InferenceBatch
spec:
  model: meta-llama/Llama-3.1-70B
  input: s3://my-bucket/prompts.jsonl
  output: s3://my-bucket/results.jsonl
  config:
    max_tokens: 1024
    temperature: 0.0
  resources:
    gpu_type: h100
    min_replicas: 4
    max_replicas: 16

Batch jobs use spot/preemptible instances and scale aggressively for throughput.

---

5. Large Model Serving

For models that require multi-node serving (DeepSeek-R1, GPT-class):

Wide Expert Parallelism

llm-d orchestrates MoE expert parallelism across nodes:

DeepSeek-R1 (671B MoE):
  Node 0-3: Attention layers (TP=4 per node)
  Node 4-11: Expert layers (EP=64 across 8 nodes)
  
llm-d manages:
  - Expert placement and load balancing
  - All-to-All routing for token dispatch
  - Autoscaling expert nodes based on routing patterns

---

6. Relationship to Other Projects

┌─────────────────────────────────────────────────┐
│  llm-d                                          │
│  (Kubernetes orchestration, routing, autoscaling)│
├─────────────────────────────────────────────────┤
│  Dynamo (optional)                              │
│  (PD disaggregation, KV transfer engine)         │
├─────────────────────────────────────────────────┤
│  LMCache / Mooncake (optional)                  │
│  (Tiered KV storage, cross-instance sharing)     │
├─────────────────────────────────────────────────┤
│  vLLM / SGLang                                  │
│  (Single-instance inference engine)              │
└─────────────────────────────────────────────────┘

These layers are composable — you can use llm-d with vLLM alone (simplest), or add Dynamo for disaggregation, or add LMCache for tiered storage. Each layer addresses a different concern.

---

For the PD disaggregation that llm-d orchestrates, see Dynamo. For the engines underneath, see vLLM Architecture.

---

Self-Check

• Explain what llm-d adds above vLLM/SGLang (routing, autoscaling, multi-tenancy)
• Describe prefix-cache-aware routing and its cluster-wide benefit
• Explain SLO-driven autoscaling signals (TTFT, TPOT, KV utilization)
• Describe the global KV cache index and tiered offload
• Draw the stack: llm-d → Dynamo → LMCache → vLLM/SGLang

---

References

• llm-d Project
• llm-d Documentation
• CNCF Sandbox Project