Dynamo

Dynamo is NVIDIA’s open-source framework for disaggregated LLM inference — it orchestrates the split between prefill and decode workers, manages KV cache routing, and handles the distributed systems complexity that individual engines (vLLM, SGLang) don’t address.

---

1. What Dynamo Adds Above the Engine

vLLM and SGLang are single-node engines. Dynamo adds the multi-node orchestration layer:

Without Dynamo:
  [vLLM instance 0] [vLLM instance 1] [vLLM instance 2]
  Each independent, no KV sharing, no PD split
  Load balancer does round-robin → misses cache locality

With Dynamo:
  ┌──────────────────────────────────────────────┐
  │  Dynamo Planner                              │
  │  Request routing, KV placement, PD assignment│
  ├──────────────┬───────────────────────────────┤
  │  KV Router   │  GPU Memory Service (KVBM)   │
  │  Cache-aware │  Block-level KV management    │
  │  routing     │  across all workers           │
  ├──────────────┴───────────────────────────────┤
  │  Prefill Workers     │  Decode Workers       │
  │  (vLLM/SGLang)       │  (vLLM/SGLang)        │
  └──────────────────────┴───────────────────────┘

---

2. Core Components

Planner

The Planner is the brain — it decides:

• Which worker handles which request
• Whether to assign prefill and decode to different workers (PD disaggregation)
• Which worker has the most relevant cached KV (cache-affinity routing)

KV Router

Routes requests based on KV cache state:

Request arrives: "System prompt A + User query X"

KV Router checks:
  Worker 0: has "System prompt A" cached (2K tokens) ✅
  Worker 1: has "System prompt B" cached (no match)
  Worker 2: no cache

→ Route to Worker 0: only needs to prefill "User query X" (skip 2K tokens)

This is the key optimization — without KV-aware routing, a random load balancer would send the request to any worker, causing redundant prefill of the system prompt.

GPU Memory Service (KV Block Manager)

Dynamo’s KV Block Manager (KVBM) provides a cluster-wide view of KV cache blocks:

KVBM tracks:
  Worker 0: 500/1000 blocks used, 200 blocks = "System prompt A" KV
  Worker 1: 800/1000 blocks used, 150 blocks = "System prompt B" KV
  Worker 2: 100/1000 blocks used (mostly idle)
  
Decisions:
  - New "System prompt A" request → Worker 0 (cache hit)
  - Memory-heavy request → Worker 2 (most free blocks)
  - Worker 1 overloaded → migrate some requests to Worker 2

---

3. PD Disaggregation in Dynamo

Dynamo’s primary use case is managing PD split at scale:

1. Request arrives at Planner
2. Planner assigns to Prefill Worker (choose: most compute-available)
3. Prefill Worker runs prefill → KV cache produced
4. KV cache transferred to Decode Worker via Dynamo's transfer layer
5. Decode Worker generates tokens
6. When complete, KV freed on both workers

Key: Planner balances prefill and decode workers independently
  Prefill pool can be smaller (bursts of compute)
  Decode pool can be larger (many concurrent long requests)

Why Disaggregate?

	Unified (same GPU)	Disaggregated (PD split)
Prefill interference	Long prefill blocks decode for all requests	Prefill and decode don’t interfere
GPU utilization	Prefill compute-bound, decode memory-bound — both on same GPU	Each pool optimized for its workload
Throughput	Limited by worst-case interference	1.5-2× higher at scale
Complexity	Simple	Requires KV transfer infrastructure

---

4. Integration with Engines

Dynamo doesn’t replace vLLM or SGLang — it orchestrates them:

# Dynamo deployment config (simplified)
deployment:
  prefill_pool:
    engine: vllm
    model: meta-llama/Llama-3.1-70B
    tensor_parallel: 4
    replicas: 4
    gpu_type: h100  # compute-optimized
    
  decode_pool:
    engine: vllm
    model: meta-llama/Llama-3.1-70B
    tensor_parallel: 4
    replicas: 8
    gpu_type: h100
    max_concurrent_requests: 256
    
  kv_transfer:
    backend: rdma  # or tcp
    
  router:
    cache_aware: true
    load_balancing: least_kv_blocks

---

5. When to Use Dynamo

Scenario	Dynamo needed?
Single vLLM instance, small model	No — overkill
Multiple instances, same model, need cache sharing	Yes — KV-aware routing
PD disaggregation at scale	Yes — core use case
Mixed workloads (some prefill-heavy, some decode-heavy)	Yes — separate pools
Kubernetes deployment (need autoscaling, routing)	Consider llm-d instead (or with Dynamo)

For the conceptual foundation of PD disaggregation, see Prefill-Decode Disaggregation (covered in the Distributed Inference section).

---

For the engines Dynamo orchestrates, see vLLM Architecture and SGLang Architecture. For KV cache infrastructure that complements Dynamo, see LMCache & Mooncake.

---

Self-Check

• Explain what Dynamo adds above a single vLLM/SGLang instance
• Describe the Planner, KV Router, and GPU Memory Service roles
• Explain KV-aware routing and why it avoids redundant prefill
• Describe the PD disaggregation flow in Dynamo
• Configure a Dynamo deployment with prefill and decode pools

---

References

• NVIDIA Dynamo
• Dynamo Documentation