Prefill-Decode Disaggregation

Prefill and decode have fundamentally different compute characteristics — mixing them on the same GPU causes interference. Disaggregation separates them into specialized pools, connected by KV cache transfer.

---

1. The Interference Problem

When prefill and decode share a GPU, long prefills block decode for all co-located requests:

Metric	Unified (mixed)	Impact
Decode P99 TPOT	3-5× worse during long prefills	Users see freezing/stuttering
Prefill throughput	Reduced by decode interruptions	Lower overall processing
GPU utilization	Suboptimal (neither workload fully served)	Wasted capacity

Even with chunked prefill (which mitigates the worst case), there’s still interference because prefill chunks consume compute budget that decode needs.

---

2. The Disaggregated Architecture

                    ┌────────────────────────┐
                    │  Request Router        │
                    │  (cache-aware, SLO)    │
                    └──────┬─────────────────┘
                           │
              ┌────────────┴────────────┐
              │                         │
    ┌─────────▼─────────┐   ┌──────────▼──────────┐
    │  Prefill Pool      │   │  Decode Pool         │
    │  (compute-optimized)│  │  (memory-optimized)   │
    │  • High batch size  │   │  • High concurrency  │
    │  • Short-lived req  │   │  • Long-lived req    │
    │  • GPU: compute-hot │   │  • GPU: memory-full  │
    └─────────┬──────────┘   └──────────▲──────────┘
              │                         │
              └───── KV Cache Transfer ─┘
                    (RDMA / NVLink / TCP)

How It Works

1. Request arrives at router
2. Router assigns request to a prefill worker (based on compute availability)
3. Prefill worker processes the entire prompt → produces KV cache
4. KV cache transferred to a decode worker (via RDMA, shared memory, or TCP)
5. Decode worker continues token generation using the transferred KV
6. When complete, decode worker frees KV and slot

---

3. KV Cache Transfer: The Critical Path

The transfer time determines whether disaggregation is practical:

$$T_{\text{transfer}} = \frac{\text{KV cache size}}{\text{transfer bandwidth}}$$

Model	Context	KV size	RDMA (50 GB/s)	TCP (10 GB/s)
70B (GQA-8)	2K	0.63 GB	13 ms	63 ms
70B (GQA-8)	8K	2.5 GB	50 ms	250 ms
70B (GQA-8)	32K	10 GB	200 ms	1000 ms

RDMA is essential for disaggregation — TCP transfer adds hundreds of milliseconds to TTFT, often negating the benefit.

KV Connector Abstraction

vLLM’s KV Connector provides a pluggable interface for transfer:

# Transfer backends:
# - NIXL: NVIDIA's high-performance transfer library
# - NCCL: GPU-to-GPU via collective communication
# - Shared memory: same-node transfer (fastest)
# - TCP: fallback for cross-datacenter

---

4. Key Research: DistServe and Splitwise

DistServe (OSDI 2024)

Systematic analysis proving that disaggregation improves goodput (throughput meeting SLA targets):

• Key finding: Under the same SLA (e.g., P99 TPOT < 100ms), disaggregated serving achieves 1.5-2× the goodput of unified serving
• Mechanism: Decode pool runs without prefill interference → P99 TPOT stays flat even under load
• Architecture: Prefill and decode as separate vLLM instances with KV transfer via NCCL

Splitwise (ISCA 2024)

Proposes optimizing GPU pool composition for each phase:

• Prefill pool: Compute-optimized GPUs (high TFLOPS, can use older/cheaper GPUs)
• Decode pool: Memory-optimized GPUs (high HBM bandwidth, more KV cache capacity)
• Heterogeneous hardware: Different GPU types for different phases

---

5. Goodput and SLO-Aware Scheduling

Goodput = throughput of requests that meet the SLA target. It’s the metric that matters:

$$\text{Goodput} = \text{RPS} \times \Pr(\text{TPOT} < \text{SLA target})$$

Configuration	Raw RPS	SLA compliance	Goodput
Unified, aggressive batching	100	60%	60
Unified, conservative	70	95%	66.5
Disaggregated	90	95%	85.5

Disaggregation wins on goodput because decode TPOT stays predictable — no prefill interference spikes.

---

6. P/D Pool Ratio

How many GPUs for prefill vs decode?

Workload	Prefill fraction	Decode fraction	Recommended P:D ratio
Short prompts, long outputs (chat)	10% compute	90% compute	1:8
Long prompts, short outputs (summarization)	70% compute	30% compute	2:1
Balanced (general API)	30% compute	70% compute	1:3

The ratio depends on your workload — measure prefill and decode compute fractions from your actual traffic, then allocate GPU pools proportionally.

---

7. When (Not) to Disaggregate

Scenario	Disaggregate?	Why
Single-user low-latency	No	Transfer overhead > interference cost
Mixed workloads, SLA targets	Yes	Decode TPOT stability crucial
Long prompts + concurrent users	Yes	Prefill interference is worst here
Small model (8B) on one GPU	No	No interference (fast prefill)
All prompts same length	Maybe	Less interference variance, less benefit

For the systems that implement disaggregation, see Dynamo and LMCache & Mooncake.

---

Self-Check

• Explain why prefill and decode interfere when mixed on the same GPU
• Calculate KV cache transfer time for a given model and context length
• Explain why RDMA is essential (vs TCP) for disaggregation
• Define goodput and calculate it for given RPS and SLA compliance
• Determine P/D GPU ratio for a given workload mix
• Identify when disaggregation helps vs adds unnecessary complexity

---

References

• DistServe: Disaggregating Prefill and Decoding (Zhong et al., OSDI 2024)
• Splitwise: Efficient Generative LLM Inference (Patel et al., ISCA 2024)