Navigation
⚡ Inference Systems· 12 min read

LMCache & Mooncake

Tiered KV storage (GPU→CPU→disk→remote), transfer engine, cross-instance KV sharing — the infrastructure layer for KV cache at cluster scale.

lmcachemooncakekv-cachetiered-storagetransfer-enginedistributeddisaggregation

As inference scales beyond a single GPU, KV cache management becomes a distributed systems problem. LMCache and Mooncake provide the infrastructure layer — tiered storage, high-speed transfer, and cross-instance sharing — that makes disaggregated serving practical.

1. The Problem: KV Cache Doesn’t Fit on One GPU

At scale, three scenarios demand KV cache infrastructure beyond PagedAttention:

ScenarioProblemWhat’s needed
Long context (128K+)KV cache exceeds GPU memory for many concurrent requestsOffload cold KV to CPU/disk
PD disaggregationPrefill and decode on different GPUsTransfer KV between machines at high speed
Multi-instance servingMultiple vLLM/SGLang instances serving same modelShare cached prefixes across instances (avoid redundant prefill)

2. LMCache: KV Cache Management Layer

KV Cache Infrastructure

LMCache is a middleware that sits between the inference engine (vLLM/SGLang) and the storage hierarchy:

┌─────────────────────────────────────────────┐
│  Inference Engine (vLLM / SGLang)           │
│  Generates and consumes KV cache            │
├─────────────────────────────────────────────┤
│  LMCache Integration Layer                  │
│  KV event tracking, cache-aware scheduling  │
├──────────┬──────────┬───────────┬───────────┤
│  GPU     │  CPU     │  Local    │  Remote   │
│  HBM     │  DRAM    │  SSD      │  Network  │
│  (hot)   │  (warm)  │  (cold)   │  (shared) │
│  3350    │  64      │  7        │  25-100   │
│  GB/s    │  GB/s    │  GB/s     │  GB/s     │
└──────────┴──────────┴───────────┴───────────┘

Tiered Storage

LMCache manages KV cache across multiple storage tiers:

  1. GPU HBM (hot): Active KV for currently generating requests — fastest, most expensive
  2. CPU DRAM (warm): Recently evicted KV — fast to restore (~1ms for a block via PCIe)
  3. Local SSD (cold): KV from older requests — slower but much larger capacity
  4. Remote store (shared): KV accessible by other instances — enables prefix sharing across the cluster

Cache-Aware Scheduling

LMCache integrates with the inference engine’s scheduler to make KV-aware decisions:

# When a new request arrives:
# 1. Check if prefix KV exists in local GPU cache → skip prefill (fastest)
# 2. Check CPU DRAM → restore to GPU (fast)  
# 3. Check local SSD → load to GPU (slower)
# 4. Check remote store → fetch via network (slowest, but avoids full prefill)
# 5. No cache hit → full prefill (most expensive)

Integration with vLLM

# vLLM with LMCache
from vllm import LLM

llm = LLM(
    model="meta-llama/Llama-3.1-70B",
    kv_connector="lmcache",
    kv_connector_config={
        "cpu_memory_gb": 32,        # 32 GB CPU for warm KV
        "disk_path": "/local_ssd/kv_cache",
        "remote_url": "redis://kv-store:6379",
    }
)

3. Mooncake: KV Cache-Centric Disaggregated Architecture

Mooncake takes a different approach — instead of adding caching to an existing engine, it builds the entire serving architecture around KV cache management.

Core Components

┌────────────────────────────────────────────────────┐
│  Conductor (orchestrator)                          │
│  Request routing, KV placement decisions,          │
│  prefill/decode assignment                         │
├─────────────┬──────────────────────────────────────┤
│  Transfer   │  Mooncake Store                      │
│  Engine     │  Distributed KV cache pool           │
│  RDMA/TCP   │  across all nodes                    │
│  direct     │  (CXL, RDMA, or TCP transport)       │
│  GPU-GPU    │                                      │
├─────────────┴──────────────────────────────────────┤
│  Prefill Workers    │    Decode Workers             │
│  (compute-heavy)    │    (memory-heavy)             │
│  Generate KV cache  │    Consume KV, generate tokens│
└─────────────────────┴──────────────────────────────┘

Transfer Engine

The heart of Mooncake — a high-performance data transfer layer for moving KV cache between GPUs:

TransportBandwidthLatencyUse case
GPU-GPU (NVLink)900 GB/s~5 μsSame node
RDMA (InfiniBand)50 GB/s per NIC~2 μsCross-node
TCP10-25 GB/s~50 μsFallback

Key optimization: Zero-copy transfer — KV blocks are sent directly from one GPU’s memory to another GPU’s memory via GPUDirect RDMA, bypassing CPU entirely.

KV Cache Transfer for PD Disaggregation

1. Request arrives at Conductor
2. Conductor assigns Prefill Worker (GPU with most compute capacity)
3. Prefill Worker processes prompt → KV cache produced in GPU memory
4. Transfer Engine moves KV cache to Decode Worker via RDMA
   (e.g., 1.25 GB for 70B 4K context → ~25ms at 50 GB/s)
5. Decode Worker starts generating tokens using transferred KV

The transfer latency (25ms for 4K context) is comparable to the prefill time itself — so PD disaggregation roughly doubles throughput by letting prefill and decode run in parallel on separate GPUs.

4. LMCache vs Mooncake

LMCacheMooncake
ApproachMiddleware layer added to existing enginesComplete disaggregated architecture
IntegrationPlugin for vLLM/SGLangOwn orchestrator (Conductor)
StorageTiered (GPU → CPU → disk → remote)Distributed KV store (Mooncake Store)
TransportStandard networkingCustom Transfer Engine (RDMA-optimized)
Best forAdding caching to existing deploymentGreenfield disaggregated deployment
MaturityProduction-ready with vLLMProduction at Moonshot AI, open-source

Complementary Use

In practice, these tools complement each other:

  • LMCache for tiered storage within a node (hot/warm/cold KV)
  • Mooncake for high-speed KV transfer between nodes in a disaggregated setup
  • Both can be used with vLLM/SGLang as the underlying engine

5. Practical Impact

MetricWithout KV infraWith LMCache/Mooncake
Prefix cache hit rate (multi-instance)0% (each instance independent)60-90% (shared across instances)
Max concurrent requests (128K context)1 per 80GB GPU5-10× more (offload cold KV)
PD disaggregation feasible?No (can’t transfer KV fast enough)Yes (RDMA transfer ~25ms for 4K)
System throughput at scaleLimited by single-instance memory2-3× improvement from sharing + disaggregation

For the conceptual foundation of prefix sharing, see Prefix Caching. For PD disaggregation concepts, see Distributed KV Cache (covered in the Distributed Inference section).

Self-Check

  • Explain the three scenarios where KV cache infrastructure is needed beyond PagedAttention
  • Describe LMCache’s tiered storage hierarchy and cache-aware scheduling
  • Describe Mooncake’s Conductor + Transfer Engine architecture
  • Calculate KV transfer time for PD disaggregation (70B, 4K context, RDMA)
  • Compare LMCache vs Mooncake and when to use each

References