Cross-GPU Memory Pooling

Traditional parallelism splits computation. Cross-GPU memory pooling splits memory — allowing one GPU to use another GPU’s spare HBM for KV cache, activations, or model weights. This is emerging as a key technique for long-context serving.

For KV cache sizing details, see Back-of-Envelope Estimation. For PagedAttention (the virtual memory system for KV cache), see PagedAttention.

1. The Problem: Uneven Memory Usage

In LLM serving, memory usage varies dramatically per request:

Short request (100 tokens): 50 MB KV cache
Long request (128K tokens): 64 GB KV cache

If a GPU has a mix of short and long requests, some GPUs OOM while others have spare memory.

2. Memory Borrowing via NVLink

Within a node, GPUs can access each other’s HBM through NVLink:

# GPU 0 allocates memory on GPU 1's HBM
with torch.cuda.device(1):
    remote_buffer = torch.empty(size, device='cuda:1')

# GPU 0 reads/writes to GPU 1's memory via NVLink
# Bandwidth: 450-900 GB/s (NVLink), vs 3350 GB/s (local HBM)

Tradeoff: Remote HBM access is ~4-8× slower than local HBM. Acceptable for:

KV cache overflow (accessed infrequently per token)
Cold model weights (not in the hot path)
NOT for activations in the compute hot path

3. Shared KV Cache Across GPUs

Mooncake Architecture (Kimi)

Kimi’s Mooncake separates KV cache from compute:

Prefill GPU: generates KV cache → writes to shared KV store
KV Store: distributed across multiple GPUs' HBM (or CPU DRAM)
Decode GPU: reads KV cache from the store via RDMA

Benefits:

KV cache can grow beyond any single GPU’s memory
Multiple decode GPUs can share the same KV cache (for beam search, parallel generation)
Prefill and decode GPUs don’t need to be the same

When multiple requests share a system prompt:

Request A: [system (2K tokens)] + [user A (100 tokens)]
Request B: [system (2K tokens)] + [user B (200 tokens)]

The system prompt’s KV cache is computed once and stored in the shared pool. Both requests reference the same physical pages — like shared memory pages in an OS.

4. Disaggregated Memory Architectures

Emerging systems separate GPU compute from GPU memory:

Compute GPUs: process tokens (attention, FFN)
Memory GPUs: store KV cache, cold weights
  Connected via NVLink or CXL

CXL (Compute Express Link): A new interconnect standard that allows CPUs and GPUs to share a unified memory pool. Lower bandwidth than NVLink but higher capacity (can pool CPU DRAM + GPU HBM).

Why This Matters

Long-context serving: 1M-token context requires ~500 GB KV cache — far beyond one GPU. Disaggregated memory lets the KV cache span multiple memory devices.
Cost efficiency: Memory-optimized GPUs (like H200 with 141 GB) can serve as KV cache storage while compute-optimized GPUs handle processing.
Elasticity: Add memory independently of compute as context lengths grow.

5. Current State

Approach	Status	Used by
NVLink memory borrowing	Available now	vLLM (limited), custom systems
Shared KV cache store	Production	Kimi/Mooncake, internal systems
CXL memory pooling	Early/experimental	Research systems
CPU DRAM offloading	Available now	vLLM, FlexGen

CPU DRAM offloading is the simplest form: move cold KV cache pages to CPU memory, bring them back when needed. Bandwidth is limited by PCIe (~64 GB/s), but it works for long-context requests where most KV pages are accessed rarely.

Self-Check

Explain why uneven memory usage is a problem in LLM serving
Explain the bandwidth tradeoff of remote vs local HBM access
Explain shared KV cache and prefix sharing across requests
Explain disaggregated memory and why CXL enables it
Explain when CPU DRAM offloading is acceptable (access frequency)