vLLM Architecture

vLLM is the most widely deployed open-source LLM inference engine. Understanding its architecture — how the concepts from previous chapters (PagedAttention, continuous batching, chunked prefill) are implemented as interacting components — is essential for anyone operating or extending an inference system.

---

1. Architecture Overview

vLLM V1 has a multi-process architecture with clear separation of concerns:

┌─────────────────────────────────────────────────────────┐
│  AsyncLLM (API layer)                                   │
│  OpenAI-compatible HTTP server, request tracking        │
├────────────────────────┬────────────────────────────────┤
│  InputProcessor        │  OutputProcessor / Detokenizer │
│  Tokenize, validate    │  Detokenize, stream to client  │
├────────────────────────┴────────────────────────────────┤
│  EngineCore (separate process, ZMQ communication)       │
│  ┌──────────────────┐  ┌─────────────────────────────┐  │
│  │ Scheduler        │  │ KV Cache Manager            │  │
│  │ Token budget,    │  │ BlockPool, prefix caching,  │  │
│  │ request queues,  │  │ KV cache coordinator        │  │
│  │ preemption       │  │                             │  │
│  └──────────────────┘  └─────────────────────────────┘  │
├─────────────────────────────────────────────────────────┤
│  Worker(s) (one per GPU, may be separate processes)     │
│  ┌─────────────────────────────────────────────────┐    │
│  │ GPUModelRunner                                  │    │
│  │ Forward pass, attention backends, CUDA graphs,  │    │
│  │ sampling, KV cache pool on GPU                  │    │
│  └─────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────┘

Key Design Decisions

• EngineCore runs in a separate process from the API server — CPU-heavy scheduling doesn’t compete with I/O-heavy HTTP handling
• ZMQ for IPC — EngineCore and AsyncLLM communicate via ZeroMQ sockets (faster than Python multiprocessing queues)
• Scheduler is CPU-only — no GPU operations in the scheduling path, keeping it fast
• Worker owns the GPU — all CUDA operations (model forward, KV cache allocation, sampling) happen in the Worker process

---

2. The Main Loop

Every decode iteration:

# Simplified EngineCore.step()
def step(self):
    # 1. Scheduler decides what to run
    scheduler_output = self.scheduler.schedule()
    
    # 2. Send scheduled batch to GPU worker
    model_output = self.model_executor.execute(scheduler_output)
    
    # 3. Process outputs (new tokens, finish reasons)
    engine_outputs = self.process_outputs(model_output, scheduler_output)
    
    # 4. Send outputs to AsyncLLM (via ZMQ) for streaming
    return engine_outputs

This loop runs continuously — there’s no “wait for a batch to fill.” Every iteration, the scheduler assembles the best possible batch from waiting and running requests, and executes it immediately.

---

3. The Scheduler

The V1 scheduler (v1/core/sched/scheduler.py) manages three queues:

Queue	Contains	Transitions
Waiting	New requests not yet started	→ Running (when resources available)
Running	Active requests (prefill or decode)	→ Finished (EOS/max_tokens) or → Waiting (preempted)
Completed	Requests that hit EOS/max_tokens	→ output sent, resources freed

Scheduling Algorithm

def schedule(self):
    budget = self.token_budget  # e.g., 2048
    batch = []
    
    # Priority 1: Running decode requests (1 token each)
    for req in self.running:
        if not self.kv_cache_manager.can_append(req):
            self.preempt(req)  # out of KV blocks → preempt
            continue
        batch.append(ScheduleDecodeOp(req))
        budget -= 1
    
    # Priority 2: Waiting requests get prefill chunks
    for req in self.waiting:
        chunk_size = min(req.remaining_prefill, budget)
        blocks_needed = ceil(chunk_size / self.block_size)
        if not self.kv_cache_manager.can_allocate(blocks_needed):
            break  # no memory → stop admitting
        batch.append(SchedulePrefillOp(req, chunk_size))
        budget -= chunk_size
        if budget <= 0:
            break
    
    return SchedulerOutput(batch)

The scheduler implements all the concepts from earlier chapters:

• Continuous batching: running requests are always included, new requests admitted per-step
• Chunked prefill: chunk_size = min(remaining, budget) — long prefills are broken into pieces
• Token budget: decode priority, prefill fills the remainder
• Preemption: when KV blocks run out, evict LRU requests

---

4. KV Cache Manager & Block Pool

The BlockPool is the physical memory allocator for KV cache:

class BlockPool:
    """Pool of physical KV cache blocks on GPU."""
    def __init__(self, num_blocks, block_size):
        self.free_blocks: list[int]     # indices of available blocks
        self.ref_counts: dict[int, int] # reference counting for sharing
    
    def allocate(self, n) -> list[int]:
        """Allocate n physical blocks from the free pool."""
        return [self.free_blocks.pop() for _ in range(n)]
    
    def free(self, block_ids):
        """Return blocks to free pool (if ref_count reaches 0)."""
        for bid in block_ids:
            self.ref_counts[bid] -= 1
            if self.ref_counts[bid] == 0:
                self.free_blocks.append(bid)

The KV Cache Coordinator sits above the block pool and handles:

• Prefix caching: hash-based lookup of cached KV blocks
• KV cache sharing: reference counting for shared prefix blocks
• Preemption policy: which requests to evict when blocks run out

---

5. GPU Model Runner

The GPUModelRunner (v1/worker/gpu_model_runner.py) executes the actual model forward pass:

class GPUModelRunner:
    def execute(self, scheduler_output: SchedulerOutput):
        # 1. Prepare input tensors
        input_ids, positions, block_tables = self.prepare_inputs(scheduler_output)
        
        # 2. Run forward pass (possibly via CUDA graph)
        if self.can_use_cuda_graph(scheduler_output):
            logits = self.cuda_graph_runner.run(input_ids, positions, block_tables)
        else:
            logits = self.model.forward(input_ids, positions, 
                                         kv_caches=self.kv_cache_pool,
                                         block_tables=block_tables)
        
        # 3. Sample next tokens
        new_tokens = self.sampler.sample(logits, scheduler_output.sampling_params)
        
        return ModelOutput(new_tokens, logits)

Key optimizations:

• CUDA graphs for decode: Fixed-shape decode batches are captured as CUDA graphs — eliminating kernel launch overhead
• Piecewise graph capture: Attention (variable KV length) is excluded from graphs; FFN and other fixed-shape ops are captured
• In-place KV cache updates: New K,V values are written directly into the pre-allocated KV cache pool at the physical block locations specified by the block table

---

6. Tensor Parallelism & Multi-GPU

For models that don’t fit on one GPU, vLLM uses TP across GPUs within a node:

Node with 4 GPUs (TP=4):
  Worker 0: 1/4 of weight matrices, full KV cache for its heads
  Worker 1: 1/4 of weight matrices, full KV cache for its heads
  Worker 2: 1/4 of weight matrices
  Worker 3: 1/4 of weight matrices
  
  After each attention + FFN layer: AllReduce across workers (NCCL)
  Scheduler runs on Worker 0 only → broadcasts schedule to all workers

---

7. V1 vs Legacy Architecture

Aspect	Legacy (V0)	V1
Process model	Single process	Multi-process (EngineCore + AsyncLLM)
IPC	Python queues	ZMQ (faster)
Scheduler	Synchronous	Asynchronous (separate thread)
CUDA graphs	Basic	Piecewise (torch.compile integration)
KV cache	Single-type	Multi-type (supports different KV formats per layer)
Disaggregation	Not supported	Supported (KV connector, external KV stores)

For how the scheduler concepts work in theory, see Continuous Batching and Chunked Prefill. For how the nano-vllm walkthrough relates, see Engine Walkthrough.

---

For SGLang’s alternative approach, see SGLang Architecture. For distributed KV cache infrastructure that extends vLLM, see LMCache & Mooncake.

---

Self-Check

• Draw the vLLM V1 multi-process architecture (AsyncLLM ↔ EngineCore ↔ Worker)
• Trace a request through the engine: HTTP → tokenize → schedule → execute → sample → stream
• Explain the scheduler’s three queues and two priorities (decode first, prefill fills budget)
• Explain BlockPool’s allocate/free with reference counting
• Explain why CUDA graphs are used for decode but not prefill
• Compare V0 vs V1 architecture differences

---

References

• vLLM V1 Architecture
• vLLM Source Code
• Efficient Memory Management for Large Language Model Serving (Kwon et al., 2023)