SGLang Architecture

SGLang is a high-performance inference engine that innovates on three fronts: RadixAttention (automatic KV cache reuse via radix tree), constrained decoding (structured output at near-zero overhead), and overlap scheduling (hiding CPU overhead by overlapping scheduling with GPU execution).

---

1. Architecture Overview

SGLang has a two-tier architecture: a frontend for structured generation programs and a backend (SRT — SGLang Runtime) for high-performance serving:

┌─────────────────────────────────────────────────┐
│  Frontend (Python SDK)                          │
│  Structured programs: gen(), select(), fork()   │
│  Prompt templates, multi-call orchestration     │
├─────────────────────────────────────────────────┤
│  TokenizerManager                               │
│  Async tokenization, dynamic batching           │
├─────────────────────────────────────────────────┤
│  Scheduler                                      │
│  Request queues, token budget, prefix matching, │
│  overlap scheduling, constrained decoding FSM   │
├─────────────────────────────────────────────────┤
│  TP Worker(s)                                   │
│  Model forward pass, RadixAttention backend,    │
│  CUDA graphs, sampling                          │
├─────────────────────────────────────────────────┤
│  RadixCache (Radix Tree for KV Cache)           │
│  Automatic prefix caching, LRU eviction,        │
│  tree-based lookup for any shared prefix        │
└─────────────────────────────────────────────────┘

---

2. RadixAttention: Automatic KV Cache Reuse

SGLang’s signature innovation — a radix tree (trie) that indexes all cached KV by their token sequences. Every request automatically benefits from any previously cached prefix, without explicit configuration.

How It Works

After processing several requests, the radix tree holds:

Root
├── [sys_prompt tokens 0..500]
│   ├── [user_A query tokens] → KV cached for full prompt
│   ├── [user_B query tokens] → KV cached
│   └── [few_shot tokens 501..1200]
│       ├── [math_q1 tokens] → KV cached
│       └── [math_q2 tokens] → KV cached
└── [doc_X context tokens]
    └── [summary query] → KV cached

For any new request: Walk the trie matching tokens. The longest match gives the cached KV to reuse. Only the unmatched suffix needs prefill.

Difference from vLLM Prefix Caching

	vLLM Automatic Prefix Caching	SGLang RadixAttention
Data structure	Hash table (block-level)	Radix tree (token-level)
Lookup granularity	Block-aligned (16 tokens)	Exact token-level
Multi-turn reuse	Supported	More efficient (tree naturally extends)
Complex branching	Limited	Natural (tree supports any branching pattern)

---

3. Constrained Decoding

SGLang integrates grammar-based constrained decoding — ensuring outputs conform to a schema (JSON, regex, etc.) — with near-zero overhead:

import sglang as sgl

@sgl.function
def extract_info(s, text):
    s += "Extract name and age from: " + text + "\n"
    s += sgl.gen("result", 
                 regex=r'\{"name": "[^"]+", "age": \d+\}')

How It Works

1. Compile grammar to FSM: The regex/JSON schema is compiled into a finite state machine (FSM) at request creation time
2. Per-token masking: At each decode step, the FSM determines which tokens are valid next → create a logit mask
3. Fused with sampling: The mask is applied to logits before sampling — no separate filtering step

Why it’s fast: The FSM is pre-compiled and the masking operation is a single GPU kernel. SGLang’s xgrammar library makes this essentially free compared to unconstrained generation.

---

4. Overlap Scheduling

SGLang’s scheduler overlaps CPU scheduling work with GPU execution:

Standard (sequential):
  [GPU idle]  → [CPU schedule] → [GPU execute] → [CPU schedule] → [GPU execute]

SGLang overlap:
  [GPU execute batch N] → [GPU execute batch N+1] → [GPU execute batch N+2]
  [CPU schedule N+1]       [CPU schedule N+2]        [CPU schedule N+3]
       ↑ overlapped!            ↑ overlapped!

Scheduling runs in a separate thread while the GPU processes the current batch. By the time the GPU finishes, the next batch is already prepared.

This matters most at small batch sizes where per-step latency is ~5ms — even 0.5ms of scheduling overhead is 10% of the step time.

---

5. Data Parallelism Controller

For multi-GPU serving with data parallelism (multiple model replicas):

DataParallelController
  ├── Scheduler 0 (GPU 0-3, TP=4)  → model replica 0
  ├── Scheduler 1 (GPU 4-7, TP=4)  → model replica 1
  └── Load balancer: route requests to least-loaded replica

Each replica has its own scheduler, RadixCache, and worker group. The controller routes incoming requests based on:

• Cache affinity: Route to the replica most likely to have the prefix cached
• Load balancing: Route to the replica with the most available KV cache blocks

---

6. Disaggregation Support

SGLang supports prefill-decode disaggregation — separate GPU pools for prefill and decode:

Prefill pool (compute-optimized):
  Processes prompts, produces KV cache
  → Transfer KV to decode pool via RDMA/shared memory

Decode pool (memory-optimized):
  Receives KV cache, continues generation
  Optimized for concurrent requests (more KV cache memory)

The disaggregation/ module handles KV transfer, state synchronization, and routing between pools.

---

7. SGLang vs vLLM: When to Choose Each

Dimension	vLLM	SGLang
Maturity	More mature, larger community	Newer, faster iteration
Prefix caching	Hash-based (block-aligned)	RadixAttention (token-level, more flexible)
Constrained output	Limited (outlines integration)	Native, near-zero overhead
Structured programs	Not supported	sgl.function SDK
Overlap scheduling	Not in V1	Built-in
Ecosystem integration	Wider (many frameworks)	Growing
Best for	General-purpose serving	Structured output, multi-call agents

For RadixAttention concepts, see Prefix Caching. For the engine walkthrough that covers scheduler concepts, see Engine Walkthrough.

---

For vLLM’s architecture, see vLLM Architecture. For cluster-level orchestration above both engines, see Dynamo and llm-d.

---

Self-Check

• Draw SGLang’s two-tier architecture (frontend SDK + SRT backend)
• Explain RadixAttention’s trie-based KV lookup vs vLLM’s hash-based approach
• Explain how constrained decoding works (grammar → FSM → logit mask per step)
• Explain overlap scheduling and why it matters for small-batch latency
• Choose between vLLM and SGLang for a given workload

---

References

• SGLang: Efficient Execution of Structured Language Model Programs (Zheng et al., 2023)
• SGLang Documentation
• SGLang Source Code