Navigation
⚡ Inference Systems· 10 min read

Chunked Prefill

Breaking long prompts into chunks to prevent head-of-line blocking — trading slightly higher TTFT for stable TPOT across all requests.

chunked-prefillschedulingtoken-budgetttfttpothead-of-line-blocking

When a long prompt arrives, its prefill can monopolize the GPU for hundreds of milliseconds — causing all decode requests to stall and their TPOT to spike. Chunked prefill solves this by breaking long prefills into fixed-size chunks interleaved with decode steps.

1. The Head-of-Line Blocking Problem

Without chunked prefill, a 10K-token prompt requires one massive prefill step:

Step t:   [Decode(A), Decode(B), Decode(C)]  — normal, ~5ms per step
Step t+1: [Prefill(D, 10K tokens)]            — 50ms! A, B, C get NO decode
Step t+2: [Decode(A), Decode(B), Decode(C), Decode(D)]  — back to normal

During step t+1, requests A, B, C receive no tokens for 50ms. Their users see a stutter — TPOT spikes from 5ms to 50ms. This is head-of-line blocking: one heavy request stalls everyone.

Why 50ms? Prefill is compute-bound. For 70B on H100 with TP=4: 6×70B×10K/(989T×4×0.5)≈42ms6 \times 70B \times 10K / (989\text{T} \times 4 \times 0.5) \approx 42\text{ms} plus overhead. Larger models or longer prompts make this worse.

2. The Solution: Chunk the Prefill

Break the 10K-token prefill into chunks of chunk_size tokens (e.g., 512):

Step t:   [Decode(A,B,C) + PrefillChunk(D, tokens 0-511)]      — ~6ms
Step t+1: [Decode(A,B,C) + PrefillChunk(D, tokens 512-1023)]   — ~6ms
Step t+2: [Decode(A,B,C) + PrefillChunk(D, tokens 1024-1535)]  — ~6ms
...
Step t+19: [Decode(A,B,C) + PrefillChunk(D, tokens 9728-9999)] — ~5ms
Step t+20: [Decode(A,B,C,D)]  — D fully prefilled, joins decode

Every step includes decode for A, B, C — their TPOT stays at ~6ms consistently. D’s prefill takes 20 steps instead of 1, but the system stays smooth.

The Tradeoff

Without chunkingWith chunking
D’s TTFT~50ms (fast, one shot)~120ms (20 × 6ms, slightly slower)
A/B/C’s TPOT during D’s prefill50ms spike~6ms (stable)
P99 TPOTVery high (spikes)Low (smoothed)

The tradeoff is deliberate: slightly slower TTFT for one request in exchange for stable TPOT for all requests. For streaming applications, this is almost always the right trade.

3. Token Budget: The Unified Scheduler

Token Budget

Chunked prefill needs a scheduling framework to decide how much prefill and decode work fits in each step. The answer is a token budget — a unified “currency” for all work:

∑decode requests⏟1 token each+∑prefill chunks⏟chunk_size tokens each≤token_budget\underbrace{\sum \text{decode requests}}_{\text{1 token each}} + \underbrace{\sum \text{prefill chunks}}_{\text{chunk\_size tokens each}} \leq \text{token\_budget}

Scheduling Priority

  1. Decode first: All active decode requests get their 1-token slot (guaranteed)
  2. Prefill fills remaining budget: Whatever budget remains goes to prefill chunks
  3. Fairness: Multiple pending prefills are served round-robin
# Simplified token budget scheduler
def schedule_step(running_decode, pending_prefill, token_budget):
    batch = []
    budget_remaining = token_budget
    
    # Priority 1: guarantee all decode requests
    for req in running_decode:
        batch.append(DecodeToken(req))
        budget_remaining -= 1
    
    # Priority 2: fill remaining budget with prefill chunks
    for req in pending_prefill:
        chunk_size = min(req.remaining_prefill, budget_remaining)
        if chunk_size > 0:
            batch.append(PrefillChunk(req, chunk_size))
            budget_remaining -= chunk_size
        if budget_remaining <= 0:
            break
    
    return batch

Choosing the Token Budget

BudgetEffect
512Very smooth TPOT, but slow prefill throughput
2048Good balance (vLLM default range)
8192Fast prefill, but decode may see latency bumps
max_model_lenEffectively disables chunking

4. Interaction with Other Techniques

TechniqueInteraction with chunked prefill
Continuous batchingChunked prefill is an extension — scheduling granularity moves from “request” to “token budget”
PagedAttentionKV blocks are allocated incrementally as chunks are processed
Prefix cachingCached prefix blocks skip chunked prefill entirely (already computed)
CUDA graphsEach unique batch shape may need its own captured graph

For how the attention kernel handles mixed prefill+decode batches, see Attention Backends.

Self-Check

  • Explain head-of-line blocking and how it causes TPOT spikes
  • Describe the token budget scheduling algorithm
  • Calculate the TTFT increase from chunking a 10K prompt into 512-token chunks
  • Explain why decode requests get scheduling priority over prefill chunks
  • Choose an appropriate token budget for a latency-sensitive vs throughput-sensitive workload

References