Multimodal Pipelines
Vision-language models, cross-modal scheduling, vision token impact on KV cache and prefill cost — serving models that see and speak.
Multimodal models (VLMs like GPT-4V, Llama-3.2-Vision, Qwen-VL) process both images and text — creating unique serving challenges where vision and language workloads intersect.
1. VLM Architecture for Serving
Image → Vision Encoder (ViT/SigLIP) → Image tokens (hundreds to thousands)
↓
Text prompt → Tokenizer → Text tokens → [image_tokens + text_tokens]
↓
LLM (standard autoregressive) → Output textThe vision encoder converts an image into “image tokens” — vectors in the same space as text tokens. These image tokens are then concatenated with text tokens and processed by the LLM.
Token Counts
| Model | Image resolution | Image tokens | Equivalent text |
|---|---|---|---|
| Llama-3.2-Vision | 560×560 | ~1,600 | ~1,600 words of context |
| Qwen-VL-2 | 448×448 | ~256 | ~256 words |
| GPT-4V | Variable | 85-1,700 | Varies with resolution |
| InternVL-2 | Dynamic | 256-2,048 | Up to 2K words |
Key impact: Image tokens consume KV cache just like text tokens. A single image can use as much KV as a 2K-word prompt.
2. Serving Challenges
KV Cache Explosion
A multimodal request with a 1024×1024 image might produce 1,600 image tokens + 500 text tokens = 2,100 tokens of context — even for a simple question like “What’s in this image?”
For Llama-3.1 70B: 2,100 tokens × 320 KB/token = 656 MB of KV cache just from the image.
Multiple images: A document understanding task with 5 pages (5 images) = 8,000+ image tokens → 2.5+ GB of KV cache per request.
Prefill Cost
Image tokens are processed during prefill. 1,600 image tokens is equivalent to a 1,600-token text prompt — compute-heavy prefill even for short questions.
| Request type | Total context tokens | Prefill time (70B, TP=4) |
|---|---|---|
| Text-only (short prompt) | ~100 | ~2 ms |
| 1 image + question | ~2,100 | ~40 ms |
| 5 images + question | ~8,500 | ~160 ms |
Vision Encoder Overhead
The vision encoder (ViT) runs separately before the LLM:
| Encoder | Parameters | Time per image (H100) |
|---|---|---|
| SigLIP-400M | 400M | ~5 ms |
| InternViT-6B | 6B | ~50 ms |
This adds to TTFT but is a one-time cost per image (image tokens can be cached if the same image appears in multiple requests).
3. Cross-Modal Scheduling
The vision encoder and LLM have different compute profiles:
Vision encoder: compute-bound (ViT forward pass on image)
LLM prefill: compute-bound (process image + text tokens)
LLM decode: memory-bound (standard autoregressive)Scheduling Strategy
1. Receive request (image + text)
2. Vision encoder processes image → image tokens (GPU)
While: LLM serves decode for existing requests
3. Concatenate [image_tokens + text_tokens]
4. LLM prefill (chunked — image tokens can be large)
5. LLM decode (standard)Key optimization: Pipeline the vision encoder with LLM decode — they use different resources (encoder is compute-heavy, decode is memory-heavy).
4. Optimization Techniques
| Technique | Savings | How |
|---|---|---|
| Image token compression | 2-4× fewer image tokens | Reduce vision encoder output resolution |
| Dynamic resolution | Adjust tokens by image complexity | Simple images get fewer tokens |
| Image embedding caching | Skip vision encoder on repeated images | Cache vision encoder output |
| Quantized vision encoder | 2× faster encoding | INT8/FP8 ViT (minimal quality loss) |
| Separate encoder pool | Better GPU utilization | Dedicated GPUs for vision encoding |
vLLM Multimodal Configuration
from vllm import LLM
llm = LLM(
model="meta-llama/Llama-3.2-11B-Vision-Instruct",
max_model_len=8192,
max_num_seqs=16, # fewer concurrent requests (each uses more KV)
limit_mm_per_prompt={"image": 4}, # max images per request
)5. The Multimodal Serving Stack
Image preprocessing (resize, normalize) → CPU
Vision encoder (ViT) → GPU (can be separate pool)
Token concatenation → CPU
LLM serving (prefill + decode) → GPU (standard vLLM/SGLang)The multimodal pipeline adds a vision preprocessing stage before the standard LLM serving pipeline. The key decision: co-locate vision encoder and LLM on the same GPU (simpler) or separate (better utilization).
For the LLM serving pipeline that processes the combined tokens, see Prefill & Decode Phases.
Self-Check
- Calculate image token count’s impact on KV cache memory
- Compare prefill time for text-only vs multimodal requests
- Explain cross-modal scheduling (pipeline vision encoder with LLM decode)
- List optimization techniques for reducing multimodal serving overhead
- Configure vLLM for multimodal serving with image limits