Workload Characterization

The single most important skill in AI infrastructure is asking: “what kind of workload is this?” — because the answer determines whether you’re bottlenecked by compute, memory capacity, memory bandwidth, or network. Different workloads stress completely different parts of the hardware stack.

1. The Four Bottlenecks

Every AI workload hits one of these walls first:

Bottleneck	What it means	Limited by	Example
Compute-bound	GPU math units are the limit	TFLOPS	Large-batch prefill, training forward pass
Memory-BW-bound	Data can’t be fed fast enough	HBM GB/s	Single-token decode, LayerNorm, activation functions
Memory-capacity-bound	Not enough memory	HBM GB	Large model weights, long-context KV cache
Network-bound	Communication between GPUs is the limit	NVLink/IB GB/s	Tensor parallelism AllReduce, expert parallelism All-to-All

Most infrastructure mistakes come from optimizing the wrong bottleneck.

2. Inference Workloads

Prefill vs Decode

The two phases of every LLM request have opposite resource profiles:

Property	Prefill	Decode
What happens	Process entire prompt in parallel	Generate one token at a time
GEMM M dimension	$M = B \times S$ (all prompt tokens)	$M = B$ (one new token per request)
Bottleneck	Compute-bound	Memory-BW-bound
GPU utilization	High (80%+ at sufficient batch)	Low (<30% for single request)
Arithmetic intensity	High (large GEMMs)	~1 (thin GEMMs)
Key metric	TTFT (Time to First Token)	TPOT (Time Per Output Token)

Infrastructure consequence: Prefill and decode benefit from different hardware configurations. This is why prefill-decode disaggregation uses separate GPU pools.

Short Context vs Long Context

Property	Short (< 4K tokens)	Long (32K–128K+)
Attention compute	Small ( $S^2$ is manageable)	Dominates ( $S^2$ explodes)
KV cache	Fits easily	Can consume most of GPU memory
Bottleneck	FFN compute or memory-BW	Attention compute + KV cache capacity
Typical use	Chat, Q&A	Document analysis, code repos

At 128K context with LLaMA-2-7B, KV cache alone is ~64 GB — nearly all of an H100’s memory.

Chat vs Agent

Property	Chat	Agent
Turn structure	Human → Model → Human → …	Model → Tool → Model → Tool → …
Latency tolerance	Interactive (< 2s TTFT)	Can be async (minutes OK)
Context management	Conversation history	+ tool outputs, file contents, execution results
State	Stateless (per request)	Stateful (sandbox, files, processes persist)
Compute pattern	Single model call per turn	Multiple calls, branching, retries
Infrastructure	Standard serving	Sandbox orchestration, state management

Agent workloads are fundamentally different — they require stateful compute environments (sandboxes), tool execution, and multi-step orchestration. This is covered in the Apps & Agents module.

Reasoning Models (Extended Thinking)

Models like o1, DeepSeek-R1, and QwQ generate long “thinking” traces before answering:

Property	Standard generation	Reasoning
Output length	100–1000 tokens	1,000–50,000+ tokens
Decode time	Seconds	Minutes
KV cache	Moderate	Very large (thinking + answer)
Compute pattern	Mostly one decode sequence	May involve tree search, verification
Cost per request	$0.001–0.01	$0.01–1.00+

Infrastructure consequence: Reasoning models need much longer decode, much more KV cache, and potentially speculative verification (multi-token prediction). The cost-per-request is 10–100× higher than standard chat.

Multimodal Workloads

Modality	Input processing	Impact on compute	Impact on memory
Vision (images)	Vision encoder → patch embeddings → projector → LLM	Prefill gets a burst of image tokens (256–2048 per image)	KV cache inflated by image tokens
Audio (speech)	Whisper-style encoder → audio embeddings → LLM	Must process faster than audio plays (real-time factor < 1)	Streaming chunks, low latency
Image generation	LLM → diffusion model (iterative denoising)	Separate GPU pool for diffusion	VAE + denoising latents
Video	Many frames → vision encoder → very long context	Extremely long prefill	Massive KV cache

Multimodal serving requires heterogeneous infrastructure — different model components on different hardware, with orchestration between them.

3. Training Workloads

Pre-training

The most resource-intensive phase:

Property	Value
Data	Terabytes of tokenized text
Compute	10K+ GPUs for weeks/months
Pattern	Forward → backward → gradient sync → optimizer step, repeat
Bottleneck	Compute-bound (large batch GEMMs) + network-bound (gradient sync)
Key metric	MFU (Model FLOPS Utilization) — typically 30–55%
Cost	$10M–$ 100M+ for frontier models

Supervised Fine-Tuning (SFT)

Property	Value
Data	Thousands to millions of (instruction, response) pairs
Compute	8–64 GPUs for hours/days
Pattern	Same as pre-training but smaller scale
Bottleneck	Often memory-capacity-bound (fitting model + optimizer)
Key optimization	LoRA/QLoRA to reduce trainable parameters

Reinforcement Learning (RLHF/GRPO)

Property	Value
Pattern	Generate rollouts (inference) → score (inference) → update (training)
Bottleneck	Interleaving inference and training — different resource profiles
Complexity	Must manage separate GPU pools for generation vs training
Key challenge	Dynamic resource allocation between rollout and training phases

RL training is uniquely hard for infrastructure — it combines inference workloads (rollout generation, reward model scoring) with training workloads (policy gradient updates) in the same training loop, requiring the system to dynamically shift resources.

Embedding & Retrieval Workloads

Often overlooked but extremely common in production:

Property	Value
Task	Encode text/images into dense vectors for search/RAG
Pattern	Prefill-only (no generation), high batch size
Bottleneck	Compute-bound at high batch sizes (bulk encoding)
Key optimization	Large batches, variable-length packing, FP16/INT8
Typical throughput	10K–100K documents/second on one H100

Embedding is the “silent majority” of AI inference — every RAG system embeds queries and documents, often at millions of documents per day.

Batch vs Real-Time Inference

Property	Real-time	Batch
Latency target	< 2s TTFT, < 50ms TPOT	Don’t care (hours OK)
Optimization focus	Latency → smaller batches, speculative decoding	Throughput → max batch size, max utilization
SLO	P99 latency	Tokens/second/dollar
Typical use	Chat, API, interactive	Document processing, evaluation, data labeling

Infrastructure consequence: batch inference can use spot instances (2-3× cheaper) because latency doesn’t matter and restarts are acceptable.

Arithmetic Intensity: A Worked Example

To prove prefill is compute-bound and decode is memory-BW-bound, calculate arithmetic intensity for a 7B model on H100:

Prefill (S=2048, B=1): The dominant operation is FFN GEMM: $(2048, 4096) \times (4096, 11008)$ .

FLOPs: $2 \times 2048 \times 4096 \times 11008 \approx 1.8 \times 10^{11}$
Bytes read: $(2048 \times 4096 + 4096 \times 11008) \times 2 \approx 1.1 \times 10^8$
AI \approx 1636 — well above H100’s ops:byte of 295 → compute-bound ✓

Decode (S=1, B=1): Same weight matrix, but M=1: $(1, 4096) \times (4096, 11008)$ .

FLOPs: $2 \times 1 \times 4096 \times 11008 \approx 9.0 \times 10^7$
Bytes read: $(1 \times 4096 + 4096 \times 11008) \times 2 \approx 9.0 \times 10^7$
AI \approx 1 — far below 295 → memory-BW-bound ✓

The weight matrix is the same size in both cases — but prefill reuses it across 2048 tokens while decode uses it for just 1.

4. Workload → Infrastructure Mapping

Workload Bottleneck Map

Workload	Primary bottleneck	Key infra decision
Prefill (short)	Compute	Pack requests, maximize batch utilization
Prefill (long)	Compute + memory	Chunked prefill, sequence parallelism
Decode	Memory bandwidth	Batch many requests to amortize weight reads
Agent	Latency + state	Sandbox infra, async execution
Reasoning	KV cache + decode time	Large memory GPUs, long-request scheduling
Vision-language	Heterogeneous compute	Separate encoder/LLM pools
Pre-training	Compute + network	3D/4D parallelism, fault tolerance
SFT	Memory capacity	LoRA, ZeRO, gradient checkpointing
RLHF	Mixed (inference + training)	Dynamic resource allocation
Embedding/retrieval	Compute	Large batches, packing, quantization
Batch inference	Throughput	Spot instances, max batch size

The correct infrastructure choice always starts with: what is the workload?

Self-Check

Name the four hardware bottlenecks and give an example workload for each
Explain why prefill is compute-bound but decode is memory-BW-bound
Explain how KV cache size changes with context length and why long context is hard
Explain how agent workloads differ from chat workloads in infrastructure requirements
Explain why reasoning models are 10–100× more expensive per request
Explain why RLHF is uniquely challenging for infrastructure (mixed inference + training)