Diffusion Models — infra.bea.ai

Diffusion model inference is fundamentally different from LLM inference — instead of autoregressive token generation, it performs iterative denoising over a fixed number of steps. This changes every aspect of serving: batching, memory, latency, and optimization.

1. How Diffusion Inference Works

Input: text prompt + noise tensor (random seed)

Step 1: Denoise noisy latent (U-Net or DiT forward pass)
Step 2: Denoise again (slightly less noisy)
...
Step N: Final denoised latent (20-50 steps typical)

Post-process: VAE decoder (latent → pixel image)

Output: image (512×512, 1024×1024, etc.)

Each denoising step runs the same model (U-Net or DiT) on the progressively cleaner latent — the compute cost is N_steps × forward_pass_time.

2. Key Differences from LLM Inference

Dimension	LLM	Diffusion
Generation pattern	Autoregressive (1 token/step)	Iterative (full image refined each step)
Steps per request	100-10K (output tokens)	20-50 (denoising steps)
Memory growth	KV cache grows with context	Fixed (latent tensor constant size)
Batching benefit	Share weight reads across requests	Share weight reads across requests AND steps
Bottleneck	Memory bandwidth (decode)	Compute (each step is a full forward)
Streaming	Token-by-token streaming	No streaming (image appears at end)

Diffusion is compute-bound, not memory-bound — the opposite of LLM decode. This means quantization and batching have different effects.

3. Classifier-Free Guidance (CFG)

Most diffusion models use CFG — running two forward passes per step (one with text conditioning, one without), then combining:

\text{output} = \text{uncond} + \text{guidance\_scale} \times (\text{cond} - \text{uncond})

CFG doubles compute cost. A 50-step generation at CFG=7.5 requires 100 forward passes.

Optimizations:

Batched CFG: Process conditional and unconditional in the same batch (2× batch, 1× launch overhead)
Distilled models (SDXL Turbo, FLUX Schnell): Reduce steps to 1-4 with distillation, eliminating CFG entirely
CFG rescaling: Apply guidance only on certain steps

4. Batching Strategies

Across Requests

Multiple image requests batched together — each at potentially different denoising steps:

Batch of 4 requests, all at step 15/50:
  [latent_A, latent_B, latent_C, latent_D] → U-Net → [denoised_A, B, C, D]

Standard batching works well because all requests use the same model with the same-size tensors.

Across Steps (Pipelining)

Different requests at different stages — while request A is at step 30, request B enters at step 1:

Step t:   UNet(A_step30, B_step1, C_step15)
Step t+1: UNet(A_step31, B_step2, C_step16)

This works because each step is independent — no “KV cache” equivalent to manage.

5. Latent Caching

For applications where users iterate on a prompt (same seed, slightly different text):

Cache intermediate latents at step $k$
When user modifies prompt, restart from step $k$ instead of step 0
Saves $k/N$ of the compute

6. Practical Serving

# Using ComfyUI / Diffusers for serving
from diffusers import StableDiffusionXLPipeline
import torch

pipe = StableDiffusionXLPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16,
    variant="fp16",
)
pipe = pipe.to("cuda")
pipe.enable_model_cpu_offload()  # offload when not in use

# For high-throughput: batch requests
images = pipe(
    prompt=["cat", "dog", "bird", "fish"],
    num_inference_steps=30,
    guidance_scale=7.5,
).images

Optimization	Effect	When to use
FP16	2× faster, minimal quality loss	Always
torch.compile	10-30% speedup	Production
xFormers attention	Memory efficient	Large resolutions
Distilled models (Turbo/Schnell)	10-50× fewer steps	When quality allows
TensorRT	Maximum throughput	Fixed resolution serving

For quantization of diffusion models (INT8 weights, FP8 attention), see Quantization Fundamentals.

Self-Check

Explain the denoising loop and how it differs from autoregressive generation
Calculate total forward passes with CFG (steps × 2)
Compare batching strategies: across requests vs across steps
Explain latent caching for iterative prompt refinement
Choose optimization strategy for a given throughput/quality target