Inference Systems
The core of modern AI infrastructure — from KV cache management to edge deployment.
Inference Fundamentals
Prefill & Decode Phasescoming soon
Compute-bound prefill vs memory-bandwidth-bound decode — the two-phase lifecycle of every request.
KV Cachecoming soon
What it stores, how it grows with sequence length, and why it dominates memory.
Model Formatscoming soon
SafeTensors, GGUF/GGML, ONNX, TensorRT engines — when each is appropriate.
Latency Metricscoming soon
TTFT, TPOT, end-to-end latency, throughput (tokens/second) — defining what to measure.
Inference Engine Internals
PagedAttention
Virtual memory for KV cache — non-contiguous allocation, eliminating fragmentation.
📄 PagedAttention / vLLM (Kwon et al., 2023)
Continuous Batchingcoming soon
Dynamically adding/removing requests mid-generation instead of waiting for a batch to finish.
Chunked Prefillcoming soon
Breaking long prompts into chunks to prevent head-of-line blocking on decode requests.
Prefix Cachingcoming soon
RadixAttention tree-based lookup — reusing KV cache across requests with shared prompts.
Attention Backends & Graph Optimizationcoming soon
How engines select attention kernels and optimize the compute graph.
Engine Walkthrough: nano-vllmcoming soon
Tracing a request through scheduler → model runner → attention backend → sampler.
Quantization for Serving
Quantization Fundamentalscoming soon
Precision formats, calibration methods, and the quality-speed tradeoff.
Weight-Only Quantizationcoming soon
INT4/INT8 with GPTQ, AWQ, GGML — smaller model, faster memory reads.
Weight-Activation Quantizationcoming soon
W8A8 (SmoothQuant), FP8 (native on Hopper+) — quantizing both for faster Tensor Core math.
KV Cache Quantizationcoming soon
FP8/INT8 KV cache — doubling concurrent sequences at minimal quality cost.
Emerging Formatscoming soon
NVFP4, MXFP4 (Blackwell), microscaling — what is next for quantized inference.
Quantization Selection Guidecoming soon
Decision tree: hardware generation × quality requirement → recommended format.
Speculative Decoding
Core Idea: Draft & Verifycoming soon
Small model drafts K tokens, target model verifies all K in one forward pass.
Draft Strategiescoming soon
Separate draft model, Medusa heads, EAGLE feature prediction, prompt lookup, n-gram matching.
Acceptance Rate & Speedup Boundscoming soon
Why 2-3× is typical and when speculative decoding does not help.
Practical Integrationcoming soon
vLLM/SGLang configuration, benchmarking speculative decoding, tuning draft quality.
Distributed Inference
Tensor-Parallel Servingcoming soon
Splitting a model across 2/4/8 GPUs within a node for latency reduction.
Prefill-Decode Disaggregationcoming soon
Separate GPU pools for prefill vs decode, KV cache transfer between them.
Expert Parallelism for MoE Inferencecoming soon
Expert placement, replication strategies, load balancing for skewed routing.
Attention-FFN Separationcoming soon
Placing memory-bound attention and compute-bound FFN on different hardware.
Distributed KV Cachecoming soon
Cross-instance prefix sharing, cache-affinity routing, distributed KV stores.
Serving Beyond LLMs
Diffusion Modelscoming soon
Iterative denoising, classifier-free guidance, batching strategies, latent caching.
Speech Modelscoming soon
ASR chunked streaming (Whisper), TTS real-time factor, voice cloning pipelines.
Vision Modelscoming soon
Object detection, segmentation, dynamic input resolution, high-throughput batch inference.
Embedding Modelscoming soon
Bulk embedding for search/RAG, throughput optimization, variable-length packing.
Multimodal Pipelinescoming soon
Vision-language models, cross-modal scheduling, vision token impact on KV cache.
Inference Economics
The Inference Cost Stackcoming soon
GPU amortization, memory bandwidth pricing, energy costs — where money actually goes.
Custom Siliconcoming soon
Groq LPU, Cerebras WSE, AWS Trainium, Google TPU — inference-specific chip tradeoffs.
Multi-Model Servingcoming soon
Router architectures, LoRA multiplexing, model co-location on shared GPUs.
Benchmarking & Performance Analysis
Metrics Definitioncoming soon
TTFT, TPOT, ITL, throughput, goodput — precise definitions and measurement.
Benchmarking Methodologycoming soon
Open-loop vs closed-loop, realistic distributions (ShareGPT), warmup, percentile reporting.
Roofline Analysis for Inferencecoming soon
Is this configuration compute-bound or bandwidth-bound, and how close to theoretical limit?
Benchmarking Toolscoming soon
vLLM benchmarking scripts, GenAI-Perf, custom harnesses.
Common Pitfallscoming soon
Zero-load testing, fixed lengths, ignoring cold-start, unfair comparisons.
Edge & On-Device Inference
On-Device Constraintscoming soon
Limited memory (8-16GB shared), thermal throttling, battery life — the edge environment.
Model Compression for Edgecoming soon
Aggressive quantization (INT4, 2-bit), distillation, structured pruning for mobile.
Edge Inference Enginescoming soon
llama.cpp/GGML, MLC-LLM, Core ML, QNN, TFLite — cross-platform compilation.
Edge-Cloud Collaborationcoming soon
Split inference, session migration, WAN optimization, wireless bandwidth adaptation.