Foundations
Transformer architecture, hardware landscape, workload characterization, and back-of-envelope estimation.
Prerequisites
Linear Algebra for AI
Tensors, shapes, matrix multiplication, GEMM tiling, and the math that connects model architecture to hardware.
Calculus & Probability
Softmax stability, cross-entropy, backpropagation, gradient problems, and numerical precision.
Programming Essentials
Python internals, C/C++ for extensions, Linux basics, GPU software stack, containers.
PyTorch Fundamentals
Tensors, autograd, training loop, GPU memory, mixed precision, and profiling.
Transformer Architecture
Transformer Architecture Overview
High-level map: components, data flow, architecture evolution, and connections to infra.
Tokenization & Vocabulary
BPE, SentencePiece, tiktoken — how text becomes numbers and why it affects cost.
The Transformer as a Compute Graph
Trace the complete forward pass — tensor shapes, FLOP counts, and parameter counting at every step.
Self-Attention Mechanism
QKV projections, scaled dot-product, multi-head attention — step by step with shapes and PyTorch code.
Feed-Forward Networks & Normalization
SwiGLU, LayerNorm vs RMSNorm, residual connections, Pre-Norm vs Post-Norm.
Positional Encoding
Sinusoidal, RoPE, ALiBi, and long-context extension techniques.
From Transformer to Autoregressive LLM
Generation loop, prefill vs decode, KV cache, MTP, sampling, streaming.
Attention Variants
MHA to GQA to MLA, linear attention, Mamba — how attention evolves for infrastructure constraints.
MoE Architecture
Routing, expert design, load balancing, shared experts, and infrastructure implications.
Workloads & Scaling Laws
Workload Characterization
Prefill vs decode, chat vs agent, reasoning, multimodal, training — identifying the dominant bottleneck.
Scaling Laws
Kaplan, Chinchilla, compute-optimal vs inference-optimal, and the data wall.
Full Lifecycle Cost
Pre-training, fine-tuning, inference cost breakdown — why inference optimization matters most.
Hardware
From Silicon to Chips
Transistors, lithography, process nodes, die size, packaging, HBM stacking — why chip economics shape AI infrastructure.
How Processors Execute
Logic gates to instruction pipelines — how CPUs and GPUs turn transistors into computation.
GPU Architecture
CUDA cores, Tensor Cores, memory hierarchy (registers → shared memory → L2 → HBM), and streaming multiprocessors.
GPU Evolution & Tensor Cores
Tensor Core architecture, precision formats from FP32 to FP4, GPU generation comparison.
GPU Memory & Interconnects
Training memory breakdown, ZeRO, checkpointing, NVLink/InfiniBand topology.
Non-NVIDIA Accelerators Overview
TPU, Trainium, Gaudi, Ascend, CPU inference — the evaluation framework for alternative hardware.
Back-of-Envelope Estimation
Estimate memory, compute time, and bottlenecks before running anything — with worked examples.