Distributed GPU Systems
Making many GPUs work together — communication primitives, parallelism strategies, and network interconnects.
Collective Communication
Communication Primitivescoming soon
Point-to-point send/recv, AllReduce, AllGather, ReduceScatter, Broadcast, All-to-All.
Ring & Tree Algorithmscoming soon
How collectives are implemented — latency vs bandwidth regimes, theoretical lower bounds.
Communication Cost Analysiscoming soon
Bytes moved as a function of message size and GPU count — when startup cost dominates vs data volume.
Communication Libraries & RDMA
NCCL Internalscoming soon
Channels, rings, trees, protocol selection (Simple, LL, LL128).
GPUDirect RDMAcoming soon
Zero-copy GPU↔NIC paths, GPUDirect Storage for checkpoint I/O.
RDMA Conceptscoming soon
Queue pairs, completion queues, memory registration — InfiniBand vs RoCE v2 tradeoffs.
Custom Collectivescoming soon
MSCCL (programmable collectives), NVIDIA Unified Bus — fusing communication into compute kernels.
Debugging Distributed Hangscoming soon
NCCL environment variables, flight recorder, torch.distributed diagnostics, nccl-tests.
Parallelism Strategies
Data Parallelismcoming soon
Replicate model, split batch — the simplest and most common strategy.
Tensor Parallelismcoming soon
Column/row parallel splits within a layer — intra-operator parallelism.
Sequence Parallelismcoming soon
Splitting LayerNorm and dropout along the sequence dimension.
Context Parallelismcoming soon
Ring Attention, Ulysses — splitting long-sequence attention across GPUs.
Pipeline Parallelismcoming soon
Splitting layers across stages — GPipe, 1F1B, interleaved schedules.
Expert Parallelismcoming soon
MoE expert distribution across GPUs, All-to-All dispatch patterns.
Cross-GPU Memory Poolingcoming soon
Memory borrowing, access boundaries, shared KV cache across GPUs within a supernode.
Interconnects & Network Topology
NVLink & NVSwitchcoming soon
Generations, bandwidth per link, full-bisection topology, NVLink domain in GB200 NVL72.
InfiniBandcoming soon
NDR/XDR, multi-rail topology, why 8 NICs per node, rail-optimized vs full fat-tree.
RoCE & Ethernetcoming soon
DCQCN, Swift, ECN — congestion control for AI workloads.
Data Center Network Designcoming soon
Spine-leaf, multi-tier topologies, oversubscription ratios, mapping parallelism to physical paths.
Reading Cluster Topology Diagramscoming soon
How to read a topology diagram and determine which parallelism strategies it can support.
Parallelism Planning
Decision Processcoming soon
Fit check → TP degree → DP/PP/EP split → communication overhead → memory verification.
Memory Budget Verificationcoming soon
Calculating per-GPU memory requirements for a given parallelism configuration.
Communication Overhead Estimationcoming soon
Predicting communication time for different parallelism strategies on different interconnects.
Worked Examplescoming soon
8B dense on 8 GPUs, 70B on 32, 600B+ MoE on 256 — step-by-step walkthroughs.
Consumer GPU Feasibilitycoming soon
4090 clusters — where interconnect and memory limitations create hard walls.