CUDA & Kernel Engineering
From writing your first CUDA kernel to implementing FlashAttention — making one GPU go fast.
CUDA Programming
CUDA Programming Modelcoming soon
Threads, warps, blocks, grids, and streaming multiprocessors — how code maps to hardware.
CUDA Memory Modelcoming soon
Registers, shared memory, L1/L2 cache, global memory (HBM) — capacity, bandwidth, and access rules.
Warp Execution & Schedulingcoming soon
Warp divergence, instruction-level parallelism, latency hiding through occupancy.
CUDA Streams, Events & Graphscoming soon
Asynchronous execution, reducing launch overhead, persistent kernels.
Environment Setupcoming soon
Driver/toolkit installation, version compatibility, and verification.
Memory & Performance Optimization
Memory Coalescingcoming soon
How thread access patterns map to memory transactions — the #1 optimization.
Shared Memory & Bank Conflictscoming soon
Using shared memory effectively and avoiding bank conflicts.
Occupancy & Resource Allocationcoming soon
The balance between registers, shared memory, and active warps per SM.
Tiling Strategiescoming soon
Breaking large problems into shared-memory-sized chunks to maximize data reuse.
Synchronization & Atomic Operationscoming soon
Thread synchronization primitives and when to use atomics.
Core Kernel Implementation
Parallel Reductioncoming soon
Warp-level primitives, avoiding bank conflicts — the foundation of many kernels.
GEMM Optimizationcoming soon
Naive → tiled → double-buffered → Tensor Core aware — understanding the connection to cuBLAS.
Softmax Implementationcoming soon
Naive → online/stable → fused with scaling — numerical stability meets performance.
Operator Fusioncoming soon
Eliminating intermediate tensors by chaining operations — principles, limits, and compilation complexity.
Attention Kernels
Naive Attentioncoming soon
Full N×N materialization — why it is memory-bound and where the bottleneck is.
FlashAttention V1coming soon
Tiled computation with online softmax — no materialization of the attention matrix.
FlashAttention V2coming soon
Improved parallelism and warp partitioning for higher GPU utilization.
FlashInfercoming soon
Page-level attention for heterogeneous request handling in serving engines.
Block-Sparse & Linear Attention Kernelscoming soon
Attention patterns beyond dense: sparse masks, linear attention, and their kernel implementations.
Compilers & Profiling
AI Compilerscoming soon
Triton, TVM, XLA, torch.compile/Inductor — automated tiling, fusion, and scheduling.
Nsight Systemscoming soon
Timeline profiling — finding launch overhead, memory copy stalls, and idle gaps.
Nsight Computecoming soon
Kernel-level analysis — roofline charts, memory throughput, compute utilization.
PyTorch Profilercoming soon
Trace collection, operator-level timing, and bottleneck identification.