⚙️ AI Mechanics

The Omni Model: How a Tiny LLM Learns to Listen, See, Think, and Speak

A visual deep-dive into MiniMind-O — a 0.1B-parameter Omni model that processes text, audio, and images as input, and generates text and streaming speech simultaneously. Covering the Thinker-Talker architecture, Multi-Token Prediction for audio codebooks, training data anatomy, and the complete evolution from text LLM to multimodal Omni.

From Text to Vision: What Changes When an LLM Learns to See

A visual walkthrough of how a text-only LLM becomes a vision-language model — the vision encoder, projector, token replacement trick, attention patterns on image tokens, and the two-stage training strategy. Using real data from a 160M-parameter VLM.

MoE Inference Visualized — How Expert Routing Actually Works Inside a Real Model

A visual deep-dive into Mixture-of-Experts inference using a real 198M-parameter MoE transformer. See the router decisions, expert assignments, load balancing, and the exact compute tradeoffs — compared side-by-side with a dense model.

Visualizing Every Step of LLM Training — How 64M Parameters Learn to Predict Text

A concrete, visual walkthrough of every computation in LLM training — loss calculation, backpropagation, gradient flow, weight updates, and the difference between pretraining and SFT — using real gradients from a 64M-parameter transformer.

Visualizing Every Step of LLM Inference — Inside a 64M Parameter Model

A concrete, visual walkthrough of every computation that happens when a language model generates text — using real weights, real attention scores, and real probabilities from a 64M-parameter transformer.

Building GPT from Scratch — Part 8: FSDP — When the Model Won't Fit

DDP replicates the full model on every GPU — which fails when the model exceeds one GPU's memory. FSDP (Fully Sharded Data Parallel) shards parameters, gradients, and optimizer states across GPUs. ...

Building GPT from Scratch — Part 10: Disaggregated Inference — Splitting Prefill and Decode

Prefill is compute-bound (process all prompt tokens in parallel). Decode is memory-bound (read KV cache for one new token at a time). Running both on the same GPU forces a compromise — you can't op...

Building GPT from Scratch — Part 7: Distributed Data Parallel Training

One GPU trains sequentially. Multiple GPUs split the batch — each processes a portion, computes gradients locally, then AllReduce averages the gradients across all GPUs before updating. Same conver...

Building GPT from Scratch — Part 9: Tensor Parallelism — Splitting Layers Across GPUs

Data parallelism replicates the model and splits data. Tensor parallelism splits individual layers across GPUs — each GPU computes a portion of every matrix multiplication. The attention layer spli...

Building GPT from Scratch — Part 3: The Transformer Architecture

The GPT architecture is a stack of transformer blocks, each containing causal self-attention (lets each token attend to all previous tokens) and a feed-forward network (per-token nonlinear processi...

Building GPT from Scratch — Part 5: Fine-tuning

Fine-tuning doesn't teach new knowledge — it teaches new behavior. Same model, same loss function, different data format. By training on instruction-response pairs instead of raw text, the model le...

Building GPT from Scratch — Part 6: Inference

Naive generation recomputes all previous tokens every step — O(n²) total. KV cache stores past Keys and Values so each new token only computes one query against the cached context — O(n) total. Thi...

Building GPT from Scratch — Part 1: What Is GPT?

We build a working GPT from scratch — tokenizer, transformer, pretraining, fine-tuning, and inference with KV cache — in ~500 lines of PyTorch on a MacBook. The model is tiny (3.3M params, trains i...

Building GPT from Scratch — Part 4: Pretraining

Pretraining has one objective: predict the next token. That's it. No grammar rules, no world knowledge injected — just next-token prediction via cross-entropy loss. To predict well, the model must ...

Building GPT from Scratch — Part 2: Tokenization

Neural networks need numbers, not text. Byte Pair Encoding (BPE) solves this by iteratively merging the most frequent character pairs into tokens — a compression algorithm repurposed for NLP. It hi...

Build GPT from Scratch in 500 Lines of PyTorch

We build a working GPT from scratch — tokenizer, transformer, pretraining, fine-tuning, and inference with KV cache — in ~500 lines of PyTorch on a MacBook. The model is tiny (3.3M params, trains i...

Why Next-Token Prediction Produces Intelligence

LLMs have one objective: predict the next token. Critics call this 'glorified autocomplete.' They're right about the objective function — and completely wrong about what happens inside. To predict ...