Blog
Deep dives, paper reviews, and opinions on AI infrastructure — personal voice, anchored by real systems.
The Omni Model: How a Tiny LLM Learns to Listen, See, Think, and Speak
A visual deep-dive into MiniMind-O — a 0.1B-parameter Omni model that processes text, audio, and images as input, and generates text and streaming speech simultaneously. Covering the Thinker-Talker architecture, Multi-Token Prediction for audio codebooks, training data anatomy, and the complete evolution from text LLM to multimodal Omni.
From Text to Vision: What Changes When an LLM Learns to See
A visual walkthrough of how a text-only LLM becomes a vision-language model — the vision encoder, projector, token replacement trick, attention patterns on image tokens, and the two-stage training strategy. Using real data from a 160M-parameter VLM.
MoE Inference Visualized — How Expert Routing Actually Works Inside a Real Model
A visual deep-dive into Mixture-of-Experts inference using a real 198M-parameter MoE transformer. See the router decisions, expert assignments, load balancing, and the exact compute tradeoffs — compared side-by-side with a dense model.
Visualizing Every Step of LLM Training — How 64M Parameters Learn to Predict Text
A concrete, visual walkthrough of every computation in LLM training — loss calculation, backpropagation, gradient flow, weight updates, and the difference between pretraining and SFT — using real gradients from a 64M-parameter transformer.
Visualizing Every Step of LLM Inference — Inside a 64M Parameter Model
A concrete, visual walkthrough of every computation that happens when a language model generates text — using real weights, real attention scores, and real probabilities from a 64M-parameter transformer.
Agent Sandboxes: From E2B to AWS Lambda MicroVMs
AI agents need stateful, isolated compute that suspends when idle and resumes on demand — something between Lambda (too ephemeral) and EC2 (too expensive at rest). E2B solved this with Firecracker ...
Agent Memory Deep Dive: How It Works, Why It Breaks, and Where Research Is Headed
TLDR: Every major AI agent made a different bet on memory — files, vectors, graphs, virtual memory, temporal decay. Each is defensible. None is complete. The deeper question isn't which framework i...
Agent Memory Deep Dive #5: What a Brain-Inspired Memory Architecture Actually Looks Like
TLDR: Human memory has three systems (episodic, semantic, procedural), five properties (selective encoding, cue-based retrieval, offline consolidation, reconsolidation on recall, forgetting as feat...
Agent Memory Deep Dive #4: The Cross-Tool Memory Gap — Why Agents Are Siloed
TLDR: Every AI tool has its own memory island. Claude Code knows your project preferences. ChatGPT knows your writing style. Cursor knows your codebase. Gemini knows your research questions. None s...
Agent Memory Deep Dive #3: The Forgetting Problem — Why Append-Only Memory Breaks
TLDR: Every major memory system is append-only. After months of use, stale facts overwhelm retrieval. The fix isn't deletion — it's graduated strength decay: memories that are never accessed gradua...
Agent Memory Deep Dive #2: The Memory Landscape — 10 Approaches, No Winner
TLDR: I evaluated 10 open-source memory projects across 7 capability dimensions. They cluster into 5 archetypes: append-only stores, agent-managed virtual memory, graph-based systems, self-organizi...
Agent Memory Deep Dive #0: The State of Agent Memory — Why It's Still Broken
TLDR: Agent memory is the most visibly broken component of the current AI stack. I spent some time researching every major approach — from Claude Code's file-based system to Mem0's vector store to ...
Agent Memory Deep Dive #1: Why Your Agent Forgets — The 4 Failure Modes
TLDR: Agent memory failures aren't bugs — they're architectural gaps. Four distinct failure modes explain every frustrating 'I already told you this' moment: context overflow (finite window, silent...
Two Real Problems in the Agent Domain
I run multiple AI sessions simultaneously across providers:
Agent Landscape #2: Orchestration — Pick Your Reasoning Loop
TLDR: Pick LangGraph (34k) for complex branching logic, CrewAI (53k stars) for fast prototyping, PydanticAI (17k) for type-safe single-agent, AgentScope (26k) for batteries-included. Hermes (190k) ...
Agent Landscape #6: Coordination & Protocols — Agents Talking to Everything
TLDR: Three protocols cover the full agent communication surface: MCP (agent↔tools, 87k servers), A2A (agent↔agent, Linux Foundation), AG-UI (agent↔user, adopted by Google/AWS/MS). Adopt all three....
Agent Landscape #3: Durability — Why Your Agent Can't Survive a Restart
TLDR: Most agent frameworks lose all progress on crash. Temporal (21k) and Inngest (5.5k) solve durable execution for generic workflows, but neither is agent-aware. The gap: agent-native event sour...
Agent Landscape #1: The 5 Things Every Production Agent Needs
TLDR: Every production agent needs 5 components: orchestration (reasoning loop), durable state (survive crashes), memory (3-tier: working + long-term + session), sandboxed execution (isolated tool ...
Agent Landscape #4: Memory — Beyond Context Window Overflow
TLDR: Agents need three memory tiers: working (in context window), long-term (vector-searchable), and session (per-conversation). Letta (23k) is the most complete — agents explicitly manage their o...
Agent Landscape #5: Sandboxing — The Missing Security Boundary
TLDR: Most frameworks run tools in-process with full permissions — fine for demos, fatal in production. Daytona (73k) offers the fastest sandbox (<90ms, self-hostable, MCP-native). E2B (12k) provid...
Agent-Native: Research & Landscape
| Criterion | Agent-Native | AI-Enhanced (not this) |
Building GPT from Scratch — Part 8: FSDP — When the Model Won't Fit
DDP replicates the full model on every GPU — which fails when the model exceeds one GPU's memory. FSDP (Fully Sharded Data Parallel) shards parameters, gradients, and optimizer states across GPUs. ...
Building GPT from Scratch — Part 10: Disaggregated Inference — Splitting Prefill and Decode
Prefill is compute-bound (process all prompt tokens in parallel). Decode is memory-bound (read KV cache for one new token at a time). Running both on the same GPU forces a compromise — you can't op...
Building GPT from Scratch — Part 7: Distributed Data Parallel Training
One GPU trains sequentially. Multiple GPUs split the batch — each processes a portion, computes gradients locally, then AllReduce averages the gradients across all GPUs before updating. Same conver...
Building GPT from Scratch — Part 9: Tensor Parallelism — Splitting Layers Across GPUs
Data parallelism replicates the model and splits data. Tensor parallelism splits individual layers across GPUs — each GPU computes a portion of every matrix multiplication. The attention layer spli...
Agent-Native Whitespace: Underserved Directions
```
Building GPT from Scratch — Part 3: The Transformer Architecture
The GPT architecture is a stack of transformer blocks, each containing causal self-attention (lets each token attend to all previous tokens) and a feed-forward network (per-token nonlinear processi...
Building GPT from Scratch — Part 5: Fine-tuning
Fine-tuning doesn't teach new knowledge — it teaches new behavior. Same model, same loss function, different data format. By training on instruction-response pairs instead of raw text, the model le...
Building GPT from Scratch — Part 6: Inference
Naive generation recomputes all previous tokens every step — O(n²) total. KV cache stores past Keys and Values so each new token only computes one query against the cached context — O(n) total. Thi...
Building GPT from Scratch — Part 1: What Is GPT?
We build a working GPT from scratch — tokenizer, transformer, pretraining, fine-tuning, and inference with KV cache — in ~500 lines of PyTorch on a MacBook. The model is tiny (3.3M params, trains i...
Building GPT from Scratch — Part 4: Pretraining
Pretraining has one objective: predict the next token. That's it. No grammar rules, no world knowledge injected — just next-token prediction via cross-entropy loss. To predict well, the model must ...
Building GPT from Scratch — Part 2: Tokenization
Neural networks need numbers, not text. Byte Pair Encoding (BPE) solves this by iteratively merging the most frequent character pairs into tokens — a compression algorithm repurposed for NLP. It hi...
Build GPT from Scratch in 500 Lines of PyTorch
We build a working GPT from scratch — tokenizer, transformer, pretraining, fine-tuning, and inference with KV cache — in ~500 lines of PyTorch on a MacBook. The model is tiny (3.3M params, trains i...
Prefill vs Decode — Why One GPU Can't Do Both Well
LLM inference has two phases that look nothing alike: prefill (process the entire prompt — compute-heavy, parallel, fast) and decode (generate tokens one at a time — memory-bound, sequential, slow)...
Why Next-Token Prediction Produces Intelligence
LLMs have one objective: predict the next token. Critics call this 'glorified autocomplete.' They're right about the objective function — and completely wrong about what happens inside. To predict ...
Attention at Scale — From FlashAttention to FlashAttention-4
Attention is the single most expensive operation in a transformer. It's also where the most performance is left on the table. FlashAttention (2022) showed you could make it 2-4x faster by respectin...
The KV Cache Problem — Why Memory Is the New Compute
Every LLM serving system is bottlenecked by the same thing: KV cache memory. Not model weights. Not compute. Memory. The KV cache grows linearly with context length, persists for the duration of ge...
Naval's Playbook, Re-read in the Age of AI
I've been re-reading Naval's ideas — the almanack, the tweetstorms, the podcasts. What strikes me is how much more relevant they become when AI rewrites the rules of leverage.
Naval's Playbook for Engineers
Most engineers have heard Naval's ideas. Few have thought about how to actually apply them — day to day, in the terminal, in sprint planning, in career decisions. This is my attempt to bridge that ...
When Inference Becomes Training
The line between training and inference is blurring. A smaller model that 'thinks harder' at test time can beat a model 14x its size. But 'thinking harder' takes three very different forms — from j...
Why Serving LLMs Is Nothing Like Serving Traditional ML
Traditional ML serving is stateless: request comes in, model does a forward pass, response goes out. LLM serving is stateful, sequential, and memory-dominated. Every token generated depends on ever...
RLHF Is Dead, Long Live RL
Post-training used to be 'collect human preferences, train a reward model, run PPO.' That whole pipeline is collapsing. DeepSeek-R1 proved you can get reasoning to emerge from pure RL without human...
What Actually Happens When You Train on 10,000 GPUs
Training a frontier model is no longer a modeling problem — it's a distributed systems problem. Think of it like running a single stateful application across 10,000 nodes that must stay perfectly s...