⚡ AI Infra

Prefill vs Decode — Why One GPU Can't Do Both Well

LLM inference has two phases that look nothing alike: prefill (process the entire prompt — compute-heavy, parallel, fast) and decode (generate tokens one at a time — memory-bound, sequential, slow)...

Attention at Scale — From FlashAttention to FlashAttention-4

Attention is the single most expensive operation in a transformer. It's also where the most performance is left on the table. FlashAttention (2022) showed you could make it 2-4x faster by respectin...

The KV Cache Problem — Why Memory Is the New Compute

Every LLM serving system is bottlenecked by the same thing: KV cache memory. Not model weights. Not compute. Memory. The KV cache grows linearly with context length, persists for the duration of ge...

Naval's Playbook, Re-read in the Age of AI

I've been re-reading Naval's ideas — the almanack, the tweetstorms, the podcasts. What strikes me is how much more relevant they become when AI rewrites the rules of leverage.

Naval's Playbook for Engineers

Most engineers have heard Naval's ideas. Few have thought about how to actually apply them — day to day, in the terminal, in sprint planning, in career decisions. This is my attempt to bridge that ...

When Inference Becomes Training

The line between training and inference is blurring. A smaller model that 'thinks harder' at test time can beat a model 14x its size. But 'thinking harder' takes three very different forms — from j...

Why Serving LLMs Is Nothing Like Serving Traditional ML

Traditional ML serving is stateless: request comes in, model does a forward pass, response goes out. LLM serving is stateful, sequential, and memory-dominated. Every token generated depends on ever...

RLHF Is Dead, Long Live RL

Post-training used to be 'collect human preferences, train a reward model, run PPO.' That whole pipeline is collapsing. DeepSeek-R1 proved you can get reasoning to emerge from pure RL without human...

What Actually Happens When You Train on 10,000 GPUs

Training a frontier model is no longer a modeling problem — it's a distributed systems problem. Think of it like running a single stateful application across 10,000 nodes that must stay perfectly s...