Navigation
📐 Foundations· 14 min read

How Processors Execute

From logic gates to instruction pipelines — how CPUs and GPUs turn transistors into computation, and why parallelism requires fundamentally different hardware.

processorcpugpuinstruction-pipelinesimdsimtparallelismclock-cycle

A transistor switches on and off. A logic gate combines switches into decisions. But how do billions of gates turn into a machine that can multiply matrices, run Python, and generate text? This chapter bridges the gap from “what is a transistor” to “what is a CUDA Core.”

1. From Logic Gates to Arithmetic

The Adder: Computing 1 + 1

The simplest computation — adding two 1-bit numbers — requires just two gates:

Inputs: A=1, B=1

  A ──┬── XOR ── Sum = 0
      │
  B ──┼── AND ── Carry = 1
      │
  Result: 10 (binary) = 2 (decimal)

A full adder handles a carry-in from the previous bit. Chain 32 full adders together and you get a 32-bit integer adder — the foundation of all arithmetic.

The Multiplier

Multiplication is repeated addition with shifts:

    1101  (13)
  × 1011  (11)
  ──────
    1101  (13 × 1, shift 0)
   1101   (13 × 1, shift 1)
  0000    (13 × 0, shift 2)
 1101     (13 × 1, shift 3)
──────────
10001111  (143)

Hardware multipliers do all partial products in parallel using a tree of adders. A single-cycle integer multiplier needs ~4000 gates for 32-bit numbers.

Fused Multiply-Add (FMA)

The FMA computes a×b+ca \times b + c as a single operation — the atomic operation of deep learning:

a ──→ ┌──────────┐
      │ Multiply │──→ ┌──────┐
b ──→ └──────────┘    │  Add │──→ result
                      │      │
c ──────────────────→ └──────┘

FMA is “fused” because it does the multiply and add with a single rounding step (instead of rounding after multiply, then rounding after add). This gives better numerical precision and counts as 2 floating-point operations (1 multiply + 1 add).

Every CUDA Core is essentially an FMA unit. Every Tensor Core is a grid of FMA units operating on matrix tiles. Every TFLOPS metric counts FMA operations.

2. The Clock: Synchronizing Everything

All digital circuits are synchronized by a clock signal — a square wave that toggles between 0 and 1 at a fixed frequency.

Clock:  ─┐  ┌──┐  ┌──┐  ┌──┐  ┌──
         └──┘  └──┘  └──┘  └──┘
         ← one cycle →
  • Clock frequency: how many cycles per second (Hz). An H100 runs at ~1.8 GHz = 1.8 billion cycles/second.
  • Each cycle: all circuits read their inputs, compute, and write outputs.
  • Higher frequency = faster but generates more heat (power ∝ frequency × voltage²)

Why Clock Speed Stopped Scaling

CPUs hit ~4 GHz around 2005 and stopped. Why? The power wall:

P=C×V2×fP = C \times V^2 \times f

Where CC = capacitance (chip area), VV = voltage, ff = frequency. Higher frequency requires higher voltage, and power grows with voltage squared. At 4+ GHz, chips melt.

The solution: instead of faster clocks, add more parallel units. This is why GPUs exist — thousands of slower cores instead of a few fast ones.

3. How a CPU Executes Instructions

The Instruction Pipeline

A CPU processes instructions in stages, like a factory assembly line:

Cycle 1:  [Fetch₁] 
Cycle 2:  [Decode₁] [Fetch₂]
Cycle 3:  [Execute₁] [Decode₂] [Fetch₃]
Cycle 4:  [Write₁]  [Execute₂] [Decode₃] [Fetch₄]
StageWhat happens
FetchRead the next instruction from memory
DecodeFigure out what the instruction does (add? multiply? load?)
ExecuteDo the computation (ALU, FMA, etc.)
Write-backStore the result to a register or memory

Pipelining means the CPU starts a new instruction every cycle, even though each instruction takes 4+ cycles to complete. Throughput: ~1 instruction per cycle (even though latency is ~4 cycles).

Branch Prediction: The CPU’s Gamble

When the CPU encounters an if-else:

if score > threshold:
    result = expensive_function()
else:
    result = cheap_function()

The CPU doesn’t know which branch to take until score > threshold is evaluated. But the pipeline has already fetched the next instruction — which one?

Branch prediction: the CPU guesses which branch is more likely (based on history) and speculatively executes it. If the guess is right → no delay. If wrong → flush the pipeline and restart (penalty: ~15-20 cycles).

Modern CPUs predict correctly ~95-99% of the time. This is one of the key reasons CPUs are fast at serial, branchy code — and why GPUs avoid branches entirely.

Out-of-Order Execution

CPUs don’t execute instructions in strict order. If instruction 3 is waiting for data but instruction 4 is ready, the CPU executes 4 first:

Instruction 1: a = load(memory)     ← waiting for memory (~600 cycles)
Instruction 2: b = a + 1            ← depends on a, must wait
Instruction 3: c = 5 * 7            ← independent! Execute now
Instruction 4: d = c + 3            ← depends on c, execute after 3

This out-of-order execution keeps the CPU busy even when some data is slow to arrive. It requires complex hardware (reorder buffer, reservation stations) — which is why CPU cores are “big” and there are few of them.

4. CPU vs GPU: Two Design Philosophies

CPU vs GPU Design

CPU: Optimize for Latency

CPU Core (big, complex):
┌─────────────────────────────────────────┐
│  Branch Predictor │ Out-of-Order Engine │  ← complex control
│  Large L1/L2 Cache                      │  ← hide memory latency
│  ┌─────┐ ┌─────┐                       │
│  │ ALU │ │ FMA │  (few, powerful units) │
│  └─────┘ └─────┘                       │
└─────────────────────────────────────────┘
× 4-64 cores per chip

Design goal: Execute a single thread as fast as possible. Complex control logic (branch prediction, out-of-order) + large caches to hide memory latency.

GPU: Optimize for Throughput

GPU SM (small, simple, many):
┌─────────────────────────────┐
│  Simple Scheduler            │  ← no branch prediction
│  Tiny Cache                  │  ← rely on parallelism to hide latency
│  ┌───┐┌───┐┌───┐┌───┐...   │
│  │FMA││FMA││FMA││FMA│ ×128 │  (many, simple units)
│  └───┘└───┘└───┘└───┘      │
└─────────────────────────────┘
× 132 SMs per chip (H100)

Design goal: Execute thousands of threads simultaneously. Simple control + small caches, but massive parallelism. Hide memory latency by switching between threads (when one warp waits for data, another warp executes).

SIMD vs SIMT

ModelWhatUsed byExample
SIMD (Single Instruction, Multiple Data)One instruction operates on a vector of dataCPU (AVX-512)add [a0,a1,...,a7], [b0,b1,...,b7] — process 8 elements at once
SIMT (Single Instruction, Multiple Threads)One instruction dispatched to many threads, each with own registersGPU (CUDA)32 threads in a warp execute add r1, r2, r3 — same instruction, different register values

The key difference: SIMD operates on a fixed-width vector. SIMT operates on independent threads that happen to run the same instruction — they can diverge (at a performance cost).

Why SIMT suits deep learning: Matrix multiplication is naturally parallel — each output element is an independent dot product. You don’t need branch prediction or out-of-order execution. You just need thousands of FMA units running in lockstep.

5. Memory Hierarchy: Why Data Movement Dominates

Every processor, CPU or GPU, faces the same problem: compute is fast, memory is slow.

                Speed           Size        Cost
Registers:      0 cycles        KB          $$$$  (in the core)
L1 Cache:       ~4 cycles       64 KB       $$$   (on die)
L2 Cache:       ~12 cycles      1-50 MB     $$    (on die)
Main Memory:    ~200-600 cycles GB-TB       $     (off die, separate chips)

A compute unit can do an FMA every cycle, but loading the data from main memory takes hundreds of cycles. The entire memory hierarchy (caches, registers, shared memory, HBM) exists to solve this gap.

This is the fundamental constraint of AI infrastructure: not how fast you can compute, but how fast you can feed data to the compute units. A GPU that can do 989 TFLOPS is useless if it spends 90% of cycles waiting for data from HBM. This is why:

  • FlashAttention computes in SRAM tiles instead of reading/writing to HBM
  • Kernel fusion combines operations to avoid writing intermediate results to memory
  • Tensor Cores do more arithmetic per memory access (one MMA = 128+ FMAs from a small tile)
  • HBM provides ~10× the bandwidth of DDR — but is still the bottleneck for decode

6. Putting It Together: From Transistor to TFLOPS

The complete stack:

Transistors (80B in H100)
  → Logic gates (AND, OR, NOT)
  → FMA units (multiply-add in one cycle)
  → CUDA Cores (one FMA unit each, 128 per SM)
  → Tensor Cores (matrix of FMA units, 4 per SM)
  → Streaming Multiprocessor / SM (128 CUDA + 4 Tensor + shared memory)
  → GPU chip (132 SMs in H100)
  → Module (GPU die + HBM via CoWoS + NVLink connector)
  → Server node (8 GPUs + NVSwitch + CPUs + NICs)
  → Cluster (thousands of nodes + InfiniBand network)
  → Your training job

At every level, the design choices are driven by the same tension: more compute vs more memory bandwidth vs more communication bandwidth, constrained by power and cost.

Self-Check

  • Explain how a full adder works and how chaining them creates an integer adder
  • Explain FMA and why it’s the fundamental operation of deep learning
  • Explain the clock frequency power wall and why it led to parallel architectures
  • Explain the CPU instruction pipeline (fetch, decode, execute, write-back)
  • Explain branch prediction and why CPUs need it but GPUs don’t
  • Explain the difference between SIMD and SIMT
  • Explain why GPU SMs are simpler than CPU cores and why that’s a feature
  • Explain why memory hierarchy exists and how it relates to AI performance (compute vs memory-bound)

References