The Memory Wall — infra.bea.ai

The memory wall is a hard physical constraint: as models scale, memory requirements grow faster than GPU memory capacity. This chapter quantifies the wall for every model size and shows exactly where infrastructure decisions are forced.

1. The Wall in Numbers

Weights Only (Inference Baseline)

M_{\text{weights}} = \Psi \times \text{bytes\_per\_param}

Model	BF16 (2B)	INT8 (1B)	INT4 (0.5B)	Fits on 1× 80 GB GPU?
1.5B	3 GB	1.5 GB	0.75 GB	✅ (any precision)
7B	14 GB	7 GB	3.5 GB	✅ (any precision)
13B	26 GB	13 GB	6.5 GB	✅ (any precision)
70B	140 GB	70 GB	35 GB	❌ BF16, ✅ INT8, ✅ INT4
405B	810 GB	405 GB	202 GB	❌ ❌ ❌
671B (MoE, all experts)	1,342 GB	671 GB	336 GB	❌ ❌ ❌

Training (Full State)

Training multiplies the problem by 8-16×:

Model	Weights (BF16)	Full training (16Ψ)	Min GPUs (80 GB, FSDP)
1.5B	3 GB	24 GB	1 (barely)
7B	14 GB	112 GB	2
13B	26 GB	208 GB	3
70B	140 GB	1,120 GB	14
405B	810 GB	6,480 GB	81

2. The Three Memory Walls

Wall 1: Single-GPU Capacity

When: Weight memory (BF16) exceeds GPU memory. Where: ~40B parameters on 80 GB GPU, ~13B on 24 GB consumer GPU. Force: Must use TP (split model across GPUs) or PP (split layers across GPUs).

Wall 2: Single-Node Capacity

When: Weight memory exceeds total node memory (8 × 80 GB = 640 GB). Where: ~320B parameters in BF16. Force: Must use multi-node training with InfiniBand. Cannot use NVLink-only strategies.

Wall 3: Cluster Memory

When: Total training state exceeds cluster memory. Where: Trillion-parameter models on small clusters. Force: Must use aggressive optimization: FSDP + activation checkpointing + CPU offloading.

3. How Model Scaling Hits the Walls

Modern LLMs follow scaling laws that increase all dimensions:

Generation	Parameters	Hidden dim	Layers	Wall hit
GPT-2 (2019)	1.5B	1600	48	None — fits on one V100
GPT-3 (2020)	175B	12288	96	Wall 2 — multi-node required
LLaMA-2 (2023)	70B	8192	80	Wall 1 (BF16), borderline Wall 2
LLaMA-3.1 (2024)	405B	16384	126	Wall 2 — ~81 GPUs minimum
DeepSeek-V3 (2024)	671B MoE	7168	61	Wall 2 — but MoE reduces active params

MoE Softens the Wall

MoE models have many total parameters but only activate a subset per token:

Model	Total params	Active params/token	Training memory (active)
LLaMA-70B (dense)	70B	70B	1,120 GB
Mixtral 8×7B (MoE)	47B	13B	752 GB total, but experts are distributed
DeepSeek-V3 (MoE)	671B	37B	Shared params (11B) + 4 experts per GPU

With EP, each GPU only holds a subset of experts — the memory wall is determined by per-GPU parameters, not total parameters.

4. Memory Wall vs GPU Evolution

Are GPUs growing fast enough to keep up?

GPU	Year	Memory	Model that fits (BF16 inference)	Model that fits (training, FSDP×8)
V100	2017	32 GB	16B	10B
A100	2020	80 GB	40B	25B
H100	2022	80 GB	40B	25B
H200	2024	141 GB	70B	44B
B200	2024	192 GB	96B	60B
GB200 NVL72 (aggregate)	2024	13.8 TB	Effectively unlimited (single domain)	500B+ in one NVLink domain

Memory per GPU grows ~2× per generation (every 2-3 years). But models grow 10-100× per generation. The gap widens — distributed training is permanently necessary.

The GB200 NVL72’s 13.8 TB NVLink domain fundamentally changes this — for the first time, a 500B model can train within a single NVLink domain without InfiniBand. See NVLink & NVSwitch.

5. Practical Decision Points

Use this table to quickly determine your infrastructure needs:

Model size	Training approach	Min hardware
≤1.5B	Single GPU, no sharding	1× 24 GB GPU
3-7B	FSDP across 2-8 GPUs	1 node (8× 80 GB)
7-13B	FSDP, activation checkpointing	1-2 nodes
13-70B	TP=8 + FSDP, checkpointing	2-8 nodes
70-405B	TP=8 + PP + FSDP	16-64 nodes
>405B	TP=8 + PP + EP + FSDP	100+ nodes

For the complete decision algorithm, see Parallelism Planning Decision Process.

Self-Check

Calculate weights-only and training memory for a 13B model
Identify which memory wall (1, 2, or 3) a 175B model hits
Explain how MoE reduces the effective memory wall
Compare GPU memory growth rate vs model size growth rate
Determine minimum hardware for training a 70B model