Pipeline Schedules — infra.bea.ai

Pipeline parallelism splits a model across stages, but the schedule — the order of forward and backward passes across micro-batches — determines how much time is wasted in the “pipeline bubble.”

1. The Pipeline Bubble Problem

With $P$ pipeline stages and $M$ micro-batches per step:

Warm-up: First micro-batch must traverse all $P$ stages before any backward can start
Drain: Last backward must traverse all $P$ stages
Bubble: Stages sit idle during warm-up and drain

\text{Bubble fraction} = \frac{P - 1}{M}

For PP=4, M=4: bubble = 3/4 = 75% waste! For PP=4, M=32: bubble = 3/32 = 9.4%.

Rule: Use at least 4× as many micro-batches as pipeline stages to keep bubble < 25%.

Pipeline Schedules

2. GPipe Schedule (All-Forward-Then-All-Backward)

Stage 0: F1 F2 F3 F4 __________ B4 B3 B2 B1
Stage 1: __ F1 F2 F3 F4 _______ B4 B3 B2 B1
Stage 2: __ __ F1 F2 F3 F4 ____ B4 B3 B2 B1
Stage 3: __ __ __ F1 F2 F3 F4 _ B4 B3 B2 B1
         ↑                     ↑
         warm-up bubble        drain bubble

Problem: All activations from all micro-batches are stored simultaneously — massive memory spike. Stage 0 holds activations for all $M$ micro-batches before any backward starts.

Memory: $O(M)$ activation memory per stage.

3. 1F1B Schedule (One-Forward-One-Backward)

The key innovation: interleave forward and backward so each stage starts backward as soon as possible.

Stage 0: F1 F2 F3 F4 B1 B2 B3 B4
Stage 1: __ F1 F2 F3 B1 F4 B2 B3 B4
Stage 2: __ __ F1 F2 B1 F3 B2 F4 B3 B4
Stage 3: __ __ __ F1 B1 F2 B2 F3 B3 F4 B4

Steady state: Each stage alternates one forward and one backward — hence “1F1B.”

Memory: Only $P$ micro-batches’ activations in flight simultaneously (not $M$ ). For PP=4: 4 micro-batches’ activations instead of all $M$ .

Bubble: Same as GPipe: $(P-1)/M$ . But memory is much better.

4. Interleaved 1F1B (Virtual Stages)

The key insight: instead of each device handling one contiguous chunk of layers, assign multiple non-contiguous chunks (virtual stages). This allows more forward/backward interleaving opportunities.

Standard PP=4: Device 0 runs layers 1-20, device 1 runs layers 21-40, etc. Interleaved PP=4 with v=2: Device 0 runs layers 1-10 AND 41-50, device 1 runs layers 11-20 AND 51-60, etc.

Why This Helps

The real benefit is reducing the number of physical pipeline stages while maintaining a deep pipeline:

With standard PP=4: bubble = $(4-1)/M$
With interleaved PP=2 physical, v=2 virtual (4 total): bubble = $(4-1)/M$ but only 2 nodes involved instead of 4
Fewer physical stages = fewer inter-node P2P hops = less communication

The tradeoff: each micro-batch makes more P2P trips (traversing virtual stages on the same device requires sending to the next device, then back). So communication volume increases, but the pipeline is deeper relative to the physical stage count.

Configuration	Physical stages	Virtual stages	Bubble	Communication
Standard PP=4	4	4	$(3)/M$	1 P2P per stage
Interleaved PP=2, v=2	2	4	$(3)/M$	2 P2P per physical stage
Interleaved PP=4, v=2	4	8	$(7)/M$	2 P2P per physical stage

Best use case: When you want deep pipeline (many virtual stages for small bubble) but can only afford a few physical stages (limited inter-node bandwidth).

5. Zero-Bubble Schedules

Recent research eliminates the bubble entirely by overlapping forward, backward-for-weights ( $B_W$ ), and backward-for-inputs ( $B_I$ ):

Standard backward: B = compute dL/dW (weight grad) + dL/dx (input grad)
Zero-bubble split: B_W and B_I can be scheduled independently

B_I must happen before the previous stage's backward
B_W can be delayed — only needed before optimizer step

By scheduling $B_W$ into bubble slots:

Stage 0: F1 F2 F3 F4 B1i B2i B3i B4i [B1w B2w B3w B4w fill bubbles]

Result: Near-zero bubble at the cost of higher memory (delayed weight gradients accumulate).

Schedule	Bubble	Memory	Communication	Complexity
GPipe	$(P-1)/M$	$O(M)$ per stage	Minimal	Simple
1F1B	$(P-1)/M$	$O(P)$ per stage	Same as GPipe	Moderate
Interleaved 1F1B	$(P_v-1)/(M \times V)$	$O(P_v)$	More P2P	High
Zero-bubble	~0	Higher (delayed $B_W$ )	Same	Very high

6. Choosing a Schedule

Scenario	Recommended	Why
PP=2, large M	1F1B	Small bubble, simple
PP=4-8, want minimum bubble	Interleaved 1F1B	Standard for Megatron-LM
PP=16+, bubble is significant	Zero-bubble	Research frontier
Memory-limited	1F1B	Lowest activation memory

For how to quantify bubble overhead in your training, see Pipeline Bubble Analysis.

Self-Check

Calculate bubble fraction for GPipe with PP=4, M=16
Explain why 1F1B has the same bubble as GPipe but better memory
Explain interleaved 1F1B’s virtual stage concept
Compare zero-bubble’s split backward ( $B_W$ vs $B_I$ ) approach
Choose a schedule for PP=8 with 64 micro-batches