Navigation
💾 Data & Storage· 8 min read

Sequence Packing

Combining variable-length sequences into fixed-length training examples to eliminate padding waste — 2-5× throughput gain.

sequence-packingpaddingthroughputtrainingefficiency

Standard batching pads all sequences to the longest in the batch — wasting 30-70% of compute on padding tokens. Sequence packing eliminates this waste by concatenating multiple sequences into each training example.

1. The Padding Problem

Standard batching (max_len=2048):
  Seq A (200 tokens):  [A₁...A₂₀₀, PAD, PAD, ..., PAD]  ← 1848 PAD tokens (90% waste)
  Seq B (150 tokens):  [B₁...B₁₅₀, PAD, PAD, ..., PAD]  ← 1898 PAD tokens (93% waste)
  Seq C (1800 tokens): [C₁...C₁₈₀₀, PAD, ..., PAD]      ← 248 PAD tokens (12% waste)
  Seq D (500 tokens):  [D₁...D₅₀₀, PAD, PAD, ..., PAD]  ← 1548 PAD tokens (76% waste)

  Average utilization: (200+150+1800+500) / (4×2048) = 32% ← 68% wasted!

2. Packing: Concatenate Sequences

Sequence Packing

Packed batching (max_len=2048):
  Pack 1: [A₁...A₂₀₀ | B₁...B₁₅₀ | D₁...D₅₀₀ | E₁...E₃₀₀ | ...]  ← ~95% utilized
  Pack 2: [C₁...C₁₈₀₀ | F₁...F₂₄₈]                                   ← ~100% utilized

  Average utilization: ~95% ← almost no waste!

Throughput improvement: From 32% utilization to 95% → ~3× more useful tokens processed per step.

Attention Masking

Packed sequences must not attend to each other — use a block-diagonal attention mask:

Pack: [Seq A (200 tokens) | Seq B (150 tokens) | Seq C (500 tokens)]

Attention mask:
  A tokens can attend to: A tokens only
  B tokens can attend to: B tokens only
  C tokens can attend to: C tokens only

  Implemented as: document_ids = [0,0,...,0, 1,1,...,1, 2,2,...,2]
  mask[i][j] = (document_ids[i] == document_ids[j])

Most modern training frameworks (Megatron-LM, Llama training code) support this natively.

3. Packing Algorithms

AlgorithmHow it worksPacking efficiency
First-fit decreasingSort by length (longest first), pack greedily90-95%
Best-fitFind the pack with least remaining space95-98%
RandomRandom assignment with length constraint85-90%
Sorted batchingGroup similar lengths, no packing70-85%

First-fit decreasing is the most common — simple, fast, and achieves near-optimal packing.

4. Packing for Different Workloads

WorkloadPacking benefitNotes
Pre-training (web text)High (2-5×)Short documents dominate, lots of padding waste
SFT (instruction tuning)High (2-4×)Conversations vary wildly in length
Long-context trainingLow (1.1-1.3×)Most sequences already near max_len
Fixed-length (evaluation)NoneAll sequences same length

Implementation

# Simple first-fit-decreasing packing
def pack_sequences(sequences, max_len):
    sorted_seqs = sorted(sequences, key=len, reverse=True)
    packs = []
    
    for seq in sorted_seqs:
        # Try to fit into existing pack
        placed = False
        for pack in packs:
            if sum(len(s) for s in pack) + len(seq) <= max_len:
                pack.append(seq)
                placed = True
                break
        if not placed:
            packs.append([seq])  # new pack
    
    return packs

For how packed data is loaded into training, see Data Loading at Scale.

Self-Check

  • Calculate padding waste for a batch with sequences of lengths [200, 150, 1800, 500] at max_len=2048
  • Explain block-diagonal attention masking for packed sequences
  • Implement first-fit-decreasing packing
  • Estimate throughput improvement from packing for a given length distribution

References