Navigation
🏋️ Training Systems· 10 min read

SFT Pipelines

Chat templates, sequence packing, curriculum design, and evaluation checkpoints — the practical infrastructure of supervised fine-tuning.

sftfine-tuningchat-templatepackingcurriculumevaluation

Supervised fine-tuning (SFT) turns a base language model into an instruction-following assistant. The infrastructure involves data formatting, efficient batching, and evaluation — not just the training loop.

1. Chat Templates

Every model family has its own chat format:

# Llama-3 format
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
You are a helpful assistant.<|eot_id|>
<|start_header_id|>user<|end_header_id|>
What is CUDA?<|eot_id|>
<|start_header_id|>assistant<|end_header_id|>
CUDA is NVIDIA's parallel computing platform...<|eot_id|>

Key: Loss is computed only on assistant tokens — the model learns to generate responses, not to predict system prompts or user messages.

# Masking non-assistant tokens
labels = input_ids.clone()
labels[~assistant_mask] = -100  # PyTorch ignores -100 in cross-entropy

2. Sequence Packing

Naive batching wastes GPU compute on padding:

Batch without packing:
  [Turn 1 (200 tokens) | PAD PAD PAD ... PAD]  ← 1848 PAD tokens wasted
  [Turn 2 (150 tokens) | PAD PAD PAD ... PAD]  ← 1898 PAD tokens wasted
  
Batch with packing:
  [Turn 1 | Turn 2 | Turn 3 | Turn 4 | Turn 5]  ← all 2048 tokens are useful

Packing concatenates multiple conversations into one sequence (up to max length), with attention masks ensuring conversations don’t attend to each other.

Throughput improvement: 2-5× for datasets with short conversations.

# Attention mask for packing (block-diagonal)
# Conversation 1: tokens 0-199, Conversation 2: tokens 200-349, ...
# Each conversation only attends to itself
attention_mask = block_diagonal_mask(conversation_boundaries)

3. Curriculum Design

Order of training examples affects quality:

PhaseData typeEpochsLearning rate
Phase 1: Basic instructionSimple Q&A, formatting1-2Higher (2e-5)
Phase 2: Complex reasoningMath, code, multi-step2-3Lower (1e-5)
Phase 3: Style refinementSafety, helpfulness1Lowest (5e-6)

Data Mixing

Real SFT datasets mix multiple sources:

dataset = {
    "general_instructions": 40%,   # ShareGPT, OpenAssistant
    "code": 20%,                   # CodeAlpaca, self-instruct
    "math": 15%,                   # GSM8K, MATH
    "safety": 10%,                 # safety-tuning data
    "conversation": 15%,           # multi-turn dialogue
}

4. Evaluation Checkpoints

Save and evaluate at regular intervals:

training_args = TrainingArguments(
    eval_strategy="steps",
    eval_steps=500,
    save_steps=500,
    load_best_model_at_end=True,
    metric_for_best_model="eval_loss",
)

Evaluation benchmarks: MMLU (knowledge), HumanEval (code), GSM8K (math), MT-Bench (conversation quality). Run all benchmarks at each checkpoint to catch quality regressions.

For LoRA-specific fine-tuning setup, see LoRA. For post-SFT alignment, see RLHF/GRPO Pipeline.

Self-Check

  • Format a multi-turn conversation in Llama-3 chat template
  • Implement loss masking for assistant-only tokens
  • Explain sequence packing and its throughput benefit
  • Design a curriculum for a domain-specific SFT job
  • Set up evaluation checkpoints with multiple benchmarks

References