Continual Learning — infra.bea.ai

When you fine-tune a model on new data, it tends to forget old capabilities — “catastrophic forgetting.” Continual learning techniques allow incremental updates without starting from scratch.

1. The Forgetting Problem

Fine-tuning on domain B degrades performance on domain A:

Model state	Domain A (general)	Domain B (medical)
Base model	85%	40%
After SFT on B	60% (↓25%)	92%
Target	85%	92%

The weight updates that improve medical knowledge overwrite the representations used for general knowledge.

2. Mitigation Strategies

Replay Buffers

Mix old data into new training:

# 80% new domain data, 20% replay from old training data
dataset = ConcatDataset([
    new_domain_data,                  # 80%
    Subset(original_training_data, indices=replay_indices),  # 20%
])

Effectiveness: Simple, works well, but requires access to original training data.

Elastic Weight Consolidation (EWC)

Add a regularization term that penalizes changing important weights:

L_{\text{total}} = L_{\text{task}} + \frac{\lambda}{2} \sum_i F_i (\theta_i - \theta_i^*)^2

Where $F_i$ is the Fisher information (importance) of parameter $i$ , and $\theta^*$ are the original weights. Important weights are penalized more for changing.

LoRA for Continual Learning

Train separate LoRA adapters for each domain:

Domain	Approach	Forgetting?
Medical	LoRA adapter A	No (base unchanged)
Legal	LoRA adapter B	No (base unchanged)
Both	Merge or switch adapters	No forgetting

LoRA naturally prevents catastrophic forgetting because the base model is frozen. Each adapter learns domain-specific adjustments without interfering.

Knowledge Distillation

Use the old model as a teacher to regularize the new model:

L = \alpha \cdot L_{\text{task}} + (1-\alpha) \cdot \text{KL}(p_{\text{old}} \| p_{\text{new}})

The KL divergence term ensures the updated model’s output distribution stays close to the original on general inputs. This is used in practice by teams that do periodic “model merges” — combining capabilities from separately fine-tuned models.

Data Mixing (Most Common in Practice)

The simplest and most effective strategy for production: always train on a mix of old and new data, with proportions tuned by evaluation:

mix_ratios = {
    "original_pretraining_sample": 0.10,   # prevent forgetting
    "previous_sft_data": 0.20,              # maintain old skills
    "new_domain_data": 0.60,                # learn new capability
    "safety_data": 0.10,                    # maintain alignment
}

This is what most production teams actually do — it’s simple, effective, and doesn’t require complex algorithms. Llama-3.1 used extensive data mixing during continued pre-training to add multilingual capabilities without losing English performance.

For LoRA as a continual learning strategy, see LoRA.

3. Practical Recommendations

Scenario	Best approach
Adding a new capability	LoRA adapter (no forgetting)
Improving existing capabilities	Replay buffer (mix old + new)
Periodic model refresh	Full retrain on combined data
Continuous data stream	EWC + replay buffer

Self-Check

Explain catastrophic forgetting with a concrete example
Implement a replay buffer for continual fine-tuning
Explain EWC’s Fisher information regularization
Explain why LoRA naturally prevents forgetting