LoRA — infra.bea.ai

LoRA (Low-Rank Adaptation) freezes the pretrained model and adds small trainable matrices to selected layers. It reduces trainable parameters by 100-1000× while achieving quality comparable to full fine-tuning.

1. The Core Idea

LoRA Decomposition

For a pretrained weight matrix $W_0 \in \mathbb{R}^{d \times k}$ , LoRA adds a low-rank update:

W = W_0 + \Delta W = W_0 + BA

where $B \in \mathbb{R}^{d \times r}$ , $A \in \mathbb{R}^{r \times k}$ , and $r \ll \min(d, k)$ .

	Full fine-tune	LoRA (r=16)
Trainable params (per layer, $d=4096$ )	$d \times k = 16.8$ M	$(d + k) \times r = 131$ K
Reduction	1×	128×
Memory for optimizer	Full 12Ψ	12 × Ψ_adapter (tiny)
Training speed	Baseline	Often faster (smaller gradients)

Why Low-Rank Works

Weight updates during fine-tuning live in a low-dimensional subspace. The pretrained model has learned the task-general representation; fine-tuning only needs to steer it — a small adjustment in a low-rank direction.

2. Where to Apply LoRA

Target module	Effect	Recommendation
Q, K, V projections	Modifies attention patterns	✅ Always include
Output projection	Modifies attention output mix	✅ Include
FFN up/gate/down	Modifies knowledge retrieval	✅ Include for best quality
Embedding	Modifies token representations	⚠️ Usually skip
LM head	Modifies output distribution	⚠️ Usually skip

Best practice: Apply LoRA to all linear layers in attention AND FFN. This is “LoRA on all linear” — the default in modern frameworks.

from peft import LoraConfig, get_peft_model

config = LoraConfig(
    r=16,                    # rank
    lora_alpha=32,           # scaling factor (alpha/r applied to ΔW)
    target_modules=[         # apply to all attention + FFN
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ],
    lora_dropout=0.05,
)
model = get_peft_model(model, config)
model.print_trainable_parameters()
# trainable params: 83,886,080 || all params: 6,738,415,616 || trainable%: 1.245%

3. Rank Selection

Rank (r)	Trainable params	Quality	When to use
4	Very few	Good for simple tasks	Classification, simple format
8-16	Few	Good for most tasks	Default choice
32-64	Moderate	Near full fine-tune	Complex tasks, large domain shift
128-256	Many	~Full fine-tune	When quality is paramount

Rule of thumb: Start with r=16. Increase if quality is insufficient; decrease if you need fewer parameters for multi-adapter serving.

Alpha and Scaling

The actual update applied is:

\Delta W = \frac{\alpha}{r} \cdot BA

The $\alpha/r$ scaling ensures the LoRA contribution is consistent across different ranks. Common: $\alpha = 2r$ (so scaling factor = 2).

4. Training Configuration

# Typical LoRA fine-tuning setup
from transformers import TrainingArguments

training_args = TrainingArguments(
    learning_rate=2e-4,       # higher LR than full fine-tune (1e-5)
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    num_train_epochs=3,
    bf16=True,
    optim="adamw_torch",     # only optimizes adapter params
)

Higher learning rate (10-100× vs full fine-tune) because adapter params are initialized near zero and need larger steps.

Memory Comparison

Configuration	7B model
Full fine-tune (BF16 + Adam)	112 GB
LoRA r=16 (frozen base + adapter)	14 GB (base) + 0.6 GB (adapter + optimizer) ≈ 15 GB
LoRA on 24 GB GPU	✅ Fits easily

5. DoRA: Weight-Decomposed Low-Rank Adaptation

DoRA decomposes the weight update into magnitude and direction components:

W = m \cdot \frac{W_0 + BA}{\|W_0 + BA\|}

Where $m$ is a learnable magnitude vector. This better mimics how full fine-tuning updates weights (which change both magnitude and direction), leading to ~1-2% quality improvement over standard LoRA at the same rank.

When to use DoRA: When you need LoRA-level efficiency but slightly better quality. Supported in PEFT: LoraConfig(use_dora=True).

6. When Full Fine-Tuning Still Wins

LoRA doesn’t always match full fine-tuning:

Scenario	LoRA quality vs full FT	Recommendation
Simple task adaptation (formatting, style)	99%+	LoRA (no reason to full FT)
Domain shift (medical, legal)	95-98%	LoRA usually sufficient
Significant new knowledge	90-95%	Consider higher rank or full FT
Pre-training on new language	80-90%	Full FT required
Architecture changes (adding layers)	N/A	Full FT required

The rule: If the task requires the model to learn fundamentally new representations (not just steer existing ones), full fine-tuning wins. LoRA works by adjusting within the subspace the model already knows; it can’t create entirely new representations.

For fine-tuning with quantized base models, see QLoRA. For serving multiple adapters on a shared base, see Multi-Adapter Serving.

7. Merging and Deployment

After training, merge LoRA weights back into the base model:

W_{\text{merged}} = W_0 + \frac{\alpha}{r} \cdot BA

model = model.merge_and_unload()  # merge adapter into base weights
model.save_pretrained("merged_model")
# Result: standard model with no adapter overhead at inference

No inference overhead after merging — the model is the same size and speed as the original, just with different weights.

For quantized base models with LoRA adapters, see QLoRA. For serving multiple adapters on a shared base, see Multi-Adapter Serving.

Self-Check

Explain the low-rank decomposition $\Delta W = BA$ and why it works
Calculate trainable parameters for LoRA r=16 on a 7B model’s attention layers
Choose rank $r$ for a given task complexity
Configure LoRA with PEFT library, targeting all linear layers
Merge LoRA weights and deploy as a standard model