QLoRA

QLoRA combines 4-bit quantization of the base model with standard LoRA adapters, enabling fine-tuning of models 4-8× larger than what fits in GPU memory with standard approaches.

---

1. How QLoRA Works

Component	Precision	Memory per param
Base model weights	NF4 (4-bit)	0.5 bytes
LoRA adapters (A, B)	BF16	2 bytes
Adapter gradients	BF16	2 bytes
Adapter optimizer (Adam)	FP32	12 bytes per adapter param

Key Innovation: NF4 (Normal Float 4-bit)

NF4 is a 4-bit quantization format optimized for normally-distributed weights:

• Quantization levels are placed at the quantiles of a normal distribution
• Information-theoretically optimal for neural network weights (which are approximately Gaussian)
• Better than uniform INT4 for the same bit budget

Double Quantization

QLoRA also quantizes the quantization constants themselves:

• Block size 64: one FP32 scale per 64 weights → 0.5 bits/weight overhead
• Double quantize: quantize those scales to FP8 → 0.125 bits/weight overhead
• Total: 4 + 0.125 = 4.125 bits per weight

---

2. Memory Budget

Model	BF16	QLoRA (NF4)	Fits on 24 GB?	Fits on 48 GB?
7B	14 GB	3.8 GB + adapters	✅	✅
13B	26 GB	7.1 GB + adapters	✅	✅
70B	140 GB	38 GB + adapters	❌	✅
405B	810 GB	220 GB + adapters	❌	❌

QLoRA makes 70B fine-tuning feasible on a single A6000 (48 GB) — impossible with standard LoRA (needs at least 14 GB for BF16 base).

---

3. Implementation

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model

# Load base model in 4-bit
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",          # NF4 format
    bnb_4bit_compute_dtype=torch.bfloat16,  # compute in BF16
    bnb_4bit_use_double_quant=True,     # double quantization
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-70b-hf",
    quantization_config=bnb_config,
    device_map="auto",
)

# Add LoRA adapters (these are in BF16)
lora_config = LoraConfig(
    r=16, lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                     "gate_proj", "up_proj", "down_proj"],
    lora_dropout=0.05,
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)

The Backward Pass Through Quantization

During backward, gradients must flow through the quantized base weights:

1. Quantized weights (NF4) are dequantized to BF16 on-the-fly
2. Forward/backward computed in BF16
3. Only adapter gradients are computed and updated
4. Base weights remain frozen in NF4

No training of base model — gradient only flows through adapters. The NF4 base acts as a fixed feature extractor.

---

4. Quality Considerations

Method	7B quality (MMLU)	70B quality	Memory (70B)
Full fine-tune (BF16)	Baseline	Baseline	1,120 GB
LoRA r=16 (BF16 base)	~98% of full	~98%	140 GB
QLoRA r=16 (NF4 base)	~96% of full	~97%	38 GB

The ~1-2% quality gap comes from the quantized base: the feature extractor is slightly degraded. For most practical applications, this is acceptable.

When QLoRA Is NOT Enough

• Tasks requiring very precise base model outputs (e.g., code generation with exact syntax)
• When the domain shift from pretraining is very large
• When you need the best possible quality and have enough hardware

For serving fine-tuned models efficiently, see SFT Pipelines.

---

Self-Check

• Explain NF4 and why it’s better than uniform INT4 for neural network weights
• Calculate QLoRA memory for a 70B model with r=16 adapters
• Explain double quantization and its memory savings
• Implement QLoRA with bitsandbytes and PEFT
• Identify the quality tradeoff vs standard LoRA

---

References

• QLoRA: Efficient Finetuning of Quantized LLMs (Dettmers et al., 2023)
• bitsandbytes Library