Model Compression for Edge

Edge devices require aggressive compression — INT4 quantization, knowledge distillation, and pruning — to fit models into limited memory and compute budgets while maintaining acceptable quality.

1. The Compression Toolkit

Technique	Size reduction	Quality impact	Effort
INT8 quantization (PTQ)	2×	<1%	Minutes
INT4 quantization (PTQ)	4×	1-3%	Minutes-hours
INT4 quantization (QAT/GPTQ)	4×	<1%	Hours
2-bit quantization	8×	3-10%	Hours + careful tuning
Knowledge distillation	Train smaller model	Depends on capacity gap	Days (full training)
Structured pruning	1.5-3×	1-5%	Hours-days
Architecture choice	Use smaller model	Design-time decision	—

Most practical for edge: Start with the smallest model that meets quality requirements, then apply INT4 quantization.

2. Quantization for Edge

PTQ (Post-Training Quantization)

Fastest path — no retraining:

# GGUF quantization for llama.cpp
./llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M

# ExecuTorch quantization
python -m executorch.examples.models.llama.export_llama \
    --model llama-3.2-1b \
    --quantization int4 \
    --output llama-1b-int4.pte

QAT (Quantization-Aware Training)

Better quality at low bit widths:

# Simulate quantization during training
from torchao.quantization import quantize_, int4_weight_only

model = load_model("llama-3.2-1b")
quantize_(model, int4_weight_only(group_size=32))
# Fine-tune for a few hundred steps on calibration data

Edge-Specific Considerations

Consideration	Cloud quantization	Edge quantization
Target precision	FP8, INT8	INT4, INT4+INT8 mixed
Hardware support	Tensor Cores (uniform)	Varies by device (NPU, CPU, GPU)
Operator coverage	Full	Partial — some ops may not quantize
Calibration	Standard dataset	Must match on-device distribution

3. Knowledge Distillation

Train a smaller “student” model to mimic a larger “teacher”:

Teacher (70B): high quality, too large for device
Student (1B): small enough for device, trained to match teacher outputs

Training: minimize KL(teacher_output || student_output) on large dataset
Result: 1B model with quality approaching 3-7B model

Teacher → Student	Quality retained	Size reduction
70B → 8B	~95%	9×
8B → 1B	~85%	8×
3B → 0.5B	~80%	6×

Distillation + quantization: Distill to 1B, then quantize to INT4 = 0.5 GB model with decent quality.

4. Structured Pruning

Remove entire neurons, attention heads, or layers — reducing actual compute (unlike unstructured pruning which creates sparse matrices that may not speed up):

Target	Method	Speedup
Attention heads	Remove low-importance heads	1.2-1.5×
FFN neurons	Remove columns/rows from weight matrices	1.3-2×
Entire layers	Remove deepest layers (less important)	1.5-3×

Practical approach: Width pruning (remove neurons) is more hardware-friendly than depth pruning (remove layers) because it doesn’t change the architecture.

5. Choosing a Compression Strategy

Start: What's my device memory budget?

If budget > 4 GB:  3B model + INT4 → 1.5 GB (comfortable)
If budget > 2 GB:  1B model + INT4 → 0.5 GB (comfortable)
If budget > 1 GB:  1B model + INT4 + pruning → 0.3-0.4 GB
If budget < 1 GB:  0.5B distilled + INT4 → 0.25 GB
                   Or: use cloud inference instead

For quantization techniques in detail, see Quantization Fundamentals. For edge runtime options, see Edge Inference Engines.

Self-Check

Choose a compression strategy for a 3 GB memory budget
Compare PTQ vs QAT for INT4 on edge devices
Explain knowledge distillation: teacher-student training
Explain structured pruning and why it’s more hardware-friendly than unstructured
Calculate final model size: distilled 1B + INT4 + group_size=32