Navigation
🏋️ Training Systems· 8 min read

Multi-Adapter Serving

S-LoRA unified paging, Punica batched GEMM, adapter hot-swapping — serving hundreds of LoRA adapters on a shared base model.

multi-adapters-lorapunicaservinglorahot-swap

In production, you serve many fine-tuned variants (one per customer, task, or language) from a shared base model. Multi-adapter serving makes this efficient.

1. The Problem

Without multi-adapter serving, each LoRA variant needs its own model instance:

  • 100 adapters × 14 GB base = 1,400 GB — 18 H100 GPUs just for base model copies
  • Each adapter adds only ~100 MB, but the base model is duplicated

Multi-adapter serving: One base model in GPU memory, swap LoRA adapters per request.

2. S-LoRA: Unified Paging

S-LoRA stores all adapters in a unified memory pool and pages them in/out like virtual memory:

GPU Memory:
  [Base model: 14 GB] [Adapter page pool: 4 GB]
  
Request arrives for adapter_42:
  1. Check if adapter_42 is in page pool → if yes, use it
  2. If not, evict LRU adapter, load adapter_42 from CPU/disk
  3. Compute: y = W₀x + (α/r) · B₄₂ · A₄₂ · x
MetricSeparate instancesS-LoRA
GPU memory for 100 adapters100 × 14 GB = 1,400 GB14 GB + 4 GB pool = 18 GB
Adapter switch timeLoad full model (~seconds)Page in adapter (~ms)
Max concurrent adaptersLimited by GPU countThousands

3. Punica: Batched Multi-LoRA GEMM

When batching requests with different adapters, the naive approach is one GEMM per adapter:

Request 1 (adapter A): y₁ = W₀x₁ + B_A · A_A · x₁
Request 2 (adapter B): y₂ = W₀x₂ + B_B · A_B · x₂
Request 3 (adapter A): y₃ = W₀x₃ + B_A · A_A · x₃

Punica’s SGMV (Segmented Gather Matrix-Vector) kernel batches these into one operation:

Base: Y_base = W₀ · [x₁, x₂, x₃]           (one batched GEMM)
LoRA: Y_lora = SegmentedGatherMV(adapters, inputs, segments)  (one fused kernel)
Y = Y_base + Y_lora

Throughput improvement: 2-4× vs naive per-adapter GEMM for diverse adapter batches.

4. vLLM Multi-LoRA Support

from vllm import LLM, SamplingParams
from vllm.lora.request import LoRARequest

llm = LLM(
    model="meta-llama/Llama-2-7b-hf",
    enable_lora=True,
    max_lora_rank=16,
    max_loras=100,           # max concurrent adapters in memory
    max_cpu_loras=1000,      # max adapters on CPU (paged in on demand)
)

# Different requests use different adapters
output_1 = llm.generate("Hello", lora_request=LoRARequest("customer_a", 1, "path/to/adapter_a"))
output_2 = llm.generate("Bonjour", lora_request=LoRARequest("french", 2, "path/to/adapter_french"))

For managing adapter lifecycle and A/B testing, see Adapter Registry & Versioning.

Self-Check

  • Explain the memory waste problem with separate model instances per adapter
  • Describe S-LoRA’s unified paging approach
  • Explain Punica’s batched multi-LoRA GEMM optimization
  • Configure vLLM for multi-LoRA serving with adapter hot-swapping

References