Multi-Model Hosting

Production AI platforms serve multiple models — different sizes, different fine-tunes, different modalities. The challenge: GPU memory is finite, model loading takes seconds to minutes, and traffic patterns shift throughout the day. Multi-model hosting must decide which models to keep loaded, when to swap, and how to minimize cold-start impact.

---

1. The Hosting Problem

A typical platform serves:

Model	Size (INT4)	Usage pattern	Priority
Llama-70B (chat)	40 GB	Constant, high traffic	Critical
Llama-8B (simple)	6 GB	Constant, medium traffic	High
Embedding (bge-large)	1.3 GB	Bursty (RAG queries)	High
Code model (32B)	18 GB	Business hours only	Medium
Vision model (LLaVA)	14 GB	Sporadic	Low
Whisper-large	3 GB	Batch + sporadic	Low
5× LoRA adapters	0.5 GB each	Variable per tenant	Medium

Total if all loaded: ~85 GB — fits on one H100 (80 GB) only with tight memory management. But not all models are needed simultaneously.

---

2. Model Loading/Unloading Policies

LRU (Least Recently Used)

class LRUModelManager:
    """Keep recently-used models loaded, evict stale ones."""
    
    def __init__(self, gpu_memory_gb=80, reserved_gb=10):
        self.available_gb = gpu_memory_gb - reserved_gb
        self.loaded = OrderedDict()  # model_name → ModelEntry (LRU order)
    
    async def get_model(self, name: str) -> Model:
        if name in self.loaded:
            self.loaded.move_to_end(name)  # Mark as recently used
            return self.loaded[name].model
        
        # Need to load — evict if necessary
        needed_gb = MODEL_SIZES[name]
        while self._used_memory() + needed_gb > self.available_gb:
            await self._evict_oldest()
        
        model = await self._load_model(name)
        self.loaded[name] = ModelEntry(model=model, size_gb=needed_gb, loaded_at=time.time())
        return model
    
    async def _evict_oldest(self):
        name, entry = self.loaded.popitem(last=False)  # Remove LRU
        await entry.model.unload()
        logger.info(f"Evicted {name} ({entry.size_gb} GB freed)")

Priority-Based with Pinning

Pin critical models (always loaded) and use LRU for the rest:

class PriorityModelManager:
    PINNED = ["llama-70b", "llama-8b", "embedding"]  # Always loaded
    
    async def _evict_oldest(self):
        """Evict LRU model, but never evict pinned models."""
        for name in self.loaded:
            if name not in self.PINNED:
                entry = self.loaded.pop(name)
                await entry.model.unload()
                return
        raise MemoryError("All loaded models are pinned — cannot evict")

Traffic-Predictive Loading

Pre-load models before traffic arrives:

class PredictiveModelManager:
    SCHEDULE = {
        "08:00": ["code-model"],      # Engineers start work
        "09:00": ["vision-model"],     # Design team meetings
        "18:00": [],                    # Unload work-hour models
    }
    
    async def check_schedule(self):
        """Pre-load models based on time-of-day patterns."""
        current = datetime.now().strftime("%H:%M")
        if current in self.SCHEDULE:
            for model_name in self.SCHEDULE[current]:
                if model_name not in self.loaded:
                    await self.get_model(model_name)

---

3. Cold-Start Mitigation

Model loading is slow — the dominant component is reading weights from storage:

Model size	Storage → GPU load time	Impact
1 GB (embedding)	0.5-2s	Acceptable
8 GB (8B INT4)	3-8s	Noticeable
40 GB (70B INT4)	15-60s	Unacceptable for interactive
140 GB (70B BF16)	45-120s	Major incident if cold

Mitigation Strategies

1. Keep hot models always loaded (most important):

# Never evict models with traffic in the last hour
MIN_KEEPALIVE = 3600  # seconds

2. Use quantized fallback during loading:

async def get_model_with_fallback(self, name: str) -> Model:
    if name in self.loaded:
        return self.loaded[name].model
    
    # Serve from smaller quantized version while full model loads
    fallback = self.get_fallback(name)  # e.g., 70B-INT4 while 70B-BF16 loads
    if fallback:
        # Start loading full model in background
        asyncio.create_task(self._load_model(name))
        return fallback
    
    # No fallback — must wait
    return await self._load_model(name)

3. Pre-warm from local NVMe (instead of network storage):

# Stage model weights on local NVMe for fast loading
# NVMe: 3-7 GB/s read → 40 GB model loads in 6-13s
# Network (S3): 1-2 GB/s → 40 GB model loads in 20-40s
LOCAL_CACHE = "/nvme/model-cache/"

4. Tensor parallel with staged loading:

# Load model shards in parallel across GPUs
# 4 GPUs × 3.5 GB/s each from NVMe = 14 GB/s effective
# 40 GB model → ~3s load time with TP=4

---

4. Memory Management

Memory Accounting

class GPUMemoryManager:
    def __init__(self, total_gb=80):
        self.total = total_gb
        self.allocations = {
            "cuda_overhead": 2.0,        # CUDA runtime, driver
            "kv_cache_pool": 0.0,        # Dynamic — vLLM manages this
            "model_weights": {},          # model_name → GB
        }
    
    def available_for_models(self) -> float:
        used = (self.allocations["cuda_overhead"] + 
                sum(self.allocations["model_weights"].values()))
        return self.total - used
    
    def available_for_kv_cache(self) -> float:
        """Memory left for KV cache after all models loaded."""
        model_total = sum(self.allocations["model_weights"].values())
        return self.total - self.allocations["cuda_overhead"] - model_total

Fragmentation

Repeated load/unload cycles fragment GPU memory. Mitigate with:

# 1. Use CUDA memory pools (PyTorch)
torch.cuda.memory.set_per_process_memory_fraction(0.95)

# 2. Periodic defragmentation (requires brief pause)
async def defragment():
    """Unload all models, clear cache, reload."""
    for model in list(self.loaded.values()):
        await model.unload()
    torch.cuda.empty_cache()
    # Reload pinned models
    for name in self.PINNED:
        await self.get_model(name)

---

5. Scheduling Across Multiple GPUs

With a GPU pool, decide where to place each model:

class ModelPlacement:
    """Decide which GPU(s) each model runs on."""
    
    def plan(self, models: list[ModelSpec], gpus: list[GPU]) -> dict:
        placement = {}
        
        for model in sorted(models, key=lambda m: -m.priority):
            if model.tp_size > 1:
                # Large model needs multiple GPUs (tensor parallel)
                gpu_group = self._find_connected_gpus(gpus, model.tp_size)
                if gpu_group:
                    placement[model.name] = gpu_group
                    for gpu in gpu_group:
                        gpu.allocated += model.memory_per_gpu
            else:
                # Small model — find GPU with most free memory
                best_gpu = max(gpus, key=lambda g: g.free_memory())
                if best_gpu.free_memory() >= model.total_memory:
                    placement[model.name] = [best_gpu]
                    best_gpu.allocated += model.total_memory
        
        return placement
    
    def _find_connected_gpus(self, gpus, count):
        """Find GPUs connected via NVLink for TP."""
        # Prefer GPUs on the same NVLink domain
        for group in nvlink_groups(gpus):
            free_in_group = [g for g in group if g.free_memory() > 0]
            if len(free_in_group) >= count:
                return free_in_group[:count]
        return None

---

6. Monitoring

hosting_metrics = {
    # Model inventory
    "models_loaded": len(loaded_models),
    "gpu_memory_used_pct": used / total,
    "gpu_memory_for_kv_cache_gb": kv_available,
    
    # Cold start tracking
    "cold_starts_last_hour": cold_start_count,
    "avg_load_time_sec": avg_load_duration,
    "max_load_time_sec": max_load_duration,
    
    # Eviction tracking
    "evictions_last_hour": eviction_count,
    "evicted_then_reloaded": reload_after_evict,  # Thrashing indicator
    
    # Per-model
    "per_model_requests": {"llama-70b": 5000, "embedding": 12000},
    "per_model_memory_gb": {"llama-70b": 40, "embedding": 1.3},
}

# Alert on thrashing: model evicted then reloaded within 5 minutes
if evicted_then_reloaded > 3:
    alert("Model thrashing detected — increase GPU memory or reduce model count")

---

Self-Check

• Design an LRU model manager with pinning for critical models
• Calculate cold-start time for different model sizes from NVMe vs network
• Implement a fallback strategy that serves a smaller model during loading
• Plan model placement across a multi-GPU pool with tensor parallelism
• Set up monitoring to detect model thrashing (evict-then-reload cycles)

---

References

• NVIDIA Triton Inference Server: Model Management
• AlpaServe: Statistical Multiplexing with Model Parallelism
• For GPU memory sharing, see Multi-Model GPU Sharing
• For model format considerations, see Model Formats