Multi-Model GPU Sharing

Running one model per GPU is simple but wasteful. An 8B model uses 16 GB of an 80 GB GPU — the remaining 64 GB sits idle. Multi-model GPU sharing co-locates models to maximize utilization, but introduces scheduling complexity and performance interference.

---

1. The Utilization Problem

A typical inference fleet has models of different sizes:

Model	Memory (INT4)	Typical GPU util	Wasted capacity on 80GB GPU
Llama-8B	6 GB	20-40%	74 GB memory, 60-80% compute
Embedding (e5-large)	2 GB	10-30%	78 GB memory, 70-90% compute
Reranker (BGE)	1 GB	5-15%	79 GB memory, 85-95% compute
Whisper-large	3 GB	15-25%	77 GB memory, 75-85% compute

Giving each model a dedicated GPU means 4 GPUs at 15-30% utilization. Co-locating all four on one GPU saves 3 GPUs:

Before (dedicated):         After (co-located):
GPU 0: [8B_______________]  GPU 0: [8B|Embed|Rerank|Whisper]
GPU 1: [Embed____________]    → 12 GB used, 68 GB free
GPU 2: [Rerank___________]    → Still room for more models
GPU 3: [Whisper__________]
  4 GPUs, ~20% avg util       1 GPU, ~60% avg util

---

2. Memory Partitioning Strategies

Static Partitioning

Pre-allocate a fixed memory budget per model:

# Static memory allocation
GPU_MEMORY = 80  # GB (H100)

allocations = {
    "llama-8b":    {"memory_gb": 20, "priority": "high"},   # 6GB weights + KV cache
    "embedding":   {"memory_gb": 8,  "priority": "medium"}, # 2GB weights + batch buffer
    "reranker":    {"memory_gb": 8,  "priority": "medium"}, # 1GB weights + batch buffer
    "whisper":     {"memory_gb": 10, "priority": "low"},    # 3GB weights + audio buffer
    # Reserved:     34 GB for CUDA overhead + growth
}

Pros: Predictable, no interference between models’ memory.
Cons: Rigid — if the 8B model gets a traffic spike, it can’t use Whisper’s idle memory.

Dynamic Partitioning

Load and unload models based on demand:

class DynamicModelManager:
    def __init__(self, gpu_memory_gb=80, reserved_gb=10):
        self.available = gpu_memory_gb - reserved_gb  # 70 GB usable
        self.loaded_models = {}  # name → (model, memory_gb, last_used)
    
    def get_model(self, name: str):
        if name in self.loaded_models:
            self.loaded_models[name].last_used = time.time()
            return self.loaded_models[name].model
        
        needed_gb = MODEL_SIZES[name]
        
        # Evict LRU models until enough memory is free
        while self.free_memory() < needed_gb:
            self._evict_lru()
        
        model = load_model(name)
        self.loaded_models[name] = ModelEntry(model, needed_gb, time.time())
        return model
    
    def _evict_lru(self):
        """Evict least-recently-used model."""
        lru = min(self.loaded_models.values(), key=lambda m: m.last_used)
        lru.model.unload()  # Free GPU memory
        del self.loaded_models[lru.name]

Pros: Adapts to traffic patterns — popular models stay loaded, rare ones get evicted.
Cons: Cold-start latency when loading evicted models (seconds to minutes for large models).

Comparison

Strategy	Memory efficiency	Latency predictability	Complexity
Static	Low (pre-allocated waste)	High (all models always loaded)	Low
Dynamic (LRU)	High (only active models loaded)	Low (cold-start on eviction)	Medium
Hybrid (pin hot + LRU cold)	High	Medium	Medium

Production recommendation: Pin high-traffic models (always loaded) + LRU for the rest:

PINNED_MODELS = ["llama-8b", "embedding"]   # always loaded
LRU_MODELS = ["whisper", "reranker", "code-model"]  # loaded on demand

---

3. GPU Sharing Mechanisms

CUDA MPS (Multi-Process Service)

MPS allows multiple processes to share GPU compute concurrently:

# Start MPS daemon
nvidia-cuda-mps-control -d

# Now multiple processes can share the GPU
CUDA_VISIBLE_DEVICES=0 python serve_8b.py &
CUDA_VISIBLE_DEVICES=0 python serve_embedding.py &
CUDA_VISIBLE_DEVICES=0 python serve_reranker.py &

Without MPS, CUDA time-slices between processes (one runs, others wait). With MPS, kernels from different processes can execute simultaneously on different SMs.

Feature	Time-slicing	MPS	MIG
Concurrent execution	No (round-robin)	Yes (shared SMs)	Yes (dedicated SMs)
Memory isolation	Shared (can OOM each other)	Shared	Isolated (hardware)
Fault isolation	No	No	Yes
Max processes	Unlimited	48 (Hopper)	7 (A100/H100)
Overhead	Context switch ~1ms	~0%	~0%
Best for	Dev/test	Production co-location	Multi-tenant isolation

For detailed MIG/MPS/time-slicing comparison, see GPU Virtualization.

MIG for Multi-Model (Hardware Isolation)

MIG creates isolated GPU instances with dedicated memory and compute:

# Create MIG instances on H100 (80GB)
# Profile: 3g.40gb (3/7 of compute, 40GB memory) + 2× 2g.20gb
nvidia-smi mig -cgi 9,14,14 -C

# Instance 0 (3g.40gb): Run 70B INT4 model (needs ~40GB)
# Instance 1 (2g.20gb): Run 8B model + embedding
# Instance 2 (2g.20gb): Run reranker + whisper

When MIG over MPS: When you need fault isolation (one model crashing can’t kill others) or performance isolation (one model’s bursty traffic can’t starve others).

---

4. Scheduling Co-Located Models

When multiple models share a GPU, a scheduler must decide which requests to process:

class CoLocationScheduler:
    def __init__(self, models: dict, gpu_compute_budget_ms=1000):
        self.queues = {name: deque() for name in models}
        self.priorities = {name: model.priority for name, model in models.items()}
        # Per-model compute budget per scheduling window
        self.budgets = self._allocate_budgets(gpu_compute_budget_ms)
    
    def _allocate_budgets(self, total_ms):
        """Allocate compute time proportional to priority."""
        total_priority = sum(self.priorities.values())
        return {
            name: total_ms * (p / total_priority)
            for name, p in self.priorities.items()
        }
    
    def schedule_next(self):
        """Pick next request respecting priorities and budgets."""
        # Strict priority: high-priority models always go first if queue non-empty
        for name in sorted(self.queues, key=lambda n: -self.priorities[n]):
            if self.queues[name] and self.budgets[name] > 0:
                request = self.queues[name].popleft()
                return name, request
        
        # All budgets exhausted — reset for next window
        self._reset_budgets()
        return self.schedule_next()

Interference Management

Co-located models compete for shared resources:

Resource	Interference type	Mitigation
GPU compute (SMs)	Bursty model starves others	Priority scheduling, compute budgets
GPU memory bandwidth	High-throughput model saturates HBM	Stagger batch processing
L2 cache	One model’s working set evicts another’s	Smaller batch sizes, cache partitioning (Hopper)
PCIe bandwidth	CPU↔GPU transfers contend	Batch data transfers, pin memory

Practical rule: Co-locate complementary workloads:

• ✅ LLM (compute-heavy prefill) + embedding (memory-bandwidth-heavy) — use different bottleneck resources
• ❌ Two LLMs both doing prefill — both want all SMs simultaneously

---

5. Monitoring Co-Located Models

Track per-model metrics even when sharing a GPU:

# Per-model metrics (not just per-GPU)
metrics = {
    "model_name": "llama-8b",
    "gpu_id": 0,
    
    # Latency — detect interference
    "p50_latency_ms": 45,
    "p99_latency_ms": 120,   # Watch for spikes from co-tenant interference
    
    # Throughput — detect starvation
    "tokens_per_second": 850,
    "queue_depth": 3,         # Growing queue = model is being starved
    
    # Memory — detect leaks
    "memory_allocated_gb": 18.5,
    "memory_peak_gb": 22.1,
}

# Alert if P99 latency exceeds isolation threshold
# (compare against dedicated-GPU baseline)
INTERFERENCE_THRESHOLD = 1.5  # 50% latency increase = too much interference
if p99_collocated / p99_dedicated > INTERFERENCE_THRESHOLD:
    alert("Model experiencing excessive co-location interference")

---

Self-Check

• Calculate memory savings from co-locating 4 small models on one GPU vs dedicated GPUs
• Compare static vs dynamic memory partitioning tradeoffs
• Explain when to use MPS vs MIG for multi-model serving
• Design a scheduling policy for co-located models with different priorities
• Describe how to detect and mitigate co-location interference

---

References

• NVIDIA Multi-Process Service (MPS) Documentation
• NVIDIA Multi-Instance GPU (MIG) User Guide
• AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving (OSDI 2023)