GPU Isolation

When multiple tenants share GPU infrastructure, isolation prevents one tenant from affecting another — whether through resource contention (noisy neighbor), data leakage (side-channel), or fault propagation (one crash kills all). GPU isolation is harder than CPU isolation because GPUs weren’t designed for multi-tenancy.

---

1. The Isolation Problem

Without isolation, tenants sharing a GPU can:

Threat	Mechanism	Impact
Resource contention	One tenant’s large batch starves others	Latency spikes for neighbors
Memory snooping	Read uncleared GPU memory from previous tenant	Data leakage
Cache pollution	Large working set evicts neighbor’s cached data	Performance degradation
Fault propagation	CUDA error in one context crashes the GPU	All tenants lose service
Timing attacks	Measure execution time to infer model/data	Information leakage

---

2. MIG: Hardware Partitioning

Multi-Instance GPU (A100, H100, B200) creates physically isolated GPU instances:

# Enable MIG mode
sudo nvidia-smi -i 0 -mig 1

# Create instances on H100 (80GB)
# 7 × 1g.10gb — 7 small instances (10GB each)
sudo nvidia-smi mig -i 0 -cgi 19,19,19,19,19,19,19 -C

# Or: 3g.40gb + 2×2g.20gb — mixed sizes
sudo nvidia-smi mig -i 0 -cgi 9,14,14 -C

# List instances
nvidia-smi mig -lgi

MIG Profiles (H100 80GB)

Profile	SMs	Memory	Compute	Use case
1g.10gb	1/7 (16 SMs)	10 GB	~14%	Small embedding models
2g.20gb	2/7 (32 SMs)	20 GB	~28%	8B models (INT4)
3g.40gb	3/7 (48 SMs)	40 GB	~43%	70B models (INT4)
4g.40gb	4/7 (64 SMs)	40 GB	~57%	70B with more compute
7g.80gb	7/7 (132 SMs)	80 GB	100%	Full GPU (no MIG)

MIG Isolation Guarantees

Property	MIG guarantee
Memory isolation	Hardware-enforced — separate memory controllers
Compute isolation	Hardware-enforced — dedicated SMs per instance
L2 cache	Partitioned — each instance gets its own slice
Error isolation	Fault-contained — ECC error in one instance doesn’t affect others
Performance isolation	Strong — one instance can’t starve another

# Kubernetes: assign MIG instances to pods
resources:
  limits:
    nvidia.com/mig-2g.20gb: 1  # Request a specific MIG profile

For MIG configuration details, see GPU Virtualization.

---

3. MPS: Software Time-Sharing

CUDA Multi-Process Service allows concurrent GPU sharing without hardware partitioning:

# Start MPS daemon
export CUDA_MPS_PIPE_DIRECTORY=/tmp/nvidia-mps
export CUDA_MPS_LOG_DIRECTORY=/tmp/nvidia-log
nvidia-cuda-mps-control -d

MPS vs MIG

Property	MPS	MIG
Concurrent execution	Yes (shared SMs)	Yes (dedicated SMs)
Memory isolation	None (shared address space)	Hardware-enforced
Compute isolation	None (best-effort sharing)	Hardware-enforced
Fault isolation	None (one crash kills all)	Isolated
Number of tenants	Up to 48 (Hopper)	Up to 7
Performance overhead	~0%	~0%
GPU support	All NVIDIA GPUs	A100+ only
Best for	Trusted workloads (internal)	Multi-tenant (external)

Security recommendation: Use MIG for external/untrusted tenants. Use MPS only when tenants are trusted (same organization, different teams).

---

4. Kubernetes GPU Quotas

Enforce GPU allocation limits at the namespace level:

# ResourceQuota: limit GPUs per team namespace
apiVersion: v1
kind: ResourceQuota
metadata:
  name: gpu-quota
  namespace: team-alpha
spec:
  hard:
    nvidia.com/gpu: "8"           # Max 8 GPUs for this team
    requests.nvidia.com/gpu: "8"

---
# LimitRange: per-pod GPU limits
apiVersion: v1
kind: LimitRange
metadata:
  name: gpu-limits
  namespace: team-alpha
spec:
  limits:
  - type: Container
    max:
      nvidia.com/gpu: "4"         # No single pod can take more than 4 GPUs
    default:
      nvidia.com/gpu: "1"

Topology-Aware Scheduling

Ensure multi-GPU pods get GPUs on the same NVLink domain:

# Use topology manager for GPU placement
# kubelet configuration
topologyManagerPolicy: "best-effort"  # or "restricted" for strict
topologyManagerScope: "pod"

---

5. Memory Clearing

GPU memory is not automatically cleared between tenants:

# PROBLEM: After tenant A's model is unloaded, memory may still contain weights
# Tenant B could theoretically read stale data

# SOLUTION: Zero GPU memory between tenant switches
import torch

def secure_gpu_clear(device_id: int):
    """Clear all GPU memory to prevent data leakage."""
    torch.cuda.set_device(device_id)
    
    # Free all cached allocations
    torch.cuda.empty_cache()
    
    # Allocate and zero-fill entire GPU memory
    total_mem = torch.cuda.get_device_properties(device_id).total_memory
    usable = int(total_mem * 0.95)
    try:
        zeroed = torch.zeros(usable // 4, dtype=torch.float32, device=f'cuda:{device_id}')
        del zeroed
    except RuntimeError:
        pass  # Best effort
    
    torch.cuda.empty_cache()
    torch.cuda.synchronize()

With MIG: Memory isolation is hardware-enforced — clearing is not necessary between MIG instances on the same GPU (they cannot access each other’s memory).

---

6. Choosing an Isolation Strategy

Is this multi-tenant (different customers)?
├── Yes
│   ├── Have A100/H100+ GPUs?
│   │   ├── Yes → MIG (strongest isolation)
│   │   └── No → Dedicated GPUs per tenant (no sharing)
│   └── Need more than 7 tenants per GPU?
│       └── Use dedicated GPUs + time-sharing with memory clearing
└── No (internal teams)
    ├── Need fault isolation?
    │   ├── Yes → MIG or dedicated GPUs
    │   └── No → MPS (best utilization)
    └── Simple GPU quotas sufficient?
        └── Kubernetes ResourceQuota + LimitRange

---

Self-Check

• Compare MIG and MPS on memory, compute, and fault isolation guarantees
• Configure MIG profiles on H100 for mixed workloads (70B + 8B + embedding)
• Set up Kubernetes GPU quotas and per-pod limits
• Implement GPU memory clearing between tenant switches
• Choose an isolation strategy for multi-tenant vs internal-team scenarios

---

References

• NVIDIA MIG User Guide
• NVIDIA MPS Documentation
• For detailed MIG/MPS/time-slicing comparison, see GPU Virtualization