GPU Virtualization — infra.bea.ai

A single H100 has 80 GB of memory and 989 TFLOPS — far more than many workloads need. GPU virtualization shares one physical GPU across multiple workloads, improving utilization from 20-30% to 70-90%.

1. The Utilization Problem

Typical GPU utilization without virtualization:

Workload	GPU memory used	Compute used	Waste
8B model inference (INT4)	4 GB / 80 GB	5%	95% idle
Dev notebook (experiment)	10 GB / 80 GB	2% (bursty)	95% idle most of the time
Embedding model	2 GB / 80 GB	15%	85% idle

These workloads could share a single GPU — if the isolation is sufficient.

2. MIG (Multi-Instance GPU)

MIG partitions the GPU in hardware — dedicated SMs, memory controllers, and L2 cache per partition. The strongest isolation available.

How MIG Works

H100 (80 GB, 132 SMs):
  └── Partition into instances:
      ├── Instance 0: 1g.10gb (1/7 GPU, 10 GB, ~19 SMs)
      ├── Instance 1: 1g.10gb
      ├── Instance 2: 1g.10gb
      ├── Instance 3: 1g.10gb
      ├── Instance 4: 1g.10gb
      ├── Instance 5: 1g.10gb
      └── Instance 6: 1g.10gb

  Or: mixed profiles
      ├── Instance 0: 3g.40gb (3/7 GPU, 40 GB) → 70B INT4 inference
      ├── Instance 1: 2g.20gb (2/7 GPU, 20 GB) → 13B model
      └── Instance 2: 2g.20gb (2/7 GPU, 20 GB) → embedding service

Configuration

# Enable MIG mode (requires GPU reset)
sudo nvidia-smi -i 0 -mig 1

# Create instances
sudo nvidia-smi mig -i 0 -cgi 9,14,14    # 3g.40gb + 2g.20gb + 2g.20gb
sudo nvidia-smi mig -i 0 -cci             # create compute instances

# List instances
nvidia-smi mig -i 0 -lgi
# +---+------+--------+------+
# | 0 | 9    | 3g.40gb| 40GB |
# | 1 | 14   | 2g.20gb| 20GB |
# | 2 | 14   | 2g.20gb| 20GB |
# +---+------+--------+------+

# Use in Kubernetes
# Each MIG instance appears as a separate GPU resource:
# nvidia.com/mig-3g.40gb: 1
# nvidia.com/mig-2g.20gb: 2

MIG Profile Options (H100)

Profile	SMs	Memory	Use case
1g.10gb	~19	10 GB	Small inference (1-3B INT4), embedding
2g.20gb	~38	20 GB	Medium inference (8B), fine-tuning experiments
3g.40gb	~57	40 GB	Larger inference (13-30B INT4)
4g.40gb	~76	40 GB	Same memory, more compute
7g.80gb	132	80 GB	Full GPU (no partitioning)

MIG Limitations

Requires GPU reset to enable/disable MIG mode
Fixed profiles — can’t create arbitrary sizes
Not all GPUs support MIG — A100, H100, H200, B200 only
No NVLink between MIG instances — each instance is isolated

3. MPS (Multi-Process Service)

MPS allows multiple CUDA processes to share a GPU simultaneously via time-slicing at the CUDA context level:

Without MPS:
  Process A runs → context switch → Process B runs → context switch
  (expensive context switches, ~ms each)

With MPS:
  Process A and B share the GPU simultaneously
  MPS server multiplexes their CUDA work onto the GPU
  (no context switches, overlapped execution)

# Start MPS server
nvidia-cuda-mps-control -d

# All CUDA processes on this GPU now go through MPS
# Set memory limit per process
echo "set_active_thread_percentage 50" | nvidia-cuda-mps-control

When MPS helps: Multiple small training jobs (hyperparameter search), each using <50% of the GPU. MPS lets them overlap instead of context-switching.

When MPS doesn’t help: If one process already uses 100% of compute, MPS adds overhead with no benefit.

Isolation: Weak — processes share the same memory space. A crash in one process can affect others. Not suitable for multi-tenant production.

4. Time-Slicing (Kubernetes)

The simplest approach — Kubernetes schedules multiple pods onto the same GPU, and NVIDIA’s time-slicing alternates between them:

# NVIDIA device plugin config for time-slicing
apiVersion: v1
kind: ConfigMap
metadata:
  name: nvidia-device-plugin
data:
  config.yaml: |
    version: v1
    sharing:
      timeSlicing:
        resources:
        - name: nvidia.com/gpu
          replicas: 4    # each GPU appears as 4 "virtual GPUs"

Now each H100 appears as 4 nvidia.com/gpu resources — 4 pods can each request 1 GPU and share the physical device.

Isolation: None — pods share all GPU memory. If one pod allocates too much, others OOM. Only suitable for development/non-critical workloads.

5. Comparison and Selection

	MIG	MPS	Time-Slicing
Isolation	Hardware (strong)	Weak (shared memory)	None
Memory partitioning	Hardware-enforced	Voluntary limits	None (shared)
Overhead	Near-zero	Low	Medium (context switch)
Configuration	GPU reset required	Start MPS daemon	K8s config only
Failure isolation	One instance crashes, others fine	Crash can propagate	Crash can propagate
Best for	Multi-tenant production	Multiple small training jobs	Dev/test only

Decision: If you’re running production multi-tenant inference → MIG. If you’re maximizing a dev cluster → time-slicing. MPS is niche (multiple small training experiments).

For dynamic scaling of GPU resources, see Dynamic Allocation.

Self-Check

Explain the GPU utilization problem that virtualization solves
Configure MIG on an H100 with mixed profiles (3g.40gb + 2×2g.20gb)
Compare MIG vs MPS vs time-slicing on isolation, overhead, and use case
Explain why MIG requires a GPU reset and time-slicing doesn’t
Choose a virtualization approach for: (a) multi-tenant inference, (b) dev cluster