Cluster Managers

GPU clusters need a manager that allocates hardware to workloads, handles failures, and enforces priorities. Two systems dominate: Slurm for training and Kubernetes for serving. Most organizations run both.

---

1. Slurm: The Training Standard

Slurm (Simple Linux Utility for Resource Management) is a job scheduler designed for HPC — submit a script, request resources, wait in queue, run.

Why Training Uses Slurm

Distributed training has a unique requirement: all GPUs must be allocated simultaneously (gang scheduling). If you request 64 GPUs and only 60 are free, the job waits rather than starting with a partial allocation. Slurm handles this natively.

Job Submission

#!/bin/bash
#SBATCH --job-name=llama-70b-pretrain
#SBATCH --partition=gpu-a100
#SBATCH --nodes=8
#SBATCH --ntasks-per-node=8
#SBATCH --gpus-per-node=8
#SBATCH --cpus-per-task=12
#SBATCH --mem=0                    # all memory on node
#SBATCH --time=72:00:00
#SBATCH --exclusive                # no sharing

# Environment
module load cuda/12.4 nccl/2.21
export MASTER_ADDR=$(scontrol show hostname $SLURM_NODELIST | head -1)
export MASTER_PORT=29500

# Launch distributed training
srun torchrun \
    --nproc_per_node=8 \
    --nnodes=$SLURM_NNODES \
    --node_rank=$SLURM_NODEID \
    --master_addr=$MASTER_ADDR \
    --master_port=$MASTER_PORT \
    train.py --config configs/llama-70b.yaml

Key Slurm Concepts

Concept	What it does	Example
Partition	Named group of nodes with shared properties	gpu-a100, gpu-h100, cpu-only
QOS (Quality of Service)	Priority level, time limits, resource limits	high-priority, normal, preemptible
Fair-share	Prevents one team from monopolizing the cluster	Team A used 60% this week → lower priority
Job arrays	Submit many similar jobs efficiently	Hyperparameter sweeps
Dependencies	Job B starts only after Job A completes	Pipeline: preprocess → train → evaluate

Slurm + Containers

Slurm doesn’t natively run containers. Use Pyxis + Enroot (NVIDIA) for container support:

#SBATCH --container-image=nvcr.io/nvidia/pytorch:24.10-py3
#SBATCH --container-mounts=/data:/data,/checkpoints:/checkpoints

srun python train.py  # runs inside the container

---

2. Kubernetes: The Serving Standard

Kubernetes orchestrates containerized workloads with auto-scaling, health checks, and rolling deployments — essential for serving.

GPU Support Stack

NVIDIA GPU Operator (installs everything automatically):
  ├── GPU Driver (kernel module)
  ├── CUDA Toolkit (container runtime)
  ├── NVIDIA Container Toolkit (GPU passthrough to containers)
  ├── GPU Device Plugin (exposes GPUs to K8s scheduler)
  ├── GPU Feature Discovery (labels nodes: gpu-model, memory, MIG)
  └── DCGM Exporter (GPU metrics for monitoring)

Deploying a GPU Workload

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama-70b
spec:
  replicas: 4
  selector:
    matchLabels:
      app: vllm
  template:
    metadata:
      labels:
        app: vllm
    spec:
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        args: ["--model", "meta-llama/Llama-3.1-70B", "--tensor-parallel-size", "4"]
        resources:
          limits:
            nvidia.com/gpu: 4     # request 4 GPUs
            memory: "200Gi"
          requests:
            nvidia.com/gpu: 4
            memory: "200Gi"
        ports:
        - containerPort: 8000
      nodeSelector:
        nvidia.com/gpu.product: "NVIDIA-H100-SXM"  # specific GPU type
      tolerations:
      - key: "nvidia.com/gpu"
        operator: "Exists"
        effect: "NoSchedule"

Kubernetes Advantages for Serving

Feature	How it helps serving
Horizontal Pod Autoscaler	Scale replicas based on queue depth or GPU utilization
Liveness/Readiness probes	Restart crashed pods, drain unhealthy ones
Rolling updates	Deploy new model version with zero downtime
Service mesh	Traffic splitting for canary deployments
Namespaces + RBAC	Isolate teams, control who deploys what

---

3. Hybrid: Running Both

Workload	Where it runs	Why
Pre-training (weeks-long)	Slurm	Gang scheduling, simplicity, --exclusive nodes
Fine-tuning (hours-days)	Slurm or K8s	Either works; K8s if you want auto-retry
RLHF (mixed train+infer)	Kubernetes	Need inference serving alongside training
Inference serving	Kubernetes	Auto-scaling, health checks, rolling updates
Batch evaluation	Slurm	Job arrays, simple queue management
Data preprocessing	Either	Slurm for HPC-style, K8s for cloud-native

The Typical Setup

On-prem cluster (DGX SuperPOD):
  └── Slurm manages all nodes
      ├── gpu-train partition (256 H100s for pre-training)
      ├── gpu-finetune partition (64 H100s for fine-tuning)
      └── gpu-eval partition (16 H100s for evaluation)

Cloud (AWS/GCP):
  └── Kubernetes manages all pods
      ├── vllm-serving namespace (auto-scaled inference)
      ├── batch-inference namespace (offline processing)
      └── dev namespace (experiments, notebooks)

For how GPUs are scheduled within these managers, see GPU Scheduling.

---

Self-Check

• Write a Slurm job script for 8-node distributed training with containers
• Deploy a GPU inference workload on Kubernetes with node selection
• Explain gang scheduling and why Slurm is preferred for training
• Design a hybrid Slurm + K8s setup for a team doing both training and serving
• List the NVIDIA GPU Operator components and what each does

---

References

• Slurm Documentation
• NVIDIA GPU Operator
• Pyxis Container Plugin for Slurm