GPU Scheduling

GPU scheduling determines who gets which GPUs, when, and for how long. The policies differ dramatically between training (all-or-nothing) and inference (fractional, elastic) — getting this wrong means either wasted GPUs or stalled jobs.

---

1. The Two Worlds

Property	Training scheduling	Inference scheduling
Allocation unit	All N GPUs at once (gang)	1-8 GPUs per pod (fractional)
Duration	Hours to weeks	Indefinite (always running)
Failure handling	Checkpoint + restart	Health check + restart pod
Scaling	Fixed size for duration	Auto-scale with load
Preemption	Acceptable (with checkpoint)	Avoid (user-facing)

---

2. Gang Scheduling

Distributed training needs ALL requested GPUs simultaneously. 63 out of 64 is useless — the job can’t start.

Cluster: 128 GPUs total

Job A (submitted first): needs 64 GPUs → starts immediately (64/128 used)
Job B (submitted second): needs 64 GPUs → starts immediately (128/128 used)
Job C (submitted third): needs 96 GPUs → WAITS (only free after A or B finish)

Without gang scheduling:
  Job C might get 64 GPUs → can't use them → deadlock
  
With gang scheduling (Slurm default):
  Job C waits until 96 GPUs are simultaneously free → starts atomically

Backfill Scheduling

While Job C waits for 96 GPUs, smaller jobs can backfill the gap:

Job C waiting for 96 GPUs (ETA: 12 hours when Job A finishes)
Job D (needs 8 GPUs, runs 2 hours): can backfill! Fits in the gap.
Job E (needs 32 GPUs, runs 1 hour): can backfill!

Backfill condition: Job D/E must FINISH before Job C needs those GPUs
→ Requires accurate time estimates (--time in Slurm)

Rule: Always set accurate --time limits — Slurm uses them for backfill decisions. Overestimate = blocks backfill. Underestimate = job gets killed.

---

3. Bin Packing for Inference

Inference pods need 1-8 GPUs each. Pack them tightly to minimize wasted GPUs:

8-GPU node:
  Bin packing:
    Pod A: 4 GPUs (70B model, TP=4)
    Pod B: 2 GPUs (13B model, TP=2)
    Pod C: 1 GPU (8B model)
    Pod D: 1 GPU (embedding model)
    → 100% utilized, 0 GPUs wasted

  Spread (anti-affinity):
    Pod A: 4 GPUs on Node 0
    Pod B: 2 GPUs on Node 1
    Pod C: 1 GPU on Node 2
    → Fault tolerant but 11 GPUs idle across 3 nodes

# Kubernetes: prefer bin packing (pack onto fewer nodes)
apiVersion: v1
kind: Pod
spec:
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        preference:
          matchExpressions:
          - key: gpu-utilization
            operator: Lt
            values: ["80"]  # prefer nodes with available GPUs

---

4. Priority and Preemption

Not all workloads are equal. Production serving must never be preempted; dev experiments can be:

Tier	Priority	Preemptible	Examples
P0: Production	Highest	Never	Live inference serving
P1: Training (scheduled)	High	Only by P0	Pre-training, scheduled fine-tuning
P2: Training (ad-hoc)	Medium	By P0, P1	Experiment runs, hyperparameter search
P3: Development	Low	By anyone	Notebooks, debugging, exploration

Preemption in Practice

P3 job running on 8 GPUs (dev experiment)
P1 job submitted (scheduled training, needs 64 GPUs, only 56 free)

→ Scheduler preempts the P3 job (checkpoints, then frees 8 GPUs)
→ P1 job now has 64 GPUs → starts
→ P3 job re-queued, waits for resources

---

5. Fair-Share Scheduling

Prevent one team from monopolizing the cluster:

Cluster: 256 GPUs
Team quotas: Team A = 40%, Team B = 35%, Team C = 25%

This week:
  Team A used 60% (over quota) → priority reduced next week
  Team B used 20% (under quota) → priority boosted next week
  Team C used 20% (under quota) → priority boosted

Fair-share adjusts priorities dynamically so everyone gets their share over time.

# Slurm fair-share configuration
sacctmgr add account team-a fairshare=40
sacctmgr add account team-b fairshare=35
sacctmgr add account team-c fairshare=25

# Check current fair-share status
sshare -l -a

For sharing a single GPU across workloads, see GPU Virtualization. For dynamic scaling, see Dynamic Allocation.

---

Self-Check

• Explain gang scheduling and why training requires it
• Explain backfill scheduling and why accurate time estimates matter
• Compare bin packing vs spread for inference pods
• Design a 4-tier priority scheme with preemption rules
• Configure Slurm fair-share for 3 teams with different quotas

---

References

• Slurm Priority/Multifactor
• Slurm Backfill Scheduling
• Kubernetes Priority and Preemption