Dynamic Allocation

Static GPU allocation wastes money — inference load varies 5× between peak and trough, training jobs don’t run 24/7, and spot instances offer 60-80% savings. Dynamic allocation adjusts GPU assignment in real-time.

---

1. Auto-Scaling Inference

Scaling Signals

Not all signals are equal for LLM inference:

Signal	Measures	Response time	Recommended?
GPU utilization	Compute usage	Lagging (already saturated when high)	⚠️ Too late
Request queue depth	Pending requests	Leading (predicts saturation)	✅ Best
TTFT P99	User experience	Direct SLA metric	✅ Best
KV cache utilization	Memory pressure	Leading (OOM before GPU saturates)	✅ Good
Throughput (tok/s)	System capacity	Lagging	⚠️ Supplementary

Kubernetes HPA Configuration

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: vllm-autoscaler
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: vllm-serving
  minReplicas: 2        # always keep 2 replicas warm
  maxReplicas: 16
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60   # wait 60s before scaling up
      policies:
      - type: Pods
        value: 2          # add max 2 pods at a time
        periodSeconds: 120
    scaleDown:
      stabilizationWindowSeconds: 300  # wait 5 min before scaling down
      policies:
      - type: Pods
        value: 1
        periodSeconds: 300
  metrics:
  - type: Pods
    pods:
      metric:
        name: vllm_num_requests_waiting  # custom DCGM/Prometheus metric
      target:
        type: AverageValue
        averageValue: "5"   # scale up when >5 requests queued per pod

Scale-up fast, scale-down slow: You want to respond quickly to load spikes (add pods in 60s) but avoid oscillation on temporary dips (wait 5 min before removing).

Cold Start Problem

New inference pods take 1-5 minutes to load the model:

Mitigation	How	Latency
Pre-cached model on NVMe	Model already on node disk	~30s (load to GPU)
Warm standby pods	Idle pods with model loaded, 0 traffic	~0s (route traffic immediately)
Predictive scaling	Scale up before peak (based on time-of-day pattern)	0s (already scaled)

---

2. Spot/Preemptible Instances

Cloud spot instances cost 60-80% less but can be reclaimed with 30s-2min warning:

Spot-Friendly Workloads

Workload	Spot viable?	Interruption strategy
Pre-training	✅	Checkpoint every 10 min, resume from last checkpoint
Fine-tuning	✅	Same — short runs, frequent checkpoints
Batch inference	✅	Retry failed batches on new instance
Online inference	❌	User-facing — can’t tolerate random interruption
Data preprocessing	✅	Stateless, easily retried

Spot Integration Architecture

Spot instance pool (70% of training GPUs):
  ├── On-demand fallback: 30% always-on GPUs
  ├── Checkpoint service: save every 10 min to S3
  ├── Spot interruption handler:
  │     When 2-min warning received:
  │       1. Save emergency checkpoint
  │       2. Gracefully exit training
  │       3. Re-queue job on available capacity
  └── Auto-scaling group: request new spots when capacity available

Cost Savings

Configuration	GPU cost/hour	Savings vs on-demand
AWS p5.48xlarge on-demand	~$98 (8×H100)	Baseline
AWS p5.48xlarge 1yr reserved	~$62	37%
AWS p5 spot (when available)	~$30-50	50-70%
Mix: 30% reserved + 70% spot	~$42 average	57%

---

3. Elastic Training

Resize a training job without restarting:

Hour 0:  48 GPUs available → start with DP=6, TP=8
Hour 3:  16 GPUs reclaimed (spot) → shrink to DP=4, TP=8 (32 GPUs)
Hour 5:  64 GPUs available → expand to DP=8, TP=8 (64 GPUs)

Requirements for Elastic Training

Requirement	Why	Implementation
Elastic launch (torchrun)	Adjust world_size at runtime	--nnodes=min:max
FSDP/ZeRO that reshards	Redistribute parameters when DP changes	PyTorch DCP
Frequent checkpoints	Survive resize without losing progress	Every 5-10 min
Batch size adjustment	Effective batch = micro_batch × DP × accum_steps	Adjust accum_steps when DP changes

For the cluster managers that implement these policies, see Cluster Managers.

---

Self-Check

• Choose auto-scaling signals for LLM inference (why queue depth > GPU utilization)
• Configure Kubernetes HPA with scale-up fast / scale-down slow behavior
• Design a spot instance strategy for pre-training (checkpoint + fallback)
• Calculate cost savings from a 30% reserved + 70% spot mix
• List the requirements for elastic training

---

References

• Kubernetes HPA
• AWS Spot Best Practices
• TorchElastic