Dynamic Resource Allocation
Async vs sync policy updates, elastic allocation between rollout and training phases, and maximizing GPU utilization in RLHF.
RLHF alternates between inference (rollout generation) and training (policy gradient updates). Maximizing GPU utilization requires intelligent allocation between these phases.
1. The Utilization Problem
| Phase | GPU operation | Arithmetic intensity | Typical utilization |
|---|---|---|---|
| Rollout generation | Autoregressive decode | Low (memory-bound) | 30-50% MFU |
| Reward scoring | Forward pass | Moderate | 40-60% MFU |
| Policy training | Forward + backward | High (compute-bound) | 50-70% MFU |
With time-multiplexing (all GPUs do one phase, then the next), GPUs alternate between low-utilization inference and high-utilization training — average utilization is poor.
2. Synchronous vs Asynchronous Updates
Synchronous (Standard)
Step 1: All GPUs generate rollouts → All GPUs train → All GPUs generate → ...Pros: Simple, deterministic. Cons: GPUs idle during phase transitions.
Asynchronous
GPUs 0-3: Continuously generate rollouts with policy v_k
GPUs 4-7: Train on rollouts from v_{k-1}, producing v_{k+1}
When training finishes: swap — 4-7 generate with v_{k+1}, 0-3 train on new rolloutsPros: Higher utilization, pipeline always running. Cons: Training on slightly stale rollouts (from policy not ).
Staleness is usually acceptable: policy changes slowly between iterations, so rollouts from are close enough. PPO’s clipping constraint naturally handles this — it rejects updates from too-different distributions.
3. Elastic Allocation
Adjust the rollout/training GPU split based on which phase is the bottleneck:
def allocate_gpus(total_gpus, rollout_time, training_time):
"""Allocate GPUs to balance rollout and training time."""
# Ideal: rollout_time/rollout_gpus ≈ training_time/training_gpus
ratio = rollout_time / (rollout_time + training_time)
rollout_gpus = max(1, round(total_gpus * ratio))
training_gpus = total_gpus - rollout_gpus
return rollout_gpus, training_gpus| Scenario | Rollout GPUs | Training GPUs | Rationale |
|---|---|---|---|
| Long generation (512 tokens) | 75% | 25% | Generation is bottleneck |
| Short generation (32 tokens) | 25% | 75% | Training is bottleneck |
| Balanced | 50% | 50% | Equal time in each phase |
Frameworks Supporting Dynamic Allocation
| Framework | Allocation strategy | Notes |
|---|---|---|
| OpenRLHF | Separate Ray actors for rollout/training | Manual split, easy to configure |
| DeepSpeed-Chat | Hybrid Engine (same GPUs) | Switches modes, no split needed |
| ReaLHF | Parameter reallocation between workers | Optimizes GPU assignment per phase |
| veRL (Volcano Engine) | Flexible worker placement | Production-grade elastic allocation |
For the overall RLHF pipeline architecture, see RLHF/GRPO Pipeline.
Self-Check
- Explain why RLHF has lower GPU utilization than standard training
- Compare synchronous vs asynchronous policy updates and the staleness tradeoff
- Calculate optimal GPU split for a given rollout and training time
- Choose between OpenRLHF, DeepSpeed-Chat, and veRL for a given scenario