Navigation
🏭 Platform & Infrastructure· 10 min read

Capacity Planning

Predicting GPU needs from traffic forecasts — sizing formulas, growth modeling, procurement lead times, and headroom strategy.

capacity-planninggpu-sizingdemand-forecastingprovisioningheadroom

Running out of GPUs means degraded service or lost revenue. Over-provisioning wastes $25-40K per idle GPU per year. Capacity planning predicts future demand and provisions the right amount of hardware — with enough headroom for spikes but not enough to bankrupt you.

1. The Sizing Formula

def calculate_gpus_needed(
    peak_requests_per_sec: float,
    avg_input_tokens: int,
    avg_output_tokens: int,
    target_ttft_p99_ms: float,
    target_tpot_p99_ms: float,
    model_name: str,
    gpu_type: str,
    quantization: str,
    headroom_pct: float = 0.25,
) -> int:
    """Calculate GPUs needed to serve traffic within SLO."""
    
    # Step 1: Per-GPU throughput (from benchmarks)
    per_gpu_throughput = BENCHMARK_DATA[model_name][gpu_type][quantization]
    # e.g., Llama-70B INT4 on H100: ~2500 output tokens/sec at batch=64
    
    # Step 2: Total token throughput needed
    total_output_tok_s = peak_requests_per_sec * avg_output_tokens
    total_prefill_tok_s = peak_requests_per_sec * avg_input_tokens
    
    # Step 3: GPUs for throughput
    gpus_for_throughput = math.ceil(total_output_tok_s / per_gpu_throughput.output_tok_s)
    
    # Step 4: GPUs for latency SLO (may need more GPUs at lower batch size)
    if per_gpu_throughput.p99_ttft_at_target_batch > target_ttft_p99_ms:
        # Need smaller batches → more GPUs
        gpus_for_latency = math.ceil(gpus_for_throughput * 
            per_gpu_throughput.p99_ttft_at_target_batch / target_ttft_p99_ms)
    else:
        gpus_for_latency = gpus_for_throughput
    
    # Step 5: GPUs for tensor parallelism
    tp_size = TP_REQUIREMENTS[model_name][quantization]  # e.g., 70B INT4 → TP=1, 70B BF16 → TP=4
    gpus_base = max(gpus_for_throughput, gpus_for_latency)
    gpus_with_tp = math.ceil(gpus_base / tp_size) * tp_size  # Round up to TP multiple
    
    # Step 6: Add headroom
    gpus_total = math.ceil(gpus_with_tp * (1 + headroom_pct))
    
    return gpus_total

Worked Example

# Customer support chatbot
result = calculate_gpus_needed(
    peak_requests_per_sec=100,
    avg_input_tokens=800,      # System prompt + conversation history
    avg_output_tokens=200,     # Typical response
    target_ttft_p99_ms=2000,   # 2s TTFT
    target_tpot_p99_ms=50,     # 50ms per token
    model_name="llama-70b",
    gpu_type="h100",
    quantization="int4",       # TP=1, ~2500 tok/s per GPU
    headroom_pct=0.25,
)
# total_output: 100 × 200 = 20,000 tok/s
# GPUs for throughput: ceil(20000/2500) = 8
# TP=1, so 8 GPUs
# With 25% headroom: 10 GPUs

2. Growth Modeling

Traffic doesn’t stay constant. Model growth patterns:

Linear Growth (Internal Tools)

# Monthly growth rate from historical data
monthly_growth_rate = 0.08  # 8% month-over-month
current_tok_s = 20_000
months_ahead = 6

projected = current_tok_s * (1 + monthly_growth_rate) ** months_ahead
# 20,000 × 1.08^6 = 31,737 tok/s in 6 months

Exponential Growth (Consumer Products)

# Doubling time model
doubling_months = 3  # Traffic doubles every 3 months
projected = current_tok_s * (2 ** (months_ahead / doubling_months))
# 20,000 × 2^(6/3) = 80,000 tok/s in 6 months

Seasonal Patterns

# Business hours pattern
traffic_profile = {
    "weekday_peak":    1.0,   # 9am-5pm baseline
    "weekday_evening": 0.3,   # 5pm-10pm
    "weekday_night":   0.05,  # 10pm-6am
    "weekend":         0.15,  # All day
}

# Peak capacity must handle weekday peak
# Overnight: run batch jobs on idle capacity

3. Procurement Timeline

GPU procurement has significant lead times:

SourceLead timeCostFlexibility
Cloud on-demand (AWS/GCP)Minutes$$$$ ($2-4/GPU-hr)Instant up/down
Cloud reserved (1yr)Days (contract)$$$ (40-60% discount)Committed
Cloud spot/preemptibleMinutes$$ (60-70% discount)Can be reclaimed
Colo + own hardware3-6 months$ (amortized)Fixed capacity
New data center build12-24 months$ (at scale)Very fixed

Capacity Procurement Strategy

Timeline approach:
  ├── 0-2 weeks:  Cloud on-demand (burst capacity)
  ├── 1-3 months: Cloud reserved (predictable baseline)
  ├── 3-6 months: Colo hardware (cost-optimized baseline)
  └── 6-24 months: Own facility (if >1000 GPUs)

Capacity layers:
  [Reserved (60% of peak)] + [Spot (20%)] + [On-demand (20% burst)]

4. Headroom Strategy

How much spare capacity to maintain:

ComponentHeadroomRationale
Peak traffic buffer20-30% above forecasted peakTraffic spikes are unpredictable
GPU failure reserve5-10% of fleetGPUs fail; need hot spares
Maintenance window10-15%Rolling updates, firmware patches
Experimentation5-10%A/B testing new models, eval runs

Total recommended headroom: 25-40% above steady-state peak demand.

def headroom_calculation(peak_gpus: int) -> dict:
    return {
        "peak_demand": peak_gpus,
        "traffic_buffer": int(peak_gpus * 0.25),
        "failure_reserve": int(peak_gpus * 0.05),
        "maintenance": int(peak_gpus * 0.10),
        "experimentation": int(peak_gpus * 0.05),
        "total_provisioned": int(peak_gpus * 1.35),  # Rounded
    }

# Example: 100 GPUs at peak
# provision: ~135 GPUs total

5. Capacity Dashboard

-- Key capacity planning queries

-- Current utilization vs capacity
SELECT 
    model,
    count(DISTINCT gpu_id) AS gpus_allocated,
    avg(gpu_utilization) AS avg_util,
    max(queue_depth) AS peak_queue,
    sum(tokens_per_sec) AS total_throughput
FROM gpu_metrics
WHERE timestamp > now() - interval '1 hour'
GROUP BY model;

-- Projection: when will we run out?
-- If growing 8%/month and currently at 75% utilization:
-- Months until 95%: ln(0.95/0.75) / ln(1.08) = 3.0 months
SELECT 
    current_utilization,
    growth_rate_monthly,
    ln(0.95 / current_utilization) / ln(1 + growth_rate_monthly) AS months_until_95pct
FROM capacity_summary;

Automated Capacity Alerts

# Alert 3 months before projected capacity exhaustion
- alert: CapacityExhaustionProjected
  expr: |
    predict_linear(gpu_utilization_avg[30d], 90 * 86400) > 0.95
  for: 1d
  labels:
    severity: warning
  annotations:
    summary: "GPU capacity projected to reach 95% in ~90 days"
    action: "Begin procurement for additional GPUs"

6. Right-Sizing Review

Regularly verify that provisioned capacity matches actual need:

async def right_sizing_review():
    """Weekly review of capacity allocation."""
    for model in get_all_models():
        metrics = await get_weekly_metrics(model)
        
        peak_util = metrics.peak_gpu_utilization
        avg_util = metrics.avg_gpu_utilization
        allocated_gpus = metrics.allocated_gpus
        
        if avg_util < 0.30 and peak_util < 0.60:
            recommendation = "DOWNSIZE"
            suggested = math.ceil(allocated_gpus * peak_util / 0.70)
        elif peak_util > 0.90:
            recommendation = "UPSIZE"
            suggested = math.ceil(allocated_gpus * 1.30)
        else:
            recommendation = "OK"
            suggested = allocated_gpus
        
        report.append({
            "model": model,
            "allocated": allocated_gpus,
            "avg_util": f"{avg_util:.0%}",
            "peak_util": f"{peak_util:.0%}",
            "recommendation": recommendation,
            "suggested_gpus": suggested,
        })
    
    return report

Self-Check

  • Calculate GPUs needed for 100 req/s of Llama-70B with TTFT <2s SLO
  • Model traffic growth for both linear (internal) and exponential (consumer) patterns
  • Design a procurement strategy combining reserved, spot, and on-demand
  • Calculate appropriate headroom for a 100-GPU fleet
  • Set up automated alerting for projected capacity exhaustion

References