Demand Forecasting

Demand forecasting predicts future GPU needs so you can procure capacity before you need it — without over-buying. The challenge: AI traffic patterns are volatile, new features cause step-function jumps, and GPU procurement has 3-6 month lead times.

---

1. What to Forecast

Metric	Why it matters	Forecasting method
Daily token volume	Determines GPU fleet size	Time series + trend
Peak concurrent requests	Determines minimum online capacity	Peak-to-average ratio
Average prompt length	Affects prefill cost	Distribution tracking
Average completion length	Affects decode cost	Distribution tracking
New model adoption	New models need new capacity	Feature launch planning

---

2. Traffic Patterns

Daily Pattern (Business SaaS)

Tokens/sec profile:
  6am:  200  ▁
  9am:  800  ▃  (work starts)
 12pm: 1200  ▅  (peak — lunch + meetings)
  3pm: 1000  ▄
  6pm:  500  ▂  (work ends)
 10pm:  100  ▁
  2am:   50  ▁  (batch jobs)

Peak-to-average ratio: ~2.5×

Growth Patterns

def forecast_tokens(
    historical_daily_tokens: list[float],
    forecast_days: int = 90,
    method: str = "linear",
) -> list[float]:
    """Forecast future daily token volumes."""
    import numpy as np
    
    days = np.arange(len(historical_daily_tokens))
    
    if method == "linear":
        # Fit linear trend
        slope, intercept = np.polyfit(days, historical_daily_tokens, 1)
        future_days = np.arange(len(days), len(days) + forecast_days)
        return (slope * future_days + intercept).tolist()
    
    elif method == "exponential":
        # Fit exponential: y = a * e^(b*x)
        log_tokens = np.log(np.array(historical_daily_tokens) + 1)
        slope, intercept = np.polyfit(days, log_tokens, 1)
        future_days = np.arange(len(days), len(days) + forecast_days)
        return np.exp(slope * future_days + intercept).tolist()

# Internal tool: linear growth (predictable)
# forecast_tokens(history, 90, "linear")
# → Steady 5-10% monthly growth

# Consumer product: exponential growth (viral)  
# forecast_tokens(history, 90, "exponential")
# → Traffic doubles every 2-3 months

---

3. Step-Function Events

Some events cause sudden traffic jumps:

Event	Typical impact	Lead time
New feature launch	20-100% increase	2-4 weeks
New customer onboarding (enterprise)	10-50% increase	1-2 months
Marketing campaign	50-200% spike (temporary)	1-2 weeks
Model upgrade (better → more usage)	10-30% increase	1 week
Competitor outage	50-500% spike (unpredictable)	0

def forecast_with_events(baseline_forecast, planned_events):
    """Overlay planned events on baseline forecast."""
    adjusted = baseline_forecast.copy()
    
    for event in planned_events:
        start_day = event["start_day"]
        multiplier = event["traffic_multiplier"]
        duration = event.get("duration_days", 30)
        ramp_days = event.get("ramp_days", 7)
        
        for day in range(start_day, start_day + duration):
            if day < len(adjusted):
                # Ramp up gradually
                ramp = min(1.0, (day - start_day) / ramp_days)
                adjusted[day] *= 1 + (multiplier - 1) * ramp
    
    return adjusted

events = [
    {"start_day": 30, "traffic_multiplier": 1.5, "duration_days": 365, "ramp_days": 14},
    {"start_day": 60, "traffic_multiplier": 1.2, "duration_days": 365, "ramp_days": 7},
]

---

4. From Tokens to GPUs

def tokens_to_gpus(
    peak_tokens_per_sec: float,
    model: str,
    gpu_type: str,
    quantization: str,
    headroom: float = 0.3,
) -> int:
    """Convert token forecast to GPU count."""
    # Per-GPU throughput at target batch size
    throughput = BENCHMARK[model][gpu_type][quantization]["tokens_per_sec"]
    
    # GPUs needed for peak
    gpus_base = math.ceil(peak_tokens_per_sec / throughput)
    
    # Add headroom
    gpus_total = math.ceil(gpus_base * (1 + headroom))
    
    # Round up to TP multiple
    tp_size = TP_REQUIREMENTS[model][quantization]
    gpus_total = math.ceil(gpus_total / tp_size) * tp_size
    
    return gpus_total

# Current: 20K tok/s → 8 H100s
# 3 months: 35K tok/s → 14 H100s (need to procure 6 more)
# 6 months: 55K tok/s → 22 H100s (need to procure 8 more)

---

5. Procurement Planning

Month 0: Current demand, current capacity
Month 3: Predicted demand, procurement order placed NOW
Month 6: Predicted demand, hardware arrives and deployed

Timeline:
  [Now]                  [Order]              [Deploy]         [Needed]
    │                       │                     │               │
    ├── Cloud on-demand ────┤                     │               │
    ├── Cloud reserved ─────┤ (negotiate)         │               │
    └── Own hardware ───────┤── 3-6 month lead ───┤── online ─────┤

Critical insight: If hardware lead time is 4 months and your forecast horizon is 6 months, you must order at month 2 based on month 6 predictions. Forecast errors compound.

Buffer Strategy

provisioning_strategy = {
    "baseline": "reserved_instances",      # 60% of forecast
    "growth_buffer": "on_demand",          # 20% of forecast
    "burst": "spot_with_fallback",         # 20% peak buffer
    "emergency": "reduce_quality",         # Degradation ladder
}

---

Self-Check

• Identify daily and weekly traffic patterns in your workload
• Forecast 90-day token volume using linear and exponential models
• Account for step-function events (launches, onboarding) in forecasts
• Convert token forecasts to GPU count with headroom
• Design a procurement timeline accounting for 3-6 month hardware lead times

---

References

• For capacity planning integration, see Capacity Planning
• For GPU pricing, see GPU Cloud Economics