GPU Cloud Economics — infra.bea.ai

GPU cloud pricing is complex — the sticker price is just the beginning. Networking, storage, data transfer, and commitment discounts change the economics dramatically. Understanding these costs is the foundation for every infrastructure decision.

1. GPU Instance Pricing (2024-2025)

GPU	Provider	Instance	On-Demand $/hr	1yr Reserved	3yr Reserved	Spot/Preemptible
H100 80GB	AWS	p5.48xlarge (8 GPU)	$98.32 ($ 12.29/GPU)	~ $65 (~$ 8.13/GPU)	~ $42 (~$ 5.25/GPU)	$30-50
H100 80GB	GCP	a3-highgpu-8g	$98.35 ($ 12.29/GPU)	~$62	~$40	$30-40
H100 80GB	Azure	ND H100 v5	$97.98 ($ 12.25/GPU)	~$63	~$41	$30-45
H100 80GB	CoreWeave	—	$2.49/GPU	—	$2.06/GPU (1yr)	—
A100 80GB	AWS	p4de.24xlarge (8 GPU)	$40.97 ($ 5.12/GPU)	~$27	~$18	$12-20
A100 80GB	GCP	a2-ultragpu-8g	$40.22 ($ 5.03/GPU)	~$25	~$17	$12-16
L40S 48GB	AWS	g6e.48xlarge (8 GPU)	$38.80 ($ 4.85/GPU)	~$25	—	$12-18
T4 16GB	GCP	n1-standard-8 + T4	$1.40/GPU	~$0.88	~$0.63	$0.42

Prices are approximate and change frequently. Check provider pricing pages for current rates.

Hidden Costs

Cost	Typical range	Often overlooked?
GPU instance	$2-13/GPU-hr	No (headline price)
Networking (inter-AZ)	$0.01/GB	Yes — distributed training moves TB
Data egress	$0.05-0.12/GB	Yes — serving results to users
Storage (EBS/Persistent Disk)	$0.08-0.16/GB-month	Somewhat — model weights + checkpoints
Data transfer to GPU	Free (in-region)	No
IP addresses	$3.65/month (AWS)	Yes
Load balancer	$16-25/month + per-GB	Somewhat

A seemingly cheap instance can become expensive when you factor in 100 TB of checkpoint storage and multi-AZ network traffic.

2. Pricing Model Comparison

On-Demand

Cost: Highest (100%)
Commitment: None
Flexibility: Start/stop anytime
Best for: Experimentation, variable workloads, burst capacity
Risk: None

Reserved Instances (1yr / 3yr)

Cost: 35-60% discount
Commitment: 1 or 3 year contract
Flexibility: Fixed capacity for term
Best for: Steady-state production inference, known training schedules
Risk: Paying for capacity you don't use

Spot / Preemptible

Cost: 60-90% discount
Commitment: None
Flexibility: Can be reclaimed with 30s-2min notice
Best for: Fault-tolerant training (with checkpointing), batch inference
Risk: Interruption during critical work

When to Use Each

Training (days-weeks):
  ├── Can tolerate interruption? → Spot + checkpointing (biggest savings)
  └── Must be uninterrupted? → Reserved (1yr if recurring)

Inference (always-on):
  ├── Baseline (predictable load) → Reserved
  ├── Burst (above baseline) → On-demand or spot with fallback
  └── Batch (offline processing) → Spot

Development / Experimentation:
  └── On-demand (short bursts, stop when idle)

3. Spot Instance Strategy for Training

class SpotTrainingManager:
    """Manage spot-based training with checkpointing."""
    
    def __init__(self, checkpoint_interval_min=15):
        self.checkpoint_interval = checkpoint_interval_min
    
    async def run_training(self, config):
        while not training_complete:
            try:
                # Request spot instances
                instances = await cloud.request_spot(
                    instance_type="p5.48xlarge",
                    count=config.num_nodes,
                    max_price=config.max_spot_price,
                )
                
                # Resume from latest checkpoint
                checkpoint = await self.find_latest_checkpoint(config.run_id)
                
                # Train with periodic checkpointing
                await self.train_with_checkpoints(
                    instances, config, checkpoint,
                    checkpoint_every=self.checkpoint_interval,
                )
                
            except SpotInterruption:
                logger.warning("Spot interrupted — saving checkpoint")
                await self.emergency_checkpoint()
                # Loop will retry with new spot instances
            
            except SpotUnavailable:
                logger.warning("Spot unavailable — waiting or switching region")
                await asyncio.sleep(300)  # Wait 5 min and retry

4. Provider Comparison Beyond Price

Factor	AWS	GCP	Azure	CoreWeave
GPU availability	Often limited	Better H100 supply	Good with commitments	GPU-focused, good supply
Network for training	EFA (100 Gbps)	GPUDirect-TCPX	InfiniBand	InfiniBand native
Kubernetes	EKS	GKE	AKS	Native K8s (GPU-optimized)
ML tools	SageMaker	Vertex AI	Azure AI	Minimal (BYOT)
Spot stability	Variable	Better	Variable	N/A
Egress cost	$0.09/GB	$0.12/GB	$0.087/GB	$0.05/GB

5. Rent vs Own Decision

def rent_vs_own(
    num_gpus: int,
    utilization_pct: float,
    years: int = 3,
) -> dict:
    """Compare cloud rental vs owned hardware."""
    # Cloud costs (reserved 3yr H100)
    cloud_per_gpu_hr = 5.25  # 3yr reserved
    cloud_hours = years * 365.25 * 24
    cloud_total = num_gpus * cloud_per_gpu_hr * cloud_hours
    
    # Owned costs
    gpu_cost = num_gpus * 30_000          # $30K per H100
    server_cost = (num_gpus / 8) * 50_000 # $50K per 8-GPU server (non-GPU components)
    network_cost = num_gpus * 5_000        # $5K per GPU for networking
    capex = gpu_cost + server_cost + network_cost
    
    power_per_gpu_kw = 0.9  # H100 + overhead
    power_cost_kwh = 0.08   # $/kWh
    power_annual = num_gpus * power_per_gpu_kw * 8760 * power_cost_kwh
    
    colo_per_kw_month = 150  # $/kW/month
    colo_annual = num_gpus * power_per_gpu_kw * 12 * colo_per_kw_month
    
    staff_per_100_gpus = 200_000  # Annual cost for partial FTE
    staff_annual = (num_gpus / 100) * staff_per_100_gpus
    
    opex_annual = power_annual + colo_annual + staff_annual
    own_total = capex + opex_annual * years
    
    return {
        "cloud_total": cloud_total,
        "own_total": own_total,
        "savings_pct": (cloud_total - own_total) / cloud_total * 100,
        "breakeven_utilization": own_total / cloud_total,
    }

# 100 H100s for 3 years:
# Cloud (reserved): 100 × $5.25 × 26,298 hrs = $13.8M
# Owned: CapEx $3.5M + OpEx $2.1M = $5.6M
# Savings: 59% — but only if utilization > ~40%

Rule of thumb: If sustained utilization exceeds 60-70%, owning is cheaper within 2-3 years.

Self-Check

Compare on-demand, reserved, and spot pricing for H100 across major providers
Calculate hidden costs (networking, storage, egress) for a distributed training run
Design a spot training strategy with checkpointing and failover
Perform a rent-vs-own analysis for 100 GPUs over 3 years
Explain the utilization threshold where owning becomes cheaper than renting

References

AWS EC2 GPU Instances Pricing
GCP GPU Pricing
Azure GPU VM Pricing
For cost optimization techniques, see Optimization Levers