Navigation
⚡ Inference Systems· 10 min read

The Inference Cost Stack

GPU amortization, memory bandwidth pricing, energy costs, cloud vs self-hosted TCO — where money actually goes in inference.

costeconomicstcogpucloudpricinginference

Inference is where most AI compute spending goes — and the cost structure is fundamentally different from training. Understanding the cost stack helps make optimization decisions that matter for the business.

1. Where the Money Goes

Cost componentShare of totalScales with
GPU hardware (amortization)50-70%GPU count × depreciation period
Energy (power + cooling)15-25%GPU power × PUE × electricity rate
Networking5-10%Bandwidth for multi-node, egress
Staff (ML + infra engineers)5-15%Team size (often underestimated)
Software (licenses, API costs)2-5%Usage-based

The GPU Dominance

An H100 SXM GPU costs 30,000(bareGPU).Butthefullsystemcost—includingserverchassis,networking,CPUs,NVSwitch,powersupply—ismuchhigher.ADGXH100(8GPUs)costs 30,000 (bare GPU). But the full system cost — including server chassis, networking, CPUs, NVSwitch, power supply — is much higher. A DGX H100 (8 GPUs) costs ~250,000, or **$31,000 per GPU slot** including system overhead.

Amortized over 3 years:

GPU cost/hour=$31,0003×365×24≈$1.18/hour\text{GPU cost/hour} = \frac{\$31{,}000}{3 \times 365 \times 24} \approx \$1.18\text{/hour}

Add power (700W × 0.10/kWh×1.3PUE=0.10/kWh × 1.3 PUE = 0.09/hr), data center space, networking, staff allocation → ~$2.00-2.50/hour per GPU self-hosted (fully-loaded cost).

Cloud pricing for comparison:

ProviderGPUOn-demand $/hrReserved (1yr) $/hr
AWS (p5.48xlarge)8× H100~$98~$65
GCP (a3-highgpu-8g)8× H100~$99~$62
CoreWeave8× H100~$23~$18
Lambda8× H100~$24~$19

Self-hosted at ~16−20/hr(8GPUs)vscloudat16-20/hr (8 GPUs) vs cloud at 65-98/hr — 2-3× cheaper to own, but requires upfront capital and operational expertise.

2. Cost per Token

Cost per 1M tokens=GPU cost/hourthroughput (tokens/hour)×106\text{Cost per 1M tokens} = \frac{\text{GPU cost/hour}}{\text{throughput (tokens/hour)}} \times 10^6
ConfigurationThroughputGPU cost/hrCost/1M output tokens
70B BF16, TP=4, 4×H1001,500 tok/s$16$2.96
70B INT4, TP=2, 2×H1002,000 tok/s$8$1.11
8B INT4, 1×H1005,000 tok/s$4$0.22

Quantization halves GPU count AND increases throughput — compound savings.

3. Optimization Levers (Ranked by Impact)

LeverCost reductionEffort
Right-size the model (8B instead of 70B)5-10×Low (if quality allows)
Quantization (INT4)2-4×Low (PTQ, one-time)
Batching optimization2-3×Medium (tuning)
Prefix caching1.5-2× (for shared prefixes)Low (config change)
Spot/preemptible instances2-3×Medium (fault tolerance needed)
PD disaggregation1.5-2×High (architecture change)
Custom silicon2-5×High (vendor lock-in risk)

4. Build vs Buy

FactorSelf-hostedCloud API (OpenAI, Anthropic)
Upfront costHigh ($250K+ for 8×H100)Zero
Per-token costLow ($0.22-3/M tokens)$0.15-60/M tokens (varies by model)
Break-even volume~50M tokens/monthBelow break-even: cloud cheaper
ControlFullLimited (rate limits, model changes)
Ops burdenHighZero

For custom silicon options, see Custom Silicon. For multi-model cost optimization, see Multi-Model Serving.

Self-Check

  • Calculate GPU amortization cost per hour (self-hosted)
  • Calculate cost per million tokens for a given configuration
  • Rank optimization levers by cost impact
  • Determine build vs buy break-even for a given token volume