🏭 Platform & Infrastructure· 10 min read
TCO Modeling
Total Cost of Ownership for AI infrastructure — CapEx, OpEx, amortization, and a worked example for a 1000 H100 cluster.
tcototal-cost-ownershipcapexopexamortizationgpu-cluster
TCO (Total Cost of Ownership) captures the full cost of running AI infrastructure — not just GPU hardware, but power, cooling, networking, staff, facility, and amortization. TCO per GPU-hour is the metric that determines whether to build or rent, and which optimizations have the biggest impact.
1. TCO Components
TCO = CapEx (amortized) + OpEx (recurring)
CapEx:
├── GPUs 40-50% of CapEx
├── Servers (CPU, RAM, NVMe) 15-20%
├── Networking (NICs, switches, cables) 15-20%
└── Facility buildout 10-15%
OpEx (annual):
├── Power 30-40% of OpEx
├── Cooling 10-15% (included in PUE)
├── Facility lease/colo 15-20%
├── Staff (ops, ML eng) 15-25%
├── Software licenses 5-10%
└── Maintenance/spares 5-10%2. CapEx Breakdown
GPU Cluster Cost (1000 H100s)
| Component | Per-GPU cost | Total (1000 GPUs) | % of CapEx |
|---|---|---|---|
| H100 SXM 80GB | $25,000-35,000 | $30M | 47% |
| Server chassis (DGX H100) | $3,000-5,000/GPU | $4M | 6% |
| CPU, RAM, NVMe per server | $2,000-3,000/GPU | $2.5M | 4% |
| InfiniBand NDR NICs (8/node) | $1,500-2,500/GPU | $2M | 3% |
| InfiniBand switches (leaf+spine) | $2,000-4,000/GPU | $3M | 5% |
| NVLink cables | $500-1,000/GPU | $0.75M | 1% |
| InfiniBand cables | $500-1,000/GPU | $0.75M | 1% |
| Storage (parallel FS) | $1,000-2,000/GPU | $1.5M | 2% |
| Racks + PDUs | $500-1,000/GPU | $0.75M | 1% |
| Facility buildout (if new) | $5,000-15,000/GPU | $10M | 16% |
| Total CapEx | ~$55K/GPU | ~$55M | 100% |
Amortization
| Component | Useful life | Annual depreciation |
|---|---|---|
| GPUs | 3-5 years | $6-10M/year |
| Servers | 4-5 years | $1.3M/year |
| Networking | 5-7 years | $0.8M/year |
| Facility | 10-15 years | $0.7M/year |
| Total depreciation | ~$10M/year |
3. OpEx Breakdown
Annual Operating Costs (1000 H100s)
| Cost category | Formula | Annual cost |
|---|---|---|
| Power (GPUs) | 1000 × 700W × 8760hr × $0.08/kWh | $4.9M |
| Power (overhead, PUE 1.3) | GPU power × (PUE - 1) | $1.5M |
| Facility lease | 125 racks × $2K/rack/month | $3.0M |
| Staff | 5 SREs + 3 ML infra eng @ $250K loaded | $2.0M |
| Network bandwidth | 100 Gbps internet @ $0.50/Mbps/month | $0.6M |
| Software licenses | Monitoring, schedulers, security | $0.5M |
| Maintenance + spares | 5% of CapEx annually | $2.75M |
| Insurance + misc | $0.5M | |
| Total annual OpEx | ~$15.75M |
4. TCO Per GPU-Hour
def tco_per_gpu_hour(
capex: float,
annual_opex: float,
num_gpus: int,
amortization_years: int = 3,
utilization: float = 0.75,
) -> float:
"""Calculate all-in cost per usable GPU-hour."""
annual_capex = capex / amortization_years
total_annual = annual_capex + annual_opex
total_gpu_hours = num_gpus * 8760 * utilization
return total_annual / total_gpu_hours
# 1000 H100 cluster:
cost = tco_per_gpu_hour(
capex=55_000_000,
annual_opex=15_750_000,
num_gpus=1000,
amortization_years=3,
utilization=0.75,
)
# annual_capex = $18.3M
# total_annual = $34.1M
# usable GPU-hours = 1000 × 8760 × 0.75 = 6.57M hours
# TCO per GPU-hour = $5.19
# Compare to cloud:
# H100 on-demand: ~$12.29/GPU-hr
# H100 reserved 3yr: ~$5.25/GPU-hr
# → Owned at 75% utilization is competitive with 3yr reservedUtilization Sensitivity
| Utilization | Own TCO/GPU-hr | Cloud reserved | Cloud wins? |
|---|---|---|---|
| 50% | $7.78 | $5.25 | Cloud wins |
| 65% | $5.98 | $5.25 | Close |
| 75% | $5.19 | $5.25 | Own wins slightly |
| 85% | $4.58 | $5.25 | Own wins clearly |
| 95% | $4.10 | $5.25 | Own wins significantly |
Breakeven utilization: ~70% for 3-year amortization against reserved instances.
5. Worked Example: 1000 H100 Cluster, 3-Year TCO
Year 0 (Procurement + Setup):
Hardware purchase: $45M
Facility buildout: $10M
Setup labor: $1M
─────────────────────────
Total CapEx: $56M
Annual Operating (Years 1-3):
Power: $6.4M/yr
Facility lease: $3.0M/yr
Staff: $2.0M/yr
Maintenance + spares: $2.75M/yr
Software + network: $1.1M/yr
Insurance + misc: $0.5M/yr
─────────────────────────
Annual OpEx: $15.75M/yr
3-Year TCO:
CapEx: $56.0M
OpEx (3 years): $47.25M
─────────────────────────
Total 3-Year TCO: $103.25M
Per GPU: $103,250
Per GPU-hour (75% util): $5.24
Cloud comparison (3yr reserved, same 1000 GPUs):
$5.25/GPU-hr × 1000 × 8760 × 3 = $138M
Own saves: $35M (25%) over 3 years
But: Cloud has zero upfront cost and flexibility to scale down6. Hidden TCO Factors
| Factor | Impact | Often missed? |
|---|---|---|
| GPU failure rate (2-5%/year) | Replacement cost + downtime | Yes |
| Firmware/driver updates | Downtime for rolling updates | Yes |
| Power price increases | 5-10% annual in many regions | Sometimes |
| GPU depreciation curve | Resale value drops 50% after 2 years | Sometimes |
| Opportunity cost of CapEx | $56M could be invested elsewhere | Yes |
| Scaling delay | 3-6 months to add capacity vs minutes in cloud | Yes |
Self-Check
- Break down CapEx into GPU, server, networking, and facility components
- Calculate annual OpEx including power, cooling, staff, and maintenance
- Compute TCO per GPU-hour at different utilization levels
- Find the break-even utilization point between owned and cloud infrastructure
- List hidden TCO factors that are commonly underestimated
References
- NVIDIA DGX H100 Specifications
- Uptime Institute: Data Center Cost Benchmarks
- For cloud pricing comparison, see GPU Cloud Economics