Carbon & Energy Awareness

AI infrastructure is energy-intensive: a single H100 draws 700W, and a 1000-GPU cluster consumes ~1 MW continuously. As AI scales, energy cost becomes a significant OpEx component, and carbon footprint becomes a regulatory and reputational concern.

---

1. GPU Power Consumption

GPU	TDP (Watts)	Typical inference (W)	Typical training (W)
T4	70	40-60	60-70
A100 SXM	400	200-300	350-400
H100 SXM	700	350-550	600-700
H200 SXM	700	350-550	600-700
B200 SXM	1000	500-800	900-1000
GB200 NVL72 (per GPU)	1200	600-900	1000-1200

Cluster Power

def cluster_power_kw(
    num_gpus: int,
    gpu_tdp_watts: int,
    utilization: float = 0.75,
    pue: float = 1.3,
) -> dict:
    """Calculate total facility power draw."""
    gpu_power_kw = num_gpus * gpu_tdp_watts * utilization / 1000
    
    # Non-GPU server power (CPU, RAM, NVMe, NICs): ~30% of GPU power
    server_overhead_kw = gpu_power_kw * 0.30
    
    # IT load
    it_power_kw = gpu_power_kw + server_overhead_kw
    
    # Total with cooling + facility overhead (PUE)
    total_power_kw = it_power_kw * pue
    
    return {
        "gpu_power_kw": gpu_power_kw,
        "it_power_kw": it_power_kw,
        "total_facility_kw": total_power_kw,
        "cooling_overhead_kw": total_power_kw - it_power_kw,
        "annual_kwh": total_power_kw * 8760,
        "annual_cost_usd": total_power_kw * 8760 * 0.08,  # $0.08/kWh
    }

# 1000 H100 cluster:
result = cluster_power_kw(1000, 700, 0.75, 1.3)
# GPU power: 525 kW
# IT power: 682 kW  
# Total facility: 887 kW (~1 MW)
# Annual: 7.77 GWh
# Annual cost: $621K

---

2. PUE (Power Usage Effectiveness)

PUE = Total facility power / IT equipment power. Lower is better:

Cooling type	Typical PUE	Notes
Air-cooled (basic)	1.4-1.6	Cheapest, lowest density
Air-cooled (optimized)	1.2-1.4	Hot/cold aisle containment
Direct liquid cooling (DLC)	1.05-1.15	Required for H100/B200 dense racks
Immersion cooling	1.02-1.08	Lowest PUE, highest upfront cost

Impact: Going from PUE 1.4 to 1.1 saves ~21% on total power cost.

# Annual savings from PUE improvement
it_power_kw = 682  # 1000 H100s + servers
pue_old, pue_new = 1.4, 1.1
savings_kw = it_power_kw * (pue_old - pue_new)  # 204 kW saved
annual_savings = savings_kw * 8760 * 0.08  # $143K/year

---

3. Carbon Intensity

The same kWh produces vastly different carbon depending on the grid:

Grid / Region	Carbon intensity (gCO2/kWh)	Example locations
Hydro-dominant	20-50	Quebec, Norway, Iceland
Nuclear + renewable	50-100	France, Sweden, Ontario
Mixed (US average)	350-450	Most US states
Coal-heavy	500-900	Parts of India, China, Poland

def annual_carbon_tonnes(
    annual_kwh: float,
    carbon_intensity_g_per_kwh: float,
) -> float:
    return annual_kwh * carbon_intensity_g_per_kwh / 1_000_000

# 1000 H100 cluster, 7.77 GWh/year:
# Quebec (hydro):   7.77M × 30  / 1M = 233 tonnes CO2
# US average:       7.77M × 400 / 1M = 3,108 tonnes CO2
# Coal grid:        7.77M × 800 / 1M = 6,216 tonnes CO2
# → 27× difference based on location

---

4. Carbon-Aware Scheduling

Shift flexible workloads to low-carbon hours:

class CarbonAwareScheduler:
    """Schedule batch jobs during low-carbon grid periods."""
    
    async def schedule(self, job: BatchJob):
        if job.is_latency_sensitive:
            return await self.run_immediately(job)
        
        # Get carbon forecast for next 24 hours
        forecast = await self.carbon_api.get_forecast(
            region=self.region,
            hours=24,
        )
        
        # Find lowest-carbon window that fits the job
        best_window = min(
            forecast.windows,
            key=lambda w: w.carbon_intensity
                if w.duration >= job.estimated_hours else float('inf')
        )
        
        if best_window.carbon_intensity < forecast.current.carbon_intensity * 0.7:
            # Wait for 30%+ carbon reduction
            return await self.schedule_at(job, best_window.start_time)
        else:
            # Not worth waiting
            return await self.run_immediately(job)

What Can Be Shifted

Workload	Shiftable?	Typical flexibility
Interactive inference	No	Must serve immediately
Batch inference	Yes	Hours to days
Training (long-running)	Partially	Can pause/resume
Evaluation runs	Yes	Hours
Data preprocessing	Yes	Hours to days
Model export/conversion	Yes	Hours

---

5. Renewable Energy Strategies

Strategy	Cost	Carbon reduction	Complexity
Renewable Energy Certificates (RECs)	Low	Accounting only (no physical change)	Low
Power Purchase Agreement (PPA)	Medium	Real if additive	Medium
On-site solar/wind	High upfront	Real, limited by space	High
Grid-matched 24/7	High	Highest	Very high
Low-carbon location	Varies	Highest impact decision	Medium

Biggest single decision: Where to build. Quebec at 30g CO2/kWh vs Texas at 400g CO2/kWh means 13× less carbon for the same workload.

---

6. Water Usage (WUE)

Cooling systems consume water:

Water Usage Effectiveness (WUE) = Annual water usage (liters) / IT energy (kWh)

Air cooling (evaporative): WUE = 1.0-2.0 L/kWh
Direct liquid cooling:     WUE = 0.1-0.5 L/kWh (mostly closed-loop)
Immersion cooling:         WUE ≈ 0 L/kWh (fully closed-loop)

1000 H100 cluster, air cooled:
  Annual water: 6M kWh × 1.5 L/kWh = 9M liters (2.4M gallons)
  Same cluster, DLC: 6M × 0.2 = 1.2M liters

---

Self-Check

• Calculate total facility power for a 1000-GPU cluster including PUE
• Compare annual carbon emissions across different grid regions
• Design a carbon-aware scheduler for batch workloads
• Estimate the PUE improvement from switching air cooling to DLC
• Evaluate renewable energy strategies by cost, impact, and complexity

---

References

• Electricity Maps: Real-time Carbon Intensity
• Google Carbon-Free Energy
• WattTime API: Marginal Carbon Intensity
• For cooling systems, see Cooling Systems