Fault Detection & Recovery

At scale, failures are not exceptions — they’re the norm. A 10K GPU cluster with 1,250 nodes experiences ~1 failure per hour. Training infrastructure must detect failures quickly, recover automatically, and minimize wasted work.

1. Failure Taxonomy

Failure type	Frequency (10K GPUs)	Detection time	Impact
GPU error (ECC, thermal)	Every few hours	~seconds (driver reports)	1 GPU offline
NIC failure	Every few days	~seconds (link down)	Node loses network
Node crash (OS, PSU)	Daily	~10 seconds (heartbeat timeout)	8 GPUs offline
Network partition	Rare	~seconds (NCCL timeout)	Partial communication loss
Silent data corruption	Very rare	Not detected without checks	Wrong gradients, bad training
Storage failure	Rare	~seconds (I/O error)	Can’t save/load checkpoints

2. Detection Mechanisms

Heartbeat Monitoring

Each training process sends periodic heartbeats to a controller:

class HeartbeatMonitor:
    def __init__(self, timeout_seconds=30):
        self.last_heartbeat = {}
        self.timeout = timeout_seconds
    
    def report_alive(self, rank):
        self.last_heartbeat[rank] = time.time()
    
    def check_failures(self):
        now = time.time()
        failed = [
            rank for rank, last in self.last_heartbeat.items()
            if now - last > self.timeout
        ]
        return failed

NCCL Timeout

NCCL operations have configurable timeouts. If any rank fails to participate in a collective:

export NCCL_TIMEOUT=300  # 5 minutes (default: 10 minutes in newer versions)

When a timeout occurs, the surviving ranks get a NCCL error — signaling that recovery is needed.

GPU Health Checks

# Periodic GPU health verification
nvidia-smi --query-gpu=ecc.errors.uncorrected.volatile --format=csv
# If uncorrected ECC errors > 0, the GPU is unreliable

# Check XID errors in dmesg
dmesg | grep "NVRM: Xid"
# Xid 48, 74, 79: hardware errors requiring GPU reset or replacement

3. Recovery Strategies

Strategy 1: Full Restart

The simplest approach — kill all processes, resume from last checkpoint:

1. Detect failure (heartbeat timeout or NCCL error)
2. Kill all remaining training processes
3. Replace failed node (if hardware) or restart it
4. Load last checkpoint on all ranks
5. Resume training

Downtime: 2-5 minutes (checkpoint load + process restart). Lost work: time since last checkpoint.

Strategy 2: Rank Replacement

Replace only the failed rank without restarting all processes:

1. Detect that rank 42 has failed
2. Launch new process on a spare node
3. New process loads rank 42's state from checkpoint
4. Rejoin the communication group
5. Resume training

Faster than full restart, but requires spare nodes and sophisticated orchestration. Frameworks like Oobleck and Bamboo support this.

Strategy 3: Elastic Training

Continue training with fewer GPUs after failure:

1. Detect that node 5 (ranks 40-47) has failed
2. Surviving 248 GPUs re-rendezvous (torchrun elastic)
3. Recreate process groups with new world_size
4. Adjust DP degree (was 32, now 31)
5. Resume from last checkpoint

No spare nodes needed, but requires:

Elastic-aware training code (variable world_size)
Checkpoint format that supports resharding
Batch size adjusts with fewer GPUs (or gradient accumulation compensates)

4. Practical Recovery Setup

#!/bin/bash
# Fault-tolerant training launcher
MAX_RETRIES=10
RETRY_COUNT=0

while [ $RETRY_COUNT -lt $MAX_RETRIES ]; do
    echo "Training attempt $((RETRY_COUNT + 1))"
    
    torchrun \
        --nproc_per_node=8 \
        --nnodes=2:32 \           # elastic: min 2, max 32 nodes
        --rdzv_backend=c10d \
        --rdzv_endpoint=$MASTER:29500 \
        --max_restarts=3 \         # torchrun-level restarts
        train.py \
            --resume_from_checkpoint auto \
            --checkpoint_dir /shared/checkpoints
    
    EXIT_CODE=$?
    if [ $EXIT_CODE -eq 0 ]; then
        echo "Training completed successfully"
        break
    fi
    
    RETRY_COUNT=$((RETRY_COUNT + 1))
    echo "Training failed (exit $EXIT_CODE), retrying in 60s..."
    sleep 60
done

5. Proactive Failure Mitigation

Strategy	How it works	Benefit
ECC monitoring	Replace GPUs with rising ECC error counts	Prevent failures before they happen
Thermal monitoring	Throttle or migrate workloads from hot nodes	Prevent thermal shutdowns
Network monitoring	Track packet loss, retransmits	Detect degraded links early
Spare node pool	Keep 5-10% extra nodes ready	Instant replacement
Staggered maintenance	Rolling firmware updates, avoid cluster-wide impact	Reduce correlated failures

For measuring efficiency lost to failures, see Cluster Efficiency Metrics.

Self-Check

List common failure types in a large GPU cluster and their frequencies
Implement heartbeat monitoring for a distributed training job
Compare full restart vs rank replacement vs elastic training recovery
Set up a fault-tolerant training launcher with automatic retries
Design a proactive failure mitigation strategy