💾 Data & Storage· 10 min read
Parallel File Systems
Lustre, GPFS/Spectrum Scale — metadata performance, striping, and tuning for training data reads vs checkpoint writes.
lustregpfsparallel-filesystemstripingmetadatastorage
Parallel file systems (Lustre, GPFS) are the backbone of on-prem AI clusters — providing high-bandwidth shared storage for training data and checkpoints. Understanding their architecture is essential for avoiding I/O bottlenecks.
1. Why Parallel File Systems
Standard NFS can’t handle AI workloads:
| Requirement | NFS | Parallel FS (Lustre/GPFS) |
|---|---|---|
| Aggregate bandwidth | 1-10 GB/s | 100+ GB/s |
| Concurrent clients | Tens | Thousands |
| File striping | No | Yes (data across many servers) |
| Metadata scalability | Single server | Distributed metadata |
2. Lustre Architecture
Clients (compute nodes with GPUs)
↕
Network (InfiniBand / Ethernet)
↕
┌─────────────────────────────────────────┐
│ MDS (Metadata Servers) │
│ File names, directories, permissions │
│ Stripe layout (which OSTs hold data) │
├─────────────────────────────────────────┤
│ OSS/OST (Object Storage Servers/Targets)│
│ OST 0: [data blocks] │
│ OST 1: [data blocks] │
│ ... │
│ OST N: [data blocks] │
│ (each OST is a RAID group or NVMe pool)│
└─────────────────────────────────────────┘Striping
A single file is split across multiple OSTs — enabling parallel reads:
File: training_data_shard_042.bin (256 MB)
Stripe size: 1 MB
Stripe count: 4 OSTs
OST 0: MB 0, 4, 8, 12, ...
OST 1: MB 1, 5, 9, 13, ...
OST 2: MB 2, 6, 10, 14, ...
OST 3: MB 3, 7, 11, 15, ...
Read bandwidth: 4× single OST bandwidthTuning for AI Workloads
| Workload | Access pattern | Optimal tuning |
|---|---|---|
| Training data read | Large sequential reads | Wide striping (stripe_count=8-16), large stripe_size (4 MB) |
| Checkpoint write | Burst parallel writes | Wide striping, dedicated OSTs for checkpoints |
| Small file access | Random reads (configs, scripts) | Narrow striping (stripe_count=1), small stripe_size |
# Set striping for training data directory
lfs setstripe -c 8 -S 4M /lustre/training_data/
# Set striping for checkpoints
lfs setstripe -c 16 -S 1M /lustre/checkpoints/
# Check current striping
lfs getstripe /lustre/training_data/shard_042.bin3. GPFS/Spectrum Scale
IBM’s parallel file system — similar concepts, different implementation:
| Feature | Lustre | GPFS |
|---|---|---|
| Metadata | Separate MDS servers | Distributed across all nodes |
| Striping | OST-based | Disk-based (NSDRAID) |
| Small files | Weak (MDS bottleneck) | Better (distributed metadata) |
| Snapshots | Limited | Built-in |
| Best for | Large sequential I/O | Mixed workloads |
4. Common Problems and Fixes
| Problem | Symptom | Fix |
|---|---|---|
| MDS overload | ls is slow, file create stalls | Reduce small file count, use fewer directories |
| OST imbalance | Some OSTs full, others empty | Rebalance with lfs migrate, check stripe policy |
| Checkpoint storm | All GPUs write simultaneously → saturate | Stage to local NVMe, async flush to Lustre |
| Metadata storm | Thousands of workers open() simultaneously | Pre-compute file lists, use streaming datasets |
For cloud-based alternatives, see Cloud Object Storage. For checkpoint-specific design, see Checkpoint Storage Design.
Self-Check
- Draw Lustre architecture (MDS + OSS/OST + clients)
- Explain striping and calculate read bandwidth for a given stripe count
- Choose striping parameters for training data vs checkpoints
- Diagnose MDS overload and propose fixes
- Compare Lustre vs GPFS for mixed AI workloads