Cloud Object Storage

Cloud object stores (S3, GCS) offer virtually unlimited capacity at low cost — but with high latency that can bottleneck training. Caching layers bridge the gap between cloud latency and GPU speed.

---

1. Object Storage Characteristics

Property	Parallel FS (Lustre)	Cloud Object Store (S3)
Latency (first byte)	~100 μs	~50-200 ms (1000× slower)
Throughput (single stream)	1-5 GB/s	100-200 MB/s
Throughput (parallel)	100+ GB/s	10-100+ GB/s (many streams)
Capacity	1-10 PB (hardware limited)	Unlimited
Cost (per TB/month)	$50-200 (hardware amortized)	$20-25 (S3 Standard)
Durability	RAID-dependent	99.999999999% (11 nines)

Key tradeoff: Object stores have high throughput (with parallelism) but high latency. AI training needs low-latency access to the current batch — caching solves this.

---

2. Access Patterns for AI

Workload	Pattern	Cloud storage fit
Training data	Sequential read, high throughput	✅ Good (prefetch hides latency)
Checkpoints	Burst write, then read-back	⚠️ OK (write latency acceptable, read on restart only)
Random access (shuffle)	Small random reads	❌ Bad (high latency per read)
Model artifacts	One-time large read at startup	✅ Good (download once, cache)

---

3. Caching Layers

The Problem

Without cache:
  Every batch → S3 read → 50ms latency → GPU starved

With cache:
  First read → S3 → local NVMe cache (50ms)
  Subsequent reads → NVMe → 0.1ms → GPU fed!

Caching Solutions

Solution	How it works	Best for
S3 Mountpoint	FUSE mount S3 as filesystem	Simple reads, no write
Alluxio	Distributed cache layer (memory + SSD)	Multi-node shared cache
JuiceFS	POSIX filesystem backed by S3 + metadata DB	Full POSIX compatibility
Local NVMe + streaming	Download shards → local SSD	Simplest, most reliable

Local NVMe Caching (Recommended)

# Mosaic StreamingDataset with local cache
dataset = StreamingDataset(
    remote='s3://bucket/data/',
    local='/local_nvme/cache/',   # fast local NVMe
    cache_limit='200gb',          # evict LRU when full
)
# First epoch: download from S3 (slow)
# Subsequent: read from NVMe cache (fast)

---

4. Cost Optimization

Strategy	Savings	How
S3 Infrequent Access	40% cheaper storage	For old checkpoints, archived data
S3 Intelligent-Tiering	Auto-tier based on access	Set-and-forget
Lifecycle policies	Auto-delete old checkpoints	Keep last N checkpoints, archive rest
Transfer acceleration	Faster long-distance transfer	For cross-region training
Same-region access	Avoid egress fees	Keep compute and data in same region

For checkpoint-specific storage patterns, see Checkpoint Storage Design.

---

Self-Check

• Compare cloud object storage vs parallel FS on latency, throughput, cost
• Identify which AI workloads fit cloud storage well vs poorly
• Choose a caching strategy for a given training setup
• Design a cost-optimized storage policy for checkpoints (lifecycle rules)

---

References

• AWS S3 Performance
• Alluxio
• JuiceFS