Model Artifact Distribution
Distributing multi-GB model files across a cluster — cache hierarchies, peer-to-peer, and fast model loading at deployment time.
Deploying a model means loading 35-810 GB of weights onto every serving instance. At cluster scale with hundreds of instances, naive downloading from a central store creates a massive bottleneck.
1. The Distribution Challenge
| Model | Size (BF16) | Size (INT4) | 100 instances (BF16) | Download from S3 @ 10 GB/s |
|---|---|---|---|---|
| 8B | 16 GB | 4 GB | 1.6 TB total | 2.7 min per instance |
| 70B | 140 GB | 35 GB | 14 TB total | 23 min per instance |
| 405B | 810 GB | 202 GB | 81 TB total | 135 min per instance |
100 instances downloading 70B simultaneously: 14 TB from S3. Even at aggregate 100 GB/s, that’s 2.3 minutes — if S3 can sustain it. In practice, cold starts take 5-15 minutes.
2. Distribution Strategies
Central Download (Simplest)
S3 → Instance 0 (download 140 GB)
S3 → Instance 1 (download 140 GB)
...
S3 → Instance 99 (download 140 GB)
Time: max(S3 throughput / 100 instances, network bandwidth per instance)Problem: S3 request rate limits, network congestion, slow at scale.
Tiered Cache
Tier 1: S3 (origin)
Tier 2: Regional NFS/Lustre cache (pre-populated)
Tier 3: Local NVMe cache (per instance)
First instance on a node: download from Tier 2 (fast, local network)
Same model on same node later: read from Tier 3 (instant)Peer-to-Peer (BitTorrent-style)
Instance 0 downloads first 10 GB from S3
Instance 1 downloads different 10 GB from S3
Instance 0 shares its 10 GB with Instance 1 (and vice versa)
...
N instances each download 1/N of the model, share the rest
Time: download_time / N + transfer_time
→ Scales linearly with instance count!Dragonfly (Uber), Kraken (Uber), and similar P2P systems distribute container images and model artifacts at scale.
3. Model Loading Optimization
Memory-Mapped Loading
# SafeTensors: memory-mapped loading (no full copy into RAM)
from safetensors.torch import load_file
tensors = load_file("model.safetensors", device="cpu")
# Only pages actually accessed are loaded from disk
# Much faster startup than torch.load()Tensor Parallelism-Aware Loading
Instead of loading the full model then sharding:
# Bad: load full model → shard (wastes memory)
model = load_full_model() # 140 GB in CPU RAM
model = shard_for_tp(model, tp_rank=0, tp_size=8) # keep 1/8
# Good: load only the shard you need
model = load_tp_shard("model.safetensors", tp_rank=0, tp_size=8)
# Only 17.5 GB loaded per GPU4. Fast Cold Start for Serving
| Technique | Cold start time (70B) | How |
|---|---|---|
| S3 download | 5-15 min | Full download |
| Pre-cached on NVMe | 10-30 sec | Read from local SSD |
| Memory-mapped SafeTensors | 5-15 sec | mmap, load on demand |
| Pre-loaded in shared memory | <1 sec | Already in RAM from previous instance |
Production pattern: Keep popular models pre-cached on NVMe across the cluster. New instances check local cache first.
Self-Check
- Calculate distribution time for 100 instances of a 70B model
- Compare central download vs tiered cache vs P2P distribution
- Explain memory-mapped model loading and why it’s faster
- Design a caching strategy for fast cold-start serving