Edge-Cloud Collaboration
Split inference, session migration, speculative on-device + verify in cloud, and when to use which approach.
Not every workload fits entirely on-device or entirely in the cloud. Edge-cloud collaboration splits the work — running what you can locally and offloading what you can’t.
1. The Spectrum
| Approach | On-device | Cloud | Latency | Privacy | Cost |
|---|---|---|---|---|---|
| Full cloud | Nothing | Everything | High (network) | Low | Per-API-call |
| Full on-device | Everything | Nothing | Low | High | Device hardware |
| Edge-cloud hybrid | Some processing | Heavy lifting | Medium | Medium | Balanced |
2. Split Inference Patterns
Pattern 1: Small Model On-Device, Large Model in Cloud
User query → On-device 1B model (fast, private)
If confidence high → return result (no cloud needed)
If confidence low → send to cloud 70B model → return resultCascading: Try the cheap local model first, escalate only when needed. Saves 60-80% of cloud API calls for simple queries.
Pattern 2: On-Device Prefill, Cloud Decode
User input (private) → On-device prefill → KV cache
Transfer KV cache to cloud → Cloud decode → Stream tokens backPrivacy benefit: Raw input stays on device. Only the KV cache (not the original text) is sent to the cloud. But KV transfer adds latency.
Pattern 3: On-Device Draft, Cloud Verify
Speculative decoding across the edge-cloud boundary:
On-device small model: draft 5 tokens (fast, local)
Send candidates to cloud: large model verifies (accurate)
Accept/reject: commit accepted tokensBenefit: Multiple tokens per cloud round-trip. Instead of 1 token per request, get 3-5 tokens — effectively hiding network latency.
Pattern 4: Embedding On-Device, Generation in Cloud
RAG pipeline:
Document → On-device embedding model → query vector
Vector → Cloud vector DB search → relevant chunks
Chunks + query → Cloud LLM → responseBenefit: Embedding is small enough for on-device. Only the query vector (not the document) leaves the device.
3. Session Migration
When context length exceeds device capacity:
Start: On-device inference (1B model, context fits in 3 GB)
After 10K tokens: KV cache exceeds device memory
→ Migrate session to cloud (transfer KV + model state)
→ Continue generation in cloud
Or: Drop to sliding window attention on device (lose distant context)4. Network-Aware Scheduling
| Network condition | Strategy |
|---|---|
| WiFi (50+ Mbps, <20ms) | Offload freely to cloud |
| 4G (10-50 Mbps, 30-80ms) | Selective offload (heavy tasks only) |
| 3G / poor signal (<5 Mbps, >100ms) | On-device only (cloud too slow) |
| Offline | Full on-device (no choice) |
Adaptive: Monitor network quality in real-time, switch strategies dynamically.
5. When to Use Each Pattern
| Use case | Best pattern | Why |
|---|---|---|
| Privacy-critical (medical, legal) | Full on-device or Pattern 2 | Data never leaves device |
| Latency-critical (voice assistant) | Full on-device + cascade | No network round-trip |
| Quality-critical (complex reasoning) | Full cloud or cascade | Large model needed |
| Cost-sensitive (high volume) | Cascade (1B + 70B) | 60-80% requests handled locally |
| Offline-capable | Full on-device | No network dependency |
For on-device model optimization, see Model Compression for Edge. For on-device runtime options, see Edge Inference Engines.
Self-Check
- Compare the 4 edge-cloud collaboration patterns
- Design a cascading pipeline (small on-device + large in cloud)
- Explain speculative on-device drafting with cloud verification
- Choose a strategy based on network conditions
- Identify when full on-device vs full cloud vs hybrid is best