On-Device Constraints

Edge inference operates under constraints that don’t exist in data centers: shared memory with the OS, thermal throttling from sustained workloads, battery drain, and heterogeneous accelerators with limited operator coverage.

---

1. Why On-Device

Four reasons to run models on the device rather than the cloud:

Reason	Example
Low latency	Camera detection, voice wake-word — can’t wait for network round-trip
Offline capability	Airplane mode, rural areas, factory floors
Privacy	Photos, voice, health data stay on device
Cost	High-frequency lightweight queries — cheaper locally than cloud API calls

The tradeoff: the engineering constraints expand from “throughput and cost” to “latency, memory, package size, power, temperature, accuracy, AND compatibility.”

---

2. The Four Constraint Categories

Memory & Storage

Device memory is shared between the OS, apps, and the model:

$$M_{\text{available}} = M_{\text{physical}} - M_{\text{OS}} - M_{\text{apps}} \approx 40\text{-}60\% \text{ of physical}$$

Device	Physical RAM	Available for model	3B model (INT4)
iPhone 15 Pro	8 GB	~3-4 GB	✅ (1.5 GB)
iPhone 16 Pro	8 GB	~3-4 GB	✅
Pixel 9 Pro	16 GB	~6-8 GB	✅
Budget Android	4 GB	~1.5-2 GB	⚠️ Tight

Model file size also matters — users don’t want a 10 GB app download:

Model	FP16	INT8	INT4
1B	2 GB	1 GB	0.5 GB
3B	6 GB	3 GB	1.5 GB
7B	14 GB	7 GB	3.5 GB

Compute

Mobile SoCs have lower TFLOPS than data center GPUs:

SoC	CPU (INT8)	GPU	NPU	Total
Apple A17 Pro	~2 TOPS	~2 TOPS	35 TOPS	~39 TOPS
Snapdragon 8 Gen 3	~1.5 TOPS	~3 TOPS	45 TOPS	~50 TOPS
MediaTek Dimensity 9300	~1 TOPS	~2 TOPS	36 TOPS	~39 TOPS
H100 (comparison)	—	—	—	~4,000 TOPS (INT8)

The NPU is the powerhouse — but only for operations it supports (see §3).

Thermal & Power

Sustained inference causes heating → thermal throttling → performance drops:

Start:        100% performance, 25°C
After 30s:    95% performance, 35°C
After 2 min:  70% performance, 42°C (throttling)
After 5 min:  50% performance, 45°C (severe throttling)

Battery impact: Continuous LLM inference can drain 10-20% battery per hour on a phone.

Compatibility

Every device has different hardware (CPU ISA, GPU vendor, NPU model) and software (OS version, driver version, runtime support). An operation that runs on one device may not run on another.

---

3. Hardware Accelerators on Device

Accelerator	Strengths	Weaknesses	Best for
CPU	Universal compatibility, broadest op coverage	Slowest for matrix math	Fallback, small models
GPU (mobile)	Good parallelism, Vulkan/Metal	Scheduling overhead, shared memory	Medium models, flexible workloads
NPU	Highest TOPS, best power efficiency	Limited operator coverage, fixed precision	INT8/INT4 inference, standard architectures
DSP	Ultra-low power, always-on	Very limited ops	Audio, sensor, wake-word

The NPU challenge: NPUs are fastest but only support a subset of operations. When the model uses an unsupported op:

Model graph: [Conv → ReLU → LayerNorm → Attention → ...]
NPU supports: [Conv ✅, ReLU ✅, LayerNorm ❌, Attention ❌]

→ Graph partitioned: [Conv+ReLU on NPU] → [LayerNorm+Attention on CPU]
→ Data transfer between NPU and CPU adds latency
→ Benefit of NPU partially lost

Always benchmark on the actual target device — not just on a development machine.

For model compression techniques specific to edge, see Model Compression for Edge. For edge runtime options, see Edge Inference Engines.

---

Self-Check

• Calculate available memory for a model on a given device (physical - OS - apps)
• Estimate model file size at different precisions
• Explain thermal throttling and its impact on sustained inference
• Compare CPU, GPU, NPU, and DSP for on-device inference
• Explain the graph partitioning problem when NPU doesn’t support all ops

---

References

• ExecuTorch Documentation
• Apple Core ML Performance