Edge Inference Engines

Edge inference engines bridge the gap between trained models and on-device execution — handling model loading, graph optimization, hardware delegation, and memory management on constrained devices.

---

1. The Model-to-Device Pipeline

Training framework (PyTorch/TF)
  → Export (torch.export / ONNX)
  → Graph transforms (constant folding, op fusion, quantization)
  → Backend compilation (CPU/GPU/NPU/DSP delegation)
  → Runtime package (.pte / .gguf / .mlpackage / .tflite)
  → On-device execution

Three concepts to distinguish:

• Model format (IR): Describes the computation graph — ONNX, TorchScript, GGUF
• Runtime: Loads the model, manages memory, launches execution — ExecuTorch, ONNX Runtime
• Backend/Delegate: Actually computes operations — XNNPACK (CPU), Core ML (Apple), Vulkan (GPU), QNN (Qualcomm NPU)

The runtime doesn’t compute every op itself — it delegates supported subgraphs to hardware backends and falls back to CPU for unsupported ops.

---

2. Runtime Comparison

Runtime	Source	Backends	LLM support	Best for
llama.cpp	GGUF	CPU, Metal, Vulkan, CUDA	✅ Native	Local/edge LLM, simplest setup
ExecuTorch	torch.export	XNNPACK, Vulkan, Core ML, QNN	✅	PyTorch ecosystem, multi-backend
ONNX Runtime Mobile	ONNX	CPU, NNAPI, Core ML, DML	⚠️ Limited	Cross-framework, broad compatibility
Core ML	Core ML model	Apple CPU, GPU, Neural Engine	✅ (via converters)	iOS/macOS native apps
LiteRT (TFLite)	TFLite	CPU, GPU delegate, NNAPI	⚠️ Limited	Google/Android ecosystem
MLC-LLM	Compiled	CPU, Metal, Vulkan, WebGPU	✅ Native	Cross-platform compiled LLM
ncnn	ONNX/pnnx	ARM CPU, Vulkan	❌	Lightweight C++, mobile vision
MNN	ONNX/TFLite	CPU, GPU, Metal	✅	Chinese mobile ecosystem

Choosing a Runtime

LLM on device?
  ├── Yes, simplest setup → llama.cpp (GGUF)
  ├── Yes, PyTorch ecosystem → ExecuTorch
  ├── Yes, cross-platform compiled → MLC-LLM
  └── Yes, iOS only → Core ML

Vision/audio model?
  ├── PyTorch model → ExecuTorch or ONNX Runtime
  ├── TensorFlow model → LiteRT
  ├── iOS only → Core ML
  └── Lightweight C++ → ncnn

---

3. llama.cpp: The Default for Edge LLMs

# Build
cmake -B build -DGGML_METAL=ON  # macOS with Metal GPU
cmake --build build

# Run 3B model in Q4_K_M quantization
./build/bin/llama-cli \
    -m llama-3.2-3b-Q4_K_M.gguf \
    -p "Explain CUDA in one paragraph" \
    -n 200 \
    --threads 4 \
    --gpu-layers 33  # offload all layers to GPU

Key features: Single-file GGUF format, built-in quantization, CPU+GPU hybrid (split layers between CPU and GPU), Metal/Vulkan GPU backends, runs on everything from Raspberry Pi to M4 Mac.

---

4. ExecuTorch: PyTorch Native

# Export model
import torch
from executorch.exir import to_edge

model = load_model("llama-3.2-1b")
example_input = torch.randint(0, 32000, (1, 128))
exported = torch.export.export(model, (example_input,))

# Lower to edge with XNNPACK backend
edge_program = to_edge(exported)
edge_program = edge_program.to_backend("XnnpackPartitioner")
et_program = edge_program.to_executorch()

# Save
with open("model.pte", "wb") as f:
    f.write(et_program.buffer)

Key features: Native PyTorch workflow (torch.export), pluggable backends (XNNPACK for CPU, Core ML, Vulkan, QNN), graph partitioning (unsupported ops fall back to CPU).

---

5. Graph Partitioning: The Real-World Challenge

No backend supports every operation. The runtime must:

1. Analyze which ops the backend supports
2. Partition the graph: supported subgraphs → backend, rest → CPU
3. Execute with data transfers between partitions

Model graph: [Embedding → Attention → LayerNorm → FFN → ...]

NPU supports: Attention ✅, FFN ✅
NPU doesn't support: Embedding ❌, LayerNorm ❌

Partitioned:
  CPU: [Embedding] → transfer → NPU: [Attention] → transfer →
  CPU: [LayerNorm] → transfer → NPU: [FFN] → transfer → CPU: [...]

Each transfer adds latency. A model with 50% NPU coverage may be slower than 100% CPU due to transfer overhead. Always benchmark the partitioned execution, not just the supported ops.

For edge-cloud collaboration (when the device isn’t powerful enough), see Edge-Cloud Collaboration.

---

Self-Check

• Distinguish model format, runtime, and backend/delegate
• Choose a runtime for: (a) LLM on iPhone, (b) vision model on Android, (c) LLM on Linux laptop
• Set up llama.cpp with GPU layer offloading
• Export a PyTorch model to ExecuTorch with XNNPACK backend
• Explain graph partitioning and why partial NPU coverage can be slower than full CPU

---

References

• llama.cpp
• ExecuTorch
• MLC-LLM