Model Formats

Before a model can serve requests, it must be loaded into an inference engine in the right format. Different formats serve different purposes — understanding them prevents costly format mismatches and conversion overhead.

---

1. Format Comparison

Format	Created by	Strengths	Weaknesses	Best for
SafeTensors	Hugging Face	Safe (no code execution), fast mmap load, standard	No quantization metadata	Training checkpoints, HF ecosystem
GGUF	llama.cpp	Built-in quantization, CPU+GPU hybrid, single file	CPU-focused, limited GPU optimization	Local/edge inference, consumer hardware
ONNX	Microsoft	Cross-platform, wide framework support	Large graphs, limited LLM optimizations	Cross-framework deployment
TensorRT-LLM	NVIDIA	Maximum GPU performance, compiled graphs	NVIDIA-only, offline compilation	Production GPU serving
PyTorch (pickle)	PyTorch	Native, easy	Security risk (arbitrary code execution)	Legacy, avoid for distribution

---

2. SafeTensors

The standard format for model distribution (Hugging Face Hub):

# Load SafeTensors model
from safetensors.torch import load_file
tensors = load_file("model.safetensors")

# Memory-mapped loading (fast, no copy)
# The file is mapped into virtual memory — tensors are read on demand

Key properties:

• No arbitrary code execution — unlike pickle, SafeTensors is a pure data format
• Memory-mapped — the OS loads tensors on demand, no need to read the entire file
• Sharded — large models split across model-00001-of-00004.safetensors files
• No quantization support — weights are stored in their original precision

When to Use

• Distributing models on Hugging Face Hub
• Loading into vLLM, SGLang, TGI (they accept SafeTensors directly)
• Starting point for conversion to other formats

---

3. GGUF (GPT-Generated Unified Format)

The format for llama.cpp and local inference:

# Convert from SafeTensors/HF to GGUF
python convert_hf_to_gguf.py ./model-dir --outtype f16 --outfile model.gguf

# Quantize
./llama-quantize model.gguf model-Q4_K_M.gguf Q4_K_M

Key properties:

• Built-in quantization — Q4_K_M, Q5_K_S, Q8_0, etc. stored in the file
• Single file — everything (weights, tokenizer config, metadata) in one .gguf
• CPU + GPU hybrid — can split layers between CPU and GPU for models that don’t fit in VRAM
• Active ecosystem — llama.cpp, ollama, LM Studio all use GGUF

Quantization Naming Convention

Q{bits}_K_{size}
  Q4 = 4-bit     K = k-quants (block quantization)
  Q5 = 5-bit     S = small (less precision, faster)
  Q8 = 8-bit     M = medium (balanced)
                  L = large (more precision, slower)

When to Use

• Running models on consumer hardware (laptops, desktops)
• CPU inference or CPU+GPU hybrid
• Quick local experimentation

---

4. TensorRT-LLM

NVIDIA’s compiled engine format for maximum GPU throughput:

# Build TensorRT-LLM engine (offline, takes minutes)
trtllm-build \
    --checkpoint_dir ./model-ckpt \
    --output_dir ./engine \
    --gemm_plugin bfloat16 \
    --max_batch_size 64 \
    --max_input_len 4096 \
    --max_seq_len 8192 \
    --tp_size 2

# Engine is compiled for specific:
# - GPU architecture (H100 vs A100)
# - Max batch size and sequence length
# - TP/PP configuration
# - Quantization format

Key properties:

• Pre-compiled — graph optimizations, kernel selection, memory planning done offline
• Hardware-specific — engine compiled for specific GPU, not portable
• Maximum throughput — 10-30% faster than dynamic engines for fixed configurations
• Inflexible — changing max batch size or sequence length requires rebuilding

When to Use

• Production deployment where configuration is fixed
• Maximum throughput is critical
• NVIDIA GPUs exclusively

---

5. Conversion Pipeline

Training (PyTorch) → SafeTensors (distribution)
                     │
                     ├→ vLLM/SGLang (load directly, apply runtime quantization)
                     ├→ GGUF (convert + quantize for local inference)
                     ├→ TensorRT-LLM (compile for production GPU serving)
                     └→ ONNX (cross-platform deployment)

For quantization techniques applied during or after format conversion, see Quantization Fundamentals.

---

Self-Check

• Compare SafeTensors vs GGUF vs TensorRT-LLM for a production GPU deployment
• Explain why SafeTensors is safer than pickle format
• Decode a GGUF quantization name (e.g., Q4_K_M)
• Explain why TensorRT-LLM engines are not portable across GPU architectures
• Map a model from training checkpoint to each serving format

---

References

• SafeTensors
• GGUF Format Specification
• TensorRT-LLM