Vision Models

Vision model inference differs from language models: inputs are fixed-size tensors (images), there’s no autoregressive generation, and throughput requirements can be extreme (millions of images per day for content moderation or search indexing).

---

1. Vision Inference Patterns

Task	Model	Input	Output	Latency requirement
Classification	ViT, ConvNeXt	Single image	Class label + confidence	5-50ms
Object detection	YOLO, DETR	Single image	Bounding boxes + labels	5-30ms (real-time video)
Segmentation	SAM, Mask2Former	Image + prompts	Pixel-level masks	50-200ms
Embedding	CLIP, SigLIP	Image (or text)	Feature vector	1-10ms

Key Difference from LLMs

• No autoregressive generation: One forward pass produces the full output
• Fixed compute: Same cost regardless of output complexity (number of objects, etc.)
• High batch efficiency: Many images can be batched (no variable-length KV cache)
• No KV cache: No memory growth during inference

---

2. Dynamic Input Resolution

Real-world images have varying aspect ratios and resolutions:

Input images: 1920×1080, 640×480, 3000×3000, 800×600
Model expects: 224×224 (ViT) or dynamic (SAM)

Strategies

Strategy	How	Quality	Throughput
Fixed resize	Resize all to 224×224	Distortion for non-square	Highest (uniform batch)
Pad to max	Pad all to largest in batch	No distortion, wasted compute on padding	Good
Bucket by aspect ratio	Group similar aspect ratios	Good quality, fewer padding waste	Good (requires bucketing logic)
Dynamic resolution	Model accepts any size (SAM)	Best	Varies (harder to batch)

---

3. High-Throughput Batch Inference

For offline processing (search indexing, content moderation):

# Process millions of images with high throughput
from torch.utils.data import DataLoader

dataloader = DataLoader(
    image_dataset,
    batch_size=256,        # large batch for GPU utilization
    num_workers=16,        # parallel I/O
    pin_memory=True,       # faster CPU→GPU transfer
    prefetch_factor=4,     # prefetch batches
)

model = model.half().cuda()  # FP16 for 2× throughput
model = torch.compile(model)  # fuse operations

for batch in dataloader:
    with torch.no_grad(), torch.autocast('cuda'):
        outputs = model(batch.cuda())

Throughput target: ViT-B/16 on H100 can process 10,000+ images/second at batch=256, FP16.

Pipeline: Overlap I/O and Compute

CPU: Load batch N+1 → preprocess → to GPU
GPU:                   [process batch N] [process batch N+1]
     ↑ overlapped ↑

---

4. SAM (Segment Anything)

SAM is unique — it separates the image encoder (heavy, one-time) from the mask decoder (lightweight, per-prompt):

Image → Image Encoder (ViT-H, ~0.6s) → Image embedding (cached!)
  + Point/box prompt → Mask Decoder (~10ms) → Segmentation mask
  + Different prompt → Mask Decoder (~10ms) → Different mask

Serving optimization: Encode the image once, cache the embedding, serve multiple mask queries against the same image. The decoder is 60× cheaper than the encoder.

---

5. Real-Time Video Inference

For video (30 fps = 33ms per frame budget):

Approach	Latency	Use case
Per-frame inference	5-30ms	Object detection (YOLO)
Keyframe + tracking	5ms detection + 1ms tracking	Reduce compute by 5-10×
Temporal batching	Batch 4-8 frames together	Higher throughput, slightly more latency

---

Self-Check

• Compare vision inference to LLM inference (no autoregression, fixed compute, no KV cache)
• Choose a resolution strategy for images with varying aspect ratios
• Set up a high-throughput batch inference pipeline with overlapped I/O
• Explain SAM’s encoder-decoder split and how to cache the image embedding
• Calculate throughput for ViT at a given batch size and GPU

---

References

• Segment Anything (Kirillov et al., 2023)
• YOLOv8 (Ultralytics)