GPU-Accelerated Preprocessing

When CPU-based preprocessing can’t keep up with GPU training speed, NVIDIA DALI moves transforms (decode, resize, augment) to the GPU — eliminating the CPU bottleneck entirely.

---

1. The CPU Preprocessing Bottleneck

For vision and multimodal workloads, preprocessing is expensive:

Standard pipeline (CPU):
  Read JPEG → CPU decode (10ms) → CPU resize (5ms) → CPU augment (3ms)
  → CPU→GPU transfer (1ms) → GPU training step (5ms)
  Total: ~24ms, GPU waits 19ms (79% idle!)

DALI pipeline (GPU):
  Read JPEG → GPU decode (0.5ms) → GPU resize (0.3ms) → GPU augment (0.2ms)
  → Already on GPU! → GPU training step (5ms)
  Total: ~6ms, GPU waits <1ms (~83% utilized)

---

2. NVIDIA DALI Pipeline

from nvidia.dali.pipeline import pipeline_def
import nvidia.dali.fn as fn
import nvidia.dali.types as types

@pipeline_def(batch_size=64, num_threads=4, device_id=0)
def training_pipeline():
    jpegs, labels = fn.readers.file(
        file_root="/data/imagenet/train",
        random_shuffle=True,
    )
    images = fn.decoders.image(jpegs, device="mixed")  # decode on GPU
    images = fn.resize(images, size=[224, 224])
    images = fn.crop_mirror_normalize(
        images,
        mean=[0.485 * 255, 0.456 * 255, 0.406 * 255],
        std=[0.229 * 255, 0.224 * 255, 0.225 * 255],
        mirror=fn.random.coin_flip(),
    )
    return images, labels

pipe = training_pipeline()
pipe.build()

Key Benefits

Benefit	Details
GPU decode	JPEG/video decode on GPU (nvJPEG, nvVideoCodec)
Zero CPU bottleneck	Transforms run on GPU, no CPU→GPU transfer
Prefetching	Automatic pipeline prefetch hides I/O
PyTorch integration	DALIGenericIterator wraps DALI as a PyTorch DataLoader

---

3. When to Use DALI

Workload	DALI helps?	Why
Vision (ImageNet, detection)	✅ Best use case	Heavy decode + augmentation
Video training	✅	GPU video decode is much faster
Multimodal (image + text)	✅ For image part	Text preprocessing stays on CPU
Pure text (LLM pre-training)	❌	No image decode, tokenization is CPU-efficient
Audio	⚠️ Limited	Some audio ops supported

For text-focused data loading, see PyTorch DataLoader and Streaming Datasets.

---

Self-Check

• Explain the CPU preprocessing bottleneck for vision workloads
• Compare CPU vs GPU decode/resize latency
• Write a DALI pipeline for ImageNet training
• Identify when DALI helps vs when it’s unnecessary (text vs vision)

---

References

• NVIDIA DALI Documentation
• DALI PyTorch Plugin