NCCL Internals — infra.bea.ai

NCCL (NVIDIA Collective Communications Library) is the de facto standard for GPU-to-GPU communication. Every dist.all_reduce() in PyTorch, every gradient sync in DDP, every AllGather in FSDP — they all call NCCL. This chapter explains what happens inside.

1. What NCCL Does

NCCL provides optimized implementations of collective operations (AllReduce, AllGather, ReduceScatter, etc.) that:

Automatically detect GPU topology (NVLink, PCIe, InfiniBand)
Choose the best algorithm (ring, tree, or hybrid) per operation
Exploit all available bandwidth (multiple NVLink/IB connections simultaneously)
Handle multi-node communication transparently

import torch.distributed as dist

# Under the hood: NCCL handles everything
dist.init_process_group(backend='nccl')
dist.all_reduce(tensor)  # NCCL selects algorithm, topology, protocol

2. Channels: Parallel Communication Paths

NCCL doesn’t use a single ring or tree — it uses multiple channels running in parallel.

A channel is an independent communication path through the GPU topology. If a DGX H100 has 18 NVLink connections between GPU 0 and GPU 1, NCCL can create multiple channels each using different NVLink links.

Channel 0: GPU0 → GPU1 → GPU2 → GPU3 → GPU0  (ring on links 0-3)
Channel 1: GPU0 → GPU1 → GPU2 → GPU3 → GPU0  (ring on links 4-7)
Channel 2: GPU0 → GPU3 → GPU2 → GPU1 → GPU0  (reverse ring on links 8-11)
...

Benefit: $C$ channels × per-channel bandwidth = $C \times B$ aggregate bandwidth. NCCL automatically determines the optimal number of channels based on available links.

Typical: 8-16 channels on NVLink, 1-4 channels on InfiniBand.

3. Protocol Selection

NCCL has three protocols for different message sizes:

Protocol	How it works	Best for	Latency	Bandwidth
LL (Low Latency)	8-byte inline sends, no buffering	Tiny messages (< 8 KB)	Lowest	Low
LL128	128-byte chunked sends with flags	Small messages (8 KB - 256 KB)	Low	Medium
Simple	Large buffer copies with GPU-side flow control	Large messages (> 256 KB)	Higher	Highest

NCCL selects the protocol automatically based on message size. You can override with:

export NCCL_PROTO=Simple   # force Simple protocol
export NCCL_PROTO=LL128    # force LL128

But in practice, NCCL’s automatic selection is almost always correct.

4. Algorithm Selection

NCCL also chooses between ring and tree algorithms:

Algorithm	NCCL name	Used when
Ring	`NCCL_ALGO=Ring`	Large messages (> 256 KB), bandwidth matters
Tree	`NCCL_ALGO=Tree`	Small messages (< 256 KB), latency matters
CollNet	Direct network offload	InfiniBand SHARP (hardware AllReduce)

NCCL Tuning

# Force ring algorithm
export NCCL_ALGO=Ring

# Force tree algorithm  
export NCCL_ALGO=Tree

# Set number of channels
export NCCL_NCHANNELS_PER_NET_PEER=8

# Set chunk size (affects pipelining)
export NCCL_BUFFSIZE=4194304  # 4 MB

Usually don’t tune these — NCCL’s defaults are well-optimized. Only tune if profiling shows communication as the bottleneck and you understand the topology.

Chunking and Pipelining

For large messages, NCCL doesn’t send the entire tensor at once — it chunks it into smaller pieces and pipelines them through the ring/tree:

Without pipelining: [send entire 1GB] → [wait] → [receive entire 1GB]
With pipelining:    [chunk1: 4MB] [chunk2] [chunk3] ...
                         [chunk1] [chunk2] [chunk3] ...  ← overlapped

This hides per-chunk latency behind transfer of other chunks. The chunk size is controlled by NCCL_BUFFSIZE (default: 4 MB).

export NCCL_BUFFSIZE=8388608  # 8 MB chunks (larger = fewer chunks, less latency overhead)

Larger chunks: fewer latency penalties, but less pipelining overlap. NCCL’s default is tuned for most workloads.

5. Topology Detection

When dist.init_process_group('nccl') is called, NCCL:

Discovers all GPUs — enumerates PCIe/NVLink/IB connections
Builds a topology graph — which GPUs connect to which, via what links
Selects rings/trees — finds optimal paths that use all available bandwidth
Creates channels — assigns links to channels, balancing load

You can see NCCL’s topology detection:

export NCCL_DEBUG=INFO
python train.py

# Output includes:
# NCCL INFO Ring 0 : 0 -> 1 -> 2 -> 3 -> 4 -> 5 -> 6 -> 7 -> 0
# NCCL INFO Ring 1 : 0 -> 7 -> 6 -> 5 -> 4 -> 3 -> 2 -> 1 -> 0
# NCCL INFO Trees [0] ...

6. NCCL vs Other Libraries

Library	Target	Strengths
NCCL	NVIDIA GPUs	Best GPU-to-GPU performance, NVLink-aware
Gloo	CPU, mixed	Fallback for CPU tensors, broader hardware
MPI (OpenMPI)	General HPC	Standard interface, many backends
RCCL	AMD GPUs	NCCL-compatible API for ROCm

In AI infrastructure, NCCL is used for all GPU communication. Gloo is used only for CPU-side coordination (process group setup, metadata exchange).

Self-Check

Explain what NCCL channels are and why multiple channels improve bandwidth
Name the three NCCL protocols and when each is used
Explain how NCCL selects between ring and tree algorithms
Use NCCL_DEBUG=INFO to inspect topology and ring/tree selection
Explain when to tune NCCL settings vs using defaults