Navigation
🌐 Distributed GPU Systems· 10 min read

RDMA Concepts

Queue pairs, memory registration, InfiniBand vs RoCE v2 β€” understanding RDMA networking without writing verbs code.

rdmainfinibandrocequeue-pairsnetworking

RDMA (Remote Direct Memory Access) is the networking technology behind GPU cluster communication. You don’t need to program RDMA directly β€” NCCL handles it. But understanding the concepts helps you debug network issues and make informed hardware decisions.

1. What RDMA Does Differently

Traditional networking (TCP/IP):

App β†’ Kernel β†’ Socket buffer β†’ TCP stack β†’ NIC β†’ Network

RDMA:

App β†’ NIC β†’ Network  (bypass kernel entirely)

Three key properties:

  1. Zero-copy: Data moves directly between app memory and NIC β€” no kernel buffers
  2. Kernel bypass: No system calls for data transfer β€” userspace talks to NIC directly
  3. CPU offload: The NIC handles the protocol β€” CPU is free for other work

2. RDMA Programming Model

Queue Pairs (QP)

Communication happens through queue pairs β€” a send queue + receive queue:

Node A                              Node B
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”                        β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Send Q   β”‚ ──── RDMA send ────→  β”‚ Recv Q   β”‚
β”‚ Recv Q   β”‚ ←── RDMA send ────── β”‚ Send Q   β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜                        β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

The application posts work requests to the send queue. The NIC processes them asynchronously. Completions appear in a completion queue (CQ).

Memory Registration

Before RDMA can access memory, it must be registered with the NIC:

// Conceptual (not actual API β€” NCCL handles this)
struct ibv_mr *mr = ibv_reg_mr(pd, buffer, size, IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_WRITE);

Registration pins the memory (prevents OS from swapping it) and gives the NIC a translation table for virtual β†’ physical addresses. This is why RDMA requires pinned memory.

RDMA Operations

OperationDescriptionUsed by
Send/RecvTraditional message passingControl messages
RDMA WriteWrite to remote memory without remote CPU involvementNCCL data transfer
RDMA ReadRead from remote memory without remote CPU involvementLess common
AtomicRemote atomic operations (CAS, Add)Synchronization

NCCL primarily uses RDMA Write β€” the sender writes data directly into the receiver’s GPU memory. The receiver’s CPU never knows it happened until it checks a completion flag.

3. InfiniBand vs RoCE v2

Two RDMA transports, each with tradeoffs:

InfiniBandRoCE v2 (RDMA over Converged Ethernet)
NetworkDedicated InfiniBand fabricStandard Ethernet switches
BandwidthNDR: 400 Gb/s, XDR: 800 Gb/s100-400 GbE
Latency~1 ΞΌs~2-5 ΞΌs
Congestion controlCredit-based (lossless)ECN/DCQCN (lossy network + SW control)
CostHigher (dedicated switches)Lower (reuse existing Ethernet)
ReliabilityExcellent (hardware flow control)Good (requires careful tuning)
Used byTop AI clusters (Meta, NVIDIA DGX)Cloud providers (Azure, some AWS)

When to Use Which

  • InfiniBand: Dedicated AI training clusters where performance is critical. The standard for top-tier training (Meta’s 24K GPU cluster, NVIDIA DGX SuperPOD).
  • RoCE v2: Cloud environments where Ethernet infrastructure exists. Azure uses RoCE for its AI clusters. Requires Priority Flow Control (PFC) and DCQCN tuning.

The Practical Difference

For NCCL AllReduce of 1 GB across 2 nodes:

InfiniBand NDRRoCE v2 (100 GbE)
Bandwidth50 GB/s12.5 GB/s
Time20 ms80 ms
Latency overhead~2 ΞΌs~5 ΞΌs

InfiniBand is 4Γ— faster for this example β€” and the gap widens with more nodes (InfiniBand’s credit-based flow control handles congestion better at scale).

Congestion Control

The biggest practical difference between IB and RoCE:

InfiniBand: Uses credit-based flow control β€” the receiver tells the sender how much buffer space is available. No packet loss, no congestion collapse. β€œLossless” by design.

RoCE v2: Runs on standard Ethernet which can drop packets. Requires:

  • PFC (Priority Flow Control): Ethernet-level pause frames to prevent buffer overflow
  • DCQCN (Data Center QCN): End-to-end congestion control that reduces send rate when ECN marks are detected

Why this matters: RoCE without proper PFC/DCQCN tuning can suffer from PFC storms β€” cascading pauses that stall the entire network. This is why RoCE deployments require careful switch configuration, while InfiniBand β€œjust works.”

4. NCCL Network Configuration

Selecting Network Interface

# Use InfiniBand
export NCCL_IB_DISABLE=0           # enable IB (default)
export NCCL_IB_HCA=mlx5_0,mlx5_1  # specific IB devices

# Use TCP/Ethernet (fallback, slow)
export NCCL_IB_DISABLE=1
export NCCL_SOCKET_IFNAME=eth0

# Multiple NICs (multi-rail)
export NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3  # 4 NICs

Multi-Rail: Using All NICs

DGX H100 has 8 ConnectX-7 NICs (one per GPU pair). NCCL uses all 8 simultaneously:

GPU 0,1 β†’ NIC 0 (400 Gb/s)
GPU 2,3 β†’ NIC 1 (400 Gb/s)
GPU 4,5 β†’ NIC 2 (400 Gb/s)
GPU 6,7 β†’ NIC 3 (400 Gb/s)
...
Total: 8 Γ— 50 GB/s = 400 GB/s aggregate inter-node bandwidth

This rail-optimized topology ensures each GPU has dedicated NIC bandwidth.

5. Verifying RDMA Setup

# Check IB devices
ibv_devinfo

# Test RDMA bandwidth
ib_write_bw --size=1048576    # on server
ib_write_bw --size=1048576 <server_ip>  # on client

# NCCL performance test
cd /path/to/nccl-tests
./build/all_reduce_perf -b 1M -e 1G -f 2 -g 8
# Shows bandwidth for AllReduce at various message sizes

Self-Check

  • Explain the 3 key RDMA properties (zero-copy, kernel bypass, CPU offload)
  • Explain queue pairs and why memory registration (pinning) is required
  • Explain the difference between InfiniBand and RoCE v2
  • Configure NCCL to use specific InfiniBand devices
  • Run nccl-tests to measure AllReduce bandwidth

References