Navigation
🌐 Distributed GPU Systems· 8 min read

RoCE & Ethernet

RDMA over Converged Ethernet — DCQCN congestion control, PFC, ECN, and when RoCE is used instead of InfiniBand for AI clusters.

roceethernetdcqcnpfcecncongestion-control

RoCE (RDMA over Converged Ethernet) provides RDMA capability on standard Ethernet infrastructure — lower cost than InfiniBand but requiring careful congestion control configuration for AI workloads.

1. Why RoCE?

FactorInfiniBandRoCE v2
Network hardwareDedicated IB switches (~$15K+)Standard Ethernet switches (~$5K)
BandwidthNDR 400 Gb/s100-400 GbE
Existing infrastructureNone (new build)Can reuse existing Ethernet
Cloud availabilityLimitedWidely available (Azure, some AWS)
ConfigurationPlug-and-playRequires PFC/ECN tuning

Azure uses RoCE for its AI clusters (ND A100v4, ND H100v5). InfiniBand is used by NVIDIA DGX, Meta, and most dedicated training clusters.

2. The Congestion Problem

Ethernet is a lossy network — packets can be dropped when buffers overflow. For TCP, this is fine (retransmits). For RDMA, dropped packets are catastrophic — RDMA assumes reliable delivery.

Priority Flow Control (PFC)

PFC makes Ethernet lossless for RDMA traffic:

When switch buffer fills → send PAUSE frame to upstream sender → sender stops
When buffer drains → send RESUME → sender continues

PFC classes: Traffic is classified by priority (CoS). Only RDMA traffic (typically CoS 3) gets PFC — other traffic can still be dropped.

PFC storms: If PAUSE frames cascade across switches, the entire network can stall. This is the #1 operational risk with RoCE.

ECN (Explicit Congestion Notification)

ECN marks packets instead of dropping them:

Switch detects congestion → marks packet with ECN CE bit
Receiver sends ECN feedback to sender → sender reduces rate

DCQCN (Data Center QCN)

DCQCN combines ECN with a rate-control algorithm:

  1. Sender starts at full line rate
  2. ECN marks trigger rate reduction (multiplicative decrease)
  3. Without ECN marks, rate slowly increases (additive increase)
  4. Converges to fair share of bandwidth
# Typical RoCE switch configuration
# Enable PFC on CoS 3 (RDMA traffic)
# Enable ECN marking at 50% buffer threshold
# Configure DCQCN on NICs

For details on InfiniBand’s credit-based flow control (which avoids these issues), see RDMA Concepts.

3. RoCE Performance for AI

MetricInfiniBand NDRRoCE v2 (100 GbE)RoCE v2 (400 GbE)
Per-port BW50 GB/s12.5 GB/s50 GB/s
Latency~1 μs~2-5 μs~2 μs
AllReduce 1 GB (8 nodes)~20 ms~80 ms~20 ms
StabilityExcellentRequires tuningGood with tuning

400 GbE RoCE closes the bandwidth gap with InfiniBand NDR. The remaining difference is congestion control reliability and multi-rail support.

When RoCE Works Well

  • Cloud environments (Azure ND series)
  • Moderate-scale training (≤256 GPUs)
  • When existing Ethernet infrastructure exists
  • When InfiniBand switches are unavailable

When RoCE Struggles

  • Very large clusters (1000+ GPUs) — PFC storm risk increases
  • Shared networks (RDMA and non-RDMA traffic on same fabric)
  • Without dedicated network engineering for PFC/ECN tuning

4. NCCL with RoCE

# Force NCCL to use RoCE instead of IB
export NCCL_IB_DISABLE=1
export NCCL_SOCKET_IFNAME=eth0

# Or use RoCE explicitly
export NCCL_IB_GID_INDEX=3  # RoCEv2 GID
export NCCL_IB_DISABLE=0     # keep IB verbs (RoCE uses IB verbs)

NCCL treats RoCE as InfiniBand at the API level (both use IB verbs). The configuration difference is at the network layer.

Self-Check

  • Explain why Ethernet needs PFC and ECN for RDMA (lossy vs lossless)
  • Explain PFC storms and why they’re the #1 RoCE risk
  • Explain DCQCN and how it achieves fair bandwidth sharing
  • Compare RoCE v2 100 GbE vs 400 GbE vs InfiniBand NDR for AllReduce
  • Configure NCCL for RoCE vs InfiniBand

References