RoCE on NVIDIA Spectrum-X: Building Lossless Ethernet for AI Fabrics

RDMA over Converged Ethernet lets a GPU read memory on another host without either CPU touching the data. The hard part is not the NIC — it is that RDMA was designed for InfiniBand, a fabric that does not drop packets, and Ethernet does. Everything below is about closing that gap.

Why RDMA is fast, and what it assumes#

A conventional TCP transfer copies data several times: application buffer to kernel socket buffer, socket buffer to driver, driver to NIC. Every copy costs CPU cycles and memory bandwidth, and every packet costs an interrupt and a trip through the network stack.

RDMA removes all of it. The NIC reads directly from registered application memory, puts the data on the wire, and the receiving NIC writes it directly into registered memory on the far side. The kernel is not in the path. The remote CPU does not know a transfer happened until it is complete. This is what “zero copy” and “kernel bypass” mean in practice, and it is why an AI training job can saturate a 400G link without spending its CPU on the network.

The cost is that RDMA’s transport is much less forgiving than TCP. The original InfiniBand transport assumes a lossless fabric, so it has no fast retransmit, no selective acknowledgement, and no sophisticated congestion response. A dropped packet on a Reliable Connection queue pair means the transfer goes back to the last acknowledged point. Under the go-back-N behaviour of classic RoCE, a single drop in a large collective can cost far more than the one packet.

So the network has to do work that TCP would otherwise do for itself.

RoCEv1 and RoCEv2#

Only RoCEv2 matters in practice. A GPU cluster of any size is a routed leaf-spine fabric, and RoCEv1 cannot leave its own subnet. Just as importantly, RoCEv1 has nowhere to put ECN marks, which rules out the congestion control described below.

The 4791 destination port is worth remembering. It is what you filter on when capturing, what ACLs and QoS classification match on, and what tells you at a glance whether traffic on a link is RDMA or something else.

PFC and ECN do different jobs#

These two are constantly discussed together and are often confused. They operate at different layers, on different timescales, and solve different problems.

PFC stops the last hop from dropping#

Priority-based Flow Control (IEEE 802.1Qbb) is link-local backpressure. When a switch’s ingress buffer for a given priority fills past a threshold, it sends a PAUSE frame to the device upstream on that link, naming the priority and a duration. The upstream device stops sending that priority and keeps sending everything else.

It is fast — it acts within microseconds, on one link — and it is blunt. It does not know which flow caused the congestion, so it stops all of them at that priority. Three consequences follow, and they are the reason PFC has a bad reputation:

• Head-of-line blocking. An innocent flow sharing the priority is paused along with the culprit.
• Congestion spreading. The paused upstream switch now fills its own buffers and pauses its upstream. Backpressure propagates toward the source, hop by hop, and a hotspot on one link can stall a region of the fabric.
• Deadlock. If the paused dependencies form a cycle, nothing can drain and the fabric locks up. This is rare in a clean leaf-spine topology and much less rare once you have unusual routing or a cabling mistake.

PFC is therefore a floor, not a strategy. It should be preventing the rare drop, not running continuously.

ECN slows the sender down#

Explicit Congestion Notification works end to end and on a much longer timescale.

1. A switch’s egress queue builds past a configured threshold. Instead of dropping, it sets the Congestion Experienced bits in the IP header of packets it forwards.
2. The receiving NIC sees CE-marked packets and generates a Congestion Notification Packet back to the sender.
3. The sending NIC reduces its injection rate for that queue pair, then ramps back up as CNPs stop arriving.

On NVIDIA NICs this loop is DCQCN, and it is the part that actually controls congestion. It is per-flow rather than per-link, so it does not punish innocent traffic, and it pushes back to the source rather than pushing back one hop at a time.

CNPs deserve their own treatment: they are the control signal, so they must not be stuck behind the very congestion they are reporting. They are normally carried on their own priority, and that priority is not paused.

The division of labour is: ECN is the control loop, PFC is the safety net. If your PFC counters are climbing steadily under normal load, ECN is not doing its job — the correct response is to tune the ECN thresholds down, not to tune PFC up.

Configuring it on Cumulus Linux#

NVUE keeps this to one decision. Lossless mode:

nv set qos roce mode lossless
nv config apply

Lossy mode:

nv set qos roce mode lossy
nv config apply

A single command carries a whole QoS profile: traffic classification by DSCP, a switch priority for RoCE traffic and another for CNPs, ECN thresholds on the RoCE queue, PFC on the RoCE priority in lossless mode, and the buffer allocation behind all of it.

To remove it:

nv unset qos roce
nv config apply

Which mode#

Lossless turns on both PFC and ECN. It is the conservative choice and the one to use when the workload cannot tolerate loss or when the NICs are configured expecting it. It brings the PFC failure modes described above along with it.

Lossy runs ECN only. No PAUSE frames, therefore no congestion spreading and no risk of PFC deadlock. Drops are possible and the transport has to recover from them — which is acceptable when the NICs support a modern RoCE transport with selective retransmission rather than go-back-N, and when ECN is tuned well enough that drops stay rare.

The industry has been moving toward lossy for exactly the reasons in the PFC section. Spectrum-4 adds packet trimming, which softens the trade-off further: instead of dropping a packet outright under congestion, the switch can truncate it to its headers and forward that, so the receiver learns about the loss immediately rather than waiting for a timeout.

Both modes share a single buffer pool rather than carving the buffer into static per-port reservations. On a fabric where congestion is bursty and moves around, a shared pool absorbs a transient hotspot far better than a fixed allocation that is idle everywhere else.

Verifying it#

Configuration that is not verified under load is a guess. Three things to look at.

The profile that is actually applied:

nv show qos roce

Confirm the mode, which switch priorities carry RoCE and CNP traffic, and the ECN minimum and maximum thresholds. Compare this against what the NICs are configured to send — a DSCP mismatch between host and switch is the single most common RoCE misconfiguration. The traffic classifies into the wrong queue, misses the RoCE policy entirely, and everything looks fine in show output on both sides while performance is quietly terrible.

Per-interface counters:

nv show interface <interface> qos-roce-counters

What to read:

• ECN-marked packets rising under load — normal and healthy. This is the control loop working.
• PFC PAUSE frames sent or received, rising steadily — not healthy. ECN is too slow or its thresholds are too high, and PFC is carrying load it should not be.
• Any drops on the RoCE priority in lossless mode — the lossless guarantee is not holding. Buffer headroom, PFC thresholds, or cable and link errors.

And a baseline from the hosts:

ib_write_bw -d mlx5_0 -x 3 --report_gbits <server>
ib_write_lat -d mlx5_0 -x 3 <server>

Run these on a quiet fabric and keep the numbers. The value of a baseline is entirely in having taken it before you needed it — “throughput is 340 Gb/s” means nothing on its own, and everything if you know it was 392 Gb/s last month on the same pair of hosts.

What I would tell someone starting#

Decide DSCP first and write it down. Host NIC, switch classification, and any intermediate device must agree on which DSCP value carries RoCE and which carries CNP. Most RoCE problems that present as “it works but it is slow” are this.

Do not tune PFC to fix congestion. If PAUSE counters are high, that is a symptom. Look at ECN thresholds.

Watch CNPs as a first-class signal. A rising CNP rate tells you congestion control is engaging and where. It is a much earlier warning than throughput dropping.

Take baselines before you need them. ib_write_bw between a fixed pair of hosts, monthly, costs a minute and turns “is the fabric slow?” from an argument into a measurement.

References

• RFC 3168 — The Addition of Explicit Congestion Notification (ECN) to IP
• NVIDIA networking documentation — Cumulus Linux QoS and RoCE configuration
• perftest — ib_write_bw, ib_read_lat and the rest of the RDMA benchmark suite
• IEEE 802.1Qbb — Priority-based Flow Control

Configuration syntax here follows NVUE on recent Cumulus Linux releases; verify command form and available options against your own version before applying. ECN thresholds and buffer sizing are workload-dependent — the defaults are a starting point, not an answer.