RoCE succeeded because it offered a practical migration path. Operators could retain Ethernet cabling and switching while applications used RDMA verbs, kernel bypass, and direct memory access. Lossless behavior was reconstructed with priority flow control (PFC), and end hosts added congestion control around a transport inherited from InfiniBand. The approach was attractive when replacing the entire network was harder than adapting the one already deployed.
The 2023 IEEE Computer article by Hoefler and colleagues asks whether that compromise still holds at hyperscale[1]. Its authors come from academic high-performance computing and from HPE, Broadcom, Google, and Microsoft. They do not benchmark one RoCE product. Instead, they examine the assumptions that connect Ethernet flow control, RDMA recovery, packet format, NIC state, security, and physical-link error correction. Their conclusion is that these are no longer independent tuning problems.
This distinction matters for AI infrastructure. A faster Ethernet generation doubles serialization bandwidth, but it also increases the number of bytes in flight before a pause can take effect, the packet rate presented to the NIC, and the cost of a recovery mechanism that retransmits more than the lost packet. A fabric cannot be evaluated by line rate alone when its control mechanisms scale with bandwidth, distance, and connection count.
PFC turns distance into dedicated switch memory
PFC prevents packet loss by stopping a traffic priority before an ingress buffer overflows. A receiver crosses a threshold, sends a pause frame upstream, and continues receiving data until that frame reaches the sender and the sender stops. The reserved headroom must absorb bytes transmitted during this round trip, plus a maximum packet. In simplified form, the requirement follows bandwidth multiplied by round-trip time, with separate accounting for priorities and ports.
This memory is not an ordinary shared packet buffer. If it is reused and a pause arrives late, packets can be dropped and the lossless contract fails. Link speed, cable distance, switch pipeline delay, arbitration, and forward error correction all increase the amount that must be held. The article models a 600 ns average hop, 9 KB packets, eight traffic classes, and a three-tier fat tree to show how headroom grows across switch generations. At 800 Gbit/s, longer campus or geo-replicated paths can push per-port requirements from megabytes toward much larger values.
Credit-based fabrics such as InfiniBand also reserve in-flight capacity, so buffering is not unique to RoCE. The difference is operational timing. Credits travel proactively, while PFC reacts when a buffer is already near its threshold. PFC can share buffers flexibly across sources, but configuration errors and propagation delay become part of correctness. Increasing shared buffer capacity does not remove the headroom that must remain available for the worst permitted path.
One paused priority can grow a congestion tree
PFC acts on a priority class, not one flow. A congested destination can pause unrelated flows that happen to share that class. Those victim flows stop at an upstream switch, fill another buffer, and trigger another pause. The blocked region expands backward through the topology as a congestion tree, often called a PFC storm. A single slow receiver can therefore consume switching resources well beyond its path.
Lossless networks are also vulnerable to deadlock when routing contains cycles and buffers wait on one another. Cycle-free routing or dedicated virtual channels can prevent stable cycles, but failures and transient route changes can create states that were absent from the normal topology. Switch timeouts can break a deadlock by dropping traffic, which restores progress by temporarily violating losslessness. The operator is consequently managing a distributed safety mechanism, not just a queue threshold.
End-to-end congestion control is expected to react before PFC. In practice, algorithms such as DCQCN depend on marking thresholds, rate updates, switch buffers, and traffic distributions. Parameters that work for one topology or workload can be unstable in another. Multi-vendor operation is harder because endpoints and switches may interpret telemetry and rate changes differently. PFC remains the last line of defense, so a mistake in end-to-end control reappears as a link-level pause problem.
Recovery and path use inherit InfiniBand assumptions
RoCE reliable connections use go-back-N style recovery. When a packet is lost, later packets from that queue pair may be discarded and retransmission restarts from the missing sequence. This is simple for an in-order lossless fabric, but expensive when high bandwidth-delay products place many packets behind the lost one. A single corrupt packet can turn into a burst of retransmissions and increase incast pressure.
Strict in-order delivery also complicates multipath. Ordinary Ethernet equal-cost multipath hashes a flow to one route because spraying packets across routes would reorder them. The result is a mismatch between abundant path diversity and queue-pair granularity. Large elephant flows can collide on one link while other equal-cost links remain available. Newer designs add packet spraying and receiver-side reordering, but that changes assumptions in the transport and NIC rather than merely adjusting a switch hash.
Connection state is another scaling dimension. A reliable queue pair maintains sequence numbers, acknowledgments, retransmission state, memory protection, and scheduling context. The article notes that an RDMA connection can consume up to roughly 1 KB of NIC state. At cloud scale, millions of tenant and service connections exceed on-device cache and force metadata transfers over PCIe. Misses add latency and consume the same host link used by data DMA. A nominally network-only scaling problem thus reaches the server I/O hierarchy.
Packet efficiency is visible only with a small payload
Large transfers make almost any fast link look efficient. AI control traffic, graph updates, atomics, and storage metadata can be much smaller. RoCEv2 carries 22 bytes of Layer-2 information, 20 bytes of IP, 8 bytes of UDP, a 12-byte base transport header, and a 4-byte invariant CRC. The total is 66 bytes before optional protocol headers. Locally routed InfiniBand uses a 20-byte total header in the comparison.
On an 800 Gbit/s link, an idealized stream of 8-byte payloads could reach 12.5 billion packets per second if headers did not exist. With the 20-byte InfiniBand header, the rate falls to about 3.5 Gpps. With the 66-byte RoCEv2 header, it falls to about 1.4 Gpps, and close to 90% of transmitted bytes are overhead. Real systems add framing and implementation limits, so these values are protocol bounds rather than measured application rates.
The correct conclusion is not that RoCE is always slower than InfiniBand. Large messages amortize headers, and Ethernet deployment economics may dominate. The calculation identifies the workload boundary where packet processing and protocol bytes, rather than line rate, determine useful work. Procurement tests should therefore include the application’s message-size distribution and messages per second, not only a bandwidth sweep with large buffers.
Security and FEC cannot remain afterthoughts
RDMA exposes memory semantics to the network. Authentication, encryption, key revocation, tenant isolation, and protection against malformed operations must be designed together. The article argues that security was a minor design objective in early RoCE and InfiniBand environments compared with today’s multi-tenant clouds. Adding encryption later consumes NIC silicon, state, and latency, while connection-oriented protection can make state scaling worse.
Physical links add another fixed cost. Higher signaling rates operate closer to channel limits, so forward error correction is required. Reed-Solomon FEC must accumulate a codeword, compute parity or decode errors, and release corrected data. The article cites practical computation delays of roughly 20 to 100 ns, in addition to serialization. Faster links reduce codeword accumulation time but cannot reduce computation below the hardware floor. Stronger correction improves bit-error tolerance while adding latency and energy.
This creates a cross-layer recovery question. Link FEC corrects bit errors before packets are visible. Ethernet CRC detects remaining corruption. PFC tries to avoid drops. RoCE transport recovers loss with go-back-N. Each layer assumes behavior from the one below, and each adds buffering or state. Optimizing one layer without measuring the combined tail can move cost rather than remove it.
A new Ethernet RDMA contract
The article predicts convergence between datacenter Ethernet and high-performance interconnects, but not by preserving RoCE unchanged. A successor needs selective recovery instead of retransmitting a long suffix, multipath that tolerates reordering, efficient small-message headers, scalable or connectionless state, explicit security, and congestion control designed for large topologies. It must also coordinate with physical FEC and switch buffering.
For an operator, this framework changes how a fabric is evaluated. PFC headroom should be calculated from the longest supported path and actual switch latency, not copied from a reference topology. Failure drills should include congestion trees and transient routing states. NIC tests should measure context-cache behavior at production connection counts. Small-message workloads should report useful payload rate. Security and FEC must be included in latency and power budgets.
RoCE remains useful because it is deployed, interoperable, and integrated into application stacks. The paper is not evidence that every RoCE fabric should be replaced. It is evidence that adding faster links to the same transport contract produces diminishing returns in the workloads that stress control state and small messages. The engineering decision is whether an existing deployment’s operational maturity outweighs protocol costs, and when those costs justify a transport that changes more than congestion-control parameters.
Source and copyright notice
This article is an editorial analysis by Silicon & Systems. It restates the authors’ technical argument and numerical examples in our own words. No source sentence, table, or figure is reproduced; all figures were created for this article. The reviewed author version is available on arXiv, and the journal record is available through the IEEE DOI. (c) IEEE 2023.