Cloud servers strand more than memory. A host stops accepting virtual machines when one resource fills, even if its other resources remain idle. Microsoft Azure measurements cited by this HotOS paper put average stranded SSD capacity at 54% and NIC bandwidth at 29%.[1] Hardware pooling can recover some of that capacity, but a large PCIe switch adds its own cost, limits the failure domain and ties the deployment to a fixed port topology.
The paper makes a more unusual proposal: if the datacenter is already buying a CXL memory pool for DRAM efficiency, use that shared memory as the data path for remote PCIe devices. The NIC, SSD or FPGA remains physically attached to one server. Other servers place I/O buffers in CXL memory that both the device and the requesting host can reach. Device pooling then becomes a software indirection over memory rather than a hardware rerouting of PCIe links.
This is a HotOS position paper, so its value lies in the architectural reframing and the prototype that tests the hardest first-order objection. It is not a completed production system. The right review must therefore separate what the paper measures, what it estimates and what it leaves to an orchestrator that has not yet been built.
The stranded-resource argument
Local attachment couples the ratio of CPUs, DRAM, SSDs and NIC bandwidth at procurement time. Real workloads do not consume those dimensions in the same ratio. Once one dimension is exhausted, the remaining capacity cannot be offered to another host. The paper illustrates square-root aggregation with eight hosts: the model leaves 19% of SSD capacity unused instead of 54%, and 10% of NIC bandwidth unused instead of 29%, provided that demands are independent.
Those numbers are not deployment measurements of the proposed system. They are an upper-level capacity argument based on observed averages and an independence assumption. Placement constraints, availability zones and synchronized traffic can correlate peaks. The useful point is narrower: even a half-rack pool can reduce the reserve required for independent bursts, so a flexible software pool may have value before it reaches an entire rack.[2]
PCIe pooling also changes failure provisioning. A server with one NIC becomes unreachable when that NIC fails. A shared pool could redirect the workload to another NIC rather than reserving a second NIC in every host. Rare accelerators are another fit: a provider may deploy one FPGA or smart SSD for 16 servers and keep it useful without wiring every host through a PCIe switch.
Shared buffers instead of remote PCIe
CXL.mem gives CPUs load/store access to device memory and lets capable endpoints DMA into it. The proposal allocates transmit and receive buffers in a shared CXL range. A NIC attached to server A performs DMA into that range; an application on server B reads or writes the same buffer through its CXL link. Neither device firmware nor the PCIe endpoint changes. Only the userspace I/O stack chooses a different allocator.
The scheme deliberately avoids multi-host hardware cache coherence, which was not available in the evaluated devices. CPU writes use non-temporal stores so data reaches the pool instead of remaining in a private cache. The authors reserve a 64-byte-slot ring in CXL memory for control messages. This channel forwards events and remote MMIO requests to the host that physically owns the device.
There are two paths, not one. Bulk packet or block data moves through shared buffers. Control operations still execute at the owning host because a remote CPU cannot directly issue MMIO to a device behind another root complex. A correct implementation must keep the two ordered: the consumer must not observe a completion flag before the corresponding buffer contents are globally visible.
What the 100 Gbps prototype proves
The test uses a dual-socket server and a Mellanox ConnectX-5 adapter rated at 100 Gbps, connected alongside a multi-headed CXL pod. Each CPU reaches the pool over PCIe 5.0 x8. The NIC sits behind socket 0, while the modified Junction userspace network stack runs on socket 1. Junction places TX and RX buffers, but not queue structures, in CXL memory. A second server generates UDP traffic through a network switch operating at the same line rate.
Across 75-byte, 1,500-byte and 9,000-byte payloads, the paper reports negligible difference between local DDR5 buffers and CXL buffers in the latency-throughput curves. Maximum throughput remains unchanged because two PCIe 5.0 x8 paths provide enough bandwidth to saturate the 100 Gbps NIC. This answers an important objection: a few hundred nanoseconds of memory latency does not automatically appear as the same increase in end-to-end packet latency when batching, queueing and the network dominate.
The control ring reaches a median one-way message latency of roughly 600 ns. That is close to one CXL write plus one CXL read and remains below a microsecond without hardware coherence. The result makes remote event signaling plausible, but it is a ping-pong microbenchmark. It does not include connection migration, queue reconstruction, retransmission state or a failed owner.
The authors estimate that current Xeon 6 platforms with 64 PCIe 5.0/CXL lanes per socket have enough links to pool a 200 Gbps NIC with x8 or a 400 Gbps NIC with x16. An estimate based on lane bandwidth is not the same as a two-port 400 Gbps experiment. Root-complex peer traffic, read-write asymmetry, DMA ordering and shared-pool contention can all reduce usable throughput.
The orchestrator is the real system
The proposed control plane runs a pooling agent on every host and a management container in the CXL pod. It inventories devices, monitors utilization and health, assigns devices to hosts and moves workloads after overload or failure. A request uses a local device if one is available below a threshold; otherwise it selects the least utilized reachable device.
That description is sufficient for a position paper but leaves the most consequential engineering choices open. NIC state includes queues, doorbells, flow steering, congestion-control state and in-flight packets. An SSD includes namespaces, submission and completion queues, write ordering and data durability. An accelerator may retain application state in local memory. Moving only the buffer pointer does not migrate these states.
Isolation is equally important. A shared physical address range becomes a new DMA trust boundary. The system needs per-tenant address protection, revocation after reassignment, rate limiting and protection from a device that continues DMA after its lease expires. The pool’s fabric manager and the host IOMMU must agree on ownership transitions. Otherwise software flexibility can widen the blast radius beyond that of a hardware PCIe switch.
Where the proposal fits
General-purpose network and storage devices are the strongest initial targets. Their end-to-end operations are long enough to hide CXL’s additional load latency, and their utilization varies across hosts. The paper itself is cautious about HPC and machine-learning devices. GPUs and high-rate accelerators can consume hundreds of gigabytes per second, use peer-to-peer paths with platform-specific restrictions and require tighter completion semantics.
The idea also depends on a CXL pool having a positive return for memory alone.[3] Building CXL solely to avoid a PCIe switch changes the cost equation. Conversely, if an operator already has multi-ported memory devices or a sparse pod such as Octopus,[4] the incremental software path may be attractive because the links, address translation and fabric management are already present.
A production acceptance test
The first test should reproduce the paper’s UDP result while both hosts generate simultaneous traffic and a third workload saturates memory bandwidth. It should report p50, p99 and p99.9 packet latency, CXL read and write traffic, root-complex counters and fairness between local and remote users. The question is not whether one NIC reaches line rate in isolation, but whether shared memory remains predictable under mixed DMA and CPU access.
The second test should fail the owning host during active connections. A successful demonstration identifies the failure, freezes or invalidates its buffers, reconstructs device state elsewhere and resumes traffic without exposing stale memory. The same exercise for SSDs must verify acknowledged-write durability and ordering.
The final test is revocation. After a tenant releases a device, the old process and device must be unable to reach the reassigned range, even with queued DMA. These tests turn the paper’s compelling data-path shortcut into an operational system. Until then, the right conclusion is that a CXL pool can carry PCIe buffers efficiently, not that it has already replaced the PCIe switch.
Source and attribution
This article is an editorial analysis prepared by Silicon & Systems. It restates the paper in our own words and does not reproduce its text, tables or figures. Both figures were independently created for this article. The original paper is available from ACM SIGOPS HotOS 2025, DOI 10.1145/3713082.3730393. Copyright is held by the owners/authors, (c) 2025.