PCIe devices are commonly purchased per server and stranded per server. A host may need a high-bandwidth NIC or local SSD for its peak, even if average utilization is low. Dedicated PCIe pooling switches can connect devices to several hosts, but they add specialized hardware, fixed cabling, and a separate management domain. Oasis asks whether a CXL memory pool already justified for memory utilization can carry device traffic as well[1].
The key idea is software routing over shared memory. Hosts in a CXL pod map regions of the same pooled memory. CPUs access those regions with loads and stores, and PCIe devices access them with DMA. A local driver writes device requests and descriptors into shared buffers. An engine on the host that physically owns the device consumes them, programs the device, and returns completions and data through the same pool. The remote host sees a logical device even though PCIe transactions do not cross between root complexes.
Oasis demonstrates the design with NIC pooling. Its evaluation reports twice the NIC utilization and a 38 ms interruption during failover. The more important architectural result is that PCIe pooling does not require routing PCIe itself. CXL memory becomes a common data plane, while software translates each device class. The trade is additional software and memory-copy protocol in exchange for flexible placement on commodity CXL 2.0 components.
A CXL pod as a device interconnect
CXL memory pools expose capacity to multiple hosts inside a rack-scale pod. The hardware used by Oasis is not cross-host cache coherent. A CPU can cache a line that another host or PCIe device changes without receiving an automatic invalidation. The system therefore cannot use an ordinary shared-memory queue that assumes coherent atomics and cache visibility.
Oasis exploits a useful property of device I/O: DMA already bypasses CPU caches. Request and completion buffers can be treated as ownership-transferred regions. Before a producer publishes a descriptor, it flushes written cache lines and issues the required ordering operation. The consumer reads only after observing the ownership change, then transfers ownership back. Data buffers are not simultaneously modified by both sides.
The paper builds a message channel specifically for non-coherent CXL memory. Existing channels often assume coherent shared memory or omit fences that happen to work on an emulator. Oasis measures the real pool and separates the control word from payload cache lines to reduce false sharing and flush traffic. The optimized channel is reported to be up to 29× faster than evaluated alternatives.
This is different from CXL 3.0 hardware coherence. Oasis deliberately targets available CXL 2.0 pooling devices and implements correctness in software. It also differs from RDMA. A network round trip and RNIC transport are unnecessary within the pod; CPUs and devices use memory semantics. The authors compare conventional cloud remote-I/O paths of roughly 50 to 110 microseconds with a CXL route whose additional cost stays in the single-digit-microsecond range for the evaluated operations.
A device engine defines what can be pooled
Shared memory moves bytes, but it does not reproduce an arbitrary PCIe device. Oasis requires a software engine for each device class. The engine understands its descriptor format, queue semantics, memory registration, interrupts or polling, reset behavior, and completion rules. For NICs, the Oasis network engine maps remote virtual queues to a physical NIC, handles packet buffers, and preserves the host networking interface expected by applications.
This translation creates flexibility. Hosts are not wired permanently to one NIC, and the allocator can assign queue capacity according to demand. One high-bandwidth adapter can serve several hosts whose peaks do not coincide. It also creates an implementation boundary. SSD, accelerator, and FPGA pooling require separate engines, and a device feature that exposes unusual ordering or peer-to-peer behavior may be difficult to virtualize.
The control plane tracks device ownership, available queue capacity, and host membership. When a host requests a device, it installs mappings and connects its logical queues to an engine. Data-plane operations then avoid the central allocator. This separation keeps common I/O from waiting for a management service while allowing reassignment and failover.
Pooling changes the denominator from ports to use
The prototype evaluation uses off-the-shelf CXL 2.0 memory-pooling hardware and commercial NICs. Under the tested workload mix, Oasis doubles NIC utilization. This does not mean one NIC produces twice its line rate. The pool serves demand that would otherwise be spread across more lightly used adapters, so useful work per purchased NIC increases.
The benefit depends on statistical multiplexing. If every host reaches peak network demand simultaneously, a shared NIC remains the same finite port and queue resource. AI collectives are often synchronized, which can reduce pooling gain compared with independently varying cloud services. The allocator needs workload-aware reservations so one host cannot consume every queue or all pool bandwidth.
Remote I/O adds latency. The client communicates through CXL memory, the engine schedules work, and the physical device completes locally. The paper reports single-digit microseconds of additional overhead, much lower than the 50 to 110 microseconds cited for cloud network paths but higher than a native local queue. This makes pooling attractive for throughput-oriented or moderately latency-sensitive devices, while the tightest microsecond services may still require a local adapter.
The 38 ms failover result includes detecting a failed serving path, assigning another engine or device, and resuming the logical interface. It is fast enough to avoid VM or container restart but not invisible to every transport. Applications with shorter timeout budgets may observe a stall or retransmission. Failure containment is nonetheless a major advantage over a device physically bound to one host: the logical attachment can move without recabling or rebooting the client.
Non-coherence is both a cost and a simplifier
Hardware coherence would let hosts use familiar shared queues, but it adds protocol, processor, and device requirements. Oasis chooses explicit ownership and flushes. That increases software responsibility but makes behavior visible and compatible with deployed pools. The data path is safe only if every producer follows the flush and ordering contract. A missed fence can become a rare corruption that is difficult to reproduce.
Cache-line layout matters because a control flag and payload in the same line can cause unnecessary writebacks. Polling frequency trades latency against CPU and CXL traffic. Large payloads may consume pool bandwidth needed by memory applications. CXL memory is therefore not a free transport added to a pool; device I/O and pooled-memory loads share links, switch ports, and device media.
Security also moves into software. The pool must restrict which host can map a command or data region, and the engine must validate descriptors before programming a physical device. A malicious client should not DMA into another host or reset a shared NIC. IOMMU protection still applies on the serving host, while CXL address permissions protect the shared region. Two protection domains must agree.
Where the model fits
Oasis is strongest when CXL memory pooling is already deployed, device demand varies among hosts, and an extra few microseconds is acceptable. In that setting, device pooling reuses the CXL investment and can reduce the number of NICs or SSDs purchased for peaks. It also enables fast reassignment after failure or maintenance.
It is less attractive when the CXL pool would be purchased only for I/O, when all hosts peak together, or when a device depends on local peer-to-peer access with GPUs. A GPU Direct path that normally stays within one PCIe tree may lose its main advantage if commands and data are relayed through host memory and a software engine. Each device class needs an end-to-end comparison rather than assuming the NIC result transfers.
Scale is bounded by the pod. CXL 2.0 pooling is typically rack-local, and shared-memory latency and routing do not replace a datacenter network. The architecture creates a device pool among a modest number of nearby hosts, then applications still use Ethernet or InfiniBand beyond that boundary.
Buy fewer devices only after measuring simultaneous demand
The business case for Oasis is utilization, not peak bandwidth. An operator should collect per-host device demand at fine time resolution, calculate how often peaks overlap, and reserve capacity for failures. The denominator is useful queue and byte service per physical device under an SLO, including CXL path overhead and failover interruption.
The engineering review should also price CXL pool bandwidth. Memory and I/O may contend, and the serving host consumes CPU cores for the device engine. Savings from fewer devices must exceed that compute cost and the complexity of maintaining device-class software. The 2× utilization result demonstrates potential, not a universal consolidation ratio.
Oasis broadens the role of CXL. It is not only a way to add remote memory or coherence. Non-coherent shared memory can be a fast, explicitly managed message and DMA medium. By leaving PCIe local and moving device semantics into software, the system obtains a flexible pool without a dedicated PCIe fabric. That is a different architecture, and its success depends on choosing devices whose latency, concurrency, and failure requirements fit the additional layer.
Source and copyright notice
This article is an editorial analysis by Silicon & Systems. It restates the paper’s architecture, measurements, and limits in our own words. No source sentence, table, or figure is reproduced; all figures were created for this article. The paper is available through its ACM DOI under CC BY 4.0. Copyright remains with the authors, 2025.