RDMA is often described as a way to access a remote machine’s memory without interrupting its CPU. That statement is accurate only after memory has been registered, assigned, and mapped. A disaggregated-memory service must also decide which 4 KB page belongs to which compute node, reclaim it after eviction, and keep concurrent requests from corrupting the allocator. Native one-sided RDMA verbs do not provide those operations. The usual workaround is therefore to reserve large slabs, often 1 GB or more, for each compute node. Access remains fast, but unused pages inside each slab cannot be lent to another node.
ODRP asks whether this allocation boundary can be moved into the RDMA data path itself[1]. It does not add code to the memory node and does not require a programmable SmartNIC. Instead, the compute node builds ordered chains of ordinary RDMA work requests. Reads, writes, compare-and-swap operations, and completion ordering collectively implement allocation, page lookup, writeback, and reclamation. The memory-node RNIC executes the chain while its CPU remains outside the access path.
This is a narrow but useful result. ODRP does not make a network link behave like local DRAM, and it does not remove page-fault latency. It changes the economic unit of a remote-memory pool. Capacity can be assigned at 4 KB granularity without paying a remote CPU on every decision. That distinction matters when several compute nodes have time-varying memory demand, because a pool is valuable only when idle capacity can move between tenants.
The control-path gap inside one-sided RDMA
An RDMA memory region is not an arbitrary range that software can create for free. The operating system pins physical pages, the RNIC records address translations and access keys, and applications exchange the resulting metadata before issuing data verbs. Registration belongs to the control path because it changes protection and translation state. Repeating it for every swapped page would turn a 4 KB data movement into a CPU-intensive sequence of pinning, registration, and coordination.
Existing remote paging systems therefore choose between two imperfect operating points. Static one-sided designs reserve a large region per client and keep the memory-node CPU idle. Their access path is efficient, but fragmentation grows when clients reserve more than they use. Dynamic two-sided designs let a memory-node service allocate smaller units. They recover capacity, but requests enter the remote CPU scheduler and contend for a limited core budget. A third option dynamically registers 4 KB regions, which combines fine granularity with especially high control-path cost. ODRP’s contribution is to separate ownership bookkeeping from memory registration. A large registered pool remains stable, while page ownership changes through data already visible to the RNIC.
The allocator lives in ordinary memory on the memory node. It contains a free-page structure, per-client metadata, and translation entries that connect swap offsets to remote page addresses. These are data structures, not a hidden processor. The compute node reads and updates them through atomic and ordered verbs. Consequently, the RNIC needs no new opcode, yet the sequence must remain correct when multiple clients allocate or free pages concurrently.
A work-request chain as a small program
The important mechanism is not merely batching. A chain carries dependencies between work requests so that an RNIC can use a result produced by an earlier operation. To store a page that has no mapping, for example, the chain obtains an entry from the free list, installs the mapping conditionally, writes the page, and repairs state if another request won the race. A mapped-page store takes a shorter route because the address is already known. Fetch and invalidation use corresponding lookup and reclamation paths.
RDMA hardware offers only limited branching and temporary state. ODRP deals with that limit through a client-assisted rule: the RNIC performs the common path and exposes an exceptional completion to the compute node when a rare race or depleted cache requires a decision. The client then submits the appropriate continuation. The memory node remains passive. This is different from pretending that a verb chain is a general-purpose program. The design works because OS swapping provides constrained semantics: pages are fixed at 4 KB, a client knows its swap offset, and operations can be separated into a few predictable cases.
The chains also exploit ordering rather than a global lock. A conditional atomic operation protects the mapping transition, while later writes become visible only after ownership has been established. Temporary addresses and small client-side caches reduce repeated metadata reads. The paper’s broader lesson is that programmable RDMA does not necessarily mean adding an embedded core. A carefully chosen state machine can be encoded in the ordering and data dependencies of verbs already implemented in commodity RNICs.
What the experiment actually compares
The evaluation uses one memory node and as many as eight compute nodes connected through an InfiniBand switch. Each node contains a 12-core Intel Xeon E5-2650 processor, and the paper evaluates swap-intensive microbenchmarks and applications while varying the share of local memory. The relevant comparison is not simply ODRP versus local memory. It is among four remote-memory allocation policies that expose the cost of granularity.
The static one-sided baseline preallocates remote memory in large chunks and represents the fastest, least flexible path. A dynamic one-sided baseline allocates coarse chunks on demand through the memory-node CPU. The two-sided baseline performs 4 KB management through remote processing. A dynamic 4 KB registration baseline repeatedly uses the RDMA control path. ODRP alone combines a registered pool, page-granular ownership, and a passive memory-node CPU.
This setup produces two groups of results. First, ODRP, the two-sided design, and dynamic 4 KB registration reach 100% remote-memory utilization because they can assign exactly the pages requested. Relative to static one-sided reservation, ODRP improves utilization by 1.72× to 12× across the tested demand patterns. A coarse dynamic design can still waste capacity inside its allocation units, so ODRP reaches up to 1.82× higher utilization than that alternative.
Second, the three fine-grained designs do not have the same operating cost. Two-sided allocation drives memory-node CPU use to 86% or 99% for workloads with frequent swap activity. Per-page registration can require more than the single core allocated to the service and some cases do not finish. ODRP keeps measured memory-node CPU use near zero because its metadata operations stay on the RNIC. Its application slowdown relative to the static one-sided baseline ranges from 0.8% to 14.6%. At eight compute nodes, it reaches 87.3% of the static baseline’s swap throughput and reports a 14.6% application overhead.
These percentages answer different questions. The 1.72× to 12× range is an allocation-efficiency result and depends on how unevenly clients use their reservations. The 0.8% to 14.6% range is a performance cost relative to a baseline that buys speed by reserving capacity. Combining them into one score would hide the choice the system makes. ODRP spends additional RNIC operations and network round trips so the pool can reclaim unused pages.
Where the cost moves
Removing the memory-node CPU does not eliminate work. It moves work into longer verb sequences, more metadata traffic, and client recovery logic. Small swap operations are sensitive to those extra round trips. As compute-node count rises, the allocator metadata and the RNIC’s ability to execute chains become shared resources. The eight-node result shows that ODRP remains close to the static path, but it does not establish behavior at hundreds of clients or across multiple memory nodes.
Failure handling also changes. A memory service with a CPU can run leases, garbage collection, and recovery protocols locally. A passive pool depends on compute nodes and external management to reclaim pages after a client failure. ODRP focuses on allocation and access during normal operation; a production pool still needs authentication, isolation, telemetry, and a durable ownership protocol. Likewise, its security inherits RDMA’s registered-memory model. A correct allocator does not by itself protect against a compromised client with an incorrectly scoped key.
The testbed uses an older CPU generation and InfiniBand RNICs. Newer links increase bandwidth, but faster serialization does not remove the sequential dependencies inside an allocation chain. Conversely, RNICs with better atomic throughput or deeper work queues may reduce the cost. The most portable result is therefore the structure, not a fixed latency number: keep registration coarse, change ownership fine, and execute common allocator transitions next to the data path.
The procurement question is memory, not only bandwidth
Remote-memory proposals are frequently compared by link latency and peak read bandwidth. ODRP shows why that comparison is incomplete. A fast remote read is useful only after capacity has been assigned. If a design reserves one large slab per server, the operator pays for idle capacity even when the link is busy. If it invokes a memory-node CPU for every 4 KB decision, the pool acquires a separate compute bottleneck.
For an operator, the relevant denominator is useful allocated bytes per memory-node CPU and per RNIC, under the actual distribution of tenant demand. ODRP is attractive when demand is bursty, memory nodes are intentionally CPU-light, and a modest paging slowdown buys a much higher pool occupancy. It is less compelling when each client has stable demand, remote capacity is already fully used, or application latency cannot absorb extra verb-chain round trips. In those cases, static one-sided allocation may remain the better operating point.
The paper also suggests a general design method for RDMA systems. Start with a narrow state transition that has predictable cases, map its common path to verbs and ordering guarantees, and leave rare decisions to the client. This method cannot implement arbitrary remote services, but it can remove a CPU from data structures whose semantics are already constrained by the caller. The result is not a universal programmable network. It is a practical way to make remote memory shareable at the same granularity that the operating system already manages it.
Source and copyright notice
This article is an editorial analysis by Silicon & Systems. It restates the paper’s mechanisms, measurements, and limits in our own words. No sentence, table, or figure from the proceedings is reproduced; the figures on this page were created for this review. The paper is available from the USENIX NSDI 2025 page. Copyright remains with the authors, 2025.