Serving a large language model is increasingly a memory problem wearing a compute costume. The KVCache, which stores per-token attention state so that shared context is computed once rather than repeatedly, grows linearly with context length; the paper opens with the observation that reaching a maximal hit ratio for 50M tokens of context in a service like Kimi takes roughly 20 TB of DRAM. Since no GPU server holds that, production systems such as Mooncake[2] and NVIDIA’s Dynamo[4] spill the KVCache into RDMA-attached memory pools, and an entire layer of engineering (bounce buffers, completion polling, locality-aware schedulers) exists to hide what RDMA really is: a network protocol asked to impersonate a memory bus.
A paper from Alibaba Cloud, posted to arXiv on November 25, 2025 and accepted to SIGMOD 2026, asks what happens when the impersonation stops[1]. Beluga builds a shared memory pool out of the first commercially available CXL 2.0 switch and lets both CPUs and GPUs reach it with plain load/store semantics, then rebuilds vLLM’s KVCache path[3] on top. We summarize the work in our own words and place it on the map we have been drawing in earlier coverage: Meta’s Vistara[7] showed single-host CXL expansion at hyperscale, the one-chip review[8] sketched rack-scale coherent fabrics, and Beluga occupies the middle rung: multi-host pooling through a real switch.
Why RDMA is the wrong shape for a KVCache
The paper’s critique of the status quo is concrete rather than rhetorical. In the CPU-driven model that vLLM-based stacks use, every KVCache block travels GPU to host bounce buffer to remote pool, and back again on reads. In the GPU-driven alternative (GPUDirect RDMA), a dedicated kernel occupies streaming multiprocessors just to poll for completions, and the feature is absent on consumer GPUs anyway. Either way, the control path dominates: on an H20, moving 16 KB end to end costs 10.55 µs, of which the actual data movement is 2.68 µs. Roughly 75% of the time is synchronization.
The shape of the data makes this worse. A single KVCache block for Qwen-32B with grouped-query attention scatters into 128 non-contiguous 20 KB pieces, while a ConnectX-7 NIC caps scatter-gather lists at 30 entries, so one logical operation fragments into several requests. To amortize that overhead, RDMA stacks batch blocks into 256-token super-blocks; forced down to vLLM’s native 16-token granularity, Mooncake’s cache-hit TTFT balloons from 13.0 s to 76.8 s, which is worse than simply recomputing. Lastly, because remote access is so much slower than local, schedulers must route requests toward the nodes that already hold the right blocks, and this locality bookkeeping brings skew, load imbalance and operational complexity of its own.
A switch changes what CXL means
Until recently, CXL in practice meant a single-host expander, which is exactly the Vistara design point. What changes the game is the XConn XC50256[5], a switch chip that forwards CXL.mem across 256 PCIe 5.0 lanes with 2 TB/s of capacity and a minimum 64-byte access latency of about 750 ns. Beluga’s pool combines two such chips into one switch node: up to 16 servers can share an 8 TB memory box with 1 TB/s of aggregate bandwidth. The evaluation cluster attaches two 8×H20 GPU servers to that pool, with each CPU socket reaching the switch through a PCIe 5.0 x16 CXL adapter.
Software sees something refreshingly boring. The BIOS reserves a contiguous physical range for the pool, each host onlines it in DAX mode, and processes mmap() the region straight into their address spaces. Partitioning is an offset convention; sharing is mapping the same region twice. In addition, the economics favor the switch: the host adapter costs $210 where a dual-200 Gbps NIC costs $1,745, and normalized per 64 GB/s of connectivity the CXL path comes to $218.75 against $800 for the RDMA path (the switch figure is a B1 sample price).
Coherence becomes software’s job
The honest core of the paper is that CXL 2.0 gives you the address space but not the coherence. A switch presents one logical memory to every host, yet each host’s cache hierarchy operates in isolation, so a write parked in one CPU’s cache is invisible to its neighbors. Beluga does not pretend otherwise. It narrows the problem to the pattern a KVCache actually needs (a single writer inserts a block, many readers consume it) and enforces consistency with explicit cache management on both sides.
The measured recipes are worth recording. For CPU writes, non-temporal stores that bypass the cache take 2.41 µs for 16 KB, whereas marking the region uncacheable makes an ordinary store take 281.56 µs, because every store stalls the pipeline for a full CXL round trip. For CPU reads, a CLFLUSH before the load lands at 5.98 µs against 166.49 µs for uncacheable loads. Intel’s DSA copy engine is fastest of all (1.69 to 2.12 µs) and is indifferent to the uncacheable attribute, while GPU transfers work best with the region uncacheable and DDIO disabled, so that inbound writes bypass the CPU’s last-level cache. Note that these are one-time configuration choices plus a flush instruction, which compares favorably to managing RDMA queue-pair ordering and asynchronous completions.
What the microbenchmarks teach
Three latency lessons generalize beyond this system. First, the CPU should issue loads and stores itself only below 4 KB; the DSA engine’s setup cost pays off above that, with the crossover completing by 16 KB. Second, the GPU’s problem is not the wire but the launcher: kernel launch overhead dominates small transfers, so Beluga fuses many scattered copies into one custom CUDA kernel instead of calling cudaMemcpy repeatedly. Third, there is a genuine trap: cudaMemcpy from uncacheable host-side memory collapses to about 1.23 ms for transfers under 24 KB, apparently because the CUDA runtime optimizes small copies with CPU instructions that uncacheable memory punishes. The custom kernel sidesteps this. With these fixes, a 64 KB pool-to-GPU copy takes 11.73 µs against 10.32 µs from local host memory, which is the paper’s basis for calling the pool near-local.
Bandwidth needed two more interventions. A single adapter should deliver PCIe 5.0 x16 rates, yet GPU access initially reached only 26 GB/s; the authors trace the ceiling to the CPU root complex’s peer-to-peer forwarding, not to CXL itself, and verify it by reproducing the same limits on a pure GPU-to-NIC path. The workaround is parallelism: more adapters per server, plus software interleaving of data across the pool’s memory devices at 2 MB granularity (each device tops out at 22.5 GB/s). Interleaving alone lifts end-to-end serving throughput by 33.2%. The clean fix, connecting GPUs to the CXL switch directly, is left to a future architecture.
Rebuilding the KVCache path on load/store
With the pool in place, Beluga-KVCache replaces three RDMA-era mechanisms. Data movement becomes a gather-write and scatter-read problem handled inside one CUDA kernel, with no scatter-gather list limits; this trims dense KVCache latency by 36.2% on the write side and 38.7% on the read side. The advantage widens dramatically when serving exploits sparsity: keeping only the 256 highest-scoring tokens for each head and layer turns a read into more than a thousand 160-byte fragments (over 74% of chosen tokens are non-contiguous in Qwen-32B), and fetching 16 such tokens takes 211 µs over CXL against 5,260 µs over RDMA, a 95.9% reduction.
Metadata traffic moves onto the pool as well. Beluga implements its RPC as shared-memory slots with status flags, polled entirely in user space: a 64-byte round trip costs 2.11 µs against 8.39 µs for RDMA reliable-connection RPC, and a single server thread sustains 12.13 Mops at queue depth 128, 2.7× the RDMA figure. Since every node now sees uniform access latency, the scheduler simply stops caring where blocks live: requests are spread by ordinary load balancing, and nodes join or leave without rebalancing any cache partitions.
The numbers, end to end
The evaluation runs 16 vLLM instances across two 8-GPU servers on long-context QA traces (LV-Eval, inputs beyond 15K tokens), against Mooncake and Dynamo with the pool capacity fixed at 2 TB for all systems. On the first pass, while the cache is being populated, Beluga-KVCache edges out Mooncake by 12.4% on mean TTFT and 21.5% on throughput. On the second pass, when every request hits the cache, the gap becomes categorical: mean TTFT falls from 13.0 s to 1.36 s (an 89.6% reduction) and throughput rises from 1.54 to 11.32 queries per second, a 7.35× gain. Under prefill-decode disaggregation the advantage ranges from 3.41× to 9.47×, and it grows with context length, since cache movement occupies a larger share of end-to-end time.
Limits we should be honest about
The scope is a rack. One switch domain reaches 16 servers and 8 TB, and nothing here addresses coherence or pooling across switches; the deployment measured is two servers, so the 16-host envelope is architectural rather than demonstrated. The coherence protocol is hand-built for single-writer, multiple-reader data and would not generalize to contended shared state without more machinery. The CXL-based RPC explicitly trades away the reliability guarantees of RDMA transports and leans on upper layers. GPU bandwidth to the pool remains root-complex-bound at 26 GB/s per path until GPUs attach to the fabric directly. Lastly, part of the end-to-end gap reflects the RDMA baselines’ software maturity, which the authors themselves acknowledge, and the switch price is a sample quote rather than volume pricing.
What we take from it
We believe the significant result is not the 7.35× but what it attributes the win to. RDMA’s tax on LLM serving was never headline bandwidth; it was the control path, the 75% of a small transfer spent coordinating rather than moving, multiplied by the fragmentation of attention-state layouts, and Beluga shows that memory semantics deletes that category of cost rather than shaving it. The paper also quietly rebalances the pooling debate: the case against CXL pooling[6] was argued when no multi-host switch existed to measure, and the first commercial one delivers latency within range of local DRAM for exactly the read-mostly, block-granular workload LLM serving generates. Taken together with Vistara[7] one rung below and the one-chip fabric vision[8] one rung above, the CXL ladder now has a measured middle. What remains open is whether software-managed coherence stays a reasonable price as sharing patterns grow richer, or whether that burden ultimately forces the hardware coherence that CXL 3.x promises.
Source and attribution
This article is an editorial summary prepared for Silicon and Systems. It restates the argument of the paper cited below in our own words. No text, figures or tables from the paper are reproduced here, and the figures on this page were created for this summary. The summary is based on the arXiv preprint (v1 posted November 25, 2025; v2 on November 27, 2025), available at arXiv:2511.20172; the paper has been accepted to SIGMOD 2026, with the version of record at DOI 10.1145/3786627. Copyright of the published article is held by the owner/author(s), with publication rights licensed to ACM, (c) 2026.