The phrase “CXL memory is slower than DRAM” is true and insufficient. A vector scan can be limited by aggregate read bandwidth. A B+tree lookup can be limited by dependent-load latency. A high-selectivity filter reads a column and writes a result list, making write bandwidth decisive. A complete analytical query touches hot columns repeatedly while most base-table bytes remain cold. One average latency ratio cannot predict all four.
This PVLDB paper, written by researchers from Hasso Plattner Institute, SAP, IT University of Copenhagen and Technical University of Munich, builds a measurement ladder from links to a full database.[1] The server has eight DDR5-5200 channels and four real PCIe 5.0 x16 CXL Type-3 devices. The authors measure raw access, vectorized scans, a concurrent B+tree and TPC-H on Hyrise. The result is a method for deciding placement, not a claim that far memory is universally near-local.
The hardware gives two kinds of scale-out
Local DRAM scales through memory channels close to the CPU. CXL adds independent controllers and links. In the evaluated configuration, each DDR channel has a nominal 41.6 GB/s data rate, while each x16 CXL link has a nominal 64 GB/s line rate. Protocol overhead, device controllers and CPU request generation determine usable throughput, so the nominal figures are not interchangeable.
The important experiment varies the number of CXL devices from one to four. If performance rises with more devices, the workload was constrained by CXL bandwidth or controller queues. If it barely changes, dependent latency, CPU execution or another resource dominates. This device-count sweep is more informative than comparing one local NUMA node with one CXL NUMA node.
CXL memory is integrated into the host’s cache-coherence domain at 64-byte cache-line granularity. The CPU home agent resolves coherence, and the operating system exposes device memory as NUMA nodes. This is direct-attached expansion, not a switch-based multi-host pool. It supplies the placement evidence a pool would need, but it does not include switch latency, fabric sharing or another host’s traffic.
Vector scans reveal the read-write split
The scan microbenchmark gives every thread a 512 MiB column of four-byte integers and applies an AVX-512 less-than predicate. Matching rows produce four-byte tuple identifiers (TIDs). Selectivity controls the ratio: at 0.1% the operator mostly reads; at 100% it writes one TID for nearly every value.
With data in CPU memory, the read-dominant scan approaches 260 GB/s. At 100% selectivity, throughput plateaus around 84 GB/s because result writes become the limit. Placing both columns and TIDs on one CXL device is the weakest configuration. Four devices add bandwidth and improve it, especially where the workload is bandwidth-bound.
The more useful placement separates inputs and outputs. Cold, read-only columns can live on CXL while TID lists remain in local DRAM. This avoids spending limited CXL write bandwidth on short-lived results. The lesson generalizes beyond scans: read-mostly durable state and write-heavy temporary state should not move as one allocation merely because one operator owns both.
A B+tree changes the answer
The authors evaluate BTreeOLC with 100 million records and operations drawn from a Zipf distribution. One workload is 95% reads and 5% inserts; the other reverses the ratio. Tree nodes initially occupy 1 KiB. With all nodes on CXL, four devices provide more aggregate bandwidth than one.
The read-heavy case remains sensitive to dependent pointer-chasing latency. A lookup cannot request the next node until it has read the current node, so adding devices cannot create unlimited parallelism. The write-heavy case becomes more bandwidth-bound and benefits more from four devices. Intel top-down counters distinguish retiring, front-end, speculation and back-end stalls, then divide memory-bound work into latency- and bandwidth-bound shares.
Node size is not a fixed software detail under heterogeneous memory. The paper varies nodes from 256 B to 4 KiB. Small nodes reduce bytes per access but increase dependent traversals; large nodes fetch and modify more data and push the workload toward bandwidth limits. The throughput-optimal node size differs for local DRAM, one CXL device and four CXL devices. Moving an unchanged data structure to CXL can therefore leave performance on the table even when placement is otherwise correct.
From access counters to a TPC-H placement policy
Hyrise is an open-source columnar in-memory database. It divides tables into chunks and column segments, uses polymorphic memory resources for allocation and records access counts for sequential, monotonic, random and point accesses. The study compares page interleaving with column-granular placement.
Round-robin striping across more CXL devices raises TPC-H throughput because it raises available bandwidth. Weighted page interleaving between CPU and CXL memory can trade local capacity for performance, but pages do not express which data is valuable. A frequently read column and a cold column may be mixed at the same ratio.
Column placement uses measured access frequency. The hottest columns remain in CPU memory; colder base-table columns move to one CXL device. Temporary data generated during query processing stays local. Under the paper’s concurrent TPC-H workload, this policy stores more than 80% of table data in CXL memory while retaining at least 85% of the all-local throughput. The threshold is an editorially useful operating point, not a universal constant. A different query mix changes which columns are hot and how much local capacity is required.
What this says about pooling
A shared memory pool needs this placement layer before it needs a larger switch. Pooling creates capacity flexibility, but the database must identify what can move, how much bandwidth it will consume and which temporary writes must stay local.[2][3] Sending pages to the pool solely because they are old can misclassify a cold column that will soon be scanned by every query.
Multiple devices also create a striping decision. Interleaving improves bandwidth only when requests are parallel and the downstream paths are independent. In a switched pool, two hosts may stripe across the same devices and turn local bandwidth scaling into cross-tenant interference. The paper’s four-device curve is thus an upper bound on what a contended pool might deliver.
The cost conclusion is outside the measured scope. The study argues that CXL can reduce server cost by using cheaper or reused memory, but it does not price the four devices, switches, cables or local DRAM reduction. Pond and subsequent pool studies model that fleet-level tradeoff.[5] This paper supplies the application performance function that such a cost model should use.
A database acceptance test
Start by sweeping one, two and four devices for four patterns: sequential read, sequential write, random read and mixed read-write. Record latency, bandwidth and CPU stall counters. This establishes whether the platform bottleneck is the device, root complex or CPU request generation.
Then run application structures with data and temporary results placed independently. For a scan, separate the column from its output list. For an index, sweep node size and read-write ratio. For a query engine, keep a time series of column access counts and verify that the placement controller adapts after the workload changes.
Finally, test the pool condition that this paper does not: introduce a second host that saturates one shared CXL device. A valid service policy must preserve the database’s p99 query SLO or relocate the affected columns. The headline ratio only matters if it survives interference, rebalance and failure.
The paper’s lasting insight is simple but demanding. CXL should not be treated as a slower heap. It is a set of bandwidth domains with a different latency, and database structures must expose enough meaning for the system to place inputs, outputs and temporary state separately.
Source and attribution
This article is an editorial analysis prepared by Silicon & Systems. We restate the paper’s arguments and results in our own words and reproduce no paper text, figure or table. Both figures were created for this article. The original is available from PVLDB, DOI 10.14778/3746405.3746432, under CC BY-NC-ND 4.0. The authors retain copyright and have granted the VLDB Endowment the right to publish the work, (c) 2025.