Memory sits at an uncomfortable intersection of three fleet-level problems. In Meta’s fleet of millions of servers, 43.7% are limited by memory capacity before any other resource, DRAM accounts for 69% of embodied carbon emissions, and a DIMM remains serviceable for 10 to 14 years while the server around it retires in 5 to 7. Each problem has its own literature. What makes the intersection interesting is that a single mechanism can address all three at once: take the DDR4 out of a decommissioned server and attach it to a new one as expansion memory behind CXL.
That mechanism has been on the table since CXL appeared, and after six years of characterization papers, kernel patches and commercial devices, no hyperscaler had reported actually running it in production at scale. A paper presented at ISCA 2026 by Meta’s infrastructure organization closes that gap[1]. It describes the full path (i.e., a custom ASIC, the kernel integration and per-workload policy) and reports measurements from production services running on the result. We summarize the argument in our own words here, and note what we think it settles and what it leaves open. Readers who followed our earlier coverage of CXL scale-up fabrics will recognize the same infrastructure group among the authors; the two papers bracket opposite ends of the same design space.
Three problems, one recycled resource
Capacity exhaustion rarely shows up as an out-of-memory error first. It shows up as stranding: memory runs out before cores do, so CPU, storage and network sit idle behind a full DRAM budget. On earlier hardware generations, Meta reports that memory-bound services strand 25 to 40% of CPU resources, and that utilization on 64 GB configurations drops to the 25 to 35% range that a 128 GB configuration avoids. In large-scale ML serving, parameters that do not fit force wider fan-out, more shards and higher tail latency, which inflates the total cost of ownership.
The carbon and lifetime arguments push in the same direction. Since DRAM is present in essentially every server, it dominates the fleet’s embodied emissions, and retiring a 10-year DIMM with a 6-year server discards working silicon. Reusing that DIMM in a newer machine avoids both the new DRAM purchase and the new DRAM’s manufacturing footprint.
However, none of the commercial CXL expanders available when this project started could serve as the vehicle. Most bundle the controller with new DRAM, which defeats reuse, and many omit DDR4 support entirely. Power and price further limit their appeal. This is why the paper’s hardware contribution exists at all: Meta built its own expander ASIC, named Vistara, because the market offered no device optimized for old memory.
Cold pages make a slow tier tolerable
The expanded tier is substantially slower than local memory, and the paper is direct about the numbers. In production its bandwidth is roughly a tenth of what the local DDR5 tier provides, and its latency runs about 60% higher. Notably, the penalty does not come from the age of the DIMMs. About 150 ns of it is the expander datapath itself (controller, PHY and bridge), and the rest follows from running the DDR4 at 2400 MT/s deliberately, for power and for compatibility across mixed-vintage modules.
The reason this is acceptable is the shape of the workloads. Across the fleet, most memory is cold most of the time: measured idle-time distributions show that at least 75% of pages have sat untouched for more than 4 seconds across every workload studied, and median idle times range from minutes to tens of minutes. A tiering layer that keeps the small hot fraction in local DRAM and parks the cold majority behind CXL therefore pays the latency penalty on accesses that rarely happen. In deployed systems the operating point confirms this: local memory runs near 60% bandwidth utilization while the CXL tier stays below 10%.
An ASIC whose spec sheet reads like a constraint list
Vistara is a Type-3 memory expander speaking CXL 2.0/1.1, sitting behind a PCIe Gen5 x16 interface that is deployed as x8. Each chip drives two independent 72-bit DDR4 channels at up to 3200 MT/s, for a ceiling of 256 GB with 64 GB DIMMs; production uses 4×32 GB, or 128 GB per chip, because 32 GB was the largest module available for reuse in volume. Reliability is handled in the expander rather than delegated to the old DIMMs: a Reed-Solomon RS(36,32) code corrects two symbols and provides x4 chip-kill. Three RISC-V cores (secure boot, control and initialization, respectively) run the management plane, which speaks the standard CXL Command Interface to both host and BMC. The chip draws about 9 W.
The headline figure is an idle round-trip latency of roughly 50 ns through the ASIC. The paper attributes this to configuration rather than magic: CXL and DDR controller IP tuned for latency, a minimized number of clock-domain crossings, and physical design that floorplans the latency-critical blocks tightly and spends low-Vt cells on the critical datapaths. Under load, the completion buffers, flow-control windows and replay queues are all sized generously to keep the loaded latency bounded, while firmware rate-limits error interrupts so that a misbehaving DIMM cannot stall the pipeline.
The deployment vehicle, called MemServer, pairs a single-socket 158-core AMD Turin with 768 GB of DDR5-6400 across 12 channels (614 GB/s peak, ~130 ns idle) and two Vistara cards holding 256 GB of DDR4-2400 (~76 GB/s peak, ~250 ns idle), for 1 TB per node. Accesses interleave across the two devices at 256-byte granularity, and the whole CXL subsystem (two ASICs plus eight DIMMs) fits in a ~50 W budget inside a 450 to 560 W server. The economics are the point: per gigabyte, the recycled tier costs 0.13× and consumes 0.7× the power of local DRAM.
The kernel, not the BIOS, owns the tier
On the software side, the design principle is that the operating system should see and manage the asymmetry rather than have it hidden. CXL memory appears as a CPU-less NUMA node, and the Linux CXL driver (rather than BIOS address-map tricks) onlines it as ZONE_MOVABLE. That single decision keeps page tables, slabs and other non-migratable kernel structures off the slow tier, which protects both reliability and the tiering machinery’s freedom to move pages. Getting the ACPI tables (CEDT and HMAT) right turned out to be foundational, since the kernel derives its latency and bandwidth model from them.
Placement itself runs on two existing mechanisms, TPP for reactive demotion and promotion[2] and TMO for proactive offload[3], with three sysctl knobs changed from defaults and everything else stock. Two refinements are worth noting. A policy that sends file-cache and tmpfs allocations straight to the CXL node let an experimental cache configuration with 20 GB local plus 76 GB CXL match the throughput of 96 GB of all-local DRAM. In addition, Meta implemented and upstreamed auto-tuned weighted interleave for bandwidth-bound consumers.
Fleet policy is deliberately uniform: every MemServer is always a 1 TB machine, and CXL is never disabled in firmware. Workloads that cannot tolerate the tier opt out in software, through the cpuset.mems cgroup controller driven by the cluster orchestrator, which requires neither a BIOS change nor a reboot. Thus a node can serve a latency-critical service today and rejoin the general capacity pool tomorrow, and background services default to the CXL tier to keep local DRAM free.
Three objections, answered with production data
The paper spends unusual effort refuting published concerns, and this may be its most useful contribution.
The first concern is tail latency. A widely cited characterization found highly unstable tails on CXL hardware[4], and the paper’s response is a direct measurement: as concurrency sweeps from 1 to 100 threads, the latency distribution of CXL-attached memory tracks that of local DRAM, with comparable variability under contention. The authors argue the earlier anomaly is an artifact of FPGA-based test devices (e.g., insufficient credit buffering for outstanding transactions) rather than a property of the protocol. A purpose-built ASIC behaves; a prototype does not.
The second concern is software overhead. Reports of substantial TPP cost led to a series of more elaborate OS designs, yet in Meta’s production telemetry the CPU overhead of tiering, including the minor faults used for hotness detection, stays below 0.5%. Promotion traffic is similarly small (single digits to a few thousand pages per minute, depending on the service). Since migration volume is minimal, the authors also conclude that DMA-engine offload for page migration solves a problem their fleet does not have, and that simple LRU-grade hotness detection is sufficient in practice.
The third concern is whether real services regress. That answer takes the form of A/B tests, which the next section summarizes.
Production numbers
For caching services, expansion translates directly into hit rate and endurance. One cache grew its memory tier from 680 GB to 890 GB, sustained 33% more queries per server, and stretched object retention from about 1 minute to 5 to 10 minutes; another grew from 590 GB to 820 GB and cut average query processing time by 29%, while moving 2.3% of hits from flash to DRAM, which directly reduces flash wear. For data warehouse services, capacity means packing: Spark fits as many as 33% more executors on each server, the Cosco shuffle service gave back 30% of its compute capacity alongside an 11% MIPS gain, and OOM-related failures fell by 33% (by 50% for some ML workloads). Developer infrastructure packs 33% more CI containers and 33% more developer VMs per server, and ML parameter servers cut compute capacity by 25% while gaining 12% throughput; the paper projects 4 to 12% throughput gains and 20 to 25% fewer servers as models scale toward 20 TB.
Two stress results bound the envelope. First, at the deployed 3:1 and even 2:1 local-to-CXL capacity ratios, iso-capacity comparisons against all-local DRAM stay within 1% on application metrics; measurable regressions appear only at 1:1, where 35 to 40% of the working set lives on CXL and a web workload loses 6% request rate while a preprocessing service loses 19% throughput. Second, a synthetic sweep of the hot fraction of the footprint finds a knee near 75%: below it, average latency stays flat around 260 ns, and beyond it, latency climbs to +22% at a fully hot footprint. Production services sit far below the knee, with fleet CXL bandwidth 10 to 100× lower than local bandwidth.
Multi-tenant hosts needed one genuine addition. With stock TPP, two identical cache containers on one node ended up with 90% and 70% local-DRAM residency, respectively, a 20% gap in p99 latency, and a noisy-neighbor scenario in which one container lost 65% of its throughput and was eventually OOM-killed. A per-container accounting layer (Fair Share) that bounds each container’s local-DRAM usage restored a balanced 3:1 split, improved the two containers’ p99 latencies by 1.6× and 1.8×, respectively, and limited the worst-case dip to 12% with no OOM kill.
What it does not claim
Precision about scope matters here, because “CXL at scale” has meant many things. This deployment is memory expansion on a single host: no pooling across hosts, no shared coherent memory, no fabric, and nothing beyond CXL 2.0 running in a 1.1-equivalent topology. The long-running debate about pooling economics, from Pond[5] to the case against pooling[6], is untouched by this evidence. Likewise, the paper’s own learnings flag the fragility of the input stream: the fleet’s reusable DDR4 spans nine component types (the top three at 29%, 28% and 15%), which forced qualification tooling and just-in-time inventory management, and the 0.13× cost per gigabyte holds only while decommissioned DDR4 keeps arriving. A future iteration built on purchased DDR5 will have to re-argue the economics, and workloads hotter than the 75% knee will need wider links (the current x8 leaves half the lanes dark) or faster media.
What we take from it
We believe the paper’s deepest result is a shift in where the burden of proof lies. For years the presumption was that tiered memory needed cleverer operating systems, and that CXL devices had disqualifying tails; this deployment shows a mostly stock kernel path holding overhead under 0.5% and an ASIC holding tails next to local DRAM, across services that bill real traffic. Proposals in this space now have to beat a deployed baseline, not a strawman. It is equally telling what made this possible: fleet-scale idle-time data to justify the design point, and the ability to build an ASIC when the market would not. To this end, the contrast with the scale-up fabric review we covered previously[7] is instructive. The same infrastructure organization backs a modest, deployed CXL 2.0 expander today and sketches an ambitious coherent fabric for tomorrow, and the distance between those two papers (i.e., between recycling cold capacity and disciplining hot latency) is a fair map of where CXL actually stands.
Source and attribution
This article is an editorial summary prepared for Silicon and Systems. It restates the argument of the paper cited below in our own words. No text, figures or tables from the published article are reproduced here, and the figures on this page were created for this summary. The paper was presented in the industry session of ISCA 2026 (Raleigh, June 29, 2026); the authoritative version appears in the ISCA 2026 proceedings, and an author-posted copy is available here. The published article is (c) IEEE 2026.