A long-context serving system can report a cache hit and still recompute the prefix. The requested KV blocks may be in CPU memory, but loading them can take longer than the new-token prefill that was expected to hide the transfer. Several requests may also ask for the same prefix while the first cache miss is still unresolved, so every request behaves like a miss. Strata, presented by NVIDIA and university collaborators at OSDI 2026, treats these cases as a scheduling failure rather than a cache-capacity failure[1].
The system combines a data plane and a control plane. GPU-assisted I/O moves many small, fragmented KV pages efficiently while allowing host memory and GPU HBM to use different layouts. A cache-aware scheduler estimates transfer and compute demand before forming a batch, delays redundant work, and inserts decode work when a prefill would otherwise wait for data. Across the paper’s long-context tests, Strata reaches up to 5× the throughput of vLLM-LMCache and 3.75× that of TensorRT-LLM at comparable time to first token (TTFT). The largest gains occur where long reused contexts make I/O the dominant resource.
Small pages solve one problem and create another
PagedAttention divides KV state into small units so variable-length requests do not reserve large contiguous HBM regions. SGLang can use one-token pages, while the evaluated vLLM and TensorRT-LLM hierarchical configurations use 32-token pages. Small pages improve allocation granularity and cache reuse, but the same logical context becomes thousands of non-contiguous transfers. For Llama-3.1-8B, moving the KV state of 8,192 tokens with 32-token pages uses only about 22% of theoretical PCIe 5.0 bandwidth. On Grace Hopper, whose CPU-GPU link is much faster, the measured utilization falls to roughly 5% because software submits transfers too finely.
Increasing page size is not a free fix. In the Mistral-24B and ShareGPT experiment, moving from small pages toward 512-token pages lowers the cache hit rate and raises average TTFT by up to 2× and P90 TTFT by up to 2.9×. Coarser pages move efficiently but retain and fetch tokens that the request may not reuse. The runtime needs fine cache granularity and large physical transfers at the same time.
Strata launches a GPU kernel whose threads gather small host pages into efficient transfers. It also stores host-side KV state in a page-first layout while preserving the layer-oriented GPU layout needed by attention kernels. The result separates cache policy from transfer geometry. On the evaluated H200 microbenchmark, the GPU-assisted path reaches 48 GB/s, compared with 38 GB/s for CUDA 12.8’s batched copy API. On Grace Hopper, Strata sustains 150 GB/s; the measured starting point is 40 GB/s.
A scheduler must know when the cache will be ready
Transfer efficiency alone does not remove every stall. In the reported profile, loading cached state still consumes as much as 24% of a prefill’s runtime after I/O optimization. Strata therefore extends SGLang’s RadixTree metadata into a hierarchy-aware index and lets the scheduler query where each page resides, whether a load is in progress, and how much bandwidth the next batch will require.
Three policies follow from that state. First, delay-hit mitigation holds requests that share a context until the initial miss resolves, avoiding duplicate prefix computation. In the Mooncake tool-agent trace, 38% of requests share at least 6,000 prefix tokens with another request arriving within one second. Second, balanced batching pairs cache-heavy requests with enough new-token computation to cover transfer time. Third, stall hiding runs useful decode batches when no prefill composition can hide the I/O.
These policies have different operating regions. At low arrival rates, scheduling contributes more because small batches leave room to choose complementary work. At high rates, the transfer path becomes the bottleneck. In the ablation on Qwen2.5-14B and LooGLE, scheduling alone raises peak throughput by up to 1.8× and GPU-assisted I/O alone by up to 2.3× over the SGLang hierarchical baseline. Combining them is necessary because a faster link with an unaware scheduler still forms the wrong batch.
The 5× result belongs to long reused contexts
The principal testbed is an eight-H200 node with an Intel Sapphire Rapids CPU, 1.6 TB of DRAM, and PCIe 5.0 x16 per GPU. The evaluation covers Llama-3.1-8B, Qwen2.5-14B, and Llama-3.1-70B on LooGLE, NarrativeQA, ReviewMT, and ShareGPT. The 8B and 14B models use one GPU; the 70B model uses four-way tensor parallelism. CPU-caching cases receive 1 TB of pinned DRAM, and requests arrive according to a Poisson process because the datasets do not provide timestamps.
On LooGLE at the same TTFT, Strata improves throughput over vLLM-LMCache by up to 2.6× for Llama-8B, 2.1× for Qwen-14B, and 5× for Llama-70B. Against TensorRT-LLM hierarchical caching, the corresponding maxima are 1.9×, 1.9×, and 3.75×. ReviewMT has longer decoding, so prefill I/O is a smaller fraction; the Llama-8B gain over the two baselines is 2.3×. With NarrativeQA’s CPU cache prewarmed, Strata reaches 2.3× to 2.6× vLLM-LMCache throughput across the three models.
Short-context ShareGPT performance remains comparable after accounting for underlying engine differences. That result means the mechanisms do not impose an obvious penalty in the tested setup. It does not mean short contexts benefit from Strata. When reuse distance is minimal, hierarchical offload is unnecessary. When similar requests are far apart, delay-hit mitigation adds nothing. The paper’s own breakdown shows that each mechanism activates only where its resource is constrained.
Faster copies consume GPU resources
GPU-assisted I/O uses streaming multiprocessor capacity that could execute model kernels. Strata controls this interference, but the trade remains. The newer batched DMA API is slower for the critical load path yet avoids SM competition, making it attractive for asynchronous GPU-to-CPU backup. A production system may need to select a different mechanism in each direction rather than standardize on one copy primitive.
The scheduler also optimizes aggregate transfer and compute efficiency, not strict fairness. It includes starvation prevention, but individual requests can still receive unequal treatment and violate per-request SLOs. Disk prefetch is only partially addressed, and the current design targets dense KV caches from standard attention rather than sparse or linear-attention state. Most end-to-end comparisons also use CPU memory; disk is isolated to a separate H20 experiment because the baselines have limited disk support.
Serving capacity should be quoted with cache distance
Strata changes what an operator must put beside a throughput number. Model, context length, page size, host memory, CPU-GPU link, cache distance, arrival rate, warm or cold state, and TTFT target all change the answer. A 95% hit rate is not sufficient if the cached bytes arrive after prefill needs them, and a 150 GB/s link is not sufficient if the scheduler cannot form work that consumes it.
The practical design rule is to promote cache readiness to a first-class scheduling signal. HBM capacity decides what can remain local. Host capacity decides what can be reused. Transfer time decides whether reuse beats recomputation. Request order decides whether the first load benefits the next request. Systems that expose only the first two quantities are measuring stored context, not deliverable context.
Source and attribution
This article is an editorial summary prepared by Silicon & Systems. It restates the cited paper’s mechanisms, evaluation conditions, results, and limitations in our own words. No sentences, tables, or figures from the paper are reproduced; all three figures and the card image were created for this article from reported facts. Copyright (c) 2026 the authors. The paper is publicly available through the USENIX OSDI 2026 page.