Loading a large language model is not one sequential copy from an SSD into accelerator memory. A serving framework opens weight files, parses metadata, assigns tensors to workers, reads disjoint ranges through the kernel page cache, and transfers those pages across a NUMA topology. The SSD can be idle while the framework does CPU work, then become the critical path when a worker reaches data that the kernel did not predict. Huawei’s FAST 2026 paper treats that mismatch as a cache-policy problem rather than another reason to replace the model format or inference runtime[1].
The paper contributes two layers. Programmable Page Cache (PPC) lets a userspace policy control prefetch and eviction while applications continue to use ordinary POSIX file access. Model-Accelerated I/O (MAIO) is a PPC policy that records the repeatable I/O sequence of one inference-service configuration. On later starts, it prefetches from the current position, places pages in memory near the accelerator that will consume them, and discards host pages after their final transfer.
Across the authors’ five-model test, MAIO reduces model-loading latency by as much as 79% with sufficient host memory and 74% under a 64 GB page-cache limit. In a separate customer deployment, loading DeepSeek-R1-671B across 16 Ascend NPUs falls from 649 seconds to 452 seconds. These are conditional maxima, not a promise for every framework or GPU. The more durable result is architectural: a transparent filesystem path can still use application-specific knowledge if the knowledge is expressed as a replaceable policy.
Native readahead sees files, not an inference service
The Linux page cache normally predicts access from recent file reads. That works well for conventional sequential streams, but model loading interleaves metadata parsing, tensor construction, several files, and multiple accelerator workers. In the paper’s Qwen2.5-72B and Llama-70B measurements, average SSD bandwidth during loading is about 17% of the observed maximum. The drive is not inherently too slow. The software fails to issue enough correct work early enough to overlap I/O with framework initialization and tensor processing.
NUMA placement creates a second penalty. A kernel readahead worker allocates pages near the CPU that executes the prefetch, even though another CPU socket and accelerator may consume them. The authors’ controlled tmpfs experiment reduces model-loading latency by about 20% when each accelerator reads from memory on its corresponding NUMA node instead of one shared node. Ordinary readahead has no model-worker identity with which to make that decision.
Memory pressure creates a third mismatch. Once a weight page has reached accelerator HBM, its host copy often has no value for the lifetime of that service. However, an LRU-like policy only observes generic page recency. With page-cache capacity near 45% of the Qwen2.5-72B model size, the native path takes 38% longer than the sufficient-memory case. Useful future pages compete with weight pages whose only consumer has already finished.
These problems share one missing input: the kernel knows which byte was requested, but not the service-level sequence, destination accelerator, or point at which a host page becomes dead.
PPC inserts policy at the cache miss
PPC uses a read-only routing filesystem (RFS) stacked above an existing filesystem such as EXT4 or XFS. The mounted namespace mirrors the underlying files, so the serving container reads the same model through the PPC path. RFS checks the native page cache. A hit continues through the underlying filesystem; a miss emits the file handle, offset, length, and process identifier to a userspace procedure-call queue before the native read completes.
The event path is non-blocking and uses per-core queues. In userspace, the cache policy runtime (CPRT) maps the event to a registered policy. The policy returns lists of regions to load or reclaim rather than manipulating kernel structures directly. PPC performs loading through a core-bound thread pool and requests clean-page reclamation through standard kernel interfaces. The policy can fail without placing custom logic on the application’s call stack or inside the underlying filesystem implementation.
This boundary is the principal compatibility claim. PPC is an independent kernel module, not an upstream patch to VFS or an inference-framework fork. A directory can receive one policy, and the policy can be switched without changing model code. The current implementation is read-only, which matches weight loading but limits the claim. Extending the same mechanism to writable paths would have to define dirty-page ordering, writeback errors, and durability, none of which this evaluation establishes.
An empty PPC policy measures the cost of the interception itself. For a 1 MB memcpy-after-mmap workload, the paper reports up to 3.7% overhead on EXT4 and 6.4% on XFS, compared with up to 14% and 15% for the evaluated RFUSE path. PPC process memory stays around 30 MB. Event-listening CPU overhead ranges from about 1% to 11% as concurrency changes. These figures price the control plane before MAIO’s useful prefetch work is counted.
MAIO turns a deployment specification into an I/O trace
A Model-as-a-Service platform already identifies a deployment by model and runtime parameters such as tensor-parallel degree and prefill-decode arrangement. MAIO hashes that specification into a service identifier. The first run observes cache misses and records, for each accelerator worker, a sequence of file paths, offsets, and lengths. Later instances with the same identifier load the resulting template before startup.
This is neither statistical prediction nor a universal model profile. It assumes that the same service specification repeats the same weight-read order. When a model file, parallel strategy, or worker mapping changes, the platform must generate another template. The lifecycle of the I/O evidence therefore belongs with the deployment specification, not with the physical server.
The metadata is small because consecutive reads are consolidated. Templates range from 11 KB for a 15 GB Qwen2.5-7B deployment to 118 KB for a 132 GB Llama-70B deployment. The two-node, 662 GB DeepSeek-R1-671B case uses 545 KB. Storage capacity is not the constraint. Correct invalidation is: a stale trace can request the wrong ranges early, consume bandwidth, and undermine the locality that the policy was meant to create.
Three mechanisms spend the trace differently
Interruptible prefetch begins at the current miss and may issue all remaining I/O in that worker’s template. The apparently aggressive choice is bounded by the PPC loader. A new foreground miss advances the cursor and interrupts obsolete prefetch work. Memory pressure or exhausted loader threads can also stop the request. The policy therefore tries to fill idle SSD time without insisting that yesterday’s prefetch queue finish before today’s foreground read.
XPU-affine loading maps the template’s logical worker to the actual accelerator assigned to this instance, identifies the accelerator’s nearby NUMA node, and allocates prefetched pages there. This does not make PCIe transfer free. It removes an avoidable cross-socket hop and preserves the model framework’s worker-to-device relationship below the POSIX interface.
Burn-after-reading (BAR) eviction follows the opposite cursor. Weight data before the current read position is likely to have reached HBM and can leave the host cache. MAIO preserves a default 1 GB distance behind the foreground position to avoid reclaiming pages that have been read by the host but not completely transferred. That distance is empirical. A framework with deeper asynchronous staging or repeated host access needs a different safety margin.
The ablation clarifies when each mechanism matters. With enough memory, interruptible prefetch cuts loading latency by more than 65% for Qwen2.5-72B and Llama-70B; affinity adds more than 8.5% on top, while BAR changes little. Under the 64 GB limit, the three additions contribute sequential improvements of more than 47%, 6%, and 19% for Qwen2.5-72B, and more than 44%, 4%, and 23% for Llama-70B. Eviction becomes a performance feature only when retaining the wrong page displaces future work.
The evaluation rewards overlap, not only storage bandwidth
The main test node has four 48-core Kunpeng 920 processors, eight Ascend 910B2 NPUs, 1 TB of DRAM, and a 3.75 TB SSD. Software includes Linux 5.10, PyTorch 2.5.1, and vLLM-Ascend 0.9.2. Most experiments use four NPUs and five Qwen2.5 or Llama models from 7B to 72B parameters. The memory-constrained case uses cgroups to cap memory available for loading at 64 GB.
MAIO is compared with the native cache, eager whole-model prefetch, complete DRAM precaching, and an NPU adaptation of ServerlessLLM. The last comparison is narrower because the adapted system runs with Transformers rather than vLLM and requires enough free pinned host memory for the model. MAIO’s 17% maximum advantage over that path for large models does not establish superiority over every GPU implementation. It shows that beginning I/O before the framework’s explicit loading phase can overlap more startup work.
End-to-end service startup improves by up to 38% against native loading when memory is sufficient and up to 51% in the constrained case. The smaller percentages than model-loading results are expected because container initialization and service setup remain. In an elastic sequence of short-lived services, MAIO raises token throughput by up to 13% with sufficient memory and 28% with constrained memory against the native path. As each service remains active longer, startup occupies less of the denominator and the advantage narrows.
The customer system provides a separate reality check. Two nodes with eight Ascend 910B3 NPUs each load DeepSeek-R1-671B from local SSDs. MAIO reduces loading from 649 to 452 seconds and even finishes before the reported 561-second fully cached DRAM case. The paper attributes the difference to earlier overlap and NUMA-aware transfer. It does not mean SSD is faster than DRAM in isolation; the compared startup pipelines schedule CPU parsing, file access, and device copies differently.
Repeatability is both the source of speed and the boundary
MAIO is strongest when model weights are immutable, service specifications are versioned, and the platform launches the same configuration often enough to amortize one profiling run. It is less certain when tensor layout changes dynamically, workers read conditionally, adapters are composed per request, or several services compete for one SSD. The paper proposes using the platform’s cgroups for contention, but a multi-tenant QoS evaluation remains future work.
Operators should measure template accuracy before treating the policy as infrastructure. Useful counters include prefetched bytes consumed before eviction, interrupted I/O, cross-NUMA traffic, reclaimed pages later reread, and startup tail latency under concurrent launches. A high hit rate can still be inefficient if it fills memory too early and suppresses another service’s critical reads.
Failure behavior deserves the same attention. If CPRT stops, PPC can fall back to the kernel policy, but the serving platform needs to observe that transition. A malformed template should degrade performance rather than data integrity because ordinary filesystem reads still define correctness. That is a valuable separation, provided monitoring distinguishes a transparent fallback from a successful optimized start.
The architectural lesson extends beyond LLM weights. Page-cache policy can sit between a stable application contract and a native filesystem without forcing either side to own every workload-specific optimization. The contract works because the policy recommends prefetch and clean-page eviction while the kernel remains responsible for correct reads. PPC and MAIO show the performance available at that boundary. Their production value will be determined by template governance, contention control, and how reliably the platform recognizes when yesterday’s I/O sequence no longer describes today’s service.
Source and copyright notice
This article is an independent editorial analysis of the FAST 2026 paper by Yubo Liu, Hongbo Li, Xiaojia Huang, Yongfeng Wang, Hanjun Guo, Hui Chen, Yuxin Ren, and Ning Jia of Huawei Technologies. The source is openly available from the USENIX presentation page. We restated the mechanisms and measurements in our own words and created the figure specifically for this article. No source text, table, or figure is reproduced. Copyright (c) 2026 the paper authors.