Every multi-turn conversation with a chatbot re-reads its own history, and every re-read is a prefill the provider pays for in GPU seconds. Moonshot AI, whose Kimi service made 128 K-token contexts a consumer feature in China, decided that this arithmetic should run the other way: computation that has been done once should be stored, found, and shipped rather than repeated. Mooncake, the serving platform described in the company’s FAST 2025 paper with Tsinghua University (and honored as a best paper there), is the systems expression of that decision[1]. It pools the CPU memory, DRAM, SSDs and RDMA NICs that sit idle inside every GPU server into a cluster-wide cache of KV state, schedules every request around where that cache lives, and, by the company’s accounting, lets the same A800 and H800 fleets absorb 115% and 107% more requests than the systems they replaced, while moving over 100 billion tokens a day across thousands of nodes.
We summarize the argument in our own words below. The paper’s slogan, trading more storage for less computation, is also its proof obligation: the trade only clears when the bandwidth to fetch a cached prefix beats the FLOPS to regenerate it, and the paper does that arithmetic in public.
The inequality that justifies the pool
Because each token’s KV entries depend only on the tokens before it, any request sharing a prefix with earlier traffic can skip that prefix’s computation entirely, provided the cached state arrives in HBM fast enough. The paper reduces the question to a bandwidth threshold: for LLaMA3-70B on an 8×A800 machine, fetching a cached prefix pays off for time-to-first-token as long as the cache moves at 6 GB/s or better, rising to 19 GB/s on H800s. Those numbers decide the architecture. A single node’s spare DRAM (about 1 TB, or roughly 3 million tokens at 320 KB of KV state per token) captures less than half of the theoretically achievable hit rate on Kimi’s traces; reaching effectively all of it takes around 50 million tokens of cache, which is 20-plus nodes of pooled memory. Since the required bandwidth is well within a 100 Gbps NIC, the conclusion writes itself: the cache should be global, disaggregated across the cluster, and treated as the primary object the serving system schedules.
Mooncake therefore splits the fleet three ways. Prefill instances and decoding instances are separate pools, in the now-familiar disaggregated style[3][5], and beneath both sits Mooncake Store, the distributed cache holding KV blocks (256 tokens each in the evaluation) keyed by prefix-aware hashes, with hot blocks replicated across nodes and cold ones evicted by LRU. A request flows through four steps: the global scheduler, called Conductor, picks a prefill and a decoding instance; reusable cache blocks stream into the prefill node; the incremental prefill runs, layer by layer, with its new KV state streaming onward to the decoding node as it is produced; and the request joins continuous batching on the decoding side. For long contexts, prefill parallelizes across nodes with chunked pipeline parallelism, which passes activations only at chunk boundaries, avoiding both the per-layer all-reduce cost of cross-node tensor parallelism and the constant re-partitioning that sequence parallelism demands.
A transfer engine built like a storage system
The unglamorous heart of the paper is the machinery that moves cache at memory-class speeds. Each server publishes a topology matrix mapping which NICs sit closest to which memory (DRAM per socket, VRAM per GPU), and every transfer picks its path accordingly, splitting each request into 16 KB slices sprayed across all available NICs, with endpoint pooling and automatic failover around dead links. On hardware with four 200 Gbps NICs the engine sustains 87 GB/s; give it eight 400 Gbps NICs and it reaches 190 GB/s, beating TCP-based transfer by 2.4× and 4.6×, respectively, on a 40 GB cache payload (a 128 K-token context). Below about 100 Gbps of network, the system visibly congests and time-to-first-token degrades toward recomputation, which the authors state plainly as a deployment floor. This transfer engine has been open-sourced and has since become the seed of a community project reused well beyond Kimi.
Scheduling is where the cache stops being a passive pool. For each arriving request, Conductor estimates time-to-first-token per candidate prefill node as the sum of queue time, prefill time (a regression on input length and prefix hit length), and any cache transfer time, then places the request where the total is smallest; if no node can meet the SLO, the request is rejected up front rather than admitted to fail slowly. Hot prefixes migrate automatically: when the best-matching cache sits on an overloaded node, the chosen node pulls a copy, so system prompts and other shared prefixes end up replicated on nearly every prefill instance without any usage forecasting. In the paper’s replay experiment, this cache-and-load-aware policy cuts mean time-to-first-token to 3.07 s against 5.27 s for load balancing and 19.65 s for random placement.
What the numbers say
On replayed production traces with a LLaMA3-70B-class stand-in model across 16 8-GPU nodes, Mooncake’s advantage scales with how much history the workload carries. Against vLLM variants (plain, prefix caching, chunked prefill)[2][4], the conversation trace shows 59 to 498% more requests served within SLO depending on the tightness of the between-token deadline; the tool-and-agent trace, whose repetitive system prompts push the cache ratio to 59%, shows a 42% gain over even local prefix caching; and a long-context synthetic mix gains 40%. The mechanism is visible in the prefill accounting: global caching cuts prefill GPU time by 36 to 64% across the three workloads, and a 95% prefix hit on a 128 K-token input removes 92% of the prefill time. Isolating the store design, the global pool lifts cache hit rates by up to 2.36× over an equal-capacity local cache and cuts prefill computation by up to 48%. A practical footnote for anyone provisioning such a system: sweeping the split between prefill and decoding nodes lands the optimum at roughly one to one, and Moonshot pins the ratio rather than flipping node roles dynamically, since production traffic statistics move slowly.
Boundaries are worth marking. The reproducible numbers come from a dummy model on replayed traces (the production claims are historical statistics against the company’s previous vLLM-based system), the gains are proportional to prefix hit ratios that Kimi’s chat-heavy workloads keep high (40 to 66% in the published traces), and the effective-capacity metric counts requests the system chose to admit, with early rejection doing quiet work in the background. The 320 KB-per-token cache size is also a property of LLaMA3-70B’s attention layout: architectures built to shrink KV state, such as the latent attention we saw in DeepSeek’s co-design story[8], shift the trade’s exchange rate, though in the direction of caching more tokens in the same pool.
What we take from it
Mooncake reads today as the founding paper of the KVCache-centric school of serving design. Its successors are visible across this site’s coverage: Beluga rebuilds the pooled cache on CXL-attached memory so the fetch path stops being a network problem, and Aegaeon[7] extends per-token scheduling from cache placement to model residency itself. What we find most durable is the paper’s insistence on doing economics in the open: a bandwidth inequality that says when the trade clears, a capacity curve that says why local caching cannot clear it, and open-sourced traces (the first public dataset with real prefix-reuse structure) that let outsiders check both. The storage-for-computation trade will keep being renegotiated as attention architectures compress KV state and as memory fabrics change the cost of the fetch, but the ledger this paper introduced is the one those negotiations will be recorded in.
Source and attribution
This article is an editorial summary prepared for Silicon and Systems. It restates the argument of the paper cited below in our own words. No text, figures or tables from the paper are reproduced here, and the figures on this page were created for this summary. The paper appeared at the 23rd USENIX Conference on File and Storage Technologies (FAST 2025) and is available open access from USENIX; (c) 2025 the authors.