Every multi-turn conversation with a chatbot re-reads its own history, and every re-read is a prefill the provider pays for in GPU seconds. Moonshot AI, whose Kimi service made 128 K-token contexts a consumer feature in China, decided that this arithmetic should run the other way: computation that has been done once should be stored, found, and shipped rather than repeated. Mooncake, the serving platform described in the company’s FAST 2025 paper with Tsinghua University (and honored as a best paper there), is the systems expression of that decision[1]. It pools the CPU memory, DRAM, SSDs and RDMA NICs that sit idle inside every GPU server into a cluster-wide cache of KV state, schedules every request around where that cache lives, and, by the company’s accounting, lets the same A800 and H800 fleets absorb 115% and 107% more requests than the systems they replaced, while moving over 100 billion tokens a day across thousands of nodes.
We summarize the argument in our own words below. The paper’s slogan, trading more storage for less computation, is also its proof obligation: the trade only clears when the bandwidth to fetch a cached prefix beats the FLOPS to regenerate it, and the paper does that arithmetic in public.
The inequality that justifies the pool
Because each token’s KV entries depend only on the tokens before it, any request sharing a prefix with earlier traffic can skip that prefix’s computation entirely, provided the cached state arrives in HBM fast enough. The paper reduces the question to a bandwidth threshold: for LLaMA3-70B on an 8×A800 machine, fetching a cached prefix pays off for time-to-first-token as long as the cache moves at 6 GB/s or better, rising to 19 GB/s on H800s. Those numbers decide the architecture. A single node’s spare DRAM (about 1 TB, or roughly 3 million tokens at 320 KB of KV state per token) captures less than half of the theoretically achievable hit rate on Kimi’s traces; reaching effectively all of it takes around 50 million tokens of cache, which is 20-plus nodes of pooled memory. Since the required bandwidth is well within a 100 Gbps NIC, the conclusion writes itself: the cache should be global, disaggregated across the cluster, and treated as the primary object the serving system schedules.
Mooncake therefore splits the fleet three ways. Prefill instances and decoding instances are separate pools, in the now-familiar disaggregated style[3][5], and beneath both sits Mooncake Store, the distributed cache holding KV blocks (256 tokens each in the evaluation) keyed by prefix-aware hashes, with hot blocks replicated across nodes and cold ones evicted by LRU. A request flows through four steps: the global scheduler, called Conductor, picks a prefill and a decoding instance; reusable cache blocks stream into the prefill node; the incremental prefill runs, layer by layer, with its new KV state streaming onward to the decoding node as it is produced; and the request joins continuous batching on the decoding side. For long contexts, prefill parallelizes across nodes with chunked pipeline parallelism, which passes activations only at chunk boundaries, avoiding both the per-layer all-reduce cost of cross-node tensor parallelism and the constant re-partitioning that sequence parallelism demands.
A transfer engine built like a storage system
The unglamorous heart of the paper is the machinery that moves cache at memory-class speeds. Each server publishes a topology matrix mapping which NICs sit closest to which memory (DRAM per socket, VRAM per GPU), and every transfer picks its path accordingly, splitting each request into 16 KB slices sprayed across all available NICs, with endpoint pooling and automatic failover around dead links. On hardware with four 200 Gbps NICs the engine sustains 87 GB/s; give it eight 400 Gbps NICs and it reaches 190 GB/s, beating TCP-based transfer by 2.4× and 4.6×, respectively, on a 40 GB cache payload (a 128 K-token context). Below about 100 Gbps of network, the system visibly congests and time-to-first-token degrades toward recomputation, which the authors state plainly as a deployment floor. This transfer engine has been open-sourced and has since become the seed of a community project reused well beyond Kimi.
Scheduling is where the cache stops being a passive pool. For each arriving request, Conductor estimates time-to-first-token per candidate prefill node as the sum of queue time, prefill time (a regression on input length and prefix hit length), and any cache transfer time, then places the request where the total is smallest; if no node can meet the SLO, the request is rejected up front rather than admitted to fail slowly. Hot prefixes migrate automatically: when the best-matching cache sits on an overloaded node, the chosen node pulls a copy, so system prompts and other shared prefixes end up replicated on nearly every prefill instance without any usage forecasting. In the paper’s replay experiment, this cache-and-load-aware policy cuts mean time-to-first-token to 3.07 s against 5.27 s for load balancing and 19.65 s for random placement.
What the numbers say
On replayed production traces with a LLaMA3-70B-class stand-in model across 16 8-GPU nodes, Mooncake’s advantage scales with how much history the workload carries. Against vLLM variants (plain, prefix caching, chunked prefill)[2][4], the conversation trace shows 59 to 498% more requests served within SLO depending on the tightness of the between-token deadline; the tool-and-agent trace, whose repetitive system prompts push the cache ratio to 59%, shows a 42% gain over even local prefix caching; and a long-context synthetic mix gains 40%. The mechanism is visible in the prefill accounting: global caching cuts prefill GPU time by 36 to 64% across the three workloads, and a 95% prefix hit on a 128 K-token input removes 92% of the prefill time. Isolating the store design, the global pool lifts cache hit rates by up to 2.36× over an equal-capacity local cache and cuts prefill computation by up to 48%. A practical footnote for anyone provisioning such a system: sweeping the split between prefill and decoding nodes lands the optimum at roughly one to one, and Moonshot pins the ratio rather than flipping node roles dynamically, since production traffic statistics move slowly.
Boundaries are worth marking. The reproducible numbers come from a dummy model on replayed traces (the production claims are historical statistics against the company’s previous vLLM-based system), the gains are proportional to prefix hit ratios that Kimi’s chat-heavy workloads keep high (40 to 66% in the published traces), and the effective-capacity metric counts requests the system chose to admit, with early rejection doing quiet work in the background. The 320 KB-per-token cache size is also a property of LLaMA3-70B’s attention layout: architectures built to shrink KV state, such as the latent attention we saw in DeepSeek’s co-design story[8], shift the trade’s exchange rate, though in the direction of caching more tokens in the same pool.
What we take from it
Mooncake reads today as the founding paper of the KVCache-centric school of serving design. Its successors are visible across this site’s coverage: Beluga rebuilds the pooled cache on CXL-attached memory so the fetch path stops being a network problem, and Aegaeon[7] extends per-token scheduling from cache placement to model residency itself. What we find most durable is the paper’s insistence on doing economics in the open: a bandwidth inequality that says when the trade clears, a capacity curve that says why local caching cannot clear it, and open-sourced traces (the first public dataset with real prefix-reuse structure) that let outsiders check both. The storage-for-computation trade will keep being renegotiated as attention architectures compress KV state and as memory fabrics change the cost of the fetch, but the ledger this paper introduced is the one those negotiations will be recorded in.
Cache economics begin with avoided prefill
A KV cache has value only when retrieving it costs less than recreating it under the active service objective. The comparison must include more than GPU arithmetic. Recompute consumes prefill capacity, may delay unrelated requests in the same batch, and can increase queueing on a cluster already near its SLO boundary. Retrieval consumes host or SSD bandwidth, network capacity, destination memory, and scheduler attention. The cheaper path changes with prefix length, cache location, link load, and the amount of prefill work waiting elsewhere.
This is why a single cache-hit ratio is not enough. An operator should record avoided prefill tokens, bytes transferred, transfer time, destination queueing, and the TTFT effect for both the reused request and neighboring requests. A short prefix can count as a hit while saving less time than its metadata and movement cost. A long prefix can be valuable even when it returns infrequently because one reuse avoids substantial computation. The cache policy should price saved GPU time against the full retrieval path.
The same ledger exposes when SSD is a capacity tier rather than a latency tier. SSD can retain a much larger working set, but a request with a tight first-token objective may not be able to wait for it. The scheduler can prefetch an entry whose reuse is predictable, route the request to a prefill instance near an existing copy, or choose recomputation. Treating all three as cache hits hides the decision that actually determines service quality.
Overload control is part of the architecture
Mooncake’s production setting does not assume that every request must be served immediately. When demand exceeds the feasible region, accepting all work can make every request miss its SLO. Prediction-based early rejection protects useful throughput by declining work before it consumes expensive partial execution. This is not a secondary safety valve. It is part of the capacity model for a service whose prefill and decode resources are separated and whose requests carry different context lengths.
An admission predictor must be calibrated against consequences. A false acceptance wastes compute and may push several admitted requests past their objectives. A false rejection leaves capacity unused and denies a request that could have completed. The costs are asymmetric and can differ by customer class. Production policy should therefore report acceptance rate, SLO attainment among accepted requests, false-rejection estimates, and completed requests per accelerator-hour under the same load trace.
Disaggregation adds another failure mode to overload. Prefill may finish while no decode instance has enough KV capacity to accept the result. The system then owns a large intermediate state with nowhere useful to place it. Admission control should reserve or probabilistically account for downstream decode capacity before launching expensive prefill. Otherwise, the architecture moves queueing from requests to KV objects without reducing it.
The cache pool needs locality and failure domains
Pooling idle CPU, DRAM, and SSD across GPU servers turns stranded capacity into a service, but distance matters. A cache copy on the same host, the same rack, or another failure domain has different transfer time and availability. Replicating a hot prefix can reduce network hotspots, while excessive replication consumes the capacity that made pooling attractive. The Conductor therefore needs a placement policy that balances predicted reuse, link contention, and resilience.
Operators should test node loss, cache-server restart, and partial network isolation while requests are active. Losing a cache entry should normally cause recomputation rather than an incorrect answer, but the latency consequence can still cascade through an overloaded service. Metadata must not claim that an entry is available after its last replica disappears. Versioning is also required when prompts, model weights, tokenization, or attention implementation change, because a byte-compatible key does not guarantee semantically compatible KV state.
The failure boundary clarifies what disaggregated caching can promise. It is an optimization layer, not the authoritative record of a conversation. The service should remain correct when a cached prefix is absent and should degrade in a measured way when a tier is slow. This makes recovery testing simpler: verify cache invalidation and fallback first, then measure how much capacity disappears under each failure.
Serving should be planned in completed conversations
Mooncake demonstrates that adding storage can buy back compute, but storage utilization is not the final objective. Prefill throughput, decode throughput, KV movement, and rejection decisions meet at the user-visible conversation. Capacity planning should therefore start from context and output-length distributions, reuse distance, TTFT and token-latency objectives, and the fraction of offered requests the service intends to accept.
From those inputs, the operator can derive prefill GPUs, decode GPUs, cache bytes, and fabric demand for each traffic class. The ratios should be revisited as model architecture changes. Grouped-query attention, latent attention, or a smaller KV representation can lower cache bytes per token and change whether disaggregation is worthwhile. Faster prefill kernels can also make recomputation more competitive. A system named after its cache must still be evaluated against the option of not retrieving the cache.
We read Mooncake as a shift from cache capacity to cache economics. The architecture is valuable because it puts reuse, movement, overload, and phase-specific compute in one scheduling decision. Its production result cannot be reduced to a universal percentage for every long-context service. The transferable method is to measure which prefix saves which computation, whether it can arrive before the SLO, and how that choice changes the number of conversations completed by the whole cluster.
Source and attribution
This article is an editorial summary prepared for Silicon & Systems. It restates the argument of the paper cited below in our own words. No text, figures or tables from the paper are reproduced here, and the figures on this page were created for this summary. The paper appeared at the 23rd USENIX Conference on File and Storage Technologies (FAST 2025) and is available open access from USENIX; (c) 2025 the authors.