An ordered message service is easy to sketch as a log with producers on one side and consumers on the other. The operational shape is less tidy. A single stream may feed thousands of replicas in one cluster and hundreds of thousands globally. Each consumer advances at its own pace, yet every one expects the same byte order and at-least-once delivery. If they all tail the same broker or distributed-file block, read amplification converts a modest producer rate into a network and CPU bottleneck.
Google built Fast ACS for this high-fan-out case[1]. It is not a sub-50-millisecond trading bus and does not claim exactly-once delivery. It targets real-time services such as advertising and shopping whose state must converge within seconds or sub-seconds while surviving regional placement, consumer restarts, and variable demand. The system has run in dozens of production clusters and carries terabit-scale aggregate consumer traffic within a cluster.
The architectural decision is to preserve a file as the source of ordering, then build a separate hot path for repeated tail reads. Colossus remains the durable record and recovery path. A replicated in-memory layer stores the recent chunks and current length. Cross-cluster workers copy bytes with ordinary RPC over the WAN, whereas consumers inside the destination cluster use one-sided remote memory access (RMA). The cache avoids executing a server-side request for every reader, which is the resource that ordinary broker designs exhaust first.
A durable file and a volatile acceleration layer
Each message-stream shard is a sequence of chronologically named append-only files. Producers serialize messages and append bytes; consumers interpret those bytes using their own protocol. File rollover bounds file size without changing the ordering rule. A destination cluster receives a tail copy of the files and exposes the same sequence to local readers.
Fast ACS shadows every recent file into two caches. The data cache holds 4 KB chunks keyed by the globally unique file path and chunk number. The metadata cache holds the visible file length and a best-effort writer lock. Four kilobytes match the observed message granularity and fit one packet on the cluster fabric. Older chunks expire by time-to-live or capacity pressure, so the cache is deliberately not the durable copy.
The write ordering is important. A writer first makes chunk data available, then advances the length visible in metadata. A reader polls that length and requests only bytes below it. The last partially filled chunk is always rewritten from its beginning, which means a reader either observes a correct prefix or misses and falls back. Colossus resolves missing or expired chunks. Correctness therefore does not depend on every cache write succeeding; cache availability changes latency and load, not the authoritative byte sequence.
The cache uses replication with three full copies plus a fractional capacity replica. Reads can choose a relaxed single replica for speed and retry with a quorum when necessary. Metadata deserves stronger handling because publishing a length before its bytes are reachable would create a false promise. The design separates the cheap common case from the validation and fallback paths instead of requiring every access to pay the strongest consistency cost.
This split is more consequential than the choice of cache product. Durable storage absorbs recovery, lagging readers, and exceptional cache loss. Volatile memory absorbs the repeated operation that scales with consumer count. Because the cache serves only readers near the head, it can stay small even when the retained message history is large.
Order at the reader does not require order on the wire
A naïve copy worker would read and transmit each file sequentially. That preserves order but leaves network bandwidth idle whenever one read waits. Fast ACS divides files into chunks and fetches them in parallel, allowing network completion to arrive out of order. The destination writes chunks at their offsets, while consumers expose only the complete prefix indicated by metadata. The externally visible stream stays ordered even though its transfer does not.
This distinction removes a server bottleneck. Global order is a property of the published file position, not a demand that every network packet traverse one serialized queue. Distributing chunk keys also avoids placing every tail read on the node that happens to own the last large file-system block. The paper contrasts this with broker and large-block file designs whose hot partition or tail block receives the entire fan-out.
Geographic distribution adds a second amplification problem. Sending every source byte directly to every destination would multiply expensive intercontinental traffic. Fast ACS builds a copy tree using a minimum-spanning-tree-style optimizer. The first hop commonly crosses a continent, later hops cross metropolitan links, and a typical route is capped at four hops for a few hundred clusters. Reader and writer jobs are separated at every hop so ingress and egress capacity can scale independently.
The tree is a cost optimization with a failure consequence. A slow parent delays every descendant. When a cluster or link is unhealthy, the controller penalizes it, reconstructs the tree, and tends to move it to a leaf. Consumers in the affected cluster can temporarily read from a nearby healthy cluster, giving up local RMA until normal routing returns. The design limits the blast radius but cannot erase the delay already accumulated downstream.
What the production measurements establish
The main controlled experiment used nine data-cache replicas and six metadata-cache replicas in each cluster. Two streams originated in US-central and Europe-west, were copied to 15 destinations, and together produced about 240 Mb/s. Each stream had 120 shards. A consumer read four shards at 8 Mb/s, and the evaluated leaf was three copy-tree hops from its sources.
Fast ACS scaled smoothly to 7,950 active consumers. The data cache reached 70 Gb/s, about 13% above the simple 62 Gb/s estimate, and 4.5 million requests per second, 2.25 times the predicted 2 million. Re-reading a partially filled trailing chunk and retrying relaxed misses with consistent reads explain the difference. These are not bookkeeping details: a capacity plan based only on useful payload would materially underprovision cache packet rate.
At that operating point, monitors reported approximately 500 ms p99 end-to-end delivery. The paper attributes about 120 ms to producer buffering and serial data/index flush, 180 ms to network transit, 100 ms to consumer polling and serial reads, and about 25 ms per copy hop to processing and I/O. The numbers show why labeling the result simply as “RMA latency” would be misleading. The user-visible delay is dominated by batching, polling, WAN distance, and the number of hops rather than a single memory access.
An abrupt jump from zero to 4,000 consumers raised cache read rate to 4.7 million operations per second. Reads and writes shared operating-system network buffers, so the burst delayed cache writes, triggered misses, and forced Colossus fallbacks. Delivery rose by about one second for roughly 150 seconds before recovering. When the same burst occurred along a long copy-tree branch, p99 usually remained below 600 ms but spiked to 1.8 seconds as parent delays propagated.
Fault tests expose a similar boundary. Losing one data-cache replica preserved quorum and caused no read or write failures. Losing two replicas for a key shard created unavailability while a backup rebuilt, drove fallback traffic to 60,000 reads per second, and pushed latency as high as two seconds. Reader and writer process failures were less disruptive because the scheduler restarted their operations in around 50 ms. The cache protects durability by falling back, but the durable layer must be provisioned for the correlated surge that protection creates.
The daily tail is wider than the steady-state graph
Production traffic adds conditions that a scaling chart cannot hold constant. For migrated advertising streams measured over one day, delivery latency was 500 ms at p95, 630 ms at p99, 730 ms at p99.9, and 5.71 seconds at p99.99. The largest stream reached 8.59 seconds at p99.99. Those tails came from varied stream rates, bad machines, software deadlocks, and occasional intercontinental network disruption, not only cache service time.
This evidence changes the operational objective. The system should not be sized merely to keep median delivery low. It needs telemetry that separates producer stalls from transport stalls, cache misses from Colossus throttling, and a slow copy-tree parent from a destination-local problem. The paper reports that some long latency events were initially difficult to distinguish because a cluster-local monitor could not tell whether a producer had stopped or a worker had deadlocked.
The source also reports a production cost reduction exceeding one third after migration, attributed to lower CPU consumption. One-sided RMA matters because thousands of readers no longer require matching server-side request execution. The figure should not be interpreted as a universal infrastructure discount: it reflects Google’s prior stack, RMA-capable machines, cache layout, and workload mix.
Where this pattern fits
Fast ACS is a strong fit when readers greatly outnumber writers, recent bytes receive most reads, and seconds-to-subseconds delivery is acceptable. Its file contract gives lagging consumers a durable recovery route. The small-chunk cache spreads tail reads and the one-sided path reduces serving CPU. A global tree reduces WAN replication cost while preserving a local read path in healthy clusters.
It is a weaker fit for consumers that require less than 50 ms, for streams whose readers scan old history, or for environments without efficient one-sided networking. More aggressive polling can reduce latency but raises metadata-cache QPS. Larger chunks reduce metadata pressure but revive hot-block and overfetch costs. The 4 KB decision is therefore an operating point, not a format that transfers unchanged to every fabric.
At-least-once delivery also places deduplication responsibility on the consumer. Exactly-once semantics across regions would require a transaction spanning producer state, transport, and every consumer checkpoint, which Fast ACS explicitly leaves out. Applications must decide whether an ordered duplicate is tolerable and make updates idempotent where necessary.
The most reusable lesson is the separation of contracts. Ordering comes from monotonic file positions. Durability comes from the distributed file system. Fan-out throughput comes from replicated memory and RMA. Geographic efficiency comes from a copy tree. Treating those properties as independent layers makes each failure visible and gives the system a conservative fallback. It also reveals the capacity equation operators must use: useful producer bytes are only the beginning; partial-chunk rereads, quorum retries, downstream fan-out, and fallback bursts determine the real network and request budget.
Source and copyright notice
This article is an editorial analysis by Silicon & Systems. It restates the source paper’s architecture, production evidence, and limitations in our own words. No source sentence, table, or figure is reproduced; the explanatory figure was created for this article. The complete paper is available from the USENIX ATC 2025 presentation page. Copyright remains with the authors, 2025.