Warehouse-scale efficiency has an unusual denominator. A code change that saves a tiny fraction of CPU in one common function can return more capacity than a dramatic microbenchmark win in an obscure path. Finding that function, understanding its semantics, and delivering a safe edit across a monorepo are separate problems. A language model addresses only part of the middle problem.
ECO treats optimization as a production workflow. It learns recurring anti-patterns from historical performance changes and external guidance, uses fleet profiles to identify where those patterns are expensive, generates candidate edits, and passes them through existing tests, reviewers, owners, and monitoring. Google reports more than 25,000 changed lines across over 6,400 commits during a year of deployment.
The system’s important abstraction is the commit, not the generated completion. Every proposal must have a target justified by measured cost, a narrow transformation, automated evidence, an accountable reviewer, and a rollback path. This framing allows increasingly capable models to improve one stage without granting them production authority.
Search begins with fleet cost
Static scanners can flag every use of a costly-looking idiom, but most occurrences may be cold. ECO starts from Google Wide Profiling data and maps aggregate cycles, cache misses, or application-specific metrics into source functions. It recursively attributes cost through call trees while pruning shared utilities that would otherwise absorb every signal.
The localization step changes expected value. If two functions contain the same unnecessary allocation, the one executed across a large service fleet receives priority. It also constrains the amount of source sent to the model and reduces irrelevant proposals.
Profiles are imperfect. Inlining, symbolization, sampling bias, and shared libraries can misattribute cost. ECO therefore uses them to rank opportunities, not prove that an edit is correct. A candidate must later establish performance under a controlled benchmark or production experiment.
Historical changes become retrieval material
ECO mines commits whose messages and review context indicate an efficiency improvement. It removes reverted or poor examples and groups remaining transformations into patterns such as reserving vector capacity, avoiding unnecessary copies, replacing unsuitable maps, or eliminating redundant sorting.
These examples serve two roles. They define an anti-pattern dictionary used to search source, and they provide semantically similar before-and-after code for model context. Retrieval grounds a proposal in changes that engineers previously accepted inside the same codebase.
The data still carries selection bias. A historical commit may be specific to one container, ownership rule, or compiler era. A successful edit is not automatically a universally safe rewrite. ECO keeps source types and local context and asks the model for a candidate rather than performing blind text substitution.
The architecture can use different models. The evaluated deployment used a Gemini-family code model, but the paper emphasizes that source localization, retrieval, validation, and rollout are separable from model identity. Better generation can improve yield without redesigning production controls.
Conservative generation is an engineering choice
The evaluation compares zero-shot, few-shot, chain-of-thought, and tool-like prompting on microbenchmarks and production code. More elaborate prompts can find larger speedups but also create broader and less predictable changes. A conservative patch that touches fewer lines can be easier to validate and may deliver most of the available saving.
ECO generates multiple candidates, checks whether hunks apply, builds and tests them, and uses similarity or quality signals to rank outputs. A self-review stage asks the model to identify semantic or performance risks in its own patch. These filters remove obvious failures, but they do not replace workload measurement.
CodeBLEU and related similarity metrics can help select edits close to known good transformations. They are not correctness proofs. High similarity can preserve the same hidden bug, while a novel but correct optimization may score lower. The system therefore treats model scores as triage signals.
Human review remains part of the measured system
Every production change is sent to a human code owner. The paper’s table separates changes accepted without feedback, accepted after feedback, rejected by users, and later reverted. Roughly 40% of copy-pattern proposals and 41% of vector-pattern proposals were submitted without required reviewer correction; map changes were substantially lower, near 5%, illustrating that pattern semantics matter.
This is not a weakness hidden outside the evaluation. Review burden is a key throughput metric. A tool that emits many low-quality patches transfers optimization work into reviewer interruption and can lose organizational trust. ECO uses reviewer outcomes to refine prompts, patterns, and validation.
Accepted code proceeds through normal testing and production monitoring. The reported rollback rate caused by regression is below 0.5%. That low rate reflects the whole pipeline, including cautious targeting and human approval, rather than model reliability in isolation.
Capacity savings need attribution
Production efficiency is harder to measure than a local benchmark. Workload mix, traffic, compiler versions, and unrelated releases change concurrently. ECO integrates with existing experiment and monitoring systems to compare relevant service metrics after rollout and to revert regressions.
The paper reports deployment breadth rather than one headline fleet-wide percentage. More than 6,400 commits changed over 25,000 lines, spanning copy, map, vector, and other patterns. Generating 10,000 commits is estimated to cost about $3,000 in model inference, which is small relative to the fleet resources and engineering time at stake.
Inference cost, however, is not total cost. Profile processing, builds, tests, reviewer attention, canary capacity, and incident risk belong in the denominator. A useful dashboard should track candidates per accepted commit, reviewer minutes, test compute, measured resource savings, rollback frequency, and time to payback.
The difficult failures are semantic
An allocation reduction can change iterator validity, exception behavior, memory lifetime, or synchronization. Replacing a container can alter ordering and worst-case complexity. Removing a copy can expose aliasing. Unit tests may not cover rare concurrency or overload behavior.
Performance changes can also move costs. Saving CPU while increasing memory, cache footprint, power, tail latency, or network traffic is not an unconditional improvement. ECO’s target metrics and review template must state the resource being optimized and the budgets that cannot regress.
Generated code has provenance and security implications. The service needs access controls for source, prompt logging without leaking secrets, model-version records, and reproducible patch context. Owners should be able to inspect why a function was selected and which examples influenced the proposal.
Scale the control loop, not merely generation
ECO’s transferable design is a funnel. Begin with measured fleet cost, retrieve an organization-specific transformation, generate a small edit, eliminate mechanical failures, obtain owner judgment, measure a canary, and keep rollback automatic. Each stage narrows risk while preserving evidence for the next stage.
Organizations without Google’s monorepo and profiling infrastructure can reproduce parts using continuous profiles, source indexes, code review, and CI. The likely bottleneck is data integration rather than access to a language model. Historical changes must be labeled, symbols mapped, owners found, and performance experiments made routine.
The paper shows that AI-assisted optimization becomes valuable when it is boring enough to behave like normal maintenance. The model expands the number of candidate improvements; the surrounding system decides which ones deserve to become software.
Portfolio management determines the realized saving
An efficiency program should rank patches by expected annual resource return multiplied by acceptance probability, then divide by engineering and validation cost. A tiny edit in a fleet-wide primitive may outrank a larger optimization in one service. Conversely, a risky shared-library change can have such a high blast radius that separate owners and a longer canary are justified even when expected savings are large.
Metrics should distinguish proposed, buildable, reviewed, deployed, retained, and measured-effective changes. Counting generated diffs rewards noise, while counting merged commits can hide changes whose production benefit disappears. ECO’s year-long record is useful because it exposes several of these stages. The next maturity step is a durable ledger connecting each retained patch to its measured CPU, memory, energy, or latency return and to the confidence of that attribution.
Source and copyright notice
This article is an editorial analysis by Silicon & Systems. It restates the system design, production measurements, review outcomes, and limitations in our own words. No source sentence, table, or figure is reproduced; the figure was created for this article. The paper is available from the USENIX OSDI 2026 presentation page. Copyright remains with the authors, 2026.