An online language model has at least two clocks. Time to first token (TTFT) covers admission, cache lookup and prefill. Time per output token (TPOT) covers a repeated decode path whose latency may drift one step at a time. Modern services add prefix caches, prefill-decode disaggregation, tensor parallelism and expert parallelism[2][3]. A request that misses its SLO can therefore be delayed by scheduling, communication, a slow kernel, an overloaded link, cache behavior or interference in one narrow interval.
Detailed tracing appears to be the obvious answer, but tracing every kernel and communication event can consume enough CPU, GPU memory and storage to disturb the service. Alibaba’s StriaTrace chooses a selective evidence path[1]. It records low-cost structure continuously and expands detail when an abnormality appears. In production for six months, the system instrumented more than 1,700 instances across two flagship services processing over 180 million requests per day. Reported overhead stays below 1%, 97.8% lower than the evaluated alternatives, and the system identified 19 classes of root cause.
An inference trace needs a different shape
Training observability usually summarizes long iterations and repeated collective phases. Online inference mixes requests with different prompt lengths, output lengths, cache states and arrival times. Prefill and decode may run on different machines. A short anomaly can disappear in an iteration average, while a full event stream produces too much data to retain.
StriaTrace starts from three production rules. First, synchronization points expose waiting and therefore preserve causal boundaries. Second, only events on a request’s critical path can extend its latency. Third, fine-grained events are most valuable near an abnormal interval. The always-on layer records enough timing to reconstruct stages and detect that a step moved outside its expected behavior. A triggered layer then captures deeper kernel and communication detail for the suspicious region.
This design is closer to a flight recorder than a continuous video. The sparse record answers when and where the service slowed. The expanded record answers what consumed the interval. Because the trigger is driven by request and step behavior, it can catch intermittent events that disappear before an operator attaches a general-purpose profiler.
From a slow step to a cause
Collection alone does not explain a miss. StriaTrace builds a regression-based dynamic roofline for inference steps. A conventional roofline relates arithmetic work, data movement and hardware limits. Here the expected boundary must also change with request state, batch composition and model phase. The model distinguishes a step that is slow because it contains more legitimate work from one that underperforms for its conditions.
The diagnosis stage correlates the abnormal step with events and metrics across the serving stack. A compute-bound departure may point to kernel execution or contention. A communication-shaped departure can be tested against collective and network signals. Waiting at a synchronization boundary identifies the component that arrived late rather than the ranks that merely waited for it. The output is not a universal proof of causality, but a narrowed, ranked explanation with the trace context an engineer needs.
This matters in a distributed inference service because a symptom is replicated. If one expert-parallel rank slows, every peer can show a longer collective. Looking at utilization alone produces many identical victims. Critical-path and synchronization evidence separates the late producer from the waiters.
What the production numbers establish
The deployment scale is unusually specific: over 1,700 instances, two large services, more than 180 million requests each day, and six months of production operation. It includes both monolithic and prefill-decode-disaggregated deployments, as well as tensor- and data-parallel configurations. Across development, testing and release cycles, StriaTrace diagnosed hundreds of abnormalities belonging to 19 distinct root causes.
The overhead result also needs its denominator. The paper reports less than 1% tracing overhead and a 97.8% reduction relative to alternative tracing approaches. The second number is not a 97.8% latency improvement for inference. It describes the reduction in observability overhead. Its value is operational: tracing can remain enabled rather than being attached after the anomaly has passed.
The blind spots remain important
Selective tracing depends on the trigger. If a quality regression changes generated content without creating a timing abnormality, StriaTrace is not the detector. A fault that changes every request slowly may also become part of the learned normal boundary unless release comparison or an external SLO catches it. Correlation narrows a diagnosis, but correlated telemetry can still share a downstream cause.
There is another practical limit. The paper reports root-cause classes and production cases, not a public labeled corpus from which an independent reader can calculate diagnosis precision and recall. The deployment evidence is strong for usefulness and coverage, while the exact false-diagnosis rate is not summarized as one comparable metric. Operators evaluating a similar design should ask how triggers are calibrated, how long detailed windows are retained, and how often the ranked cause led to the final fix.
StriaTrace’s transferable idea is an observability budget. At inference scale, recording everything is not neutral and recording only aggregate counters is not explanatory. The useful middle ground preserves causal structure continuously and purchases detail only when the request gives a reason. That makes diagnosis a designed part of the serving path rather than an emergency profiler session.
A trace budget should follow causality, not traffic volume
The production result supports a more general observability rule. The value of an event is not proportional to how often it occurs. Synchronization points are sparse but tell the system which component could have delayed the critical path. A detailed kernel event is plentiful but often irrelevant until an SLO miss has narrowed the time and request. StriaTrace spends its always-on budget on causal structure, then spends detail only after the posterior probability of usefulness rises. That is why the design can lower tracing cost without merely sampling fewer requests.
This approach also introduces a new failure mode: the trigger can be wrong. If the sparse layer fails to retain the synchronization edge that explains an anomaly, no amount of detailed tracing after the trigger will reconstruct it. The production evidence should therefore be read with two coverage numbers, not one overhead number. Operators need the fraction of true incidents that generated a usable trigger and the fraction of triggered windows that produced a correct diagnosis. The published 19 root-cause classes demonstrate breadth, while an adopter still has to validate coverage for its own kernels, parallelism schemes, and request-routing path.
For capacity planning, the relevant output is not a prettier trace. It is the reduction in time spent below an SLO and the engineering hours required to restore it. A mature deployment should link each diagnosis to mitigation, recurrence, and customer impact, then retain exemplars as regression tests. This turns tracing from a postmortem tool into a feedback system for releases. StriaTrace’s insight is that observability at inference scale is a staged decision process: preserve just enough structure to know where to look, open the expensive window only when evidence warrants it, and close the loop by proving that the same cause no longer reaches production.
Tail latency also needs a causal denominator. A p99 value mixes queueing, prefill, token-by-token decode, cross-instance transfer, kernel execution, and network delay. Optimizing the largest average component can leave the rare critical path unchanged. StriaTrace’s dynamic boundary is useful because it asks whether a component was slow relative to the request’s own operating condition, not whether it exceeded one fleet-wide constant. Adopters should verify that the model remains calibrated across model versions, sequence lengths, batching policies, and hardware generations. A boundary trained on yesterday’s traffic can convert normal workload drift into an incident flood.
Selective detail creates a data-governance advantage and obligation. Capturing fewer events reduces storage and the surface on which prompts, identifiers, or model behavior may leak, but the triggered windows are precisely the abnormal requests operators are most likely to retain and inspect. A production design should specify redaction, access, and retention at the same granularity as the trace stages. The analytical point is that observability cost is not only runtime overhead. It includes storage growth, privacy exposure, and the human attention consumed by false triggers, all of which must be counted when comparing an always-on tracer with a selective one.
Source and attribution
This article is an editorial summary prepared for Silicon & Systems. It restates the cited paper in our own words. No text, tables or figures from the paper are reproduced; both figures were created for this article. USENIX provides the paper through its open presentation page. Copyright (c) 2026 the authors.