Silent data corruption (SDC) is difficult because the first visible error is rarely the first wrong operation. One GPU flips a bit in logic not protected by ECC, a collective distributes the result, and the loss diverges many layers or iterations later. The same symptom can also come from a software race. An operator must answer three separate questions: did a corruption occur, which device or component produced it, and where did two executions first become different?
ByteDance and its university collaborators presented three adjacent OSDI 2026 papers that map directly onto those questions[1][2][3]. AEGIS watches live training and verifies suspicious results. SDCHunter isolates a defective GPU by replaying the workload and input that triggered the incident. OpGuard compares executions at stable operator boundaries and identifies the first bitwise mismatch. Read together, they form an operational chain rather than three interchangeable detectors.
Why the usual hardware test misses the culprit
The SDCs in the Wild study examines 23 confirmed defective GPUs removed from a production cluster[1]. Synthetic microbenchmarks missed more than 60% of them. The devices were not necessarily faulty on arrival. Some developed errors with age, and many failed only for particular data values or functional units. Standard ECC and thermal protection do not cover every logic path, so a device can look healthy under memory checks and generic stress while producing a wrong tensor in one model operation.
SDCHunter changes the diagnostic input. Instead of asking the suspect GPU to run a broad synthetic suite, it replays the exact training workload and data associated with the failure. This preserves the kernel path, operation shape and values likely to activate the defective logic. The production deployment mitigated 40 SDC incidents. The paper’s contribution is not that generic tests have no value, but that a negative generic result cannot clear a device whose failure is workload-specific.
This creates a data-retention requirement. Workload-faithful replay needs enough evidence from the original execution to reconstruct the trigger. A training platform must retain identifiers, inputs or reproducible seeds, software versions and placement information before it knows that an SDC has occurred.
Detect online without checking everything twice
AEGIS targets the first question at production scale[2]. Its cSensor-cVerifier abstraction separates a cheap corruption signal from definitive verification. The sensor can run frequently because it does not duplicate every operation. The verifier spends more work only after the sensor produces evidence. This two-stage budget aims to avoid the usual choice between low overhead and a weak signal.
The deployment spans 35 million GPU-hours. AEGIS identified 13 GPUs behind 18 confirmed SDC events, and its performance cost was 0.86%. These numbers support the practicality of continuous detection. They do not directly yield a universal SDC rate. GPU generations, job mix, fleet age, the definition of an incident and the scope of detection all belong in the denominator. The paper reports one production environment, not a hardware reliability specification.
AEGIS and SDCHunter overlap in purpose but not timing. The former protects a running job by raising and verifying an online signal. The latter tests candidate hardware with a faithful reproduction after the platform has an incident to investigate. An online detector that says “this result is wrong” may still need replay to determine which GPU should leave the fleet.
Find the first wrong operator, not the first bad loss
OpGuard addresses the software-hardware ambiguity[3]. Developers often compare losses or gradient norms from two runs, but these aggregate values can hide a small mismatch until it spreads. Comparing every instruction is impractical and benign nondeterminism makes naïve bitwise comparison noisy. OpGuard instead discovers semantic-stable boundaries between operators, fingerprints tensors there, and maps executions even when their schedules differ.
The longest prefix of matching fingerprints defines the trusted portion of the run. The first mismatching boundary becomes the debugging pivot, accompanied by operator and execution context. By controlling known nondeterministic behavior, a bitwise mismatch at that boundary becomes stronger evidence than a later change in loss.
ByteDance deployed OpGuard in pretraining and post-training. It diagnosed more than 20 production issues, including kernel races and SDCs missed by other checks, and reduced reported debugging time from days to minutes. That count includes software and hardware problems. It should not be added to AEGIS’s 18 incidents or SDCHunter’s 40 mitigations as if all three papers observed the same event population.
One reliability pipeline, three evidence budgets
The combined architecture is economical. Continuous sensing must be cheap enough for every iteration. Definitive verification can be rarer. Workload replay can consume a quarantined device after an incident. Bitwise boundary comparison can be enabled for a reproducible pair of executions when engineers need an exact pivot. Each step buys stronger evidence at a higher or more targeted cost.
The chain also clarifies the role of broader recovery systems such as ByteRobust[4]. Rapid eviction and restart restore useful training time, but they can deliberately remove more machines than the true fault requires. SDC tools provide the slower evidence needed to quarantine the right component, protect the next run and debug a faulty kernel. Recovery and diagnosis run on different clocks and should not block each other unnecessarily.
No system here proves that an undetected SDC is impossible. Sensors cover selected invariants, replay requires a reproducible trigger, and bitwise alignment requires sufficiently controlled executions. Correlated failures or corruption in retained evidence can complicate the chain. Still, these papers replace an unrealistic requirement, continuously prove every calculation, with an operational sequence that asks the strongest affordable question at each stage.
One detector cannot own the reliability claim
The three systems expose why a single SDC detection rate would be a misleading fleet metric. AEGIS observes live execution, SDCHunter reproduces the workload on a suspected device, and OpGuard compares two executions at semantic operator boundaries. They operate with different priors, costs, and definitions of truth. Combining their incident counts would double-count some failures and miss the fact that one system detects an event while another explains it.
A production reliability pipeline therefore needs an evidence state machine. A cheap sensor raises suspicion; a verifier decides whether corruption is credible; workload-faithful replay tests whether a particular GPU reproduces it; operator alignment finds the first divergence; quarantine and replacement turn diagnosis into risk reduction. Every transition needs false-positive cost, false-negative exposure, and elapsed time. AEGIS’s 0.86% overhead prices the first stage, but the fleet still needs to account for replay capacity, duplicated runs, quarantined devices, and engineer time in later stages.
The most important operational asset may be the trigger corpus. General-purpose diagnostics failed to expose over 60% of the affected devices, so a clean stress-test result cannot return a GPU to service with high confidence. Operators should preserve the exact workload and input that exposed each incident, rerun that corpus after repair, and track whether new failures fall outside it. Over time, this converts rare corruptions into an expanding acceptance test. The deeper lesson is uncomfortable but useful: correctness is workload conditional. A GPU is not simply healthy or unhealthy; it is healthy relative to the operators, data patterns, and numerical paths the fleet has actually exercised.
At 35 million GPU-hours, 18 observed incidents are rare enough that accuracy evaluation is intrinsically difficult. A detector can look excellent on aggregate while missing the few errors with the greatest blast radius. Reliability reporting should therefore pair incident counts with exposure time, affected job scale, detection latency, corrupted work before isolation, and whether the event escaped to a checkpoint or model artifact. The expected loss from one long-undetected corruption can exceed that of many quickly contained events.
Common-mode reasoning matters as well. Comparing two executions is a strong oracle only when their failures and benign nondeterminism are sufficiently independent. The same compiler defect, deterministic kernel bug, or corrupted input can make both sides agree on a wrong answer. Bitwise alignment identifies divergence; it does not prove absolute correctness when both executions share the cause. A mature stack needs diversity in selected verification paths, provenance for software and data, and a policy for which expensive checks run after a fleet-wide change. The three papers collectively move the field from hoping ECC will surface every problem toward explicitly managing the evidence that a computation deserves trust.
Sources and attribution
This article is an independent synthesis prepared by Silicon & Systems from three ByteDance operational papers at OSDI 2026. The papers’ arguments and reported results are restated in our own words. No source text, tables or figures are reproduced; both figures were created for this article. USENIX provides the papers on the presentation pages for SDCs in the Wild, AEGIS, and OpGuard. Copyright (c) 2026 the respective authors.