A failed GPU is visible. The process exits, the job reports an error, and recovery can begin. A fail-slow device is more expensive to reason about because it continues returning results. One accelerator, CPU path, or shared network link takes longer, the synchronized ranks wait at the next boundary, and a large training job appears merely inefficient rather than broken.
GREYHOUND turns that gray state into an operational object.[1] Researchers at HKUST and Alibaba Group studied a production fleet with more than 4,000 nodes and 10,000 heterogeneous GPUs, including about 1,800 H800 and 2,600 A100 accelerators. They separate compute and communication slowdowns, measure how the population changes with scale, and build a detector plus a graduated set of mitigations.
The most important result is not 99.8% detection accuracy by itself. It is the scale gradient underneath it. Six of 392 completed single-node probes saw a compute fail-slow. Forty-two of 107 completed four-node probes saw network congestion. In a separate July 2024 trace, 16 of 27 jobs using 512 to 1,024 GPUs encountered fail-slows, adding 34.59% to completion time on average. These are three different populations, not one universal incident rate.
A live component can still set the clock for every rank
Hybrid-parallel training combines tensor, pipeline, and data parallelism. Each dimension introduces points where ranks exchange data or wait for a stage. The iteration advances at the pace of the slowest dependency. A rank that becomes 20% slower can therefore reduce useful throughput beyond its own fraction of the cluster because healthy ranks spend more time waiting.
Communication fail-slows can be more irregular. A congested RoCE link may stretch collectives for minutes, recover, and return later. At 1,024 GPUs, several degraded paths can overlap with thermal or compute events. The job remains active and may keep checkpointing, so a binary health monitor sees no failure even while the purchased GPU-hours produce much less work.
This distinction changes capacity accounting. Installed, powered, and allocated GPUs can all remain constant during a fail-slow. Useful iterations per hour decline, and the job may hold the entire gang longer. The lost capacity is distributed across healthy devices that cannot make independent progress. Reporting hardware availability without degraded-throughput time consequently overstates the fleet that customers can use.
Three measurement populations answer different questions
The single-node study launched 400 probes and completed 392. Each used four H800 GPUs to train an 11-billion-parameter GPT-2 configuration for 10,000 iterations, usually 70 to 90 minutes. The probes covered roughly 500 of the fleet’s 1,800 H800 GPUs. Six jobs showed compute fail-slows: four were attributed to CPU contention and two to GPU degradation. Their mean duration was 10 minutes, and mean job-completion-time slowdown was 11.79%.
One degraded-GPU case ran approximately 20% slower. The authors estimate this type of GPU degradation at about 0.5% in the observed population. They also warn that temperature rising with a slowdown is not sufficient causal evidence. A temperature signal may follow heavier waiting or another system condition, so thermal telemetry alone should not decide that the GPU is defective.
The four-node study launched 120 probes and completed 107. Each used eight A100 GPUs per node to run a seven-billion-parameter GPT-2 configuration for 10,000 iterations, generally about five hours, over a 400 Gb/s RoCE fabric. The probes covered 690 of approximately 2,600 A100 GPUs. Forty-two jobs suffered network-congestion fail-slows, and one additional job showed CPU contention. Network events lasted 24 minutes on average and increased completion time by 15.45%.
The 42 of 107 result is about 39.3%, but it should not be called the fleet’s annual network-failure probability. These were long probing jobs exposed to a shared multi-tenant network. The denominator is completed probe runs under that experiment, not links, nodes, customers, or calendar hours. Its value is comparative: communication degradation occurred more often and lasted longer than compute degradation in the sampled environment.
The third population contains 27 production jobs, each using 512 to 1,024 GPUs during one month. Sixteen encountered fail-slows. Thirteen were attributed to network congestion, while three combined network and GPU degradation. Events lasted 72 minutes on average, and job completion increased by 34.59%. More than 20% of the jobs were delayed by over 50%. One 1,024-GPU case combined congestion and thermal throttling until throughput fell to 10% of normal.
Scale therefore changes more than the number of failure opportunities. Larger jobs cross more shared links, wait at more global boundaries, and can experience compound events. A fleet model that extrapolates the 1.5% single-node observation directly to a 1,024-GPU job misses correlated network conditions and synchronization amplification.
Telemetry suggests a symptom but does not validate the cause
Production monitoring already contains GPU utilization, temperature, CPU load, network counters, and iteration throughput. The difficulty is attribution. Low GPU utilization can mean network waiting, input starvation, a pipeline bubble, an intentionally light phase, or a degraded device. A busy link can be normal for a collective. Even matching timestamps do not prove which signal caused the slowdown.
GREYHOUND starts closer to the training dependency. An LD_PRELOAD layer observes NCCL calls without modifying the communication library. Autocorrelation estimates the repeating iteration period. Bayesian online change-point detection identifies a shift, and a verification rule requires a sustained performance change above 10% before the system enters deeper diagnosis. The paper reports that this stage reacts within two or three iterations, under five seconds in its environment.
The profiler then compares groups that perform equal communication volume. A slower group is suspicious because it should complete comparable work in comparable time. This narrows the rank and link set before active validation. For compute, GREYHOUND runs GEMM checks in FP8, FP16, and FP32. For communication, it tests the links actually used by the ring or tree rather than scanning the whole fabric. The paper describes this validation as a constant number of passes and reports roughly five seconds of added diagnosis time.
The order is important. Continuous active benchmarking across thousands of ranks would compete with training and create its own congestion. Telemetry alone is cheap but ambiguous. GREYHOUND pays the validation cost only after application-level timing and group comparison identify a small suspect set. This turns a broad symptom into evidence that an operator can act on.
Reported accuracy has two evaluation layers
Across 499 probing jobs, GREYHOUND diagnosed 498 correctly, which the paper reports as 99.8% accuracy. The compute detector identifies all 392 completed single-node jobs correctly, with no false positives or false negatives in that population. The communication detector identifies 106 of 107 four-node jobs correctly, corresponding to 99.1% accuracy, no false positives, and a 2.3% false-negative rate among the relevant positive cases.
These values should not be collapsed into “99.8% at any scale.” The 512 to 1,024-GPU production trace is used to characterize impact, not to provide a fully labeled accuracy set. The controlled 256-H800 end-to-end test injects twelve fail-slow events and measures reaction plus mitigation. Artificial injection gives a known ground truth, but its fault shapes cannot cover every organic interaction in a multi-tenant fleet.
Continuous tracking adds 0.39% overhead on average and at most 1.1% in the reported tests. Change detection usually needs less than five seconds, followed by approximately five seconds for profiling and validation. The 256-GPU experiment reports an average total reaction of 10.56 seconds. These are system-specific values tied to iteration length, the implemented probes, and the chosen 10% verification threshold.
The claimed accuracy is nevertheless operationally meaningful because it is paired with low tracking overhead and an actionable suspect path. A detector that labels every slow iteration correctly but needs several minutes or cannot identify the affected group would preserve little training work. GREYHOUND evaluates the time to a diagnosis that can select a response.
Mitigation should become more expensive only when delay accumulates
Not every transient event justifies a restart. GREYHOUND organizes four responses. S1 observes without changing placement. S2 changes the data-parallel microbatch allocation so slower workers receive less work. S3 adjusts topology, consolidating slow workers where hybrid parallelism can isolate their effect. S4 checkpoints and restarts the job.
The system compares accumulated throughput loss with the estimated cost of the next action. A short event may end before movement pays back. A continuing event eventually justifies rebalancing, then topology adjustment, and finally restart. This policy protects the job from repeated expensive reactions while avoiding unlimited tolerance of a persistent slowdown.
Data-parallel rebalancing reaches as much as 1.59 times the fail-slow baseline in the paper’s experiments. Topology adjustment reaches up to 1.23 times. In the 256-H800 test, a 40-billion-parameter GPT-2 configuration uses 8-way tensor parallelism, 16-way data parallelism, and two pipeline stages. With twelve injected events, the rate declines by nearly half: the normal 37.4 iterations per minute becomes 18.9. GREYHOUND recovers it to 29.8 iterations per minute, a 1.58-times improvement over the degraded baseline.
The remaining gap to 37.4 iterations per minute matters. Mitigation does not repair a slow device or remove shared congestion. It limits how much the slow path controls synchronized work. The output should therefore be reported as recovered throughput and residual loss, not as a resolved hardware incident.
A fleet contract needs a degraded state
Cloud and neocloud capacity contracts usually count a device as available until it fails a health check. GREYHOUND shows why that binary state is insufficient for large training. A GPU can pass basic diagnostics while running below its peer group. A network can deliver packets while collective completion expands. The service still consumes the full gang allocation.
A useful SLO should add degraded-throughput minutes. For every job, record expected iterations or tokens per minute, observed P50 and P5 throughput, the duration below a declared fraction of baseline, ranks affected, cause confidence, and the mitigation applied. Fleet reports should disclose accelerator-hours allocated, accelerator-hours inside acceptable throughput, and healthy devices stranded behind a slow synchronization dependency.
Topology belongs in the incident record. A slow GPU inside one data-parallel group has a different recovery option from congestion on a link shared across pipeline stages. The scheduler needs the job’s tensor, pipeline, and data-parallel mapping together with the physical node and network route. Without that join, an operator may replace a GPU when the shared fabric is the cause or restart into the same congested domain.
The financial denominator is completed training work. Extending a 1,024-GPU job by 34.59% ties up healthy GPUs, delays dependent experiments, and may miss a reservation window. The loss includes more than the degraded component’s power. A provider should price or credit the gang time that fell below the contracted throughput, while a buyer should include fail-slow allowance in schedule and checkpoint planning.
The implementation and evidence have clear limits
GREYHOUND’s communication tracking uses a library-injection technique, but mitigation is integrated through a Megatron-LM plugin. It is not a framework-independent control plane. Another runtime would need access to its parallel mapping, microbatch policy, and topology changes before receiving the same benefits.
The end-to-end mitigation evaluation injects faults. This gives exact event timing but cannot reproduce every organic congestion, firmware, thermal, CPU, and compound pattern in production. The paper also notes that its detector does not cover special cases where compute and communication execute concurrently in ways that hide the expected timing signature.
Relative comparison requires a healthy reference. If every equal-volume group or every candidate link slows together, GREYHOUND may have no normal peer against which to isolate the cause. Platform-wide power capping, synchronized storage stalls, or fabric-wide congestion therefore need independent baselines. The authors planned to release anonymized traces, but the paper did not provide that data set at publication time.
GREYHOUND’s operational lesson survives those boundaries. Fail-slow is not a minor hardware-health category. In synchronized AI training it is a capacity state that can consume thousands of otherwise healthy accelerator-hours. Detection must begin from application timing, narrow the suspect topology, and validate before acting. Mitigation should then spend no more disruption than the accumulated loss can justify.
Source and attribution
This article is an independent editorial digest prepared by Silicon & Systems. It restates the cited work in our own words. No sentences, tables, or figures from the paper are reproduced. The material hardware base was generated for this article and annotated deterministically; both evidence graphics were drawn in code from reported results. The paper is openly available in the USENIX ATC 2025 proceedings. Copyright (c) 2025 the authors. Alibaba Group is listed because authors were affiliated with the company; its inclusion is not an endorsement. Read the complete source on the USENIX paper page.