An AI training job and a network operator describe the same incident differently. The job reports a slow or stuck collective. The operator sees links, switch ports, overlay tunnels, adaptive paths and queues. One failed component may affect only a container placement, while one congested path may be healthy in the binary sense but still destroy collective goodput. A useful operating stack must answer three questions: which part of the expected training graph stopped behaving, which physical path carried the affected packets, and where traffic should move next.
Three industry-led SIGCOMM 2025 papers cover those control points. Alibaba Cloud’s SkeletonHunter infers failures from the regular, sparse traffic of large-model training[1]. ByteDance’s ByteTracker sends centralized probes and uses switch mirroring to recover their paths without installing agents on every server[2]. ByteDance and Broadcom’s SGLB gives commodity switches a global congestion-aware load-balancing mechanism[3]. They should not be ranked by one metric. The first two diagnose production networks; the third evaluates a forwarding prototype that acts before or during congestion.
The workload already draws a skeleton
Containers make placement dynamic and add an overlay network over the physical fabric. Comprehensive monitoring of every possible path is expensive, while opportunistic samples may miss the paths that matter to the current job. SkeletonHunter uses a property of distributed training: its communication is large but structurally sparse and repetitive. Tensor, pipeline, data and expert parallel groups repeatedly traverse a limited set of paths. This recurring set is the traffic skeleton.
The system identifies the expected skeleton and reasons about deviations in its signals. Because the selected paths are tied to active training communication, a missing or degraded observation has stronger workload context than a generic background probe. The overlay-to-underlay relationship also helps translate a container-visible symptom into candidate physical components.
Alibaba reports six months of production deployment. SkeletonHunter uncovered 4,816 network failures with 98.2% precision and 99.3% recall, and localized them with 95.7% accuracy. After operators fixed 98% of the identified problematic components, the monthly network failure rate fell by 99.1%. The last number is an operational before-after result, not an algorithmic localization score. Maintenance action sits between detection and the reduction.
Recover a path without managing an agent fleet
ByteTracker begins from a different gap. End-host probes add processes, configuration and host noise across a data center that may contain an enormous number of servers. A timeout can come from the host rather than the network, and adaptive forwarding makes the actual path harder to infer from endpoints alone.
The system places a small number of dedicated probers centrally. It constructs packets that cause the destination operating system to return a response without a resident probing agent. North-south and east-west probe patterns cover links and internal forwarding paths. Switch packet mirroring records the actual route and enables hop-by-hop comparison, including anomalies such as silent drops or changed packet contents.
ByteTracker was deployed in all ByteDance data centers for more than half a year. The official paper abstract reports that it detected almost all network anomalies and localized them within five seconds with 100% accuracy during the deployment. That result should retain its context: it is the authors’ operational evaluation over the anomalies observed in that period, not a mathematical guarantee for every failure mode. Its advantage over SkeletonHunter is not a higher percentage. It covers general data-center paths without relying on an active training pattern; SkeletonHunter adds training-topology semantics.
Steer around congestion before it becomes a failure
SGLB acts in the forwarding plane rather than the incident queue. Local adaptive routing sees nearby queues but can push traffic toward congestion farther along a path. Global state is more informative but expensive to distribute and store in commodity switch hardware. Link failure also demands fast convergence, while asymmetric path capacity can make naïve traffic spreading reduce throughput.
The system’s SyncMesh control protocol distributes compact congestion profiles to a Global Load Balancing engine in switches. The design combines global path condition with available capacity and updates forwarding after failures. Prototype experiments report recovery in as little as 45 microseconds and up to 60% faster All-to-All communication by avoiding globally congested paths.
These are not six-month production figures like those for SkeletonHunter and ByteTracker. The paper prototypes SGLB and evaluates it experimentally. It establishes that the abstraction can fit commodity switch constraints and improve selected collectives under tested conditions. Long-term behavior under changing job mixes, control-plane loss and interacting traffic remains a deployment question.
A combined operating loop
The systems fit together as a loop. Training traffic supplies high-value passive evidence. Dedicated probes test paths independently of the workload and retain coverage during job changes or outages. The load balancer uses current congestion state to avoid damage, while both monitoring systems confirm whether steering restored expected behavior. When a component is truly bad, localization feeds maintenance and quarantine rather than endless rerouting.
There are seams to engineer. Mirrored probes must follow paths representative of GPU traffic. Adaptive routing can make one probe observation stale quickly. A traffic skeleton changes when the scheduler moves containers or parallel groups. Global congestion summaries have delay and finite precision. The operator therefore needs common identifiers across job, container, endpoint, switch and path, plus timestamps accurate enough to join their evidence.
The larger lesson is that “network reliability” is not one percentage. Detection precision, recall, localization accuracy, time to localization, convergence time and collective throughput belong to different stages. An AI cloud that publishes one of them has not automatically demonstrated the others. The industry papers are valuable precisely because they expose several stages of the operating system behind the fabric.
The integration problem is larger than any one result
The three papers should not be read as competing monitoring products. They observe different clocks. SkeletonHunter follows the stable path structure of a long-running job and detects a change. ByteTracker resolves the physical path of active probes within seconds. SGLB reacts inside the network on a microsecond scale before a control-room diagnosis could arrive. A complete fabric needs all three clocks because detecting, explaining, and steering around a fault are different deadlines.
Their metrics also belong to different denominators. Precision and recall describe a classifier over observed failures. Five seconds describes operational localization latency across a deployed estate. Forty-five microseconds and 60% describe recovery and collective performance in a prototype experiment. The useful synthesis is not a ranking but an interface: the workload layer should identify the affected skeleton, the path layer should map it to components, and the control layer should keep traffic productive while repair proceeds. Without that handoff, each tool can be locally correct while the job still stalls.
This suggests a stronger acceptance test for an AI fabric. Measure completed training steps during injected link, switch, and congestion events, then decompose the loss into detection delay, localization delay, route convergence, degraded-path bandwidth, and repair time. Include false alarms that trigger unnecessary rerouting, because a control loop that reacts quickly to the wrong signal can create the outage it was meant to avoid. The headline should be the fraction of target collective goodput preserved through the event, not merely the time when an alarm appeared. The larger insight is that network operations are becoming part of the distributed runtime. Job topology supplies context to telemetry, telemetry supplies evidence to routing, and routing buys time for physical repair.
The workload skeleton also changes what deserves monitoring. A conventional network tool tries to cover every path uniformly. Large distributed jobs repeatedly exercise a sparse and structured subset, so the marginal value of observing those paths is higher. However, the subset changes when a job is rescheduled, a collective algorithm changes, or elastic membership moves ranks. The monitoring system must version the skeleton with the job plan; otherwise it can confidently watch yesterday’s critical paths while today’s traffic fails elsewhere.
Control-loop stability is the corresponding SGLB concern. Global congestion information is more powerful than a local queue sample, but it is older by the time it reaches the switch. If many switches react to the same delayed summary, they can move traffic together and create a new hot path. Prototype recovery time does not by itself establish stability under many overlapping jobs and failures. Production evidence should include oscillation, fairness, and convergence under stale or partial state. The operator’s goal is not globally perfect information. It is information fresh and bounded enough that distributed decisions improve the next interval instead of correcting the last one.
Sources and attribution
This article is an independent synthesis prepared by Silicon & Systems from three industry-led SIGCOMM 2025 papers. Their arguments and results are restated in our own words. No source text, tables or figures are reproduced; both figures were created for this article. Paper metadata and abstracts are available from the official SIGCOMM 2025 program. Copyright (c) 2025 the respective authors and publishers.