An Ethernet switch can choose the emptier of two output queues and still make the wrong decision. The queue one hop away may be clear while a downstream link is already carrying background traffic. When thousands of accelerators enter an All-to-All phase together, many locally reasonable choices can converge on the same remote bottleneck.
SGLB, a SIGCOMM 2025 prototype from ByteDance, Broadcom, and university collaborators, gives commodity switches a bounded view beyond their immediate ports[1]. A control protocol called SyncMesh distributes compact congestion information. A Global Load Balancing (GLB) engine combines that state with path capacity and failure status when selecting a route. The goal is not a centralized controller choosing every packet. It is distributed forwarding with enough shared context to avoid globally bad choices.
The prototype recovers from evaluated link failures in as little as 45 microseconds and improves All-to-All communication by up to 60% in the reported experiments. On a 32-GPU Llama 3 training test with one congested link, communication time per iteration falls from 1.177 to 0.704 seconds, reducing total iteration time by 12.3%. These results establish a mechanism and its operating region. They are not long-term production telemetry.
AI collectives expose the weakness of local routing
Equal-cost multipath hashes flows without observing congestion. It is simple and stable, but several elephant flows can collide on one path while another remains empty. Dynamic load balancing reads local queue or utilization state and can react more intelligently. However, the best local port may lead toward a busy remote link.
AI training makes this limitation visible because communication arrives in synchronized bursts. All-to-All traffic from expert parallelism creates many simultaneous sources and destinations. AllReduce builds a different hierarchical pattern. A route that works for one collective or phase can become poor during another. The system must adapt quickly enough that the burst has not ended before the information arrives.
Failure is a second clock. Conventional routing can require seconds to remove a dead link. A large collective can lose packets, stall, or time out during that interval. Fast local failure detection helps at the adjacent switch, but switches farther away still need a consistent alternative. SGLB treats congestion distribution and failure convergence as one forwarding problem.
A third issue is capacity asymmetry. If one path has half the bandwidth of another, spreading packets equally can lower throughput. Global congestion information is not sufficient unless the decision also weights how much traffic each path can carry. The paper explicitly designs against this suppression.
SyncMesh sends a profile, not a full network map
Global visibility collides with switch memory. A fabric with thousands of accelerators and many destinations cannot install per-flow state for every possible route in every device. Control messages also cannot consume a meaningful fraction of the links they protect.
SyncMesh limits the state to path segments toward destination switches of interest. Neighboring GLB engines exchange local changes, and each switch derives a compact congestion profile for the alternatives it may select. The state is tied to topology rather than current training flows, so it does not grow with the number of active jobs.
In the topology evaluated by the authors, an engine keeps 1,024 path-segment entries. Their reduction method targets clusters on the order of ten thousand GPUs within current hardware constraints. This is a scale argument about state representation, not evidence that the prototype ran one production job across that many devices.
The update interval is configurable. Experiments from 1 to 20 microseconds keep completion time within 98.5% to 100% of the best tested case. Message traffic consumes between 0.01% and 0.26% of link capacity at the slow and fast ends of that range. Another burstiness experiment uses a 10-microsecond interval and reports 0.03% fixed control overhead.
Freshness still matters. A profile is historical by the time a packet uses it. SGLB relies on training bursts lasting longer than the update interval and on decisions being robust to small state differences. A deployment with much shorter microbursts or synchronized reactions across many switches must test stability separately.
Capacity-aware choice prevents a slower path from winning
The GLB engine cannot compare queue occupancy alone. A lightly loaded 400 Gb/s path and a moderately loaded 800 Gb/s path may offer different remaining capacity. SGLB represents congestion relative to path bandwidth and chooses among viable alternatives without sending an equal share into unequal pipes.
This distinction becomes important after failure. Removing one link changes the capacity of some path groups. A policy that retains old equal weights can overload the surviving smaller group. SyncMesh propagates both availability and updated path condition so that forwarding converges to a capacity-compatible state.
Load-balancing granularity adds another tradeoff. Per-packet decisions react most quickly but can reorder traffic. Flowlet routing keeps a burst of closely spaced packets together and changes the path only after a gap. The paper finds that SGLB still beats the baselines in flowlet mode, although the advantage narrows because later packets cannot use newly discovered information until the flowlet ends.
An operator should therefore treat granularity as part of the transport contract. RDMA deployments sensitive to reordering may accept slower adaptation. A switch or NIC that supports out-of-order placement can use finer decisions. Reporting only the routing algorithm without the packet, flowlet, or flow unit leaves the result incomplete.
The collective determines the visible benefit
On the testbed, all three schemes reach about 636 Gb/s of an 800 Gb/s link before injected congestion. As background load rises, SGLB degrades more slowly. At 82% load, normalized All-to-All bus bandwidth declines by 48% with SGLB, 55% with local dynamic balancing, and 65% with ECMP. Relative to those baselines, SGLB’s advantage grows as the remote bottleneck becomes stronger.
AllReduce behaves differently. The evaluated implementation first reduces within a rack, so a smaller fraction of its traffic crosses the injected leaf-to-ToR bottleneck. At 85% background load, normalized bandwidth remains 99.5% for SGLB, compared with 67.6% for the local scheme and 57% for ECMP. The result does not mean AllReduce is universally immune. It means global steering avoids the particular remote link while the collective’s hierarchy limits exposure.
The 32-GPU Llama 3 experiment connects communication to application time. Each iteration spends 2.66 seconds in computation under both policies. With one 500 Gb/s RDMA background flow in each direction on a ToR-to-leaf link, SGLB communication takes 0.704 seconds and ECMP takes 1.177 seconds. Total training time falls 12.3%, smaller than the communication-only difference because compute is unchanged.
This denominator discipline matters. A 60% All-to-All gain cannot be applied to an application whose iteration is mostly compute or whose traffic never reaches the congested tier. Buyers need the collective mix, communication fraction, topology, background-load location, and routing granularity beside the headline.
Failure convergence moves below the routing protocol
The paper measures convergence from shutting down a link until packet loss is no longer observed. BGP-backed ECMP and local dynamic balancing take more than one second on the small testbed. SGLB reacts at submillisecond scale because SyncMesh and GLB state remove the failed segment without waiting for the conventional routing control plane.
The exact result depends on topology. A spine-to-leaf failure in the tested two-layer fabric converges in about 45 microseconds. The analogous failure in the three-layer case can be about one microsecond when the affected state is local to the relevant decision. A leaf-to-ToR failure in that topology returns to roughly 45 microseconds. The minimum should not be quoted as a universal failover time.
Fast convergence protects the interval before network operators diagnose hardware. It does not repair the link or explain why it failed. A monitoring system must still associate repeated steering events with a component, prevent oscillation around a marginal link, and decide when to quarantine it. SGLB buys useful traffic time for that slower process.
The failure test also needs packet-loss and ordering context. “No longer observing loss” marks route convergence, while transport recovery, retransmission, and collective completion can extend the application-visible interruption. Production acceptance should measure both switch convergence and the first healthy collective after the event.
Global information can create global oscillation
Shared state is useful because local signals are incomplete, but many switches can react to the same delayed profile. If they all move traffic toward one apparently empty path, the system can create a new hot spot and move again. The paper’s burstiness experiments show robustness in the tested ranges, not every combination of jobs, update delay, and asymmetric capacity.
A deployment should test phase-aligned collectives from several tenants, lost or delayed SyncMesh messages, switches running different profile versions, and repeated link flapping. Metrics should include route changes, oscillation amplitude, fairness between jobs, packet reordering, control traffic, and collective completion time. Average link utilization alone can hide one tenant repeatedly displaced by others.
Failure state needs hysteresis. A marginal link that alternates between up and down faster than control propagation can generate churn. The GLB engine should prefer a stable, slightly busier path until the recovered link has remained healthy for a defined interval. That policy trades some capacity for control-loop stability.
Security and fault containment also enter the design. Congestion summaries influence forwarding across the cluster. A corrupt switch or malformed update should not advertise false capacity and attract traffic. Operators need bounded values, authenticated control messages, versioning, and a fallback state when global information disappears.
The procurement unit is preserved collective work
SGLB suggests a more useful network evaluation than peak bisection bandwidth. Inject a remote bottleneck, unequal path capacities, and link failures while replaying the target collective mix. Measure collective completion, training iteration time, packet loss, reordering, and convergence. Then repeat under stale and missing control state.
The baseline must distinguish ECMP from local adaptive routing. Beating a static hash shows the value of congestion awareness. Beating a local scheme shows the additional value of remote state. Comparing the two gaps reveals whether a workload needs a global mechanism or merely better local queues.
Hardware feasibility should be reported with the same care. Entry count, update bandwidth, pipeline stages, supported decision granularity, and fallback behavior determine whether an advertised switch can run the design. “Commodity” means the abstraction fits existing programmable resources, not that every installed switch exposes the required GLB engine.
The lasting contribution is an information boundary. Local queues are fast but incomplete. A centralized network map is rich but too slow and large for packet decisions. SGLB places a compressed, frequently updated path profile between them. When that profile is accurate enough and the control loop remains stable, commodity Ethernet can preserve more collective work through congestion and failure without waiting for a new end-to-end routing calculation.
One final acceptance table can keep the claim auditable. For normal load, report the cost of SyncMesh and any change in uncongested collective time. For remote congestion, report remaining path capacity and completion time by collective. For asymmetric paths, report the traffic split and whether the smaller path saturates. For a hard failure, report adjacent detection, GLB convergence, transport recovery, and the first successful application barrier. For stale state, report oscillation and fairness. Each row tests a different reason to deploy global information.
The table should include scale labels. Switch-state capacity can be analyzed for a ten-thousand-GPU topology while the forwarding experiment runs on a smaller prototype. Both results are useful, but they answer different questions. Separating implementation scale, emulated scale, and analytic scale prevents a state-compression result from being presented as application throughput at the same fleet size.
This separation also guides rollout. Begin with shadow state distribution and compare the route SGLB would choose against current forwarding. Enable it for one traffic class, retain a rapid fallback to ECMP, and expand only after control loss and link flapping have been tested. A global control loop earns production authority by showing bounded behavior when its information is wrong, not only better throughput when every message arrives.
Source and attribution
This article is an editorial summary prepared by Silicon & Systems. It restates the paper’s architecture, prototype measurements, and limits in our own words. No source sentences, tables, or publisher figures are reproduced; both figures are Silicon & Systems originals first made for the multi-paper network-operations review. The version of record was published by ACM. Copyright (c) 2025 the authors, with publication rights licensed to ACM. ACM’s eReader provides free access through the DOI page.