Pipeline parallelism is easiest when a model resembles a wall of identical bricks. Split the layers evenly, send microbatches through the stages, and overlap one stage’s communication with another stage’s computation. Qwen3-Next does not offer that regularity. It interleaves three linear-attention layers with one full-attention layer, and every layer is followed by a sparse mixture-of-experts (MoE) block. Each combination has a different compute-to-communication ratio. A stage that looks balanced when its operations are timed serially can become the slow stage after overlap is enabled.
Alibaba Cloud’s OSDI 2026 operational systems paper presents Tessera, the pipeline framework used for Qwen3 and Qwen3-Next pretraining[1]. The result is production-scale rather than a small prototype: five workloads on 4,096 to 12,288 Hopper GPUs improve throughput by 20% to 33% over Alibaba’s production baseline. The peak is 39% model FLOP utilization (MFU) on a trillion-parameter Qwen3 workload. Those headline numbers matter, but the paper is more useful for explaining why a pipeline planner must now optimize a schedule and a partition together.
Why equal layers produce unequal stages
Sparse MoE training dispatches tokens to experts with All-to-All communication. A pipeline can hide part of this transfer behind matrix multiplication from another microbatch. The amount hidden, however, depends on which operations meet in the same execution window. Tessera’s measurements find a 3× difference in overlap gain among layer combinations. Counting layers or adding their standalone times therefore assigns the wrong cost to a stage.
The dependency runs in both directions. A partition determines which layer combinations can overlap, while the quality of a partition can be known only after those combinations have been scheduled and measured. Human experts had been resolving this cycle by hand for each new architecture, a process the authors say could take weeks. At more than 10,000 GPUs, a second source of imbalance appears. MoE routing changes the number of tokens sent to each expert from one iteration to the next, opening short idle slots that no static plan can predict exactly.
Profile, partition, then fill the bubbles
Tessera decomposes a layer into schedulable tasks such as expert dispatch, matrix multiplication and result combination. Its overlap scheduler searches legal interleavings for each pair of chunks and profiles the resulting makespan on the target hardware. This is not a purely analytical model. The authors found that GPU resource contention and kernel behavior make measured post-overlap cost more reliable than summing isolated task durations.
The partitioner then balances stages using those measured costs. This order breaks the circular dependency: synthesize and profile candidate overlap schedules first, then choose layer boundaries based on the time the combinations actually take in parallel. Lastly, the Dynamic Bubble Optimizer observes routing metadata during training, predicts a near-term idle window, and moves eligible deferred work into it. It operates independently within expert-parallel groups, so a larger cluster adds instances rather than a single global control bottleneck.
The distinction among the three mechanisms is useful. Scheduling decides how work can overlap. Partitioning decides where model chunks live. Runtime filling recovers slack left by stochastic routing. Calling all three simply “pipeline optimization” would hide where each gain comes from and when it may disappear.
Five production workloads, not one peak number
The production table covers Qwen3 and Qwen3-Next models from medium to trillion scale. At 8,192 GPUs, Qwen3-L moves from 29.7% to 36.3% MFU, a 22.0% throughput gain. Qwen3-XL rises from 32.0% to 39.0%, or 21.8%. Qwen3-Next-M on 4,096 GPUs improves from 16.7% to 20.0%, while the largest Qwen3-Next-XL run on 12,288 GPUs moves from 15.9% to 21.1%, a 32.8% gain. In a separate controlled comparison on 256 GPUs, its MFU reaches as much as 1.24 times that of Megatron-Core MoE with public recipes[3].
One deployment trace also separates the contribution. Enabling the static Tessera plan produced about 13% more throughput, including roughly 9% from improved stage balance. Adding dynamic bubble filling later raised the Qwen3-XL run to 39.0% MFU. On another 6,144-GPU workload, the dynamic component alone removed 641 ms of pipeline idle time and improved throughput by 3.4%.
Where the method runs out of computation
Overlap cannot hide communication without enough useful computation beside it. On an 8,192-GPU trillion-scale Qwen3-Next workload, Tessera hides 73% of expert-parallel communication and leaves only 8.3% of iteration time exposed to it. A 5,120-GPU medium workload has smaller matrix multiplications per expert, hides only 26%, and spends 38.9% of iteration time in exposed expert communication. The software did not fail; the workload lacked a long enough compute interval to cover the transfer.
Even the favorable case retains a visible gap. Exposed expert communication and pipeline idle time together account for 17% of iteration time. Warmup and cooldown phases contribute substantially because the pipeline has fewer independent microbatches available at its edges. Tessera therefore demonstrates a planner that follows changing model structure, not a universal removal of communication.
The broader systems lesson is that model architecture has become part of cluster scheduling. Linear attention, full attention and sparse experts are algorithmic choices, but at 10,000 GPUs they determine which transfers can be hidden, which stage becomes the bottleneck and how much of the installed silicon performs useful arithmetic. A training framework that assumes interchangeable Transformer blocks is optimizing a model generation that is already passing.
The number that should follow the GPU count
The usual description of a training run starts with model size and GPU count. Tessera shows why that pair is no longer enough. Two jobs with the same accelerator generation and parallelism degree can convert installed FLOPS into tokens at substantially different rates because the model supplies different opportunities to cover communication. A useful capacity report should therefore add at least three quantities: post-overlap stage imbalance, the exposed fraction of each collective, and the pipeline time left idle after dynamic repair. MFU says how much was lost in total; these three values say where another engineering month can recover it.
This distinction changes how an infrastructure team should evaluate a new model architecture. A 32.8% gain on Qwen3-Next-XL does not imply that Tessera is intrinsically better on larger jobs. The largest reported job also begins from the lowest baseline MFU and presents a particular mix of attention and MoE work. The correct transfer test is to profile the candidate model’s layer pairs, then ask whether the resulting stage costs and compute windows resemble the reported cases. GPU count is a deployment scale, not an explanation of the gain.
There is also a hardware implication. More fabric bandwidth still helps the 5,120-GPU case whose exposed expert communication reaches 38.9%, but bandwidth alone cannot repair a partition chosen from the wrong cost model. Conversely, a better planner cannot cover a transfer when the expert matrix multiplication is too short. The next generation of training systems must expose this boundary to model designers: changing expert size, routing balance or attention order changes not only model quality but the price of every collective. Tessera’s most durable contribution is making that price measurable before a multi-week run begins.
The ablation numbers suggest a practical optimization order. First remove persistent stage imbalance, because it taxes every microbatch. Then recover routing-induced bubbles, whose value depends on moment-to-moment variation. Only after both should the operator attribute the remaining exposed time to the fabric. This order prevents an expensive network upgrade from being credited for a scheduling problem, or a scheduler from being judged against communication that has no covering computation. It also gives regression testing a stable hierarchy: a model or kernel change should not ship until the post-overlap cost table, chosen partition, and residual communication ledger have all been regenerated.
The method raises a deployment question the paper does not fully price: how often the profile becomes stale. Kernel versions, batch shapes, routing distributions, and GPU firmware can change the overlap behavior even when the model graph is fixed. Profiling too rarely preserves a bad plan; profiling too often consumes scarce cluster time. A production implementation should record the prediction error between profiled stage time and observed stage time, trigger re-planning when the error persists, and report the profiling cost as part of the gain. Without that lifecycle, an overlap-aware plan risks becoming another static configuration that was accurate only on launch day.
Source and attribution
This article is an editorial summary prepared for Silicon & Systems. It restates the cited paper in our own words. No text, tables or figures from the paper are reproduced; both figures were created for this article from reported results. The authors retain copyright to the OSDI 2026 paper, which USENIX makes openly available on the presentation page. Copyright (c) 2026 the authors.