Alibaba’s ASI platform exposes 155,410 GPUs to 81 internal departments, which sounds like a pool large enough to make fragmentation disappear. A six-month production trace shows the opposite[1]. More than 99% of jobs request an exact GPU model, large jobs also need contiguous network topology, and a free accelerator can remain unusable because its host lacks the requested CPUs. At one point, jobs asking for eight GPUs and 128 CPU cores could not start while as much as 28% of the relevant GPUs sat idle.
The OSDI 2026 paper is valuable because it describes the shared fleet as an operating system rather than an inventory. Heterogeneous hardware, high-priority services, overnight slack and application portability are all scheduling state. Alibaba combines placement repair, topology-aware packing, reclaimable workloads and device-specific software optimization. GPU allocation is 68% when only high-priority work is counted, then reaches 93% after low-priority work fills the gaps. The difference between those numbers is not more silicon. It is a more accurate definition of available.
One fleet, several incompatible shapes
The workload mix differs from earlier public traces[2][3]. Online inference accounts for 54.6% of jobs, offline inference for 21.6%, training for 19.5% and development for 3.4%. Generative AI dominates training at 71% and offline inference at 98%, while recommendation models still represent 63% of online inference. The median execution time is five hours, compared with 23 minutes in an earlier Alibaba PAI trace and two minutes in Microsoft’s Acme trace. The scheduler therefore manages long-lived placements whose mistakes persist.
Priority divides the fleet into two markets. Online inference and most training are high priority, while offline inference fills the low-priority side. Development is also treated as high priority because an engineer waiting for a GPU is part of the service objective. Median scheduling latency is only one second, yet the high-priority 90th percentile reaches 101 seconds. Task duration is much longer, with a 90th percentile near 24 hours, so even modest fragmentation accumulates around placements.
Hardware diversity does not automatically create application flexibility. Fewer than 1% of jobs span different GPU models, and more than 99% pin one model exactly. The fat-tree network adds another shape constraint. One access switch connects 32 to 64 eight-GPU nodes, or 256 to 512 GPUs, and all-reduce within one access-switch domain performs up to 27% better than communication that crosses domains. A request for 256 GPUs is therefore not satisfied by any 256 free devices. It wants one model, sufficient host CPUs and preferably one network neighborhood.
Repair the placement without stopping the service
Alibaba’s in-place compaction (IPC) repairs fragmentation by moving existing allocations. A global integer program would find better arrangements but does not finish at this scale. The paper reports that an optimal solver can take five minutes for 25 nodes and two days for 100, while a production partition may contain 500 nodes. IPC instead divides the cluster, applies a recursive ejection chain with depth three and repeats at most five times.
An ejection chain moves the allocation occupying a desired node, then finds a new home for whatever that move displaces. The mechanism is make-before-break: it starts the replacement container before releasing the original, thereby preserving high-priority service. Decisions complete in under two minutes and reduce partially occupied nodes by 20.2%. A separate entropy-based placement policy packs large distributed jobs into fewer access-switch domains, preserving the remaining contiguous regions for future requests.
This repair exposes an important change in fragmentation. Earlier GPU clusters devoted substantial attention to fractional sharing, but the ASI trace finds it negligible. Generative workloads consume whole devices, and the dominant losses now come from stranded GPUs on partly occupied nodes, insufficient CPU cores and network topology. Scheduling policy has to follow the workload rather than continue optimizing yesterday’s bottleneck.
Sell the slack twice, but preserve the first buyer
High-priority demand leaves a strong daily cycle, including as many as 10,000 standby GPU-hours around midnight. Alibaba recovers that capacity through two mechanisms. The first keeps online-inference containers warm but detaches traffic and places them into a sleep state. When demand returns, eviction takes 13 seconds on average and 48 seconds at the 95th percentile; fewer than 5% require a forced termination. Low-priority work harvests 90% of the standby GPU-hours without paying the cold-start cost of rebuilding the service.
SpotGPU handles general preemption. Instead of evicting the smallest or newest job, it estimates lost work as the number of GPUs multiplied by the time since the last checkpoint. The scheduler then takes capacity from jobs with the lowest recomputation cost. In production experiments, this policy reduces low-priority completion time by 24% without measurable damage to high-priority performance. The result is the 68% to 93% allocation step reported for combining both priority classes.
Portability remains the harder form of slack recovery. One alternative accelerator, identified as XPU-A, has a theoretical advantage over H20 for a target workload but initially delivers only 80% of H20 performance. Kernel and framework work improves its throughput by as much as 43%, including a 1.58× prefill-kernel gain, after which high-priority demand for the device grows by 2.5×. A heterogeneous fleet therefore needs a software budget for every hardware option. Otherwise, the scheduler sees capacity that applications will not request.
Utilization is not one number
The remaining telemetry warns against treating allocation as useful computation. Online inference has median streaming-multiprocessor utilization of only 6% and memory use of 30%. Generative inference reaches 94% memory use but only 5% SM use, while conventional deep neural networks show about 20% and 6%, respectively. These complementary profiles suggest colocation, but performance isolation and memory pressure make it harder than adding two percentages.
CPU colocation creates a less visible interference path. CPU-only jobs dominate CPU use on 80% of GPU nodes. Their presence reduces median training SM utilization by 10% and the 90th percentile by 18%. A scheduler can therefore report a highly allocated GPU fleet while co-located CPU work quietly lowers completed training tokens.
The paper explicitly excludes giant foundation-model pretraining on dedicated clusters, so its findings should not be applied unchanged to every hyperscale system. It instead describes the broad middle of a multi-tenant AI cloud, where inference, development and training compete across several accelerator generations. For that environment, the durable lesson is an accounting rule. Physical idle, schedulable idle and productive idle are different quantities. A neocloud that sells only the first number will discover the other two in its queue.
Source and attribution
This article is an editorial summary prepared for Silicon & Systems. It restates the cited paper in our own words. No text, tables or figures from the paper are reproduced; both figures were created for this article from reported results. The paper is openly available in the OSDI 2026 proceedings. Copyright (c) 2026 the authors. The source is available from the USENIX paper page.