Accelerators are usually shared below the level where applications can see the contention. A foreground request enters one software stack, a background job enters another, and both eventually become commands in a device queue. If that queue is first-come, first-served, an urgent task can wait behind work that has already been admitted. XSched asks whether one software scheduler can impose priority and bandwidth policies across devices that were never designed to expose the same controls[1].
The evaluation is deliberately broad. It covers ten XPUs through seven software platforms: NVIDIA, AMD and Intel GPUs; Intel and Ascend NPUs; NVIDIA DLA, PVA and OFA fixed-function accelerators; and a Xilinx FPGA. The devices range from older hardware with little scheduling support to a Volta GPU with command interruption. This variety is the point. A scheduler that works only on a recent CUDA GPU does not solve contention in an AI PC, embedded system or heterogeneous server.
XQueue separates policy from the native queue
XSched presents applications with a preemptible command queue called XQueue. Commands first enter software queues. A scheduler chooses which XQueue may feed a native hardware queue and launches commands progressively rather than submitting an entire task at once. Applications can call XQueue directly, or an interception layer called XShim can wrap existing runtime APIs.
The design divides hardware support into three levels. Level 1 stops launching commands from a low-priority XQueue but must wait for commands already pending in the hardware queue. It is broadly portable because most runtimes expose queue submission and synchronization. Level 2 can prevent pending and in-flight commands from starting, leaving only the command currently executing. Level 3 can interrupt that running command. Each stronger level reduces the amount of old work that an urgent task must wait behind.
This hierarchy gives policy code a stable contract without pretending that all devices are equal. Basic Level 1 ports required 214 to 841 lines of C++ across the seven platforms. The authors implemented fixed-priority scheduling in 104 lines and bandwidth partitioning in 200 lines. Integrating XSched into Triton and a Paella baseline took 10 and 15 lines, respectively. Much of the device-specific code still resides in queue wrapping, synchronization and driver interfaces.
Priority improves only if the scheduler can remove old work
The first experiment runs a latency-sensitive foreground process beside a continuously issuing background process. With native device scheduling, foreground P99 latency rises to 1.60× through 2.19× its standalone value across the tested XPUs. XSched keeps the range to 1.02× through 1.30× and lowers P99 by as much as 2.11× relative to native scheduling.
That result should not be read as one identical mechanism on every device. With a progressive-launch threshold of eight commands, Level 1 has a P99 preemption delay of roughly eight command times, written as 8T in the paper. Level 2 reduces it to about 1T. Level 3 on the NVIDIA GV100 reaches 32 microseconds and no longer scales with command duration. The abstraction is portable; the interruption point is not.
A second experiment asks for a 75:25 throughput split between foreground and background tasks. Native schedulers generally divide service close to equally. XSched approaches the requested ratio with 1.5% average aggregate-throughput overhead. On the AMD MI50, Ascend 910b and Xilinx VU9P, native aggregate throughput can exceed a single process’s standalone result because two processes expose more parallel work. XSched’s policy guarantee therefore carries a measurable utilization tradeoff on hardware that benefits from extra concurrency.
The framework also coordinates different accelerators. When background tasks run on both an NPU and GPU, two independent native schedulers cannot see the combined interference experienced by the foreground NPU task. A shared XSched policy keeps P99 to 1.18× and 1.09× standalone in two tested combinations, improving it by up to 2.63× over the native schedulers.
The overhead is small, except when the driver spins
Level 1 runtime overhead remains below 3.4% on all ten XPUs. Raising the progressive-launch threshold above ten commands reduces it below 1%, at the cost of a longer Level 1 preemption window. Level 2 adds overhead for guardian code on the tested GPUs, while Level 3 uses a device-specific driver interface.
CPU cost is below 5% of one core in most tests. Two exceptions expose a systems detail that an average would hide: the Ascend 910b reaches 18.3% and NVIDIA PVA 11.9% because their drivers spin while synchronizing the hardware queue. XSched’s scheduling logic is portable, but it inherits the waiting behavior of each runtime.
Three cases show why policy must match the application
In the GPU-harvesting case, a production job receives priority while opportunistic work uses the remaining GV100 capacity. TGS harvests 7.3% of the GPU during deep-learning training. XSched harvests 20.0%, a 2.74× increase, while limiting production degradation to 1.0%. A quota-based vCUDA setup degrades the production job by 15.1% for training and 80.0% for a financial workload. TGS also loses 70.0% on the financial case because its workload-specific submission estimate does not generalize. On an AMD MI50 with only Level 1 support, XSched limits degradation to 4.1% for training and 0.4% for finance.
The AI PC case is more revealing. Background blur runs at 25 frames per second on an Intel NPU while speech recognition executes every three seconds. Native first-come scheduling pushes the blur task’s P99 frame interval to 880 ms, or 20.12× standalone. Fixed priority stabilizes video but can starve speech long enough to lose transcription. A laxity-aware XSched policy instead keeps speech within its period and reduces video P99 to 95 ms, a 9.26× improvement over native scheduling.
For two BERT-large models in Triton, vanilla serving and Triton’s priority setting produce foreground P99 values of 1.53× and 1.51× standalone. XSched reaches 1.07×, 30% below vanilla. Against the GPU-specific Paella system, it is comparable at lower rates and 1.3× better at 1,000 requests per second. These experiments do not prove that one policy fits every service. They show why a reusable mechanism matters: fixed priority, bandwidth shares and laxity can be changed without rebuilding the device runtime.
Portability does not remove hardware limits
XSched applies to host-managed accelerators that receive commands from a CPU. It does not directly control autonomous devices such as some DPUs and FPGAs. A task represented by one indivisible command gains little from Levels 1 or 2; it needs Level 3 support or must be split into smaller commands. The current system schedules computation and assumes enough device memory for all tasks. It does not solve memory oversubscription.
Trust is another boundary. An application could bypass XQueue or flood it with commands unless the platform intercepts all device access through API remoting or a hypervisor. Some advanced interfaces are also undocumented or unstable, including the driver path used for GV100 Level 3 support. XSched therefore demonstrates a useful software contract, not a universal guarantee. Its strongest result is that heterogeneous accelerators can share one policy vocabulary while still exposing exactly where hardware support ends.
Source and attribution
This article is an editorial summary prepared by Silicon & Systems. It restates the OSDI 2025 paper in our own words and preserves the evaluation conditions attached to each reported value. No paper text, table or figure is reproduced; both figures were created for this article. Copyright (c) 2025 the authors. The paper and artifact links are openly accessible from the USENIX conference page.