Accelerators are usually shared below the level where applications can see the contention. A foreground request enters one software stack, a background job enters another, and both eventually become commands in a device queue. If that queue is first-come, first-served, an urgent task can wait behind work that has already been admitted. XSched asks whether one software scheduler can impose priority and bandwidth policies across devices that were never designed to expose the same controls[1].
The evaluation is deliberately broad. It covers ten XPUs through seven software platforms: NVIDIA, AMD and Intel GPUs; Intel and Ascend NPUs; NVIDIA DLA, PVA and OFA fixed-function accelerators; and a Xilinx FPGA. The devices range from older hardware with little scheduling support to a Volta GPU with command interruption. This variety is the point. A scheduler that works only on a recent CUDA GPU does not solve contention in an AI PC, embedded system or heterogeneous server.
XQueue separates policy from the native queue
XSched presents applications with a preemptible command queue called XQueue. Commands first enter software queues. A scheduler chooses which XQueue may feed a native hardware queue and launches commands progressively rather than submitting an entire task at once. Applications can call XQueue directly, or an interception layer called XShim can wrap existing runtime APIs.
The design divides hardware support into three levels. Level 1 stops launching commands from a low-priority XQueue but must wait for commands already pending in the hardware queue. It is broadly portable because most runtimes expose queue submission and synchronization. Level 2 can prevent pending and in-flight commands from starting, leaving only the command currently executing. Level 3 can interrupt that running command. Each stronger level reduces the amount of old work that an urgent task must wait behind.
This hierarchy gives policy code a stable contract without pretending that all devices are equal. Basic Level 1 ports required 214 to 841 lines of C++ across the seven platforms. The authors implemented fixed-priority scheduling in 104 lines and bandwidth partitioning in 200 lines. Integrating XSched into Triton and a Paella baseline took 10 and 15 lines, respectively. Much of the device-specific code still resides in queue wrapping, synchronization and driver interfaces.
Priority improves only if the scheduler can remove old work
The first experiment runs a latency-sensitive foreground process beside a continuously issuing background process. With native device scheduling, foreground P99 latency rises to 1.60× through 2.19× its standalone value across the tested XPUs. XSched keeps the range to 1.02× through 1.30× and lowers P99 by as much as 2.11× relative to native scheduling.
That result should not be read as one identical mechanism on every device. With a progressive-launch threshold of eight commands, Level 1 has a P99 preemption delay of roughly eight command times, written as 8T in the paper. Level 2 reduces it to about 1T. Level 3 on the NVIDIA GV100 reaches 32 microseconds and no longer scales with command duration. The abstraction is portable; the interruption point is not.
A second experiment asks for a 75:25 throughput split between foreground and background tasks. Native schedulers generally divide service close to equally. XSched approaches the requested ratio with 1.5% average aggregate-throughput overhead. On the AMD MI50, Ascend 910b and Xilinx VU9P, native aggregate throughput can exceed a single process’s standalone result because two processes expose more parallel work. XSched’s policy guarantee therefore carries a measurable utilization tradeoff on hardware that benefits from extra concurrency.
The framework also coordinates different accelerators. When background tasks run on both an NPU and GPU, two independent native schedulers cannot see the combined interference experienced by the foreground NPU task. A shared XSched policy keeps P99 to 1.18× and 1.09× standalone in two tested combinations, improving it by up to 2.63× over the native schedulers.
The overhead is small, except when the driver spins
Level 1 runtime overhead remains below 3.4% on all ten XPUs. Raising the progressive-launch threshold above ten commands reduces it below 1%, at the cost of a longer Level 1 preemption window. Level 2 adds overhead for guardian code on the tested GPUs, while Level 3 uses a device-specific driver interface.
CPU cost is below 5% of one core in most tests. Two exceptions expose a systems detail that an average would hide: the Ascend 910b reaches 18.3% and NVIDIA PVA 11.9% because their drivers spin while synchronizing the hardware queue. XSched’s scheduling logic is portable, but it inherits the waiting behavior of each runtime.
Three cases show why policy must match the application
In the GPU-harvesting case, a production job receives priority while opportunistic work uses the remaining GV100 capacity. TGS harvests 7.3% of the GPU during deep-learning training. XSched harvests 20.0%, a 2.74× increase, while limiting production degradation to 1.0%. A quota-based vCUDA setup degrades the production job by 15.1% for training and 80.0% for a financial workload. TGS also loses 70.0% on the financial case because its workload-specific submission estimate does not generalize. On an AMD MI50 with only Level 1 support, XSched limits degradation to 4.1% for training and 0.4% for finance.
The AI PC case is more revealing. Background blur runs at 25 frames per second on an Intel NPU while speech recognition executes every three seconds. Native first-come scheduling pushes the blur task’s P99 frame interval to 880 ms, or 20.12× standalone. Fixed priority stabilizes video but can starve speech long enough to lose transcription. A laxity-aware XSched policy instead keeps speech within its period and reduces video P99 to 95 ms, a 9.26× improvement over native scheduling.
For two BERT-large models in Triton, vanilla serving and Triton’s priority setting produce foreground P99 values of 1.53× and 1.51× standalone. XSched reaches 1.07×, 30% below vanilla. Against the GPU-specific Paella system, it is comparable at lower rates and 1.3× better at 1,000 requests per second. These experiments do not prove that one policy fits every service. They show why a reusable mechanism matters: fixed priority, bandwidth shares and laxity can be changed without rebuilding the device runtime.
Portability does not remove hardware limits
XSched applies to host-managed accelerators that receive commands from a CPU. It does not directly control autonomous devices such as some DPUs and FPGAs. A task represented by one indivisible command gains little from Levels 1 or 2; it needs Level 3 support or must be split into smaller commands. The current system schedules computation and assumes enough device memory for all tasks. It does not solve memory oversubscription.
Trust is another boundary. An application could bypass XQueue or flood it with commands unless the platform intercepts all device access through API remoting or a hypervisor. Some advanced interfaces are also undocumented or unstable, including the driver path used for GV100 Level 3 support. XSched therefore demonstrates a useful software contract, not a universal guarantee. Its strongest result is that heterogeneous accelerators can share one policy vocabulary while still exposing exactly where hardware support ends.
Preemption is a procurement feature, not only a runtime feature
XSched reveals a hardware contract that accelerator comparisons usually omit. Peak throughput says how fast one task can run; preemption level says how much unrelated work an urgent task must inherit. In a shared system, that inherited work sets the latency floor. Two devices with similar throughput can therefore support different service densities because one can stop at a command boundary and the other must wait through an entire admitted queue.
Procurement should request a preemption profile alongside memory and bandwidth. The profile needs the finest interruptible unit, worst-case command duration, queue-depth controls, context-switch cost, memory isolation, and whether the interface is documented and stable. XSched can provide a common policy vocabulary, but it cannot manufacture Level 3 behavior from Level 1 hardware. If a vendor exposes only coarse queues, the platform must compensate with smaller commands, conservative admission, or dedicated capacity.
The business implication is that heterogeneous devices should be assigned by interference tolerance as well as kernel efficiency. A fixed-function accelerator may be ideal for a predictable pipeline but unsuitable for mixed interactive work. A GPU with stronger interruption can host lower-priority harvesting while preserving a foreground SLO. The correct fleet optimization therefore prices both standalone performance and the opportunity value of safe sharing. XSched’s 20% harvested capacity on GV100 is not merely a scheduling speedup; it is evidence that a better preemption contract can turn stranded time into usable inventory without transferring the risk to the production workload.
Preemption must preserve application semantics
Stopping a command queue is not equivalent to pausing a stateless CPU thread. An accelerator may hold partial outputs, local memory, synchronization state, descriptors, or external side effects. The scheduler needs a contract for the point at which execution can stop, what state remains valid, and whether the command may resume or must restart.
Applications can classify work accordingly. Idempotent inference kernels may restart cheaply. A long FPGA pipeline may need a checkpoint boundary. A device command that updates shared memory or communicates with another accelerator may require coordinated preemption. XQueue can expose a common scheduling surface, but each backend must state which guarantees it can implement.
Timeout and cancellation behavior should be tested under load. If a driver cannot interrupt a spinning command, priority policy becomes advisory until the command returns. The system should detect this condition, bound the affected queue, and prevent one tenant from converting a high-priority service into an unbounded wait. A capability bit without a measured worst case is not sufficient.
Fairness needs more than priority numbers
Strict priority can protect urgent work while starving background jobs. Weighted sharing can preserve progress while missing a tight deadline. The policy should connect priority, deadline, minimum share, and preemption cost to a service objective. A short urgent task may justify immediate interruption, while repeated small arrivals may be batched to avoid thrashing a large resident job.
State size and switch cost belong in the decision. Preempting a GPU context with large memory or an accelerator that requires reprogramming can cost more than waiting for the current command. The scheduler should estimate payback and use non-preemptive intervals where hardware cannot switch economically. One API can host different policies without pretending the devices have identical costs.
Multi-tenant accounting should include work discarded by preemption, state movement, cache warmup, and queue delay. Charging only executed device time exports scheduling cost to the platform. Reporting completed requests or jobs per device-hour, with SLO attainment and interruption, makes policy comparisons more meaningful.
Portability depends on a verified capability matrix
Supporting ten accelerators demonstrates breadth, but production portability requires a matrix of queue operations, preemption granularity, state preservation, priority enforcement, isolation, reset, and failure behavior for every driver and device. Unsupported operations should fail visibly rather than degrade to a different semantic path.
Driver versions can change the matrix. The paper’s spinning-driver case shows that software below the abstraction can dominate latency. Qualification should rerun worst-case preemption, cancellation, and recovery after driver or firmware updates. Telemetry should identify the backend and version for every violation so a fleet-wide policy does not conceal one weak implementation.
Security is part of portability. A preempted tenant’s device memory and queue state must not become visible to the next tenant. Reset and context destruction should have bounded time and verified cleanup. Some devices may require a stronger reset that reduces availability; the scheduler must price that rather than skipping isolation.
Adoption should begin with one service conflict
The first use case should be concrete: an urgent inference request competing with batch work, an interactive graphics or media task sharing an accelerator, or a control workload that must interrupt background processing. Measure current queue delay, deadline misses, command duration, state size, and device reset behavior. Then apply XSched only to that conflict and compare completed work and SLO attainment.
Shadow scheduling can reveal how often the desired preemption point exists and how much work would be discarded. The next stage enables cooperative boundaries, followed by true device preemption where the backend proves it. This order separates policy value from hardware capability and avoids deploying a universal queue before the critical driver behavior is known.
Failure injection should include a hung command, driver restart, device reset, process exit, and tenant handoff. The system must preserve isolation, release resources, and restore priority service within a measured bound. These events determine whether preemption is an operational feature rather than a benchmark mechanism.
We read XSched as a common language for heterogeneous accelerator scheduling. Its durable value is the separation of policy from native queues, provided that device-specific semantics remain visible. One scheduler can coordinate GPUs, NPUs, FPGAs, and other XPUs without claiming they pause alike. The acceptance claim should be the set of service conflicts resolved within SLO and isolation bounds, with the full capability and cost matrix attached.
Source and attribution
This article is an editorial summary prepared by Silicon & Systems. It restates the OSDI 2025 paper in our own words and preserves the evaluation conditions attached to each reported value. No paper text, table or figure is reproduced; both figures were created for this article. Copyright (c) 2025 the authors. The paper and artifact links are openly accessible from the USENIX conference page.