Supervised model training has a regular inner loop: load a batch, execute forward and backward passes, synchronize gradients, and update parameters. Reinforcement-learning training adds systems whose work does not scale together. Rollout engines generate variable-length responses, inference or reward models score them, tools wait on external services, and trainers consume the accepted data. The slowest stage sets the iteration rate.
Static allocation assumes the same component remains limiting. In a reasoning task, rollout demand can be high at the start and collapse as sequences finish. In an agentic task, one episode re-enters the model across several tool calls, creating heavy-tailed queues. GPUs assigned to a shrinking component can sit idle while another stage gates the barrier.
DynaRL puts a scheduler inside the job. It discovers pipeline components and dependencies, observes queues and throughput, predicts which component benefits from another worker, and moves resources among rollout, inference, tools, and training. The control objective is end-to-end progress rather than utilization of any one engine.
A synchronized pipeline is a max-min problem
If an iteration requires output from every component, its steady throughput is bounded by the minimum component throughput. Adding a GPU to a non-bottleneck stage may raise local utilization yet change nothing at the barrier. Conversely, removing a GPU from an overprovisioned stage can be free until that stage approaches the minimum.
DynaRL models a component’s throughput as a function of its current resources and observed state. The planner considers releasing a small resource increment from an overprovisioned component and allocating it to another candidate. It predicts the resulting minimum throughput and commits only when the estimated global improvement is positive.
The function is not fixed. Batching, sequence length, KV-cache occupancy, parallelism degree, and queue depth change the marginal value of a GPU. DynaRL continually recalibrates from recent measurements instead of relying solely on an offline model.
Worker groups hide framework-specific migration
The runtime represents each logical component as a worker group bound to a resource set. A common migration interface can suspend a rollout engine, resize trainer parallelism, move state, or restart a component whose framework lacks fine-grained support. This separation lets the global scheduler reason about resources without embedding every engine’s internal API.
Migration strategies have different costs. Weakly stateful rollout and tool workers can often suspend and resume quickly. A trainer holds optimizer state, model shards, and communication groups; changing its layout may require resharding or rebuilding collective contexts. Components without a native path fall back to reboot migration and reload state.
Most reported cases change within roughly one second, though cost varies with model size and worker count. Trainer migration contributes less than 0.5% of end-to-end latency in evaluated settings. That percentage is conditional on iterations long enough to amortize the move.
The implementation uses about seven thousand lines of Python atop RLinf, with separate modules for the global scheduler, migration, graph extraction, and routing. The software investment indicates that dynamic allocation is not merely a Kubernetes replica-count change; it reaches into model and communication state.
Two-stage decisions reduce oscillation
Telemetry is noisy, and immediately moving a worker after a queue dip can make the system chase transients. DynaRL first detects sustained overprovisioning over a window. It then evaluates candidate reallocations using a throughput predictor and applies a change only when the predicted global result improves.
The planner’s search is proportional to the number of candidate resource reductions and pipeline components, not the total GPU count. Since an RL pipeline typically has only several components, scheduling remains under 200 milliseconds even at 128 GPUs. Reported online scheduling overhead is below 1%.
Thresholds still encode an operational tradeoff. A conservative threshold misses short opportunities; an aggressive one pays migration cost and can destabilize caches and collective groups. Deployment should expose rejected plans, prediction error, time since the last move, and benefit realized after each move.
Agentic tails require request scheduling too
Resource counts alone cannot remove head-of-line blocking inside rollout. Multi-turn episodes alternate model generation and tool execution. Early turns tend to be longer, while later turns are shorter and closer to completion. Treating every request equally can leave nearly finished episodes behind long initial turns.
DynaRL assigns priority using completed tool-call count, favoring requests nearer the end of an episode. This drains short remaining work and releases the iteration barrier sooner. The choice improves end-to-end completion but can delay new episodes, so fairness and maximum wait should accompany the priority rule in a shared service.
KV-cache behavior is part of the decision. Moving or reordering requests can reduce cache reuse and increase memory pressure. A local scheduler that only sees tokens per second may choose a policy that looks fast briefly while causing later recomputation.
Results depend on workload phase and scale
The evaluation uses H100 clusters up to 128 GPUs, math-reasoning reinforcement learning, and a multi-turn agent workload. For math reasoning, reported gains over the static RLinf baseline include approximately 1.43× to 1.55× in several settings and reach 1.98× overall. Agentic cases improve from 1.06× to 1.53× depending on model and cluster size.
Larger clusters create more room to reassign whole workers and therefore can show larger gains. A small job whose minimum parallel unit consumes much of the cluster has fewer choices. Workloads where rollout, inference, and training remain balanced will also see little benefit.
The timeline analysis shows why the gains occur. As active rollout sequences shrink, the scheduler removes rollout workers and gives resources to inference or training. The next stage overlaps with the tail instead of waiting for all rollout GPUs to become idle at once.
Dynamic allocation expands the failure surface
Every migration can fail after releasing resources but before the destination is ready. The control plane needs idempotent operations, lease ownership, timeouts, and a recovery path that reconstructs the previous allocation. Checkpoints must bind model version, optimizer state, random state, and sample position to prevent silent training divergence.
The scheduler also becomes performance-critical infrastructure. A faulty predictor can repeatedly move large jobs, while a stale dependency graph can deadlock data flow. Safe mode should freeze the last valid allocation and let the original RL framework continue.
Multi-tenant integration adds another layer. DynaRL reallocates within a job, but a cluster scheduler decides the job’s total quota. Borrowing GPUs temporarily can conflict with preemption, topology, power, and fairness policies. The two controllers need a contract for elastic bounds and migration cost.
Schedule progress, not devices
DynaRL’s central contribution is to make RL pipeline state visible to a scheduler. GPU utilization alone cannot distinguish useful work on the critical path from work accumulating behind a barrier. Queue pressure, component throughput, episode progress, and migration cost provide the missing signals.
The approach is most compelling for long-running, synchronized jobs with several elastic stages and strong within-iteration variation. It is less attractive when components are rigid, iterations are too short to amortize migration, or external tool latency dominates beyond the cluster’s control.
As post-training workloads become more agentic, the scheduling unit shifts from a homogeneous training job to a changing graph of services. The operator’s question is no longer how many GPUs a job owns. It is which stage can convert the next GPU-second into earlier validated training data.
The scheduler needs an accounting plane
Every reallocation should produce an event containing the old and new topology, the predicted bottleneck, expected gain, migration bytes, pause time, cache loss, and realized iteration-time change. Aggregating those events reveals whether gains come from better allocation or simply from extra resources. It also identifies a component whose migration cost repeatedly exceeds its predicted benefit.
Training correctness must remain outside the optimizer’s discretion. A resource move cannot change sample membership, reward association, update order, or random-state lineage without declaring a new experiment. Deterministic replay on a small job, checksum continuity across migration, and periodic comparison against a static schedule can detect silent drift. Throughput is valuable only when the resulting policy is the policy the training program intended to produce.
Source and copyright notice
This article is an editorial analysis by Silicon & Systems. It restates the architecture, scheduling logic, measurements, and limitations in our own words. No source sentence, table, or figure is reproduced; the figure was created for this article. The paper is available from the USENIX OSDI 2026 presentation page. Copyright remains with the authors, 2026.