Reinforcement learning from human feedback looks tidy on a whiteboard: an actor generates responses, a critic, a reference policy and a reward model score them, and the actor and critic learn from the result. On a GPU cluster the tidiness evaporates, since each of those boxes is itself a distributed LLM sharded with its own tensor, pipeline and data parallelism, and each arrow is a many-to-many reshuffle between two different sharding layouts. ByteDance and the University of Hong Kong published the systems answer at EuroSys 2025[1], and the answer has since become infrastructure: HybridFlow is the design inside verl, the open-source framework that a large share of the reasoning-model wave now runs its RL post-training on. The paper’s claim is a throughput improvement of 1.53 to 20.57× over the RLHF systems of the day; its longer-lived contribution is a programming model that made new RL algorithms a few-line change instead of a rewrite.
We summarize the argument in our own words below.
Two paradigms, each half right
Distributed ML has two ways to run a dataflow. A single controller owns the whole graph, dispatching every operation to remote workers: maximally flexible, and ruinously chatty when a node is an LLM with billions of operators, since dispatch messages cross the cluster for every step. A multi-controller design instead gives every device its own program: dispatch cost disappears (control stays on the CPU-to-GPU path), which is why Megatron-style training and modern serving engines all work this way, but the dataflow logic now lives nowhere. Each model’s script must hand-code its sends and receives to its neighbors, so changing one edge of the RLHF graph means editing every dependent model’s program. The 2024-era RLHF systems (DeepSpeed-Chat, OpenRLHF, NeMo-Aligner)[2][3][4] all chose the multi-controller side and inherited its rigidity: one placement plan, one execution pattern, PPO only, and a codebase where communication and computation are braided together.
HybridFlow’s observation is that the choice is only forced if made globally. An RLHF graph has a handful of nodes, so a single controller orchestrating at the node level costs almost nothing; the expense was only ever in dispatching operators inside a node. The framework therefore runs one Ray-based controller[6] over the graph and a multi-controller worker group inside each model. Model classes (built on Megatron-LM, FSDP or DeepSpeed for training, vLLM for generation) encapsulate the distributed computation behind primitive calls like generate_sequences or update_actor, and every call is tagged with a transfer protocol: a collect function that gathers the model’s output according to its own sharding, and a distribute function that scatters inputs according to the consumer’s. Data moves GPU-to-GPU with only futures passing through the controller. The payoff is measured in lines of code: PPO is 8 lines of controller script[5], Safe-RLHF adds 5, ReMax deletes the critic and adds a generation call, and none of them touch the models’ internals.
Moving the actor without paying for it twice
The heaviest node is the actor, which trains (compute-bound, wanting wide model parallelism) and generates (memory-bound, wanting many small replicas) in every iteration. Prior systems handled the mismatch badly in three distinct ways: NeMo-Aligner reused the training layout for generation and idled its GPUs; OpenRLHF kept two copies of the actor on different devices and synchronized weights each iteration; DeepSpeed-Chat resharded on the same devices but gathered the full model on every GPU in the process, a transfer that can eat a third of the iteration for a 70 B model. HybridFlow’s 3D-HybridEngine keeps one copy of the weights on one set of GPUs and reshards in place between a training layout (p-t-d) and a generation layout with smaller tensor-parallel groups plus micro data-parallel replicas. The trick is in how generation groups are drawn: ranks are assigned at strides chosen so that each GPU’s generation shard overlaps its training shard, which makes the transition an all-gather confined within each micro group, with zero duplicated weights resident anywhere. Against the baselines this cuts transition time by 55.2% on average and up to 89.1% (78.2 seconds per iteration) at 70 B, and the freedom to shrink generation tensor parallelism pays directly: dropping from the training width to 2-way cuts 7 B generation latency by 60.3%.
Placement, the remaining degree of freedom, gets an optimizer rather than a convention. Because models are placed by binding them to virtual resource pools, any partition of models onto device sets is expressible, and an auto-mapping algorithm enumerates the placements (15 for PPO’s four models), sizes each colocated set against memory floors, searches parallelism per model with a simulator, and picks the plan with the shortest iteration. The sweep reproduces an operational truth the fixed-placement systems could not act on: colocating everything wins up to about 64 GPUs, splitting training and scoring models wins in the middle, and fully standalone placement wins at 128 GPUs, with the algorithm finding each crossover on its own in under half an hour of CPU time.
What the numbers say
On 128 A100s with Llama-family models from 7 B to 70 B, HybridFlow outperforms DeepSpeed-Chat by 3.67× on average (up to 7.84×), OpenRLHF by an average of 3.25× with a 5.93× peak, and NeMo-Aligner by 12.52× on average (peaking at 20.57×) on PPO throughput, with similar margins on ReMax and Safe-RLHF and the largest average gain (9.64×) at 70 B, where transition overhead dominates the baselines. Strong-scaling efficiency averages 66.8% across algorithms and sizes. Note the caveats the setup implies: all four models share a size in the headline runs, response lengths are fixed for fairness against baselines that lack continuous batching, and the largest speedups are measured against NeMo-Aligner’s generation path, whose engine lacked a KV cache and spent up to 81.2% of each iteration generating. The fairest reading is not the top of the range but the shape of it: every baseline is beaten everywhere, by construction rather than by tuning, because each model in the dataflow gets the parallelism and placement its own workload wants.
What we take from it
The measure of this paper in 2026 is that its artifact outgrew its benchmark. verl is now the default substrate for open RL post-training work, the reasoning-model boom made RL throughput a first-order economic quantity, and the ideas here (hybrid control, protocol-mediated resharding, in-place actor transition) are what let one framework absorb algorithm after algorithm without rewrites. There is also a supply-chain observation worth making: the same company’s Seed team trains frontier models, and the demand side we covered in DeepSeek’s hardware wishlist[7] pointed out that RL ties training progress to inference token rates. HybridFlow is what that dependency looks like from the software side, an admission that generation is now inside the training loop, and that the boundary between serving systems and training systems has stopped being an organizational convenience and become a resharding problem. We believe the hybrid-controller idea generalizes past RLHF: any workload whose graph is small but whose nodes are enormous (agentic pipelines and multi-model systems increasingly qualify) faces the same choice this paper refused to make.
Algorithm semantics must survive every placement
The abstraction is valuable only if changing placement does not change the learning procedure. PPO, ReMax, and Safe-RLHF differ in which models they invoke and which tensors cross an edge, but a placement optimizer should not alter sample identity, model version, or update order. Every transfer protocol therefore represents more than a copy routine. It is the contract that says how a logical batch is collected from one sharding scheme and reconstructed for the next model.
This matters most around the actor. Generation may use many narrow replicas while training uses one wider partition. A rollout must still be associated with the precise actor version that produced it, and the next update must consume the intended reward and reference outputs. If an asynchronous copy or failed rank leaves one generation group on an older version, the framework can remain fast while the algorithm silently becomes different. A production system should tag samples and weight states with iteration identifiers, verify them at each transfer boundary, and fail the iteration rather than mix versions.
The same contract should include failure behavior. HybridFlow’s paper evaluates steady execution, while long post-training runs also encounter worker replacement, engine restart, and partial communication failure. The controller can see the small graph, so it is the natural place to decide which nodes must be replayed after a failure. Worker groups should expose whether an operation is idempotent and whether their output has become visible downstream. Without this information, flexible orchestration can increase the number of ambiguous recovery states.
A placement has a useful lifetime
Auto-mapping chooses among measured or simulated configurations, but its answer is tied to the profile that produced it. Response length, policy quality, reward-model load, sequence packing, kernel versions, and cluster contention can all change during one campaign. The actor may generate longer traces as the policy improves, or a new inference engine may lower generation time enough to move the bottleneck back to training. The plan that minimized the first thousand iterations need not minimize the final thousand.
Operators should retain the optimizer’s inputs and compare predicted node time with observed node time. A persistent error should trigger a bounded remapping experiment, not an immediate global move. The experiment must subtract transition cost, including weight movement, process-group creation, cache warmup, and any paused iterations. A configuration that is 5% faster after a ten-minute transition is unattractive for a job with twenty minutes remaining and compelling for a run with several days remaining.
This lifecycle also clarifies which performance number can travel. The reported 1.53× to 20.57× range demonstrates that the framework avoids several severe baseline mismatches. It does not set a universal speedup for a current verl deployment. A portable result is the reduction in actor-transition work and the ability to search colocated, partially split, and standalone mappings under one execution model. Each new cluster should rerun that search with its own network and workload.
The capacity unit is a completed policy update
GPU utilization can rise while useful RL progress falls. A rollout engine may produce tokens rapidly but leave the trainer waiting for a synchronized batch. A trainer may report high MFU while the generation pool waits for a new actor version. HybridFlow’s graph makes these dependencies visible, so capacity should be measured at the graph boundary: completed policy updates per hour, together with samples, generated tokens, and accelerator-hours consumed for each update.
This accounting prevents one component from exporting cost to another. Narrower generation replicas can improve token throughput but add resharding or synchronization. Colocation can eliminate network transfers but reduce memory available for batching. Standalone placement can maximize each kernel while creating an expensive edge between models. The optimizer should price all of these effects in the same iteration ledger rather than maximizing a local metric.
The operational lesson is straightforward. Keep algorithm specification, execution protocol, and hardware mapping as separate objects, but record the version of all three for every run. Then a new RL method can change the graph without rewriting distributed kernels, a new GPU can change the mapping without changing learning semantics, and a failed experiment can be reproduced from more than a list of device counts. That separation is the reason HybridFlow became infrastructure rather than remaining a single benchmark result.
For an adoption review, we would ask for one iteration ledger before requesting a large cluster test. The ledger should divide time among rollout, reward and reference inference, critic and actor updates, each data redistribution, and idle gaps. It should also state the weight and optimizer bytes resident on every resource pool. This view reveals whether the candidate workload has the alternating layouts and expensive edges that HybridFlow is designed to repair.
The next test should change one condition at a time: response length, model-size asymmetry, GPU count, and network topology. If the selected placement changes for a reason that matches the ledger, the abstraction is doing useful work. If one plan wins everywhere regardless of the measured bottleneck, the search space or cost model is probably missing a constraint. This is a stronger acceptance criterion than reproducing the paper’s largest multiplier.
Source and attribution
This article is an editorial summary prepared for Silicon & Systems. It restates the argument of the paper cited below in our own words. No text, figures or tables from the paper are reproduced here, and the figures on this page were created for this summary. The paper appeared at EuroSys 2025; the authoritative version is in the Proceedings of the Twentieth European Conference on Computer Systems, (c) 2025 the authors, publication rights licensed to ACM. An author-prepared version is openly available at arXiv:2409.19256.