Reinforcement learning from human feedback looks tidy on a whiteboard: an actor generates responses, a critic, a reference policy and a reward model score them, and the actor and critic learn from the result. On a GPU cluster the tidiness evaporates, since each of those boxes is itself a distributed LLM sharded with its own tensor, pipeline and data parallelism, and each arrow is a many-to-many reshuffle between two different sharding layouts. ByteDance and the University of Hong Kong published the systems answer at EuroSys 2025[1], and the answer has since become infrastructure: HybridFlow is the design inside verl, the open-source framework that a large share of the reasoning-model wave now runs its RL post-training on. The paper’s claim is a throughput improvement of 1.53 to 20.57× over the RLHF systems of the day; its longer-lived contribution is a programming model that made new RL algorithms a few-line change instead of a rewrite.
We summarize the argument in our own words below.
Two paradigms, each half right
Distributed ML has two ways to run a dataflow. A single controller owns the whole graph, dispatching every operation to remote workers: maximally flexible, and ruinously chatty when a node is an LLM with billions of operators, since dispatch messages cross the cluster for every step. A multi-controller design instead gives every device its own program: dispatch cost disappears (control stays on the CPU-to-GPU path), which is why Megatron-style training and modern serving engines all work this way, but the dataflow logic now lives nowhere. Each model’s script must hand-code its sends and receives to its neighbors, so changing one edge of the RLHF graph means editing every dependent model’s program. The 2024-era RLHF systems (DeepSpeed-Chat, OpenRLHF, NeMo-Aligner)[2][3][4] all chose the multi-controller side and inherited its rigidity: one placement plan, one execution pattern, PPO only, and a codebase where communication and computation are braided together.
HybridFlow’s observation is that the choice is only forced if made globally. An RLHF graph has a handful of nodes, so a single controller orchestrating at the node level costs almost nothing; the expense was only ever in dispatching operators inside a node. The framework therefore runs one Ray-based controller[6] over the graph and a multi-controller worker group inside each model. Model classes (built on Megatron-LM, FSDP or DeepSpeed for training, vLLM for generation) encapsulate the distributed computation behind primitive calls like generate_sequences or update_actor, and every call is tagged with a transfer protocol: a collect function that gathers the model’s output according to its own sharding, and a distribute function that scatters inputs according to the consumer’s. Data moves GPU-to-GPU with only futures passing through the controller. The payoff is measured in lines of code: PPO is 8 lines of controller script[5], Safe-RLHF adds 5, ReMax deletes the critic and adds a generation call, and none of them touch the models’ internals.
Moving the actor without paying for it twice
The heaviest node is the actor, which trains (compute-bound, wanting wide model parallelism) and generates (memory-bound, wanting many small replicas) in every iteration. Prior systems handled the mismatch badly in three distinct ways: NeMo-Aligner reused the training layout for generation and idled its GPUs; OpenRLHF kept two copies of the actor on different devices and synchronized weights each iteration; DeepSpeed-Chat resharded on the same devices but gathered the full model on every GPU in the process, a transfer that can eat a third of the iteration for a 70 B model. HybridFlow’s 3D-HybridEngine keeps one copy of the weights on one set of GPUs and reshards in place between a training layout (p-t-d) and a generation layout with smaller tensor-parallel groups plus micro data-parallel replicas. The trick is in how generation groups are drawn: ranks are assigned at strides chosen so that each GPU’s generation shard overlaps its training shard, which makes the transition an all-gather confined within each micro group, with zero duplicated weights resident anywhere. Against the baselines this cuts transition time by 55.2% on average and up to 89.1% (78.2 seconds per iteration) at 70 B, and the freedom to shrink generation tensor parallelism pays directly: dropping from the training width to 2-way cuts 7 B generation latency by 60.3%.
Placement, the remaining degree of freedom, gets an optimizer rather than a convention. Because models are placed by binding them to virtual resource pools, any partition of models onto device sets is expressible, and an auto-mapping algorithm enumerates the placements (15 for PPO’s four models), sizes each colocated set against memory floors, searches parallelism per model with a simulator, and picks the plan with the shortest iteration. The sweep reproduces an operational truth the fixed-placement systems could not act on: colocating everything wins up to about 64 GPUs, splitting training and scoring models wins in the middle, and fully standalone placement wins at 128 GPUs, with the algorithm finding each crossover on its own in under half an hour of CPU time.
What the numbers say
On 128 A100s with Llama-family models from 7 B to 70 B, HybridFlow outperforms DeepSpeed-Chat by 3.67× on average (up to 7.84×), OpenRLHF by an average of 3.25× with a 5.93× peak, and NeMo-Aligner by 12.52× on average (peaking at 20.57×) on PPO throughput, with similar margins on ReMax and Safe-RLHF and the largest average gain (9.64×) at 70 B, where transition overhead dominates the baselines. Strong-scaling efficiency averages 66.8% across algorithms and sizes. Note the caveats the setup implies: all four models share a size in the headline runs, response lengths are fixed for fairness against baselines that lack continuous batching, and the largest speedups are measured against NeMo-Aligner’s generation path, whose engine lacked a KV cache and spent up to 81.2% of each iteration generating. The fairest reading is not the top of the range but the shape of it: every baseline is beaten everywhere, by construction rather than by tuning, because each model in the dataflow gets the parallelism and placement its own workload wants.
What we take from it
The measure of this paper in 2026 is that its artifact outgrew its benchmark. verl is now the default substrate for open RL post-training work, the reasoning-model boom made RL throughput a first-order economic quantity, and the ideas here (hybrid control, protocol-mediated resharding, in-place actor transition) are what let one framework absorb algorithm after algorithm without rewrites. There is also a supply-chain observation worth making: the same company’s Seed team trains frontier models, and the demand side we covered in DeepSeek’s hardware wishlist[7] pointed out that RL ties training progress to inference token rates. HybridFlow is what that dependency looks like from the software side, an admission that generation is now inside the training loop, and that the boundary between serving systems and training systems has stopped being an organizational convenience and become a resharding problem. We believe the hybrid-controller idea generalizes past RLHF: any workload whose graph is small but whose nodes are enormous (agentic pipelines and multi-model systems increasingly qualify) faces the same choice this paper refused to make.
Source and attribution
This article is an editorial summary prepared for Silicon and Systems. It restates the argument of the paper cited below in our own words. No text, figures or tables from the paper are reproduced here, and the figures on this page were created for this summary. The paper appeared at EuroSys 2025; the authoritative version is in the Proceedings of the Twentieth European Conference on Computer Systems, (c) 2025 the authors, publication rights licensed to ACM. An author-prepared version is openly available at arXiv:2409.19256.