Recommendation training is not one GPU job. Sparse feature processing and sample construction consume large CPU fleets, embedding and model updates use CPUs or GPUs, and examples arrive both as historical batches and live events. At ByteDance, daily data for one model grew from 20 TB to 160 TB in five years, while daily CPU virtual-core demand grew from 1.5 million to 9 million. A single model can process more than 20 PB over its training history.
Primus is the system built to keep that growth from producing a separate stack for every scheduler, store, and learning mode[1]. It presents one declarative training contract above YARN and Kubernetes, multiple storage systems, and offline, online, or mixed training. As of 2025, the deployment covered over ten million virtual CPU cores, a GPU estate numbering in the tens of thousands, and 7 EB of recommendation-training data.
The paper’s three “unified” layers solve different bottlenecks. Resource unification controls heterogeneous executors and changes their number or size. Data unification describes sources, transformations, and tasks without embedding each store in a framework. Training unification mixes fresh streams with historical batches so model updates can remain timely without forgetting older distributions.
A common resource object above two schedulers
Primus represents a job with custom resource definitions rather than exposing YARN or Kubernetes details to the training framework. A training master watches job objects. A resource controller allocates executors in the available pool, monitors CPU, memory, I/O, and progress, then applies horizontal or vertical scaling. The training code sees a consistent executor contract even when the underlying scheduler differs.
Horizontal scaling changes the executor count. It suits parallel preprocessing and other work that can repartition. Vertical scaling changes CPU or memory assigned to existing executors and avoids a restart when the bottleneck is per-process capacity. Primus selects a strategy from job and cluster conditions rather than assuming that adding GPUs is the only elastic action.
One production experiment raised CPU utilization per job from about 50% to 80%. In a cluster rollout, training throughput per core rose from 30.26 to 35.44, reported as a 17.1% improvement and cost reduction. Another vertical-scaling case increased throughput from 275 to 496 minibatches per second after correcting memory allocation. These results measure resource matching, not a faster recommendation algorithm.
Elasticity still needs a stable unit of progress. Changing executor count can disturb data ownership or collective communication, and vertical changes can move the bottleneck from memory to I/O. Primus uses production telemetry and conservative adjustment windows. The paper’s savings are therefore evidence about a managed fleet, not proof that every job should scale continuously.
Data is a task graph, not a path string
Recommendation examples can live in distributed files, tables, feature stores, and streams. Training systems that accept a single path force preprocessing logic into framework code and make a new storage format an invasive change. Primus separates logical source definition, physical partitions, and executable tasks into a three-tier data description.
A planner expands a requested time range and source mixture into a data-task graph. Data executors read, transform, and serve samples to training executors over RPC. The graph represents dependencies and parallelism, while adapters contain storage-specific access. Batch and stream inputs can appear in one job without teaching TensorFlow or PyTorch every backend.
For a 20-day production input from two sources, the original serial task generation took 58 minutes. The proposed distributed generation reduced it to 149 seconds, 23 times faster, and four planner threads reduced it further to 42 seconds. Task creation had become a control-plane bottleneck before training began; parallelizing it removes idle accelerators without changing model math.
Data execution remains sensitive to skew. A partition with more examples or slower storage can hold the step boundary. Primus redistributes tasks and separates planning from loading, but the operator must still measure useful samples per second, storage traffic, and executor CPU. A faster graph generator does not guarantee balanced graph execution.
The task graph also becomes the accounting boundary. A platform can attribute bytes read, transformation CPU, retries, and completed samples to a source rather than charging every cost to the final trainer. That visibility is necessary when one model combines a cheap archived table with an expensive live feature stream. Without it, unified access can hide cross-team subsidies and make a technically efficient job economically unpredictable. Quotas should therefore apply to data tasks as well as executors, and retries should preserve identifiers so failed storage work is not counted as useful training progress.
Schema evolution needs similar discipline. A feature definition may change while an offline window spans the old and new form. The logical data object must pin versions, transformations, and time ranges so a rerun observes the same sample meaning. Otherwise faster planning produces a graph that is operationally valid but scientifically irreproducible.
Freshness without abandoning history
Pure online training updates quickly from streams but can overfit a shifting recent distribution and forget older behavior. Pure offline training preserves broad history but publishes updates slowly and handles delayed feedback poorly. Running two independent systems adds model dump, load, and consistency boundaries.
Primus supports a mixed training runtime that combines multiple batch and stream sources in one update process. It controls parameter updates and assigns fine-grained priorities to sources. When stream ingestion lags or load changes, the policy can protect fresh samples without discarding historical batches. The model does not cross a separate offline-online serialization boundary.
Four production recommendation models showed offline evaluation AUC improvements of 0.03% to 0.07%. Online advertising A/B tests reported revenue gains from 0.4% to 2.4%. The difference in scale is normal: small ranking changes can affect many auctions, and revenue is a product metric rather than a direct transformation of AUC. The range should not be averaged with infrastructure savings.
Freshness also introduces bias and feedback risk. Source priorities determine which population influences each update. A delayed conversion label may belong with an older impression, and a temporary traffic spike can dominate a stream. Primus provides the mechanism to mix sources; business owners still need sampling, attribution, and rollback rules.
Five years of operation are the strongest evidence
Primus has supported ByteDance recommendation training for five years across products including Douyin, Xigua, and Toutiao. Thousands of models share the platform. This history establishes that the abstractions survived scheduler and storage evolution, which is more informative than a single benchmark for an infrastructure layer.
It also makes the results environment-specific. ByteDance owns the schedulers, feature pipelines, resource telemetry, and advertising feedback loop. A smaller organization may not have enough jobs for horizontal pooling or enough source diversity to justify the same control plane. The useful question is whether one common contract removes duplicated integrations, not whether another fleet can reproduce 10 million cores.
The central architecture can become an organizational dependency. A change to JobCRD or DataCRD affects many frameworks and teams. Compatibility, versioning, quota isolation, and debugging tools matter as much as planner speed. A unified platform succeeds only if it preserves enough escape hatches for unusual models without fragmenting into private extensions.
Reading the three denominators correctly
Primus reports infrastructure efficiency, control-plane speed, and model outcome. The 17.1% value concerns training throughput per core and cluster cost. The 23-times value concerns task-graph generation time. The 0.4% to 2.4% values concern advertising revenue after changing the training mixture. They answer different questions and should never be multiplied into one platform return.
For a deployment review, an operator should establish an equivalent job on the old and new resource controller, measure executor utilization and completed samples, then test planner latency as source and day count grow. Hybrid training needs a separate online experiment with freshness, model quality, and business guardrails. Only the last step can attribute a product result.
The reusable lesson is that heterogeneous training becomes manageable when resources, data, and update semantics have independent but composable contracts. Primus can change a scheduler without rewriting data logic, add a store without changing model code, and mix streams without launching another training stack. Its scale shows the value of the separation; its metrics remind readers to evaluate each layer on its own denominator.
Source and copyright notice
This article is an editorial analysis by Silicon & Systems. It restates the architecture, measurements, and limits in our own words. No source sentence, table, or figure is reproduced; the figure was created for this article. The paper is available from the USENIX ATC 2025 presentation page. Copyright remains with the authors, 2025.