GPU training systems optimize collective communication, parallelism, and model FLOP utilization, yet every step begins with bytes. ByteDance’s production environment stores exabytes in HDFS and supports both text models measured in billions of parameters and much larger multimodal training. Replacing that data lake would require moving data shared with many non-AI pipelines.
The authors analyze 30,000 training traces across 90 days and find three different synchronization failures. Evaluation reads checkpoints across datacenters through many small tensor accesses. Job startup makes thousands of workers request the same metadata and parameters. Multimodal training decodes and transforms video and images on training CPUs, with the slowest host delaying every GPU at the step barrier.
The response is not one faster filesystem. Each bottleneck exposes information that the storage layer did not previously receive: which checkpoint a remote evaluation will need, which files a job will fan out at startup, and which samples the next deterministic step will consume.
Cross-datacenter evaluation is latency-bound, not bandwidth-bound
Companion evaluation loads recent training checkpoints and runs quality suites while training continues. Evaluation hardware and schedules can place it in another datacenter. A checkpoint contains many tensor shards and the reader issues thousands of small accesses, so WAN round trips dominate even when aggregate bandwidth is available.
A global namespace already knows file identity across sites. The new service observes checkpoint creation and evaluation scheduling, predicts the target datacenter, and replicates the complete checkpoint before the evaluator opens it. Consumers retain the same path; namespace placement changes which physical copy serves the read.
Prediction must bind to a committed checkpoint. Copying shards while training is still writing can expose a mixed version. The replication controller verifies completion and checksum metadata before publishing the remote location. It also expires replicas after evaluation so periodic checkpoints do not become permanent multi-site capacity.
For each detected regression, the evaluated path lowered wasted accelerator time from 16,800 GPU-hours to 4,000. The unit includes accelerators allocated while remote data delays the workflow, not only storage-server time. A cost model should include WAN transfer and replica capacity, but one proactive sequential copy can be cheaper than many latency-bound reads holding a large GPU allocation.
Startup is a flash crowd over a small shared set
Large jobs restart for debugging, faults, preemption, and configuration changes. At startup, every worker needs global metadata and some replicated parameters before sharded steady-state I/O begins. A few storage replicas therefore receive a synchronized read storm from thousands of clients.
The controller distinguishes globally shared hot files from per-worker shards and raises their replication factor proactively. In the 2,048-GPU evaluation, hot metadata and replicated-parameter files were expanded to 128 replicas. Checkpoint loading fell from 38.5 to 22.78 seconds, a 40.8% improvement.
Replication is temporary and workload-scoped. Leaving 128 copies of every shard would multiply capacity and write work. The scheduler or training service needs to announce job size and startup phase early enough for copies to complete; a sudden recovery can still arrive before preparation.
The result also depends on storage topology. Replicas should spread across failure domains and network paths, not just node names. A rack-local concentration can move the bottleneck from disks to a top-of-rack link. Telemetry needs request fanout, replica readiness, per-node queue depth, and client completion distribution.
Multimodal transformation becomes the step’s straggler
Text tokenization can hide behind GPU compute. Multimodal batches add video decode, frame sampling, resize, crop, and image transformation. The deterministic loader stores raw media in large binary bins and maps logical samples to offsets. Training hosts read the bytes, transform them on CPU, and transfer tensors to GPUs.
For a large multimodal trace, transformation accounted for 94.4% of data-loading latency. The slowest sampled host took 42.72 seconds while average loading was far smaller. Synchronous training waits for that host, wasting more than 10,000 GPU hours per day in the reported fleet.
Pretransforming every possible output is impractical. Decoded video can be 40 to 100 times larger, and crop size, resolution, frame count, and augmentation change with model experiments. Regenerating petabytes would consume capacity and delay research.
Storage nodes, however, often waited on network and disk with CPU utilization around 20% to 30%. The deterministic loader knows future sample identifiers and step progress. It sends that schedule to storage, whose nodes read raw bins and execute registered transformation operators for step N+1 while GPUs compute step N.
For video, frame selection occurs beside the stored bytes so unused frames do not cross the network. Returned payloads remain bounded tensors rather than complete decoded video. The training client accepts either a transformed tensor or raw binary, allowing transparent fallback.
Backpressure protects the shared data lake
Storage-side compute can steal CPU from metadata, replication, recovery, and other tenants. The service sets a safety threshold, cited as 80% CPU. Above it, a node aborts transformation and returns raw bytes; the training host detects the format and performs its normal local path.
This fallback makes the optimization opportunistic. It prevents one training job from turning an exabyte shared store into an unbounded compute cluster. It also means performance can vary with unrelated storage tenants, so the scheduler should expose offload availability rather than assuming constant throughput.
Offload reduced p99 data-loading latency 85.7%, cut stalls caused by loading 63.2%, and improved relative training performance 10.8%. Training-host data-loading CPU fell 94%. These values measure a multimodal workload and a storage fleet with spare CPU; lean object appliances or CPU-saturated erasure coding nodes may not reproduce them.
Processed-data caching had limited value for one-pass pretraining because the next epoch may not revisit a sample before parameters or augmentation change. Repeated fine-tuning and reinforcement learning can have more reuse. Cache keys must include dataset version, transform code, parameters, and random seed to prevent training on stale tensors.
Determinism is the optimization interface
All three techniques rely on future knowledge. A checkpoint lifecycle predicts evaluation input. A scheduler predicts startup fanout. A deterministic loader predicts samples. The storage system becomes proactive without embedding model-training logic into HDFS’s block path.
That interface needs explicit uncertainty. A canceled evaluation leaves a replica unused. A failed startup can create hot copies for a job that never runs. Dynamic data mixing or online sampling can invalidate the next-batch list. Controllers should record prediction hit rate, bytes prepared but unused, lead time, and fallback rate.
Privacy and governance follow the copy. Cross-DC replication may cross jurisdiction or data-residency boundaries. Checkpoint metadata must carry placement policy, encryption keys, retention, and deletion obligations. A global namespace cannot treat every visible path as transferable.
Transform execution also introduces supply-chain risk. Registered decoders process untrusted or malformed media on storage nodes. Operators need sandboxing, resource limits, signed transform versions, and deterministic numerical tests. A decoder crash must not affect the storage daemon or corrupt raw data.
Evaluation freshness is a capacity metric
The checkpoint path matters because evaluation is part of the training control loop rather than a reporting task. Over a 30-day sample, 19 pretraining tasks launched 3,589 companion evaluations and identified 156 regressions. Training continued while those results were delayed. The authors estimate that timely evaluation saved 5.5 million GPU-hours, while slow checkpoint delivery allowed roughly 2.6 million GPU-hours to be spent on runs that would later be rejected.
Those numbers have different denominators from startup and data-loading results. The 2.6 million figure is attributed to delayed decisions across detected regressions; the 40.8% result measures a 2,048-worker checkpoint load; the 63.2% stall reduction comes from multimodal transformation. They should not be added or multiplied into a fleet-wide percentage. Their common meaning is that a synchronization boundary converts seconds of data-path delay into many accelerator-hours.
An operator should therefore record evaluation age at the moment a regression is declared. The useful timeline begins when a checkpoint becomes committed, continues through replica readiness and evaluator startup, and ends when the result reaches the training controller. That view distinguishes a storage delay from queued evaluation hardware, slow benchmark execution, or the policy that decides whether to stop training. Faster I/O has value only when the decision loop consumes the saved time.
A rollout needs three separate resource budgets
The mechanisms should not be deployed as one undifferentiated storage feature. Predictive checkpoint transfer spends wide-area bandwidth and remote capacity before its value is certain. Hot-file expansion spends temporary replica byte-seconds inside the source cluster. Storage-side transformation spends CPU, memory, and egress on a service whose first obligation remains durable data access. Each path needs its own admission rule, quota, and abort condition.
For checkpoint transfer, the useful control variable is lead time. Record when a checkpoint becomes readable, when its local copy is complete, and when evaluation requests it. A copy that routinely finishes too early holds duplicate capacity; one that finishes late leaves GPUs on the WAN path. For restart replication, measure barrier reduction per temporary byte and remove extra copies after fan-out ends. Otherwise an incident-specific optimization silently becomes a permanent cache policy.
Transformation requires the strongest isolation. The transform version, input range, output checksum, CPU time, memory peak, and fallback decision should be traceable for every batch. A failed or slow transform must return the original bytes quickly enough for training to continue on its established path. Disabling the optimization should change performance, not sample identity or checkpoint correctness.
Rollout also needs a control group. Enable one mechanism at a time for comparable jobs, preserve the original path, and measure productive GPU time after subtracting replica storage, WAN transfer, and storage CPU. WAN bytes avoided or CPU consumed are diagnostics, not the objective. The deployment succeeds when evaluation decisions arrive sooner, the last recovery rank starts earlier, or synchronized training completes more useful steps without weakening the shared data service.
The controller must know when prediction is unsafe
Proactive work has a failure mode that reactive I/O does not: it can consume resources for demand that never arrives. An evaluation can be canceled, a restart can move to another checkpoint, and dynamic sampling can change the next batch. The controller should attach an expiry and confidence to every hint, then expose prepared-but-unused bytes and CPU time. A high miss rate is evidence to shorten the look-ahead window or disable prediction for that workload.
Freshness must be enforced independently of placement. A complete remote copy of the wrong checkpoint is worse than a slower read of the right one. Namespace publication should require an immutable checkpoint identity, completion marker, and content verification. Transformation should bind the dataset version, operator version, parameters, and random seed to its output. If any identity is missing, the safe choice is the original client path rather than a best-effort transformed tensor.
Shared-service pressure can also invalidate a previously good decision. A storage node with 30% CPU utilization when the plan is issued may enter recovery or receive another tenant’s burst before step N+1. The backpressure threshold handles the last moment, but admission should also consider queue depth, memory pressure, and network egress. This two-stage control avoids accepting more speculative work simply because one resource still appears idle.
Finally, cross-datacenter copying is subject to policy as well as capacity. A globally visible checkpoint may contain data or weights that cannot move into the evaluator’s region. Placement hints must pass residency, encryption, retention, and access-control checks before bandwidth scheduling begins. The optimization is useful only when every speculative copy and transform remains auditable, revocable, and no less protected than the ordinary HDFS path it accelerates.
Upgrade the old system where the mismatch is measured
The paper argues against a greenfield AI store because HDFS already holds exabytes and serves other processing. The three extensions are narrow: namespace-guided copy, replica count adjustment, and a bounded compute sidecar. They preserve existing paths and file formats.
This is strongest when data gravity dominates migration cost and workload intent is available. A new deployment without legacy data may prefer an AI-native store with collective loading, GPU-direct paths, or richer sample services. The reported work does not claim HDFS is intrinsically optimal; it shows legacy architecture can meet training objectives when control-plane determinism is exposed.
Capacity planning should price GPU waiting against storage resources. One second saved on 2,048 GPUs can justify much more replication and CPU than one second saved on a small job. Policies should allocate proactive work by expected accelerator-hour return rather than treat every file equally.
The larger lesson is that training data pipelines are coordinated distributed systems, not background readers. At scale, a small shared file, a remote namespace decision, or one slow decoder gates thousands of accelerators. ByteDance’s reductions come from predicting those barriers and moving work before they form.
Source and copyright notice
This article is an editorial analysis by Silicon & Systems. It restates the production analysis, mechanisms, measurements, and limits in our own words. No source sentence, table, or figure is reproduced; the figure was created for this article. The paper is available from the USENIX OSDI 2026 presentation page. Copyright remains with the authors, 2026.