GPU training systems optimize collective communication, parallelism, and model FLOP utilization, yet every step begins with bytes. ByteDance’s production environment stores exabytes in HDFS and runs jobs from billion-parameter text models to trillion-parameter multimodal models. Replacing that data lake would require moving data shared with many non-AI pipelines.
The authors analyze 30,000 training traces across 90 days and find three different synchronization failures. Evaluation reads checkpoints across datacenters through many small tensor accesses. Job startup makes thousands of workers request the same metadata and parameters. Multimodal training decodes and transforms video and images on training CPUs, with the slowest host delaying every GPU at the step barrier.
The response is not one faster filesystem. Each bottleneck exposes information that the storage layer did not previously receive: which checkpoint a remote evaluation will need, which files a job will fan out at startup, and which samples the next deterministic step will consume.
Cross-datacenter evaluation is latency-bound, not bandwidth-bound
Companion evaluation loads recent training checkpoints and runs quality suites while training continues. Evaluation hardware and schedules can place it in another datacenter. A checkpoint contains many tensor shards and the reader issues thousands of small accesses, so WAN round trips dominate even when aggregate bandwidth is available.
A global namespace already knows file identity across sites. The new service observes checkpoint creation and evaluation scheduling, predicts the target datacenter, and replicates the complete checkpoint before the evaluator opens it. Consumers retain the same path; namespace placement changes which physical copy serves the read.
Prediction must bind to a committed checkpoint. Copying shards while training is still writing can expose a mixed version. The replication controller verifies completion and checksum metadata before publishing the remote location. It also expires replicas after evaluation so periodic checkpoints do not become permanent multi-site capacity.
The evaluated path reduced wasted GPU hours per evaluation from 16,800 to 4,000. The unit includes accelerators allocated while remote data delays the workflow, not only storage-server time. A cost model should include WAN transfer and replica capacity, but one proactive sequential copy can be cheaper than many latency-bound reads holding a large GPU allocation.
Startup is a flash crowd over a small shared set
Large jobs restart for debugging, faults, preemption, and configuration changes. At startup, every worker needs global metadata and some replicated parameters before sharded steady-state I/O begins. A few storage replicas therefore receive a synchronized read storm from thousands of clients.
The controller distinguishes globally shared hot files from per-worker shards and raises their replication factor proactively. In the 2,048-GPU evaluation, hot metadata and replicated-parameter files were expanded to 128 replicas. Checkpoint loading fell from 38.5 to 22.78 seconds, a 40.8% improvement.
Replication is temporary and workload-scoped. Leaving 128 copies of every shard would multiply capacity and write work. The scheduler or training service needs to announce job size and startup phase early enough for copies to complete; a sudden recovery can still arrive before preparation.
The result also depends on storage topology. Replicas should spread across failure domains and network paths, not just node names. A rack-local concentration can move the bottleneck from disks to a top-of-rack link. Telemetry needs request fanout, replica readiness, per-node queue depth, and client completion distribution.
Multimodal transformation becomes the step’s straggler
Text tokenization can hide behind GPU compute. Multimodal batches add video decode, frame sampling, resize, crop, and image transformation. The deterministic loader stores raw media in large binary bins and maps logical samples to offsets. Training hosts read the bytes, transform them on CPU, and transfer tensors to GPUs.
For a large multimodal trace, transformation accounted for 94.4% of data-loading latency. The slowest sampled host took 42.72 seconds while average loading was far smaller. Synchronous training waits for that host, wasting more than 10,000 GPU hours per day in the reported fleet.
Pretransforming every possible output is impractical. Decoded video can be 40 to 100 times larger, and crop size, resolution, frame count, and augmentation change with model experiments. Regenerating petabytes would consume capacity and delay research.
Storage nodes, however, often waited on network and disk with CPU utilization around 20% to 30%. The deterministic loader knows future sample identifiers and step progress. It sends that schedule to storage, whose nodes read raw bins and execute registered transformation operators for step N+1 while GPUs compute step N.
For video, frame selection occurs beside the stored bytes so unused frames do not cross the network. Returned payloads remain bounded tensors rather than complete decoded video. The training client accepts either a transformed tensor or raw binary, allowing transparent fallback.
Backpressure protects the shared data lake
Storage-side compute can steal CPU from metadata, replication, recovery, and other tenants. The service sets a safety threshold, cited as 80% CPU. Above it, a node aborts transformation and returns raw bytes; the training host detects the format and performs its normal local path.
This fallback makes the optimization opportunistic. It prevents one training job from turning an exabyte shared store into an unbounded compute cluster. It also means performance can vary with unrelated storage tenants, so the scheduler should expose offload availability rather than assuming constant throughput.
Offload reduced p99 data-loading latency 85.7%, cut stalls caused by loading 63.2%, and improved relative training performance 10.8%. Training-host data-loading CPU fell 94%. These values measure a multimodal workload and a storage fleet with spare CPU; lean object appliances or CPU-saturated erasure coding nodes may not reproduce them.
Processed-data caching had limited value for one-pass pretraining because the next epoch may not revisit a sample before parameters or augmentation change. Repeated fine-tuning and reinforcement learning can have more reuse. Cache keys must include dataset version, transform code, parameters, and random seed to prevent training on stale tensors.
Determinism is the optimization interface
All three techniques rely on future knowledge. A checkpoint lifecycle predicts evaluation input. A scheduler predicts startup fanout. A deterministic loader predicts samples. The storage system becomes proactive without embedding model-training logic into HDFS’s block path.
That interface needs explicit uncertainty. A canceled evaluation leaves a replica unused. A failed startup can create hot copies for a job that never runs. Dynamic data mixing or online sampling can invalidate the next-batch list. Controllers should record prediction hit rate, bytes prepared but unused, lead time, and fallback rate.
Privacy and governance follow the copy. Cross-DC replication may cross jurisdiction or data-residency boundaries. Checkpoint metadata must carry placement policy, encryption keys, retention, and deletion obligations. A global namespace cannot treat every visible path as transferable.
Transform execution also introduces supply-chain risk. Registered decoders process untrusted or malformed media on storage nodes. Operators need sandboxing, resource limits, signed transform versions, and deterministic numerical tests. A decoder crash must not affect the storage daemon or corrupt raw data.
Upgrade the old system where the mismatch is measured
The paper argues against a greenfield AI store because HDFS already holds exabytes and serves other processing. The three extensions are narrow: namespace-guided copy, replica count adjustment, and a bounded compute sidecar. They preserve existing paths and file formats.
This is strongest when data gravity dominates migration cost and workload intent is available. A new deployment without legacy data may prefer an AI-native store with collective loading, GPU-direct paths, or richer sample services. The reported work does not claim HDFS is intrinsically optimal; it shows legacy architecture can meet training objectives when control-plane determinism is exposed.
Capacity planning should price GPU waiting against storage resources. One second saved on 2,048 GPUs can justify much more replication and CPU than one second saved on a small job. Policies should allocate proactive work by expected accelerator-hour return rather than treat every file equally.
The larger lesson is that training data pipelines are coordinated distributed systems, not background readers. At scale, a small shared file, a remote namespace decision, or one slow decoder gates thousands of accelerators. ByteDance’s reductions come from predicting those barriers and moving work before they form.
Source and copyright notice
This article is an editorial analysis by Silicon & Systems. It restates the production analysis, mechanisms, measurements, and limits in our own words. No source sentence, table, or figure is reproduced; the figure was created for this article. The paper is available from the USENIX OSDI 2026 presentation page. Copyright remains with the authors, 2026.