Llama 3 405B consumed about 3.8 × 10^25 floating-point operations during pretraining on 16,384 H100 GPUs. At that scale, the question is not whether to parallelize but which dimension should pay each communication cost. Meta’s ISCA 2025 paper documents a four-dimensional answer and, more importantly, why the answer must change during training[1].
Four partitions solve different limits
Fully sharded data parallelism distributes parameters, gradients, and optimizer state. Tensor parallelism splits individual layers and pays frequent collective communication. Pipeline parallelism assigns layer groups to stages but creates bubbles when microbatches are scarce. Context parallelism divides long sequences, reducing per-GPU activation pressure while introducing attention communication.
No one dimension dominates. Llama 3 has phases with different sequence lengths, batch sizes, model structures, and GPU allocations. Long-context training can reduce the global batch so far that a conventional pipeline schedule lacks enough microbatches to fill stages. Meta therefore developed a pipeline schedule that tolerates changing batch shapes and a context-parallel method that supports document-mask attention rather than assuming one continuous sequence.
Efficiency includes diagnosis
The implementation reports about 400 TFLOPS per GPU for an 8K sequence and 380 TFLOPS per GPU at 131K. One context-parallel attention experiment achieves 3.89× scaling from one to four GPUs. These figures are useful operating points, not isolated kernel peaks. They include schedules designed around a model whose batch and architecture evolve.
The paper gives debugging equal architectural weight. A numerical error or straggler that appears only after thousands of ranks synchronize cannot be treated as an ordinary application bug. Profiling, deterministic comparison, and identifying the responsible rank are necessary to preserve useful machine time. At 16,384 GPUs, observability sits on the critical path to performance.
What we take from it
The common description of distributed training as a choice among data, tensor, or pipeline parallelism is obsolete at frontier scale. The actual system is a changing composition of all four, constrained by batch semantics and the network hierarchy. Future accelerators should therefore expose fast collectives, enough memory for flexible partition changes, and counters that make numerical and performance faults attributable. Raw arithmetic is only one term in the training-time equation.
Source and attribution
This article independently summarizes the paper in new language. The figure is original and no source figure or table is reproduced. The public paper is licensed under CC BY 4.0; copyright is held by the authors, © 2025.