Reliability work often starts from the largest training run because its failure probability is easiest to see. Meta’s eleven-month study covers two clusters, four million jobs, and more than 150 million A100 GPU-hours, then shows why that focus is incomplete[1].
Job size changes both risk and impact
Jobs using at least 4,000 GPUs were fewer than 1% of submissions but consumed 12% of GPU resources. The fitted model gives a 1,024-GPU job a mean time to failure of 7.9 hours. Extrapolation makes still larger jobs increasingly checkpoint-bound. However, small jobs dominate count and can be preempted or delayed when large jobs fail and restart. The study attributes 16% of total failure overhead to these second-order effects.
The paper uses Effective Training Time Ratio (ETTR), useful training time divided by total occupied time, to connect failure rate, checkpoint interval, restart cost, and queue delay. For a projected 12,288-GPU run, reaching about 0.9 ETTR requires either checkpoint writes near ten seconds or a material reduction in failure rate. Health checks reduced failures for jobs above 512 GPUs from 14% to 4% in one reported intervention.
What we take from it
MTTF alone cannot guide a shared research cluster. Operators need ETTR, job mix, preemption cascades, and the time needed to identify unhealthy nodes. Hardware redundancy, scheduler policy, checkpoint systems, and debugging tools therefore share one reliability budget. The important operational conclusion is that optimizing a heroic training run while destabilizing thousands of small experiments can reduce total research output.
Source and attribution
This digest uses independent wording and a newly created figure. It reproduces no source prose, figure, or table. The public preprint is available on arXiv; the proceedings version is © IEEE 2025.