The H800 is a Hopper GPU with its wings clipped for export compliance: FP64 throughput is reduced, and NVLink bandwidth drops from the 900 GB/s of an H100 to 400 GB/s. DeepSeek trained its 671-billion-parameter V3 model on 2,048 of them, backfilling the missing intra-node bandwidth with eight 400 Gbps InfiniBand NICs per node. At the ISCA 2025 industry track, the company’s infrastructure team published what amounts to the engineering ledger of that constraint[1]: which model choices were forced by the interconnect, what the resulting speed limits are, and, unusually for an architecture venue, a numbered list of demands addressed to the people who design the next generation of hardware. Frontier labs rarely publish this kind of document. Hyperscalers describe their clusters after the fact[6][7], and vendors describe their roadmaps in advance, but a customer writing down exactly where the silicon failed them is a rarer artifact, and it reads as a demand-side specification for the scale-up fabrics now being standardized.
We summarize the argument in our own words below. The paper spans model architecture, numerics and networking, but its center of gravity is one claim: for Mixture of Experts (MoE) inference, the decode speed of a reasoning model is a property of the interconnect, not of the FLOPS.
Co-design under a bandwidth ceiling
The model-side story is compression at every point where hardware is weakest. Multi-head Latent Attention (MLA) projects the key-value state of all attention heads into one jointly trained latent vector, so decoding caches 70 KB per token where grouped-query designs cache far more (327 KB for Qwen-2.5 72B and 516 KB for LLaMA-3.1 405B, a 4.66× and 7.28× gap, respectively)[1]. DeepSeekMoE activates 37 B of V3’s 671 B parameters per token, which the paper prices at roughly 250 GFLOPS per token of training compute against 2,448 GFLOPS for a 405 B dense model. FP8 arrives not as an inference afterthought but as the training format: V3 was, by the authors’ account, the first open-source large model trained end to end in FP8, using 1×128 tile-wise quantization on activations and 128×128 block-wise on weights, with ablations on 16 B and 230 B models holding the accuracy loss below 0.25% relative to BF16[2]. A Multi-Token Prediction module rounds out the set, drafting a second token per step with an 80 to 90% acceptance rate and lifting generation speed by 1.8×.
The interconnect shapes the model most visibly in expert placement. On an H800 node, the effective bandwidth ratio between NVLink (about 160 GB/s in practice) and one InfiniBand NIC (about 40 GB/s) is roughly 4:1. V3 therefore groups its 256 routed experts into 8 per-node clusters and caps each token’s expert set at 4 nodes: a token crosses InfiniBand once per destination node, and NVLink fans it out to sibling GPUs from there, deduplicating the expensive traffic. The routing function of the model was, in other words, co-authored by a bandwidth ratio.
The decode ceiling, in microseconds
The paper’s most quotable passage is arithmetic. In expert-parallel decoding, every layer performs two all-to-all exchanges (dispatch in FP8, combine in BF16), and each token addresses nine experts (eight routed plus one shared) with a hidden size near 7 K. At a batch of 32 tokens per device over a 50 GB/s (400 Gbps) NIC, one exchange costs about 121 µs; two exchanges across V3’s 61 layers put the floor at 14.76 ms per output token even if computation is perfectly hidden behind communication. That is a hard ceiling near 67 tokens per second per user, before any real-world loss. Run the identical formula on a GB200 NVL72 domain (900 GB/s of unidirectional scale-up bandwidth spanning 72 GPUs) and the per-exchange cost falls to 6.7 µs, the floor to 0.82 ms, and the ceiling to roughly 1,200 tokens per second, an 18× headroom that the authors are careful to label theoretical, since GPU efficiency collapses at small batches long before the network lets go[1].
Note what this calculation makes explicit. Test-time scaling ties model quality to how many tokens a system can emit per second, and reinforcement learning ties training progress to the same number. Since the token rate of an MoE model is bounded by all-to-all latency, the market value of a scale-up domain (an NVL72, or whatever an open consortium eventually ships) can be read directly off this envelope. The paper effectively prices the fabric in tokens per second, which is a more legible unit than GB/s.
Paying the fabric tax in compute
What makes the current arrangement expensive is not only latency but where the patches run. Because NVLink and InfiniBand are separate worlds, forwarding between them is software: GPU Streaming Multiprocessors fill work queues, shuttle data between RDMA buffers and tensor buffers, execute the reduction in the combine step, manage chunked memory layouts, and convert data types on the way in and out. In training, this bookkeeping consumes up to 20 of an H800’s SMs, compute silicon doing a NIC’s job. For inference the team routes all-to-all traffic entirely through NIC RDMA to win those SMs back, and IBGDA (InfiniBand GPUDirect Async) lets GPU threads write doorbells directly, cutting the CPU proxy out of the control path. However, these are workarounds for a boundary the authors argue should not exist.
The wishlist follows from the diagnosis. First, converge scale-up and scale-out into one framework: a unified network adapter or I/O die that fronts both domains under a single address, with enough switching capability to forward an arriving scale-out packet to the right GPU over the scale-up fabric. Second, move the chores into hardware: a dedicated communication co-processor that accelerates memory load/store the way a tensor memory accelerator does, plus in-network broadcast for dispatch and in-network reduction for combine. Third, replace software synchronization with hardware ordering guarantees, ideally acquire/release semantics scoped to memory regions, so that memory-semantic communication does not pay a fence-and-flag round trip per message. The authors name the efforts they expect to deliver this: UALink[3], Ultra Ethernet[4], and Unified Bus, whose UB-Mesh work they cite as a novel take on merging the two domains[5].
Numerics and the network, same complaint
The precision sections file the same grievance against different silicon. Hopper tensor cores accumulate FP8 products in what is effectively FP22 (a 13-bit fraction), which is why DeepSeek’s framework hoists partial sums out to CUDA cores for high-precision accumulation, and why the paper asks for FP32 or at least configurable accumulation precision, plus native support for fine-grained scaling factors so dequantization never leaves the tensor core (Blackwell’s microscaling formats are acknowledged as the first industrial answer). A homegrown logarithmic number format, LogFMT, illustrates the cost of missing hardware from the other direction: at 8 bits it beat E4M3 and E5M2 on training accuracy in the team’s tests, and at 10 bits it approximated BF16, but encoding and decoding on general-purpose GPU pipelines added 50 to 100% overhead to communication kernels, so the format was shelved. A compression codec in the NIC would have shipped it.
The scale-out chapter is the deployed evidence that co-design pays. V3 trained on an eight-plane two-layer fat tree (each GPU-NIC pair pinned to its own plane), a topology the paper costs at $4.39 k per endpoint against $7.5 k for the three-layer tree it replaced, while scaling in principle to 16,384 GPUs; policy constraints kept the actual deployment just above 2,000. Measured NCCL and expert-parallel traffic match a single-plane multi-rail network, with the DeepEP library sustaining above 40 GB/s per GPU across 16 to 128 GPUs. The RoCE recommendations are equally concrete: Ethernet switches stripped for RDMA latency (Broadcom’s scale-up Ethernet work is cited approvingly[8]), adaptive routing in place of ECMP’s collision-prone hashing, and virtual output queues per flow so all-to-all incast cannot starve an adjacent all-reduce. InfiniBand keeps its 2.8 µs latency edge over RoCE’s 3.6 µs in their measurements, but 64-port switch radix and cost keep it from being the long-term answer.
What we take from it
Two readings coexist in this paper. Read as a systems report, it shows a lab compensating for a halved scale-up link so effectively (routing co-design, FP8 end to end, multi-plane networking) that V3’s training cost became an industry talking point. Read as a market signal, it is the most precise demand curve yet published for the fabrics that UALink, Ultra Ethernet and Unified Bus are racing to supply: the customer has already translated bandwidth into tokens per second and named the features it will pay for, from in-network reduction to region-scoped ordering. We believe the second reading matters more in 2025. The supply side of this story (a 200 Gbps per-lane, 1,024-accelerator scale-up standard on one hand, and memory-semantic CXL fabrics reaching for the same rack on the other) has appeared in our earlier coverage of one-chip-like datacenter design and Panmnesia’s silicon for switched CXL, and this paper supplies the missing half: what the buyer actually needs, stated in microseconds. Note that the headline numbers are envelopes, not measurements (the authors say so themselves), and the H800’s 400 GB/s constraint is a policy artifact rather than a law of physics. The wishlist, however, does not depend on the handicap: every item on it would pay off identically on unrestricted hardware.
Source and attribution
This article is an editorial summary prepared for Silicon and Systems. It restates the argument of the paper cited below in our own words. No text, figures or tables from the paper are reproduced here, and the figures on this page were created for this summary. The paper appeared in the ISCA 2025 industry track; the authoritative version is in the Proceedings of the 52nd International Symposium on Computer Architecture, (c) 2025 the authors, publication rights licensed to ACM. An author-prepared version is openly available at arXiv:2505.09343.