Reasoning inference is not ordinary chat with more tokens. In the paper’s 100,000-sample workload, 45% of outputs exceed 5,000 tokens, while inputs remain much shorter. Every generated token extends persistent KV state, so memory capacity can saturate before GPU compute reaches its limit[1].

Data parallelism does not pool memory

The study evaluates models from 8B to 671B on eight-H200 nodes. Data parallelism replicates weights and partitions requests. It delivers near-linear aggregate throughput for a small model, including about 4× from two to eight GPUs, but each replica keeps an independent HBM pool. Long and variable requests fragment that pool; new admissions trigger preemption and recomputation even when another replica has unused memory.

Reasoning inference’s capacity transition. Long outputs grow KV state until data-parallel replicas throttle independently. Tensor parallelism pools HBM and becomes worthwhile near the 32B model crossover, while dense 405B and sparse 671B models require different communication strategies. Original figure created for this article.

Tensor parallelism shares model and KV state across GPU memory but pays collectives on every layer. The paper observes a crossover near 32B where added capacity outweighs communication. Dense Llama 405B remains bandwidth and interconnect sensitive and favors high-degree TP. Sparse DeepSeek-R1 671B adds routing and synchronization costs, so hybrid TP and pipeline arrangements can be better. No one parallelism degree fits both.

What we take from it

Reasoning service must tune time to first token, time per output token, end-to-end latency, KV occupancy, and throughput together. Increasing admission can improve TTFT while making long decodes slower and tails unstable. The infrastructure implication is explicit memory control: pooled or tiered HBM, DRAM, CXL, and storage, plus schedulers that move KV state according to temperature. Arithmetic capacity alone will not remove this wall.

Source and attribution

This independent summary uses original language and a new figure. It reproduces no source prose, chart, or table. The authors provide a public arXiv manuscript; the ISCA proceedings version is © IEEE 2026.