Reasoning inference is not ordinary chat with more tokens. In the paper’s 100,000-sample workload, 45% of outputs exceed 5,000 tokens, while inputs remain much shorter. Every generated token extends persistent KV state, so memory capacity can saturate before GPU compute reaches its limit[1].
That reversal changes capacity planning. A provider must size for concurrent sequence state and completion-time objectives, not only peak tokens per second or model weights. This review connects the workload distribution to memory occupancy and shows why the resulting GPU requirement is a scenario calculation rather than a universal ratio for every reasoning service.
Data parallelism does not pool memory
The study evaluates models from 8B to 671B on eight-H200 nodes. Data parallelism replicates weights and partitions requests. It delivers near-linear aggregate throughput for a small model, including about 4× from two to eight GPUs, but each replica keeps an independent HBM pool. Long and variable requests fragment that pool; new admissions trigger preemption and recomputation even when another replica has unused memory.
Tensor parallelism shares model and KV state across GPU memory but pays collectives on every layer. The paper observes a crossover near 32B where added capacity outweighs communication. Dense Llama 405B remains bandwidth and interconnect sensitive and favors high-degree TP. Sparse DeepSeek-R1 671B adds routing and synchronization costs, so hybrid TP and pipeline arrangements can be better. No one parallelism degree fits both.
Output length changes the resource reservation
Conventional serving often estimates work from prompt length and a modest maximum response. Reasoning requests reverse that balance. In the evaluated 100,000-sample distribution, 45% of outputs exceed 5,000 tokens. The scheduler cannot know the realized length at admission, yet each active sequence retains state for every processed token until completion.[1]
This uncertainty creates a capacity tail. Reserving the maximum for every request wastes HBM; admitting against the mean can overcommit when several long traces arrive together. Once memory is full, the runtime may preempt sequences, move their state, or discard and recompute it later. These actions consume bandwidth and compute exactly when latency is already growing.
Service policy therefore needs a length model and a response when the model is wrong. Admission can use quantiles, user or task classes, and dynamic headroom. Long requests may enter a separate pool or receive an explicit token budget. The product decision is not purely technical because truncation, queueing, and price affect user-visible quality.
KV cache is persistent working state
Autoregressive attention stores keys and values for prior tokens so the model does not recompute the entire prefix at every step. The allocation grows with active sequences, sequence length, layer count, head structure, data type, and parallel layout. Unlike a temporary matrix workspace, it remains live across thousands of decode iterations.
Weight memory establishes a fixed floor. KV state, fragmentation, communication buffers, and runtime reserve occupy the remaining HBM. A model can fit at startup but fail under a realistic concurrency and output-length distribution. Capacity must be calculated at the serving point, not from parameter count alone.
Paged allocation reduces external fragmentation and lets sequences grow incrementally, but pages do not pool memory across independent data-parallel replicas. One replica can run out while another has free pages. Global routing can redirect new requests, yet it cannot cheaply move an in-progress sequence unless its KV state is transferred or reconstructed.
Data parallelism scales throughput until memory fragments
Replicating a model gives each GPU or group an independent request queue. For small models that fit comfortably, additional replicas raise aggregate throughput with little cross-replica communication. The paper reports roughly 4× throughput when scaling a small model from two to eight GPUs under its evaluated load.
The limitation is physical independence. Each copy holds weights and its own KV allocations. Variable-length requests finish at different times, leaving holes that cannot serve a sequence located on another replica. Load balancing at admission helps, but a request that turns out much longer than predicted can strand capacity after placement.
Queue policy also changes the result. Favoring short jobs increases completed requests and can starve long reasoning. Fair service may accept lower aggregate throughput. A benchmark should publish the arrival and length distribution, admission limit, preemption policy, and latency percentiles rather than reporting a maximum request rate alone.
Tensor parallelism exchanges communication for pooled HBM
Tensor parallelism shards layers and their associated state across GPUs, so one request can use the combined memory of the group. This reduces the independent-replica fragmentation and allows larger models or longer contexts. The cost is collective communication within nearly every transformer layer and synchronization at token cadence.
For small models, that communication overwhelms the capacity benefit, which explains why data parallelism remains attractive. Around the reported 32B crossover, model and KV pressure make pooling worthwhile. The crossover is not a universal parameter count. It shifts with HBM capacity, link speed, quantization, batch, context, and implementation.
Placement is decisive. A high tensor degree within a fast node can be effective, while crossing a slower network for every layer can destroy token latency. Throughput can sometimes hide this cost by batching, but interactive time per output token exposes it. The parallel group must follow the physical hierarchy.
Dense and sparse models stress different paths
Dense Llama 405B uses all its major weight matrices for every token. Weight bandwidth and tensor collectives dominate, making high-degree TP a reasonable way to fit and feed the model. Sparse DeepSeek-R1 671B activates a subset of experts, reducing arithmetic per token relative to its total parameters but adding routing and load-balance behavior.
Expert traffic can be uneven, and tokens may cross devices to reach selected experts. Increasing tensor parallelism does not automatically solve that all-to-all communication. Pipeline or hybrid layouts can keep expert groups local and divide layers while using TP where dense components require it. The best design depends on expert placement, capacity factor, token distribution, and link topology.
Calling both models “large” hides these mechanisms. Procurement and serving plans should distinguish total parameters, active parameters per token, weight residency, KV size, and communication pattern. A dense 405B and sparse 671B can need different networks even when both require several GPUs.
TTFT and token latency pull the scheduler differently
Time to first token includes queueing and prompt processing. Time per output token measures the decode stream after it begins. Increasing admission and batching can improve accelerator utilization and prompt throughput while slowing individual decode steps or increasing queue delay. Optimizing one metric can damage another.
Long reasoning makes token latency cumulative. A small per-token penalty multiplied by thousands of outputs can dominate end-to-end response time. Preempting a sequence may admit a short request quickly but forces the long request to reload or recompute state. Tail latency needs separate reporting by output-length bucket.
The serving objective should therefore specify a service curve: TTFT, token latency, completion time, and throughput under a declared arrival distribution. Capacity is the maximum load that meets that curve, not the point at which GPUs stop accepting allocations.
Memory tiering is useful only with a movement policy
Host DRAM, CXL memory, local SSD, and remote storage can extend apparent KV capacity beyond HBM. Their bandwidth and latency are much lower than on-package memory, so moving an active sequence at every token would erase throughput. Tiering requires predicting which state will be reused soon.
Cold or preempted sequences can move outward, while actively decoding state stays in HBM. Prefix sharing may avoid duplicate KV for common prompts, provided isolation and lifetime are correct. Compression or reduced precision lowers movement and capacity but must preserve model quality. Each technique adds metadata and scheduler complexity.
The right metric is avoided recomputation and admitted useful work per byte moved. A tier that increases nominal capacity but saturates PCIe or CXL during bursts can make all requests slower. Bandwidth reservation and backpressure are necessary so eviction does not compete uncontrollably with model weights and network traffic.
Capacity planning needs distributions
An operator can combine the arrival rate, prompt-length distribution, output-length distribution, model state, KV bytes per token, and service-level objectives into a simulation or queueing model. The model should reproduce preemption, fragmentation, batch formation, and memory transfer, then be validated against traces.
Mean values are not enough because correlated long requests create the worst pressure. Scenario tests should include high-percentile length bursts and model-specific traffic. Separate pools can isolate unpredictable reasoning from short chat, but fragmentation across pools becomes another cost. Dynamic boundaries may improve utilization if they remain safe under bursts.
Hardware procurement follows from the bottleneck breakdown. More arithmetic helps prompt processing or compute-bound models. More HBM admits longer and more concurrent sequences. Faster scale-up links support tensor pooling, while scale-out bandwidth helps routing and state movement. Buying peak FLOPS without this workload model can leave capacity unchanged.
What the evaluation does and does not establish
The study covers models from 8B to 671B on eight-H200 nodes and uses a 100,000-request reasoning workload. It compares several parallel layouts and reports crossovers under that software and hardware. Different engines, quantization, speculative decoding, KV compression, or future larger-memory GPUs can shift the curves.
The 45% long-output statistic characterizes the selected data, not every reasoning product. The 32B crossover is an observed region, not an architectural constant. The robust conclusion is the method: measure persistent state and latency components under a length distribution, then choose parallelism separately for dense and sparse models.
What serving platforms should expose
Operators need live KV occupancy and fragmentation by replica or group, request length estimates, preemption and recomputation counts, bytes moved between tiers, collective time, TTFT, token latency, and completion tails. Averages should be broken out by model and length bucket.
Schedulers should make the admission and fairness policy explicit. Users and product teams need to know whether long reasoning is queued, throttled, truncated, or charged differently. Infrastructure cannot hide a product trade-off inside an out-of-memory retry.
Model servers should support topology-aware group placement, versioned configurations, and controlled experiments across parallel degrees. The goal is a capacity plan that can be revised when the request distribution changes, not one static layout selected from a peak benchmark.
An admission controller should expose its error budget
Length prediction will be wrong, so the controller needs explicit budgets for underestimation and overestimation. Underestimation causes preemption, recomputation, or rejected allocations; overestimation leaves HBM idle. Traces can measure both by request class and show whether a new model or prompt style has shifted the distribution.
The controller can reserve a shared emergency pool rather than padding every request. When several sequences exceed their estimates, the pool absorbs growth; if it empties, policy determines which state moves or waits. The reserve has an opportunity cost that should appear in capacity reports. A nominal occupancy of 95% can be unsafe if the remaining 5% cannot cover correlated long outputs.
Fairness needs a separate error budget. A scheduler optimized for completed requests may repeatedly preempt the longest reasoning traces. Per-class completion, slowdown, and abandonment reveal this behavior. Product teams can then decide whether to reserve a long-context tier, change prices, or cap outputs rather than allowing infrastructure heuristics to make the policy implicitly.
Model and hardware changes should trigger recalibration
KV bytes per token can change with grouped-query attention, precision, layer structure, or cache compression. Speculative decoding changes the amount of temporary work and accepted output. A new GPU changes HBM and link ratios. Each modification can move the data-to-tensor-parallel crossover and invalidate stored capacity tables.
Recalibration should run a fixed workload trace across candidate layouts, record all latency and memory components, and compare the predicted and observed bottlenecks. Canary traffic then confirms behavior under real arrivals. Configuration should be versioned so a regression can restore the prior model-server and placement policy together.
This process turns the paper’s 32B observation into a method rather than a rule. The infrastructure remains responsive to model evolution because it remeasures the capacity transition instead of assuming parameter count alone determines it.
Cost analysis should price memory and links by the useful requests they unlock. A higher tensor degree may use more GPUs per replica yet admit long sequences that otherwise fail. More data-parallel copies may complete short requests cheaply while rejecting the reasoning tail. Neither resource efficiency can be judged without the service mix.
Operators should also separate overload from model failure. When KV capacity is exhausted, repeated retries can amplify load and resemble instability. Admission must reject or queue before allocation, communicate a clear reason, and avoid recomputing a prompt that cannot be placed. These controls turn memory capacity into a predictable service boundary.
Finally, privacy and isolation apply to shared prefixes and tiered state. Reusing KV or moving it through host and remote memory must preserve tenant boundaries, encryption, deletion, and accounting. A capacity optimization that creates cross-request leakage is not deployable regardless of throughput. Audit logs should record state movement without retaining prompt contents. They should also prove timely deletion when a request completes or a tenant leaves the service.
What we take from it
Reasoning service must tune time to first token, time per output token, end-to-end latency, KV occupancy, and throughput together. Increasing admission can improve TTFT while making long decodes slower and tails unstable. The infrastructure implication is explicit memory control: pooled or tiered HBM, DRAM, CXL, and storage, plus schedulers that move KV state according to temperature. Arithmetic capacity alone will not remove this wall.
Procurement must price the session mix
Capacity planning for reasoning inference cannot begin with one model size and one context length. The memory demand of a live service is a distribution over prompt length, generated tokens, concurrent sessions, KV-cache precision, batching policy, and the fraction of requests that reuse prefixes. Two services running the same weights can reach different capacity limits because one retains long histories while the other produces short independent answers.
The purchasing trace should therefore preserve these session attributes and replay them against candidate memory hierarchies. Operators need tail TTFT, inter-token latency, completed requests per accelerator-hour, rejected admissions, and bytes moved between memory tiers. Quantization or eviction can raise capacity, but its accuracy and recomputation costs must remain attached to the saved bytes. The correct comparison is the cost of meeting the service’s latency and quality objectives across the full session mix, not peak tokens per second on a prompt chosen to fit memory.
Source and attribution
This independent summary uses original language and two new figures. It reproduces no source prose, chart, or table. The authors provide a public arXiv manuscript; the ISCA proceedings version is © IEEE 2026.