Parameter-efficient fine-tuning updates a small adapter while most model weights remain frozen. That lowers optimizer state and checkpoint size, but it does not make distributed execution communication-free. Fully sharded runtimes still gather frozen weights before a layer executes and discard them afterward to preserve memory. The same bytes can cross GPUs in every iteration even when memory becomes temporarily available elsewhere in the schedule.
mTuner’s premise is that memory capacity is not one static number[1]. Different tensors are needed at different moments, and parameter-efficient fine-tuning has an unusual asset: most weights never change. If the runtime can retain selected gathered weights, release them before a peak, and use spare capacity for a larger accumulation window, it can reduce communication without changing model quality.
The paper introduces an elastic tensor abstraction with four operations: gather, discard, execute, and checkpoint. Each tensor has a tunable availability or accumulation ratio. mTuner uses these controls to improve temporal memory utilization, relax dependencies between computation and communication, and adapt gradient accumulation to the memory actually free at runtime. The most revealing result is the interconnect split. Against the strongest baseline, average throughput improves by 28.3% on PCIe servers and 14.5% on NVLink servers. The same software idea is worth more when communication is slower.
Frozen weights are a communication cache opportunity
In fully sharded data parallelism, each GPU keeps a partition of model parameters. Before computing a layer, GPUs all-gather the full weight tensor. After use, the full copy is discarded and only the shard remains. This enables large models but repeats communication. During parameter-efficient fine-tuning, the gathered base weight is immutable, so retaining it is equivalent to caching clean data. No coherence or writeback is needed.
Static policies choose either to keep a weight or discard it. The correct choice changes during an iteration. Activation memory grows during the forward pass, then is released or recomputed. Gradient buffers appear during backward execution. Some layers have enough computation to hide an all-gather; others expose it directly. mTuner can keep more weights when memory pressure is low, then discard selected copies before a peak.
The execute operation separates a tensor’s logical use from its physical residency. Gather makes the required fraction available, discard releases it, and checkpoint retains a chosen state across a boundary. For runtime tensors such as gradients, an accumulation ratio controls how much work is combined before synchronization. These operations let the scheduler express time rather than only placement.
This does not create memory. Retaining a frozen weight reduces capacity available for activations, batches, and temporary communication buffers. A poor choice can force recomputation or an out-of-memory failure. mTuner profiles tensor sizes and operation timing, chooses static ratios, and adjusts runtime accumulation according to observed capacity.
Communication dependencies are also scheduling choices
Traditional runtimes often require an all-gather to finish immediately before its layer. Prefetching can overlap communication with earlier computation, but a fixed order may leave the link or compute engine idle. Elastic availability allows parts of a tensor to be gathered earlier and kept until execution. It also lets the runtime avoid gathering a tensor that is already resident from the previous iteration.
Checkpointing is applied selectively. Activations can be recomputed to save memory, while a frozen parameter can be retained to save communication. The scheduler weighs recompute time against interconnect time. On a PCIe server, avoiding a collective often wins even if memory use rises. On an NVLink server, the same collective is cheaper, so retaining the tensor has a smaller return and may lose to a larger batch.
Gradient accumulation adds another dimension. Larger accumulation reduces synchronization frequency and can improve arithmetic intensity, but stores more runtime state. mTuner changes the accumulation choice as memory availability changes instead of fixing it for the entire run. The goal is not maximum occupancy at every instant. It is to spend each free byte on the operation that removes the most exposed time.
Two interconnects reveal the mechanism
The PCIe test server has eight NVIDIA A100 PCIe GPUs with 40 GB each. Groups of four GPUs sit behind a tree-like PCIe topology and traffic between NUMA domains crosses QPI. The NVLink evaluation uses four servers, each with eight NVIDIA H100 SXM GPUs and 80 GB per GPU. The models are Llama 2 variants from 7B through 70B, with input sequence lengths and parallel configurations varied across the experiment.
On the PCIe server, communication is the dominant constraint. DeepSpeed is the strongest baseline because it already overlaps computation and communication effectively. mTuner raises throughput by 28.3% on average over that baseline and by up to 51.2% across tested cases. Compared with the Torch FSDP base on which it is built, average speedup is 4.15×. The larger number includes weaknesses of unoptimized FSDP and should not be used as the primary comparison.
For the 7B model, more spare memory lets mTuner retain more state and improve throughput by about 40%. Even at 30B and 70B, where parameters consume a larger share of capacity, it reports roughly 27% improvement in the discussed configurations. Long sequences of at least 4096 tokens create more compute and activation work; the paper reports an average 34% acceleration in that range.
On the four NVLink servers, the best baseline is often Flux, whose communication kernel is optimized for NVLink. mTuner improves throughput by 14.5% on average and up to 24.8%. The smaller gain supports the mechanism rather than weakening it: elastic tensors primarily reduce communication, so the avoided operation is less expensive on a faster fabric.
These are throughput improvements, not reductions in model convergence time at a fixed statistical target. The paper evaluates execution efficiency and retains the same fine-tuning operations, but a deployment should still verify optimizer behavior, effective batch, and checkpoint semantics. Results also combine different server counts and GPU generations, so PCIe versus NVLink is a mechanism comparison rather than a controlled hardware price comparison.
The boundary conditions
mTuner benefits from immutable base weights. Full-parameter training changes those weights and requires synchronization or invalidation before a cached copy can be reused. Inference has a different lifetime and no backward pass. Elastic tensors remain a useful abstraction, but the profitable operations and correctness rules change.
Memory fragmentation and allocator behavior can reduce the capacity predicted by tensor sizes. Collectives also need temporary buffers, and the runtime must avoid retaining a tensor that blocks NCCL progress. Dynamic decisions add control overhead and can become unstable if memory demand fluctuates rapidly. The paper’s implementation is integrated with PyTorch and FSDP; another framework needs equivalent visibility into tensor lifetime and collective dependencies.
Topology matters inside the PCIe server. Four GPUs sharing one branch have different costs from a collective that crosses QPI and another PCIe root. A deployment should profile the actual topology and bind processes accordingly. Elasticity cannot recover bandwidth lost to poor NUMA placement, though it may reduce the number of transfers that cross the boundary.
The comparison also assumes a single tuning job or a controlled multi-server environment. In a shared cluster, keeping more frozen weights on one job may prevent another job from entering. A cluster scheduler must decide whether higher per-job throughput or higher aggregate occupancy is the objective. Memory saved by parameter-efficient tuning can be spent on caching, batch size, or another tenant, and those choices have different business denominators.
A practical decision rule for PCIe servers
Organizations often assume that multi-GPU fine-tuning requires NVLink. mTuner provides a more precise rule. First measure exposed collective time after existing overlap. Then measure transient free memory across the iteration. If significant free capacity and repeated all-gathers coexist, caching immutable tensors can recover part of the PCIe gap. The software return rises with the cost of the avoided transfer.
This does not make PCIe equivalent to NVLink. NVLink still provides higher bandwidth and supports workloads whose communication cannot be removed. mTuner changes the workload presented to the interconnect. It is most attractive for parameter-efficient fine-tuning, where frozen weights create clean reusable state and organizations already own PCIe GPU servers.
The procurement implication is to benchmark the runtime and tuning method together. Peak interconnect bandwidth does not determine useful tokens per second if software repeatedly discards data it could retain. Conversely, a large-memory GPU has additional value when the runtime can convert spare bytes into fewer communications. Memory capacity, tensor lifetime, and fabric bandwidth form one scheduling problem.
Source and copyright notice
This article is an editorial analysis by Silicon & Systems. It restates the paper’s abstraction, evaluation, and limits in our own words. No source sentence, table, or figure is reproduced; all figures were created for this article. The paper and source code are available from the USENIX ATC 2025 page. Copyright remains with the authors, 2025.