Serverless pricing promises that an application pays when it runs. Conventional GPU inference often breaks that promise before the first request arrives. A model is loaded into a dedicated accelerator, the GPU remains reserved during quiet periods, and the customer pays for a long-lived instance because reloading the model would take too long. The function looks serverless at the API and serverful in the hardware ledger.
Torpor asks whether the binding point can move from function deployment to request arrival.[1] The system was developed by researchers from CUHK-Shenzhen, HKUST, Alibaba Group, and Nokia Bell Labs, then evaluated in an Alibaba Cloud pilot. Its answer is not to eliminate model state. Torpor moves the idle state into host DRAM, shares four local GPUs across many functions, and copies a model onto an accelerator only when work appears.
That distinction is the paper’s strongest systems idea. GPU memory becomes an execution cache instead of permanent tenancy. However, the economic claim is valid only if model movement, tail latency, host-memory cost, and burst handling are measured together. The reported 70% average user saving is therefore a result of one production policy and workload population, not a universal serverless discount.
Early binding converts a sparse workload into a reserved GPU
The production trace motivating Torpor is highly sparse. Across one week, 85% of observed inference functions were called no more than once per minute, and 97% were called no more than once per second. The publication combines CPU and GPU functions in that confidentiality-preserving trace, while stating that both groups show similar frequency patterns. It does not provide a GPU-only histogram, so the percentages should be read as demand-shape evidence rather than a direct GPU capacity forecast.
Early binding is costly under this shape because each function receives a GPU-serving instance before its requests are known. Keeping the instance alive avoids a cold start but leaves memory and compute stranded between calls. Reclaiming the device reverses the problem: the next request waits for a container, runtime, and model image. The paper reports conventional cold starts between 8 seconds for ResNet-152 and 61 seconds for Llama-2-13B in its production path. A service objective measured in tens or hundreds of milliseconds cannot absorb those delays.
This is also why simply increasing a keep-alive timeout is not a serverless solution. A longer timeout buys latency by turning uncertain future demand into prepaid accelerator time. A shorter timeout improves reclamation but pushes more requests into cold startup. Torpor changes the allocation unit instead. It keeps a model image available without keeping that model resident on a particular GPU.
Host DRAM is the idle tier and four GPUs form the execution tier
Each Torpor worker has a host-memory pool and four GPUs. Inactive model images remain in host DRAM. When a request arrives, the platform may select any suitable GPU in the worker rather than sending the request to a device chosen at deployment. If a copy already resides on a GPU, the request can reuse it. Otherwise, the system loads the model and may evict another image when device memory is full.
The data path matters because host-to-device transfer is not free. Torpor redirects asynchronous CUDA calls, uses pinned host memory, and pipelines model transfer with execution. The four-GPU topology also permits a model to move from a peer over NVLink when that route is useful, rather than always returning to host memory over PCIe. The policy considers link contention because simultaneous transfers can lengthen inference time, particularly for larger models.
Runtime treatment differs between the paper’s testbed and its pilot. Testbed experiments share one GPU runtime to isolate Torpor’s model-management mechanisms. The production pilot gives each user a dedicated runtime for isolation, so startup includes both model loading and runtime resumption. This boundary is important: a laboratory result about copy scheduling does not, by itself, establish multi-tenant security or production startup latency.
Host DRAM is consequently not free capacity. It is a cheaper state tier that allows the expensive execution tier to be multiplexed. In the pilot, users are charged 10% of the GPU price while a function is inactive and retained in host memory. Extremely infrequent functions can still accumulate material cost. A complete comparison must therefore include DRAM reservation and the duration of idle retention, not only active GPU seconds.
The scheduler optimizes compliant functions, not raw utilization
Late binding creates a coupled queueing problem. A worker may host many functions with different request rates, deadlines, model sizes, and loading costs. Choosing the first free GPU can cause a large transfer to block a smaller request, while maximizing immediate occupancy can push multiple functions beyond their tail-latency objectives.
Torpor uses a metric called required request count (RRC) to estimate how much successful service a function needs to remain within its SLO. A function close to violating its objective receives higher priority than one with more margin. The threshold adapts to load rather than permanently assigning one class of models to a high-priority lane. This policy shifts the objective from keeping every GPU busy to maximizing the number of functions whose measured tail remains acceptable.
Placement and eviction then account for interference. The scheduler avoids overlapping transfers that contend for the same PCIe path and can use the worker’s NVLink relationships when a peer copy exists. Its eviction policy distinguishes models whose loading footprint makes them expensive from lighter models that can return more quickly. The mechanism is not a general proof that one priority formula fits every service. It is a concrete example of how the model cache and the request queue must be managed as one system.
That coupling changes the platform metric. Average GPU utilization may rise while a small set of large models misses every deadline. Conversely, leaving a transfer path temporarily quiet can admit more SLO-valid requests later. The useful numerator is requests completed inside the declared latency objective, and the denominator includes GPUs, host memory, and the state retained for future calls.
Startup results separate model production from runtime isolation
The paper reports model loading and runtime resumption separately in the pilot path. Llama-3-8B requires 1.6 seconds for model production and 1.4 seconds for runtime isolation, totaling 3.0 seconds, compared with a reported 48-second conventional cold start. Qwen-14B totals 3.6 seconds rather than 57 seconds. Llama-2-13B totals 4.4 seconds rather than 61 seconds. Smaller examples are also reduced: ResNet-152 totals 0.29 seconds, and BERT question answering totals 0.33 seconds.
These are large reductions, but they are not request latency for a permanently warm model. A three-second startup still exceeds an 80-millisecond or 200-millisecond inference SLO if the whole delay lands on one request. Torpor relies on queue policy, retained copies, pipelining, and the fact that many functions can amortize setup over subsequent requests. Buyers should ask for the fraction of requests that trigger each startup path, not compare only the endpoints of a cold-start bar.
The controlled cluster uses as many as six workers. Each worker has 48 virtual CPUs, 384 GB of host memory, and four NVIDIA V100 GPUs with 32 GB each. Eight representative models cover image, question-answering, diffusion, and language-model workloads. The default objectives are P98 latency below 80 milliseconds for computer vision and below 200 milliseconds for BERT question answering.
At one worker with 160 functions, Torpor executes every function and reports all of them as SLO-compliant. Native execution reaches 72 functions. INFless with keep-alive executes 112, but only seven satisfy the objective in that experiment. At a 1,000-function cluster scale, Torpor meets nearly every deadline, while the paper reports deadline tails above four times for a simple swapping baseline and above seven times for a non-swapping baseline. Those comparisons demonstrate the value of coordinated policy under the chosen traces. They do not establish equivalent ratios on newer accelerators, different interconnects, or very large tensor-parallel models.
The pilot result is operational evidence with a defined denominator
The Alibaba Cloud pilot served more than 150 users on more than 350 GPUs and reached as many as 465,000 requests per day. Compared with the preceding long-running GPU approach, the paper reports average user cost savings of 70%. Consolidation reduced the platform’s required GPU count and associated cost by 65%. A production case study reports an 84% cost reduction; model and runtime loading account for roughly 30% of its end-to-end time.
Those figures answer three different questions. The 70% value concerns customer charges under the pilot billing policy. The 65% value concerns platform provisioning. The 84% case is one function with its own arrival pattern. Combining them into a single efficiency number would hide who saves money and which workload produced the result.
The pilot also reveals the commercial mechanism. An idle model is not billed at zero because it occupies host memory. It is billed at one-tenth of the GPU rate, while active execution uses GPU pricing. The claimed saving therefore depends on the ratio between idle duration, active duration, model size, and request bursts. A frequently invoked model may gain less because its GPU remains productive. A function called a few times per hour may still pay enough DRAM rent to justify moving the model to cheaper storage.
The result is credible evidence that late binding can work beyond a simulator. It is not a randomized cloud-price study. The provider designed the platform, billing policy, and comparison baseline, and the paper does not expose a workload-level cost distribution. An operator considering the mechanism should request percentiles by model size and invocation rate rather than rely on the fleet average alone.
What a buyer or platform team should measure
The first requirement is a state inventory. For each function, record model bytes, runtime state, retained host-memory bytes, GPU-resident time, load source, transfer path, and eviction count. This makes the 10% idle charge auditable and reveals whether DRAM has become the next scarce resource. Report host-memory occupancy at P50, P95, and peak, together with the number of functions that could not be admitted.
Second, separate request paths. A hit on an existing GPU copy, a host-to-GPU load, a peer-GPU transfer, and a full runtime resumption have different latency and contention. For each path, disclose its request share and P50, P95, and P98 completion time. A single average can look healthy even when rare cold paths consume the entire SLO budget.
Third, preserve the workload shape. Invocation rate alone is insufficient because bursts determine how many copies and workers are needed simultaneously. Report inter-arrival distributions, burst duration, model popularity, input and output sizes, batching, and the number of replicas created across nodes. A platform should show the savings after reserving enough capacity for the declared burst percentile.
Finally, price successful work. The denominator should include active GPU time, inactive host-memory retention, failed or late requests, data transfer, and the capacity margin needed for bursts. Cost per SLO-compliant request is more useful than cost per GPU-second. It prevents a platform from appearing cheaper by admitting more functions while allowing their tails to fail.
Three boundaries keep Torpor from being a universal runtime
Very infrequent functions remain an awkward fit because the host copy still costs money. A multi-tier design could place the coldest images on cheaper storage and promote them as demand rises, but that mechanism is future work in the paper. It would also reintroduce a longer startup path that needs a separate SLO contract.
The implementation does not execute one very large model across multiple GPUs. Late binding is demonstrated for functions that fit the supported device path, not for the largest model-parallel inference services. Extending it would require coordinated partition loading, parallel-runtime state, and placement across a topology rather than one available GPU.
Highly bursty functions may need replicas on several workers. Replication protects latency but consumes more host memory and may duplicate transfer traffic. The paper proposes exploring faster inter-node movement, including RDMA, but does not evaluate that design. In addition, the controlled system uses V100 GPUs. Modern accelerators have different memory capacities, host links, peer fabrics, partitioning features, and loading ratios, so the absolute startup numbers should not be used as a current hardware forecast.
Torpor’s durable insight is narrower and more useful than a universal speedup claim. Serverless GPU inference becomes economically serverless only when model state and accelerator ownership are decoupled. Host memory can hold the option to run, while a local GPU pool supplies execution on demand. The platform must then prove that the new state tier, queue policy, and transfer fabric deliver more compliant requests at a lower complete cost.
Source and attribution
This article is an independent editorial digest prepared by Silicon & Systems. It restates the cited work in our own words. No sentences, tables, or figures from the paper are reproduced. The material hardware base was generated for this article and then annotated deterministically; the evidence graphic was replotted in code from reported values. The paper is openly available in the USENIX ATC 2025 proceedings. Copyright (c) 2025 the authors. Alibaba Group and Nokia Bell Labs are listed because authors were affiliated with those organizations; their inclusion is not an endorsement. Read the complete source on the USENIX paper page.