A serverless cold start is often reduced to one operation: create an isolated execution environment. Once a sandbox can fork in less than a millisecond, the problem appears solved. Ant Group’s production measurements show why that view is incomplete. A request also crosses the scheduler and node gateway, traverses a high-level and low-level runtime, acquires kernel resources, initializes a network stack and security policy, loads a language runtime and dependencies, runs user code, and eventually tears the instance down.
Ant Group operated more than 50,000 unique functions and about 100 million calls per day in the studied environment. More than half of functions had a cold-start probability above 0.75, and over 35% were cold on every invocation. Hot requests dominated the aggregate because a small group of popular functions received much of the traffic. Keeping every infrequent function warm would spend memory on idle instances, so the platform retained hot containers for only one minute[1].
Catalyzer already provided a secure VM-process fork in under one millisecond under favorable conditions. End-to-end cold starts still ranged from hundreds of milliseconds to seconds. AFaaS begins from the observation that making one stage fast exposes the remaining stages. Its design addresses three gaps: runtime control-path overhead, resource contention under sustained concurrency, and user-code initialization.
The control path outlives the optimized fork
An OCI-compatible container stack separates containerd, a shim, and a low-level runtime. The high-level process issues RPCs, loads a runtime binary, prepares arguments, asks the seed to fork, and later activates the sandbox. This modular interface supports many container types, but in the measured Catalyzer path it consumed 18 to 25 milliseconds, about 30% to 40% of cold-start time after sandbox creation had been optimized.
AFaaS introduces a specialized fork runtime interface (FRI). A containerd plug-in calls a long-lived low-level runtime directly and moves work that is invariant across function instances into seed preparation. This shortens the chain and replaces repeated process and binary setup with function calls and prepared state. The trade is deliberate: FRI gives up part of OCI’s generality in return for a FaaS-specific lifecycle.
Specialization is defensible only if the boundary remains auditable. A deployment needs to know which OCI semantics are omitted, which fields can differ per invocation, and how upgrades coordinate between containerd, the plug-in, and the sandbox. The latency saving comes from declaring some flexibility unnecessary, not from implementing the same general interface more quickly.
Concurrency turns kernel setup into a shared bottleneck
Sequential microbenchmarks hide locks and allocation paths that appear when many instances start together. Creating virtual Ethernet pairs, cgroups, namespaces, network state, and seccomp filters can contend on host-kernel structures. Catalyzer’s throughput degraded during sustained execution as cache misses and global lock contention pushed setup from fast paths to slow paths.
AFaaS divides resources into objects that can be pooled and state that can be inherited. It pre-allocates veth devices and recycles cgroups. Seeds and children can share selected network and IPC namespaces because user code remains isolated inside the guest OS. Network structures are split into common pieces prepared in the seed and per-instance identities such as addresses and backend devices. Seccomp rules are parsed and compiled before requests, leaving a smaller installation step.
Pooling is not free capacity. AFaaS prepared 1,000 veth devices and 600 cgroups in the reported configuration. A pool can exhaust, and refilling it can become a noisy neighbor. The paper notes that even serial pool preparation may contend with other workloads. Operators need low-water marks, preparation rate limits, fallback latency, and isolation between maintenance work and user traffic.
Sharing also requires a threat model. Host network namespaces are acceptable here because isolation is enforced inside secure containers, but that conclusion is not transferable to an ordinary process-container design. A reused device must close old connections before reassignment, and inherited memory must not expose another tenant’s function state.
A seed tree moves application work before arrival
Language and framework initialization can dominate short functions. Loading Python or Node.js, importing libraries, compiling code, and parsing framework configuration may take longer than executing the handler. A seed with only the guest OS avoids VM setup but leaves all application work. A seed for every function minimizes latency but consumes memory and is difficult to keep warm.
AFaaS uses a hierarchy. Level 0 holds the guest OS. Level 1 children initialize a language runtime. Level 2 children add a function’s framework, dependencies, and compiled user code. Copy-on-write allows descendants to share physical pages with their ancestors. When an exact function seed is absent, the scheduler forks from the closest available language or root seed and performs only the remaining initialization.
The hierarchy converts a binary hot-or-cold choice into graded reuse. Popular functions can justify level-2 seeds, while the long tail shares language state. Controlled measurements showed 28.11% to 84.91% less seed memory than Catalyzer for functions where shared components were substantial. Production seed sizes ranged from 6.9 MB to 135.03 MB, depending on the initialized code.
AFaaS also prefilled extended page tables. A forked VM-process shares physical pages but would otherwise fault while reconstructing guest-to-host mappings. Copying the seed’s EPT and protecting the last directory level avoids many read-side VM exits. This optimization reminds us that copy-on-write memory sharing does not automatically share every translation structure needed to access that memory.
Unique state is the dangerous part of fork
Fork duplicates more than useful code pages. Buffered random-number state can cause children to emit repeated values. Open sockets, identifiers, timers, credentials, and cached process assumptions may also be invalid after cloning. AFaaS explicitly terminates long-lived connections and reinitializes unique state. The paper describes a buffered getrandom() path that could otherwise repeat random sequences across children.
This is a correctness and security gate, not a performance detail. Each runtime and library upgrade can introduce new state that must be classified as shareable, copy-on-write, or per-instance. A seed catalog therefore needs a manifest of open descriptors, entropy sources, network state, JIT caches, and hooks to run after fork. Testing only the first response will miss correlations that appear across many children.
Early destruction has similar assumptions. AFaaS considers a function complete when the unified handler returns, pauses the guest, disconnects TCP sessions, and reclaims resources without waiting for the full language-runtime shutdown. That is safe for stateless functions whose durable effects happen through remote services. It is unsafe for handlers that leave background work, buffered local writes, or asynchronous finalizers outside the contract.
End-to-end evaluation separates short and long functions
For short functions, AFaaS improved average latency 3.76 to 6.68 times and P99 latency 6.31 to 11.74 times over the Catalyzer-only configuration. Functions with expensive initialization gained more from function-specific seeds: average speedup was 4.09 to 31.48 times and P99 speedup 6.19 to 34.51 times. Long-running handlers improved only 1.05 to 1.14 times on average because execution dominated the response.
At concurrency from 1 through 24, a JavaScript benchmark completed in 16.34 to 39.56 milliseconds end to end under AFaaS, compared with 51.32 to 117.92 milliseconds for Catalyzer alone. AFaaS cold-start work within those requests ranged from 6.97 to 14.55 milliseconds, while Catalyzer ranged from 38.39 to 74.05 milliseconds. Kata and gVisor were slower in this selected secure-container setup.
Production measurements used eight representative Node.js functions over one day. AFaaS delivered 1.80 to 8.14 times end-to-end speedup and held startup latency between 5.45 and 9.41 milliseconds. The system had been deployed for more than 18 months. These observations are stronger than a one-machine benchmark, although the CataOnly comparison used matched hardware with mocked peer responses rather than a simultaneous production A/B test.
The workload context matters. Ant Group reports that more than 80% of requests finished user execution within 221 milliseconds. A platform dominated by multi-second functions would see a smaller user-visible fraction from a ten-millisecond startup. Conversely, latency-sensitive APIs with very short handlers make every control-path millisecond visible.
The remaining limits are operational
A seed itself serializes forks, so very high concurrency may require multiple replicas of the same seed. Repeated creation and destruction encountered seccomp installation failures related to bpf_jit_limit leaks. Co-located workloads can still hold cgroup locks needed when a pooled object is assigned or recycled. The optimized path reduces these bottlenecks; it does not remove shared host state.
Too many function-specific seeds raise memory pressure and can push pages to disk, destroying the assumed startup latency. Admission should therefore account for invocation probability, saved initialization time, seed size, and contention at the parent. The best seed set changes with demand and library versions.
AFaaS’s reusable lesson is to measure cold start from the scaling request to the response. A fast fork is one mechanism inside that path. Interface specialization removes control work, pools move contended allocation out of bursts, and hierarchical seeds move language and function initialization ahead of demand. Each technique borrows latency from another budget: compatibility, reserved resources, preparation work, or memory.
For a production release, the key checks are end-to-end percentiles under sustained concurrency, pool depletion behavior, uniqueness reinitialization, seed provenance, memory pressure, fallback paths, and cross-tenant isolation. Sub-millisecond cloning is meaningful only when those surrounding contracts keep the whole request in the millisecond range.
Source and copyright notice
This article is an editorial analysis by Silicon & Systems. It restates the production design, measurements, and limits in our own words. No source sentence, table, or figure is reproduced; the figure was created for this article. The paper is available from the USENIX OSDI 2025 presentation page. Copyright remains with the authors, 2025.