Record-replay is valuable when a crash depends on an interrupt, race, or data value that a developer cannot reproduce. A recorder captures every nondeterministic input and later injects the same sequence, allowing repeated debugging and reverse analysis. Whole-machine systems place the boundary around a virtual machine. This is complete, but it records application and I/O activity even when the failure belongs to the guest kernel.
The cost grows with current datacenter workloads. Multicore replay often serializes execution to make races deterministic, so overhead can exceed the core count. I/O-intensive software creates large traces. Kernel-bypass frameworks make the mismatch sharper: application threads move data directly to a device, yet whole-VM replay still captures and orders that traffic even though it does not enter the guest kernel[1].
KRR changes the unit of replay from the VM to the kernel execution slice. User space runs during recording, but only the values and events that cross into the kernel are part of the trace. Replay emulates the user-side boundary and reconstructs kernel behavior. The reduction creates a faster recorder, while also making completeness dependent on identifying every way external state can influence the kernel.
Two recorders observe two boundaries
The guest kernel receives inputs from user space and hardware. No single layer sees both efficiently. A hypervisor can intercept interrupts and emulated-device activity, but recording every system call buffer through introspection would cause expensive exits. Guest instrumentation naturally sees copies and shared-memory reads but cannot independently control all hardware timing.
KRR therefore splits responsibility. An in-guest recorder captures system calls, copied user buffers, shared queues, nondeterministic instructions, and kernel scheduling. An in-hypervisor recorder captures interrupts, I/O reads, and device DMA visible to emulation. Their events enter one total order so replay can inject each input at the same kernel point.
User-memory capture exploits hardened kernel APIs. Linux funnels untrusted reads through hardened helpers, including copy_from_user alongside get_user. KRR records the destination kernel bytes after the copy, avoiding a second user-memory read and its time-of-check/time-of-use race. For io_uring, it instruments kernel reads of the shared submission queue and entries. Page-table accessed and dirty bits are recorded because user execution can change them and thereby alter kernel replacement decisions.
Hardware events need a timing signature. KRR reserves a kernel-mode performance counter and records the number of executed kernel instructions when an interrupt or DMA event arrives. Replay injects the event at the corresponding count. The counter is hidden from the guest’s available set through CPUID masking, leaving other counters for ordinary kernel use.
The design is architecture-specific in important places. Counter behavior, interrupt state, repeating instructions, page faults, and device emulation must be modeled correctly. Porting KRR is not just recompiling kernel probes; it requires validating the timing and state contract on each processor and virtual-device model.
Serializing only kernel execution preserves application parallelism
Kernels contain benign and intentional data races, so recording only synchronization order is insufficient. KRR serializes kernel execution across virtual CPUs and records which thread enters next. User-space threads may continue in parallel when they are outside the kernel. This creates a deterministic kernel schedule without forcing the complete VM into single-core execution.
The distinction is especially useful for polling applications. An SPDK or DPDK thread can exchange bulk data with a passthrough device without making system calls. Whole-VM serialization may stop the polling core while another thread waits for it, producing extreme slowdown. KRR does not record data that bypasses the kernel and permits those user threads to run, while logging the smaller control path that still enters Linux.
Serialization changes timing and can mask a bug. KRR models concurrency that can arise through context switches and interrupts on one core. It cannot reproduce a failure that strictly requires simultaneous kernel execution on separate cores, including some weak-memory interactions or narrow parallel windows. The recorder is therefore a diagnostic coverage tool, not a guarantee that every field failure becomes reproducible.
Initial state is part of deterministic replay
Inputs and order are insufficient if replay begins from a different kernel state. KRR saves VM snapshots during recording and associates later trace segments with those states. Replay can start at an intermediate snapshot instead of interpreting the complete execution from boot. This reduces turnaround for long traces and bounds how much state must be reconstructed.
Snapshots also create storage and privacy responsibilities. Trace entries may contain application data copied into the kernel, network payloads, file blocks, credentials, and memory mappings. KRR narrows exposure relative to whole-VM recording, but the remaining slice is still sensitive. Production collection needs encryption, access control, retention limits, tenant consent, and deletion linkage to the original incident.
Trace completeness should be tested independently of bug replay. The implementation can run deterministic workloads, compare kernel outputs and state hashes, and deliberately exercise every input class. A newly added shared-memory API or passthrough driver may cross the boundary without existing hooks. Kernel upgrades should therefore trigger an interface audit, not only a build test.
Eight-core workloads show the scaling difference
On eight-core VMs, RocksDB and Linux kernel compilation under KRR slowed by 1.52 to 2.79 times relative to native execution. A traditional whole-VM recorder slowed the same class of workloads by 8.97 to 29.94 times. KRR’s overhead still rises as kernel activity and core count increase, and the paper identifies one to eight virtual CPUs as its practical range.
Kernel bypass changes the denominator. A two-core RocksDB-SPDK configuration saw KRR slow operations by 1.17 to 1.27 times, while whole-VM replay reached 29.36 to 64.51 times in the reported workloads. SPDK removed 77% to 94% of system calls and 94% to 99% of user-to-kernel data copies for relevant cases, directly reducing KRR’s event volume.
Redis over DPDK on a four-core VM lost 0.26% of GET throughput and 1.14% of SET throughput on average. P99 latency changes ranged from a measured improvement of 5.19% to a slowdown of 11.27%, which is close enough that run-to-run variance and workload details matter. The bulk data path stayed outside the recorded kernel slice.
Nginx-DPDK exposed the opposite case. Small 1 KB and 4 KB files spent much of their work inside kernel file reads, and slowdown exceeded 46% while worsening with more cores. At 16 KB and 64 KB, network transfer dominated and overhead fell to about 2% and 5%. “Kernel bypass” does not by itself predict recorder cost; the fraction of request time and input bytes that still cross the kernel boundary does.
Bug reproduction validates scope, not universality
The authors selected 12 Linux 6.1 bugs from Syzbot and five high-impact CVEs, for 17 cases. Six of the Syzbot cases were nondeterministic. KRR recorded and reproduced all but one nondeterministic bug, yielding 16 successful cases overall. The set includes concurrency faults and security-relevant failures across Linux versions.
Selection matters. Reproducer programs already existed for the Syzbot population, and five CVEs were chosen from previously evaluated high-severity issues. The experiment demonstrates breadth across mechanisms, not a field success rate for arbitrary crashes. A deployment still needs a trigger strategy, trace budget, and a way to capture the interval before an unknown failure.
Recording can also perturb the event it seeks. Instrumentation and serialization alter timing. No software-only recorder can promise to preserve every race probability. Practical use should combine KRR with crash dumps, hardware traces, sanitizers, and schedule-independent detectors. KRR is strongest after a failure is captured inside its boundary, because deterministic replay then enables expensive offline analysis.
Devices and scale define the deployment envelope
KRR relies on hypervisor emulation to observe DMA and timing. Passthrough devices and SR-IOV paths that interact with the kernel but remain outside emulation are not currently supported. Extending the guest recorder into driver interfaces could capture their buffers and events, but each driver adds a new completeness surface.
Beyond eight virtual CPUs, serialized kernel execution reduces scalability. Chunk-based recording could preserve more parallelism, although it would need to log memory-access order and expand replay complexity. Operators should choose VM size based on diagnostic value, not assume that a 64-core production guest can be recorded at the eight-core overhead.
A practical rollout can keep recording off by default, enable it for a canary or incident cohort, and use background snapshots so activation retains pre-failure context. The trace collector must apply backpressure without blocking the kernel. When storage or ordering fails, the recorder should mark the trace incomplete instead of presenting it as replayable evidence.
KRR’s reusable systems idea is boundary selection. Recording less is safe only when the omitted execution cannot influence the component being replayed without crossing an instrumented interface. For kernel-bypass workloads, that condition turns a liability of whole-machine replay into an advantage: most high-rate data is irrelevant to the kernel slice.
The result is not free debugging. The kernel boundary must be maintained as APIs, devices, processors, and virtualization change. Within that boundary, however, KRR makes deterministic kernel diagnosis compatible with workloads that whole-VM replay slows by an order of magnitude or more.
Source and copyright notice
This article is an editorial analysis by Silicon & Systems. It restates the recording architecture, measurements, and limitations in our own words. No source sentence, table, or figure is reproduced; the figure was created for this article. The paper is available from the USENIX OSDI 2025 presentation page. Copyright remains with the authors, 2025.