Composable infrastructure extends PCIe beyond one server so processors can attach remote GPUs, storage devices, or accelerators as if they were local. A non-transparent bridge (NTB) joins separate PCIe domains and translates transactions across external switches. The programming model remains familiar, but the physical path now contains adapters, cables, switch ports, and independent credit domains.

rPCIeBench was built because local PCIe intuition is insufficient for that path[1]. The framework combines FPGA traffic generation with transaction-level tracing and reconfigurable topology. It measures host-to-device, device-to-host, device-to-device, and MMIO operations, then varies transaction size, outstanding requests, direction, and shared links. The study extracts rules that can be used by a scheduler rather than stopping at a list of bandwidth numbers.

Its first result is a clear latency cost. A one-way transaction takes 379.0 ns inside the server and 868.6 ns across the routable fabric, an additional 489.6 ns or 129.2%. Roughly 105 ns is attributed to each host adapter, external switch, and target adapter, with the remainder coming from propagation and other stages. A remote device is therefore not local simply because software reaches it through PCIe semantics.

Small transactions expose every additional stage

The fixed path cost is most visible in MMIO and small transfers. A local 64-byte MMIO read takes 766.0 ns in the testbed. The remote path adds the NTB chain and completion return. At 1 KB, sixteen cache-line transactions produce 11.9 microseconds locally and 27.8 microseconds remotely. Software that polls a device register or emits many small commands can therefore pay the fabric cost repeatedly.

Larger DMA transfers amortize setup and protocol overhead. At 64 KB, the relative difference between local and remote host-to-device or device-to-host movement becomes much smaller. The paper consequently recommends batching. This is not merely a software optimization; it changes whether a disaggregated device is latency-bound or bandwidth-bound.

NUMA placement remains relevant. Even before the external fabric, reaching a root complex from the wrong socket adds host interconnect traffic. Across the evaluated transfers, NUMA adds about 10.7% and 7.0% for local host-to-device and device-to-host traffic, then another 7.5% and 11.2% on remote paths. A resource manager must place CPU threads near the adapter that owns the routable PCIe connection.

The measured routable PCIe path. Local one-way PCIe takes 379.0 ns; crossing a host adapter, external switch, and target adapter raises it to 868.6 ns. Small MMIO operations pay every hop, while 64 KB DMA amortizes the fixed path cost. NUMA placement remains an additional host-side variable. Original figure created for this article.

Credits make request size a scheduling weight

PCIe uses hop-by-hop credit flow control. A sender can issue transactions only while the next receiver advertises buffer capacity for the relevant packet class. When several flows share a link, the amount of outstanding data each flow can present affects its share. rPCIeBench finds an approximate max-min allocation whose weights follow outstanding bytes.

In one host-to-device case, a 4 KB flow competing with a 128-byte flow reaches 9.5 GB/s while the smaller flow receives 0.39 GB/s, a 32.8 ratio. Two 4 KB flows each sustain about 5.5 GB/s. The behavior appears in local and routable paths and in device-to-host and device-to-device movement. A scheduler cannot treat all active PCIe flows as equal when transaction size and outstanding depth differ.

The fabric also synchronizes quickly across bottlenecks. When a downstream edge limits one flow, upstream rates settle to the constraint rather than filling indefinitely. The authors encode this as edge capacities and iteratively relax constraints until flow rates converge. Their model estimates transmission performance from topology, direction, and transaction properties without replaying every packet.

This is useful for placement. If two accelerators share an oversubscribed external port, moving one workload to an orthogonal branch can be more effective than changing a software queue. The model provides a way to test that choice before allocation, though it remains calibrated to the measured hardware.

PCIe has separate transmit and receive directions. Two flows that traverse the same physical connection in opposite directions can use different link capacity. In the testbed, a latency-sensitive reverse flow changes little while a throughput flow saturates the other direction. Device-to-host latency stays around 2.2 to 2.3 microseconds remotely, and device-to-device latency remains around 3.0 to 3.1 microseconds across the external switch.

The same independence can appear between separate branches of a switch. Flows that do not share a directed edge interfere little even when the diagram looks physically close. Conversely, two paths that share one hidden adapter port can contend although their endpoints are different. Topology should therefore be modeled as directed capacity edges, not as a count of devices behind one switch.

Three routable PCIe scheduling rules. Shared directed edges divide bandwidth approximately by outstanding bytes; a downstream bottleneck rapidly constrains the whole end-to-end flow; and opposite directions or disjoint directed edges show little interference. The rules support an iterative edge-constraint model for placement. Original figure created for this article.

Latency rises sharply near maximum bandwidth because credits are exhausted and transactions wait in queues. Tail latency grows more than the average, especially for paths with more hops. This creates the same operating choice seen in networks: peak throughput is not the best point for a latency-sensitive service. Admission control needs headroom below the measured link maximum.

What rPCIeBench does not generalize automatically

The testbed represents one generation of adapters, FPGA endpoints, and external switches. NTB implementations differ in translation pipelines, buffering, cable interfaces, and firmware. PCIe Gen5 or Gen6 changes serialization and encoding, while CXL adds protocol classes and coherence behavior. The measured 489.6 ns increment is not a universal routable PCIe constant.

The structural findings are more portable. Additional domains add fixed processing stages; credit exhaustion links queue depth to tail latency; outstanding bytes influence sharing; and direction matters. A new platform should rerun the benchmark and fit its own edge capacities. The open-source framework is valuable because it turns these assumptions into measurements.

Correctness and failure are outside most performance experiments. An NTB fabric also needs device discovery, address-window management, access isolation, reset containment, and recovery when a remote chassis disappears. A GPU or SSD that looks local to a driver may fail with network-like causes. Software must distinguish a device reset from a broken fabric path and prevent one tenant from mapping another tenant’s BAR space.

The model predicts steady sharing, not arbitrary burst sequences. Real accelerators issue phase-dependent traffic, and switch firmware may contain undocumented arbitration. Validation should include the production command-size distribution, concurrency, and fault states rather than relying on a single large DMA benchmark.

The resource unit is a directed path

The paper changes how composable devices should be scheduled. Allocating one remote GPU is not enough. The scheduler should allocate its host adapter, directed switch edges, target adapter, and a latency budget. It should know the root complex and NUMA node of the consuming CPU. Two allocations are compatible when their directed paths are orthogonal or have enough credit and bandwidth headroom.

The same path description should follow a device into observability. A utilization alert needs to identify the directed edge whose credits are exhausted, not merely the endpoint. A latency alert should separate time spent in the local root complex, NTB translation, external switch, target adapter, and device. Without that breakdown, an operator may migrate a workload whose actual bottleneck is a host-side NUMA crossing. Composable infrastructure therefore needs topology-aware telemetry as much as topology-aware placement.

Capacity planning also needs two envelopes. The bandwidth envelope describes long DMA transfers with enough outstanding work to fill the link. The command envelope describes MMIO, completions, and synchronization operations that pay fixed hop latency. A device can fit the first and violate the second. Reporting both prevents an aggregate bandwidth result from hiding a control path that determines application progress.

For latency-sensitive command traffic, the operator should prefer fewer fabric stages, batch MMIO where the device permits it, and reserve headroom before saturation. For streaming transfers, the extra sub-microsecond fixed cost may be negligible and remote pooling can improve utilization. The decision depends on transaction size and synchronization frequency, not the PCIe generation printed on the chassis.

Routable PCIe is therefore best understood as a fabric that happens to carry PCIe transactions. It preserves a device programming model, not local physical behavior. rPCIeBench provides the missing measurement contract: trace the complete path, express contention on directed edges, and schedule devices together with the interconnect that makes them reachable.

This article is an editorial analysis by Silicon & Systems. It restates the paper’s measurements and model in our own words. No source sentence, table, or figure is reproduced; all figures were created for this article. The paper and artifacts are available from the USENIX NSDI 2024 page. Copyright remains with the authors, 2024.