Most AI accelerators expose a matrix multiplication shape and ask the compiler to decompose everything into it. FuriosaAI’s ISCA 2024 paper reverses that contract. Its Tensor Contraction Processor (TCP) starts from the more general operation that underlies matrix multiplication and many tensor transforms, then lets hardware regroup its compute resources around the current axes[1].
The design is worth examining because flexibility can either remove graph transformations or move more responsibility into compiler search. The useful comparison is therefore completed model work after layout changes, fallback paths, memory traffic, and power, not peak arithmetic alone. This review uses that boundary to read the reported Llama-2 result without turning one tested configuration into a universal accelerator ranking.
Flexible slices without tiny-core programming
TCP uses coarse processing elements so software does not have to schedule a sea of small cores. Each element can act as one large unit or split into independent slices. A circuit-switched fetch network connects those slices to data sources, allowing an operand to be reused across several compute units. Broadcast and local input buffers add reuse inside the contraction engine. The compiler searches tensor shapes, loop order, partitioning, and the available slice topology together.
This middle ground matters. One rigid array is easy to program but wastes lanes on mismatched shapes. Hundreds of tiny cores adapt better but move scheduling and network complexity into software. TCP keeps a coarse programming surface while allowing the physical engine to fragment only when the operation benefits.
A memory-heavy 150 W design
The 5 nm chip integrates 256 MB of SRAM with 384 TB/s of internal bandwidth and 48 GB of HBM3 delivering 1.5 TB/s. Peak arithmetic is reported as 256 TOPS for BF16, 512 TOPS for FP8 or INT8, and 1,024 TOPS for INT4, within a 150 W thermal design power. Those numbers describe ceilings; the architecture’s claim is that flexible contraction mapping keeps a larger fraction useful across prefill and decode shapes.
For Llama-2 7B, the paper reports 2.7× the performance per watt of H100 and 4.1× that of L40S over the tested configurations. The authors use public NVIDIA results and note computational equivalence between Llama and Llama-2 7B in the compared path. The result is therefore evidence for the selected sequence lengths, batch sizes, precisions, and power assumptions, not a blanket ranking across models.
Tensor contraction is the algebraic contract
Matrix multiplication contracts one index shared by two matrices, but neural networks also reshape, transpose, batch, reduce, and combine tensors across several axes. A compiler can lower these operations into a sequence of GEMMs and data transformations, yet the intermediate layouts and underfilled matrix shapes can consume much of the expected accelerator efficiency.[1]
TCP exposes contraction more directly. The operation identifies input axes, reduction axes, and output axes, allowing the compiler to choose how they map onto compute slices and movement paths. Matrix multiplication remains a common case rather than the only native shape. This can reduce decomposition overhead when a model’s natural tensor form does not match one fixed array.
The abstraction still needs precise semantics for layout, accumulation, precision, and boundary conditions. A general operation is useful only if the compiler can generate predictable code and libraries can express common models without many special cases. Flexibility in the ISA does not guarantee utilization; it enlarges the mapping space that software must search.
Slices provide physical divisibility
Each coarse processing element can operate as a larger unit or divide into smaller slices. A large contraction can combine resources for throughput, while a small or irregular shape can use slices independently and avoid leaving most lanes idle. The design keeps software from scheduling hundreds of tiny general cores directly.
Divisibility has overhead. Slices need control, operand delivery, partial-result handling, and synchronization. If every operation fragments differently, setup can consume the saved compute. TCP’s architecture makes the divisions limited and structured so the compiler can reason about them and the fetch network can provide data without a fully general packet fabric.
The useful metric is lane-time utilization across a model distribution, not peak occupancy on one matrix. Small batch decode, attention projections, convolutions, and elementwise or reduction work present different axes. A slice scheme succeeds when it maintains enough utilization across these phases to offset its routing and control cost.
A circuit-switched fetch network targets reuse
Operand movement often consumes more energy than arithmetic. TCP configures paths from data sources to slices so one fetched value can feed several consumers. Broadcast and local buffers provide additional reuse inside an element. Circuit switching avoids per-packet routing decisions after a schedule is installed.
This is efficient for predictable tensor traffic. The compiler knows the contraction loop nest and can reserve paths before execution. Irregular dynamic routing would fit less naturally and may require conservative schedules or fallback handling. Contention must be resolved at mapping time because two transfers cannot assume the same circuit resource simultaneously.
The network should be evaluated with setup time, path occupancy, multicast capability, and blocked transfers. An arithmetic unit can be nominally active while waiting for an operand. Internal bandwidth of 384 TB/s describes the fabric and SRAM capability; achieved reuse determines how much external HBM traffic it prevents.
The 256 MB SRAM is a scheduling resource
Large on-chip SRAM can hold weights, activations, or tiles close to compute, reducing HBM energy and latency. Its value depends on allocation and lifetime. The compiler decides which tensors remain resident, which are double-buffered, and when storage can be reused after the last consumer.
Capacity is not the only constraint. Banks and ports determine simultaneous access, while placement determines whether the target slices can receive data at the advertised rate. A schedule that fits in 256 MB may still conflict on banks or fetch paths. Compiler cost models need physical contention rather than a flat capacity number.
The SRAM also distinguishes prefill from decode. Prefill can reuse blocks across many tokens and create large contractions. Decode may stream a large weight set for little work per token, making HBM the dominant source. On-chip capacity can retain selected layers or hot tensors but cannot hold every large model.
Precision changes both arithmetic and movement
TCP reports peaks of 256 TOPS at BF16, 512 TOPS at FP8 or INT8, and 1,024 TOPS at INT4. The doubling reflects more low-precision operations in similar hardware width. Actual acceleration requires a model whose accuracy permits the format and a software path that avoids expensive conversions.
Lower precision also reduces weight and activation bytes, which can improve a bandwidth-bound phase more than the arithmetic peak suggests. Mixed-precision accumulation, scales, zero points, and outlier handling add metadata and operations. A TOPS comparison should state whether these are included and whether the model reaches the same quality.
The paper’s Llama comparison uses selected configurations and public competitor data. Reproducibility requires model variant, sequence, batch, precision, decoding policy, software version, and measured versus specified power. Differences in any of these can change performance per watt independently of architecture.
Prefill and decode select different strengths
Prompt prefill processes many tokens in parallel and often produces matrix shapes large enough to use broad compute. Decode processes one next-token step for each active sequence and repeatedly reads weights. Batch can increase reuse, but interactive latency limits how long requests wait to form it.
TCP’s divisible contraction engine aims to cover both: large combined elements for prefill and smaller slices for awkward decode shapes. The 48 GB HBM3 and 1.5 TB/s external bandwidth still set a weight-streaming ceiling. A 150 W chip can be efficient if it sustains useful work at that ceiling rather than powering idle lanes.
Performance curves across batch and sequence are therefore more informative than one ratio. They show where the engine shifts from compute to memory, how slicing affects small shapes, and whether the compiler changes layouts smoothly. A product deployment needs tail latency and concurrency as well as throughput.
Peak specifications need three denominators
The 5 nm chip combines arithmetic, 256 MB SRAM, HBM interfaces, and the fetch network within a 150 W thermal design point. Peak TOPS describes arithmetic capability, 384 TB/s describes internal movement, and 1.5 TB/s describes external movement. None can predict application performance alone.
An operator should ask achieved operations per second, external bytes per useful operation, and sustained board or system power. A high internal bandwidth is valuable when it converts HBM reads into reuse; otherwise it can circulate data without increasing completed inference. TDP is a design envelope and may differ from measured workload power.
Thermal and memory power boundaries matter in competitor comparisons. Whether HBM, host, or cooling is included can alter performance per watt. The paper’s ratios are useful evidence within its stated setup, not a replacement for matched system measurement.
Compiler search is the operational risk
Contraction mapping includes axis order, tiling, slice grouping, data placement, circuit paths, buffering, and precision. The search space grows quickly. Heuristics and analytical models can prune it, while autotuning validates candidates on silicon. Compilation time must remain practical for frequently changing models.
Dynamic shapes require several prepared schedules or a runtime path that can select and adjust safely. A schedule optimized for one batch can waste slices on another. Unsupported operations need fallbacks that do not move tensors through a CPU or break graph fusion unnecessarily.
The compiler should expose why a mapping is limited: HBM, SRAM capacity, bank conflict, network path, or arithmetic occupancy. This observability lets engineers improve the model and allows buyers to test whether the claimed flexibility survives workloads outside the paper.
What the comparison proves and leaves open
The reported 2.7× and 4.1× performance-per-watt results for Llama-2 7B support the architecture under selected sequence, batch, precision, software, and power assumptions. They do not prove that TCP outranks H100 or L40S on training, larger models, every latency target, or a later software stack.
Using public competitor results introduces another boundary. Even when the computation is equivalent, system versions and measurement methods may differ. A matched laboratory comparison would strengthen the claim by holding model implementation, quality, traffic, and meter boundary constant.
The silicon specifications and end-to-end model results nevertheless establish more than a paper design. The open question is breadth: how quickly the compiler can support new attention variants, sparse routing, multimodal operators, and changing quantization without losing the utilization the slices were built to preserve.
What a deployment evaluation should include
Test a portfolio rather than one model: prefill and decode, convolution, attention, reductions, irregular axes, dynamic shapes, and precision transitions. Publish achieved utilization, latency distributions, SRAM residency, HBM bandwidth, internal-network stalls, and measured power.
Run compiler cold-start and tuning time, then introduce unseen operators and shapes. Verify numerical quality at each precision. Test concurrent models because edge and datacenter inference often share the device. The circuit-switched network and SRAM allocation must remain efficient when several schedules coexist.
Finally, compare total system cost. Include host requirements, HBM capacity, board power, cooling, software support, and the cost of a fallback path. A 150 W accelerator is valuable when its algebraic flexibility reduces the number and type of devices needed, not only when one benchmark reports more TOPS per watt.
The abstraction should reduce graph-level transformations
One way to validate tensor contraction as an interface is to count the transformations required to map a model. Compare intermediate tensors, transposes, padding, kernel launches, and HBM bytes against a GEMM-centered implementation. A contraction engine should win by preserving useful algebraic structure, not merely by assigning a different name to the same lowered sequence.
The analysis should distinguish transformations fused into an engine from those materialized in memory. A logical reshape may cost nothing, while a layout conversion that reads and writes a large activation can dominate. Compiler reports can make this difference visible and help model authors choose layouts that remain efficient across operators.
Graph-level evidence also tests composability. Optimizing one contraction can leave its output in a layout that slows the next. The best schedule minimizes end-to-end movement and latency, even when one operator runs below its isolated peak.
Ecosystem maturity determines usable flexibility
Deployment needs debuggers, profilers, numerical libraries, model import, quantization tools, serving integration, and stable compiler releases. A flexible engine with a narrow software surface can spend most of its life on fallback paths. Coverage should be reported as the fraction of model time and operators executed natively across released workloads.
Reproducible artifacts and regression suites are especially important because mapping search can change after a compiler update. Operators need to pin a known schedule, test a new one on canaries, and roll back. Long-term performance depends on the cadence at which the ecosystem turns new model patterns into validated contractions.
These software costs belong beside silicon efficiency in total ownership. They determine whether TCP’s divisible hardware becomes a general platform or remains an accelerator for the configurations already tuned in the paper. That distinction is the commercial test of the architecture.
What we take from it
TCP’s strongest idea is not a peak TOPS number. It is that the abstraction visible to the compiler should match the algebra and should remain divisible at runtime. If this software stack continues to cover new operators, a coarse but shape-flexible engine can offer a credible alternative to both rigid systolic arrays and general GPUs. The remaining test is deployment breadth: compiler maturity and stable utilization across rapidly changing models will decide whether the silicon’s flexibility survives outside the case study.
Compiler coverage determines realized utilization
A flexible tensor engine earns its area only when the software stack maps a broad operator mix onto it without excessive data conversion or fallback. Contraction shapes, layouts, sparsity, precision, and fusion boundaries change across models. A compiler can keep the arithmetic units active for a favorable kernel while the full graph loses time reshaping tensors or moving them through external memory. Peak efficiency therefore describes a hardware capability, not the fraction of an application that can use it.
Evaluation should report graph coverage and the cost of leaving the native path. For each representative model, buyers need the share of execution time mapped to TCP, bytes moved at every memory level, time spent in layout transformations, fallback latency, and the effort required to support a new operator. Versioned model traces can reveal whether compiler improvements expand coverage or merely optimize the existing subset. The decisive metric is completed application work per watt after mapping and data movement, measured across the model portfolio the accelerator is expected to serve.
Source and attribution
This article is an editorial summary prepared in our own words. It reproduces no paper text, figures, or tables. The explanatory figure was created for this article. The authoritative proceedings version is © IEEE 2024.