betaFeedback

WorksComputer Science

Fuse-All MoE Inference Kernel with Load Balancing on TPUs

Abstract

Mixture-of-Experts (MoE) architectures activate only a subset of experts for each token, allowing large language models (LLMs) to scale parameter capacity without a proportional increase in computation per token. However, intermediate data movement and uneven expert loads limit the efficiency of MoE inference on TPUs. To address these bottlenecks, we introduce Fuse-All MoE, a TPU kernel design that jointly optimizes expert computation, output communication, and load balancing. The kernel fuses token collection, expert projections, gated activation, and output writes, retaining expert computation intermediates on chip. Fixed token-slot destinations allow scheduling and expert placement to change while preserving routing semantics. In prefill-decode (PD) disaggregated serving, Fuse-All MoE balances EP16 prefill workloads through scheduling and expert redistribution, while symmetric EP8TP2 shares active expert computation between the two TensorCore devices within each chip during decode. Our experiments show that Fuse-All MoE reduces kernel runtime for a measured skewed EP16 prefill workload from 2.3x to 1.09x the same fixed reference time for a balanced workload. Across 32-512 decode tokens, symmetric EP8TP2 reduces kernel latency by up to 21.3% relative to fused EP16. With accompanying serving stack optimizations, the integrated system improves serving throughput by up to 34.5% over the Megablox EP16 baseline.

Introduction

Mixture-of-Experts (MoE) models increase model capacity while activating only a subset of experts for each token [1, 2, 3]. Expert parallelism distributes these experts across devices, but also makes inference sensitive to where the router sends work. A device with many active experts can delay the layer even when other devices are lightly loaded. Efficient execution must therefore address computation, communication, and load imbalance together.

These costs change between prefill and decode. Large prefill batches offer substantial weight reuse, but skewed routing can concentrate computation on a few devices. Small decode batches provide less work over which to amortize weight reads and communication. In this regime, the number of active experts can matter more than the number of assigned tokens. Balancing token counts alone may leave the dominant cost unchanged.

We present Fuse-All MoE, a Pallas implementation [5] that co-designs expert computation, output transport, and load management. Its central contribution is a fused compute-and-return pipeline whose token-slot output contract remains unchanged as tile order and expert execution placement change. Indexed input collection, both projections, gated activation, and direct writes execute in one expert launch, retaining intermediates and reusable weights in VMEM while overlapping computation with output communication. Figure 1 shows this boundary; routing and upstream input exchange remain separate.

Fusion boundary diagram showing separate-stage baseline and fused expert computation with direct output writes

Figure 1. Fusion boundary. Gate/up and activated intermediates stay in VMEM; expert results go directly to final token-slot buffers. Routing, input exchange, and postprocessing remain outside the expert launch. Boxes show logical stages, not measured kernel counts.

The second part addresses uneven routing. Row bucketing avoids padded computation, hot-expert scheduling exposes weight-prefetch overlap, and destination interleaving reduces traffic bursts. Static placement, runtime chipmate redistribution, and preloaded hot-expert copies change where work executes. A symmetric TP2 decode variant shares each active expert across chipmates. The implemented cross-chip copy path targets decode; prefill integration is an extension.

The paper develops two complementary contributions:

  • A fused EP16 compute-and-return pipeline. We integrate token collection, expert computation, and final-destination output exchange under explicit buffer-lifetime and completion rules. This removes intermediate materialization and receive-side redistribution while preserving output ownership when work moves.

  • Regime-specific balancing within that pipeline. For PD serving, EP16 uses load-aware tile and traffic scheduling, placement, and runtime redistribution for prefill, while symmetric EP8TP2 shares active experts across chipmates with aligned tile computation for decode. Preloaded copies provide a complementary cross-chip placement mechanism. The designs preserve the router's expert choices and a common token-slot output layout.

Section 3 develops the base EP16 kernel. Section 4 introduces imbalance-handling mechanisms and the decode variant. Sections 5 and 6 present evaluation and remaining limitations, distinguishing expert-kernel latency, full-layer cost, and serving throughput.

Background and Problem Definition

TPU Hardware and the Unit of Parallelism

A TPU7x (Ironwood) chip exposes two TensorCore devices to JAX, each with its own HBM address space. The chiplets communicate over a die-to-die link; physical chips use the inter-chip interconnect (ICI). A rank denotes one EP device: the base EP16 configuration spans sixteen devices on eight chips. A topology map identifies chipmates. § 4.7 distinguishes this layout from EP8 x TP2.

Each TensorCore contains a matrix unit (MXU), a vector processing unit (VPU), and scalar control. TPU7x's MXU uses a 256×256256 \times256 systolic array [4]. VMEM holds vector tiles and intermediates; SMEM holds scalar metadata. SparseCores have separate irregular-access hardware and memory. Below, KK denotes the matmul reduction dimension and NN its output width.

ResourceSpecificationConsequence
TensorCore devices / SparseCores2 / 4 per chipEP device count and physical chip count differ.
VMEM64 MiB per TensorCoreWeights, intermediates, and DMA staging compete for capacity.
HBM192 GiB capacity; approximately 7.38 TB/sUse chip-level bytes and chip-level peak throughput consistently.
Dense compute2,307 TFLOPs/s BF16; 4,614 TFLOPs/s FP8A roofline must match the arithmetic mode actually used.
ICI1,200 GB/s bidirectional per chipThis is aggregate bandwidth, not a per-destination guarantee.
Intra-chip linkDocumented as six times a 1-D ICI linkChipmate weight transfers avoid inter-chip torus traffic.

Table 1. TPU7x specifications relevant to this design [4, 9, 10]. Compute and HBM bandwidth are per physical chip, unless stated otherwise.

The peak compute-to-HBM ratios are approximately 313 FLOPs/byte for BF16 (2307e12/7.38e122307\mathrm{e}{12} / 7.38\mathrm{e}{12}) and 625 for FP8. These roofline ridge points bound the intensity needed to reach peak compute. Actual mixed FP8×\timesFP4 throughput also depends on generated instructions, MXU utilization, and achieved bandwidth.

Hardware and representation: physical chip boundary with two illustrative logical devices, TensorCores, VMEM, MXU, VPU, SMEM, HBM, and D2D within-chip links; packed FP4 weight representation; row-leading logical layout

Figure 2. Hardware and representation. Chipmates have separate address spaces; scalar metadata and vector tiles use separate memories. FP4 packing reduces weight storage, and a leading row axis supports row-addressed DMA.

Pallas, Layout, and Software Pipelines

Pallas expresses tiled kernels that Mosaic compiles into statically scheduled TPU programs. Fuse-All MoE explicitly issues local and remote DMAs ahead of use and waits on completion semaphores. JAX Megablox provides a public Pallas grouped-matmul implementation [15].

HBM fetches, MXU/VPU computation, and output writes can overlap subject to buffer dependencies. Shared bandwidth, scalar descriptor issue, and interconnect contention can still expose stalls.

Activations and outputs have a leading token-row axis and vector-sized trailing dimensions, such as [rows, N/128N/128, 128]. Weights pack eight FP4 values per 32-bit word along K. Logical matrices, packed storage, and MXU tiles are distinct representations; packing does not determine arithmetic precision.

MoE Semantics and Notation

A routed assignment is a (token, selected-expert) pair; its slot is the position among the token's kk choices, not a global expert ID. W1 combines gate/up weights; W2 holds down-projection weights. Token collection denotes indexed reads from the input table into VMEM, not inter-device input exchange.

SymbolMeaning
TT, TvalidT_{\mathrm{valid}}Token count of the compiled tensor shape, including padding, and the valid-token count used to plan work.
kk, MMExperts selected per token and total (token, expert-choice) assignments, M = Tk for complete routing.
EE, PPTotal experts and EP devices/ranks. Initial expert assignment has E/PE/P experts per rank.
dd, ffHidden width and expert intermediate width.
cc, ssOwned tokens per rank, c=T/Pc = T/P, and routing slot ss in [0,k)[0, k).
rer_e, tmt_mRows assigned to expert e and the maximum rows in one compute tile for that expert.
qq, geg_eQuantization-block width along K, and an expert's per-projection global scale.

Table 2. Notation. Unless explicitly stated otherwise, token and routed-row counts are global across the EP mesh.

For token tt, let e(t,s)e(t, s) denote the expert in slot ss and at,sa_{t,s} its router weight. A standard SiLU- or GELU-gated expert has d×fd \times f gate/up matrices and an f×df \times d down matrix:

he(x)=act⁡(xWgate,e)⊙(xWup,e),h_e(x) = \operatorname{act}(xW_{\mathrm{gate},e}) \odot(xW_{\mathrm{up},e),}
ze(x)=he(x)Wdown,e,z_e(x) = h_e(x)W_{\mathrm{down},e},
yt=∑sat,sze(t,s)(xt).y_t = \sum_s a_{t,s} z_{e(t,s)}(x_t).

This example activates the gate before multiplication by the up vector; variants that also transform the up branch require a matching fused activation. Routing remains outside the kernel, and per-expert normalization precedes the weighted sum (§ 3.1).

Expert placement determines the producer; token ownership determines the destination. For contiguous token partitions:

destination(t) = floor(t / c)
local_token(t) = t mod c
output_row(t,s) = local_token(t) * k + s

Rebalancing changes the producer without changing these destinations, allowing direct writes to compose with load balancing.

Routing example showing expert-major work and token-major destination slots

Figure 3. Routing example: T=4T = 4, E=4E = 4, P=2P = 2, k=2k = 2. Rank 0 owns A/B and rank 1 owns C/D; tokens 0–3 choose [A,C], [B,C], [A,D], and [B,D]. A's result for token 2, slot 0 returns to rank 1, row 0; C's result for token 0, slot 1 returns to rank 0, row 1. Expert sorting changes compute order, not destinations. This is not a benchmark workload.

Baseline and Serving Boundary

The baseline uses the Megablox grouped-matrix-multiplication kernels for expert computation under EP16 on a 2x2x2 TPUv7 slice, comprising eight physical chips and sixteen TensorCore devices. It does not use our fused compute-and-return kernel. Full-layer comparisons include routing, activation, exchange, and combination. Weight format, activation quantization, expert normalization, routes, and output semantics must be matched; exact baseline launch and HBM-transfer counts require compiled-trace confirmation.

The model runner sorts assignments by expert and replicates the full input-row table in every device's HBM before the expert launch. Sequence-parallel inputs are all-gathered first; prequantized FP8 inputs include scales. Token collection then reads selected rows locally, including for borrowed experts without additional token exchange. The launch returns one row per token-slot; postprocessing forms the weighted sum.

Related Work

Models and routing. GShard [2] demonstrates large-scale TPU MoE training; Switch Transformers [3] studies simplified routing and stable low-precision training. Expert Choice [13] balances load through fixed per-expert token buckets, allowing variable expert counts per token. Fuse-All MoE instead preserves token-choice top-kk assignments.

Expert computation. MegaBlocks [11] uses block-sparse GPU training to handle unequal expert loads without token dropping, while retaining block-granularity padding. Tutel [12] optimizes GPU dispatch/combine and adaptive MoE execution. On TPUs, JAX Megablox [15] provides Pallas grouped matmuls, integrated with separate routing and gating in MaxText [16]. MegaBlocks and Megablox are distinct references.

Communication and serving. DeepEP [14] specializes GPU expert-parallel dispatch and combine. SGLang [6] and vLLM [7] address runtime execution and attention-cache management. Fuse-All MoE targets the TPU expert launch: token collection, both projections, activation, and addressed output writes. Routing, upstream exchange, and the router-weighted sum remain separate.

Relative to the standalone TPU grouped-matmul building blocks in Megablox and MaxText [15, 16], our contribution couples the full expert-compute-and-return pipeline with load-aware scheduling and execution placement. The stable output contract connects these mechanisms: expert producers may change while final token-slot destinations remain fixed. We preserve token-choice routing rather than introducing a new routing algorithm.

Base EP16 Fused Kernel Design

We first describe the EP16 kernel: experts are partitioned across sixteen TensorCore devices on eight chips, and each expert is computed entirely by its owning device. Each device runs a fused tile pipeline that writes results directly to token-slot destinations. This section establishes the dataflow, precision, memory reuse, and communication contract for fixed expert ownership. Section 4 adds load-aware scheduling and placement, including the EP8 x TP2 decode variant.

Fusion and the Output Contract

Each device consumes activations, expert weights, row counts, token indices, and output destinations to produce one row per routed assignment:

  1. Collect token rows. Indexed row DMAs gather inputs directly from HBM into VMEM, avoiding an expert-sorted activation tensor in HBM.

  2. Gate/up matmul. The kernel unpacks W1, applies scales, and computes gate and up vectors together.

  3. Gated activation. The standard act(gate) * up operation writes directly to the down projection's VMEM input, removing a separate activation kernel and intermediate copy.

  4. Down-projection matmul. Multiplication by W2 produces a hidden-width result, staged for output DMA while later tiles compute.

  5. Direct write. Local or remote DMA writes each result to its final token-slot row, removing auxiliary receive storage and receive-side scatter-add redistribution.

Each device's output buffer holds ckc_k rows, one per token-slot. Transport writes but does not sum these rows, unlike conventional reduce-scatter. Separate postprocessing converts to FP32, combines router-weighted results, and casts to the requested dtype. For RMS-normalized expert blocks, scaling and normalization precede the sum:

u[t,s] = (1 + norm_weight) * z[t,s]                 # scale expert result
v[t,s] = u[t,s] / sqrt(mean(u[t,s]^2) + 1e-8)       # normalize each result
y[t]   = sum_s a[t,s] * v[t,s]                      # combine expert choices

RMSNorm of the weighted sum is not equivalent to this per-expert normalization. Postprocessing consumes the tiled slots directly; its logical reshape need not materialize an HBM transpose.

Boundary or storageFused path
Grouped activation materializationRemoved: token collection goes directly into VMEM.
gate/up matmul / activation / down-projection matmul intermediates in HBMRemoved within the fused expert-compute path.
Auxiliary receive-and-scatter bufferRemoved: writes target final token-slot addresses.
Final expert-slot outputRetained; the main HBM-output path materializes it before postprocessing.
Routing preparation and upstream activation exchangeOutside this fused launch; include their cost in full-layer measurements.
Weighted sum and per-expert output normalizationOutside this launch, implemented by a Pallas postprocessing kernel or JAX path.

Table 3. What fusion removes and what it retains.

Low-Precision Representation and Arithmetic

Packed weights and scales

Weights use FP4 e2m1, packing eight values per uint32 word along KK and reinterpreting them without a weight copy. FP32 block scales have shape [experts,Kblocks,1,N][\mathrm{experts}, K_{\mathrm{blocks}}, 1, N]; each scales a group of KK values. A separate global scale applies to each expert's gate, up, or down projection.

Block width is recovered from the scale shape under a power-of-two policy, with trailing blocks padded alongside weights. It affects scaling and tile selection and must be reported with the weight format.

Where scaling occurs

Weight scales follow each block matmul when their width is an integer multiple of the inner KK step; otherwise they are folded into BF16 weight slices before multiplication. This is independent of activation precision: W4A8 uses eight-bit activations, W4A16 BF16. At K=4096K=4096, compatible 256/512-wide blocks postscale in either mode; 128-wide blocks prescale.

W4A8 quantizes each activation row by KK-block absmax, handling zero blocks without division by zero. Prequantized FP8 inputs carry scales and bypass input quantization; the newly computed down-projection input still requires precision conversion.

Precision boundaries for standard gated activation

Figure 4. Precision boundaries for standard gated activation. FP4 block matmuls request FP32 results, cast to BF16 before scaling and summation. The K=4096 outer tile contains eight 512-wide quantization blocks; the down input is converted to its matmul precision. This shows source-level dataflow, not native instructions or measured error.

Accumulation Precision

Quantization-block matmuls request FP32 results but cast them into a BF16 shared accumulator in the BF16-output FP4 path. Block scaling, summation, global scaling, activation, and stored intermediates therefore do not remain FP32 throughout.

BF16 accumulation saves 16 KiB per row for an 8192-column accumulator relative to FP32, less for narrower NN tiles. An outer KK tile may contain multiple conversion-and-accumulation steps. Layer- and model-level accuracy validation is therefore essential, especially for long KK or multiple outer KK tiles.

Global scales apply after each projection's final KK tile, separately for gate and up columns. Completing all KK reductions for one output-column chunk before the next exposes scaling and stores for overlap with later matmuls.

Software Pipeline and Weight Residency

Buffers and dependencies

The outer loop processes expert tiles of at most tmt_m rows, including alignment padding. Each iteration pipelines next-input collection and weight prefetch with current computation and output writes. Expert identity and weight-buffer capacity determine reuse.

BufferMultiplicityLifetime or purpose
Collected input rows2 tilesCurrent compute and next-tile token collection use different slots.
W1 and its block scalesCircular weight buffersCurrent-expert consumption, cross-tile prefetch, and conditional residency.
W2 and its block scalesTuned ringIssued early enough to overlap token collection/gate/up matmul; repeated-expert hits skip loads.
Activated intermediate1 tileWritten by gate/up matmul's activation and consumed by down-projection matmul.
Shared accumulator / down-projection matmul tileShared scratch / 1 output tileComputation and conversion before temporary send buffers.
Temporary send buffers3 tilesRows remain stable until local/remote source DMA completion.
Output destination metadata2 bounded SMEM arraysDestination rank and output-row index for the active window.

Table 4. VMEM data buffers and SMEM destination metadata. Ring depths and tile shapes share a VMEM budget.

Before reusing a send slot, the kernel drains its outstanding DMAs. It then starts destination-metadata and next-input fetches. Down weights are requested before the current-input wait; gate/up weights are usually prefetched by the preceding tile.

Prefetch may overwrite a weight buffer only after its last consumer. A separate bank or sufficient remaining capacity permits early next-expert prefetch. Down weights follow the same wait-and-refill discipline. Completed rows enter a send buffer and leave once destination metadata is ready. Full-matrix residency requires a ring large enough to retain every weight tile.

Same-expert caching

Let S1S_1 and S2S_2 count the gate/up and down weight tiles. A ring holding an entire matrix lets consecutive tiles of that expert skip loads and waits; a smaller ring overwrites earlier tiles. For LL consecutive input tiles and packed expert size BexpertB_{\mathrm{expert}}, excluding scales:

cacheable_i = (weight_ring_depth_i >= S_i)
weight bytes for L consecutive tiles:
    without reuse: L * B_expert
    full residency: B_expert
Pipeline dependencies

Figure 5. Pipeline dependencies. Start-to-wait intervals show available overlap, not measured DMA durations [4]. Same-expert hits omit weight transfers and waits; scalar issue and late transfers may remain exposed.

With both projections resident, L=16L = 16 cuts this run's weight reads 16-fold, not total layer traffic. Activations, outputs, scales, and communication remain. The example uses 16 MiB of packed W1, 8 MiB of W2, and 0.375 MiB of FP32 scales at block width 512.

Arithmetic intensity and its limits

A gated expert performs approximately 6⋅re⋅d⋅f6 \cdot r_e \cdot d \cdot f matmul FLOPs and stores 3⋅d⋅f/23 \cdot d \cdot f / 2 packed weight bytes, excluding scales. One load gives weight-only intensity near 4⋅re4 \cdot r_e FLOPs/byte; per-tile reloads amortize only over tile rows.

At T=8192T = 8192, k=8k = 8, E=256E = 256, there are 65,536 global assignments: 256 rows per expert and 4096 per EP16 rank under balanced routing. Skew and alignment determine actual group lengths and tile counts.

Compute-bound claims must account for actual HBM traffic, achieved mixed-precision throughput, padding, scalar work, and communication. Resource work divided by achievable rate gives lower bounds; approaching their maximum requires sufficient buffering and overlap.

Direct-Write Exchange and Synchronization

Unique Output Ownership

If each (global token, slot) appears once, exactly one producer writes each output row, even when two slots choose the same expert. The later router-weighted sum combines expert results [1, 11]. Direct writes eliminate auxiliary receive-and-scatter storage, not the final landing buffer.

Resident weights can serve repeated expert tiles

Figure 6. Consecutive tiles retain W1/W2 when both rings fit. The hot-expert extension in § 4.2 reserves a separate W1 bank. A hit removes that projection's reload, not all tile traffic.

The wrapper skips output zero-fill only for complete routing with even token sharding. Invalid or padded routes map to a real expert with zero router weight, preserving coverage. Callers that drop assignments must initialize unwritten rows; buffer shape does not establish that a custom route list is a permutation.

Destination-Aware DMA Issue

The generic path issues complementary local/remote descriptors per valid row, with sizes (1,0)(1,0) or (0,1)(0,1), avoiding a destination-dependent branch. Zero-size descriptors carry no payload. Per-buffer counts track local and remote completions.

Large destination-sorted tiles instead use contiguous destination ranges and compile-time device addresses, issuing one applicable descriptor per row. This reduces address-encoding work but does not apply to interleaved destinations.

Output Staging

Three rotating send buffers leave two intervening tiles before reuse, versus one with two buffers. They are transport storage, distinct from the permanent kk expert-choice slots. Greater depth buys overlap slack at a VMEM cost.

Synchronization

  1. Entry readiness. Each rank completes any zero-fill and signals entry. The wait occurs before tile 0's first remote activity, overlapping initial compute. Even without zero-fill, the peer must enter before its output or receive semaphore is addressed.

  2. Source lifetime. Send-buffer reuse waits for recorded local-copy and remote-send completions, then clears the counters. Source completion does not establish receiver-wide completion.

  3. Tail completion. Exit drains output staging. Ranks without compute still perform the entry wait.

Send-buffer lifetime diagram showing two and three send buffers, stage reuse, DMA waits, and overwrite stalls

Figure 7. Send-buffer lifetime. Each buffer is reused every third iteration only after its outgoing DMAs complete; late transfers safely delay reuse.

4. Receive completion and exit. An EP barrier follows local drains; each rank then drains expected receive completions before exposing outputs.

The loop has no periodic cross-rank barriers, but retains input, weight, metadata, and buffer-reuse waits. Early issue reduces exposed waits without weakening dependencies.

Algorithm 1: base EP16 tile pipeline

Dependency-level outline for fixed expert ownership. Asynchronous stages overlap subject to buffer lifetimes. Section 4 changes schedules and producers while preserving this output and completion contract.

prepare routes, destinations, counts, and the fixed-ownership tile schedule
derive expected remote rows, excluding local copies
initialize unwritten outputs when required; signal entry readiness

for each scheduled expert tile:
    drain prior sends before reusing output staging
    fetch destinations and gather current/upcoming inputs
    prefetch weights without overwriting live buffers; reuse resident weights
    await input and W1; compute gate/up and activation in VMEM
    await W2; compute the down projection into output staging
    await destinations and entry readiness before first remote activity
    direct-write valid rows to unique token-slot destinations
    record source-completion counts

complete entry readiness even with no local tiles
drain all final-write sources
rendezvous across EP ranks; drain expected incoming output rows
outside the launch: normalize expert results, then apply the router-weighted sum
Protocol ordering and temporary send-buffer lifetimes

Figure 8. Entry readiness protects destinations, send drains protect sources, and receive completion protects consumers. Section 4.3 adds rebalancing-specific weight-transfer obligations. Safety does not depend on transfers fitting the overlap window.

Routing Metadata and Bounded SMEM Windows

Routing Preparation

Device-side JAX operations prepare expert-sorted assignments, token indices, per-expert counts, and output destinations. Full-layer timing includes this preparation.

Kernel setup uses replicated expert counts to build each device's tile schedule without another collective. In the base EP16 path, the schedule follows fixed expert ownership; the planners in Section 4 modify that schedule when work moves.

Replicated counts determine the runtime loop bound, including alignment padding. Compilation fixes scratch capacities and tile shapes. A shared tile selector and caller/kernel check enforce consistency. Section 4.3 describes the bound adjustment required when a device receives redistributed work.

Bounded Metadata Windows

Full int32 destination and output-row arrays would occupy 8×M8 \times M SMEM bytes. Instead, HBM stores them in 1024-entry slabs, and each tile loads a slab-aligned SMEM window addressed relative to its base.

slab_base=floor⁡(tile_row_start/1024)×1024\texttt{slab\_base} = \operatorname{floor}(\texttt{tile\_row\_start} / 1024) \times1024
window_slabs=min⁡(available_slabs,floor⁡((2046+tm)/1024))\texttt{window\_slabs} = \min(\texttt{available\_slabs}, \operatorname{floor}((2046 + t_m) / 1024))

For 64–512-row tiles, sufficiently long inputs use two slabs per array: 2048 entries or 8 KiB, totaling 16 KiB of SMEM. Short inputs cap the window; HBM padding makes tail DMAs safe. Metadata fetch begins early in the tile and completes before send-address lookup.

Token indices still cost about 4×M4 \times M bytes, and tile schedules grow with workload size. At top-8, 32,768 global tokens require 262,144 indices, or 1 MiB before other metadata. Only the destination window is bounded for a fixed tile configuration.

A bounded metadata window for a routed tile crossing a slab boundary

Figure 9. A tile starting at routed row 1000 crosses a slab boundary. The bounded two-array window supplies its send addresses; token indices and tile schedules remain workload-dependent.

Receive accounting

Expected remote-row counts exclude local copies. Under-counting leaves completion state outstanding; over-counting can wait for nonexistent rows. Section 4.3 extends this accounting when the producer changes.

Without a precomputed count, the kernel counts receives internally for M≤2048M \le2048 (T≤256T \le256 at top-8); larger shapes use outer JAX preparation. This changes accounting location, not necessarily routing-operand VMEM residency. Its benefit must include preparation cost.

Tile Selection and Static Shapes

Tile selection jointly chooses row height tmt_m, K/N sizes for both matmuls, and both weight-ring depths under VMEM, alignment, spill-space, model-shape, and workload constraints. The caller uses the same row height to count tiles, checked at compilation.

Balanced and skewed routes can favor different tile shapes. Reproducible comparisons should report the selected shapes, precision mode, routing distribution, and source revision.

Static buckets avoid unexpected recompilation only when serving precompiles supported shape, dtype, and configuration combinations. Routing varies at runtime. Padding must preserve output coverage but be excluded from valid-load planning, so padded shape does not select an inappropriate policy.

MoE Imbalance Handling

The base kernel removes intermediate traffic but does not by itself equalize work. We address imbalance through tile scheduling, execution placement, and communication ordering while preserving the router's expert choices and the output contract of Section 3.

With prefill-decode disaggregation (PD), the evaluated serving configuration selects EP16 for prefill at T≥1024T \ge1024 and symmetric EP8TP2 for decode at T≤512T \le512, where T is the global token count defined in Section 2.3. This phase-specific layout selection applies only to PD; it is not a general token-countbased switching policy for colocated prefill and decode. The reported decode measurements cover TT from 32 to 512.

Sources of Imbalance and Cost Models

Uneven routing produces partially occupied tiles, unequal per-device workloads, and bursts of output traffic. For prefill, row or compute-tile counts estimate the work to redistribute. For decode, the number of active experts and their tiles captures weight-streaming costs that token counts alone can miss. These are scheduling proxies, not guarantees of a compute- or bandwidth-bound regime.

Row bucketing and hot-expert interleaving improve local execution (§ 4.2); chipmate redistribution shifts work between devices (§ 4.4); destination interleaving reduces output bursts (§ 4.5). Static placement and preloaded copies address placement and cross-chip skew (§ 4.6), while symmetric TP2 shares decode work within a chip (§ 4.7). These are distinct execution options, not a claim that every mechanism runs in one launch.

Load-Aware Tile Scheduling

Row bucketing for partial tiles

For uneven expert loads, the kernel selects the smallest precompiled row bucket covering a tile's valid rows and leading alignment padding, avoiding full-height computation for every tile.

For 41 valid rows at offset eight, the required extent is 49. A 64-row body computes eight leading and fifteen trailing padding rows while skipping half a 128-row allocation. Bucketing reduces but does not eliminate padding, which also remains at block granularity in MegaBlocks [11].

Bucket selection covers valid rows and leading alignment padding

Figure 10. Bucket selection covers valid rows and leading alignment padding, separating computed padding from skipped allocation rows.

Each row bucket adds a separately compiled branch of the expert computation. The implementation caps bucket count times the row cost factor for the quantization configuration times tmt_m at 768, falling back to full-height computation; a 512-row allocation therefore cannot use two buckets. One experiment crossed an instruction-overlay threshold around 66K–70K bundles, triggering code reloads in the loop. This compiler- and shape-dependent observation requires remeasurement and is not total physical instruction-memory capacity.

Hot-Expert Interleaving

Alternating tiles of the busiest (hot) expert with other experts uses longer hot-expert computation to hide short-lived experts' weight loads. A stable merge preserves each stream's order and leaves unmatched tiles in a tail.

Scheduling uses the maximum, across devices, of busiest-expert load divided by the mean expert load on that device. Counts exclude invalid routes and precede chipmate redistribution. If the hot expert moves, another local expert supplies hot tiles without recomputing the global skew test.

Hot/non-hot expert interleaving and separate W1 residency and streaming

Figure 11. Hot/non-hot interleaving creates prefetch windows. A dedicated W1 bank retains hot weights; W2 follows consecutive-expert residency rules. Unmatched non-hot tiles form a tail, and weight waits remain.

Separate VMEM banks retain hot W1 weights while other experts reload theirs on each return. W2 may also reload on expert changes. The added prefetch overlap costs VMEM and weight traffic. Borrowed work remains at the schedule's end.

Intra-Chip Expert Redistribution

Chipmates offer a low-cost rebalance target. Moving an expert's work copies its weights into receiver HBM and changes the tile schedule for one invocation, not permanent ownership. Inputs are already replicated, so no additional token exchange is needed.

Planning. Replicated valid-row counts and the topology map determine at most one expert move within each physical chip's pair of TensorCore devices per invocation, without a planning collective. Different eligible pairs can move concurrently. The sender removes the selected tiles and the receiver appends them; SMEM holds the move description. A move must improve the chosen load estimate.

Regime-specific cost. For large prefill batches, the planner chooses a move that lowers the pair's maximum row load. Small-batch planning estimates weight cost by active-expert count: it chooses a few-row expert from a globally maximal-count device, requires an active-count gap of at least two, and rejects increases in the pair's maximum row count. The planner does not explicitly model transfer cost.

These are cost proxies, not regime guarantees. Few rows limit added compute, not weight bytes; improving one tied maximal sender need not reduce the global maximum.

Schematic of one expert move between paired devices, runtime planning and sender-pushed weights, and receiver borrowed-weight prefetch

Figure 12. Moving a four-row expert changes example loads from 24/12 to 20/16 but still transfers full W1/W2W1/W2 and scales. The receiver appends the work and waits before its first weight prefetch.

Weight transfer. The sender copies gate/up and down weights plus block scales into receiver HBM landing buffers. Local and borrowed weights feed the same matmul body.

Transfer starts after entry readiness and drains at sender exit. The receiver waits before its last local tile prefetches borrowed W1W1, or at tile 0 if it has no local work. Global scales come from a replicated table prepared outside the kernel.

Overlap. The example expert transfers 24 MiB of packed W1/W2W1/W2, excluding scales. Long resident schedules can hide more of this cost; short schedules may expose most of it. Chip-local transfers still contend for VMEM/HBM bandwidth.

Routing and completion. Moving tiles leaves token-slot destinations unchanged but changes local/remote classification and receive counts. The receiver extends its loop bound beyond the shared pre-move bound if needed. Weight transfers start only after entry readiness and drain at exit; idle devices retain these obligations. The exit barrier waits for the slowest peer.

Balancing Output Traffic

Expert-major, global-token order can create bursts to one destination under contiguous token ownership. Sorting within each expert by (local_token∗P+destination)∗k+slot(\text{local\_token} * P + \text{destination}) * k + \text{slot} encourages interleaving while preserving expert grouping and original token-slot identities. Here PP denotes the number of EP ranks (TensorCore devices), not physical chips.

The same workload-skew decision controls destination interleaving for all expert groups and hot-expert tile rotation. Noncontiguous destinations require per-row DMA issue (§ 3.5), trading address work for less bursty traffic. Benefits depend on routing and topology [8].

Destination runs in an expert group and interleaved logical destinations

Figure 13. Destination interleaving reduces multi-shard bursts but need not produce round-robin traffic under sparse or skewed routes. Logical destinations share physical links; no P-fold bandwidth gain is implied.

Static Placement and Preloaded Expert Copies

Static Expert Placement

Chipmate moves cannot change a chip's total load. Static placement uses observed expert loads to permute router checkpoint columns and expert weights consistently before distribution, retaining fixed expert-slot counts per device. Each expert is assigned once; this is neither replication nor explicit optimization of chip-pair totals.

Cross-chip balancing with preloaded hot experts

Intra-chip balancing cannot change a chip's aggregate load. The cross-chip design reserves one additional expert slot per chip. During model weight loading, calibration counts identify hot experts to copy onto other chips. Placement considers the holder's load and gives each copied expert one non-home holder. The copy and its scales reside in HBM; only working tiles are staged in VMEM. Under TP2, the copied expert is partitioned across the chipmates like the home experts.

At runtime, replicated routing counts determine whether a copied expert executes on its home chip or its holder. This changes execution placement, not the router's expert choices or weights. A selected expert's routed work moves as a unit: the home suppresses its tiles and the holder reads the preloaded bank. Token-slot destinations are unchanged, and completion accounting follows the actual writers. No runtime weight transfer or additional input exchange is required.

Figure 14 contrasts the implemented home-or-holder decision with a split-token extension. Splitting would assign disjoint rows of one expert to both chips and requires finer-grained placement and completion accounting; it is not implemented by the current planner.

Preloaded expert routing: placement and runtime routing

Figure 14. Preloaded expert routing. Weights are copied at startup. Left: the current planner assigns the entire expert batch to its home or holder. Right: a proposed extension distributes disjoint token rows across both copies. Arrows denote execution assignment, not an additional activation collective; original token-slot destinations are preserved.

The objective depends on the inference regime. A prefill extension would balance per-chip computation, estimated from routed rows or compute tiles. For decode, the number of activated experts captures the cost of fetching many weight matrices for few tokens. The implemented cross-chip decoder uses tile count, which approximates active-expert count for one-tile experts and also accounts for multi-tile experts. For expert cost tt and chip loads LhomeL_{\mathrm{home}}, LholderL_{\mathrm{holder}}, it accepts a move when Lhome−t≥Lholder+tL_{\mathrm{home}} - t \ge L_{\mathrm{holder}} + t, avoiding a transfer that reverses the imbalance.

One copy per chip bounds the added HBM capacity to one expert, including scales. It only helps experts with a pre-positioned copy and does not split a single expert across chips. Replica placement balances aggregate chip load; symmetric TP2 scheduling (§ 4.7) balances execution within the selected chip. Both remain sensitive to routing skew and the slowest chip.

Implementation scope. The current optional bank supports a configurable number of copies; the design above is the one-copy configuration. Cross-chip serving is implemented for decode/verification shapes up to 512 tokens and is disabled by default. Applying the same preloaded-copy mechanism to prefill with a row- or compute-based objective is an extension, not a result of the current evaluation.

Symmetric EP8TP2 Decode

Chip-Local Expert Partitioning

Decode cost can be dominated by the number of active experts and weight tiles, even when each expert receives few tokens. EP8 x TP2 shares each chip-owned expert across two cores: W1 is partitioned across matching gate/up channels and W2 across the reduction dimension. Both cores process the same input rows using complementary weight halves, without migrating weights during an invocation.

For input tile XX and core pp in {0,1}\{0,1\}:

Hp=act⁡(XWgate,p)⊙(XWup,p),Pp=HpWdown,p,Z=P0+P1.\begin{aligned} H_p &= \operatorname{act}(XW_{\mathrm{gate},p}) \odot(XW_{\mathrm{up},p}), \\ P_p &= H_pW_{\mathrm{down},p}, \\ Z &= P_0 + P_1. \end{aligned}

The partial sum reconstructs one expert result before normalization and router weighting. Compared with EP16 fixed ownership, both chipmates participate in every active expert, reducing intra-chip imbalance. Cross-chip skew is addressed separately by placement and preloaded copies (§ 4.6).

Symmetric Tile Scheduling

The symmetric schedule computes the same tile on both cores with identical valid-row bounds and row-bucket selection. Complementary weight partitions therefore carry matched matmul work, avoiding waits caused by unequal tile shapes. This aligns computation without requiring clock-step lockstep. Reduction of tile tt is delayed until the next iteration so that partial exchange overlaps later computation. Additional partial buffering and a separate reduction step provide the overlap window.

Cooperative gather and delayed reduction

After both cores gather the first tile, subsequent input gathers alternate between cores; the on-chip bridge forwards each gathered tile to its partner. Weight prefetching runs ahead subject to buffer lifetimes. Each core keeps one valid-row half of its output partial and sends the other half, then adds the received contribution for its owned rows and writes those completed rows to their original token-slot destinations. Each routed row therefore has one final writer, although both cores contribute to the expert computation.

TP partials stay in VMEM and use the on-chip bridge rather than an HBM intermediate or a separate TP-reduction launch. FP32 partial storage and transport do not imply full FP32 accumulation: the shared inner kernel retains its BF16 rounding boundaries. Figure 15 shows the dependency schedule.

Symmetric TP2 pipeline

Figure 15. Symmetric TP2 pipeline. Both cores compute the same tile with complementary weight halves; input sharing, weight prefetch, partial exchange, and delayed reduction overlap subject to buffer-lifetime dependencies. The diagram is a dependency schematic, not a measured timeline.

Algorithm 2: symmetric decode expert pipeline

Dependency-level outline. Finishing a tile waits for its partner partial, destination metadata, and output entry readiness before reducing and writing owned rows.

establish chipmate VMEM readiness; build the shared tile schedule
prime weights and inputs; both cores gather tile 0, if present
for tile t:
    gather t+2 and forward t+1 on their assigned cores
    await inputs, weights, and safe buffer reuse; compute P_p(t)
    send the partner-owned partial rows over the bridge
    if t > 0: finish tile t-1
finish the final tile, if any; drain bridge and output DMA
complete entry readiness even when idle; report remote-row counts
rendezvous across EP ranks; wait for incoming output rows

Scope and validation

The evaluated symmetric EP8TP2 path accepts FP4 weights and BF16 inputs for up to 512 global input tokens, with the token count divisible by the participating device count. § 5.2 reports its kernel-operation latency relative to fused EP16 at 32-512 tokens. Its benefit depends on whether reduced intra-chip imbalance and waiting outweigh bridge and reduction costs.

Empirical Evaluation and Validation Protocol

Model and Measurement Scope

Evaluation uses a TPUv7 (Ironwood) 2×2×22 \times2 \times2 slice with eight chips and sixteen TensorCore devices. The FP4 model has 256 experts, top-8 routing, and widths d=f=4096d=f=4096. EP16 assigns 16 experts to each device; EP8TP2 assigns 32 experts to each chip, partitioned across its two devices. The Megablox-only EP16 implementation is the baseline for the reported serving comparison; decode kernel-operation results compare symmetric EP8TP2 with our fused EP16 kernel. Scale-block width, full model/deployment configuration, and source revisions remain to be documented.

ParameterValue
Hidden / intermediate width4096 / 4096
Experts / top-k256 / 8
Hardware sliceTPUv7 (Ironwood), 2×2×22 \times2 \times2 physical chips
EP16 devices / experts per device16 / 16
W1 / W2 shape per expert4096×81924096 \times8192 / 4096×40964096 \times4096
Packed W1 / W2 bytes per expert16 MiB / 8 MiB, excluding scales
Assignments at T tokensM=8TM = 8T globally; mean M/PM/P per rank under balanced routing
Arithmetic configurationFP4 block-scaled weights; W4A8 or W4A16 must be reported explicitly

Table 5. Model and evaluation hardware. Both EP16 and EP8TP2 use sixteen TensorCore devices on eight TPUv7 chips.

Report expert-launch, full-MoE-layer, and serving timings separately. Full-layer timing includes preparation, exchange, and combination; distinguish inclusive/exclusive preprocessing costs [4]. Kernel attribution requires a controlled backend swap.

Precompile and warm up shapes, synchronize asynchronous completion, match inputs/routes and topology, and report repetitions and variability. Prefill, decode, and mixed workloads require valid/padded token counts, concurrency, and route distributions. Uniform routing provides a bound, not representative skew.

Performance Results

EP16 prefill imbalance. For the measured skewed prefill workload, imbalance handling reduces kernel runtime from 2.3x to 1.09x the balanced-kernel reference. Both ratios use the same fixed balanced-workload kernel time.

Symmetric EP8TP2 decode. Across decode token counts from 32 to 512, symmetric EP8TP2 reduces kernel-operation latency by up to 21.3% relative to our fused EP16 kernel. This kernel-operation comparison is distinct from the Megablox-based whole-stack serving result.

End-to-end serving. Relative to the Megablox-based EP16 baseline on the same 2x2x2 slice, whole-stack serving throughput improves by 12–34.5% across different request mixes and batch sizes. These gains include serving-stack changes and do not isolate the MoE kernel's contribution.

Ablation and Bottleneck Analysis

FLOPs and traffic. For TT valid global tokens, top-kk routing, and widths dd and ff, the three expert projections perform Fuseful=6TkdfF_{\mathrm{useful}} = 6T kdf FLOPs. Issued matmul work additionally reflects row and width padding; vector work for quantization and activation is separate. With AA active experts and one read of each packed FP4 weight, the weight-only arithmetic intensity is IW=4Tk/AI_W = 4T k/A FLOPs/byte. This optimistic intensity excludes scales, activations, metadata, output traffic, and weight reloads.

Global tokensUseful projection GFLOPsWeight-only FLOPs/byte
3225.84
128103.116
512412.364
20481649.3256
81926597.11024

Table 6. Analytic projection work and weight-only intensity, not measured performance. Balanced routing across all 256 experts, top-8, d=f=4096d = f = 4096, and one weight read per expert are assumed. FLOPs cover the entire sixteen-device invocation.

Decode. Few rows per active expert give little weight reuse per FLOP; row-bucket padding and fixed metadata, DMA-issue, and synchronization costs can dominate useful computation. At fixed token count, activating more experts can increase weight traffic without increasing useful FLOPs. Active-expert counts, issued-to-valid row ratios, and the slowest rank therefore complement token count as predictors of latency.

Prefill. More rows per expert amortize weight traffic when the weight panels remain resident across row tiles. If the selected panel set does not fit the buffers, or execution order breaks reuse, repeated loads can limit this gain. Higher token count consequently does not guarantee compute-bound execution; output traffic, exposed staging waits, and skew on the busiest rank can still determine runtime.

SOL accounting. Bounds must use the actual schedule and a consistent per-device scope. An optimistic compute/HBM bound is

t≥max⁡rmax⁡(FMXU,rPMXU,r,BHBM,rBWHBM,r),t \ge\max_{r} \max\left(\frac{F_{\mathrm{MXU},r}}{P_{\mathrm{MXU},r}},\frac{B_{\mathrm{HBM},r}}{\mathrm{BW}_{\mathrm{HBM},r}}\right),

where FMXU,rF_{\mathrm{MXU},r} counts issued matmul work, not vector operations. Interconnect and synchronization constraints can raise runtime further; overlapping component times must not be added as if serialized.

Report useful FLOP/s separately from issued matmul throughput. Total HBM utilization requires measured traffic; a weight-byte model yields effective weight bandwidth, not a hardware-counter utilization reading.

The aggregate results above do not isolate each scheduling and communication mechanism. The following ablations are further validation, not completed results. Each configuration must enter the intended compiled path: small shapes can bypass an optimization even when its flag is enabled.

  • Fusion boundary. Measure expert-launch and full-layer latency, including routing, exchange, and postprocessing; obtain baseline launches and materialized bytes from compiled traces.

  • Weight reuse. Hold row order and tile shape fixed while retaining or reloading weights; report weight bytes, reuse runs, and layer latency.

  • Staging depth. Compare two and three send buffers under a matched memory budget where possible; measure exposed reuse waits and exit tails, not only ICI bytes.

  • Direct-write issue. Compare generic and ordered issue on destination-sorted tiles, then separately evaluate interleaving, which removes that ordering condition.

  • Row specialization. Compare bucketed and full-row bodies on identical routes; record padded computation, instruction footprint, and overlays.

  • Load management. Isolate static placement, hot/non-hot rotation, and chipmate rebalance; report per-rank rows, active experts, transfer bytes, and longest-rank tails.

  • Preloaded expert copies. Compare zero and one copy per chip under identical routes and a fixed calibration profile, separate from evaluation traffic. Report per-chip tiles and active experts, slowest-chip latency, added HBM capacity, and startup loading cost. Prefill extensions additionally require compute-load measurements.

  • Precision and capacity. Compare FP8/BF16 activations with FP4 weights, accumulator precision, and accuracy; separate numerical changes from scheduling gains.

Compute-only and communication-only timings are diagnostic bounds, not additive costs, because they alter dependencies and contention. Measured timelines require profiler data; the figures in § 3 and § 4 show dependencies only.

Model Quality

All production evaluations passed, with no numerical issues observed. This reports production acceptance, not bitwise equivalence between different arithmetic or reduction paths. Detailed evaluation-suite settings and numerical-error summaries remain to be documented. § 3.5 states the output-ownership and DMA-completion invariants.

Production Integration

Reported production use requires a rollout record specifying model coverage, numerical/scheduling settings, integration boundaries, and monitored correctness/performance signals.

Discussion and Future Directions

Fusion avoids intermediate HBM traffic; routing identity and separate source/destination completion make the pipeline safe. Preloaded copies and symmetric TP2 address different levels of imbalance. Scalar overhead, padding, and the slowest-chip tail remain measurement priorities.

Upstream Activation Exchange

Consumer-directed upstream exchange could send tokens only to ranks owning selected experts and deduplicate local uses. Savings depend on distinct remote consumers, potentially numerous under top-8 routing.

Per-forward token caching requires invocation isolation and readiness before collection; relay routing requires topology-aware byte/hop and contention analysis. Neither is an established FP4-kernel capability.

Adaptive Placement and Prefill Extension

Preloaded copies cover a fixed set of hot experts. Extending cross-chip placement to prefill requires a compute-aware objective and serving integration. Further work includes adapting copy placement as routing changes, supporting multiple holders, and splitting a dominant expert across chips. These extensions must preserve unique token-slot producers and safe weight and output lifetimes across invocations.

Precision, Tuning, and Compiler Visibility

Further opportunities include mixed-precision lowering, more accurate memory-budgeted accumulation, instruction-aware specialization, and representative-route tuning. Generated-instruction and DMA-issue visibility would improve reproducibility; the objective remains the validated layer-level tradeoff.

References

  1. [1]N. Shazeer et al. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. arXiv:1701.06538, 2017. Paper.arxiv.org/abs/1701.06538
  2. [2]D. Lepikhin et al. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. ICLR 2021. Paper.
  3. [3]W. Fedus, B. Zoph, N. Shazeer. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. JMLR 2022. Paper.arxiv.org/abs/2101.03961
  4. [4]J. Jiang, Y. Chen, B. A. Hechtman, F. Zhang, Y. Mu. Ragged Paged Attention: A High-Performance and Flexible LLM Inference Kernel for TPU. arXiv:2604.15464, 2026.arxiv.org/abs/2604.15464
  5. [5]JAX Team. Pallas: a JAX kernel language. JAX documentation.
  6. [6]L. Zheng et al. SGLang: Efficient Execution of Structured Language Model Programs. NeurIPS 2024. NeurIPS paper.arxiv.org/abs/2312.07104
  7. [7]W. Kwon et al. Efficient Memory Management for Large Language Model Serving with Page-dAttention. SOSP 2023. Paper.
  8. [8]N. P. Jouppi et al. TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings. ISCA 2023. Paper.arxiv.org/abs/2304.01433
  9. [9]Google Cloud. TPU7x (Ironwood). Architecture and specifications. Accessed October 7, 2026.
  10. [10]Google Cloud. TPU7x (Ironwood) performance optimizations. VMEM and performance guidance. Accessed October 7, 2026.
  11. [11]T. Gale, D. Narayanan, C. Young, M. Zaharia. MegaBlocks: Efficient Sparse Training with Mixture-of-Experts. Proceedings of Machine Learning and Systems 5, 288–304, 2023. MLSys paper.arxiv.org/abs/2211.15841
  12. [12]C. Hwang et al. Tutel: Adaptive Mixture-of-Experts at Scale. Proceedings of Machine Learning and Systems 5, 269–287, 2023. MLSys paper.arxiv.org/abs/2206.03382
  13. [13]Y. Zhou et al. Mixture-of-Experts with Expert Choice Routing. Advances in Neural Information Processing Systems 35, 7103–7114, 2022. NeurIPS paper.arxiv.org/abs/2202.09368
  14. [14]DeepSeek. DeepEP: An Efficient Expert-Parallel Communication Library. Project repository; version-pinned documentation. Accessed October 7, 2026.
  15. [15]The JAX Authors. JAX Megablox: Grouped Matrix Multiplication Kernels for TPU Written in Pallas. Version-pinned implementation. Accessed October 7, 2026.
  16. [16]Google / AI Hypercomputer. MaxText: Optimizing with Pallas Kernels. Version-pinned documentation. Accessed October 7, 2026.

Paper details

vlnc:W_fthm52ae
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents