betaFeedback

WorksComputer Science

RankLane: Efficient MoE Prefill on Accelerators with Ranked Expert Capacities

Abstract

Mixture-of-experts (MoE) models expand model capacity by activating only a subset of experts for each token. On accelerators with static execution plans, expert capacities and placement are fixed before routing reveals each expert's load. Uniform worst-case capacities waste work on padding; smaller capacities tied to expert identities can overflow. We present RankLane, which ranks each device's experts by current load and assigns them to fixed-capacity lanes. Runtime indices select experts and token rows while tier populations, shapes and device placement remain fixed. An aggregation graph preserves device ownership and reduces each token's local contributions on one core. Routing calibration and measured execution costs guide tier selection; overflowing chunks rerun at full capacity. We evaluate full-model prefill of Qwen3-30B-A3B and Gemma 4 26B-A4B with MXFP6 weights on four Cloud AI 100 Ultra SoCs, using 128-2048-token chunks and including overflow detection and in-program recovery support. Against a dense baseline, RankLane achieves end-to-end prefill speedups of 1.27-4.18x on Qwen3 and 1.93-3.46x on Gemma.

Introduction

Mixture-of-experts (MoE) models expand model capacity while activating only a subset of experts for each token ([33, 11]). Efficient prefill matters for serving these models because the prompt must be processed before generation begins. Chunked prefill bounds the tokens processed per invocation ([2]), but each expert’s load still varies with the input and layer. Specialized inference accelerators target high performance per watt and lower serving cost ([7 Amazon Web Services, n.d.]). Turning MoE’s sparse computation into efficient execution on these platforms requires accommodating this changing demand.

This becomes difficult when the accelerator executes a static plan that fixes tensor shapes, expert capacities and placement before routing. We study this setting on Qualcomm Cloud AI 100, where fixed-capacity lanes execute expert GEMMs (Figure 1(a)). GPU block-sparse kernels adapt their work to routed tokens ([12]); the evaluated graph interface instead executes expert GEMMs in slots with compiled capacities. Runtime indices select weights and tokens within these slots; GEMM shapes and dispatch remain fixed.

Figure 1. Static execution and changing expert demand. (a) The evaluated Cloud AI 100 graph path fixes lane capacities, shapes and placement. (b) For eight top-2 tokens on one device, uniform capacity reserves 32 rows for 16 assignments; smaller capacities tied to expert identities overflow when the busy experts change. (c) Ranking fits both illustrated chunks into the same 16-row lane layout, followed by local aggregation.

Two coupled challenges limit efficiency. First, capacities tied to expert identities must accommodate their changing peak loads. Qwen3-30B-A3B selects 8 of 128 experts per token ([37]): capacity TT per expert covers any routing of a TT-token chunk but reserves 128T128T rows for 8T8T assignments. Smaller fixed capacities risk overflow when the busy experts change (Figure 1(b)). Second, returning expert outputs to tokens can erase the savings. Dense aggregation retains large lane-indexed buffers, and compiler placement can redistribute outputs between devices and cores. The latency benefit of smaller expert capacities therefore depends on the storage and locality of the return path (§ 3).

We present RankLane, which assigns capacities by current load rank while preserving a static execution plan (Figure 1(c)). Our key observation is that the busiest experts change across inputs, while the measured load envelope falls sharply below the first ranks. Ranking each device’s experts and assigning fixed rank intervals to tiers gives the largest lanes to the currently busiest experts. Runtime gathers select their weights and token rows while tier populations, tensor shapes and expert-to-device placement remain fixed. To realize these capacity savings, token-owned aggregation gathers a token’s local expert outputs and sums them on one core within each device, limiting intermediate storage and communication.

The paper makes three contributions. First, we characterize the capacity savings available from runtime load ranks and show how aggregation governs their latency benefit (§ 3). Second, we design a compiled program that combines adaptive expert assignment with device-local, token-owned reduction (§ 4). Third, we integrate calibration and measured execution costs to select tiers, with overflow detection and capacity-TT reruns for loads beyond calibration. Together, these mechanisms preserve token–expert assignments and keep the normal path within the compiled program, without host dispatch between layers.

We evaluate full-model prefill of Qwen3-30B-A3B and Gemma 4 26B-A4B with MXFP6 weights and an on-device KV cache on four Cloud AI 100 Ultra SoCs. Matched ablations isolate the contributions of tiering and reduction, and held-out routing and recovery experiments test capacity coverage (§ 6). Across 128–2048-token chunks, RankLane includes overflow detection and in-program recovery support and achieves end-to-end prefill speedups over a dense baseline of 1.27–4.18× on Qwen3 and 1.93–3.46× on Gemma.

Background and Related Work

MoE Inference

An MoE layer replaces the feed-forward block with EE experts and a router. For a token with hidden state xt∈Rd\boldsymbol{x}_t \in\mathbb{R}^{d}, the router selects the set Kt\mathcal{K}_t of kk experts with the largest gate values and returns

yt=∑e∈Ktwt,eWedown(SiLU⁡(Wegatext)⊙Weupxt),(1)\boldsymbol{y}_t = \sum_{e \in\mathcal{K}_t} w_{t,e} W_e^{\mathrm{down}} \left(\operatorname{SiLU}\left(W_e^{\mathrm{gate}}\boldsymbol{x}_t\right) \odot W_e^{\mathrm{up}}\boldsymbol{x}_t\right), \tag*{(1)}

with normalized gate weights wt,ew_{t,e}. Qwen3-30B-A3B uses E=128E = 128, k=8k = 8, d=2048d = 2048 and an expert width of 768,

so an expert’s three matrices hold 9 MiB in FP16. Implementations group assignments by expert, run each expert’s GEMMs over its gathered token rows, and return the weighted rows to their tokens; under expert parallelism the experts are spread over devices and dispatch and combine become cross-device exchanges ([24, 30]). Training systems bound each expert’s work by a capacity factor and drop the tokens beyond it ([24, 11]); at inference, a dropped assignment changes the model’s output. Prefill processes the prompt in chunks of TT tokens and writes their keys and values to the KV cache; decode then generates one token per step ([23, 2]). A chunk gives each layer 8T8T assignments, 2T2T per SoC on average when four SoCs hold 32 experts each.

MoE Systems

MegaBlocks represents routed expert work as a runtime block-sparse topology and executes it with GPU kernels without dropping tokens ([12]). DeepSpeed-MoE, FasterMoE and Lina optimize or schedule the all-to-all exchange ([30] (He et al. 2022; Li et al. 2023)). The combine has been made sparse on GPUs too: Tutel replaces GShard’s dense dispatch and combine einsums with sparse encode and decode kernels that assign tokens to thread groups ([20]), and vLLM sums each token’s top-kk expert rows in a single kernel, with fixed-size buffers and device-side indices that suit CUDA graphs ((vLLM Project, 2026; NVIDIA, 2026)). A kernel author decides there which thread reads which rows and where partial sums live; a graph compiler decides that for the program, which is the gap § 3.2 examines. Placement studies exploit routing structure (Yao et al., [38]; Yu et al., 2025), Huang et al. [19] replace static capacity with dynamic gating, and expert-choice routing fixes each expert’s tokens by construction but changes the model (Zhou et al., [40]). On Trainium, expert work is packed into fixed-size blocks whose weight loads are shared across an expert’s blocks and whose unused blocks are skipped (Amazon Web Services, [3]). DynaMoE recompiles a static-shape program when capacities change during training (Kossmann et al., [22]); our full-model programs take 7–26 minutes to compile, so capacities cannot follow each chunk by recompilation.

NPUMoE targets Apple Silicon NPUs, assigning experts to static capacity tiers using offline popularity and performing routing and aggregation on the CPU (Benazir & Lin, [6]). RankLane assigns experts to tiers by current load rank and expresses assignment and aggregation within the compiled multi-device program.

Accelerators with Static Execution Plans

Groq statically schedules instructions and data movement (Abts et al., [1]); Neuron compiles fixed-shape graphs for traced inference on Inferentia and Trainium (Amazon Web Services, [4]). Compiler-managed graphs also appear in SambaNova’s RDU and Graphcore’s IPU. SambaFlow maps dataflow graphs onto RDUs and produces a compiled executable (SambaNova Systems, 2024); Poplar compiles static graphs with tensor dimensions determined before execution (Graphcore, n.d.). Compilation ahead of time does not itself require fixed expert capacities: runtime control flow and custom-kernel interfaces vary across platforms. RankLane targets the constraints in § 1; transfer requires cost and locality checks (Appendix § 10).

A Cloud AI 100 SoC has 16 AI cores and 136 GB/s of LPDDR4x bandwidth to up to 32 GB (Chatha, [7]; Qualcomm Technologies, n.d.); each core has a scalar VLIW processor, a vector unit, a tensor unit with 4K FP16 MACs and 8 MB of software-managed vector memory. The Ultra card carries four SoCs under a shared 150 W limit.

The compiler takes an ONNX graph and one shape specialization per chunk length and produces a static program, partitioned over the cores of a SoC and, by multi-device tensor slicing, over the SoCs. Matrix weights can be MXFP6 (Rouhani et al., [31]), dequantized by the vector unit while the tensor unit computes. A KV cache stays on device when each cache input can be paired with its updated output (retained state). The evaluated graph interface supports runtime weight gathers but rejects ONNX If with a non-constant condition. Dynamic-shape compilers for CPUs and GPUs provide run-time dispatch (Shen et al., [34]; Zhu et al., [41]); here, expert GEMMs execute in slots with compiled capacities.

Motivation

Under this execution model, expert capacity is allocated before routing reveals demand. We study this problem on Qwen3-30B-A3B compiled for the four SoCs of one Cloud AI 100 Ultra card (setup in § 6.1). The exported graph (Qualcomm Innovation Center, [28]) runs a layer’s 128 experts as two stages of 64 lanes: each lane executes one expert’s gate, up and down GEMMs over a fixed number of token rows, its capacity CC. Compilation fixes the lanes’ shapes, placement and schedule. Figure 1 connects this hardware setting to the capacity mismatch and the opportunity to adapt expert assignments within the compiled plan. We first examine the capacity this can save, then the aggregation needed to realize the saving.

Fixed Capacities for Runtime Load Ranks

A fixed assignment binds a lane’s capacity to one expert. Capacity TT accommodates any routing of a TT-token chunk, but reserves 128T128T rows for Qwen3’s 8T assignments. Smaller capacities must cover each expert’s peak demand over the inputs the program will serve; average popularity alone does not determine that peak. Figure 1(b–Figure 1c) shows four experts with token counts (8,4,2,2)(8, 4, 2, 2) in one chunk and (2,2,8,4)(2, 2, 8, 4) in another. Fixed expert assignments require capacities (8,4,8,4)(8, 4, 8, 4) to cover both. Assigning lanes by current load rank requires (8,4,2,2)(8, 4, 2, 2) in either chunk: the expert needing the largest lane changes, while the lane capacities stay fixed.

We measure this opportunity using routing captures of all 48 layers for 50 prompts of 128 tokens from each of eight workloads, GSM8K, MMLU, SWE-bench Lite issues, HumanEval and its JavaScript version, Alpaca instructions, CNN/DailyMail articles and Chinese XNLI sentence pairs (Cobbe et al., [9]; Hendrycks et al., [16]; Jimenez et al., [21]; Chen et al., [8]; Muennighoff et al., [26]; Taori et al., [35]; Hermann et al., [17]; Conneau et al., [10]), and for chunks of 256 and 512 tokens from four of them (Appendix § 12). An expert in some layer receives 127 of 128, 256 of 256 or 508 of 512 tokens. Which experts are busy also varies: across workloads the 16 most loaded experts of a layer agree with a Jaccard index of 0.00–0.33, against 0.52–0.88 between two halves of one workload.

Figure 2 compares the capacity envelopes using device-captured routing. With each SoC’s 32 experts fixed in order of their calibration mean load, the held-out maximum is 90–126 of 128 rows at every position; covering this envelope reserves 93% of the rows of capacity-TT padding. With the same experts ordered by their load in the current chunk, ranks 17–32 receive at most 12 rows at T=128T = 128. The measured rank envelope therefore supports much smaller capacities below the first ranks.

Figure 2. Expert-load envelopes from device routing captures. Each panel uses 200 calibration and 200 held-out chunks, with matched counts per workload across eight workloads. Calibration fixes the mean-load expert order; curves summarize held-out loads. Red: fixed order. Blue: runtime rank. Black: deployed RankLane capacities. Dotted: capacity TT.

Ranks also fix how many lanes need each capacity. Assigning an expert independently to a capacity bucket by its load can change the number of experts in that bucket. Assigning fixed ranges of ranks to tiers keeps each tier’s lane count constant: eight ranks always occupy eight lanes, whatever their expert identities. Ranking within each SoC’s expert set keeps expert computation on its assigned device. A sort and gathers can then select the weights and token rows for fixed-shape lanes. The small capacities depend on the measured load envelope; inputs outside it require overflow detection and recovery § 5.2.

The benefit also depends on executed padding. The compiler skips some empty work, so reserved capacity does not directly determine executed work § 5.1. Smaller capacities also change buffering and memory traffic. We therefore evaluate compiled latency as well as provisioned rows.

Aggregation Limits the Benefit of Smaller Capacities

After expert execution, the program must return each expert’s weighted outputs to their tokens and sum the contributions. Runtime assignment changes which expert occupies each lane, and tiers produce outputs with different shapes. Reducing padded expert work leaves this aggregation to be performed; its storage and communication costs determine how much of the saving becomes lower latency.

The exported combine scatters each stage’s rows into a zero-filled accumulator of shape [lanes,T,d][\text{lanes}, T, d], reduces over each SoC’s lanes and sums the four SoC partials. Each lane still reserves TT output rows in this accumulator even when its expert capacity is smaller. Tiers of different widths require separate accumulators, so dense aggregation need not become cheaper as expert padding shrinks.

Placement adds another cost. Although each expert stays on its assigned device, the compiled aggregation can redistribute its outputs across devices; within a device, a reduction spread over cores requires exchanges between them. Sparse aggregation must therefore also preserve locality. Section 5.4 measures these effects with equivalent aggregation expressions that produce different compiled placements.

The interaction is visible in full-model latency. At T=512T = 512, replacing uniform capacity TT with ranked tiers lowers Qwen3’s latency by only 4% with dense aggregation, compared with 27% with token-owned aggregation. All four programs use the same runtime ranking, weight access and final cross-device sum (§ 6.4). The capacity savings are the same, but the aggregation determines the realized speedup.

These observations motivate two requirements: assign fixed lane capacities by current load rank, and return outputs to tokens with limited intermediate storage and communication. Section 4 develops the program structure that meets these requirements while retaining fixed shapes and placement.

RankLane

RankLane addresses both challenges of § 3: it assigns experts to fixed lanes by current load rank (§ 4.1), then gathers their outputs for token-local reduction (§ 5). Figure 3 shows the full pipeline within a fixed compiled graph. Calibration and measured costs select the tiers (§ 5.1); overflow recovery handles loads beyond calibration (§ 5.2). Appendix § 11 gives the Cloud AI 100 realization.

Figure 3. RankLane overview. Offline calibration and measured costs select fixed lane counts and capacities. At run time, each SoC (1) ranks its experts, (2) gathers weights and tokens into fixed-capacity lanes, and (3) combines outputs by token on local cores; (4) an elementwise sum combines device partials. After all layers, a host check triggers a capacity-TT rerun if any lane overflows.

Adapting Expert Assignment within Fixed Tiers

Assigning experts independently to capacity buckets would change each bucket’s population, violating the fixed lane count. RankLane instead assigns fixed intervals of load ranks to tiers: busy experts receive larger lanes while the number and shape of lanes remain constant.

Bind capacities to ranks. Device cc owns a fixed expert set Ec\mathcal{E}_c, 32 experts per SoC here. The router assigns nen_e tokens to expert ee and Nc=∑e∈EcneN_c = \sum_{e \in\mathcal{E}_c} n_e to the device. Sorting these counts gives πc\pi_c, with nπc(1)≥nπc(2)≥⋯n_{\pi_c(1)} \ge n_{\pi_c(2)} \ge\cdots. Tier ii executes one stage of LiL_i lanes per device at capacity CiC_i, covering ranks ai,…,ai+Li−1a_i,\ldots,a_i + L_i - 1, where ∑iLi=∣Ec∣\sum_i L_i = |\mathcal{E}_c|. Lane j∈{0,…,Li−1}j \in\{0,\ldots,L_i-1\} serves expert πc(ai+j)\pi_c(a_i+j). Each lane executes that expert’s gate, up and down GEMMs over its fixed row capacity.

Adapt through indices. The rank permutation selects each lane’s expert weights and token table (Figure 3, steps 1–2). The routing compacts these tables in expert order; gathers place the selected tokens into the fixed lane inputs. Tier counts, widths, capacities, tensor shapes, weight banks and expert-to-device placement are fixed at compilation. Only indices and data change within the compiled program. Every expert keeps one lane and its original tokens. If nen_e fits that lane’s capacity, the exact-arithmetic output is unchanged; Qwen3’s tested tiering variants are also bit-identical, although compilation can change summation order. A device provisions ∑iLiCi\sum_i L_i C_i rows instead of ∣Ec∣T|\mathcal{E}_c|T.

Why ranks. The rr busiest experts each hold at least nπc(r)n_{\pi_c(r)} assignments, and together at most NcN_c, so

nπc(r)≤Ncr,Nc≤min⁡(k,∣Ec∣)T.(2)n_{\pi_c(r)} \le\frac{N_c}{r}, \qquad N_c \le\min(k,|\mathcal{E}_c|)T. \tag*{(2)}

NcN_c averages kT∣Ec∣/E=2TkT|\mathcal{E}_c|/E = 2T here and reaches at most 1.9×1.9\times that in our Qwen3 256- and 512-token captures. Lower ranks therefore admit smaller capacities, whereas fixed expert identities can peak near TT (§ 3.1). Eq. (2) holds for any routing; our tighter empirical envelope is tested across eight workloads (Figure 2, § 7.1).

Returning Outputs through Local Token Aggregation

The dense return path still reserves TT rows per lane (§ 3.2). RankLane instead gathers each token’s contributions, at most kk, and sums them locally. The graph layout must preserve locality as well as remove the lane-dense accumulator.

Gather contributions by token. Let OcO_c hold device cc’s weighted expert outputs and row(t,j)(t,j) address the output of token tt’s jj-th expert. Each device computes a token partial, followed by a sum across devices:

Pc[t]=∑j:et,j∈EcOc[row⁡(t,j)],y[t]=∑cPc[t],(3)P_c[t] = \sum_{j:e_{t,j}\in\mathcal{E}_c} O_c[\operatorname{row}(t,j)], \qquad y[t] = \sum_c P_c[t], \tag*{(3)}

Figure 4 shows how the graph expression of this standard sparse combine determines device and core locality.

Figure 4. Aggregation placement on the evaluated compiler. (a) Global indices cause expert-output redistribution. (b) A SoC-batched gather preserves device locality, but lane-indexed reduction exchanges operands across cores. (c) RankLane also completes each local token sum on one core. All variants retain the final cross-SoC sum; §5.4 measures these placement effects.

Keep expert outputs on their device. Global indices and per-device slices cause our compiler to redistribute expert outputs (Figure 4(a)). RankLane exposes device ownership as the gather’s leading batch axis (Figure 4(b–Figure 4c)): tier outputs reshape from [D,Ls,Cs,d][D,L_s,C_s,d] to [D,LsCs,d][D,L_sC_s,d], without moving data, where DD is the device count and LsL_s the lanes per device. Batched indices select each token’s expert rows; contributions outside the device or tier are masked.

Sum each token’s contributions on one core. A lane-indexed reduction can stay on one device yet still exchange operands across cores (Figure 4(b)). RankLane arranges the gathered rows as [D,Tk,d][D,Tk,d] and sums over kk into [D,T,d][D,T,d] partials. The compiler splits this work by token tiles: each core gathers and sums its tokens’ contributions, without a cross-core reduction (Figure 4(c)). Section 5.4 measures both placement effects.

Combine device partials once. After accumulating all tiers’ local partials, the program sums (P0+P1)+(P2+P3)(P_0 + P_1) + (P_2 + P_3) per token tile (Figure 3, step 4). The compiler lowers this elementwise tree as a reduce-scatter with no root device; the next layer’s input normalization completes it. Algorithm 1 in Appendix B gives the complete path.

Each tier gathers TkT_k rows per device, including masked slots, versus LsTL_sT lane-dense rows. Only [T,d][T,d] partials enter the final cross-device sum. Added tiers still pay for gathers and local partials, which tier selection must include.

Choosing Tiers for Lower Latency

Ranking and local aggregation make smaller capacities useful, but do not determine how many tiers to compile. More tiers follow the rank envelope more closely while adding expert stages, aggregation gathers and transitions. We choose their widths and capacities together (Figure 3, top): calibration bounds the observed demand at each rank, and measured execution costs determine which groups of ranks are worth separating.

Calibrating capacities. A configuration S=((Li,Ci))i=1KS = ((L_i, C_i))_{i=1}^{K} has KK tiers in rank order. Let m(r)m(r) be the largest load at rank rr over calibration chunks, layers and devices. Tier ii starts at row aia_i:

ai=1+∑j<iLj,C1=T,a_i = 1 + \sum_{j<i} L_j,\qquad C_1 = T,
Ci=min⁡{T, gmax⁡(1,⌈h m(ai)g⌉)},i>1.(4)C_i = \min\left\{T,\ g\max\left(1,\left\lceil\frac{h\,m(a_i)}{g}\right\rceil\right)\right\},\qquad i>1. \tag*{(4)}

Here hh is headroom and gg the row granularity; each lane reserves at least one quantum. Sorted counts make the first rank the largest in its tier, so widths determine capacities. The deployed rule uses h=1.28h=1.28 and g=16g=16 (Appendix § 12). The main T=512T=512 search uses g=32g=32 to match its profiling grid; the other searches use g=16g=16, interpolating table entries where necessary. Headroom is empirical, so overflow still requires recovery (§4.4).

Account for executed padding. Cloud AI 100 can skip empty work, so provisioned rows alone do not predict latency. Capacities also affect buffering, weight loads and activation spills (Appendix § 13). We therefore use measured compiled execution costs to evaluate candidate tiers.

Include the return path in the cost. Width changes core placement and memory traffic as well as padding. For a stage of WW lanes per device at capacity CC, we measure its GEMM time run⁡(W,C)\operatorname{run}(W,C) and the gap to the next stage switch(W,C)(W,C) in complete two-layer programs with real routing and deployment settings. Let σ(j)\sigma(j) identify the jj-th tier in predicted compiler execution order. The estimated layer time is

τ^(S)=τpre+∑i=1Krun⁡(Li,Ci)+∑j=1K−1switch⁡(Lσ(j),Cσ(j))+A(Lσ(K),Cσ(K))+Δ(S,σ).(5)\begin{aligned} \widehat{\tau}(S)=\tau_{\mathrm{pre}}+\sum_{i=1}^{K}\operatorname{run}(L_i,C_i) \\ &+\sum_{j=1}^{K-1}\operatorname{switch}(L_{\sigma(j)},C_{\sigma(j)}) \\ &+A(L_{\sigma(K)},C_{\sigma(K)})+\Delta(S,\sigma). \tag*{(5)} \end{aligned}

τpre\tau_{\mathrm{pre}} covers sort-to-first-GEMM time; AA covers aggregation after the last GEMM. Δ\Delta contains fitted corrections for the first transition, later-stage slowdown and activation spills.

The last stage pays no transition. Rank order sets capacities; execution order sets costs. We predict decreasing-width execution, breaking ties by rank order (Appendix § 15).

Searching structures. With N=∣Ec∣N = |\mathcal{E}_c| experts per device, the search objective is

S⋆∈arg⁡min⁡Sτ^(S)\mathcal{S}^{\star} \in\arg\min_{S} \widehat{\tau}(S)

subject to

∑i=1KLi=N,Li∈{1,…,Wmax⁡},(6)\sum_{i=1}^{K} L_i = N,\qquad L_i \in\{1,\ldots,W_{\max}\}, \tag*{(6)}

K∈{1,…,Kmax⁡}K \in\{1,\ldots,K_{\max}\}, CiC_i from Eq. (4).

We score 27,466 structures for N=32N = 32, Wmax⁡=16W_{\max} = 16 and Kmax⁡=5K_{\max} = 5, then compile leading candidates to update spill corrections; unchecked candidates remain provisional (Algorithm 2, Appendix § 15). The objective excludes recovery and does not guarantee unseen loads fit. § 7 evaluates selection quality and offline cost.

Deployed configuration. Qwen3 uses 8/8/16 lanes per device across chunk lengths, varying only capacities (Table 1). It is among the search leaders at T=128T = 128 and 512; alternatives are close at T=256T = 256 (§5.5).

TTC1/C2/C3C_1/C_2/C_3Tiered rowsUniform rows
128128/48/161,6644,096
256256/64/323,0728,192
512512/128/646,14416,384

Table 1. Qwen3 tiers: 8/8/16 lanes per device. Provisioned rows are 8C1+8C2+16C38C_1 + 8C_2 + 16C_3, compared with 32T32T for uniform capacity-TT lanes.

Recovering from Capacity Overflow

Future inputs can exceed the calibrated envelope. RankLane reruns overflowing chunks with capacity TT, preserving smaller normal-path tiers while ensuring an undersized lane never produces an accepted result.

Detect and rerun the chunk. Detection reuses every lane’s token count; the host compares it with the lane’s capacity after each chunk (Figure 3, bottom). Recovery reruns that chunk at the same positions. Prefill writes the KV cache only at the chunk’s own positions, so the rerun overwrites every entry the failed attempt wrote, later chunks see only the rerun’s cache, and the result is that of an overflow-free run. Our recovery profile specializes capacities through the lengths of small row inputs, sharing the normal profile’s weights and cache.

Charge recovery to latency. For in-program recovery with overflow probability pp, a chunk costs tready+p tretryt_{\mathrm{ready}} + p\,t_{\mathrm{retry}} on average: treadyt_{\mathrm{ready}} includes profile-support and host-check overhead, and tretryt_{\mathrm{retry}} includes the capacity-TT rerun and its check.

A separate program also pays activation and prefix replay (Appendix § 17).

Evaluation

We measure full-model performance and separate reduction, tiering and recovery contributions, then evaluate runtime assignment, capacity, locality, latency-guided selection, robustness and recovery cost.

Experimental Setup

We evaluate Qwen3-30B-A3B and the text path of Gemma 4 26B-A4B on one four-SoC Cloud AI 100 Ultra card, at batch 1 with MXFP6 weights. The dense baseline uses fixed expert assignments, static weights, capacity TT and dense aggregation; it shares expert-parallel execution, head-parallel attention and tree routing scans with RankLane. RankLane includes overflow detection and in-program recovery. We measure uninstrumented full-model prefill on GSM8K chunks, using 60 timed inferences per pass after warm-up. Pass medians are normalized across mirrored sessions through a shared reference program. Unless identified as Gemma, ablations use Qwen3. Appendix § 10 gives the configurations, input selection and fitting procedure.

End-to-End Performance

RankLane prefills 1.27–4.18× faster than the dense baseline on Qwen3 and 1.93–3.46× on Gemma (Figure 5). These gains include token-owned aggregation, ranked capacities, runtime assignment and recovery.

Figure 5. We measure batch-one prefill with MXFP6 weights on a four-SoC Cloud AI 100 Ultra card across 128–2048-token chunks. RankLane speeds up full-model prefill by 1.27–4.18× on Qwen3 and 1.93–3.46× on Gemma over the dense baseline.

Contribution of reduction and tiering. Figure 6 decomposes the end-to-end speedup: (1) token-owned reduction and its elementwise final sum, retaining static weights and capacity TT; (2) runtime ranking, weight gathers and deployed tiers; and (3) overflow detection and in-program recovery. The tiering increment includes assignment and weight access; § 6.4 tests the interaction.

Latency relative to the dense baseline for Qwen3-30B-A3B

(a)

Latency relative to the dense baseline for Gemma 4 26B-A4B

(b)

Figure 6. Under the setup of Figure 5, we successively add token-owned reduction, ranked tiers and recovery support to the dense baseline. Across T=128T = 128–20482048, reduction gives 1.211.21–2.32×2.32\times on Qwen3 and 1.221.22–1.95×1.95\times on Gemma over the dense baseline. Ranked tiers add a further 1.101.10–2.11×2.11\times and 1.601.60–2.00×2.00\times, respectively, over reduction alone, before recovery overhead.

At T=128T = 128–512512, reduction gives 1.211.21–1.56×1.56\times and 1.221.22–1.80×1.80\times on Qwen3 and Gemma; ranked tiers add 1.101.10–1.39×1.39\times and 1.631.63–2.00×2.00\times, with 4.44.4–5.6%5.6\% and 2.32.3–3.5%3.5\% recovery overhead, respectively. At T=512T = 512, Gemma’s 982.6→546.9→274.0→283.7982.6 \to546.9 \to274.0 \to283.7 ms sequence saves 436436 ms through reduction and 273273 ms through tiering, then adds 1010 ms for recovery, delivering 3.46×3.46\times end-to-end. At T=2048T = 2048, Gemma’s recovery overhead grows to 10.2%10.2\%, reducing the delivered speedup to 2.78×2.78\times.

Runtime Expert-to-Tier Assignment

We test whether calibration can fix each expert’s tier for all inputs. Both alternatives in Table 2 use Qwen3’s deployed 8/8/16 tier widths and capacities, the same weight gathers, token-owned reduction and final sum. The compiled capacities stay fixed in both; the experiment changes how experts are assigned to them.

TTAssignmentRows/SoCAttempt (ms)Overflow chunks
128Calibration-fixed1,664111.1782/782
Runtime-ranked1,664105.40/782
256Calibration-fixed3,072145.3609/609
Runtime-ranked3,072141.70/609
512Calibration-fixed6,144235.3443/443
Runtime-ranked6,144225.50/443

Table 2. Qwen3 assignment at matched tier capacities. In held-out trace replay, fixed assignment overflows every chunk; runtime ranking overflows none. Attempt timings exclude recovery, required by all fixed-assignment timing runs.

Assignment policies. Calibration-fixed tiers freeze a separate expert order for each layer and SoC. We test orders based on mean, peak, 99th- and 99.9th-percentile calibration load and select among them using calibration alone; the selected order is the 99th-percentile policy at all three lengths. Runtime-ranked tiers (RankLane) instead sort the current routing counts and assign the busiest experts to the largest capacities.

Coverage at the same budget. We replay held-out device routing counts from 782/609/443 chunks at T=128/256/512T = 128/256/512, after the exclusions in § 7.1. A chunk overflows if any lane in any layer exceeds its capacity. All four calibration-fixed policies overflow every chunk at the deployed capacities; runtime ranking overflows none of the 1,834 chunks. Both alternatives provision 59–63% fewer rows than uniform capacity TT. Among the tested policies, adapting expert identities to current loads is what makes these smaller budgets cover the held-out routing.

Interpreting latency. The fixed policy also overflows on each timing input, so its listed attempt must be followed by recovery before an output can be accepted. Table 2 excludes recovery support from both programs; the complete-system results in § 6.2 include detection and recovery support. Appendix § 18 gives the capacity sweep, supplied-order timing control and recovery-cost estimates; Appendix § 13 gives additional ranking and tier-capacity controls.

Token-Owned Aggregation

We test how aggregation affects the benefit of smaller expert capacities, then isolate the device and core locality choices profiling protocol; these traffic measurements explain the placement effects without decomposing full-model latency into additive phase costs.

Latency-Guided Tier Selection

Tier count alone is insufficient. In full-model Qwen3 controls, three tiers improve on two by 1.01× at T=128T = 128 and 1.20× at T=512T = 512 (Appendix § 13). The benefit varies because each tier changes execution and transition costs as well as padding. We therefore select widths and capacities using measured execution costs rather than minimizing provisioned rows alone.

Selecting among candidates. Table 8 in Appendix § 15 compares predicted and measured Qwen3 MoE spans, fixing calibration and headroom within each chunk length. At T=128T = 128 and 512, the deployed 8/8/16 structure is fastest among the 8 and 21 measured configurations following the respective capacity rules. Its T=512T = 512 predicted score is within 0.1% of the search minimum. At T=256T = 256, the exact-fit search selects 8/16/8, measured at 2.35 ms versus 2.32 ms for the fastest of five tested candidates, a 1.3% gap. These single-sample profiles do not resolve close alternatives.

TThhWidthsPredictedMeasured
1281.288/8/16*2.041.94
16/162.091.98
2561.008/16/8*2.382.35
9/7/162.432.32
8/8/162.482.39
5121.288/8/16*3.383.32
6/6/4/163.593.33
8/8/8/83.964.03
16/165.125.67

Table 8. Qwen3 candidate selection: predicted and measured layer-1 MoE spans (ms) in instrumented two-layer cuts. Each group fixes calibration and hh; * marks the selected candidate (T=512T = 512: the deployed choice, within 0.1% of the minimum score). Capacities follow Eq. (4); The comparison excludes arbitrary capacity sweeps and the 12-layer cut.

Full-model selection benefit. Table 3 tests whether the cost model improves on Gemma’s manually selected, profiled four-tier structure. At T=512T = 512, the search chooses 11/12/9 lanes with capacities 512/112/32 from the calibration split (h=1.28h = 1.28). In an anchored session separate from Figure 5, the selected structure reduces full-model latency by 8.5% despite provisioning 18.2% more rows. It also reduces latency by 3.6% relative to 12/11/9 at 512/96/32, calibrated on the same split with h=1.28h = 1.28. All three programs include overflow detection and in-program recovery; none overflows on the timing chunk, and their logits are bit-identical. The 11/12/9 widths are selected in all five calibration splits, with capacities 512/112/32 in four and 512/96/32 in one. Table 3 fixes the primary split; the latter is a separate sensitivity point (Appendix § 19.§ 3).

SelectionWidthsRows/SoCms
Manual8/8/8/86,144309.9
Alternative12/11/97,488294.2
Latency-guided11/12/97,264283.5

Table 3. Gemma prefill at T=512T = 512 with recovery support. Latency-guided tiers cut latency by 8.5% versus manual tiers despite 18.2% more rows, showing that reserved capacity alone does not predict performance.

The selected 11/12/9 structure fits one replicated weight bank and avoids short 16-lane stages that use too few cores (Appendix § 19). We transfer these widths to the other chunk lengths and recalibrate capacities with h=1.28h = 1.28; this does not establish a separate width optimum at each length.

The model predicts 6.65 versus 6.83 ms per MoE layer for 11/12/9 and 12/11/9. Two-layer measurements reverse that order (7.91 versus 7.01 ms), while full-model timing favors the selected structure. This supports full-model validation of cost-guided candidates, without establishing a global optimum.

Prediction accuracy. Figure 8 compares predicted and measured Qwen3 MoE spans; the full-fit mean absolute percentage error is 4.3% across 48 programs. Excluding

Figure 8. Predicted versus measured Qwen3 MoE spans at T=512T = 512 for 48 programs, using the full fitted cost table. Predictions track the measured cost differences with 4.3% mean absolute percentage error; close candidates still require measurement to resolve their ordering.

each program’s own cost-table observations raises it to 4.9%; auxiliary terms still share the fitting pool. Gemma’s errors are 3.9% on 19 independent structures and 10.9% on 20 search-selected candidates, both excluded from fitting. All search-candidate errors are optimistic. The models predict MoE spans, not full-model latency; full-model validation remains necessary. Appendices § 15 and § 19.§ 3 give validation details.

Search cost and reuse. Qwen3’s T=512T = 512 cost table uses 256 grid programs, each taking about a minute to compile and a minute to profile, with the two phases overlapped. Gemma uses 251 programs with four samples each. Once the table is available, scoring all 27,466 Qwen3 structures takes seconds; spill checks take about a minute per candidate. The table can score new calibration profiles for the same model, precision and compiler settings. This concentrates device profiling on the cost grid and selected candidates, rather than every enumerated structure.

Capacity Robustness and Recovery Cost

Qwen3. Replaying held-out device routing for 782/609/443 chunks at T=128/256/512T = 128/256/512 across eight workloads gives no capacity exceedances (1,834 chunks total). The evaluation excludes calibration documents and HumanEval/JavaScript problem overlaps; fixed policies use the same filtered set without recalibration (Appendices § 12 and § 18).

Gemma. Executing the deployed 11/12/9 program on 430 held-out inputs at each of T=128/256/512T = 128/256/512 across eight workloads produces zero overflows. Tiered and capacity-TT profiles give identical logits on every tested input. Appendix § 19.§ 1 reports per-workload latency and output checks; Appendix § 19.§ 2 gives calibration-split, headroom and source-holdout studies.

Overflow recovery cost. With an injected host overflow signal, total chunk latency is 238.3/349.9/579.0 ms for Qwen3 and 248.1/376.2/674.3 ms for Gemma at T=128/256/512T = 128/256/512: 2.1–2.5× a normal recovery-enabled chunk. These measurements include the rejected tiered attempt, both host checks and the in-program capacity-TT rerun. Normal-path overhead is already included in Figure 5 and Figure 6. Gemma additionally passes 180 deliberate actual-overflow recoveries on four-chunk prompts, with accepted outputs and continuation matching direct capacity-TT execution at the stressed chunk (Appendix § 17).

Conclusion

RankLane enables efficient MoE prefill on accelerators with static execution plans. It binds lane capacities to current load ranks and uses token-owned aggregation to preserve device and core locality. Calibration estimates capacity needs, latency-guided search accounts for the execution cost of tier boundaries, and overflow recovery handles loads beyond the calibrated envelope.

We evaluate full-model, batch-one prefill on Cloud AI 100, using MXFP6 weights and 128–2048-token chunks. Including overflow detection and in-program recovery support, RankLane achieves end-to-end prefill speedups over a dense baseline of 1.27–4.18× on Qwen3 and 1.93–3.46× on Gemma.

AI USAGE DISCLOSURE

We used OpenAI’s ChatGPT/Codex and Anthropic’s Claude Code during research and manuscript preparation. Their use was iterative and substantial, including research planning, implementation assistance, and drafting and revision of the manuscript.

Ideation and literature search. ChatGPT/Codex helped refine the problem framing, motivation, method presentation, and formulation of tier selection. Both ChatGPT/Codex and Claude Code were used for literature search, including identifying related systems and accelerator architectures, summarizing relevant material, and developing comparisons with prior work.

Experiment design and code. The tools helped propose and refine baselines, matched-budget comparisons, tiering and reduction ablations, calibration and held-out evaluation protocols, and overflow-detection and recovery experiments. ChatGPT/Codex also helped discuss candidate evaluation models and the interpretation and presentation of results. Both tools were used to generate and revise code for experimental workflows, including program construction, compilation, experiment orchestration, data processing, result analysis, and visualization. Codex additionally executed build, measurement, and analysis commands through its tool interface and assisted with debugging, reproduction instructions, and checks against experiment artifacts.

Data preparation and results. AI-assisted code was used for workload preparation, tokenization, calibration/test partitioning, and analysis of routing counts and performance logs. No study-specific dataset examples or benchmark labels were generated with these tools. Timing and routing results were obtained from program executions and recorded experiment artifacts; simulator predictions were computed by the cost-model implementation.

Writing and presentation. ChatGPT/Codex and Claude Code were used for writing and revision. ChatGPT/Codex assistance covered the abstract, introduction, motivation, method, evaluation, conclusion, and appendices. It provided suggested equations and pseudocode, reviewer-style feedback, and help with figures, tables, captions, LaTeX formatting, and the separate submission PDFs. The authors are responsible for the final technical claims, citations, experimental results, and manuscript.

Experimental Setup and Compilation

Hardware and models. One Cloud AI 100 Ultra card has four SoCs with 16 cores and 32 GB each, using SDK 1.21.6. We evaluate all 48 layers of Qwen3-30B-A3B and the full 30-layer text path of Gemma 4 26B-A4B-IT (Gemma Team, [13]), including embedding, final normalization and language-model head. Gemma selects eight of 128 experts (hidden/expert widths 2816/704), adds a parallel dense MLP, and alternates five sliding-attention layers with one global layer. Main results use batch 1, MXFP6 weights, FP16 activations and retained KV state; ablations use Qwen3 unless identified as Gemma.

Comparisons. The dense baseline uses fixed expert assignments, static weights, capacity TT and dense output aggregation. Both programs share expert-parallel execution, head-parallel attention and tree routing scans (Appendix § 11); Gemma also shares direct activation gathers and logit softcapping. RankLane adds token-owned aggregation, ranked tiers and in-program recovery. The detailed tier controls already use token-owned reduction and omit recovery support (Appendix § 13); they are distinct from the dense baseline.

Timing. Primary measurements use first-chunk GSM8K chat prefills, outside the calibration range at T=256,512T = 256, 512 for Qwen3. Mirrored passes share a uniform-capacity, token-owned anchor; a program-plus-session fit of log latency separates program and session effects. Qwen3’s T=128/256/512T = 128/256/512 fit uses 74–83 passes, each the median of three 20-inference rounds after warm-up; maximum residuals are 0.63/0.88/1.7%. Dense and reduction-only controls use additional mirrored passes of 60 requests after two warm-ups, normalized through the same anchor. Longer-chunk Qwen3 sessions are in Appendix § 13. Full-model latencies are uninstrumented; phase times and traffic come from instrumented cuts. Numerical and multi-workload checks are reported separately.

Gemma uses held-out GSM8K chunks at all five lengths: four mirrored sessions, 24 passes and 60 timed requests after two warm-ups per pass. Fit residuals are 0.58/0.57/0.67/0.57/0.42%. Each length includes 240 injected recoveries; single-profile, recovery-enabled and recovered logits match exactly on the timing input.

Compilation. Qwen3 uses the command below with four-SoC partitioning and one specialization per chunk length (sequence TT, context 2T2T). All compared programs use -mdts-mos=1. Exact per-program commands are retained in the reproduction scripts.

qaic-compile -aic-hw -aic-hw-version=ai100
-retained-state -convert-to-fp16
-mxfp6-matmul -aic-num-cores=16 -mos=1
-aic-enable-depth-first
-mdp-load-partition-config=<4 SoCs>
-network-specialization-config=<T, 2T>
-mdts-mos=1

Gemma at T=2048T = 2048. Both compared programs use unwrapped cache-update positions within the retained context (exported modulus 2302^{30}), preserving the sliding-attention mask and avoiding within-chunk overwrites beyond the 1024-token window. The embedding table is split into four column blocks with concatenated lookups to fit the compiler’s per-core address space at context 4096; this split preserves logits bit for bit. Capacities and coverage are in Appendix § 19.

Scope. RankLane targets fixed expert capacities and placement with costly padding. Weight reuse and scheduling policies (Amazon Web Services, [3]) can change the preferred tiers; transfer requires new cost measurements and locality checks. Our evidence covers two models, batch-one prefill and one compiler, without task-quality, decode, larger-batch or other-platform evaluation. Calibration cannot guarantee unseen loads fit; reference-routing checks and final-program executions have distinct coverage (Appendix § 19).

Realization on Cloud AI 100

RankLane transforms QEfficient ONNX exports (Qualcomm Innovation Center, [28]). Compiler cost measurements are in Appendices § 13 and § 15.

Expert Placement and Retained State

Both programs keep each expert’s three matrices on one SoC (-mdts-mos=1). Head-parallel attention uses a leading device batch axis to colocate cache inputs and outputs as retained state. The four attention partials are combined as (G0+G1)+(G2+G3)(G_0 + G_1) + (G_2 + G_3) per token tile. These choices are shared by the compared programs.

Routing and Aggregation

Routing assigns compact rows through log⁡2T\log_2 T Hillis–Steele scan steps (Hillis & Steele, [18]). The dense baseline scatters the first stage’s weighted outputs into a zero-filled accumulator; later stages gather, add and scatter, and the per-SoC lane sum uses 16 token tiles.

Algorithm 1 instead gathers each token’s selected expert rows with a leading device batch axis. Other-device or other-tier entries use a safe index and zero mask because the gather does not zero-fill invalid indices. The final elementwise sum lowers to a reduce-scatter completed by the next layer’s normalization. Appendix § 14 isolates device placement and

Algorithm 1 Ownership-preserving token aggregation for one tier $s$ ($D$ devices)

Require: top-$k$ experts $\mathcal{K}[t,1..k]$ of each token $t$; map $\ell(e) \mapsto (s,c,l)$ (tier, device, lane); slot table $\operatorname{slot}[e,t]$; tier outputs $O_s$ of shape $[DL_s,C_s,d]$, $L_s$ lanes per device
1: $\widehat{O}_s \leftarrow \operatorname{reshape}(O_s,[D,L_sC_s,d])$ {no data movement}
2: for all $t \le T$, $j \le k$, devices $c$ in parallel do
3: $e \leftarrow \mathcal{K}[t,j]$; $(s',c',l) \leftarrow \ell(e)$
4: $m[c,t,j] \leftarrow [s'=s \wedge c'=c]$; $\operatorname{idx}[c,t,j] \leftarrow m ? lC_s+\operatorname{slot}[e,t]:0$
5: end for
6: $G \leftarrow \operatorname{gather}(\widehat{O}_s,\operatorname{idx}) \odot m$ {batched over devices: $[D,Tk,d]$}
7: $P_s \leftarrow \sum_j G[\cdot,\cdot,j]$ {per token, on its core: $[D,T,d]$}
8: after all tiers: $P \leftarrow \sum_s P_s$; $y \leftarrow (P_0+P_1)+(P_2+P_3)$ {elementwise}

core-local summation.

Runtime Assignment and Weight Access

Each SoC sorts 32 counts with TopK; masks, scans and slot tables remain in native expert order and do not depend on sorting. Each tier’s weight gathers lower to indirect DMA from a constant containing all 128 experts. The compiler replicates this bank across SoCs although each computes only 32 experts; Qwen3 occupies about 22.5 GiB per SoC versus 6.7 GiB for static weights. Replication is a lowering property, not an algorithmic requirement. Gemma’s constraints are in Appendix § 19. Each chunk length is separately specialized, and every program returns every layer’s lane counts for overflow detection.

Routing Captures and Tier Calibration

The original capacity-calibration captures run the model’s FP32 layers on the CPU and record every layer’s top-8 experts. At T=128T=128 we use 50 chat prompts per workload; at 512 tokens, 8 GSM8K chunks and 4 each from HumanEval, MMLU and SWE-bench Lite; the 256-token calibration set holds 16 GSM8K chunks and both halves of each 512-token chunk (56 in total). The held-out GSM8K chunks of the evaluation use test questions 1000–1003 (256 tokens) and 1004–1011 (512 tokens), outside the calibration range. Table 4 gives the largest per-SoC load at selected ranks, the input to the tier rule of § 5.1.

TTr=1r=12489161732
128127122105333214123
256256223134585025254
51250842424010189515010

Table 4. Largest load of the rank-rr expert of a SoC over all calibration chunks, layers and SoCs.

Figure 2 instead uses the device-routing captures audited below. Each panel has 200 calibration and 200 held-out chunks from the same eight workloads. Documents are split before chunking; evaluation applies the exclusions below.

We match each workload’s count across both lengths and both splits, allocating as evenly as availability permits, then sample without replacement with seed 20261030, independently of routing loads. Calibration fixes one mean-load expert order per layer and SoC; both assignment policies use the same held-out chunks. The capacity lines retain the original calibration above. The diagnostic is a subset of the audited data; coverage results use the full retained set.

Evaluation-set audit. We reconstruct all 436 original calibration prompts from source documents and verify their token IDs against the saved captures. Of the original 805/618/449 device-evaluation chunks, 23/5/4 contain a document used in capacity calibration. Excluding these leaves 782/613/445; additionally grouping HumanEval problems with their JavaScript translations leaves 782/609/443 (1,834 total). The latter set is used throughout the main-text Qwen3 overflow comparison. It is also disjoint from the fixed policies’ calibration documents under these groups. No retained chunk exceeds the deployed capacities. The audit checks source-document identity and known translations; it does not establish exhaustive semantic deduplication or a prospective test corpus.

Tier Assignment and Compiler Effects

Table 5 fixes token-owned reduction and gathered weights. B and C use capacity TT; D uses 16/16 lanes at TT and T/8T/8; E uses the deployed 8/8/16 tiers. Programs within each column share a compilation configuration. All give bit-identical logits on the tested chunks.

T=128T = 128T=256T = 256T=512T = 512
Qwen3-30B-A3B
B gathered, fixed order, TT—160.2302.4
C gathered, ranked, TT115.6163.3308.3
D ranked, two tiers106.4138.1269.8
E RankLane, three tiers105.1—224.8
C/E speedup1.10×—1.37×
Rows, B–C4,0968,19216,384
Rows, D2,3044,6089,216
Rows, E1,6643,0726,144

Table 5. Tier-assignment ablation: full-model latency (ms per chunk), with token-owned reduction in every program. B→C adds ranking; C→D/E adds tiers. Dashes omit historical variants outside the common configuration. All tiered programs here are single-profile; recovery is included in Figure 6. Row budgets are per SoC.

Ranking and capacity controls. With uniform capacity TT and gathered weights, B→C adds ranking and increases latency by 2.0/1.9% at T=256/512T=256/512. C→E retains ranking and weight gathers while introducing tiers, giving 1.10/1.37× speedups at T=128/512T=128/512. These are whole-program latency differences.

Executed padding. Instrumented layer-1 profiles of two-layer T=512T=512 cuts, with an einsum attention head-group sum, show that Where(r<ne,y,0)(r<n_e,y,0) gates activation gatherers, GEMMs and SiLU. B/C execute 6.9–7.7k rows per SoC, versus 3.3–3.5k for E. Thus, executed rows differ from reserved capacity even in the uniform-capacity programs. These trace-derived observations explain work differences but are not an additive decomposition of full-model latency. The search also checks activation spills because buffer placement can add DRAM traffic (Appendix § 15).

Longer chunks. Qwen3 at T=1024/2048T = 1024/2048 uses one held-out GSM8K chunk and a 2T2T cache per length. Three anchored sessions (22 passes; residuals 0.21/0.94%) establish ranked-tier timing; three further mirrored sessions normalize dense/reduction controls through a shared reduction anchor and recovery through single-profile RankLane (residuals 0.16/0.37%). Capacities TT, T/4T/4, T/8T/8 are extrapolated, not recalibrated; neither timing chunk overflows.

All programs share a 150 W card limit rather than fixed clocks. Median clocks are 1018–1100 MHz through T=512T = 512; at 2048, reduction-only runs at 912 MHz, single-profile RankLane at 995 MHz and recovery-enabled RankLane at 984 MHz. The dense and reduction controls can throttle at the longer lengths.

Aggregation Formulations and Phase Profiles

These controls fix capacity TT, static weights and the final cross-device sum. The reference uses SoC-batched, token-owned aggregation; Figure 9 tests device placement and Table 6 tests core locality.

Figure 9. Equivalent arithmetic, different placement. Three expressions of the same per-SoC token aggregation, with bit-identical logits at uniform capacity TT with static weights. Left: full-model latency relative to the layout-preserving SoC axis. Right: cross-SoC traffic inside one MoE layer at T=512T = 512, from instrumented two-layer profiles.

Lane-indexedToken-owned
Full-model latency (ms)401.9349.3
Cross-SoC traffic (MiB)12.114.0
Local-sum inter-core traffic (MiB)241.91.9
Local-sum busy time (core-ms)47.12.4

Table 6. Local-reduction ablation at T=512T = 512, uniform capacity TT and static weights. Full-model latency is uninstrumented; traffic and local-sum busy time are from layer 1 of instrumented two-layer cuts, summed across SoCs.

Device ownership. Figure 9 compares global row indices, explicit SoC slices and the SoC batch axis while holding the selected rows, summation order and final sum fixed. All three produce bit-identical logits. Global indices and explicit slices make the compiler redistribute expert outputs: 1539 MiB per layer against 14 MiB for the SoC batch axis at T=512T = 512. Their full-model latencies are respectively 4.4–5.3× and 3.4–4.1× higher across chunk lengths. This isolates the graph expression; redistribution can also change scheduling, buffering and tiling.

Token-local reduction. The lane-indexed formulation preserves device locality but reduces each token tile over lanes distributed across 16 cores. It is 11–15% slower across chunk lengths. Table 6 shows why device locality alone is insufficient: cross-SoC traffic is similar, but the local sum moves far more data between cores. Token-owned reduction gathers and sums each token’s expert rows on one core. These two formulations differ in FP16 summation order; the device-ownership variants above preserve the order exactly, while this comparison permits FP16 rounding differences.

Both forms gather similar amounts of data (898/924 MiB), but lane-indexed sums pull operands from 16 cores. Instrumented phase gaps exceed uninstrumented per-layer gaps and cannot be added to reconstruct full-model latency. Figure 7’s dense control uses one accumulator per tier width, writes the first stage without reading zeros, and sums lanes in 16 token tiles (§ 6.4).

Figure 7. Qwen3 full-model prefill under uniform and ranked capacities. Token-owned aggregation enables 9–27% latency reductions from tiering; dense aggregation yields a 37% slowdown at T=128T = 128 and only a 4% reduction at T=512T = 512.

Tier Search: Cost Tables and Validation

Cost Tables and Fitting

Figure 10 summarizes 256 programs at T=512T = 512, covering W=1,…,16W = 1,\ldots,16 lanes and C=32,64,…,512C = 32,64,\ldots,512 rows. Each combines one W×CW \times C stage with 16-row well for the other experts. Layer 1 of a two-layer cut is measured on one prompt using the third of five iterations: one instrumented sample per cell. Stage time is the mean first-to-last-GEMM span over SoCs; transition time is the mean gap to the next stage.

Figure 10. Qwen3 cost table at T=512T = 512: (a) stage GEMM span and (b) transition gap, measured on layer 1 of instrumented two-layer cuts; boxes mark deployed tiers. Figure 8 compares predicted and measured MoE spans.

Equation (5) adds sort-to-first-GEMM work, first-transition and final-aggregation corrections, and a spill correction. The first-transition and aggregation terms are least-squares fits over capacity sweeps and structures with known boundary stages; the spill term uses five spilling four-/five-tier programs. Copy counts of at least 60 trigger the additional spill correction, except when the cells already include it (14-/16-lane first stages at capacity 512). Stages are ordered by decreasing width, ties in rank order; this matches 44 of 48 validation programs. A compile check is needed because small capacity changes can alter spilling, adding 0.5–0.9 ms per layer.

Validation Protocol

The 48-program set includes capacity sweeps, 37 structures with two to five tiers, and a 12-layer cut. Measurements use layer 1 of two-layer cuts, except the labeled deployed structure, which uses the median of 11 layers in the 12-layer cut. In Figure 8, shading marks ±10% error and hollow markers indicate spills. The measurement spans rank-sort start to final aggregation; starting at routing would include hoisted work overlapping attention. Of the programs, 31 were predicted before execution and 17 supplied cost-table observations. Figure 8 in the main text shows the full-fit predictions. Mean absolute percentage error is 4.3%, or 4.9% when each program’s own cell observations are excluded. Auxiliary-term fitting still uses this pool, so the latter is not a fully independent fit. Stage context and single-sample variability limit fine ranking: the same 16×51216 \times512 stage takes 1.98–2.93 ms across four programs.

Enumeration and Calibration

Algorithm 2 scores candidates, compiles a leading batch for spill checks, then rescores. Cell costs interpolate linearly in capacity at fixed width, using the nearest endpoint outside the measured range. Unchecked candidates omit only the additional spill term; the returned ranking retains unknown spill statuses. Enumeration needs no device execution; the compile-check batch defaults to ten candidates and can be repeated.

Algorithm 2 Tier enumeration and a batch of compile-only spill checks

Require: rank maxima mm; TT, hh, qq, NN, Wmax⁡W_{\max}, Kmax⁡K_{\max}; measured tables and fitted terms; cached copy counts QQ; compile-check budget bb

  1. P←∅\mathcal{P} \leftarrow\varnothing

  2. for each ordered width tuple (L1,…,LK)(L_1,\ldots,L_K) satisfying the width constraints in Eq. (6) do Derive CiC_i by Eq. (4); form S=((Li,Ci))i=1K\mathcal{S} = ((L_i,C_i))_{i=1}^{K}

  3. Set σ\sigma by decreasing width, ties in rank order

  4. Score S\mathcal{S} by Eq. (5), using QQ when available; otherwise omit the additional spill term

  5. Add S\mathcal{S}, its score and spill-check status to P\mathcal{P}

  6. end for

  7. for up to bb lowest-scoring unchecked candidates in P\mathcal{P} do Compile a two-layer candidate with deployment settings

  8. Cache its layer-input copy count in QQ; classify a spill at 60 or more copies

  9. end for

  10. Rescore P\mathcal{P} using QQ, without double-counting spills already represented in the stage cells

  11. return P\mathcal{P} sorted by predicted latency, retaining unknown spill statuses

TTMaximaDeployedh=1.28h = 1.28h=1h = 1
128127/30/12128/48/16128/48/16128/32/16
256254/56/26256/64/32256/80/48256/64/32
512509/98/44512/128/64512/128/64512/112/48

Table 7. Qwen3 search inputs: maxima at ranks 1/9/17 from expanded captures (401/400/252 chunks). Deployed capacities use the original captures in Appendix C. Headroom rules define separate comparisons.

The T=128,256T = 128, 256 tables use their respective prompts, refitting sort-to-first-GEMM and final aggregation while retaining the T=512T = 512 first-gap and spill terms. Their 10/15 validation programs have mean absolute errors of 3.2/4.1%, including exact-fit and fixed-headroom variants. Table 8 compares only candidates under the same rule: eight at T=128T = 128, five at 256 and 21 at 512. Arbitrary capacity sweeps and the 12-layer result are excluded from those sets. All 27,466 structures are scored, but selection quality is relative to the measured subset, not a global optimum. At T=512T = 512, the model identifies the slower two-tier structure, but 6/6/4/16 nearly matches the deployed structure despite its higher predicted cost. Single-sample profiles do not resolve these close alternatives.

Qwen3 numerical checks

Table 9 checks the programs of Figure 5 against FP32 CPU inference on the timing chunks. These last-position checks do not establish task-level quality.

ProgramT=128T = 128T=256T = 256T=512T = 512
Dense baseline0.199 ✓0.072 ✓0.087 ✓
RankLane0.188 ✓0.091 ×0.096 ✓

Table 9. Relative L2L_2 error of last-position logits against FP32; ✓ indicates the same next token.

The tier-assignment controls produce bit-identical logits on these chunks. Changes to aggregation or compiler placement can change FP16 summation order; at T=256T = 256 the leading two FP32 logits are 0.58 apart. No lane exceeds its capacity. Recovery-enabled and recovered outputs match single-profile RankLane bit for bit (Appendix § 17).

At T=1024,2048T = 1024, 2048, the evaluated Qwen3 programs’ next token and top five agree with FP32. Across those programs, last-position KL divergence is 0.0039–0.0048 and 0.0022–0.0031, respectively. Recovery-enabled, recovered and single-profile RankLane give identical logits. These remain single-chunk checks.

Overflow recovery protocol and validation

Primary cost. Using Appendix § 10’s timing protocol, we inject a host overflow signal after an otherwise fitting tiered attempt, then check a capacity-TT rerun. Latency includes the rejected attempt, both checks and rerun; it measures the recovery path, not deployment overflow frequency. Recovery-cost passes use 60 timed requests after two warm-ups; Gemma repeats the mirrored schedule in two sessions. Normal-path overhead includes compiler profile effects as well as host checks.

Shared state. Shape-specialized row indices, formed from len(cap_rows), let normal and capacity-TT profiles share weights and KV state. The host checks every lane count after each attempt. A rerun overwrites the rejected chunk’s positions; later chunks return to normal tiers. Recovery-enabled and recovered outputs match single-profile RankLane exactly on both models’ timing inputs.

Model / TT12825651210242048
Qwen3127.4202.2344.41065.92285.2
Gemma140.1214.7390.61197.62476.6

Table 10. Additional total latency when recovery is invoked (ms), including the capacity-TT rerun and host checks. Adding it to the normal-path latency in Figure 5 gives the recovered-chunk cost in §5.6, up to rounding.

Qwen3 continuation. Final 48-layer programs process one four-chunk GSM8K prompt per T=128,256,512T = 128, 256, 512 with a 4T4T cache, mirrored passes and ten repetitions. Injecting overflow at chunk 0, 1 or 3 preserves every accepted output, including later chunks, bit for bit against static-capacity execution. Peak sampled additional allocation is 22.59–22.72 GiB per SoC relative to pre-load state. A separate resident fallback requires activation and prefix replay, with 9.26–9.38 s switching in these tests; the deployed path shares state instead. A two-layer cut also provokes actual overflow in 28 (layer, lane) pairs and recovers all four chunks exactly. Full-model Qwen3 cost measurements use injected signals.

Gemma actual overflow. Three math/coding/news inputs run as four T=256T = 256 chunks through all 30 layers. Besides normal 256/64/16 and capacity-TT profiles, an undersized 256/16/4 profile forces actual overflow at chunk 0, 1 or 3. Every stressed attempt is rejected and rerun, without injected signals. Ten repetitions per input in each mirrored pass give 180 timed recoveries. All accepted outputs and continuation match a reference that uses capacity TT directly at the stressed chunk and normal tiers elsewhere. This checks retained-state recovery; undersized profiles do not estimate deployment overflow probability, and the tests do not cover decode or broad continuing-prefix distributions.

Table 11. Gemma actual-overflow validation at T=256T = 256: median four-chunk prompt latency (ms), pooled over three inputs and two mirrored passes. The direct reference uses capacity TT at the specified chunk, with no failed attempt. The three-profile normal schedule takes 659.4 ms; these stress timings are separate from the deployed two-profile results.

Stressed chunkDirect referenceWith actual overflow
0710.5863.7
1709.3859.5
3706.8861.7

Table 11.

Fixed assignments at matched capacity budgets

Policies and capacity search. We reuse the device counts of § 7.1: 782/608/441 calibration chunks and 782/609/443 evaluation chunks at T=128/256/512T = 128/256/512, after the exclusions in Appendix § 12. Policy orders and capacity choices remain frozen from the original calibration split. Each fixed policy orders experts separately per layer and SoC by mean, peak, 99th- or 99.9th-percentile calibration load; ties use mean load. At each capacity pair, calibration-best selects the policy with the fewest overflowing calibration chunks, breaking ties by excess assignments. The search fixes tier widths at 8/8/16 and the first capacity at TT, and enumerates 16≤C3≤C2≤T16 \le C_3 \le C_2 \le T in 16-row steps. Neither policy nor capacity selection uses evaluation overflow.

At the deployed capacities, calibration-best selects the 99th-percentile policy at all three lengths. All four fixed policies also overflow every evaluation chunk at every other capacity pair within those row budgets. When minimizing rows subject to zero calibration overflow, calibration-best instead selects peak order. Its smaller budgets sacrifice evaluation coverage, especially at T=512T = 512 (Table 12). These comparisons cover the four tested calibration policies, not all possible fixed assignments.

T=128T = 128T=256T = 256T=512T = 512
Same capacities as deployed RankLane
Rows1,6643,0726,144
Runtime rank, overflow0%0%0%
Each of four fixed policies100%100%100%
Capacities selected for zero calibration overflow
Calibration-best, rows3,8407,04013,056
(evaluation overflow)(1.4%)(2.1%)(15.8%)
Fixed 99th, rows4,0967,68015,488
(evaluation overflow)(0.0%)(0.3%)(0.5%)
Uniform capacity TT, rows4,0968,19216,384

Table 12.

Device controls. The fixed-99th and supplied-order programs share RankLane’s tier capacities, weight-bank gathers, aggregation and final sum. Both remove runtime ranking while retaining runtime-valued gather indices. The supplied order comes from the timing chunk’s FP32 routing and produces bit-identical logits to RankLane, but disagrees with the MXFP6 device order, including strictly busier experts placed below tier boundaries. The latency difference includes placement effects and does not isolate sorting.

The supplied-order program takes 103.1/139.8/229.0 ms at T=128/256/512T = 128/256/512, against the 105.4/141.7/225.5 ms ranked control in Table 2. RankLane differs by +2.3%, +1.3% and −1.5%, respectively. These are whole-program differences that include tier-placement effects, rather than an isolated measurement of sorting.

Table 12. Fixed assignment versus runtime ranking. Overflow is the percentage of evaluation chunks with any lane over capacity. Rows are per SoC. The lower panel reports minimum row budgets with zero calibration overflow and their evaluation overflow in parentheses. The evaluation-set qualification in § 7.1 applies.

Timing and recovery estimate. Mirrored anchor/control passes extend the log-latency fit, with maximum residuals 0.50/0.88/1.69%. In this expanded fit, a separately optimized uniform-capacity recovery program takes Lfull=113.9/169.0/288.3L_{\mathrm{full}} = 113.9/169.0/288.3 ms at T=128/256/512T = 128/256/512. It uses fixed expert assignments, static weights and token-owned reduction; capacity TT guarantees no overflow.

Fixed-99th overflows on the timing inputs and is charged as a failed attempt. The estimate L^=Lattempt+pLfull\hat{L} = L_{\mathrm{attempt}} + pL_{\mathrm{full}} combines these single-input times with test-set overflow probability (p=1p = 1 for fixed-99th, 0 for RankLane). It gives 225.0/314.3/523.6 ms for fixed-99th, or 2.13–2.32× RankLane’s single-profile latency. It excludes host checks, switching and prefix replay and is not a measured recovery sequence or a corpus-wide latency average. Appendix § 17’s measured recovery paths concern RankLane only.

Gemma 4: Calibration and Validation

Model, compilation and timing settings are consolidated in Appendix Appendix § 10.

Tiers and capacity coverage. The delivered program uses 11/12/9 lanes per SoC, selected at T=512T = 512 and transferred to other lengths. Capacities are calibrated separately with h=1.28h = 1.28 and g=16g = 16 (Table 13), using uniform-capacity device routing from 449/449/451/232/126 calibration chunks. The corresponding held-out sets contain 430/430/430/224/120 chunks, with no offline exceedances. Splits retain source-document groups and exclude timing-prompt documents; longer inputs concatenate groups within their workload and split. These are retrospective partitions. Direct final-program coverage at the first three lengths is reported below; longer-length checks cover reference routing and the timing inputs, not corpus-wide tiered execution.

TTC1/C2/C3C_1/C_2/C_3Tiered rowsUniform rows
128128/32/161,9364,096
256256/64/163,7288,192
512512/112/327,26416,384
10241024/192/6414,14432,768
20482048/384/12828,28865,536

Table 13. Table 13. Delivered Gemma capacities for 11/12/9 lanes. Rows are provisioned per SoC; uniform capacity reserves 32T32T.

Width and memory constraints. Two-layer probes show that a 16-lane stage with 16 or 32 rows uses only four cores per SoC, serializing expert work. Qwen3’s 8/8/16 layout also induces duplicate replicated MXFP6 banks, projected to require about 36 GiB per SoC. The selected 11/12/9 structure fits with one replicated bank. Its benefit over a manually selected four-tier layout is measured in §5.5. Transferring the selected widths does not ensure the best latency at every length: at T=256T = 256, the earlier four-tier complete system took 151.8 ms versus 161.6 ms for the frozen transferred layout, in separate anchored fits.

Final-Program Corpus Validation

All 430 held-out inputs run at each of T=128,256,512T = 128, 256, 512 on the selected-layout program, its capacity-TT profile and Dense baseline: twice per input in each of two mirrored passes, after two warm-ups, starting at position zero. None overflows; repeated outputs match exactly. Table 14 gives per-workload ratios of median request latency including checks, separate from the primary single-input fit. Repetitions and lengths reuse documents.

WorkloadT=128T = 128T=256T = 256T=512T = 512
GSM8K1.91×2.23×3.45×
HumanEval1.94×2.25×3.46×
JavaScript1.93×2.25×3.48×
MMLU1.91×2.23×3.45×
SWE-bench1.93×2.24×3.45×
Alpaca1.87×2.18×3.37×
CNN/DailyMail1.91×2.23×3.44×
Chinese XNLI2.01×2.28×3.44×

Table 14. Table 14. Final Gemma program: per-workload dense-to-complete speedup at 11/12/9 widths. Each entry uses all held-out inputs of that workload at the given length; zero actual overflows occur.

Tiered and capacity-TT profiles give identical logits on every input. Dense and tiered programs agree on the next token for 384/430, 378/430 and 369/430 inputs (89.3/87.9/85.8%); maximum relative logit L2L_2 differences are 0.356/0.463/0.731. These differences occur without overflow. The comparison establishes neither their cause nor task accuracy or numerical equivalence of the dense and tiered graphs.

Device-Routing Calibration at 512 Tokens

Capture and splits. The 30-layer MXFP6 static-capacity program captures 916 first-chunk prefills across eight workloads, with 8TT assignments per layer and no capacity truncation. Five seeds split six workloads into random chunk halves and HumanEval/JavaScript into paired problem halves, keeping document groups disjoint. Tests exclude calibration near-duplicates and timing/cost-table documents, leaving 451 calibration chunks and 430/429/432/427/431 test chunks. These are retrospective resplits of the earlier document pool.

Coverage and headroom. With 11/12/9 widths, fixed h=1.28h = 1.28 gives zero overflows in every mixed split. Calibration-only cross-validation (CV) selects h=1.0h = 1.0 in the mixed splits and h=1.0h = 1.0–1.05 in the source holdouts. It gives one overflow per mixed split and one across the source holdouts. All failures across mixed splits and source holdouts concern the same HumanEval chunk, whose rank-12 load reaches 84 rows against capacity 80. On the primary split, fixed headroom raises provisioned rows from 6,880 to 7,264 (5.6%) and covers this outlier. Zero calibration overflow alone is therefore insufficient for held-out coverage.

Source holdouts. Table 15 holds out each of seven sources, pairing HumanEval with its translation. After excluding three timing-prompt chunks, 913 chunks appear once as held-out tests. Calibration alone freezes capacities and headroom; these holdouts reuse the mixed-split document pool and do not constitute independent trials alongside those splits.

Held-out sourceCV headroomFixed h=1.28h = 1.28
GSM8K0/1490/149
HumanEval + JavaScript1/810/81
MMLU0/4000/400
SWE-bench0/1530/153
Alpaca0/140/14
CNN/DailyMail0/840/84
Chinese XNLI0/320/32
Total1/9130/913

Table 15. Gemma T=512T = 512, 11/12/9 widths: offline overflow chunks / tested chunks under source-held-out calibration. CV chooses headroom using calibration only.

Interpretation. Offline overflow is a lane exceeding capacity under reference-program routing, not a recovery event. Uniform-capacity and tiered routing differ by up to six rows at a rank on the reference chunk; that check does not bound other inputs. Final-program tests cover the primary split, not every holdout configuration. Calibration coverage is limited to T=512T = 512, empty KV state and MXFP6, with small Alpaca/Chinese samples.

Calibration size and headroom. Five seeds draw nested subsets of 32/64/128/256/451 calibration chunks against the same 430 tests. Every subset has one exceedance at h=1.0h = 1.0 and none at 1.28. At 1.15, three 32-chunk subsets and one 64-chunk subset have one exceedance; larger subsets have none. At 1.28, capacities are 512/96/32 for size 32, 512/(96–112)/32 for 64–256, and 512/112/32 for 451. All 120 rows are independently reproduced from counts and frozen capacities. Reusing tests does not add independent coverage or establish that 32 chunks suffice generally.

Gemma Cost Model and Full-Model Tier Selection

Fitting and validation. Gemma uses 251 grid programs at T=512T = 512: recovery-enabled two-layer cuts, MXFP6 and four instrumented samples across two rounds, averaged per program before pooling. All 19 independent structures and 20 search candidates are excluded from fitting. Predictions span layer-1 rank sort through aggregation, not full-model latency. Table 16 shows optimistic errors for search-selected candidates; stage context and transition costs are not fully captured.

Validation programsMean abs.Max abs.Mean signed
19 independent structures3.99.5+1.0
20 search candidates10.916.8−10.9

Table 16. Gemma simulator errors relative to measured layer-1 MoE spans. All values are percentages; signed error is (predicted − measured) / measured.

Full-model validation. Table 3’s separate 30-layer mirrored session uses 60 prompts per pass after two warm-ups and a static-capacity anchor (maximum residual 0.28%). All tiered programs include recovery, fit the timing input and give identical logits. The primary comparison uses seed 0; seed 3’s 512/96/32 capacities take 280.6 ms with 7,072 rows per SoC, a separate sensitivity point. Two-layer and full-model ordering differs (§ 7), so layer-level scores require full-model validation.

References

  1. [1]Abts, D., Ross, J., Sparling, J., Wong-VanHaren, M., Baker, M., Hawkins, T., Bell, A., et al. ThinkFast: A tensor streaming processor (TSP) for accelerating deep learning workloads. In 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA), pp. 145–158, 2020. doi: 10.1109/ISCA45697.2020.00023.
  2. [2]Agrawal, A., Kedia, N., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B. S., Tumanov, A., and Ramjee, R. Tam-ing throughput-latency tradeoff in LLM inference with Sarathi-Serve. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI), pp. 117–134, 2024.arxiv.org/abs/2403.02310
  3. [3]Amazon Web Services. MoE inference architecture deep dive, AWS Neuron SDK 2.30.0. https://awsdocs-neuron.readthedocs-hosted.com/en/v2.30.0/libraries/nxd-inference/developer_guides/moe-arch-deep-dive.html, 2026a. Accessed September 30, 2026.
  4. [4]Amazon Web Services. Comparison of traced inference versus XLA lazy tensor inference, AWS Neuron SDK 2.30.0. https://awsdocs-neuron.readthedocs-hosted.com/en/v2.30.0/frameworks/torch/torch-neuronx/programming-guide/inference/trace-vs-xla-lazytensor.html, 2026b. Accessed October 24, 2026.
  5. [6]Benazir, A. and Lin, F. X. Efficient mixture-of-experts LLM inference with apple silicon NPUs. arXiv preprint arXiv:2604.18788, 2026. URL https://arxiv.org/abs/2604.18788.
  6. [7]Chatha, K. Qualcomm cloud AI 100: 12 TOPS/W scalable, high performance and low latency deep learning inference accelerator. In 2021 IEEE Hot Chips 33 Symposium (HCS), pp. 1–19, 2021. doi: 10.1109/HCS52781.2021.9567417.
  7. [8]Chen, M., Tworek, J., Jun, H., Yuan, Q., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.arxiv.org/abs/2107.03374
  8. [9]Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.arxiv.org/abs/2110.14168
  9. [10]Conneau, A., Rinott, R., Lample, G., Williams, A., Bowman, S., Schwenk, H., and Stoyanov, V. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2475–2485, 2018.arxiv.org/abs/1809.05053
  10. [11]Fedus, W., Zoph, B., and Shazeer, N. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022.
  11. [12]Gale, T., Narayanan, D., Young, C., and Zaharia, M. MegaBlocks: Efficient sparse training with mixture-of-experts. In Proceedings of Machine Learning and Systems (MLSys), volume 5, 2023.
  12. [13]Gemma Team. Gemma 4 technical report. arXiv preprint arXiv:2607.02770, 2026. URL https://arxiv.org/abs/2607.02770.
  13. [16]Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In International Conference on Learning Representations (ICLR), 2021.
  14. [17]Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., and Blunsom, P. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems (NeurIPS), volume 28, 2015.arxiv.org/abs/1506.03340
  15. [18]Hillis, W. D. and Steele, Jr., G. L. Data parallel algorithms. Communications of the ACM, 29(12):1170–1183, 1986.DOI
  16. [19]Huang, H., Ardalani, N., Sun, A., Ke, L., Lee, H.-H. S., Bhosale, S., Wu, C.-J., and Lee, B. Toward efficient inference for mixture of experts. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, 2024.DOI
  17. [20]Hwang, C., Cui, W., Xiong, Y., Yang, Z., Liu, Z., Hu, H., Wang, Z., Salas, R., Jose, J., Ram, P., Chau, J., Cheng, P., Yang, F., Yang, M., and Xiong, Y. Tutel: Adaptive mixture-of-experts at scale. In Proceedings of Machine Learning and Systems (MLSys), volume 5, 2023.arxiv.org/abs/2206.03382
  18. [21]Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations (ICLR), 2024.
  19. [22]Kossmann, F., Jia, Z., and Aiken, A. Optimizing mixture of experts using dynamic recompilations. arXiv preprint arXiv:2205.01848, 2022.arxiv.org/abs/2205.01848
  20. [23]Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP), pp. 611–626, 2023.arxiv.org/abs/2309.06180
  21. [24]Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., and Chen, Z. GShard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations (ICLR), 2021.
  22. [26]Muennighoff, N., Liu, Q., Zebaze, A., Zheng, Q., Hui, B., Zhuo, T. Y., Singh, S., Tang, X., von Werra, L., and Longpre, S. OctoPack: Instruction tuning code large language models. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2308.07124
  23. [28]Qualcomm Innovation Center. Efficient transformers library (QEfficient). https://github.com/qualcomm/efficient-transformers, 2024.
  24. [30]Rajbhandari, S., Li, C., Yao, Z., Zhang, M., Aminabadi, R. Y., Awan, A. A., Rasley, J., and He, Y. DeepSpeed-MoE: Advancing mixture-of-experts inference and training to power next-generation AI scale. In Proceedings of the 39th International Conference on Machine Learning (ICML), volume 162 of Proceedings of Machine Learning Research, pp. 18332–18346, 2022.arxiv.org/abs/2201.05596
  25. [31]Rouhani, B. D., Zhao, R., More, A., Hall, M., Khodamoradi, A., Deng, S., Choudhary, D., Cornea, M., Dellinger, E., Denolf, K., et al. Microscaling data formats for deep learning. arXiv preprint arXiv:2310.10537, 2023.arxiv.org/abs/2310.10537
  26. [33]Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations (ICLR), 2017.arxiv.org/abs/1701.06538
  27. [34]Shen, H., Roesch, J., Chen, Z., Chen, W., Wu, Y., Li, M., Sharma, V., Tatlock, Z., and Wang, Y. Nimble: Efficiently compiling dynamic neural networks for model inference. In Proceedings of Machine Learning and Systems (ML-Sys), volume 3, pp. 208–222, 2021.arxiv.org/abs/2006.03031
  28. [35]Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford Alpaca: An instruction-following LLaMA model. https://github.com/tatsu-lab/stanford_alpaca, 2023.
  29. [37]Yang, A. et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.arxiv.org/abs/2505.09388
  30. [38]Yao, J., Anthony, Q., Shafi, A., Subramoni, H., and Panda, D. K. Exploiting inter-layer expert affinity for accelerating mixture-of-experts model inference. arXiv preprint arXiv:2401.08383, 2024.arxiv.org/abs/2401.08383
  31. [40]Zhou, Y., Lei, T., Liu, H., Du, N., Huang, Y., Zhao, V., Dai, A., Chen, Z., Le, Q., and Laudon, J. Mixture-of-experts with expert choice routing. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, 2022.arxiv.org/abs/2202.09368
  32. [41]Zhu, K., Zhao, W., Zheng, Z., Guo, T., Zhao, P., Bai, J., Yang, J., Liu, X., Diao, L., and Lin, W. DISC: A dynamic shape compiler for machine learning workloads. In Proceedings of the 1st Workshop on Machine Learning and Systems (EuroMLSys), pp. 89–95, 2021.

Paper details

vlnc:W_rhq2jgb7
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

ChatGPT, Codex, Claude Code

View statement
Looking for related papers…
Contents