WorksComputer Science
RankLane: Efficient MoE Prefill on Accelerators with Ranked Expert Capacities
Abstract
Mixture-of-experts (MoE) models expand model capacity by activating only a subset of experts for each token. On accelerators with static execution plans, expert capacities and placement are fixed before routing reveals each expert's load. Uniform worst-case capacities waste work on padding; smaller capacities tied to expert identities can overflow. We present RankLane, which ranks each device's experts by current load and assigns them to fixed-capacity lanes. Runtime indices select experts and token rows while tier populations, shapes and device placement remain fixed. An aggregation graph preserves device ownership and reduces each token's local contributions on one core. Routing calibration and measured execution costs guide tier selection; overflowing chunks rerun at full capacity. We evaluate full-model prefill of Qwen3-30B-A3B and Gemma 4 26B-A4B with MXFP6 weights on four Cloud AI 100 Ultra SoCs, using 128-2048-token chunks and including overflow detection and in-program recovery support. Against a dense baseline, RankLane achieves end-to-end prefill speedups of 1.27-4.18x on Qwen3 and 1.93-3.46x on Gemma.
Introduction
Mixture-of-experts (MoE) models expand model capacity while activating only a subset of experts for each token ([33, 11]). Efficient prefill matters for serving these models because the prompt must be processed before generation begins. Chunked prefill bounds the tokens processed per invocation ([2]), but each expert’s load still varies with the input and layer. Specialized inference accelerators target high performance per watt and lower serving cost ([7 Amazon Web Services, n.d.]). Turning MoE’s sparse computation into efficient execution on these platforms requires accommodating this changing demand.
This becomes difficult when the accelerator executes a static plan that fixes tensor shapes, expert capacities and placement before routing. We study this setting on Qualcomm Cloud AI 100, where fixed-capacity lanes execute expert GEMMs (Figure 1(a)). GPU block-sparse kernels adapt their work to routed tokens ([12]); the evaluated graph interface instead executes expert GEMMs in slots with compiled capacities. Runtime indices select weights and tokens within these slots; GEMM shapes and dispatch remain fixed.
Figure 1. Static execution and changing expert demand. (a) The evaluated Cloud AI 100 graph path fixes lane capacities, shapes and placement. (b) For eight top-2 tokens on one device, uniform capacity reserves 32 rows for 16 assignments; smaller capacities tied to expert identities overflow when the busy experts change. (c) Ranking fits both illustrated chunks into the same 16-row lane layout, followed by local aggregation.
Two coupled challenges limit efficiency. First, capacities tied to expert identities must accommodate their changing peak loads. Qwen3-30B-A3B selects 8 of 128 experts per token ([37]): capacity per expert covers any routing of a -token chunk but reserves rows for assignments. Smaller fixed capacities risk overflow when the busy experts change (Figure 1(b)). Second, returning expert outputs to tokens can erase the savings. Dense aggregation retains large lane-indexed buffers, and compiler placement can redistribute outputs between devices and cores. The latency benefit of smaller expert capacities therefore depends on the storage and locality of the return path (§ 3).
We present RankLane, which assigns capacities by current load rank while preserving a static execution plan (Figure 1(c)). Our key observation is that the busiest experts change across inputs, while the measured load envelope falls sharply below the first ranks. Ranking each device’s experts and assigning fixed rank intervals to tiers gives the largest lanes to the currently busiest experts. Runtime gathers select their weights and token rows while tier populations, tensor shapes and expert-to-device placement remain fixed. To realize these capacity savings, token-owned aggregation gathers a token’s local expert outputs and sums them on one core within each device, limiting intermediate storage and communication.
The paper makes three contributions. First, we characterize the capacity savings available from runtime load ranks and show how aggregation governs their latency benefit (§ 3). Second, we design a compiled program that combines adaptive expert assignment with device-local, token-owned reduction (§ 4). Third, we integrate calibration and measured execution costs to select tiers, with overflow detection and capacity- reruns for loads beyond calibration. Together, these mechanisms preserve token–expert assignments and keep the normal path within the compiled program, without host dispatch between layers.
We evaluate full-model prefill of Qwen3-30B-A3B and Gemma 4 26B-A4B with MXFP6 weights and an on-device KV cache on four Cloud AI 100 Ultra SoCs. Matched ablations isolate the contributions of tiering and reduction, and held-out routing and recovery experiments test capacity coverage (§ 6). Across 128–2048-token chunks, RankLane includes overflow detection and in-program recovery support and achieves end-to-end prefill speedups over a dense baseline of 1.27–4.18× on Qwen3 and 1.93–3.46× on Gemma.
Background and Related Work
MoE Inference
An MoE layer replaces the feed-forward block with experts and a router. For a token with hidden state , the router selects the set of experts with the largest gate values and returns
with normalized gate weights . Qwen3-30B-A3B uses , , and an expert width of 768,
so an expert’s three matrices hold 9 MiB in FP16. Implementations group assignments by expert, run each expert’s GEMMs over its gathered token rows, and return the weighted rows to their tokens; under expert parallelism the experts are spread over devices and dispatch and combine become cross-device exchanges ([24, 30]). Training systems bound each expert’s work by a capacity factor and drop the tokens beyond it ([24, 11]); at inference, a dropped assignment changes the model’s output. Prefill processes the prompt in chunks of tokens and writes their keys and values to the KV cache; decode then generates one token per step ([23, 2]). A chunk gives each layer assignments, per SoC on average when four SoCs hold 32 experts each.
MoE Systems
MegaBlocks represents routed expert work as a runtime block-sparse topology and executes it with GPU kernels without dropping tokens ([12]). DeepSpeed-MoE, FasterMoE and Lina optimize or schedule the all-to-all exchange ([30] (He et al. 2022; Li et al. 2023)). The combine has been made sparse on GPUs too: Tutel replaces GShard’s dense dispatch and combine einsums with sparse encode and decode kernels that assign tokens to thread groups ([20]), and vLLM sums each token’s top- expert rows in a single kernel, with fixed-size buffers and device-side indices that suit CUDA graphs ((vLLM Project, 2026; NVIDIA, 2026)). A kernel author decides there which thread reads which rows and where partial sums live; a graph compiler decides that for the program, which is the gap § 3.2 examines. Placement studies exploit routing structure (Yao et al., [38]; Yu et al., 2025), Huang et al. [19] replace static capacity with dynamic gating, and expert-choice routing fixes each expert’s tokens by construction but changes the model (Zhou et al., [40]). On Trainium, expert work is packed into fixed-size blocks whose weight loads are shared across an expert’s blocks and whose unused blocks are skipped (Amazon Web Services, [3]). DynaMoE recompiles a static-shape program when capacities change during training (Kossmann et al., [22]); our full-model programs take 7–26 minutes to compile, so capacities cannot follow each chunk by recompilation.
NPUMoE targets Apple Silicon NPUs, assigning experts to static capacity tiers using offline popularity and performing routing and aggregation on the CPU (Benazir & Lin, [6]). RankLane assigns experts to tiers by current load rank and expresses assignment and aggregation within the compiled multi-device program.
Accelerators with Static Execution Plans
Groq statically schedules instructions and data movement (Abts et al., [1]); Neuron compiles fixed-shape graphs for traced inference on Inferentia and Trainium (Amazon Web Services, [4]). Compiler-managed graphs also appear in SambaNova’s RDU and Graphcore’s IPU. SambaFlow maps dataflow graphs onto RDUs and produces a compiled executable (SambaNova Systems, 2024); Poplar compiles static graphs with tensor dimensions determined before execution (Graphcore, n.d.). Compilation ahead of time does not itself require fixed expert capacities: runtime control flow and custom-kernel interfaces vary across platforms. RankLane targets the constraints in § 1; transfer requires cost and locality checks (Appendix § 10).
A Cloud AI 100 SoC has 16 AI cores and 136 GB/s of LPDDR4x bandwidth to up to 32 GB (Chatha, [7]; Qualcomm Technologies, n.d.); each core has a scalar VLIW processor, a vector unit, a tensor unit with 4K FP16 MACs and 8 MB of software-managed vector memory. The Ultra card carries four SoCs under a shared 150 W limit.
The compiler takes an ONNX graph and one shape specialization per chunk length and produces a static program, partitioned over the cores of a SoC and, by multi-device tensor slicing, over the SoCs. Matrix weights can be MXFP6 (Rouhani et al., [31]), dequantized by the vector unit while the tensor unit computes. A KV cache stays on device when each cache input can be paired with its updated output (retained state). The evaluated graph interface supports runtime weight gathers but rejects ONNX If with a non-constant condition. Dynamic-shape compilers for CPUs and GPUs provide run-time dispatch (Shen et al., [34]; Zhu et al., [41]); here, expert GEMMs execute in slots with compiled capacities.
Motivation
Under this execution model, expert capacity is allocated before routing reveals demand. We study this problem on Qwen3-30B-A3B compiled for the four SoCs of one Cloud AI 100 Ultra card (setup in § 6.1). The exported graph (Qualcomm Innovation Center, [28]) runs a layer’s 128 experts as two stages of 64 lanes: each lane executes one expert’s gate, up and down GEMMs over a fixed number of token rows, its capacity . Compilation fixes the lanes’ shapes, placement and schedule. Figure 1 connects this hardware setting to the capacity mismatch and the opportunity to adapt expert assignments within the compiled plan. We first examine the capacity this can save, then the aggregation needed to realize the saving.
Fixed Capacities for Runtime Load Ranks
A fixed assignment binds a lane’s capacity to one expert. Capacity accommodates any routing of a -token chunk, but reserves rows for Qwen3’s 8T assignments. Smaller capacities must cover each expert’s peak demand over the inputs the program will serve; average popularity alone does not determine that peak. Figure 1(b–Figure 1c) shows four experts with token counts in one chunk and in another. Fixed expert assignments require capacities to cover both. Assigning lanes by current load rank requires in either chunk: the expert needing the largest lane changes, while the lane capacities stay fixed.
We measure this opportunity using routing captures of all 48 layers for 50 prompts of 128 tokens from each of eight workloads, GSM8K, MMLU, SWE-bench Lite issues, HumanEval and its JavaScript version, Alpaca instructions, CNN/DailyMail articles and Chinese XNLI sentence pairs (Cobbe et al., [9]; Hendrycks et al., [16]; Jimenez et al., [21]; Chen et al., [8]; Muennighoff et al., [26]; Taori et al., [35]; Hermann et al., [17]; Conneau et al., [10]), and for chunks of 256 and 512 tokens from four of them (Appendix § 12). An expert in some layer receives 127 of 128, 256 of 256 or 508 of 512 tokens. Which experts are busy also varies: across workloads the 16 most loaded experts of a layer agree with a Jaccard index of 0.00–0.33, against 0.52–0.88 between two halves of one workload.
Figure 2 compares the capacity envelopes using device-captured routing. With each SoC’s 32 experts fixed in order of their calibration mean load, the held-out maximum is 90–126 of 128 rows at every position; covering this envelope reserves 93% of the rows of capacity- padding. With the same experts ordered by their load in the current chunk, ranks 17–32 receive at most 12 rows at . The measured rank envelope therefore supports much smaller capacities below the first ranks.
Figure 2. Expert-load envelopes from device routing captures. Each panel uses 200 calibration and 200 held-out chunks, with matched counts per workload across eight workloads. Calibration fixes the mean-load expert order; curves summarize held-out loads. Red: fixed order. Blue: runtime rank. Black: deployed RankLane capacities. Dotted: capacity .
Ranks also fix how many lanes need each capacity. Assigning an expert independently to a capacity bucket by its load can change the number of experts in that bucket. Assigning fixed ranges of ranks to tiers keeps each tier’s lane count constant: eight ranks always occupy eight lanes, whatever their expert identities. Ranking within each SoC’s expert set keeps expert computation on its assigned device. A sort and gathers can then select the weights and token rows for fixed-shape lanes. The small capacities depend on the measured load envelope; inputs outside it require overflow detection and recovery § 5.2.
The benefit also depends on executed padding. The compiler skips some empty work, so reserved capacity does not directly determine executed work § 5.1. Smaller capacities also change buffering and memory traffic. We therefore evaluate compiled latency as well as provisioned rows.
Aggregation Limits the Benefit of Smaller Capacities
After expert execution, the program must return each expert’s weighted outputs to their tokens and sum the contributions. Runtime assignment changes which expert occupies each lane, and tiers produce outputs with different shapes. Reducing padded expert work leaves this aggregation to be performed; its storage and communication costs determine how much of the saving becomes lower latency.
The exported combine scatters each stage’s rows into a zero-filled accumulator of shape , reduces over each SoC’s lanes and sums the four SoC partials. Each lane still reserves output rows in this accumulator even when its expert capacity is smaller. Tiers of different widths require separate accumulators, so dense aggregation need not become cheaper as expert padding shrinks.
Placement adds another cost. Although each expert stays on its assigned device, the compiled aggregation can redistribute its outputs across devices; within a device, a reduction spread over cores requires exchanges between them. Sparse aggregation must therefore also preserve locality. Section 5.4 measures these effects with equivalent aggregation expressions that produce different compiled placements.
The interaction is visible in full-model latency. At , replacing uniform capacity with ranked tiers lowers Qwen3’s latency by only 4% with dense aggregation, compared with 27% with token-owned aggregation. All four programs use the same runtime ranking, weight access and final cross-device sum (§ 6.4). The capacity savings are the same, but the aggregation determines the realized speedup.
These observations motivate two requirements: assign fixed lane capacities by current load rank, and return outputs to tokens with limited intermediate storage and communication. Section 4 develops the program structure that meets these requirements while retaining fixed shapes and placement.
RankLane
RankLane addresses both challenges of § 3: it assigns experts to fixed lanes by current load rank (§ 4.1), then gathers their outputs for token-local reduction (§ 5). Figure 3 shows the full pipeline within a fixed compiled graph. Calibration and measured costs select the tiers (§ 5.1); overflow recovery handles loads beyond calibration (§ 5.2). Appendix § 11 gives the Cloud AI 100 realization.
Figure 3. RankLane overview. Offline calibration and measured costs select fixed lane counts and capacities. At run time, each SoC (1) ranks its experts, (2) gathers weights and tokens into fixed-capacity lanes, and (3) combines outputs by token on local cores; (4) an elementwise sum combines device partials. After all layers, a host check triggers a capacity- rerun if any lane overflows.
Adapting Expert Assignment within Fixed Tiers
Assigning experts independently to capacity buckets would change each bucket’s population, violating the fixed lane count. RankLane instead assigns fixed intervals of load ranks to tiers: busy experts receive larger lanes while the number and shape of lanes remain constant.
Bind capacities to ranks. Device owns a fixed expert set , 32 experts per SoC here. The router assigns tokens to expert and to the device. Sorting these counts gives , with . Tier executes one stage of lanes per device at capacity , covering ranks , where . Lane serves expert . Each lane executes that expert’s gate, up and down GEMMs over its fixed row capacity.
Adapt through indices. The rank permutation selects each lane’s expert weights and token table (Figure 3, steps 1–2). The routing compacts these tables in expert order; gathers place the selected tokens into the fixed lane inputs. Tier counts, widths, capacities, tensor shapes, weight banks and expert-to-device placement are fixed at compilation. Only indices and data change within the compiled program. Every expert keeps one lane and its original tokens. If fits that lane’s capacity, the exact-arithmetic output is unchanged; Qwen3’s tested tiering variants are also bit-identical, although compilation can change summation order. A device provisions rows instead of .
Why ranks. The busiest experts each hold at least assignments, and together at most , so
averages here and reaches at most that in our Qwen3 256- and 512-token captures. Lower ranks therefore admit smaller capacities, whereas fixed expert identities can peak near (§ 3.1). Eq. (2) holds for any routing; our tighter empirical envelope is tested across eight workloads (Figure 2, § 7.1).
Returning Outputs through Local Token Aggregation
The dense return path still reserves rows per lane (§ 3.2). RankLane instead gathers each token’s contributions, at most , and sums them locally. The graph layout must preserve locality as well as remove the lane-dense accumulator.
Gather contributions by token. Let hold device ’s weighted expert outputs and row address the output of token ’s -th expert. Each device computes a token partial, followed by a sum across devices:
Figure 4 shows how the graph expression of this standard sparse combine determines device and core locality.
Figure 4. Aggregation placement on the evaluated compiler. (a) Global indices cause expert-output redistribution. (b) A SoC-batched gather preserves device locality, but lane-indexed reduction exchanges operands across cores. (c) RankLane also completes each local token sum on one core. All variants retain the final cross-SoC sum; §5.4 measures these placement effects.
Keep expert outputs on their device. Global indices and per-device slices cause our compiler to redistribute expert outputs (Figure 4(a)). RankLane exposes device ownership as the gather’s leading batch axis (Figure 4(b–Figure 4c)): tier outputs reshape from to , without moving data, where is the device count and the lanes per device. Batched indices select each token’s expert rows; contributions outside the device or tier are masked.
Sum each token’s contributions on one core. A lane-indexed reduction can stay on one device yet still exchange operands across cores (Figure 4(b)). RankLane arranges the gathered rows as and sums over into partials. The compiler splits this work by token tiles: each core gathers and sums its tokens’ contributions, without a cross-core reduction (Figure 4(c)). Section 5.4 measures both placement effects.
Combine device partials once. After accumulating all tiers’ local partials, the program sums per token tile (Figure 3, step 4). The compiler lowers this elementwise tree as a reduce-scatter with no root device; the next layer’s input normalization completes it. Algorithm 1 in Appendix B gives the complete path.
Each tier gathers rows per device, including masked slots, versus lane-dense rows. Only partials enter the final cross-device sum. Added tiers still pay for gathers and local partials, which tier selection must include.
Choosing Tiers for Lower Latency
Ranking and local aggregation make smaller capacities useful, but do not determine how many tiers to compile. More tiers follow the rank envelope more closely while adding expert stages, aggregation gathers and transitions. We choose their widths and capacities together (Figure 3, top): calibration bounds the observed demand at each rank, and measured execution costs determine which groups of ranks are worth separating.
Calibrating capacities. A configuration has tiers in rank order. Let be the largest load at rank over calibration chunks, layers and devices. Tier starts at row :
Here is headroom and the row granularity; each lane reserves at least one quantum. Sorted counts make the first rank the largest in its tier, so widths determine capacities. The deployed rule uses and (Appendix § 12). The main search uses to match its profiling grid; the other searches use , interpolating table entries where necessary. Headroom is empirical, so overflow still requires recovery (§4.4).
Account for executed padding. Cloud AI 100 can skip empty work, so provisioned rows alone do not predict latency. Capacities also affect buffering, weight loads and activation spills (Appendix § 13). We therefore use measured compiled execution costs to evaluate candidate tiers.
Include the return path in the cost. Width changes core placement and memory traffic as well as padding. For a stage of lanes per device at capacity , we measure its GEMM time and the gap to the next stage switch in complete two-layer programs with real routing and deployment settings. Let identify the -th tier in predicted compiler execution order. The estimated layer time is
covers sort-to-first-GEMM time; covers aggregation after the last GEMM. contains fitted corrections for the first transition, later-stage slowdown and activation spills.
The last stage pays no transition. Rank order sets capacities; execution order sets costs. We predict decreasing-width execution, breaking ties by rank order (Appendix § 15).
Searching structures. With experts per device, the search objective is
subject to
, from Eq. (4).
We score 27,466 structures for , and , then compile leading candidates to update spill corrections; unchecked candidates remain provisional (Algorithm 2, Appendix § 15). The objective excludes recovery and does not guarantee unseen loads fit. § 7 evaluates selection quality and offline cost.
Deployed configuration. Qwen3 uses 8/8/16 lanes per device across chunk lengths, varying only capacities (Table 1). It is among the search leaders at and 512; alternatives are close at (§5.5).
| Tiered rows | Uniform rows | ||
| 128 | 128/48/16 | 1,664 | 4,096 |
| 256 | 256/64/32 | 3,072 | 8,192 |
| 512 | 512/128/64 | 6,144 | 16,384 |
Table 1. Qwen3 tiers: 8/8/16 lanes per device. Provisioned rows are , compared with for uniform capacity- lanes.
Recovering from Capacity Overflow
Future inputs can exceed the calibrated envelope. RankLane reruns overflowing chunks with capacity , preserving smaller normal-path tiers while ensuring an undersized lane never produces an accepted result.
Detect and rerun the chunk. Detection reuses every lane’s token count; the host compares it with the lane’s capacity after each chunk (Figure 3, bottom). Recovery reruns that chunk at the same positions. Prefill writes the KV cache only at the chunk’s own positions, so the rerun overwrites every entry the failed attempt wrote, later chunks see only the rerun’s cache, and the result is that of an overflow-free run. Our recovery profile specializes capacities through the lengths of small row inputs, sharing the normal profile’s weights and cache.
Charge recovery to latency. For in-program recovery with overflow probability , a chunk costs on average: includes profile-support and host-check overhead, and includes the capacity- rerun and its check.
A separate program also pays activation and prefix replay (Appendix § 17).
Evaluation
We measure full-model performance and separate reduction, tiering and recovery contributions, then evaluate runtime assignment, capacity, locality, latency-guided selection, robustness and recovery cost.
Experimental Setup
We evaluate Qwen3-30B-A3B and the text path of Gemma 4 26B-A4B on one four-SoC Cloud AI 100 Ultra card, at batch 1 with MXFP6 weights. The dense baseline uses fixed expert assignments, static weights, capacity and dense aggregation; it shares expert-parallel execution, head-parallel attention and tree routing scans with RankLane. RankLane includes overflow detection and in-program recovery. We measure uninstrumented full-model prefill on GSM8K chunks, using 60 timed inferences per pass after warm-up. Pass medians are normalized across mirrored sessions through a shared reference program. Unless identified as Gemma, ablations use Qwen3. Appendix § 10 gives the configurations, input selection and fitting procedure.
End-to-End Performance
RankLane prefills 1.27–4.18× faster than the dense baseline on Qwen3 and 1.93–3.46× on Gemma (Figure 5). These gains include token-owned aggregation, ranked capacities, runtime assignment and recovery.
Figure 5. We measure batch-one prefill with MXFP6 weights on a four-SoC Cloud AI 100 Ultra card across 128–2048-token chunks. RankLane speeds up full-model prefill by 1.27–4.18× on Qwen3 and 1.93–3.46× on Gemma over the dense baseline.
Contribution of reduction and tiering. Figure 6 decomposes the end-to-end speedup: (1) token-owned reduction and its elementwise final sum, retaining static weights and capacity ; (2) runtime ranking, weight gathers and deployed tiers; and (3) overflow detection and in-program recovery. The tiering increment includes assignment and weight access; § 6.4 tests the interaction.

(a)

(b)
Figure 6. Under the setup of Figure 5, we successively add token-owned reduction, ranked tiers and recovery support to the dense baseline. Across –, reduction gives – on Qwen3 and – on Gemma over the dense baseline. Ranked tiers add a further – and –, respectively, over reduction alone, before recovery overhead.
At –, reduction gives – and – on Qwen3 and Gemma; ranked tiers add – and –, with – and – recovery overhead, respectively. At , Gemma’s ms sequence saves ms through reduction and ms through tiering, then adds ms for recovery, delivering end-to-end. At , Gemma’s recovery overhead grows to , reducing the delivered speedup to .
Runtime Expert-to-Tier Assignment
We test whether calibration can fix each expert’s tier for all inputs. Both alternatives in Table 2 use Qwen3’s deployed 8/8/16 tier widths and capacities, the same weight gathers, token-owned reduction and final sum. The compiled capacities stay fixed in both; the experiment changes how experts are assigned to them.
| Assignment | Rows/SoC | Attempt (ms) | Overflow chunks | |
| 128 | Calibration-fixed | 1,664 | 111.1 | 782/782 |
| Runtime-ranked | 1,664 | 105.4 | 0/782 | |
| 256 | Calibration-fixed | 3,072 | 145.3 | 609/609 |
| Runtime-ranked | 3,072 | 141.7 | 0/609 | |
| 512 | Calibration-fixed | 6,144 | 235.3 | 443/443 |
| Runtime-ranked | 6,144 | 225.5 | 0/443 |
Table 2. Qwen3 assignment at matched tier capacities. In held-out trace replay, fixed assignment overflows every chunk; runtime ranking overflows none. Attempt timings exclude recovery, required by all fixed-assignment timing runs.
Assignment policies. Calibration-fixed tiers freeze a separate expert order for each layer and SoC. We test orders based on mean, peak, 99th- and 99.9th-percentile calibration load and select among them using calibration alone; the selected order is the 99th-percentile policy at all three lengths. Runtime-ranked tiers (RankLane) instead sort the current routing counts and assign the busiest experts to the largest capacities.
Coverage at the same budget. We replay held-out device routing counts from 782/609/443 chunks at , after the exclusions in § 7.1. A chunk overflows if any lane in any layer exceeds its capacity. All four calibration-fixed policies overflow every chunk at the deployed capacities; runtime ranking overflows none of the 1,834 chunks. Both alternatives provision 59–63% fewer rows than uniform capacity . Among the tested policies, adapting expert identities to current loads is what makes these smaller budgets cover the held-out routing.
Interpreting latency. The fixed policy also overflows on each timing input, so its listed attempt must be followed by recovery before an output can be accepted. Table 2 excludes recovery support from both programs; the complete-system results in § 6.2 include detection and recovery support. Appendix § 18 gives the capacity sweep, supplied-order timing control and recovery-cost estimates; Appendix § 13 gives additional ranking and tier-capacity controls.
Token-Owned Aggregation
We test how aggregation affects the benefit of smaller expert capacities, then isolate the device and core locality choices profiling protocol; these traffic measurements explain the placement effects without decomposing full-model latency into additive phase costs.
Latency-Guided Tier Selection
Tier count alone is insufficient. In full-model Qwen3 controls, three tiers improve on two by 1.01× at and 1.20× at (Appendix § 13). The benefit varies because each tier changes execution and transition costs as well as padding. We therefore select widths and capacities using measured execution costs rather than minimizing provisioned rows alone.
Selecting among candidates. Table 8 in Appendix § 15 compares predicted and measured Qwen3 MoE spans, fixing calibration and headroom within each chunk length. At and 512, the deployed 8/8/16 structure is fastest among the 8 and 21 measured configurations following the respective capacity rules. Its predicted score is within 0.1% of the search minimum. At , the exact-fit search selects 8/16/8, measured at 2.35 ms versus 2.32 ms for the fastest of five tested candidates, a 1.3% gap. These single-sample profiles do not resolve close alternatives.
| Widths | Predicted | Measured | ||
| 128 | 1.28 | 8/8/16* | 2.04 | 1.94 |
| 16/16 | 2.09 | 1.98 | ||
| 256 | 1.00 | 8/16/8* | 2.38 | 2.35 |
| 9/7/16 | 2.43 | 2.32 | ||
| 8/8/16 | 2.48 | 2.39 | ||
| 512 | 1.28 | 8/8/16* | 3.38 | 3.32 |
| 6/6/4/16 | 3.59 | 3.33 | ||
| 8/8/8/8 | 3.96 | 4.03 | ||
| 16/16 | 5.12 | 5.67 |
Table 8. Qwen3 candidate selection: predicted and measured layer-1 MoE spans (ms) in instrumented two-layer cuts. Each group fixes calibration and ; * marks the selected candidate (: the deployed choice, within 0.1% of the minimum score). Capacities follow Eq. (4); The comparison excludes arbitrary capacity sweeps and the 12-layer cut.
Full-model selection benefit. Table 3 tests whether the cost model improves on Gemma’s manually selected, profiled four-tier structure. At , the search chooses 11/12/9 lanes with capacities 512/112/32 from the calibration split (). In an anchored session separate from Figure 5, the selected structure reduces full-model latency by 8.5% despite provisioning 18.2% more rows. It also reduces latency by 3.6% relative to 12/11/9 at 512/96/32, calibrated on the same split with . All three programs include overflow detection and in-program recovery; none overflows on the timing chunk, and their logits are bit-identical. The 11/12/9 widths are selected in all five calibration splits, with capacities 512/112/32 in four and 512/96/32 in one. Table 3 fixes the primary split; the latter is a separate sensitivity point (Appendix § 19.§ 3).
| Selection | Widths | Rows/SoC | ms |
| Manual | 8/8/8/8 | 6,144 | 309.9 |
| Alternative | 12/11/9 | 7,488 | 294.2 |
| Latency-guided | 11/12/9 | 7,264 | 283.5 |
Table 3. Gemma prefill at with recovery support. Latency-guided tiers cut latency by 8.5% versus manual tiers despite 18.2% more rows, showing that reserved capacity alone does not predict performance.
The selected 11/12/9 structure fits one replicated weight bank and avoids short 16-lane stages that use too few cores (Appendix § 19). We transfer these widths to the other chunk lengths and recalibrate capacities with ; this does not establish a separate width optimum at each length.
The model predicts 6.65 versus 6.83 ms per MoE layer for 11/12/9 and 12/11/9. Two-layer measurements reverse that order (7.91 versus 7.01 ms), while full-model timing favors the selected structure. This supports full-model validation of cost-guided candidates, without establishing a global optimum.
Prediction accuracy. Figure 8 compares predicted and measured Qwen3 MoE spans; the full-fit mean absolute percentage error is 4.3% across 48 programs. Excluding
Figure 8. Predicted versus measured Qwen3 MoE spans at for 48 programs, using the full fitted cost table. Predictions track the measured cost differences with 4.3% mean absolute percentage error; close candidates still require measurement to resolve their ordering.
each program’s own cost-table observations raises it to 4.9%; auxiliary terms still share the fitting pool. Gemma’s errors are 3.9% on 19 independent structures and 10.9% on 20 search-selected candidates, both excluded from fitting. All search-candidate errors are optimistic. The models predict MoE spans, not full-model latency; full-model validation remains necessary. Appendices § 15 and § 19.§ 3 give validation details.
Search cost and reuse. Qwen3’s cost table uses 256 grid programs, each taking about a minute to compile and a minute to profile, with the two phases overlapped. Gemma uses 251 programs with four samples each. Once the table is available, scoring all 27,466 Qwen3 structures takes seconds; spill checks take about a minute per candidate. The table can score new calibration profiles for the same model, precision and compiler settings. This concentrates device profiling on the cost grid and selected candidates, rather than every enumerated structure.
Capacity Robustness and Recovery Cost
Qwen3. Replaying held-out device routing for 782/609/443 chunks at across eight workloads gives no capacity exceedances (1,834 chunks total). The evaluation excludes calibration documents and HumanEval/JavaScript problem overlaps; fixed policies use the same filtered set without recalibration (Appendices § 12 and § 18).
Gemma. Executing the deployed 11/12/9 program on 430 held-out inputs at each of across eight workloads produces zero overflows. Tiered and capacity- profiles give identical logits on every tested input. Appendix § 19.§ 1 reports per-workload latency and output checks; Appendix § 19.§ 2 gives calibration-split, headroom and source-holdout studies.
Overflow recovery cost. With an injected host overflow signal, total chunk latency is 238.3/349.9/579.0 ms for Qwen3 and 248.1/376.2/674.3 ms for Gemma at : 2.1–2.5× a normal recovery-enabled chunk. These measurements include the rejected tiered attempt, both host checks and the in-program capacity- rerun. Normal-path overhead is already included in Figure 5 and Figure 6. Gemma additionally passes 180 deliberate actual-overflow recoveries on four-chunk prompts, with accepted outputs and continuation matching direct capacity- execution at the stressed chunk (Appendix § 17).
Conclusion
RankLane enables efficient MoE prefill on accelerators with static execution plans. It binds lane capacities to current load ranks and uses token-owned aggregation to preserve device and core locality. Calibration estimates capacity needs, latency-guided search accounts for the execution cost of tier boundaries, and overflow recovery handles loads beyond the calibrated envelope.
We evaluate full-model, batch-one prefill on Cloud AI 100, using MXFP6 weights and 128–2048-token chunks. Including overflow detection and in-program recovery support, RankLane achieves end-to-end prefill speedups over a dense baseline of 1.27–4.18× on Qwen3 and 1.93–3.46× on Gemma.
AI USAGE DISCLOSURE
We used OpenAI’s ChatGPT/Codex and Anthropic’s Claude Code during research and manuscript preparation. Their use was iterative and substantial, including research planning, implementation assistance, and drafting and revision of the manuscript.
Ideation and literature search. ChatGPT/Codex helped refine the problem framing, motivation, method presentation, and formulation of tier selection. Both ChatGPT/Codex and Claude Code were used for literature search, including identifying related systems and accelerator architectures, summarizing relevant material, and developing comparisons with prior work.
Experiment design and code. The tools helped propose and refine baselines, matched-budget comparisons, tiering and reduction ablations, calibration and held-out evaluation protocols, and overflow-detection and recovery experiments. ChatGPT/Codex also helped discuss candidate evaluation models and the interpretation and presentation of results. Both tools were used to generate and revise code for experimental workflows, including program construction, compilation, experiment orchestration, data processing, result analysis, and visualization. Codex additionally executed build, measurement, and analysis commands through its tool interface and assisted with debugging, reproduction instructions, and checks against experiment artifacts.
Data preparation and results. AI-assisted code was used for workload preparation, tokenization, calibration/test partitioning, and analysis of routing counts and performance logs. No study-specific dataset examples or benchmark labels were generated with these tools. Timing and routing results were obtained from program executions and recorded experiment artifacts; simulator predictions were computed by the cost-model implementation.
Writing and presentation. ChatGPT/Codex and Claude Code were used for writing and revision. ChatGPT/Codex assistance covered the abstract, introduction, motivation, method, evaluation, conclusion, and appendices. It provided suggested equations and pseudocode, reviewer-style feedback, and help with figures, tables, captions, LaTeX formatting, and the separate submission PDFs. The authors are responsible for the final technical claims, citations, experimental results, and manuscript.
Experimental Setup and Compilation
Hardware and models. One Cloud AI 100 Ultra card has four SoCs with 16 cores and 32 GB each, using SDK 1.21.6. We evaluate all 48 layers of Qwen3-30B-A3B and the full 30-layer text path of Gemma 4 26B-A4B-IT (Gemma Team, [13]), including embedding, final normalization and language-model head. Gemma selects eight of 128 experts (hidden/expert widths 2816/704), adds a parallel dense MLP, and alternates five sliding-attention layers with one global layer. Main results use batch 1, MXFP6 weights, FP16 activations and retained KV state; ablations use Qwen3 unless identified as Gemma.
Comparisons. The dense baseline uses fixed expert assignments, static weights, capacity and dense output aggregation. Both programs share expert-parallel execution, head-parallel attention and tree routing scans (Appendix § 11); Gemma also shares direct activation gathers and logit softcapping. RankLane adds token-owned aggregation, ranked tiers and in-program recovery. The detailed tier controls already use token-owned reduction and omit recovery support (Appendix § 13); they are distinct from the dense baseline.
Timing. Primary measurements use first-chunk GSM8K chat prefills, outside the calibration range at for Qwen3. Mirrored passes share a uniform-capacity, token-owned anchor; a program-plus-session fit of log latency separates program and session effects. Qwen3’s fit uses 74–83 passes, each the median of three 20-inference rounds after warm-up; maximum residuals are 0.63/0.88/1.7%. Dense and reduction-only controls use additional mirrored passes of 60 requests after two warm-ups, normalized through the same anchor. Longer-chunk Qwen3 sessions are in Appendix § 13. Full-model latencies are uninstrumented; phase times and traffic come from instrumented cuts. Numerical and multi-workload checks are reported separately.
Gemma uses held-out GSM8K chunks at all five lengths: four mirrored sessions, 24 passes and 60 timed requests after two warm-ups per pass. Fit residuals are 0.58/0.57/0.67/0.57/0.42%. Each length includes 240 injected recoveries; single-profile, recovery-enabled and recovered logits match exactly on the timing input.
Compilation. Qwen3 uses the command below with four-SoC partitioning and one specialization per chunk length (sequence , context ). All compared programs use -mdts-mos=1. Exact per-program commands are retained in the reproduction scripts.
qaic-compile -aic-hw -aic-hw-version=ai100
-retained-state -convert-to-fp16
-mxfp6-matmul -aic-num-cores=16 -mos=1
-aic-enable-depth-first
-mdp-load-partition-config=<4 SoCs>
-network-specialization-config=<T, 2T>
-mdts-mos=1
Gemma at . Both compared programs use unwrapped cache-update positions within the retained context (exported modulus ), preserving the sliding-attention mask and avoiding within-chunk overwrites beyond the 1024-token window. The embedding table is split into four column blocks with concatenated lookups to fit the compiler’s per-core address space at context 4096; this split preserves logits bit for bit. Capacities and coverage are in Appendix § 19.
Scope. RankLane targets fixed expert capacities and placement with costly padding. Weight reuse and scheduling policies (Amazon Web Services, [3]) can change the preferred tiers; transfer requires new cost measurements and locality checks. Our evidence covers two models, batch-one prefill and one compiler, without task-quality, decode, larger-batch or other-platform evaluation. Calibration cannot guarantee unseen loads fit; reference-routing checks and final-program executions have distinct coverage (Appendix § 19).
Realization on Cloud AI 100
RankLane transforms QEfficient ONNX exports (Qualcomm Innovation Center, [28]). Compiler cost measurements are in Appendices § 13 and § 15.
Expert Placement and Retained State
Both programs keep each expert’s three matrices on one SoC (-mdts-mos=1). Head-parallel attention uses a leading device batch axis to colocate cache inputs and outputs as retained state. The four attention partials are combined as per token tile. These choices are shared by the compared programs.
Routing and Aggregation
Routing assigns compact rows through Hillis–Steele scan steps (Hillis & Steele, [18]). The dense baseline scatters the first stage’s weighted outputs into a zero-filled accumulator; later stages gather, add and scatter, and the per-SoC lane sum uses 16 token tiles.
Algorithm 1 instead gathers each token’s selected expert rows with a leading device batch axis. Other-device or other-tier entries use a safe index and zero mask because the gather does not zero-fill invalid indices. The final elementwise sum lowers to a reduce-scatter completed by the next layer’s normalization. Appendix § 14 isolates device placement and
Algorithm 1 Ownership-preserving token aggregation for one tier $s$ ($D$ devices)
Require: top-$k$ experts $\mathcal{K}[t,1..k]$ of each token $t$; map $\ell(e) \mapsto (s,c,l)$ (tier, device, lane); slot table $\operatorname{slot}[e,t]$; tier outputs $O_s$ of shape $[DL_s,C_s,d]$, $L_s$ lanes per device
1: $\widehat{O}_s \leftarrow \operatorname{reshape}(O_s,[D,L_sC_s,d])$ {no data movement}
2: for all $t \le T$, $j \le k$, devices $c$ in parallel do
3: $e \leftarrow \mathcal{K}[t,j]$; $(s',c',l) \leftarrow \ell(e)$
4: $m[c,t,j] \leftarrow [s'=s \wedge c'=c]$; $\operatorname{idx}[c,t,j] \leftarrow m ? lC_s+\operatorname{slot}[e,t]:0$
5: end for
6: $G \leftarrow \operatorname{gather}(\widehat{O}_s,\operatorname{idx}) \odot m$ {batched over devices: $[D,Tk,d]$}
7: $P_s \leftarrow \sum_j G[\cdot,\cdot,j]$ {per token, on its core: $[D,T,d]$}
8: after all tiers: $P \leftarrow \sum_s P_s$; $y \leftarrow (P_0+P_1)+(P_2+P_3)$ {elementwise}
core-local summation.
Runtime Assignment and Weight Access
Each SoC sorts 32 counts with TopK; masks, scans and slot tables remain in native expert order and do not depend on sorting. Each tier’s weight gathers lower to indirect DMA from a constant containing all 128 experts. The compiler replicates this bank across SoCs although each computes only 32 experts; Qwen3 occupies about 22.5 GiB per SoC versus 6.7 GiB for static weights. Replication is a lowering property, not an algorithmic requirement. Gemma’s constraints are in Appendix § 19. Each chunk length is separately specialized, and every program returns every layer’s lane counts for overflow detection.
Routing Captures and Tier Calibration
The original capacity-calibration captures run the model’s FP32 layers on the CPU and record every layer’s top-8 experts. At we use 50 chat prompts per workload; at 512 tokens, 8 GSM8K chunks and 4 each from HumanEval, MMLU and SWE-bench Lite; the 256-token calibration set holds 16 GSM8K chunks and both halves of each 512-token chunk (56 in total). The held-out GSM8K chunks of the evaluation use test questions 1000–1003 (256 tokens) and 1004–1011 (512 tokens), outside the calibration range. Table 4 gives the largest per-SoC load at selected ranks, the input to the tier rule of § 5.1.
| 2 | 4 | 8 | 9 | 16 | 17 | 32 | ||
| 128 | 127 | 122 | 105 | 33 | 32 | 14 | 12 | 3 |
| 256 | 256 | 223 | 134 | 58 | 50 | 25 | 25 | 4 |
| 512 | 508 | 424 | 240 | 101 | 89 | 51 | 50 | 10 |
Table 4. Largest load of the rank- expert of a SoC over all calibration chunks, layers and SoCs.
Figure 2 instead uses the device-routing captures audited below. Each panel has 200 calibration and 200 held-out chunks from the same eight workloads. Documents are split before chunking; evaluation applies the exclusions below.
We match each workload’s count across both lengths and both splits, allocating as evenly as availability permits, then sample without replacement with seed 20261030, independently of routing loads. Calibration fixes one mean-load expert order per layer and SoC; both assignment policies use the same held-out chunks. The capacity lines retain the original calibration above. The diagnostic is a subset of the audited data; coverage results use the full retained set.
Evaluation-set audit. We reconstruct all 436 original calibration prompts from source documents and verify their token IDs against the saved captures. Of the original 805/618/449 device-evaluation chunks, 23/5/4 contain a document used in capacity calibration. Excluding these leaves 782/613/445; additionally grouping HumanEval problems with their JavaScript translations leaves 782/609/443 (1,834 total). The latter set is used throughout the main-text Qwen3 overflow comparison. It is also disjoint from the fixed policies’ calibration documents under these groups. No retained chunk exceeds the deployed capacities. The audit checks source-document identity and known translations; it does not establish exhaustive semantic deduplication or a prospective test corpus.
Tier Assignment and Compiler Effects
Table 5 fixes token-owned reduction and gathered weights. B and C use capacity ; D uses 16/16 lanes at and ; E uses the deployed 8/8/16 tiers. Programs within each column share a compilation configuration. All give bit-identical logits on the tested chunks.
| Qwen3-30B-A3B | |||
| B gathered, fixed order, | — | 160.2 | 302.4 |
| C gathered, ranked, | 115.6 | 163.3 | 308.3 |
| D ranked, two tiers | 106.4 | 138.1 | 269.8 |
| E RankLane, three tiers | 105.1 | — | 224.8 |
| C/E speedup | 1.10× | — | 1.37× |
| Rows, B–C | 4,096 | 8,192 | 16,384 |
| Rows, D | 2,304 | 4,608 | 9,216 |
| Rows, E | 1,664 | 3,072 | 6,144 |
Table 5. Tier-assignment ablation: full-model latency (ms per chunk), with token-owned reduction in every program. B→C adds ranking; C→D/E adds tiers. Dashes omit historical variants outside the common configuration. All tiered programs here are single-profile; recovery is included in Figure 6. Row budgets are per SoC.
Ranking and capacity controls. With uniform capacity and gathered weights, B→C adds ranking and increases latency by 2.0/1.9% at . C→E retains ranking and weight gathers while introducing tiers, giving 1.10/1.37× speedups at . These are whole-program latency differences.
Executed padding. Instrumented layer-1 profiles of two-layer cuts, with an einsum attention head-group sum, show that Where gates activation gatherers, GEMMs and SiLU. B/C execute 6.9–7.7k rows per SoC, versus 3.3–3.5k for E. Thus, executed rows differ from reserved capacity even in the uniform-capacity programs. These trace-derived observations explain work differences but are not an additive decomposition of full-model latency. The search also checks activation spills because buffer placement can add DRAM traffic (Appendix § 15).
Longer chunks. Qwen3 at uses one held-out GSM8K chunk and a cache per length. Three anchored sessions (22 passes; residuals 0.21/0.94%) establish ranked-tier timing; three further mirrored sessions normalize dense/reduction controls through a shared reduction anchor and recovery through single-profile RankLane (residuals 0.16/0.37%). Capacities , , are extrapolated, not recalibrated; neither timing chunk overflows.
All programs share a 150 W card limit rather than fixed clocks. Median clocks are 1018–1100 MHz through ; at 2048, reduction-only runs at 912 MHz, single-profile RankLane at 995 MHz and recovery-enabled RankLane at 984 MHz. The dense and reduction controls can throttle at the longer lengths.
Aggregation Formulations and Phase Profiles
These controls fix capacity , static weights and the final cross-device sum. The reference uses SoC-batched, token-owned aggregation; Figure 9 tests device placement and Table 6 tests core locality.
Figure 9. Equivalent arithmetic, different placement. Three expressions of the same per-SoC token aggregation, with bit-identical logits at uniform capacity with static weights. Left: full-model latency relative to the layout-preserving SoC axis. Right: cross-SoC traffic inside one MoE layer at , from instrumented two-layer profiles.
| Lane-indexed | Token-owned | |
| Full-model latency (ms) | 401.9 | 349.3 |
| Cross-SoC traffic (MiB) | 12.1 | 14.0 |
| Local-sum inter-core traffic (MiB) | 241.9 | 1.9 |
| Local-sum busy time (core-ms) | 47.1 | 2.4 |
Table 6. Local-reduction ablation at , uniform capacity and static weights. Full-model latency is uninstrumented; traffic and local-sum busy time are from layer 1 of instrumented two-layer cuts, summed across SoCs.
Device ownership. Figure 9 compares global row indices, explicit SoC slices and the SoC batch axis while holding the selected rows, summation order and final sum fixed. All three produce bit-identical logits. Global indices and explicit slices make the compiler redistribute expert outputs: 1539 MiB per layer against 14 MiB for the SoC batch axis at . Their full-model latencies are respectively 4.4–5.3× and 3.4–4.1× higher across chunk lengths. This isolates the graph expression; redistribution can also change scheduling, buffering and tiling.
Token-local reduction. The lane-indexed formulation preserves device locality but reduces each token tile over lanes distributed across 16 cores. It is 11–15% slower across chunk lengths. Table 6 shows why device locality alone is insufficient: cross-SoC traffic is similar, but the local sum moves far more data between cores. Token-owned reduction gathers and sums each token’s expert rows on one core. These two formulations differ in FP16 summation order; the device-ownership variants above preserve the order exactly, while this comparison permits FP16 rounding differences.
Both forms gather similar amounts of data (898/924 MiB), but lane-indexed sums pull operands from 16 cores. Instrumented phase gaps exceed uninstrumented per-layer gaps and cannot be added to reconstruct full-model latency. Figure 7’s dense control uses one accumulator per tier width, writes the first stage without reading zeros, and sums lanes in 16 token tiles (§ 6.4).
Figure 7. Qwen3 full-model prefill under uniform and ranked capacities. Token-owned aggregation enables 9–27% latency reductions from tiering; dense aggregation yields a 37% slowdown at and only a 4% reduction at .
Tier Search: Cost Tables and Validation
Cost Tables and Fitting
Figure 10 summarizes 256 programs at , covering lanes and rows. Each combines one stage with 16-row well for the other experts. Layer 1 of a two-layer cut is measured on one prompt using the third of five iterations: one instrumented sample per cell. Stage time is the mean first-to-last-GEMM span over SoCs; transition time is the mean gap to the next stage.
Figure 10. Qwen3 cost table at : (a) stage GEMM span and (b) transition gap, measured on layer 1 of instrumented two-layer cuts; boxes mark deployed tiers. Figure 8 compares predicted and measured MoE spans.
Equation (5) adds sort-to-first-GEMM work, first-transition and final-aggregation corrections, and a spill correction. The first-transition and aggregation terms are least-squares fits over capacity sweeps and structures with known boundary stages; the spill term uses five spilling four-/five-tier programs. Copy counts of at least 60 trigger the additional spill correction, except when the cells already include it (14-/16-lane first stages at capacity 512). Stages are ordered by decreasing width, ties in rank order; this matches 44 of 48 validation programs. A compile check is needed because small capacity changes can alter spilling, adding 0.5–0.9 ms per layer.
Validation Protocol
The 48-program set includes capacity sweeps, 37 structures with two to five tiers, and a 12-layer cut. Measurements use layer 1 of two-layer cuts, except the labeled deployed structure, which uses the median of 11 layers in the 12-layer cut. In Figure 8, shading marks ±10% error and hollow markers indicate spills. The measurement spans rank-sort start to final aggregation; starting at routing would include hoisted work overlapping attention. Of the programs, 31 were predicted before execution and 17 supplied cost-table observations. Figure 8 in the main text shows the full-fit predictions. Mean absolute percentage error is 4.3%, or 4.9% when each program’s own cell observations are excluded. Auxiliary-term fitting still uses this pool, so the latter is not a fully independent fit. Stage context and single-sample variability limit fine ranking: the same stage takes 1.98–2.93 ms across four programs.
Enumeration and Calibration
Algorithm 2 scores candidates, compiles a leading batch for spill checks, then rescores. Cell costs interpolate linearly in capacity at fixed width, using the nearest endpoint outside the measured range. Unchecked candidates omit only the additional spill term; the returned ranking retains unknown spill statuses. Enumeration needs no device execution; the compile-check batch defaults to ten candidates and can be repeated.
Algorithm 2 Tier enumeration and a batch of compile-only spill checks
Require: rank maxima ; , , , , , ; measured tables and fitted terms; cached copy counts ; compile-check budget
for each ordered width tuple satisfying the width constraints in Eq. (6) do Derive by Eq. (4); form
Set by decreasing width, ties in rank order
Score by Eq. (5), using when available; otherwise omit the additional spill term
Add , its score and spill-check status to
end for
for up to lowest-scoring unchecked candidates in do Compile a two-layer candidate with deployment settings
Cache its layer-input copy count in ; classify a spill at 60 or more copies
end for
Rescore using , without double-counting spills already represented in the stage cells
return sorted by predicted latency, retaining unknown spill statuses
| Maxima | Deployed | |||
| 128 | 127/30/12 | 128/48/16 | 128/48/16 | 128/32/16 |
| 256 | 254/56/26 | 256/64/32 | 256/80/48 | 256/64/32 |
| 512 | 509/98/44 | 512/128/64 | 512/128/64 | 512/112/48 |
Table 7. Qwen3 search inputs: maxima at ranks 1/9/17 from expanded captures (401/400/252 chunks). Deployed capacities use the original captures in Appendix C. Headroom rules define separate comparisons.
The tables use their respective prompts, refitting sort-to-first-GEMM and final aggregation while retaining the first-gap and spill terms. Their 10/15 validation programs have mean absolute errors of 3.2/4.1%, including exact-fit and fixed-headroom variants. Table 8 compares only candidates under the same rule: eight at , five at 256 and 21 at 512. Arbitrary capacity sweeps and the 12-layer result are excluded from those sets. All 27,466 structures are scored, but selection quality is relative to the measured subset, not a global optimum. At , the model identifies the slower two-tier structure, but 6/6/4/16 nearly matches the deployed structure despite its higher predicted cost. Single-sample profiles do not resolve these close alternatives.
Qwen3 numerical checks
Table 9 checks the programs of Figure 5 against FP32 CPU inference on the timing chunks. These last-position checks do not establish task-level quality.
| Program | |||
| Dense baseline | 0.199 ✓ | 0.072 ✓ | 0.087 ✓ |
| RankLane | 0.188 ✓ | 0.091 × | 0.096 ✓ |
Table 9. Relative error of last-position logits against FP32; ✓ indicates the same next token.
The tier-assignment controls produce bit-identical logits on these chunks. Changes to aggregation or compiler placement can change FP16 summation order; at the leading two FP32 logits are 0.58 apart. No lane exceeds its capacity. Recovery-enabled and recovered outputs match single-profile RankLane bit for bit (Appendix § 17).
At , the evaluated Qwen3 programs’ next token and top five agree with FP32. Across those programs, last-position KL divergence is 0.0039–0.0048 and 0.0022–0.0031, respectively. Recovery-enabled, recovered and single-profile RankLane give identical logits. These remain single-chunk checks.
Overflow recovery protocol and validation
Primary cost. Using Appendix § 10’s timing protocol, we inject a host overflow signal after an otherwise fitting tiered attempt, then check a capacity- rerun. Latency includes the rejected attempt, both checks and rerun; it measures the recovery path, not deployment overflow frequency. Recovery-cost passes use 60 timed requests after two warm-ups; Gemma repeats the mirrored schedule in two sessions. Normal-path overhead includes compiler profile effects as well as host checks.
Shared state. Shape-specialized row indices, formed from len(cap_rows), let normal and capacity- profiles share weights and KV state. The host checks every lane count after each attempt. A rerun overwrites the rejected chunk’s positions; later chunks return to normal tiers. Recovery-enabled and recovered outputs match single-profile RankLane exactly on both models’ timing inputs.
| Model / | 128 | 256 | 512 | 1024 | 2048 |
| Qwen3 | 127.4 | 202.2 | 344.4 | 1065.9 | 2285.2 |
| Gemma | 140.1 | 214.7 | 390.6 | 1197.6 | 2476.6 |
Table 10. Additional total latency when recovery is invoked (ms), including the capacity- rerun and host checks. Adding it to the normal-path latency in Figure 5 gives the recovered-chunk cost in §5.6, up to rounding.
Qwen3 continuation. Final 48-layer programs process one four-chunk GSM8K prompt per with a cache, mirrored passes and ten repetitions. Injecting overflow at chunk 0, 1 or 3 preserves every accepted output, including later chunks, bit for bit against static-capacity execution. Peak sampled additional allocation is 22.59–22.72 GiB per SoC relative to pre-load state. A separate resident fallback requires activation and prefix replay, with 9.26–9.38 s switching in these tests; the deployed path shares state instead. A two-layer cut also provokes actual overflow in 28 (layer, lane) pairs and recovers all four chunks exactly. Full-model Qwen3 cost measurements use injected signals.
Gemma actual overflow. Three math/coding/news inputs run as four chunks through all 30 layers. Besides normal 256/64/16 and capacity- profiles, an undersized 256/16/4 profile forces actual overflow at chunk 0, 1 or 3. Every stressed attempt is rejected and rerun, without injected signals. Ten repetitions per input in each mirrored pass give 180 timed recoveries. All accepted outputs and continuation match a reference that uses capacity directly at the stressed chunk and normal tiers elsewhere. This checks retained-state recovery; undersized profiles do not estimate deployment overflow probability, and the tests do not cover decode or broad continuing-prefix distributions.
Table 11. Gemma actual-overflow validation at : median four-chunk prompt latency (ms), pooled over three inputs and two mirrored passes. The direct reference uses capacity at the specified chunk, with no failed attempt. The three-profile normal schedule takes 659.4 ms; these stress timings are separate from the deployed two-profile results.
| Stressed chunk | Direct reference | With actual overflow |
| 0 | 710.5 | 863.7 |
| 1 | 709.3 | 859.5 |
| 3 | 706.8 | 861.7 |
Table 11.
Fixed assignments at matched capacity budgets
Policies and capacity search. We reuse the device counts of § 7.1: 782/608/441 calibration chunks and 782/609/443 evaluation chunks at , after the exclusions in Appendix § 12. Policy orders and capacity choices remain frozen from the original calibration split. Each fixed policy orders experts separately per layer and SoC by mean, peak, 99th- or 99.9th-percentile calibration load; ties use mean load. At each capacity pair, calibration-best selects the policy with the fewest overflowing calibration chunks, breaking ties by excess assignments. The search fixes tier widths at 8/8/16 and the first capacity at , and enumerates in 16-row steps. Neither policy nor capacity selection uses evaluation overflow.
At the deployed capacities, calibration-best selects the 99th-percentile policy at all three lengths. All four fixed policies also overflow every evaluation chunk at every other capacity pair within those row budgets. When minimizing rows subject to zero calibration overflow, calibration-best instead selects peak order. Its smaller budgets sacrifice evaluation coverage, especially at (Table 12). These comparisons cover the four tested calibration policies, not all possible fixed assignments.
| Same capacities as deployed RankLane | |||
| Rows | 1,664 | 3,072 | 6,144 |
| Runtime rank, overflow | 0% | 0% | 0% |
| Each of four fixed policies | 100% | 100% | 100% |
| Capacities selected for zero calibration overflow | |||
| Calibration-best, rows | 3,840 | 7,040 | 13,056 |
| (evaluation overflow) | (1.4%) | (2.1%) | (15.8%) |
| Fixed 99th, rows | 4,096 | 7,680 | 15,488 |
| (evaluation overflow) | (0.0%) | (0.3%) | (0.5%) |
| Uniform capacity , rows | 4,096 | 8,192 | 16,384 |
Table 12.
Device controls. The fixed-99th and supplied-order programs share RankLane’s tier capacities, weight-bank gathers, aggregation and final sum. Both remove runtime ranking while retaining runtime-valued gather indices. The supplied order comes from the timing chunk’s FP32 routing and produces bit-identical logits to RankLane, but disagrees with the MXFP6 device order, including strictly busier experts placed below tier boundaries. The latency difference includes placement effects and does not isolate sorting.
The supplied-order program takes 103.1/139.8/229.0 ms at , against the 105.4/141.7/225.5 ms ranked control in Table 2. RankLane differs by +2.3%, +1.3% and −1.5%, respectively. These are whole-program differences that include tier-placement effects, rather than an isolated measurement of sorting.
Table 12. Fixed assignment versus runtime ranking. Overflow is the percentage of evaluation chunks with any lane over capacity. Rows are per SoC. The lower panel reports minimum row budgets with zero calibration overflow and their evaluation overflow in parentheses. The evaluation-set qualification in § 7.1 applies.
Timing and recovery estimate. Mirrored anchor/control passes extend the log-latency fit, with maximum residuals 0.50/0.88/1.69%. In this expanded fit, a separately optimized uniform-capacity recovery program takes ms at . It uses fixed expert assignments, static weights and token-owned reduction; capacity guarantees no overflow.
Fixed-99th overflows on the timing inputs and is charged as a failed attempt. The estimate combines these single-input times with test-set overflow probability ( for fixed-99th, 0 for RankLane). It gives 225.0/314.3/523.6 ms for fixed-99th, or 2.13–2.32× RankLane’s single-profile latency. It excludes host checks, switching and prefix replay and is not a measured recovery sequence or a corpus-wide latency average. Appendix § 17’s measured recovery paths concern RankLane only.
Gemma 4: Calibration and Validation
Model, compilation and timing settings are consolidated in Appendix Appendix § 10.
Tiers and capacity coverage. The delivered program uses 11/12/9 lanes per SoC, selected at and transferred to other lengths. Capacities are calibrated separately with and (Table 13), using uniform-capacity device routing from 449/449/451/232/126 calibration chunks. The corresponding held-out sets contain 430/430/430/224/120 chunks, with no offline exceedances. Splits retain source-document groups and exclude timing-prompt documents; longer inputs concatenate groups within their workload and split. These are retrospective partitions. Direct final-program coverage at the first three lengths is reported below; longer-length checks cover reference routing and the timing inputs, not corpus-wide tiered execution.
| Tiered rows | Uniform rows | ||
| 128 | 128/32/16 | 1,936 | 4,096 |
| 256 | 256/64/16 | 3,728 | 8,192 |
| 512 | 512/112/32 | 7,264 | 16,384 |
| 1024 | 1024/192/64 | 14,144 | 32,768 |
| 2048 | 2048/384/128 | 28,288 | 65,536 |
Table 13. Table 13. Delivered Gemma capacities for 11/12/9 lanes. Rows are provisioned per SoC; uniform capacity reserves .
Width and memory constraints. Two-layer probes show that a 16-lane stage with 16 or 32 rows uses only four cores per SoC, serializing expert work. Qwen3’s 8/8/16 layout also induces duplicate replicated MXFP6 banks, projected to require about 36 GiB per SoC. The selected 11/12/9 structure fits with one replicated bank. Its benefit over a manually selected four-tier layout is measured in §5.5. Transferring the selected widths does not ensure the best latency at every length: at , the earlier four-tier complete system took 151.8 ms versus 161.6 ms for the frozen transferred layout, in separate anchored fits.
Final-Program Corpus Validation
All 430 held-out inputs run at each of on the selected-layout program, its capacity- profile and Dense baseline: twice per input in each of two mirrored passes, after two warm-ups, starting at position zero. None overflows; repeated outputs match exactly. Table 14 gives per-workload ratios of median request latency including checks, separate from the primary single-input fit. Repetitions and lengths reuse documents.
| Workload | |||
| GSM8K | 1.91× | 2.23× | 3.45× |
| HumanEval | 1.94× | 2.25× | 3.46× |
| JavaScript | 1.93× | 2.25× | 3.48× |
| MMLU | 1.91× | 2.23× | 3.45× |
| SWE-bench | 1.93× | 2.24× | 3.45× |
| Alpaca | 1.87× | 2.18× | 3.37× |
| CNN/DailyMail | 1.91× | 2.23× | 3.44× |
| Chinese XNLI | 2.01× | 2.28× | 3.44× |
Table 14. Table 14. Final Gemma program: per-workload dense-to-complete speedup at 11/12/9 widths. Each entry uses all held-out inputs of that workload at the given length; zero actual overflows occur.
Tiered and capacity- profiles give identical logits on every input. Dense and tiered programs agree on the next token for 384/430, 378/430 and 369/430 inputs (89.3/87.9/85.8%); maximum relative logit differences are 0.356/0.463/0.731. These differences occur without overflow. The comparison establishes neither their cause nor task accuracy or numerical equivalence of the dense and tiered graphs.
Device-Routing Calibration at 512 Tokens
Capture and splits. The 30-layer MXFP6 static-capacity program captures 916 first-chunk prefills across eight workloads, with 8 assignments per layer and no capacity truncation. Five seeds split six workloads into random chunk halves and HumanEval/JavaScript into paired problem halves, keeping document groups disjoint. Tests exclude calibration near-duplicates and timing/cost-table documents, leaving 451 calibration chunks and 430/429/432/427/431 test chunks. These are retrospective resplits of the earlier document pool.
Coverage and headroom. With 11/12/9 widths, fixed gives zero overflows in every mixed split. Calibration-only cross-validation (CV) selects in the mixed splits and –1.05 in the source holdouts. It gives one overflow per mixed split and one across the source holdouts. All failures across mixed splits and source holdouts concern the same HumanEval chunk, whose rank-12 load reaches 84 rows against capacity 80. On the primary split, fixed headroom raises provisioned rows from 6,880 to 7,264 (5.6%) and covers this outlier. Zero calibration overflow alone is therefore insufficient for held-out coverage.
Source holdouts. Table 15 holds out each of seven sources, pairing HumanEval with its translation. After excluding three timing-prompt chunks, 913 chunks appear once as held-out tests. Calibration alone freezes capacities and headroom; these holdouts reuse the mixed-split document pool and do not constitute independent trials alongside those splits.
| Held-out source | CV headroom | Fixed |
| GSM8K | 0/149 | 0/149 |
| HumanEval + JavaScript | 1/81 | 0/81 |
| MMLU | 0/400 | 0/400 |
| SWE-bench | 0/153 | 0/153 |
| Alpaca | 0/14 | 0/14 |
| CNN/DailyMail | 0/84 | 0/84 |
| Chinese XNLI | 0/32 | 0/32 |
| Total | 1/913 | 0/913 |
Table 15. Gemma , 11/12/9 widths: offline overflow chunks / tested chunks under source-held-out calibration. CV chooses headroom using calibration only.
Interpretation. Offline overflow is a lane exceeding capacity under reference-program routing, not a recovery event. Uniform-capacity and tiered routing differ by up to six rows at a rank on the reference chunk; that check does not bound other inputs. Final-program tests cover the primary split, not every holdout configuration. Calibration coverage is limited to , empty KV state and MXFP6, with small Alpaca/Chinese samples.
Calibration size and headroom. Five seeds draw nested subsets of 32/64/128/256/451 calibration chunks against the same 430 tests. Every subset has one exceedance at and none at 1.28. At 1.15, three 32-chunk subsets and one 64-chunk subset have one exceedance; larger subsets have none. At 1.28, capacities are 512/96/32 for size 32, 512/(96–112)/32 for 64–256, and 512/112/32 for 451. All 120 rows are independently reproduced from counts and frozen capacities. Reusing tests does not add independent coverage or establish that 32 chunks suffice generally.
Gemma Cost Model and Full-Model Tier Selection
Fitting and validation. Gemma uses 251 grid programs at : recovery-enabled two-layer cuts, MXFP6 and four instrumented samples across two rounds, averaged per program before pooling. All 19 independent structures and 20 search candidates are excluded from fitting. Predictions span layer-1 rank sort through aggregation, not full-model latency. Table 16 shows optimistic errors for search-selected candidates; stage context and transition costs are not fully captured.
| Validation programs | Mean abs. | Max abs. | Mean signed |
| 19 independent structures | 3.9 | 9.5 | +1.0 |
| 20 search candidates | 10.9 | 16.8 | −10.9 |
Table 16. Gemma simulator errors relative to measured layer-1 MoE spans. All values are percentages; signed error is (predicted − measured) / measured.
Full-model validation. Table 3’s separate 30-layer mirrored session uses 60 prompts per pass after two warm-ups and a static-capacity anchor (maximum residual 0.28%). All tiered programs include recovery, fit the timing input and give identical logits. The primary comparison uses seed 0; seed 3’s 512/96/32 capacities take 280.6 ms with 7,072 rows per SoC, a separate sensitivity point. Two-layer and full-model ordering differs (§ 7), so layer-level scores require full-model validation.
References
- [1]Abts, D., Ross, J., Sparling, J., Wong-VanHaren, M., Baker, M., Hawkins, T., Bell, A., et al. ThinkFast: A tensor streaming processor (TSP) for accelerating deep learning workloads. In 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA), pp. 145–158, 2020. doi: 10.1109/ISCA45697.2020.00023.
- [2]Agrawal, A., Kedia, N., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B. S., Tumanov, A., and Ramjee, R. Tam-ing throughput-latency tradeoff in LLM inference with Sarathi-Serve. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI), pp. 117–134, 2024.arxiv.org/abs/2403.02310
- [3]Amazon Web Services. MoE inference architecture deep dive, AWS Neuron SDK 2.30.0. https://awsdocs-neuron.readthedocs-hosted.com/en/v2.30.0/libraries/nxd-inference/developer_guides/moe-arch-deep-dive.html, 2026a. Accessed September 30, 2026.
- [4]Amazon Web Services. Comparison of traced inference versus XLA lazy tensor inference, AWS Neuron SDK 2.30.0. https://awsdocs-neuron.readthedocs-hosted.com/en/v2.30.0/frameworks/torch/torch-neuronx/programming-guide/inference/trace-vs-xla-lazytensor.html, 2026b. Accessed October 24, 2026.
- [6]Benazir, A. and Lin, F. X. Efficient mixture-of-experts LLM inference with apple silicon NPUs. arXiv preprint arXiv:2604.18788, 2026. URL https://arxiv.org/abs/2604.18788.
- [7]Chatha, K. Qualcomm cloud AI 100: 12 TOPS/W scalable, high performance and low latency deep learning inference accelerator. In 2021 IEEE Hot Chips 33 Symposium (HCS), pp. 1–19, 2021. doi: 10.1109/HCS52781.2021.9567417.
- [8]Chen, M., Tworek, J., Jun, H., Yuan, Q., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.arxiv.org/abs/2107.03374
- [9]Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.arxiv.org/abs/2110.14168
- [10]Conneau, A., Rinott, R., Lample, G., Williams, A., Bowman, S., Schwenk, H., and Stoyanov, V. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2475–2485, 2018.arxiv.org/abs/1809.05053
- [11]Fedus, W., Zoph, B., and Shazeer, N. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022.
- [12]Gale, T., Narayanan, D., Young, C., and Zaharia, M. MegaBlocks: Efficient sparse training with mixture-of-experts. In Proceedings of Machine Learning and Systems (MLSys), volume 5, 2023.
- [13]Gemma Team. Gemma 4 technical report. arXiv preprint arXiv:2607.02770, 2026. URL https://arxiv.org/abs/2607.02770.
- [16]Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In International Conference on Learning Representations (ICLR), 2021.
- [17]Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., and Blunsom, P. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems (NeurIPS), volume 28, 2015.arxiv.org/abs/1506.03340
- [18]Hillis, W. D. and Steele, Jr., G. L. Data parallel algorithms. Communications of the ACM, 29(12):1170–1183, 1986.DOI
- [19]Huang, H., Ardalani, N., Sun, A., Ke, L., Lee, H.-H. S., Bhosale, S., Wu, C.-J., and Lee, B. Toward efficient inference for mixture of experts. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, 2024.DOI
- [20]Hwang, C., Cui, W., Xiong, Y., Yang, Z., Liu, Z., Hu, H., Wang, Z., Salas, R., Jose, J., Ram, P., Chau, J., Cheng, P., Yang, F., Yang, M., and Xiong, Y. Tutel: Adaptive mixture-of-experts at scale. In Proceedings of Machine Learning and Systems (MLSys), volume 5, 2023.arxiv.org/abs/2206.03382
- [21]Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations (ICLR), 2024.
- [22]Kossmann, F., Jia, Z., and Aiken, A. Optimizing mixture of experts using dynamic recompilations. arXiv preprint arXiv:2205.01848, 2022.arxiv.org/abs/2205.01848
- [23]Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP), pp. 611–626, 2023.arxiv.org/abs/2309.06180
- [24]Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., and Chen, Z. GShard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations (ICLR), 2021.
- [26]Muennighoff, N., Liu, Q., Zebaze, A., Zheng, Q., Hui, B., Zhuo, T. Y., Singh, S., Tang, X., von Werra, L., and Longpre, S. OctoPack: Instruction tuning code large language models. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2308.07124
- [28]Qualcomm Innovation Center. Efficient transformers library (QEfficient). https://github.com/qualcomm/efficient-transformers, 2024.
- [30]Rajbhandari, S., Li, C., Yao, Z., Zhang, M., Aminabadi, R. Y., Awan, A. A., Rasley, J., and He, Y. DeepSpeed-MoE: Advancing mixture-of-experts inference and training to power next-generation AI scale. In Proceedings of the 39th International Conference on Machine Learning (ICML), volume 162 of Proceedings of Machine Learning Research, pp. 18332–18346, 2022.arxiv.org/abs/2201.05596
- [31]Rouhani, B. D., Zhao, R., More, A., Hall, M., Khodamoradi, A., Deng, S., Choudhary, D., Cornea, M., Dellinger, E., Denolf, K., et al. Microscaling data formats for deep learning. arXiv preprint arXiv:2310.10537, 2023.arxiv.org/abs/2310.10537
- [33]Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations (ICLR), 2017.arxiv.org/abs/1701.06538
- [34]Shen, H., Roesch, J., Chen, Z., Chen, W., Wu, Y., Li, M., Sharma, V., Tatlock, Z., and Wang, Y. Nimble: Efficiently compiling dynamic neural networks for model inference. In Proceedings of Machine Learning and Systems (ML-Sys), volume 3, pp. 208–222, 2021.arxiv.org/abs/2006.03031
- [35]Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford Alpaca: An instruction-following LLaMA model. https://github.com/tatsu-lab/stanford_alpaca, 2023.
- [37]Yang, A. et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.arxiv.org/abs/2505.09388
- [38]Yao, J., Anthony, Q., Shafi, A., Subramoni, H., and Panda, D. K. Exploiting inter-layer expert affinity for accelerating mixture-of-experts model inference. arXiv preprint arXiv:2401.08383, 2024.arxiv.org/abs/2401.08383
- [40]Zhou, Y., Lei, T., Liu, H., Du, N., Huang, Y., Zhao, V., Dai, A., Chen, Z., Le, Q., and Laudon, J. Mixture-of-experts with expert choice routing. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, 2022.arxiv.org/abs/2202.09368
- [41]Zhu, K., Zhao, W., Zheng, Z., Guo, T., Zhao, P., Bai, J., Yang, J., Liu, X., Diao, L., and Lin, W. DISC: A dynamic shape compiler for machine learning workloads. In Proceedings of the 1st Workshop on Machine Learning and Systems (EuroMLSys), pp. 89–95, 2021.








