betaFeedback

WorksComputer Science

Channel Selection for Mixture-of-Experts Decoding Needs No Calibration Data

Abstract

Mixture-of-Experts (MoE) models are bottlenecked by loading routed-expert weights from memory during interactive decoding at small batch sizes. Because most intermediate channels in a routed expert contribute little to any single token, channel selectors predict which channels matter and read only their weights, typically keeping channels above a threshold calibrated on offline data. We show that channel selection needs no calibration data. First, a calibrated threshold does not fix how much a selector reads, because the number of channels passing it varies across inputs; Prox with thresholds calibrated for an 18% read budget actually reads only 12.9% on average across our benchmarks. Second, top-k keeps each token's k highest-scoring channels and needs no calibration data. At the calibrated budget, a selector's top-k version matches or beats its threshold version on the same score in 40 of 43 comparisons at 13% and 18% budgets, including all 23 on generation tasks. The top-k versions also avoid the repetition-loop collapse that calibrated thresholds cause on Gemma 4. Calibrating a threshold thus provides neither of the two things selectors use it for: a known read budget and a consistent accuracy gain at that budget. We furthermore introduce a reader built entirely from the model's weights, which screens channels with a ternary sketch and recovers the skipped ones with a low-rank map. Across three post-trained MoE models, our reader retains 98.1% of full-model accuracy at 17.8% of routed-expert reads, with Triton kernels running MoE blocks 1.60 to 1.90x faster than optimized vLLM at batch one and up to 2.69x at larger batches on the same three models.

Introduction

Mixture-of-Experts (MoE) models scale capacity by routing each token to only a few experts ([57, 21]). At the small batch sizes of interactive decoding, latency is dominated by reading routed-expert weights from GPU memory (DRAM) to on-chip SRAM. Within each routed expert’s gated MLP, however, most intermediate channels contribute little to any single token and can be skipped with minimal loss ([52]). A channel selector exploits this by predicting which channels matter and reading only their weights.

Every channel selector couples two components: a channel score that estimates each channel’s contribution to the output, and a keep rule that decides which channels to compute. To balance quality and sparsity without retraining, most selectors use a calibrated threshold, setting a fixed per-layer cutoff on offline calibration data so that the model reads a target budget on that data ([37, 42, 76, 43]). The alternative keep rule is top-kk, which keeps each token’s kk highest-scoring channels and needs no calibration data itself ([44, 13]). Prior studies disagree on which rule is preferable. Prox ([43]) argues that a threshold’s variable channel count adapts computation to each token, and both Prox and the Sparsing Law ([48]) find their threshold versions ahead of top-kk on dense models. Conversely, other studies find top-kk ahead ([44]), while calibration introduces sensitivity to the choice of calibration data ([70] (Ji et al. 2025)).

We show that channel selection for frozen MoE decoding needs no calibration data. Selectors calibrate a threshold for two reasons, and it delivers neither. First, a calibrated threshold does not fix how much a selector reads (§ 4.1). The number of channels above a fixed cutoff changes with the input, so selectors with calibrated thresholds read less than the budget they were calibrated for in all 72 benchmark runs we made, and calibrating on the models’ own reasoning does not fix this. On Qwen3-Thinking, the count follows the answer written so far more than how much each token needs. Second, calibration gives no consistent accuracy gain over top-kk (§ 4.2). We run each selector twice on its own score, once with its calibrated threshold and once with top-kk, at the budget the threshold was calibrated for. The top-kk version matches or beats the threshold version in 40 of 43 comparisons across four MoE models at 13% and 18% budgets. Under heavy sparsity, Gemma 4 with calibrated thresholds falls into repetition loops during long reasoning, which the same selectors with top-kk avoid (Figure 1).

Figure 1. At heavy sparsity, calibrated selectors lose most of their accuracy on at least one task, whereas data-free top-kk selectors stay accurate and our reader performs best and decodes faster. (a) Worst accuracy over three MoE models and three benchmarks; dashed lines use calibrated thresholds, placed at the budget they were calibrated for. See Appendix Figure 11 for the same curves with calibrated selectors placed at the reads actually used. (b) Whole-model decoding speedup of our reader at 17.8% of reads over optimized vLLM.

With top-kk as the keep rule, performance depends on how well a selector ranks channels before reading them and handles skipped channels. Our reader is built entirely from the model’s weights (§ 5). For each token, it screens all candidate channels with a rotated ternary sketch of the weights (about 7% of the BF16 checkpoint), computes the top-kk exactly, and recovers the skipped channels through a low-rank map of the down projections.

We evaluate our reader in vLLM ([35]) on Qwen3-30B-A3B-Thinking-2507 ([75]), Gemma-4-26B-A4B-it ([26]), and Qwen3.5-35B-A3B ([53]), charging every weight and parameter read in 32-byte hardware sectors and checking counts with a GPU profiler (§ 2). At both 12.8% and 17.8% of routed-expert reads, it is the most accurate tested selector, on average and in its worst model-benchmark pair, which keeps 95.0% of full-model accuracy at 17.8%. On OLMoE ([50]), it matches the full model more closely than nine prior selectors at the budgets at which each was tested (Appendix § 16). Our Triton kernels make whole models decode up to 1.88×1.88 \times faster than optimized vLLM, with the gain peaking at batch size eight or sixteen (§ 6.2).

Background

In an MoE layer, a router sends each token to a few experts, each a gated MLP ([56]). We call each intermediate dimension of a routed expert a channel: channel jj of expert ee owns row jj of the gate and up projections GeG_e and VeV_e and column jj of the down projection DeD_e (Figure 2a). The routed output sums all channel outputs, each weighted by its expert’s router weight. In Qwen3-Thinking ([75]), a token goes to 8 experts of 768 channels, so a selector chooses among 6,144 channels per layer. Under a calibrated threshold, the token’s scores decide how many channels pass the cutoff; under top-kk, the deployer fixes this number in advance (Figure 2b).

Figure 2. A channel selector computes only a subset of each routed expert’s channels, and hardware charges every 32-byte sector it touches. (a) Channel jj owns row jj of the gate and up projections (GeG_e, VeV_e) and column jj of the down projection (DeD_e); HH is the hidden width, II the number of channels per expert, ϕ\phi the gate activation. (b) On the same scores, a calibrated threshold τ\tau keeps six channels on token A and two on token B, while top-kk keeps four on both. (c) A selector that needs 80 bytes spread over four sectors is charged 128 bytes.

We measure cost in reads: the bytes loaded per token as a percentage of the full model’s routed-expert reads. Reads include kept-channel weights, sketch or proxy weights, and auxiliary parameters. Modern NVIDIA GPUs fetch memory in 32-byte sectors ([51]), so we charge every touched sector in full (Figure 2c). A GPU profiler confirms the count: at batch one on Qwen3-Thinking, it measures cold-cache DRAM reads of 18.3% against our counted 17.8% (Appendix § 10).

A calibrated threshold is not a selector by itself but a keep rule that selectors such as Prox and TEAL use. To compare keep rules, we run each such selector twice on the same score: its threshold version keeps the calibrated threshold, and its top-kk version replaces the threshold with top-kk. We compare the two at the budget the threshold was calibrated for, because a deployer sets this budget in advance, while the reads the threshold version actually uses, its realized reads, are known only afterward (§ 4.1). As a check, we also run the top-kk version at these realized reads. Threshold versions read less than their budget, so this check favors them (§ 4.2).

Related work

Four families of channel selectors. Prior selectors differ in what they read before choosing channels (Figure 3). Trained predictors use an auxiliary network (Liu et al., [46]; Song et al., 2024; Akhauri et al. [3], and SparseInfer uses sign bits without training Shin et al. [58].

Figure 3. Prior channel selectors, grouped by what they read before choosing channels. Dashed rows or columns are skipped; chips show each keep rule, and a dot marks budgets or rotations fitted on data. HiRE and Mixture-of-Channels (MoC) train the model for top-kk; TopK-Guided bounds a threshold by a top-kk window. Our reader (bold) fits nothing to the model.

Full-projection selectors compute a whole gate or up projection first, so they read at least a third of each routed expert [37, 28, 52], as does M-COUNTDOWN [16]. Input-sparsity selectors skip the weight columns that multiply small inputs [42, 76, 44, 13, 29, 12]. Low-bit-copy selectors, including our reader, score channels with a compressed copy of the weights and then load the kept rows [78, 43] (Samaga B L et al. 2024); FloE computes its whole up projection from an INT2 copy.

Calibration and prior keep-rule comparisons. Most selectors for frozen models calibrate on sample data, while HiRE and Mixture-of-Channels [73] train the model for top-kk. Six earlier studies compare the two keep rules, and they disagree (Appendix Table 2). La RoSA [44], Sparser-MoE [9], and batch-wise pruning [34] favor top-kk. The Sparsing Law [48] and Prox [43] favor thresholds, Prox by 1.8 points on dense Qwen3 models. TopK-Guided [2] finds a hybrid best. However, five of the six studies test dense models, mostly on short tasks, and two change the score along with the keep rule. Several works note that reads under a calibrated threshold drift at test time [44, 29, 8], yet none of the six runs top-kk at the reads a threshold actually used. Prox’s result does not carry over to OLMoE: on its likelihood tasks at 13% and 18% budgets, its two versions tie in 7 of 10 comparisons, and top-kk leads in 2 (Appendix § 13.§ 1).

Scope. Our conclusions concern calibrated thresholds on the channels of frozen MoE experts. Task-dependent sparsity also motivates data-free selection [64], and SparseDecoding finds that calibration and decoding activations differ [68]. Thresholds can work elsewhere, as in sparse attention [6] or trained MoE routers [7]. Appendix § 8 covers recovery, batching, and other levels of sparsity.

Channel selection needs no calibration data

Selectors calibrate a threshold for two reasons: to read a known budget, and to gain accuracy at that budget by giving each token the channels it needs.

A calibrated threshold does not fix how much a selector reads

Channel scores change across tokens, tasks, and answer length, so a different number of channels passes a fixed cutoff τ\tau each time, while top-kk always keeps kk.

Reads vary across tokens and fall during generation. Figure 4a–Figure 4b isolate the keep rule from the errors of an approximate score: channels are scored by their exact contribution to the output, and τ\tau is calibrated on UltraChat to keep about 12% (12.3% on Qwen3-Thinking). On held-out chat, the mean kept count matches the target, consistent with Prox’s small drift on five short tasks [43]. Yet even on chat, the middle 90% of Qwen3-Thinking tokens keep 9.0 to 16.2% of channels. On benchmark answers, scores are smaller and the median falls to 10.0% (Figure 4a, orange). Counts also fall as the answer grows (Figure 4b): around 6,000 tokens, Qwen3-Thinking keeps 71% of its calibrated count, Gemma 4 92%, Qwen3.5 88%.

Figure 4. Under a calibrated threshold, the number of kept channels changes from token to token, and selectors read less than their budget on benchmarks. (a) Share of channels each token keeps under an exact-score threshold calibrated on UltraChat for about 12%, on held-out chat and on the model’s answers to GPQA-Diamond ([54]), IFEval ([77]), and MMLU-Pro ([69]); top-kk keeps the same share on every token (line). (b) Median kept count per 500-token window, relative to the calibrated count (about 94,000 answer tokens per model). (c) In all 72 benchmark runs, Prox and our reader with calibrated thresholds read below their budget (the diagonal).

Calibrated budgets are missed at deployment. Selectors with calibrated thresholds read less than their budget in all 72 benchmark runs we made: Prox and our reader, each at targets of about 13%, 18%, 28%, and 38% (Figure 4c; Appendix § 11). Prox with its threshold calibrated for 18% reads 12.9% on average and as little as 9.9%. About 1.7 percentage points of this gap are built into Prox’s published rule for splitting the budget between inputs and channels, once reads are counted in sectors (Appendix § 9). Our reader with calibrated thresholds hits its target on calibration data but reads 0.4 to 5.8 percentage points less on benchmarks. Calibrating on the models’ own reasoning does not stop this drift (Appendix § 11.§ 1).

Kept channels follow the earlier answer, not token need. Why do counts fall? We take a 256-token window from position 6,000 of a Qwen3-Thinking GPQA-Diamond answer and place it directly after the question. Under a Prox threshold calibrated for 13%, the same tokens then keep 57% more channels, close to the count at the start of the answer. Removing earlier context raises the count even for the same tokens. This fall barely helps: keeping each layer’s count the same early and late in the answer, at equal total reads, changes KLKL divergence by at most 1.0±0.5%1.0 \pm0.5\% on all three models. Nor do harder tokens get more channels: kept counts correlate with the full model’s next-token loss at −0.29-0.29 to +0.14+0.14 (Spearman). Does the remaining variation from token to token help? Giving every token in a layer the same count, at equal reads, lowers KLKL divergence on reasoning by 3.3±1.0%3.3 \pm1.0\% on Qwen3-Thinking and 12.5±1.5%12.5 \pm1.5\% on Gemma 4, so there the variation hurts; on Qwen3.5 it raises KLKL by 10.5±1.1%10.5 \pm1.1\%, so there it helps (Appendix § 12).

Calibration gives no consistent accuracy gain over top-kk

We compare the threshold and top-kk versions of existing selectors (§ 2) at the budget each threshold was calibrated for. Prox also selects inputs first; Appendix § 13 gives the input rules.

Top-kk matches or beats thresholds across models and scores. We make 43 paired comparisons at 13% and 18% budgets, on generation tasks on four MoE models and likelihood tasks on OLMoE, with three channel scores: Prox, TEAL, and Prox scaled to output units (Appendix § 13). The top-kk version ties the threshold version in 30 (within two paired standard errors), leads in 10, and trails in 3, each time by at most 2.8 points (Figure 5). We exclude two TEAL comparisons where both versions collapse. Pooled per model, no model favors the threshold version (Appendix Table 14). On the three post-trained models alone, the top-kk version leads in 5 of 11 comparisons and the threshold version in none.

Figure 5. On the same score, a selector’s top-kk version matches or beats its threshold version in 40 of 43 comparisons and avoids Gemma 4’s collapse. Each dot is the accuracy of a selector’s threshold version minus that of its top-kk version, at the budget the threshold was calibrated for, with ±2\pm2 paired SE. Gray marks ties, blue top-kk leads (the four on Gemma 4 by 38 to 45 points), and orange threshold leads (at most 2.8 points). See Appendix Figure 8 for the same comparisons with each top-kk version run at the reads its threshold version actually used.

Robustness at realized reads. We also run each top-kk version at the reads its threshold version actually used, a check that favors the threshold version (§ 2). The top-kk version still matches or beats it in 37 of 41 comparisons (Appendix § 13). The largest lead of a threshold version, +11.5±1.9+11.5 \pm1.9 points for Prox on Qwen3-Thinking IFEval, disappears at the calibrated budget, where Prox scores 85.4 with top-kk and 84.3 with its threshold. At similar reads, controls show roughly equal gains from per-layer budgets and from per-token counts (Appendix § 12). Per-token variation helps most on Qwen3.5 GPQA-Diamond, by +7.1±2.7+7.1 \pm2.7 points. But this gain comes at a budget the deployer did not choose: with its threshold calibrated for 13%, Prox reads only 9.1% there and scores 80.6, which ties the 79.9 it scores with top-kk at 13.5% of reads.

Other scores and calibrated parts. Our reader behaves the same way. Replacing its top-kk with calibrated per-layer thresholds gives 34 ties and 2 comparisons where the top-kk version leads, both on Gemma 4, out of 36. CATS ([37]), which thresholds exact gate activations, ties in 16 of 18 comparisons at 53% and 67% of reads, and its threshold version leads in the other 2, both on Gemma 4 (Appendix § 13.§ 3). Some selectors also calibrate layer budgets (TEAL, WINA) or recovery terms (R-Sparse, GRAIL). Calibrated versions of these parts lower KL divergence, calibrated layer budgets even on reasoning text unlike their calibration data, but improve no benchmark beyond two standard errors; a calibrated recovery map beats our weight-only map in 0 of 18 comparisons (Appendix § 13.§ 4).

Long reasoning can collapse under calibrated thresholds

Channel thresholds trigger repetition loops on Gemma 4. The four largest gaps in Figure 5 share one cause: on Gemma 4 at the 13% budget, the top-kk versions of Prox and its output-scaled variant beat their threshold versions by 38 to 45 points. With a calibrated threshold, Gemma 4 repeats phrases until the 32,768-token limit. Reading too little does not explain this. Over four samples per question, Prox with its threshold calibrated for 18% reads 13.3% and scores 11.7. With top-kk at nearly the same reads, 13.5%, Prox scores 56.3 (Appendix Figure 10), and 145 rather than 674 of 792 answers hit the limit. Replacing only Prox’s channel threshold with top-kk avoids the collapse; replacing only its input threshold does not (Appendix § 13.§ 2). Neither a 4% floor on channels nor calibration on Gemma 4’s own reasoning prevents it (Appendices § 11.§ 1 and § 13.§ 2). Similar loops appear in other compressed models: pruning amplifies loops at stable perplexity (Lee et al. [39]), 2-bit quantization often makes reasoning loop or exhaust its token budget (Alimaskina et al. [4]), and batch-shared TEAL masks collapse reasoning (Kim et al. [34]).

Collapse behaves like a cliff. We suspect that once a sparse model starts a repetition, it rarely stops on its own. If so, accuracy drops sharply, not gradually, once the error sparsity adds to each token passes a level that depends on the model, and a less accurate selector needs more reads to stay below it. On Gemma 4 IFEval, where the full model scores 93.5, WINA as published scores 40.9 at 20% of reads, and 53% of its answers stop at the length limit. Prox’s score with top-kk reaches 57.1 at 13.5% of reads, and our reader 88.0 at 12.8%. Prox still collapses when each layer’s count is fixed to its average under the threshold (Appendix § 12). With a calibrated threshold, a deployer therefore cannot see this level coming: the selector’s reads are not known in advance (§ 4.1), and even at equal reads its accuracy can fall far below that of its top-kk version (11.7 against 56.3 above).

A reader built from the weights alone

With top-kk as the keep rule, a selector’s quality depends on two things: how well it scores channels before reading them, and how it handles the channels it skips. We build our reader around both.

Scoring matters more than the keep rule. Switching Prox from its calibrated threshold to top-kk changes next-token KL divergence by only −17% to −3% on held-out OLMoE documents and by −6% to +24% on chat. Changing the score moves KL much more. On the same 4-bit weight copy, scaling Prox’s score to output units, with down-projection norms and router weights, divides KL by 1.3 to 1.5. On OLMoE development data at 17.8% of reads, with no recovery, selectors that score with an INT4 copy of the gate projection or with TEAL’s input magnitudes reach 2.8 and 5.1 times the KL divergence of our reader’s full-sketch screen (Appendix Table 17). Across four scores and four budgets on OLMoE, KL divergence falls as the kept channels carry a larger share of the routed output’s energy (Spearman −0.96; Appendix § 14).

Our reader therefore puts its effort into the score and into recovering skipped channels. It is built once from the model’s weights, with nothing fitted to the target model. Three settings were chosen once on OLMoE development data and reused on every model: 128-coordinate rotation blocks, 80-coordinate partial screens, and rank-32 recovery (Appendix § 14). For each token and layer, it screens all candidate channels, computes the top-kk exactly, and recovers the rest (Figure 6).

Figure 6. Our reader screens candidate channels with a ternary sketch, computes the top-kk channels exactly, and recovers the rest using only the full model’s weights. Top: the full model reads every channel of a routed expert; our reader reads a ternary sketch, the kk kept channels, and a low-rank recovery map. Middle: the three stages. Bottom: reads at 17.8% on Qwen3-Thinking.

Screening channels with a rotated ternary sketch

The screen estimates every candidate channel’s gate and up activations from a ternary sketch of the weights, built once at load time. Weights and hidden states contain outliers, which make low-bit codes inaccurate. We therefore rotate each gate row gjg_j and up row vjv_j with a randomized Hadamard transform RR ([10, 66]), applied in blocks of 128 coordinates, which spreads large values across each block. Because RR is orthogonal, the rotation leaves activations unchanged: gjTx=(Rgj)T(Rx)g_j^\mathsf{T}x=(Rg_j)^\mathsf{T}(Rx). Each block of RgjRg_j and RvjRv_j is stored as ternary codes {−1,0,1}\{-1,0,1\} ([41]), together with one FP16 scale and the block’s mean-squared quantization error ρb\rho_b. This shrinks a block from 256 bytes in BF16 to about 30 bytes, so the sketch adds about 7% to resident memory. At decode time, the reader rotates xx once and multiplies it by the sketch, giving estimates g^j≈gjTx\hat{g}_j\approx g_j^\mathsf{T}x and v^j≈vjTx\hat{v}_j\approx v_j^\mathsf{T}x.

At our two lowest budgets, 12.8% and 17.8% of reads, reading the full sketch would take 44 to 61% of all reads on OLMoE. At these two budgets the reader runs a partial screen: in each block, it reads ternary codes only at the 80 coordinates where ∣Rx∣\lvert Rx\rvert is largest. This buys 20 to 50% more exact channels.

Output-scaled scoring and exact top-kk computation

The reader scores each candidate channel by the size of what it would add to the MoE output. Channel jj adds wjajdjw_j a_j d_j, where wjw_j is the router weight of channel jj’s expert, aj=ϕ(gjTx)(vjTx)a_j=\phi(g_j^\mathsf{T}x)(v_j^\mathsf{T}x) its activation, and djd_j its down-projection column (Figure 2a). The size of this contribution is wj∥dj∥∣aj∣w_j\lVert d_j\rVert\lvert a_j\rvert, and replacing aja_j with its sketch estimate gives the score

sj=wj∥dj∥∣ϕ(g^j)v^j∣(1)s_j = w_j\lVert d_j\rVert\left\lvert\phi(\hat{g}_j)\hat{v}_j\right\rvert \tag*{(1)}

where the norms ∥dj∥\lVert d_j\rVert are computed once at load time. Because the score is in output units, channels from different experts are directly comparable. The reader therefore picks the kk best channels across all of the token’s experts, and one expert may receive many channels while another receives none. From 28.7% of reads upward, the reader reads the full sketch and also uses its block errors ρb\rho_b to rank channels by the expected squared output error left after recovery if skipped (Appendix § 14).

For the kk kept channels, the reader loads the full weights and computes the exact activations aja_j. To make each kept channel’s down weights contiguous, we transpose every down projection DeD_e once at load time, so that column jj becomes a row. Prefill and decode both use this one layout.

Weight-only low-rank recovery of skipped channels

Each skipped channel contributes little, but thousands of them per layer together matter. The screen has already estimated the activation pj=ϕ(g^j)v^jp_j=\phi(\hat{g}_j)\hat{v}_j of every skipped channel j∈Sj\in\mathcal{S}, so the reader can reuse these estimates without rereading any gate or up weights. It still needs each skipped channel’s down column djd_j to turn the estimate into output. To avoid loading these full-width columns, we approximate them with a small map built once before deployment. We stack the down columns of all experts in a layer, compute their top 32 left singular vectors U∈RH×32U\in\mathbb{R}^{H\times32}, and store each channel’s coefficients bj=UTdjb_j=U^\mathsf{T}d_j. With kept channels K\mathcal{K}, the reader then outputs

y^=∑j∈Kwjdjaj+U∑j∈Swjbjpj.(2)\hat{y}=\sum_{j\in\mathcal{K}}w_jd_ja_j+U\sum_{j\in\mathcal{S}}w_jb_jp_j. \tag*{(2)}

The first sum is exact; the second adds the skipped channels’ estimated output through the rank-32 map. The map adds about 0.5% to resident memory and 0.5 to 0.8 percentage points of reads.

Each component earns its reads. On OLMoE at equal reads, spending 0.6 to 0.7 percentage points on recovery lowers KL divergence more than spending them on more exact channels. The partial screen lowers KL relative to the full screen, from 0.488 to 0.310 at 13.5% of reads and from 0.245 to 0.187 at 18.5%. The same components help benchmark accuracy on Gemma 4. With the full screen at 18% of reads, recovery raises IFEval accuracy by 11.6 ± 1.6 points and reduces repetition loops. At 13%, the partial screen beats the full screen by 13.7 ± 1.3 points on MMLU-Pro.

Accuracy and speed on post-trained MoE models

Accuracy

Experimental setup and baselines. We evaluate our reader in vLLM (Kwon et al., [35]) on three post-trained MoE models, Qwen3-Thinking, Gemma 4, and Qwen3.5, and three benchmarks, GPQA-Diamond, IFEval, and MMLU-Pro (Appendix § 9). The full model processes each prompt, and the selector runs on every decoded token. We compare against three published selectors. Prox (Liu et al., [43]) uses calibrated thresholds, and we also run its score with top-kk. TEAL (Liu et al., [42]) uses calibrated thresholds with greedy per-layer budgets. WINA (Chen et al., [13]) as published keeps the top-kk with calibrated per-projection budgets, and we also run its score with calibrated thresholds. Every calibrated part is fitted on the same UltraChat assistant turns (Ding et al., [19]). We report accuracy as a percentage of the full model’s accuracy, over nine model-benchmark pairs (Appendix Table 18), and count a difference only beyond two question-clustered paired standard errors.

Accuracy at heavy sparsity. Our reader has the highest mean accuracy at both low budgets, and its weakest pair beats every other selector’s weakest pair (Table 1; Appendix Figure 9 shows every pair).

SelectorTargetRealizedMeanWorstTargetRealizedMeanWorst
\multicolumn{4}{c}{About 13% of reads}\multicolumn{4}{c}{About 18% of reads}
Our reader–12.894.586.9–17.898.195.0
Prox score, top-kk–13.587.061.1–18.596.190.4
Prox as published, calibrated138.551.50.01812.968.512.1
WINA as published, top-kk–152.40.0–2028.41.1
WINA score, calibrated157.80.20.02011.25.00.0
TEAL as published, calibrated158.90.20.02012.69.90.0

Table 1. Our reader is the most accurate tested selector at the two lowest budgets. Mean and worst accuracy over nine model-benchmark pairs, as a percentage of the full model’s accuracy. Selectors with calibrated thresholds list the budget they were calibrated for (Target) and the reads they used (Realized); WINA as published uses top-kk and meets its budget to the precision shown.

Prox, with its threshold calibrated for 18%, reads 12.9%, about as much as our reader at 12.8%. Yet it reaches only 68.5% of full-model accuracy on average, against 94.5% for our reader. Most of this gap comes from Gemma 4, where its threshold version falls into the repetition loops of § 4.3; on the two Qwen models, Prox is about as accurate as our reader at similar reads. Top-kk cannot rescue a weak score: WINA as published uses top-kk and reaches only 28.4% of full-model accuracy at 20% of reads. From 28% of reads upward, our reader and Prox, with either keep rule, stay within 10% of the full model.

Speed

Our reader accelerates MoE blocks by up to 2.69×2.69 \times in optimized vLLM. Our Triton kernels select exactly the channels of the reader evaluated above, at 17.8% of reads on one RTX PRO 6000 Blackwell GPU. We compare against unmodified vLLM in its fastest configuration, default or tuned, at each batch size, with the same router (Figure 7). On Qwen3-Thinking, Gemma 4, and Qwen3.5, MoE blocks run 1.90×1.90 \times, 1.71×1.71 \times, and 1.60×1.60 \times faster at batch one, and 2.39×2.39 \times, 2.34×2.34 \times, and 2.51×2.51 \times at four.

Figure 7. Our reader accelerates MoE blocks by 1.60 to 2.69× over optimized vLLM across three post-trained models. At 17.8% of routed-expert reads, median of three processes. (a) MoE-block decode speedup; both arms share the unmodified router, and Gemma 4’s dense MLP and Qwen3.5’s shared expert are timed inside the block. (b) Whole model, batch sizes 1 to 128: right is faster per user, up is more output per GPU.

The whole model gains less, because attention runs as before. Whole models decode 1.15×1.15 \times to 1.36×1.36 \times faster at batch one and up to 1.75×1.75 \times, 1.74×1.74 \times, and 1.88×1.88 \times at batch eight or sixteen, in the same model order, and 1.08×1.08 \times to 1.54×1.54 \times with 8k to 32k tokens of context. At 100 tokens per second per user, one GPU serves 3.3 to 4.3 times as many tokens (Figure 7b; Appendix § 17).

At batch one, the speedup is smaller than the saved reads suggest. On Qwen3-Thinking, the MoE block moves only 18.3% of the full model’s memory traffic, yet it runs 1.90×1.90 \times faster, not about 5×5 \times. Part of each step is a fixed cost, such as launching kernels, that fewer reads do not shrink. Larger batches share these fixed costs, so the block speedup grows up to batch four or eight (Table 20).

Batch size1248163264128
Qwen3-Thinking
MoE block (ms) — Full model3.0845.5258.51711.96918.370
MoE block (ms) — Our reader1.6192.6043.5625.1319.260not measured
MoE block (ms) — Speedup (×)1.902.122.392.331.98
Per output token (ms) — Full model6.2539.45412.45015.66622.04427.36634.10738.196
Per output token (ms) — Our reader4.6056.6097.4538.95712.85019.83128.92538.363
Per output token (ms) — Speedup (×)1.361.431.671.751.721.381.181.00
Gemma 4
MoE block (ms) — Full model3.6465.3998.01010.75015.244
MoE block (ms) — Our reader2.1362.7043.4284.8467.591not measured
MoE block (ms) — Speedup (×)1.712.002.342.222.01
Per output token (ms) — Full model6.4598.26011.06014.00519.12223.25029.89036.345
Per output token (ms) — Our reader4.9155.6916.5158.05811.10916.38524.40833.937
Per output token (ms) — Speedup (×)1.311.451.701.741.721.421.221.07
Qwen3.5
MoE block (ms) — Full model2.1633.5115.5039.13215.085
MoE block (ms) — Our reader1.3502.0492.1933.3945.760not measured
MoE block (ms) — Speedup (×)1.601.712.512.692.62
Per output token (ms) — Full model5.8628.14110.17414.12720.39729.02940.66853.605
Per output token (ms) — Our reader5.0926.6806.9438.54010.83716.68030.81243.353
Per output token (ms) — Speedup (×)1.151.221.471.651.881.741.321.24

Table 20. Speedups peak at batch sizes four to sixteen, and the whole-model gain fades by batch 128. Decode times at 17.8% of routed-expert reads, in milliseconds per decoding step, median of three processes. The MoE block covers all MoE layers with the router, Gemma 4’s dense MLP, and Qwen3.5’s shared expert. Whole-model times use 128-token prompts.

In dense models, the channels a batch needs together grow with batch size and approach the dense set in deeper layers, so channel selection helps mainly at small batches (Shrestha et al., [59]). MoE models with many small experts push this limit to larger batches: at batch 16, each expert serves about one token or fewer on average. Our reader still decodes 1.72×1.72 \times to 1.88×1.88 \times faster as a whole model at this batch size; Park et al. [52] likewise find their MoE-layer speedup peaks at batch 16 to 128. At batch 128, the whole-model speedup falls to 1.00×1.00 \times to 1.24×1.24 \times.

Limitations and conclusion

Limitations. Our kernels turn only part of the saved reads into speed (§ 6.2), and they were tuned for one budget on one GPU type (Appendix § 17). Selecting channels for a whole batch at once, so the gain lasts to larger batches, is future work. The share of coordinates the partial screen reads was chosen on OLMoE and checked only on Gemma 4 (Table 17). Our reader changes only routed experts in decoding. Its sketch and recovery map add about 7% and 0.5% of a BF16 checkpoint, a larger share of a quantized one (Table 3). We tested models of up to 35B parameters.

ModelTotalActiveExpertHiddenlayersfullSketchRecovery
Qwen3-Thinking1288768204848–7.60.50
Gemma 41288704281630dense MLP7.00.35
Qwen3.52568512204840shared expert7.00.47
OLMoE6481024204816–7.30.50

Table 3. Evaluated MoE models and the resident memory our reader adds. Each row gives the number of routed experts and how many each token uses, the channels per expert and the hidden width, the number of MoE layers, any module that always runs in full, and the resident memory of our reader’s ternary sketch and recovery map as a percentage of the BF16 checkpoint. OLMoE’s sketch size is computed analytically.

Variant at 17.8% of reads (OLMoE, development)KL
Full sketch, no recovery (rotated, top-kk over all routed experts)0.286
Without rotation0.348
Fixed share of channels per expert0.491
Score from an INT4 copy of the gate projection, same top-kk0.811
Score from TEAL’s input magnitudes, top-kk inputs1.471
Coordinates read per 128-coordinate block
ModelBudget (% of reads)1289680644832
OLMoE130.4430.3360.3160.3170.3430.424
180.2240.1980.1940.1980.2290.297
280.1020.0940.0980.1060.1270.173
Gemma 4130.2350.1880.1760.1770.1970.260
180.1240.1100.1060.1110.1290.179
280.0560.0520.0520.0570.0700.103

Table 17. Rotation and ranking channels across all routed experts lower KL divergence, and reading 80 of 128 coordinates is at or near the best. Next-token KL divergence from the full model on development data (lower is better). Top: at 17.8% of reads on OLMoE without recovery, the full sketch, two of its design variants, and two other scores under top-kk. Bottom: coordinates the partial screen reads per 128-coordinate block, with recovery, at equal reads; bold marks the lowest in each row.

What our results suggest. We suspect two patterns hold beyond our models. First, short likelihood evaluations hide failures of long generation. On likelihood tasks, the two versions of a selector differ little and in both directions. On generation tasks at a 13% or 18% budget, no threshold version beats its top-kk version, and Gemma 4 falls into repetition loops under thresholds. REAP (Lasby et al., [36]) saw the same contrast when merging experts, and pruning can add loops at largely unchanged perplexity (Lee et al., [39]). Second, lower KL divergence need not mean higher accuracy: calibrated layer budgets lower KL divergence even on reasoning text unlike their calibration data, yet neither they nor calibrated recovery maps raise a benchmark score.

Conclusion. Channel selection for MoE decoding needs no calibration data. With a calibrated threshold, a selector’s reads follow the input rather than the budget. At the calibrated budget, its threshold version gains no consistent accuracy over its top-kk version on the same score. Built from the weights alone, our reader retains 98.1% of full-model accuracy at 17.8% of routed-expert reads on three models and runs MoE blocks up to 2.69×2.69 \times faster and whole models up to 1.88×1.88 \times faster than optimized vLLM.

AI USE STATEMENT

We used LLM-based coding assistants to write the experiment code and kernels, run experiments, and write the scripts that summarize results, and to help draft and edit the text. The authors designed the study, checked every reported number against the experiment outputs, revised the manuscript, and take full responsibility for its content.

REPRODUCIBILITY STATEMENT

All models, benchmarks and calibration data we use are publicly available. Sections 2 and 5 define the selectors and our reader. Appendix § 9 gives the models, benchmarks, generation settings, statistics, calibration protocols, baseline adaptations and hardware, and Appendix § 10 defines how reads are counted. Our reader, its Triton kernels, the vLLM patches, the baselines, the evaluation scripts and the scripts that compute the paper’s numbers from the evaluation results are in https://github.com/AidenGeunGeun/calibration-free-moe.

StudyRegimeModelsSame scoreMatchedAhead
La RoSA (2025b)dense, PPLLLaMA2-7Byestargettop-kk (PPL)
Batch-wise pruning (2026c)dense, CoT2 distilled 7–8Byestargettop-kk (batch 4)
Sparsing Law (2025)dense, short tasks0.1B, 0.8Bnotargetthreshold
Prox (2026)dense, short tasksQwen3-8B, 14Byestargetthreshold (+1.8)
TopK-Guided (2026)dense, short tasksLlama-2/3, 7–8Bnotargethybrid; top-kk vs. τ\tau mixed
Sparser-MoE (2026d)MoE, short tasksOLMoEuncleartargettop-kk (<1 pt)
This paper § 4.2MoE, short + CoT4 MoE modelsyesbothtie; top-kk on Gemma 4

Table 2. Prior keep-rule comparisons reach opposite conclusions, focus mostly on short tasks on dense models, and none runs top-kk at the reads a threshold actually uses. Unlike most prior comparisons, ours holds the channel score fixed (Same score), tests long chain-of-thought reasoning on MoE models (Regime), and runs top-kk both at the target budget and at realized reads (Matched).

Extended related work

Prior comparisons of calibrated thresholds and top-kk. Earlier studies disagree on whether a calibrated threshold or top-kk selects channels better (Table 2). La RoSA, Sparser-MoE, and batch-wise adaptive pruning find top-kk ahead; batch-wise adaptive pruning is the only prior study in the table that tests long chain-of-thought reasoning. The Sparsing Law and Prox find calibrated thresholds ahead. TopK-Guided finds top-kk (WINA) ahead of thresholded TEAL on Llama-2-7B but not consistently on Llama-3-8B, and its best rule is a hybrid: a calibrated threshold whose per-token count stays within ±δ\pm\delta of kk, combined with calibrated block budgets. In our MoE experiments, calibrated layer budgets give no held-out benchmark gain (Appendix F.4), and a per-token floor on the count does not prevent Gemma 4’s collapse (Appendix F.2). We do not test a two-sided ±δ\pm\delta band.

Approximating skipped channels. Instead of simply dropping the channels they skip, several methods approximate what those channels would have added. R-Sparse adds low-rank weight factors from an SVD and chooses how to split the budget between them and sparse inputs on calibration data. GRAIL [65] fits linear compensation maps on calibration data to approximate omitted activations. ReComp [8] distills a calibrated low-rank estimate of the skipped MLP output. Other methods send medium-importance columns through a compressed copy, as in SpAx [33], or through a pruned or low-precision path, as in DAWS [61] and SharQ [49]. Our recovery map needs no calibration data. It reuses each skipped channel’s activation estimate from the ternary screen and passes it through the top singular vectors of the down projections.

Speed under batching. In batched decoding, the GPU must load every channel that any token in the batch selects, so per-token selection saves fewer reads as the batch grows [59]. SERE [71] addresses this by shrinking the union of routed experts across a batch. Weak dense baselines can also inflate reported speedups [38]. We therefore charge reads in hardware sectors, check them with a GPU profiler, time against optimized vLLM, and report speedups at batch sizes one to sixteen.

Other levels of sparsity. Other methods act at a different granularity and can be combined with our reader. Expert-skipping and expert-merging methods drop, prune, or fold whole experts [47, 30, 17, 36, 14, 74, 72, 1, 32]. Static compression methods prune or quantize weights ahead of time [23, 62, 5, 45, 11] (Frantar & Alistarh 2023). Finally, prior work has studied per-token channel selection (Samaga B L et al., 2024), adaptive pruning with little or no calibration data [20, 67, 34], and sparsity inside MoE experts [78, 52], including top-kk neuron selection trained into experts during pretraining [15].

ModelBenchmarkItemsSamplesTemperatureTop-ppTop-kkMax tokens
Qwen3-ThinkingGPQA-D19840.60.952032,768
IFEval54110.60.952032,768
MMLU-Pro1,00010.60.952032,768
Gemma 4GPQA-D19841.00.956432,768
IFEval54111.00.956432,768
MMLU-Pro1,00011.00.956432,768
Qwen3.5†GPQA-D19841.00.952081,920
IFEval54111.00.952032,768
MMLU-Pro1,00011.00.952032,768

Table 4. Evaluation protocol on the post-trained models. Each model runs in thinking mode with its model card’s sampling parameters and random seed 20260923. GPQA-Diamond reports accuracy over four samples per question, IFEval reports prompt-level strict accuracy, and MMLU-Pro reports accuracy on a fixed 1,000-question subset.

Experimental setup

Models and runtime. We evaluate Qwen3-30B-A3B-Thinking-2507 (Qwen3-Thinking), Gemma-4-26B-A4B-it (Gemma 4), and Qwen3.5-35B-A3B (Qwen3.5) in their official BF16 releases in vLLM 0.29.0. For every prompt, the full model processes the prompt and generates the first output token, and the channel selector runs on every later decoded token. Shared experts and parallel dense MLPs always run in full and are not counted in routed-expert reads.

Benchmarks. GPQA-Diamond uses all 198 questions with four sampled answers per question, in each model card’s prompt format and with its sampling parameters. IFEval uses all 541 prompts and reports prompt-level strict accuracy. MMLU-Pro uses a fixed stratified subset of 1,000 questions. Answers may be up to 32,768 tokens long, except on Qwen3.5 GPQA-Diamond, where its model card recommends 81,920 tokens. The full Qwen3-Thinking model also ran GPQA-Diamond with an 81,920-token cap, but none of its answers exceeds 23,500 tokens. On OLMoE, GSM8K ([18]) (1,319 questions, eight-shot chain of thought) and BBH ([63]) (6,511 questions, three-shot chain of thought) use greedy decoding up to 512 tokens under the task definitions of the OLМES evaluation suite ([27]).

Answers that hit the length limit. On the post-trained models, an answer that reaches the token limit without finishing counts as wrong, for every method. On IFEval, most such answers contain no final answer at all. Of the 10,872 IFEval answers that hit the limit in our main runs, 10,649 never left their reasoning trace. Of the other 223, 202 ended in repetition loops and the rest stopped mid-sentence. IFEval’s format checkers can still pass such fragments on rules such as lowercase-only or no commas, and crediting them would change single model-benchmark results by up to 3.3 points. Counting them as wrong avoids crediting collapsed outputs.

Statistics. Benchmark comparisons use paired standard errors clustered by question. We count two methods as different when their paired difference exceeds two standard errors, without correcting for the number of comparisons. A tie therefore means no difference was detected, not that the methods are equivalent. Unless stated otherwise, ±\pm denotes one standard error. Next-token KL divergence is KL⁡(P∥P^)\operatorname{KL}(P \mathbin{\|} \hat{P}), from the full model’s next-token distribution PP to the sparse model’s P^\hat{P}, averaged over scored positions; top-1 agreement is the share of positions where both put the same token first.

Calibration on the post-trained models. Every calibrated method is fitted on the same UltraChat conversations ([19]). We split them with a fixed seed into 448 fitting, 64 tuning, and 128 development conversations; the tuning set only selects the checkpoint of the calibrated recovery map. Each conversation is rendered with the model’s chat template and contributes one 128-token window from an assistant turn. Prox fits its thresholds on 20,480 token positions per layer, sampled from the full text of 30 fitting conversations. TEAL’s greedy budget allocation and both WINA variants use 512 fitting states per layer. UltraChat turns contain no reasoning traces, so these thresholds are fitted outside thinking mode and evaluated in thinking mode. Appendix § 11.§ 1 repeats the calibration on the models’ own reasoning instead.

How Prox splits its budget. Prox splits each target budget between input and channel sparsity with its published rule, which prices its 4-bit weight copy with Prox’s latency model. Counted in sectors, this rule plans 1.6 to 1.7 percentage points of reads below the target at budgets from 13 to 38%, and 5.6 points below at 58 and 78%. Qwen3.5’s 13% thresholds were fitted for a 12.5% target. At test time, every Prox run reads even less than this plan.

Calibration on OLMoE. OLMoE is a base model, so its thresholds are fitted on Pile text ([24]). Prox uses the first 30 fitting documents (20,480 positions per layer), TEAL uses 177 documents (512 states per layer), and 40 further documents serve as tuning data. Our reader’s constants and the baselines’ input fractions were chosen on a separate set of 192 development documents, and the held-out OLMoE evaluation uses another 192. The development and held-out sets are disjoint from each other and from all fitting and tuning documents. Fitting Prox takes 3 to 19 minutes on a single GPU.

Baselines. Prox ([43]) was designed for dense models. We apply it directly to each routed expert and store its 4-bit copy of the gate and up weights in the memory layout that needs the fewest sector reads. Prox’s top-kk version keeps each token’s 30% largest input coordinates (50% from 28% of reads upward) and then its top-kk channels; these input fractions were also chosen on OLMoE development data. TEAL ([42]) uses its greedy search to divide sparsity among the projections of each layer; we apply it to the gate, up, and down projections of the routed experts. WINA’s score with a calibrated threshold uses one threshold per projection.

WINA as published ([13]) scores each input coordinate by its magnitude times the matching column norm of the weight matrix that consumes it (the gate projection in a gated MLP), keeps the top-kk per token, and fits a sparsity level for each projection with TEAL’s greedy search. We adapt this search to routed experts. It minimizes the reconstruction error of the routed-expert output, uses empirical quantile thresholds measured on the fitting data, steps in increments of 0.01, and stops at the first sparsity level that reaches the target budget. Neither WINA variant applies the orthogonal weight transform that WINA adopts to maximize its gains. CATS ([37]) computes the full gate projection and keeps channels by exact gate magnitude with a calibrated threshold; we also run a per-token top-kk version. Published HiRE trains the model with a group-sparse FFN; on frozen OLMoE we test only its screen, as a HiRE-style control that screens channels with an INT4 copy of the gate projection under our top-kk.

Hardware. Benchmark runs used a single RTX PRO 6000 Blackwell GPU, two A6000 GPUs, or four A5000 or RTX 3090 GPUs. In the multi-GPU setups, attention is split by tensor parallelism and routed experts by expert parallelism. Gemma 4 ran only on the first two setups. Before pooling results across these layouts, we ran a consistency test fixed in advance. Every layout replayed 192 held-out chat documents of 128 tokens each for the full model, our reader at 17.8% of reads, and Prox calibrated for 18%. For each sparse method, the 90% confidence interval of the change in its next-token KL divergence from the full model lay within 10% of that divergence, and its realized reads changed by less than 0.5 percentage points. For the full model, its KL divergence from the single-GPU RTX PRO 6000 run stayed below 10% of our reader’s divergence. So the layout does not materially change the numerics.

Sketch readpartialpartialfullfullfullfull
Channels computed exactly (%)7.0–7.512.0–12.520305070
Reads (% of full model)12.817.828.738.758.678.5

Table 6. Our reader reads part of the sketch at the two smallest budgets and all of it above. Each column is one budget. It gives the share of candidate channels computed exactly across each token’s routed experts and the resulting total reads, which include the ternary sketch, exact channels, recovery map, and metadata, averaged over the three post-trained models and three benchmarks.

Read accounting and profiler check

For each token and MoE layer, our reader is charged for every weight and parameter byte it loads from memory. This includes the ternary sketch codes and scales read during screening, the full-precision gate, up, and down rows of the kk kept channels, the recovery map’s coefficient vectors for the skipped channels and its shared basis UU, the precomputed per-channel norms, and the Hadamard rotation signs. Every read is counted in whole 32-byte hardware sectors at batch one, without modeling cache reuse across tokens. Every baseline is charged under the same sector model, in the weight layout that needs the fewest reads, including its thresholds, low-bit copies, and other metadata. For selectors with thresholds, we report the reads they realize on each benchmark, counted on the GPU. Table 5 breaks down our reader’s reads by component at the four main budgets.

ModelSketch readSketchExact channelsRecoveryMetadataTotal
Qwen3-Thinkingpartial4.867.290.660.0412.85
partial4.8612.300.630.0417.83
full7.9920.000.590.1028.68
full7.9930.000.540.1038.62
Gemma 4partial4.787.490.540.0312.84
partial4.7812.500.520.0317.83
full7.8619.990.490.0828.42
full7.8630.010.450.0738.40
Qwen3.5partial5.086.980.740.0412.84
partial5.0812.040.720.0417.87
full8.3320.000.680.1129.11
full8.3330.000.620.1039.06

Table 5. Most of our reader’s reads go to exact channels, and the sketch takes most of the rest. Routed-expert reads per token at batch one, as a percentage of the full model, by component at the four main budgets. At the two smallest budgets, the partial screen reads part of the ternary sketch.

To check that the sector count matches what the hardware actually transfers, we profiled 12 of Qwen3-Thinking’s 48 MoE layers with NVIDIA Nsight Compute. The profiled kernels ran on weights, recovery maps, and hidden states captured from a live decoding step, with the cache flushed before each kernel launch. Table 7 reports the measured DRAM reads of the MoE block without the router. At batch one, our reader’s DRAM traffic is 18.3% of the full model’s, close to the 17.8% our sector count gives. At larger batches the share rises, to 21.8% at batch four and 25.7% at batch eight, because tokens in a batch select partly different channels.

Batch size148
Full model75.6229.1359.3
Our reader13.949.992.3
Share18.3%21.8%25.7%

Table 7. At batch one, the profiler measures 18.3% of the full model’s DRAM reads, close to the 17.8% we count. Measured DRAM reads per MoE layer on Qwen3-Thinking in MB, averaged over 12 layers, without the router, with a cold cache.

Calibrated thresholds read less than their budget

§ 4.1 counts 72 benchmark runs: Prox and our reader’s threshold version, each at four budgets. Table 8 extends this count to every selector we ran with a calibrated threshold on the post-trained models, 24 threshold settings on all nine model-benchmark pairs. All 216 runs read less than the budget they were calibrated for. For Prox and our reader’s threshold version, this budget is the read target of the calibration fit. For TEAL and WINA’s score with a threshold, it is the target share of inputs kept. For CATS, it is the reads of its top-kk version at the same target sparsity.

MethodCalibrated for (%)MinMeanMax
CATS53.351.152.153.0
66.763.865.166.0
Our reader, calibrated12.811.211.812.4
17.815.216.517.1
28.724.526.527.7
38.732.935.837.2
Prox as published, calibrated13.05.68.510.1
18.09.912.914.7
28.018.922.524.6
38.029.632.634.1
58.047.349.851.0
78.067.170.071.4
TEAL as published, calibrated15.06.48.911.0
20.09.812.615.3
30.019.922.624.9
40.032.635.036.2
50.043.746.247.4
60.054.756.958.0
WINA score, calibrated15.05.47.89.6
20.08.211.213.9
30.017.220.723.5
40.032.034.335.8
50.043.546.147.7
60.054.857.158.2

Table 8. Every calibrated threshold reads less than the budget it was calibrated for. Each row is one selector at one calibrated budget. The columns give the lowest, mean, and highest realized routed-expert reads (%) over the nine model-benchmark pairs.

Calibrating on the model's own generations

UltraChat assistant turns contain no reasoning traces, so we also fitted Prox’s thresholds on each full model’s own reasoning, for targets of 13.4, 15, 18.4, and 20%. The full model wrote one thinking-mode trace, up to 32,768 tokens, for each of 1,024 UltraChat first user turns. These turns lie outside every fitting, tuning, and evaluation split, and none shares a 13-gram with a benchmark prompt. The traces average 1,669 generated tokens on Gemma 4 and 2,302 on Qwen3-Thinking. As in our chat calibration, Prox’s recipe uses 30 of these sequences and 20,480 evenly spaced positions. Each sequence is a prompt plus its trace, cut at a random point and capped at 8,192 tokens.

Prox with these trace-calibrated thresholds still reads less than its targets on Gemma 4 GPQA-Diamond (four samples per question): 10.5, 11.6, 14.0, and 15.2% of reads. It scores 0.0, 0.3, 23.9, and 45.1, and 100, 100, 65, and 35% of its answers hit the 32,768-token limit. For comparison, Prox with its threshold calibrated on chat for 18% scores 11.7 at 13.3% of reads. On Gemma 4 IFEval, Prox with a trace-calibrated threshold scores 31.2 at 14.9% of reads, against 11.3 with chat calibration.

On Qwen3-Thinking GPQA-Diamond, Prox with trace-calibrated thresholds scores 57.1 at 9.6% of reads and 62.8 at 11.2%, against 59.7 at 9.9% with chat calibration. On IFEval it scores 82.1, against 82.8. Calibrating on the model’s own reasoning therefore neither stops the reads from drifting nor prevents Gemma 4’s collapse into repetition loops.

What a calibrated threshold’s channel count follows

With a calibrated threshold, the number of channels kept changes from token to token. This appendix asks what that number follows, and whether the variation spends reads where tokens need them. Unless stated otherwise, we use Prox’s INT4 score on Qwen3-Thinking with the thresholds from our main evaluation, calibrated on UltraChat for 13% and 18% of reads. Where stated, we also use thresholds calibrated on the model’s own reasoning (Appendix § 11.§ 1). Gemma 4 repeats the context test, and Gemma 4 and Qwen3.5 repeat the equal-read and generation tests, with thresholds calibrated on UltraChat. Every variant ranks channels by the same score; only the rule that sets each token’s count changes.

Setup. We use 48 GPQA-Diamond reasoning traces written by the full Qwen3-Thinking model and teacher-force 256-token windows starting at answer positions 0, 2,000, and 6,000. The full model processes the question and the earlier answer. Inside the window, each variant runs on hidden states from its own sparse layers, and we measure its next-token KL divergence from the full model. We compare the threshold with two controls. Per-layer fixed counts give every token in a layer the same count, set so that reads in the window match the threshold’s; a two-pass procedure matches them to a relative residual below 0.05%. Shuffled counts randomly permute the threshold’s per-token counts across positions. To measure how much each token needs extra channels, a separate single-step experiment scores 576 tokens, four positions in each of the three windows of 48 traces. Each token is scored after a full-model prefix at 0.5, 0.7, 1, 1.4, and 2 times the top-kk count, and its need is how much its KL divergence falls from 0.5 to 2 times the count.

The earlier answer sets the count. Table 9 takes the window at position 6,000 and changes only what comes before it. Removing the earlier answer, so that the window follows the question directly, raises the kept count by 56.6 ± 2.7%. This brings it to 89% of the count at the start of the answer, where the tokens themselves differ. Among identical token IDs, Prox keeps 41% fewer channels at position 6,000 than at the start, and the count falls equally in every band of the full model’s next-token loss. On Gemma 4, whose reads fall less (from 10.5% to 9.7%), removing the earlier answer raises the kept count by 12.5 ± 0.5%. With thresholds calibrated on Qwen3-Thinking’s own traces, reads fall similarly, from 11.3% to 7.7% over 6,000 tokens. Both the length and the content of the earlier answer matter. With the exact scores and chat-calibrated threshold of Figure 4a, the window keeps 30.1 ± 1.3% more channels after the question alone and 10.4 ± 0.6% more after the last 1,000 answer tokens. Placing an answer to a different question, also 6,000 tokens long, before the window instead of the model’s own answer raises the count by 19.3 ± 1.3%.

Context before the windowKept channelsChange
Question and all 6,000 earlier answer tokens9,983–
Question and the last 1,000 answer tokens11,423+15.0±1.2%+15.0 \pm1.2\%
Question only15,422+56.6±2.7%+56.6 \pm2.7\%
Start of the answer (different tokens)17,283

Table 9. The same tokens keep more channels when the earlier answer is removed. The first three rows change only the context before the 256-token windows at answer position 6,000 of Qwen3-Thinking GPQA-Diamond traces, under Prox’s threshold calibrated for 13%. Kept channels are per token, summed over 48 layers (of 294,912 candidates) and averaged over 48 traces; ±\pm is one paired SE. The last row shows the start of the answer, which has different tokens.

What the count follows on Qwen3-Thinking. In the single-step experiment, the threshold’s per-token count correlates with answer position at −0.57-0.57 (Spearman, at both budgets), with need at only −0.13-0.13 (13%) and −0.19-0.19 (18%), and with the full model’s next-token loss at −0.29-0.29 and −0.26-0.26. Need itself follows loss (+0.58+0.58, +0.68+0.68) rather than position (+0.09+0.09, +0.14+0.14). Even among tokens at the same position, the count correlates slightly negatively with need (−0.11±0.04-0.11 \pm0.04 and −0.12±0.05-0.12 \pm0.05 over the 12 positions) and with loss (−0.19±0.05-0.19 \pm0.05 and −0.16±0.06-0.16 \pm0.06). On this model, removing the dependence on answer position would therefore leave variation that slightly favors tokens with lower need. On each model’s own sparse states in the three windows, the count correlates with loss at

−0.16 to −0.23 on Qwen3-Thinking, −0.21 to −0.28 on Gemma 4, and 0.00 to +0.14 on Qwen3.5. Yet on Qwen3.5, variable counts still improve quality at equal reads (below), so loss alone does not decide whether the variation helps.

Equal reads. Does the variation help at equal reads? Pooled over the three windows, per-layer fixed counts lower KL divergence relative to the chat-calibrated threshold by 3.3±1.0%3.3 \pm1.0\% at 13% and 3.4±1.2%3.4 \pm1.2\% at 18%, and relative to the trace-calibrated threshold by 4.0±0.9%4.0 \pm0.9\% and 2.9±1.2%2.9 \pm1.2\%. On Gemma 4 at 13%, they lower it by 12.5±1.5%12.5 \pm1.5\%. On Qwen3.5 they raise it, by 10.5±1.1%10.5 \pm1.1\% at 13% and 14.8±1.2%14.8 \pm1.2\% at 18%, and the threshold is ahead in all three windows at both budgets. So on Qwen3-Thinking and Gemma 4 reasoning the variation hurts, and on Qwen3.5 it helps. The threshold’s counts still carry information on every model: shuffling them across positions raises pooled KL on each. On held-out chat, the distribution the threshold was calibrated on, fixed counts instead raise KL by 6.5±1.9%6.5 \pm1.9\% and 9.2±2.2%9.2 \pm2.2\%.

Moving reads across positions. Making each layer’s count constant across the three windows barely changes KL at equal pooled reads. We compare per-window fixed counts, matched to the threshold version’s reads in each window, with counts held at their three-window mean. At equal pooled reads, the constant counts change KL by −0.3±0.5%-0.3 \pm0.5\% and −0.2±0.6%-0.2 \pm0.6\% on Qwen3-Thinking, +0.1±0.8%+0.1 \pm0.8\% on Gemma 4, and +1.0±0.5%+1.0 \pm0.5\% (just beyond two SE) and +0.1±0.7%+0.1 \pm0.7\% on Qwen3.5. A reversed schedule, which reflects each layer’s counts about their three-window mean, is worse: +6.8±0.7%+6.8 \pm0.7\% and +7.1±1.1%+7.1 \pm1.1\% on Qwen3-Thinking, and +4.2±0.7%+4.2 \pm0.7\% and +3.5±0.8%+3.5 \pm0.8\% on Qwen3.5.

Predicting need. An oracle that knows every token’s loss could halve KL divergence at the same total reads, but a predictor of need does not get close. A per-layer ridge regression from the token’s position and summary statistics of its screened channel scores (total and largest score mass, share in the top channels, entropy, and count above the threshold) explains about 20% of the variance in log KL. Our offline estimate of the allocation it implies is worse than fixed counts.

Generation. Qwen3-Thinking IFEval is the benchmark where a threshold beats top-kk at its realized reads. There, Prox keeps its top-kk on inputs and uses a channel threshold calibrated for 13%. With this threshold, it scores 84.4 at 10.0% of reads. With per-layer fixed counts matched to the threshold’s per-layer budgets, it scores 77.6 at 10.3% (−4.8±1.8-4.8 \pm1.8), and with the same count in every layer, 72.8 at 10.3%. On GPQA-Diamond (one sample), it scores 59.1 at 9.9% with the threshold and 56.6 at 9.9% with fixed counts (−2.5±2.8-2.5 \pm2.8, a tie). Per-token counts thus help on IFEval but show no detectable gain on GPQA-Diamond. IFEval answers are shorter than GPQA-Diamond reasoning and fail on a single violated instruction, which may make a few tokens decisive; we did not test this.

On Qwen3.5, with its threshold calibrated for 13%, Prox scores 89.5 at 9.7% of reads on IFEval, against 88.5 at 9.7% with fixed counts (−0.9±1.3-0.9 \pm1.3, a tie). On GPQA-Diamond (one sample), it scores 79.8 at 9.4%, against 72.7 at 9.1% (−7.1±2.7-7.1 \pm2.7). Without per-token counts, Qwen3.5’s answers grow longer, 16.2k against 11.3k tokens on average, and 5.1% of them hit the length limit, against 1.0%.

On Gemma 4, removing the per-token variation does not prevent the collapse. On IFEval, with per-layer fixed counts matched to the threshold’s per-layer budgets, Prox scores 0.0 at 10.1% of reads, and 99.8% of its answers stop at the length limit. Prox’s threshold version also scores 0.0 at 10.1%; replacing only its input threshold with top-kk gives 3.5 (Appendix § 13.§ 2). With the same count in every layer, Prox’s top-kk version scores 16.5 at 11.7%. On GPQA-Diamond, Prox with fixed counts also scores 0.0 at 9.8% of reads with every answer stopped at the limit, as does its threshold version (0.0 at 9.9%). Gemma 4’s collapse therefore comes with the threshold’s calibrated per-layer budgets or the low reads they settle at, rather than with per-token variation; this test cannot separate the two.

Every keep-rule comparison

Tables 10 and 12 list every paired comparison of a selector’s threshold and top-kk versions, and Tables 11 and 13 repeat them at realized reads; Figure 5 summarizes the 43 at 13% and 18% budgets. Each row fixes one channel score and compares the two versions at the budget the threshold was calibrated for. When the threshold version’s realized reads differ from that budget by more than one percentage point, the top-kk version is also rerun at those reads (Figure 8).

Figure 8. Figure 5 with each top-kk version run at the reads its threshold version actually used. Each dot is one comparison, as in Figure 5. When a threshold version read more than one percentage point away from its calibrated budget, the top-kk version is rerun at those realized reads. The figure has 41 comparisons at 13% and 18% budgets; four TEAL comparisons at heavy sparsity, where both versions collapse, are left out. Threshold versions read less than their budget, so this rerun lowers the top-kk version’s reads too, yet the top-kk version matches or beats the threshold version in 37 of 41.

In the 23 generation-task comparisons, the top-kk version leads beyond two standard errors in 8 and the threshold version in none. Two TEAL comparisons at heavy sparsity, where both versions fall below 10% of the full model, are marked and not counted. We fixed this set of comparisons in advance, not by outcome: Prox’s score and its output-scaled variant on IFEval at 13% and on GPQA-Diamond at 13% for Gemma 4 and Qwen3-Thinking, TEAL’s score on IFEval at 18%, and GSM8K and BBH on OLMoE at both budgets. Table 18 covers the remaining pairs for Prox: over all nine model-benchmark pairs at 13% and 18%, its threshold version scores higher than its top-kk version in 2 of 18, both on Qwen3.5 GPQA-Diamond and by at most 1.5 points. In the 20 likelihood-task comparisons on OLMoE at 13% and 18%, the top-kk version leads in 2 and the threshold version in 3. At the 38% budget, Prox’s top-kk version keeps 50% of inputs and its threshold version 30%, so the two differ in more than the keep rule. These 10 comparisons (top-kk version ahead in 3, threshold version in none) appear in parentheses and are left out of every count. Table 14 pools each model’s counted comparisons.

ModelAt the calibrated budget: ComparisonsAt the calibrated budget: τ−k\tau-kAt realized reads: ComparisonsAt realized reads: τ−k\tau-k
Qwen3-Thinking4−0.3±1.0-0.3\pm1.04+4.5±1.1+4.5\pm1.1
Qwen3.53−1.7±0.7-1.7\pm0.72−0.2±0.8-0.2\pm0.8
OLMoE32−0.3±0.1-0.3\pm0.131+0.0±0.1+0.0\pm0.1
Gemma 44−41.5±1.3-41.5\pm1.34−13.3±1.1-13.3\pm1.1

Table 14. No model favors the threshold version at the calibrated budget. Threshold-version minus top-kk-version accuracy, pooled over each model’s counted comparisons with inverse-variance weights, at the calibrated budget and at realized reads. The ±\pm is one SE that treats comparisons as independent; accounting for shared questions changes each SE by at most 0.1. The realized-read columns favor the threshold version (§ 2). Only Qwen3-Thinking leads beyond two SE there, a lead that comes from Prox on IFEval and reverses at the calibrated budget.

MethodBudget (%)GPQA-DIFEvalMMLU-Pro
Qwen3-Thinking
Full model10070.887.880.5
Our reader12.863.885.076.9
17.867.387.278.2
28.767.985.478.6
38.770.886.180.3
Prox score, top-kk13.563.885.874.9
18.564.086.376.3
28.569.386.379.3
Prox as published, calibrated1330.843.364.9
1859.782.876.3
2867.286.576.9
WINA as published, top-kk208.60.928.4
3058.679.573.8
TEAL as published, calibrated200.00.00.2
3035.550.561.7
WINA score, calibrated200.00.00.0
3017.833.340.0
Gemma 4
Full model10079.993.585.1
Our reader12.869.488.080.4
17.876.692.682.5
28.777.994.584.2
38.780.294.584.4
Prox score, top-kk13.556.357.167.3
18.573.790.882.0
28.577.593.983.9
Prox as published, calibrated130.00.00.0
1811.711.316.7
2872.391.781.9
WINA as published, top-kk2014.640.921.8
3069.694.378.8
TEAL as published, calibrated200.00.00.0
3022.538.328.6
WINA score, calibrated200.00.00.0
3025.827.227.9
Qwen3.5
Full model10083.692.185.5
Our reader12.881.390.083.7
17.883.391.985.2
28.782.792.484.9
38.784.591.986.2
Prox score, top-kk13.579.991.383.0
18.581.691.584.4
28.583.391.785.5
Prox as published, calibrated1380.689.382.8
1883.191.184.1
2883.391.785.7
WINA as published, top-kk2015.437.052.2
3070.188.583.3
TEAL as published, calibrated2010.717.449.1
3072.788.783.6
WINA score, calibrated204.84.629.0
3058.281.781.0

Table 18. Accuracy of every selector on every model and task. Scores in percent at heavy sparsity. Budget is the mean reads for selectors without calibration data and the calibrated budget for the others. GPQA-Diamond (GPQA-D) averages four samples per question.

In the generation-task comparisons with Prox’s score and its output-scaled variant, both versions select inputs by top-kk, so only the channel threshold is replaced. In the comparisons with TEAL’s score, the threshold on inputs is replaced by top-kk.

Applying Prox to OLMoE

Prox ([43]) was evaluated on dense models, and five of its six tasks are multiple-choice likelihood tasks. We apply Prox to the routed experts of OLMoE-1B-7B-0924 and evaluate the same five tasks in lm-eval 0.4.3 ([25]) (Table 15). Each task runs under two prompt settings. In the first, as in Prox, the whole sequence is sparse. In the second, the full model computes the prompt except its last token, so only the positions that predict an answer token are sparse.

TaskShotsMetricQuestionsFull model (%)
MMLU5accuracy14,04252.71
HellaSwag10normalized accuracy10,04279.40
ARC-E0accuracy2,37678.11
ARC-C25normalized accuracy1,17255.20
PIQA0accuracy1,83879.98

Table 15. Prox’s five likelihood tasks on OLMoE. Few-shot examples, metric, number of questions, and full-model accuracy for each task.

We fit the thresholds with Prox’s sequential procedure. Layers are calibrated in order, with all earlier layers already sparse, and each layer gets one input threshold and one channel threshold from empirical quantiles of up to 200,000 activation samples pooled across its routed experts. Three settings differ from Prox’s dense setup. First, OLMoE is a base model, so we calibrate on 20,480 token positions from 30 Pile documents instead of Alpaca text. Second, we set the budgets (13, 18, and 38%) in charged sector reads instead of through Prox’s latency model. Third, the threshold version’s input threshold is fitted to keep 30% of inputs at all three budgets, while the top-kk version keeps 30% of inputs at 13% and 18% and 50% at 38%.

At realized reads and the 13% and 18% budgets, with the whole sequence sparse as in Prox, the threshold version leads in 1 of 10 comparisons and the top-kk version in 2. With the prompt computed by the full model, the threshold version leads in 2 and the top-kk version in none (Table 12 and Table 13).

Table 12: Keep-rule comparisons on Prox’s five likelihood tasks on OLMoE. Prox’s threshold and top-kk versions, with the prompt either sparse or computed by the full model. Columns follow Table 10. At 38%, the top-kk version keeps 50% of inputs and the threshold version 30%; these verdicts are in parentheses and not counted.

PromptTaskBudget (%)τ\taukkτ−k\tau-k
full modelARC-C1349.750.0−0.3±0.9-0.3\pm0.9=
ARC-E1373.973.2+0.7±0.6+0.7\pm0.6=
HellaSwag1375.474.9+0.6±0.3+0.6\pm0.3τ\tau
MMLU1350.350.3−0.0±0.3-0.0\pm0.3=
PIQA1377.776.4+1.3±0.7+1.3\pm0.7=
ARC-C1851.951.4+0.5±0.9+0.5\pm0.9=
ARC-E1875.875.8+0.0±0.6+0.0\pm0.6=
HellaSwag1877.276.6+0.6±0.2+0.6\pm0.2τ\tau
MMLU1850.450.6−0.2±0.3-0.2\pm0.3=
PIQA1878.677.5+1.1±0.6+1.1\pm0.6=
ARC-C3855.155.3−0.2±0.7-0.2\pm0.7(=)
ARC-E3877.878.6−0.8±0.4-0.8\pm0.4(=)
HellaSwag3878.778.9−0.2±0.2-0.2\pm0.2(=)
MMLU3851.652.1−0.5±0.3-0.5\pm0.3(=)
PIQA3879.280.1−0.9±0.5-0.9\pm0.5(=)
sparseARC-C1344.643.0+1.6±1.0+1.6\pm1.0=
ARC-E1371.470.3+1.1±0.7+1.1\pm0.7=
HellaSwag1373.673.3+0.3±0.3+0.3\pm0.3=
MMLU1335.739.5−3.8±0.4-3.8\pm0.4kk
PIQA1376.076.1−0.2±0.8-0.2\pm0.8=
ARC-C1849.646.8+2.8±1.0+2.8\pm1.0τ\tau
ARC-E1874.973.7+1.2±0.6+1.2\pm0.6=
HellaSwag1876.476.0+0.4±0.3+0.4\pm0.3=
MMLU1842.143.1−1.0±0.4-1.0\pm0.4kk
PIQA1877.878.4−0.6±0.7-0.6\pm0.7=
ARC-C3855.254.4+0.8±0.8+0.8\pm0.8(=)
ARC-E3877.477.9−0.5±0.5-0.5\pm0.5(=)
HellaSwag3878.879.4−0.6±0.2-0.6\pm0.2(kk)
MMLU3849.550.7−1.1±0.3-1.1\pm0.3(kk)
PIQA3879.080.1−1.1±0.5-1.1\pm0.5(kk)

Table 12.

PromptTaskBudget (%)τ\taukkτ−k\tau-k
full modelARC-C1349.750.0†−0.3±0.9-0.3\pm0.9=
ARC-E1373.973.2†+0.7±0.6+0.7\pm0.6=
HellaSwag1375.474.9†+0.6±0.3+0.6\pm0.3τ\tau
MMLU1350.349.7+0.5±0.3+0.5\pm0.3=
PIQA1377.776.4†+1.3±0.7+1.3\pm0.7=
ARC-C1851.952.8−0.9±0.9-0.9\pm0.9=
ARC-E1875.875.7+0.1±0.5+0.1\pm0.5=
HellaSwag1877.276.6†+0.6±0.2+0.6\pm0.2τ\tau
MMLU1850.450.6−0.2±0.3-0.2\pm0.3=
PIQA1878.678.0+0.6±0.6+0.6\pm0.6=
ARC-C3855.154.9+0.2±0.8+0.2\pm0.8(=)
ARC-E3877.878.7−1.0±0.4-1.0\pm0.4(kk)
HellaSwag3878.779.0−0.2±0.2-0.2\pm0.2(=)
MMLU3851.651.8−0.2±0.3-0.2\pm0.3(=)
PIQA3879.279.7−0.4±0.5-0.4\pm0.5(=)
sparseARC-C1344.643.0†+1.6±1.0+1.6\pm1.0=
ARC-E1371.470.3†+1.1±0.7+1.1\pm0.7=
HellaSwag1373.673.3†+0.3±0.3+0.3\pm0.3=
MMLU1335.739.5†−3.8±0.4-3.8\pm0.4kk
PIQA1376.076.1†−0.2±0.8-0.2\pm0.8=
ARC-C1849.646.8†+2.8±1.0+2.8\pm1.0τ\tau
ARC-E1874.973.7†+1.2±0.6+1.2\pm0.6=
HellaSwag1876.476.0†+0.4±0.3+0.4\pm0.3=
MMLU1842.143.1†−1.0±0.4-1.0\pm0.4kk
PIQA1877.878.4†−0.6±0.7-0.6\pm0.7=
ARC-C3855.254.9+0.3±0.8+0.3\pm0.8(=)
ARC-E3877.477.9†−0.5±0.5-0.5\pm0.5(=)
HellaSwag3878.879.1−0.3±0.2-0.3\pm0.2(=)
MMLU3849.550.7†−1.1±0.3-1.1\pm0.3(kk)
PIQA3879.080.3−1.3±0.5-1.3\pm0.5(kk)

Table 13. The likelihood-task comparisons with each top-kk version run at its threshold version’s realized reads. Columns follow Table 10. †The threshold version read within one point of its budget, so the top-kk run at the budget is reused.

Repetition loops on Gemma 4

Prox selects in two stages: a threshold on input coordinates, then a threshold on expert channels. § 4.3 showed that Gemma 4 falls into repetition loops under Prox’s thresholds even at nearly equal reads. To find which stage causes the collapse, we replaced one threshold at a time with top-kk, at the 13% budget, on the first GPQA-Diamond sample of each question (realized reads in parentheses). With both thresholds, Prox scores 0.0 (10.9% of reads), and every answer hits the length limit. With top-kk on inputs and the threshold on channels, it scores 0.0 (11.4%). With the threshold on inputs and top-kk on channels, it scores 43.9 (12.4%), and with top-kk at both stages, 51.0 (13.0%). On IFEval, the same four settings score 0.0, 3.5, 43.1, and 45.8.

Replacing the channel threshold with top-kk therefore raises GPQA-Diamond accuracy by 43.9 to 44.9 points while adding only 1.5 to 1.6 percentage points of reads, and raises IFEval accuracy by 42.3 to 43.1 points. Replacing the input threshold changes accuracy by at most 7.1 points on either benchmark. So replacing the channel threshold, not the input threshold, is what avoids the collapse. Giving every token in a layer the same count, set to the average the threshold kept in that layer, does not avoid it either (Appendix § 12); that test cannot tell whether the calibrated layer budgets or the low reads they settle at drive the collapse.

The generation-task comparisons with each top-kk version run at its threshold version’s realized reads. Columns follow Table 10. The top-kk version is rerun only where the threshold version’s realized reads differ from the budget by more than one percentage point. †The threshold version read within one point of its budget, so the top-kk run at the budget is reused.

ModelTaskScoreBudget (%)τ\taukkτ−k\tau-k
Gemma 4GPQA-DProx, output-scaled1316.230.3−14.1±3.5-14.1\pm3.5 kk
Prox136.124.2−18.2±2.9-18.2\pm2.9 kk
IFEvalProx, output-scaled1323.134.0−10.9±2.1-10.9\pm2.1 kk
Prox133.516.5−12.9±1.6-12.9\pm1.6 kk
TEAL180.00.2−0.2±0.2-0.2\pm0.2 –
OLMoEBBHProx, output-scaled1325.425.3+0.1±0.6+0.1\pm0.6 =
Prox1326.126.2†26.2^\dagger−0.1±0.6-0.1\pm0.6 =
TEAL133.03.0+0.1±0.2+0.1\pm0.2 –
Prox, output-scaled1828.528.0+0.6±0.6+0.6\pm0.6 =
Prox1828.128.5†28.5^\dagger−0.4±0.5-0.4\pm0.5 =
TEAL1810.812.9−2.1±0.4-2.1\pm0.4 kk
GSM8KProx, output-scaled139.610.4†10.4^\dagger−0.8±1.0-0.8\pm1.0 =
Prox138.39.9†9.9^\dagger−1.5±0.9-1.5\pm0.9 =
TEAL131.82.2†2.2^\dagger−0.4±0.4-0.4\pm0.4 =
Prox, output-scaled1812.111.8+0.2±1.0+0.2\pm1.0 =
Prox189.611.0†11.0^\dagger−1.4±0.9-1.4\pm0.9 =
TEAL182.11.5+0.6±0.4+0.6\pm0.4 =
Qwen3.5IFEvalProx, output-scaled1391.191.9†91.9^\dagger−0.7±1.0-0.7\pm1.0 =
Prox1391.590.9†90.9^\dagger+0.6±1.1+0.6\pm1.1 =
TEAL183.00.0+3.0±0.7+3.0\pm0.7 –
Qwen3-ThinkingGPQA-DProx, output-scaled1362.657.1+5.6±3.0+5.6\pm3.0 =
Prox1357.153.5+3.5±3.3+3.5\pm3.3 =
IFEvalProx, output-scaled1385.485.2+0.2±1.5+0.2\pm1.5 =
Prox1384.372.8+11.5±1.9+11.5\pm1.9 τ\tau
TEAL180.00.0†0.0^\dagger+0.0±0.0+0.0\pm0.0 –

Table 11.

ModelTaskScoreBudget (%)τ\taukkτ−k\tau-k
Gemma 4GPQA-DProx, output-scaled1316.254.0−37.9±3.8-37.9\pm3.8 kk
Prox136.151.0−44.9±3.5-44.9\pm3.5 kk
IFEvalProx, output-scaled1323.163.6−40.5±2.3-40.5\pm2.3 kk
Prox133.545.8−42.3±2.2-42.3\pm2.2 kk
TEAL180.07.0−7.0±1.1-7.0\pm1.1 –
OLMoEBBHProx, output-scaled1325.426.1−0.7±0.6-0.7\pm0.6 =
Prox1326.126.2−0.1±0.6-0.1\pm0.6 =
TEAL133.05.4−2.4±0.3-2.4\pm0.3 kk
Prox, output-scaled1828.530.1−1.6±0.6-1.6\pm0.6 kk
Prox1828.128.5−0.4±0.5-0.4\pm0.5 =
TEAL1810.817.6−6.8±0.5-6.8\pm0.5 kk
GSM8KProx, output-scaled139.610.4−0.8±1.0-0.8\pm1.0 =
Prox138.39.9−1.5±0.9-1.5\pm0.9 =
TEAL131.82.2−0.4±0.4-0.4\pm0.4 =
Prox, output-scaled1812.111.0+1.1±1.0+1.1\pm1.0 =
Prox189.611.0−1.4±0.9-1.4\pm0.9 =
TEAL182.12.1+0.0±0.5+0.0\pm0.5 =
Qwen3.5IFEvalProx, output-scaled1391.191.9−0.7±1.0-0.7\pm1.0 =
Prox1391.590.9+0.6±1.1+0.6\pm1.1 =
TEAL183.010.4−7.4±1.5-7.4\pm1.5 kk
Qwen3-ThinkingGPQA-DProx, output-scaled1362.659.6+3.0±3.0+3.0\pm3.0 =
Prox1357.158.1−1.0±3.0-1.0\pm3.0 =
IFEvalProx, output-scaled1385.485.6−0.2±1.4-0.2\pm1.4 =
Prox1384.385.4−1.1±1.6-1.1\pm1.6 =
TEAL180.00.0+0.0±0.0+0.0\pm0.0 –

Table 10. Keep-rule comparisons on generation tasks. Each row fixes one model, task, score, and budget. It gives the accuracy of the threshold version (τ\tau) and the top-kk version (kk), their paired difference ± one standard error, and which version is ahead (= within two standard errors, – both collapsed).

The collapse is not limited to 13%: with its threshold calibrated for 18%, Prox finishes its reasoning in only 132 of 792 GPQA-Diamond answers, against 787 of 792 for the full model. At the 13% budget, a per-token floor on channels does not prevent the collapse either. When every token keeps at least 4% of its candidate channels on top of the calibrated threshold, Prox still scores only 2.0 on GPQA-Diamond (11.1% of reads). Finally, when set to keep all channels, our implementations of Prox and TEAL match the full model on held-out chat, which rules out a gross implementation error in these baselines.

Our reader with a calibrated threshold, and CATS

Our reader. To test the keep-rule swap on our own reader, we replace its per-token top-kk with one threshold per layer on the same score. The threshold is fitted on UltraChat with Prox’s recipe, so that it keeps kk channels per token on average. On the benchmarks, this threshold version reads 0.4 to 5.8 percentage points less than its budget. Of 36 paired comparisons, across three post-trained models, three benchmarks, and four budgets, 34 are ties and none favors the threshold version. Both remaining comparisons favor the top-kk version, on Gemma 4 at 12.8% of reads. There the threshold version reads 11.7% and scores 62.4 against 69.4 on GPQA-Diamond (−7.1±1.7-7.1 \pm1.7) and 73.2 against 80.4 on MMLU-Pro (−7.2±1.2-7.2 \pm1.2). It also hits the length limit on 15 to 17% of answers, against 4 to 6% for the top-kk version.

CATS. CATS computes the gate projection exactly and keeps channels by gate magnitude. We compare its threshold and top-kk versions at two budgets, about 53% and 67% of reads, on all nine model-benchmark pairs. Of the 18 comparisons, 16 are ties. The threshold version leads in the other two, both on Gemma 4 at 53%: by 5.8±1.75.8\pm1.7 points on GPQA-Diamond and 5.5±1.15.5\pm1.1 points on MMLU-Pro, while reading slightly less (52.7% and 52.8%, against 53.3%).

Calibrated layer budgets and calibrated recovery

Layer budgets. We compare our reader’s uniform layer budget with non-uniform per-layer budgets, derived either from the weights or from calibration data, at the same total reads of 17.8%. Calibrated minus uniform, accuracy changes by −0.76±1.10-0.76\pm1.10 on GPQA-Diamond (four samples) and +1.11±1.01+1.11\pm1.01 on IFEval on Gemma 4, and by −1.77±1.18-1.77\pm1.18 and −1.85±1.31-1.85\pm1.31 on Qwen3-Thinking. On OLMoE, calibrated layer budgets lower next-token KL divergence by about 0.02 on held-out Pile text from the calibration distribution. Yet they change GSM8K accuracy by at most 0.1 points and BBH accuracy by −1.23±0.50-1.23\pm0.50 and −0.28±0.50-0.28\pm0.50 at the two smallest budgets. The KL gain is not confined to the calibration data. On Qwen3-Thinking with our reader at 17.8% of reads, we measure teacher-forced KL divergence on 144 windows from the full model’s GPQA-Diamond answers, with every layer sparse inside each window, fitting budgets on half of the answers and evaluating on the other half. Relative to the uniform budget, the chat-calibrated budgets lower KL divergence by 15.7±1.9%15.7\pm1.9\%, close to the 18.7±2.1%18.7\pm2.1\% of budgets fitted on GPQA-Diamond answers, while budgets derived from the weights change it by +2.4±1.7%+2.4\pm1.7\%. Lower KL divergence on reasoning text thus did not turn into a detected benchmark gain.

Recovery. We also fit the low-rank recovery map on calibration text by minimizing next-token KL divergence from the full model, at the same reads as our weight-only map. We make 18 comparisons on the three post-trained models: GPQA-Diamond, IFEval, MMLU-Pro, and GSM8K at 17.8% of reads, plus IFEval and MMLU-Pro at 12.8%. The calibrated map never beats the weight-only map beyond two standard errors, and the weight-only map leads on Qwen3-Thinking MMLU-Pro at 12.8% (+2.1 ± 1.0). On OLMoE, the calibrated map lowers next-token KL divergence on held-out Pile text from its calibration distribution (0.158 against 0.200 at 17.8% of reads) but gives no benchmark gain beyond two standard errors.

What each part of the reader contributes

Score with the full sketch. When the reader reads the full ternary sketch (from 28.7% of reads upward), each 128-coordinate block also comes with the mean-squared quantization error ρb\rho_b of its ternary code. This gives each screened inner product an error variance σ2=∑bρb∥(Rx)b∥2\sigma^2 = \sum_b \rho_b \lVert(Rx)_b\rVert^2 (all norms are Euclidean). We model the true gate and up pre-activations as independent Gaussians centered at g^j\hat{g}_j and v^j\hat{v}_j, with variance σ2\sigma^2 computed from each row’s own residuals. Five-node Gauss–Hermite quadrature then gives the mean μj\mu_j and second moment mjm_j of the gated activation aj=ϕ(gjTx)(vjTx)a_j = \phi(g_j^\mathsf{T}x)(v_j^\mathsf{T}x). Let nj=∥dj∥n_j = \lVert d_j\rVert, and let qj=Ubjq_j = Ub_j be channel jj’s low-rank recovery vector, with component βj∥=djTqj/nj\beta_j^\parallel= d_j^\mathsf{T}q_j/n_j along djd_j and orthogonal remainder of norm βj⊥=∥qj−βj∥dj/nj∥\beta_j^\perp= \lVert q_j - \beta_j^\parallel d_j/n_j\rVert. We score channel jj by the expected squared output error of skipping it after recovery,

sj=wj2[nj2(mj−μj2)+(njμj−βj∥pj)2+(βj⊥pj)2].s_j = w_j^2\left[n_j^2(m_j-\mu_j^2) + (n_j\mu_j-\beta_j^\parallel p_j)^2 + (\beta_j^\perp p_j)^2\right].

Output energy and KL divergence. Next-token KL divergence closely follows how much of the routed output’s energy the kept channels preserve, E=1−∥y−yK∥2/∥y∥2E = 1 - \lVert y-y_K\rVert^2/\lVert y\rVert^2. Here yy is the full routed-expert output and yKy_K the output of the kept channels alone. We measure EE on all 16 OLMoE layers, at 3,000 positions per layer, for four scores under per-token top-kk, each at four read budgets without recovery. The four scores are our output-scaled ternary score, two versions of Prox’s score using each token’s largest 30% and 50% of input coordinates, and an INT4 screen of the gate projection. Across these 16 settings, EE and held-out next-token KL divergence have a rank correlation of −0.96-0.96, and KL divergence scales roughly as (1−E)1.2–1.9(1-E)^{1.2\text{--}1.9}.

Ablations. Table 16 compares variants of our reader on 192 held-out OLMoE documents, by next-token KL divergence from the full model at equal total reads.

Reads (%)13.518.528.538.4
Full sketch, no recovery0.5530.2770.1070.051
Full sketch with recovery0.4880.2450.0960.047
Partial sketch with recovery0.3100.1870.0900.053
Prox’s 4-bit copy, output-scaled top-k, recovery0.3050.1870.0700.035

Table 16. Partial screening helps at small budgets, and recovery helps at every budget. Next-token KL divergence from the full model on held-out OLMoE documents at equal reads (lower is better). Each row is one variant of our reader.

Our reader uses the partial sketch at the two smallest budgets and the full sketch at higher budgets. We also replace our ternary sketch (1.0 GB next to the OLMoE checkpoint) with Prox’s 4-bit weight

copy (2.16 GB), keeping our output-scaled top-kk and weight-only recovery. This variant also needs no calibration data. It matches our reader at the two smallest budgets and has lower KL divergence from 28.5% of reads upward, at about twice the resident auxiliary memory.

Quantized checkpoints. Our reader also works on quantized checkpoints. On OLMoE held-out documents, with reads charged in BF16-equivalent routed bytes including quantization scales, our reader on a GPTQ ([23]) 4-bit checkpoint reaches a next-token KL divergence of 0.045 at 18.9% of reads. AWQ ([40]) 3-bit weights alone reach 0.075 at 19.7% (paired difference −0.030 ± 0.003). At 21.5% of reads our reader reaches 0.033, against 0.019 for GPTQ 4-bit weights alone at 26.0%. On a quantized checkpoint, the ternary sketch takes a larger share of resident memory: a 4-bit checkpoint plus the sketch occupies about 34% of the BF16 routed bytes, against 20% for 3-bit weights.

Design of the screen. Table 17 tests the screen’s design choices on development data. Removing the Hadamard rotation, or forcing each routed expert to keep a fixed share of channels, raises KL divergence on OLMoE. For the partial screen, we chose to read 80 of every 128 rotated coordinates per block on OLMoE development data. There, 80 gives the lowest KL divergence, or lies within about 0.004 of it, at each of three budgets. On Gemma 4 development data, 80 is lowest at two budgets and within 0.0003 of the lowest at the third.

Every model and task

Figures 9 and 10 show accuracy against reads for every model and benchmark, with calibrated selectors at their calibrated budgets and at their realized reads. Table 18 lists the scores at heavy sparsity. In a rerun of GPQA-Diamond with eight samples per question, our reader at 17.8% of reads trails the full model by 3.9 ± 1.0 points on Gemma 4 and 4.6 ± 1.4 points on Qwen3-Thinking.

Answer length. Our reader does not trade speed per token for longer answers. At 17.8% of reads, its mean answer length stays within about 15% of the full model’s on every model-benchmark pair. On Qwen3-Thinking, Gemma 4, and Qwen3.5, it changes by −3, +3, and +1% on GPQA-Diamond, +15, +9, and −12% on IFEval, and −14, +14, and −10% on MMLU-Pro. At most 1.3% of its answers stop at the length limit on any pair, and the same holds for the full model.

Figure 10. Figure 9 with calibrated selectors placed at the reads they actually used. Selectors with calibrated thresholds move left, because they read less at test time than the budget they were calibrated for.

Figure 9. Accuracy against reads on every model and task. Each panel is one model-benchmark pair, and its vertical axis runs from zero to above the full model’s accuracy (dotted line). Blue solid lines use no calibration data. Warm dashed lines are selectors with thresholds fitted on calibration data, placed at the budget they were calibrated for. WINA as published (red solid) keeps a top-kk per token, with a sparsity level for each projection fitted on calibration data as in TEAL. GPQA-Diamond averages four samples per question, IFEval reports prompt-level strict accuracy, and MMLU-Pro uses a fixed subset of 1,000 questions.

Figure 11. Figure 1 with calibrated selectors placed at the reads they actually used. Each curve is the worst accuracy over the nine model-benchmark pairs, as in Figure 1, but each calibrated selector sits at the mean reads it used on the pair that gives its worst accuracy, instead of the budget it was calibrated for. Selectors with calibrated thresholds read less than that budget (§ 4.1), so their points move left.

Baseline comparison on OLMoE

Table 19 compares our reader with nine prior channel selectors on OLMoE, by next-token KL divergence from the full model and top-1 token agreement on 192 held-out Pile documents (12,096 decoded positions), with the prompt processed by the full model. Calibrated baselines are fitted on disjoint Pile text for budgets near our reader’s and reported at the reads they realize. Prox divides each read budget between its input screen and the channels it computes; we report both Prox’s published division and the division with the lowest KL divergence on development data. At each budget, our reader has lower KL divergence than every baseline evaluated there, and it reads less than Prox’s nearest point at the 13% and 18% budgets. Among baselines evaluated at 13% to 18% of reads, every one except Prox has at least four times our reader’s KL divergence.

BudgetMethodReads (%)KL (↓\downarrow)Top-1 (↑\uparrow)
∼\sim13%Our reader12.80.3340.751
∼\sim13%Prox, calibrated, division tuned on development data13.20.4880.709
∼\sim13%Prox as published, calibrated13.70.4620.723
∼\sim13%HiRE-style gate screen12.81.7090.469
∼\sim18%Our reader17.80.1990.808
∼\sim18%Prox, calibrated, division tuned on development data18.30.2980.774
∼\sim18%TEAL as published, calibrated18.51.1090.613
∼\sim18%WINA score, calibrated17.61.2950.583
∼\sim18%R-Sparse18.11.1320.577
∼\sim18%HiRE-style gate screen17.80.8600.602
∼\sim28%Our reader28.50.0960.859
∼\sim28%Prox as published, calibrated29.10.1360.842
∼\sim28%TEAL as published, calibrated29.10.3780.760
∼\sim28%WINA score, calibrated28.50.4610.731
∼\sim28%R-Sparse28.80.4450.717
∼\sim28%HiRE-style gate screen27.80.3760.720
∼\sim38%Our reader38.40.0470.900
∼\sim38%Prox, calibrated, division tuned on development data38.70.0570.891
∼\sim38%Prox as published, calibrated38.30.0600.892
∼\sim38%HiRE-style gate screen37.80.2030.783
∼\sim38%CHESS40.40.6430.663
∼\sim38%CATS40.51.1130.555
∼\sim38%M-COUNTDOWN40.91.1230.612
∼\sim38%Park et al. [52]40.41.1200.557

Table 19. On OLMoE, our reader is closer to the full model than every baseline evaluated at each budget. Next-token KL divergence from the full model (lower is better) and top-1 agreement on 192 held-out Pile documents, at the reads each method realizes. HiRE-style screens channels with an INT4 copy of the gate projection under our top-kk. R-Sparse adds a low-rank term for skipped inputs. CHESS, CATS, M-COUNTDOWN (the up-projection variant of COUNTDOWN [16]), and Park et al. [52] compute one projection in full before thresholding, so they read at least a third of each routed expert.

Speed details

Setup. We measure decoding speed for Qwen3-Thinking, Gemma 4, and Qwen3.5 at 17.8% of routed-expert reads, on a single RTX PRO 6000 Blackwell Max-Q GPU (300 W) with vLLM 0.29.0, PyTorch 2.13, and Triton 3.7.1. The full model runs vLLM’s fused-MoE Triton kernels. At each batch size, we take the faster of vLLM’s default kernel configuration and one tuned for this GPU and model. Our reader and the full model use the same unmodified router.

Always-active modules. Gemma 4’s dense MLP and Qwen3.5’s shared expert are part of the timed block. Our implementation runs them on a second CUDA stream, alongside the router and the reader. Unmodified vLLM does not overlap Gemma 4’s dense MLP this way. If it did, Gemma 4’s MoE-block speedup would be 1.61 to 2.19×\times instead of 1.71 to 2.34×\times.

Strength of the baseline. Unmodified vLLM is a strong baseline. At batch one, its fused-MoE expert kernels stream weights at 1.35 to 1.44 TB/s, which is 83% to 88% of the 1.63 TB/s read bandwidth we measured as achievable on this GPU.

MoE block. To time the MoE blocks alone, we capture the layer inputs from seven real decoding steps on 128-token chat prompts and replay all MoE layers, routing included, in one CUDA graph. After four warmup replays, each process times seven repetitions with CUDA events and keeps the median. We report the median over three fresh processes.

Whole model. For whole-model timing, vLLM generates 64 output tokens from 128-token prompts at batch sizes 1 to 128, with full CUDA graphs, one warmup run, and three timed repetitions in each of three processes. Time per output token is the wall-clock time between the first and last generated tokens divided by 63, and we report the median across processes. This time includes attention, the language-model head, sampling, and vLLM’s scheduler, but not prompt prefill.

Same outputs. The timed Triton kernels compute the same function as the evaluated reader, up to the order of floating-point operations. On held-out chat documents at equal reads on Qwen3-Thinking, the two implementations’ next-token KL divergences from the full model differ by −0.0011±0.0013-0.0011 \pm0.0013 (90% confidence interval). On all three models, the kernels keep the same channels as the evaluated reader at equal reads: 0 of 25,606 checked selection decisions differ.

Long contexts. In single-process runs at 8,192 to 32,768 context tokens and batch sizes one and four, whole-model decoding is 1.20 to 1.55× faster on Qwen3-Thinking, 1.16 to 1.48× on Gemma 4, and 1.08 to 1.40× on Qwen3.5. We leave out batch four at 32,768 tokens on Gemma 4 and Qwen3.5, where neither version fits in memory. The gain shrinks at longer contexts, because attention takes longer while the MoE work per token stays the same.

Where the time goes. Traces of whole decoding steps at batch one show that each Qwen3-Thinking MoE layer takes 64.7 μ\mus in the full model and 34.4 μ\mus with our reader, router included. Within our reader, screening takes 9.5 μ\mus, top-kk selection 5.0 μ\mus, and computing the kept gate, up, and down rows 16.4 μ\mus. Projecting the skipped channels onto the rank-32 recovery map takes 8.0 μ\mus, partly overlapped on a second CUDA stream. Attention takes 48 μ\mus per layer in both, so the whole model speeds up less than the MoE block.

Larger batches. Tokens in a batch together select more channels, so the gain shrinks at large batches. Whole-model speedup peaks at batch 8 or 16 (1.75×\times on Qwen3-Thinking, 1.74×\times on Gemma 4, and 1.88×\times on Qwen3.5) and falls to 1.00×\times, 1.07×\times, and 1.24×\times at batch 128 (Table 20).

Serving trade-off. Figure 12 shows the trade-off between each user’s speed and the GPU’s total throughput. At batch one, our reader generates 196 to 217 tokens per second per user. The full model reaches this speed at no batch size; at batch one it generates 155 to 171 tokens per second. At the same per-user speed, between 80 and 120 tokens per second, one GPU generates 3.3 to 4.4 times as many tokens per second in total with our reader (3.3 to 4.3 at 100), interpolating log-linearly between measured batch sizes. The full model must run smaller batches to stay that fast. At longer contexts, attention takes a larger share of each step, and this advantage shrinks.

Figure 12. Our reader raises per-GPU throughput at the same per-user speed. Whole-model decoding of three post-trained models at 17.8% of routed-expert reads, against optimized vLLM, from Table 20 (128-token prompts, one RTX PRO 6000 Blackwell GPU). Each point is one batch size, labeled for our reader. Per-user speed is the inverse of the time per output token, and per-GPU throughput is the batch’s total output rate.

References

  1. [1]Ali Abbasi, Justin Shi, and Soheil Kolouri. BASE: Batch-aware selection of experts using predicted removal error for efficient MoE decoding. arXiv preprint arXiv:2609.36222, 2026.arxiv.org/abs/2609.36222
  2. [2]Mukund Agarwalla and Chih-Jen Lin. TopK-Guided: Adaptive, budget-aware activation sparsity for efficient LLM inference. arXiv preprint arXiv:2610.01763, 2026.arxiv.org/abs/2610.01763
  3. [3]Yash Akhauri, Ahmed F. AbouElhamayed, Jordan Dotzel, Zhiru Zhang, Alexander M. Rush, Safeen Huda, and Mohamed S. Abdelfattah. ShadowLLM: Predictor-based contextual sparsity for large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 19154–19167, 2024.
  4. [4]Ekaterina Alimaskina, Darya Rudas, Denis Shevykin, Gleb Molodtsov, Pavel Vasiliev, and Aleksandr Beznosikov. Extreme low-bit inference in reasoning models: Failure modes and targeted recovery. arXiv preprint arXiv:2606.02011, 2026.arxiv.org/abs/2606.02011
  5. [5]Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. Fluctuation-based adaptive structured pruning for large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 10865–10873, 2024.arxiv.org/abs/2312.11983
  6. [6]Anonymous. Where does attention mass live? A matched-budget study of query-adaptive attention sparsification for decoder LLMs. Submitted to International Conference on Learning Representations 2027, 2026a. URL https://openreview.net/forum?id=A9VqlTtV6t.
  7. [7]Anonymous. Qual-MoE: Practical dynamic latent mixture-of-experts with quantile balancing. Submitted to International Conference on Learning Representations 2027, 2026b. URL https://openreview.net/forum?id=5r5dBeWjfx.
  8. [8]Anonymous. ReComp: Low-rank residual compensation for contextual sparsity in language models. Submitted to International Conference on Learning Representations 2027, 2026c. URL https://openreview.net/forum?id=o6YIvAjXBd.
  9. [9]Anonymous. Sparser-MoE: Beyond expert sparsity with heterogeneous neuron-level sparsity. Submitted to International Conference on Learning Representations 2027, 2026d. URL https://openreview.net/forum?id=GRUZkymchi.
  10. [10]Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. QuaRot: Outlier-free 4-bit inference in rotated LLMs. In Advances in Neural Information Processing Systems, volume 37, pp. 100213–100240, 2024.arxiv.org/abs/2404.00456
  11. [11]Sankar Behera, Dhruv Singh, Anshika Agnihotri, Raj Kumar Choudhary, Satyadev Ahlawat, and Yamuna Prasad. LILA: Calibration-free structured pruning of large language models via latent spectral geometry. arXiv preprint arXiv:2609.11163, 2026.arxiv.org/abs/2609.11163
  12. [12]Lei Chen, Yuan Meng, Xiaoyu Zhan, Zhi Wang, and Wenwu Zhu. WiSparse: Boosting LLM inference efficiency with weight-aware mixed activation sparsity. arXiv preprint arXiv:2602.14452, 2026a.arxiv.org/abs/2602.14452
  13. [13]Sihan Chen, Dan Zhao, Jongwoo Ko, Colby Banbury, Huiping Zhuang, Luming Liang, Pashmina Cameron, and Tianyi Chen. WINA: Weight informed neuron activation for accelerating large language model inference. In International Conference on Learning Representations, pp. 122858–122880, 2026b.arxiv.org/abs/2505.19427
  14. [14]Xing Chen and Hengshuai Yao. Training-free halving of activated experts in fine-grained mixture-of-experts models. arXiv preprint arXiv:2609.04575, 2026.arxiv.org/abs/2609.04575
  15. [15]Runxi Cheng, Yuchen Guan, Yucheng Ding, Qingguo Hu, Yongxian Wei, Chun Yuan, Yelong Shen, Weizhu Chen, and Yeyun Gong. Mixture of neuron experts. arXiv preprint arXiv:2510.05781, 2025.arxiv.org/abs/2510.05781
  16. [16]Jaewon Cheon and Pilsung Kang. COUNTDOWN: Contextually sparse activation filtering out unnecessary weights in down projection. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 28393–28409, 2025.arxiv.org/abs/2505.17701
  17. [17]Krishna Teja Chitty-Venkata, Sandeep Madireddy, Murali Emani, and Venkatram Vishwanath. LExI: Layer-adaptive active experts for efficient MoE model inference. arXiv preprint arXiv:2509.02753, 2025.
  18. [18]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reihiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.arxiv.org/abs/2110.14168
  19. [19]Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 3029–3051, 2023.arxiv.org/abs/2305.14233
  20. [20]Harry Dong, Beidi Chen, and Yuejie Chi. Prompt-prompted adaptive structured pruning for efficient LLM generation. In Conference on Language Modeling, 2024.arxiv.org/abs/2404.01365
  21. [21]William Fedus, Barret Zoph, and Noam Shazeer. Switch Transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022.arxiv.org/abs/2101.03961
  22. [23]Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. OPTQ: Accurate post-training quantization for generative pre-trained Transformers. In International Conference on Learning Representations, 2023.arxiv.org/abs/2210.17323
  23. [24]Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.arxiv.org/abs/2101.00027
  24. [25]Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. Long language model evaluation harness, 2024. URL https://zenodo.org/records/12608602.
  25. [26]Gemma Team. Gemma 4 technical report. arXiv preprint arXiv:2607.02770, 2026.
  26. [27]Yuling Gu, O​​yvind Tafjord, Bailey Kuehl, Dany Haddad, Jesse Dodge, and Hannah Hajishirzi. OLMES: A standard for language model evaluations. In Findings of the Association for Computational Linguistics: NAACL 2025, pp. 5020–5048, 2025.
  27. [28]Junhui He, Shangyu Wu, Weidong Wen, Chun Jason Xue, and Qingan Li. CHESS: Optimizing LLM inference via channel-wise thresholding and selective sparsification. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 18658–18668, 2024.arxiv.org/abs/2409.01366
  28. [29]Wenwen Hou, Xinyuan Song, Wenqian Weng, and Shiwei Liu. ActTail: Global activation sparsity in large language models. arXiv preprint arXiv:2603.12272, 2026.arxiv.org/abs/2603.12272
  29. [30]Yushi Huang, Zining Wang, Zhihang Yuan, Yifu Ding, Ruihao Gong, Jinyang Guo, Xianglong Liu, and Jun Zhang. MoDES: Accelerating mixture-of-experts multimodal large language models via dynamic expert skipping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 30205–30215, 2026.arxiv.org/abs/2511.15690
  30. [32]Dohyeon Kim, Bedionita Soro, and Sung Ju Hwang. Distribution-consistent inference for dynamic sparse mixture-of-experts. In Findings of the Association for Computational Linguistics: EMNLP 2026, 2026a.arxiv.org/abs/2609.09241
  31. [33]JuneHyung Kim, Sankeerth Durvasula, and Nandita Vijaykumar. Activation sparsity with weight approximation for faster LLM decoding on offloaded weights. arXiv preprint arXiv:2610.02598, 2026b.arxiv.org/abs/2610.02598
  32. [34]Yongmin Kim, Shota Takashiro, Yusuke Iwasawa, Takeshi Kojima, and Yutaka Matsuo. Batch-wise adaptive pruning: Periodic neuron activation-aware weight pruning for language reasoning model. In Conference on Language Modeling, 2026c.arxiv.org/abs/2608.14003
  33. [35]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles, pp. 611–626, 2023.arxiv.org/abs/2309.06180
  34. [36]Mike Lasby, Ivan Lazarevich, Nish Sainnadurai, Sean Lie, Yani Ioannou, and Vithursan Thangarasa. REAP the experts: Why pruning prevails for one-shot MoE compression. In International Conference on Learning Representations, pp. 146883–146912, 2026.arxiv.org/abs/2510.13999
  35. [37]Donghyun Lee, Jaeyong Lee, Genghan Zhang, Mo Tiwari, and Azalia Mirhoseini. CATS: Context-aware thresholding for sparsity in large language models. In Conference on Language Modeling, 2024.arxiv.org/abs/2404.08763
  36. [38]Jungseob Lee, Seungyoon Lee, Seongtae Hong, Sugyeong Eo, and Heuis eok Lim. Where activation sparsity and KV-cache sparsity cross in LLM decoding. arXiv preprint arXiv:2609.33889, 2026a.arxiv.org/abs/2609.33889
  37. [39]Junyoung Lee, Sehyeon Park, Shinhyoung Jang, Seonha Ryu, Hojeong Kim, Hyunsei Lee, Il Hong Suh, and Yeseong Kim. FOCUS & RePAIR: Mitigating text degeneration via token-level guidance for pruned large language models. In International Conference on Machine Learning, 2026b.arxiv.org/abs/2608.26676
  38. [40]Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. AWQ: Activation-aware weight quantization for on-device LLM compression and acceleration. In Proceedings of Machine Learning and Systems, volume 6, pp. 87–100, 2024.DOI
  39. [41]Bin Liu, Fengfu Li, Xiaoxing Wang, Bo Zhang, and Junchi Yan. Ternary weight networks. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5, 2023a.
  40. [42]James Liu, Pragaash Ponnusamy, Tianle Cai, Han Guo, Yoon Kim, and Ben Athiwaratkun. Training-free activation sparsity in large language models. In International Conference on Learning Representations, pp. 98302–98322, 2025a.arxiv.org/abs/2408.14690
  41. [43]Jinyi Liu, Wei Chen, Pengyu Chen, Xinyi Yuan, Minghe Bai, Guoquan Wu, and Jun Wei. Prox: Training-free FFN activation sparsity via approximate intermediate-channel salience in LLMs. arXiv preprint arXiv:2607.27591, 2026.arxiv.org/abs/2607.27591
  42. [44]Kai Liu, Bowen Xu, Shaoyu Wu, Xin Chen, Hao Zhou, Yongliang Tao, and Lulu Hu. La RoSA: Enhancing LLM efficiency via layerwise rotated sparse activation. In International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 39968–39986. PMLR, 2025b.arxiv.org/abs/2507.01299
  43. [45]Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. SpinQuant: LLM quantization with learned rotations. In International Conference on Learning Representations, 2025c.arxiv.org/abs/2405.16406
  44. [46]Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Ré, and Beidi Chen. Deja Vu: Contextual sparsity for efficient LLMs at inference time. In International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 22137–22176. PMLR, 2023b.arxiv.org/abs/2310.17157
  45. [47]Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang, Bo Zhang, Junchi Yan, and Hongsheng Li. Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experts large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6159–6172, 2024.arxiv.org/abs/2402.14800
  46. [48]Yuqi Luo, Chenyang Song, Xu Han, Yingfa Chen, Chaojun Xiao, Xiaojun Meng, Liqun Deng, Jiansheng Wei, Zhiyuan Liu, and Maosong Sun. Sparsing law: Towards large language models with greater activation sparsity. In International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 41311–41330. PMLR, 2025.arxiv.org/abs/2411.02335
  47. [49]Haoqian Meng, Yilun Luo, Yafei Zhao, Wenyuan Liu, Huaqing Zheng, Xindian Ma, and Peng Zhang. SharQ: Bridging activation sparsity and FP4 quantization for LLM inference. arXiv preprint arXiv:2606.26587, 2026.arxiv.org/abs/2606.26587
  48. [50]Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, Yuling Gu, Shane Arora, Akshita B_hagia, Dustin Schwenk, David Wadden, Alexander Wettig, Binyuan Hui, Tim Dettmers, Douwe Kiela, Ali Farhadi, Noah A. Smith, Pang Wei Koh, Amanpreet Singh, and Hannaneh Hajishirzi. OLMoE: Open mixture-of-experts language models. In International Conference on Learning Representations, pp. 62061–62121, 2025.arxiv.org/abs/2409.02060
  49. [51]NVIDIA. CUDA C++ best practices guide, 2025. URL https://docs.nvidia.com/cuda/archive/13.0.0/cuda-c-best-practices-guide/index.html. Version 13.0, August 4, 2025. Section “Coalesced Access to Global Memory”.
  50. [52]Jongseok Park, Sunga Kim, Zhenyu Gu, Ion Stoica, and Alvin Cheung. Uncovering intra-expert activation sparsity for efficient mixture-of-expert model execution. arXiv preprint arXiv:2605.08575, 2026.arxiv.org/abs/2605.08575
  51. [53]Qwen Team. Qwen3.5-35B-A3B. Official model card; accessed 2026-09-24, 2026. URL https://huggingface.co/Qwen/Qwen3.5-35B-A3B.
  52. [54]David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julien Michael, and Samuel R. Bowman. GPQA: A graduate-level Google-proof Q&A benchmark. In Conference on Language Modeling, 2024.arxiv.org/abs/2311.12022
  53. [56]Noam Shazeer. GLU variants improve Transformer. arXiv preprint arXiv:2002.05202, 2020.arxiv.org/abs/2002.05202
  54. [57]Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations, 2017.arxiv.org/abs/1701.06538
  55. [58]Jiho Shin, Hoeseok Yang, and Youngmin Yi. SparseInfer: Training-free prediction of activation sparsity for fast LLM inference. In 2025 Design, Automation & Test in Europe Conference (DATE), pp. 1–7, 2025.arxiv.org/abs/2411.12692
  56. [59]Susav Shrestha, Bradley Settlemyer, Nikoli Dryden, and Narasimha Reddy. Polar Sparsity: High throughput batched LLM inferencing with scalable contextual sparsity. In Advances in Neural Information Processing Systems, volume 38, pp. 84842–84870, 2025.arxiv.org/abs/2505.14884
  57. [61]Luoyang Sun, Guangyan Li, Cheng Deng, Haifeng Zhang, Jian Zhao, Yongqiang Tang, Wensheng Zhang, and Jun Wang. Dual activation–weight sparsity: A training-free framework for efficient large language model compression. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8350–8366, 2026.
  58. [62]Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. A simple and effective pruning approach for large language models. In International Conference on Learning Representations, 2024.arxiv.org/abs/2306.11695
  59. [63]Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging BIG-Bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 13003–13051, 2023.arxiv.org/abs/2210.09261
  60. [64]Filip Szatkowski, Patryk Będkowski, Alessio Devoto, Jan Dubiński, Pasquale Minervini, Mikołaj Piórczyński, Simone Scardapane, and Bartosz Wojcik. Universal properties of activation sparsity in modern large language models. In International Conference on Learning Representations, pp. 146247–146269, 2026.arxiv.org/abs/2509.00454
  61. [65]Wenwu Tang, Dong Wang, Lothar Thiele, and Olga Saukh. GRAIL: Post-hoc compensation by linear reconstruction for compressed networks. In Conference on Parsimony and Learning, volume 328 of Proceedings of Machine Learning Research, pp. 881–895. PMLR, 2026.arxiv.org/abs/2602.23795
  62. [66]Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. QuIP#: Even better LLM quantization with Hadamard incoherence and lattice codebooks. In International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 48630–48656. PMLR, 2024.arxiv.org/abs/2402.04396
  63. [67]Abhishek Tyagi, Yunuo Cen, Shrey Dhorajivja, Bharadwaj Veeravalli, and Xuanyao Fong. DART-ing through the drift: Dynamic tracing of knowledge neurons for adaptive inference-time pruning. arXiv preprint arXiv:2601.22632, 2026.arxiv.org/abs/2601.22632
  64. [68]Qitong Wang, Xinwei Niu, Mingluo Su, Shanwei Zhao, Shiai Zhu, and Huan Wang. SparseDecoding: Decoding-aware pruning for accurate and efficient LLM inference. arXiv preprint arXiv:2610.12327, 2026.arxiv.org/abs/2610.12327
  65. [69]Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-Pro: A more robust and challenging multi-task language understanding benchmark. In Advances in Neural Information Processing Systems, volume 37, pp. 95266–95290, 2024.arxiv.org/abs/2406.01574
  66. [70]Miles Williams and Nikolaos Aletras. On the impact of calibration data in post-training quantization and pruning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 10100–10118, 2024.DOI
  67. [71]Juntong Wu, Jialiang Cheng, Fuyu Lv, Dan Ou, and Li Yuan. SERE: Similarity-based expert re-routing for efficient batch decoding in MoE models. In International Conference on Learning Representations, 2026a.arxiv.org/abs/2602.07616
  68. [72]Juntong Wu, Yifei Liu, Junyi Chen, Siqi Fan, Chaoran Feng, Minghao Li, Liujie Zhang, Weihang Chen, and Li Yuan. ExFold: Unified expert folding for training-free MoE prefill-decode acceleration. arXiv preprint arXiv:2608.24938, 2026b.arxiv.org/abs/2608.24938
  69. [73]Tong Wu, Yutong He, Bin Wang, and Kun Yuan. Mixture-of-Channels: Exploiting sparse FFNs for efficient LLMs pre-training and inference. arXiv preprint arXiv:2511.09323, 2025.arxiv.org/abs/2511.09323
  70. [74]Zukang Xu, Zhixiong Zhao, Xing Hu, Jiangyong Yu, Houji Wen, Jun Li, Zhe Jiang, and Dawei Yang. ACE: Adaptive calibration-free expert skipping for MoE-based LLMs. arXiv preprint arXiv:2609.05228, 2026.arxiv.org/abs/2609.05228
  71. [75]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chenghen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, T ianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyuan Zhang, Xuanchen Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.arxiv.org/abs/2505.09388
  72. [76]Zhenyu Zhang, Zechun Liu, Yuandong Tian, Harshit Khaitan, Zhangyang Wang, and Steven Li. R-Sparse: Rank-aware activation sparsity for efficient LLM inference. In International Conference on Learning Representations, pp. 77452–77466, 2025.arxiv.org/abs/2504.19449
  73. [77]Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2301.07911, 2023.
  74. [78]Yuxin Zhou, Zheng Li, Jun Zhang, Jue Wang, Yiping Wang, Zhongle Xie, Ke Chen, and Lidan Shou. FloE: On-the-fly MoE inference on memory-constrained GPU. In International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 78859–78882. PMLR, 2025.arxiv.org/abs/2505.05950

Paper details

vlnc:W_rdg2n663
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

LLM-based coding assistants

View statement
Looking for related papers…
Contents