betaFeedback

WorksComputer Science

GEAR2^2: Pruned Experts Live On through Gated Reuse of Activated Experts and a Missing-Mass Residual

Abstract

Mixture-of-Experts (MoE) Large Language Models (LLMs) activate only a few experts per token while generally keeping all the experts in GPU memory for high efficiency. Expert pruning reduces this GPU memory footprint cost, yet discards the contribution of pruned experts. Existing remedies either rely on static substitutes, merge experts at the cost of altering their functions, or require costly recovery training. In this paper, we find that the functionalities of pruned experts can be efficiently reconstructed from the activated remaining experts. We propose GEAR (Gated Expert Activation Reuse), which reuses the scaled outputs of reliable activated experts without any training or additional expert computation. In addition, we propose GEAR2^2 to further recover what reuse cannot express with an ultra-lightweight residual. On five MoE LLMs from 16B to 106B parameters with 25% and 50% of experts pruned, GEAR improves over its training-free counterparts in nearly every setting (up to 4 pp.) in terms of accuracy, and GEAR2^2 outperforms baseline approaches (up to 2.75 pp.) in every setting on nine zero-shot benchmarks, with larger advantages (up to 2.72 pp. higher) under more aggressive pruning (up to 50%).

Introduction

Mixture-of-Experts (MoE) Large Language Models (LLMs) activate only a few experts for each token, trading bounded computation for vast parameter capacity, and MoE has become the mainstream architecture of LLMs ([33, 21, 8, 44, 12]). However, this design comes with two notable challenges: (i) MoE trades memory for FLOPs, so all experts are generally loaded to GPU memory at deployment for efficiency, which incurs a large GPU memory footprint and makes decoding bound by the memory traffic of expert weights ([47]); (ii) expert usage is imbalanced, leaving substantial redundancy across experts ([3]). Expert pruning, which removes redundant experts entirely, is the most direct remedy for both ([30, 24]), but it discards the contribution of pruned experts to the tokens routed to them. In this work, we focus on compensating for the pruned experts under a limited budget, so that pruned MoE LLMs retain the memory savings and inference speedup of pruning while approaching the capabilities of the original models.

Existing work mitigates the performance degradation from pruning along three routes (Figure 1b–Figure 1d). First, pruning methods design better criteria for which experts to remove ([30, 24, 20]), and some other approaches replace each pruned expert with a substitute, such as its mean output or an affine transform of a retained expert ([47, 38]). These substitutes are static, i.e., they do not adapt to the experts activated for the current token, so the functionalities of pruned experts are only partly restored, and the performance degradation grows with the pruning ratio. Second, merging approaches fold functionally similar experts into one, so that their knowledge survives in the merged weights ([27, 3]). However, since experts are nonlinear functions, averaged parameters cannot reproduce the weighted sum of the original outputs ([24]). In addition, the difference between the original experts and the newly generated merged experts propagates across layers ([19]). Third, recovery training regains most of the performance ([20, 39]), but it updates all parameters with substantial costs.

Existing routes for mitigating the loss from expert pruning

Figure 1. Existing routes for mitigating the loss from expert pruning. (a) An MoE Transformer block. (b) Pruning discards the contribution of pruned experts. (c) Merging alters the functions of merged experts. (d) Recovery fine-tuning is costly.

Our key hypothesis is that the functionality of a pruned expert does not vanish with its parameters: experts in an MoE LLM form groups of functionally similar experts ([3, 49, 25]). The functionality of a pruned expert can be covered by other experts in the same Top-kk combination [48]. Thus, it can be largely reconstructed from the experts activated for the same token.

We thus propose a training-free Gated Expert Activation Reuse (GEAR) approach and its lightweight-training extension GEAR2^{2}, which dynamically compensate for pruned experts (Figure 2). To recover the functionalities of the pruned experts, we decompose the functionalities of a pruned expert into a shared component and a non-collinear residual. The shared component represents the functionalities of pruned experts that exist in other retained experts. The non-collinear residual represents the other functionalities of pruned experts that are hard to recover from the retained experts. To reconstruct the shared component of pruned experts, GEAR performs expert reuse, i.e., the shared functionalities of each pruned expert are largely recovered by reusing the scaled output of its best proxy among the activated experts. A reliability gate falls back to the mean output when the proxy is unreliable. The gate is essential as unreliable proxies make reuse worse than no reuse at all. Then, in order to recover the non-collinear residual, we propose GEAR2^{2} to exploit an ultra-lightweight compensation module trained by a small amount of end-to-end distillation.

Overview of GEAR and GEAR2

Figure 2. Overview of GEAR and GEAR². (a) The pruned model keeps its router and activated retained experts; (b) each pruned expert reuses the scaled output of a reliable proxy and falls back to the mean otherwise; (c) GEAR² adds a missing-mass-weighted low-rank residual, trained by light distillation with all original parameters frozen.

Our contributions are summarized as follows:

  • We propose GEAR, which compensates for pruned experts by reusing their best proxies among the activated experts behind a reliability gate, with no training or extra computation.

  • We propose GEAR2^{2}, i.e., an ultra-lightweight compensation module, which recovers the non-collinear residual by missing-mass weighting and light distillation.

  • We conduct extensive experiments with five MoE models ranging from 16B to 106B and nine zero-shot tasks. Experimental results demonstrate superior performance of GEAR compared with its training-free counterparts in nearly every setting, and GEAR2^{2} consistently outperforms four baseline approaches, with larger advantages at higher pruning ratios.

Related Work

Expert pruning. Weight-level pruning removes individual weights within experts [10, 37, 43], but the resulting irregular sparsity is hard to translate into actual speedup. Expert pruning instead removes entire experts and directly reduces storage. Existing methods differ mainly in how they select the experts to remove, using routing statistics [23, 15], the impact on layer outputs [30, 25, 24], or collaboration among experts [48]. Some methods further replace each pruned expert with a mean output [47] or an affine transform of a fixed proxy expert [38]. Such substitutes are fixed offline and do not adapt to the experts actually activated for the current token.

Expert merging. Merging methods fold functionally similar experts into one, so that the knowedge of pruned experts is retained in the merged parameters ([27, 3, 26, 50]. This line of work also reveals that experts form groups with similar functions ([3, 49]). However, merging rewrites the retained experts and alters their own functions. Since experts are non-linear, merged parameters cannot reproduce the input-dependent mix of the original outputs, and this error causes the functional subspace of later layers to collapse ([24]); the merged result is also fixed offline.

Recovery training after compression. Recovery training continues to train the compressed model ([20, 14, 39]), or updates only a small set of parameters such as the router ([19]) or low-rank adapters ([31, 7]). However, continued training approaches the cost of retraining, often with hundreds of billions of tokens ([39]), consuming the very cost that pruning was meant to save. Router calibration only re-allocates the remaining experts and does not restore the outputs of pruned ones. Low-rank adapters apply their correction regardless of how much routing weight a token loses, giving no stronger correction to tokens that lose more.

Dynamic expert skipping at inference. Another line of work keeps all experts and reduces those executed per token, by using a variable number of experts ([18, 46, 22]), skipping low-weight routes ([30]), or folding routes onto executed experts ([13, 41 b]). These methods do not reduce memory; after pruning, the contribution of removed experts can only be recovered through compensation.

Preliminaries

MoE layer. An MoE layer ([36, 9]) consists of NN routed experts {E1,…,EN}\{E_1,\ldots,E_N\}, a router, and optionally shared experts. Given a layer input x∈Rdx \in\mathbb{R}^{d}, the router scores every expert, selects the kk highest-scoring ones as the Top-kk set T(x)T(x), and assigns routing weights gi(x)g_i(x). The layer output is the weighted sum of the selected experts’ outputs:

y(x)=ys(x)+∑i∈T(x)gi(x)Ei(x),(1)y(x) = y_s(x) + \sum_{i \in T(x)} g_i(x) E_i(x), \tag*{(1)}

where ys(x)y_s(x) is the output of the shared experts (zero if absent).

Expert pruning. Expert pruning keeps a subset of experts K⊂{1,…,N}\mathcal{K} \subset\{1,\ldots,N\} in each layer according to some criterion and removes the rest. We keep the router unchanged, so each token still selects T(x)T(x) by its original scores. This set splits into the activated retained experts R(x)=T(x)∩K\mathcal{R}(x)=T(x)\cap\mathcal{K} and the pruned experts D(x)=T(x)∖K\mathcal{D}(x)=T(x)\setminus\mathcal{K}, and Eq. (1) becomes

y(x)=ys(x)+∑j∈R(x)gj(x)Ej(x)+∑i∈D(x)gi(x)Ei(x)⏟c(x).(2)\begin{aligned} y(x) &= y_s(x) + \sum_{j \in\mathcal{R}(x)} g_j(x) E_j(x) \\ &\quad+ \underbrace{\sum_{i \in\mathcal{D}(x)} g_i(x) E_i(x)}_{c(x)}. \tag*{(2)} \end{aligned}

The first two terms are computed as before, while the last term, the contribution of the pruned experts c(x)c(x), disappears with their parameters.

Compensation objective. Given a pruning mask, we compensate for c(x)c(x) under three constraints: (i) the retained experts and the router stay unchanged, so compensation applies to any mask; (ii) the original routing weights gi(x)g_i(x) are kept without renormalization, so the router still weights the compensation of each pruned expert; and (iii) compensation executes no additional experts at inference and uses little calibration and training data.

Method

Figure 2 illustrates the pipeline. We keep the output of the pruned model, y(x)−c(x)y(x)-c(x), and replace the lost contribution c(x)c(x) with an estimate c^(x)\hat{c}(x), built from an estimate E^i(x)\hat{E}_i(x) of each pruned expert and a residual term Δ(x)\Delta(x):

y^(x)=y(x)−c(x)+c^(x),\hat{y}(x) = y(x) - c(x) + \hat{c}(x),
c^(x)=∑i∈D(x)gi(x)E^i(x)+Δ(x).(3)\hat{c}(x) = \sum_{i \in\mathcal{D}(x)} g_i(x) \hat{E}_i(x) + \Delta(x). \tag*{(3)}

GEAR and GEAR2^{2} provide the two terms of c^(x)\hat{c}(x). GEAR (§ 4.1) constructs E^i(x)\hat{E}_i(x) by reusing the outputs of the experts activated for the current token; it needs only one offline calibration and no training, with Δ(x)=0\Delta(x)=0. GEAR2^{2} (§ 4.2) learns a missing-mass-weighted low-rank residual Δ(x)\Delta(x) for the part that GEAR cannot express, using distillation on a small amount of data.

GEAR: Gated Expert Activation Reuse

Scaled reuse. With its parameters removed, a pruned expert can only be estimated from what remains after pruning. Experts in an MoE LLM form groups with similar activation patterns § 1: within the Top-kk combination of a token, the output of a pruned expert often points in a direction similar to that of an activated expert and differs mainly in scale [42]. GEAR therefore estimates it by scaling the output of an activated expert:

Ei(x)≈Si←jEj(x),j∈R(x).(4)E_i(x) \approx S_{i\leftarrow j} E_j(x), \qquad j \in\mathcal{R}(x). \tag*{(4)}

We call Si←jS_{i\leftarrow j} the directed scale, since approximating ii with jj and jj with ii generally yield different coefficients. Let CijC_{ij} denote the calibration tokens on which experts ii and jj are both in the Top-kk set. One forward pass of the original model over a small calibration corpus yields SS and the relative reconstruction error by weighted least squares:

Lij(s)=∑t∈Cijwt∥Ei(xt)−sEj(xt)∥22,L_{ij}(s) = \sum_{t\in C_{ij}} w_t \left\lVert E_i(x_t) - s E_j(x_t) \right\rVert_2^2,
Si←j=arg⁡min⁡s  Lij(s),εi←j=Lij(Si←j)Lij(0).(5)S_{i\leftarrow j} = \underset{s}{\arg\min}\; L_{ij}(s), \qquad\varepsilon_{i\leftarrow j} = \frac{L_{ij}(S_{i\leftarrow j})}{L_{ij}(0)}. \tag*{(5)}

Here ε\varepsilon is the error relative to dropping the term, with 0 meaning exact reconstruction; both quantities have closed forms.

Best proxy. For each pruned expert i∈D(x)i \in\mathcal{D}(x), the best proxy is the activated expert with the lowest calibrated error among those co-activated with ii during calibration:

ji∗(x)=arg⁡min⁡j∈R(x)∩Pi  εi←j,(6)j_i^*(x) = \underset{j\in\mathcal{R}(x)\cap\mathcal{P}_i}{\arg\min}\; \varepsilon_{i\leftarrow j}, \tag*{(6)}

where Pi={j:Cij≠∅}\mathcal{P}_i = \{j : C_{ij} \ne\varnothing\} is the set of experts co-activated with ii during calibration. Since Eji∗(x)E_{j_i^*}(x) is already computed for the current token, reuse adds no expert computation; since the candidates follow the Top-kk combination of each token, E^i(x)\hat{E}_i(x) varies from token to token.

Reliability-gated reuse. The best proxy is the lowest-error expert among the available candidates, yet its error varies widely across models and expert pairs. A large error injects bias into the residual stream that accumulates over subsequent layers. A fallback that needs no expert computation is the mean output μi\mu_i, the constant that minimizes the expected squared error over the tokens routed to expert ii, recorded in the same calibration pass. GEAR adopts the best proxy only when its error falls below a threshold τ\tau and falls back to μi\mu_i otherwise:

E^i(x)={Si←ji∗Eji∗(x),εi←ji∗<τ,μi,otherwise.(7)\hat{E}_i(x) = \begin{cases} S_{i\leftarrow j_i^*} E_{j_i^*}(x), & \varepsilon_{i\leftarrow j_i^*} < \tau,\\ \mu_i, & \text{otherwise}. \end{cases} \tag*{(7)}

With Dτ(x)⊆D(x)\mathcal{D}_{\tau}(x) \subseteq\mathcal{D}(x) the pruned experts that pass the gate and yμ(x)y_\mu(x) the output of Eq. (3) with E^i=μi\hat{E}_i=\mu_i and Δ=0\Delta=0,

yGEAR(x)=yμ(x)+∑i∈Dτ(x)gi(x)[Si←ji∗Eji∗(x)−μi].(8)y_{\mathrm{GEAR}}(x) = y_\mu(x) + \sum_{i\in\mathcal{D}_{\tau}(x)} g_i(x)\left[S_{i\leftarrow j_i^*}E_{j_i^*}(x)-\mu_i\right]. \tag*{(8)}

GEAR thus builds on the mean estimate yμy_{\mu}, which coincides with MoNE [47] on the same mask, and replaces it with the reused output only where the proxy is reliable; τ→0\tau\to0 recovers the mean estimate, and τ=1\tau= 1 disables the gate.

Threshold calibration. The threshold τ\tau depends on the model and the pruning ratio. With the mask and calibration statistics fixed, we select τ∗=arg min⁡τ∈GNLL⁡dev(fτ)\tau^{*} = \operatorname{arg\,min}_{\tau\in\mathcal{G}} \operatorname{NLL}_{\mathrm{dev}}(f_{\tau}) from a small candidate set G\mathcal{G}, where fτf_{\tau} denotes GEAR with threshold τ\tau and NLL⁡dev\operatorname{NLL}_{\mathrm{dev}} is the next-token NLL on a development set. The search involves only forward passes. Overall, GEAR requires one calibration pass and a few development passes, and executes no additional experts at inference.

GEAR2: Missing-Mass Residual

Residual target. We write the output of each pruned expert as the GEAR estimate plus a remainder, Ei(x)=E^i(x)+δi(x)E_i(x) = \hat{E}_i(x) + \delta_i(x). When GEAR reuses a proxy, E^i\hat{E}_i is the shared component aligned with the proxy and δi\delta_i is the non-collinear residual that no scaling of the proxy can express; when GEAR falls back to the mean, δi=Ei(x)−μi\delta_i = E_i(x) - \mu_i is the input-dependent part that the mean misses. The layer output therefore still lacks ∑i∈D(x)gi(x) δi(x)\sum_{i \in\mathcal{D}(x)} g_i(x)\,\delta_i(x), which grows with the pruning ratio as each token loses more experts and keeps fewer proxies. This part has no closed form from calibration, so GEAR2^2 learns only it and leaves the reusable part to GEAR.

Missing-mass weighting. We factor the missing part by the missing mass m(x)m(x):

∑i∈D(x)gi(x) δi(x)=m(x)∑i∈D(x)gˉi(x) δi(x),m(x)=∑i∈D(x)gi(x),gˉi(x)=gi(x)/m(x).(9)\sum_{i \in\mathcal{D}(x)} g_i(x)\,\delta_i(x) = m(x)\sum_{i \in\mathcal{D}(x)} \bar{g}_i(x)\,\delta_i(x), \\ m(x) = \sum_{i \in\mathcal{D}(x)} g_i(x), \qquad\bar{g}_i(x) = g_i(x)/m(x). \tag*{(9)}

The missing mass is the total routing weight that the original router assigns to the pruned experts; it measures how much routed contribution the token loses and is available at inference. The second factor, a routing-weighted average of the residuals, is a function of xx, which we approximate with a low-rank linear map shared within each layer:

Δ(x)=m(x) BAx,A∈Rr×d,B∈Rd×r(10)\Delta(x) = m(x)\,BAx,\qquad A \in\mathbb{R}^{r \times d},\quad B \in\mathbb{R}^{d \times r} \tag*{(10)}

The correction vanishes when m(x)=0m(x) = 0 and scales with m(x)m(x) otherwise. Each layer adds a single pair of AA and BB with 2dr2dr parameters, independent of the number of pruned experts.

Distillation. Rather than fitting δi\delta_i layer by layer, we train the residuals of all layers end to end against the output distribution of the original model [17]. We freeze all original parameters and calibration statistics, initialize BB to zero so that training starts from GEAR, and minimize the token-level forward KL divergence:

LKD=1∣V∣∑t∈VKL⁡(porig(t)∥pGEAR2(t)),(11)\mathcal{L}_{\mathrm{KD}} = \frac{1}{|\mathcal{V}|}\sum_{t \in\mathcal{V}} \operatorname{KL}\left(p_{\mathrm{orig}}^{(t)} \mathbin{\|} p_{\mathrm{GEAR2}}^{(t)}\right), \tag*{(11)}

where V\mathcal{V} is the set of valid tokens. § 5.3 validates both choices: the same residual performs better on top of GEAR than on top of the mean estimate, and removing m(x)m(x) markedly degrades performance.

Implementation details. In Eq. (5), we set wt=∥Ei(xt)∥2w_t = \lVert E_i(x_t)\rVert_2 so that tokens with larger output norms receive larger weights, add 10−310^{-3} to the denominator of SS for numerical stability, and clip SS to [−4,4][-4,4]. A pruned expert with no candidate in R(x)∩Pi\mathcal{R}(x) \cap\mathcal{P}_i falls back to μi\mu_i, and ties are broken by the original Top-kk order. The candidate set of τ\tau and its rationale are given in Appendix § 7.

Experiments

Experimental Setup

Models. We evaluate five open-source MoE models from 16B to 106B parameters: Moonlight-16B-A3B [28], Qwen3-30B-A3B-Instruct-2507 [44], DeepSeek-V2-Lite [8], Gemma-4-26B-A4B [11], and GLM-4.5-Air [12]. Their expert configurations are listed in Appendix § 7.

Pruning and baselines. We prune 25% or 50% of the routed experts in each layer. To examine whether compensation depends on the pruning criterion, we use the REAP mask [24] on GLM-4.5-Air and the MoNE mask [47] on the other models; Appendix § 8.§ 3 reports REAP masks on three more models, where the gains persist. We compare with methods from three families. Native REAP removes the pruned experts and reroutes tokens among the retained ones. MoNE replaces each pruned expert with its mean output on the same mask and is the direct training-free counterpart of GEAR, while MoRA [38] approximates pruned experts with affine transforms of retained ones. HC-SMoE [3] clusters experts and merges each cluster into one. Results of MoRA and HC-SMoE follow Sun et al. [38] and are shown in italics.

Implementation details. Following common practice (Frantar and Alistarh, [10]; Sun et al., [37]), we calibrate on C4 (Raffel et al., [34]): 100 packed sequences of 4,096 tokens from the training set for the statistics, and 16 documents for selecting τ\tau. The residual of GEAR² has rank 32 and is distilled for 64 steps on 64 sequences of 512 tokens from C4. Further details are given in Appendix § 7.

Evaluation. We use lm-evaluation-harness (Bider­man et al., [1]) to evaluate nine zero-shot tasks: ARC-Challenge and ARC-Easy (Clark et al., [5]), BoolQ (Clark et al., [4]), HellaSwag (Zellers et al., [45]), MMLU (Hendrycks et al., [16]), OpenBookQA (Mihaylov et al., [32]), PIQA (Bisk et al., [2]), RTE (Dagan et al., [6]; Wang et al., [40]), and WinoGrande (Sakaguchi et al., [35]). We report acc_norm for ARC, HellaSwag, OpenBookQA, and PIQA and acc for the others, and we report the unweighted average over the nine tasks. Bold marks the best pruned result in each column of a model.

Main Results

Table 1 compares the nine-task average accuracy on five models at two pruning ratios. Our methods achieve the best pruned result in all 10 settings, 9 of them by GEAR², and GEAR² outperforms

MethodMaskAvg.Δ\Delta
Mean–62.39−0.59
No gateNo61.76−1.22
Random #1Yes62.02−0.96
Random #2Yes61.41−1.57
Reverse ε\varepsilonYes61.98−1.00
GEARYes62.98ref.

caption: tbl:3: Reliability-gate ablation on Gemma-4 at 50% pruning. Mean: every pruned expert falls back to μi\mu_i; masked variants mask as many pairs as GEAR. Δ\Delta: relative to GEAR.

BaseRes.m(x)m(x)Avg.Δ\Delta
Mean––70.25−1.99
GEAR––71.41−0.83
Zero✓✓71.62−0.62
Mean✓✓71.61−0.63
GEAR✓✗68.62−3.62
GEAR✓✓72.24ref.

caption: tbl:4: Residual ablation on Qwen3 at 50% pruning. Res.: low-rank residual; ✗: m(x)m(x) replaced by 1. Last row: GEAR2^{2}. Δ\Delta: relative to GEAR2^{2}.

both same-mask MoNE and native REAP in every setting. At 50% pruning, GEAR2^{2} improves over MoNE by 0.96–3.44 points. Without any training, GEAR already surpasses MoNE in 9 of the 10 settings, by 1.49 and 1.16 points on DeepSeek-V2-Lite and Qwen3 at 50%. GEAR2^{2} also exceeds the substitution method MoRA and the merging method HC-SMoE on every model they report, by 2.75–6.41 and 3.31–25.34 points at 50% pruning.

The gain from compensation grows with the damage caused by pruning. Across the 10 settings, the accuracy lost to pruning and the gain of GEAR2^{2} over MoNE reach a Spearman correlation of 0.90 (Figure 3b): at 25%, pruning costs only 0.6–2.6 points, leaving little to recover, whereas at 50% the loss rises to 4.2–13.3 points and the gain widens accordingly. Per task at 50% (Figure 3a and Table 2), the largest gains appear on ARC-Challenge and ARC-Easy, which pruning damages by 16.5 and 16.9 points on average: GEAR2^{2} recovers 4.3 and 3.9 points and improves on all five models. Overall, GEAR2^{2} improves over MoNE on 41 of the 45 model–task pairs, and on every model for five of the nine tasks. By source, the residual improves over GEAR on 41 pairs and reuse improves over MoNE on 30: reuse recovers what the activated experts can express, and the residual recovers the rest. These gains are significant under a paired bootstrap and robust to seeds and pruning masks (Appendix § 8.§ 2–§ 8.§ 4).

MethodRTEBoolQOBQAPIQAARC-CARC-EWino.Hella.MMLUAvg.
Original66.0680.2445.2080.7957.9482.4171.4378.3667.4869.99
HC-SMoE53.4361.3127.8062.7326.0250.6354.0634.9629.9344.54
MoRA61.7370.9235.4077.3136.4365.3261.0161.4728.6655.36
REAP46.2156.5140.8077.4242.8366.2570.8072.7624.7655.37
MoNE59.5772.2639.2078.2936.4359.8969.5372.0122.9756.68
GEAR60.6572.1139.6078.5136.2660.0670.4072.2222.9756.98
GEAR²59.5773.7940.8080.1441.3065.0771.1975.7624.1659.09
Original76.9088.5644.8080.0962.6383.8473.6479.7680.1174.48
HC-SMoE58.1272.2927.4056.4730.0347.3953.4335.6141.3346.90
MoRA75.0982.5141.8080.4746.2569.0768.9875.2053.1465.83
REAP81.2384.8938.4077.1552.3071.1769.8570.0561.6467.41
MoNE79.7887.8943.8081.1853.5074.9274.0378.2758.8770.25
GEAR79.7888.3544.2081.4556.1477.0274.1178.4963.1071.41
GEAR²84.1288.2044.4081.6656.4878.4574.2779.1363.4872.24
Original61.0179.7243.8080.0348.5576.2271.0377.8655.1165.93
HC-SMoE57.0466.5134.5671.5833.6960.5765.5660.7534.3253.84
MoRA51.6358.9642.2077.8038.6563.1762.5171.4723.2554.40
REAP54.5159.1740.0076.2840.7861.8363.6969.3123.6354.36
MoNE54.1566.2136.8076.2835.5856.0667.4867.8422.9653.71
GEAR55.9665.4738.6077.3738.5760.4866.6970.6723.0255.20
GEAR²58.1268.7840.4078.3541.1363.5167.7273.1723.1957.15
Original67.1585.0848.2083.7368.6989.5676.0183.2074.3175.10
REAP58.8477.7441.2079.5442.1564.5266.1471.7130.9759.20
MoNE60.6581.1347.0082.1042.4167.2674.5180.0026.5062.39
GEAR62.8279.1447.6082.0544.8068.5674.3580.4827.0062.98
GEAR²67.5182.0548.6082.4345.8268.7375.3081.5428.8264.53
Original75.0988.2348.2083.4661.8682.6677.5185.8778.9375.76
REAP62.0984.2845.4080.9052.2273.7477.1982.1661.2968.81
MoNE66.4387.1943.4081.1249.2372.0578.1480.8559.6668.67
GEAR63.1886.1842.6081.0750.8572.8176.8780.8260.2068.29
GEAR²68.5985.6643.4081.1854.0173.9976.8081.6361.4469.63

Table 2. Accuracy (%) on nine zero-shot tasks at 50% pruning.

Figure 3. (a) Per-task gain of GEAR2\mathrm{GEAR}^{2} over MoNE at 50% pruning (pp). (b) Accuracy lost to pruning versus the gain of GEAR2\mathrm{GEAR}^{2} over MoNE: the gain grows with the loss. Dashed line: least-squares fit.

Ablation Study

This section examines the key design of each method: the reliability gate of GEAR, and the base and weighting of the residual in GEAR2^{2}.

Reliability gate. The gate must identify reliable proxies rather than merely reduce reuse: masking the pairs (i←j)(i \leftarrow j) whose error exceeds τ\tau does both at once. To separate the two, on Gemma-4 at 50% we keep the number of masked pairs equal to that of GEAR and change only which pairs are masked, either at random or in reverse order of ε\varepsilon, which masks the lowest-error pairs and keeps the highest. Table 3 shows that the two random masks and the reverse mask fall 0.96, 1.57, and 1.00 points below GEAR, all below the mean estimate. The same reuse frequency thus yields opposite outcomes depending on which pairs are kept, so the gain of GEAR comes from keeping reliable proxies according to ε\varepsilon rather than from reusing less. Without the gate, always reusing the best proxy falls below the mean estimate by 0.63 points on Gemma-4 and by 5.01 points on Moonlight at 50%, while GEAR with the gate exceeds it on both (Table 1). Unreliable reuse is more harmful than no reuse: μi\mu_i is the minimum-error constant and introduces no input-dependent bias, whereas a biased proxy output enters the residual stream at every layer and accumulates. The gate is also easy to set: every τ<1\tau< 1 in GG recovers 4.86–5.30 points over ungated reuse on Moonlight (Appendix § 8.§ 4), and replacing the ε\varepsilon-selected proxy with an oracle choice reduces the reconstruction error by at most 0.17% (Appendix § 9.§ 1).

MethodMoonlight 25%Moonlight 50%Qwen3 25%Qwen3 50%DeepSeek 25%DeepSeek 50%Gemma-4 25%Gemma-4 50%GLM 25%GLM 50%
Original69.9974.4865.9375.1075.76
HC-SMoE56.8744.5464.9846.9062.6653.84––––
MoRA67.4855.3672.5065.8363.6854.40––––
REAP66.6855.3773.5667.4163.1954.3669.6159.2074.3368.81
MoNE67.3756.6873.8770.2564.4753.7173.1162.3974.3168.67
GEAR67.6656.9873.9171.4164.4955.2073.2562.9874.4368.29
GEAR²67.9659.0973.9772.2464.5057.1573.4364.5374.3969.63

Table 1. Nine-task average accuracy (%) after pruning 25% or 50% of routed experts.

Residual design. The residual of GEAR2^{2} is built on the estimate of GEAR and weighted by the missing mass m(x)m(x) (Eq. (9)). Table 4 tests both on Qwen3 at 50% under the same parameter count and training budget. For the base, the same residual built on zero or on the mean estimate scores 0.62 and 0.63 points lower than on GEAR, although it then faces a larger remaining error. This is consistent with the decomposition in § 4.2: a low-rank map is better suited to the non-collinear remainder after reuse than to the full contribution, so reuse and the residual are complementary. For the weighting, replacing m(x)m(x) with the constant 1 lowers the average to 68.62, below the mean estimate without any residual. With m(x)m(x), the correction vanishes on tokens that lose no routing weight and grows with the weight a token loses; a constant weight applies the same correction to every token, including those whose Top-kk set contains no pruned expert.

Representation Analysis

To illustrate how compensation acts on the forward computation, Figure 4 projects the layer-wise hidden states of the original and compensated models onto two dimensions, with examples from Qwen3, GLM-4.5-Air, and DeepSeek-V2-Lite at 50% pruning. After pruning, the trajectory with mean compensation drifts away from the original model layer by layer. GEAR recovers the pruned contributions from the experts already executed for the current token and pulls the trajectory back toward the original. The residual of GEAR2\mathrm{GEAR}^{2} further corrects what reuse cannot express and brings the endpoint closer to the original model.

Figure 4. Layer-wise hidden states at 50% pruning, projected onto two dimensions. Qualitative examples, selected by the error reduction of GEAR2\mathrm{GEAR}^{2}. Err: normalized L2L_2 error of the hidden state relative to the original model at the shown layer.

Overhead

Table 5 summarizes the cost on Qwen3 at 50% pruning: GEAR needs no training, and GEAR2\mathrm{GEAR}^{2}

MetricMoNEGEARGEAR2\mathrm{GEAR}^{2}
Training––64 steps, 32K tokens
Stats.μi\mu_iμi\mu_i, SS, ε\varepsilonμi\mu_i, SS, ε\varepsilon
Storage (GiB)0.0470.0560.079
Extra experts000
Latency (s)2.543.153.47

Table 5. Recovery cost and inference overhead on Qwen3 at 50% pruning; the pruned model occupies 29.9 GiB in bf16.

trains only 64 steps and adds about 0.08 GiB of storage, under 0.3% of the pruned model. At inference, both only reuse expert outputs that have already been computed; the extra table lookups, scalar scaling, weighted summation, and low-rank projection can be fused into the output aggregation kernel of the MoE layer. In our unfused implementation (batch size 1, 128 prompt and 32 generated tokens), MoNE, GEAR, and GEAR2\mathrm{GEAR}^{2} take 2.54, 3.15, and 3.47 seconds, respectively.

Conclusion

This paper shows that the contribution of pruned experts can be recovered from the outputs a pruned MoE already computes. GEAR reuses the scaled outputs of experts already activated for the same token behind a reliability gate, without training or additional expert computation, and GEAR2\mathrm{GEAR}^{2} adds a missing-mass-weighted low-rank residual distilled for only 64 steps. Both leave the router and retained experts untouched and apply on top of any pruning mask. On five MoE models, GEAR2\mathrm{GEAR}^{2} outperforms MoNE and REAP in every setting, and the more aggressive the pruning, the larger the gain. Recovery after pruning need not rely on retraining: by making full use of what a pruned model already computes, the accuracy loss from pruning can be narrowed at very low cost.

Limitations

Our evaluation covers nine zero-shot understanding and reasoning tasks on MoE models from 16B to 106B parameters. Generative and long-context tasks and larger models are left for future work, where calibration and distillation data matched to the target domain are a natural extension. Our reference implementation is not yet fused into the output aggregation of the MoE layer, which would further reduce the inference overhead.

Ethical Considerations

This work reduces the memory and computation needed to deploy MoE models at little training cost. All models and datasets we use are publicly available and used under their licenses.

Implementation Details

Models. Moonlight-16B-A3B and DeepSeek-V2-Lite activate 6 of 64 routed experts alongside 2 shared experts in each MoE layer; Qwen3 and Gemma-4 activate 8 of 128 routed experts; GLM-4.5-Air activates 8 of 128 routed experts alongside 1 shared expert.

Calibration. All statistics of a layer are collected in a single forward pass of the original model over 100 packed sequences of 4,096 tokens from the C4 training set. The mean output μi\mu_i is averaged over the tokens routed to expert ii, and Si←jS_{i\leftarrow j} and ϵi←j\epsilon_{i\leftarrow j} are computed over the tokens CijC_{ij} on which ii and jj are co-activated.

Threshold selection. We search τ\tau over G={0.90,0.95,0.98,0.99,1}\mathcal{G} = \{0.90, 0.95, 0.98, 0.99, 1\}, where τ=1\tau= 1 disables the gate. Since ϵ\epsilon measures the error relative to dropping the term, a proxy with ϵ\epsilon close to 1 offers little over the mean estimate, so the candidates concentrate near 1. For each model and pruning ratio, we select the threshold with the lowest next-token NLL on 16 development documents from C4 and break ties toward the larger τ\tau, which keeps more reuse. The search involves only forward passes. The selection transfers to held-out text: on Gemma-4 at 50% pruning, the threshold selected on the development documents (τ=0.98\tau= 0.98) also yields the lowest NLL on 16 further C4 documents.

Training. The residual is trained with AdamW (Loshchilov and Hutter [29]) at a learning rate of 5×10−45 \times10^{-4}, with gradient clipping at 1.0 and a distillation temperature of 1.

Residual size. With rank 32, the residual of GEAR2\mathrm{GEAR}^{2} adds 3.4M parameters to Moonlight and DeepSeek-V2-Lite, 6.3M to Qwen3, 5.4M to Gemma-4, and 11.8M to GLM-4.5-Air, about 0.02% of each model or less.

Evaluation. All tasks are evaluated zero-shot with lm-evaluation-harness. MMLU accuracy is averaged over all its questions, and the reported average weights the nine tasks equally.

Additional Results

Per-Task Results at 25% Pruning

Table 6 reports the per-task accuracy at 25% pruning in the same format as Table 2. GEAR2\mathrm{GEAR}^{2} stays within 2.1 points of the original model on every model and attains the best pruned average on four of the five models.

MethodRTEBoolQOBQAPIQAARC-CARC-EWino.Hella.MMLUAvg.
Original66.0680.2445.2080.7957.9482.4171.4378.3667.4869.99
HC-SMoE59.5770.3436.4068.9941.7270.4556.5953.9353.8056.87
MoRA70.7679.5743.0080.7953.1579.1269.3874.7656.7967.48
REAP55.6079.1145.6081.1254.2778.2872.6178.1155.3966.68
MoNE62.0979.2445.6081.3955.4679.4671.8278.0253.2267.37
GEAR63.1879.1446.0081.2855.4679.5571.6777.6854.9967.66
GEAR²62.4579.2045.4081.4556.4080.3571.7478.3556.2967.96
Original76.9088.5644.8080.0962.6383.8473.6479.7680.1174.48
HC-SMoE80.5083.5538.0072.1447.1871.8465.2761.9864.3564.98
MoRA76.1788.8144.8081.1857.4280.1872.3879.2872.3272.50
REAP74.7389.0844.4080.8563.4083.5473.7277.8274.5473.56
MoNE75.8188.2344.4080.9061.6984.3974.3579.7275.3573.87
GEAR76.5388.0443.6080.7962.0384.4374.0379.7076.0473.91
GEAR²77.2688.1043.6080.8561.7784.0174.5979.7375.8373.97
Original61.0179.7243.8080.0348.5576.2271.0377.8655.1165.93
HC-SMoE61.3773.3740.6078.7844.8872.1071.0374.4947.2962.66
MoRA58.1272.2644.4080.2045.9977.1970.0177.2447.7363.68
REAP57.4068.1744.2079.9247.6175.5170.4877.2148.1863.19
MoNE62.0978.4743.8079.7147.1874.7970.2475.7448.1664.47
GEAR62.8276.3343.6080.5247.3575.1370.3276.2948.0364.49
GEAR²63.5476.2744.4080.0947.7074.3370.7276.5646.9464.50
Original67.1585.0848.2083.7368.6989.5676.0183.2074.3175.10
REAP66.0682.9446.8083.1357.3481.4074.1980.6354.0269.61
MoNE68.5984.0747.2083.7963.2386.8375.4582.7866.0373.11
GEAR69.6883.5246.8084.0664.4286.2875.5382.7266.2773.25
GEAR²68.9584.3749.0084.2863.4886.2076.4882.7365.3573.43
Original75.0988.2348.2083.4661.8682.6677.5185.8778.9375.76
REAP69.3187.3448.6083.9060.6779.4279.0186.0674.7074.33
MoNE68.9587.7447.2083.6261.7780.3978.6985.8774.5374.31
GEAR69.6887.7747.6083.5761.8680.4778.4585.8774.6174.43
GEAR²70.0487.8647.8083.7361.3579.9778.7785.7574.2374.39

Table 6. Accuracy (%) on nine zero-shot tasks at 25% pruning.

Statistical Significance

We estimate 95% confidence intervals of the nine-task average difference with a paired bootstrap over test questions, resampling within each task (4,000 resamples). At 50% pruning, the improvements of GEAR2\mathrm{GEAR}^{2} over MoNE, REAP, and GEAR are significant on all five models (Table 7). With disjoint question halves for estimating pruning damage and recovery gain, cross-fitted Spearman correlations remain positive across the nine available model–pruning settings (median 0.87; 2.5–97.5% split quantiles [0.72, 0.95]) and across the 45 model–task pairs at 50% pruning (median 0.45; [0.34, 0.58]).

Modelvs. MoNEvs. REAPvs. GEAR
Moonlight+2.41 [1.76, 3.03]+3.72 [2.82, 4.62]+2.11 [1.48, 2.75]
Qwen3+1.99 [1.46, 2.56]+4.83 [4.02, 5.67]+0.83 [0.34, 1.36]
DeepSeek+3.44 [2.65, 4.27]+2.79 [1.84, 3.75]+1.95 [1.22, 2.69]
Gemma-4+2.14 [1.29, 2.96]+5.33 [4.32, 6.35]+1.55 [0.79, 2.32]
GLM+0.96 [0.31, 1.64]+0.82 [0.06, 1.56]+1.34 [0.68, 2.02]

Table 7. Improvement of GEAR2\mathrm{GEAR}^{2} at 50% pruning (points), with 95% paired bootstrap confidence intervals.

Results with REAP Masks

To test whether compensation depends on the pruning criterion, we replace the MoNE mask with the REAP mask on Qwen3, DeepSeek-V2-Lite, and Gemma-4 at 50% pruning and follow the same procedure otherwise. GEAR and GEAR2\mathrm{GEAR}^{2} improve over the mean estimate by 1.16–2.44 and 1.83–3.61 points, respectively (Table 8), so the gains do not rely on a specific mask.

ModelMeanGEARGEAR2\mathrm{GEAR}^{2}
Qwen364.7965.9566.62
DeepSeek-V2-Lite53.5555.3757.16
Gemma-452.1254.5655.28

Table 8. Nine-task average accuracy (%) with REAP masks at 50% pruning. Compensation consistently improves over the mean estimate.

Per task, GEAR2\mathrm{GEAR}^{2} improves over the mean estimate on 23 of the 27 model–task pairs (Table 9).

TaskQwen3 MeanQwen3 GEARQwen3 GEAR²DeepSeek MeanDeepSeek GEARDeepSeek GEAR²Gemma-4 MeanGemma-4 GEARGemma-4 GEAR²
RTE78.3475.4575.0953.4355.6057.0452.3555.9653.43
BoolQ86.4285.6684.0769.4568.9672.3569.2068.3569.76
OBQA35.6038.2040.4036.8036.8039.4036.0040.6040.20
PIQA76.8877.1577.5875.1977.0478.2475.8476.8278.24
ARC-C47.6149.8349.4934.2237.9740.1032.0036.8637.63
ARC-E67.9769.9570.5455.9360.9064.8155.9358.2561.95
Wino.70.2470.0970.4068.0367.8867.0963.8561.7263.14
Hella.66.1167.8170.3965.9270.0072.0360.9368.3269.76
MMLU59.9059.3961.6652.9753.1553.3752.2554.2453.44
Avg.64.7965.9566.6253.5555.3757.1652.1254.5655.28

Table 9. Accuracy (%) with REAP masks at 50% pruning.

Robustness to Seed, Rank, and τ\tau

Table Table 10 retrains the residual of GEAR² with three seeds on Moonlight and DeepSeek-V2-Lite at 50% pruning: every seed improves over MoNE by at least 1.83 and 2.80 points, respectively, and rank 16 retains most of the gain. Table Table 11 varies the threshold of GEAR on Moonlight: every τ<1\tau< 1 in G\mathcal{G} recovers 4.86–5.30 points over ungated reuse (τ=1\tau= 1), so the gate does not hinge on a finely tuned τ\tau.

GEAR²MoonlightDeepSeek
Seed 1, rank 3258.5157.15
Seed 2, rank 3259.0956.51
Seed 3, rank 3258.8256.58
Seed 1, rank 1658.4456.27
MoNE56.6853.71

Table 10. Nine-task average accuracy (%) of GEAR² at 50% pruning across training seeds and ranks.

τ\tau10.990.980.950.90
GEAR51.6856.9856.5556.5456.67

Table 11. Nine-task average accuracy (%) of GEAR on Moonlight at 50% pruning under different thresholds; τ=1\tau= 1 disables the gate.

Robustness across Calibration Batches

On GLM-4.5-Air, we repeat calibration and residual training with three calibration batches. GEAR² outperforms REAP on all six combinations of batch and pruning ratio, by 0.82–1.49 points at 50% pruning (Table Table 12).

25%50%
BatchREAPGEAR2\mathrm{GEAR}^{2}REAPGEAR2\mathrm{GEAR}^{2}
174.3374.3968.8169.63
274.1674.3667.7469.23
374.0974.1668.5769.85

Table 12. Nine-task average accuracy (%) of GLM-4.5-Air with three calibration batches. GEAR2\mathrm{GEAR}^{2} outperforms REAP on every batch.

Additional Analysis

Proxy Selection

We examine the two choices behind the best proxy on Moonlight and DeepSeek-V2-Lite at 50% pruning, measuring the reconstruction error of the pruned contribution on 8 sequences of 512 tokens. First, enlarging the candidate pool from the co-activated experts to all retained experts changes this error by about 0.1%, while it would require 18.0% and 30.2% more expert evaluations, respectively. Second, replacing the ε\varepsilon-selected proxy with an oracle that picks, for every slot, the candidate with the smallest true error reduces the per-slot error by only 0.17% and 0.04%. Restricting reuse to co-activated experts and selecting proxies by ε\varepsilon is therefore nearly optimal at no extra cost.

Hidden-State Trajectories

The examples in Figure 4 (§ 5.4) are chosen for illustration; Table 13 confirms the trend over all token windows. Over 150 windows per model, GEAR2\mathrm{GEAR}^{2} lowers the average hidden-state error of the mean estimate on every model and is closer to the original model in 111, 142, and 99 windows on Moonlight, DeepSeek-V2-Lite, and GLM-4.5-Air, respectively.

ModelWindowsMeanGEAR2\mathrm{GEAR}^{2}Closer
Moonlight1500.2650.232111/150
DeepSeek1500.2220.168142/150
GLM1500.2490.23999/150

Table 13. Average hidden-state error at 50% pruning over token windows, and the number of windows in which GEAR2\mathrm{GEAR}^{2} is closer to the original model than the mean estimate.

References

  1. [1]Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, Anthony DiPorfi, Julen Etxaniz, Benjamin Fattori, Jessica Zosa Forde, Charles Foster, Jeffrey Hsu, Mimansa Jaiswal, Wilson Y. Lee, Haonan Li, and 11 others. 2024. Lessons from the Trenches on Reproducible Evaluation of Language Models. Preprint, arXiv:2405.14782.arxiv.org/abs/2405.14782
  2. [2]Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. PIQA: Reasoning about Physical Commonsense in Natural Language. In AAAI Conference on Artificial Intelligence, volume 34, pages 7432–7439.arxiv.org/abs/1911.11641
  3. [3]I-Chun Chen, Hsu-Shen Liu, Wei-Fang Sun, Chen-Hao Chou, Yen-Cheng Hsu, and Chun-Yi Lee. 2025. Retraining-free merging of sparse MoE via hierarchical clustering. In International Conference on Machine Learning, volume 267, pages 8594–8620. PMLR.arxiv.org/abs/2410.08589
  4. [4]Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2924–2936, Minneapolis, Minnesota. Association for Computational Linguistics.arxiv.org/abs/1905.10044
  5. [5]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. Preprint, arXiv:1803.05457.arxiv.org/abs/1803.05457
  6. [6]Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL recognising textual entailment challenge. In Machine Learning Challenges, pages 177–190. Springer Berlin Heidelberg.
  7. [7]Devleena Das, Rajeev Patwari, and Ashish Sirasao. 2025. Recover-LoRA: Data-free accuracy recovery of degraded language models via low-rank adaptation. In Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 2377–2386, Suzhou (China). Association for Computational Linguistics.arxiv.org/abs/2510.08600
  8. [8]DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guangbo Hao, Guang Chen, and 138 others. 2024. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. Preprint, arXiv:2405.04434.arxiv.org/abs/2405.04434
  9. [9]William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39.arxiv.org/abs/2101.03961
  10. [10]Elias Frantar and Dan Alistarh. 2023. SparseGPT: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, volume 202, pages 10323–10337. PMLR.arxiv.org/abs/2301.00774
  11. [11]Gemma Team, Sherif El Abd, Vaibhav Aggarwal, Robin Algayres, Alek Andreev, Olivier Bachem, Ian Ballantyne, Cormac Brick, Victor Cărbune, Michelle Casbon, Mayank Chaturvedi, Aditya Chawla, Victor Cotruta, Alice Coucke, Phil Culliton, Robert Dadashi, Lucas Dixon, Mohamed Elhawaty, Utku Evci, and 304 others. 2026. Gemma 4 Technical Report. Preprint, arXiv:2607.02770.
  12. [12]GLM-4.5 Team, Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Dayi Yin, Hao Zeng, Jiajie Zhang, Kedong Wang, Lucen Zhong, Mingdao Liu, Rui Lu, Shulin Cao, Xiaohan Zhang, Xuancheng Huang, Yao Wei, and 152 others. 2025. GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models. Preprint, arXiv:2508.06471.
  13. [13]Vima Gupta, Jae Hyung Ju, Kartik Sinha, Ada Gavrilovska, and Anand Padmanabha Iyer. 2026. Lynx: Enabling Efficient MoE Inference through Dynamic Batch-Aware Expert Selection. Preprint, arXiv:2411.08982.arxiv.org/abs/2411.08982
  14. [14]Jiawei Hao, Zhiwei Hao, Jianyuan Guo, Li Shen, Yong Luo, Han Hu, and Dan Zeng. 2026. LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacing. Preprint, arXiv:2603.12645.arxiv.org/abs/2603.12645
  15. [15]Shwai He, Daize Dong, Liang Ding, and Ang Li. 2025. Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques. Transactions on Machine Learning Research.arxiv.org/abs/2406.02500
  16. [16]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Understanding. In International Conference on Learning Representations.
  17. [17]Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the Knowledge in a Neural Network. Preprint, arXiv:1503.02531.arxiv.org/abs/1503.02531
  18. [18]Quzhe Huang, Zhenwei An, Nan Zhuang, Mingxu Tao, Chen Zhang, Yang Jin, Kun Xu, Kun Xu, Liwei Chen, Songfang Huang, and Yansong Feng. 2024. Harder task needs more experts: Dynamic routing in MoE models. In Annual Meeting of the Association for Computational Linguistics, pages 12883–12895, Bangkok, Thailand. Association for Computational Linguistics.
  19. [19]Sieun Hyeon and Jaeyoung Do. 2026. Is Retraining-Free Enough? The Necessity of Router Calibration for Efficient MoE Compression. Preprint, arXiv:2603.02217.
  20. [20]Ajay Jaiswal, Jianyu Wang, Yixiao Li, Pingzhi Li, Tianlong Chen, Zhangyang Wang, Chong Wang, Ruoming Pang, and Xianzhi Du. 2025. Finding Fantastic Experts in MoEs: A Unified Study for Expert Dropping Strategies and Observations. Preprint, arXiv:2504.05586.arxiv.org/abs/2504.05586
  21. [21]Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, and 7 others. 2024. Mixtral of Experts. Preprint, arXiv:2401.04088.arxiv.org/abs/2401.04088
  22. [22]Peng Jin, Bo Zhu, Li Yuan, and Shuicheng Yan. 2025. MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts. In International Conference on Learning Representations.arxiv.org/abs/2410.07348
  23. [23]Yeskendir Koishekenov, Alexandre Berard, and Vassilina Nikolova. 2023. Memory-efficient NLLB-200: Language-specific expert pruning of a massively multilingual machine translation model. In Annual Meeting of the Association for Computational Linguistics, pages 3567–3585, Toronto, Canada. Association for Computational Linguistics.
  24. [24]Mike Lasby, Ivan Lazarevich, Nish Sinnadurai, Sean Lie, Yani Ioannou, and Vithursan Thangarasa. 2026. REAP the Experts: Why Pruning Prevails for One-Shot MoE compression. In International Conference on Learning Representations.arxiv.org/abs/2510.13999
  25. [25]Jaeseong Lee, Seung-won Hwang, Aurick Qiao, Daniel F Campos, Zhewei Yao, and Yuxiong He. 2025. STUN: Structured-then-unstructured pruning for scalable MoE pruning. In Annual Meeting of the Association of the Association for Computational Linguistics, pages 13660–13676, Vienna, Austria. Association for Computational Linguistics.
  26. [26]Lujun Li, Qiyuan Zhu, Jiacheng Wang, Xiaoyu Qin, Wei Li, Hao Gu, Sirui Han, and Yike Guo. 2026. Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Merging. In AAAI Conference on Artificial Intelligence, volume 40, pages 22994–23002.
  27. [27]Pingzhi Li, Zhenyu Zhang, Prateek Yadav, Yi-Lin Sung, Yu Cheng, Mohit Bansal, and Tianlong Chen. 2024. Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy. In International Conference on Learning Representations.arxiv.org/abs/2310.01334
  28. [28]Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, En-zhe Lu, Junjie Yan, Yanru Chen, Huabin Zheng, Yibo Liu, Shaowei Liu, Bohong Yin, Weiran He, Han Zhu, Yuzhi Wang, Jianzhou Wang, and 9 others. 2025. Muon is Scalable for LLM Training. Preprint, arXiv:2502.16982.arxiv.org/abs/2502.16982
  29. [29]Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. In International Conference on Learning Representations.
  30. [30]Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang, Bo Zhang, Junchi Yan, and Hongsheng Li. 2024. Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experts large language models. In Annual Meeting of the Association for Computational Linguistics, pages 6159–6172, Bangkok, Thailand. Association for Computational Linguistics.arxiv.org/abs/2402.14800
  31. [31]Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. LLM-Pruner: On the Structural Pruning of Large Language Models. In Advances in Neural Information Processing Systems, volume 36, pages 21702–21720.arxiv.org/abs/2305.11627
  32. [32]Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Conference on Empirical Methods in Natural Language Processing, pages 2381–2391, Brussels, Belgium. Association for Computational Linguistics.arxiv.org/abs/1809.02789
  33. [33]OpenAI. 2025. gpt-oss-120b & gpt-oss-20b Model Card. Preprint, arXiv:2508.10925.
  34. [34]Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67.arxiv.org/abs/1910.10683
  35. [35]Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. WinoGrande: An adversarial winograd schema challenge at scale. In AAAI Conference on Artificial Intelligence, volume 34, pages 8732–8740.arxiv.org/abs/1907.10641
  36. [36]Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. In International Conference on Learning Representations.
  37. [37]Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. 2024. A Simple and Effective Pruning Approach for Large Language Models. In International Conference on Learning Representations.arxiv.org/abs/2306.11695
  38. [38]Yushuai Sun, Zikun Zhou, Lin Gao, Jun Yu, and Wenjie Pei. 2026. MoRA: MoE Pruning via Router Bias Learning and Expert Approximation. Preprint, arXiv:2610.00367.arxiv.org/abs/2610.00367
  39. [39]Shengkun Tang, Zekun Wang, Bo Zheng, Liangyu Wang, Rui Men, Siqi Zhang, Xiulong Yuan, Zihan Qiu, Zhiqiang Shen, and Dayiheng Liu. 2026. SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training. Preprint, arXiv:2605.08738.arxiv.org/abs/2605.08738
  40. [40]Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems. In Advances in Neural Information Processing Systems, volume 32.arxiv.org/abs/1905.00537
  41. [41]Juntong Wu, Jialiang Cheng, Fuyu Lv, Dan Ou, and Li Yuan. 2026a. SERE: Similarity-based Expert Rerouting for Efficient Batch Decoding in MoE Models. In International Conference on Learning Representations.arxiv.org/abs/2602.07616
  42. [42]Juntong Wu, Yifei Liu, Junyi Chen, Siqi Fan, Chaoran Feng, Minghao Li, Liujie Zhang, Weihang Chen, and Li Yuan. 2026b. ExFold: Unified Expert Folding for Training-Free MoE Prefill-Decode Acceleration. Preprint, arXiv:2608.24938.arxiv.org/abs/2608.24938
  43. [43]Yanyue Xie, Zhi Zhang, Ding Zhou, Cong Xie, Ziang Song, Xin Liu, Yanzhi Wang, Xue Lin, and An Xu. 2024. MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router. Preprint, arXiv:2410.12013.arxiv.org/abs/2410.12013
  44. [44]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Cheng'en Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Gao He, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 Technical Report. Preprint, arXiv:2505.09388.arxiv.org/abs/2505.09388
  45. [45]Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy. Association for Computational Linguistics.arxiv.org/abs/1905.07830
  46. [46]Zihao Zeng, Yibo Miao, Hongcheng Gao, Hao Zhang, and Zhijie Deng. 2024. AdaMoE: Token-adaptive routing with null experts for mixture-of-experts language models. In Findings of the Association for Computational Linguistics: EMNLP, pages 6223–6235, Miami, Florida, USA. Association for Computational Linguistics.arxiv.org/abs/2406.13233
  47. [47]Geng Zhang, Yuxuan Han, Yuxuan Lou, Yiqi Zhang, Wangbo Zhao, and Yang You. 2026. MoNE: Replacing Redundant Experts with Lightweight Novices for Structured Pruning of MoE. In International Conference on Learning Representations.arxiv.org/abs/2507.00390
  48. [48]Yuhao Zhang. 2026. SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMs. Preprint, arXiv:2606.09886.arxiv.org/abs/2606.09886
  49. [49]Zeliang Zhang, Xiaodong Liu, Hao Cheng, Chenliang Xu, and Jianfeng Gao. 2025. Diversifying the expert knowledge for task-agnostic pruning in sparse mixture-of-experts. In Findings of the Association for Computational Linguistics: ACL, pages 86–102, Vienna, Austria. Association for Computational Linguistics.arxiv.org/abs/2407.09590
  50. [50]Yixiao Zhou, Ziyu Zhao, Dongzhou Cheng, Zhiliang Wu, Jie Gui, Yi Yang, Fei Wu, Yu Cheng, and Hehe Fan. 2025. Dropping experts, recombining neurons: Retraining-free pruning for sparse mixture-of-experts LLMs. In Findings of the Association for Computational Linguistics: EMNLP, pages 15169–15186, Suzhou, China. Association for Computational Linguistics.arxiv.org/abs/2509.10377

Paper details

vlnc:W_3k3fn72j
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY-SA 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents