SPARE: Sparsity Allocation via Routing Entropy for Efficient MoEs

Introduction

The pursuit of stronger reasoning capabilities has pushed Large Language Models (LLMs) to the trillion-parameter scale, incurring prohibitive computational costs. The Mixture-of-Experts (MoEs) architecture addresses this challenge through conditional computation, which activates only a sparse subset of parameters per input token [16, 9, 7] (Shazeer et al. 2017). While conditional computation drastically reduces FLOPs, the memory overhead necessitates loading the full set of experts. Furthermore, the requirement to distribute these experts across devices complicates the communication topology, leading to efficiency bottlenecks. This highlights the urgent need for compression techniques capable of shrinking pre-trained MoEs.

To address this, recent research has focused on expert pruning, which removes entire experts to reduce model size. Existing approaches usually estimate an expert-level importance score from activation frequency, routing statistics, or reconstruction error [8, 15]. These scores provide useful criteria for selecting experts within a layer. However, using expert importance directly for cross-layer pruning is less reliable. Scores computed in different layers are not necessarily calibrated, since activation scales, expert functions, and token distributions vary across depth. The calibration domain can further change which experts appear important. As a result, globally ranking experts across all layers may over-prune sensitive layers [4, 15].

This limitation comes from conflating two decisions: inter-layer allocation, which determines how many experts to remove from each layer under a global budget, and intra-layer selection, which determines which experts to remove within a layer. Existing criteria such as REAP and EASY-EP address the second; SPARE addresses the first and can be paired with any of them. Non-uniform allocation has been explored in dense LLMs (Men et al., 2025; Yin et al., 2024) and, more recently, in MoE expert pruning [17] (Yang et al. 2024), using signals such as calibration loss or expert-usage frequency. SPARE instead allocates budgets by how gate weight is distributed among each token’s active experts, a training-free signal distinct from usage statistics across inputs.

To capture these routing dynamics, we use routing entropy: the Shannon entropy of each token’s normalized Top-$K$ gate weights, averaged over calibration tokens. As illustrated in Figure 1, high-entropy layers spread gate weight broadly among the active experts, so removing one active expert discards only a small share of each token’s weight, which is then reassigned to the remaining experts. Low-entropy layers concentrate weight on fewer experts, suggesting stronger input-conditioned specialization and making aggressive pruning riskier. This signal is layer-level rather than expert-level: it assumes neither that particular experts are indispensable nor that the same experts are selected across domains. Although expert identities shift between web text, code, math, and science, layer-wise entropy rankings remain stable across calibration domains, making them a robust basis for allocation.

Leveraging this insight, we introduce SPARE, a training-free sparsity allocation framework that assigns non-uniform pruning budgets across MoE layers. Our main contributions are:

• Entropy-guided, plug-and-play allocation. We formulate MoE expert pruning as an inter-layer budget allocation problem and replace uniform allocation with entropy-aware budgets. SPARE decides how many experts to prune in each layer, while existing criteria such as REAP and EASY-EP decide which ones, without retraining or modifying the router.

• Theoretical foundation. We derive a perturbation bound that, under uniform random pruning, grows with the routing entropy gap. The analysis provides a layer-level sensitivity signal and an existence argument for deterministic masks.

• Empirical validation. With the same selector and budget, SPARE improves average accuracy over uniform allocation on Mixtral, Qwen3, and DeepSeek models, especially at high pruning ratios. At 50% expert pruning, it raises the eight-task average of Qwen3-30B-A3B from 57.2% to 63.8%, and the math, code, and science average of DeepSeek-R1 from 80.1 to 82.8.

Related Work

Routing Heterogeneity. MoEs scale model capacity through conditional computation [9] (Shazeer et al. 2017). Despite load-balancing objectives, expert behavior can remain non-uniform: Su et al. (2025) identify “super-experts” that produce extreme activation outliers, and Lo et al. (2024) examine relationships between routing weights and expert outputs. These observations motivate studying routing heterogeneity across layers.

Expert Pruning and Compression. Expert pruning reduces the memory footprint of pretrained MoEs. Selection criteria use routing statistics [8, 14], router-weighted activation saliency [15], or reconstruction loss (Lu et al., 2024); EEP (Liu et al., 2024) instead uses evolutionary search. Shapley-MoE [11] ranks experts globally across layers via router-guided sampling; we exclude it from comparisons under fixed layer budgets, which would alter its selection rule. Chowdhury et al. [4] provide a pruning guarantee for fine-tuned sparse MoEs based on changes in router distributions. SEER-MoE (Muzio et al., 2024), MoE-SVD [18], and AdaWhiten [1] explore further pruning and decomposition strategies, alongside task-specific pruning [3] and benchmarks such as MC-Suite [13]. Expert merging, skipping, dynamic Top-$K$ routing, and intra-expert pruning (Xie et al., 2024) operate at other granularities; SPARE instead targets budget allocation.

Layer-wise Sparsity Allocation. In dense LLMs, ShortGPT (Men et al., 2025) studies block removal, whereas OWL (Yin et al., 2024) allocates weight sparsity using activation-aware outlier statistics. SlimGPT (Ling et al., 2024) also adopts non-uniform pruning schedules. For MoEs, MoE-I$^{2}$ (Yang et al., 2024) estimates layer importance from calibration losses after removing individual experts, assigns non-uniform pruning ratios, and selects experts through genetic and block-wise search; its full pipeline additionally includes low-rank decomposition and LoRA recovery. C-GNN-PRUNE [17] computes the entropy of expert-selection frequencies pooled over calibration samples and retains more experts in layers with higher usage entropy. Its expert-selection procedure uses learned graph embeddings. SPARE averages, over tokens, the entropy of each token’s normalized Top-$K$ gate weights and prunes more experts in layers with higher gate entropy. Usage entropy describes how broadly expert identities are used across inputs; gate entropy describes how evenly weight is spread among the experts active for a single token. The rules therefore treat high entropy in opposite ways without contradiction: they measure different distributions. SPARE contributes this routing signal and a selector-independent allocation rule.

Routing Entropy and Heterogeneity of MoE Layers

We analyze layer-wise routing heterogeneity in MoEs and define routing entropy as the signal used by SPARE for inter-layer budget allocation.

Problem Formulation

Consider an MoE layer $\ell$ with $N$ experts. For token $\mathbf{x}$, the router selects the Top-$K$ expert set $\mathcal{T}_{\ell}(\mathbf{x})$ and computes the normalized gating distribution

Let $\mathbf{m} = [m_0, \ldots, m_{L-1}]^{\top}$ denote the layer-wise pruning profile, where $m_{\ell}$ is the number of experts removed from layer $\ell$. Given a global target $M_{\mathrm{target}}$, SPARE solves the inter-layer allocation problem

while leaving the intra-layer choice of removed experts to an expert-ranking rule such as EASY-EP (Dong et al., [8]) or REAP (Lasby et al., [15]).

Routing Entropy Gap

For each token, we define the Top-$K$ routing entropy as

The layer-average entropy and routing entropy gap are

Equivalently,

where $u_{\mathcal{T}_{\ell}(\mathbf{x})}$ is the uniform distribution over the active experts. A larger $S_{\ell}$ means sharper routing concentration and motivates a more conservative pruning budget.

Layer-wise Routing Heterogeneity

Figure 1 shows that gating concentration varies across layers (left), while selected expert identities vary across domains (right). SPARE therefore allocates budgets with the layer-level statistic $\overline{H}_{\ell}$ rather than domain-specific expert frequencies. Routing entropy indicates how much gate weight a token loses when an active expert is removed, but not whether the remaining experts can compensate, so we test pruning tolerance directly.

We divide the 48 layers of Qwen3-30B-A3B into entropy terciles and use REAP to prune 50% of experts within one tercile at a time, removing about 16.7% of all routed experts. From an unpruned eight-task average of 72.1%, pruning the high-, medium-, and low-entropy terciles yields 68.2%, 67.2%, and 66.4%, respectively, supporting entropy as a pruning-tolerance signal in this setting (see also § 6.3 and § 6.6).

Theoretical Analysis: Routing-Entropy Perturbation Bounds

We analyze how routing entropy can guide the allocation of a fixed expert-pruning budget across layers. Under the stated routing and bounded-output assumptions, we first bound the output perturbation for any fixed pruning mask using the routing mass assigned to removed experts. For uniform random pruning, this bound takes an explicit form involving the routing entropy gap. We then give a sufficient condition under which a deterministic expert selector inherits the same entropy-dependent bound, with an additive term when the condition holds approximately. Finally, we describe how local pruning perturbations propagate through the network.

Single-Layer Pruning Perturbation

Consider a layer $\ell$ with output $y_\ell(\mathbf{x}) = \sum_j g_{\ell,j}(\mathbf{x})f_{\ell,j}(\mathbf{x})$. After removing $P \subseteq\{1,\ldots,N\}$, $|P| = m \le N-K$, we recompute Top-$K$ among remaining experts with unchanged logits and fixed tie-breaking, then apply softmax on the new support to obtain $g_{\ell}^{(P)}$. Write $y_\ell^{(P)}(\mathbf{x}) = \sum_j g_{\ell,j}^{(P)}(\mathbf{x})f_{\ell,j}(\mathbf{x})$ and $\delta_\ell(\mathbf{x},P) = y_\ell^{(P)}(\mathbf{x}) - y_\ell(\mathbf{x})$.

The following lemma quantifies the routing change.

Proof. The new softmax normalizer cannot increase, so surviving original Top-$K$ weights cannot decrease. Unpruned experts gain exactly the removed mass $E_\ell$, yielding an $\ell_1$ change of $2E_\ell$ (Appendix § 11.§ 3).

We first separate the mask-wise perturbation bound from the distribution used to select pruning masks.

Condition on fixed calibration data. For any token-independent distribution $q_\ell$ on $\mathcal{P}_{\ell,m}$, let $\mathcal{M}_\ell(q_\ell) = \mathbb{E}_{\mathbf{x},P\sim q_\ell}[E_\ell(\mathbf{x},P)^2]$. Then

A deterministic selector has $q_\ell(P) = \mathbf{1}\{P=P_A\}$.

Uniform random pruning gives a closed-form special case of Proposition 4.2.

If this second moment is no greater than the uniform random pruning baseline, $\varepsilon_{A,\ell}=0$ suffices.

Proofs of Proposition 4.2 and Corollary 4.4 are in Appendix § 11.§ 3. The condition in Eq. (10) is not established for REAP or EASY-EP.

End-to-End Propagation as a Weighting Factor

Consider an $L$-layer residual network with original states $h_\ell$, pruned states $\hat{h}_\ell$, and identical initial states. Assume deterministic Lipschitz constants $\lambda_\ell$ hold uniformly over admissible masks on a common domain containing both trajectories. Unrolling the error recursion gives:

The proof follows by discrete Gronwall unrolling; see Appendix § 11.§ 4. Taking expectations and applying Proposition 4.2 yields

Here $\mathcal{M}_\ell$ uses the original layer-input distribution, with masks independent of the evaluation input given fixed calibration data. Its bound is $\Psi_\ell(m_\ell)$ for uniform random pruning and $\Psi_\ell(m_\ell)+\varepsilon_{A,\ell}$ under Eq. (10). The decomposition motivates a propagation regularizer, without identifying an exact estimator of $\Gamma_\ell$.

Implications for Pruning Strategy

For fixed $(m,N,K,B_\ell)$ and selector residual, the bound increases with $S_\ell=\log K-\bar{H}_\ell$. SPARE does not minimize the summed bound; it uses $S_\ell$ as a layer-ranking signal and ignores the layer-dependent scale $B_\ell$. The allocation score in § 5 is therefore a heuristic.

Since exact Lipschitz products are intractable for large MoEs, SPARE uses a cosine-based proxy as an empirical propagation regularizer; entropy remains the primary allocation signal.

Entropy-Aware Sparsity Allocation

SPARE converts layer-wise routing entropy into an integer pruning profile under the global budget. It uses entropy as the primary signal and an optional propagation-aware term as a lightweight regularizer.

Entropy-Based Allocation Score

Since $S_\ell=\log K-\bar{H}_\ell$, smaller entropy gap is equivalent to larger routing entropy. We therefore define the normalized allocation score

Layers with larger $\mathcal{R}_\ell$ are assigned larger pruning budgets. A contrast coefficient $p$ controls the sharpness of allocation:

Propagation-Aware Regularization

Equation (12) motivates accounting for downstream propagation, but exact Lipschitz products are intractable for large MoEs. We use cosine similarity only as a data-dependent proxy. Let $c_k$ be the average cosine similarity between the input and output activations of layer $k$. We define

This term mildly reduces the pruning weight of layers whose perturbations pass through more transformative downstream blocks. When $\beta=0$, SPARE reduces to entropy-only allocation.

Discrete Budget Projection

The final continuous allocation weight is

We convert $\{\mathcal{W}_\ell\}$ into an integer pruning profile by solving for a global scaling factor $\lambda^*$:

where $\mathcal{P}_{\mathbb{Z}}(\cdot)$ rounds to the nearest feasible integer under $0\le m_\ell\le N-K$. Since the left-hand side is monotone in $\lambda^*$, we solve it by bisection and obtain the final profile $\mathbf{m}$.

Experiments

We ask three questions: (RQ1) Does entropy-guided allocation improve benchmark performance over uniform allocation at the same pruning budget? (RQ2) Does the gain persist across the evaluated expert selectors, and does entropy correlate with pruning risk when the selector is fixed? (RQ3) How stable are entropy profiles across calibration domains, and how does non-uniform allocation affect throughput in the tested implementation?

Experimental Setup

Models. We evaluate Mixtral-8x7B-Instruct-v0.1, Qwen3-30B-A3B, Qwen3-235B-A22B, and DeepSeek-R1.

Benchmarks. We assess capabilities across four distinct domains: (i) Multiple-choice tasks: MMLU (Hendrycks et al., [10]), ARC-Challenge (ARC-c) / ARC-Easy (ARC-e) (Clark et al., [6]), BoolQ (Clark et al., [5]), HellaSwag (Hella) (Zellers et al., 2019), OpenBookQA (OBQA) (Mihaylov et al., 2018), Recognizing Textual Entailment Challenge (RTE) (Bentivogli et al., [2]), and WinoGrande (WinG) (Sakaguchi et al., 2021); (ii) Reasoning: MATH-500 (Lightman et al., [19]); (iii) Science question answering: GPQA-Diamond (Rein et al., 2023); and (iv) Code: LiveCodeBench (Jain et al., [12]).

Calibration. For Mixtral and Qwen3-30B-A3B, we use a calibration set of 1024 randomly sampled sequences from C4 to estimate layer-average routing entropy ($\bar{H}_{\ell}$). For large-scale reasoning models such as DeepSeek-R1, we sample 25 instances each from AIME (math), LiveCodeBench (code), and GPQA (science).

Baselines and Protocols. To disentangle the benefits of sparsity allocation from expert selection, we structure our comparison along two orthogonal dimensions. 1) Intra-layer Expert Ranking: Determines expert priority within a layer. We employ: (i) Random; (ii) EASY-EP (Dong et al., [8]); and (iii) REAP (Lasby et al., [15]). 2) Inter-layer Sparsity Allocation: Determines the pruning budget $m_{\ell}$ per layer. We compare: (i) Uniform; (ii) OWL (Yin et al., 2024); and (iii) SPARE.

Main Results

Table 1 shows improvements in macro-average accuracy in all ten evaluated combinations of model, pruning ratio, and intra-layer selector. The largest gain occurs on Qwen3-30B-A3B at 50% pruning: with EASY-EP fixed, SPARE raises mean accuracy from 57.2% to 63.8%, a gain of 6.6 percentage points. At the same pruning ratio, gains with Random and REAP are 0.9 and 2.3 points, respectively, showing that the benefit depends on the selector. Improvements at 25% pruning are smaller, ranging from 0.1 to 0.7 points.

Model	PR	Inter.	Intra.	ARC-c	ARC-e	BoolQ	Hella	MMLU	OBQA	RTE	WinG	Avg.↑
Mixtral-8x7B -Instruct-v0.1	0%	Baseline		65.0	84.2	88.7	86.1	69.1	49.6	72.2	74.0	73.6
Mixtral-8x7B -Instruct-v0.1	25%	Uni.	Random	59.6	81.4	86.9	82.9	62.7	46.8	71.1	75.6	70.9
Mixtral-8x7B -Instruct-v0.1	25%	Uni.	REAP	60.6	80.7	87.5	83.5	63.3	48.6	71.9	70.9	70.9
Mixtral-8x7B -Instruct-v0.1	25%	SPARE	Random	60.1	81.9	87.1	83.2	63.1	47.5	71.9	75.8	71.3 (↑0.4)
Mixtral-8x7B -Instruct-v0.1	25%	SPARE	REAP	61.8	81.5	87.9	84.0	64.5	49.2	71.9	72.1	71.6 (↑0.7)
Mixtral-8x7B -Instruct-v0.1	50%	Uni.	Random	51.4	73.3	81.5	75.7	52.1	41.0	70.2	69.5	64.3
Mixtral-8x7B -Instruct-v0.1	50%	Uni.	REAP	50.9	76.2	85.3	78.1	51.9	44.6	71.6	72.5	66.4
Mixtral-8x7B -Instruct-v0.1	50%	SPARE	Random	50.9	75.5	81.7	77.2	54.4	42.0	70.8	70.2	65.3 (↑1.0)
Mixtral-8x7B -Instruct-v0.1	50%	SPARE	REAP	53.6	78.4	86.9	78.2	53.1	44.4	70.4	74.1	67.4 (↑1.0)
Qwen3-30B-A3B	0%	Baseline		56.2	79.1	88.3	77.6	77.9	44.6	81.6	71.1	72.1
Qwen3-30B-A3B	25%	Uni.	Random	50.8	75.6	84.8	70.7	69.7	40.2	80.5	68.8	67.6
Qwen3-30B-A3B	25%	Uni.	REAP	55.8	78.3	87.1	75.8	71.9	44.6	79.4	70.5	70.4
Qwen3-30B-A3B	25%	Uni.	Easy-EP	53.2	78.0	88.8	76.6	74.2	44.0	81.9	70.7	70.9
Qwen3-30B-A3B	25%	SPARE	Random	50.6	76.8	85.5	70.8	68.8	41.2	79.0	69.0	67.7 (↑0.1)
Qwen3-30B-A3B	25%	SPARE	REAP	56.0	78.4	87.5	76.0	72.1	44.8	80.1	70.6	70.7 (↑0.3)
Qwen3-30B-A3B	25%	SPARE	Easy-EP	53.8	78.4	89.0	76.9	74.4	44.6	81.5	69.4	71.0 (↑0.1)
Qwen3-30B-A3B	50%	Uni.	Random	40.0	62.8	77.9	55.5	49.6	25.4	74.1	61.1	55.8
Qwen3-30B-A3B	50%	Uni.	REAP	40.5	61.2	82.9	64.4	51.1	36.2	61.0	64.5	57.7
Qwen3-30B-A3B	50%	Uni.	Easy-EP	37.2	58.1	81.0	66.7	49.0	37.6	63.2	64.7	57.2
Qwen3-30B-A3B	50%	SPARE	Random	39.8	63.1	77.1	55.9	50.6	31.8	74.0	61.2	56.7 (↑0.9)
Qwen3-30B-A3B	50%	SPARE	REAP	43.1	64.6	83.3	64.5	48.9	37.0	74.7	63.6	60.0 (↑2.3)
Qwen3-30B-A3B	50%	SPARE	Easy-EP	42.5	65.2	84.9	73.2	59.1	41.2	75.8	68.4	63.8 (↑6.6)

Controlled Analysis of Pruning Risk

Entropy groups with a fixed selector. On Mixtral-8×7B, we estimate routing entropy using 128 C4 samples and divide layers into high-, middle-, and low-entropy groups. Within each group, we use the same selector, REAP, to prune four of the eight experts per layer while retaining the expert with the highest REAP score. Table 3 reports normalized output perturbation on held-out samples. The low-entropy group has higher perturbation than the high-entropy group in all three domains, although the middle group does not follow a uniformly monotonic ordering. The point estimates indicate an association under this selection procedure.

Domain	High	Middle	Low
WikiText2	0.721	0.641	0.726
MATH-500	0.673	0.736	0.728
HumanEval	0.726	0.725	0.763

Single-layer pruning and output-distribution changes. We next test whether this association holds when each layer is pruned in isolation. For each MoE layer, we remove its lowest-saliency experts under REAP, with saliency estimated on C4, while keeping all other layers intact; the pruning ratios in Table 4 apply only to the intervened layer. We measure the resulting output-distribution change on C4, Math, Code, and MMLU with KL divergence truncated to the unpruned model’s top-64 tokens, and summarize each layer by its mean over the four domains and by its worst (maximum) domain. We define routing concentration as $C_{\ell}=1-\bar{H}_{\ell}/\log K=S_{\ell}/\log K$, so a larger $C_{\ell}$ corresponds to lower routing entropy. In both models, at both pruning ratios, and under both aggregations, the correlation between $C_{\ell}$ and pruning-induced KL is positive, ranging from 0.26 to 0.45. These correlations are moderate: routing concentration accounts for only part of the layer-to-layer variation in pruning risk. We therefore use it to rank layers for budget allocation, not to predict the perturbation of an individual layer.

Model	PR	Mean-domain KL	Worst-domain KL
DeepSeek-V2-Lite	25%	+0.435	+0.450
DeepSeek-V2-Lite	50%	+0.340	+0.272
Qwen3-30B-A3B	25%	+0.288	+0.257
Qwen3-30B-A3B	50%	+0.314	+0.329

Scalability to Large-scale Reasoning MoEs

At 50% global expert pruning, we compare SPARE and uniform allocation on Qwen3-235B-A22B and DeepSeek-R1, using EASY-EP for both (Table 2). On both models, SPARE narrows the gap to the unpruned model in the unweighted three-score mean. It raises this mean from 80.1 to 82.8 on DeepSeek-R1, compared with 83.4 before pruning, and from 86.4 to 87.9 on Qwen3-235B-A22B. On DeepSeek-R1, GPQA-Diamond improves by 5.5 points and

Model	PR	Inter.	LCB	Math	GPQA	Avg.↑
DS-R1	–	Baseline	84.5	96.0	69.7	83.4
DS-R1	50%	Uni.	79.0	94.7	66.7	80.1
DS-R1	50%	SPARE	82.0	94.3	72.2	82.8 (↑2.7)
Qwen3	–	Baseline	92.5	96.0	77.5	88.7
Qwen3	50%	Uni.	89.0	95.7	74.4	86.4
Qwen3	50%	SPARE	93.0	95.7	75.1	87.9 (↑1.5)

LiveCodeBench by 3.0 points over uniform allocation, while MATH-500 declines by 0.4 points. Deployment efficiency is reported in Appendix § 9.

Comparison with Layer-wise Allocation Baselines

We compare SPARE with OWL ((Yin et al., 2024)) adapted from weight-sparsity allocation to expert pruning, holding EASY-EP fixed for both methods. On Qwen3-30B-A3B, average accuracy is comparable at 12.5% pruning and higher with SPARE at 25%, 37.5%, and 50% (Appendix § 10). Direct comparisons with other MoE-specific allocation pipelines remain outside the evaluation.

Calibration Robustness

We compare DeepSeek-R1 layer-average entropy across four calibration domains: C4 (web), MATH-500 (math), LiveCodeBench (code), and GPQA-Diamond (science). Table 5 reports correlations relative to C4. Spearman correlations range from 0.958 to 0.974, indicating similar layer rankings across these domains. This analysis measures entropy-profile stability; it does not establish identical integer budgets or unchanged downstream performance under every calibration choice.

Model	Calibration analysis	Pearson $r$	Spearman $\rho$
DeepSeek-R1	C4 vs. MATH-500	0.988	0.962
DeepSeek-R1	C4 vs. LiveCodeBench	0.991	0.974
DeepSeek-R1	C4 vs. GPQA-Diamond	0.979	0.958

Ablation on Hyper-parameters

We run two separate sweeps on Qwen3-30B-A3B with EASY-EP at 50% global expert pruning (Table 6). The $p$ sweep fixes $\beta= 0$: mean accuracy peaks at $p = 2.0$ (63.2%) and falls to 58.5% at $p = 2.5$, showing that stronger contrast need not help. The $\beta$ sweep fixes $p = 2.0$: scores range

Param	0.8	1.0	1.5	2.0	2.5	Param	0.0	0.2	0.4	0.6
Avg.$\uparrow$	61.9	61.5	62.8	63.2	58.5	Avg.$\uparrow$	63.2	63.1	63.1	63.8

from 63.1% to 63.8%, peaking at $\beta= 0.6$. These are separate sweeps, not a joint parameter grid. The allocation profiles in Figure 2 show that larger $p$ increases the contrast between more- and less-pruned layers, while the profiles for $\beta\in[0, 0.6]$ remain close. We selected $p = 2.0$ and $\beta= 0.6$ on the reported Qwen3-30B-A3B evaluation benchmarks and fixed them for all other models.

Conclusion and Future Work

We introduced SPARE, a training-free method that uses token-level routing entropy to allocate expert-pruning budgets across MoE language-model layers and can be paired with existing intra-layer selectors. A perturbation analysis and controlled interventions with a fixed selector associate concentrated routing with greater pruning risk. Under the same expert selector and global budget, SPARE improves average performance in the evaluated settings, with larger gains at high pruning ratios: at 50% expert pruning, it raises the Qwen3-30B-A3B eight-task average from 57.2% to 63.8%, and the DeepSeek-R1 three-benchmark average from 80.1 to 82.8, close to the unpruned 83.4. Future work includes studying calibration shift and combining entropy-guided allocation with quantization and expert merging.

Limitations

SPARE yields its largest gains at aggressive pruning ratios; at moderate ratios such as 25%, its improvements over uniform allocation are small. We study SPARE only for expert pruning; combining it with quantization or expert merging is left for future work.

Additional results

To further validate the applicability of SPARE to fine-grained MoE architectures, we extend our evaluation to the recent GLM-5. Our main experiments already cover representative fine-grained MoEs, including Qwen3-30B-A3B, Qwen3-235B-A22B, and DeepSeek-R1. As summarized in Table 7, these models differ in the number of routed experts, the use of shared experts, and the number of MoE layers. GLM-5 follows a structure similar to DeepSeek-R1, with 256 routed experts, top-8 routing, and one shared expert, but increases the number of layers from 61 to 78.

Model	Routed Experts	Top-$K$	Shared	Layers
DeepSeek-R1 (671B)	256	8	Yes (1)	61
Qwen3-30B-A3B	128	8	No	48
Qwen3-235B-A22B	128	8	No	94
GLM-5	256	8	Yes (1)	78

As in § 6.4, both SPARE and uniform allocation use EASY-EP at 50% global expert pruning. On GLM-5, SPARE improves the average accuracy from 87.5% to 89.8%, yielding a +2.3 point gain over uniform allocation. This additional result extends the observed average improvement to another fine-grained MoE architecture.

Deployment efficiency. At the same global pruning budget, SPARE and uniform allocation remove the same total number of experts, giving equal routed-expert parameter storage when expert sizes and precision match. We use modified fused MoE Triton kernels to support variable per-layer expert counts. Table 8 reports throughput on a DeepSeek-style architecture at 50% pruning. Relative differences from uniform allocation range from a 1.16% decrease to a 0.56% increase across the tested batch sizes. Throughput is similar in these measured configurations.

Batch size	1	16	32	64
Uniform	8.61	172.32	181.54	162.31
SPARE	8.51	172.57	182.56	162.24

Additionally, non-uniform allocation opens future optimization possibilities: layers retaining more experts can use padded expert placement to co-locate the most frequently activated experts, potentially reducing HBM transfer beyond what uniform pruning achieves.

Comparison with Layer-wise Allocation Methods

We compare SPARE with the adapted OWL allocation on Qwen3-30B-A3B, holding EASY-EP fixed for both methods. Figure 3 plots average benchmark accuracy against the global expert-pruning ratio. The methods are comparable at 12.5% pruning; SPARE has higher average accuracy at 25%, 37.5%, and 50%. OWL uses activation-aware weight outlier statistics, whereas SPARE uses token-level routing entropy.

Proof of Main Theorem

Notation and Mathematical Definitions

Data and Tokens

Let $\mathcal{D}$ denote the target token distribution (e.g., pruning-evaluation domain). We write $x \sim\mathcal{D}$ for a random token representation entering a given MoE layer. In practice, $\mathcal{D}$ is approximated by a finite calibration set of $M$ tokens, $\mathcal{D} = \{x_1, \ldots, x_M\}$.

MoE Layer, Experts, and Router

Consider a fixed MoE layer $\ell$ with:

• $N$ experts indexed by $j \in\{1, \ldots, N\}$, each mapping token input $x$ to an output vector $f_{\ell,j}(x) \in\mathbb{R}^{d}$ (dimension $d$ depends on the model).

• Router logits $z_{\ell}(x) = (z_{\ell,1}(x), \ldots, z_{\ell,N}(x)) \in\mathbb{R}^{N}$.

Top-K Routing and Gating Distribution

Let $K < N$ be the number of selected experts per token. Define the Top-K index set

where $\operatorname{TopK}(\cdot)$ returns indices of the $K$ largest components, using a fixed ordering of expert indices to break ties before and after pruning.

The Top-K gating distribution $g_{\ell}(x)$ is defined by softmax restricted to $T_{\ell}(x)$:

Thus $g_{\ell,j}(x) \ge0$ and $\sum_{j=1}^{N} g_{\ell,j}(x) = 1$, but only $K$ entries are nonzero.

Layer Output (MoE Branch)

The MoE layer output (restricted to the expert mixture branch) is

Pruning Operation and Top-K Rerouting

Let $P \subseteq\{1,\ldots,N\}$ denote the removed expert set, with $|P| = m \le N-K$. Router logits are unchanged by pruning. We recompute Top-K over the remaining experts, retaining their original indices:

This uses the same tie-breaking rule as the original routing. The removed experts that were originally active and their gate mass are

where $\mathcal{E}_{\ell}(x,P)$ is the total probability mass assigned to pruned experts within the original Top-K support.

Top-K Routing and Gating Distribution

The post-pruning gating distribution is softmax over the recomputed support:

Since $N-m \ge K$, this remains well defined even when every originally active expert is removed. The post-pruning output is $y_{\ell}^{(P)}(x) = \sum_j g_{\ell,j}^{(P)}(x)f_{\ell,j}(x)$, and the perturbation is $\delta_{\ell}(x,P) = y_{\ell}^{(P)}(x) - y_{\ell}(x)$.

Routing Entropy and Concentration

For a Top-K sparse distribution $g_{\ell}(x)$, define Shannon entropy

where the convention $0\log0 = 0$ is used. Since only $K$ entries are nonzero, we have $0 \le H(g_{\ell}(x)) \le\log K$.

Define the layer-average entropy and routing entropy gap as in Eq. (4). Equivalently, let $u_{T_{\ell}(x)}$ denote the uniform distribution on the Top-K support $T_{\ell}(x)$: $u_{T_{\ell}(x)}(j) = 1/K$ if $j \in T_{\ell}(x)$ and $0$ otherwise. Then

Interpretation. Larger $S_{\ell}$ means greater routing concentration: the gating distribution is further from uniform on its active support.

Assumptions

We state the assumption used to relate routing concentration to pruning perturbation.

Assumption 1 (Output scale control). For each layer $\ell$, there exists $B_{\ell} > 0$ such that

where $P$ ranges over admissible pruning sets with $|P| \le N-K$.

Single-Layer Pruning Perturbation

Proofs for general and deterministic selectors. For Proposition 4.2, the triangle inequality, bounded expert outputs, and the routing identity proved below give, pointwise for every admissible $x$ and $P$,

Squaring before taking expectations yields Eqs. (7) and (8) for any mask distribution $q_\ell$. A deterministic selector is the special case with all probability on its returned mask. All expectations are conditional on fixed calibration data and are taken over the stated evaluation-token distribution.

For Corollary 4.4, the bracketed term in Eq. (9) is

Lemmas C.3 and C.4 give

Combining this with Eqs. (7) and (10) proves Eq. (11). In particular, a selector that minimizes $\mathbb{E}_x[\mathcal{E}_\ell(x,P)^2]$ over the finite mask family satisfies the condition with $\varepsilon_{A,\ell}=0$, since its minimum cannot exceed the uniform average. This observation does not establish the condition for selectors optimizing a different saliency criterion. A residual measured on calibration samples is an empirical estimate and does not by itself certify the condition on the evaluation distribution.

Proof Sketch for Main Theorem. Pruning a subset $P$ changes the Top-$K$ routing distribution from $g_\ell(x)$ to $g_\ell^{(P)}(x)$. Let $\mathcal{E}_\ell(x,P)$ denote the total gate mass assigned to experts that are both pruned and originally selected. A key identity shows that the routing change is exact: $\lVert g_\ell^{(P)}(x)-g_\ell(x)\rVert_1=2\mathcal{E}_\ell(x,P)$ (Lemma C.1). Since the MoE output is a convex combination of bounded expert outputs, the output perturbation satisfies $\lVert\delta_\ell(x,P)\rVert_2^2\le4B_\ell^2\mathcal{E}_\ell(x,P)^2$ (Lemma C.2). Under uniform random pruning, $\mathbb{E}_P[\mathcal{E}_\ell(x,P)^2]$ decomposes into a constant term and a collision term proportional to $Q_\ell(x)=\sum_j g_{\ell,j}(x)^2$ (Lemma C.3). Finally, for Top-$K$ sparse routing, the collision probability is upper bounded by the entropy gap $Q_\ell(x)\le\frac{1}{K}+2(\log K-H(g_\ell(x)))$ (Lemma C.4), which yields the stated bound and establishes monotonic dependence on routing entropy gap.

Proof. For brevity, suppress layer $\ell$ and token $x$ in notation. Let $T\triangleq T_\ell(x)$ and $T_P\triangleq T_{\ell,P}(x)$. Define

Unchanged logits and consistent tie-breaking imply $U\subseteq T_P$. Thus $T=U\cup D$ and $T_P=U\cup R$, with $|R|=|D|$ since both supports have size $K$.

Define the Top-$K$ softmax normalizers:

For any $u\in U$, we have

The factor $\alpha$ is common to all $u\in U$ because softmax shares the denominator $Z_P$; if $U$ is empty, the statements about its entries are vacuous.

Next, since $T$ consists of the top-$K$ logits, every $r\in R$ satisfies $z_r\le\min_{t\in T}z_t$, while every $d\in D$ satisfies $z_d\ge\min_{t\in T}z_t$. Because $|R|=|D|$, it follows that

Hence $\alpha=Z/Z_P\ge1$.

Now compute the $\ell_1$ difference by partitioning indices into $D$, $U$, $R$ and the remainder:

• For $d\in D$, $g_d>0$ but $g_d^{(P)}=0$, so $|g_d^{(P)}-g_d|=g_d$.

• For $u\in U$, $g_u^{(P)}=\alpha g_u\ge g_u$, so $|g_u^{(P)}-g_u|=(\alpha-1)g_u$.

• For $r\in R$, $g_r=0$ and $g_r^{(P)}>0$, so $|g_r^{(P)}-g_r|=g_r^{(P)}$.

• All other indices have both weights 0 and contribute nothing.

Therefore,

By definition, we have $L\coloneqq\sum_{d\in D}g_d$. Because $\sum_j g_j=1$ and $\sum_j g_j^{(P)}=1$, and $g_d^{(P)}=0$ for $d\in D$, we have

Moreover, for $i\notin D$ we have $g_i^{(P)}\ge g_i$ (since $U$ entries scale up and $R$ entries become positive), so

Combining with $\sum_{d\in D}\lvert g_d^{(P)}-g_d\rvert=\sum_{d\in D}g_d=L$ yields

which proves the claim.

Intuition. The pruned experts in $D$ lose exactly $L$ probability mass (their original total gate mass), and because the gating distribution must still sum to 1, the remaining indices must gain exactly $L$ mass in total. Since the gain occurs only on indices where $g^{(P)}\ge g$ (common scaling on retained Top-K plus newly activated replacements), the $\ell_1$ distance equals (loss $L$) + (gain $L$) = $2L$.

Lemma C.2 (Perturbation bound: bounded-norm variant). Under Assumption 1 in (26), for any pruned set $P$, we have

Consequently,

Proof. By definition,

Apply triangle inequality we have:

Assumption 1 gives the pointwise inequality

Taking expectation over $x$ yields

The equality in (29) follows from Lemma C.1. Squaring the pointwise inequality before taking expectation yields (30).

Intuition. The MoE output is a convex combination of expert outputs. Pruning changes only the mixing weights. Thus, the output perturbation is controlled by (i) how much the gating distribution changes ($\lVert g^{(P)}-g\rVert$), and (ii) the scale of expert outputs. Lemma C.1 makes (i) exact in terms of the pruned gate mass $L$.

Lemma C.3 (uniform random pruning). Fix $m\in\{1,\ldots,N-1\}$. Let $P$ be drawn uniformly at random from all subsets of size $m$. Then for any fixed $x$,

where

Proof. Let $I_j=1\{j\in P\}$ be the inclusion indicator. Then

Therefore

where we suppress $\ell,x$ in $g$ for readability. For uniform sampling without replacement, $\mathbb{E}[I_j]=m/N$ and $\mathbb{E}[I_jI_k]=\frac{m(m-1)}{N(N-1)}$ for $j\ne k$. Taking expectation over $P$ conditional on $x$:

Note that $\sum_{j<k} g_jg_k = \frac{1}{2}\left[\left(\sum_j g_j\right)^2-\sum_j g_j^2\right] = \frac{1}{2}(1-Q)$ since $\sum_j g_j=1$. Hence

Simplifying the coefficient yields $c_1(m,N)=\frac{m(N-m)}{N(N-1)}$, proving (31).

Intuition. Randomly pruning $m$ experts is like sampling $m$ indices. The second moment of the removed probability mass depends on how concentrated $g$ is. Concentration is captured by collision mass $Q=\lVert g\rVert_2^2$.

Proof. Let $T=T_\ell(x)$ be the support (Top-K set) of $g=g_\ell(x)$. Let $u$ be the uniform distribution on $T$, i.e., $u_j=1/K$ for $j\in T$. Then

where the equality 1 holds because:

Using $\lVert v\rVert_2\leq\lVert v\rVert_1$ and Pinsker’s inequality $\lVert g-u\rVert_1\leq2\operatorname{KL}(g\Vert u)$, we have

Finally, $\operatorname{KL}(g\Vert u)=\log K-H(g)$ (by direct algebra, see (25)), which yields (33).

Intuition. Among distributions on $K$ points, the uniform distribution minimizes collision mass ($Q=1/K$). Deviations from uniformity increase $Q$, and the entropy gap $\log K-H(g)$ quantifies such deviation.

With these lemmas, we can now prove our main theorem.

Proof. Combine Lemma C.2 with Lemma C.3 and Lemma C.4. Specifically, for any admissible pruned set $P$, Lemma C.2 gives $\mathbb{E}_x[\lVert\delta_\ell(x,P)\rVert_1^2]\leq4B_\ell^2\mathbb{E}_x[\mathcal{E}_\ell(x,P)^2]$. Taking expectation over random $P$ conditional on $x$ and applying Lemma C.3 yields dependence on $Q_\ell(x)$. Finally, Lemma C.4 upper bounds $Q_\ell(x)$ by a function of $\log K-H(g_\ell(x))$, and taking expectation over $x$ gives $\log K-\bar{H}_\ell$.

Interpretation. For fixed $(m,N,K)$ and layer scale $B_\ell$, the upper bound increases monotonically with

Thus, for fixed $(m,N,K,B_\ell)$, a larger entropy gap increases this upper bound, without establishing an ordering of actual perturbations across layers.

The minimum over the finite mask family cannot exceed its uniform average. This proves the existence of at least one deterministic mask satisfying the entropy-dependent bound, without identifying a particular selector’s output.

End-to-End Expert Pruning Error Bound via Discrete Gronwall

Model: Residual Network with MoE Blocks

Let $h_0 \in\mathbb{R}^d$ denote the input representation. We define a residual network of depth $L$ with residual scaling factor $\alpha> 0$ according to the recursion:

where $F_\ell$ represents the layer transition map. We assume $F_\ell$ decomposes into an unpruned component $A_\ell$ (encapsulating operations such as attention and normalization) and a modifiable component $y_\ell$ (e.g., a Mixture-of-Experts branch). Under a pruning policy $P_\ell$, the original and pruned transformations are defined respectively as:

Consequently, for any input $h$, the approximation error introduced by pruning is given by:

The dynamics of the pruned network are therefore governed by the recursion:

initialized with the same state $\widehat{h}_0 = h_0$.

Key Lemmas: One-step recursion and discrete Gronwall bound

Proof. Subtracting (36) from (39) and applying the triangle inequality yields:

We decompose the functional difference by adding and subtracting $\widehat{F}_\ell(h_\ell)$:

where the second inequality follows from the Lipschitz property (Assumption A2) and the definition of the injection error {(xref)}(38). Substituting this back into the expression for $E_{\ell+1}$ concludes the proof.

Lemma C.10 (Global error bound via Discrete Gronwall). Suppose $\hat{h}_0 = h_0$ (i.e., $E_0 = 0$). Then, for any fixed sequence of pruning masks $\{P_\ell\}$, the final layer error satisfies:

Here, we adopt the convention that the empty product $\prod_{k=L}^{L-1}(\cdot)$ equals 1.

Note that the product $\prod_{k=\ell+1}^{L-1}(1+\alpha\lambda_k)$ appears because the perturbation injected at layer $\ell$ is subsequently amplified only by later layers.

Proof. We proceed by induction or simply unrolling the recurrence from Lemma C.9. For $E_0 = 0$, we have:

Iterating this substitution $L$ times yields the summation in {(xref)}(45). The term $\lVert\delta_\ell\rVert_1$ is introduced at step $\ell$ and is subsequently scaled by the expansion factors $(1+\alpha\lambda_k)$ for all future layers $k>\ell$.

Lemma C.11 (Exponential upper bound). For any non-negative sequence $\{\lambda_k\}$, the product term admits the exponential upper bound:

Proof. The result follows immediately from the elementary inequality $1+x\le e^x$ for $x\ge0$. Taking the product over $k$ yields $\prod(1+\alpha\lambda_k)\le\prod e^{\alpha\lambda_k}=e^{\alpha\sum\lambda_k}$.

Main Theorem: End-to-end Expected Perturbation Bound

Theorem C.12 (End-to-end perturbation upper bound under layer-wise random pruning). Suppose Assumptions A1–A3 hold. Let the network follow the recursion defined in {(xref)}(36)–(39). The expected $\ell_1$-norm perturbation at the final layer satisfies:

where $w_\ell$ denotes the downstream propagation factor for layer $\ell$:

For compactness, define the entropy-dependent local term

Substituting the single-layer bound {(xref)}(34) then yields

Finally, applying the exponential upper bound from Lemma C.11, we obtain:

Proof. We commence with the unrolled discrete Gronwall bound established in Lemma C.10 (Eq. (45)):

Taking the expectation with respect to the initialization $h_0\sim\mathcal{D}_0$ and the sequence of independent pruning masks $\{P_\ell\}$, we have:

Since $f(z)=z^2$ is convex, Jensen’s inequality implies $\mathbb{E}[X]\le\sqrt{\mathbb{E}[X^2]}$ for any non-negative random variable $X$. Applying this to the error term:

By definition, the distribution of the layer input $h_\ell$ is denoted by $\mathcal{D}_\ell$. Under Assumption A3, the mask $P_\ell$ is independent of $h_\ell$ and uniformly distributed. We can therefore rewrite the expectation over the joint distribution:

Substituting (55) into (54) yields the generic bound (48).

To derive the explicit forms, we substitute the single-layer result (34) for the squared error term into (48), which directly yields (51). The final exponential form (52) follows by applying the inequality $\prod(1+x)\le\exp(\sum x)$ to the propagation weights $w_\ell$.

Corollaries and Design Implications

Corollary C.13 (Stability under Depth-scaled Initialization). Consider a scaling regime where $\alpha=\frac{1}{L}$. The error bound (52) simplifies to:

If the Lipschitz constants satisfy $\lambda_k\le\lambda_{\max}$, the propagation factor obeys $\exp\left(\sum_{k=\ell+1}^{L-1}\frac{\lambda_k}{L}\right)\le e^{\lambda_{\max}}$. Under the stated depth scaling, this factor is bounded independently of $L$.

Remark C.14 (Scope of the allocation surrogate). For large $N_\ell$ at low removal ratios $m_\ell/N_\ell\ll1$, $c_0$ is second order in the removal ratio and $c_1\approx m_\ell/N_\ell$. This approximation does not apply to the main 50% pruning setting, where both coefficients approach $1/4$ as $N_\ell$ grows. Moreover, the coefficients depend on the removal budget and cannot be discarded when optimizing the bound. SPARE uses routing concentration only as a layer-ranking heuristic:

The allocation rule does not minimize the summed perturbation bound and omits the layer-dependent output scale $B_\ell$.

Communication cost analysis

2 nodes $\rightarrow$ 1 node: cross-node EP traffic savings. In EP, each MoE layer incurs two all-to-all exchanges (dispatch + combine), whose activation payload is $2Nkdb$ bytes per layer, where $N$ is the number of routed tokens per step (prefill: $N=B\times T$, decode: $N=B$), $d$ is hidden width, $k$ is top-$k$ routing, and $b$ is bytes/element. When the EP group spans $S=2$ nodes and expert ownership is balanced, the expected cross-node fraction is $p_{\mathrm{cross}}\approx1-\frac{1}{2}=\frac{1}{2}$. Therefore, the cross-node all-to-all volume per step across $L_{\mathrm{moe}}$ MoE layers is

Plugging DeepSeek-R1-like settings ($k=8$, $d=7168$, $b=2$ bytes, $L_{\mathrm{moe}}=58$),

If pruning enables consolidating EP from $S=2$ to $S=1$, then $p_{\mathrm{cross}}=0$ and the cross-node traffic vanishes. Hence the saved cross-node communication is simply

For example, in decode with batch $B$, $N=B$ so the saving is $\approx6.34$ MiB $\times B$ per step.