betaFeedback

WorksComputer Science

From Local Prediction Error to Final Outputs: A Theory of Expert Sharing in Multi-Step Reasoning

Abstract

Mixture-of-experts (MoE) models can share experts across layers or computational steps, pooling training information from different contexts. This can reduce local prediction error, but multi-step systems are ultimately evaluated by their final outputs. We study when these local gains survive composition. In specified linear regression models, we derive exact sharing and merging criteria by balancing variance savings against bias from sharing across roles with different target behavior. Under a common fixed linear propagation map, we characterize expected squared final-output risk, retaining error directions and correlations. We construct cases where sharing lowers average local error yet increases final-output error. Within this setting, we also identify when orthogonal full pooling with isotropic noise preserves every local improvement at the final output. For general routed computations, we derive conditional end-to-end error bounds that account for distribution shifts, errors in surrounding components, and changes in expert selection. Separately, we show that correct predictions and route labels alone do not establish that individual experts implement the attributed operations. Controlled experiments realize a strict local-to-final risk reversal. In low-data symbolic settings, trained Transformers exhibit statistically supported teacher-forced one-step gains without statistically resolved free-running final-output gains. Separate intervention tests distinguish successful multi-layer actions from single-expert attribution and generalization to unseen transitions. Together, these results distinguish when sharing improves local estimation, when those gains survive to the final output, and what evidence is needed to attribute reusable operations to individual experts.

Introduction

A mixture-of-experts (MoE) Transformer uses a router to select among neural modules, called experts, when processing an input representation [34, 21, 8, 5]. Different layers can maintain separate experts, each with its own parameters, or use a common set of expert parameters across layers, which is called expert sharing [37, 14, 2, 17, 32]. Training information from different layers then shapes the same learned function. For multi-step reasoning, evaluating this choice requires understanding both the accuracy of individual computations and their contribution to the final output.

Choosing which computations should share an expert requires comparing the final-output errors of the resulting shared and separate models. A natural statistical rationale for sharing is that combining information from related contexts can reduce estimation variance, at the cost of bias when their required transformations differ. Multi-task learning provides a foundation for understanding this trade-off [38, 39, 6, 15]. However, even when this trade-off favors sharing in average local prediction error, the corresponding final-output comparison remains a separate question. Shared parameters can change the dependence between errors at different calls, while subsequent computations determine how those errors combine. A criterion based on local error therefore needs an additional connection to the accuracy of the complete computation. The comparison we seek must account jointly for how sharing changes estimation errors and how downstream computation transforms them, identifying when a local preference for shared experts remains justified at the final output. We therefore ask:

When does expert sharing reduce average local prediction error, and under what conditions is this advantage preserved at the final output?

Our analysis shows that the preference can reverse even when both models use the same fixed linear propagation map. Sharing can lower average local prediction error while increasing expected squared final-output error. The reason is that a local improvement can coexist with greater bias in directions amplified downstream, or with a change in how errors across calls reinforce or cancel. We derive exact risk comparisons that retain these effects and characterize the propagation conditions under which every strict local improvement is preserved in a specified sharing model.

Beyond its effect on accuracy, sharing raises a question about what an individual expert computes. Repeated selection of an expert may suggest that it implements an operation common to several contexts. Causal abstraction evaluates such interpretations through interventions on internal computation [9, 10]. We apply this perspective by changing expert choices while holding the pre-action context fixed and comparing the resulting effects with proposed operations.

Contributions

  1. Exact risk comparisons for expert sharing. In specified linear regression models, we derive necessary and sufficient conditions for a common estimate to outperform separate estimates by balancing variance savings against bias from differences between targets. The same analysis gives exact criteria for merging expert groups and evaluates fixed combinations of separate and common estimates (Theorems 1 and 2), with extensions to cross-layer sharing and general estimation geometry.

  1. When local improvements survive to the output. Under a common fixed linear propagation map, exact expected squared-risk comparisons yield a strict local-to-final reversal (Proposition 47). For nontrivial fixed groupwise averaging and centered estimation noise with equal positive variance in every direction, we characterize the propagation maps that preserve every strict local improvement (Section N.4). The condition requires equal positive scaling of squared lengths on the coefficient differences removed by sharing. For general routed computations, we derive conditional error bounds accounting for distribution shifts, changes in expert selection, and errors in surrounding components (§ 5).

  1. Separating predictive success from expert-level operations. We construct a model with correct predictions and correct expert labels whose output remains unchanged when a different expert is forced (Proposition 5). We then analyze the state information and intervention coverage needed to evaluate operation-level effects, distinguishing individual expert calls from actions spanning several layers (§ 6).

Additionally, we conduct experiments to reproduce the local risk comparisons and realize a local-to-final reversal. In low-data symbolic tasks, trained Transformers show one-step gains when given correct preceding states, while final-output gains remain statistically unresolved when the models use their own predictions (Figure 1). Separate intervention and held-out-transition tests examine expert-level attribution and generalization beyond observed transitions. These results provide a basis for evaluating sharing patterns through both local estimation and final-output performance, while identifying the evidence needed to interpret individual experts as executable operations.

Figure 1. Local and final gains in trained Transformers. Positive differences mean lower error for tied experts. (a) At 64 traces, one-step gains are supported, while the final-output difference is not statistically resolved. (b) All forty tying contrasts at 64, 96, 128, and 192 traces. Bars retain the reported simultaneous intervals of the full 240-endpoint follow-up family (N=32768N = 32768, 480 tails, δ=0.05\delta= 0.05, radius =2.3657= 2.3657)

Related Work

We introduce prior work in expert sharing, statistical pooling, and causal abstraction discussing how local sharing gains change under composition, routing, and intervention-based evaluation.

Expert reuse and routed architectures. Classical MoE studies learned gating over specialized predictors [16, 19]; sparse Transformers scale conditional computation [34, 21, 8, 5]. Recent systems share experts across adjacent layers, global pools, or depth groups [37, 14, 2, 17, 32]. Expert-choice, soft dispatch, and sequential expert calls use different execution mechanisms [44, 31, 42]. Our event-level comparison retains their functional costs; Sections B and I give the architecture representation and cost dictionary.

Statistical sharing and shrinkage. Multi-task learning explains gains from shared structure and negative transfer from misspecified relatedness [38, 39, 6, 15]. MoE theory studies generalization, robustness, expressivity, and task-specialized routing [3, 43, 30, 41, 28]. Our local formulas and known-noise Gaussian plug-in guarantee specialize classical shrinkage [1, 36, 35]. The fixed-rule audit uses unbiased risk estimation, and additive transfer uses the standard Wasserstein coupling argument [33].

Reasoning and causal abstraction. Chain-of-thought theory studies serial computation through intermediate states [27, 23]; latent-module work formalizes reusable components in reasoning traces [20]. Causal abstraction distinguishes behavioral agreement from intervention agreement [9, 10]. Applied to route actions, this requires sufficient pre-action states and distinguishable forced effects. We also analyze randomized action audits via inverse-propensity estimation [12], with a matching lower-bound dependence on per-action coverage.

Routed Roles and Event-Lifted Risk

We distinguish call environments from expert blocks and measure their complete-event effects. This makes tied and separate libraries comparable.

Roles, experts, and the reference. Consider one prediction request: the current state is 3 and the next operation is multiplication by two, so the target next state is 6. In the illustrative two-layer Transformer of Appendix § 14, both layers process this same request, rather than applying the task operation twice. At the chosen prediction position, operation i=2i = 2 is labelled by call type k=i+1=3k = i + 1 = 3, giving two roles (1,3)(1, 3) and (2,3)(2, 3).

A role aa in a finite set J\mathcal{J} specifies a controlled call environment and its target behavior. Different input numbers can instantiate the same role. The two layers define distinct roles here because their input sources and surrounding computation differ. An expert library Eθ={Eθ,e:Xe→Ye}e=1K\mathcal{E}_\theta= \{E_{\theta,e} : \mathcal{X}_e \to\mathcal{Y}_e\}_{e=1}^{K} contains parameterized functions indexed by ee. Separate fitting uses two independent parameter blocks for these roles, and reuse assigns the same block. The reused block may receive different inputs across roles. Its outputs and the roles’ target updates may also differ.

In general, a fixed assignment αm:J→[Km]\alpha_m : \mathcal{J} \to[K_m] identifies the expert serving each role in comparator m∈{reuse,sep}m \in\{\mathrm{reuse},\mathrm{sep}\}, with [Km]={1,…,Km}[K_m] = \{1,\ldots,K_m\}. Reuse groups Je={a:αreuse(a)=e}\mathcal{J}_e = \{a : \alpha_{\mathrm{reuse}}(a) = e\} share functions through compatible interfaces or fixed adapters. The separate model is a function per role. In this role-assignment comparison, expert sharing is the common-function use represented by the reuse comparator. The assignment specifies who should serve a role. The router decides who actually does, without redefining the role.

The reference is the declared target computation, not the fitted separate model. In the example, knowing the target next state is 6 does not determine a layer’s target residual. A local target fa⋆f_a^\star is specified separately to map hidden vectors to residuals, whereas the symbolic task operator FiF_i maps task states to task states.

Complete events and aligned instructions. An event is a complete computational block, not necessarily a token or reasoning step. For j=0,…,H−1j = 0,\ldots,H-1, states Xj,X^jm∈SjX_j,\widehat{X}_j^m \in\mathcal{S}_j retain everything needed for execution, including prefixes, hidden-state arrays, and caches. A canonical program Γj∈Gj⋆\Gamma_j \in\mathcal{G}_j^\star specifies roles, call order, input selection, and combination. A fixed implementation map ιjm:Gj⋆→Gjm\iota_j^m : \mathcal{G}_j^\star\to\mathcal{G}_j^m translates it into model-specific instructions. Conditional on fitted parameters and external randomness,

Xj+1=Φj⋆(Xj;Γj),X^j+1m=Φjm(X^jm;Γ^jm),Γ^jm=Rjm(Hjm).(1)X_{j+1}=\Phi^\star_j(X_j;\Gamma_j), \qquad\widehat{X}^{m}_{j+1}=\Phi^m_j(\widehat{X}^{m}_j;\widehat{\Gamma}^{m}_j), \qquad\widehat{\Gamma}^{m}_j=\mathcal{R}^{m}_j(\mathcal{H}^{m}_j). \tag*{(1)}

The maps Φj⋆\Phi^\star_j and Φjm\Phi^m_j return complete post-event states in Sj+1\mathcal{S}_{j+1}, including shared paths and aggregation. The router uses available history Hjm\mathcal{H}^{m}_j to choose the actual plan Γ^jm\widehat{\Gamma}^{m}_j, which may differ from the aligned plan ιjm(Γj)\iota^m_j(\Gamma_j). Decisions use only information available when made. Adaptive internal decisions must be separate events for fixed-sequence replacement, or the whole policy must be one event lift. Section B.1 gives the complete history and randomness interface.

Role risk in its call environment. Role aa has probe-record space Za\mathcal{Z}_a, distribution PaP_a, and an event lift

Ta[f]:Za⟶S,(2)\mathcal{T}_a[f]:\mathcal{Z}_a\longrightarrow\mathcal{S}, \tag*{(2)}

which inserts expert function ff into a specified environment and returns the complete post-event state. For example, Ta[f](h,u,c)=h+cf(u)\mathcal{T}_a[f](h,u,c)=h+cf(u) is a residual update with state hh, expert input uu, and scalar gain cc. Dimensions agree after any fixed adapter. Let Ta⋆\mathcal{T}^{\star}_a be the target-call counterpart. For a state metric dd,

ℓa(f;z)=d(Ta[f](z),Ta⋆(z)),Ra(f)=EZ∼Paℓa(f;Z).(3)\ell_a(f;z)=d(\mathcal{T}_a[f](z),\mathcal{T}^{\star}_a(z)), \qquad R_a(f)=\mathbb{E}_{Z\sim P_a}\ell_a(f;Z). \tag*{(3)}

Both arguments are complete states. The worked example inserts one residual into the full array. The bounded-loss theorem additionally requires ℓa∈[0,1]\ell_a\in[0,1] and probe laws, lifts, targets, classes, and permitted expert inputs fixed independently of fitting data, or a valid joint uniform bound.

The Local Sharing Frontier

This section mainly discuss that, with call environments fixed, when should roles share a function. The linear model prices pooling and merges by removable variance versus sharing bias. Beyond that, we compare certificates rather than exact risks.

An exact linear-operator frontier

In a specified linear estimation problem, consider M≥1M\ge1 equally weighted roles with target operators Aa∈Rp×dA_a\in\mathbb{R}^{p\times d} and responses Ya=XaAaT+EaY_a=X_aA_a^{\mathsf{T}}+E_a. Each fixed design Xa∈Rn×dX_a\in\mathbb{R}^{n\times d} stacks the training inputs and satisfies n≥dn\ge d, XaTXa=nIdX_a^{\mathsf{T}}X_a=nI_d, which are not the event states XjX_j. Designs are held constant as fitting noise varies. Orthogonal columns are a model assumption. All noise rows are mutually independent across observations and roles, centered, and have covariance σ2Ip\sigma^2I_p. Gaussianity is not required. Define

A‾=1M∑a=1MAa,B=1M∑a=1M∥Aa−A‾∥F2.(4)\overline{A}=\frac{1}{M}\sum_{a=1}^{M}A_a,\qquad B=\frac{1}{M}\sum_{a=1}^{M}\lVert A_a-\overline{A}\rVert_F^2. \tag*{(4)}

Thus BB measures dispersion among the true role targets. Ordinary least squares (OLS) gives A^a=YaTXa/n\widehat{A}_a=Y_a^{\mathsf{T}}X_a/n and A^pool=M−1∑aA^a\widehat{A}_{\mathrm{pool}}=M^{-1}\sum_a\widehat{A}_a. For a fixed coefficient λ∈[0,1]\lambda\in[0,1], partial pooling uses

A^a(λ)=(1−λ)A^a+λA^pool.(5)\widehat{A}^{(\lambda)}_a=(1-\lambda)\widehat{A}_a+\lambda\widehat{A}_{\mathrm{pool}}. \tag*{(5)}

Separate fitting and full sharing with λ=0\lambda=0 and 11, and intermediate matrices generally remain role-specific. An independent input has second moment E[xxT]=Id\mathbb{E}[xx^{\mathsf{T}}]=I_d. The excess prediction risk is

Rλ=1M∑a=1MEE,x∥(A^a(λ)−Aa)x∥22.(6)\mathcal{R}_{\lambda}=\frac{1}{M}\sum_{a=1}^{M}\mathbb{E}_{\mathcal{E},x}\left\lVert\left(\widehat{A}^{(\lambda)}_a-A_a\right)x\right\rVert_2^2. \tag*{(6)}

It averages fitting noise and test inputs relative to noiseless targets, excluding fresh label noise. Only under this isotropic test geometry does conditional prediction error equal squared Frobenius parameter error. This specified squared loss is distinct from the bounded metric loss in Equation (3).

Theorem 1 (Exact sharing and partial-pooling risks). Let D=pdD=pd, V=Dσ2/nV=D\sigma^2/n, and C=V(1−1/M)C=V(1-1/M). For the excess risk in Equation (6),

Rsep=V,(7)\mathcal{R}_{\mathrm{sep}}=V, \tag*{(7)}
Rshare=B+VM,(8)\mathcal{R}_{\mathrm{share}}=B+\frac{V}{M}, \tag*{(8)}
Rλ=V−2Cλ+(B+C)λ2.(9)\mathcal{R}_{\lambda}=V-2C\lambda+(B+C)\lambda^2. \tag*{(9)}

Consequently, full reuse has smaller expected excess prediction risk than separate estimation if and only if

B<C=pdσ2n(1−1M).(10)B<C=\frac{pd\sigma^2}{n}\left(1-\frac{1}{M}\right). \tag*{(10)}

If B+C>0B+C>0, the optimal fixed coefficient is λ⋆=C/(B+C)\lambda^\star=C/(B+C). The minimum risk, endpoint comparisons, and degenerate cases are given in Section D.1.

References

  1. [1]Alvin J Baranchik. A family of minimax estimators of the mean of a multivariate normal distribution. The Annals of Mathematical Statistics, pp. 642–645, 1970.DOI
  2. [2]Yilong Chen, Naibin Gu, Junyuan Shang, Zhenyu Zhang, Yuchen Feng, Jiawei Sheng, Tingwen Liu, Shuhuowan Wang, Yu Sun, Hua Wu, et al. Mixture of universal experts: Scaling virtual width via depth-width transformation. arXiv preprint arXiv:2603.04971, 2026.arxiv.org/abs/2603.04971
  3. [3]Mohammed Nowaz Rabbani Chowdhury, Shuai Zhang, Meng Wang, Sijia Liu, and Pin-Yu Chen. Patch-level routing in mixture-of-experts is provably sample-efficient for convolutional neural networks. In International Conference on Machine Learning, pp. 6074–6114. PMLR, 2023.arxiv.org/abs/2306.04073
  4. [4]Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wanding Zeng, Xingkai Yu, Yu Wu, et al. Deepspeekmo: Towards ultimate expert specialization in mixture-of-experts language models. In Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers), pp. 1280–1297, 2024.
  5. [5]Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. In International conference on machine learning, pp. 5547–5569. PMLR, 2022.arxiv.org/abs/2112.06905
  6. [6]Yaqi Duan and Kaizheng Wang. Adaptive and robust multi-task learning. The Annals of Statistics, 51 (5):2015–2039, 2023.arxiv.org/abs/2202.05250
  7. [7]Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychometrika, 1(3):211–218, 1936.DOI
  8. [8]William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022.arxiv.org/abs/2101.03961
  9. [9]Atticus Geiger, Zhengxuan Wu, Hanson Lu, Josh Rozner, Elisa Kreiss, Thomas Icard, Noah Goodman, and Christopher Potts. Inducing causal structure for interpretable neural networks. In International Conference on Machine Learning, pp. 7324–7338. PMLR, 2022.arxiv.org/abs/2112.00826
  10. [10]Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah Goodman. Finding alignments between interpretable causal variables and distributed neural representations. In Causal Learning and Reasoning, pp. 160–187. PMLR, 2024.arxiv.org/abs/2303.02536
  11. [11]Bruce E Hansen. Efficient shrinkage in parametric models. Journal of Econometrics, 190(1):115–132, 2016.DOI
  12. [12]Daniel G Horvitz and Donovan J Thompson. A generalization of sampling without replacement from a finite universe. Journal of the American statistical Association, 47(260):663–685, 1952.DOI
  13. [13]Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.arxiv.org/abs/2106.09685
  14. [14]Minbin Huang, Han Shi, Chuanyang Zheng, Yimeng Wu, Guoxuan Chen, Xintong Yu, Yichun Yin, and Hong Cheng. Unipool: A globally shared expert pool for mixture-of-experts. arXiv preprint arXiv:2605.06665, 2026.
  15. [15]Xinmeng Huang, Kan Xu, Donghwan Lee, Hamed Hassani, Hamsa Bastani, and Edgar Dobriban. Optimal multitask linear regression and contextual bandits under sparse heterogeneity. Journal of the American Statistical Association, 120(552):2262–2275, 2025.
  16. [16]Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79–87, 1991.DOI
  17. [17]Martin Jaggi. Tying the loop-tied expert layers in mixture-of-experts language models. arXiv preprint arXiv:2606.16825, 2026.arxiv.org/abs/2606.16825
  18. [18]Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024.arxiv.org/abs/2401.04088
  19. [19]Michael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181–214, 1994.
  20. [20]Lingjing Kong, Xin Liu, Guangyi Chen, Martin Q Ma, Xiangchen Song, Yuekai Sun, Mikhail Yurochkin, Taylor W Killian, Ruslan Salakhutdinov, Kun Zhang, et al. From reasoning traces to reusable modules: Understanding compositional generalization in language model reasoning. arXiv preprint arXiv:2606.18089, 2026.arxiv.org/abs/2606.18089
  21. [21]Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020.arxiv.org/abs/2006.16668
  22. [22]Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. In International Conference on Machine Learning, pp. 6265–6274. PMLR, 2021.arxiv.org/abs/2103.16716
  23. [23]Zhiyuan Li, Hong Liu, Denny Zhou, and Tengyu Ma. Chain of thought empowers transformers to solve inherently serial problems. In International Conference on Learning Representations, volume 2024, pp. 11911–11943, 2024.arxiv.org/abs/2402.12875
  24. [24]BG Manjunath and Stefan Wilhelm. Moments calculation for the doubly truncated multivariate normal density. Journal of Behavioral Data Science, 1(1):17–33, 2021.arxiv.org/abs/1206.5387
  25. [25]Yuzo Maruyama and Akimichi Takemura. Non-minimacity of debiased shrinkage estimators. Japanese Journal of Statistics and Data Science, 7(1):361–375, 2024.
  26. [26]Takeru Matsuda and William E Strawderman. Estimation under matrix quadratic loss and matrix superharmonicity. Biometrika, 109(2):503–519, 2022.arxiv.org/abs/2005.12479
  27. [27]William Merrill and Ashish Sabharwal. The expressive power of transformers with chain of thought. In International Conference on Learning Representations, volume 2024, pp. 7690–7706, 2024.arxiv.org/abs/2310.07923
  28. [28]Vinoth Nandakumar, Yongli Xiang, Yunzhi Yao, Peike Li, and Tongliang Liu. A theoretical model for task routing in mixture-of-expert transformers. arXiv preprint arXiv:2606.14398, 2026.arxiv.org/abs/2606.14398
  29. [29]Gabriel Peyré and Marco Cuturi. Computational optimal transport with applications to data sciences. Foundations and trends® in machine learning, 11(5-6):355–607, 2019.
  30. [30]Joan Puigcerver, Rodolphe Jenatton, Carlos Riquelme, Pranjal Awasthi, and Srinadh Bhojanapalli. On the adversarial robustness of mixture of experts. Advances in neural information processing systems, 35:9660–9671, 2022.arxiv.org/abs/2210.10253
  31. [31]Joan Puigcerver, Carlos Riquelme Ruiz, Basil Mustafa, and Neil Houlsby. From sparse to soft mixtures of experts. In International Conference on Learning Representations, volume 2024, pp. 28435–28445, 2024.arxiv.org/abs/2308.00951
  32. [32]Eric S Qiu, Utku Umur Acik alin, Justin Lovelace, Christian Belardi, Arjun B Mulchandani, Carla P Gomes, and Kilian Q Weinberger. More: Mixture of reused experts. arXiv preprint arXiv:2609.18176, 2026.arxiv.org/abs/2609.18176
  33. [33]Ievgen Redko, Amaury Habrard, and Marc Sebban. Theoretical analysis of domain adaptation with optimal transport. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 737–753. Springer, 2017.arxiv.org/abs/1610.04420
  34. [34]Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017.arxiv.org/abs/1701.06538
  35. [35]K Pavan Srinath and Ramji Venkataramanan. Cluster-seeking james–stein estimators. IEEE Transactions on Information Theory, 64(2):853–874, 2017.DOI
  36. [36]Charles M Stein. Estimation of the mean of a multivariate normal distribution. The annals of Statistics, pp. 1135–1151, 1981.DOI
  37. [37]Zheyue Tan, Zhiyuan Li, Tao Yuan, Dong Zhou, Weilin Liu, Yueqing Zhuang, Yadong Li, Guowei Niu, Cheng Qin, Zhuyu Yao, et al. Rexmoe: Reusing experts with minimal overhead in mixture-of-experts. arXiv preprint arXiv:2510.17483, 2025.arxiv.org/abs/2510.17483
  38. [38]Nilesh Tripuraneni, Michael Jordan, and Chi Jin. On the theory of transfer learning: The importance of task diversity. Advances in neural information processing systems, 33:7852–7862, 2020.arxiv.org/abs/2006.11650
  39. [39]Nilesh Tripuraneni, Chi Jin, and Michael Jordan. Provable meta-learning of linear representations. In International conference on machine learning, pp. 10434–10443. Pmlr, 2021.arxiv.org/abs/2002.11684
  40. [40]Alexandre B Tsybakov and Vladimir Zaiats. Introduction to nonparametric estimation, volume 11. Springer, 2009.
  41. [41]Mingze Wang et al. On the expressive power of mixture-of-experts for structured complex tasks. Advances in Neural Information Processing Systems, 38:68579–68606, 2026.arxiv.org/abs/2505.24205
  42. [42]Zihan Wang, Rui Pan, Jiarui Yao, Robert Csordas, Linjie Li, Lu Yin, Jiajun Wu, Tong Zhang, Manling Li, and Shiwei Liu. Chain-of-experts: unlocking the communication power of mixture-of-experts models. arXiv preprint arXiv:2506.18945, 2025.arxiv.org/abs/2506.18945
  43. [43]Jinze Zhao, Peihao Wang, and Zhangyang Wang. Generalization error analysis for sparse mixture-of-experts: A preliminary study. arXiv preprint arXiv:2403.17404, 2024.arxiv.org/abs/2403.17404
  44. [44]Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al. Mixture-of-experts with expert choice routing. Advances in Neural Information Processing Systems, 35:7103–7114, 2022.arxiv.org/abs/2202.09368

Paper details

vlnc:W_zap65cgk
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

generative AI tools

View statement
Looking for related papers…
Contents