WorksComputer Science
Multi-View Latent MoE: Letting Every Router Shape Expert Computation
Abstract
Sparse mixture-of-experts (MoE) models expand parameter capacity by activating a small subset of experts per token. Expert selection directly affects model capabilities under a fixed active-expert budget. Dense evaluation of all experts reveals weak agreement between single-router rankings and expert-utility evaluation rankings, especially in middle and deep layers. Richer signals for selection and computation incur additional cost: selection based on internal expert activations requires preliminary computation, while processing multiple full representations separately through the same expert repeats its computation. We propose Multi-View Latent MoE to improve expert selection with multiple routers and reuse their preferences to modulate expert computation under a fixed expert budget. Its Multi-Router submodule aggregates sigmoid scores from independently initialized routers. These routers develop highly differentiated expert preferences during training, allowing experts strongly favored by one router or jointly supported by several routers to be selected. Multi-View further uses each selected expert's relative scores across routers to mix learned latent views into its input, reusing routing preferences for both selection and computation. Each selected expert executes one forward pass, preserving the active-expert count, expert FLOPs, and all-to-all communication volume. On a 24-layer 10B-A1B model trained on 708B tokens, Multi-View Latent MoE improves the nine-task downstream mean by 1.18 percentage points over the baseline. Multi-Router alone yields a 0.6-percentage-point gain and improves the rank correlation between router scores and evaluated expert utility, with relative increases of 8.1% globally and approximately 58% in the layer with the lowest single-router alignment. Controlled mixing interventions show the model uses input differences that depend on the expert and token.
Figure 1. Router-conditioned expert inputs in Multi-View. Equal total scores can express different preferences across routing heads and yield different mixed inputs. In this illustrative example, experts and have score pairs and , both summing to . Normalizing across routing heads for each expert gives view proportions and , respectively. Routing heads and view projections read the same hidden state; each selected expert runs once. The balancing bias is set to zero. Shared experts, output gating, and up-projection are omitted.
Introduction
Figure 2. Agreement between single-router scores and empirical expert-utility evaluation on the 24-layer 10B-A1B backbone. (a) Spearman correlation across training. The logarithmic training-step axis expands the early stage and compresses later intervals to show early-training changes more clearly. (b) Layer-wise means over the 201 logged measurements used in the router comparison; shaded bands show standard deviation. Scores exclude the load-balancing bias. Expert utility is evaluated using each expert’s output norm, normalized by its mean over sampled tokens (App. § 7.5).
Sparse mixture-of-experts (MoE) models expand parameter capacity by activating a small subset of experts for each token [17, 18]. This conditional computation enables larger language models under a limited active-computation budget. Routing scores determine which experts process each token and how their outputs are combined. The resulting capability depends on both expert selection and the computation performed by the selected experts.
Traditional MoE uses a single routing head followed by hard top- selection. Dense evaluation of all experts reveals limited agreement between the router’s ranking and the ranking obtained from empirical expert-utility evaluation, especially in middle and deep layers (Figure 2). Unselected experts receive no direct task-loss gradient from that token, so poor selections may become self-reinforcing. This motivates assessing experts from multiple scoring perspectives rather than relying on one router.
Standard MoE uses routing scores to select experts and weight their outputs, but does not reuse them to construct expert inputs. Load-balancing methods regulate expert utilization [22], but do not themselves provide multiple assessments of each expert. Activation-based routing such as AoE obtains expert-specific signals through partial evaluations of all experts, using low-rank factorizations to control the cost [30]. Evaluating multiple input representations separately would also add expert computation. The challenge is therefore to obtain multiple assessments of experts and use routing preferences to modulate their computation while preserving the active-expert count, expert FLOPs, and communication volume.
We propose Multi-View Latent MoE, a framework that reuses router preferences to improve expert selection and modulate expert computation. Its Multi-Router submodule combines multiple assessments of a shared expert pool: independently initialized sigmoid routing heads produce scores that are summed before one top- selection. These routers develop highly differentiated preferences during training. Their aggregation allows experts strongly favored by one router or jointly supported by multiple routers to be selected. This submodule retains a shared expert input while enriching the information used for selection.
Multi-View further uses these differentiated preferences to modulate each selected expert’s computation. The summed score alone does not retain the routers’ relative preferences: score vectors and both sum to but express opposite preferences (Figure 1). Each router is paired with a learned latent projection, and each selected expert’s raw sigmoid scores are normalized across routers to blend the views into its input before evaluation. The projections are shared across experts, while the mixing proportions depend on both the token and the expert. Mixing takes place before dispatch, and each selected expert performs one forward computation, preserving the active-expert count, expert FLOPs, and all-to-all communication volume.
On a 24-layer 10B-A1B model trained for 708B tokens, the full Multi-View Latent MoE improves the nine-task downstream mean by percentage points over the baseline, reaching ; its Multi-Router submodule alone provides a -point gain. Routing diagnostics show that this submodule improves global score–utility rank correlation by a relative , with a relative gain of approximately in the layer with the lowest single-router alignment (§ 4.4). Measurements show distinct expert inputs for the same token. At the final MoE layer, permuting view-mixing proportions across experts while fixing expert selection and output gates raises held-out NLL by nats/token; permutation across tokens for the same expert also increases NLL (§ 4.3). These results show that the trained model uses the correspondence between mixing proportions, experts, and tokens.
Contributions.
Multi-Router for improved expert ranking. Dense evaluation of all experts reveals weak agreement between single-router score rankings and rankings from expert-utility evaluations, especially in middle and deep layers. Motivated by this observation, the Multi-Router submodule aggregates sigmoid scores from independently initialized routers. These routers develop highly differentiated expert preferences during training, and their aggregated scores achieve closer agreement with the evaluated utility rankings.
Multi-View Latent MoE at a fixed expert budget. The framework couples its Multi-Router selection submodule with router-conditioned latent-view mixing. Mixing before dispatch retains one forward computation per selected expert, the active-expert count, expert FLOPs, and all-to-all communication volume.
Quantitative gains. On a 24-layer 10B-A1B model trained for 708B tokens, full Multi-View Latent MoE improves the nine-task downstream mean by percentage points over the baseline; Multi-Router alone provides a -point gain. Multi-Router improves global score–utility rank correlation by a relative , with a relative gain of approximately in the layer with the lowest single-router alignment.
Related Work
Sparse MoE and expert specialization.
Sparse MoE layers route each token to a small subset of experts and combine their outputs [17]. Token routing with a linear routing head and top- selection remains common in sparse MoE models [1, 26, 25]. Switch Transformer uses top-1 routing [18]. Recent large language models also adopt MoE architectures [3, 4]. Approaches such as fine-grained experts with an always-active shared expert increase the flexibility of expert specialization [24]. In these architectures, however, a discrete routing decision determines the active experts, which process a shared token representation.
Router design.
Router design addresses routing quality, stability, and load balance through methods including router -loss [19], expert-choice routing [20], auxiliary-loss-free balancing [22], mitigation of representation collapse [23], and attention over experts [5]. Expert assignments can fluctuate during training, motivating methods for more stable routing [21, 7]. Router Upcycling initializes query projections and expert keys from preceding attention layers for dense-to-MoE upcycling, then sums query–key matching scores across routers before softmax selection [27]. MoR uses a learnable main router to weight sub-router outputs in LoRA expert mixtures for parameter-efficient fine-tuning [29]. Our Multi-Router submodule sums scores from separately parameterized sigmoid routing heads during pretraining. Multi-View further uses each expert’s relative scores across routing heads to construct its input.
Multi-representation and latent-space MoE.
A separate line of work changes the representations processed by experts. LatentMoE computes router scores from the full hidden state and evaluates experts in a low-dimensional latent space [12]. Stable LatentMoE modifies this design for training stability while retaining a single down-projection [2]. We build on this shared-projection architecture. MoLAE factorizes expert weights into shared latent mappings, optionally within expert groups, and expert-specific transformations [15]. AoE removes the separate router: experts are ranked by the norms of their low-dimensional internal activations, and only selected experts continue computation; low-rank factorization reduces the cost of this initial evaluation [30]. MH-MoE splits projected tokens into sub-tokens that share a router and expert pool [10, 11]. Multi-Head LatentMoE instead assigns each sub-token an independent MoE module with its own router and experts [13]; SliceMoE also explores token slicing [28]. MoH applies shared and routed components to attention heads rather than to the FFN [14].
Multi-View Latent MoE uses projections of the full hidden state as a shared basis. For each selected expert, raw scores are normalized across routers to determine a token- and expert-dependent mixture before expert computation. The number of projections depends on , rather than on the number of experts, and expert parameters are shared across views. This changes the representation each expert receives. Prior analyses relate hidden-state geometry to expert-usage patterns [9]. Multi-Router retains the shared latent input of LatentMoE while aggregating multiple routing heads; Multi-View further reuses their preferences to modulate expert inputs.
Method
Preliminaries and framework overview
We study autoregressive language modeling and modify the routing and expert-input interface of a latent-space MoE backbone [12]. The router reads the hidden state and selects of routed experts. A down-projection supplies the shared input . Each expert maps to . With selected set and output gates , the latent expert aggregation is
The backbone up-projects this aggregation to dimension and combines it with the always-active shared-expert output.
Multi-View retains the shared expert pool and , while reusing the preferences of individual routing heads for both selection and input construction. Its Multi-Router submodule aggregates scores; for each selected expert, scores normalized across routers mix latent views before one expert evaluation. Figure 3 shows the computation.
Figure 3. Comparison of latent-space MoE architectures. Top: (a) LatentMoE uses a router and a shared latent input; (b) Multi-Router sums scores from separately parameterized sigmoid routing heads for top- selection while retaining the shared input; (c) Multi-View adds distinct down-projections and expert-specific view mixing. Bottom: expansion of the boxed routed path in (c), with matching colors and numbered markers. Balancing bias is added to summed scores only for selection. Output gates follow Eq. (3). For each selected expert, raw sigmoid scores are normalized across routers to form view proportions , whose sum is one. Each selected expert processes its mixed input once. The aggregated representation is up-projected and combined with the shared-expert output shown above. views and selected experts are illustrative; is the number of active experts out of . Bars depict example routing scores. Normalization modules and residual paths are omitted for clarity.
Multi-Router: score aggregation before selection
Multi-Router uses routing heads , , to score the shared expert pool from the full hidden state . Each routing head’s output passes through a sigmoid, and the resulting scores are summed before top- selection:
Here is the backbone’s expert-balancing bias, added only for selection. Output gates normalize the uncorrected score sums over the selected experts and apply the backbone’s routed-output scale:
Here is the backbone’s learnable scalar for expert ; it scales the expert output, not the view mixture.
Routing heads are initialized with distinct seeds to break symmetry, without a diversity penalty. We use by default. In Multi-View (§ 3.3), each router is paired with one view, so and the two symbols refer to the same count.
Summation after the sigmoid preserves the routing heads as distinct nonlinear scoring functions. Summing logits before the sigmoid instead gives a single linear scorer, since . Post-sigmoid summation lets a strong score from one routing head or support across routing heads affect the shared ranking and supplies the relative contributions used for input mixing. Alternatives and their properties appear in Apps. § 6.3–§ 6.5, with training comparisons in § 4.5.
Multi-View: router-conditioned expert inputs
Multi-View pairs routing heads with latent views in a one-to-one correspondence, so . Selection and output gating follow the summed-score rules of Multi-Router, while each selected expert’s scores normalized across routers construct its input . Views are fused before expert evaluation, and each selected expert runs once. This parameterization avoids evaluating experts separately for each view; for nonlinear , it is generally not equivalent to mixing outputs after separate evaluations:
Views.
We form latent views , with each matching the shape of the backbone down-projection. View uses the backbone’s projection ; the remaining views use independently initialized perturbations, as described below. Each view projects the full hidden state rather than a subset of its channels. Expert architectures and input dimensions are unchanged.
Routing, mixing, and gating.
All routing heads read . Routing head produces for every expert , and these scores determine three operations. (i) Selection: sum across routers, , add the balancing bias, and take one top-, , as in Multi-Router. (ii) View mixing: for each selected expert , normalize its raw sigmoid scores across the routing heads:
These proportions form the convex mixture . Neither the balancing bias nor the routed-output scale enters this mixture. (iii) Output gating: use from Eq. (3) to combine the selected experts’ outputs. Thus, output gating normalizes summed scores over experts, whereas input mixing normalizes each expert’s scores across routers. The resulting expert aggregation is
Equivalently, the input uses a token- and expert-dependent effective projection,
The projection basis is shared across experts; only its mixing coefficients depend on and . The implementation mixes the projected vectors, rather than materializing a separate matrix for every token–expert pair.
Normalizing each expert’s scores across routers to derive the mixing weights requires no additional mixing network and makes view ’s contribution to expert ’s input depend on the corresponding routing preference. Algorithm alg:mv gives the forward pass. Learned mixing weights and gradient coupling are evaluated in the corresponding ablation configurations in § 4.5.
\caption{Multi-View{} routed-expert branch forward pass}
\label{alg:mv}
\begin{algorithmic}[1]
\Require hidden state $h$; projections $\{P_m\}_{m=1}^{K}$; routing heads $\{w^{(m)}\}_{m=1}^{R}$; experts $\{E_i\}_{i=1}^{N}$; active count $k$; balancing bias $b$; routed-output scales $\{\alpha_i\}_{i=1}^{N}$
\State $v^{(m)} \gets P_m h$ \textbf{for } $m=1,\dots,K$ \Comment{$K$ latent views}
\State $s_{m,i} \gets \sigma(w_i^{(m)\top} h)$ \Comment{per-router scores on full $h$}
\State $\bar{s}_i \gets \sum_{m} s_{m,i}$ \Comment{summed score per expert}
\State $\mathcal{S} \gets \mathrm{TopK}_k(\bar{s}+b)$ \Comment{bias only for selection}
\For{$i \in \mathcal{S}$}
\State $a_{m,i} \gets s_{m,i}/\sum_{n=1}^{R}s_{n,i}$ \Comment{normalize across routers}
\State $\hat{v}_i \gets \sum_{m} a_{m,i}\, v^{(m)}$ \Comment{per-expert view blend}
\State $g_i \gets \alpha_i\bar{s}_i / \sum_{j\in\mathcal{S}} \bar{s}_j$ \Comment{output gate, Eq.~\ref{eq:gate}}
\EndFor
\State \Return $y = \sum_{i\in\mathcal{S}} g_i\, E_i(\hat{v}_i)$
\end{algorithmic}
Expert-dependent mixing weights.
The index in allows input modulation to vary across experts. If all selected experts for the same token share , they receive the same input . This remains token-conditioned modulation, but no longer distinguishes different experts for that token. If the weights are further fixed to constants, such as , the mixture can be absorbed into a fixed projection. Multi-View derives from the raw scores for expert , normalized across the routing heads, making modulation depend on both the token and the expert (App. § 6.10).
Initialization.
To introduce the additional projections gradually, we retain the backbone down-projection for view and parameterize the others as ramped perturbations, , with increasing from to early in training. At , all view projections equal the backbone projection; after the ramp, the view projections are unconstrained. The expert inputs are weighted combinations of these views. This procedure can also initialize the views from an existing checkpoint (App. § 6.9).
Shared expert parameters.
All views use one expert pool, so expert parameters are not duplicated or partitioned across views. Each selected expert processes one mixed input, and gradients propagate through every view with nonzero mixing weight. Multi-Router uses the shared input for all selected experts; Multi-View adds input modulation through the router-derived coefficients (App. § 6.10).
Experiments
Setup
We compare the latent-space MoE baseline, Multi-Router, and score-derived Multi-View on a 24-layer 10B-A1B backbone with top-8 routing and one always-active shared expert [12]. The three configurations share the training data, token budget, backbone, and optimization settings (App. § 7.2). Downstream evaluation uses the 708B-token checkpoints on nine tasks (protocol in App. § 7.4). We use for the number of routing heads and for the number of views. The main method Multi-View() pairs them one to one, so ; Multi-Router() uses routing heads with a shared expert input.
Training ablations examine router aggregation, learned input-mixing weights, and view-fed scoring. Their losses are averaged over a token window around a common ablation checkpoint. A pair of near-matched runs differs after smoothing by at most , with a mean absolute difference of . Architecture, optimization, and comparison settings are detailed in App. § 7.3.
Downstream performance
| Exams | Knowledge | Science | Reasoning | |||||||||
| Method | CMMLU | MMLU | Redux | Pro | Avg. | HotpotQA | SimpleQA | PreciseWiki | Avg. | GPQA | BBH | Overall |
| Baseline (LatentMoE) | 53.71 | 57.48 | 59.03 | 27.28 | 49.38 | 22.00 | 3.35 | 8.64 | 11.33 | 30.00 | 36.10 | 33.07 |
| orange!12 Multi-Router () | 53.77 | 57.50 | 58.90 | 27.71 | 49.47 | 22.13 | 3.85 | 9.16 | 11.71 | 32.86 | 37.31 | 33.69 |
| blue!10 Multi-View () | 55.20 | 58.67 | 60.93 | 27.12 | 50.48 | 22.54 | 3.41 | 9.02 | 11.66 | 35.60 | 35.76 | 34.25 |
| (MVBaseline) |
Table 1. Downstream scores (%) at 708B training tokens. Baseline: latent-space MoE; MR: Multi-Router (); MV: score-derived Multi-View (; Eq. (5)). Tasks are grouped into exams, knowledge, science, and reasoning. Avg. is the unweighted within-group mean; Overall is the unweighted mean across all nine tasks. Bold marks the highest displayed score in each column. The final row reports MV minus Baseline in percentage points. Prompt and scoring protocols appear in App. § 7.4.
Table 1 compares all three configurations at the same training budget. The nine-task means are for the baseline, for Multi-Router, and for Multi-View. Multi-View improves the mean by percentage points over the baseline and by over Multi-Router. Gains over Multi-Router reach points on GPQA and on MMLU-Redux; CMMLU and MMLU improve by and points, respectively. These results indicate an aggregate benefit from extending selection with multiple routing heads to router-conditioned expert computation.
Training convergence.
Figure 4 compares the three configurations through the 708B-token checkpoint used for downstream evaluation. Over the final 1,000 steps, the mean losses are for the baseline, for Multi-Router, and for Multi-View. Multi-Router reduces loss by relative to the baseline, and Multi-View increases this reduction to . Multi-View’s lower loss in late training is consistent with its downstream gains.
Figure 4. Training loss of the baseline, Multi-Router, and Multi-View on 10B-A1B through 708B tokens. (a) Loss difference from the baseline; the inset shows the full loss trajectories. (b) Magnification of the blue-bordered window in (a), covering 50–230B tokens. (c) Magnification of the purple-bordered window in the full-loss inset, showing absolute loss over 620–708B tokens. All curves use the same causal exponential moving average (EMA), with a 2,000-step span (), initialized at the first observation. The final losses reported in the text average the last 1,000 unsmoothed steps.
Router preferences shape expert computation
Figure 5. Expert-specific inputs and controlled mixing interventions in the score-derived Multi-View checkpoint at 708B tokens. (a) Differences among the eight experts selected for the same token: the mean absolute pairwise difference in view-1 proportions (top) and the mean pairwise input distance normalized by the mean input norm (bottom). Zero denotes identical proportions or inputs, respectively. Curves show the median across tokens at each layer; bands show the 25th–75th percentiles. (b) Final-layer interventions on 256 sequences, changing view proportions while holding and output gates fixed to numerical precision. Whiskers are 95% sequence-bootstrap CIs; crosses mark permutation-seed means. (c) Single-layer interventions on 80 sequences. Bands are pointwise 95% CIs; downstream routing is recomputed. NLL is intervention loss minus the original loss.
We probe the score-derived Multi-View checkpoint used for downstream evaluation at 708B tokens. The analysis covers all 24 MoE layers on 256 held-out sequences of 4096 tokens. It examines expert-specific inputs and whether the trained model uses the correspondence between mixing proportions, experts, and tokens.
From preferences to distinct expert inputs.
The two routing heads have markedly different rankings: across layers, their top-8 Jaccard index averages and their top-1 agreement averages . For each token, the probe exports the actual view-1 mixing proportion and mixed input of each of its eight selected experts. We compare all 28 pairs of these experts: is their mean absolute proportion difference, and is their mean input distance divided by their mean input norm (App. § 7.6). Both are zero when all eight experts receive the same proportions or inputs, respectively. Figure 5(a) summarizes these within-token differences across tokens at each layer. The layer-wise medians are – for and – for . The selected experts therefore receive distinct inputs according to their view preferences.
The trained model uses the correspondence.
At the final MoE layer, we hold the selected experts and their output gates fixed to numerical precision and intervene only on the view-mixing proportions. For a replacement proportion , the two dispatcher channels are rebuilt as , preserving their sum . Permuting the view proportions among the selected experts of the same token raises NLL by nats/token (95% sequence-bootstrap CI: ). Using uniform mixing proportions raises NLL by ; permuting proportions across tokens for the same expert raises it by . The respective CIs are and . Expert permutation increases sequence-average NLL for all 256 sequences. These interventions show that the trained checkpoint uses the correspondence between mixing proportions, experts, and tokens. Expert permutation produces the largest increase among these three interventions.
Both views contribute across depth.
Applying the view 1-only or view 2-only proportions at the final layer increases NLL by and nats/token, respectively. The trained checkpoint therefore uses both view branches. A separate 80-sequence scan perturbs one layer at a time: expert permutation and uniform mixing increase NLL, with pointwise 95% CIs above zero at all 24 layers. Only the intervened layer’s selections and output gates are fixed; subsequent routing is recomputed, so these are total effects including downstream responses. Additional preference, single-view, and token-group analyses appear in App. § 7.6.
Alignment of router scores and expert outputs
Figure 6. Router-score alignment with normalized expert-output norms. (a) Global Spearman correlation over training; solid curves use a 51-record moving mean and pale curves show raw measurements. (b) Layer-wise means and one-standard-deviation bands over 201 matched logged measurements; metric and aggregation definitions appear in App. § 7.5. (c) Correlation changes across all 24 layers. Multi-Router improves global correlation by a relative , with the largest relative layer-wise gain of at layer 20.
| Method | Spearman |
| Single router | |
| Multi-Router |
Table 2. Global alignment between pre-bias router scores and normalized expert-output norms, averaged over 201 matched logged measurements. Multi-Router improves rank correlation by a relative . Definitions appear in App. § 7.5.
We compare the baseline and Multi-Router using the normalized expert-output norm. For each sampled token, all 256 experts process its latent input; each output’s norm is divided by that expert’s mean norm over the sampled tokens. Spearman correlation measures agreement between the resulting expert ranking and the router scores before balancing bias (App. § 7.5). Related work also examines routing quality through expert-pathway optimization, routing-weight analysis, and expert-importance measurements [34, 35, 8, 36].
Table 2 reports an relative improvement in global rank correlation across 24 MoE layers, from to . Figure 6 shows the training trajectories and layer-wise comparison, with gains in 17 of 24 layers. The largest relative gain occurs at layer 20, where correlation rises from to (); layer 7 shows the largest absolute gain, from to . These measurements show that aggregating differentiated routing heads improves alignment between router scores and normalized expert outputs, complementing the downstream and training-loss gains of Multi-Router.
Training ablations
| Method | loss vs. baseline () |
| Single router (baseline; loss) | |
| Multi-Router (, sigmoid-sum) | |
| Multi-Router () | |
| Fusion position () | |
| Logit-sum | collapses to one router |
| Probability-normalized sum | |
| Fusion operator (post-sigmoid,) | |
| Max | |
| Max to select, sum to gate | |
| Geometric mean | |
| Product | |
| Sum(agreement) | |
| Sum(disagreement) | |
| Observed run-pair difference |
| Variant () | loss vs. baseline () |
| Multi-Router () | |
| Learned per-expert view weights | |
| Decoupled gate, init | |
| Decoupled gate, asym. init | |
| Two independent gates | |
| Coupling between scoring and views | |
| Coupled (gradient flows to views) | |
| Detached (gradient stopped) | |
| Observed run-pair difference |
Fusion position and routing heads.
Multi-Router reduces loss by with two routing heads and with three routing heads (Table tab:main). The additional reduction of is smaller than the observed run-pair difference of . We use two routing heads by default. Fusion position determines whether the routing heads contribute distinct scoring functions: summing logits before sigmoid collapses them into one linear scorer (§ 3.2), whereas applying sigmoid to each routing head before summation preserves their nonlinear contributions. The latter distinguishes equal-mean logits with different distributions across routing heads.
Fusion operators and routing-head contributions.
Different fusion rules determine how the scoring views contribute to expert selection. Maximum fusion retains only the strongest score, whereas products and geometric means emphasize joint support. Summation accumulates bounded contributions from all routing heads, allowing either a strong response from one routing head or support across routing heads to increase the aggregate. All evaluated fusion variants reduce loss by – (Figure 7). The operator gaps lie within the observed run-pair difference. Beyond selection, Multi-View normalizes each selected expert’s raw scores across routing heads to construct input-mixing proportions (§ 3.3), connecting selection with multiple routing heads to router-conditioned expert computation.
Learned input mixing.
Table tab:mv_abl compares three learned per-expert mixing parameterizations: symmetric initialization at , asymmetric initialization, and two independent gates. They reduce loss by – relative to the baseline while retaining the same active-expert budget. The main Multi-View method instead derives mixing weights directly from router scores, reusing the selection signal without an additional mixing network.
Gradient coupling in view-fed scoring.
A separate ablation feeds the views into both the scoring and expert-computation paths. Stopping the gradient from mixing weights to views changes the loss difference from to , an increase of . The forward weight values at fixed parameters are preserved, isolating the benefit of this gradient path within the view-fed configuration.
Parameter and computational efficiency
Parameter overhead.
For the 24-layer 10B-A1B backbone with two routing heads, the Multi-Router submodule adds approximately parameters. Multi-View adds one latent-view projection per layer, contributing a further , for a total increase of approximately .
Expert computation and communication.
Each selected expert processes one mixed input, preserving the active-expert count and activated expert FLOPs. Input mixing takes place on the source device before dispatch; each token still sends vectors of dimension , preserving all-to-all communication volume. Additional work comes from routing, projection, and input mixing; for fixed , the extra scoring and projection cost scales as . Appendix § 6.6 gives the parameter and operation counts, together with same-cluster throughput measurements for the router and view-fed ablation configurations.
Conclusion
Multi-View Latent MoE reuses routing preferences to improve expert selection and shape expert computation. Motivated by weak agreement between single-router rankings and empirical expert-utility rankings, its Multi-Router submodule combines independently initialized routing heads that learn highly differentiated preferences. Routing diagnostics show relative rank-correlation gains of globally and approximately in the layer with the lowest single-router alignment. Multi-View further uses each selected expert’s relative scores across routing heads to blend latent views into its input before dispatch. Each selected expert runs once, preserving the active-expert count, expert FLOPs, and all-to-all communication volume. On a 24-layer 10B-A1B model trained for 708B tokens, full Multi-View Latent MoE improves the nine-task downstream mean by percentage points over the baseline; its Multi-Router submodule alone provides a -point gain. Controlled mixing interventions show that the model makes effective use of expert- and token-dependent input differences. These findings establish routing preferences as a signal for both selecting experts and shaping how they process each token.
Limitations
The experiments use a 24-layer 10B-A1B backbone with a shared training seed and evaluate nine downstream tasks at the 708B-token checkpoint. The controlled interventions measure the trained model’s use of view mixing at that checkpoint.
Extended Method Discussion
This appendix provides supporting analyses for the main-text method. Algorithm alg:mv in § 3.3 gives the forward pass of the Multi-View routed-expert branch.
LatentMoE as the backbone
Because the experts operate in latent space, replacing a selected expert’s input with a view leaves its architecture, width, and hyperparameters unchanged. The existing expert blocks can therefore be reused without increasing expert capacity. Latent-space designs have also been studied in other MoE architectures [12, 15].
Contextual representations and expert inputs
Attention produces a contextual representation , but the baseline MoE still evaluates all selected experts at the same projected input . The attention representation and the expert inputs are separate architectural choices: multiple projections can give the selected experts different inputs derived from the same . We use the term view for this learned latent projection or the resulting expert input.
Choice of fusion position
We consider three positions for combining router scores. (1) Before the nonlinearity (logit sum): is equivalent to a single scorer with weights (§ 3.2). (2) After the sigmoid (ours): each router first applies its own nonlinearity, after which the resulting scores are combined. (3) After normalization into probabilities: each router’s scores are normalized before summation, removing differences in their overall scale. Table tab:main reports the corresponding training comparisons.
Choice of fusion operator
For post-sigmoid fusion, different aggregation operators emphasize different score patterns. Let denote one router’s score for an expert. An additive operator, , allows a high score from one router to increase the aggregate even when other scores are low. Multiplicative operators, and the geometric mean , suppress an expert when any router assigns it a near-zero score. We also consider additive operators with an agreement term, , or a disagreement penalty, ; both reduce to summation at .
Summation has two useful properties, with empirical comparisons in Table tab:main and Figure 7. First, multiplicative aggregation is sensitive to low scores: one near-zero score can suppress an expert regardless of the other routers’ scores. Summation does not impose this requirement for agreement. Second, with , the expert’s output gate in Eq. (3) can be decomposed into per-router contributions, . A product does not have this additive decomposition, and its scale depends on . The same raw sigmoid scores also supply the input-mixing weights in § 3.3: for each selected expert, its scores are normalized across routers, , so that .
We also evaluate a variant that selects experts using and gates their outputs using . Selection therefore depends on an expert’s highest router score, while the output weight uses scores from all routers. This separates the effects of score aggregation on selection and gating. Conceptually, it corresponds to choosing one router at the selection stage, analogous to choosing one view for an expert’s input in a hard single-view variant. Table tab:mv_abl separately reports the evaluated view-mixing variants.
Aggregation before selection
The layer aggregates all router scores before a single top- selection. Every router scores the full expert set , and the number of activated experts remains , independent of the number of routers or of views. All routing heads contribute to one shared ranking over the full expert pool.
An alternative is to distribute a fixed budget of selections adaptively among routers. Although this preserves the total budget, each router still ranks experts independently and selects only up to its assigned quota. An expert may therefore be excluded from each local selection even if its aggregate score would place it in the global top-. We instead aggregate the scores and perform one selection with a fixed total of experts.
Cost accounting in detail
Let be the number of MoE layers. The hidden dimension is , the latent dimension is , and there are routed experts with active per token. For the main configuration with , each layer adds projections of shape and routing heads of shape . Thus,
With , , , , and , Multi-View adds parameters per layer and in total, approximately of the 10B-parameter backbone. Multi-Router adds only the routing heads: parameters per layer and in total, approximately .
The expert path still evaluates unchanged experts on -dimensional inputs. For each token and layer, the additional projections and scoring matrices require multiply–accumulate operations (MACs). Counting one MAC as two FLOPs, and a direct mixture as multiplications and additions per coordinate, the additional forward cost for is approximately
The estimate covers scoring and projection matrix multiplications and input blending per token and layer. Expert-block FLOPs are unchanged. The implementation still dispatches vectors of dimension per token, preserving all-to-all volume (App. § 6.7).
Table 3 reports throughput for scoring-only and coupled/detached view-fed variants on one cluster.
| Variant | Expert FLOPs | Throughput |
| Multi-Router () | ||
| Geometric mean | ||
| Product | ||
| Coupled views | ||
| Detached views |
Table 3. Same-cluster throughput in tokens/s/card for the listed 10B-A1B ablation configurations. Coupled and detached variants use view-fed scoring. Expert FLOPs are normalized to the backbone. Main-framework parameter counts are given in Eq. (6).
Implementation
Standard MoE implementations permute tokens before all-to-all communication, copying each token’s latent vector once per selected expert into a buffer organized by expert. Multi-View instead constructs the -dimensional mixed input for each selected (token, expert) pair on the token’s device before dispatch. For tokens, the implementation flattens the token–expert slots, each with one selected expert as its destination. Each token still dispatches vectors of dimension , so the all-to-all volume is unchanged. Dispatching all views and combining them on the expert’s device would instead multiply that volume by . The return all-to-all communication, inverse permutation, and gated sum of expert outputs are unchanged.
Because selection uses no per-view quota, load balancing is applied once over the experts using the backbone’s existing scheme. No auxiliary loss is added for the views.
The expert bias is added after summing the routing-head scores and is used only by top- selection, ; it enters neither the input-mixing weights nor the output gates. The output gates follow Eq. (3): the uncorrected are normalized over the selected set and then multiplied by the backbone’s per-expert output scale. For input mixing, Eq. (4) normalizes each selected expert’s raw sigmoid scores across routers, and the resulting coefficients directly form . The input-ratio computation uses for numerical stability, omitted from the mathematical formulas. At fixed parameters, replacing the score sum by its mean and scaling by preserves selection, output gates, and view weights. To maintain the bias correspondence during training, its initialization and update increments must also be scaled by . In Multi-Router, repeating one routing head times and multiplying its balancing bias by preserves the forward computation with a single routing head.
Execution-path verification.
For the checkpoint probes, we verify the expert IDs, output gates, and mixing weights received by the dispatcher. Appendix § 7.6 reports these execution checks and identity-replay errors.
Interaction with other components
Multi-View modifies the MoE down-projection, router, and permutation step. Attention, including multi-head latent attention (MLA) [16], is unchanged, as is the KV cache. Hyper-connections and their manifold-constrained variant mHC use multiple residual streams with learned mixing between sublayers [31, 32]. Multi-View accepts the resulting sublayer input and returns a -dimensional output, so these designs are architecturally compatible. Multi-token prediction (MTP) [26] adds prediction modules and a loss term, while Multi-View leaves the training objective unchanged. In our configuration, the MTP module uses a dense FFN and is not modified. The method is also compatible in design with fine-grained and shared experts, auxiliary-loss-free balancing, and dense or hash-routed early layers [6]. It leaves the vocabulary, embeddings, routed and active expert counts, expert blocks, up-projection, and expert-parallel layout unchanged.
Warm-starting from a LatentMoE checkpoint
An existing LatentMoE checkpoint provides a possible initialization for Multi-View. View 1 retains the checkpoint’s down-projection and routing head. The remaining views are initialized from the same projection with small perturbations, and their routing heads are initialized to zero. These additional routing heads add the same constant to every expert’s raw score, preserving the raw-score ordering but generally changing normalized output gates. Keeping the balancing bias unchanged also preserves the actual bias-corrected selection, because the same constant is added to every entry of . The views can then differentiate during continued training.
Properties: tied views and view differentiation
Tied-view reduction.
Since , tying all gives for every selected expert. With the same router scores, selected set, and output gates, this recovers Multi-Router with the shared input . All routers continue to score the same expert pool, independently of whether the view projections are tied.
Expert-dependent mixing.
For a given token, mixing weights that vary across experts allow those experts to receive different inputs. If all selected experts share , then
All experts then share an input, but can still vary with the token, so input modulation remains. When is constant, including , the mixture can be absorbed into a fixed projection. The main method’s retains both token and expert dependence, providing modulation at this granularity without an additional mixing network.
Per-view load.
Aggregate load balancing (App. § 6.7) controls the number of tokens assigned to each expert, which affects dispatch cost and expert utilization. For the per-view diagnostics, let denote the recorded weight of view at expert . Define as the fraction of view ’s recorded weight mass assigned to expert . Aggregate assignment counts can remain unchanged while varies. Different views can therefore concentrate on different subsets even when the aggregate expert load is uniform. We report the coefficient of variation of and the effective expert count , where is the entropy of (App. § 7.7). A low effective count indicates concentration on fewer experts. We also report the ratio of an expert’s selection share to its gate-mass share, which distinguishes changes in assignment frequency from changes in output weighting.
Differentiation.
The tied configuration is symmetric, but it is preserved by a gradient update only when the gradient components of the views coincide. Each receives gradients through the selected experts that assign nonzero mixing weight to view , scaled by in Eq. (5). Because these weights can vary across experts and tokens, the views can receive different updates without an explicit diversity penalty. We assess router disagreement (§ 4.3) and per-view load (App. § 7.7). The checkpoint probes in § 4.3 measure mixing-weight and expert-input diversity. Controlled interventions in App. § 7.6 test whether the trained model uses the correspondence between experts, tokens, and mixtures.
Experimental Details and Additional Analyses
Full configuration
Appendix § 7.2 gives the complete architecture and training configuration. The model has hidden dimension , 24 layers, latent dimension , and routed experts with top-8 selection and one always-active shared expert. Matrix parameters use Muon, while embeddings, the output head, biases, and router weights use AdamW. Training uses a warmup–stable–decay schedule with peak learning rate , 4096-token sequences, approximately M tokens per optimizer step, and BF16 with FP8 (E4M3) matrix multiplications under 8-way expert parallelism. Each variant in § 4 changes one experimental setting.
The 10B-A1B configuration
Table tab:arch10b specifies the 10B-A1B latent-space MoE, with approximately 10B total and 1B activated parameters. This configuration is used for all main results, ablations, and diagnostics in § 4.
| Component | Setting |
| Layers / hidden size | 24 / 1024 |
| Normalization / embeddings | RMSNorm / untied |
| Attention | MLA, 16 attention heads, gated output |
| Query / KV latent rank | 512 / 512 |
| QK dims per attention head (no RoPE + RoPE) | 192 + 64 |
| Value dim per attention head | 256 |
| QK-Norm / RoPE base | yes / |
| MoE layers | all 24 |
| Latent dim | 512 |
| Routed experts / active | 256 / 8 |
| Routed expert FFN | SwiGLU, intermediate 1024 |
| Shared experts | 1, intermediate 1024 |
| Balancing | expert bias, rate |
| MTP | 1 module, dense FFN 5120, weight 0.3 |
Attention.
Each layer uses multi-head latent attention (MLA) [16] with 16 attention heads. Queries are compressed to rank 512, and keys and values share a rank-512 latent representation. Each attention head has 192 query–key dimensions without rotary embeddings, 64 decoupled rotary dimensions, and a 256-dimensional value. QK-normalization is applied, rotary embeddings use base , and the attention output is gated. The cache stores the 512-dimensional KV latent and the 64-dimensional rotary key for each token and layer.
MoE.
All 24 FFN layers are latent-space MoE layers [12], with no preceding dense FFN layers. Each token is projected from to , and 8 of 256 routed experts are activated. Each routed expert is a latent-space SwiGLU FFN with intermediate dimension 1024; one shared expert with the same intermediate dimension is always active. Expert biases are updated at rate for load balancing [22], and the auxiliary-loss coefficient is 0.
MTP.
Following [26], one multi-token prediction module predicts one additional future token. Its FFN is a dense SwiGLU with intermediate dimension 5120. Its loss is added to the language-modeling loss with weight 0.3.
Training.
The sequence length is 4096 tokens, and the global batch contains 1920 sequences, or approximately 7.9M tokens per step. Matrix parameters use Muon [33] with momentum 0.95 and five Newton–Schulz steps. Embeddings, the output head, biases, and router weights use AdamW with and . The learning rate reaches after 1000 warmup steps and follows a warmup–stable–decay schedule ending at 0. Weight decay is 0.1, and gradients are clipped at 1.0. Training uses BF16 with FP8 (E4M3) matrix multiplications, 8-way expert parallelism, and seed 42. The main baseline, Multi-Router, and Multi-View comparisons use the 708B-token checkpoints; training ablations use a common checkpoint.
Comparability protocol
The loss differences are of order , making configuration consistency particularly important. We use the following comparison protocol.
Run-pair reference. A pair of near-matched runs exhibits a systematic loss offset. Its observed difference provides an empirical reference for the ablation magnitudes in Table tab:main.
Matched code and configuration. Each variant uses a common code snapshot with one experimental setting changed. Comparisons use configurations recorded during training, including initialization and its scale, optimizer, schedule, and data order.
Complementary diagnostics. We report expert load and utilization (App. § 7.7) alongside loss to characterize how the expert pool is used.
Additional measurements
Downstream protocol.
We evaluate the baseline, Multi-Router, and score-derived Multi-View checkpoints from training step 90K, corresponding to 708B tokens at 7.86M tokens per step, on the nine tasks in Table 1. CMMLU [37], MMLU [38], MMLU-Redux [39], MMLU-Pro [40], and BBH [42] use 5-shot prompts. GPQA [41] uses 10-shot prompts. These tasks are answered without chain of thought. HotpotQA [43], SimpleQA [44], and PreciseWikiQA [45] use 10-shot prompts with chain of thought and are scored by an LLM judge (DeepSeek-V4-Flash). Group means and the overall mean are unweighted.
Training trajectories and aggregation operators.
Figure 4 in the main text shows the training trajectories of the baseline, Multi-Router, and Multi-View through 708B tokens and their loss differences. Figure 7 compares the post-sigmoid fusion operators at the shared ablation checkpoint.
Figure 7. Training-loss changes for router-fusion variants with . All evaluated variants improve over the single-router baseline. The shaded range shows the observed run-pair difference.
Router-score alignment
For a layer and a sampled token set , let be the latent input of token . Evaluating all experts gives the normalized output-norm proxy
The normalization removes each expert’s average output scale. Let be the score before balancing bias: the score from a single routing head for the baseline and the summed scores from all routing heads for Multi-Router. Correlation is computed over the experts for each token:
Token values are averaged within each microbatch to obtain for layer , rank , and microbatch . With participating ranks and microbatches , the layer and global averages are
For layer-wise summaries, we match the two diagnostic exports by logged step and average the last 201 records available in both exports. Layer-wise bands show one standard deviation over these measurements.
Expert-input diversity and controlled mixing interventions
Evaluation protocol.
All probes in this subsection use the score-derived two-view checkpoint at step 90K, the same checkpoint used in Table 1. The main evaluation contains 256 held-out sequences of 4096 input tokens, giving 1,048,320 next-token predictions; the clean checkpoint’s NLL is nats/token. The layer scan uses 80 sequences, giving 327,600 predictions. No model parameters are updated.
Figure 8. Additional analyses of the 708B-token Multi-View checkpoint. (a) Mean view-1 share by expert, with expert ranks determined independently in each layer. (b) Agreement between the two routing heads on 256 sequences. (c) Single-view interventions on 80 sequences, with downstream routing recomputed; bands are pointwise 95% CIs. (d) Final-layer effects grouped by clean token-NLL quintile on 256 sequences. The x-axis indexes five groups with equal numbers of predictions; error bars show 95% intervals.
Diversity definitions.
For token , let be its selected experts. The probe uses the main method’s actual mixing proportions, , so and . The actual input to expert is
We compute
The bands in Figure 5(a) are the 25th–75th percentiles across tokens. Mean cosine similarity between the two views ranges from to across layers. The median view-separation norm, divided by the mean of the two view norms, ranges from to . The projections therefore produce distinct views at this checkpoint.
Expert preferences.
Figure 8(a) shows each expert’s mean over tokens selecting it; experts are sorted independently within each layer. All 256 experts are observed in each layer. The within-token component accounts for – of the measured share variance, while grouping by expert ID explains –. Across layers, top-8 Jaccard between routing heads ranges from to , and top-1 agreement ranges from to (Figure 8(b)).
Interventions and uncertainty.
Expert permutation shuffles the view proportions among the eight selected experts of each token. Token permutation shuffles them among tokens selecting the same expert; of selected assignments are permutable. For each selected assignment, a replacement is implemented through the dispatcher channels . The output gate remains attached to the original assignment and is not permuted. Uniform mixing uses ; the two single-view interventions use or . View projections, expert parameters, the selected expert set, and output gates at the intervened layer are retained. All reported changes use paired predictions on the same sequences. For the final-layer permutation tests, we average three permutation seeds within each sequence and bootstrap sequences to obtain 95% CIs. Table 4 reports the final-layer results. The probe reports mean absolute changes in the view-1 share of , , and for expert permutation, token permutation, and uniform mixing, respectively.
| Intervention | NLL | 95% CI |
| Identity replay | ||
| Token permutation | ||
| Uniform mixture | ||
| Expert permutation | ||
| View 1 only | ||
| View 2 only |
Table 4. Final-layer interventions on 256 held-out sequences. All values, including CIs, are in nats/token. Expert selection and output gates are fixed to numerical precision; only view-mixing proportions change.
Execution and replay checks.
On a 16-sequence execution-check subset, the dispatcher receives the same expert IDs across interventions; output gates differ by at most in relative error, and the realized view proportions at the dispatcher differ from their intended values by at most . Two clean passes agree exactly on 32 sequences. At layer 24, identity replay gives a mean NLL change of nats/token, with a 95% CI spanning zero, and a mean absolute token-level difference of . Using replay rather than the clean pass as the paired reference leaves the reported effects unchanged at the displayed precision.
For shallower interventions, rounding-level changes can alter later top- decisions. At layer 1, identity replay changes the final-layer expert set for of tokens in the execution check. On the 80-sequence scan, its mean NLL change is , with 95% CI nats/token. The scan estimates total effects with downstream routing recomputed. Its intervals use its own 80-sequence sample and are pointwise intervals for individual layers.
Single-view and token-group results.
Both single-view interventions increase NLL at every scanned layer (Figure 8(c)); their final-layer effects on 256 sequences appear in Table 4. Both branches contribute to prediction in this trained checkpoint. Figure 8(d) groups predictions into equal-sized quintiles of clean token NLL, with 209,664 predictions per group. Expert permutation has its largest absolute effect in the highest-loss quintile ( versus in the lowest). Uniform mixing gives a middle-quintile NLL of . The largest intervention effects occur in the highest-loss group.
Aggregate and per-view expert load
Per-view load measurements show how each view’s recorded weight mass is distributed across the shared expert pool. Load balancing acts once on the aggregated assignment over the experts (App. § 6.7), so different views may concentrate on different subsets while the aggregate load remains balanced. We therefore report, for each view, the coefficient of variation (CV) of its recorded weight mass across experts and an effective expert count , where is the entropy of this normalized weight mass (App. § 6.10). The aggregate load CV is , compared with for the baseline, and the aggregate effective count is , compared with out of . The individual views have CVs of and and effective counts of and . These counts correspond to approximately of the pool for each view and in aggregate. This pattern occurs across MoE layers (layer 2: in aggregate versus and ; layer 15: versus and ). Together, the views combine concentrated expert preferences with broader aggregate expert usage.
References
- [1]Cai, W., Jiang, J., Wang, F., Tang, J., Kim, S., & Huang, J. (2024). A Survey on Mixture of Experts in Large Language Models. https://arxiv.org/abs/2407.06204
- [2]Kimi Team, et al. (2026). Kimi K3: Open Frontier Intelligence. https://arxiv.org/abs/2607.24653
- [3]GLM-5-Team, et al. (2026). GLM-5: from Vibe Coding to Agentic Engineering. https://arxiv.org/abs/2602.15763
- [4]Wang, H., et al. (2026). ERNIE 5.0 Technical Report. https://arxiv.org/abs/2602.04705
- [5]Wu, S., Luo, J., Chen, X., Li, L., Zhao, X., Yu, T., Wang, C., Wang, Y., Wang, F., Qiao, W., He, H., Zhang, Z., Sun, Z., Mao, J., & Shen, C. (2024). Yuan 2.0-M32: Mixture of Experts with Attention Router. https://arxiv.org/abs/2405.17976
- [6]Roller, S., Sukhbaatar, S., Szlam, A., & Weston, J. (2021). Hash Layers For Large Sparse Models. https://arxiv.org/abs/2106.04426
- [7]Nguyen, T., Tran, N. N., Nguyen, K., & Baraniuk, R. G. (2025). Improving Routing in Sparse Mixture of Experts with Graph of Tokens. https://arxiv.org/abs/2505.00792
- [8]Li, Z., Li, Z., & Zhou, T. (2025). Routing Manifold Alignment Improves Generalization of Mixture-of-Experts LLMs. https://arxiv.org/abs/2511.07419
- [9]Wang, X., Hayou, S., & Nalisnick, E. (2026). The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise. https://arxiv.org/abs/2604.09780
- [10]Wu, X., Huang, S., Wang, W., Ma, S., Dong, L., & Wei, F. (2024). Multi-Head Mixture-of-Experts. Advances in Neural Information Processing Systems, 37. https://doi.org/10.52202/079017-2984
- [11]Huang, S., Wu, X., Ma, S., & Wei, F. (2024). MH-MoE: Multi-Head Mixture-of-Experts. https://arxiv.org/abs/2411.16205
- [12]Elango, V., Bhatia, N., Waleffe, R., Shafipour, R., Asida, T., Khattar, A., Assaf, N., Golub, M., Guman, J., Mitra, T., Zhao, R., Borkar, R., Zilberstein, R., Patwary, M., Shoeybi, M., & Rouhani, B. (2026). LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts. https://arxiv.org/abs/2601.18089
- [13]Cui, C., Jackson, R., Herrera, B. J., Tárano, A. M., & Kerner, H. (2026). Multi-Head LatentMoE and Head Parallel: Communication-Efficient and Deterministic MoE Parallelism. https://arxiv.org/abs/2602.04870
- [14]Jin, P., Zhu, B., Yuan, L., & Yan, S. (2024). MoH: Multi-Head Attention as Mixture-of-Head Attention. https://arxiv.org/abs/2410.11842
- [15]Liu, Z., Wu, H., She, R., Fu, X., Han, X., Zhong, T., & Yuan, M. (2025). MoLAE: Mixture of Latent Experts for Parameter-Efficient Language Models. https://arxiv.org/abs/2503.23100
- [16]DeepSeek-AI, et al. (2024). DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. https://arxiv.org/abs/2405.04434
- [17]Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., & Dean, J. (2017). Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. https://arxiv.org/abs/1701.06538
- [18]Fedus, W., Zoph, B., & Shazeer, N. (2021). Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. https://arxiv.org/abs/2101.03961
- [19]Zoph, B., Bello, I., Kumar, S., Du, N., Huang, Y., Dean, J., Shazeer, N., & Fedus, W. (2022). ST-MoE: Designing Stable and Transferable Sparse Expert Models. https://arxiv.org/abs/2202.08906
- [20]Zhou, Y., Lei, T., Liu, H., Du, N., Huang, Y., Zhao, V., Dai, A., Chen, Z., Le, Q., & Laudon, J. (2022). Mixture-of-Experts with Expert Choice Routing. https://arxiv.org/abs/2202.09368
- [21]Dai, D., Dong, L., Ma, S., Zheng, B., Sui, Z., Chang, B., & Wei, F. (2022). StableMoE: Stable Routing Strategy for Mixture of Experts. https://arxiv.org/abs/2204.08396
- [22]Wang, L., Gao, H., Zhao, C., Sun, X., & Dai, D. (2024). Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts. https://arxiv.org/abs/2408.15664
- [23]Chi, Z., Dong, L., Huang, S., Dai, D., Ma, S., Patra, B., Singhal, S., Bajaj, P., Song, X., Mao, X.-L., Huang, H., & Wei, F. (2022). On the Representation Collapse of Sparse Mixture of Experts. https://arxiv.org/abs/2204.09179
- [24]Dai, D., Deng, C., Zhao, C., Xu, R. X., Gao, H., Chen, D., Li, J., Zeng, W., Yu, X., Wu, Y., Xie, Z., Li, Y. K., Huang, P., Luo, F., Ruan, C., Sui, Z., & Liang, W. (2024). DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models. https://arxiv.org/abs/2401.06066
- [25]Jiang, A. Q., et al. (2024). Mixtral of Experts. https://arxiv.org/abs/2401.04088
- [26]DeepSeek-AI, et al. (2024). DeepSeek-V3 Technical Report. https://arxiv.org/abs/2412.19437
- [27]Ran, J., Zhao, G., Wu, Y., Zhu, D., Wu, L., Zhao, Y., Yang, T., Sun, L., Zhang, X., & Li, S. (2025). Router Upcycling: Leveraging Mixture-of-Routers in Mixture-of-Experts Upcycling. https://arxiv.org/abs/2509.00679
- [28]Vejendla, H. (2025). SliceMoE: Routing Embedding Slices Instead of Tokens for Fine-Grained and Balanced Transformer Scaling. https://arxiv.org/abs/2510.04286
- [29]Zhang, J.-C., Xiong, Y.-J., Qiu, X.-H., Xia, C.-M., Dai, F., & Zhou, Z. (2025). Mixture of Routers. https://arxiv.org/abs/2503.23362
- [30]Lv, A., Xie, R., Qian, Y., Wu, S., Sun, X., Kang, Z., Wang, D., & Yan, R. (2025). Autonomy-of-Experts Models. https://arxiv.org/abs/2501.13074
- [31]Zhu, D., Huang, H., Huang, Z., Zeng, Y., Mao, Y., et al. (2024). Hyper-Connections. https://arxiv.org/abs/2409.19606
- [32]Xie, Z., Wei, Y., Cao, H., Zhao, C., Deng, C., et al. (2025). mHC: Manifold-Constrained Hyper-Connections. https://arxiv.org/abs/2512.24880
- [33]Liu, J., Su, J., Yao, X., Jiang, Z., Lai, G., et al. (2025). Muon is Scalable for LLM Training. https://arxiv.org/abs/2502.16982
- [34]Li, Z., Li, Z., & Zhou, T. (2025). C3PO: Critical-Layer, Core-Expert, Collaborative Pathway Optimization for Test-Time Expert Re-Mixing. Conference on Language Modeling (COLM). https://arxiv.org/abs/2504.07964
- [35]Li, Z., Li, Z., & Zhou, T. (2025). R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Experts. Proceedings of the 42nd International Conference on Machine Learning (ICML). https://arxiv.org/abs/2502.20395
- [36]Chen, Y., Wang, P., Shao, Y., Zeng, N., Xu, C., & Cheng, J. (2026). Ban&Pick: Enhancing Performance and Efficiency of MoE-LLMs via Smarter Routing. Findings of the Association for Computational Linguistics: EMNLP 2026, 14228–14253. https://aclanthology.org/2026.findings-emnlp.678/aclanthology.org/2026.findings-emnlp.678
- [37]Li, H., Zhang, Y., Koto, F., Yang, Y., Zhao, H., Gong, Y., Duan, N., & Baldwin, T. (2023). CMMLU: Measuring massive multitask language understanding in Chinese. https://arxiv.org/abs/2306.09212
- [38]Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020). Measuring Massive Multitask Language Understanding. https://arxiv.org/abs/2009.03300
- [39]Gema, A. P., Leang, J. O. J., Hong, G., Devoto, A., Mancino, A. C. M., Saxena, R., He, X., Zhao, Y., Du, X., Madani, M. R. G., Barale, C., McHardy, R., Harris, J., Kaddour, J., Krieken, E. van ., & Minervini, P. (2024). Are We Done with MMLU?. https://arxiv.org/abs/2406.04127
- [40]Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., Li, T., Ku, M., Wang, K., Zhuang, A., Fan, R., Yue, X., & Chen, W. (2024). MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark. https://arxiv.org/abs/2406.01574
- [41]Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., & Bowman, S. R. (2023). GPQA: A Graduate-Level Google-Proof Q&A Benchmark. https://arxiv.org/abs/2311.12022
- [42]Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q. V., Chi, E. H., Zhou, D., & Wei, J. (2022). Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them. https://arxiv.org/abs/2210.09261
- [43]Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W. W., Salakhutdinov, R., & Manning, C. D. (2018). HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. https://arxiv.org/abs/1809.09600
- [44]Wei, J., Nguyen, K., Chung, H. W., Jiao, Y. J., Papay, S., Glaese, A., Schulman, J., & Fedus, W. (2024). Measuring short-form factuality in large language models. https://arxiv.org/abs/2411.04368
- [45]Bang, Y., Ji, Z., Schelten, A., Hartshorn, A., Fowler, T., Zhang, C., Cancedda, N., & Fung, P. (2025). HalluLens: LLM Hallucination Benchmark. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 24128–24156. https://doi.org/10.18653/v1/2025.acl-long.1176


