WorksComputer Science
MoE–LoRA Methods for Multilingual Adaptation: A Systematic Empirical Study
Abstract
Routed low-rank adapters offer a parameter-efficient means to improve the performance of large language models (LLMs) across tasks. Hence, it is assumed that they can also be used to improve the support of low-resource languages in LLMs, which currently serve English far better than the thousands of other languages spoken worldwide. However, this assumption has never been verified empirically under a shared multilingual evaluation protocol. We address exactly this gap by comparing eight low-rank adaptation (LoRA) methods on a frozen base model, using 146 training runs across 200 unique languages and more than 10,000 evaluations. Our results suggest that a reduced-scale search can identify viable configurations of LoRA methods but does not reliably predict their relative performance, as the top-ranking configuration is never best at scale. LoRA reliably improves language-model fit on translation data, consistently degrades reading comprehension, and benefits low-resource languages most. These findings suggest that LoRA methods should be regarded as targeted tools rather than general-purpose means for improving the multilingual capabilities of LLMs.
Introduction
Large language models (LLMs) are trained on data that are highly unevenly distributed across languages. English constitutes a substantial share of the training corpora used for widely deployed models such as Llama 3, whereas many other languages are represented by far less data (Dubey et al., 2024). This disparity contributes to lower and less reliable performance beyond English, particularly for low-resource languages [4] (Üstün et al. 2024; Ojo et al. 2025). Addressing this gap therefore requires compute-efficient methods for extending existing models to many languages, rather than training a separate model for each one.
Continued pretraining (CPT) on multilingual data can add language coverage without training a new model from scratch. However, joint training also creates competition for limited model capacity. Updates that help one language can have little effect on another or can reduce its performance, a pattern described as negative interference or the curse of multilinguality [3, 13] (Wu and Dredze 2020; Wang et al. 2020). Multilingual adaptation therefore needs to balance parameter sharing, language-specific capacity, and training cost.
Running CPT (Gururangan et al., 2020; Ke et al., 2023) at scale is expensive, which has made parameter-efficient fine-tuning (PEFT) the practical way to adapt large models. Low-Rank Adaptation (LoRA) freezes the base model and learns a single low-rank update (Hu et al., 2022). Because every language passes through those same parameters, LoRA cannot allocate capacity per language and leaves the interference problem largely unresolved. Mixture-of-Experts (MoE) extensions of LoRA target this limitation directly. They replace the single update with several low-rank experts that are shared or selected by a routing rule (Liu et al., 2024; Zadouri et al., 2024), adding capacity at low cost and letting different experts specialize. Such specialization can align with language structure. Indeed, routing in multilingual MoE models has been observed to correlate with implicit phylogenetic language groups (Srivastava et al., 2025), making MoE–LoRA a natural candidate for balancing shared and language-specific capacity during multilingual CPT. However, whether this additional structure actually leads to significantly better results remains an unresolved problem.
This question is all the more relevant when one considers that most MoE–LoRA methods were developed and validated for multi-task settings such as medical adaptation and instruction tuning (Liu et al., 2024; Zadouri et al., 2024), rather than for multilingual learning. The few efforts that extend them to multilingual settings each commit to a single design. MLAS-LoRA uses language-general and language-specific LoRA modules for machine translation (Dong et al., 2025), and MoE-LPR adds experts to a pretrained model with language-aware routing (Zhou et al., 2025a). However, no prior work systematically compares MoE–LoRA methods using the same base model, training data, and evaluation protocol for multilingual CPT. We address this gap through a controlled comparison that evaluates both aggregate performance and variation across tasks and languages.
Concretely, we compare eight parameter-efficient adaptation methods using a frozen Llama-3.1-8B base model. The search stages comprise 132 training runs on ten languages, including the longer follow-up on 22 combined configurations. We then carry the five best configurations, together with two plain-LoRA baselines, to fourteen broader runs covering 96 and 200 training languages, for 146 independent training runs in total. Four multilingual benchmarks evaluate reading comprehension, language-model fit on translation data, topic classification, and text classification, which gives each trained configuration around 70 benchmark and language measurements, 10,842 altogether.
Research questions. We organize the study around four questions:
How do rank and expert count affect multilingual performance?
How do MoE–LoRA methods compare under a shared multilingual evaluation?
Do the strongest methods remain effective under broader multilingual training conditions?
Which tasks and languages benefit most from MoE–LoRA adaptation?
Contributions. We contribute a controlled comparison of routed low-rank experts under massively multilingual continued pretraining, which yields three key findings: (i) low ranks win consistently, so the best-performing setting also trains the fewest parameters; (ii) a reduced-scale search finds configurations that remain viable at scale but does not reliably predict how they rank there; and (iii) what transfers instead is the direction of the effect, with gains concentrated on translation fit and on lower-resource languages. We release the protocol, per-language results, and code so that a new method can enter the same comparison.1
Related Work
In this section, abbreviations in bold indicate the baselines used in our experiments.
Mixture-of-Experts. Jacobs et al. (1991) introduced a modular supervised learning framework in which multiple expert networks are trained on disjoint subsets of the data. Following the transformer architecture (Vaswani et al., 2017), sparse MoE became a primary approach for reducing the computational cost of LLMs (Shazeer et al., 2017; Fedus et al., 2022; Ludziejewski et al., 2024), and MoEs empirically outperform dense counterparts at a fixed computational budget, particularly at large scale and with well-optimized routing [11] (Zoph et al. 2022; Zhang et al. 2024).
Low-Rank Adaptation. PEFT methods (Ding et al., 2023) attain comparable performance to full fine-tuning while updating only a fraction of the parameters. The most widely adopted of them, LoRA (Hu et al., 2022), trains only low-rank perturbations to selected weight matrices, which reduces memory usage. It also generalizes robustly (Xin et al., 2024) and better maintains the base model’s performance outside the target domain [9]. CoLA (Zhou et al., 2025b) extends it with an efficient initialization scheme and three collaborative strategies between the low-rank matrices.
MoE–LoRA Family. LoRA scales poorly to multi-task settings, where each task typically requires a separate adapter. To address this, Liu et al. (2024) proposed MOELoRA for multi-task medical applications, and numerous MoE–LoRA methods have followed with different routing strategies and expert architectures.
Initial contributions proposed dynamic LoRA composition and highly parameter-efficient MoE for instruction tuning (Huang et al., 2024; Feng et al., 2024; Zadouri et al., 2024), followed by token-level and layer-wise variants [10], multi-task ones, first proposed for vision with MTLoRA [2], and routing designed to mitigate world-knowledge forgetting (Dou et al., 2024; Wu et al., 2024). Among the methods we compare, AdaMoLE (Liu and Luo, 2024) and AdaMoE (Zeng et al., 2024)2 select experts adaptively through threshold networks or null-expert designs, HydraLoRA Tian et al., 2024 reduces redundancy through an asymmetric structure, HMoRA Liao et al., 2025 combines token- and task-level routing across layers, MTL-LoRA Yang et al., 2025b separates task-specific from shared representations, and MoV Zadouri et al., 2024 mixes vectors that inhibit and amplify inner activations Liu et al., 2022.
These methods target instruction tuning and multi-task learning, not languages. Multilingual adaptation has instead relied on modular methods such as MAD-X Pfeiffer et al., 2020, X-MOD Pfeiffer et al., 2022, and MLAS-LoRA Dong et al., 2025, which assign modules per language by construction rather than by routing. Routed multilingual work mostly targets speech recognition [5] (Song et al. 2024; Kwok et al. 2025), and MoE-LPR routes between full feed-forward rather than low-rank experts Zhou et al., 2025a. LoRALib benchmarks MoE–LoRA methods, but not for multilingual adaptation Wang et al., 2025. Routed low-rank experts under a shared multilingual CPT protocol at several language scales remain unstudied.
Multilingual Language Modeling. Early multilingual models showed that a single pretrained transformer can learn shared representations across dozens of languages [13] (Devlin et al. 2019), and large-scale multilingual LLMs have since extended this to hundreds of languages (Workshop, 2024; Srivastava et al., 2025; Yang et al., 2025a; Team et al., 2026).
A complementary line of work uses CPT as a cheaper route to multilinguality. Sailor2 Dou et al., 2025 applies block expansion and CPT on roughly 500 billion tokens for 15 Southeast Asian languages, and comparable efforts cover Japanese, Korean, and Indic languages (Fujii et al., 2024; Vo et al., 2024; Nag et al., 2025). Others study how language mixing interacts with CPT Li et al., 2025 or target selected layers with LoRA when adding new languages Owodunni and Kumar, 2026. We apply solutions from the multi-task domain to this setting and compare MoE–LoRA methods across languages and tasks.
Methodology
Problem Setting
Let denote the parameters of a pretrained autoregressive language model and a CPT corpus of token sequences drawn from languages. The goal is to adapt the model to the joint distribution of all languages while mitigating catastrophic forgetting Kirkpatrick et al., 2017 of the capabilities it already has. Rather than updating , we keep it frozen and train a separate set of parameters on top of it § 3.3.3.
Causal Language Model Objective
Given a sequence of tokens, the CLM objective maximizes the probability of each token conditioned on its causal prefix using the standard autoregressive next-token prediction mechanism and cross-entropy loss over target words [8] (Vaswani et al. 2017):
where is the set of trainable parameters (§ 3.3.3) and is the usual softmax over the output projection of the final hidden state.
MoE–LoRA Architectures
Expert Modules
All pretrained weights are frozen (Dou et al., 2024; Zhou et al., 2025a). For each target weight matrix at layer , we introduce low-rank expert modules following the LoRA formulation Hu et al., 2022:
where for each expert , is the down-projection, the up-projection, the rank, and the input to the layer.
Routing and Aggregation
The adapted layer output combines the frozen weights with a weighted sum of experts, utilizing the LoRA scaling factor Hu et al., 2022 and a sparse routing mechanism (Shazeer et al., 2017; Fedus et al., 2022):
where is the LoRA scaling hyperparameter, tied to the rank as described in § 4.1, and is the routing weight for expert , determined by a routing function that may condition on the token representation , a language identifier, the layer index , or any combination thereof.
Trainable Parameters
The set of trainable parameters is defined as:
where contains the routing mechanism parameters (e.g., gating weights, threshold networks, or language embeddings) at layer , and is the total number of transformer layers. Pretrained weights receive no gradient.
Training Objective
To prevent expert collapse and ensure balanced utilization, the CLM loss is augmented with an auxiliary loss as described in sparse MoE literature (Shazeer et al., 2017; Fedus et al., 2022):
where is a balancing coefficient and is an auxiliary loss that regularizes the routing distribution (e.g., encouraging balanced expert utilization or routing certainty). Gradients of flow through the expert modules and routing parameters only, with each expert’s learning signal scaled by its routing weight .
Experimental Setup
The study runs in three stages, described in § 4.4. The first two are exploratory, because they use the same observations both to select and to compare configurations.
The design is organized by the two quantities of § 3, the per-expert rank and the expert count . We use expert count throughout for what some methods call heads or projections, and where a method divides a total rank budget among its experts. We report the per-expert value, so that a total rank of 64 over 32 experts appears as . Configurations are named by method, , , and top-, the number of experts routed per token, so MOELoRA r2/16 k8 has , , and .
Model and Training Data
All experiments start from the frozen meta-llama/Llama-3.1-8B checkpoint (Dubey et al., 2024), which supplies the pretrained parameters of § 3. English training text comes from FineWeb and the remaining languages from FineWeb2 (Penedo et al., 2024, 2025). We use training sets of , 96, or 200 languages (see Appendix § 8.§ 1). Each set takes the same fraction of every language’s available documents, so languages keep their relative sizes in FineWeb2 and low-resource languages receive no extra weight. Table A4 gives the resulting proportions.
| Training set | Labels | Ex. (M) | Top-1 (%) | Top-5 (%) | 50% | 90% | Min (%) |
| 10 (search) | 13 | 0.10 | 35.5 | 85.9 | 2 | 7 | 0.0040 |
| 10 (follow-up) | 13 | 0.40 | 35.5 | 85.9 | 2 | 7 | 0.0040 |
| 96 | 102 | 1.51 | 23.3 | 71.8 | 3 | 16 | 0.0001 |
| 200 | 215 | 3.88 | 18.2 | 57.4 | 5 | 26 | 0.0001 |
Table A4. Composition of each prepared training set.
Unless a method requires a different adapter layout, we attach adapters to the attention and feed-forward projections of every layer. The shared settings include a dropout rate of 0.05, a learning rate of , BF16 training, a sequence length of 1,024, and a seed of 42. We set [9], so the scaling stays at 2 across ranks for every method except HMoRA (Appendix § 8.§ 3). The base model stays frozen, so training updates only the expert and routing parameters , and every run uses four NVIDIA A100 40GB GPUs. Settings that only one method defines, such as its routing rule, are recorded per run in the released configuration files.
Methods
The candidate pool covers AdaMoE, AdaMoLE, HMoRA, HydraLoRA, LoRA, MOELoRA, MTL-LoRA, and MoV, which routes vectors rather than low-rank matrices but demonstrates stronger performance than LoRA (Zadouri et al., 2024). They represent different ways of sharing or routing experts, and each can be trained in a single multilingual run.
The search space is not a complete factorial grid, because not every method defines every factor and the number of settings per method differs. We therefore restrict controlled comparisons to configurations with shared coverage and give the full matrix in Appendix § 8.§ 2. Plain LoRA takes part in the shared rank panel, and its two best ranks there also serve as baselines in both broader regimes. They help separate the effect of the multi-expert structure from that of adaptation itself.
Evaluation
We evaluate the final checkpoint with the LM Evaluation Harness (Gao et al., 2024) on four multilingual benchmarks. Belebele (Bele.) measures reading comprehension [6], FLORES (FLO.) supplies parallel translation data (Team et al., 2022), SIB-200 (SIB) measures topic classification [1], and Taxi1500 (Taxi) measures text classification (Ma et al., 2025). Together they cover both task accuracy and language-model fit.
We count a training run as one configuration trained once and an evaluation run as one model measured on one benchmark and language pair, counting the two FLORES directions separately (Table A1).
| Stage | Languages | Hours/run | Runs | Purpose |
| One-axis rank sweep | 10 | 3.6 | 47 | Effect of rank |
| One-axis expert-count sweep | 10 | 3.6 | 63 | Effect of expert count |
| Mixed follow-up | 10 | 7.2 | 22 | Joint configurations and shortlist |
| Broader validation (96 languages) | 96 | 34.5 | 7 | Test the shortlist at scale |
| Broader validation (200 languages) | 200 | 72 | 7 | Test the shortlist at scale |
| Total | – | – | 146 | 129 unique configurations; 10,842 evaluation runs |
Table A1. Stages of the study, with training coverage, per-run budget, and run counts.
We use accuracy for Belebele, SIB-200, and Taxi1500. FLORES is evaluated as a language-modeling task rather than by decoding, so we score it with bits-per-byte over the reference translation (Gao et al., 2020). Normalizing by bytes rather than tokens keeps this score comparable across languages. It measures how well a model fits translation data (Magnusson et al., 2024), so a FLORES gain is not evidence of better translation output. Lower values are better on FLORES, while higher values are better on the other three. Therefore, we invert FLORES before any comparison.
Experimental Stages
One-factor search (RQ1). We change one design choice at a time, first varying the rank while holding each method’s expert count fixed and then varying the expert count while holding the rank fixed. The balanced rank analysis uses across the six methods that cover every one of those values, and the expert-count analysis uses .
Mixed follow-up (RQ2). The first stage leaves the two best values of each factor, and . Combining them per method and removing duplicates gives 22 configurations, which we train for 7.2 hours on the same ten-language data (Appendix § 8.§ 5). Eight are new rank–expert combinations and the remaining fourteen repeat an earlier sweep setting at the longer budget, which separates the effect of combining choices from the effect of simply training longer. The five best benchmark-balanced configurations go to the broader validation, so that the broader regimes test the shortlist itself rather than a single bet on the top position.
Broader validation (RQ3–RQ4). We train each of the five promoted configurations, plus plain LoRA at and , once with 96 training languages (34.5 hours per run) and once with 200 training languages (72 hours per run), and evaluate them against the unadapted base model. Runs within a regime are matched on wall clock, hardware, data, and optimizer settings, which are the usual constraints in practice. Equal wall clock is not equal data exposure, because methods differ in throughput. Details of resource accounting are provided in Table A3. The two regimes differ in several factors at once, so we compare systems, that is, the adapted configurations and the base model, within each regime only.
| Configuration | Train. (M) | Active (M) | Steps (k) | Tokens (B) | Tok./s (k) | Corpus (%) | GPU (h) | ||||
| 96 training languages | |||||||||||
| MTL-LoRA r8/16 | 16 | dense | 8 | 16 | 188.0 | 186.1 | 20.0 | 0.98 | 8.0 | 63 | 136 |
| MOELoRA r2/16 k8 | 16 | 8 | 32 | 64 | 27.3 | 13.6 | 7.6 | 0.37 | 3.0 | 24 | 136 |
| HydraLoRA r2 | 8 | dense | 2 | 4 | 34.5 | 34.5 | 10.3 | 0.51 | 4.1 | 33 | 136 |
| MOELoRA r2/32 k8 | 32 | 8 | 64 | 128 | 54.5 | 13.6 | 6.7 | 0.33 | 2.7 | 21 | 136 |
| AdaMoE r2/8 k6 | 8 | 6 | 2 | 4 | 71.8 | 30–61 | 6.5 | 0.32 | 2.6 | 21 | 136 |
| LoRA r2 | 1 | – | 2 | 4 | 5.2 | 5.2 | 24.1 | 1.19 | 9.7 | 77 | 136 |
| LoRA r4 | 1 | – | 4 | 8 | 10.5 | 10.5 | 24.1 | 1.19 | 9.7 | 77 | 136 |
| 200 training languages | |||||||||||
| MTL-LoRA r8/16 | 16 | dense | 8 | 16 | 190.0 | 186.1 | 30.6 | 1.50 | 5.8 | 38 | 287 |
| MOELoRA r2/16 k8 | 16 | 8 | 32 | 64 | 27.3 | 13.6 | 15.9 | 0.78 | 3.0 | 20 | 288 |
| HydraLoRA r2 | 8 | dense | 2 | 4 | 34.5 | 34.5 | 21.8 | 1.07 | 4.1 | 27 | 288 |
| MOELoRA r2/32 k8 | 32 | 8 | 64 | 128 | 54.5 | 13.6 | 10.2 | 0.50 | 1.9 | 13 | 288 |
| AdaMoE r2/8 k6 | 8 | 6 | 2 | 4 | 71.8 | 30–61 | 10.4 | 0.51 | 2.0 | 13 | 288 |
| LoRA r2 | 1 | – | 2 | 4 | 5.2 | 5.2 | 48.9 | 2.40 | 9.3 | 61 | 287 |
| LoRA r4 | 1 | – | 4 | 8 | 10.5 | 10.5 | 48.7 | 2.40 | 9.3 | 60 | 287 |
Table A3. Training budget consumed by each configuration in the broader regimes.
Aggregation and Statistical Tests
We reserve for the rank of a low-rank expert and use the term average position for the ordering of systems, which runs from 1 for the best system downwards. We compare systems only on matching benchmark, task, language, and metric cases, and within each case we order the systems from best to worst and record the position of each. We then average the two FLORES directions and give each of the four benchmark families equal weight, which keeps benchmarks with more evaluation cases from dominating the comparison.
For RQ and RQ we use one Friedman test per benchmark and report Kendall’s , which measures how far the individual evaluation cases agree on an ordering, from 0 (they disagree entirely) to 1 (every case gives the same ordering), so that a low means a benchmark does not separate the systems rather than that the systems are equal. Follow-up pairwise comparisons use two-sided Wilcoxon signed-rank tests, with each stated family of tests adjusted using Holm’s method. We call a result significant when its -value, Holm-adjusted () within such a family, is below .05.
In each broader regime we run one Friedman test per benchmark over all eight systems, followed by two-sided paired Wilcoxon tests on every one of the twenty-eight system pairs, with Holm correction inside each regime and benchmark. A non-significant result means only that the available evaluation does not distinguish two systems, and never that those systems are equivalent.
Results
All comparisons are reported as average positions (§ 4.5), where position 1 is best, so lower is always better, and the value says where a system places relative to the others rather than by how much. For brevity, the complete panels and the per-comparison statistics are in Appendix § 8, as tables numbered A1 onwards, alongside all the related details.
Rank and Expert Count (RQ1)
We varied one factor at a time on the ten-language data, comparing on balanced panels, that is, restricted to the values at which all compared methods ods were actually run. Table A6 gives the average positions and Table A7 the tests behind this section.
| Setting | Belebele | FLORES | SIB-200 | Taxi1500 | Overall |
| Rank | |||||
| 2 | 2.76 | 2.00 | 2.89 | 2.31 | 2.49 |
| 4 | 2.85 | 2.37 | 3.41 | 2.62 | 2.81 |
| 8 | 2.41 | 2.36 | 3.32 | 2.79 | 2.72 |
| 16 | 3.01 | 3.85 | 3.63 | 3.46 | 3.49 |
| 32 | 4.94 | 4.83 | 3.60 | 4.60 | 4.49 |
| 64 | 5.03 | 5.58 | 4.15 | 5.22 | 5.00 |
| Expert count | |||||
| 2 | 3.16 | 3.59 | 3.40 | 3.68 | 3.46 |
| 4 | 2.80 | 2.89 | 2.67 | 3.00 | 2.84 |
| 8 | 3.76 | 2.94 | 3.27 | 3.47 | 3.36 |
| 16 | 2.81 | 3.50 | 2.19 | 2.51 | 2.75 |
| 32 | 2.47 | 2.08 | 3.48 | 2.33 | 2.59 |
Table A6. Average position of each rank and expert count on the balanced panels. Lower is better, best per column within each factor in bold.
| Benchmark | ||||
| Rank | ||||
| Belebele | 54 | 106.24 | 0.393 | *** |
| FLORES | 108 | 341.43 | 0.632 | *** |
| SIB-200 | 54 | 13.60 | 0.050 | 0.018* |
| Taxi1500 | 36 | 74.52 | 0.414 | *** |
| Expert count | ||||
| Belebele | 54 | 20.87 | 0.097 | *** |
| FLORES | 108 | 62.99 | 0.146 | *** |
| SIB-200 | 54 | 28.47 | 0.132 | *** |
| Taxi1500 | 36 | 20.54 | 0.143 | *** |
Table A7. Friedman tests for the balanced rank and expert-count panels. is the Friedman statistic and Kendall’s coefficient of concordance.
Low ranks win, and they are also the smallest. A rank of has the best overall average position (2.49), with second (2.72). Rank 2 leads on FLORES, SIB-200, and Taxi1500, whereas rank 8 leads on Belebele. Since adapter size grows with , the winning setting is also the smallest. Rank 2 trains roughly an order of magnitude fewer adapter parameters than rank 64. How much rank matters depends strongly on the benchmark, as the evaluation cases agree closely on the rank ordering on FLORES () and barely at all on SIB-200 (). The tests establish only that the ranks differ from each other, not that beats .
Expert count has a weaker and flatter effect. takes the best average position (2.59) ahead of (2.75). Agreement between evaluation cases stays between and on all four benchmarks, against a range of 0.05 to 0.63 for rank. Expert count therefore matters less than rank, and it matters about equally little on every benchmark.
MoE–LoRA Methods (RQ2)
Two comparisons answer RQ2: the first holds rank coverage fixed and compares methods (Table 1), while the second combines the values selected above and produces the shortlist from the 22 configurations of that stage (Appendix § 8.§ 5).
| Method | Bele. | FLO. | SIB | Taxi | Overall |
| MOELoRA | 2.04 | 3.32 | 2.81 | 2.56 | 2.68 |
| AdaMoE | 2.62 | 2.80 | 2.63 | 2.83 | 2.72 |
| HydraLoRA | 3.36 | 3.12 | 2.74 | 3.79 | 3.25 |
| MTL-LoRA | 3.78 | 2.97 | 4.18 | 3.75 | 3.67 |
| LoRA | 3.35 | 2.88 | 4.76 | 3.93 | 3.73 |
| HMoRA | 5.85 | 5.91 | 3.88 | 4.14 | 4.94 |
Table 1. Average position of the six methods with complete coverage of ranks 2–64 on Belebele, FLORES, SIB-200 and Taxi1500. Lower is better, best per column in bold. Coverage matrix in Table A2.
At matched ranks, MOELoRA and AdaMoE lead. Table 1 compares the six methods that were run at every rank from 2 to 64, which is the largest set of methods we can compare without letting differences in grid coverage decide the outcome. MOELoRA and AdaMoE take the best overall average positions (2.68 and 2.72) ahead of HydraLoRA (3.25), and the per-benchmark leaders differ, with MOELoRA leading on Belebele and Taxi1500 and AdaMoE on FLORES and SIB-200. MoV has no rank to vary and AdaMoLE was not run in the rank search, so this panel compares six methods rather than summarizing the full search.
After combining factors, the shortlist is tight. The longer follow-up identifies five leading configurations within a narrow band of average positions among the 22 evaluated: MTL-LoRA (7.28), MOELoRA with 16 experts (7.48), HydraLoRA (7.50), MOELoRA with 32 experts (8.18), and AdaMoE (9.24). Their performance profiles vary by task, but only FLORES distinguishes the top three, with MOELoRA outperforming both MTL-LoRA () and HydraLoRA (). The other three benchmarks show no significant differences among them. Given the narrow overall spread and limited task-level separation, we carry all five configurations forward rather than selecting only the top-ranked one.
Broader Multilingual Validation (RQ3)
Table 2 orders the five promoted configurations, the two plain-LoRA baselines, and the unadapted base model within each regime, marking each position with the outcome of that system’s direct test against the base model, which we return to in RQ4.
| System | Belebele | FLORES | SIB-200 | Taxi1500 | Overall |
| 96 training languages | |||||
| MOELoRA r2/32 k8 | 4.25 ▼ | 3.35 ▲ | 1.49 ▲ | 5.36 • | 3.61 |
| LoRA r2 | 4.08 ▼ | 4.38 ▲ | 4.19 • | 2.86 ▲ | 3.88 |
| MOELoRA r2/16 k8 | 4.28 ▼ | 4.11 ▲ | 4.14 • | 3.04 ▲ | 3.89 |
| MTL-LoRA r8/16 | 5.22 ▼ | 3.76 ▲ | 5.92 ▼ | 3.27 ▲ | 4.54 |
| Base model | 2.08 | 6.87 | 4.00 | 5.24 | 4.55 |
| HydraLoRA r2 | 4.00 ▼ | 5.28 ▲ | 3.59 • | 6.12 • | 4.75 |
| AdaMoE r2/8 k6 | 6.30 ▼ | 4.23 ▲ | 5.76 ▼ | 4.53 • | 5.21 |
| LoRA r4 | 5.80 ▼ | 4.02 ▲ | 6.91 ▼ | 5.58 • | 5.58 |
| 200 training languages | |||||
| HydraLoRA r2 | 4.35 ▼ | 2.62 ▲ | 1.64 ▲ | 5.18 • | 3.45 |
| LoRA r4 | 5.71 ▼ | 4.28 ▲ | 5.36 ▼ | 1.41 ▲ | 4.19 |
| Base model | 1.79 | 6.04 | 3.54 | 5.59 | 4.24 |
| MOELoRA r2/32 k8 | 3.89 ▼ | 4.77 ▲ | 4.01 • | 4.51 • | 4.29 |
| MTL-LoRA r8/16 | 4.77 ▼ | 5.54 ▲ | 3.44 • | 4.72 • | 4.62 |
| LoRA r2 | 4.01 ▼ | 3.14 ▲ | 7.78 ▼ | 3.91 ▲ | 4.71 |
| AdaMoE r2/8 k6 | 6.47 ▼ | 3.48 ▲ | 3.88 • | 5.98 • | 4.95 |
| MOELoRA r2/16 k8 | 5.01 ▼ | 6.14 • | 6.36 ▼ | 4.70 • | 5.55 |
Table 2. Each system within its training regime, showing its average position on every benchmark (lower is better, best per column in bold) followed by the outcome of its paired comparison against the base model, ▲ better, • not distinguishable, ▼ worse. Underlying tests in Table A12 and Table A13.
The unadapted model is difficult to outperform. The base model places fifth of eight after training on 96 languages and third of eight after training on 200. No adapted configuration is best on more than two of the four benchmarks in either regime, and none is best in both regimes. Adaptation with these methods is therefore not a general improvement over the starting model, and performance varies between tasks, which we quantify in RQ4.
Which method wins depends on the regime. HydraLoRA is fifth of the seven adapted configurations at 96 training languages and first at 200, where it is significantly better than the base model on FLORES and SIB-200. MOELoRA with 16 experts moves in the opposite direction, from third to last. MOELoRA with 32 experts leads at 96 languages. Since the regimes differ in several factors at once, we describe this as an interaction between method and regime rather than an effect of language count. Appendix § 8.§ 6 reports the omnibus test behind each benchmark column and the pairwise comparisons against the base model.
The selection stage did not predict either outcome. Table 3 follows each configuration across the three stages. The configuration ranked first after the ten-language follow-up finishes third in both broader regimes, and the two configurations that end at the extremes at 200 languages were adjacent in the middle of the shortlist. Spearman correlations between the stage orderings are (ten languages to 96), (ten languages to 200), and (96 to 200). With five configurations these values are rough indicators, but they match the tables, which show little rank-order agreement across stages. The reduced-scale search identified a band of workable configurations without identifying the ordering inside it.
| Configuration | 10 | 96 | 200 |
| MTL-LoRA r8/16 | 1 | 3 | 3 |
| MOELoRA r2/16 k8 | 2 | 2 | 5 |
| HydraLoRA r2 | 3 | 4 | 1 |
| MOELoRA r2/32 k8 | 4 | 1 | 2 |
| AdaMoE r2/8 k6 | 5 | 5 | 4 |
Table 3. Position of each promoted configuration among the five, at the selection stage and in the two broader regimes. Columns give the number of training languages.
Benefits and Regressions by Task and Language (RQ4)
The marks in Table 2 reduce the comparison against the base model to one verdict per configuration, regime, and benchmark. RQ1–RQ3 ask only which system ranks highest, which average position answers. RQ4 asks how much adaptation gives and takes, on benchmarks whose units do not compare, so we add two measures to the same tests. One is the rank-biserial effect, the share of ranked evidence for a system minus the share against it, ranging from −1 to +1. The other highlights each difference against the base model’s own level. Table A12 lists the medians, effects, and adjusted values.
| Configuration | 96 training languages: Median diff. | 96 training languages: Effect | 96 training languages: | 200 training languages: Median diff. | 200 training languages: Effect | 200 training languages: |
| Belebele | ||||||
| MTL-LoRA r8/16 | -0.0250 | -0.87 | *** | -0.0222 | -0.90 | *** |
| MOELoRA r2/16 k8 | -0.0200 | -0.85 | *** | -0.0244 | -0.98 | *** |
| HydraLoRA r2 | -0.0167 | -0.82 | *** | -0.0233 | -0.92 | *** |
| MOELoRA r2/32 k8 | -0.0228 | -0.82 | *** | -0.0178 | -0.86 | *** |
| AdaMoE r2/8 k6 | -0.0344 | -0.95 | *** | -0.0344 | -0.99 | *** |
| LoRA r2 | -0.0144 | -0.78 | *** | -0.0178 | -0.89 | *** |
| LoRA r4 | -0.0228 | -0.92 | *** | -0.0267 | -0.91 | *** |
| FLORES | ||||||
| MTL-LoRA r8/16 | +0.0058 | +0.78 | *** | +0.0023 | +0.25 | 0.027* |
| MOELoRA r2/16 k8 | +0.0059 | +0.94 | *** | +0.0006 | +0.17 | 0.18 |
| HydraLoRA r2 | +0.0047 | +0.73 | *** | +0.0064 | +0.81 | *** |
| MOELoRA r2/32 k8 | +0.0064 | +0.98 | *** | +0.0019 | +0.40 | *** |
| AdaMoE r2/8 k6 | +0.0056 | +0.79 | *** | +0.0060 | +0.72 | *** |
| LoRA r2 | +0.0052 | +0.87 | *** | +0.0060 | +0.83 | *** |
| LoRA r4 | +0.0058 | +0.84 | *** | +0.0049 | +0.67 | *** |
| SIB-200 | ||||||
| MTL-LoRA r8/16 | -0.0441 | -0.60 | 0.003** | -0.0000 | +0.06 | 1 |
| MOELoRA r2/16 k8 | -0.0000 | -0.09 | 1 | -0.0588 | -0.87 | *** |
| HydraLoRA r2 | +0.0049 | +0.01 | 1 | +0.0392 | +0.70 | *** |
| MOELoRA r2/32 k8 | +0.0343 | +0.75 | *** | -0.0147 | -0.10 | 1 |
| AdaMoE r2/8 k6 | -0.0294 | -0.52 | 0.011* | -0.0049 | -0.07 | 1 |
| LoRA r2 | -0.0147 | -0.09 | 1 | -0.1324 | -0.99 | *** |
| LoRA r4 | -0.0490 | -0.78 | *** | -0.0441 | -0.64 | *** |
| Taxi1500 | ||||||
| MTL-LoRA r8/16 | +0.0180 | +0.56 | 0.006** | +0.0090 | +0.32 | 0.092 |
| MOELoRA r2/16 k8 | +0.0270 | +0.90 | *** | +0.0090 | +0.36 | 0.066 |
| HydraLoRA r2 | -0.0000 | -0.36 | 0.288 | +0.0090 | +0.16 | 1 |
| MOELoRA r2/32 k8 | -0.0000 | +0.20 | 1 | +0.0090 | +0.28 | 0.192 |
| AdaMoE r2/8 k6 | +0.0090 | +0.23 | 1 | -0.0000 | -0.10 | 1 |
| LoRA r2 | +0.0270 | +0.72 | *** | +0.0180 | +0.63 | *** |
| LoRA r4 | -0.0000 | -0.08 | 1 | +0.0721 | +1.00 | *** |
Table A12. Every configuration against the base model in the broader regimes. Median difference and rank-biserial effect are signed so that positive favours the configuration. Median differences are in each benchmark’s own units and do not compare across benchmarks.
Adaptation is not uniformly beneficial. All fourteen configuration–regime combinations, plain LoRA included, are significantly worse than the base model on Belebele, where the median regression runs from 1.4 to 3.4 accuracy points, roughly 7 to 18% of the base model’s margin above chance. Thirteen of fourteen are significantly better on FLORES, where the median gain of 0.002 to 0.006 bits per byte is a relative reduction of 0.4 to 1.4%. The effect sizes are what can be compared across the two scales: at 96 languages those on FLORES (+0.73 to +0.98) mirror the Belebele ones (−0.78 to −0.95), so the benefit is as consistent as the regression, while at 200 languages it attenuates and becomes configuration-dependent, holding for HydraLoRA, AdaMoE, and both plain-LoRA baselines (+0.67 to +0.83) while the other three shrink towards zero (+0.17 to +0.40). SIB-200 is where the methods actually diverge, with outcomes ranging from significantly better to significantly worse depending on configuration and regime. On Taxi1500 no configuration is significantly worse than the base model, but at 200 languages only the two plain-LoRA baselines are significantly better, and none of the five routed configurations can be distinguished from it. Because each configuration is trained once, these tests describe consistency across evaluation cases and not uncertainty across training runs.
The gain tracks how weak the base model already was. In Figure 1 the languages the base model handles worst sit on the left, and vertical position is how much adaptation changes that score, so languages above the grey line improved and those below degraded. The orange line is the trend and the rank correlation behind it, marked * where .
Figure 1. Per-language gain over the base model against its own score on that language, averaged over the seven adapted configurations. Each point is one language, coloured by its resource level.
That line slopes downward in all eight panels, with between and on Belebele, FLORES, and SIB-200 in both regimes, and only weakly on Taxi1500 ( and ). Adaptation therefore redistributes capability rather than adding it uniformly, giving most to the languages the base model served worst and least, or nothing, to the languages it already served well. The three benchmarks share that trend and differ only in where it sits relative to the grey line, high enough on FLORES that nearly every language gains, crossing it on SIB-200, and below it on Belebele. The Taxi1500 slope is close to flat, which is consistent with its inability to separate systems elsewhere. Low-resource languages cluster on the left, so the coarser tier counts point the same way, with every configuration beating the base model more often on low-resource than on high-resource languages in both regimes, rising for example from 33% to 66% for AdaMoE at 96 languages (Table A14).
| Configuration | High | Medium | Low |
| 96 training languages (17/8/55 languages) | |||
| MTL-LoRA r8/16 | 40 | 40 | 70 |
| MOELoRA r2/16 k8 | 49 | 60 | 77 |
| HydraLoRA r2 | 32 | 51 | 62 |
| MOELoRA r2/32 k8 | 58 | 66 | 68 |
| AdaMoE r2/8 k6 | 33 | 40 | 66 |
| LoRA r2 | 44 | 49 | 78 |
| LoRA r4 | 33 | 40 | 61 |
| 200 training languages (23/13/100 languages) | |||
| MTL-LoRA r8/16 | 41 | 42 | 58 |
| MOELoRA r2/16 k8 | 35 | 41 | 41 |
| HydraLoRA r2 | 51 | 54 | 69 |
| MOELoRA r2/32 k8 | 36 | 27 | 61 |
| AdaMoE r2/8 k6 | 39 | 44 | 59 |
| LoRA r2 | 46 | 47 | 55 |
| LoRA r4 | 45 | 51 | 68 |
Table A14. Percentage of evaluation cases won against the base model, by the resource level of the language.
Because the base-model score appears on both axes, we also predict the gain on each benchmark from the base score on a different benchmark, where the measurement noise is independent, and the relationship survives (Table A15). Nothing here is randomized, so we read all of this as a consistent association rather than a causal effect.
Discussion and Conclusion
What transfers is the region, not the ranking. The reduced-scale search gives a usable starting point. A low rank and a larger expert count are favoured on the ten-language data, and every configuration it promoted remained trainable at scale. What it does not give is an ordering, nor any guarantee of improving on the unadapted model, which places third of eight at 200 languages. The configuration ranked first at ten languages finished third in both broader regimes, and its ordering is uncorrelated with the one at 200. A short multilingual sweep is therefore a reasonable way to find viable configurations but an unreliable way to pick a winner. It also leaves open whether the configurations it discards are worse, since, of the two discarded configurations we retrained at scale, plain LoRA at placed second at 200 languages.
Method, configuration, and regime interact. MOELoRA and AdaMoE lead the controlled shared-rank comparison, yet different configurations lead once the selected factors are combined and the budget is extended, and different ones again in each broader regime. Combining selected values does not always help either, since the best mixed descendant fails to beat its preserved parent in two of six prespecified groups. One-factor trends therefore do not reliably compose, and no single leaderboard summarizes this method family. Routing does not cause this instability, as plain LoRA configurations also move between the two regimes. Practitioners should validate in the training regime they intend to deploy, because a shortlist assembled anywhere else does not rank correctly.
The effect is clear, but why it occurs is not. Across regimes, the effect of adaptation depends more on the benchmark than on the configuration. Belebele performance declines for all fourteen configuration–regime combinations. FLORES improves strongly at 96 training languages, although the benefit weakens at 200 languages for three of the seven configurations. Because plain LoRA behaves the same way on both benchmarks, the trade-off suggests a property of low-rank adaptation of a frozen multilingual model rather than of routing. MoE–LoRA adaptation as configured here is therefore a targeted tool, not a general multilingual upgrade.
One explanation is that continued pretraining benefits the two benchmarks differently. FLORES measures how well the model fits reference translations, so the multilingual text modelling learned during CPT can transfer to it. In contrast, CPT does not directly train the in-context discrimination required by Belebele. Adaptation may therefore improve language-model fit while weakening behaviour already present in the base model. Such forgetting has been documented when pretrained models are extended to new languages (Zhou et al., 2025a; Li et al., 2025), while low-rank adapters must balance learning new information against preserving existing capabilities [9]. Testing this would require a deeper look at sample-level performance, which we leave to future work. A short replay stage using data from the base model’s original distribution as suggested by Zhou et al. (2025a) might help recover original-language ability but still would not explain why adaptation trades comprehension for language-model fit.
Limitations
Training all 166 configurations of the eight methods would cost more than 70,000 A100-hours across both broader scales. We therefore vary one design choice at a time and retrain only a shortlist, as is common practice (Team et al., 2022; Üstün et al., 2024; Zadouri et al., 2024; Gao et al., 2025). Apart from the plain-LoRA baselines, discarded configurations were never retrained at scale, so stronger configurations may lie outside the shortlist. Each configuration is also trained once, and close rankings may change on retraining. Runs are matched on wall clock rather than on tokens, so faster low-rank configurations also see more data (Table A3). Benchmark coverage differs across stages and reaches only 136 of the 200 training languages. All results also come from a single base model and may not hold for others.
Future work can introduce seeds, change the base model, evaluate generated translations with metrics such as BLEU or COMET, or move selection closer to the target regime. We release the configurations, per-language results, and analysis code so that future work need not repeat the search.
Ethical Considerations
This study involves no human subjects, annotation, or new data collection. The training text is derived from the web, and we apply no additional screening for personal or offensive content beyond the procedures used for the source corpora (Penedo et al., 2024, 2025). Residual personal information and harmful content may therefore remain, although we do not redistribute any training text. Appendix § 8.7 documents the licenses and intended uses of the artifacts used and released in this study. In total, the 146 training runs consumed approximately 5,150 A100-hours.
Acknowledgments
This work has been funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – TRR 318/3 2026 – 438445824, by the Federal Ministry for Economic Affairs and Energy (BMWE) on the basis of a decision by the German Bundestag under the project ikDS (FKZ KK5175206LO4), and by the German Federal Ministry of Research, Technology and Space (BMFTR) within the project KI-Akademie OWL (16IS24057B).
The authors gratefully acknowledge the computing time made available to them on the high-performance computer Noctua 2 [7] at the NHR Center Paderborn Center for Parallel Computing (PC2). This center is jointly supported by the Federal Ministry of Research, Technology and Space and the state governments participating in the National High-Performance Computing (NHR) joint funding program (www.nhr-verein.de/en/our-partners).
Generative AI tools (ChatGPT, Claude, Gemma, Qwen) were used for language refinement and code assistance. All scientific contributions and conclusions are solely those of the authors, who reviewed and take full responsibility for the final manuscript.
Experimental Details
In the tables below, is the number of paired evaluation cases behind a test, and stars mark the -value in the same column, * for , ** for , *** for .
Language Selection and Resource Labels
Two released notebooks document how the training languages were chosen and how each language was labelled by resource level.
Resource labels. Resource labels are generated by data_sample/notebook/fineweb2_analysis_and_subsampling.ipynb, which builds the language table used throughout the paper. It starts from the FineWeb2 language distribution, drops the subsets marked as removed, and joins an external resource-category list from Chang et al. [12]. Its original categories establish document-count boundaries, which are merged into the three levels we report, high, medium, and low, and applied to every retained FineWeb2 subset after sorting by document count.
Language selection. The language selection is generated by data_sample/notebook/lang2vec.ipynb, which chooses the languages we train on. It reads the labelled table, keeps the languages with more than a thousand documents in FineWeb2, and intersects them with the languages for which lang2vec provides a genetic distance (Littell et al., 2017). From that pool it applies greedy farthest-point selection, repeatedly adding the language furthest from those already picked, so that each smaller set is contained in the next larger one. It also reports how much of Belebele and FLORES each set covers.
Method and Grid Coverage
Table A1 summarizes the five stages of the study and their run counts. Table A2 enumerates the grid itself, so that the 47 and 63 runs of the two one-factor searches can be reconstructed method by method. In it, is the rank of one expert, the number of experts, and the number of experts routed to per token, which methods without a router do not define. The rank search holds and fixed, and the expert-count search holds . The Panel column marks the six methods with complete rank coverage, which are the ones compared in Table 1. The grid reports observed coverage and does not imply that every method was evaluated on the same grid. Method-specific configuration files, run identifiers, and raw benchmark rows accompany the released analysis.
| Method | Rank search | Runs | Expert-count search (: top-) | Runs | Panel |
| AdaMoE | at , | 7 | 2: 1; 4: 1, 2; 8: 1, 2, 4; 16: 1, 2, 4, 8; 32: 1, 2, 4, 8, 16 | 15 | yes |
| AdaMoLE | – | – | 2, 4, 8, 16 | 4 | no |
| HMoRA | at , | 7 | 2: 1; 4: 1, 2; 8: 1, 2, 4; 16: 1, 2, 4, 8; 32: 1, 2, 4, 8, 16 | 15 | yes |
| HydraLoRA | 8 | 2, 4, 8, 16, 32 | 5 | yes | |
| LoRA | 10 | – | – | yes | |
| MOELoRA | at , | 6 | 2: 1; 4: 1, 2; 8: 1, 2, 4; 16: 1, 2, 4, 8; 32: 1, 2, 4, 8 | 14 | yes |
| MoV | no rank at | 1 | 2, 4, 8, 16, 32 | 5 | no |
| MTL-LoRA | 8 | 2, 4, 8, 16, 32 | 5 | yes | |
| Total | 47 | 63 |
Table A2. Every configuration trained in the two one-factor searches.
What the searches leave out. Four kinds of gap produce the uneven grids. First, some methods do not define a factor. Plain LoRA trains one adapter and so has no expert count, which is why it appears in the rank search only and why its two best ranks there, rather than a rank–expert pair, are what carries into the broader regimes; and MoV scales activation vectors rather than low-rank matrices and so has no rank, which is why it contributes a single run to the rank search. Second, some routing settings are degenerate. The expert-count search varies top- over the powers of two up to , because routes to every expert and removes the routing being studied, and MoV is trained dense for the same reason. Third, some configurations did not fit the hardware. AdaMoLE at 32 experts exhausted memory on four A100 40GB GPUs at a per-device batch size of one, under both the all-target and the attention-only adapter placement, and MOELoRA at 32 experts with top- 16 was likewise infeasible, which is why those two cells are missing. Fourth, one method’s grid did not train reliably. AdaMoLE was not run in the rank search, because its higher-rank configurations did not train stably, so it is excluded from the shared rank panel.
CoLA. We also implemented and trained CoLA (Zhou et al., 2025b), with eight runs in the rank search and five in the expert-count search. Those runs failed early, and we excluded the method rather than report it on unequal terms. CoLA is therefore absent from every count, table, and test reported here, and the 132 search runs of § 4.4 are the 145 trained search runs minus these 13. The released workbooks retain the CoLA rows.
Evaluation coverage is smaller than training coverage at every stage, so we state the two separately. The ten-language stages evaluate nine languages, giving 42 task and language cases per model. The broader regimes evaluate 80 languages (249 cases) after training on 96, and 136 languages (408 cases) after training on 200.
Training Budget and Language Mixture
Table A3 reports the training budget of every configuration in both broader regimes. is the expert count and the number of experts routed per token, or dense where a method combines all of them.
Steps counts optimizer steps, Tokens the input token positions processed, Tok./s the throughput, Corpus the share of the training set seen once, and GPU the hours on the run’s four A100 40GB GPUs. Every run in a regime received the same wall clock, so the spread in steps and tokens is the throughput difference between methods. That spread is large. At 200 training languages the configurations process between 0.50 and 2.40 billion token positions, or between 13% and 61% of one pass over the training set. We report several indicators because they do not agree with each other [14]. The configuration with the fewest trainable parameters sees the most data, and the configuration with the most trainable parameters is the fastest of the routed ones.
The same table gives the rank and scaling hyperparameter of every configuration, as the implementation counts them. Across all 140 training runs with a low-rank update, is twice that rank. MOELoRA counts the total rank budget, so its configurations carry and although the paper reports the per-expert rank of 2. Every implementation except HMoRA scales the update by , which is therefore 2 in every run. For these methods the rank sweep does not change update magnitude. HMoRA specifically defines its own ratio of to , scaling the update by , and we use that definition instead. With its factor is , from 2.8 at to 22.6 at , so its seven rank-sweep runs vary rank and update magnitude together. Its expert-count runs all use and share one factor. HMoRA is also not among the configurations carried to the broader regimes. MoV scales vectors rather than low-rank matrices and has no . The rank, , and applied scaling factor of every run are released with the run metadata.
The Train. and Active columns separate trainable adapter parameters from those reached on a forward pass, since an expert that is never selected for a token contributes to the first but not the second. The two MOELoRA configurations activate 13.6M of the 27.3M and 54.5M they train, because they route to 8 of 16 and 8 of 32 experts. HydraLoRA and MTL-LoRA combine every up-projection with dense weights, so their two counts nearly coincide, as they do exactly for plain LoRA. AdaMoE selects a variable number of experts per token through its null experts, so its count is given as a range rather than a point value. Counts are derived from the saved checkpoint shapes together with each run’s routing settings, and they are parameter counts rather than FLOP estimates. The frozen backbone contributes a further 8.03B parameters to every configuration and is excluded from both columns.
Table A4 gives the composition of each training set. Labels counts language–script labels, which exceed the nominal language count because a language can appear in more than one script, and Ex. counts packed training sequences of 1,024 tokens. Top-1 and Top-5 give the share of sequences held by the largest labels, and the 50% and 90% columns the number of labels needed to reach that share. Construction draws the same fraction of every language’s available documents, 0.0025 at 96 training languages and 0.005 at 200, so the mixture reproduces FineWeb2’s own distribution without upsampling low-resource languages. At 200 training languages five language–script labels account for half of the training sequences and 26 account for nine tenths, while the smallest label contributes
about one sequence in a million. English is the one deliberate exception, set to the rounded median document target of the selected non-English entries rather than to its corpus share, which leaves it with 31 training documents at 200 languages. These are shares of the prepared split rather than measured per-run consumption, so they describe what each run could draw on and not what it saw. The broader-regime results hold for this mixture, which is one reason not to read them as the best achievable at 200 languages. Per-language document, sequence, and token counts are released with the analysis.
Complete RQ1 Results
Table A6 gives the average position of every rank and every expert count on the balanced panels, blocking by method so that each method contributes its own ordering. Table A5 states the two values of each factor that this analysis carried into the mixed follow-up, and Table A7 reports the corresponding Friedman tests, with Holm correction applied over the four benchmarks separately for each factor. These tests are exploratory, because the same observations were used to select the values carried to the follow-up.
| Factor | Selection analysis | Follow-up values |
| Rank | 2 (next: 8) | 2, 8 |
| Expert count | 32 (next: 16) | 32, 16 |
Table A5. Design-factor values carried into the mixed follow-up.
Complete RQ2 Results
The 22 follow-up configurations are built as follows. We take the three best concrete configurations at each of the four selected factor values, then pair each method’s selected rank with every expert count and routing setting that the same method contributed to the expert-count selection, and remove duplicates. Table A9 orders all 22 of them, and Table A8 repeats the five that were promoted. Table A11 reports the omnibus tests over all 22 configurations together with two-sided Wilcoxon tests among the promoted top three, with Holm correction applied over the four omnibus tests and, separately, over all twelve post-hoc tests. That family was fixed after the ranking was known, so it is less conservative than the all-pairs correction used in the broader regimes. Table A10 isolates the effect of combining a selected rank with a selected expert count. It selects the best observed child within each prespecified parent group and is therefore descriptive. It should not be read as an unbiased estimate of the benefit of mixing design choices.
| Method | Rank | Experts | Top- | Belebele | FLORES | SIB-200 | Taxi1500 | Overall |
| MTL-LoRA | 8 | 16 | – | 7.11 | 10.28 | 4.33 | 7.42 | 7.28 |
| MOELoRA | 2 | 16 | 8 | 6.44 | 4.94 | 11.61 | 6.92 | 7.48 |
| HydraLoRA | 2 | – | – | 10.83 | 9.89 | 4.11 | 5.17 | 7.50 |
| MOELoRA | 2 | 32 | 8 | 4.17 | 9.22 | 14.50 | 4.83 | 8.18 |
| AdaMoE | 2 | 8 | 6 | 10.83 | 4.33 | 8.94 | 12.83 | 9.24 |
Table A8. The five configurations carried to the broader validation, with their average positions among all 22 follow-up configurations. Lower is better, best per column in bold.
| # | Run ID | Method | Rank | Experts | Top- | Belebele | FLORES | SIB-200 | Taxi1500 | Overall |
| 1 | mtllora_r8_nup16 | MTL-LoRA | 8 | 16 | – | 7.11 | 10.28 | 4.33 | 7.42 | 7.28 |
| 2 | moelora_tr32_e16_k8 | MOELoRA | 2 | 16 | 8 | 6.44 | 4.94 | 11.61 | 6.92 | 7.48 |
| 3 | hydralora_r2 | HydraLoRA | 2 | – | – | 10.83 | 9.89 | 4.11 | 5.17 | 7.50 |
| 4 | moelora_tr64_e32_k8 | MOELoRA | 2 | 32 | 8 | 4.17 | 9.22 | 14.50 | 4.83 | 8.18 |
| 5 | mola_tr16_e8_k6 | AdaMoE | 2 | 8 | 6 | 10.83 | 4.33 | 8.94 | 12.83 | 9.24 |
| 6 | moelora_tr64_e32_k4 | MOELoRA | 2 | 32 | 4 | 9.00 | 8.33 | 11.28 | 8.92 | 9.38 |
| 7 | moelora_tr256_e32_k8 | MOELoRA | 8 | 32 | 8 | 7.06 | 4.06 | 16.00 | 13.58 | 10.17 |
| 8 | mtllora_r8 | MTL-LoRA | 8 | – | – | 8.17 | 19.83 | 6.22 | 8.58 | 10.70 |
| 9 | moelora_tr128_e16_k8 | MOELoRA | 8 | 16 | 8 | 8.22 | 9.56 | 14.11 | 14.08 | 11.49 |
| 10 | mtllora_r8_nup32 | MTL-LoRA | 8 | 32 | – | 12.00 | 12.00 | 8.44 | 13.58 | 11.51 |
| 11 | mola_tr64_e32_k4 | AdaMoE | 2 | 32 | 4 | 15.00 | 9.56 | 14.89 | 7.08 | 11.63 |
| 12 | hydralora_r2_heads32 | HydraLoRA | 2 | 32 | – | 9.61 | 11.50 | 9.50 | 16.25 | 11.72 |
| 13 | hydralora_r2_heads16 | HydraLoRA | 2 | 16 | – | 16.83 | 10.67 | 7.94 | 11.67 | 11.78 |
| 14 | mola_tr256_e32_k4 | AdaMoE | 8 | 32 | 4 | 17.56 | 12.39 | 11.06 | 7.08 | 12.02 |
| 15 | moelora_tr256_e32_k4 | MOELoRA | 8 | 32 | 4 | 9.11 | 12.56 | 13.83 | 13.75 | 12.31 |
| 16 | moelora_tr64_e8_k2 | MOELoRA | 8 | 8 | 2 | 18.44 | 6.28 | 21.67 | 3.50 | 12.47 |
| 17 | mola_tr128_e16_k1 | AdaMoE | 8 | 16 | 1 | 13.44 | 20.94 | 4.44 | 12.67 | 12.87 |
| 18 | mola_tr32_e16_k1 | AdaMoE | 2 | 16 | 1 | 13.72 | 17.22 | 10.72 | 12.17 | 13.46 |
| 19 | moelora_tr16_e8_k2 | MOELoRA | 2 | 8 | 2 | 7.89 | 14.39 | 21.33 | 11.58 | 13.80 |
| 20 | mola_tr32_e16_k8 | AdaMoE | 2 | 16 | 8 | 15.78 | 12.78 | 7.28 | 20.00 | 13.96 |
| 21 | mola_tr128_e16_k8 | AdaMoE | 8 | 16 | 8 | 16.00 | 15.44 | 12.22 | 20.67 | 16.08 |
| 22 | mola_tr64_e8_k6 | AdaMoE | 8 | 8 | 6 | 15.78 | 16.83 | 18.56 | 20.67 | 17.96 |
Table A9. Average position of every mixed follow-up configuration. Lower is better, and bold marks the five configurations carried to the broader validation.
| Benchmark | Kendall’s | Omnibus | MTL-LoRA – MOELoRA | MTL-LoRA – HydraLoRA | MOELoRA – HydraLoRA | |
| Belebele | 9 | 0.410 | *** | |||
| FLORES | 18 | 0.511 | *** | * | ** | |
| SIB-200 | 9 | 0.612 | *** | |||
| Taxi1500 | 6 | 0.604 | *** |
Table A10. Effect of combining the selected rank and expert-count values. Cells are the parent’s average position minus the descendant’s, so positive values favour mixing and bold marks a positive overall effect.
Complete Broader-Regime Tests
Table A12 reports the seven comparisons against the base model for each benchmark within each broader regime, and Table A13 the corresponding omnibus tests, with Holm correction over the four benchmarks within each regime. For the pairwise tests, Holm correction covers all twenty-eight system pairs inside each regime and benchmark, not only the pairs shown here, and the remaining pairs are released with the analysis code. Correcting over every pair rather than over a shortlist keeps the size of the correction family independent of which systems come out on top, which makes these comparisons more conservative than a pre-planned finalist-versus-base design would have been.
| Benchmark | ||||
| 96 training languages | ||||
| Belebele | 40 | 79.70 | 0.285 | *** |
| FLORES | 100 | 142.71 | 0.204 | *** |
| SIB-200 | 51 | 170.87 | 0.479 | *** |
| Taxi1500 | 58 | 118.12 | 0.291 | *** |
| 200 training languages | ||||
| Belebele | 65 | 148.73 | 0.327 | *** |
| FLORES | 166 | 350.68 | 0.302 | *** |
| SIB-200 | 84 | 363.88 | 0.619 | *** |
| Taxi1500 | 93 | 225.34 | 0.346 | *** |
Table A13. Friedman tests over the eight systems in each broader regime. is the Friedman statistic and Kendall’s coefficient of concordance. Holm correction is applied over the four benchmarks within each regime.
Table A14 reports how often each configuration beats the base model, split by the resource level of the language, which follows the assignment described in Appendix § 8.§ 1. Low-resource languages dominate the evaluation population. Table A15 is the independent-noise check for Figure 1. The base-model score appears on both axes of that figure, so a negative relationship could in principle be produced by measurement noise alone. Taking the base score from a different benchmark than the gain removes that path, and the negative relationship holds everywhere except Taxi1500.
Released Code and Notebooks
The complete artifact is available at https://github.com/dice-group/moe-study/tree/master. It contains the training and evaluation pipeline behind all 146 training runs and 10,842 evaluation runs, together with the per-method configuration files and run identifiers referred to throughout this appendix. A paper_artifact/ directory holds the analysis layer, which comprises the result workbooks, the scripts that turn them into every table and figure of this paper, and the per-question notebooks listed below under paper_artifact/notebooks/.
Configuration names and run identifiers. The paper names a configuration by its method, per-expert rank, expert count, and top-, while the released runs keep the identifiers the pipeline generated, which encode the total rank budget and carry two historical prefixes. Table A16 maps between them for the configurations carried to the broader regimes. In an identifier, tr is the total rank budget, which the paper divides by the expert count e to give the per-expert rank r, so tr32_e16 is r=2 with . k is the top-, and nup is the number of up-projections that MTL-LoRA calls an expert count. The mola prefix marks the AdaMoE implementation, whose runs appear as adamoe_1_* in the expert-count sweep. The same rule applies to every identifier in Table A9.
Per-question notebooks. Each notebook reads the released workbooks and reproduces, in reporting order, the tables and figures for one research question.
00_study_overview.ipynbcovers the study funnel, method coverage, and configuration grid (Appendix § 8.§ 2).
Table A11: Omnibus and pairwise tests for the mixed follow-up. The omnibus test covers all 22 configurations, while the pairwise columns cover the top three of the five promoted, which is every pairing of those three. Arrows point toward the descriptively better configuration.
| Gain on | Belebele | FLORES | SIB-200 | Taxi1500 |
| 96 training languages | ||||
| Belebele | – | -0.52*** | -0.53*** | -0.67*** |
| FLORES | -0.58*** | – | -0.48*** | -0.56** |
| SIB-200 | -0.69*** | -0.76*** | – | -0.50** |
| Taxi1500 | +0.01 | -0.03 | +0.10 | – |
| 200 training languages | ||||
| Belebele | – | -0.56*** | -0.58*** | -0.61*** |
| FLORES | -0.46*** | – | -0.29** | -0.41** |
| SIB-200 | -0.67*** | -0.69*** | – | -0.47** |
| Taxi1500 | -0.18 | -0.25 | -0.04 | – |
Table 15. Spearman correlation between the gain on one benchmark and the base-model score on another. The diagonal is empty because it would take both values from the same benchmark.
| Name in this paper | Released run identifier |
| MTL-LoRA r8/16 | mtllora_lr5e5_r8_nup16 |
| MOELoRA r2/16 k8 | moelora_tr32_e16_k8 |
| HydraLoRA r2 | hydralora_r2 |
| MOELoRA r2/32 k8 | moelora_tr64_e32_k8 |
| AdaMoE r2/8 k6 | mola_tr16_e8_k6 |
| LoRA r2 | lora_r2 |
| LoRA r4 | lora_r4 |
| Base model | baseline::l-3-1-8b |
Table 16. Configuration names in this paper and the released run identifiers.
02_rq2_method_comparison.ipynbcovers the shared-panel method comparison, the 22 follow-up configurations, and their statistical tests (§ 5.2, Appendix § 8.§ 5).03_rq3_broader_regimes.ipynbcovers the 96- and 200-language regimes, the selection-transfer comparison, and the complete pairwise tests summarized in Appendix § 8.§ 6 (§ 5.3).04_rq4_localization.ipynbcovers gains and regressions by task, translation direction, and resource level (§ 5.4).
Licensing and intended use. Every artifact we build on is used for non-commercial research, which is the use its terms provide for. The base model is used under the Llama 3.1 Community License, the training text under the Open Data Commons Attribution License that covers FineWeb and FineWeb2, and Belebele, FLORES-200, and SIB-200 under CC BY-SA 4.0. Taxi1500 is used under the terms its authors distribute it with, which restrict it to research use. The evaluation harness is used under its MIT license. We redistribute none of this material. What we release is our own configurations, run records, per-language scores, and analysis code, under CC BY 4.0 for the results and the MIT license for the code, so that the terms above continue to govern the underlying artifacts.
Reproducing the reported numbers. paper_artifact/scripts/run_all.py regenerates every table and figure from the workbooks in paper_artifact/input/, writing the LaTeX sources and their underlying CSV files to paper_artifact/output/. The pinned environment is given in paper_artifact/env/environment.yml. The workbooks retain the rows excluded from the analysis, including the CoLA runs and the unbalanced grid cells, so the filtering decisions described above can be inspected rather than taken on trust.
References
- [1]David Ifeoluwa Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vasilyev, Jesuojoba O. Alabi, Yanke Mao, Haonan Gao, and En-Shiun Annie Lee. 2024. SIB-200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 226–245, St. Julian’s, Malta. Association for Computational Linguistics.
- [2]Ahmed Agiza, Marina Naseem, and Sherief Reda. 2024. Mtlora: Low-rank adaptation approach for efficient multi-task learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16196–16205.arxiv.org/abs/2403.20320
- [3]Roee Aharoni, Melvin Johnson, and Orhan Fırat. 2019. Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3874–3884, Minneapolis, Minnesota. Association for Computational Linguistics.arxiv.org/abs/1903.00089
- [4]Akari Asai, Sneha Kudugunta, Xinyan Yu, Terra Blevins, Hila Gonen, Machel Reid, Yulia Tsvetkov, Sebastian Ruder, and Hannah Hajishirzi. 2024. BUFFET: Benchmarking large language models for few-shot cross-lingual transfer. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1771–1800, Mexico City, Mexico. Association for Computational Linguistics.arxiv.org/abs/2305.14857
- [5]Raphaël Bagat, Irina Illina, and Emmanuel Vincent. 2025. Mixture of LoRA Experts for Low-Resourced Multi-Accent Automatic Speech Recognition. In Interspeech 2025, pages 1143–1147.DOI
- [6]Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artexte, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2024. The bebelele benchmark: a parallel reading comprehension dataset in 122 language variants. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 749–775, Bangkok, Thailand. Association for Computational Linguistics.
- [7]Carsten Bauer, Tobias Kenter, Michael Lass, Lukas Mazur, Marius Meyer, Holger Nitsche, Heinrich Riebler, Robert Schade, Michael Schwarz, Nils Winnwa, Alex Wiens, Xin Wu, Christian Plessl, and Jens Simon. 2024. Noctua 2 supercomputer. Journal of large-scale research facilities JLSRF, 9.
- [8]Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137–1155.
- [9]Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, Cody Blakeney, and John Patrick Cunningham. 2024. LoRA learns less and forgets less. Transactions on Machine Learning Research.
- [10]Eric L Buehler and Markus J Buehler. 2024. X-lora: Mixture of low-rank adapter experts, a flexible framework for large language models with applications in protein mechanics and molecular design. APL Machine Learning, 2(2):026119.arxiv.org/abs/2402.07148
- [11]Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. 2025. A survey on mixture of experts in large language models. IEEE Transactions on Knowledge and Data Engineering, 37(7):3896–3915.DOI
- [12]Tyler A. Chang, Catherine Arnett, Zhuowen Tu, and Benjamin K. Bergen. 2024. When is multilinguality a curse? language modeling for 250 high- and low-resource languages. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 4074–4096, Miami, Florida, USA. Association for Computational Linguistics.
- [13]Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451, Online. Association for Computational Linguistics.arxiv.org/abs/1911.02116
- [14]Mostafa Dehghani, Anurag Arnab, Lucas Beyer, Ashish Vaswani, and Yi Tay. 2022. The efficiency misnomer. In International Conference on Learning Representations.arxiv.org/abs/2110.12894
