betaFeedback

WorksComputer Science

Do Frontier Model Ingredients Add Up? A Replicated Factorial Study of Small Language Models

Abstract

Building a small language model requires choosing which published improvements to combine. We compare five changes in the same model, using the same data and training budget. We test selected combinations and repeat each one with a second random seed. Mixture-of-experts layers, Muon optimisation and a Gated DeltaNet hybrid give the largest gains. Extra normalisation also helps, while an attention gate with a matched parameter budget adds little on average. The gains mostly combine, but small interactions between changes appear again in the repeated runs. Repetition also reveals benefits that an initial statistical estimate treated as noise. These findings apply to our small-model training setting; the expert models also have greater total capacity. The practical lesson is to test promising components together and repeat the comparisons before deciding which ones to keep.

Disclaimer: This paper was prepared for informational purposes by the LLM Suite group of JP Morgan Chase and its affiliates (‘JPMC’) and is not a product of the Research Department of JP Morgan. JP Morgan makes no representation, warranty or undertaking whatsoever and disclaims all liability for the completeness, accuracy or reliability of the information contained herein. This document is not intended as investment research or investment advice, or a recommendation, offer or solicitation for the purchase or sale of any security, financial instrument, financial product or service, or to be used in any way for evaluating the merits of participating in any transaction, and shall not constitute a solicitation under any jurisdiction to any person, if such solicitation under such jurisdiction or to such person would be unlawful.

Introduction

Published improvements to language models are usually measured against different baselines. A builder must decide whether they will still help when used together. Two changes may provide the same benefit, or one may change how well the other works. We call the latter an interaction.

We compare five changes in one implementation. G adds a sigmoid attention-output gate [18]; M uses Muon to update hidden weight matrices [10, 15]; S normalises both the input and output of each sub-layer [5, 13]; E adds small routed experts [3, 23, 4]; and H replaces six attention layers with Gated DeltaNet [26, 25].

Testing every combination requires 32 configurations before any repeats. We use a classical fractional factorial design: a balanced selection of 16 combinations that allows several comparisons to share the same runs [7, 2]. We then repeat each configuration with a second seed. A contrast is a difference between two group averages; for a single switch, it compares average loss with the switch on and off.

The largest gains mostly combine, while smaller interactions also repeat. The design has an important limit: some effects are indistinguishable in the selected comparisons. This is called aliasing. Repeating the runs measures variability but does not remove that ambiguity. We report these limits alongside the results so that the study can guide component selection at the tested scale.

Geiping and Goldstein [8] study how to train masked language models within a single-GPU-day budget. Dodge et al. [6] show why comparisons should account for the compute spent developing a model. Wen et al. [24] show how optimiser rankings depend on tuning and the final training budget. Our study compares several choices together and repeats every selected combination.

Factorial language-model studies also predate ours. ATMA selects a long-context recipe with a complete 120-cell sweep of attention, memory, and training choices [1]. Our contribution is a repeated comparison of five different types of change in a small decoder. Both the components and the experimental-design method are established.

Model, Switches, and Training

Common protocol

The baseline is an eight-layer decoder of width 512 with eight 64-dimensional heads, SwiGLU feed-forward width 1,408, pre-RMSNorm, query/key normalisation, rotary positions, and tied embeddings. Its vocabulary has 50,304 entries and it contains 51,455,488 parameters. These are standard Transformer choices [22, 20, 27, 9] (Su et al. 2024).

We train on FineWeb [17], using GPT-2 byte-pair tokenisation and the NanoGPT-speedrun shard format [19, 11]. Sequences contain 1,024 tokens. Four micro-batches of 16 sequences give 65,536 tokens per step; 4,577 steps reach the nominal 300M-token budget. Final validation measures next-token cross-entropy over 10,485,760 held-out tokens. Seeds change initial weights and the contiguous range of training shards. Configurations with the same seed receive identical batches, checked by first-batch hashes. Loss is in nats per token; lower is better.

SwitchChange from the baseline
GAttention-output sigmoid gate; narrower feed-forward layers to keep active size similar.
MMuon for hidden matrices; AdamW for other groups, with the chosen family settings.
SEmbedding RMSNorm and output RMSNorm before each sub-layer joins the residual stream.
ELayers 2–8: 32 routed experts, top-3 selection, one shared expert; width 352 each.
HGated DeltaNet in layers 1–3 and 5–7; global attention in layers 4 and 8.

Table 1. The five switches, each of which can be on or off. They share one training implementation. Active parameter counts are approximately matched; total counts are not.

The learning rate rises for 300 steps, remains constant, then falls linearly over the last 20% of training. AdamW (Loshchilov and Hutter [16]) uses momentum coefficients (0.9, 0.95) and gradient clipping at 1.0. Preliminary sweeps select one peak rate per optimiser family: 0.002 for AdamW and 0.016 for Muon. For the relevant matrix groups, decay preserves the baseline product ηλ=2×10−4\eta\lambda= 2 \times10^{-4} of peak rate and weight decay. Muon updates hidden matrices with five Newton–Schulz iterations, momentum 0.95, and the Moonlight update scale. Embeddings, norms, routers, gates, and convolution kernels retain AdamW. Thus M compares complete optimiser recipes, including rates and parameter groups. We do not tune each architecture separately.

What each switch changes

Table 1 defines the switches. G adds a 512×512 projection to each attention layer. We pay for it by reducing feed-forward width from 1,408 to 1,240 and active expert width from 352 to 309. G therefore tests whether the gate is a useful way to spend the parameter budget.

For E, a sigmoid score and a balancing bias select the top three routed experts. The bias receives no gradient. Output weights use the original scores, normalised over the selected experts and multiplied by 3\sqrt{3}; the shared expert is always active. A sign update increases the selection bias of underused experts. Its step size is 10−210^{-2} for 1,000 steps and 10−310^{-3} thereafter. A sequence-wise balance term has coefficient 10−410^{-4}. The model has about 161.3M total and 51.6M active parameters. Its gain can therefore reflect both routing and roughly three times the total capacity.

For H, unit-normalised keys drive the gated delta update,

St=αt(I−βtktktT)St−1+βtktvtT,(1)S_t = \alpha_t(I-\beta_t k_t k_t^{\mathsf{T}})S_{t-1}+\beta_t k_t v_t^{\mathsf{T}}, \tag*{(1)}

where αt,βt∈(0,1)\alpha_t,\beta_t \in(0,1) and each head stores a 64 × 64 state (Yang, Kautz, and Hatamizadeh [25]). Causal width-four convolutions filter the projected queries, keys and values before the memory update. The two remaining attention layers use no positional encoding (NoPE) (Kazemnejad et al. [12]). H tests this whole hybrid recipe.

Experimental Design and Analysis

Choosing the combinations

Code each switch as xf∈{−1,+1}x_f \in\{-1,+1\}. We run the 16 configurations satisfying

xGxMxSxExH=+1,I=GMSEH.(2)x_Gx_Mx_Sx_Ex_H = +1,\qquad I = GMSEH. \tag*{(2)}

These comprise five single-switch configurations, ten three-switch configurations, and the all-five configuration. The all-off baseline is outside this selection. We use it only for comparisons labelled as standalone. Two seeds per configuration give 32 core training runs, or about 9.6B training tokens. This count excludes preliminary sweeps, diagnostics, and the extra hybrid repeats used below.

Let ycry_{cr} be final validation loss for configuration cc and seed rr. We describe it using a model with single-switch and pairwise terms:

ycr=β0+∑fβfxfc+∑f<gβfgxfcxgc+εcr.(3)y_{cr} = \beta_0 + \sum_f \beta_f x_{fc} + \sum_{f<g}\beta_{fg}x_{fc}x_{gc}+\varepsilon_{cr}. \tag*{(3)}

There are 16 coefficients and 16 configurations. The design balances the comparison columns so that they are orthogonal. For a single-switch or pair-product column zAcz_{Ac}, the contrast and its two-seed mean are

e^A(r)=216∑c=116zAcycr,e‾A=e‾A(0)+e‾A(1)2.(4)\widehat{e}^{(r)}_A = \frac{2}{16}\sum_{c=1}^{16} z_{Ac}y_{cr},\qquad\overline{e}_A = \frac{\overline{e}^{(0)}_A+\overline{e}^{(1)}_A}{2}. \tag*{(4)}

Negative single-switch contrasts favour turning that switch on.

This is a resolution-V design (Box and Hunter [2]): single-switch and pairwise effects can be separated from each other, but some higher-order effects remain mixed in. For example, G≡MSEHG \equiv MSEH and GM≡SEHGM \equiv SEH. The measured G contrast includes the four-switch effect MSEH; the measured GM contrast includes the three-switch effect SEH. Interpreting them as G or GM alone requires those higher-order effects to be small. Replication cannot separate them. A small measured contrast could also hide two large contributions with opposite signs.

Measuring variation between runs

With one seed, Equation 3 fits as many coefficients as there are observations. It leaves no residual estimate of run noise. We initially use Lenth’s pseudo-standard error (PSE), which estimates noise from the spread of effects under the assumption that most are inactive [14]. It gives PSE 0.0102 nats. This estimate comes from the effects themselves, rather than repeated runs.

For the repeated-run estimate, let sH2s_H^2 be the sample variance of five completed H-only runs. Let δc=yc1−yc0\delta_c = y_{c1} - y_{c0} for the other 15 configurations. Combining these gives

σ^2=4sH2+12∑c≠Hδc219,σ^=0.0036.(5)\widehat{\sigma}^2 = \frac{4s_H^2 + \frac{1}{2}\sum_{c \ne H}\delta_c^2}{19}, \qquad\widehat{\sigma} = 0.0036. \tag*{(5)}

H’s first pair is counted only once. If run errors are independent and have equal variance, the standard error of a two-seed mean contrast is

SE⁡(e‾A)=σ^22≃0.0013.(6)\operatorname{SE}(\overline{e}_A) = \frac{\widehat{\sigma}}{2\sqrt{2}} \simeq0.0013. \tag*{(6)}

With 19 nominal degrees of freedom and a Šidák adjustment for 15 comparisons, this gives a simultaneous 95% reference margin of ±0.0043 nats. This is a guide under the stated assumptions. Errors may move together because configurations share data streams, and two seeds cannot estimate their full covariance or establish equal variances. We report both seeds so readers can also judge agreement directly.

Results

The main gains repeat

Experts, Muon, and the hybrid give the largest average reductions in loss (Table 2, Figure 1). Extra normalisation also helps. Single-switch contrasts differ by at most 0.004 nats between seeds. The Pearson correlation is 0.996 across all 15 contrasts and 0.94 across interaction contrasts alone.

Average loss differences for each switch at both seeds

Figure 1. Average loss differences for each switch at both seeds. Negative values favour the switch. Markers show estimates from different seeds, not uncertainty ranges. Each comparison can also include the corresponding four-switch effect because of the selected design.

ContrastSeed 0Seed 1Mean
E−0.0964−0.1001−0.0983
M−0.0486−0.0526−0.0506
H−0.0369−0.0361−0.0365
S−0.0190−0.0198−0.0194
G+0.0021+0.0015+0.0018
SE / GMH−0.0124−0.0099−0.0111
ME / GSH−0.0094−0.0090−0.0092
GE / MSH−0.0073−0.0063−0.0068
GS / MEH−0.0063−0.0041−0.0052
MS / GEH+0.0014−0.0021−0.0004
EH / GMS+0.0012−0.0005+0.0003
GH / MSE+0.0042−0.0018+0.0012
SH / GME+0.0027+0.0033+0.0030
GM / SEH+0.0108+0.0124+0.0116
MH / GSE+0.0106+0.0139+0.0122

Table 2. Loss contrasts in nats. A slash names effects that this design cannot separate. Single-switch rows also include four-switch effects. Means use unrounded losses, so they may differ from averages of the displayed values. The reference margin for a mean is ±0.0043, under the noise assumptions in the text.

The gate shows why the common comparison matters. Adding G alone to the all-off baseline reduces seed-0 loss by 0.019 nats. Across the selected combinations, its repeated mean is instead +0.0018, within the reference margin. Spending active parameters on this gate gives little average benefit here. This does not tell us whether an extra gate without a narrower feed-forward layer would help, or which other change reduces its benefit.

Gains mostly combine, with small interactions

The largest mean pairwise contrast is MH/GSE at +0.0122 nats. Adding the reference margin gives about 0.017 nats, below the experiment’s practical bound of 0.02 nats. Even so, six pairwise means and four single-switch means exceed the reference margin. The three largest seed-0 interaction contrasts—SE/GMH, GM/SEH, and MH/GSE—keep their signs and exceed the single-seed reference margin of ±0.0060 at seed 1.

The large gains therefore mostly combine, but the smaller departures matter when comparing modest refinements. If higher-order effects are negligible, a pairwise contrast of 0.012 means that turning one switch on changes the other’s benefit by about 0.024 nats.

The all-five configuration has the lowest loss in the selected set at both seeds: 3.6792 and 3.6750 nats. The untested combinations or separately tuned recipes could do better. In particular, G’s near-zero average does not show that removing it from the all-five model would help: that four-switch configuration was not run.

Repeats reveal smaller effects

Lenth’s PSE implies a run standard deviation near 0.020 nats, using the single-seed contrast standard error σ/2\sigma/ 2. That is about six times the repeated-run estimate of 0.0036. The initial margin separates only E, M, and H from zero; the repeated-run reference also separates S and several interaction contrasts. Many effects in this experiment contain signal, so their spread overestimates noise. This explains the conservative initial result without implying that Lenth’s method generally fails.

Checking the Implementation

Two numerical failures were found during development, repaired, and excluded from the final results. First, an RMSNorm GPU kernel let a position use information from the next token at a particular row-byte width. The resulting very low loss was invalid. Tests at the actual model width, changing future tokens at both odd and even positions, exposed the leak and checked the replacement formula. Smaller-width tests had missed it.

Second, the sigmoid router could produce finite outputs but non-finite gradients when selected scores were extremely small. Replaying a saved failing batch traced this to a squared denominator that underflowed during differentiation. We use the equivalent expression

gi=ρ softmax⁡i∈K(log⁡sigmoid⁡(ℓi)),(7)g_i = \rho\,\operatorname{softmax}_{i\in K}\left(\log\operatorname{sigmoid}(\ell_i)\right), \tag*{(7)}

with a stable log-sigmoid implementation and ρ=3\rho= \sqrt{3}, instead of differentiating ρsi/∑j∈Ksj\rho s_i / \sum_{j\in K} s_j directly. The final comparison uses runs after both fixes.

Limitations and Reproducibility

Where the findings apply. The study covers about 51.5M active parameters, one corpus, a 1,024-token context, and a nominal 300M-token budget. Training time, energy and inference speed are not matched. The results do not establish downstream or long-context gains, or behaviour at larger scales. Optimiser comparisons show why size, tuning and final budget can change rankings [24].

What the comparisons measure. E adds total capacity, G spends some feed-forward capacity on a gate, H changes both recurrent mixing and nearby design choices, and M includes several optimiser settings. These are comparisons of complete recipes. Also, the selected combinations cannot separate all effects. A complementary selection or targeted extra configurations would be needed to resolve the remaining ambiguity.

Variation and model selection. Two seeds give little information about each configuration’s full distribution. The noise pool assumes a common variance, and shared data streams may correlate errors. Both seeds are reported to make their agreement visible. The same validation corpus supports preliminary tuning and final comparisons, so these are development results rather than measurements on a fresh test set.

Reproduction. An evidence ledger links definitions, settings, final losses and calculations to the source report and generated tables. It does not contain every raw log or checkpoint. The 32 core runs, learning-rate sweeps and extra H repeats must be counted separately. Reproduction should first check future-token causality at the model width, finite router gradients on extreme logits, and matching data streams. Equations 4–5 then give the calculations from completed valid runs.

Conclusion

In this training setting, experts, Muon, a Gated DeltaNet hybrid and extra normalisation reduce average loss. A gate with a matched parameter budget adds little on average. The main gains mostly combine, but small interactions repeat. Comparing components together and repeating the runs gives a clearer basis for deciding which changes to keep.

References

  1. [1]Akbar, H. 2026. ATMA: Long-Context Language Modeling via Polar Attention and Gated-Delta Compression Memory. arXiv preprint arXiv:2606.25156. Version 4.arxiv.org/abs/2606.25156
  2. [2]Box, G. E. P.; and Hunter, J. S. 1961. The 2^k−p Fractional Factorial Designs, Part I. Technometrics, 3(3): 311–351.
  3. [3]Dai, D.; Deng, C.; Zhao, C.; Xu, R. X.; Gao, H.; Chen, D.; Li, J.; Zeng, W.; Yu, X.; Wu, Y.; et al. 2024. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models. arXiv preprint arXiv:2401.06066.arxiv.org/abs/2401.06066
  4. [4]DeepSeek-AI. 2024. DeepSeek-V3 Technical Report. arXiv preprint arXiv:2412.19437.arxiv.org/abs/2412.19437
  5. [5]Ding, M.; Yang, Z.; Hong, W.; Zheng, W.; Zhou, C.; Yin, D.; Lin, J.; Zou, X.; Shao, Z.; Yang, H.; and Tang, J. 2021. CogView: Mastering Text-to-Image Generation via Transformers. In Advances in Neural Information Processing Systems, volume 34.arxiv.org/abs/2105.13290
  6. [6]Dodge, J.; Gururangan, S.; Card, D.; Schwartz, R.; and Smith, N. A. 2019. Show Your Work: Improved Reporting of Experimental Results. In Proceedings of EMNLP-IJCNLP, 2185–2194.arxiv.org/abs/1909.03004
  7. [7]Fisher, R. A. 1935. The Design of Experiments. Edinburgh: Oliver and Boyd.
  8. [8]Geiping, J.; and Goldstein, T. 2023. Cramming: Training a Language Model on a Single GPU in One Day. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, 11117–11143.arxiv.org/abs/2212.14034
  9. [9]Henry, A.; Dachapally, P. R.; Pawar, S. S.; and Chen, Y. 2020. Query-Key Normalization for Transformers. In Findings of the Association for Computational Linguistics: EMNLP 2020, 4246–4253.arxiv.org/abs/2010.04245
  10. [10]Jordan, K.; Jin, Y.; Boza, V.; You, J.; Cesista, F.; Newhouse, L.; and Bernstein, J. 2024a. Muon: An Optimizer for Hidden Layers in Neural Networks. Author’s technical article.
  11. [11]Jordan, K.; et al. 2024b. modded-nanogpt: The NanoGPT Speedrun. Software repository.
  12. [12]Kazemnejad, A.; Padhi, I.; Natesan Ramamurthy, K.; Das, P.; and Reddy, S. 2023. The Impact of Positional Encoding on Length Generalization in Transformers. In Advances in Neural Information Processing Systems, volume 36.arxiv.org/abs/2305.19466
  13. [13]Kim, J.; Lee, B.; Park, C.; Oh, Y.; Kim, B.; Yoo, T.; Shin, S.; Han, D.; Shin, J.; and Yoo, K. M. 2025. Peri-LN: Revisiting Normalization Layer in the Transformer Architecture. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, 30400–30436.arxiv.org/abs/2502.02732
  14. [14]Lenth, R. V. 1989. Quick and Easy Analysis of Unreplicated Factorials. Technometrics, 31(4): 469–473.DOI
  15. [15]Liu, J.; Su, J.; Yao, X.; Jiang, Z.; Lai, G.; Du, Y.; Qin, Y.; Xu, W.; Lu, E.; Yan, J.; et al. 2025. MoE Is Scalable for LLM Training. arXiv preprint arXiv:2502.16982.arxiv.org/abs/2502.16982
  16. [16]Loshchilov, I.; and Hutter, F. 2019. Decoupled Weight Decay Regularization. In International Conference on Learning Representations.
  17. [17]Penedo, G.; Kyliček, H.; Ben Allal, L.; Lozhkov, A.; Mitchell, M.; Raffel, C.; Von Werra, L.; and Wolf, T. 2024. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. In Advances in Neural Information Processing Systems, volume 37.arxiv.org/abs/2406.17557
  18. [18]Qiu, Z.; Wang, Z.; Zheng, B.; Huang, Z.; Wen, K.; Yang, S.; Men, R.; Yu, L.; Huang, F.; Huang, S.; Liu, D.; Zhou, J.; and Lin, J. 2025. Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free. arXiv preprint arXiv:2505.06708.arxiv.org/abs/2505.06708
  19. [19]Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; and Sutskever, I. 2019. Language Models Are Unsupervised Multitask Learners. Technical report, OpenAI.
  20. [20]Shazeer, N. 2020. GLU Variants Improve Transformer. arXiv preprint arXiv:2002.05202.arxiv.org/abs/2002.05202
  21. [22]Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017. Attention Is All You Need. In Advances in Neural Information Processing Systems, volume 30.arxiv.org/abs/1706.03762
  22. [23]Wang, L.; Gao, H.; Zhao, C.; Sun, X.; and Dai, D. 2024. Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts. arXiv preprint arXiv:2408.15664.arxiv.org/abs/2408.15664
  23. [24]Wen, K.; Hall, D.; Ma, T.; and Liang, P. 2025. Fantastic Pretraining Optimizers and Where to Find Them. arXiv preprint arXiv:2509.02046.arxiv.org/abs/2509.02046
  24. [25]Yang, S.; Kautz, J.; and Hatamizadeh, A. 2025. Gated Delta Networks: Improving Mamba2 with Delta Rule. In International Conference on Learning Representations.arxiv.org/abs/2412.06464
  25. [26]Yang, S.; Wang, B.; Zhang, Y.; Shen, Y.; and Kim, Y. 2024. Parallelizing Linear Transformers with the Delta Rule over Sequence Length. In Advances in Neural Information Processing Systems, volume 37.arxiv.org/abs/2406.06484
  26. [27]Zhang, B.; and Sennrich, R. 2019. Root Mean Square Layer Normalization. In Advances in Neural Information Processing Systems, volume 32.arxiv.org/abs/1910.07467

Paper details

vlnc:W_c3fmytkx
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents