betaFeedback

WorksComputer Science

Pretraining Data Scale Reverses the Muon-AdamW Fine-Tuning Gap

Abstract

Muon has recently emerged as a promising optimizer for pretraining large language models, offering a speedup over AdamW in minimizing pretraining loss. However, it remains unclear whether Muon's advantage persists after fine-tuning. In this paper, we study the downstream behavior of Muon and AdamW and uncover a surprising trend: beyond a sufficiently large pretraining data budget, AdamW outperforms Muon after fine-tuning. Moreover, as model size increases, AdamW overtakes Muon at progressively smaller pretraining token-parameter ratios. To explain this reversal, we view fine-tuning as a perturbation of the outputs and introduce the margin-to-spread ratio, which measures how far the correct logit lies above the wrong ones relative to how widely the wrong logits are spread. In language models, Muon has the larger margin-to-spread ratio at small data budgets and AdamW at large ones, mirroring the fine-tuning trend. In a linear associative memory, we prove that after one update the margin-to-spread ratio of gradient descent gains on that of Muon as the data budget grows, and simulations show that this trend persists after extended training.

Introduction

The Muon optimizer has recently shown strong empirical performance for pretraining LLMs, challenging AdamW’s status as the standard optimizer for minimizing pretraining loss ([13]). Prior work has shown that the pretraining optimization choices that minimize pretraining loss do not necessarily yield the best model after fine-tuning ([38]). What ultimately matters downstream is the learning–forgetting tradeoff: how well a fine-tuned model learns the target task while retaining its pretrained knowledge. Motivated by this, we ask:

Does Muon’s pretraining advantage over AdamW extend to the learning–forgetting tradeoff?

To answer this question, we measure the Muon–AdamW fine-tuning gap, which compares the two optimizers on this tradeoff; a negative gap indicates that the Muon pretrained model dominates the AdamW pretrained model after fine-tuning, and a positive gap indicates the reverse. We show that the answer depends on the pretraining data budget, and find that pretraining data scale reverses the Muon–AdamW fine-tuning gap. At small budgets, Muon achieves lower fine-tuning loss with less forgetting than AdamW and the fine-tuning gap is negative. At larger budgets, this trend reverses: AdamW achieves a better learning–forgetting tradeoff than Muon, and the fine-tuning gap becomes positive.

In § 3, we pretrain OLMo-style 30M–600M models ([11, 34]) on 150M to 154B DCLM tokens ([19]), comparing AdamW and Muon, and fine-tune each base model on four datasets while sweeping the fine-tuning optimizer and learning rate, for nearly 600 pretraining runs and 29,000 fine-tuning runs in total. We show that when tuning pretraining hyperparameters for the lowest pretraining loss, Muon consistently achieves lower loss than AdamW. However, with sufficient pretraining data, AdamW outperforms Muon after fine-tuning, reversing the Muon–AdamW fine-tuning gap. This reversal also holds when we disregard pretraining loss, instead sweeping pretraining hyperparameters to obtain the best learning–forgetting tradeoff after fine-tuning. We additionally show that as model size scales, AdamW surpasses Muon at progressively smaller data budgets.

In § 4, we investigate why the Muon–AdamW fine-tuning gap reverses as the pretraining data budget grows. Fine-tuning changes the output distribution, so we generalize fine-tuning updates with perturbations to the output distribution and study which optimizer produces base models that are more robust to such perturbations. To quantify sensitivity to output perturbations, we introduce the margin-to-spread ratio (MSR), a simple metric derived from the model’s output logits that characterizes robustness. For each prediction, the MSR is the difference between the correct logit and the average of the incorrect logits, normalized by the spread of the incorrect logits (Definition 4.1): a prediction is more robust when the MSR is larger.

To understand where this trend comes from, we study a linear associative memory trained with gradient descent (GD) and Muon (§ 4.2). Analyzing a single update from zero initialization, we prove that the ratio of GD’s MSR to Muon’s increases with the data budget (Theorem 4.2). Simulations of many training steps in the linear associative memory setting confirm the same trend: Muon has the larger MSR at small data budgets, while GD has the larger MSR at large data budgets (§ 4.2). Measuring the MSR of our pretrained language models, we find that Muon has the larger MSR at small data budgets, while AdamW has the larger MSR at large data budgets, mirroring the reversal of the fine-tuning gap (§ 4.3).

Preliminaries

Our goal is to compare AdamW- and Muon-pretrained models after fine-tuning. Since fine-tuning can induce forgetting of the pretraining distribution, performance on the fine-tuning task alone is not sufficient: one base model may learn more but also forget more. We therefore compare base models by the tradeoff between the two, and define metrics to track how this comparison changes with model size and pretraining data budget. Let θPT\theta_{\mathrm{PT}} denote a pretrained model.

Learning–forgetting tradeoff. Fine-tuning improves downstream performance but may degrade pretrained knowledge ([10, 17, 31]). When θPT\theta_{\mathrm{PT}} is fine-tuned to obtain θFT\theta_{\mathrm{FT}}, we measure learning by the fine-tuning loss LFT(θFT)\mathcal{L}_{\mathrm{FT}}(\theta_{\mathrm{FT}}) and forgetting by the pretraining loss LPT(θFT)\mathcal{L}_{\mathrm{PT}}(\theta_{\mathrm{FT}}). Since both depend on the fine-tuning hyperparameters, a single fine-tuned checkpoint does not capture how well θPT\theta_{\mathrm{PT}} can be adapted. We therefore summarize each base model by the outcomes reachable by fine-tuning it,

T(θPT):={(LFT(θFT),LPT(θFT))∣θFT∈ΘFT(θPT)},\mathcal{T}(\theta_{\mathrm{PT}}) := \left\{ \left(\mathcal{L}_{\mathrm{FT}}(\theta_{\mathrm{FT}}), \mathcal{L}_{\mathrm{PT}}(\theta_{\mathrm{FT}})\right) \mid\theta_{\mathrm{FT}} \in\Theta_{\mathrm{FT}}(\theta_{\mathrm{PT}}) \right\},

where ΘFT(θPT)\Theta_{\mathrm{FT}}(\theta_{\mathrm{PT}}) is the set of models obtained by fine-tuning θPT\theta_{\mathrm{PT}}, and compare base models through the lower-left Pareto frontier of this set, which we call the learning–forgetting frontier. For a collection of base models B\mathcal{B}, we compare the corresponding sets

T(B):={T(θPT)∣θPT∈B}.\mathcal{T}(\mathcal{B}) := \left\{ \mathcal{T}(\theta_{\mathrm{PT}}) \mid\theta_{\mathrm{PT}} \in\mathcal{B} \right\}.

Fine-tuning gap. We compare the two optimizers on this tradeoff with a single metric, fine-tuning gap GFTG_{\mathrm{FT}}, which we define as the signed mean minimum Euclidean distance between the AdamW and Muon frontier. The fine-tuning gap is positive when the AdamW frontier dominates the Muon frontier, and the reverse else wise. Thus GFT>0G_{\mathrm{FT}} > 0 when AdamW dominates and GFT<0G_{\mathrm{FT}} < 0 when Muon dominates (Section C.5 gives the details).

Crossover token–parameter ratio (TPR). We quantify pretraining data budget using token–parameter ratio (TPR), the number of pretraining tokens per model parameter. For a fixed model size and fine-tuning dataset, the crossover TPR is the minimum TPR required for the AdamW frontier to overtake the Muon frontier.

Experiments

We now compare how the learning–forgetting tradeoffs of Muon and AdamW scale with pretraining data budget and model size. In § 3.2, we show that as the pretraining data budget increases, Muon’s advantage over AdamW shrinks and eventually reverses in favor of AdamW. This reversal happens at progressively smaller data budgets as model size grows.

Experimental setup

Pretraining. We pretrain OLMo-style 30M–600M models on 150M to 154B DCLM tokens, comparing AdamW and Muon under a warmup–stable–decay (WSD) schedule. We sweep hyperparameters broadly and report the complete pretraining methodology in Section C.

Fine-tuning. For each base model, we evaluate the learning–forgetting tradeoff on four standard datasets: GSM8K ([4]), StackMathQA ([43]), Alpaca ([33]), and MusicPile ([41]), sweeping AdamW learning rates from 10−410^{-4} to 10−210^{-2} and Muon learning rates from 6×10−46 \times10^{-4} to 2×10−22 \times10^{-2} (Section C.2). We fine-tune AdamW-pretrained models with AdamW and Muon-pretrained models with both AdamW and Muon: AdamW fine-tuning consistently outperforms Muon fine-tuning on AdamW-pretrained models, whereas neither optimizer is consistently better on Muon-pretrained models (Section D.3). We then plot the learning–forgetting frontier over the resulting runs. We choose a broad learning-rate range to ensure we capture the complete learning–forgetting tradeoff spectrum: higher learning rates favor learning at the expense of forgetting, while lower rates preserve pretraining knowledge but learn less.

Evaluating the learning–forgetting tradeoff. For each fine-tuned checkpoint, we compute the fine-tuning loss and pretraining loss on held-out fine-tuning and pretraining (DCLM) data, and use these checkpoints to approximate the learning–forgetting tradeoff. Although this approximation is constructed from a finite set of fine-tuning runs, we conduct a comprehensive sweep over four datasets and across a wide range of fine-tuning learning rates to obtain a close estimate. We evaluate the learning–forgetting tradeoff in two settings: lowest-pretraining-loss setting, and the best-frontier setting.

  1. Lowest-pretraining-loss setting: Pretraining hyperparameters are typically selected to minimize pretraining loss. Therefore, for each optimizer, we tune the pretraining hyperparameters to minimize pretraining loss for a given pretraining data budget and model size. We then sweep across all fine-tuning configurations for each of the lowest-loss pretrained models, and evaluate the resulting frontier.

  1. Best-frontier setting: However, prior studies find that the lowest-loss pretrained model is not always the best model after fine-tuning: models pretrained with other hyperparameters can offer a better learning–forgetting tradeoff ([38]). Therefore, for each pretraining optimizer, we additionally sweep

across all pretraining and fine-tuning configurations to achieve the best possible learning–forgetting frontier for a given pretraining data budget and model size (Section C.1.3).

Main result

We now investigate how the Muon–AdamW fine-tuning gap scales with pretraining data budget and model size.

Main result: Pretraining data scale reverses the Muon–AdamW fine-tuning gap. We compare the fine-tuning gap for AdamW and Muon at different pretraining data budgets in the lowest-pretraining-loss setting, using StackMathQA as the fine-tuning dataset (Figure 3a). At small pretraining data budgets (2.4B tokens), Muon Pareto-dominates AdamW (Figure 1) and the fine-tuning gap is negative. At larger pretraining data budgets, this trend reverses: beyond 2.4B tokens, AdamW achieves lower fine-tuning loss with less forgetting than Muon. The trend also holds in the best-frontier setting (Figure 12), and for additional fine-tuning datasets in Section D.

Figure 1. Beyond a sufficiently large data budget, AdamW outperforms Muon. We pretrain OLMo-60M models with Muon and AdamW and fine-tune on StackMathQA. For each pretraining optimizer, we take the pretrained model with the lowest pretraining loss, sweep across all fine-tuning configurations, and plot the resulting learning–forgetting frontier, from 2.4B to 38.4B pretraining tokens. At 2.4B tokens, Muon’s frontier dominates AdamW’s; at larger budgets the trend reverses and AdamW surpasses Muon. This reversal happens despite Muon achieving lower pretraining loss across all data budgets.

Figure 3. The Muon–AdamW fine-tuning gap widens with increasing pretraining data budget, and Muon’s advantage shrinks with model size. (a) The fine-tuning gap after fine-tuning OLMo-60M models on StackMathQA, in the lowest-pretraining-loss setting (each pretraining optimizer’s lowest-loss pretrained model swept across all fine-tuning configurations). As the pretraining data budget increases, the fine-tuning gap transitions from negative to positive, with AdamW outperforming Muon beyond the crossover TPR. (b) Pretraining loss at 40 TPR. Muon’s advantage over AdamW shrinks as model size grows. (c) Degradation at 40 TPR, when we accept a StackMathQA loss 10% above the best in exchange for less forgetting. Muon degrades increasingly more than AdamW as model size grows.

Figure 12. Fine-tuning gap on every fine-tuning dataset, best-frontier setting (60M). Negative values mean the Muon frontier dominates.

We next investigate how the reversal of the Muon–AdamW fine-tuning gap scales with model size. Specifically, we ask how the amount of pretraining data needed for AdamW to overtake Muon changes as model size grows.

As model size scales, AdamW surpasses Muon at progressively smaller data budgets. We find that the crossover TPR decreases with model size. On StackMathQA, the crossover TPR is 320 TPR for 30M models but only 10 TPR for 600M models (Figure 2). The specific crossover TPR depends on the fine-tuning dataset, but the broader trend persists; as model size grows, the crossover TPR decreases, and Muon’s advantage reverses with substantially smaller data budgets.

Figure 2. Crossover TPR decreases with model size. In the lowest-pretraining-loss setting, where each pretraining optimizer’s lowest-loss pretrained model is swept across all fine-tuning hyperparameters, Muon’s advantage reverses at substantially smaller pretraining data budgets as model size increases. The exact crossover TPR varies depending on the fine-tuning dataset, but the high-level trend is consistent.

We hypothesize that in the lowest-pretraining-loss setting, this scaling trend is driven by two factors: Muon’s shrinking pretraining loss advantage and increased degradation relative to AdamW (Figure 3b,Figure 3c). For each optimizer and model size, we compare its base pretraining loss and its degradation LPT(θPT)−LPT(θPT∗)\mathcal{L}_{\mathrm{PT}}(\theta_{\mathrm{PT}}) - \mathcal{L}_{\mathrm{PT}}(\theta_{\mathrm{PT}}^{*}), after fine-tuning on StackMathQA. First, for a fixed TPR, the gap in pretraining loss between Muon and AdamW shrinks as model size increases, reducing Muon’s advantage as a lower-loss base model. Additionally, for a fixed TPR and fine-tuning loss, Muon forgets increasingly more than AdamW as model size grows. Consequently, as model size scales, Muon’s pretraining advantage over AdamW narrows, while its forgetting relative to AdamW increases, enabling AdamW to overtake Muon at progressively smaller data budgets.

Analyzing the Muon–AdamW Forgetting Gap

In § 3, we show that Muon offers a better learning–forgetting tradeoff at small token budgets, but AdamW overtakes it beyond a crossover budget. This reversal occurs even though Muon achieves lower pretraining loss at

every budget we consider. In this section, we develop a theoretical explanation for this reversal. We compare Muon with exact orthogonalization to full-batch gradient descent (GD), a tractable comparison that captures the qualitative budget trend of Muon and AdamW in language models (§ 4.3).

To formalize the relationship between GD and Muon, we study a linear associative memory model of fact learning. In this setting, GD and Muon learn the facts to a similar degree, but their checkpoints can differ in their sensitivity to perturbations of the output distribution. Since fine-tuning changes the output distribution, we expect checkpoints that preserve factual accuracy under a wider range of perturbations to be more resistant to forgetting during fine-tuning. Our central goal is therefore to characterize this sensitivity across pretraining budgets and use it to explain why the forgetting gap reverses.

To quantify sensitivity to output perturbations, we introduce the margin-to-spread ratio (MSR): the difference between the correct answer’s logit and the average incorrect logit, normalized by the spread of the incorrect logits. The MSR captures a checkpoint’s robustness to output perturbations and provides an analytically tractable basis for comparing GD and Muon.

Roadmap. We introduce the linear associative memory setting, define the MSR, and establish its connection to sensitivity (§ 4.1). We then prove that, after one update step, the ratio of Muon’s MSR to GD’s decreases as the data budget grows, indicating that Muon becomes more sensitive to output perturbations relative to GD (§ 4.2). Finally, we show that the MSR tracks the robustness of AdamW and Muon in real language models (§ 4.3).

Setup and the margin-to-spread ratio

We begin by introducing our linear associative memory setup and defining the margin-to-spread ratio.

Linear associative memory. To capture the behavior of Muon and GD in a tractable setting, we consider a linear associative memory, in which a single weight matrix is trained to associate each of NN keys, represented by a vector, with its corresponding value, also represented by a vector. Formally, we represent each fact by a key–value pair (ui,vi)∈Rd×Rd(u_i, v_i) \in\mathbb{R}^{d} \times\mathbb{R}^{d}, with all keys and values drawn independently from N(0,Id)\mathcal{N}(0, I_d). The memory is parameterized by a matrix W∈Rd×dW \in\mathbb{R}^{d \times d}. Given a key uiu_i, it scores each candidate value vjv_j as sij(W):=vjTWuis_{ij}(W) := v_j^{\mathsf{T}} W u_i, which we treat as logits, and retrieves the highest-scoring value. Fact ii is recovered when sii(W)>sij(W)s_{ii}(W) > s_{ij}(W) for all j≠ij \ne i.

We train with the softmax cross-entropy loss. Given a query distribution q=(q1,…,qN)q = (q_1, \ldots, q_N) over the facts, the predicted probabilities and cross-entropy loss are

p^W(⋅∣i):=softmax⁡(si1(W),…,siN(W)),L(W):=−∑i=1Nqilog⁡p^W(i∣i).(1)\widehat{p}_W(\cdot\mid i) := \operatorname{softmax}\left(s_{i1}(W), \ldots, s_{iN}(W)\right), \qquad\mathcal{L}(W) := -\sum_{i=1}^{N} q_i \log\widehat{p}_W(i \mid i). \tag*{(1)}

Data budget. We denote the memory dimension by dd and the number of facts by NN, and grow the two together, keeping their ratio γ=N/d>1\gamma= N/d > 1 fixed, so that N=γdN = \gamma d as d→∞d \to\infty. This mirrors the typical scaling behavior of language models, which store many more facts than they have dimensions. We call γ\gamma the data budget: it measures facts per dimension and plays the role of token-parameter ratio in § 2. We compare data budgets by taking this limit at different fixed values of γ\gamma. Since γ>1\gamma> 1, we have N−1≥dN - 1 \ge d, and log⁡N/d→0\log N/d \to0, written log⁡N=o(d)\log N = o(d). For random variables XX and a deterministic a>0a > 0, X=Op(a)X = O_p(a) means that for every ε>0\varepsilon> 0 there is a CC with Pr⁡[∣X∣>Ca]≤ε\Pr[|X| > Ca] \le\varepsilon eventually, and X=op(a)X = o_p(a) means that Pr⁡[∣X∣>εa]→0\Pr[|X| > \varepsilon a] \to0 for every ε>0\varepsilon> 0.

Optimization. For simplicity, we consider the exact orthogonalization variant of Muon. For a matrix AA with singular value decomposition A=PΣQTA = P\Sigma Q^{\mathsf{T}}, let polar⁡(A):=PQT\operatorname{polar}(A) := P Q^{\mathsf{T}}. Muon updates Wt+1=Wt−ηpolar⁡(∇L(Wt))W_{t+1} = W_t - \eta\operatorname{polar}(\nabla\mathcal{L}(W_t)). We compare Muon to full-batch gradient descent (GD), whose updates descend along the gradient itself, Wt+1=Wt−η∇L(Wt)W_{t+1} = W_t - \eta\nabla\mathcal{L}(W_t).

Margin-to-spread ratio. In a binary classification problem, a prediction is robust if the correct logit is far from the incorrect logit: the larger the margin, the more robust the prediction. In a multiclass classification problem, such as our linear associative memory setting, there are many incorrect logits. The robustness of a correct output to perturbations of the logits thus depends on its separation from all of the incorrect logits. As it turns out, the degree of separation can be usefully summarized by a single statistic, which we call the margin-to-spread ratio (MSR). MSR captures the average separation between the correct logit and all other logits, normalized by the spread of the incorrect logits. A smaller spread, or a larger average separation, would both place the incorrect logits further below the correct logit relative to their spread, which can increase the robustness of the prediction to perturbations. To capture this intuition, we formally define the margin-to-spread ratio as follows:

Definition 4.1 (Margin-to-spread ratio). For fact ii, define the pairwise margins mij:=sii(W)−sij(W)m_{ij} := s_{ii}(W) - s_{ij}(W) for j≠ij \ne i, with mean and variance

hi:=mean⁡j≠imij,κi2:=mean⁡j≠i(mij−hi)2.(2)h_i := \operatorname{mean}_{j \ne i} m_{ij}, \qquad\kappa_i^2 := \operatorname{mean}_{j \ne i}(m_{ij} - h_i)^2. \tag*{(2)}

The margin-to-spread ratio (MSR) for fact ii is ρi:=hi/κi\rho_i := h_i/\kappa_i, for κi>0\kappa_i > 0. At the model level, we average the mean margins and variances uniformly over facts before taking the ratio:

hˉ:=mean⁡ihi,κˉ2:=mean⁡iκi2,ρ:=hˉκˉ,(3)\bar{h} := \operatorname{mean}_i h_i, \qquad\bar{\kappa}^2 := \operatorname{mean}_i \kappa_i^2, \qquad\rho:= \frac{\bar{h}}{\bar{\kappa}}, \tag*{(3)}

where ρ\rho is defined for κˉ>0\bar{\kappa} > 0.

Since sii(W)s_{ii}(W) is the same for every competitor jj, κi\kappa_i is also the standard deviation of the incorrect logits sij(W)s_{ij}(W), so the MSR measures the average margin in units of their spread. The definition applies to any model that assigns a logit to each candidate answer; for a language model, sij(W)s_{ij}(W) are the next-token logits (§ 4.3).

Under isotropic Gaussian noise added to the logits, the MSR gives a lower bound on the fraction of incorrect answers that stay below the correct one: with noise of standard deviation σ\sigma on each logit, this fraction is at least ρi2/(ρi2+2σ2/κi2)\rho_i^2/(\rho_i^2 + 2\sigma^2/\kappa_i^2) in expectation. For a fixed noise strength relative to the incorrect-logit spread, this bound improves as the MSR increases (Corollary B.5).

One-step analysis of the margin-to-spread ratio

We have so far introduced our linear associative memory setup and built intuition for how the MSR captures the robustness of a model to perturbations of its logits. Here, we argue that the MSR of GD is greater than that of Muon for sufficiently large data budgets. We precisely track the one-step dynamics of the MSR ratio R:=ρGD/ρMuon\mathcal{R} := \rho^{\mathrm{GD}}/\rho^{\mathrm{Muon}} between GD and Muon in the linear associative memory.

We analyze a single update from zero initialization, using uniform query weights qi=1/Nq_i = 1/N. The weights after many optimization steps depend on the whole training trajectory and are hard to characterize precisely, so theoretical studies of how an optimizer shapes the learned weights often analyze its first step ([2, 5, 25]). Kim et al. [14] take the same approach to compare a single step of Muon, SGD, and Newton’s method in a linear associative memory.

Theorem 4.2 (Informal). Consider one GD step and one Muon step from zero in the Gaussian associative memory model, with data budget γ=N/d\gamma= N/d. For sufficiently large γ\gamma:

  • GD has a larger MSR than Muon.

  • Increasing the data budget γ\gamma by any fixed percentage increases the ratio R=ρGD/ρMuon\mathcal{R} = \rho^{\mathrm{GD}}/\rho^{\mathrm{Muon}}, with high probability as the dimension grows.

  • The ratio approaches 3π/8≈1.183\pi/8 \approx1.18 from below at a rate of 1/γ1/\gamma:

−Cγ≤R−3π8≤−cγ,-\frac{C}{\gamma} \le\mathcal{R} - \frac{3\pi}{8} \le-\frac{c}{\gamma},

where 0<c<C0 < c < C are absolute constants.

All claims hold with probability tending to one as d→∞d \to\infty, for sufficiently large fixed γ\gamma.

Proof sketch. Our goal is to compare one update step for Muon and for GD.

  1. Since the MSR does not change when WW is multiplied by a positive constant, the one-step weights can be written as

WGD=G,WMuon=O:=polar⁡(G)W^{\mathrm{GD}} = G, \qquad W^{\mathrm{Muon}} = O := \operatorname{polar}(G)

(Appendix, Lemma F.1). We now want to compute ρ=hˉ/κˉ\rho= \bar{h}/\bar{\kappa} for these matrices. This requires the mean margins and their standard deviations; we begin with the means.

References

  1. [1]Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European conference on computer vision (ECCV), pp. 139–154, 2018.arxiv.org/abs/1711.09601
  2. [2]Jimmy Ba, Murat A. Erdogdu, Taiji Suzuki, Zhichao Wang, Denny Wu, and Greg Yang. High-dimensional asymptotics of feature learning: How one gradient step improves the representation. In Advances in Neural Information Processing Systems, 2022. URL https://arxiv.org/abs/2205.01445.
  3. [3]Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with A-GEM. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=HkF2_sc5FX.arxiv.org/abs/1812.00420
  4. [4]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/abs/2110.14168.
  5. [5]Yatin Dandi, Florent Krzakala, Bruno Loureiro, Luca Pesce, and Ludovic Stephan. How two-layer neural networks learn, one (giant) step at a time. Journal of Machine Learning Research, 25(349):1–65, 2024. URL https://jmlr.org/papers/v25/23-1543.html.arxiv.org/abs/2305.18270
  6. [6]Sara Dragutinović, Yedi Zhang, and Rajesh Ranganath. To use or not to use Muon: How simplicity bias in optimizers matters, 2026. URL https://arxiv.org/abs/2603.00742.
  7. [7]Essential AI, Ishaan Shah, Anthony M. Polloreno, Karl Stratos, Philip Monk, Adarsh Chaluvaraju, et al. Practical efficiency of Muon for pretraining, 2025. URL https://arxiv.org/abs/2505.02222.
  8. [8]Chen Fan, Mark Schmidt, and Christos Thrampoulidis. Implicit bias of spectral descent and Muon on multiclass separable data. In Advances in Neural Information Processing Systems, 2025. URL https://arxiv.org/abs/2502.04664.
  9. [10]Ian J. Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgetting in gradient-based neural networks. In International Conference on Learning Representations, 2014. URL https://arxiv.org/abs/1312.6211.
  10. [11]Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhatia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, and Hannaneh Hajishirzi. OLMo: Accelerating the science of language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15789–15809, 2024. URL https://aclanthology.org/2024.acl-long.841.arxiv.org/abs/2402.00838
  11. [12]Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. In Advances in Neural Information Processing Systems, volume 35, 2022. URL https://arxiv.org/abs/2203.15556.
  12. [13]Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks. Blog post, 2024. URL https://kellerjordan.github.io/posts/muon/.
  13. [14]Juno Kim, Eshaan Nichani, Denny Wu, Alberto Bietti, and Jason D. Lee. Sharp capacity scaling of spectral optimizers in learning associative memory, 2026. URL https://arxiv.org/abs/2603.26554.
  14. [15]Kimi Team. Kimi K2: Open agentic intelligence, 2025. URL https://arxiv.org/abs/2507.20534.
  15. [16]Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015. URL https://arxiv.org/abs/1412.6980.
  16. [17]James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521–3526, March 2017. ISSN 1091-6490. doi: 10.1073/pnas.1611835114. URL http://dx.doi.org/10.1073/pnas.1611835114.
  17. [18]Binghui Li, Kaifei Wang, Han Zhong, Pinyan Lu, and Liwei Wang. Muon in associative memory learning: Training dynamics and scaling laws. In International Conference on Machine Learning, 2026. URL https://arxiv.org/abs/2602.05725.
  18. [19]Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Kumar Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee F Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Kamal Mohamed Abbas, Cheng-Yu Hsieh, Dhruba Ghosh, Joshua P Gardner, Maciej Kilian, Hanlin Zhang, Rulin Shao, Sarah M Pratt, Sunny Sanyal, Gabriel Ilharco, Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jieyu Zhang, Khyathi Chandu, Thao Nguyen, Igor Vasiljevic, Sham M. Kakade, Shuran Song, Sujay Sanghavi, Fartash Faghri, Sewoong Oh, Luke Zettlemoyer, Kyle Lo, Aalaaeldin El-Nouby, Hadi Pouransari, Alexander T Toshev, Stephanie Wang, Dirk Groeneveld, Luca Soldaini, Pang Wei Koh, Jenia Jitsev, Thomas Kollar, Alex Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, and Vaishaal Shankar. Datacomp-LM: In search of the next generation of training sets for language models. In The Thirty-eighth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https://openreview.net/forum?id=CNWdNw47IE.
  19. [20]Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, Yanru Chen, Huabin Zheng, Yibo Liu, Shaowei Liu, Bohong Yin, Weiran He, Han Zhu, Yuzhi Wang, Jianzhou Wang, Mengnan Dong, Zheng Zhang, Yongsheng Kang, Hao Zhang, Xinran Xu, Yutao Zhang, Yuxin Wu, Xinyu Zhou, and Zhilin Yang. Muon is scalable for LLM training, 2025. URL https://arxiv.org/abs/2502.16982.
  20. [21]Yuxing Liu, Jianyu Wang, and Tong Zhang. Optimizer-model consistency: Full finetuning with the same optimizer as pretraining forgets less, 2026. URL https://arxiv.org/abs/2605.06654.
  21. [22]David Lopez-Paz and Marc’Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
  22. [23]Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019.
  23. [24]Sushant Mehta, Raj Dandekar, Rajat Dandekar, and Sreedath Panat. Muon: Training and trade-offs with latent attention and MoE, 2025. URL https://arxiv.org/abs/2509.24406.
  24. [25]Behrad Moniri, Donghwan Lee, Hamed Hassani, and Edgar Dobriban. A theory of non-linear feature learning with one gradient step in two-layer neural networks. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, 2024. URL https://proceedings.mlr.press/v235/moniri24a.html.
  25. [26]Razvan Pascanu, Clare Lyle, Ionut-Vlad Modoranu, Naima Elosегui Borras, Dan Alistarh, Petar Velickovic, Sarath Chandar, Soham De, and James Martens. Optimizers qualitatively alter solutions and we should leverage this, 2025. URL https://arxiv.org/abs/2507.12224.
  26. [27]Tianyu Ruan, Fengzhuo Zhang, Shuche Wang, and Shihua Zhang. Muon learns more robust and transferable features than Adam, 2026. URL https://arxiv.org/abs/2606.09658.
  27. [28]Naoki Sato, Hiroki Naganuma, and Hideaki Iiduka. Convergence bound and critical batch size of Muon optimizer, 2025. URL https://arxiv.org/abs/2507.01598.
  28. [29]Wei Shen, Ruichuan Huang, Minhui Huang, Cong Shen, and Jiawei Zhang. On the convergence analysis of Muon, 2025. URL https://arxiv.org/abs/2505.23737.
  29. [30]Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Advances in Neural Information Processing Systems, volume 30, 2017. URL https://arxiv.org/abs/1705.08690.
  30. [31]Jacob Mitchell Springer, Sachin Goyal, Kaiyue Wen, Tanishq Kumar, Xiang Yue, Sadhika Malladi, Graham Neubig, and Aditi Raghunathan. Overtrained language models are harder to fine-tune. In International Conference on Machine Learning, 2025. URL https://arxiv.org/abs/2503.19206.
  31. [32]Weijie Su. Isotropic curvature model for understanding deep learning optimization: Is gradient orthogonalization optimal?, 2025. URL https://arxiv.org/abs/2511.00674.
  32. [33]Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford Alpaca: An instruction-following LLaMA model. https://github.com/tatsu-lab/stanford_alpaca, 2023.
  33. [34]Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groenveld, Kyle Lo, Shane Arora, Akshita Bhatia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tajford, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Allyson Ettinger, Michal Guerguin, David Heineman, Hamish Ivison, Pang Wei Koh, Jiacheng Liu, Saumya Malik, William Merrill, Lester James V. Miranda, Jacob Morrison, Tyler Murray, Crystal Nam, Jake Poznanski, Valentina Pyatkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Christopher Wilhelm, Michael Wilson, Luke Zettlemoyer, Ali Farhadi, Noah A. Smith, and Hannaneh Hajishirzi. 2 OLMo 2 furious. In Conference on Language Modeling, 2025. URL https://arxiv.org/abs/2501.00656.
  34. [35]Bhavya Vasudeva, Puneesh Deora, Yize Zhao, Vatsal Sharan, and Christos Thrampoulidis. How Muon’s spectral design benefits generalization: A study on imbalanced data. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=YzjS4jcfmS.arxiv.org/abs/2510.22980
  35. [36]Shuche Wang, Fengzhuo Zhang, Jiaxiang Li, Dirk Bergemann, and Zhuoran Yang. Why Muon outperforms Adam: A curvature perspective, 2026a. URL https://arxiv.org/abs/2606.04662.
  36. [37]Shuche Wang, Fengzhuo Zhang, Jiaxiang Li, Cunxiao Du, Chao Du, Tianyu Pang, Zhuoran Yang, Mingyi Hong, and Vincent Y. F. Tan. Muon outperforms Adam in tail-end associative memory learning. In The Fourteenth International Conference on Learning Representations, 2026b. URL https://arxiv.org/abs/2509.26030.
  37. [38]Ishaan Watts, Catherine Li, Sachin Goyal, Jacob Mitchell Springer, and Aditi Raghunathan. Sharpness-aware pretraining mitigates catastrophic forgetting. In Proceedings of the 43rd International Conference on Machine Learning, 2026. URL https://arxiv.org/abs/2605.02105.
  38. [40]Kaiyue Wen, David Hall, Tengyu Ma, and Percy Liang. Fantastic pretraining optimizers and where to find them. In The Fourteenth International Conference on Learning Representations, 2026b. URL https://openreview.net/forum?um?id=2J51qUZ0iG.arxiv.org/abs/2509.02046
  39. [41]Ruibin Yuan, Hanfeng Lin, Yi Wang, Zeyue Tian, Shangda Wu, Tianhao Shen, Ge Zhang, Yuhang Wu, Cong Liu, Ziya Zhou, Ziyang Ma, Liumeng Xue, Ziyu Wang, Qin Liu, Tianyu Zheng, Yizhi Li, Yinghao Ma, Yiming Liang, Xiaowei Chi, Ruibo Liu, Zili Wang, Pengfei Li, Jingcheng Wu, Chenghua Lin, Qifeng Liu, Tao Jiang, Wenhao Huang, Wenhu Chen, Emmanouil Benetos, Jie Fu, Gus Xia, Roger Dannenberg, Wei Xue, Shiyin Kang, and Yike Guo. ChatMusician: Understanding and generating music intrinsically with LLM. In Findings of the Association for Computational Linguistics: ACL 2024, pp. 6252–6271, 2024. URL https://aclanthology.org/2024.findings-acl.373/.
  40. [42]Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International conference on machine learning, pp. 3987–3995. PMLR, 2017.arxiv.org/abs/1703.04200
  41. [43]Yifan Zhang. StackMathQA: A Curated Collection of 2 Million Mathematical Questions and Answers Sourced from Stack Exchange. Technical report, Math-AI, 2024.

Paper details

vlnc:W_c75kzsm8
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 15

downloads · 1

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents