MUON-RT Reusing the Teacher's Trajectory for Momentum-Free Muon for LLM Self-Distillation

Introduction

Self-distillation ([42, 17]) provides an effective approach to adapting large language models (LLMs) by using privileged information to elicit supervision from the model’s existing capabilities. The student learns from its own generated responses by matching a teacher’s predictions conditioned on additional context. This formulation supports skill acquisition and policy improvement while helping preserve previously learned capabilities. Such retention is an advantage over conventional finetuning, which can cause forgetting ([4] (Zheng et al. 2025a)), a recognized challenge when updating LLMs ([43] (Zheng et al. 2025b)). Within this family, Self-Distillation Fine-Tuning (SDFT) uses demonstrations to guide the acquisition of new skills ([42]), while Self-Distillation Policy Optimization (SDPO) extends the approach to on-policy post-training using feedback from successful attempts ([17]). In this settings, the teacher is maintained as an exponential moving average (EMA) of student weights, following approaches developed in semi-supervised and self-supervised learning ([45, 15, 6]). The teacher therefore evolves alongside the student, providing an averaged model whose predictions guide subsequent learning.

Optimizing EMA-based self-distillation incurs substantial memory costs because training must retain student and teacher parameters alongside optimizer state. Adam Kingma & Ba [22] and AdamW (Loshchilov & Hutter, 2019) store two running moment estimates per parameter. Muon ([20]) reduces this overhead by maintaining one momentum buffer per hidden weight matrix and approximately orthogonalizing update directions using Newton–Schulz iterations. Muon and its variants have been used to train Kimi K2 [3] and GLM-4.5 [46]. Nevertheless, optimizing self-distillation with Muon still requires three persistent states for each hidden matrix: student weights, teacher weights, and momentum. AdamW requires four, including its two moment buffers. For example Qwen2.5-7B-Instruct, momentum Muon and AdamW require 103.2 GB and 129.0 GB of GPU memory for persistent states, respectively, compared with 75.4 GB for Muon-RT (Figure 1). Existing memory-efficient optimizers address state storage through low-rank projection, quantization, or shared statistics [25] (Zhao et al. 2024; Zhang et al. 2025; Zhu et al. 2025). The teacher offers another opportunity: reusing history already maintained by self-distillation to remove a separate optimizer buffer. We therefore ask:

Can the teacher’s existing parameter trajectory replace Muon’s momentum buffer reducing persistent memory while improving task accuracy in self-distillation?

Why could the teacher replace momentum? In SDFT and SDPO, the teacher is not fixed; its weights continuously follow the evolving student through an EMA. Unlike momentum, which averages gradients, the teacher averages weights. The momentum-like information therefore lies in the teacher–student displacement, not the teacher itself. As the student progresses, this displacement records an exponentially weighted history of applied updates (Lemma 1), smoothing noisy steps while preserving persistent directions (Lemma 2). Under gradient descent, this history is proportional to gradient momentum; under Muon, it captures previously orthogonalized updates. As optimization settles, the displacement shrinks as the teacher catches up with the student. Although this cannot recover AdamW’s second-moment statistics, however, it can replace Muon’s momentum buffer.

Even though this approach appears simple, reusing the teacher–student displacement in Muon is not straightforward. First, Muon orthogonalizes its updates, so the displacement averages past optimizer outputs rather than raw gradients. Using it directly as momentum would feed these outputs back into the optimizer. Second, mixing the displacement with gradients before Newton–Schulz leaves its influence uncontrolled, as Newton–Schulz can amplify weak singular directions. This distinction matters in self-distillation, where the teacher and learning targets continuously evolve with the student. While momentum averages gradients from outdated targets, the displacement captures the updates actually applied to the student, providing a different source of historical guidance.

We therefore propose Muon with Reused Teacher (Muon-RT), which replaces the dedicated momentum buffer with information from the existing EMA teacher. For each hidden weight matrix, Muon-RT orthogonalizes the current gradient without momentum and separately computes the teacher–student displacement. It removes the component parallel to the student weights and scales the remainder by an estimate of its largest singular value. The two directions are then combined after orthogonalization using a small fixed mixing coefficient. This preserves current gradient information while adding historical guidance without changing the training objective. Only one small vector per matrix is retained to warm-start power iteration across successive optimization steps. The method applies wherever a parameter-aligned EMA teacher is already maintained, including the studied SDFT and SDPO settings and mean-teacher training [45]. Independent teachers lacking this alignment fall outside its scope. Our contributions are:

• We establish that the teacher–student displacement stores a discounted history of applied updates, is proportional to gradient momentum under gradient descent, and reduces variance under a local model with a fixed direction and temporally uncorrelated noise.

• We introduce Muon-RT, eliminating hidden-matrix momentum buffers through normalized teacher–student displacement added after orthogonalization. Under regularity, bounded objective drift, and alignment assumptions, we establish linear contraction of an expected objective-gap bound toward a residual error level.

• With SDFT, Muon-RT improves average accuracy over AdamW by 2.6 points with 42% less persistent memory, and over momentum Muon by 5.4 points with 27% less (Figure 1). With SDPO, it gains 2.0 points over AdamW. Muon-RT ranks first among Muon variants across both hosts’ six settings.

Preliminaries

Self-distillation with an EMA-teacher: Let $S$ and $T$ denote the student and teacher LLMs, with policies $\pi_S$ and $\pi_T$ parameterized by $\theta_S$ and $\theta_T$, respectively. Both share the same architecture and initialization. For a prompt $x$, the student samples a completion $y \sim\pi_S(\cdot\mid x)$. The teacher evaluates the same tokens using privileged context $c$: an expert demonstration in SDFT (Shenfeld et al., [42]), or rollout feedback, potentially including a successful sibling response, in SDPO (Hübotter et al., [17]). At token position $j$, their next-token distributions are

The student minimizes the token-level distillation objective

where the expectation uses student-generated completions and their associated teacher contexts, $\operatorname{sg}$ stops teacher gradients, and $\mathcal{D}$ denotes the chosen divergence. SDFT formulates reverse KL but favors forward KL empirically, while SDPO employs Jensen–Shannon or reverse KL. To stabilize distillation, the EMA-teacher is updated after each student optimization step:

Our method assumes only that the EMA-teacher is explicitly maintained and parameter-wise aligned with the student.

Muon Optimizer: Muon updates each student hidden weight matrix $W_S \in\mathbb{R}^{A \times B}$, while embeddings, output heads, norms, and biases typically use AdamW (Jordan et al., [20]; Loshchilov & Hutter, [28]; Liu et al., 2025). Given gradient $G = \nabla_{W_S}\mathcal{L}(\theta_S)$ and momentum coefficient $\mu$, Muon maintains $M \leftarrow\mu M + G$ and orthogonalizes the Nesterov direction:

Here, $\operatorname{NS}_K$ applies $K$ quintic Newton–Schulz iterations from $X_0 = Z/(\lVert Z\rVert_F+\varepsilon)$, with $Z = G+\mu M$:

The fixed coefficients approximately equalize nonzero singular values, yielding an approximate polar factor (Jordan et al., [20]). With decoupled weight decay, the update is

where $\eta$ is the learning rate, $\lambda$ is weight decay, and $c_{A,B}=0.2\sqrt{\max(A,B)}$ approximately matches AdamW’s update RMS (Liu et al., 2025). Combined with the EMA-teacher, Muon maintains 3 persistent states per hidden matrix: student weights $W_S$, teacher weights $W_T$, and momentum $M$. We do not count the gradient $G$ as a persistent state: it is recomputed at every step and freed after the update, and every optimizer needs it in the same way. During a step, Muon therefore holds 4 matrix-sized tensors, and between steps it keeps 3.

Momentum-Free Muon Optimization

We first characterize the information retained by the EMA-teacher, derive the proposed update, analyze memory efficiency and sharded implementation, and present convergence guarantees. Throughout, we consider a student hidden weight matrix $W_S \in\mathbb{R}^{A \times B}$ and its corresponding teacher matrix $W_T$. Each matrix is processed independently. Figure 2 illustrates the proposed framework.

Vector (PDF)

The Teacher Stores Applied Updates

Muon keeps a momentum buffer as large as each hidden matrix, while the EMA-teacher already stores a history of the same trajectory. If this history can take the role of momentum, the buffer becomes unnecessary. We therefore first characterize what the teacher stores and why it cannot simply replace momentum. Let $U^{(s)}$ denote the applied update direction at step $s$, with $W_S^{(s+1)} =$ $W_S^{(s)}-\eta c_{A,B}U^{(s)}$, assuming constant learning rate $\eta$ and zero weight decay. Unrolling the EMA update in Eq. (3), initialized with $W_T^{(0)}=W_S^{(0)}$, gives (Lemma 1)

Thus, the teacher lags behind the student, with its displacement encoding a discounted history of applied update directions. Unlike Muon’s momentum, which accumulates raw gradients, this history contains optimizer outputs that have already been orthogonalized under standard Muon. Directly using this displacement as momentum would therefore mix historical updates with current gradients, potentially reinforcing stale directions as the self-distillation target evolves. Moreover, combining both signals before Newton–Schulz entangles their contributions through nonlinear orthogonalization, making their relative influence difficult to control. Muon-RT instead processes the two signals separately and combines them after normalization, explicitly controlling the historical contribution through a mixing weight.

The Muon-RT Update

Gradient branch. We eliminate Muon’s momentum buffer and orthogonalize the current student gradient $G=\nabla_{W_S}\mathcal{L}(\theta_S)$:

This corresponds to Muon with $\mu=0$. Since the EMA-teacher and its distillation targets evolve during training, accumulated gradients may lag behind the current objective (§ 4.5).

Teacher branch. Let $D=W_T-W_S$ denote the teacher–student displacement. We remove its component parallel to the student weights:

This suppresses radial changes associated with parameter scaling, including weight decay, while retaining the remaining trajectory information. We normalize the residual by its estimated largest singular value:

This brings its spectral norm approximately to one, comparable to the orthogonalized gradient $O$. We estimate $\sigma_{\max}$ using KPI power iterations, warm-started from the previous right singular vector, retaining only one vector of length $B$ per matrix.

Mixing after orthogonalization. We combine the orthogonalized gradient and normalized teacher directions:

Algorithm 1 One optimization step of Muon-RT

Require: Student $\theta_S$, EMA-teacher $\theta_T$, loss $\mathcal{L}$, mixing weight $\alpha$, EMA decay $\tau$, learning rate $\eta$, weight decay

1. Compute student gradients $\nabla_{\theta_S}\mathcal{L}$

2. for each Muon-optimized hidden matrix $\ell$ do

3. $G_\ell\leftarrow\nabla_{W_{S,\ell}}\mathcal{L}$

4. $O_\ell\leftarrow\operatorname{NSK}(G_\ell)$

5. $D_\ell\leftarrow W_{T,\ell} - W_{S,\ell}$

6. Compute $H_\ell$ using Eq. (9)

7. Estimate $\widehat{\sigma}_{\max}(H_\ell)$ by power iteration

8. Compute $P_\ell$ using Eq. (10)

9. $U_\ell\leftarrow(1-\alpha)O_\ell+ \alpha P_\ell$

10. Update $W_{S,\ell}$ in place via Eq. (12)

11. Release temporary matrix buffers

12. end for

13. Update remaining student parameters with AdamW

14. $\theta_T \leftarrow\tau\theta_T + (1-\tau)\theta_S$

Here, $\alpha\in[0,1]$ controls the teacher contribution, with $\lVert\alpha P\rVert_2 \le\alpha$ under exact spectral normalization and an approximate bound with power iteration. Setting $\alpha= 0$ recovers momentum-free Muon. The EMA-teacher is updated afterward using Eq. (3).

Memory and Sharded Implementation

Muon-RT reduces the persistent states of each Muon-optimized hidden matrix from $\{W_S,W_T,M\}$ to $\{W_S,W_T\}$, eliminating the momentum buffer. Including gradient storage $G$, this reduces the matrix-sized tensors from 4 to 3. Importantly, $O$, $D$, $H$, $P$, and $U$ are temporary computational tensors, not additional parameters or persistent states. As summarized in Algorithm 1, they are computed one matrix at a time, with buffers reused or released after updating $W_S$ in place. Only a power-iteration vector of length $B$ per matrix is retained for warm-starting. Thus, the temporary workspace scales with the processed matrix size rather than requiring additional model-wide buffers. We report measured persistent memory in § 4.4 and detailed accounting in Appendix § 8.§ 6.

We use Fully Sharded Data Parallel (FSDP) ((Zhao et al., 2023)), with matching sharding layouts for student and EMA-teacher parameters. The displacement $D = W_T - W_S$ is computed locally, while the Frobenius inner products for radial projection are all-reduced across ranks. Full matrices $G$ and $H$ are temporarily gathered for Newton–Schulz orthogonalization and power iteration, respectively. Each rank then updates only its corresponding shard of $W_S$, followed by the sharded EMA update. With consistent global reductions and power-iteration initialization, the sharded implementation matches the unsharded computation up to numerical differences (Appendix § 8.§ 4).

Theoretical Analysis

We now give the guarantees behind Muon-RT; Appendix § 8.§ 2 contains the proofs and the full assumptions. All results hold for any differentiable host loss with a parameter-aligned EMA-teacher.

Lemma 2 shows that the teacher holds a smoothed estimate of persistent motion, with the noise variance reduced by a factor of up to $(1-\tau)/(1+\tau)$. The closer $\tau$ is to 1, the more stable the teacher direction is than any single step.

Proposition and Corollary in the Appendix § 8.§ 1 further shows where gain must come from and when the reuse of the teacher parameters helps.

Experiments

We evaluate Muon-RT as a drop-in optimizer for two self-distillation hosts: fine-tuning with SDFT and on-policy post-training with SDPO. Our evaluation covers task accuracy, retention of prior capabilities, convergence speed and memory. Finally, ablations isolate each design choice, and we test the theory of § 3.4 against empirical measurements.

Experimental Setup

Tasks and data. For fine-tuning with self-distillation, we follow SDFT (Shenfeld et al., [42]) on Qwen2.5-7B-Instruct (Yang et al., 2024a) with its two tasks: tool use from ToolAlpaca (Tang et al., [44]) and science questions from SciKnowEval (Feng et al., [14]). For on-policy post-training, we follow SDPO (Hübottter et al., [17]) on Olmo-3-7B-Instruct (Ettinger et al., [13]) with the biology and materials subsets of SciKnowEval. In both settings, we use the default splits and metrics of the original papers.

Baselines. For SDFT, we compare with SFT, SFT with re-invoked prior knowledge, DFT (Wu et al., [50]) and SDFT with AdamW, as reported by Shenfeld et al. [42]. We additionally run SDFT with Muon and with MuonClip, which adds the query–key logit clipping (QK-Clip) of Kimi K2 (Bai et al., [3]). Both Muon variant use $\mu=0$, their stronger setting; momentum $\mu=0.95$ is evaluated as an ablation (Table 3). For SDPO, we compare with Group Relative Policy Optimization (GRPO; Shao et al., [39]), on-policy GRPO and SDPO with AdamW, as reported by Hübottter et al. [17], and run SDPO with Muon, MuonClip and Muon-RT. The training and evaluation details are provide in the Appendix § 8.9 and Appendix § 8.§ 5 lists all hyperparameter settings.

Variant	Tool use	Science
(a) Optimizer state
Muon ($\mu=0.95$)	66.9	68.3
MuonClip ($\mu=0.95$)	67.0	68.8
Muon ($\mu=0$)	69.1	69.0
MuonClip ($\mu=0$)	73.2	71.0
Muon-RT (ours)	74.2	71.8
(b) Teacher weight $\alpha$
$\alpha=0$ (Muon, $\mu=0$)	69.1	69.0
$\alpha=0.05$	67.0	66.5
$\alpha=0.1$ (ours)	74.2	71.8
$\alpha=0.2$	68.0	66.8
$\alpha=0.4$	67.0	65.4
(c) Teacher branch
Muon-RT (ours)	74.2	71.8
Mix before Newton–Schulz	72.5	69.2
Frobenius instead of spectral norm	71.8	68.5
Keep radial part	70.5	67.3
Cosine gate on $\alpha$	69.2	66.8
(d) Logit clipping
Muon-RT (ours)	74.2	71.8
Muon-RT + QK-Clip (student only)	66.0	70.6
Muon-RT + QK-Clip (student & teacher)	67.3	70.1

Fine-Tuning with Self-Distillation

Task accuracy. Table 1 shows that Muon-RT achieves the highest new-task accuracy on both tasks: 74.2% on tool use and 71.8% on science, compared with 70.6% and 70.2% for SDFT with AdamW. It improves on momentum-free Muon by 5.1 and 2.8 points, respectively. The two methods share the same 32-bit teacher and differ only in whether the teacher branch is reused, so we attribute this gain to teacher reuse. Muon-RT also outperforms MuonClip without any logit clipping. Across both host models, Muon-RT ranks first among the Muon variants in all six task and budget settings (Table 1 and Table 2).

Task	Method	Optimizer	W/O CL Accuracy	HS	HE	IFE	MMLU	TQA	Wino	Avg.
Science	Base model$^{\dagger}$	–	32.1	62.0	65.8	74.3	71.7	47.9	71.1	65.5
Science	SFT$^{\dagger}$	AdamW	66.2	55.0	54.8	35.3	64.6	36.8	73.7	53.4
Science	SFT + re-invoke$^{\dagger}$	AdamW	66.0	61.6	63.4	52.9	68.7	45.2	70.0	60.3
Science	DFT$^{\dagger}$	AdamW	54.8	57.6	67.0	60.4	69.4	38.8	68.2	60.2
Science	SDFT$^{\dagger}$	AdamW	70.2	60.9	<u>68.9</u>	66.8	70.7	46.5	<u>73.1</u>	64.5
Science	SDFT	Muon ($\mu=0$)	69.0	61.5	68.3	67.9	<u>71.6</u>	47.7	71.9	64.8
Science	SDFT	MuonClip ($\mu=0$)	<u>71.0</u>	<u>61.6</u>	67.1	<u>72.4</u>	71.7	47.5	72.1	<u>65.4</u>
Science	SDFT	Muon-RT (ours)	71.8	61.2	70.7	69.3	<u>71.6</u>	46.5	72.6	65.3
Tool use	Base model$^{\dagger}$	–	42.9	62.0	65.8	74.3	71.7	47.9	71.1	65.5
Tool use	SFT$^{\dagger}$	AdamW	63.2	57.3	50.0	49.8	70.2	37.5	73.1	56.3
Tool use	SFT + re-invoke$^{\dagger}$	AdamW	63.1	<u>61.7</u>	68.9	59.1	71.5	49.1	71.6	63.7
Tool use	DFT$^{\dagger}$	AdamW	64.2	59.7	61.4	60.2	<u>71.6</u>	40.2	71.5	60.8
Tool use	SDFT$^{\dagger}$	AdamW	70.6	61.6	<u>68.3</u>	<u>71.9</u>	71.5	47.3	71.7	<u>65.4</u>
Tool use	SDFT	Muon ($\mu=0$)	69.1	61.4	65.2	69.3	71.5	47.0	71.5	64.3
Tool use	SDFT	MuonClip ($\mu=0$)	<u>73.2</u>	61.5	61.0	69.5	<u>71.6</u>	46.0	71.6	63.5
Tool use	SDFT	Muon-RT (ours)	74.2	61.1	63.4	63.7	71.4	46.1	<u>72.3</u>	63.0

Method	Optimizer	1 h	5 h
Base model†	–		16.2
GRPO†	AdamW	35.6	55.8
GRPO (on-policy)†	AdamW	49.8	49.8
SDPO†	AdamW	48.0	52.8
SDPO	Muon ($\mu=0$)	48.7	53.1
SDPO	MuonClip ($\mu=0$)	48.5	54.4
SDPO	Muon-RT (ours)	<u>49.1</u>	<u>55.2</u>
Base model†	–		36.7
GRPO†	AdamW	70.9	75.0
GRPO (on-policy)†	AdamW	73.3	73.5
SDPO†	AdamW	73.7	79.1
SDPO	Muon ($\mu=0$)	73.2	81.3
SDPO	MuonClip ($\mu=0$)	73.8	<u>81.5</u>
SDPO	Muon-RT (ours)	74.4	82.8

Prior capabilities. Muon-RT changes only the optimizer, so the question is whether its faster learning comes at the cost of prior capabilities. Retention is set mainly by the host loss: the prior-capability averages of all self-distillation runs lie within 2.4 points of one another, and those of Muon-RT are 11.9 and 6.7 points above SFT (Table 1). On science, Muon-RT has the best HumanEval score (70.7) and a higher average than SDFT with AdamW (65.3 against 64.5). On tool use, where Muon-RT gains the most on the new task, all Muon variants average 1.1–2.4 points below AdamW, mostly on IFEval. Even so Muon-RT retains 96% of the AdamW average (63.0 against 65.4). Muon-RT thus improves new-task accuracy through optimization alone while preserving the retention advantage of self-distillation over SFT.

Convergence. Figure 3(a)–(Figure 3c) shows that Muon-RT learns faster (25-step averages for the KL). On tool use, it reaches 74.2% after 100 of 252 steps, whereas MuonClip needs 250 steps to reach 73.2% and Muon does not exceed 69.1%. On science, it leads at 50 steps and ends highest. At the same learning rate, its self-distillation KL is lower than that of both baselines for most of training, as Corollary 1 predicts when the teacher branch raises the effective descent coefficient.

Vector (PDF)

On-Policy Post-Training

It is evident from Table 2, Muon-RT is the most accurate optimizer for SDPO on both datasets and budgets, 2.0 points above AdamW on average. Its advantage grows with training time: its margin over SDPO with AdamW is 0.7–1.1 points after 1 hour and widens to 2.4–3.7 points after 5 hours. This trend is consistent with the gain arising from information accumulated over the optimization trajectory rather than from any single step update. On Materials, Muon-RT outperforms all baselines (82.8% after 5 hours). On Biology it is second only to GRPO, which uses a different training signal.

Memory and Compute

Figure 3(d) reports persistent memory per GPU, measured between steps when gradients are freed, with every optimizer using a 32-bit EMA-teacher. Because Muon-RT keeps no optimizer state for hidden matrices, it needs only 37.7 GB, 13.9 GB (27%) less than Muon with its standard momentum $\mu=0.95$ and 26.8 GB (42%) less than AdamW. Momentum-free Muon and MuonClip need the same 37.7 GB but are 0.8–5.1 points less accurate (Table 1): Muon-RT therefore obtains the benefit of momentum without storing a momentum buffer. At 10T parameters, Muon-RT needs 121 TB, 25% less than Muon or MuonClip with momentum (160 TB) and 40% less than AdamW (200 TB; Figure 5, Appendix § 8.§ 6). The additional cost is one power iteration and one extra gather per matrix. The optimizer step takes 2.3 s, about 2% of a training step dominated by generation.

Vector (PDF)

Ablations and Analysis

We ablate each component of Muon-RT with SDFT on both tasks (Table 3): the optimizer state it replaces, the teacher weight $\alpha$, the construction of the teacher branch, and its combination with logit clipping. All ablations follow the setting of Table 1, with $\alpha=0.1$ unless varied, and every block includes the full method as reference. The Appendix A.11 provides the detailed discussion about the ablation provide in the Table 1.

Empirical Checks of the Theory

Figure 4 tests each theoretical statement with a matching measurement. The details are discussed in the Appendix A.10 where we empirically justify each theoretical claims.

Vector (PDF)

Related Work

Training on the model’s own samples against a teacher improves adaptation (Agarwal et al., [1]; Ko et al., [23]), with further advances in

self-distillation [48] (Xu et al. 2024). Such approaches can reduce forgetting relative to SFT [41, 7] (Yang et al. 2024b), which often compromises prior capabilities [4, 43] (Zheng et al. 2025a). In particular, SDFT [42] and SDPO [17] use privileged context with an EMA-teacher. These methods redesign the objective but retain standard optimizers, keeping 2 model-sized histories without exploiting their interaction. We preserve their objectives and modify only the optimizer, improving accuracy with both hosts.

Muon [20] brings orthogonalized updates to LLMs [3] (Liu et al. 2025), supported by analyses of matrix updates [2, 24] and of its convergence [26, 40]. Other optimizers include SOAP [47], Sophia [27] and Lion [8]. Memory-efficient methods use low-rank projections [16] (Zhao et al. 2024; Zhu et al. 2025), quantized or shared states [25, 12] (Zhang et al. 2025), block updates [29, 36], or fused updates [30, 32]. Others introduce slow averages [35] or examine optimizer tuning (Zhao et al., 2025). These approaches redesign the optimizer’s own state rather than reuse model copies already maintained during training. Muon-RT eliminates Muon’s momentum state, saving 27% of the measured memory at 7B and 25% of the projected memory at 10T while improving accuracy.

EMA-teachers provide stable learning targets [45, 15, 6], extending to representation learning [34] and EMA scaling [5, 33]. Weight averaging also improves optimization and model selection [37, 18, 19, 38, 10] (Zhang et al. 2019), and momentum itself can be written as an average of iterates [9] or mixed with the current gradient [31]. These approaches treat the average as a model, a target or a dedicated buffer, without characterizing the information encoded by its displacement from current weights. We show that this gap stores applied updates, averages out noise, and provides a reusable optimized direction.

Conclusion

We asked whether the EMA-teacher of self-distillation can replace Muon’s momentum buffer. We showed that the teacher–student displacement is a discounted sum of applied updates (Lemma 1) that averages out their noise (Lemma 2), while the orthogonalized gradient is already optimal for a single step (Proposition 1); the teacher thus adds temporal information rather than a better instantaneous direction. On this basis, Muon-RT drops the momentum buffer and adds the radial-free, spectrally normalized displacement after orthogonalization with a bounded weight, with linear convergence under the PL condition. Empirically, it reached the highest accuracy among all optimizers with both SDFT and SDPO, converged faster with SDFT, retained prior capabilities well above SFT and reduced memory. The ablations show that gradient momentum hurts when the target moves, whereas the teacher’s history helps when it enters after orthogonalization and stays consistent with the student. Any method that keeps a parameter-aligned EMA model can lend it to the optimizer.

Limitations

Our experiments cover 7B-parameter models from 2 families and 4 tasks; larger models and further domains are left for future work. The convergence analysis relies on smoothness, PL and alignment assumptions that are standard for non-convex optimization but whose constants are not estimated for LLMs. Muon-RT applies when a parameter-aligned EMA-teacher already exists, as in SDFT and SDPO; for methods without one, a teacher kept only to replace momentum would not save memory.

Ethics Statement

This work changes how a model is optimized, not what it is trained on, and it uses public datasets (ToolAlpaca, SciKnowEval) under their released licenses and splits. Models trained on science questions may still produce wrong answers about chemical, biological or material properties and should not be used for laboratory or safety decisions without expert checks. Reducing optimizer memory lowers the hardware needed for continual adaptation; we report our compute in Appendix § 8.§ 5. AI assistants were used only for language and writing refinement; the authors checked all content and remain responsible for it.

References

Appendix

Theoretical Analysis

This section provides remaining theoretical analysis. The below proposition shows where this gain must come from.

So the Newton–Schulz direction is already optimal for the current gradient, and the teacher branch contributes what a single gradient cannot: temporal information. To make this precise, write $F_s$ for the expected host loss at step $s$ (it moves with the teacher), $F_s^* = \inf F_s$ and $\Delta_s = F_s(W_S^{(s)}) - F_s^*$. We assume (A1) $L$-smoothness, (A2) the PL condition $\frac{1}{2}\lVert\nabla F_s\rVert_{\mathrm{F}}^2 \ge\mu_{\mathrm{PL}}(F_s-F_s^*)$ ([21]), (A3) bounded teacher drift $\sup_W\lvert F_{s+1}-F_s\rvert\le\delta_s$, and (A4) alignment $\mathbb{E}\langle g^{(s)}, U^{(s)}\rangle\ge a_R\lVert g^{(s)}\rVert^2$ with second moment $\mathbb{E}\lVert U^{(s)}\rVert^2 \le b_R\lVert g^{(s)}\rVert^2+\sigma_R^2$, where $g^{(s)}=\nabla F_s(W_S^{(s)})$.

The below corollary further shows when reuse of the teacher parameters are faster:

Corollary 1 shows that reuse helps when the teacher branch raises the effective descent coefficient, i.e. adds alignment with the true gradient over time without inflating the update’s second moment.

Theoretical Analysis: Proofs and Convergence

This appendix proves the results stated in § 3.4 and gives the full assumptions behind them. Notation follows § 3.

Proof of Lemma 1

Proof. Define the displacement

Using the EMA update, Eq. (3),

where the last equality follows from the student update of Lemma 1. Since $D^{(0)}=0$, recursively expanding Eq. (15) gives

which is Eq. (7).

Proof of Lemma 2

Proof. Substituting the local model $\mathbf{U}^{(s)}=\mathbf{d}+\boldsymbol{\xi}^{(s)}$ of Lemma 2 into the definition of $\overline{\mathbf{U}}^{(s)}$,

Since

taking expectations gives the stated mean. After subtracting the mean,

Therefore,

All cross terms vanish by temporal uncorrelatedness, yielding

which proves the variance statement of Lemma 2. Letting $s\to\infty$ gives the stated limit. $\square$

Remark 1. Lemma 2 applies to the raw, rescaled EMA trajectory before the radial projection and spectral normalization in Muon-RT. Since these operations are nonlinear, the factor $(1-\tau)/(1+\tau)$ should not be interpreted as an exact variance reduction for the final normalized direction $\mathbf{P}^{(s)}$. Moreover, for $\alpha>0$ the applied direction $\mathbf{U}^{(s)}$ itself contains $\mathbf{P}^{(s)}$, so its noise is not exactly temporally uncorrelated; the lemma is exact for $\alpha=0$ and a small-$\alpha$ approximation otherwise. The lemma instead establishes that the state from which $\mathbf{P}^{(s)}$ is constructed already contains a temporally smoothed estimate of persistent parameter motion.

Instantaneous alignment is not sufficient to explain the gain

The following proposition is useful for interpreting the role of the EMA branch.

Restatement (Proposition 1). Let $\mathbf{G} = \mathbf{L}_{G}\mathbf{\Sigma}\mathbf{V}_{G}^{\top}$ be a compact singular value decomposition of $\mathbf{G} \ne0$ and define its exact polar factor

Then

and

where $\lVert\cdot\rVert_{*}$ denotes the nuclear norm. Therefore, for any spectrally normalized teacher direction $\mathbf{P}$ satisfying $\lVert\mathbf{P}\rVert_{2}\le1$,

Proof. By the duality between the nuclear and spectral norms,

For $\mathbf{Q} = \mathbf{L}_{G}\mathbf{V}_{G}^{\top}$, $\lVert\mathbf{Q}\rVert_{2}=1$, and

Thus $\mathbf{Q}$ attains the spectral-norm-constrained maximum, proving Eqs. (23)–(24). Since $\lVert\mathbf{P}\rVert_{2}\le1$, Eq. (25) follows immediately.

Remark 2. Proposition 1 rules out an overly strong interpretation of Muon-RT. In the idealized deterministic setting with an exact polar factor, the teacher branch cannot improve the current gradient’s linear alignment beyond the polar direction while remaining inside the same spectral-norm ball. Any optimization advantage must therefore arise from temporal information, stochastic denoising, finite Newton–Schulz approximation, curvature-dependent second-order effects, or a combination of these factors.

Finite-time convergence of Muon-RT

We now analyze the complete mixed direction of (11), written $\mathbf{U}_{R}^{(s)}$ to distinguish it from baseline directions,

where $\mathbf{O}^{(s)}$ is the momentum-free Muon branch and $\mathbf{P}^{(s)}$ is the normalized EMA-trajectory branch.

The host objective may itself move because the teacher changes. With $\mathbf{W}$ denoting the Muon-eligible matrix under analysis, we therefore write the expected host objective as

with

We use the following standard local conditions over the region visited during training.

A1. Smoothness. Every $F_s$ is $L$-smooth with respect to the Frobenius norm:

A2. PL condition. There exists $\mu_{\mathrm{PL}}>0$ such that

for all iterates under consideration.

Controlled teacher-induced objective drift

There exists $\delta_s \ge0$ such that

Descent alignment and second moment

Let

Conditioned on the optimization history $\mathcal{F}_s$, the Muon-RT direction satisfies

and

The constants in A4 have a direct interpretation. $a_R$ measures expected alignment with the current true descent direction, $b_R$ measures how much update energy grows with the gradient, and $\sigma_R^2$ collects residual update energy that does not scale with the gradient. Because $\mathbf{O}^{(s)}$ and $\mathbf{P}^{(s)}$ are normalized, this residual includes their gradient-independent magnitude as well as stochastic noise, and $a_R$ is understood over a region in which $\lVert\mathbf{g}^{(s)} \rVert_F$ is bounded. A conservative set of valid coefficients can be obtained from separate branch bounds. In particular, if

and

where $a_P$ may be negative, then linearity of the inner product and convexity of the squared norm imply

These are conservative because they do not exploit any favorable cancellation between $\mathbf{O}^{(s)}$ and $\mathbf{P}^{(s)}$.

If the objective drift is uniformly bounded by $\delta_s \le\bar{\delta}$, then

In particular,

For a fixed objective, $\bar{\delta}=0$.

Proof. By $L$-smoothness of $F_s$ and Eq. (43),

Conditioning on $\mathcal{F}_s$ and applying Eqs. (35) and (36),

Using the PL inequality in Eq. (32),

Substituting this into Eq. (52) gives

It remains to account for the fact that the EMA-teacher changes the host objective between steps. From A3,

Moreover,

Combining Eqs. (55) and (56),

Taking total expectation in Eq. (54) and applying Eq. (57) yields Eq. (46).

Repeatedly applying the recursion gives Eq. (48). When $\delta_s \le\bar{\delta}$, the geometric-series identity

gives Eq. (49). Finally, using

and taking $n \to\infty$ yields Eq. (50).

Comparison with a baseline optimizer

The previous result immediately gives a sufficient condition for a faster convergence guarantee than AdamW, momentum Muon, or momentum-free Muon, provided the corresponding baseline satisfies the same local regularity assumptions.

Proof. (61) implies

and therefore

Hence the geometric transient $q_R^n$ decays more rapidly than $q_B^n$. The second statement follows directly from Theorem 1: $\Phi_R$ and $\Phi_B$ are the respective asymptotic upper bounds obtained by taking the limit superior of each finite-time recursion. If $\Phi_R < \Phi_B$, the Muon-RT bound approaches a smaller optimization neighborhood.

For the fixed-objective case $\bar{\delta}_R = \bar{\delta}_B = 0$, Eq. (65) reduces to

Thus the relevant quantity is not variance alone, but the ratio between residual update energy and effective descent. This formalizes the optimization mechanism suggested by Lemma 2: a temporally smoothed trajectory is useful when the reduction in update noise is large enough relative to any loss of instantaneous descent alignment.

Iteration complexity

Because the branches are normalized, $\sigma_R^2 > 0$ and the bound converges to the neighborhood $\Phi_R$ rather than to zero. From Eq. (49),

To reach $\mathbb{E}[\Delta_n] \le\Phi_R + \epsilon_{\mathrm{tol}}$, it is sufficient that

For small $2\mu_{\mathrm{PL}}\eta\gamma_R$, $-\log q_R \approx2\mu_{\mathrm{PL}}\eta\gamma_R$, giving

Accordingly, the local iteration-speedup predicted relative to a baseline, each measured to within $\epsilon_{\mathrm{tol}}$ of its own neighborhood, is approximately

Equation (72) concerns optimization steps rather than wall-clock time; the latter additionally depends on the per-step cost of each optimizer.

Remark 3 (What the theorem does and does not establish). Corollary 1 supports two claims under explicit conditions: faster finite-time contraction and a lower optimization-error bound. It does not claim that Muon-RT universally reaches a better local minimum in an arbitrary non-convex neural-network landscape. Higher downstream accuracy can be consistent with the lower error bound and the temporal smoothing of the raw trajectory discussed above, but the mapping from optimization loss to task accuracy is data- and objective-dependent and is therefore verified empirically rather than assumed by the theorem.

Teacher Precision

Muon-RT reads the teacher–student displacement $\mathbf{W}^{(s)}_T-\mathbf{W}^{(s)}_S$, so the teacher must actually follow the EMA of Eq. (3). With $\tau=0.99$, each step moves the teacher by 1% of the displacement, and the displacement is only 0.1–0.3% of the weights (Appendix § 8.7). The per-step change is therefore about $10^{-5}$ of a weight’s magnitude, while bfloat16 can only represent relative changes larger than about $4 \times10^{-3}$ (8 significand bits). In 16-bit storage most EMA steps round to zero. We measured this on a Qwen2.5-7B-Instruct key projection matrix with 250 momentum-free Muon steps (Figure 6(a); setup in Appendix § 8.7): the student moved by 2.15 in Frobenius norm, a 32-bit teacher by 1.37, and a bfloat16 teacher by only 0.66, with 79% of its entries never changing. A 16-bit displacement would thus reflect rounding rather than the averaged trajectory that Lemmas 1 and 2 describe. For the same reason, optimizer states such as Muon’s momentum and AdamW’s moments are usually kept in 32 bits ([25] (Liu et al. 2025)); since the teacher takes over the role of optimizer state, we store it in 32 bits as well, and we do so for every optimizer we train in Table 1–Table 3, so that comparisons share the same teacher. The teacher’s forward pass still runs in bfloat16, so the self-distillation loss is computed as in SDFT. The 32-bit teacher adds 2 bytes per parameter (7.6 GB per GPU with 2 GPUs); the memory in Figure 3(d) includes this cost.

Vector (PDF)

Precision setting. Our precision differs from the original SDFT setting, which keeps the student and the teacher in bfloat16. To make sure that the gains of the Muon variants come neither from precision nor from run-to-run randomness, we train every optimizer in the same setting: a bfloat16 mixed-precision student with 32-bit master weights and a teacher stored in 32 bits, with forward passes in bfloat16. In this setting, SDFT and SDPO with AdamW reproduce their reported accuracy within run-to-run variation, so the AdamW rows taken from Shenfeld et al. [42] and Hübötter et al. [17] are comparable with our runs. All memory values in the paper are for a 32-bit teacher (Appendix § 8.§ 6).

Implementation Details

We build on the released SDFT code (TRL with colocated vLLM generation) and replace the optimizer. Hidden matrices are all two-dimensional weights inside the transformer blocks; embeddings, the output head, normalization weights and query/key/value biases use AdamW with the same learning rate, following Liu et al. (2025). Student and teacher are sharded with FSDP2 using the same wrapping policy, so each teacher shard matches a student shard. For every hidden matrix, the inner products in Eq. (9) and the norms are reduced across shards; the gradient (in bfloat16) and $H^{(s)}$ are gathered, Newton–Schulz and power iteration run on the full matrix on every rank, and each rank keeps its slice of $U^{(s)}$. The power-iteration vector of each matrix is stored across steps (1 vector of length $B$, 4.5 MB in total for Qwen2.5-7B); starting from a random vector, 8 iterations underestimate $\sigma_{\max}$ by about 8%, and the warm start removes this error within 3 steps. We verified on a small model with 2 GPUs that (i) $\alpha=0$ is bit-identical to Muon with $\mu=0$, (ii) sharded and unsharded updates are identical, and (iii) for a single matrix the update matches an exact singular-value reference to $5 \times10^{-7}$, with $\lVert P^{(s)}\rVert_2=1$.

Hyperparameters and Compute

Setting	Value
Model	Qwen2.5-7B-Instruct
Hardware	$2 \times$H100 80 GB, FSDP2
Prompts per step	32 (1 rollout each)
Epochs (steps)	2 (252 tool use, 166 science)
Learning rate	$\{3,5\}\times10^{-5}$, cosine
Warmup	10% of steps
Max. prompt / completion	1024 / 1024 tokens
Loss	forward KL, first 3 tokens masked
Student storage / compute	32 / 16 bits (bfloat16)
Teacher storage / forward	32 / 16 bits (bfloat16)
EMA rate $1-\tau$	0.01
Grad. clip / weight decay	1.0 / 0
Newton–Schulz steps $K$ / coefficients $(\kappa_1,\kappa_2,\kappa_3)$	5 / (3.4445, −4.7750, 2.0315)
Update scale $c_{A,B}$	$0.2\sqrt{\max(A,B)}$
Muon-RT: $\alpha$ / power iterations $K_{\mathrm{PI}}$	0.1 / 8 (warm-started)
AdamW (other params.)	(0.9, 0.999), $10^{-8}$
Evaluation	greedy, every 50 steps

Table 4 lists the SDFT settings. Following Shenfeld et al. [42] and Hübotter et al. [17], we ran every configuration with 3 random seeds. A SDFT run takes 1.1–1.8 hours on 2 H100 GPUs, dominated by generation. For SDPO (Table 5), the model runs without thinking and every setting is identical to the experiments of Hübotter et al. [17] without feedback on Olmo-3-7B-Instruct: model, data splits, rollout and validation settings, loss, teacher EMA rate, schedule, weight decay and gradient clipping. Only the optimizer of the hidden matrices and its learning rate change. As for SDFT, Muon, MuonClip and Muon-RT are trained with learning rates $5 \times10^{-5}$ and $3 \times10^{-5}$, and we report the better run; the AdamW rows are taken from Hübotter et al. [17]. Accuracy is the best avg@16 reached within 1 and 5 hours of wall-clock training, excluding initialization and validation, as in Hübotter et al. [17].

Setting	Value
Model	Olmo-3-7B-Instruct
Data	SciKnowEval (2 subsets)
Max. prompt / response	2048 / 8192
Questions / mini-batch	32 / 32
Rollouts / temperature	8 / 1.0
Validation / temperature / top-$p$	avg@16 / 0.6 / 0.95
Loss / top-$k$	Jensen–Shannon / 100
EMA rate $1-\tau$	0.05
Importance-sampling clip	2
Schedule / warmup	constant / 10 steps
Weight decay / grad. clip	0.01 / 1.0
Student storage / compute	32 / 16 bits (bfloat16)
Teacher storage / forward	32 / 16 bits (bfloat16)
LR, AdamW	$1 \times10^{-5}$
LR, Muon / MuonClip / ours	$\{3,5\} \times10^{-5}$
Muon, MuonClip	$\mu=0$
Muon-RT: $\alpha$ / $K_{\mathrm{PI}}$	0.1 / 8

Memory Projection

Figure 5 projects the persistent model state of self-distillation for a model with $N=10^{13}$ parameters. Per parameter, all optimizers store the student, its gradient and the EMA-teacher in 32 bits (12 bytes); every state uses the same precision, so no optimizer is favored. AdamW adds 2 32-bit moments (8 bytes). Muon and MuonClip, in their standard form with $\mu=0.95$, add a 32-bit momentum (4 bytes) on hidden matrices and AdamW states on the remaining parameters; the per-head statistics of MuonClip are negligible. Muon-RT adds no state on hidden matrices and AdamW states on the rest. We take hidden matrices as 99% of the parameters, since embeddings and output head take about 0.2% of the parameters in a 1T model [3]. This gives 20, 16.04, 16.04 and 12.08 bytes per parameter, i.e. 200, 160, 160 and 121 TB. Activations and generation buffers are excluded. Our 7B measurements on 2 GPUs, all with a 32-bit teacher, are taken between steps, when gradients are freed; without the 4-byte gradient, the same accounting gives savings of 27% against Muon and 43% against AdamW, close to the measured 27% and 42%. Muon-RT, momentum-free Muon and momentum-free MuonClip need 37.7 GB per GPU. A Muon run that allocates the 32-bit momentum of 6.5B hidden parameters needs 51.6 GB, close to 37.7 + 13.1 = 50.8 GB. Storing the teacher in 32 instead of 16 bits adds 7.6 GB per GPU: Muon with $\mu=0.95$ and a 16-bit teacher needs 43.9 GB, and 43.9 + 7.6 = 51.5 GB is close to the measurement. AdamW with a 16-bit teacher needs 56.9 GB, so with a 32-bit teacher it needs 56.9 + 7.6 = 64.5 GB, the value in Figure 1 and Figure 3(d).

Learning-Rate Sensitivity and Training Diagnostics

Learning rate. Figure 6(d) shows that Muon-RT exceeds the best momentum-free Muon run (69.1%) for every learning rate from $2 \times10^{-5}$ to $6 \times10^{-5}$ (70.1–74.2%) and peaks at $3 \times10^{-5}$. Only at $10^{-5}$ and below do 2 epochs leave too few effective steps.

Teacher branch. We log, for every hidden matrix and step, $\lVert\mathbf{H}^{(s)} \rVert_{\mathrm{F}}/\lVert\mathbf{W}^{(s)} \rVert_{\mathrm{F}}$, $\sigma_{\max}(\mathbf{H}^{(s)})/\lVert\mathbf{H}^{(s)} \rVert_{\mathrm{F}}$, $\cos(\mathbf{G}^{(s)},\mathbf{H}^{(s)})$ and $\cos(\mathbf{O}^{(s)},\mathbf{P}^{(s)})$, averaged over matrices. The displacement grows during warmup, perhaps at 0.2–0.3% of the weights and shrinks as the learning rate decays (Figure 6(b)). Its largest singular value carries about 6% of its Frobenius norm on all runs, so $\mathbf{H}^{(s)}$ is spread over many directions. After the first 20 steps, the 15-step averages of both cosines stay below 0.015 in magnitude on all runs (Figure 6(c)): the teacher branch is nearly orthogonal to the current gradient and adds directions it does not cover.

Experiments on a Qwen2.5-7B-Instruct weight matrix. Figure 4(b, Figure 4c), Figure 6(a) and the constants below use the key projection matrix of layer 10 of Qwen2.5-7B-Instruct ($512 \times3584$). The student takes momentum-free Muon steps $W \leftarrow W-\eta c_{A,B}\operatorname{NS}_{K}(G^{(s)})$ with $\eta=5\times10^{-5}$ and $G^{(s)}=d+\nu E^{(s)}/\lVert E^{(s)}\rVert_{\mathrm{F}}$, where $d$ is a fixed random direction with $\lVert d\rVert_{\mathrm{F}}=1$, $E^{(s)}$ is fresh Gaussian noise and $\nu$ is the noise level; the teacher follows Eq. (3) with $\tau=0.99$. For Figure 4(b), we run 150 steps for each $\nu$ and average, over the last 20 steps, the cosine of $O^{(s)}$ and of $P^{(s)}$ with $\operatorname{NS}_{K}(d)$. For Figure 4(c), we take $H^{(s)}$ after 150 steps with $\nu=2$ and a gradient of low stable rank, $G=Y_{1}Y_{2}^{\top}$ with $Y_{1}\in\mathbb{R}^{A\times m}$, $Y_{2}\in\mathbb{R}^{B\times m}$ Gaussian, $m\in\{4,16,64\}$, plus 1% Gaussian noise. We compare $\lVert U-O\rVert_{\mathrm{F}}/\lVert O\rVert_{\mathrm{F}}$ for Eq. (11) and for the pre-Newton–Schulz mix $U=\operatorname{NS}_{K}\left((1-\alpha)G/\lVert G\rVert_{\mathrm{F}}+\alpha H^{(s)}/\lVert H^{(s)}\rVert_{\mathrm{F}}\right)$. For Figure 6(a), the teacher is either kept in 32 bits or rounded to bfloat16 after every EMA step.

Constants of Theorem 1. With the same matrix and gradients, the student follows Muon-RT ($\alpha=0.1$), so the teacher records Muon-RT’s own trajectory. Over steps 51–150 we average $a_X=\langle d,X\rangle_{\mathrm{F}}$ and $b_X=\lVert X\rVert_{\mathrm{F}}$ for $X\in\{O^{(s)},P^{(s)},U^{(s)}\}$, where $d$ is the mean gradient. For noise levels $\nu\in\{0.5,1,2,4,8,16\}$, $a_P/a_O$ is 1.10, 1.37, 1.97, 3.20, 5.03 and 6.62, so $a_R$ exceeds $a_B=a_O$ by 1%–56%, while $b_P<b_O$ and $b_R<b_O$ (e.g. 366 against 447 at $\nu=16$). Hence $\gamma_R>\gamma_B$ for every $L\eta$. The cosine between $G^{(s)}$ and $P^{(s)}$ falls from 0.87 to 0.03 as $\nu$ grows, while $a_P/a_O$ rises, so a small single-batch cosine does not indicate a weak teacher direction.

Teacher–student displacement without normalization. For Figure 4(a), the predicted ratio $\lVert W_T^{(s)}-W_S^{(s)}\rVert_{\mathrm{F}}/\lVert W^{(s)}\rVert_{\mathrm{F}}$ is $\tau\psi_s/\operatorname{rms}(W)$, where $\psi_s$ is $\sum_j\tau^j u_{s-1-j}$ (aligned updates) or $\left(\sum_j\tau^{2j}u_{s-1-j}^{2}\right)^{1/2}$ (uncorrelated updates), $u_s$ is the logged learning rate times the update root-mean-square, and $1/\operatorname{rms}(W)$ is averaged over the 196 hidden matrices. From step 20 on, the measured mean ratio is 1.2–1.7 times the uncorrelated value and 0.12–0.34 times the aligned value.

Lemma 1 range. For Figure 4(a), let $u_s=\eta_s\rho_s$ with the logged learning rate $\eta_s$ and update root-mean-square $\rho_s$. By Eq. (7) and the triangle inequality, $\lVert W_T^{(s)}-W_S^{(s)}\rVert_{\mathrm{F}}$ is proportional to at most $\sum_j\tau^j u_{s-1-j}$ (aligned updates) and, for uncorrelated updates, to $\left(\sum_j\tau^{2j}u_{s-1-j}^{2}\right)^{1/2}$. We normalize both and the measured $\lVert H^{(s)}\rVert_{\mathrm{F}}/\lVert W^{(s)}\rVert_{\mathrm{F}}$ by their maxima.

Attention Logits and QK-Clip

Growing attention logits can destabilize training with Muon, which motivated QK-Clip (Bai et al., [3]; Wortsman et al., [49]; Dehghan et al., 2023). We measured the maximum pre-softmax logit of every head (after rotary embeddings) on 24 evaluation prompts. The base Qwen2.5-7B-Instruct already reaches about 4,800, and 26–28 heads exceed the threshold of 100 used by Bai et al. [3]; these are attention-sink heads driven by the query and key biases (Xiao et al., 2024). After SDFT, the global maximum stays within $\pm1\%$ of the base value for AdamW, Muon and Muon-RT alike, the largest heads grow by at most 13–20%, and the norms of the query and key weights change by less than 0.05%. Clipping at 100 would therefore rescale about 27 heads that are large by design. Our MuonClip baseline instead uses per-head thresholds of $1.1\times$ the largest logit observed in the first 10 steps. Because logits do not grow, QK-Clip is not needed in this setting. With Muon-RT, it is also harmful (Table 3(d)). In the tool-use run with clipping, 8 clips occur, each on 1 head, with factors down to 0.56; right after the first clip (step 20, factor 0.90), the mean $\lVert H^{(s)}\rVert_{\mathrm{F}}/\lVert W^{(s)}\rVert_{\mathrm{F}}$ over all hidden matrices rises 12% above the run without clipping, after tracking it within 2% before. To isolate the effect, we take the query matrix of layer 10 of Qwen2.5-7B-Instruct, scale the 128 rows of 1 head in the student but not in the teacher, and form $H^{(s)}$ and $P^{(s)}$ as in Eqs. (9)–(10). A factor of 0.90 (0.56) creates a gap of 1.9% (8.0%) of the matrix norm, against 0.1–0.3% from learning; radial removal changes it by under 1.5%, 97% (99%) of $\lVert H^{(s)}\rVert_{\mathrm{F}}^{2}$ lies in the clipped head, and $-P^{(s)}$ is restricted to that head has cosine $-1.00$ with its weights. The update therefore shrinks the clipped head further at every step. The offset decays by $\tau$ per step, so a 1.9% gap falls below 0.2% only after about 225 steps. Scaling the same query rows of the teacher by the same factor at every clip keeps $W_T^{(s)}-W_S^{(s)}$ free of this offset; in Table 3(d), this variant reaches 67.3 on tool use and 70.1 on science.

Training and evaluation.

Only the optimizer changes; each host method keeps its loss, batch size and EMA settings same. For every optimizer, the student is trained in bfloat16 mixed precision (bfloat16 compute, 32-bit master weights), and the EMA-teacher is stored in 32 bits with its forward pass in bfloat16. Appendix § 8.§ 3 shows that this precision does not affect accuracy. We train Muon, MuonClip and Muon-RT ($\alpha=0.1$) with learning rates $5 \times10^{-5}$ and $3 \times10^{-5}$ and report the better run. Following Shenfeld et al. [42], we report the best checkpoint under greedy decoding and measure prior capabilities on HellaSwag, HumanEval, IFEval, MMLU, TruthfulQA, and Winogrande. For SDPO, we report the best avg@16 within 1 and 5 hours of training (Hübotter et al., [17]).

Empirical Checks of the Theory

Figure 4 tests each theoretical statement with a matching measurement.

Post-Newton–Schulz mixing bounds the teacher. Mixed before Newton–Schulz, even $\alpha=0.02$ changes the update by more than its own norm, because Newton–Schulz lifts the teacher’s weak directions to unit singular value (Figure 4(c)). Mixed after Newton–Schulz, as in Muon-RT, the change grows linearly with $\alpha$, as (11) guarantees.

Ablations and Analysis

We ablate each component of Muon-RT with SDFT on both tasks (Table 3): the optimizer state it replaces, the teacher weight $\alpha$, the construction of the teacher branch, and its combination with logit clipping. All ablations follow the setting of Table 1, with $\alpha=0.1$ unless varied, and every block includes the full method as reference.

Which history helps. Gradient history and teacher history have opposite effects (Table 3(a)). Standard momentum reduces accuracy by 0.7–6.2 points for both Muon and MuonClip on both tasks, whereas the teacher branch raises the accuracy of momentum-free Muon by 5.1 and 2.8 points. Orthogonalization normalizes each update, but with momentum consecutive updates share a direction for about $1/(1-\mu)=20$ steps and add up instead of partly cancelling, so the weights move further at the same learning rate. In addition, self-distillation targets move with the teacher at every step, so an average of past gradients points partly toward targets that no longer apply. The teacher branch, by contrast, records the updates that were actually applied (Lemma 1), and the bounded mix adds this history without letting it override the current gradient.

Teacher weight. $\alpha=0.1$ is the best value on both tasks (Table 3(b)). Both smaller and larger values reduce accuracy, confirming that the teacher should guide, not replace, the gradient direction (Eq. (11)). The same weight is optimal for both tasks and is used without modification for SDPO, so $\alpha$ requires no per-task tuning.

Teacher branch. Every modification to the teacher branch reduces accuracy (Table 3(c)), by 1.7–2.6 points for mixing before Newton–Schulz up to 5.0 points for a cosine gate. The gradient and displacement are nearly orthogonal ($|\cos| \approx0.004$), causing the cosine gate to effectively switch the teacher off. On the tool-use task, even the weaker variants outperforms the momentum-free Muon (70.5–72.5 Vs. 69.1), indicating that the teacher’s history is beneficial in every form; however, across both tasks, the bounded post-Newton–Schulz formulation extracts the most from it.

QK-Clip. Logit clipping helps Muon (69.1 to 73.2 on tool use) but hurts Muon-RT (Table 3(d); 71.1 to 66.0 at the same learning rate). QK-Clip rescales a head’s query weights in the student only, so each clip introduces an offset into $W^{(s)}_{T} - W^{(s)}_{S}$ that is not learning motion. On a real query matrix, a single clip by a factor 0.9 creates a gap 6–19 times larger than the learning displacement, this gap is retained by radial removal and accounts for 97% of $H^{(s)}$ (Appendix § 8.8). As a result, the teacher branch continues shrinking the clipped head at full weight $\alpha$ for over 200 steps; in training, the first clip raises the mean displacement over all matrices by 12% within a single step. Applying the same clip to the teacher removes this offset and recovers 1.3 points on tool use (67.3), but both tasks remain below Muon-RT without clipping (67.3 and 70.1): each clip also perturbs the teacher’s attention patterns and therefore the self-distillation target. In any case, clipping is unnecessary here, as logits stay within 1% of their initial maximum for every optimizer.