Introduction
The Muon optimizer has recently shown strong empirical performance for pretraining LLMs, challenging AdamW’s status as the standard optimizer for minimizing pretraining loss ([13 ] ). Prior work has shown that the pretraining optimization choices that minimize pretraining loss do not necessarily yield the best model after fine-tuning ([38 ] ). What ultimately matters downstream is the learning–forgetting tradeoff : how well a fine-tuned model learns the target task while retaining its pretrained knowledge. Motivated by this, we ask:
Does Muon’s pretraining advantage over AdamW extend to the learning–forgetting tradeoff?
To answer this question, we measure the Muon–AdamW fine-tuning gap , which compares the two optimizers on this tradeoff; a negative gap indicates that the Muon pretrained model dominates the AdamW pretrained model after fine-tuning, and a positive gap indicates the reverse. We show that the answer depends on the pretraining data budget, and find that pretraining data scale reverses the Muon–AdamW fine-tuning gap . At small budgets, Muon achieves lower fine-tuning loss with less forgetting than AdamW and the fine-tuning gap is negative. At larger budgets, this trend reverses: AdamW achieves a better learning–forgetting tradeoff than Muon, and the fine-tuning gap becomes positive.
In § 3 , we pretrain OLMo-style 30M–600M models ([11 , 34 ] ) on 150M to 154B DCLM tokens ([19 ] ), comparing AdamW and Muon, and fine-tune each base model on four datasets while sweeping the fine-tuning optimizer and learning rate, for nearly 600 pretraining runs and 29,000 fine-tuning runs in total. We show that when tuning pretraining hyperparameters for the lowest pretraining loss, Muon consistently achieves lower loss than AdamW. However, with sufficient pretraining data, AdamW outperforms Muon after fine-tuning, reversing the Muon–AdamW fine-tuning gap. This reversal also holds when we disregard pretraining loss, instead sweeping pretraining hyperparameters to obtain the best learning–forgetting tradeoff after fine-tuning. We additionally show that as model size scales, AdamW surpasses Muon at progressively smaller data budgets .
In § 4 , we investigate why the Muon–AdamW fine-tuning gap reverses as the pretraining data budget grows. Fine-tuning changes the output distribution, so we generalize fine-tuning updates with perturbations to the output distribution and study which optimizer produces base models that are more robust to such perturbations. To quantify sensitivity to output perturbations, we introduce the margin-to-spread ratio (MSR) , a simple metric derived from the model’s output logits that characterizes robustness. For each prediction, the MSR is the difference between the correct logit and the average of the incorrect logits, normalized by the spread of the incorrect logits (Definition 4.1 ): a prediction is more robust when the MSR is larger.
To understand where this trend comes from, we study a linear associative memory trained with gradient descent (GD) and Muon (§ 4.2 ). Analyzing a single update from zero initialization, we prove that the ratio of GD’s MSR to Muon’s increases with the data budget (Theorem 4.2 ). Simulations of many training steps in the linear associative memory setting confirm the same trend: Muon has the larger MSR at small data budgets, while GD has the larger MSR at large data budgets (§ 4.2 ). Measuring the MSR of our pretrained language models, we find that Muon has the larger MSR at small data budgets, while AdamW has the larger MSR at large data budgets, mirroring the reversal of the fine-tuning gap (§ 4.3 ).
Preliminaries
Our goal is to compare AdamW- and Muon-pretrained models after fine-tuning . Since fine-tuning can induce forgetting of the pretraining distribution, performance on the fine-tuning task alone is not sufficient: one base model may learn more but also forget more. We therefore compare base models by the tradeoff between the two, and define metrics to track how this comparison changes with model size and pretraining data budget. Let θ P T \theta_{\mathrm{PT}} θ PT denote a pretrained model.
Learning–forgetting tradeoff. Fine-tuning improves downstream performance but may degrade pretrained knowledge ([10 , 17 , 31 ] ). When θ P T \theta_{\mathrm{PT}} θ PT is fine-tuned to obtain θ F T \theta_{\mathrm{FT}} θ FT , we measure learning by the fine-tuning loss L F T ( θ F T ) \mathcal{L}_{\mathrm{FT}}(\theta_{\mathrm{FT}}) L FT ( θ FT ) and forgetting by the pretraining loss L P T ( θ F T ) \mathcal{L}_{\mathrm{PT}}(\theta_{\mathrm{FT}}) L PT ( θ FT ) . Since both depend on the fine-tuning hyperparameters, a single fine-tuned checkpoint does not capture how well θ P T \theta_{\mathrm{PT}} θ PT can be adapted. We therefore summarize each base model by the outcomes reachable by fine-tuning it,
T ( θ P T ) : = { ( L F T ( θ F T ) , L P T ( θ F T ) ) ∣ θ F T ∈ Θ F T ( θ P T ) } , \mathcal{T}(\theta_{\mathrm{PT}}) := \left\{ \left(\mathcal{L}_{\mathrm{FT}}(\theta_{\mathrm{FT}}), \mathcal{L}_{\mathrm{PT}}(\theta_{\mathrm{FT}})\right) \mid\theta_{\mathrm{FT}} \in\Theta_{\mathrm{FT}}(\theta_{\mathrm{PT}}) \right\}, T ( θ PT ) := { ( L FT ( θ FT ) , L PT ( θ FT ) ) ∣ θ FT ∈ Θ FT ( θ PT ) } ,
where Θ F T ( θ P T ) \Theta_{\mathrm{FT}}(\theta_{\mathrm{PT}}) Θ FT ( θ PT ) is the set of models obtained by fine-tuning θ P T \theta_{\mathrm{PT}} θ PT , and compare base models through the lower-left Pareto frontier of this set, which we call the learning–forgetting frontier . For a collection of base models B \mathcal{B} B , we compare the corresponding sets
T ( B ) : = { T ( θ P T ) ∣ θ P T ∈ B } . \mathcal{T}(\mathcal{B}) := \left\{ \mathcal{T}(\theta_{\mathrm{PT}}) \mid\theta_{\mathrm{PT}} \in\mathcal{B} \right\}. T ( B ) := { T ( θ PT ) ∣ θ PT ∈ B } .
Fine-tuning gap. We compare the two optimizers on this tradeoff with a single metric, fine-tuning gap G F T G_{\mathrm{FT}} G FT , which we define as the signed mean minimum Euclidean distance between the AdamW and Muon frontier. The fine-tuning gap is positive when the AdamW frontier dominates the Muon frontier, and the reverse else wise. Thus G F T > 0 G_{\mathrm{FT}} > 0 G FT > 0 when AdamW dominates and G F T < 0 G_{\mathrm{FT}} < 0 G FT < 0 when Muon dominates (Section C.5 gives the details).
Crossover token–parameter ratio (TPR). We quantify pretraining data budget using token–parameter ratio (TPR), the number of pretraining tokens per model parameter. For a fixed model size and fine-tuning dataset, the crossover TPR is the minimum TPR required for the AdamW frontier to overtake the Muon frontier.
Experiments
We now compare how the learning–forgetting tradeoffs of Muon and AdamW scale with pretraining data budget and model size. In § 3.2 , we show that as the pretraining data budget increases, Muon’s advantage over AdamW shrinks and eventually reverses in favor of AdamW. This reversal happens at progressively smaller data budgets as model size grows.
Experimental setup
Pretraining. We pretrain OLMo-style 30M–600M models on 150M to 154B DCLM tokens, comparing AdamW and Muon under a warmup–stable–decay (WSD) schedule. We sweep hyperparameters broadly and report the complete pretraining methodology in Section C.
Fine-tuning. For each base model, we evaluate the learning–forgetting tradeoff on four standard datasets: GSM8K ([4 ] ), StackMathQA ([43 ] ), Alpaca ([33 ] ), and MusicPile ([41 ] ), sweeping AdamW learning rates from 10 − 4 10^{-4} 1 0 − 4 to 10 − 2 10^{-2} 1 0 − 2 and Muon learning rates from 6 × 10 − 4 6 \times10^{-4} 6 × 1 0 − 4 to 2 × 10 − 2 2 \times10^{-2} 2 × 1 0 − 2 (Section C.2). We fine-tune AdamW-pretrained models with AdamW and Muon-pretrained models with both AdamW and Muon: AdamW fine-tuning consistently outperforms Muon fine-tuning on AdamW-pretrained models, whereas neither optimizer is consistently better on Muon-pretrained models (Section D.3). We then plot the learning–forgetting frontier over the resulting runs. We choose a broad learning-rate range to ensure we capture the complete learning–forgetting tradeoff spectrum: higher learning rates favor learning at the expense of forgetting, while lower rates preserve pretraining knowledge but learn less.
Evaluating the learning–forgetting tradeoff. For each fine-tuned checkpoint, we compute the fine-tuning loss and pretraining loss on held-out fine-tuning and pretraining (DCLM) data, and use these checkpoints to approximate the learning–forgetting tradeoff. Although this approximation is constructed from a finite set of fine-tuning runs, we conduct a comprehensive sweep over four datasets and across a wide range of fine-tuning learning rates to obtain a close estimate. We evaluate the learning–forgetting tradeoff in two settings: lowest-pretraining-loss setting , and the best-frontier setting .
Lowest-pretraining-loss setting: Pretraining hyperparameters are typically selected to minimize pretraining loss. Therefore, for each optimizer, we tune the pretraining hyperparameters to minimize pretraining loss for a given pretraining data budget and model size. We then sweep across all fine-tuning configurations for each of the lowest-loss pretrained models, and evaluate the resulting frontier.
Best-frontier setting: However, prior studies find that the lowest-loss pretrained model is not always the best model after fine-tuning: models pretrained with other hyperparameters can offer a better learning–forgetting tradeoff ([38 ] ). Therefore, for each pretraining optimizer, we additionally sweep
across all pretraining and fine-tuning configurations to achieve the best possible learning–forgetting frontier for a given pretraining data budget and model size (Section C.1.3).
Main result
We now investigate how the Muon–AdamW fine-tuning gap scales with pretraining data budget and model size.
Main result: Pretraining data scale reverses the Muon–AdamW fine-tuning gap. We compare the fine-tuning gap for AdamW and Muon at different pretraining data budgets in the lowest-pretraining-loss setting, using StackMathQA as the fine-tuning dataset (Figure 3 a). At small pretraining data budgets (2.4B tokens), Muon Pareto-dominates AdamW (Figure 1 ) and the fine-tuning gap is negative. At larger pretraining data budgets, this trend reverses: beyond 2.4B tokens, AdamW achieves lower fine-tuning loss with less forgetting than Muon. The trend also holds in the best-frontier setting (Figure 12 ), and for additional fine-tuning datasets in Section D.
Figure 1. Beyond a sufficiently large data budget, AdamW outperforms Muon. We pretrain OLMo-60M models with Muon and AdamW and fine-tune on StackMathQA. For each pretraining optimizer, we take the pretrained model with the lowest pretraining loss, sweep across all fine-tuning configurations, and plot the resulting learning–forgetting frontier, from 2.4B to 38.4B pretraining tokens. At 2.4B tokens, Muon’s frontier dominates AdamW’s; at larger budgets the trend reverses and AdamW surpasses Muon. This reversal happens despite Muon achieving lower pretraining loss across all data budgets.
Figure 3. The Muon–AdamW fine-tuning gap widens with increasing pretraining data budget, and Muon’s advantage shrinks with model size. (a) The fine-tuning gap after fine-tuning OLMo-60M models on StackMathQA, in the lowest-pretraining-loss setting (each pretraining optimizer’s lowest-loss pretrained model swept across all fine-tuning configurations). As the pretraining data budget increases, the fine-tuning gap transitions from negative to positive, with AdamW outperforming Muon beyond the crossover TPR. (b) Pretraining loss at 40 TPR. Muon’s advantage over AdamW shrinks as model size grows. (c) Degradation at 40 TPR, when we accept a StackMathQA loss 10% above the best in exchange for less forgetting. Muon degrades increasingly more than AdamW as model size grows.
Figure 12. Fine-tuning gap on every fine-tuning dataset, best-frontier setting (60M). Negative values mean the Muon frontier dominates.
We next investigate how the reversal of the Muon–AdamW fine-tuning gap scales with model size. Specifically, we ask how the amount of pretraining data needed for AdamW to overtake Muon changes as model size grows.
As model size scales, AdamW surpasses Muon at progressively smaller data budgets. We find that the crossover TPR decreases with model size. On StackMathQA, the crossover TPR is 320 TPR for 30M models but only 10 TPR for 600M models (Figure 2 ). The specific crossover TPR depends on the fine-tuning dataset, but the broader trend persists; as model size grows, the crossover TPR decreases, and Muon’s advantage reverses with substantially smaller data budgets.
Figure 2. Crossover TPR decreases with model size. In the lowest-pretraining-loss setting, where each pretraining optimizer’s lowest-loss pretrained model is swept across all fine-tuning hyperparameters, Muon’s advantage reverses at substantially smaller pretraining data budgets as model size increases. The exact crossover TPR varies depending on the fine-tuning dataset, but the high-level trend is consistent.
We hypothesize that in the lowest-pretraining-loss setting, this scaling trend is driven by two factors: Muon’s shrinking pretraining loss advantage and increased degradation relative to AdamW (Figure 3 b,Figure 3 c). For each optimizer and model size, we compare its base pretraining loss and its degradation L P T ( θ P T ) − L P T ( θ P T ∗ ) \mathcal{L}_{\mathrm{PT}}(\theta_{\mathrm{PT}}) - \mathcal{L}_{\mathrm{PT}}(\theta_{\mathrm{PT}}^{*}) L PT ( θ PT ) − L PT ( θ PT ∗ ) , after fine-tuning on StackMathQA. First, for a fixed TPR, the gap in pretraining loss between Muon and AdamW shrinks as model size increases, reducing Muon’s advantage as a lower-loss base model. Additionally, for a fixed TPR and fine-tuning loss, Muon forgets increasingly more than AdamW as model size grows. Consequently, as model size scales, Muon’s pretraining advantage over AdamW narrows, while its forgetting relative to AdamW increases, enabling AdamW to overtake Muon at progressively smaller data budgets.
Analyzing the Muon–AdamW Forgetting Gap
In § 3 , we show that Muon offers a better learning–forgetting tradeoff at small token budgets, but AdamW overtakes it beyond a crossover budget. This reversal occurs even though Muon achieves lower pretraining loss at
every budget we consider. In this section, we develop a theoretical explanation for this reversal. We compare Muon with exact orthogonalization to full-batch gradient descent (GD), a tractable comparison that captures the qualitative budget trend of Muon and AdamW in language models (§ 4.3 ).
To formalize the relationship between GD and Muon, we study a linear associative memory model of fact learning. In this setting, GD and Muon learn the facts to a similar degree, but their checkpoints can differ in their sensitivity to perturbations of the output distribution. Since fine-tuning changes the output distribution, we expect checkpoints that preserve factual accuracy under a wider range of perturbations to be more resistant to forgetting during fine-tuning. Our central goal is therefore to characterize this sensitivity across pretraining budgets and use it to explain why the forgetting gap reverses.
To quantify sensitivity to output perturbations, we introduce the margin-to-spread ratio (MSR) : the difference between the correct answer’s logit and the average incorrect logit, normalized by the spread of the incorrect logits. The MSR captures a checkpoint’s robustness to output perturbations and provides an analytically tractable basis for comparing GD and Muon.
Roadmap. We introduce the linear associative memory setting, define the MSR, and establish its connection to sensitivity (§ 4.1 ). We then prove that, after one update step, the ratio of Muon’s MSR to GD’s decreases as the data budget grows, indicating that Muon becomes more sensitive to output perturbations relative to GD (§ 4.2 ). Finally, we show that the MSR tracks the robustness of AdamW and Muon in real language models (§ 4.3 ).
Setup and the margin-to-spread ratio
We begin by introducing our linear associative memory setup and defining the margin-to-spread ratio.
Linear associative memory. To capture the behavior of Muon and GD in a tractable setting, we consider a linear associative memory, in which a single weight matrix is trained to associate each of N N N keys, represented by a vector, with its corresponding value, also represented by a vector. Formally, we represent each fact by a key–value pair ( u i , v i ) ∈ R d × R d (u_i, v_i) \in\mathbb{R}^{d} \times\mathbb{R}^{d} ( u i , v i ) ∈ R d × R d , with all keys and values drawn independently from N ( 0 , I d ) \mathcal{N}(0, I_d) N ( 0 , I d ) . The memory is parameterized by a matrix W ∈ R d × d W \in\mathbb{R}^{d \times d} W ∈ R d × d . Given a key u i u_i u i , it scores each candidate value v j v_j v j as s i j ( W ) : = v j T W u i s_{ij}(W) := v_j^{\mathsf{T}} W u_i s ij ( W ) := v j T W u i , which we treat as logits, and retrieves the highest-scoring value. Fact i i i is recovered when s i i ( W ) > s i j ( W ) s_{ii}(W) > s_{ij}(W) s ii ( W ) > s ij ( W ) for all j ≠ i j \ne i j = i .
We train with the softmax cross-entropy loss. Given a query distribution q = ( q 1 , … , q N ) q = (q_1, \ldots, q_N) q = ( q 1 , … , q N ) over the facts, the predicted probabilities and cross-entropy loss are
p ^ W ( ⋅ ∣ i ) : = softmax ( s i 1 ( W ) , … , s i N ( W ) ) , L ( W ) : = − ∑ i = 1 N q i log p ^ W ( i ∣ i ) . (1) \widehat{p}_W(\cdot\mid i) := \operatorname{softmax}\left(s_{i1}(W), \ldots, s_{iN}(W)\right), \qquad\mathcal{L}(W) := -\sum_{i=1}^{N} q_i \log\widehat{p}_W(i \mid i). \tag*{(1)} p W ( ⋅ ∣ i ) := softmax ( s i 1 ( W ) , … , s i N ( W ) ) , L ( W ) := − i = 1 ∑ N q i log p W ( i ∣ i ) . (1)
Data budget. We denote the memory dimension by d d d and the number of facts by N N N , and grow the two together, keeping their ratio γ = N / d > 1 \gamma= N/d > 1 γ = N / d > 1 fixed, so that N = γ d N = \gamma d N = γ d as d → ∞ d \to\infty d → ∞ . This mirrors the typical scaling behavior of language models, which store many more facts than they have dimensions. We call γ \gamma γ the data budget : it measures facts per dimension and plays the role of token-parameter ratio in § 2 . We compare data budgets by taking this limit at different fixed values of γ \gamma γ . Since γ > 1 \gamma> 1 γ > 1 , we have N − 1 ≥ d N - 1 \ge d N − 1 ≥ d , and log N / d → 0 \log N/d \to0 log N / d → 0 , written log N = o ( d ) \log N = o(d) log N = o ( d ) . For random variables X X X and a deterministic a > 0 a > 0 a > 0 , X = O p ( a ) X = O_p(a) X = O p ( a ) means that for every ε > 0 \varepsilon> 0 ε > 0 there is a C C C with Pr [ ∣ X ∣ > C a ] ≤ ε \Pr[|X| > Ca] \le\varepsilon Pr [ ∣ X ∣ > C a ] ≤ ε eventually, and X = o p ( a ) X = o_p(a) X = o p ( a ) means that Pr [ ∣ X ∣ > ε a ] → 0 \Pr[|X| > \varepsilon a] \to0 Pr [ ∣ X ∣ > ε a ] → 0 for every ε > 0 \varepsilon> 0 ε > 0 .
Optimization. For simplicity, we consider the exact orthogonalization variant of Muon. For a matrix A A A with singular value decomposition A = P Σ Q T A = P\Sigma Q^{\mathsf{T}} A = P Σ Q T , let polar ( A ) : = P Q T \operatorname{polar}(A) := P Q^{\mathsf{T}} polar ( A ) := P Q T . Muon updates W t + 1 = W t − η polar ( ∇ L ( W t ) ) W_{t+1} = W_t - \eta\operatorname{polar}(\nabla\mathcal{L}(W_t)) W t + 1 = W t − η polar ( ∇ L ( W t )) . We compare Muon to full-batch gradient descent (GD), whose updates descend along the gradient itself, W t + 1 = W t − η ∇ L ( W t ) W_{t+1} = W_t - \eta\nabla\mathcal{L}(W_t) W t + 1 = W t − η ∇ L ( W t ) .
Margin-to-spread ratio. In a binary classification problem, a prediction is robust if the correct logit is far from the incorrect logit: the larger the margin, the more robust the prediction. In a multiclass classification problem, such as our linear associative memory setting, there are many incorrect logits. The robustness of a correct output to perturbations of the logits thus depends on its separation from all of the incorrect logits. As it turns out, the degree of separation can be usefully summarized by a single statistic, which we call the margin-to-spread ratio (MSR) . MSR captures the average separation between the correct logit and all other logits, normalized by the spread of the incorrect logits. A smaller spread, or a larger average separation, would both place the incorrect logits further below the correct logit relative to their spread, which can increase the robustness of the prediction to perturbations. To capture this intuition, we formally define the margin-to-spread ratio as follows:
Definition 4.1 (Margin-to-spread ratio). For fact i i i , define the pairwise margins m i j : = s i i ( W ) − s i j ( W ) m_{ij} := s_{ii}(W) - s_{ij}(W) m ij := s ii ( W ) − s ij ( W ) for j ≠ i j \ne i j = i , with mean and variance
h i : = mean j ≠ i m i j , κ i 2 : = mean j ≠ i ( m i j − h i ) 2 . (2) h_i := \operatorname{mean}_{j \ne i} m_{ij}, \qquad\kappa_i^2 := \operatorname{mean}_{j \ne i}(m_{ij} - h_i)^2. \tag*{(2)} h i := mean j = i m ij , κ i 2 := mean j = i ( m ij − h i ) 2 . (2) The margin-to-spread ratio (MSR) for fact i i i is ρ i : = h i / κ i \rho_i := h_i/\kappa_i ρ i := h i / κ i , for κ i > 0 \kappa_i > 0 κ i > 0 . At the model level, we average the mean margins and variances uniformly over facts before taking the ratio:
h ˉ : = mean i h i , κ ˉ 2 : = mean i κ i 2 , ρ : = h ˉ κ ˉ , (3) \bar{h} := \operatorname{mean}_i h_i, \qquad\bar{\kappa}^2 := \operatorname{mean}_i \kappa_i^2, \qquad\rho:= \frac{\bar{h}}{\bar{\kappa}}, \tag*{(3)} h ˉ := mean i h i , κ ˉ 2 := mean i κ i 2 , ρ := κ ˉ h ˉ , (3) where ρ \rho ρ is defined for κ ˉ > 0 \bar{\kappa} > 0 κ ˉ > 0 .
Since s i i ( W ) s_{ii}(W) s ii ( W ) is the same for every competitor j j j , κ i \kappa_i κ i is also the standard deviation of the incorrect logits s i j ( W ) s_{ij}(W) s ij ( W ) , so the MSR measures the average margin in units of their spread. The definition applies to any model that assigns a logit to each candidate answer; for a language model, s i j ( W ) s_{ij}(W) s ij ( W ) are the next-token logits (§ 4.3 ).
Under isotropic Gaussian noise added to the logits, the MSR gives a lower bound on the fraction of incorrect answers that stay below the correct one: with noise of standard deviation σ \sigma σ on each logit, this fraction is at least ρ i 2 / ( ρ i 2 + 2 σ 2 / κ i 2 ) \rho_i^2/(\rho_i^2 + 2\sigma^2/\kappa_i^2) ρ i 2 / ( ρ i 2 + 2 σ 2 / κ i 2 ) in expectation. For a fixed noise strength relative to the incorrect-logit spread, this bound improves as the MSR increases (Corollary B.5 ).
One-step analysis of the margin-to-spread ratio
We have so far introduced our linear associative memory setup and built intuition for how the MSR captures the robustness of a model to perturbations of its logits. Here, we argue that the MSR of GD is greater than that of Muon for sufficiently large data budgets. We precisely track the one-step dynamics of the MSR ratio R : = ρ G D / ρ M u o n \mathcal{R} := \rho^{\mathrm{GD}}/\rho^{\mathrm{Muon}} R := ρ GD / ρ Muon between GD and Muon in the linear associative memory.
We analyze a single update from zero initialization, using uniform query weights q i = 1 / N q_i = 1/N q i = 1/ N . The weights after many optimization steps depend on the whole training trajectory and are hard to characterize precisely, so theoretical studies of how an optimizer shapes the learned weights often analyze its first step ([2 , 5 , 25 ] ). Kim et al. [14 ] take the same approach to compare a single step of Muon, SGD, and Newton’s method in a linear associative memory.
Theorem 4.2 (Informal). Consider one GD step and one Muon step from zero in the Gaussian associative memory model, with data budget γ = N / d \gamma= N/d γ = N / d . For sufficiently large γ \gamma γ :
GD has a larger MSR than Muon.
Increasing the data budget γ \gamma γ by any fixed percentage increases the ratio R = ρ G D / ρ M u o n \mathcal{R} = \rho^{\mathrm{GD}}/\rho^{\mathrm{Muon}} R = ρ GD / ρ Muon , with high probability as the dimension grows.
The ratio approaches 3 π / 8 ≈ 1.18 3\pi/8 \approx1.18 3 π /8 ≈ 1.18 from below at a rate of 1 / γ 1/\gamma 1/ γ :
− C γ ≤ R − 3 π 8 ≤ − c γ , -\frac{C}{\gamma} \le\mathcal{R} - \frac{3\pi}{8} \le-\frac{c}{\gamma}, − γ C ≤ R − 8 3 π ≤ − γ c , where 0 < c < C 0 < c < C 0 < c < C are absolute constants.
All claims hold with probability tending to one as d → ∞ d \to\infty d → ∞ , for sufficiently large fixed γ \gamma γ .
Proof sketch. Our goal is to compare one update step for Muon and for GD.
Since the MSR does not change when W W W is multiplied by a positive constant, the one-step weights can be written as
W G D = G , W M u o n = O : = polar ( G ) W^{\mathrm{GD}} = G, \qquad W^{\mathrm{Muon}} = O := \operatorname{polar}(G) W GD = G , W Muon = O := polar ( G )
(Appendix, Lemma F.1 ). We now want to compute ρ = h ˉ / κ ˉ \rho= \bar{h}/\bar{\kappa} ρ = h ˉ / κ ˉ for these matrices. This requires the mean margins and their standard deviations; we begin with the means.
Compute the mean margins. Recall that h ˉ \bar{h} h ˉ averages the difference between the correct and incorrect logits over facts and incorrect answers. For uniformly weighted facts,
G = ∑ k = 1 N ( v k − v ˉ ) u k T , v ˉ = 1 N ∑ k = 1 N v k . G = \sum_{k=1}^{N}(v_k - \bar{v})u_k^{\mathsf{T}}, \qquad\bar{v} = \frac{1}{N}\sum_{k=1}^{N}v_k. G = k = 1 ∑ N ( v k − v ˉ ) u k T , v ˉ = N 1 k = 1 ∑ N v k .
Substituting into this average gives h ˉ ( W ) = ⟨ W , G ⟩ / ( N − 1 ) \bar{h}(\mathbf{W}) = \langle\mathbf{W}, \mathbf{G}\rangle/(N-1) h ˉ ( W ) = ⟨ W , G ⟩ / ( N − 1 ) , and therefore
h ˉ G D = ∥ G ∥ F 2 N − 1 , h ˉ M u o n = ∥ G ∥ ∗ N − 1 . \bar{h}^{\mathrm{GD}} = \frac{\lVert\mathbf{G}\rVert_{F}^{2}}{N-1}, \qquad\bar{h}^{\mathrm{Muon}} = \frac{\lVert\mathbf{G}\rVert_{*}}{N-1}. h ˉ GD = N − 1 ∥ G ∥ F 2 , h ˉ Muon = N − 1 ∥ G ∥ ∗ .
where ∥ ⋅ ∥ F \lVert\cdot\rVert_{F} ∥ ⋅ ∥ F is the Frobenius norm and ∥ ⋅ ∥ ∗ \lVert\cdot\rVert_{*} ∥ ⋅ ∥ ∗ is the sum of the singular values.
Compute GD’s variance. For each query i i i , we take the variance across incorrect candidates j j j , holding the dataset and weights fixed:
κ i 2 = Var j ≠ i ( v j T G u i ) . \kappa_{i}^{2} = \operatorname{Var}_{j\ne i}(v_{j}^{\mathsf{T}}G u_{i}). κ i 2 = Var j = i ( v j T G u i ) .
To calculate its typical size, expand the incorrect logit:
v j T G u i ≈ ∑ k ∉ { i , j } ( v j T v k ) ( u k T u i ) ⏟ other facts + ∥ u i ∥ 2 ( v j T v i ) + ∥ v j ∥ 2 ( u j T u i ) ⏟ query and candidate facts . v_{j}^{\mathsf{T}}G u_{i} \approx\underbrace{\sum_{k\notin\{i,j\}}(v_{j}^{\mathsf{T}}v_{k})(u_{k}^{\mathsf{T}}u_{i})}_{\text{other facts}}+\underbrace{\lVert u_{i}\rVert^{2}(v_{j}^{\mathsf{T}}v_{i})+\lVert v_{j}\rVert^{2}(u_{j}^{\mathsf{T}}u_{i})}_{\text{query and candidate facts}}. v j T G u i ≈ other facts k ∈ / { i , j } ∑ ( v j T v k ) ( u k T u i ) + query and candidate facts ∥ u i ∥ 2 ( v j T v i ) + ∥ v j ∥ 2 ( u j T u i ) .
Here ≈ \approx ≈ omits the mean-subtraction term in G G G , which only affects lower-order terms in the variance. Each other fact contributes a product of two independent, mean-zero inner products, each with variance d d d , giving variance d 2 d^{2} d 2 . In the query and candidate terms, we approximate the squared norms by d d d , dropping their fluctuations. This gives variance d 3 d^{3} d 3 each, with O ( d 2 ) O(d^{2}) O ( d 2 ) corrections. Replacing N − 2 N-2 N − 2 by N N N also discards only O ( d 2 ) O(d^{2}) O ( d 2 ) terms.
These contributions are uncorrelated over the Gaussian training data. Including the omitted corrections and concentration of the empirical average, we calculate GD’s variance (Appendix, Lemma F.15) to be
( κ ˉ G D ) 2 = N d 2 ⏟ other facts + 2 d 3 ⏟ query and candidate ( 1 + o p ( 1 ) ) = d 3 ( γ + 2 ) ( 1 + o p ( 1 ) ) . (\bar{\kappa}^{\mathrm{GD}})^{2}=\underbrace{Nd^{2}}_{\text{other facts}}+\underbrace{2d^{3}}_{\text{query and candidate}}(1+o_{p}(1))=d^{3}(\gamma+2)(1+o_{p}(1)). ( κ ˉ GD ) 2 = other facts N d 2 + query and candidate 2 d 3 ( 1 + o p ( 1 )) = d 3 ( γ + 2 ) ( 1 + o p ( 1 )) .
Compute Muon’s variance. Again, the variance is over incorrect candidates for each fixed query. Gaussian isotropy and orthogonality give the leading calculation
( κ ˉ M u o n ) 2 ≈ 1 N ( N − 1 ) ∑ i ∑ j ≠ i ( v j T O u i ) 2 ≈ 1 N ∑ i ∥ O u i ∥ 2 − 1 N ∑ i ∥ u i ∥ 2 ≈ d . (\bar{\kappa}^{\mathrm{Muon}})^{2}\approx\frac{1}{N(N-1)}\sum_{i}\sum_{j\ne i}(v_{j}^{\mathsf{T}}O u_{i})^{2}\approx\frac{1}{N}\sum_{i}\lVert O u_{i}\rVert^{2}-\frac{1}{N}\sum_{i}\lVert u_{i}\rVert^{2}\approx d. ( κ ˉ Muon ) 2 ≈ N ( N − 1 ) 1 i ∑ j = i ∑ ( v j T O u i ) 2 ≈ N 1 i ∑ ∥ O u i ∥ 2 − N 1 i ∑ ∥ u i ∥ 2 ≈ d .
The first approximation drops the squared mean of the incorrect logits for each query. The second neglects departures of the sampled incorrect values from perfect isotropy, and the last drops fluctuations in the average squared key length. We calculate the variance including the omitted corrections and the dependence of O O O on the training data (Appendix, Lemma F.31 ) to be
( κ ˉ M u o n ) 2 = d [ 1 + O ( γ − 3 / 2 ) + o p ( 1 ) ] . (\bar{\kappa}^{\mathrm{Muon}})^{2}=d\left[1+O(\gamma^{-3/2})+o_{p}(1)\right]. ( κ ˉ Muon ) 2 = d [ 1 + O ( γ − 3/2 ) + o p ( 1 ) ] .
Put the means and standard deviations together. We calculate the norms in the mean margins using the Gaussian singular-value distribution (Appendix, Lemma F.4 ). Dividing by the standard deviations gives, to first order in 1 / γ 1/\gamma 1/ γ ,
ρ G D ≈ d γ ( 1 − 1 γ ) , ρ M u o n ≈ 8 3 π d γ ( 1 − 1 10 γ ) . \rho^{\mathrm{GD}}\approx\sqrt{\frac{d}{\gamma}}\left(1-\frac{1}{\gamma}\right), \qquad\rho^{\mathrm{Muon}}\approx\frac{8}{3\pi}\sqrt{\frac{d}{\gamma}}\left(1-\frac{1}{10\gamma}\right). ρ GD ≈ γ d ( 1 − γ 1 ) , ρ Muon ≈ 3 π 8 γ d ( 1 − 10 γ 1 ) .
These approximations omit higher-order budget corrections and relative fluctuations that vanish as d → ∞ d\to\infty d → ∞ . Taking their ratio cancels the common factor d / γ \sqrt{d/\gamma} d / γ . Keeping track of the errors gives
R = 3 π 8 ( 1 − 9 10 γ ) + O ( γ − 3 / 2 ) + o p ( 1 ) . \mathcal{R}=\frac{3\pi}{8}\left(1-\frac{9}{10\gamma}\right)+O(\gamma^{-3/2})+o_{p}(1). R = 8 3 π ( 1 − 10 γ 9 ) + O ( γ − 3/2 ) + o p ( 1 ) .
Since 3 π / 8 > 1 3\pi/8>1 3 π /8 > 1 , GD has the larger MSR at sufficiently large budgets. The negative 1 / γ 1/\gamma 1/ γ term in R − 3 π / 8 \mathcal{R}-3\pi/8 R − 3 π /8 dominates the smaller remainder, giving the claimed rate. A fixed percentage increase in γ \gamma γ raises the leading expression by order 1 / γ 1/\gamma 1/ γ , which also dominates the remainder, establishing the increase with high probability. See Section F for the quantitative bound and full proof, and Figure 4 and Section E.1 for numerical checks.
Figure 4. The margin-to-spread ratio of GD gains on Muon’s as the data budget grows. (a, b) After t = 10 4 t = 10^{4} t = 1 0 4 steps with model dimension d = 256 d = 256 d = 256 , GD’s MSR ρ \rho ρ overtakes Muon’s as γ \gamma γ grows, and the ordering of the perturbed robustness σ 50 \sigma_{50} σ 50 turns over with it. (c, d) After one step with d = 512 d = 512 d = 512 , GD’s normalized MSR ρ / d / γ \rho/\sqrt{d/\gamma} ρ / d / γ overtakes Muon’s (c), so the MSR ratio R = ρ G D / ρ M u o n \mathcal{R} = \rho^{\mathrm{GD}}/\rho^{\mathrm{Muon}} R = ρ GD / ρ Muon rises with γ \gamma γ (d), following the theory of Theorem 4.2 .
This theorem demonstrates that in the linear associative memory, when the data budget is sufficiently large, checkpoints learned with GD have a larger MSR than checkpoints learned with Muon after a single step of training. Combined with the robustness bound of § 4.1 , GD’s checkpoints therefore carry the stronger guarantee of robustness to perturbations of the output distribution in our theoretical setting.
Beyond one step. We next check whether the trend persists after extended training. We train the linear memory for T = 10 4 T = 10^{4} T = 1 0 4 full-batch steps from W 0 = 0 W_{0} = 0 W 0 = 0 at constant learning rates across three seeds. We summarize the saturated MSR by the per-seed median over t ≥ 10 3 t \ge10^{3} t ≥ 1 0 3 and measure robustness by σ 50 \sigma_{50} σ 50 , the relative perturbation size at which half of the initially recovered facts are lost. We exclude data budgets past the memory’s capacity where the two optimizers recover different item sets. The full setup and results for every dimension d d d appear in Section E.1.
The results in Figure 4 show that R \mathcal{R} R increases with the data budget. Muon has the larger MSR at small γ \gamma γ , while GD has the larger MSR at large γ \gamma γ . The robustness comparison shifts in the same direction: Muon tolerates stronger perturbations at small γ \gamma γ , and GD does at large γ \gamma γ . Thus, the budget trend persists in the linear setting with GD as the baseline.
Connection to practice
We test whether the budget trend predicted by Theorem 4.2 also appears in real language models, beyond the assumptions used in our theoretical analysis.
Setup. We evaluate the AdamW and Muon models trained in § 3 , spanning five model sizes, on held-out DCLM text. We compute the MSR at each token position and report the median over correctly predicted positions. At each position, h h h and κ \kappa κ are the mean and standard deviation of the margins between the correct next-token logit and all other vocabulary logits. We use the median to reduce sensitivity to outliers; Definition 4.1 instead uses the mean margin divided by the square root of the mean variance.
Experimental Result. Figure 5 plots the MSR for the three model sizes with the widest budget ranges, from 30M to 100M. Muon has a larger MSR at small data budgets, while AdamW has a larger MSR at sufficiently large budgets. As the budget grows, the difference between AdamW’s and Muon’s MSRs increases, and the ratio of AdamW’s MSR to Muon’s crosses one. This shift toward AdamW is consistent with the theoretical budget trend and parallels the reversal in relative robustness to fine-tuning observed in § 3 .
Figure 5. In real language models, the margin-to-spread ratio of AdamW gains on Muon’s as the token budget grows. (a) MSR ρ \rho ρ of AdamW and Muon against pretraining TPR for 30M, 60M, and 100M models: Muon has the larger MSR at small budgets and AdamW at large ones. (b) The MSR ratio ρ A d a m W / ρ M u o n \rho^{\mathrm{AdamW}}/\rho^{\mathrm{Muon}} ρ AdamW / ρ Muon crosses one as the budget grows, at every model size.
The MSR and fine-tuning crossovers do not necessarily appear at exactly the same budget, since the learning–forgetting tradeoff also depends on pretraining loss. Nevertheless, the MSR captures the direction of the observed shift and connects our theoretical analysis to the behavior of trained language models.
Remark. We summarize the analysis of this section and relate MSR to other metrics used to predict post-training.
Takeaway. In a linear associative memory, we prove that after one update the ratio ρ G D / ρ M u o n \rho_{\mathrm{GD}}/\rho_{\mathrm{Muon}} ρ GD / ρ Muon of the two optimizers’ MSRs increases with the data budget (Theorem 4.2 ). Simulations of many training steps show that the same trend persists after extended training (§ 4.2 ).
Extension: Relation of MSR to Other Metrics Predicting Post-Training. Other work predicts forgetting after post-training with different metrics. For example, Springer et al. [31 ] show that the loss increase after adding Gaussian noise to the pretrained weights tracks how much a model degrades under fine-tuning. In Section B, we show that MSR is connected to this robustness to Gaussian perturbations: in the linear memory, a larger MSR provably guarantees that more competitors stay ranked below the correct answer after the perturbation (Theorem B.3 ), and in both language models and the linear memory, MSR is strongly correlated with perturbation robustness (Section B.2.2).
Related Work
Muon. Muon, an optimizer introduced by Jordan et al. [13 ] , has recently shown strong empirical performance for pretraining LLMs, challenging AdamW’s dominance as the status-quo optimizer. Recent work has sought to explain Muon’s pretraining advantage through analyses of simplified theoretical models, associative memory, convergence behavior, and loss landscape geometry ([14 , 18 , 28 , 32 , 29 , 36 ] ). Another line of work investigates the training dynamics of Muon, finding that it learns data components more uniformly than AdamW ([35 , 37 , 6 ] ), while also characterizing the implicit biases of Muon ([8 , 6 ] ). Yet fewer works have examined the downstream consequences of Muon. Ruan et al. [27 ] compare Muon and AdamW after fine-tuning; they find that for small data regimes, Muon outperforms AdamW, consistent with our findings. We show that this advantage does not generalize across pretraining data scale; after a sufficiently large pretraining data budget, the trend reverses, and AdamW outperforms Muon.
Muon versus AdamW on pretraining loss. Muon has been reported to reach lower pretraining loss than AdamW for the same compute across a wide range of settings ([20 , 7 , 24 ] ), and frontier models have since been pretrained with Muon or its variants ([15 ] (GLM-4.5 Team et al. 2025)). The size of this advantage depends on scale: under controlled tuning, Muon’s speedup over AdamW shrinks as model size and data grow, and Muon is overtaken by other matrix-based optimizers at large data-to-model ratios ([40 a] ). All of these works measure pretraining loss. Optimizers can also reach qualitatively different solutions rather than faster or slower routes to the same minimum ([26 ] ); we ask whether these differences matter after fine-tuning.
Catastrophic forgetting. Catastrophic forgetting, the degradation of previously learned knowledge during subsequent training, remains a central challenge in deep learning ([10 ] ). Numerous approaches have been proposed to mitigate catastrophic forgetting, including regularization-based methods ([17 , 42 , 1 ] ), replay techniques ([30 ] ), gradient projection methods ([22 , 3 ] ), and optimization-based approaches ([38 ] ). Our learning–forgetting tradeoff is an instance of the stability–plasticity dilemma, with fine-tuning performance capturing plasticity and knowledge retention capturing stability ([22 ] ). We study forgetting as an intrinsic property of the pretrained checkpoint, rather than as a consequence of the fine-tuning algorithm. Specifically, we investigate how optimization choices during pretraining shape downstream forgetting, showing that Muon and AdamW induce different learning–forgetting tradeoffs.
Pretraining and downstream adaptation. Better pretraining loss does not always yield a better model after fine-tuning. Springer et al. [31 ] show that extending pretraining can make language models harder to fine-tune, as the pretrained parameters become increasingly sensitive to modifications, which they measure in part with Gaussian perturbations. Watts et al. [38 ] show that sharpness-aware pretraining mitigates forgetting during fine-tuning, and Liu et al. [21 ] find that fine-tuning with the same optimizer as pretraining forgets less. We study how the choice of pretraining optimizer interacts with the pretraining data budget, and connect the resulting fine-tuning gap to robustness to weight perturbations.
Conclusion
We asked whether the quality of Muon base models persists through fine-tuning, and found that the answer depends on the pretraining budget: beyond a crossover budget, AdamW outperforms Muon after fine-tuning, even though Muon reaches lower pretraining loss. We do not necessarily expect this for every fine-tuning distribution, since fine-tuning on data close to the pretraining data is simply continued pretraining. Still, the trend holds broadly on all datasets we study, including math, instruction following, and music. Furthermore, the crossover budget decreases with model size, which suggests that the problem persists as we scale.
We explain why AdamW outperforms Muon at large budgets by tracking how robust each model is to small isotropic perturbations of its output distribution, which we characterize with the margin-to-spread ratio (MSR). AdamW becomes increasingly more robust than Muon as the data budget grows, and this robustness is a property of the output distribution that the training loss does not capture. Since the MSR measures robustness to perturbations of the output distribution in general, our theory suggests that this robustness is fundamental to the checkpoint learned by each optimizer, rather than specific to fine-tuning, to the perturbations we study, or to a particular architecture or parameterization. It should therefore apply to any update that changes the output distribution.
Although outside the scope of this work, our analysis suggests that pretraining should explicitly optimize for properties beyond the pretraining loss. Optimizer selection requires more than comparing pretraining loss, and we can design objectives that explicitly target adaptability, for which the MSR is a potential candidate.
More broadly, we expect this lesson to extend beyond Muon. Many pretraining choices, such as hyperparameters, data, and architecture, are selected by pretraining loss. Yet, our results show that pretraining loss can be a poor proxy for adaptability, increasingly so as the data budget grows. This suggests that for any of these choices, a lower pretraining loss does not by itself yield a model that can better adapt.
Acknowledgements
We gratefully acknowledge support from Apple, Google, Jane Street, the National Science Foundation and the FLAME cluster at Carnegie Mellon University.
This material is based upon work supported by the National Science Foundation Graduate Research Fellowship under Grant No. DGE2140739. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
The authors thank Kaiyue Wen, Alex Damian, Ishaan Watts, Lawrence Feng, and Gene Yang for helpful discussions.
Definitions
Optimizers
Optimizers update a model’s parameters during training so as to decrease a loss. Given the gradient of the loss with respect to the parameters, an optimizer determines the direction and the magnitude of each update, and in doing so shapes training stability, convergence speed and the solution that is finally reached. This section recalls the two optimizers compared in the paper.
AdamW
Adam ([16 ] ) combines momentum with a per-coordinate step size. With g t = ∇ θ L ( θ t ) g_t = \nabla_\theta\mathcal{L}(\theta_t) g t = ∇ θ L ( θ t ) the gradient at step t t t , it maintains exponential moving averages of the first and second moments of the gradient,
m t = β 1 m t − 1 + ( 1 − β 1 ) g t , v t = β 2 v t − 1 + ( 1 − β 2 ) g t 2 , m_t = \beta_1 m_{t-1} + (1-\beta_1)g_t,\qquad v_t = \beta_2 v_{t-1} + (1-\beta_2)g_t^2, m t = β 1 m t − 1 + ( 1 − β 1 ) g t , v t = β 2 v t − 1 + ( 1 − β 2 ) g t 2 ,
with the square taken coordinatewise, corrects their bias towards the zero initialization,
m ^ t = m t 1 − β 1 t , v ^ t = v t 1 − β 2 t , \hat{m}_t = \frac{m_t}{1-\beta_1^t},\qquad\hat{v}_t = \frac{v_t}{1-\beta_2^t}, m ^ t = 1 − β 1 t m t , v ^ t = 1 − β 2 t v t ,
and updates
θ t + 1 = θ t − α m ^ t v ^ t + ϵ , \theta_{t+1} = \theta_t - \alpha\frac{\hat{m}_t}{\sqrt{\hat{v}_t}+\epsilon}, θ t + 1 = θ t − α v ^ t + ϵ m ^ t ,
where α \alpha α is the learning rate, β 1 , β 2 ∈ [ 0 , 1 ) \beta_1,\beta_2 \in[0,1) β 1 , β 2 ∈ [ 0 , 1 ) are the decay rates of the two averages, ϵ > 0 \epsilon> 0 ϵ > 0 guards the division, and the division and the square root are coordinatewise. In effect each coordinate is normalized by the scale of its own recent gradients.
Regularizing Adam with an ℓ 2 \ell_2 ℓ 2 penalty λ 2 ∥ θ ∥ 2 \frac{\lambda}{2}\lVert\theta\rVert^2 2 λ ∥ θ ∥ 2 adds λ θ t \lambda\theta_t λ θ t to g t g_t g t , so the penalty is itself divided by v ^ t + ϵ \sqrt{\hat{v}_t}+\epsilon v ^ t + ϵ and coordinates with large past gradients are regularized less. AdamW ([23 ] ) instead decouples the weight decay from the adaptive step,
θ t + 1 = θ t − α m ^ t v ^ t + ϵ − α λ θ t , \theta_{t+1} = \theta_t - \alpha\frac{\hat{m}_t}{\sqrt{\hat{v}_t}+\epsilon} - \alpha\lambda\theta_t, θ t + 1 = θ t − α v ^ t + ϵ m ^ t − α λ θ t ,
with g t g_t g t now the gradient of the unregularized loss and λ \lambda λ the weight decay coefficient, so that every coordinate decays at the same rate. AdamW is the standard optimizer for language-model pretraining and the baseline throughout the paper.
Muon
Muon ([13 ] ) treats each weight matrix as a whole rather than coordinate by coordinate. For a matrix-valued parameter θ \theta θ with gradient G t G_t G t , it accumulates momentum M t = μ M t − 1 + G t M_t = \mu M_{t-1} + G_t M t = μ M t − 1 + G t and updates along the orthogonalized momentum,
θ t + 1 = θ t − η Orth ( M t ) , (4) \theta_{t+1} = \theta_t - \eta\,\operatorname{Orth}(M_t), \tag*{(4)} θ t + 1 = θ t − η Orth ( M t ) , (4)
where η \eta η is the learning rate and, for a matrix X X X with singular value decomposition X = U Σ V ⊤ X = U\Sigma V^\top X = U Σ V ⊤ , Orth ( X ) = U V ⊤ \operatorname{Orth}(X) = UV^\top Orth ( X ) = U V ⊤ is its polar factor: the matrix with the singular vectors of X X X and all singular values set to one. The update therefore has the same size along every singular direction, and it can be read as the steepest descent direction when the size of a step is measured in the spectral norm ([13 ] ). An exact singular value decomposition at every step is expensive, so Orth \operatorname{Orth} Orth is approximated by a few steps of a Newton–Schulz iteration,
X 0 = M t ∥ M t ∥ F , X k + 1 = a X k + b ( X k X k ⊤ ) X k + c ( X k X k ⊤ ) 2 X k , X_0 = \frac{M_t}{\lVert M_t\rVert_F},\qquad X_{k+1} = aX_k + b(X_kX_k^\top)X_k + c(X_kX_k^\top)^2X_k, X 0 = ∥ M t ∥ F M t , X k + 1 = a X k + b ( X k X k ⊤ ) X k + c ( X k X k ⊤ ) 2 X k ,
whose odd matrix polynomial pushes every singular value of X 0 X_0 X 0 towards one; the reference implementation uses ( a , b , c ) = ( 3.4445 , − 4.7750 , 2.0315 ) (a,b,c) = (3.4445,-4.7750,2.0315) ( a , b , c ) = ( 3.4445 , − 4.7750 , 2.0315 ) and five steps. Muon is meant for the two-dimensional weight matrices of the hidden layers; embeddings, the output layer and one-dimensional parameters such as normalization gains are updated by AdamW. Large-scale implementations further add decoupled weight decay and rescale the update to the root-mean-square size of AdamW’s ([20 ] ). In our experiments the hidden matrices are updated by Muon and all other parameters by AdamW, each with its own learning rate and weight decay (§ 3.1 ).
Learning-rate schedules
Warmup–stable–decay
All pretraining runs use a warmup–stable–decay (WSD) schedule: the learning rate increases linearly from zero to its peak value over the warmup steps, stays at the peak for most of training, and is annealed to zero over the final decay phase.
Cosine
All fine-tuning runs use a cosine schedule: after a linear warmup of 20 steps, the learning rate follows η t = 1 2 η max ( 1 + cos ( π t / T ) ) \eta_t = \frac{1}{2}\eta_{\max}(1+\cos(\pi t/T)) η t = 2 1 η m a x ( 1 + cos ( π t / T )) over the remaining T T T steps.
Token–parameter ratio
We measure the pretraining data budget of a model with N N N parameters trained on D D D tokens by its token–parameter ratio, T P R = D / N \mathrm{TPR}=D/N TPR = D / N . The compute-optimal budget of Hoffmann et al. [12 ] corresponds to TPR = 20; our budgets range from 5 to 2,560 TPR.
Other Metrics for Analyzing Post-Training
Besides the margin-to-spread ratio, the robustness of a base model to post-training can be probed by perturbing its weights directly. This section reports Gaussian perturbation results in language models and in the linear memory (Section B.1), and then relates them to the MSR (Section B.2).
Gaussian perturbation
We first check whether Gaussian perturbations of the weights reproduce the budget trend of the learning–forgetting tradeoff, in language models and in the linear memory.
Language models
We test whether the perturbed loss of the pretrained language models shifts from Muon toward AdamW as the token budget grows, as the learning–forgetting tradeoff does.
Setup. Springer et al. [31 ] show that adding isotropic Gaussian noise to pretrained weights reproduces key empirical trends in forgetting after fine-tuning. Motivated by this connection, we study Gaussian weight perturbations as a simpler setting in which to investigate why the learning–forgetting gap between Muon and AdamW changes with pretraining budget. At the same checkpoints as in § 3 , we add isotropic Gaussian noise to each weight matrix, scaled to a fraction δ \delta δ of the matrix’s norm, and measure the loss increase Δ n o i s e ( θ P T ; δ ) : = L P T ( θ ~ δ ) − L P T ( θ P T ) \Delta_{\mathrm{noise}}(\theta_{\mathrm{PT}};\delta) := \mathcal{L}_{\mathrm{PT}}(\widetilde{\theta}_{\delta})-\mathcal{L}_{\mathrm{PT}}(\theta_{\mathrm{PT}}) Δ noise ( θ PT ; δ ) := L PT ( θ δ ) − L PT ( θ PT ) , where θ ~ δ \widetilde{\theta}_{\delta} θ δ denotes the perturbed weights; a smaller increase indicates greater perturbation robustness. See Section E.2 for full experimental details.
Result. Figure 6 plots the gap in loss degradation, Δ n o i s e ( M u o n ; δ ) − Δ n o i s e ( A d a m W ; δ ) \Delta_{\mathrm{noise}}(\mathrm{Muon};\delta)-\Delta_{\mathrm{noise}}(\mathrm{AdamW};\delta) Δ noise ( Muon ; δ ) − Δ noise ( AdamW ; δ ) . This gap is positive across the token budgets tested and grows with the pretraining budget: AdamW incurs a smaller loss increase throughout the sweep, and its advantage grows with budget.
Figure 6. The perturbation degradation gap grows with pretraining data budget. Under a perturbation of magnitude δ = 0.02 \delta= 0.02 δ = 0.02 , Muon degrades increasingly more than AdamW as the pretraining data budget grows. Points show the mean over three noise draws.
To connect this gap to the learning–forgetting tradeoff, we also account for pretraining loss: loss after perturbation equals pretraining loss plus Δ n o i s e \Delta_{\mathrm{noise}} Δ noise . Muon starts with lower pretraining loss, so it can retain lower loss after perturbation at small budgets despite its larger loss increase. As its pretraining-loss advantage shrinks and the degradation gap grows, AdamW achieves lower loss after perturbation beyond a crossover budget. For sufficiently strong perturbations, this crossover occurs at smaller token budgets as model size increases, paralleling the crossover observed after fine-tuning (Table 2 in Section E.2).
Table 2. Crossover budgets, in TPR (tokens per parameter). Left: the smallest budget at which AdamW’s loss under a perturbation of magnitude δ \delta δ is below Muon’s (loss-optimal checkpoints, mean over three perturbation draws); > 20: Muon’s is lower up to the largest 600M budget, 20 TPR; ≤ 5 \leq5 ≤ 5 : AdamW’s is lower already at the smallest budget; †: Muon’s is lower again at 160 TPR, by 0.003 nats. Right: the fine-tuning crossover of Figure 2 .
Linear associative memory
We repeat the perturbation experiment in the linear memory, where the effect of a perturbation can be measured fact by fact.
Setup. We train the memory of § 4.1 with GD and Muon to minimize cross-entropy, varying the data budget γ = N / d \gamma= N/d γ = N / d and the dimension d d d , as in § 4.2 . The trained weights are perturbed to W ′ = W + σ ∥ W ∥ F ξ W' = W + \sigma\lVert W\rVert_F \xi W ′ = W + σ ∥ W ∥ F ξ with ∥ ξ ∥ F = 1 \lVert\xi\rVert_F = 1 ∥ ξ ∥ F = 1 , over 16 perturbation directions shared by the two optimizers. Since the memory retrieves facts, we measure degradation by the fraction of initially recovered facts that are lost. At a single perturbation magnitude, fact loss can be close to zero or one, making memories with different noise tolerances difficult to distinguish. We therefore compare the perturbation strengths needed to reach a common intermediate level of degradation: σ 50 \sigma_{50} σ 50 is the relative perturbation strength at which half of the initially recovered facts are lost. A larger σ 50 \sigma_{50} σ 50 means that more noise is needed to cause the same fraction of facts to be lost. Data budgets past the memory’s capacity, where the two optimizers recover different sets of facts, are excluded. Full details appear in Sections E.1 and E.3.
Result. The robustness ordering of the two optimizers turns over with the data budget: Muon tolerates stronger perturbations at small γ \gamma γ , and GD does at large γ \gamma γ (Figure 4 b and the bottom row of Figure 20 ). The σ 50 \sigma_{50} σ 50 ordering flips at γ ≈ 0.5 \gamma\approx0.5 γ ≈ 0.5 , 1.3 and 11 at d = 128 d = 128 d = 128 , 256 and 512, and at d = 64 d = 64 d = 64 GD is already the more robust at the smallest data budget we ran.
Figure 20. Top: saturated ρ \rho ρ against the data budget, the two curves behind Figure 21 . Bottom: perturbed robustness σ 50 \sigma_{50} σ 50 against the data budget, below capacity only. One column per d d d .
Gaussian perturbation and the margin-to-spread ratio
We next relate robustness to Gaussian perturbations to the MSR, first in theory and then in experiments.
Theoretical analysis
We use the scores s i j ( W ) = v j ⊤ W u i s_{ij}(W) = v_j^{\top} W u_i s ij ( W ) = v j ⊤ W u i of § 4.1 and the margins m i j m_{ij} m ij , h i h_i h i , κ i \kappa_i κ i and ρ i \rho_i ρ i of Definition 4.1 , for arbitrary keys, values and W W W . The weights are perturbed to W + σ E W + \sigma E W + σ E , where the entries of E E E satisfy
E [ E k l ] = 0 , E [ E k l E k ′ l ′ ] = 1 { ( k , l ) = ( k ′ , l ′ ) } . (5) \mathbb{E}[E_{kl}] = 0,\qquad\mathbb{E}[E_{kl}E_{k'l'}] = \mathbf{1}\{(k,l) = (k',l')\}. \tag*{(5)} E [ E k l ] = 0 , E [ E k l E k ′ l ′ ] = 1 {( k , l ) = ( k ′ , l ′ )} . (5)
Scores are linear in W W W , so exactly
s i i ( W + σ E ) − s i j ( W + σ E ) = m i j + σ ( v i − v j ) ⊤ E u i . s_{ii}(W + \sigma E) - s_{ij}(W + \sigma E) = m_{ij} + \sigma(v_i - v_j)^{\top} E u_i. s ii ( W + σ E ) − s ij ( W + σ E ) = m ij + σ ( v i − v j ) ⊤ E u i .
We measure robustness by the fraction of competitors still ranked below the correct value,
AUC i ( σ ) = mean j ≠ i Pr E [ s i i ( W + σ E ) > s i j ( W + σ E ) ] . \operatorname{AUC}_i(\sigma) = \underset{j \ne i}{\operatorname{mean}}\ \Pr_E[s_{ii}(W + \sigma E) > s_{ij}(W + \sigma E)]. AUC i ( σ ) = j = i mean E Pr [ s ii ( W + σ E ) > s ij ( W + σ E )] .
Let J J J be uniform on { 1 , … , N } ∖ { i } \{1,\ldots,N\} \setminus\{i\} { 1 , … , N } ∖ { i } , independent of E E E . Then
AUC i ( σ ) = Pr [ s i J ( W + σ E ) − s i J ( W + σ E ) > 0 ] = Pr [ m i J + σ ( v i − v J ) ⊤ E u i > 0 ] , \operatorname{AUC}_i(\sigma) = \Pr[s_{iJ}(W + \sigma E) - s_{iJ}(W + \sigma E) > 0] = \Pr[m_{iJ} + \sigma(v_i - v_J)^{\top} E u_i > 0], AUC i ( σ ) = Pr [ s i J ( W + σ E ) − s i J ( W + σ E ) > 0 ] = Pr [ m i J + σ ( v i − v J ) ⊤ E u i > 0 ] ,
so it suffices to bound the probability that one random variable is positive, from its mean and variance.
Lemma B.1 (Cantelli). If E X > 0 \mathbb{E}X > 0 E X > 0 and Var X < ∞ \operatorname{Var}X < \infty Var X < ∞ , then
Pr [ X ≤ 0 ] ≤ Var X Var X + ( E X ) 2 . \Pr[X \le0] \le\frac{\operatorname{Var}X}{\operatorname{Var}X + (\mathbb{E}X)^2}. Pr [ X ≤ 0 ] ≤ Var X + ( E X ) 2 Var X .
Proof. For t ≥ 0 t \ge0 t ≥ 0 , Markov’s inequality gives Pr [ X ≤ 0 ] ≤ E ( E X − X + t ) 2 / ( E X + t ) 2 = ( Var X + t 2 ) / ( E X + t ) 2 \Pr[X \le0] \le\mathbb{E}(\mathbb{E}X - X + t)^2/(\mathbb{E}X + t)^2 = (\operatorname{Var}X + t^2)/(\mathbb{E}X + t)^2 Pr [ X ≤ 0 ] ≤ E ( E X − X + t ) 2 / ( E X + t ) 2 = ( Var X + t 2 ) / ( E X + t ) 2 ; take t = Var X / E X t = \operatorname{Var}X/\mathbb{E}X t = Var X / E X .
Lemma B.2 (Noise adds variance, not bias). Let E E E satisfy Equation (5) . Then
E [ m i J + σ ( v i − v J ) ⊤ E u i ] = h i , \mathbb{E}\left[m_{iJ} + \sigma(v_i-v_J)^\top E u_i\right] = h_i, E [ m i J + σ ( v i − v J ) ⊤ E u i ] = h i , Var [ m i J + σ ( v i − v J ) ⊤ E u i ] = κ i 2 + σ 2 ∥ u i ∥ 2 m e a n j ≠ i ∥ v i − v j ∥ 2 . \operatorname{Var}\left[m_{iJ} + \sigma(v_i-v_J)^\top E u_i\right] = \kappa_i^2 + \sigma^2\lVert u_i\rVert^2 \mathop{\mathrm{mean}}_{j \ne i}\lVert v_i-v_j\rVert^2. Var [ m i J + σ ( v i − v J ) ⊤ E u i ] = κ i 2 + σ 2 ∥ u i ∥ 2 mean j = i ∥ v i − v j ∥ 2 .
Proof. For fixed j j j , ( v i − v j ) ⊤ E u i = ∑ k , l ( v i − v j ) k E k l ( u i ) l (v_i-v_j)^\top E u_i = \sum_{k,l}(v_i-v_j)_k E_{kl}(u_i)_l ( v i − v j ) ⊤ E u i = ∑ k , l ( v i − v j ) k E k l ( u i ) l has, by Equation (5) , mean 0 0 0 and second moment ∥ v i − v j ∥ 2 ∥ u i ∥ 2 \lVert v_i-v_j\rVert^2\lVert u_i\rVert^2 ∥ v i − v j ∥ 2 ∥ u i ∥ 2 . Average over E E E first, then over J J J .
Theorem B.3 (Perturbed AUC and the margin-to-spread ratio). Let E E E satisfy Equation (5) . For every W W W , all data, every σ ≥ 0 \sigma\ge0 σ ≥ 0 and every i i i with h i > 0 h_i > 0 h i > 0 ,
AUC i ( σ ) ≥ h i 2 h i 2 + κ i 2 + σ 2 ∥ u i ∥ 2 m e a n j ≠ i ∥ v i − v j ∥ 2 . \operatorname{AUC}_i(\sigma) \ge\frac{h_i^2}{h_i^2+\kappa_i^2+\sigma^2\lVert u_i\rVert^2\mathop{\mathrm{mean}}_{j\ne i}\lVert v_i-v_j\rVert^2}. AUC i ( σ ) ≥ h i 2 + κ i 2 + σ 2 ∥ u i ∥ 2 mean j = i ∥ v i − v j ∥ 2 h i 2 .
Proof. 1 − AUC i ( σ ) = Pr [ m i J + σ ( v i − v J ) ⊤ E u i ≤ 0 ] 1-\operatorname{AUC}_i(\sigma)=\Pr\left[m_{iJ}+\sigma(v_i-v_J)^\top E u_i\le0\right] 1 − AUC i ( σ ) = Pr [ m i J + σ ( v i − v J ) ⊤ E u i ≤ 0 ] . Apply Lemma B.1 with the mean and variance of Lemma B.2 .
No assumption is made on W W W , on the shape of the margin distribution, or on d d d and N N N .
Corollary B.4 (Gaussian data). If moreover u 1 , … , u N , v 1 , … , v N u_1,\ldots,u_N,v_1,\ldots,v_N u 1 , … , u N , v 1 , … , v N are i.i.d. N ( 0 , I d ) \mathcal{N}(0,I_d) N ( 0 , I d ) , then
AUC i ( σ ) ≥ h i 2 h i 2 + κ i 2 + 2 σ 2 d 2 ( 1 + O p ( d − 1 / 2 ) ) = ρ i 2 ρ i 2 + 1 + 2 ( σ d / κ i ) 2 ( 1 + O p ( d − 1 / 2 ) ) . \operatorname{AUC}_i(\sigma) \ge\frac{h_i^2}{h_i^2+\kappa_i^2+2\sigma^2d^2(1+O_p(d^{-1/2}))} = \frac{\rho_i^2}{\rho_i^2+1+2(\sigma d/\kappa_i)^2(1+O_p(d^{-1/2}))}. AUC i ( σ ) ≥ h i 2 + κ i 2 + 2 σ 2 d 2 ( 1 + O p ( d − 1/2 )) h i 2 = ρ i 2 + 1 + 2 ( σ d / κ i ) 2 ( 1 + O p ( d − 1/2 )) ρ i 2 .
Proof. ∥ u i ∥ 2 = d ( 1 + O p ( d − 1 / 2 ) ) \lVert u_i\rVert^2=d(1+O_p(d^{-1/2})) ∥ u i ∥ 2 = d ( 1 + O p ( d − 1/2 )) and m e a n j ≠ i ∥ v i − v j ∥ 2 = 2 d ( 1 + O p ( d − 1 / 2 ) ) \mathop{\mathrm{mean}}_{j\ne i}\lVert v_i-v_j\rVert^2=2d(1+O_p(d^{-1/2})) mean j = i ∥ v i − v j ∥ 2 = 2 d ( 1 + O p ( d − 1/2 )) .
At a fixed noise level in units of κ i \kappa_i κ i , a larger ρ i \rho_i ρ i therefore guarantees a larger AUC. For the relative perturbation W + δ ∥ W ∥ F E / ∥ E ∥ F W+\delta\lVert W\rVert_F E/\lVert E\rVert_F W + δ ∥ W ∥ F E / ∥ E ∥ F used in the experiments, with i.i.d. standard Gaussian entries of E E E , the bounds hold with σ \sigma σ replaced by δ ∥ W ∥ F / d \delta\lVert W\rVert_F/d δ ∥ W ∥ F / d .
Corollary B.5 (Noise on the logits). Suppose instead that each logit is perturbed directly, to s i j ( W ) + σ ξ i j s_{ij}(W)+\sigma\xi_{ij} s ij ( W ) + σ ξ ij , where the ξ i j \xi_{ij} ξ ij have mean 0 0 0 and variance 1 1 1 and are uncorrelated (for example, i.i.d. N ( 0 , 1 ) \mathcal{N}(0,1) N ( 0 , 1 ) ). Then for every W W W , every σ ≥ 0 \sigma\ge0 σ ≥ 0 and every i i i with h i > 0 h_i > 0 h i > 0 ,
m e a n j ≠ i Pr ξ [ s i i ( W ) + σ ξ i i > s i j ( W ) + σ ξ i j ] ≥ h i 2 h i 2 + κ i 2 + 2 σ 2 = ρ i 2 ρ i 2 + 1 + 2 σ 2 / κ i 2 . \mathop{\mathrm{mean}}_{j\ne i}\Pr_{\xi}\left[s_{ii}(W)+\sigma\xi_{ii}>s_{ij}(W)+\sigma\xi_{ij}\right]\ge\frac{h_i^2}{h_i^2+\kappa_i^2+2\sigma^2}=\frac{\rho_i^2}{\rho_i^2+1+2\sigma^2/\kappa_i^2}. mean j = i ξ Pr [ s ii ( W ) + σ ξ ii > s ij ( W ) + σ ξ ij ] ≥ h i 2 + κ i 2 + 2 σ 2 h i 2 = ρ i 2 + 1 + 2 σ 2 / κ i 2 ρ i 2 .
Proof. With J J J uniform on { 1 , … , N } ∖ { i } \{1,\ldots,N\}\setminus\{i\} { 1 , … , N } ∖ { i } and independent of ξ \xi ξ , the variable m i J + σ ( ξ i i − ξ i J ) m_{iJ}+\sigma(\xi_{ii}-\xi_{iJ}) m i J + σ ( ξ ii − ξ i J ) has mean h i h_i h i and variance κ i 2 + 2 σ 2 \kappa_i^2+2\sigma^2 κ i 2 + 2 σ 2 , since ξ i i − ξ i j \xi_{ii}-\xi_{ij} ξ ii − ξ ij has mean 0 0 0 and variance 2 2 2 for every j ≠ i j\ne i j = i . Apply Lemma B.1 .
A fact is mislabeled only if some incorrect answer overtakes the correct one, so by a union bound the probability of mislabeling fact i i i is at most N − 1 N-1 N − 1 times one minus the left-hand side. This bound is informative only when ρ i 2 \rho_i^2 ρ i 2 is much larger than N N N .
Empirical validation
We test whether the MSR predicts how much perturbation a checkpoint can tolerate, first in language models and then in the linear memory.
Language models. We measure MSR as in § 4.3 and degradation by the held-out loss increase of Section B.1.1. The robustness threshold δ ∗ \delta^* δ ∗ is the perturbation magnitude at which this increase reaches Δ L \Delta L Δ L nats. Across checkpoints, ρ \rho ρ and δ ∗ \delta^* δ ∗ are strongly correlated for both Δ L = 0.05 \Delta L = 0.05 Δ L = 0.05 and 0.1 0.1 0.1 (Figure 7 b): the Spearman correlation over the 72 checkpoints is + 0.83 +0.83 + 0.83 at both thresholds, and within each optimizer it is + 0.87 +0.87 + 0.87 to + 0.89 +0.89 + 0.89 for AdamW and + 0.92 +0.92 + 0.92 to + 0.93 +0.93 + 0.93 for Muon. Full experimental details appear in Section E.3.
Figure 7. The margin-to-spread ratio tracks robustness to weight perturbations. (a) For the linear models trained under GD and Muon, we plot the perturbation σ 50 \sigma_{50} σ 50 required to lose half of the recovered facts against its MSR, h / κ ˉ h/\bar{\kappa} h / κ ˉ (Definition 4.1 ). A higher MSR requires more noise to cause the same loss. (b) We measure the MSR of 72 pretrained language models (30M–600M) as the median of h / κ h/\kappa h / κ over correctly predicted tokens, and plot the perturbation δ ∗ \delta^* δ ∗ at which held-out loss rises by Δ L \Delta L Δ L . The same relationship holds.
Linear associative memory. With σ 50 \sigma_{50} σ 50 as in Section B.1.2, σ 50 \sigma_{50} σ 50 is strongly correlated with MSR for both optimizers (Figure 7 a). Within each optimizer, σ 50 \sigma_{50} σ 50 and ρ \rho ρ move together at every model dimension (Figure 23 ).
Figure 23. Top: σ 50 \sigma_{50} σ 50 against ρ \rho ρ in the linear model, one panel per model dimension, one point per γ \gamma γ cell, below capacity only. Bottom: real language models, δ ∗ \delta^{*} δ ∗ against ρ \rho ρ at every threshold; marker shape is the model size.
Experimental Details
Pretraining
Models and data. All language models are OLMo-style decoder-only transformers (Groeneveld et al., [11 ] ; Team OLMo et al., [34 ] ) with 30M, 60M, 100M, 300M and 600M parameters, trained on DCLM (Li et al., [19 ] ) with the OLMo 2 tokenizer. Budgets span 5–1,280 TPR at 30M, 5–2,560 TPR at 60M, 5–640 TPR at 100M, 5–160 TPR at 300M and 5–40 TPR at 600M.
Optimization. Every model is pretrained with sequence length 4096, a global batch of 2 20 2^{20} 2 20 tokens, a warmup–stable–decay (WSD) schedule, weight decay 0.1 and gradient clipping at 1.0. AdamW uses ( β 1 , β 2 ) = ( 0.9 , 0.98 ) (\beta_1, \beta_2) = (0.9, 0.98) ( β 1 , β 2 ) = ( 0.9 , 0.98 ) . Muon updates the hidden weight matrices; the remaining parameters (embeddings, output layer and norms) are updated by AdamW.
Lowest-pretraining-loss setting
For each model size, budget and optimizer, we sweep the peak learning rate and call the run with the lowest held-out DCLM loss the loss-optimal base model. For Muon we sweep the Muon learning rate and fix the learning rate of the AdamW-updated parameters to the tuned AdamW value of the same cell (Section C.1.2). Table 1 lists the resulting learning rates. In the lowest-pretraining-loss setting , used throughout the main text, each optimizer is represented by its loss-optimal base model alone: we fine-tune it over the full sweep of Section C.2 and compute the learning–forgetting frontier from its fine-tuning runs.
Table 1. Tuned peak pretraining learning rates per model size and budget. For Muon, the Muon learning rate; the AdamW-parameter learning rate is fixed to the tuned AdamW value.
Learning rate of Muon’s AdamW-updated parameters
Muon has a second learning rate for the parameters it leaves to AdamW. Rather than tuning it in every cell, we set it to the tuned AdamW learning rate of the same cell. To check this choice, we pretrain 60M Muon models at 20–160
TPR over a grid of Muon learning rates and three to four AdamW learning rates between 0.5 × 0.5\times 0.5 × and 2.8 × 2.8\times 2.8 × the tuned value. Figure 8 shows the best loss over Muon learning rates for each AdamW learning rate. It varies by at most 0.021 nats, and the tuned value is within 0.016 nats of the best, well below the 0.03–0.20 nats by which Muon beats AdamW at these budgets.
Figure 8. Muon is insensitive to the learning rate of its AdamW-updated parameters. Best pretraining loss over Muon learning rates for each AdamW learning rate, relative to the tuned AdamW value (dashed), for 60M models.
Best-frontier setting
The loss-optimal base model is not always the best starting point for fine-tuning: models pretrained with a larger learning rate can offer a better learning–forgetting tradeoff [38 ] . We therefore also compare the two optimizers in the best-frontier setting, which sweeps over both pretraining and fine-tuning configurations. For each model size, budget and pretraining optimizer, we fine-tune the loss-optimal base model and the base models pretrained at the next five larger learning rates of the sweep grid { 0.005 , 0.007 , 0.01 , 0.014 , 0.02 , 0.028 , 0.04 , 0.056 , 0.08 , 0.12 , 0.16 } \{0.005, 0.007, 0.01, 0.014, 0.02, 0.028, 0.04, 0.056, 0.08, 0.12, 0.16\} { 0.005 , 0.007 , 0.01 , 0.014 , 0.02 , 0.028 , 0.04 , 0.056 , 0.08 , 0.12 , 0.16 } , pool all of their fine-tuning runs, and compute a single learning–forgetting frontier from the pooled runs. This frontier is at least as good as the one of the lowest-pretraining-loss setting, since it contains the loss-optimal base model’s runs. Section D.2 repeats the main results in this setting.
Fine-tuning
Each base model is fine-tuned for 20M tokens on each of Alpaca ([33 ] ), GSM8K ([4 ] ), MusicPile ([41 ] ) and StackMathQA ([43 ] ), with sequence length 1024, a global batch of 65,536 tokens (about 305 steps), a cosine schedule with 20 warmup steps, weight decay 0.1, ( β 1 , β 2 ) = ( 0.9 , 0.95 ) (\beta_1, \beta_2) = (0.9, 0.95) ( β 1 , β 2 ) = ( 0.9 , 0.95 ) and gradient clipping at 1.0. AdamW-pretrained models are fine-tuned with AdamW, and Muon-pretrained models with both AdamW and Muon (Section D.3 motivates this choice). We sweep AdamW learning rates from 10 − 4 10^{-4} 1 0 − 4 to 10 − 2 10^{-2} 1 0 − 2 , and Muon learning rates from 6 × 10 − 4 6 \times10^{-4} 6 × 1 0 − 4 to 2 × 10 − 2 2 \times10^{-2} 2 × 1 0 − 2 with the learning rate of the AdamW-updated parameters set to 0.25 × 0.25\times 0.25 × the Muon learning rate. The sweep ranges from runs that barely move the base model to runs that reach the lowest fine-tuning loss at the cost of more than a nat of forgetting.
Fine-tuning regularization
A natural question is whether the reversal survives when fine-tuning itself is regularized to limit forgetting. We test two standard methods for mitigating catastrophic forgetting, data replay ([30 ] ) and elastic weight consolidation (EWC; [17 ] ), on the loss-optimal base models of every size. Both keep the fine-tuning recipe and learning-rate sweep of Section C.2, and each frontier pools all strengths of the regularizer. Section D.4 reports the results.
Replay. We replace a fraction r ∈ { 0.1 , 0.2 , 0.3 } r \in\{0.1, 0.2, 0.3\} r ∈ { 0.1 , 0.2 , 0.3 } of the fine-tuning tokens with DCLM data drawn from a shard that is disjoint from both the pretraining data and the held-out evaluation shard.
EWC. EWC penalizes movement away from the pretrained weights in the directions that matter for pretraining. It minimizes
L F T ( θ ) + λ 2 ∑ k F k ( θ k − θ P T , k ) 2 , (6) \mathcal{L}_{\mathrm{FT}}(\theta) + \frac{\lambda}{2}\sum_k F_k(\theta_k-\theta_{\mathrm{PT},k})^2, \tag*{(6)} L FT ( θ ) + 2 λ k ∑ F k ( θ k − θ PT , k ) 2 , (6)
where F F F is the diagonal empirical Fisher information of the pretraining loss at θ P T \theta_{\mathrm{PT}} θ PT , estimated from squared gradients over 100 batches of the disjoint DCLM shard used for replay. Following Watts et al. [38 ] , F F F is not normalized, so we sweep λ ∈ { 10 3 , 4 ⋅ 10 3 , 10 4 , 4 ⋅ 10 4 } \lambda\in\{10^3, 4 \cdot10^3, 10^4, 4 \cdot10^4\} λ ∈ { 1 0 3 , 4 ⋅ 1 0 3 , 1 0 4 , 4 ⋅ 1 0 4 } .
Gaussian perturbations
We perturb each base model by adding independent isotropic Gaussian noise to every parameter tensor W W W , scaled so that ∥ Δ W ∥ F = δ ∥ W ∥ F \lVert\Delta W\rVert_F = \delta\lVert W\rVert_F ∥ Δ W ∥ F = δ ∥ W ∥ F , and measure the perturbation degradation Δ n o i s e ( θ P T ; δ ) \Delta_{\mathrm{noise}}(\theta_{\mathrm{PT}}; \delta) Δ noise ( θ PT ; δ ) of Section B.1.1. We average over three noise draws per checkpoint and magnitude, use the loss-optimal checkpoint of each cell, and consider magnitudes from δ = 0.002 \delta= 0.002 δ = 0.002 to 0.56 0.56 0.56 .
Evaluation
Losses. Forgetting is the loss on a held-out DCLM shard of about 4.19M tokens that no model is trained on, and learning is the loss on the validation split of the fine-tuning dataset. Both are mean next-token cross-entropy in nats.
Frontiers. Each fine-tuned checkpoint θ F T \theta_{\mathrm{FT}} θ FT of a base model gives a point ( L P T ( θ F T ) , L F T ( θ F T ) ) (\mathcal{L}_{\mathrm{PT}}(\theta_{\mathrm{FT}}), \mathcal{L}_{\mathrm{FT}}(\theta_{\mathrm{FT}})) ( L PT ( θ FT ) , L FT ( θ FT )) . We compute the learning–forgetting frontier as the lower-left arc of the convex hull of these points, treated as a polyline.
Fine-tuning gap and crossover. We estimate the fine-tuning gap G F T G_{\mathrm{FT}} G FT of § 2 from 100 points sampled uniformly by arc length along the AdamW frontier, each assigned its Euclidean distance to the nearest point of the Muon frontier, negative when that point lies closer to the origin. The crossover TPR is the smallest budget at which G F T > 0 G_{\mathrm{FT}} > 0 G FT > 0 . In both settings, the gap and crossover compare the two optimizers’ frontiers as defined above: one base model per optimizer in the lowest-pretraining-loss setting, pooled base models in the best-frontier setting.
Degradation. For Figure 3 c, the degradation of a base model is the increase in held-out DCLM loss caused by fine-tuning, L P T ( θ F T ) − L P T ( θ P T ) \mathcal{L}_{\mathrm{PT}}(\theta_{\mathrm{FT}}) - \mathcal{L}_{\mathrm{PT}}(\theta_{\mathrm{PT}}) L PT ( θ FT ) − L PT ( θ PT ) . We read L P T ( θ F T ) \mathcal{L}_{\mathrm{PT}}(\theta_{\mathrm{FT}}) L PT ( θ FT ) off the base model’s frontier at a target fine-tuning loss of 1.1 × 1.1\times 1.1 × the lowest fine-tuning loss reached by either optimizer at that model size, so that both optimizers are compared at the same amount of learning.
Additional Results for Section 3
This section extends the fine-tuning results of § 3 . Unless noted otherwise, figures show 60M models, and frontiers are restricted to the region of Section C.5.
Lowest-pretraining-loss setting
Figure 9 shows that the loss-optimal Muon model reaches a lower pretraining loss than its AdamW counterpart at every budget and model size, so the reversal is not inherited from pretraining loss. Figure 10 and Figure 11 extend the frontiers and gaps of the lowest-pretraining-loss setting (Figure 1 and Figure 3 ) to all four datasets, at the budgets around each crossover. On every dataset Muon is better than or level with AdamW below the crossover and AdamW is better beyond it.
Figure 9. Muon reaches lower pretraining loss than AdamW. DCLM held-out loss of the loss-optimal base models across pretraining data budgets and model sizes.
Figure 10. Fine-tuning gap on every fine-tuning dataset, loss-optimal base models (60M). Negative values mean the Muon frontier dominates.
Figure 11. Learning–forgetting frontiers on every fine-tuning dataset, loss-optimal base models (60M). Each row shows the budgets around that dataset’s crossover TPR.
Best-frontier setting
We repeat the analyses of § 3 in the best-frontier setting (Section C.1.3), where each pretraining optimizer is swept across all pretraining and fine-tuning configurations. Figure 12 and Figure 13 show the learning–forgetting frontiers and fine-tuning gaps of 60M models on all four fine-tuning datasets, at the budgets around each crossover. On every dataset, the Muon frontier is better below the crossover; at the crossover AdamW has caught up with or overtaken Muon, and beyond it the AdamW frontier dominates. The fine-tuning gap turns positive at the crossover and stays positive, broadly growing with the data budget. Figure 14 shows that, as in the lowest-pretraining-loss setting (Figure 2 ), the crossover TPR decreases with model size.
Figure 13. Learning–forgetting frontiers on every fine-tuning dataset, best-frontier setting (60M). Each row shows the budgets around that dataset’s crossover TPR.
Figure 14. Crossover TPR decreases with model size, in the best-frontier setting. As model size increases, Muon’s advantage reverses with substantially smaller pretraining data budgets.
Choice of fine-tuning optimizer
Our frontiers fine-tune AdamW-pretrained models with AdamW and Muon-pretrained models with both AdamW and Muon. Figure 15 motivates this choice, and shows that Muon’s greater forgetting at large budgets is a property of the pretrained model rather than of the fine-tuning recipe. For each pretraining optimizer and budget, we take the best 60M base model among those fine-tuned with both optimizers, and plot its AdamW- and Muon-fine-tuned frontiers separately. We use GSM8K, the dataset on which AdamW-pretrained models were fine-tuned with both optimizers across the full range of budgets.
Figure 15. Muon’s greater forgetting comes from the pretrained model, not the fine-tuning optimizer. Learning–forgetting frontiers of the best 60M base models on GSM8K, split by fine-tuning optimizer (solid: AdamW; dashed: Muon). AdamW fine-tuning is always better for AdamW-pretrained models, while the two fine-tuning optimizers give nearly the same frontier for Muon-pretrained models. At large budgets, the AdamW-pretrained model dominates under either choice.
Two observations follow. First, the fine-tuning optimizer matters for AdamW-pretrained models but not for Muon-pretrained ones: on AdamW-pretrained models, AdamW fine-tuning dominates Muon fine-tuning at every budget, whereas the two frontiers of a Muon-pretrained model lie close together with no consistent winner. Second, the reversal does not depend on how the Muon-pretrained model is fine-tuned: at 1.2B tokens both of its frontiers lie below those of the AdamW-pretrained model, and from 19.2B tokens onward the AdamW-pretrained, AdamW-fine-tuned frontier dominates both.
Fine-tuning regularization
We repeat the comparison when fine-tuning is regularized with replay or EWC (setup in Section C.3), using the loss-optimal 60M base models. Figure 16 and Figure 17 show the frontiers on all four datasets; each row shows the budgets around that dataset’s crossover under the regularizer, a quarter and half of the crossover TPR, the crossover itself, and four times it. The crossover is the first budget at which the fine-tuning gap turns positive (Section C.5). Both regularizers reduce forgetting and can move the crossover to larger budgets (from 80 to 320 TPR on Alpaca and StackMathQA), but AdamW’s frontier is better at large budgets on every dataset under both (Figure 18), with one exception: under EWC on StackMathQA the gap turns negative again at the largest budget, 2,560 TPR. With replay, Muon’s frontier is better than or level with AdamW’s below the crossover on every dataset. With EWC the picture below the crossover is mixed: Muon is clearly better at small budgets on GSM8K, while on the other datasets the two frontiers are close.
Figure 16. Learning–forgetting frontiers with replay of 10–30% DCLM data, loss-optimal 60M base models. Each row shows the budgets around that dataset’s crossover under replay.
Figure 17. Learning–forgetting frontiers with EWC, λ ∈ { 10 3 , 4 ⋅ 10 3 , 10 4 , 4 ⋅ 10 4 } \lambda\in\{10^{3}, 4 \cdot10^{3}, 10^{4}, 4 \cdot10^{4}\} λ ∈ { 1 0 3 , 4 ⋅ 1 0 3 , 1 0 4 , 4 ⋅ 1 0 4 } , loss-optimal 60M base models. Each row shows the budgets around that dataset’s crossover under EWC.
Figure 18. Fine-tuning gap under fine-tuning regularization, with replay (solid) and EWC (dashed), loss-optimal 60M base models. Negative values mean the Muon frontier dominates.
Additional Results for Section 4 and Appendix B
This section collects details and additional results for the analysis of § 4 and for the perturbation experiments of Section B: the margin-to-spread ratio across data budgets in the linear memory (Section E.1, for § 4.2 ), perturbation robustness at every magnitude (Section E.2, for Section B.1), and the margin-to-spread ratio and robustness in the linear memory and in language models (Section E.3, for Sections B.2 and 4.3).
Margin-to-spread ratio across data budgets
One step
This is the regime Theorem 4.2 is about: the two solutions are the exact first step from W 0 = 0 W_0 = 0 W 0 = 0 , so every quantity below can be compared against a closed form.
Setup Data. Keys and values are drawn as in § 4.1 at d = 512 d = 512 d = 512 , with N = γ d N = \gamma d N = γ d and γ ∈ { 1 4 , 1 2 , 3 4 , 1 , 3 2 , 2 , 3 , 4 , 5 , 6 , 8 , 10 , 12 , 16 , 20 , 24 , 32 , 48 , 64 } \gamma\in\left\{\frac{1}{4}, \frac{1}{2}, \frac{3}{4}, 1, \frac{3}{2}, 2, 3, 4, 5, 6, 8, 10, 12, 16, 20, 24, 32, 48, 64\right\} γ ∈ { 4 1 , 2 1 , 4 3 , 1 , 2 3 , 2 , 3 , 4 , 5 , 6 , 8 , 10 , 12 , 16 , 20 , 24 , 32 , 48 , 64 } , five random seeds per data budget. § 4.1 requires N − 1 ≥ d N - 1 \ge d N − 1 ≥ d , so the points with γ ≤ 1 \gamma\le1 γ ≤ 1 lie outside the range in which Theorem 4.2 is stated and are drawn only for context.
One step. The gradient of the loss of § 4.1 at W 0 = 0 W_0 = 0 W 0 = 0 is computed exactly, with equal item weights. GD takes it as W G D W^{\mathrm{GD}} W GD ; Muon takes its polar factor W M u o n = Orth ( W G D ) W^{\mathrm{Muon}} = \operatorname{Orth}(W^{\mathrm{GD}}) W Muon = Orth ( W GD ) , computed exactly by singular value decomposition rather than by the Newton–Schulz iteration of Section A.1.2. As a baseline we also record the first AdamW step, which with zero initialization and the usual ϵ \epsilon ϵ is the entrywise sign ( W G D ) \operatorname{sign}(W^{\mathrm{GD}}) sign ( W GD ) ; Theorem 4.2 says nothing about it.
Observables. Every h i h_i h i and κ i \kappa_i κ i is evaluated in closed form for all i i i as in Definition 4.1 , giving h ˉ \bar{h} h ˉ , κ ˉ \bar{\kappa} κ ˉ , ρ \rho ρ and R \mathcal{R} R .
Plotting. Solid lines are means over the five seeds and bands are ± \pm ± one standard deviation; dashed lines are the closed forms behind Theorem 4.2 , with Φ ( γ ) \Phi(\gamma) Φ ( γ ) replaced by its limit 8 3 π \frac{8}{3\pi} 3 π 8 . In the h ˉ \bar{h} h ˉ and κ ˉ \bar{\kappa} κ ˉ panels each solution is normalized to ∥ W ∥ F = 1 \lVert W\rVert_F = 1 ∥ W ∥ F = 1 , which is needed because the two steps have different scales while ρ \rho ρ does not see them; the h ˉ \bar{h} h ˉ and ρ \rho ρ panels are divided by d / γ \sqrt{d/\gamma} d / γ , the unit in which Theorem 4.2 is stated.
Result Figure 19 (top) shows R \mathcal{R} R rising monotonically from 0.53 0.53 0.53 at γ = 1 4 \gamma= \frac{1}{4} γ = 4 1 to 1.16 1.16 1.16 at γ = 64 \gamma= 64 γ = 64 and crossing 1 1 1 near γ = 5 \gamma= 5 γ = 5 , with the closed form tracking it once γ ≳ 2 \gamma\gtrsim2 γ ≳ 2 . The remaining panels separate the two factors. Normalized, GD’s margin is constant in γ \gamma γ , which is the identity h ˉ G D = ∥ W G D ∥ F 2 / ( N − 1 ) \bar{h}_{\mathrm{GD}} = \lVert W^{\mathrm{GD}}\rVert_F^2/(N - 1) h ˉ GD = ∥ W GD ∥ F 2 / ( N − 1 ) read at ∥ W ∥ F = 1 \lVert W\rVert_F = 1 ∥ W ∥ F = 1 , while Muon’s climbs to about 8 3 π \frac{8}{3\pi} 3 π 8 ; both variances approach the same limit, GD’s from above and Muon’s from below. The limiting ratio 3 π 8 \frac{3\pi}{8} 8 3 π is therefore carried by the margin. Its approach to that limit comes from both factors: Muon’s margin contributes the + 1 10 γ +\frac{1}{10\gamma} + 10 γ 1 correction, while the − 9 10 γ -\frac{9}{10\gamma} − 10 γ 9 part of the 9 10 γ \frac{9}{10\gamma} 10 γ 9 term comes from GD’s variance, the factor γ + 2 \gamma+ 2 γ + 2 in ( κ ˉ G D ) 2 (\bar{\kappa}^{\mathrm{GD}})^2 ( κ ˉ GD ) 2 (§ 4.2 ). The bottom row runs the same comparison against the AdamW surrogate: its R \mathcal{R} R has the same shape but reaches only 0.93 0.93 0.93 at γ = 64 \gamma= 64 γ = 64 , so over the data budgets we ran the one-step reversal is a statement about GD, not about AdamW.
Figure 19. One step at d = 512 d = 512 d = 512 , GD against Muon (top) and the AdamW surrogate against Muon (bottom); solid is experiment, dashed the closed form of Theorem 4.2 .
Many steps
The theory covers one step. This experiment runs the same memory to convergence and asks whether the ordering it predicts survives training.
Setup Data. Keys and values are drawn as in § 4.1 , with item i i i queried with probability q i ∝ 1 / i q_i \propto1/i q i ∝ 1/ i (Zipf). We use d ∈ { 64 , 128 , 256 , 512 } d \in\{64, 128, 256, 512\} d ∈ { 64 , 128 , 256 , 512 } , γ \gamma γ log-spaced in [ 1 4 , 128 ] \left[\frac{1}{4}, 128\right] [ 4 1 , 128 ] , and three seeds per ( d , γ ) (d,\gamma) ( d , γ ) shared by the two optimizers.
Training. From W 0 = 0 W_0 = 0 W 0 = 0 we take T = 10 4 T = 10^4 T = 1 0 4 full-batch steps of W t + 1 = W t + 0.02 G t W_{t+1} = W_t + 0.02G_t W t + 1 = W t + 0.02 G t for GD and W t + 1 = W t + 0.005 Orth ( G t ) W_{t+1} = W_t + 0.005\operatorname{Orth}(G_t) W t + 1 = W t + 0.005 Orth ( G t ) for Muon, where G t = ∑ i q i ( 1 − p ^ W t ( i ∣ i ) ) u i v i T G_t = \sum_i q_i(1-\hat{p}_{W_t}(i\mid i))u_i v_i^{\mathsf{T}} G t = ∑ i q i ( 1 − p ^ W t ( i ∣ i )) u i v i T is the Hebbian part of the gradient of the loss of § 4.1 , at constant learning rate, without momentum or weight decay, and with the polar factor again taken by singular value decomposition.
Observables. h i h_i h i and κ i \kappa_i κ i are as in Definition 4.1 , with equal item weights rather than the q i q_i q i used in training. The saturated value of an observable is the per-seed median at t ≥ 10 3 t \ge10^3 t ≥ 1 0 3 ; R \mathcal{R} R is formed per seed and only then averaged. Curves are means over seeds and bands are ± \pm ± one standard deviation.
Perturbed robustness. We perturb the trained weights as in Section B.1.2, W ′ = W T + σ ∥ W T ∥ F ξ W' = W_T + \sigma\lVert W_T\rVert_F \xi W ′ = W T + σ ∥ W T ∥ F ξ with ∥ ξ ∥ F = 1 \lVert\xi\rVert_F = 1 ∥ ξ ∥ F = 1 , over 16 perturbation directions shared by the two optimizers, and read off σ 50 \sigma_{50} σ 50 , the σ \sigma σ at which half of the items correct at σ = 0 \sigma= 0 σ = 0 are lost. For N < d N < d N < d the Muon solution is rescaled to its component in the span of the keys, since outside that span it is an arbitrary orthogonal completion that carries norm but no information. Data budgets at which either optimizer is already past the capacity of the memory, meaning accuracy below 0.99 at σ = 0 \sigma= 0 σ = 0 , are dropped from the σ 50 \sigma_{50} σ 50 panels: there the two optimizers recover different sets of items and the two σ 50 \sigma_{50} σ 50 are not comparable.
Result Training does not undo the one-step picture. Figure 21 shows R \mathcal{R} R increasing with γ \gamma γ at every d d d , over a range from about 0.4 to about 1.2, and the crossing R = 1 \mathcal{R} = 1 R = 1 moving to larger data budgets as d d d grows: γ ∗ ≈ 4.0 \gamma^* \approx4.0 γ ∗ ≈ 4.0 , 4.4, 5.4 and 9.9 at d = 64 d = 64 d = 64 , 128, 256 and 512. Figure 20 shows the two ρ \rho ρ separately: both fall with the data budget, Muon’s from roughly twice GD’s at γ = 1 4 \gamma= \frac{1}{4} γ = 4 1 to below it past γ ⋆ \gamma^\star γ ⋆ . The bottom row shows that the robustness ordering turns over as well, GD’s σ 50 \sigma_{50} σ 50 gaining on Muon’s at every d d d . The two turnovers need not coincide, and they do not: the σ 50 \sigma_{50} σ 50 ordering flips at γ ≈ 0.5 \gamma\approx0.5 γ ≈ 0.5 , 1.3 and 11 at d = 128 d = 128 d = 128 , 256 and 512, and at d = 64 d = 64 d = 64 GD is already the more robust at the smallest data budget we ran.
Figure 21. Saturated R \mathcal{R} R against the data budget, one panel per d d d .
Perturbation robustness
Setup
For every (model size, data budget) cell and each optimizer we take the checkpoint attaining the lowest held-out DCLM loss over the full pretraining learning-rate grid. A Gaussian perturbation is added independently to each parameter tensor of that checkpoint, scaled so that ∥ Δ W ∥ F ≈ δ ∥ W ∥ F \lVert\Delta W\rVert_F \approx\delta\lVert W\rVert_F ∥ Δ W ∥ F ≈ δ ∥ W ∥ F , and we report the increase in held-out DCLM loss over the checkpoint without perturbation, averaged over three perturbation draws. The grid spans three model sizes, 30M to 100M, token–parameter ratios from 5 to 2,560 TPR, and perturbation magnitudes δ \delta δ from 0.005 to 0.28. Magnitudes below δ ≈ 0.02 \delta\approx0.02 δ ≈ 0.02 are omitted from the figures because the increase in loss is then at the level of the seed-to-seed variation for both optimizers.
Table 2 and the robustness counts below use the same protocol on the loss-optimal checkpoints of all five model sizes, 30M to 600M: 36 (model size, data budget) cells and perturbation magnitudes δ \delta δ from 0.002 to 0.56.
Result
Figure 6 fixes the perturbation magnitude; Figure 22 reports the same difference of degradations at every magnitude we sweep. The left three panels fix δ \delta δ and vary the model size, the right panel fixes the model size at 100M and varies δ \delta δ . The difference is positive at every budget, model size and magnitude we tested, and grows with the budget at every δ \delta δ . Magnitudes are not comparable across panels, since the degradation of both optimizers grows steeply with δ \delta δ ; what the panels share is the shape of the curve.
Figure 22. The perturbation degradation gap is positive at every perturbation magnitude. Muon minus AdamW increase in held-out DCLM loss. Left three panels: one δ \delta δ each; right: 100M across δ \delta δ .
Over the 36 cells, the perturbation magnitude δ ⋆ \delta^\star δ ⋆ at which the loss has risen by 0.05, 0.1 or 0.2 nats is larger for AdamW in every cell and at every threshold, by a median factor of 1.4, and Muon’s increase in loss exceeds AdamW’s in every cell at every δ \delta δ from 0.005 to 0.2. The ordering does not depend on which checkpoint of a cell is used: giving each optimizer its most perturbation-robust checkpoint within 0.02 nats of the cell’s best base loss still leaves Muon degrading 20–400% more.
Loss under perturbation. Table 2 compares the loss under perturbation itself, rather than its increase, and places the result next to the fine-tuning crossover of Figure 2 . At every model size, Muon’s loss under perturbation is the lower one at small budgets and AdamW’s at large ones, apart from the three marked entries. The crossover moves to smaller budgets as δ \delta δ grows, so its value is not comparable to the fine-tuning crossover; its trend with model size is. From δ = 0.1 \delta= 0.1 δ = 0.1 on, the crossover budget does not increase with model size, as for every fine-tuning dataset; at δ = 0.05 \delta= 0.05 δ = 0.05 it is not monotone.
Margin-to-spread ratio and robustness
Setup
Linear model. Each cell is a data budget γ = N / d \gamma= N/d γ = N / d and a model dimension d d d , run with three seeds. We train W W W from W 0 = 0 W_0 = 0 W 0 = 0 for T = 10 4 T = 10^4 T = 1 0 4 full-batch steps on Zipf-distributed item frequencies, with GD at a learning rate of 0.02 and Muon at 0.005, and read ρ \rho ρ off the trained W W W . The same W W W is then perturbed, W ′ = W + σ ∥ W ∥ F ξ W' = W + \sigma\lVert W \rVert_F \xi W ′ = W + σ ∥ W ∥ F ξ with ∥ ξ ∥ F = 1 \lVert\xi\rVert_F = 1 ∥ ξ ∥ F = 1 , over 16 perturbation directions shared by the two optimizers, and σ 50 \sigma_{50} σ 50 is the perturbation level at which half of the items recovered at σ = 0 \sigma= 0 σ = 0 are lost, interpolated between grid points on a log scale.
Real language models. The analogue of a stored fact is a next-token prediction, and the analogue of the competitors of Definition 4.1 is the rest of the vocabulary. We evaluate the checkpoint on held-out DCLM text (128 sequences of 4096 tokens, never trained on). At a position with gold token t t t and logit vector z z z , the margins are
m j = z t − z j , j ≠ t , (7) m_j = z_t - z_j,\qquad j \ne t, \tag*{(7)} m j = z t − z j , j = t , (7)
over the whole vocabulary, and h h h and κ 2 \kappa^2 κ 2 are their mean and variance over those ∣ V ∣ − 1 \lvert V\rvert- 1 ∣ V ∣ − 1 competitors, exactly as in Definition 4.1 . Two choices make this well defined. First, only the positions at which the gold token is already the argmax, min j ≠ t m j > 0 \min_{j\ne t}m_j > 0 min j = t m j > 0 , are used: elsewhere the prediction is wrong before any perturbation is added and the margin is not a robustness margin. Second, the position-level values h / κ h/\kappa h / κ are summarised by their median over positions rather than their mean, since the distribution over positions is heavy-tailed. The reduction over the vocabulary axis is carried out in double precision and chunked over positions, so the margin vector is never materialised. Both quantities are measured on the same checkpoints: for every (model size, token budget) cell, the checkpoint with the lowest held-out DCLM loss over the pretraining learning-rate grid, which gives 72 checkpoints over five model sizes. For robustness we take δ ∗ \delta^* δ ∗ , the relative weight-perturbation level at which the held-out loss rises by Δ L \Delta L Δ L nats, interpolated on a log–log scale and averaged over three perturbation draws.
Learning-rate ablation. At two data budgets, each optimizer is run at its learning rate and at that rate scaled by 1 4 \frac{1}{4} 4 1 and by 4, all other settings unchanged, at d = 128 d = 128 d = 128 and T = 10 4 T = 10^4 T = 1 0 4 over several seeds.
Result
Within each optimizer σ 50 \sigma_{50} σ 50 and ρ \rho ρ move together at every model dimension, over more than an order of magnitude in σ 50 \sigma_{50} σ 50 , and the association is monotone in ρ \rho ρ rather than in d d d or γ \gamma γ separately (Figure 23 ). In the language models, the Spearman correlation between ρ \rho ρ and δ ∗ \delta^{*} δ ∗ over the 72 checkpoints is +0.83 at Δ L = 0.05 \Delta L = 0.05 Δ L = 0.05 and 0.1 nats and +0.84 at 0.2 nats; within each optimizer it is +0.87 to +0.89 for AdamW and +0.92 to +0.93 for Muon (Figure 23 ). Across the 16× \times × learning-rate range, accuracy moves by less than one percentage point and ρ \rho ρ by less than 1%, far below the gap between the two optimizers, and the ratio R \mathcal{R} R between them by at most 1.3% (Table 3 ); the comparison in § 4.2 is therefore not an artefact of the learning rates.
Table 3. Learning-rate ablation in the linear model at d = 128 d = 128 d = 128 , T = 10 4 T = 10^4 T = 1 0 4 , mean ± \pm ± sd over seeds; “final” is ρ \rho ρ at the last step and “saturated” its median over t ≥ 10 3 t \ge10^3 t ≥ 1 0 3 . Each R \mathcal{R} R pairs the two rows beside it. Accuracy and ρ \rho ρ move by less than 1% over the 16× \times × range, and so does R \mathcal{R} R .
Proof of Theorem 4.2
This section proves the quantitative bound underlying the informal Theorem 4.2 . For every fixed data budget γ = N / d ≥ 16 \gamma=N/d\ge16 γ = N / d ≥ 16 , we establish
∣ R − 3 π 8 ( 1 − 9 10 γ ) ∣ ≤ C 1 γ − 3 / 2 + o p ( 1 ) (8) \left|\mathcal{R}-\frac{3\pi}{8}\left(1-\frac{9}{10\gamma}\right)\right|\le C_1\gamma^{-3/2}+o_p(1) \tag*{(8)} R − 8 3 π ( 1 − 10 γ 9 ) ≤ C 1 γ − 3/2 + o p ( 1 ) (8)
as d → ∞ d\to\infty d → ∞ , where C 1 C_1 C 1 is an absolute constant. The fixed-percentage increase is established in Lemma F.33; the other conclusions of Theorem 4.2 follow from the negative 1 / γ 1/\gamma 1/ γ term in R − 3 π / 8 \mathcal{R}-3\pi/8 R − 3 π /8 , as detailed below.
We first compute the one-step GD and Muon iterates and collect concentration estimates for the Gaussian data. We then evaluate the mean margin h ˉ \bar h h ˉ and the margin spread κ ˉ \bar\kappa κ ˉ of Definition 4.1 separately for the two iterates, and finally take the ratio of the two margin-to-spread ratios. The GD estimates hold for every γ > 1 \gamma>1 γ > 1 ; the Muon estimates, and hence the theorem, require γ ≥ 16 \gamma\ge16 γ ≥ 16 .
Setup. We work in the setting of § 4.1 with uniform query weights , q i = 1 / N q_i=1/N q i = 1/ N for every fact i i i ; this is the setting of Theorem 4.2 and is assumed throughout. The keys and values u 1 , … , u N , v 1 , … , v N u_1,\ldots,u_N,v_1,\ldots,v_N u 1 , … , u N , v 1 , … , v N are independent N ( 0 , I d ) \mathcal{N}(0,I_d) N ( 0 , I d ) vectors, N = γ d N=\gamma d N = γ d with the data budget γ > 1 \gamma>1 γ > 1 held fixed, N − 1 ≥ d N-1\ge d N − 1 ≥ d , and d → ∞ d\to\infty d → ∞ ; O p O_p O p and o p o_p o p are as in § 4.1 .
Notation. We collect the data in U = [ u 1 ⋯ u N ] ∈ R d × N U=[u_1\ \cdots\ u_N]\in\mathbb{R}^{d\times N} U = [ u 1 ⋯ u N ] ∈ R d × N and V = [ v 1 ⋯ v N ] ∈ R d × N V=[v_1\ \cdots\ v_N]\in\mathbb{R}^{d\times N} V = [ v 1 ⋯ v N ] ∈ R d × N , and write 1 = ( 1 , … , 1 ) ⊤ \mathbf{1}=(1,\ldots,1)^\top 1 = ( 1 , … , 1 ) ⊤ , e i e_i e i for the i i i -th standard basis vector and ⟨ X , Y ⟩ = tr ( X ⊤ Y ) \langle X,Y\rangle=\operatorname{tr}(X^\top Y) ⟨ X , Y ⟩ = tr ( X ⊤ Y ) . The operator, Frobenius and nuclear norms ∥ ⋅ ∥ o p \|\cdot\|_{\mathrm{op}} ∥ ⋅ ∥ op , ∥ ⋅ ∥ F \|\cdot\|_{\mathrm{F}} ∥ ⋅ ∥ F and ∥ ⋅ ∥ ∗ \|\cdot\|_* ∥ ⋅ ∥ ∗ satisfy
∥ X ∥ o p ≤ ∥ X ∥ F ≤ ∥ X ∥ ∗ ≤ rank ( X ) ∥ X ∥ o p , ∥ X ∥ F ≤ rank ( X ) ∥ X ∥ o p , \|X\|_{\mathrm{op}}\le\|X\|_{\mathrm{F}}\le\|X\|_*\le\operatorname{rank}(X)\|X\|_{\mathrm{op}},\qquad\|X\|_{\mathrm{F}}\le\sqrt{\operatorname{rank}(X)}\|X\|_{\mathrm{op}}, ∥ X ∥ op ≤ ∥ X ∥ F ≤ ∥ X ∥ ∗ ≤ rank ( X ) ∥ X ∥ op , ∥ X ∥ F ≤ rank ( X ) ∥ X ∥ op ,
∣ ⟨ X , Y ⟩ ∣ ≤ ∥ X ∥ ∗ ∥ Y ∥ o p . (9) |\langle X,Y\rangle|\le\|X\|_*\|Y\|_{\mathrm{op}}. \tag*{(9)} ∣ ⟨ X , Y ⟩ ∣ ≤ ∥ X ∥ ∗ ∥ Y ∥ op . (9)
For X ∈ R d × d X\in\mathbb{R}^{d\times d} X ∈ R d × d with compact SVD X = P Σ Q ⊤ X=P\Sigma Q^\top X = P Σ Q ⊤ we write Orth ( X ) = P Q ⊤ \operatorname{Orth}(X)=PQ^\top Orth ( X ) = P Q ⊤ for its polar factor, so that Orth ( c X ) = Orth ( X ) \operatorname{Orth}(cX)=\operatorname{Orth}(X) Orth ( c X ) = Orth ( X ) for c > 0 c>0 c > 0 and ∥ Orth ( X ) ∥ o p ≤ 1 \|\operatorname{Orth}(X)\|_{\mathrm{op}}\le1 ∥ Orth ( X ) ∥ op ≤ 1 .
Scores and observables. By Definition 4.1 , for any W W W ,
h ˉ = mean i u i ⊤ W v i − mean i mean j ≠ i u j ⊤ W v i , \bar h=\operatorname{mean}_{i}u_i^\top Wv_i-\operatorname{mean}_{i}\operatorname{mean}_{j\ne i}u_j^\top Wv_i, h ˉ = mean i u i ⊤ W v i − mean i mean j = i u j ⊤ W v i ,
κ ˉ 2 = 1 N ( N − 1 ) ∑ i = 1 N ∑ j ≠ i ( u j ⊤ W v i ) 2 − mean i ( mean j ≠ i u j ⊤ W v i ) 2 . (10) \bar\kappa^2=\frac{1}{N(N-1)}\sum_{i=1}^{N}\sum_{j\ne i}(u_j^\top Wv_i)^2-\operatorname{mean}_{i}\left(\operatorname{mean}_{j\ne i}u_j^\top Wv_i\right)^2. \tag*{(10)} κ ˉ 2 = N ( N − 1 ) 1 i = 1 ∑ N j = i ∑ ( u j ⊤ W v i ) 2 − mean i ( mean j = i u j ⊤ W v i ) 2 . (10)
where the scores are written as u j ⊤ W v i u_j^\top Wv_i u j ⊤ W v i rather than v j ⊤ W u i v_j^\top Wu_i v j ⊤ W u i as in § 4.1 . The two conventions are exchanged by swapping the roles of U U U and V V V : reading § 4.1 with ( U , V ) (U,V) ( U , V ) replaced by ( V , U ) (V,U) ( V , U ) turns every score v j ⊤ W u i v_j^\top Wu_i v j ⊤ W u i into u j ⊤ W v i u_j^\top Wv_i u j ⊤ W v i and its one-step iterate into Equation (11) , leaving the loss and every h i h_i h i and κ i \kappa_i κ i unchanged. Since U U U and V V V are i.i.d., the law of ρ \rho ρ is the same under either convention.
The two one-step iterates
Lemma F.1 (One step). Up to a positive multiple, one step of GD and one step of Muon from W 0 = 0 W_0=0 W 0 = 0 give
W G D = U V ⊤ − 1 N ( U 1 ) ( V 1 ) ⊤ = U ( I N − 11 ⊤ N ) V ⊤ , W M u o n = Orth ( W G D ) . (11) W^{\mathrm{GD}}=UV^\top-\frac{1}{N}(U\mathbf{1})(V\mathbf{1})^\top=U\left(I_N-\frac{\mathbf{1}\mathbf{1}^\top}{N}\right)V^\top,\qquad W^{\mathrm{Muon}}=\operatorname{Orth}\left(W^{\mathrm{GD}}\right). \tag*{(11)} W GD = U V ⊤ − N 1 ( U 1 ) ( V 1 ) ⊤ = U ( I N − N 1 1 ⊤ ) V ⊤ , W Muon = Orth ( W GD ) . (11) The positive multiple is immaterial for ρ \rho ρ .
Proof. Gradient. With ∂ W ( u j ⊤ W v i ) = u j v i ⊤ \partial_W(u_j^\top Wv_i)=u_jv_i^\top ∂ W ( u j ⊤ W v i ) = u j v i ⊤ , the gradient of the loss of § 4.1 with uniform weights q i = 1 / N q_i=1/N q i = 1/ N is
− ∇ W L ( W ) = 1 N ∑ i = 1 N ( u i − ∑ j = 1 N p ^ W ( j ∣ i ) u j ) v i ⊤ . (12) -\nabla_W\mathcal{L}(W)=\frac{1}{N}\sum_{i=1}^{N}\left(u_i-\sum_{j=1}^{N}\widehat p_W(j\mid i)u_j\right)v_i^\top. \tag*{(12)} − ∇ W L ( W ) = N 1 i = 1 ∑ N ( u i − j = 1 ∑ N p W ( j ∣ i ) u j ) v i ⊤ . (12)
At W = 0 W=0 W = 0 all scores vanish, so p ^ 0 ( j ∣ i ) = 1 / N \widehat p_0(j\mid i)=1/N p 0 ( j ∣ i ) = 1/ N ; moreover ∑ j u j = U 1 \sum_j u_j=U\mathbf{1} ∑ j u j = U 1 , ∑ i u i v i ⊤ = U V ⊤ \sum_i u_iv_i^\top=UV^\top ∑ i u i v i ⊤ = U V ⊤ and ∑ i v i ⊤ = ( V 1 ) ⊤ \sum_i v_i^\top=(V\mathbf{1})^\top ∑ i v i ⊤ = ( V 1 ) ⊤ , whence
− ∇ W L ( 0 ) = 1 N ( U V ⊤ − 1 N ( U 1 ) ( V 1 ) ⊤ ) = 1 N U ( I N − 11 ⊤ N ) V ⊤ . (13) -\nabla_W\mathcal{L}(0)=\frac{1}{N}\left(UV^\top-\frac{1}{N}(U\mathbf{1})(V\mathbf{1})^\top\right)=\frac{1}{N}U\left(I_N-\frac{\mathbf{1}\mathbf{1}^\top}{N}\right)V^\top. \tag*{(13)} − ∇ W L ( 0 ) = N 1 ( U V ⊤ − N 1 ( U 1 ) ( V 1 ) ⊤ ) = N 1 U ( I N − N 1 1 ⊤ ) V ⊤ . (13)
One step. With step size η > 0 \eta> 0 η > 0 , GD gives − η ∇ W L ( 0 ) = η N W G D -\eta\nabla_W\mathcal{L}(0) = \frac{\eta}{N}W^{\mathrm{GD}} − η ∇ W L ( 0 ) = N η W GD and Muon gives η Orth ( − ∇ W L ( 0 ) ) = η Orth ( W G D ) \eta\operatorname{Orth}(-\nabla_W\mathcal{L}(0)) = \eta\operatorname{Orth}(W^{\mathrm{GD}}) η Orth ( − ∇ W L ( 0 )) = η Orth ( W GD ) , using Orth ( c X ) = Orth ( X ) \operatorname{Orth}(cX) = \operatorname{Orth}(X) Orth ( c X ) = Orth ( X ) for c > 0 c > 0 c > 0 . Normalization. Replacing W W W by c W cW c W with c > 0 c > 0 c > 0 multiplies every score u j ⊤ W v i u_j^{\top}Wv_i u j ⊤ W v i by c c c , hence h i ↦ c h i h_i \mapsto ch_i h i ↦ c h i and κ i ↦ c κ i \kappa_i \mapsto c\kappa_i κ i ↦ c κ i , so h ˉ ↦ c h ˉ \bar{h} \mapsto c\bar{h} h ˉ ↦ c h ˉ , κ ˉ ↦ c κ ˉ \bar{\kappa} \mapsto c\bar{\kappa} κ ˉ ↦ c κ ˉ and ρ \rho ρ is unchanged.
Lemma F.2 (Full rank and orthogonality). Almost surely rank W G D = d \operatorname{rank}W^{\mathrm{GD}} = d rank W GD = d . Consequently W M u o n W^{\mathrm{Muon}} W Muon is an orthogonal matrix: ∥ W M u o n x ∥ = ∥ x ∥ \lVert W^{\mathrm{Muon}}x\rVert= \lVert x\rVert ∥ W Muon x ∥ = ∥ x ∥ , ∥ W M u o n ∥ o p = 1 \lVert W^{\mathrm{Muon}}\rVert_{\mathrm{op}} = 1 ∥ W Muon ∥ op = 1 and ∥ W M u o n ∥ F 2 = d \lVert W^{\mathrm{Muon}}\rVert_{\mathrm{F}}^2 = d ∥ W Muon ∥ F 2 = d .
Proof. det W G D \det W^{\mathrm{GD}} det W GD is a polynomial in the entries of ( U , V ) (U,V) ( U , V ) . Taking U = [ I d 0 ] U = [I_d\ 0] U = [ I d 0 ] and V = [ I d − ∑ k ≤ d e k 0 ] V = [I_d\ -\sum_{k\leq d}e_k\ 0] V = [ I d − ∑ k ≤ d e k 0 ] , which is possible because N ≥ d + 1 N \geq d+1 N ≥ d + 1 , gives V 1 = 0 V1 = 0 V 1 = 0 and W G D = U V ⊤ = I d W^{\mathrm{GD}} = UV^{\top} = I_d W GD = U V ⊤ = I d , so the polynomial is not identically zero; since ( U , V ) (U,V) ( U , V ) has a density, Pr [ det W G D = 0 ] = 0 \Pr[\det W^{\mathrm{GD}} = 0] = 0 Pr [ det W GD = 0 ] = 0 . At full rank the factors P , Q P,Q P , Q of the compact SVD are d × d d \times d d × d orthogonal matrices, so W M u o n = P Q ⊤ W^{\mathrm{Muon}} = PQ^{\top} W Muon = P Q ⊤ is orthogonal and ∥ W M u o n ∥ F 2 = rank W G D = d \lVert W^{\mathrm{Muon}}\rVert_{\mathrm{F}}^2 = \operatorname{rank}W^{\mathrm{GD}} = d ∥ W Muon ∥ F 2 = rank W GD = d .
Concentration estimates for the data
Lemma F.3 (Concentration for U , V U,V U , V ). As d → ∞ d \to\infty d → ∞ along the sequence above:
mean k ∥ u k ∥ 2 = d ( 1 + O p ( ( N d ) − 1 / 2 ) ) \operatorname{mean}_k\lVert u_k\rVert^2 = d(1 + O_p((Nd)^{-1/2})) mean k ∥ u k ∥ 2 = d ( 1 + O p (( N d ) − 1/2 )) , and likewise for V V V .
mean k ( ∥ u k ∥ ∥ v k ∥ − d ) 2 = O p ( d ) \operatorname{mean}_k(\lVert u_k\rVert\lVert v_k\rVert- d)^2 = O_p(d) mean k (∥ u k ∥ ∥ v k ∥ − d ) 2 = O p ( d ) .
∥ U 1 ∥ 2 / ( N d ) → 1 \lVert U1\rVert^2/(Nd) \to1 ∥ U 1 ∥ 2 / ( N d ) → 1 in probability; in particular ∥ U 1 ∥ 2 = O p ( N d ) \lVert U1\rVert^2 = O_p(Nd) ∥ U 1 ∥ 2 = O p ( N d ) . Likewise for V V V .
∥ U ∥ o p / N → 1 + γ − 1 / 2 \lVert U\rVert_{\mathrm{op}}/\sqrt{N} \to1 + \gamma^{-1/2} ∥ U ∥ op / N → 1 + γ − 1/2 almost surely, hence ∥ U ∥ o p 2 ≤ 4 N ( 1 + o p ( 1 ) ) \lVert U\rVert_{\mathrm{op}}^2 \leq4N(1 + o_p(1)) ∥ U ∥ op 2 ≤ 4 N ( 1 + o p ( 1 )) ; and for every m ≥ 1 m \geq1 m ≥ 1 there is C m C_m C m with E ∥ U ∥ o p 2 m ≤ C m N m \mathbb{E}\lVert U\rVert_{\mathrm{op}}^{2m} \leq C_mN^m E ∥ U ∥ op 2 m ≤ C m N m . Likewise for V V V .
∥ U U ⊤ − N I d ∥ o p = O p ( N d ) \lVert UU^{\top} - NI_d\rVert_{\mathrm{op}} = O_p(\sqrt{N}d) ∥ U U ⊤ − N I d ∥ op = O p ( N d ) . If γ ≥ 16 \gamma\geq16 γ ≥ 16 , then with probability tending to one both S = 1 N V V ⊤ S = \frac{1}{N}VV^{\top} S = N 1 V V ⊤ and S = 1 N ( V V ⊤ − 1 N ( V 1 ) ( V 1 ) ⊤ ) S = \frac{1}{N}(VV^{\top} - \frac{1}{N}(V1)(V1)^{\top}) S = N 1 ( V V ⊤ − N 1 ( V 1 ) ( V 1 ) ⊤ ) satisfy ∥ S − I d ∥ o p ≤ 3 γ − 1 / 2 \lVert S-I_d\rVert_{\mathrm{op}} \leq3\gamma^{-1/2} ∥ S − I d ∥ op ≤ 3 γ − 1/2 and λ min ( S ) ≥ 1 8 \lambda_{\min}(S) \geq\frac{1}{8} λ m i n ( S ) ≥ 8 1 .
∥ V V ⊤ ∥ F 2 = O p ( N 2 d ) \lVert VV^{\top}\rVert_{\mathrm{F}}^2 = O_p(N^2d) ∥ V V ⊤ ∥ F 2 = O p ( N 2 d ) , and E ∥ V V ⊤ ∥ F 2 m ≤ C 2 m ( N 2 d ) m \mathbb{E}\lVert VV^{\top}\rVert_{\mathrm{F}}^{2m} \leq C_{2m}(N^2d)^m E ∥ V V ⊤ ∥ F 2 m ≤ C 2 m ( N 2 d ) m . Likewise for U U ⊤ UU^{\top} U U ⊤ .
∥ U V ⊤ ∥ F 2 = N d 2 ( 1 + O p ( d − 1 ) ) \lVert UV^{\top}\rVert_{\mathrm{F}}^2 = Nd^2(1 + O_p(d^{-1})) ∥ U V ⊤ ∥ F 2 = N d 2 ( 1 + O p ( d − 1 )) and ∥ U V ⊤ ∥ F 2 − d ∑ k ∥ v k ∥ 2 = O p ( N d 3 / 2 + N d ) \lVert UV^{\top}\rVert_{\mathrm{F}}^2 - d\sum_k\lVert v_k\rVert^2 = O_p(\sqrt{N}d^{3/2} + Nd) ∥ U V ⊤ ∥ F 2 − d ∑ k ∥ v k ∥ 2 = O p ( N d 3/2 + N d ) .
U V ⊤ = d ( U U ⊤ ) 1 / 2 Ξ UV^{\top} \mathrel{\overset{d}{=}} (UU^{\top})^{1/2}\Xi U V ⊤ = d ( U U ⊤ ) 1/2 Ξ with Ξ ∈ R d × d \Xi\in\mathbb{R}^{d\times d} Ξ ∈ R d × d having i.i.d. N ( 0 , 1 ) \mathcal{N}(0,1) N ( 0 , 1 ) entries independent of U U U ; with probability tending to one, ∥ U V ⊤ ∥ o p ≤ 5 N d \lVert UV^{\top}\rVert_{\mathrm{op}} \leq5\sqrt{N}d ∥ U V ⊤ ∥ op ≤ 5 N d .
Proof. (i) ∑ k ∥ u k ∥ 2 ∼ χ N d 2 \sum_k\lVert u_k\rVert^2 \sim\chi^2_{Nd} ∑ k ∥ u k ∥ 2 ∼ χ N d 2 has mean N d Nd N d and variance 2 N d 2Nd 2 N d , so Chebyshev gives ∑ k ∥ u k ∥ 2 = N d + O p ( N d ) \sum_k\lVert u_k\rVert^2 = Nd + O_p(\sqrt{Nd}) ∑ k ∥ u k ∥ 2 = N d + O p ( N d ) ; divide by N N N .
(ii) For x ≥ 0 x \geq0 x ≥ 0 one has x ≥ 1 + x − 1 2 − ( x − 1 ) 2 8 \sqrt{x} \geq1 + \frac{x-1}{2} - \frac{(x-1)^2}{8} x ≥ 1 + 2 x − 1 − 8 ( x − 1 ) 2 : the difference g g g of the two sides satisfies g ( 0 ) = g ( 1 ) = g ′ ( 1 ) = 0 g(0) = g(1) = g'(1) = 0 g ( 0 ) = g ( 1 ) = g ′ ( 1 ) = 0 and g ′ ′ ( x ) = 1 − 1 4 x − 3 / 2 g''(x) = 1 - \frac{1}{4}x^{-3/2} g ′′ ( x ) = 1 − 4 1 x − 3/2 changes sign once, at x 0 = 4 − 2 / 3 x_0 = 4^{-2/3} x 0 = 4 − 2/3 , so g g g is concave on [ 0 , x 0 ] [0,x_0] [ 0 , x 0 ] and convex on [ x 0 , ∞ ) [x_0,\infty) [ x 0 , ∞ ) , and the two boundary values force g ≥ 0 g \geq0 g ≥ 0 . With u ∼ N ( 0 , I d ) u \sim\mathcal{N}(0,I_d) u ∼ N ( 0 , I d ) , E ∥ u ∥ 2 / d = 1 \mathbb{E}\lVert u\rVert^2/d = 1 E ∥ u ∥ 2 / d = 1 and var ( ∥ u ∥ 2 / d ) = 2 / d \operatorname{var}(\lVert u\rVert^2/d) = 2/d var (∥ u ∥ 2 / d ) = 2/ d , so E ∥ u ∥ ≥ d ( 1 − 1 d ) \mathbb{E}\lVert u\rVert\geq\sqrt{d}(1-\frac{1}{d}) E ∥ u ∥ ≥ d ( 1 − d 1 ) and ( E ∥ u ∥ ) 2 ≥ d − 2 (\mathbb{E}\lVert u\rVert)^2 \geq d-2 ( E ∥ u ∥ ) 2 ≥ d − 2 . Hence for independent u , v u,v u , v ,
E ( ∥ u ∥ ∥ v ∥ − d ) 2 = E ∥ u ∥ 2 E ∥ v ∥ 2 − 2 d E ∥ u ∥ E ∥ v ∥ + d 2 ≤ 2 d 2 − 2 d ( d − 2 ) = 4 d , (14) \mathbb{E}(\lVert u\rVert\lVert v\rVert-d)^2 = \mathbb{E}\lVert u\rVert^2\mathbb{E}\lVert v\rVert^2 - 2d\mathbb{E}\lVert u\rVert\mathbb{E}\lVert v\rVert+ d^2 \leq2d^2 - 2d(d-2) = 4d, \tag*{(14)} E (∥ u ∥ ∥ v ∥ − d ) 2 = E ∥ u ∥ 2 E ∥ v ∥ 2 − 2 d E ∥ u ∥ E ∥ v ∥ + d 2 ≤ 2 d 2 − 2 d ( d − 2 ) = 4 d , (14)
and Markov’s inequality gives the claim.
(iii) U 1 = ∑ k u k ∼ N ( 0 , N I d ) U1 = \sum_k u_k \sim\mathcal{N}(0,NI_d) U 1 = ∑ k u k ∼ N ( 0 , N I d ) , so ∥ U 1 ∥ 2 / N ∼ χ d 2 \lVert U1\rVert^2/N \sim\chi^2_d ∥ U 1 ∥ 2 / N ∼ χ d 2 has mean d d d and variance 2 d 2d 2 d ; Chebyshev gives ∥ U 1 ∥ 2 / ( N d ) = 1 + O p ( d − 1 / 2 ) \lVert U1\rVert^2/(Nd) = 1 + O_p(d^{-1/2}) ∥ U 1 ∥ 2 / ( N d ) = 1 + O p ( d − 1/2 ) .
(iv) ∥ U ∥ o p 2 = λ max ( U U ⊤ ) \lVert U\rVert_{\mathrm{op}}^2 = \lambda_{\max}(UU^{\top}) ∥ U ∥ op 2 = λ m a x ( U U ⊤ ) , and by the Bai–Yin law λ max ( U U ⊤ / N ) → ( 1 + γ − 1 / 2 ) 2 ≤ 4 \lambda_{\max}(UU^{\top}/N) \to(1+\gamma^{-1/2})^2 \leq4 λ m a x ( U U ⊤ / N ) → ( 1 + γ − 1/2 ) 2 ≤ 4 almost surely, since γ ≥ 1 \gamma\geq1 γ ≥ 1 . For the moments, U ↦ ∥ U ∥ o p U \mapsto\lVert U\rVert_{\mathrm{op}} U ↦ ∥ U ∥ op is 1-Lipschitz for ∥ ⋅ ∥ F \lVert\cdot\rVert_{\mathrm{F}} ∥ ⋅ ∥ F and E ∥ U ∥ o p ≤ N + d ≤ 2 N \mathbb{E}\lVert U\rVert_{\mathrm{op}} \leq\sqrt{N}+\sqrt{d} \leq2\sqrt{N} E ∥ U ∥ op ≤ N + d ≤ 2 N by Gordon’s theorem, so Gaussian concentration gives Pr [ ∥ U ∥ o p ≥ 2 N + t ] ≤ e − t 2 / 2 \Pr[\lVert U\rVert_{\mathrm{op}} \geq2\sqrt{N}+t] \leq e^{-t^2/2} Pr [∥ U ∥ op ≥ 2 N + t ] ≤ e − t 2 /2 ; integrating the tail bounds all moments.
(v) By Bai–Yin, λ max ( U U ⊤ / N ) → ( 1 + γ − 1 / 2 ) 2 \lambda_{\max}(UU^{\top}/N) \to(1+\gamma^{-1/2})^2 λ m a x ( U U ⊤ / N ) → ( 1 + γ − 1/2 ) 2 and λ min ( U U ⊤ / N ) → ( 1 − γ − 1 / 2 ) 2 \lambda_{\min}(UU^{\top}/N) \to(1-\gamma^{-1/2})^2 λ m i n ( U U ⊤ / N ) → ( 1 − γ − 1/2 ) 2 almost surely, so ∥ U U ⊤ / N − I d ∥ o p → max { ( 1 + γ − 1 / 2 ) 2 − 1 , 1 − ( 1 − γ − 1 / 2 ) 2 } = 2 γ − 1 / 2 + γ − 1 \lVert UU^{\top}/N-I_d\rVert_{\mathrm{op}} \to\max\{(1+\gamma^{-1/2})^2-1,1-(1-\gamma^{-1/2})^2\} = 2\gamma^{-1/2}+\gamma^{-1} ∥ U U ⊤ / N − I d ∥ op → max {( 1 + γ − 1/2 ) 2 − 1 , 1 − ( 1 − γ − 1/2 ) 2 } = 2 γ − 1/2 + γ − 1 , which is at most 3 γ − 1 / 2 < 4 γ − 1 / 2 3\gamma^{-1/2} < 4\gamma^{-1/2} 3 γ − 1/2 < 4 γ − 1/2 ; multiplying by N N N gives the first claim. For γ ≥ 16 \gamma\geq16 γ ≥ 16 we have γ − 1 / 2 ≤ 1 4 \gamma^{-1/2} \leq\frac{1}{4} γ − 1/2 ≤ 4 1 , so in probability ∥ V V ⊤ / N − I d ∥ o p → 2 γ − 1 / 2 + γ − 1 ≤ 9 4 γ − 1 / 2 < 3 γ − 1 / 2 \lVert VV^{\top}/N-I_d\rVert_{\mathrm{op}} \to2\gamma^{-1/2}+\gamma^{-1} \leq\frac{9}{4}\gamma^{-1/2} < 3\gamma^{-1/2} ∥ V V ⊤ / N − I d ∥ op → 2 γ − 1/2 + γ − 1 ≤ 4 9 γ − 1/2 < 3 γ − 1/2 and λ min ( V V ⊤ / N ) → ( 1 − γ − 1 / 2 ) 2 ≥ 9 16 > 1 8 \lambda_{\min}(VV^{\top}/N) \to(1-\gamma^{-1/2})^2 \geq\frac{9}{16} > \frac{1}{8} λ m i n ( V V ⊤ / N ) → ( 1 − γ − 1/2 ) 2 ≥ 16 9 > 8 1 , and the inequalities being strict they hold with probability tending to one. For the centred version, write 1 N ( V V T − 1 N ( V 1 ) ( V 1 ) T ) = V V T N − 1 N 2 ( V 1 ) ( V 1 ) T \frac{1}{N}\left(VV^{\mathsf{T}}-\frac{1}{N}(V\mathbf{1})(V\mathbf{1})^{\mathsf{T}}\right)=\frac{VV^{\mathsf{T}}}{N}-\frac{1}{N^{2}}(V\mathbf{1})(V\mathbf{1})^{\mathsf{T}} N 1 ( V V T − N 1 ( V 1 ) ( V 1 ) T ) = N V V T − N 2 1 ( V 1 ) ( V 1 ) T , where 1 N 2 ( V 1 ) ( V 1 ) T ⪰ 0 \frac{1}{N^{2}}(V\mathbf{1})(V\mathbf{1})^{\mathsf{T}}\succeq0 N 2 1 ( V 1 ) ( V 1 ) T ⪰ 0 has operator norm ∥ V 1 ∥ 2 / N 2 = ∥ V 1 ∥ 2 N d ⋅ d N → γ − 1 \|V\mathbf{1}\|^{2}/N^{2}=\frac{\|V\mathbf{1}\|^{2}}{Nd}\cdot\frac{d}{N}\to\gamma^{-1} ∥ V 1 ∥ 2 / N 2 = N d ∥ V 1 ∥ 2 ⋅ N d → γ − 1 in probability by (iii). By Weyl’s inequality,
λ min ( 1 N ( V V T − 1 N ( V 1 ) ( V 1 ) T ) ) ≥ λ min ( V V T N ) − ∥ V 1 ∥ 2 N 2 , \lambda_{\min}\left(\frac{1}{N}\left(VV^{\mathsf{T}}-\frac{1}{N}(V\mathbf{1})(V\mathbf{1})^{\mathsf{T}}\right)\right)\geq\lambda_{\min}\left(\frac{VV^{\mathsf{T}}}{N}\right)-\frac{\|V\mathbf{1}\|^{2}}{N^{2}}, λ m i n ( N 1 ( V V T − N 1 ( V 1 ) ( V 1 ) T ) ) ≥ λ m i n ( N V V T ) − N 2 ∥ V 1 ∥ 2 ,
∥ 1 N ( V V T − 1 N ( V 1 ) ( V 1 ) T ) − I d ∥ o p ≤ ∥ V V T N − I d ∥ o p + ∥ V 1 ∥ 2 N 2 , (15) \left\|\frac{1}{N}\left(VV^{\mathsf{T}}-\frac{1}{N}(V\mathbf{1})(V\mathbf{1})^{\mathsf{T}}\right)-I_d\right\|_{\mathrm{op}}\leq\left\|\frac{VV^{\mathsf{T}}}{N}-I_d\right\|_{\mathrm{op}}+\frac{\|V\mathbf{1}\|^{2}}{N^{2}}, \tag*{(15)} N 1 ( V V T − N 1 ( V 1 ) ( V 1 ) T ) − I d op ≤ N V V T − I d op + N 2 ∥ V 1 ∥ 2 , (15)
and in probability the two right-hand sides converge to
( 1 − γ − 1 / 2 ) 2 − γ − 1 ≥ 9 16 − 1 16 = 1 2 > 1 8 , 2 γ − 1 / 2 + 2 γ − 1 ≤ 5 2 γ − 1 / 2 < 3 γ − 1 / 2 , (16) (1-\gamma^{-1/2})^{2}-\gamma^{-1}\geq\frac{9}{16}-\frac{1}{16}=\frac{1}{2}>\frac{1}{8},\qquad2\gamma^{-1/2}+2\gamma^{-1}\leq\frac{5}{2}\gamma^{-1/2}<3\gamma^{-1/2}, \tag*{(16)} ( 1 − γ − 1/2 ) 2 − γ − 1 ≥ 16 9 − 16 1 = 2 1 > 8 1 , 2 γ − 1/2 + 2 γ − 1 ≤ 2 5 γ − 1/2 < 3 γ − 1/2 , (16)
using γ − 1 / 2 ≤ 1 4 \gamma^{-1/2}\leq\frac{1}{4} γ − 1/2 ≤ 4 1 ; the inequalities being strict, they hold with probability tending to one.
(v) V V T VV^{\mathsf{T}} V V T has rank at most d d d , so ∥ V V T ∥ F ≤ d ∥ V V T ∥ o p = d ∥ V ∥ o p 2 \|VV^{\mathsf{T}}\|_{F}\leq\sqrt{d}\|VV^{\mathsf{T}}\|_{\mathrm{op}}=\sqrt{d}\|V\|_{\mathrm{op}}^{2} ∥ V V T ∥ F ≤ d ∥ V V T ∥ op = d ∥ V ∥ op 2 by Equation (9) ; now use (iv).
(vii) Expanding, ∥ U V T ∥ F 2 = ∑ k , l ( u k T u l ) ( v k T v l ) = ∑ k ∥ u k ∥ 2 ∥ v k ∥ 2 + ∑ k ≠ l ( u k T u l ) ( v k T v l ) \|UV^{\mathsf{T}}\|_{F}^{2}=\sum_{k,l}(u_k^{\mathsf{T}}u_l)(v_k^{\mathsf{T}}v_l)=\sum_k\|u_k\|^{2}\|v_k\|^{2}+\sum_{k\ne l}(u_k^{\mathsf{T}}u_l)(v_k^{\mathsf{T}}v_l) ∥ U V T ∥ F 2 = ∑ k , l ( u k T u l ) ( v k T v l ) = ∑ k ∥ u k ∥ 2 ∥ v k ∥ 2 + ∑ k = l ( u k T u l ) ( v k T v l ) .
Mean. E ∥ u k ∥ 2 ∥ v k ∥ 2 = d 2 \mathbb{E}\|u_k\|^{2}\|v_k\|^{2}=d^{2} E ∥ u k ∥ 2 ∥ v k ∥ 2 = d 2 and, for k ≠ l k\ne l k = l , E ( u k T u l ) ( v k T v l ) = 0 \mathbb{E}(u_k^{\mathsf{T}}u_l)(v_k^{\mathsf{T}}v_l)=0 E ( u k T u l ) ( v k T v l ) = 0 , so E ∥ U V T ∥ F 2 = N d 2 \mathbb{E}\|UV^{\mathsf{T}}\|_{F}^{2}=Nd^{2} E ∥ U V T ∥ F 2 = N d 2 .
Variance. The variables ∥ u k ∥ 2 ∥ v k ∥ 2 \|u_k\|^{2}\|v_k\|^{2} ∥ u k ∥ 2 ∥ v k ∥ 2 , k ∈ [ N ] k\in[N] k ∈ [ N ] , are independent with var ( ∥ u k ∥ 2 ∥ v k ∥ 2 ) = ( d 2 + 2 d ) 2 − d 4 ≤ 8 d 3 \operatorname{var}(\|u_k\|^{2}\|v_k\|^{2})=(d^{2}+2d)^{2}-d^{4}\leq8d^{3} var ( ∥ u k ∥ 2 ∥ v k ∥ 2 ) = ( d 2 + 2 d ) 2 − d 4 ≤ 8 d 3 . Next, ∑ k ≠ l ( u k T u l ) ( v k T v l ) = 2 ∑ k < l ( u k T u l ) ( v k T v l ) \sum_{k\ne l}(u_k^{\mathsf{T}}u_l)(v_k^{\mathsf{T}}v_l)=2\sum_{k<l}(u_k^{\mathsf{T}}u_l)(v_k^{\mathsf{T}}v_l) ∑ k = l ( u k T u l ) ( v k T v l ) = 2 ∑ k < l ( u k T u l ) ( v k T v l ) ; for unordered pairs { k , l } ≠ { k ′ , l ′ } \{k,l\}\ne\{k',l'\} { k , l } = { k ′ , l ′ } one of the indices k , l , k ′ , l ′ k,l,k',l' k , l , k ′ , l ′ occurs exactly once, and integrating over the u u u with that index gives E [ ( u k T u l ) ( v k T v l ) ( u k ′ T u l ′ ) ( v k ′ T v l ′ ) ] = 0 \mathbb{E}[(u_k^{\mathsf{T}}u_l)(v_k^{\mathsf{T}}v_l)(u_{k'}^{\mathsf{T}}u_{l'})(v_{k'}^{\mathsf{T}}v_{l'})]=0 E [( u k T u l ) ( v k T v l ) ( u k ′ T u l ′ ) ( v k ′ T v l ′ )] = 0 ; since E ( u k T u l ) 2 ( v k T v l ) 2 = d 2 \mathbb{E}(u_k^{\mathsf{T}}u_l)^{2}(v_k^{\mathsf{T}}v_l)^{2}=d^{2} E ( u k T u l ) 2 ( v k T v l ) 2 = d 2 , this gives var ∑ k ≠ l ( u k T u l ) ( v k T v l ) = 4 ⋅ N ( N − 1 ) 2 ⋅ d 2 ≤ 2 N 2 d 2 \operatorname{var}\sum_{k\ne l}(u_k^{\mathsf{T}}u_l)(v_k^{\mathsf{T}}v_l)=4\cdot\frac{N(N-1)}{2}\cdot d^{2}\leq2N^{2}d^{2} var ∑ k = l ( u k T u l ) ( v k T v l ) = 4 ⋅ 2 N ( N − 1 ) ⋅ d 2 ≤ 2 N 2 d 2 . For k ′ ∈ [ N ] k'\in[N] k ′ ∈ [ N ] and k ≠ l k\ne l k = l , one of k , l k,l k , l differs from k ′ k' k ′ , and integrating over the u u u with that index gives E [ ( ∥ u k ′ ∥ 2 ∥ v k ′ ∥ 2 − d 2 ) ( u k T u l ) ( v k T v l ) ] = 0 \mathbb{E}[(\|u_{k'}\|^{2}\|v_{k'}\|^{2}-d^{2})(u_k^{\mathsf{T}}u_l)(v_k^{\mathsf{T}}v_l)]=0 E [( ∥ u k ′ ∥ 2 ∥ v k ′ ∥ 2 − d 2 ) ( u k T u l ) ( v k T v l )] = 0 , so the two sums are uncorrelated. Therefore
var ∥ U V T ∥ F 2 ≤ 8 N d 3 + 2 N 2 d 2 , ( var ∥ U V T ∥ F 2 ) 1 / 2 E ∥ U V T ∥ F 2 ≤ 8 N d + 2 d 2 ≤ 4 d , (17) \operatorname{var}\|UV^{\mathsf{T}}\|_{F}^{2}\leq8Nd^{3}+2N^{2}d^{2},\qquad\frac{\left(\operatorname{var}\|UV^{\mathsf{T}}\|_{F}^{2}\right)^{1/2}}{\mathbb{E}\|UV^{\mathsf{T}}\|_{F}^{2}}\leq\sqrt{\frac{8}{Nd}+\frac{2}{d^{2}}}\leq\frac{4}{d}, \tag*{(17)} var ∥ U V T ∥ F 2 ≤ 8 N d 3 + 2 N 2 d 2 , E ∥ U V T ∥ F 2 ( var ∥ U V T ∥ F 2 ) 1/2 ≤ N d 8 + d 2 2 ≤ d 4 , (17)
and Chebyshev gives the first claim.
Centred version. ∥ U V T ∥ F 2 − d ∑ k ∥ u k ∥ 2 = ∑ k ( ∥ u k ∥ 2 − d ) ∥ v k ∥ 2 + ∑ k ≠ l ( u k T u l ) ( v k T v l ) \|UV^{\mathsf{T}}\|_{F}^{2}-d\sum_k\|u_k\|^{2}=\sum_k(\|u_k\|^{2}-d)\|v_k\|^{2}+\sum_{k\ne l}(u_k^{\mathsf{T}}u_l)(v_k^{\mathsf{T}}v_l) ∥ U V T ∥ F 2 − d ∑ k ∥ u k ∥ 2 = ∑ k ( ∥ u k ∥ 2 − d ) ∥ v k ∥ 2 + ∑ k = l ( u k T u l ) ( v k T v l ) has mean zero. The variables ( ∥ u k ∥ 2 − d ) ∥ v k ∥ 2 (\|u_k\|^{2}-d)\|v_k\|^{2} ( ∥ u k ∥ 2 − d ) ∥ v k ∥ 2 are independent with variance E ( ∥ u ∥ 2 − d ) 2 E ∥ v ∥ 4 = 2 d ( d 2 + 2 d ) ≤ 6 d 3 \mathbb{E}(\|u\|^{2}-d)^{2}\mathbb{E}\|v\|^{4}=2d(d^{2}+2d)\leq6d^{3} E ( ∥ u ∥ 2 − d ) 2 E ∥ v ∥ 4 = 2 d ( d 2 + 2 d ) ≤ 6 d 3 ; the second sum has variance at most 2 N 2 d 2 2N^{2}d^{2} 2 N 2 d 2 as above; and for k ′ ∈ [ N ] k'\in[N] k ′ ∈ [ N ] and k ≠ l k\ne l k = l , integrating over the u u u whose index among k , l k,l k , l differs from k ′ k' k ′ gives E [ ( ∥ u k ′ ∥ 2 − d ) ∥ v k ′ ∥ 2 ( u k T u l ) ( v k T v l ) ] = 0 \mathbb{E}[(\|u_{k'}\|^{2}-d)\|v_{k'}\|^{2}(u_k^{\mathsf{T}}u_l)(v_k^{\mathsf{T}}v_l)]=0 E [( ∥ u k ′ ∥ 2 − d ) ∥ v k ′ ∥ 2 ( u k T u l ) ( v k T v l )] = 0 . Hence the variance is at most 6 N d 3 + 2 N 2 d 2 6Nd^{3}+2N^{2}d^{2} 6 N d 3 + 2 N 2 d 2 , its square root is at most 6 N d 3 / 2 + 2 N d \sqrt{6}Nd^{3/2}+\sqrt{2}Nd 6 N d 3/2 + 2 N d , and Chebyshev gives the second claim.
(viii) Conditionally on U U U , the columns of U V T UV^{\mathsf{T}} U V T are i.i.d. N ( 0 , U U T ) \mathcal{N}(0,UU^{\mathsf{T}}) N ( 0 , U U T ) , i.e. of the form ( U U T ) 1 / 2 g (UU^{\mathsf{T}})^{1/2}g ( U U T ) 1/2 g with g ∼ N ( 0 , I d ) g\sim\mathcal{N}(0,I_d) g ∼ N ( 0 , I d ) ; stacking the d d d vectors g g g as the columns of Ξ \Xi Ξ gives the stated equality in distribution, and the conditional law of Ξ \Xi Ξ does not depend on U U U , so Ξ \Xi Ξ is independent of U U U . For the norm, ∥ ( U U T ) 1 / 2 Ξ ∥ o p ≤ ∥ U ∥ o p ∥ Ξ ∥ o p \|(UU^{\mathsf{T}})^{1/2}\Xi\|_{\mathrm{op}}\leq\|U\|_{\mathrm{op}}\|\Xi\|_{\mathrm{op}} ∥ ( U U T ) 1/2 Ξ ∥ op ≤ ∥ U ∥ op ∥Ξ ∥ op , and almost surely ∥ U ∥ o p / N → 1 + γ − 1 / 2 ≤ 2 \|U\|_{\mathrm{op}}/\sqrt{N}\to1+\gamma^{-1/2}\leq2 ∥ U ∥ op / N → 1 + γ − 1/2 ≤ 2 by (iv) while ∥ Ξ ∥ o p / d → 2 \|\Xi\|_{\mathrm{op}}/\sqrt{d}\to2 ∥Ξ ∥ op / d → 2 by Bai–Yin with aspect ratio 1. □
Lemma F.4 (Norms of W G D W^{\mathrm{GD}} W GD ).
(i) ∥ W G D ∥ F 2 = ( N − 1 ) d 2 ( 1 + O p ( d − 1 ) ) \|W^{\mathrm{GD}}\|_{F}^{2}=(N-1)d^{2}(1+O_p(d^{-1})) ∥ W GD ∥ F 2 = ( N − 1 ) d 2 ( 1 + O p ( d − 1 )) , and ∥ W G D ∥ F 2 − d tr ( V V T − 1 N ( V 1 ) ( V 1 ) T ) = O p ( N d 3 / 2 + N d ) \|W^{\mathrm{GD}}\|_{F}^{2}-d\operatorname{tr}\left(VV^{\mathsf{T}}-\frac{1}{N}(V\mathbf{1})(V\mathbf{1})^{\mathsf{T}}\right)=O_p(\sqrt{N}d^{3/2}+Nd) ∥ W GD ∥ F 2 − d tr ( V V T − N 1 ( V 1 ) ( V 1 ) T ) = O p ( N d 3/2 + N d ) .
(ii) ∥ W G D ∥ o p ≤ ∥ U ∥ o p ∥ V ∥ o p \|W^{\mathrm{GD}}\|_{\mathrm{op}}\leq\|U\|_{\mathrm{op}}\|V\|_{\mathrm{op}} ∥ W GD ∥ op ≤ ∥ U ∥ op ∥ V ∥ op , and with probability tending to one ∥ W G D ∥ o p ≤ 7 N d \|W^{\mathrm{GD}}\|_{\mathrm{op}}\leq7\sqrt{N}d ∥ W GD ∥ op ≤ 7 N d .
(iii) With probability tending to one, 1 8 d N d ≤ ∥ W G D ∥ ∗ ≤ 7 d N d \frac{1}{8}d\sqrt{N}d\leq\|W^{\mathrm{GD}}\|_{*}\leq7d\sqrt{N}d 8 1 d N d ≤ ∥ W GD ∥ ∗ ≤ 7 d N d .
(iv) In probability, ∥ W G D ∥ ∗ / ( N d ) → Φ ( γ ) / γ \|W^{\mathrm{GD}}\|_{*}/(Nd)\to\Phi(\gamma)/\sqrt{\gamma} ∥ W GD ∥ ∗ / ( N d ) → Φ ( γ ) / γ , where
Φ ( γ ) : = 1 π ∫ 0 1 [ 1 − p γ 1 + p p + p ( 1 − p ) γ 1 − p / γ ] d p , (18) \Phi(\gamma):=\frac{1}{\pi}\int_{0}^{1}\left[\sqrt{1-\frac{p}{\gamma}}\frac{1+p}{\sqrt{p}}+\frac{\sqrt{p}(1-p)}{\gamma\sqrt{1-p/\gamma}}\right]\,dp, \tag*{(18)} Φ ( γ ) := π 1 ∫ 0 1 [ 1 − γ p p 1 + p + γ 1 − p / γ p ( 1 − p ) ] d p , (18) and for γ ≥ 16 \gamma\geq16 γ ≥ 16
Φ ( γ ) = 8 3 π ( 1 − 1 10 γ + r ( γ ) ) , ∣ r ( γ ) ∣ ≤ γ − 2 . (19) \Phi(\gamma)=\frac{8}{3\pi}\left(1-\frac{1}{10\gamma}+r(\gamma)\right),\qquad|r(\gamma)|\leq\gamma^{-2}. \tag*{(19)} Φ ( γ ) = 3 π 8 ( 1 − 10 γ 1 + r ( γ ) ) , ∣ r ( γ ) ∣ ≤ γ − 2 . (19)
Part (iv) is where the constant 3 π / 8 3\pi/8 3 π /8 enters: Muon replaces W G D W^{\mathrm{GD}} W GD by its polar factor, which turns the diagonal score into ∥ W G D ∥ ∗ \|W^{\mathrm{GD}}\|_{*} ∥ W GD ∥ ∗ , and 8 / ( 3 π ) 8/(3\pi) 8/ ( 3 π ) is the limiting Marčenko–Pastur integral at γ = ∞ \gamma=\infty γ = ∞ .
Proof. (i) By Equation (11) and ∥ X − Y ∥ F 2 = ∥ X ∥ F 2 − 2 ⟨ X , Y ⟩ + ∥ Y ∥ F 2 \lVert X-Y\rVert_F^2=\lVert X\rVert_F^2-2\langle X,Y\rangle+\lVert Y\rVert_F^2 ∥ X − Y ∥ F 2 = ∥ X ∥ F 2 − 2 ⟨ X , Y ⟩ + ∥ Y ∥ F 2 ,
∥ W G D ∥ F 2 = ∥ U V ⊤ ∥ F 2 − 2 N ( U 1 ) ⊤ U V ⊤ ( V 1 ) + 1 N 2 ∥ U 1 ∥ 2 ∥ V 1 ∥ 2 , (20) \lVert W^{\mathrm{GD}}\rVert_F^2=\lVert UV^\top\rVert_F^2-\frac{2}{N}(U\mathbf{1})^\top UV^\top(V\mathbf{1})+\frac{1}{N^2}\lVert U\mathbf{1}\rVert^2\lVert V\mathbf{1}\rVert^2, \tag*{(20)} ∥ W GD ∥ F 2 = ∥ U V ⊤ ∥ F 2 − N 2 ( U 1 ) ⊤ U V ⊤ ( V 1 ) + N 2 1 ∥ U 1 ∥ 2 ∥ V 1 ∥ 2 , (20)
where the cross term is O p ( N 2 d ) / N = O p ( N d ) O_p(N^2d)/N=O_p(Nd) O p ( N 2 d ) / N = O p ( N d ) and the last term is O p ( d 2 ) O_p(d^2) O p ( d 2 ) by Theorem F.3(iii),(iv),(viii). Substituting ∥ U V ⊤ ∥ F 2 = N d 2 ( 1 + O p ( d − 1 ) ) \lVert UV^\top\rVert_F^2=Nd^2(1+O_p(d^{-1})) ∥ U V ⊤ ∥ F 2 = N d 2 ( 1 + O p ( d − 1 )) from Theorem F.3(vii) and using d 2 ≤ N d d^2\le Nd d 2 ≤ N d gives the first claim. For the second, tr ( V V ⊤ ) = ∑ k ∥ v k ∥ 2 \operatorname{tr}(VV^\top)=\sum_k\lVert v_k\rVert^2 tr ( V V ⊤ ) = ∑ k ∥ v k ∥ 2 and tr ( ( V 1 ) ( V 1 ) ⊤ ) = ∥ V 1 ∥ 2 \operatorname{tr}((V\mathbf{1})(V\mathbf{1})^\top)=\lVert V\mathbf{1}\rVert^2 tr (( V 1 ) ( V 1 ) ⊤ ) = ∥ V 1 ∥ 2 , so the same expansion gives
∥ W G D ∥ F 2 − d tr ( V V ⊤ − 1 N ( V 1 ) ( V 1 ) ⊤ ) = ( ∥ U V ⊤ ∥ F 2 − d ∑ k ∥ v k ∥ 2 ) + d ∥ V 1 ∥ 2 N \lVert W^{\mathrm{GD}}\rVert_F^2-d\operatorname{tr}\left(VV^\top-\frac{1}{N}(V\mathbf{1})(V\mathbf{1})^\top\right)=\left(\lVert UV^\top\rVert_F^2-d\sum_k\lVert v_k\rVert^2\right)+\frac{d\lVert V\mathbf{1}\rVert^2}{N} ∥ W GD ∥ F 2 − d tr ( V V ⊤ − N 1 ( V 1 ) ( V 1 ) ⊤ ) = ( ∥ U V ⊤ ∥ F 2 − d k ∑ ∥ v k ∥ 2 ) + N d ∥ V 1 ∥ 2
− 2 N ( U 1 ) ⊤ U V ⊤ ( V 1 ) + ∥ U 1 ∥ 2 ∥ V 1 ∥ 2 N 2 , (21) -\frac{2}{N}(U\mathbf{1})^\top UV^\top(V\mathbf{1})+\frac{\lVert U\mathbf{1}\rVert^2\lVert V\mathbf{1}\rVert^2}{N^2}, \tag*{(21)} − N 2 ( U 1 ) ⊤ U V ⊤ ( V 1 ) + N 2 ∥ U 1 ∥ 2 ∥ V 1 ∥ 2 , (21)
whose four terms are O p ( N d 3 / 2 + N d ) O_p(\sqrt{N}d^{3/2}+Nd) O p ( N d 3/2 + N d ) by the second half of Theorem F.3(vii), O p ( d 2 ) O_p(d^2) O p ( d 2 ) , O p ( N d ) O_p(Nd) O p ( N d ) and O p ( d 2 ) O_p(d^2) O p ( d 2 ) respectively; since d 2 ≤ N d d^2\le Nd d 2 ≤ N d , the claim follows.
(ii) ∥ I N − 1 N 11 ⊤ ∥ o p = 1 \lVert I_N-\frac{1}{N}\mathbf{1}\mathbf{1}^\top\rVert_{\mathrm{op}}=1 ∥ I N − N 1 1 1 ⊤ ∥ op = 1 gives the first inequality. For the second, the triangle inequality and the operator norm of a rank-one matrix give ∥ W G D ∥ o p ≤ ∥ U V ⊤ ∥ o p + ∥ U 1 ∥ ∥ V 1 ∥ / N ≤ 5 N d + 3 2 d ≤ 7 N d \lVert W^{\mathrm{GD}}\rVert_{\mathrm{op}}\le\lVert UV^\top\rVert_{\mathrm{op}}+\lVert U\mathbf{1}\rVert\lVert V\mathbf{1}\rVert/N\le5\sqrt{Nd}+\frac{3}{2}d\le7\sqrt{Nd} ∥ W GD ∥ op ≤ ∥ U V ⊤ ∥ op + ∥ U 1 ∥ ∥ V 1 ∥ / N ≤ 5 N d + 2 3 d ≤ 7 N d with probability tending to one, by Theorem F.3(iii),(viii) and d ≤ N d d\le\sqrt{Nd} d ≤ N d .
(iii) For any X X X , ∥ X ∥ F 2 = ∑ a s a 2 ≤ s 1 ∑ a s a = ∥ X ∥ o p ∥ X ∥ ∗ \lVert X\rVert_F^2=\sum_a s_a^2\le s_1\sum_a s_a=\lVert X\rVert_{\mathrm{op}}\lVert X\rVert_* ∥ X ∥ F 2 = ∑ a s a 2 ≤ s 1 ∑ a s a = ∥ X ∥ op ∥ X ∥ ∗ , and ∥ X ∥ ∗ ≤ rank ( X ) ∥ X ∥ o p \lVert X\rVert_*\le\operatorname{rank}(X)\lVert X\rVert_{\mathrm{op}} ∥ X ∥ ∗ ≤ rank ( X ) ∥ X ∥ op . Combining with (i) and (ii),
∥ W G D ∥ ∗ ≥ ∥ W G D ∥ F 2 ∥ W G D ∥ o p ≥ ( N − 1 ) d 2 ( 1 − o p ( 1 ) ) 7 N d ≥ 1 8 d N d , (22) \lVert W^{\mathrm{GD}}\rVert_*\ge\frac{\lVert W^{\mathrm{GD}}\rVert_F^2}{\lVert W^{\mathrm{GD}}\rVert_{\mathrm{op}}}\ge\frac{(N-1)d^2(1-o_p(1))}{7\sqrt{Nd}}\ge\frac{1}{8}d\sqrt{Nd}, \tag*{(22)} ∥ W GD ∥ ∗ ≥ ∥ W GD ∥ op ∥ W GD ∥ F 2 ≥ 7 N d ( N − 1 ) d 2 ( 1 − o p ( 1 )) ≥ 8 1 d N d , (22)
∥ W G D ∥ ∗ ≤ d ∥ W G D ∥ o p ≤ 7 d N d . \lVert W^{\mathrm{GD}}\rVert_*\le d\lVert W^{\mathrm{GD}}\rVert_{\mathrm{op}}\le7d\sqrt{Nd}. ∥ W GD ∥ ∗ ≤ d ∥ W GD ∥ op ≤ 7 d N d .
(iv) Removing the centring. By the triangle inequality for the nuclear norm and the nuclear norm of a rank-one matrix, ∣ ∥ U V ⊤ ∥ ∗ − ∥ W G D ∥ ∗ ∣ ≤ ∥ U 1 ∥ ∥ V 1 ∥ / N = O p ( d ) \left|\lVert UV^\top\rVert_*-\lVert W^{\mathrm{GD}}\rVert_*\right|\le\lVert U\mathbf{1}\rVert\lVert V\mathbf{1}\rVert/N=O_p(d) ∥ U V ⊤ ∥ ∗ − ∥ W GD ∥ ∗ ≤ ∥ U 1 ∥ ∥ V 1 ∥ / N = O p ( d ) , which is o ( N d ) o(Nd) o ( N d ) ; so it suffices to prove the limit for ∥ U V ⊤ ∥ ∗ \lVert UV^\top\rVert_* ∥ U V ⊤ ∥ ∗ .
Limit. By Theorem F.3(viii), { s a ( U V ⊤ ) 2 / ( N d ) } a ≤ d \{s_a(UV^\top)^2/(Nd)\}_{a\le d} { s a ( U V ⊤ ) 2 / ( N d ) } a ≤ d has the same law as the eigenvalues of 1 d ( U U ⊤ N ) 1 / 2 Ξ Ξ ⊤ ( U U ⊤ N ) 1 / 2 \frac{1}{d}\left(\frac{UU^\top}{N}\right)^{1/2}\Xi\Xi^\top\left(\frac{UU^\top}{N}\right)^{1/2} d 1 ( N U U ⊤ ) 1/2 Ξ Ξ ⊤ ( N U U ⊤ ) 1/2 with Ξ \Xi Ξ independent of U U U : conditionally on U U U this is a sample covariance matrix with population covariance 1 N U U ⊤ \frac{1}{N}UU^\top N 1 U U ⊤ and aspect ratio 1 1 1 , and the spectral distribution of 1 N U U ⊤ \frac{1}{N}UU^\top N 1 U U ⊤ converges almost surely to the Marčenko–Pastur law H H H with ratio 1 / γ 1/\gamma 1/ γ and mean 1 1 1 . By the Marčenko–Pastur–Silverstein theorem the empirical distribution of { s a ( U V ⊤ ) 2 / ( N d ) } \{s_a(UV^\top)^2/(Nd)\} { s a ( U V ⊤ ) 2 / ( N d )} converges weakly in probability to a law ν \nu ν whose Stieltjes transform m m m satisfies
z = − 1 m + ∫ t 1 + t m d H ( t ) . (23) z=-\frac{1}{m}+\int\frac{t}{1+tm}\,dH(t). \tag*{(23)} z = − m 1 + ∫ 1 + t m t d H ( t ) . (23)
By Theorem F.3(viii) all s a 2 / ( N d ) ≤ 25 s_a^2/(Nd)\le25 s a 2 / ( N d ) ≤ 25 with probability tending to one, so both the empirical law and ν \nu ν are supported in [ 0 , 25 ] [0,25] [ 0 , 25 ] ; taking the limit for the bounded continuous function λ ↦ min ( λ , 5 ) \lambda\mapsto\min(\sqrt{\lambda},5) λ ↦ min ( λ , 5 ) gives 1 d ∑ a s a / N d → ∫ λ d ν \frac{1}{d}\sum_a s_a/\sqrt{Nd}\to\int\sqrt{\lambda}\,d\nu d 1 ∑ a s a / N d → ∫ λ d ν in probability, that is
∥ U V ⊤ ∥ ∗ N d = d N ⋅ 1 d ∑ a s a N d ⟶ 1 γ ∫ λ d ν ( λ ) . (24) \frac{\lVert UV^\top\rVert_*}{Nd}=\sqrt{\frac{d}{N}}\cdot\frac{1}{d}\sum_a\frac{s_a}{\sqrt{Nd}}\longrightarrow\frac{1}{\sqrt{\gamma}}\int\sqrt{\lambda}\,d\nu(\lambda). \tag*{(24)} N d ∥ U V ⊤ ∥ ∗ = N d ⋅ d 1 a ∑ N d s a ⟶ γ 1 ∫ λ d ν ( λ ) . (24)
Closed form. Let ψ ( x ) : = ∫ λ x 1 − λ x d ν \psi(x):=\int\frac{\lambda x}{1-\lambda x}\,d\nu ψ ( x ) := ∫ 1 − λ x λ x d ν . From λ x 1 − λ x = − 1 + 1 / x 1 / x − λ \frac{\lambda x}{1-\lambda x}=-1+\frac{1/x}{1/x-\lambda} 1 − λ x λ x = − 1 + 1/ x − λ 1/ x we get ψ ( x ) = − 1 − 1 x m ( 1 / x ) \psi(x)=-1-\frac{1}{x}m(1/x) ψ ( x ) = − 1 − x 1 m ( 1/ x ) , i.e. m ( 1 / x ) = − x ( 1 + ψ ( x ) ) m(1/x)=-x(1+\psi(x)) m ( 1/ x ) = − x ( 1 + ψ ( x )) . Taking z = 1 / x z=1/x z = 1/ x in Equation (23) and multiplying by x ( 1 + ψ ( x ) ) x(1+\psi(x)) x ( 1 + ψ ( x )) ,
ψ ( x ) = ∫ x ( 1 + ψ ( x ) ) t 1 − x ( 1 + ψ ( x ) ) t d H ( t ) . (25) \psi(x)=\int\frac{x(1+\psi(x))t}{1-x(1+\psi(x))t}\,dH(t). \tag*{(25)} ψ ( x ) = ∫ 1 − x ( 1 + ψ ( x )) t x ( 1 + ψ ( x )) t d H ( t ) . (25)
On the other hand the Stieltjes transform of the Marčenko–Pastur law with ratio 1 / γ 1/\gamma 1/ γ and mean 1 1 1 satisfies 1 γ z m H ( z ) 2 − ( 1 − 1 γ − z ) m H ( z ) + 1 = 0 \frac{1}{\gamma}zm_H(z)^2-\left(1-\frac{1}{\gamma}-z\right)m_H(z)+1=0 γ 1 z m H ( z ) 2 − ( 1 − γ 1 − z ) m H ( z ) + 1 = 0 , and u t 1 − u t = − 1 + 1 1 − u t \frac{ut}{1-ut}=-1+\frac{1}{1-ut} 1 − u t u t = − 1 + 1 − u t 1 gives m H ( 1 / u ) = − u ( 1 + ∫ u t 1 − u t d H ( t ) ) m_H(1/u)=-u\left(1+\int\frac{ut}{1-ut}\,dH(t)\right) m H ( 1/ u ) = − u ( 1 + ∫ 1 − u t u t d H ( t ) ) ; substituting z = 1 / u z=1/u z = 1/ u into the quadratic yields, for all u u u ,
u γ ( 1 + ∫ u t 1 − u t d H ( t ) ) 2 + ( 1 − 1 γ ) u ( 1 + ∫ u t 1 − u t d H ( t ) ) − ∫ u t 1 − u t d H ( t ) = 0. (26) \frac{u}{\gamma}\left(1+\int\frac{ut}{1-ut}\,dH(t)\right)^2+\left(1-\frac{1}{\gamma}\right)u\left(1+\int\frac{ut}{1-ut}\,dH(t)\right)-\int\frac{ut}{1-ut}\,dH(t)=0. \tag*{(26)} γ u ( 1 + ∫ 1 − u t u t d H ( t ) ) 2 + ( 1 − γ 1 ) u ( 1 + ∫ 1 − u t u t d H ( t ) ) − ∫ 1 − u t u t d H ( t ) = 0. (26)
Take u = x ( 1 + ψ ( x ) ) u=x(1+\psi(x)) u = x ( 1 + ψ ( x )) . By Equation (25) the integral then equals ψ ( x ) \psi(x) ψ ( x ) , so
1 γ x ( 1 + ψ ) 3 + ( 1 − 1 γ ) x ( 1 + ψ ) 2 − ψ = 0 , i.e. x = ψ ( x ) ( 1 + ψ ( x ) ) 2 ( 1 + ψ ( x ) / γ ) . (27) \frac{1}{\gamma}x(1+\psi)^3+\left(1-\frac{1}{\gamma}\right)x(1+\psi)^2-\psi=0,\qquad\text{i.e.}\quad x=\frac{\psi(x)}{(1+\psi(x))^2(1+\psi(x)/\gamma)}. \tag*{(27)} γ 1 x ( 1 + ψ ) 3 + ( 1 − γ 1 ) x ( 1 + ψ ) 2 − ψ = 0 , i.e. x = ( 1 + ψ ( x ) ) 2 ( 1 + ψ ( x ) / γ ) ψ ( x ) . (27)
Using π λ = ∫ 0 ∞ λ λ + w d w w \pi\sqrt{\lambda}=\int_0^\infty\frac{\lambda}{\lambda+w}\frac{dw}{\sqrt{w}} π λ = ∫ 0 ∞ λ + w λ w d w and Tonelli’s theorem, and then substituting p = − ψ ( − 1 / w ) ∈ ( 0 , 1 ) p=-\psi(-1/w)\in(0,1) p = − ψ ( − 1/ w ) ∈ ( 0 , 1 ) ,
∫ λ d ν = 1 π ∫ 0 ∞ − ψ ( − 1 / w ) w d w , \int\sqrt{\lambda}\,d\nu=\frac{1}{\pi}\int_0^\infty\frac{-\psi(-1/w)}{\sqrt{w}}\,dw, ∫ λ d ν = π 1 ∫ 0 ∞ w − ψ ( − 1/ w ) d w ,
w = ( 1 − p ) 2 ( 1 − p / γ ) p , − d w d p = w [ 2 1 − p + 1 / γ 1 − p / γ + 1 p ] , w=\frac{(1-p)^2(1-p/\gamma)}{p},\qquad-\frac{dw}{dp}=w\left[\frac{2}{1-p}+\frac{1/\gamma}{1-p/\gamma}+\frac{1}{p}\right], w = p ( 1 − p ) 2 ( 1 − p / γ ) , − d p d w = w [ 1 − p 2 + 1 − p / γ 1/ γ + p 1 ] ,
∫ λ d ν = 1 π ∫ 0 1 1 − p / γ 1 − p [ 2 p + p ( 1 − p ) / γ 1 − p / γ + 1 − p p ] d p . (28) \int\sqrt{\lambda}\,d\nu=\frac{1}{\pi}\int_0^1\sqrt{\frac{1-p/\gamma}{1-p}}\left[2\sqrt{p}+\frac{\sqrt{p}(1-p)/\gamma}{1-p/\gamma}+\frac{1-p}{\sqrt{p}}\right]\,dp. \tag*{(28)} ∫ λ d ν = π 1 ∫ 0 1 1 − p 1 − p / γ [ 2 p + 1 − p / γ p ( 1 − p ) / γ + p 1 − p ] d p . (28)
then 2 p + 1 − p p = 1 + p p 2\sqrt{p}+\frac{1-p}{\sqrt{p}}=\frac{1+p}{\sqrt{p}} 2 p + p 1 − p = p 1 + p and 1 − p / γ ⋅ p ( 1 − p ) / γ 1 − p / γ = p ( 1 − p ) γ 1 − p / γ \sqrt{1-p/\gamma}\cdot\frac{\sqrt{p}(1-p)/\gamma}{1-p/\gamma}=\frac{\sqrt{p}(1-p)}{\gamma\sqrt{1-p/\gamma}} 1 − p / γ ⋅ 1 − p / γ p ( 1 − p ) / γ = γ 1 − p / γ p ( 1 − p ) give Equation (18) .
Expansion. For 0 ≤ s ≤ 1 16 0\le s\le\frac{1}{16} 0 ≤ s ≤ 16 1 one has
∣ 1 − s − 1 + s 2 ∣ ≤ s 2 5 , 0 ≤ ( 1 − s ) − 1 / 2 − 1 ≤ 2 s 3 . (29) \left|\sqrt{1-s}-1+\frac{s}{2}\right|\le\frac{s^2}{5},\qquad0\le(1-s)^{-1/2}-1\le\frac{2s}{3}. \tag*{(29)} 1 − s − 1 + 2 s ≤ 5 s 2 , 0 ≤ ( 1 − s ) − 1/2 − 1 ≤ 3 2 s . (29)
For the first, g ( s ) : = 1 − s 2 − 1 − s g(s):=1-\frac{s}{2}-\sqrt{1-s} g ( s ) := 1 − 2 s − 1 − s satisfies g ( 0 ) = 0 g(0)=0 g ( 0 ) = 0 , g ′ ( s ) = 1 2 ( ( 1 − s ) − 1 / 2 − 1 ) ≥ 0 g'(s)=\frac{1}{2}((1-s)^{-1/2}-1)\ge0 g ′ ( s ) = 2 1 (( 1 − s ) − 1/2 − 1 ) ≥ 0 and g ′ ′ ( s ) = 1 4 ( 1 − s ) − 3 / 2 ≤ 1 4 ( 1 − 1 16 ) − 3 / 2 < 5 9 g''(s)=\frac{1}{4}(1-s)^{-3/2}\le\frac{1}{4}(1-\frac{1}{16})^{-3/2}<\frac{5}{9} g ′′ ( s ) = 4 1 ( 1 − s ) − 3/2 ≤ 4 1 ( 1 − 16 1 ) − 3/2 < 9 5 on [ 0 , 1 16 ] [0,\frac{1}{16}] [ 0 , 16 1 ] , so g ( s ) ≤ s 2 2 max g ′ ′ < s 2 5 g(s)\le\frac{s^2}{2}\max g''<\frac{s^2}{5} g ( s ) ≤ 2 s 2 max g ′′ < 5 s 2 . For the second, f ( s ) : = ( 1 − s ) − 1 / 2 − 1 − 2 s 3 f(s):=(1-s)^{-1/2}-1-\frac{2s}{3} f ( s ) := ( 1 − s ) − 1/2 − 1 − 3 2 s satisfies f ( 0 ) = 0 f(0)=0 f ( 0 ) = 0 and f ′ ( s ) = 1 2 ( 1 − s ) − 3 / 2 − 2 3 < 0 f'(s)=\frac{1}{2}(1-s)^{-3/2}-\frac{2}{3}<0 f ′ ( s ) = 2 1 ( 1 − s ) − 3/2 − 3 2 < 0 for s < 1 − ( 3 4 ) 2 / 3 ≈ 0.175 s<1-(\frac{3}{4})^{2/3}\approx0.175 s < 1 − ( 4 3 ) 2/3 ≈ 0.175 , hence on [ 0 , 1 16 ] [0,\frac{1}{16}] [ 0 , 16 1 ] , so f ≤ 0 f\le0 f ≤ 0 there. Take s = p / γ s=p/\gamma s = p / γ with γ ≥ 16 \gamma\ge16 γ ≥ 16 . From
1 π ∫ 0 1 1 + p p d p = 8 3 π , − 1 2 γ π ∫ 0 1 p ( 1 + p ) d p + 1 γ π ∫ 0 1 p ( 1 − p ) d p = 8 3 π ( − 1 10 γ ) , (30) \frac{1}{\pi}\int_0^1\frac{1+p}{\sqrt{p}}\,dp=\frac{8}{3\pi},\qquad-\frac{1}{2\gamma\pi}\int_0^1\sqrt{p}(1+p)\,dp+\frac{1}{\gamma\pi}\int_0^1\sqrt{p}(1-p)\,dp=\frac{8}{3\pi}\left(-\frac{1}{10\gamma}\right), \tag*{(30)} π 1 ∫ 0 1 p 1 + p d p = 3 π 8 , − 2 γ π 1 ∫ 0 1 p ( 1 + p ) d p + γ π 1 ∫ 0 1 p ( 1 − p ) d p = 3 π 8 ( − 10 γ 1 ) , (30)
and Equation (18) , the remainder is bounded by
∣ Φ ( γ ) − 8 3 π ( 1 − 1 10 γ ) ∣ ≤ 1 π γ 2 [ 1 5 ∫ 0 1 p 3 / 2 ( 1 + p ) d p + 2 3 ∫ 0 1 p 3 / 2 ( 1 − p ) d p ] = 1 π γ 2 [ 24 175 + 8 105 ] < 8 3 π γ − 2 , (31) \begin{aligned}
\left|\Phi(\gamma)-\frac{8}{3\pi}\left(1-\frac{1}{10\gamma}\right)\right|
\le\frac{1}{\pi\gamma^2}\left[\frac{1}{5}\int_0^1p^{3/2}(1+p)\,dp+\frac{2}{3}\int_0^1p^{3/2}(1-p)\,dp\right] \\
&=\frac{1}{\pi\gamma^2}\left[\frac{24}{175}+\frac{8}{105}\right]<\frac{8}{3\pi}\gamma^{-2}, \tag*{(31)}
\end{aligned} Φ ( γ ) − 3 π 8 ( 1 − 10 γ 1 ) ≤ π γ 2 1 [ 5 1 ∫ 0 1 p 3/2 ( 1 + p ) d p + 3 2 ∫ 0 1 p 3/2 ( 1 − p ) d p ] = π γ 2 1 [ 175 24 + 105 8 ] < 3 π 8 γ − 2 , (31)
which is Equation (19) .
The GD step
By Equation (10) with W = W G D W=W^{\mathrm{GD}} W = W GD ,
h ˉ G D = mean i u i T W G D v i ⏟ diagonal − mean i mean j ≠ i u j T W G D v i ⏟ off-diagonal . (32) \bar{h}^{\mathrm{GD}}=\underset{i}{\operatorname{mean}}\ \underbrace{u_i^{\mathsf T}W^{\mathrm{GD}}v_i}_{\text{diagonal}}-\underset{i}{\operatorname{mean}}\ \underset{j\ne i}{\operatorname{mean}}\ \underbrace{u_j^{\mathsf T}W^{\mathrm{GD}}v_i}_{\text{off-diagonal}}. \tag*{(32)} h ˉ GD = i mean diagonal u i T W GD v i − i mean j = i mean off-diagonal u j T W GD v i . (32)
Lemma F.5 (Diagonal-sum identity). The diagonal scores of W G D W^{\mathrm{GD}} W GD sum to
∑ i = 1 N u i T W G D v i = ∥ W G D ∥ F 2 + 1 N ( U 1 ) T W G D ( V 1 ) . \sum_{i=1}^{N}u_i^{\mathsf T}W^{\mathrm{GD}}v_i=\lVert W^{\mathrm{GD}}\rVert_F^2+\frac{1}{N}(U1)^{\mathsf T}W^{\mathrm{GD}}(V1). i = 1 ∑ N u i T W GD v i = ∥ W GD ∥ F 2 + N 1 ( U 1 ) T W GD ( V 1 ) . Proof. A scalar equals its own trace and ∑ i u i v i T = U V T \sum_i u_iv_i^{\mathsf T}=UV^{\mathsf T} ∑ i u i v i T = U V T , so ∑ i u i T W G D v i = tr ( ( W G D ) T ∑ i u i v i T ) = ⟨ W G D , U V T ⟩ \sum_i u_i^{\mathsf T}W^{\mathrm{GD}}v_i=\operatorname{tr}((W^{\mathrm{GD}})^{\mathsf T}\sum_i u_iv_i^{\mathsf T})=\langle W^{\mathrm{GD}},UV^{\mathsf T}\rangle ∑ i u i T W GD v i = tr (( W GD ) T ∑ i u i v i T ) = ⟨ W GD , U V T ⟩ . Rearranging Equation (11) gives U V T = W G D + 1 N ( U 1 ) ( V 1 ) T UV^{\mathsf T}=W^{\mathrm{GD}}+\frac{1}{N}(U1)(V1)^{\mathsf T} U V T = W GD + N 1 ( U 1 ) ( V 1 ) T , and bilinearity together with ⟨ Z , a b T ⟩ = a T Z b \langle Z,ab^{\mathsf T}\rangle=a^{\mathsf T}Zb ⟨ Z , a b T ⟩ = a T Z b gives the claim.
Lemma F.6 (Rank-one cross term). 1 N 2 ∣ ( U 1 ) T W G D ( V 1 ) ∣ = O p ( d ) \frac{1}{N^2}|(U1)^{\mathsf T}W^{\mathrm{GD}}(V1)|=O_p(d) N 2 1 ∣ ( U 1 ) T W GD ( V 1 ) ∣ = O p ( d ) .
Proof. By Lemma F.4 (ii) and Theorem F.3(iii),(iv),
∣ ( U 1 ) T W G D ( V 1 ) ∣ N 2 ≤ ∥ U 1 ∥ ∥ W G D ∥ o p ∥ V 1 ∥ N 2 = O p ( N d ) ⋅ O p ( N ) ⋅ O p ( N d ) N 2 = O p ( d ) . (33) \frac{\left|(U1)^{\mathsf{T}}W^{\mathrm{GD}}(V1)\right|}{N^{2}} \le\frac{\lVert U1\rVert\lVert W^{\mathrm{GD}}\rVert_{\mathrm{op}}\lVert V1\rVert}{N^{2}} = \frac{O_p(\sqrt{N}d)\cdot O_p(N)\cdot O_p(\sqrt{N}d)}{N^{2}} = O_p(d). \tag*{(33)} N 2 ( U 1 ) T W GD ( V 1 ) ≤ N 2 ∥ U 1 ∥ ∥ W GD ∥ op ∥ V 1 ∥ = N 2 O p ( N d ) ⋅ O p ( N ) ⋅ O p ( N d ) = O p ( d ) . (33)
Lemma F.7. mean i u i T W G D v i = d 2 ( 1 + O p ( d − 1 ) ) \operatorname{mean}_{i}u_i^{\mathsf{T}}W^{\mathrm{GD}}v_i=d^2(1+O_p(d^{-1})) mean i u i T W GD v i = d 2 ( 1 + O p ( d − 1 )) .
Proof. Divide Theorem F.5 by N N N . By Lemma F.4 (i), ∥ W G D ∥ F 2 / N = N − 1 N d 2 ( 1 + O p ( d − 1 ) ) = d 2 ( 1 + O p ( d − 1 ) ) \lVert W^{\mathrm{GD}}\rVert_F^2/N=\frac{N-1}{N}d^2(1+O_p(d^{-1}))=d^2(1+O_p(d^{-1})) ∥ W GD ∥ F 2 / N = N N − 1 d 2 ( 1 + O p ( d − 1 )) = d 2 ( 1 + O p ( d − 1 )) , using 1 N ≤ 1 d \frac{1}{N}\le\frac{1}{d} N 1 ≤ d 1 ; by Theorem F.6 the second term is O p ( d ) = d 2 ⋅ O p ( d − 1 ) O_p(d)=d^2\cdot O_p(d^{-1}) O p ( d ) = d 2 ⋅ O p ( d − 1 ) .
Lemma F.8 (Column-sum identity). For every W ∈ R d × d W\in\mathbb{R}^{d\times d} W ∈ R d × d and every i i i , ( N − 1 ) mean j ≠ i u j T W v i = ( U 1 ) T W v i − u i T W v i (N-1)\operatorname{mean}_{j\ne i}u_j^{\mathsf{T}}Wv_i=(U1)^{\mathsf{T}}Wv_i-u_i^{\mathsf{T}}Wv_i ( N − 1 ) mean j = i u j T W v i = ( U 1 ) T W v i − u i T W v i .
Proof. ( N − 1 ) mean j ≠ i u j T W v i = ∑ j u j T W v i − u i T W v i (N-1)\operatorname{mean}_{j\ne i}u_j^{\mathsf{T}}Wv_i=\sum_j u_j^{\mathsf{T}}Wv_i-u_i^{\mathsf{T}}Wv_i ( N − 1 ) mean j = i u j T W v i = ∑ j u j T W v i − u i T W v i and ∑ j u j = U 1 \sum_j u_j=U1 ∑ j u j = U 1 .
Lemma F.9 (Two sums of squares). ∑ i ( ( U 1 ) T W G D v i ) 2 = O p ( N 4 d ) \sum_i\left((U1)^{\mathsf{T}}W^{\mathrm{GD}}v_i\right)^2=O_p(N^4d) ∑ i ( ( U 1 ) T W GD v i ) 2 = O p ( N 4 d ) and ∑ i ( u i T W G D v i ) 2 = O p ( N 4 d ) \sum_i\left(u_i^{\mathsf{T}}W^{\mathrm{GD}}v_i\right)^2=O_p(N^4d) ∑ i ( u i T W GD v i ) 2 = O p ( N 4 d ) .
Proof. The numbers ( U 1 ) T W G D v i (U1)^{\mathsf{T}}W^{\mathrm{GD}}v_i ( U 1 ) T W GD v i are the entries of V T ( W G D ) T U 1 V^{\mathsf{T}}(W^{\mathrm{GD}})^{\mathsf{T}}U1 V T ( W GD ) T U 1 , so by Lemma F.4 (ii) and Theorem F.3(iii),(iv),
∥ V T ( W G D ) T U 1 ∥ 2 ≤ ∥ V ∥ o p 2 ∥ W G D ∥ o p 2 ∥ U 1 ∥ 2 = O p ( N ⋅ N 2 ⋅ N d ) = O p ( N 4 d ) . (34) \left\lVert V^{\mathsf{T}}(W^{\mathrm{GD}})^{\mathsf{T}}U1\right\rVert^2\le\lVert V\rVert_{\mathrm{op}}^2\lVert W^{\mathrm{GD}}\rVert_{\mathrm{op}}^2\lVert U1\rVert^2=O_p(N\cdot N^2\cdot Nd)=O_p(N^4d). \tag*{(34)} V T ( W GD ) T U 1 2 ≤ ∥ V ∥ op 2 ∥ W GD ∥ op 2 ∥ U 1 ∥ 2 = O p ( N ⋅ N 2 ⋅ N d ) = O p ( N 4 d ) . (34)
For the second sum, u i T W G D v i = [ U T W G D V ] i i u_i^{\mathsf{T}}W^{\mathrm{GD}}v_i=[U^{\mathsf{T}}W^{\mathrm{GD}}V]_{ii} u i T W GD v i = [ U T W GD V ] ii and ∑ i X i i 2 ≤ ∥ X ∥ F 2 \sum_iX_{ii}^2\le\lVert X\rVert_F^2 ∑ i X ii 2 ≤ ∥ X ∥ F 2 ; by Equation (11) , U T W G D V = ( U T U ) ( I N − 1 N 11 T ) ( V T V ) U^{\mathsf{T}}W^{\mathrm{GD}}V=(U^{\mathsf{T}}U)(I_N-\frac{1}{N}1 1^{\mathsf{T}})(V^{\mathsf{T}}V) U T W GD V = ( U T U ) ( I N − N 1 1 1 T ) ( V T V ) , and since the nonzero eigenvalues of V T V V^{\mathsf{T}}V V T V and V V T VV^{\mathsf{T}} V V T coincide, ∥ V T V ∥ F = ∥ V V T ∥ F \lVert V^{\mathsf{T}}V\rVert_F=\lVert VV^{\mathsf{T}}\rVert_F ∥ V T V ∥ F = ∥ V V T ∥ F , so by Theorem F.3(iv),(vi),
∑ i ( u i T W G D v i ) 2 ≤ ∥ U T W G D V ∥ F 2 ≤ ∥ U ∥ o p 4 ∥ V V T ∥ F 2 = O p ( N 2 ⋅ N 2 d ) = O p ( N 4 d ) . (35) \sum_i\left(u_i^{\mathsf{T}}W^{\mathrm{GD}}v_i\right)^2\le\lVert U^{\mathsf{T}}W^{\mathrm{GD}}V\rVert_F^2\le\lVert U\rVert_{\mathrm{op}}^4\lVert VV^{\mathsf{T}}\rVert_F^2=O_p(N^2\cdot N^2d)=O_p(N^4d). \tag*{(35)} i ∑ ( u i T W GD v i ) 2 ≤ ∥ U T W GD V ∥ F 2 ≤ ∥ U ∥ op 4 ∥ V V T ∥ F 2 = O p ( N 2 ⋅ N 2 d ) = O p ( N 4 d ) . (35)
Lemma F.10. mean i ( mean j ≠ i u j T W G D v i ) 2 = O p ( γ 2 d 2 ) \operatorname{mean}_i\left(\operatorname{mean}_{j\ne i}u_j^{\mathsf{T}}W^{\mathrm{GD}}v_i\right)^2=O_p(\gamma^2d^2) mean i ( mean j = i u j T W GD v i ) 2 = O p ( γ 2 d 2 ) , hence ∣ mean i mean j ≠ i u j T W G D v i ∣ = O p ( γ d ) \left|\operatorname{mean}_i\operatorname{mean}_{j\ne i}u_j^{\mathsf{T}}W^{\mathrm{GD}}v_i\right|=O_p(\gamma d) mean i mean j = i u j T W GD v i = O p ( γ d ) .
Proof. Square Theorem F.8 with W = W G D W=W^{\mathrm{GD}} W = W GD , use ( x − y ) 2 ≤ 2 x 2 + 2 y 2 (x-y)^2\le2x^2+2y^2 ( x − y ) 2 ≤ 2 x 2 + 2 y 2 , sum over i i i and apply Theorem F.9:
∑ i ( ( N − 1 ) mean j ≠ i u j T W G D v i ) 2 = O p ( N 4 d ) ≤ O p ( N 5 ) , (36) \sum_i\left((N-1)\operatorname{mean}_{j\ne i}u_j^{\mathsf{T}}W^{\mathrm{GD}}v_i\right)^2=O_p(N^4d)\le O_p(N^5), \tag*{(36)} i ∑ ( ( N − 1 ) mean j = i u j T W GD v i ) 2 = O p ( N 4 d ) ≤ O p ( N 5 ) , (36)
using d ≤ N d\le N d ≤ N . Dividing by N ( N − 1 ) 2 ≥ 1 4 N 3 N(N-1)^2\ge\frac{1}{4}N^3 N ( N − 1 ) 2 ≥ 4 1 N 3 and substituting N = γ d N=\gamma d N = γ d gives the first claim; the second follows from Cauchy–Schwarz, ∣ mean i x i ∣ ≤ ( mean i x i 2 ) 1 / 2 \left|\operatorname{mean}_i x_i\right|\le\left(\operatorname{mean}_i x_i^2\right)^{1/2} ∣ mean i x i ∣ ≤ ( mean i x i 2 ) 1/2 .
Lemma F.11 (h ˉ \bar{h} h ˉ for GD). h ˉ G D = d 2 ( 1 + O p ( γ / d ) ) \bar{h}^{\mathrm{GD}}=d^2(1+O_p(\gamma/d)) h ˉ GD = d 2 ( 1 + O p ( γ / d )) .
Proof. By Equation (32) , Theorem F.7 and Theorem F.10, h ˉ G D = d 2 ( 1 + O p ( d − 1 ) ) − O p ( γ d ) = d 2 ( 1 + O p ( d − 1 ) + O p ( γ / d ) ) \bar{h}^{\mathrm{GD}}=d^2(1+O_p(d^{-1}))-O_p(\gamma d)=d^2(1+O_p(d^{-1})+O_p(\gamma/d)) h ˉ GD = d 2 ( 1 + O p ( d − 1 )) − O p ( γ d ) = d 2 ( 1 + O p ( d − 1 ) + O p ( γ / d )) , and γ ≥ 1 \gamma\ge1 γ ≥ 1 absorbs the first error term into the second.
We turn to κ ˉ G D \bar{\kappa}^{\mathrm{GD}} κ ˉ GD . By Equation (10) ,
( κ ˉ G D ) 2 = 1 N ( N − 1 ) ∑ i = 1 N ∑ j ≠ i ( u j T W G D v i ) 2 ⏟ second moment − mean i ( mean j ≠ i u j T W G D v i ) 2 ⏟ squared mean . (37) (\bar{\kappa}^{\mathrm{GD}})^2=\underbrace{\frac{1}{N(N-1)}\sum_{i=1}^{N}\sum_{j\ne i}(u_j^{\mathsf{T}}W^{\mathrm{GD}}v_i)^2}_{\text{second moment}}-\underbrace{\operatorname{mean}_i\left(\operatorname{mean}_{j\ne i}u_j^{\mathsf{T}}W^{\mathrm{GD}}v_i\right)^2}_{\text{squared mean}}. \tag*{(37)} ( κ ˉ GD ) 2 = second moment N ( N − 1 ) 1 i = 1 ∑ N j = i ∑ ( u j T W GD v i ) 2 − squared mean mean i ( mean j = i u j T W GD v i ) 2 . (37)
Lemma F.12 (Expectation of the second moment).
E ∑ i = 1 N ∑ j ≠ i ( u j T W G D v i ) 2 = N ( N − 1 ) d 2 ( N + 2 d + 2 ) \mathbb{E}\sum_{i=1}^{N}\sum_{j\ne i}(u_j^{\mathsf{T}}W^{\mathrm{GD}}v_i)^2=N(N-1)d^2(N+2d+2) E i = 1 ∑ N j = i ∑ ( u j T W GD v i ) 2 = N ( N − 1 ) d 2 ( N + 2 d + 2 )
− d 2 [ ( 1 + 1 N ) ( N + d + 1 ) 2 − 2 ( d + 2 ) 2 − 2 ( N − 1 ) ] = N ( N − 1 ) d 2 ( N + 2 d + 2 ) ( 1 + O ( N − 1 ) ) . (38) \begin{aligned}
\qquad-d^2\left[\left(1+\frac{1}{N}\right)(N+d+1)^2-2(d+2)^2-2(N-1)\right] \\
&=N(N-1)d^2(N+2d+2)(1+O(N^{-1})). \tag*{(38)}
\end{aligned} − d 2 [ ( 1 + N 1 ) ( N + d + 1 ) 2 − 2 ( d + 2 ) 2 − 2 ( N − 1 ) ] = N ( N − 1 ) d 2 ( N + 2 d + 2 ) ( 1 + O ( N − 1 )) . (38)
Proof. Write u j T W G D v i = [ U T W G D V ] j i u_j^{\mathsf{T}}W^{\mathrm{GD}}v_i=[U^{\mathsf{T}}W^{\mathrm{GD}}V]_{ji} u j T W GD v i = [ U T W GD V ] j i ; by Equation (11) , U T W G D V = ( U T U ) ( I N − 11 T N ) ( V T V ) U^{\mathsf{T}}W^{\mathrm{GD}}V=(U^{\mathsf{T}}U)\left(I_N-\frac{11^{\mathsf{T}}}{N}\right)(V^{\mathsf{T}}V) U T W GD V = ( U T U ) ( I N − N 1 1 T ) ( V T V ) , i.e. u j T W G D v i = ∑ k , l ( u j T u k ) ( δ k l − 1 N ) ( v l T v i ) u_j^{\mathsf{T}}W^{\mathrm{GD}}v_i=\sum_{k,l}(u_j^{\mathsf{T}}u_k)(\delta_{kl}-\frac{1}{N})(v_l^{\mathsf{T}}v_i) u j T W GD v i = ∑ k , l ( u j T u k ) ( δ k l − N 1 ) ( v l T v i ) , with U U U and V V V independent. Two second moments, identical for u u u and v v v , are used:
E [ ( u k T u l ) ( u l T u m ) ] = { d 2 + 2 d , k = l = m , d , k = m ≠ l , 0 , otherwise , E [ ( u i T u k ) ( u i T u k ′ ) ] = δ k k ′ d [ 1 + ( d + 1 ) δ k i ] . (39) \mathbb{E}\left[(u_k^{\mathsf{T}}u_l)(u_l^{\mathsf{T}}u_m)\right]=
\begin{cases}
d^2+2d, & k=l=m,\\
d, & k=m\ne l,\\
0, & \text{otherwise},
\end{cases}
\qquad
\mathbb{E}\left[(u_i^{\mathsf{T}}u_k)(u_i^{\mathsf{T}}u_{k'})\right]=\delta_{kk'}d\left[1+(d+1)\delta_{ki}\right]. \tag*{(39)} E [ ( u k T u l ) ( u l T u m ) ] = ⎩ ⎨ ⎧ d 2 + 2 d , d , 0 , k = l = m , k = m = l , otherwise , E [ ( u i T u k ) ( u i T u k ′ ) ] = δ k k ′ d [ 1 + ( d + 1 ) δ k i ] . (39)
The first of these gives [ E ( U T U ) 2 ] k m = ∑ l E [ ( u k T u l ) ( u l T u m ) ] = δ k m [ ( d 2 + 2 d ) + ( N − 1 ) d ] = δ k m d ( d + N + 1 ) \left[\mathbb{E}(U^{\mathsf{T}}U)^2\right]_{km}=\sum_l\mathbb{E}\left[(u_k^{\mathsf{T}}u_l)(u_l^{\mathsf{T}}u_m)\right]=\delta_{km}\left[(d^2+2d)+(N-1)d\right]=\delta_{km}d(d+N+1) [ E ( U T U ) 2 ] k m = ∑ l E [ ( u k T u l ) ( u l T u m ) ] = δ k m [ ( d 2 + 2 d ) + ( N − 1 ) d ] = δ k m d ( d + N + 1 ) , i.e. E ( U T U ) 2 = d ( d + N + 1 ) I N \mathbb{E}(U^{\mathsf{T}}U)^2=d(d+N+1)I_N E ( U T U ) 2 = d ( d + N + 1 ) I N , and likewise for V V V . All pairs ( i , j ) (i,j) ( i , j ) . Since ∥ X ∥ F 2 = tr ( X T X ) \lVert X\rVert_F^2=\operatorname{tr}(X^{\mathsf{T}}X) ∥ X ∥ F 2 = tr ( X T X ) and by cyclicity,
∑ i , j ( u j T W G D v i ) 2 = ∥ U T W G D V ∥ F 2 = tr [ ( U T U ) 2 ( I N − 11 T N ) ( V T V ) 2 ( I N − 11 T N ) ] , (40) \sum_{i,j}(u_j^{\mathsf{T}}W^{\mathrm{GD}}v_i)^2=\lVert U^{\mathsf{T}}W^{\mathrm{GD}}V\rVert_F^2=\operatorname{tr}\left[(U^{\mathsf{T}}U)^2\left(I_N-\frac{11^{\mathsf{T}}}{N}\right)(V^{\mathsf{T}}V)^2\left(I_N-\frac{11^{\mathsf{T}}}{N}\right)\right], \tag*{(40)} i , j ∑ ( u j T W GD v i ) 2 = ∥ U T W GD V ∥ F 2 = tr [ ( U T U ) 2 ( I N − N 1 1 T ) ( V T V ) 2 ( I N − N 1 1 T ) ] , (40)
and by independence and tr [ ( I N − 11 T N ) 2 ] = N − 1 \operatorname{tr}\left[\left(I_N-\frac{11^{\mathsf{T}}}{N}\right)^2\right]=N-1 tr [ ( I N − N 1 1 T ) 2 ] = N − 1 ,
E ∑ i , j ( u j T W G D v i ) 2 = [ d ( d + N + 1 ) ] 2 ( N − 1 ) . (41) \mathbb{E}\sum_{i,j}(u_j^{\mathsf{T}}W^{\mathrm{GD}}v_i)^2=[d(d+N+1)]^2(N-1). \tag*{(41)} E i , j ∑ ( u j T W GD v i ) 2 = [ d ( d + N + 1 ) ] 2 ( N − 1 ) . (41)
Diagonal j = i j=i j = i . Using the independence of U U U and V V V , the second display above for both factors, and then ( δ k l − 1 N ) 2 = δ k l ( 1 − 2 N ) + 1 N 2 \left(\delta_{kl}-\frac{1}{N}\right)^2=\delta_{kl}\left(1-\frac{2}{N}\right)+\frac{1}{N^2} ( δ k l − N 1 ) 2 = δ k l ( 1 − N 2 ) + N 2 1 together with ∑ k [ 1 + ( d + 1 ) δ k i ] = N + d + 1 \sum_k[1+(d+1)\delta_{ki}]=N+d+1 ∑ k [ 1 + ( d + 1 ) δ k i ] = N + d + 1 and ∑ k [ 1 + ( d + 1 ) δ k i ] 2 = N − 1 + ( d + 2 ) 2 \sum_k[1+(d+1)\delta_{ki}]^2=N-1+(d+2)^2 ∑ k [ 1 + ( d + 1 ) δ k i ] 2 = N − 1 + ( d + 2 ) 2 ,
E ( u i T W G D v i ) 2 = ∑ k , l , k ′ , l ′ E [ ( u i T u k ) ( u i T u k ′ ) ] E [ ( v l T v i ) ( v l ′ T v i ) ] ( δ k l − 1 N ) ( δ k ′ l ′ − 1 N ) = d 2 ∑ k , l [ 1 + ( d + 1 ) δ k i ] [ 1 + ( d + 1 ) δ l i ] ( δ k l − 1 N ) 2 = d 2 [ ( 1 − 2 N ) ( N − 1 + ( d + 2 ) 2 ) + ( N + d + 1 ) 2 N 2 ] . (42) \begin{aligned}
\mathbb{E}(u_i^{\mathsf{T}}W^{\mathrm{GD}}v_i)^2=\sum_{k,l,k',l'}\mathbb{E}\left[(u_i^{\mathsf{T}}u_k)(u_i^{\mathsf{T}}u_{k'})\right]\mathbb{E}\left[(v_l^{\mathsf{T}}v_i)(v_{l'}^{\mathsf{T}}v_i)\right]\left(\delta_{kl}-\frac{1}{N}\right)\left(\delta_{k'l'}-\frac{1}{N}\right) \\
&=d^2\sum_{k,l}[1+(d+1)\delta_{ki}][1+(d+1)\delta_{li}]\left(\delta_{kl}-\frac{1}{N}\right)^2 \\
&=d^2\left[\left(1-\frac{2}{N}\right)\left(N-1+(d+2)^2\right)+\frac{(N+d+1)^2}{N^2}\right]. \tag*{(42)}
\end{aligned} E ( u i T W GD v i ) 2 = k , l , k ′ , l ′ ∑ E [ ( u i T u k ) ( u i T u k ′ ) ] E [ ( v l T v i ) ( v l ′ T v i ) ] ( δ k l − N 1 ) ( δ k ′ l ′ − N 1 ) = d 2 k , l ∑ [ 1 + ( d + 1 ) δ k i ] [ 1 + ( d + 1 ) δ l i ] ( δ k l − N 1 ) 2 = d 2 [ ( 1 − N 2 ) ( N − 1 + ( d + 2 ) 2 ) + N 2 ( N + d + 1 ) 2 ] . (42)
Subtracting. E ∑ i ∑ j ≠ i = E ∑ i , j − N E ( u i T W G D v i ) 2 \mathbb{E}\sum_i\sum_{j\ne i}=\mathbb{E}\sum_{i,j}-N\mathbb{E}(u_i^{\mathsf{T}}W^{\mathrm{GD}}v_i)^2 E ∑ i ∑ j = i = E ∑ i , j − N E ( u i T W GD v i ) 2 ; rearranging with ( N + d + 1 ) 2 − ( d + 2 ) 2 − ( N − 1 ) = ( N − 1 ) ( N + 2 d + 2 ) (N+d+1)^2-(d+2)^2-(N-1)=(N-1)(N+2d+2) ( N + d + 1 ) 2 − ( d + 2 ) 2 − ( N − 1 ) = ( N − 1 ) ( N + 2 d + 2 ) gives the first equality. Since d ≤ N d\le N d ≤ N , the discarded terms are bounded by 38 N 2 d 2 N − 1 ⋅ N ( N − 1 ) d 2 ( N + 2 d + 2 ) \frac{38N^2d^2}{N-1}\cdot N(N-1)d^2(N+2d+2) N − 1 38 N 2 d 2 ⋅ N ( N − 1 ) d 2 ( N + 2 d + 2 ) , which is the stated relative error. □
Lemma F.13 (Variance of the second moment). There is an absolute constant C C C with var [ ∑ i ∑ j ≠ i ( u j T W G D v i ) 2 ] ≤ C N 7 d \operatorname{var}\left[\sum_i\sum_{j\ne i}(u_j^{\mathsf{T}}W^{\mathrm{GD}}v_i)^2\right]\le CN^7d var [ ∑ i ∑ j = i ( u j T W GD v i ) 2 ] ≤ C N 7 d .
Proof. By Equation (11) , ∑ i ∑ j ≠ i ( u j T W G D v i ) 2 = ∥ U T W G D V ∥ F 2 − ∑ i [ U T W G D V ] i i 2 \sum_i\sum_{j\ne i}(u_j^{\mathsf{T}}W^{\mathrm{GD}}v_i)^2=\lVert U^{\mathsf{T}}W^{\mathrm{GD}}V\rVert_F^2-\sum_i[U^{\mathsf{T}}W^{\mathrm{GD}}V]^2_{ii} ∑ i ∑ j = i ( u j T W GD v i ) 2 = ∥ U T W GD V ∥ F 2 − ∑ i [ U T W GD V ] ii 2 , a smooth function of the 2 N d 2Nd 2 N d independent standard Gaussian entries of ( U , V ) (U,V) ( U , V ) . By the Gaussian Poincaré inequality the variance is at most E ∥ ∇ U ⋅ ∥ F 2 + E ∥ ∇ V ⋅ ∥ F 2 \mathbb{E}\lVert\nabla_U\cdot\rVert_F^2+\mathbb{E}\lVert\nabla_V\cdot\rVert_F^2 E ∥ ∇ U ⋅ ∥ F 2 + E ∥ ∇ V ⋅ ∥ F 2 .
The gradient of M ↦ ∥ M ∥ F 2 − ∑ i M i i 2 M\mapsto\lVert M\rVert_F^2-\sum_iM_{ii}^2 M ↦ ∥ M ∥ F 2 − ∑ i M ii 2 is 2 M − 2 diag ( M ) 2M-2\operatorname{diag}(M) 2 M − 2 diag ( M ) , of Frobenius norm at most 2 ∥ M ∥ F 2\lVert M\rVert_F 2 ∥ M ∥ F ; the directional derivative of U ↦ ( U T U ) ( I N − 11 T N ) ( V T V ) U\mapsto(U^{\mathsf{T}}U)\left(I_N-\frac{11^{\mathsf{T}}}{N}\right)(V^{\mathsf{T}}V) U ↦ ( U T U ) ( I N − N 1 1 T ) ( V T V ) along H H H has Frobenius norm at most 2 ∥ H ∥ F ∥ U ∥ o p ∥ V ∥ o p 2\lVert H\rVert_F\lVert U\rVert_{\mathrm{op}}\lVert V\rVert_{\mathrm{op}} 2 ∥ H ∥ F ∥ U ∥ op ∥ V ∥ op . With ∥ U T W G D V ∥ F ≤ ∥ U ∥ o p 2 ∥ V V T ∥ F \lVert U^{\mathsf{T}}W^{\mathrm{GD}}V\rVert_F\le\lVert U\rVert_{\mathrm{op}}^2\lVert VV^{\mathsf{T}}\rVert_F ∥ U T W GD V ∥ F ≤ ∥ U ∥ op 2 ∥ V V T ∥ F , the chain rule gives
∥ ∇ U ∑ i ∑ j ≠ i ( u j T W G D v i ) 2 ∥ F ≤ 4 ∥ U ∥ o p 3 ∥ V ∥ o p 2 ∥ V V T ∥ F . (43) \left\lVert\nabla_U\sum_i\sum_{j\ne i}(u_j^{\mathsf{T}}W^{\mathrm{GD}}v_i)^2\right\rVert_F\le4\lVert U\rVert_{\mathrm{op}}^3\lVert V\rVert_{\mathrm{op}}^2\lVert VV^{\mathsf{T}}\rVert_F. \tag*{(43)} ∇ U i ∑ j = i ∑ ( u j T W GD v i ) 2 F ≤ 4 ∥ U ∥ op 3 ∥ V ∥ op 2 ∥ V V T ∥ F . (43)
Squaring, taking expectations and applying Hölder together with Theorem F.3(iv),(vi) gives E ∥ ∇ U ⋅ ∥ F 2 ≤ 16 ( E ∥ U ∥ o p 8 ) 1 / 3 ( E ∥ V ∥ o p 12 ) 1 / 3 ( E ∥ V V T ∥ F 6 ) 1 / 3 ≤ 16 ( C 9 C 2 6 ) 1 / 3 N 7 d \mathbb{E}\lVert\nabla_U\cdot\rVert_F^2\le16(\mathbb{E}\lVert U\rVert_{\mathrm{op}}^8)^{1/3}(\mathbb{E}\lVert V\rVert_{\mathrm{op}}^{12})^{1/3}(\mathbb{E}\lVert VV^{\mathsf{T}}\rVert_F^6)^{1/3}\le16(C_9C_2^6)^{1/3}N^7d E ∥ ∇ U ⋅ ∥ F 2 ≤ 16 ( E ∥ U ∥ op 8 ) 1/3 ( E ∥ V ∥ op 12 ) 1/3 ( E ∥ V V T ∥ F 6 ) 1/3 ≤ 16 ( C 9 C 2 6 ) 1/3 N 7 d . The expression is invariant under transposition combined with swapping U U U and V V V , so the same bound holds for ∇ V \nabla_V ∇ V ; adding the two gives the claim with C = 32 ( C 9 C 2 6 ) 1 / 3 C=32(C_9C_2^6)^{1/3} C = 32 ( C 9 C 2 6 ) 1/3 . □
Lemma F.14 (Second moment). The second moment of the off-diagonal GD scores satisfies
1 N ( N − 1 ) ∑ i = 1 N ∑ j ≠ i ( u j T W G D v i ) 2 = d 3 ( 2 + γ ) ( 1 + O p ( γ d ) ) . \frac{1}{N(N-1)}\sum_{i=1}^{N}\sum_{j\ne i}(u_j^{\mathsf{T}}W^{\mathrm{GD}}v_i)^2=d^3(2+\gamma)\left(1+O_{\mathrm{p}}\left(\sqrt{\frac{\gamma}{d}}\right)\right). N ( N − 1 ) 1 i = 1 ∑ N j = i ∑ ( u j T W GD v i ) 2 = d 3 ( 2 + γ ) ( 1 + O p ( d γ ) ) .
Proof. By Equation (38) and N ( N − 1 ) d 2 ( N + 2 d + 2 ) ≥ N 2 ( N − 1 ) d 2 ≥ 1 2 N 3 d 2 N(N-1)d^2(N+2d+2) \ge N^2(N-1)d^2 \ge\frac{1}{2}N^3d^2 N ( N − 1 ) d 2 ( N + 2 d + 2 ) ≥ N 2 ( N − 1 ) d 2 ≥ 2 1 N 3 d 2 for N ≥ 2 N \ge2 N ≥ 2 , the expectation is at least 1 4 N 3 d 2 \frac{1}{4}N^3d^2 4 1 N 3 d 2 for N N N large. Combined with Theorem F.13, the relative standard deviation is at most C N 7 d / ( 1 4 N 3 d 2 ) = 4 C N / d 3 / 2 = 4 C γ / d \sqrt{CN^7d}/(\frac{1}{4}N^3d^2)=4\sqrt{C}\sqrt{N}/d^{3/2}=4\sqrt{C}\sqrt{\gamma}/d C N 7 d / ( 4 1 N 3 d 2 ) = 4 C N / d 3/2 = 4 C γ / d , so Chebyshev gives
∑ i ∑ j ≠ i ( u j T W G D v i ) 2 = ( E ∑ i ∑ j ≠ i ( u j T W G D v i ) 2 ) ( 1 + O p ( γ / d ) ) . (44) \sum_i\sum_{j\ne i}(u_j^{\mathsf T}W^{\mathrm{GD}}v_i)^2=\left(\mathbb{E}\sum_i\sum_{j\ne i}(u_j^{\mathsf T}W^{\mathrm{GD}}v_i)^2\right)(1+O_p(\sqrt{\gamma}/d)). \tag*{(44)} i ∑ j = i ∑ ( u j T W GD v i ) 2 = E i ∑ j = i ∑ ( u j T W GD v i ) 2 ( 1 + O p ( γ / d )) . (44)
Since N − 1 = γ − 1 d − 1 ≤ γ / d N^{-1}=\gamma^{-1}d^{-1}\le\sqrt{\gamma}/d N − 1 = γ − 1 d − 1 ≤ γ / d , the relative error O ( N − 1 ) O(N^{-1}) O ( N − 1 ) of Theorem F.12 is absorbed; dividing by N ( N − 1 ) N(N-1) N ( N − 1 ) leaves d 2 ( N + 2 d + 2 ) = d 3 ( γ + 2 + 2 d ) = d 3 ( 2 + γ ) ( 1 + 2 d ( 2 + γ ) ) d^2(N+2d+2)=d^3(\gamma+2+\frac{2}{d})=d^3(2+\gamma)(1+\frac{2}{d(2+\gamma)}) d 2 ( N + 2 d + 2 ) = d 3 ( γ + 2 + d 2 ) = d 3 ( 2 + γ ) ( 1 + d ( 2 + γ ) 2 ) and 2 d ( 2 + γ ) ≤ 1 d ≤ γ / d \frac{2}{d(2+\gamma)}\le\frac{1}{d}\le\sqrt{\gamma}/d d ( 2 + γ ) 2 ≤ d 1 ≤ γ / d .
Lemma F.15 (κ ˉ \bar{\kappa} κ ˉ for GD). ( κ ˉ G D ) 2 = d 3 ( 2 + γ ) ( 1 + O p ( γ / d ) ) (\bar{\kappa}^{\mathrm{GD}})^2=d^3(2+\gamma)(1+O_p(\gamma/d)) ( κ ˉ GD ) 2 = d 3 ( 2 + γ ) ( 1 + O p ( γ / d )) .
Proof. By Theorem F.10 the squared mean in Equation (37) is O p ( γ 2 d 2 ) = d 3 ( 2 + γ ) ⋅ O p ( γ 2 d ( 2 + γ ) ) = d 3 ( 2 + γ ) ⋅ O p ( γ / d ) O_p(\gamma^2d^2)=d^3(2+\gamma)\cdot O_p\left(\frac{\gamma^2}{d(2+\gamma)}\right)=d^3(2+\gamma)\cdot O_p(\gamma/d) O p ( γ 2 d 2 ) = d 3 ( 2 + γ ) ⋅ O p ( d ( 2 + γ ) γ 2 ) = d 3 ( 2 + γ ) ⋅ O p ( γ / d ) . Subtracting it from Theorem F.14 and using γ ≤ γ \sqrt{\gamma}\le\gamma γ ≤ γ gives the claim.
Lemma F.16 (ρ \rho ρ for GD). The margin-to-spread ratio of the GD iterate is
ρ G D = d 2 + γ ( 1 + O p ( γ / d ) ) = d γ ( 1 − 1 γ + O ( γ − 2 ) + o p ( 1 ) ) . \rho^{\mathrm{GD}}=\sqrt{\frac{d}{2+\gamma}}(1+O_p(\gamma/d))=\sqrt{\frac{d}{\gamma}}\left(1-\frac{1}{\gamma}+O(\gamma^{-2})+o_p(1)\right). ρ GD = 2 + γ d ( 1 + O p ( γ / d )) = γ d ( 1 − γ 1 + O ( γ − 2 ) + o p ( 1 ) ) . Both h ˉ G D \bar{h}^{\mathrm{GD}} h ˉ GD and κ ˉ G D \bar{\kappa}^{\mathrm{GD}} κ ˉ GD are a power of d d d times a function of γ \gamma γ alone, so ρ G D \rho^{\mathrm{GD}} ρ GD depends on d d d only through the prefactor d \sqrt{d} d .
Proof. By Theorem F.11, Theorem F.15 and 1 + O p ( γ / d ) = 1 + O p ( γ / d ) \sqrt{1+O_p(\gamma/d)}=1+O_p(\gamma/d) 1 + O p ( γ / d ) = 1 + O p ( γ / d ) ,
ρ G D = h ˉ G D κ ˉ G D = d 2 ( 1 + O p ( γ / d ) ) d 3 / 2 2 + γ ( 1 + O p ( γ / d ) ) = d 2 + γ ( 1 + O p ( γ / d ) ) . (45) \rho^{\mathrm{GD}}=\frac{\bar{h}^{\mathrm{GD}}}{\bar{\kappa}^{\mathrm{GD}}}=\frac{d^2(1+O_p(\gamma/d))}{d^{3/2}\sqrt{2+\gamma}(1+O_p(\gamma/d))}=\sqrt{\frac{d}{2+\gamma}}(1+O_p(\gamma/d)). \tag*{(45)} ρ GD = κ ˉ GD h ˉ GD = d 3/2 2 + γ ( 1 + O p ( γ / d )) d 2 ( 1 + O p ( γ / d )) = 2 + γ d ( 1 + O p ( γ / d )) . (45) With γ \gamma γ fixed and d → ∞ d\to\infty d → ∞ , O p ( γ / d ) = o p ( 1 ) O_p(\gamma/d)=o_p(1) O p ( γ / d ) = o p ( 1 ) ; moreover γ / ( 2 + γ ) = ( 1 + 2 / γ ) − 1 / 2 = 1 − γ − 1 + O ( γ − 2 ) \sqrt{\gamma/(2+\gamma)}=(1+2/\gamma)^{-1/2}=1-\gamma^{-1}+O(\gamma^{-2}) γ / ( 2 + γ ) = ( 1 + 2/ γ ) − 1/2 = 1 − γ − 1 + O ( γ − 2 ) .
The Muon step
Throughout this subsection γ ≥ 16 \gamma\ge16 γ ≥ 16 . By Equation (10) with W = W M u o n W=W^{\mathrm{Muon}} W = W Muon the same two decompositions hold as in Equation (32) and Equation (37) , with W G D W^{\mathrm{GD}} W GD replaced by W M u o n W^{\mathrm{Muon}} W Muon . We use throughout that W M u o n W^{\mathrm{Muon}} W Muon is orthogonal (Theorem F.2).
Lemma F.17 (Diagonal-sum identity). The diagonal scores of W M u o n W^{\mathrm{Muon}} W Muon sum to
∑ i = 1 N u i T W M u o n v i = ∥ W G D ∥ ∗ + 1 N ( U 1 ) T W M u o n ( V 1 ) . \sum_{i=1}^{N}u_i^{\mathsf T}W^{\mathrm{Muon}}v_i=\lVert W^{\mathrm{GD}}\rVert_*+\frac{1}{N}(U1)^{\mathsf T}W^{\mathrm{Muon}}(V1). i = 1 ∑ N u i T W Muon v i = ∥ W GD ∥ ∗ + N 1 ( U 1 ) T W Muon ( V 1 ) .
Proof. As in Theorem F.5, ∑ i u i T W M u o n v i = ⟨ W M u o n , U V T ⟩ = ⟨ W M u o n , W G D ⟩ + 1 N ( U 1 ) T W M u o n ( V 1 ) \sum_i u_i^{\mathsf T}W^{\mathrm{Muon}}v_i=\langle W^{\mathrm{Muon}},UV^{\mathsf T}\rangle=\langle W^{\mathrm{Muon}},W^{\mathrm{GD}}\rangle+\frac{1}{N}(U1)^{\mathsf T}W^{\mathrm{Muon}}(V1) ∑ i u i T W Muon v i = ⟨ W Muon , U V T ⟩ = ⟨ W Muon , W GD ⟩ + N 1 ( U 1 ) T W Muon ( V 1 ) . Writing the compact SVD W G D = P Σ Q T W^{\mathrm{GD}}=P\Sigma Q^{\mathsf T} W GD = P Σ Q T , so that W M u o n = P Q T W^{\mathrm{Muon}}=PQ^{\mathsf T} W Muon = P Q T , and using P T P = Q T Q = I P^{\mathsf T}P=Q^{\mathsf T}Q=I P T P = Q T Q = I ,
⟨ W M u o n , W G D ⟩ = tr ( Q P T P Σ Q T ) = tr ( Σ Q T Q ) = tr Σ = ∥ W G D ∥ ∗ . (46) \langle W^{\mathrm{Muon}},W^{\mathrm{GD}}\rangle=\operatorname{tr}(QP^{\mathsf T}P\Sigma Q^{\mathsf T})=\operatorname{tr}(\Sigma Q^{\mathsf T}Q)=\operatorname{tr}\Sigma=\lVert W^{\mathrm{GD}}\rVert_*. \tag*{(46)} ⟨ W Muon , W GD ⟩ = tr ( Q P T P Σ Q T ) = tr ( Σ Q T Q ) = tr Σ = ∥ W GD ∥ ∗ . (46)
Lemma F.18 (Rank-one cross term). 1 N 2 ∣ ( U 1 ) T W M u o n ( V 1 ) ∣ = O p ( d / N ) \frac{1}{N^2}|(U1)^{\mathsf T}W^{\mathrm{Muon}}(V1)|=O_p(d/N) N 2 1 ∣ ( U 1 ) T W Muon ( V 1 ) ∣ = O p ( d / N ) .
Proof. By ∥ W M u o n ∥ o p ≤ 1 \lVert W^{\mathrm{Muon}}\rVert_{\mathrm{op}}\le1 ∥ W Muon ∥ op ≤ 1 and Theorem F.3(iii), ∣ ( U 1 ) T W M u o n ( V 1 ) ∣ / N 2 ≤ ∥ U 1 ∥ ∥ V 1 ∥ / N 2 = O p ( N d ) / N 2 = O p ( d / N ) |(U1)^{\mathsf T}W^{\mathrm{Muon}}(V1)|/N^2\le\lVert U1\rVert\lVert V1\rVert/N^2=O_p(Nd)/N^2=O_p(d/N) ∣ ( U 1 ) T W Muon ( V 1 ) ∣/ N 2 ≤ ∥ U 1 ∥ ∥ V 1 ∥ / N 2 = O p ( N d ) / N 2 = O p ( d / N ) .
Lemma F.19. mean i u i T W M u o n v i = ∥ W G D ∥ ∗ N ( 1 + O p ( ( N d ) − 1 / 2 ) ) \operatorname{mean}_i u_i^{\mathsf T}W^{\mathrm{Muon}}v_i=\frac{\lVert W^{\mathrm{GD}}\rVert_*}{N}(1+O_p((Nd)^{-1/2})) mean i u i T W Muon v i = N ∥ W GD ∥ ∗ ( 1 + O p (( N d ) − 1/2 )) .
Proof. Divide Theorem F.17 by N N N and use Theorem F.18. By Lemma F.4 (iii), with probability tending to one ∥ W G D ∥ ∗ / N ≥ d N d / ( 8 N ) = d 3 / 2 / ( 8 N ) \lVert W^{\mathrm{GD}}\rVert_*/N\ge d\sqrt{Nd}/(8N)=d^{3/2}/(8\sqrt{N}) ∥ W GD ∥ ∗ / N ≥ d N d / ( 8 N ) = d 3/2 / ( 8 N ) , so the error term satisfies O p ( d / N ) = ∥ W G D ∥ ∗ N O p ( d N 8 N d 3 / 2 ) = ∥ W G D ∥ ∗ N O p ( ( N d ) − 1 / 2 ) O_p(d/N)=\frac{\lVert W^{\mathrm{GD}}\rVert_*}{N}O_p\left(\frac{d}{N}\frac{8\sqrt{N}}{d^{3/2}}\right)=\frac{\lVert W^{\mathrm{GD}}\rVert_*}{N}O_p((Nd)^{-1/2}) O p ( d / N ) = N ∥ W GD ∥ ∗ O p ( N d d 3/2 8 N ) = N ∥ W GD ∥ ∗ O p (( N d ) − 1/2 ) .
Lemma F.20 (Two sums of squares). ∑ i ( ( U 1 ) ⊤ W M u o n v i ) 2 = O p ( N 2 d ) \sum_i ((U1)^\top W^{\mathrm{Muon}}v_i)^2 = O_p(N^2d) ∑ i (( U 1 ) ⊤ W Muon v i ) 2 = O p ( N 2 d ) and ∑ i ( u i ⊤ W M u o n v i ) 2 = O p ( N d 2 ) \sum_i (u_i^\top W^{\mathrm{Muon}}v_i)^2 = O_p(Nd^2) ∑ i ( u i ⊤ W Muon v i ) 2 = O p ( N d 2 ) .
Proof. For the first, W M u o n W^{\mathrm{Muon}} W Muon is orthogonal, so ∥ ( W M u o n ) ⊤ U 1 ∥ = ∥ U 1 ∥ \lVert(W^{\mathrm{Muon}})^\top U1\rVert= \lVert U1\rVert ∥( W Muon ) ⊤ U 1 ∥ = ∥ U 1 ∥ , and ∑ i v i v i ⊤ = V V ⊤ \sum_i v_i v_i^\top= VV^\top ∑ i v i v i ⊤ = V V ⊤ , whence by Theorem F.3(iii),(iv)
∑ i ( ( U 1 ) ⊤ W M u o n v i ) 2 = ( U 1 ) ⊤ W M u o n V V ⊤ ( W M u o n ) ⊤ ( U 1 ) ≤ ∥ V ∥ o p 2 ∥ U 1 ∥ 2 = O p ( N ⋅ N d ) . (47) \sum_i ((U1)^\top W^{\mathrm{Muon}}v_i)^2 = (U1)^\top W^{\mathrm{Muon}}VV^\top(W^{\mathrm{Muon}})^\top(U1) \leq\lVert V\rVert_{\mathrm{op}}^2\lVert U1\rVert^2 = O_p(N\cdot Nd). \tag*{(47)} i ∑ (( U 1 ) ⊤ W Muon v i ) 2 = ( U 1 ) ⊤ W Muon V V ⊤ ( W Muon ) ⊤ ( U 1 ) ≤ ∥ V ∥ op 2 ∥ U 1 ∥ 2 = O p ( N ⋅ N d ) . (47)
For the second, Cauchy–Schwarz and ∥ W M u o n v i ∥ = ∥ v i ∥ \lVert W^{\mathrm{Muon}}v_i\rVert= \lVert v_i\rVert ∥ W Muon v i ∥ = ∥ v i ∥ give ( u i ⊤ W M u o n v i ) 2 ≤ ∥ u i ∥ 2 ∥ v i ∥ 2 (u_i^\top W^{\mathrm{Muon}}v_i)^2 \leq\lVert u_i\rVert^2\lVert v_i\rVert^2 ( u i ⊤ W Muon v i ) 2 ≤ ∥ u i ∥ 2 ∥ v i ∥ 2 , and x 2 ≤ 2 ( x − d ) 2 + 2 d 2 x^2 \leq2(x-d)^2 + 2d^2 x 2 ≤ 2 ( x − d ) 2 + 2 d 2 with Theorem F.3(ii) gives ∑ i ∥ u i ∥ 2 ∥ v i ∥ 2 ≤ 2 ∑ i ( ∥ u i ∥ ∥ v i ∥ − d ) 2 + 2 N d 2 = N ⋅ O p ( d ) + 2 N d 2 = O p ( N d 2 ) \sum_i\lVert u_i\rVert^2\lVert v_i\rVert^2 \leq2\sum_i(\lVert u_i\rVert\lVert v_i\rVert-d)^2+2Nd^2 = N\cdot O_p(d)+2Nd^2 = O_p(Nd^2) ∑ i ∥ u i ∥ 2 ∥ v i ∥ 2 ≤ 2 ∑ i (∥ u i ∥ ∥ v i ∥ − d ) 2 + 2 N d 2 = N ⋅ O p ( d ) + 2 N d 2 = O p ( N d 2 ) . □
Lemma F.21. mean i ( mean j ≠ i u j ⊤ W M u o n v i ) 2 = O p ( d / N ) \operatorname{mean}_i\left(\operatorname{mean}_{j\ne i}u_j^\top W^{\mathrm{Muon}}v_i\right)^2 = O_p(d/N) mean i ( mean j = i u j ⊤ W Muon v i ) 2 = O p ( d / N ) , hence ∣ mean i mean j ≠ i u j ⊤ W M u o n v i ∣ = O p ( d / N ) \left|\operatorname{mean}_i\operatorname{mean}_{j\ne i}u_j^\top W^{\mathrm{Muon}}v_i\right| = O_p(\sqrt{d/N}) mean i mean j = i u j ⊤ W Muon v i = O p ( d / N ) .
Proof. Square Theorem F.8 with W = W M u o n W=W^{\mathrm{Muon}} W = W Muon , use ( x − y ) 2 ≤ 2 x 2 + 2 y 2 (x-y)^2\leq2x^2+2y^2 ( x − y ) 2 ≤ 2 x 2 + 2 y 2 , sum over i i i and apply Theorem F.20 :
∑ i ( ( N − 1 ) mean j ≠ i u j ⊤ W M u o n v i ) 2 ≤ 2 O p ( N 2 d ) + 2 O p ( N d 2 ) = O p ( N 2 d ) , (48) \sum_i\left((N-1)\operatorname{mean}_{j\ne i}u_j^\top W^{\mathrm{Muon}}v_i\right)^2 \leq2O_p(N^2d)+2O_p(Nd^2)=O_p(N^2d), \tag*{(48)} i ∑ ( ( N − 1 ) mean j = i u j ⊤ W Muon v i ) 2 ≤ 2 O p ( N 2 d ) + 2 O p ( N d 2 ) = O p ( N 2 d ) , (48)
using d ≤ N d\leq N d ≤ N . Dividing by N ( N − 1 ) 2 ≥ 1 4 N 3 N(N-1)^2\geq\frac14N^3 N ( N − 1 ) 2 ≥ 4 1 N 3 gives the first claim, and Cauchy–Schwarz the second. □
Lemma F.22 (h ˉ \bar h h ˉ for Muon). The mean margin of the Muon iterate is
h ˉ M u o n = ∥ W G D ∥ ∗ N ( 1 + O p ( d − 1 ) ) = d Φ ( γ ) γ ( 1 + o p ( 1 ) ) . \bar h^{\mathrm{Muon}}=\frac{\lVert W^{\mathrm{GD}}\rVert_*}{N}(1+O_p(d^{-1}))=\frac{d\Phi(\gamma)}{\sqrt{\gamma}}(1+o_p(1)). h ˉ Muon = N ∥ W GD ∥ ∗ ( 1 + O p ( d − 1 )) = γ d Φ ( γ ) ( 1 + o p ( 1 )) .
Proof. By Theorem F.19 and Theorem F.21 , h ˉ M u o n = ∥ W G D ∥ ∗ N ( 1 + O p ( ( N d ) − 1 / 2 ) ) − O p ( d / N ) \bar h^{\mathrm{Muon}}=\frac{\lVert W^{\mathrm{GD}}\rVert_*}{N}(1+O_p((Nd)^{-1/2}))-O_p(\sqrt{d/N}) h ˉ Muon = N ∥ W GD ∥ ∗ ( 1 + O p (( N d ) − 1/2 )) − O p ( d / N ) . By Lemma F.4 (iii), with probability tending to one ∥ W G D ∥ ∗ N ≥ d 3 / 2 8 N \frac{\lVert W^{\mathrm{GD}}\rVert_*}{N}\geq\frac{d^{3/2}}{8\sqrt N} N ∥ W GD ∥ ∗ ≥ 8 N d 3/2 , so
O p ( d / N ) = ∥ W G D ∥ ∗ N ⋅ O p ( d N 8 N d 3 / 2 ) = ∥ W G D ∥ ∗ N ⋅ O p ( d − 1 ) , (49) O_p(\sqrt{d/N})=\frac{\lVert W^{\mathrm{GD}}\rVert_*}{N}\cdot O_p\left(\sqrt{\frac dN}\frac{8\sqrt N}{d^{3/2}}\right)=\frac{\lVert W^{\mathrm{GD}}\rVert_*}{N}\cdot O_p(d^{-1}), \tag*{(49)} O p ( d / N ) = N ∥ W GD ∥ ∗ ⋅ O p ( N d d 3/2 8 N ) = N ∥ W GD ∥ ∗ ⋅ O p ( d − 1 ) , (49)
and ( N d ) − 1 / 2 ≤ d − 1 (Nd)^{-1/2}\leq d^{-1} ( N d ) − 1/2 ≤ d − 1 , which gives the first equality. For the second, Lemma F.4 (iv) gives ∥ W G D ∥ ∗ N d = Φ ( γ ) γ + o p ( 1 ) \frac{\lVert W^{\mathrm{GD}}\rVert_*}{Nd}=\frac{\Phi(\gamma)}{\sqrt{\gamma}}+o_p(1) N d ∥ W GD ∥ ∗ = γ Φ ( γ ) + o p ( 1 ) , and Φ ( γ ) γ ≥ 1 8 γ > 0 \frac{\Phi(\gamma)}{\sqrt{\gamma}}\geq\frac{1}{8\sqrt{\gamma}}>0 γ Φ ( γ ) ≥ 8 γ 1 > 0 by Lemma F.4 (iii), so the additive o p ( 1 ) o_p(1) o p ( 1 ) may be written multiplicatively. □
We now turn to κ ˉ M u o n \bar\kappa^{\mathrm{Muon}} κ ˉ Muon . The orthogonality of W M u o n W^{\mathrm{Muon}} W Muon turns the second moment into three terms.
Lemma F.23 (Three-term identity). Almost surely
1 N ( N − 1 ) ∑ i = 1 N ∑ j ≠ i ( u j ⊤ W M u o n v i ) 2 = N N − 1 mean i ∥ v i ∥ 2 − mean i ( u i ⊤ W M u o n v i ) 2 N − 1 + tr [ ( U U ⊤ − N I d ) W M u o n V V ⊤ ( W M u o n ) ⊤ ] N ( N − 1 ) . (50) \frac{1}{N(N-1)}\sum_{i=1}^{N}\sum_{j\ne i}(u_j^\top W^{\mathrm{Muon}}v_i)^2=\frac{N}{N-1}\operatorname{mean}_i\lVert v_i\rVert^2-\frac{\operatorname{mean}_i(u_i^\top W^{\mathrm{Muon}}v_i)^2}{N-1}+\frac{\operatorname{tr}\left[(UU^\top-NI_d)W^{\mathrm{Muon}}VV^\top(W^{\mathrm{Muon}})^\top\right]}{N(N-1)}. \tag*{(50)} N ( N − 1 ) 1 i = 1 ∑ N j = i ∑ ( u j ⊤ W Muon v i ) 2 = N − 1 N mean i ∥ v i ∥ 2 − N − 1 mean i ( u i ⊤ W Muon v i ) 2 + N ( N − 1 ) tr [ ( U U ⊤ − N I d ) W Muon V V ⊤ ( W Muon ) ⊤ ] . (50)
Proof. Since ( U ⊤ x ) j = u j ⊤ x (U^\top x)_j=u_j^\top x ( U ⊤ x ) j = u j ⊤ x and ∥ W M u o n v i ∥ = ∥ v i ∥ \lVert W^{\mathrm{Muon}}v_i\rVert=\lVert v_i\rVert ∥ W Muon v i ∥ = ∥ v i ∥ ,
∑ j ≠ i ( u j ⊤ W M u o n v i ) 2 = ∥ U ⊤ W M u o n v i ∥ 2 − ( u i ⊤ W M u o n v i ) 2 , \sum_{j\ne i}(u_j^\top W^{\mathrm{Muon}}v_i)^2=\lVert U^\top W^{\mathrm{Muon}}v_i\rVert^2-(u_i^\top W^{\mathrm{Muon}}v_i)^2, j = i ∑ ( u j ⊤ W Muon v i ) 2 = ∥ U ⊤ W Muon v i ∥ 2 − ( u i ⊤ W Muon v i ) 2 ,
∥ U ⊤ W M u o n v i ∥ 2 = N ∥ v i ∥ 2 + ( W M u o n v i ) ⊤ ( U U ⊤ − N I d ) ( W M u o n v i ) . (51) \lVert U^\top W^{\mathrm{Muon}}v_i\rVert^2=N\lVert v_i\rVert^2+(W^{\mathrm{Muon}}v_i)^\top(UU^\top-NI_d)(W^{\mathrm{Muon}}v_i). \tag*{(51)} ∥ U ⊤ W Muon v i ∥ 2 = N ∥ v i ∥ 2 + ( W Muon v i ) ⊤ ( U U ⊤ − N I d ) ( W Muon v i ) . (51)
Summing over i i i and using x ⊤ Z x = tr ( Z x x ⊤ ) x^\top Zx=\operatorname{tr}(Zxx^\top) x ⊤ Z x = tr ( Z x x ⊤ ) with ∑ i v i v i ⊤ = V V ⊤ \sum_i v_iv_i^\top=VV^\top ∑ i v i v i ⊤ = V V ⊤ turns the middle term into the stated trace; dividing by N ( N − 1 ) N(N-1) N ( N − 1 ) and using ∑ i ∥ v i ∥ 2 = N mean i ∥ v i ∥ 2 \sum_i\lVert v_i\rVert^2=N\operatorname{mean}_i\lVert v_i\rVert^2 ∑ i ∥ v i ∥ 2 = N mean i ∥ v i ∥ 2 gives the claim. □
Lemma F.24 (Norm term). N N − 1 mean i ∥ v i ∥ 2 = d ( 1 + O ( N − 1 ) + O p ( ( N d ) − 1 / 2 ) ) \frac{N}{N-1}\operatorname{mean}_i\lVert v_i\rVert^2=d(1+O(N^{-1})+O_p((Nd)^{-1/2})) N − 1 N mean i ∥ v i ∥ 2 = d ( 1 + O ( N − 1 ) + O p (( N d ) − 1/2 )) .
Proof. Theorem F.3(i) gives mean i ∥ v i ∥ 2 = d ( 1 + O p ( ( N d ) − 1 / 2 ) ) \operatorname{mean}_i\lVert v_i\rVert^2=d(1+O_p((Nd)^{-1/2})) mean i ∥ v i ∥ 2 = d ( 1 + O p (( N d ) − 1/2 )) , and N N − 1 = 1 + 1 N − 1 = 1 + O ( N − 1 ) \frac{N}{N-1}=1+\frac{1}{N-1}=1+O(N^{-1}) N − 1 N = 1 + N − 1 1 = 1 + O ( N − 1 ) . □
The trace term requires two auxiliary facts.
Lemma F.25 (Decomposition of U U ⊤ UU^\top U U ⊤ ). Almost surely V V ⊤ − 1 N ( V 1 ) ( V 1 ) ⊤ VV^\top-\frac{1}{N}(V1)(V1)^\top V V ⊤ − N 1 ( V 1 ) ( V 1 ) ⊤ is invertible and there is Γ ∈ R ( N − 1 − d ) × d \Gamma\in\mathbb{R}^{(N-1-d)\times d} Γ ∈ R ( N − 1 − d ) × d with
U U ⊤ − 1 N ( U 1 ) ( U 1 ) ⊤ = W G D ( V V ⊤ − 1 N ( V 1 ) ( V 1 ) ⊤ ) − 1 ( W G D ) ⊤ + Γ ⊤ Γ , (52) UU^\top-\frac{1}{N}(U1)(U1)^\top=W^{\mathrm{GD}}\left(VV^\top-\frac{1}{N}(V1)(V1)^\top\right)^{-1}(W^{\mathrm{GD}})^\top+\Gamma^\top\Gamma, \tag*{(52)} U U ⊤ − N 1 ( U 1 ) ( U 1 ) ⊤ = W GD ( V V ⊤ − N 1 ( V 1 ) ( V 1 ) ⊤ ) − 1 ( W GD ) ⊤ + Γ ⊤ Γ , (52) where, conditionally on V V V , the entries of Γ \Gamma Γ are i.i.d. N ( 0 , 1 ) \mathcal{N}(0,1) N ( 0 , 1 ) and independent of W G D W^{\mathrm{GD}} W GD . Consequently, for every symmetric Y ∈ R d × d Y\in\mathbb{R}^{d\times d} Y ∈ R d × d that is a function of ( V , W G D ) (V,W^{\mathrm{GD}}) ( V , W GD ) ,
E [ tr ( Γ ⊤ Γ Y ) ∣ V , W G D ] = ( N − 1 − d ) tr Y , (53) \mathbb{E}\left[\operatorname{tr}(\Gamma^\top\Gamma Y)\mid V,W^{\mathrm{GD}}\right]=(N-1-d)\operatorname{tr}Y, \tag*{(53)} E [ tr ( Γ ⊤ Γ Y ) ∣ V , W GD ] = ( N − 1 − d ) tr Y , (53) var [ tr ( Γ ⊤ Γ Y ) ∣ V , W G D ] = 2 ( N − 1 − d ) ∥ Y ∥ F 2 . \operatorname{var}\left[\operatorname{tr}(\Gamma^\top\Gamma Y)\mid V,W^{\mathrm{GD}}\right]=2(N-1-d)\lVert Y\rVert_F^2. var [ tr ( Γ ⊤ Γ Y ) ∣ V , W GD ] = 2 ( N − 1 − d ) ∥ Y ∥ F 2 . Proof. Invertibility. V V ⊤ − 1 N ( V 1 ) ( V 1 ) ⊤ = V ( I N − 11 ⊤ N ) V ⊤ ⪰ 0 VV^\top-\frac{1}{N}(V1)(V1)^\top=V\left(I_N-\frac{11^\top}{N}\right)V^\top\succeq0 V V ⊤ − N 1 ( V 1 ) ( V 1 ) ⊤ = V ( I N − N 1 1 ⊤ ) V ⊤ ⪰ 0 and its determinant is a polynomial in V V V which is nonzero at V = [ I d − ∑ k ≤ d e k 0 ] V=[I_d\ -\sum_{k\le d}e_k\ 0] V = [ I d − ∑ k ≤ d e k 0 ] , where it equals det ( I d + ( ∑ k ≤ d e k ) ( ∑ k ≤ d e k ) ⊤ ) = 1 + d ≠ 0 \det(I_d+(\sum_{k\le d}e_k)(\sum_{k\le d}e_k)^\top)=1+d\ne0 det ( I d + ( ∑ k ≤ d e k ) ( ∑ k ≤ d e k ) ⊤ ) = 1 + d = 0 ; so it is nonzero almost surely.
Basis. If c 0 1 ⊤ + c ⊤ V = 0 c_0 1^\top+c^\top V=0 c 0 1 ⊤ + c ⊤ V = 0 for some c 0 ∈ R c_0\in\mathbb{R} c 0 ∈ R and c ∈ R d c\in\mathbb{R}^d c ∈ R d , then multiplying on the right by ( I N − 11 ⊤ N ) V ⊤ c \left(I_N-\frac{11^\top}{N}\right)V^\top c ( I N − N 1 1 ⊤ ) V ⊤ c , and using 1 ⊤ ( I N − 11 ⊤ N ) = 0 1^\top\left(I_N-\frac{11^\top}{N}\right)=0 1 ⊤ ( I N − N 1 1 ⊤ ) = 0 , gives c ⊤ ( V V ⊤ − 1 N ( V 1 ) ( V 1 ) ⊤ ) c = 0 c^\top\left(VV^\top-\frac{1}{N}(V1)(V1)^\top\right)c=0 c ⊤ ( V V ⊤ − N 1 ( V 1 ) ( V 1 ) ⊤ ) c = 0 , so c = 0 c=0 c = 0 and then c 0 = 0 c_0=0 c 0 = 0 . Hence { x ∈ R N : 1 ⊤ x = 0 , V x = 0 } \{x\in\mathbb{R}^N:1^\top x=0,Vx=0\} { x ∈ R N : 1 ⊤ x = 0 , V x = 0 } has dimension N − 1 − d N-1-d N − 1 − d almost surely; stack an orthonormal basis of it as the columns of R ∈ R N × ( N − 1 − d ) R\in\mathbb{R}^{N\times(N-1-d)} R ∈ R N × ( N − 1 − d ) , so that R ⊤ R = I R^\top R=I R ⊤ R = I , 1 ⊤ R = 0 1^\top R=0 1 ⊤ R = 0 and V R = 0 VR=0 V R = 0 , and set Γ : = R ⊤ U ⊤ \Gamma:=R^\top U^\top Γ := R ⊤ U ⊤ .
Projection identity. Because R N = span 1 ⊕ col ( ( I N − 11 ⊤ N ) V ⊤ ) ⊕ col R \mathbb{R}^N=\operatorname{span}1\oplus\operatorname{col}\left(\left(I_N-\frac{11^\top}{N}\right)V^\top\right)\oplus\operatorname{col}R R N = span 1 ⊕ col ( ( I N − N 1 1 ⊤ ) V ⊤ ) ⊕ col R with the three subspaces pairwise orthogonal and of dimensions 1 + d + ( N − 1 − d ) = N 1+d+(N-1-d)=N 1 + d + ( N − 1 − d ) = N ,
I N − 11 ⊤ N = ( I N − 11 ⊤ N ) V ⊤ ( V V ⊤ − 1 N ( V 1 ) ( V 1 ) ⊤ ) − 1 V ( I N − 11 ⊤ N ) + R R ⊤ . (54) I_N-\frac{11^\top}{N}=\left(I_N-\frac{11^\top}{N}\right)V^\top\left(VV^\top-\frac{1}{N}(V1)(V1)^\top\right)^{-1}V\left(I_N-\frac{11^\top}{N}\right)+RR^\top. \tag*{(54)} I N − N 1 1 ⊤ = ( I N − N 1 1 ⊤ ) V ⊤ ( V V ⊤ − N 1 ( V 1 ) ( V 1 ) ⊤ ) − 1 V ( I N − N 1 1 ⊤ ) + R R ⊤ . (54) Both sides are symmetric, so it suffices to check them on the three subspaces. Write P : = I N − 11 ⊤ N P:=I_N-\frac{11^\top}{N} P := I N − N 1 1 ⊤ and G : = V V ⊤ − 1 N ( V 1 ) ( V 1 ) ⊤ G:=VV^\top-\frac{1}{N}(V1)(V1)^\top G := V V ⊤ − N 1 ( V 1 ) ( V 1 ) ⊤ , and use P 2 = P P^2=P P 2 = P , P V P ⊤ = G PVP^\top=G P V P ⊤ = G and P R = R − 1 N 1 ( 1 ⊤ R ) = R PR=R-\frac{1}{N}1(1^\top R)=R P R = R − N 1 1 ( 1 ⊤ R ) = R . For x = 1 x=1 x = 1 : P x = 0 Px=0 P x = 0 , V P x = 0 VPx=0 V P x = 0 and R ⊤ x = 0 R^\top x=0 R ⊤ x = 0 , so both sides vanish. For x = P V ⊤ y x=PV^\top y x = P V ⊤ y : P x = x Px=x P x = x , while V P x = P V P ⊤ y = G y VPx=PVP^\top y=Gy V P x = P V P ⊤ y = G y , so the first term of the right-hand side is P V ⊤ G − 1 G y = x PV^\top G^{-1}Gy=x P V ⊤ G − 1 G y = x , and R ⊤ x = ( V R ) ⊤ y = 0 R^\top x=(VR)^\top y=0 R ⊤ x = ( V R ) ⊤ y = 0 , so the second vanishes. For x = R z x=Rz x = R z : P x = R z = x Px=Rz=x P x = R z = x , while V P x = V R z = 0 VPx=VRz=0 V P x = V R z = 0 kills the first term, and R R ⊤ R z = R z = x RR^\top Rz=Rz=x R R ⊤ R z = R z = x . In all three cases both sides agree. Multiplying Equation (54) on the left by U U U and on the right by U ⊤ U^\top U ⊤ , and using U ( I N − 11 ⊤ N ) U ⊤ = U U ⊤ − 1 N ( U 1 ) ( U 1 ) ⊤ U\left(I_N-\frac{11^\top}{N}\right)U^\top=UU^\top-\frac{1}{N}(U1)(U1)^\top U ( I N − N 1 1 ⊤ ) U ⊤ = U U ⊤ − N 1 ( U 1 ) ( U 1 ) ⊤ and U ( I N − 11 ⊤ N ) V ⊤ = W G D U\left(I_N-\frac{11^\top}{N}\right)V^\top=W^{\mathrm{GD}} U ( I N − N 1 1 ⊤ ) V ⊤ = W GD , gives the decomposition with U R R ⊤ U ⊤ = Γ ⊤ Γ URR^\top U^\top=\Gamma^\top\Gamma U R R ⊤ U ⊤ = Γ ⊤ Γ .
Distribution. Conditionally on V V V , the rows of U U U are i.i.d. N ( 0 , I N ) \mathcal{N}(0,I_N) N ( 0 , I N ) ; the a a a -th column of Γ \Gamma Γ is R ⊤ U ⊤ e a ∼ N ( 0 , R ⊤ R ) = N ( 0 , I N − 1 − d ) R^\top U^\top e_a\sim\mathcal{N}(0,R^\top R)=\mathcal{N}(0,I_{N-1-d}) R ⊤ U ⊤ e a ∼ N ( 0 , R ⊤ R ) = N ( 0 , I N − 1 − d ) with independent columns, so the entries of Γ \Gamma Γ are i.i.d. standard Gaussian. The a a a -th row of W G D W^{\mathrm{GD}} W GD is ( V ( I N − 11 ⊤ N ) U ⊤ e a ) ⊤ \left(V\left(I_N-\frac{11^\top}{N}\right)U^\top e_a\right)^\top ( V ( I N − N 1 1 ⊤ ) U ⊤ e a ) ⊤ and Cov ( V ( I N − 11 ⊤ N ) U ⊤ e a , R ⊤ U ⊤ e a ) = V ( I N − 11 ⊤ N ) R = 0 \operatorname{Cov}\left(V\left(I_N-\frac{11^\top}{N}\right)U^\top e_a,R^\top U^\top e_a\right)=V\left(I_N-\frac{11^\top}{N}\right)R=0 Cov ( V ( I N − N 1 1 ⊤ ) U ⊤ e a , R ⊤ U ⊤ e a ) = V ( I N − N 1 1 ⊤ ) R = 0 ; being jointly Gaussian conditionally on V V V , Γ \Gamma Γ and W G D W^{\mathrm{GD}} W GD are conditionally independent.
Moments. The rows of Γ \Gamma Γ are i.i.d. N ( 0 , I d ) \mathcal{N}(0,I_d) N ( 0 , I d ) and tr ( Γ ⊤ Γ Y ) = ∑ m ( Γ ⊤ e m ) ⊤ Y ( Γ ⊤ e m ) \operatorname{tr}(\Gamma^\top\Gamma Y)=\sum_m(\Gamma^\top e_m)^\top Y(\Gamma^\top e_m) tr ( Γ ⊤ Γ Y ) = ∑ m ( Γ ⊤ e m ) ⊤ Y ( Γ ⊤ e m ) ; for g ∼ N ( 0 , I d ) g\sim\mathcal{N}(0,I_d) g ∼ N ( 0 , I d ) and fixed symmetric Y Y Y , E [ g ⊤ Y g ] = tr Y \mathbb{E}[g^\top Yg]=\operatorname{tr}Y E [ g ⊤ Y g ] = tr Y and var [ g ⊤ Y g ] = 2 ∥ Y ∥ F 2 \operatorname{var}[g^\top Yg]=2\lVert Y\rVert_F^2 var [ g ⊤ Y g ] = 2 ∥ Y ∥ F 2 ; summing over the N − 1 − d N-1-d N − 1 − d rows gives the claim. □ \square □
Lemma F.26 (Trace of a commutator). Almost surely , with S : = 1 N ( V V ⊤ − 1 N ( V 1 ) ( V 1 ) ⊤ ) S:=\frac{1}{N}\left(VV^\top-\frac{1}{N}(V1)(V1)^\top\right) S := N 1 ( V V ⊤ − N 1 ( V 1 ) ( V 1 ) ⊤ ) ,
∣ tr [ ( W G D ( N S ) − 1 ( W G D ) ⊤ W M u o n ( N S ) ( W M u o n ) ⊤ ] − ∥ W G D ∥ F 2 ∣ ≤ 2 ∥ S − I d ∥ o p 2 λ min ( S ) ∥ W G D ∥ F 2 . (55) \left|\operatorname{tr}\left[(W^{\mathrm{GD}}(NS)^{-1}(W^{\mathrm{GD}})^\top W^{\mathrm{Muon}}(NS)(W^{\mathrm{Muon}})^\top\right]-\lVert W^{\mathrm{GD}}\rVert_F^2\right|\le\frac{2\lVert S-I_d\rVert_{\mathrm{op}}^2}{\lambda_{\min}(S)}\lVert W^{\mathrm{GD}}\rVert_F^2. \tag*{(55)} tr [ ( W GD ( N S ) − 1 ( W GD ) ⊤ W Muon ( N S ) ( W Muon ) ⊤ ] − ∥ W GD ∥ F 2 ≤ λ m i n ( S ) 2 ∥ S − I d ∥ op 2 ∥ W GD ∥ F 2 . (55) Proof. A matrix inequality. For symmetric S ≻ 0 S\succ0 S ≻ 0 and symmetric T T T ,
∣ tr [ S − 1 T S T ] − ∥ T ∥ F 2 ∣ ≤ 2 ∥ S − I d ∥ o p ∥ T ∥ F 2 λ min ( S ) . (56) \left|\operatorname{tr}[S^{-1}TST]-\lVert T\rVert_F^2\right|\le\frac{2\lVert S-I_d\rVert_{\mathrm{op}}\lVert T\rVert_F^2}{\lambda_{\min}(S)}. \tag*{(56)} tr [ S − 1 T S T ] − ∥ T ∥ F 2 ≤ λ m i n ( S ) 2 ∥ S − I d ∥ op ∥ T ∥ F 2 . (56) Indeed, cyclicity gives tr [ S − 1 T S T ] = ∥ T ∥ F 2 \operatorname{tr}[S^{-1}TST]=\lVert T\rVert_F^2 tr [ S − 1 T S T ] = ∥ T ∥ F 2 and tr [ ( T S − S T ) T ] = 0 \operatorname{tr}[(TS-ST)T]=0 tr [( T S − S T ) T ] = 0 , so with S − 1 = I d − S − 1 ( S − I d ) S^{-1}=I_d-S^{-1}(S-I_d) S − 1 = I d − S − 1 ( S − I d ) ,
tr [ S − 1 T S T ] − ∥ T ∥ F 2 = tr [ S − 1 ( T S − S T ) T ] = − tr [ S − 1 ( S − I d ) ( T S − S T ) T ] , (57) \operatorname{tr}[S^{-1}TST]-\lVert T\rVert_F^2=\operatorname{tr}\left[S^{-1}(TS-ST)T\right]=-\operatorname{tr}\left[S^{-1}(S-I_d)(TS-ST)T\right], \tag*{(57)} tr [ S − 1 T S T ] − ∥ T ∥ F 2 = tr [ S − 1 ( T S − S T ) T ] = − tr [ S − 1 ( S − I d ) ( T S − S T ) T ] , (57) and ∣ tr ( A B C ) ∣ ≤ ∥ A ∥ o p ∥ B ∥ F ∥ C ∥ F \lvert\operatorname{tr}(ABC)\rvert\le\lVert A\rVert_{\mathrm{op}}\lVert B\rVert_F\lVert C\rVert_F ∣ tr ( A B C )∣ ≤ ∥ A ∥ op ∥ B ∥ F ∥ C ∥ F , ∥ S − 1 ( S − I d ) ∥ o p ≤ ∥ S − I d ∥ o p / λ min ( S ) \lVert S^{-1}(S-I_d)\rVert_{\mathrm{op}}\le\lVert S-I_d\rVert_{\mathrm{op}}/\lambda_{\min}(S) ∥ S − 1 ( S − I d ) ∥ op ≤ ∥ S − I d ∥ op / λ m i n ( S ) and ∥ T S − S T ∥ F ≤ 2 ∥ S − I d ∥ o p ∥ T ∥ F \lVert TS-ST\rVert_F\le2\lVert S-I_d\rVert_{\mathrm{op}}\lVert T\rVert_F ∥ T S − S T ∥ F ≤ 2 ∥ S − I d ∥ op ∥ T ∥ F give Equation (56) .
Polar decomposition. With W G D = P Σ Q ⊤ W^{\mathrm{GD}}=P\Sigma Q^{\top} W GD = P Σ Q ⊤ and W M u o n = P Q ⊤ W^{\mathrm{Muon}}=PQ^{\top} W Muon = P Q ⊤ we have ( W G D ) ⊤ W M u o n = Q Σ Q ⊤ = ( W M u o n ) ⊤ W G D (W^{\mathrm{GD}})^{\top}W^{\mathrm{Muon}}=Q\Sigma Q^{\top}=(W^{\mathrm{Muon}})^{\top}W^{\mathrm{GD}} ( W GD ) ⊤ W Muon = Q Σ Q ⊤ = ( W Muon ) ⊤ W GD and ∥ Q Σ Q ⊤ ∥ F = ∥ Σ ∥ F = ∥ W G D ∥ F \lVert Q\Sigma Q^{\top}\rVert_{F}=\lVert\Sigma\rVert_{F}=\lVert W^{\mathrm{GD}}\rVert_{F} ∥ Q Σ Q ⊤ ∥ F = ∥ Σ ∥ F = ∥ W GD ∥ F .
Substitution. Moving the leading W G D W^{\mathrm{GD}} W GD to the end by cyclicity,
tr [ W G D ( N S ) − 1 ( W G D ) ⊤ W M u o n ( N S ) ( W M u o n ) ⊤ ] = tr [ ( N S ) − 1 ( Q Σ Q ⊤ ) ( N S ) ( Q Σ Q ⊤ ) ] , (58) \operatorname{tr}\left[W^{\mathrm{GD}}(NS)^{-1}(W^{\mathrm{GD}})^{\top}W^{\mathrm{Muon}}(NS)(W^{\mathrm{Muon}})^{\top}\right]=\operatorname{tr}\left[(NS)^{-1}(Q\Sigma Q^{\top})(NS)(Q\Sigma Q^{\top})\right], \tag*{(58)} tr [ W GD ( N S ) − 1 ( W GD ) ⊤ W Muon ( N S ) ( W Muon ) ⊤ ] = tr [ ( N S ) − 1 ( Q Σ Q ⊤ ) ( N S ) ( Q Σ Q ⊤ ) ] , (58)
and applying Equation (56) with T = Q Σ Q ⊤ T=Q\Sigma Q^{\top} T = Q Σ Q ⊤ (the factors N N N cancel) gives the claim.
Lemma F.27 (Trace term). If γ ≥ 16 \gamma\ge16 γ ≥ 16 , then with probability tending to one
1 N ( N − 1 ) d ∣ tr [ ( U U ⊤ − N I d ) W M u o n V V ⊤ ( W M u o n ) ⊤ ] ∣ ≤ 145 γ − 2 . (59) \frac{1}{N(N-1)d}\left|\operatorname{tr}\left[(UU^{\top}-NI_{d})W^{\mathrm{Muon}}VV^{\top}(W^{\mathrm{Muon}})^{\top}\right]\right|\le145\gamma^{-2}. \tag*{(59)} N ( N − 1 ) d 1 tr [ ( U U ⊤ − N I d ) W Muon V V ⊤ ( W Muon ) ⊤ ] ≤ 145 γ − 2 . (59)
The assumption γ ≥ 16 \gamma\ge16 γ ≥ 16 is tailored to this estimate: through Theorem F.3(v) it controls how far 1 N V V ⊤ \frac{1}{N}VV^{\top} N 1 V V ⊤ is from the identity, so that the fluctuation of U U ⊤ UU^{\top} U U ⊤ around N I d NI_{d} N I d , seen through the Muon iterate, contributes only O ( γ − 2 ) O(\gamma^{-2}) O ( γ − 2 ) to the second moment.
Proof. Write B : = W M u o n ( V V ⊤ − 1 N ( V 1 ) ( V 1 ) ⊤ ) ( W M u o n ) ⊤ B:=W^{\mathrm{Muon}}\left(VV^{\top}-\frac{1}{N}(V1)(V1)^{\top}\right)(W^{\mathrm{Muon}})^{\top} B := W Muon ( V V ⊤ − N 1 ( V 1 ) ( V 1 ) ⊤ ) ( W Muon ) ⊤ . By Theorem F.2 orthogonal conjugation preserves ∥ ⋅ ∥ F \lVert\cdot\rVert_{F} ∥ ⋅ ∥ F , ∥ ⋅ ∥ o p \lVert\cdot\rVert_{\mathrm{op}} ∥ ⋅ ∥ op and the trace; moreover V V ⊤ − 1 N ( V 1 ) ( V 1 ) ⊤ = V ( I N − 1 N 11 ⊤ ) V ⊤ VV^{\top}-\frac{1}{N}(V1)(V1)^{\top}=V\left(I_{N}-\frac{1}{N}11^{\top}\right)V^{\top} V V ⊤ − N 1 ( V 1 ) ( V 1 ) ⊤ = V ( I N − N 1 1 1 ⊤ ) V ⊤ satisfies 0 ⪯ V V ⊤ − 1 N ( V 1 ) ( V 1 ) ⊤ ⪯ V V ⊤ 0\preceq VV^{\top}-\frac{1}{N}(V1)(V1)^{\top}\preceq VV^{\top} 0 ⪯ V V ⊤ − N 1 ( V 1 ) ( V 1 ) ⊤ ⪯ V V ⊤ . Hence, by Theorem F.3(iii),(iv),(vi),
∥ B ∥ F ≤ ∥ V V ⊤ ∥ F + ∥ V 1 ∥ 2 N = O p ( N d ) , ∥ B ∥ o p ≤ ∥ V ∥ o p 2 = O p ( N ) , tr B = ∑ k ∥ v k ∥ 2 − ∥ V 1 ∥ 2 N = O p ( N d ) . (60) \begin{aligned}
\lVert B\rVert_{F}&\le\lVert VV^{\top}\rVert_{F}+\frac{\lVert V1\rVert^{2}}{N}=O_{p}(N\sqrt{d}),\\
\lVert B\rVert_{\mathrm{op}}&\le\lVert V\rVert_{\mathrm{op}}^{2}=O_{p}(N),\\
\operatorname{tr}B&=\sum_{k}\lVert v_{k}\rVert^{2}-\frac{\lVert V1\rVert^{2}}{N}=O_{p}(Nd). \tag*{(60)}
\end{aligned} ∥ B ∥ F ∥ B ∥ op tr B ≤ ∥ V V ⊤ ∥ F + N ∥ V 1 ∥ 2 = O p ( N d ) , ≤ ∥ V ∥ op 2 = O p ( N ) , = k ∑ ∥ v k ∥ 2 − N ∥ V 1 ∥ 2 = O p ( N d ) . (60)
Rank-one replacement. V V ⊤ − ( V V ⊤ − 1 N ( V 1 ) ( V 1 ) ⊤ ) = 1 N ( V 1 ) ( V 1 ) ⊤ VV^{\top}-\left(VV^{\top}-\frac{1}{N}(V1)(V1)^{\top}\right)=\frac{1}{N}(V1)(V1)^{\top} V V ⊤ − ( V V ⊤ − N 1 ( V 1 ) ( V 1 ) ⊤ ) = N 1 ( V 1 ) ( V 1 ) ⊤ , so by Theorem F.3(iii),(v),
∣ tr [ ( U U ⊤ − N I d ) W M u o n V V ⊤ ( W M u o n ) ⊤ ] − tr [ ( U U ⊤ − N I d ) B ] ∣ ≤ 1 N ∥ U U ⊤ − N I d ∥ o p ∥ V 1 ∥ 2 = N ( N − 1 ) d ⋅ O p ( d N − 3 / 2 ) . (61) \left|\operatorname{tr}\left[(UU^{\top}-NI_{d})W^{\mathrm{Muon}}VV^{\top}(W^{\mathrm{Muon}})^{\top}\right]-\operatorname{tr}\left[(UU^{\top}-NI_{d})B\right]\right|
\le\frac{1}{N}\lVert UU^{\top}-NI_{d}\rVert_{\mathrm{op}}\lVert V1\rVert^{2}
=N(N-1)d\cdot O_{p}\left(\sqrt{dN^{-3/2}}\right). \tag*{(61)} tr [ ( U U ⊤ − N I d ) W Muon V V ⊤ ( W Muon ) ⊤ ] − tr [ ( U U ⊤ − N I d ) B ] ≤ N 1 ∥ U U ⊤ − N I d ∥ op ∥ V 1 ∥ 2 = N ( N − 1 ) d ⋅ O p ( d N − 3/2 ) . (61)
Splitting. By Theorem F.25 and d + ( N − 1 − d ) + 1 = N d+(N-1-d)+1=N d + ( N − 1 − d ) + 1 = N ,
U U ⊤ − N I d = [ W G D ( N S ) − 1 ( W G D ) ⊤ − d I d ] + [ Γ ⊤ Γ − ( N − 1 − d ) I d ] + [ 1 N ( U 1 ) ( U 1 ) ⊤ − I d ] , (62) UU^{\top}-NI_{d}=\left[W^{\mathrm{GD}}(NS)^{-1}(W^{\mathrm{GD}})^{\top}-dI_{d}\right]+\left[\Gamma^{\top}\Gamma-(N-1-d)I_{d}\right]+\left[\frac{1}{N}(U1)(U1)^{\top}-I_{d}\right], \tag*{(62)} U U ⊤ − N I d = [ W GD ( N S ) − 1 ( W GD ) ⊤ − d I d ] + [ Γ ⊤ Γ − ( N − 1 − d ) I d ] + [ N 1 ( U 1 ) ( U 1 ) ⊤ − I d ] , (62)
and we bound the trace of each bracket against B B B separately.
First bracket. Since tr B \operatorname{tr}B tr B appears, write
tr [ ( W G D ( N S ) − 1 ( W G D ) ⊤ − d I d ) B ] = ( tr [ W G D ( N S ) − 1 ( W G D ) ⊤ B ] − ∥ W G D ∥ F 2 ) + ( ∥ W G D ∥ F 2 − d tr B ) . (63) \operatorname{tr}\left[\left(W^{\mathrm{GD}}(NS)^{-1}(W^{\mathrm{GD}})^{\top}-dI_{d}\right)B\right]=\left(\operatorname{tr}\left[W^{\mathrm{GD}}(NS)^{-1}(W^{\mathrm{GD}})^{\top}B\right]-\lVert W^{\mathrm{GD}}\rVert_{F}^{2}\right)+\left(\lVert W^{\mathrm{GD}}\rVert_{F}^{2}-d\operatorname{tr}B\right). \tag*{(63)} tr [ ( W GD ( N S ) − 1 ( W GD ) ⊤ − d I d ) B ] = ( tr [ W GD ( N S ) − 1 ( W GD ) ⊤ B ] − ∥ W GD ∥ F 2 ) + ( ∥ W GD ∥ F 2 − d tr B ) . (63)
By Theorem F.26, Theorem F.3(v) (γ ≥ 16 \gamma\ge16 γ ≥ 16 ) and Lemma F.4 (i), with probability tending to one the first difference is at most
2 ( 3 γ − 1 / 2 ) 2 1 / 8 ∥ W G D ∥ F 2 = 144 γ − 1 ( N − 1 ) d 2 ( 1 + O p ( d − 1 ) ) = N ( N − 1 ) d ⋅ 144 γ − 2 ( 1 + O p ( d − 1 ) ) , (64) \frac{2(3\gamma^{-1/2})^{2}}{1/8}\lVert W^{\mathrm{GD}}\rVert_{F}^{2}=144\gamma^{-1}(N-1)d^{2}(1+O_{p}(d^{-1}))=N(N-1)d\cdot144\gamma^{-2}(1+O_{p}(d^{-1})), \tag*{(64)} 1/8 2 ( 3 γ − 1/2 ) 2 ∥ W GD ∥ F 2 = 144 γ − 1 ( N − 1 ) d 2 ( 1 + O p ( d − 1 )) = N ( N − 1 ) d ⋅ 144 γ − 2 ( 1 + O p ( d − 1 )) , (64)
while the second is O p ( N d 3 / 2 + N d ) = N ( N − 1 ) d ⋅ O p ( d N − 3 / 2 + N − 1 ) O_{p}(\sqrt{N}d^{3/2}+Nd)=N(N-1)d\cdot O_{p}(\sqrt{d}N^{-3/2}+N^{-1}) O p ( N d 3/2 + N d ) = N ( N − 1 ) d ⋅ O p ( d N − 3/2 + N − 1 ) by Lemma F.4 (i).
Second bracket. B B B is symmetric and a function of ( V , W G D ) (V,W^{\mathrm{GD}}) ( V , W GD ) , so Theorem F.25 gives E [ tr [ ( Γ ⊤ Γ − ( N − 1 − d ) I d ) B ] ∣ V , W G D ] = 0 \mathbb{E}\left[\operatorname{tr}\left[(\Gamma^{\top}\Gamma-(N-1-d)I_{d})B\right]\mid V,W^{\mathrm{GD}}\right]=0 E [ tr [ ( Γ ⊤ Γ − ( N − 1 − d ) I d ) B ] ∣ V , W GD ] = 0 and, by Equation (60) , conditional variance 2 ( N − 1 − d ) ∥ B ∥ F 2 = O ( N ) ⋅ O p ( N 2 d ) = O p ( N 3 d ) 2(N-1-d)\lVert B\rVert_{F}^{2}=O(N)\cdot O_{p}(N^{2}d)=O_{p}(N^{3}d) 2 ( N − 1 − d ) ∥ B ∥ F 2 = O ( N ) ⋅ O p ( N 2 d ) = O p ( N 3 d ) . The conditional Chebyshev inequality then gives tr [ ( Γ ⊤ Γ − ( N − 1 − d ) I d ) B ] = O p ( N 3 / 2 d 1 / 2 ) = N ( N − 1 ) d ⋅ O p ( ( N d ) − 1 / 2 ) \operatorname{tr}\left[(\Gamma^{\top}\Gamma-(N-1-d)I_{d})B\right]=O_{p}(N^{3/2}d^{1/2})=N(N-1)d\cdot O_{p}((Nd)^{-1/2}) tr [ ( Γ ⊤ Γ − ( N − 1 − d ) I d ) B ] = O p ( N 3/2 d 1/2 ) = N ( N − 1 ) d ⋅ O p (( N d ) − 1/2 ) . Third bracket. By tr ( a a T Y ) = a T Y a ≤ ∥ a ∥ 2 ∥ Y ∥ o p \operatorname{tr}(aa^{\mathsf T}Y)=a^{\mathsf T}Ya\le\lVert a\rVert^2\lVert Y\rVert_{\mathrm{op}} tr ( a a T Y ) = a T Y a ≤ ∥ a ∥ 2 ∥ Y ∥ op , Equation (60) and Theorem F.3(iii),
∣ tr ( 1 N ( U 1 ) ( U 1 ) T − I d ) B ) ∣ ≤ ∥ U 1 ∥ 2 N ∥ B ∥ o p + ∣ tr B ∣ = O p ( d ) ⋅ O p ( N ) + O p ( N d ) = N ( N − 1 ) d ⋅ O p ( N − 1 ) . (65) \left|\operatorname{tr}\left(\frac{1}{N}(U1)(U1)^{\mathsf T}-I_d)B\right)\right|
\le\frac{\lVert U1\rVert^2}{N}\lVert B\rVert_{\mathrm{op}}+|\operatorname{tr}B|
=O_p(d)\cdot O_p(N)+O_p(Nd)=N(N-1)d\cdot O_p(N^{-1}). \tag*{(65)} tr ( N 1 ( U 1 ) ( U 1 ) T − I d ) B ) ≤ N ∥ U 1 ∥ 2 ∥ B ∥ op + ∣ tr B ∣ = O p ( d ) ⋅ O p ( N ) + O p ( N d ) = N ( N − 1 ) d ⋅ O p ( N − 1 ) . (65)
Total. Adding the four contributions and dividing by N ( N − 1 ) d N(N-1)d N ( N − 1 ) d , with probability tending to one the left-hand side is at most 144 γ − 2 ( 1 + O p ( d − 1 ) ) + o p ( 1 ) = 144 γ − 2 + o p ( 1 ) 144\gamma^{-2}(1+O_p(d^{-1}))+o_p(1)=144\gamma^{-2}+o_p(1) 144 γ − 2 ( 1 + O p ( d − 1 )) + o p ( 1 ) = 144 γ − 2 + o p ( 1 ) , which is at most 145 γ − 2 145\gamma^{-2} 145 γ − 2 since γ \gamma γ is fixed.
The remaining term of Theorem F.23 , the diagonal one, needs the following sandwich.
Lemma F.28 (Nuclear-norm sandwich). For all a , b ∈ R d a,b\in\mathbb{R}^d a , b ∈ R d ,
a T Orth ( W G D − a b T ) b ≤ ∥ W G D ∥ ∗ − ∥ W G D − a b T ∥ ∗ ≤ a T W M u o n b . (66) a^{\mathsf T}\operatorname{Orth}(W^{\mathrm{GD}}-ab^{\mathsf T})b
\le\lVert W^{\mathrm{GD}}\rVert_*-\lVert W^{\mathrm{GD}}-ab^{\mathsf T}\rVert_*
\le a^{\mathsf T}W^{\mathrm{Muon}}b. \tag*{(66)} a T Orth ( W GD − a b T ) b ≤ ∥ W GD ∥ ∗ − ∥ W GD − a b T ∥ ∗ ≤ a T W Muon b . (66)
Proof. For all X , Y X,Y X , Y , ⟨ X , Orth ( Y ) ⟩ ≤ ∥ X ∥ ∗ ∥ Orth ( Y ) ∥ o p ≤ ∥ X ∥ ∗ \langle X,\operatorname{Orth}(Y)\rangle\le\lVert X\rVert_*\lVert\operatorname{Orth}(Y)\rVert_{\mathrm{op}}\le\lVert X\rVert_* ⟨ X , Orth ( Y )⟩ ≤ ∥ X ∥ ∗ ∥ Orth ( Y ) ∥ op ≤ ∥ X ∥ ∗ by Equation (9) , and writing the compact SVD Y = P Σ Q T Y=P\Sigma Q^{\mathsf T} Y = P Σ Q T gives ⟨ Y , Orth ( Y ) ⟩ = tr ( Q Σ P T P Q T ) = ∥ Y ∥ ∗ \langle Y,\operatorname{Orth}(Y)\rangle=\operatorname{tr}(Q\Sigma P^{\mathsf T}PQ^{\mathsf T})=\lVert Y\rVert_* ⟨ Y , Orth ( Y )⟩ = tr ( Q Σ P T P Q T ) = ∥ Y ∥ ∗ ; also ⟨ a b T , Z ⟩ = a T Z b \langle ab^{\mathsf T},Z\rangle=a^{\mathsf T}Zb ⟨ a b T , Z ⟩ = a T Z b . Hence
a T Orth ( W G D − a b T ) b = ⟨ W G D , Orth ( W G D − a b T ) ⟩ − ⟨ W G D − a b T , Orth ( W G D − a b T ) ⟩ ≤ ∥ W G D ∥ ∗ − ∥ W G D − a b T ∥ ∗ , \begin{aligned}
a^{\mathsf T}\operatorname{Orth}(W^{\mathrm{GD}}-ab^{\mathsf T})b
=\langle W^{\mathrm{GD}},\operatorname{Orth}(W^{\mathrm{GD}}-ab^{\mathsf T})\rangle
-\langle W^{\mathrm{GD}}-ab^{\mathsf T},\operatorname{Orth}(W^{\mathrm{GD}}-ab^{\mathsf T})\rangle\\
&\le\lVert W^{\mathrm{GD}}\rVert_*-\lVert W^{\mathrm{GD}}-ab^{\mathsf T}\rVert_*,
\end{aligned} a T Orth ( W GD − a b T ) b = ⟨ W GD , Orth ( W GD − a b T )⟩ − ⟨ W GD − a b T , Orth ( W GD − a b T )⟩ ≤ ∥ W GD ∥ ∗ − ∥ W GD − a b T ∥ ∗ ,
and symmetrically
∥ W G D ∥ ∗ − ∥ W G D − a b T ∥ ∗ = ⟨ W G D , W M u o n ⟩ − ∥ W G D − a b T ∥ ∗ ≤ ⟨ W G D , W M u o n ⟩ − ⟨ W G D − a b T , W M u o n ⟩ = a T W M u o n b . \begin{aligned}
\lVert W^{\mathrm{GD}}\rVert_*-\lVert W^{\mathrm{GD}}-ab^{\mathsf T}\rVert_*
=\langle W^{\mathrm{GD}},W^{\mathrm{Muon}}\rangle-\lVert W^{\mathrm{GD}}-ab^{\mathsf T}\rVert_* \\
&\le\langle W^{\mathrm{GD}},W^{\mathrm{Muon}}\rangle
-\langle W^{\mathrm{GD}}-ab^{\mathsf T},W^{\mathrm{Muon}}\rangle
=a^{\mathsf T}W^{\mathrm{Muon}}b.
\end{aligned} ∥ W GD ∥ ∗ − ∥ W GD − a b T ∥ ∗ = ⟨ W GD , W Muon ⟩ − ∥ W GD − a b T ∥ ∗ ≤ ⟨ W GD , W Muon ⟩ − ⟨ W GD − a b T , W Muon ⟩ = a T W Muon b .
:::
Lemma F.29 (Self term). If γ ≥ 16 \gamma\ge16 γ ≥ 16 , then with probability tending to one
0 ≤ 1 ( N − 1 ) d mean i ( u i T W M u o n v i ) 2 ≤ 15 γ − 3 / 2 . (67) 0\le\frac{1}{(N-1)d}\operatorname{mean}_i\left(u_i^{\mathsf T}W^{\mathrm{Muon}}v_i\right)^2\le15\gamma^{-3/2}. \tag*{(67)} 0 ≤ ( N − 1 ) d 1 mean i ( u i T W Muon v i ) 2 ≤ 15 γ − 3/2 . (67)
This bounds the diagonal term of Theorem F.23 ; it is what forces the γ − 3 / 2 \gamma^{-3/2} γ − 3/2 error in κ ˉ M u o n \bar{\kappa}^{\mathrm{Muon}} κ ˉ Muon , and the only place where Theorem F.28 is used.
Proof. Nonnegativity is clear.
Step 1 (reduction to a first absolute moment). By ∥ W M u o n ∥ o p ≤ 1 \lVert W^{\mathrm{Muon}}\rVert_{\mathrm{op}}\le1 ∥ W Muon ∥ op ≤ 1 , ∣ u i T W M u o n v i ∣ ≤ ∥ u i ∥ ∥ v i ∥ |u_i^{\mathsf T}W^{\mathrm{Muon}}v_i|\le\lVert u_i\rVert\lVert v_i\rVert ∣ u i T W Muon v i ∣ ≤ ∥ u i ∥ ∥ v i ∥ , so
mean i ( u i T W M u o n v i ) 2 ≤ d mean i ∣ u i T W M u o n v i ∣ + ( mean i ( u i T W M u o n v i ) 2 ) 1 / 2 ( mean i ( ∥ u i ∥ ∥ v i ∥ − d ) 2 ) 1 / 2 . (68) \operatorname{mean}_i\left(u_i^{\mathsf T}W^{\mathrm{Muon}}v_i\right)^2
\le d\operatorname{mean}_i|u_i^{\mathsf T}W^{\mathrm{Muon}}v_i|
+\left(\operatorname{mean}_i(u_i^{\mathsf T}W^{\mathrm{Muon}}v_i)^2\right)^{1/2}
\left(\operatorname{mean}_i(\lVert u_i\rVert\lVert v_i\rVert-d)^2\right)^{1/2}. \tag*{(68)} mean i ( u i T W Muon v i ) 2 ≤ d mean i ∣ u i T W Muon v i ∣ + ( mean i ( u i T W Muon v i ) 2 ) 1/2 ( mean i (∥ u i ∥ ∥ v i ∥ − d ) 2 ) 1/2 . (68)
For x , c 0 , c 1 ≥ 0 x,c_0,c_1\ge0 x , c 0 , c 1 ≥ 0 , x 2 ≤ c 0 + c 1 x x^2\le c_0+c_1x x 2 ≤ c 0 + c 1 x forces x ≤ c 1 + c 1 2 + 4 c 0 2 ≤ c 1 + c 0 x\le\frac{c_1+\sqrt{c_1^2+4c_0}}{2}\le c_1+\sqrt{c_0} x ≤ 2 c 1 + c 1 2 + 4 c 0 ≤ c 1 + c 0 and hence x 2 ≤ 2 c 1 2 + 2 c 0 x^2\le2c_1^2+2c_0 x 2 ≤ 2 c 1 2 + 2 c 0 ; taking x = ( mean i ( u i T W M u o n v i ) 2 ) 1 / 2 x=(\operatorname{mean}_i(u_i^{\mathsf T}W^{\mathrm{Muon}}v_i)^2)^{1/2} x = ( mean i ( u i T W Muon v i ) 2 ) 1/2 , c 0 = d mean i ∣ u i T W M u o n v i ∣ c_0=d\operatorname{mean}_i|u_i^{\mathsf T}W^{\mathrm{Muon}}v_i| c 0 = d mean i ∣ u i T W Muon v i ∣ and c 1 = ( mean i ( ∥ u i ∥ ∥ v i ∥ − d ) 2 ) 1 / 2 c_1=(\operatorname{mean}_i(\lVert u_i\rVert\lVert v_i\rVert-d)^2)^{1/2} c 1 = ( mean i (∥ u i ∥ ∥ v i ∥ − d ) 2 ) 1/2 , and using Theorem F.3(ii),
mean i ( u i T W M u o n v i ) 2 ≤ O p ( d ) + 2 d mean i ∣ u i T W M u o n v i ∣ . (69) \operatorname{mean}_i\left(u_i^{\mathsf T}W^{\mathrm{Muon}}v_i\right)^2
\le O_p(d)+2d\operatorname{mean}_i|u_i^{\mathsf T}W^{\mathrm{Muon}}v_i|. \tag*{(69)} mean i ( u i T W Muon v i ) 2 ≤ O p ( d ) + 2 d mean i ∣ u i T W Muon v i ∣. (69)
Step 2 (a signed mean). By Theorem F.17 and Theorem F.18, mean i u i T W M u o n v i = ∥ W G D ∥ ∗ N + O p ( d / N ) \operatorname{mean}_i u_i^{\mathsf T}W^{\mathrm{Muon}}v_i=\frac{\lVert W^{\mathrm{GD}}\rVert_*}{N}+O_p(d/N) mean i u i T W Muon v i = N ∥ W GD ∥ ∗ + O p ( d / N ) . For real x ≥ ℓ x\ge\ell x ≥ ℓ one has
∣ x ∣ ≤ x + 2 ∣ ℓ ∣ ; (70) |x|\le x+2|\ell|; \tag*{(70)} ∣ x ∣ ≤ x + 2∣ ℓ ∣ ; (70)
this is clear for x ≥ 0 x\ge0 x ≥ 0 , and for x < 0 x<0 x < 0 it follows from ∣ x ∣ = − x ≤ − ℓ ≤ ∣ ℓ ∣ ≤ x + 2 ∣ ℓ ∣ |x|=-x\le-\ell\le|\ell|\le x+2|\ell| ∣ x ∣ = − x ≤ − ℓ ≤ ∣ ℓ ∣ ≤ x + 2∣ ℓ ∣ , the last step because x + 2 ∣ ℓ ∣ ≥ ℓ + 2 ∣ ℓ ∣ ≥ ∣ ℓ ∣ x+2|\ell|\ge\ell+2|\ell|\ge|\ell| x + 2∣ ℓ ∣ ≥ ℓ + 2∣ ℓ ∣ ≥ ∣ ℓ ∣ .
Step 3 (conditioning). Fix i i i and set
a : = u i − 1 N U 1 1 − 1 / N , b : = v i − 1 N V 1 1 − 1 / N . a:=\frac{u_i-\frac{1}{N}U1}{\sqrt{1-1/N}},\qquad
b:=\frac{v_i-\frac{1}{N}V1}{\sqrt{1-1/N}}. a := 1 − 1/ N u i − N 1 U 1 , b := 1 − 1/ N v i − N 1 V 1 .
Since 1 T ( e i − 1 N 1 ) = 0 \mathbf{1}^{\mathsf{T}}(e_i-\frac{1}{N}\mathbf{1})=0 1 T ( e i − N 1 1 ) = 0 and ∥ e i − 1 N 1 ∥ 2 = 1 − 1 N \left\|e_i-\frac{1}{N}\mathbf{1}\right\|^2=1-\frac{1}{N} e i − N 1 1 2 = 1 − N 1 , Equation (11) gives
W G D − a b T = U [ ( I N − 11 T N ) − ( e i − 1 N 1 ) ( e i − 1 N 1 ) T 1 − 1 / N ] V T , (71) W^{\mathrm{GD}}-ab^{\mathsf{T}}=U\left[\left(I_N-\frac{\mathbf{1}\mathbf{1}^{\mathsf{T}}}{N}\right)-\frac{\left(e_i-\frac{1}{N}\mathbf{1}\right)\left(e_i-\frac{1}{N}\mathbf{1}\right)^{\mathsf{T}}}{1-1/N}\right]V^{\mathsf{T}}, \tag*{(71)} W GD − a b T = U [ ( I N − N 1 1 T ) − 1 − 1/ N ( e i − N 1 1 ) ( e i − N 1 1 ) T ] V T , (71)
where the bracket, call it M M M , is the orthogonal projection onto { x ∈ R N : 1 T x = 0 , ( e i − 1 N 1 ) T x = 0 } \{x\in\mathbb{R}^{N}:\mathbf{1}^{\mathsf{T}}x=0,\ (e_i-\frac{1}{N}\mathbf{1})^{\mathsf{T}}x=0\} { x ∈ R N : 1 T x = 0 , ( e i − N 1 1 ) T x = 0 } . Indeed M = P − Π M=P-\Pi M = P − Π with P : = I N − 11 T N P:=I_N-\frac{\mathbf{1}\mathbf{1}^{\mathsf{T}}}{N} P := I N − N 1 1 T and Π : = ( e i − 1 N 1 ) ( e i − 1 N 1 ) T / ∥ e i − 1 N 1 ∥ 2 \Pi:=(e_i-\frac{1}{N}\mathbf{1})(e_i-\frac{1}{N}\mathbf{1})^{\mathsf{T}}/\|e_i-\frac{1}{N}\mathbf{1}\|^2 Π := ( e i − N 1 1 ) ( e i − N 1 1 ) T /∥ e i − N 1 1 ∥ 2 the orthogonal projections onto 1 ⊥ \mathbf{1}^{\perp} 1 ⊥ and onto span ( e i − 1 N 1 ) \operatorname{span}(e_i-\frac{1}{N}\mathbf{1}) span ( e i − N 1 1 ) ; since 1 T ( e i − 1 N 1 ) = 0 \mathbf{1}^{\mathsf{T}}(e_i-\frac{1}{N}\mathbf{1})=0 1 T ( e i − N 1 1 ) = 0 the range of Π \Pi Π lies in that of P P P , so P Π = Π P = Π P\Pi=\Pi P=\Pi P Π = Π P = Π and hence M T = M M^{\mathsf{T}}=M M T = M , M 2 = M M^2=M M 2 = M and M ( e i − 1 N 1 ) = 0 M(e_i-\frac{1}{N}\mathbf{1})=0 M ( e i − N 1 1 ) = 0 . Consequently a a a and b b b are standard Gaussian and independent of W G D − a b T W^{\mathrm{GD}}-ab^{\mathsf{T}} W GD − a b T . For a a a : its k k k -th coordinate is ( e i − 1 N 1 ) T U T e k / 1 − 1 / N (e_i-\frac{1}{N}\mathbf{1})^{\mathsf{T}}U^{\mathsf{T}}e_k/\sqrt{1-1/N} ( e i − N 1 1 ) T U T e k / 1 − 1/ N and the k k k -th row of U M UM U M is M U T e k MU^{\mathsf{T}}e_k M U T e k ; conditionally on V V V the vectors U T e k U^{\mathsf{T}}e_k U T e k are i.i.d. N ( 0 , I N ) \mathcal{N}(0,I_N) N ( 0 , I N ) , and
Cov ( ( e i − 1 N 1 ) T U T e k , M U T e k ) = M ( e i − 1 N 1 ) = 0 , (72) \operatorname{Cov}\left((e_i-\frac{1}{N}\mathbf{1})^{\mathsf{T}}U^{\mathsf{T}}e_k,MU^{\mathsf{T}}e_k\right)=M(e_i-\frac{1}{N}\mathbf{1})=0, \tag*{(72)} Cov ( ( e i − N 1 1 ) T U T e k , M U T e k ) = M ( e i − N 1 1 ) = 0 , (72)
so, being jointly Gaussian, a a a is independent of U M UM U M , and ∥ ( e i − 1 N 1 ) / 1 − 1 / N ∥ = 1 \|(e_i-\frac{1}{N}\mathbf{1})/\sqrt{1-1/N}\|=1 ∥ ( e i − N 1 1 ) / 1 − 1/ N ∥ = 1 gives a ∼ N ( 0 , I d ) a\sim\mathcal{N}(0,I_d) a ∼ N ( 0 , I d ) . The same argument applied to the rows of V V V gives b ∼ N ( 0 , I d ) b\sim\mathcal{N}(0,I_d) b ∼ N ( 0 , I d ) independent of V M VM V M , and U ⊥ V U\perp V U ⊥ V makes a , b a,b a , b independent of each other; since W G D − a b T = ( U M ) ( V M ) T W^{\mathrm{GD}}-ab^{\mathsf{T}}=(UM)(VM)^{\mathsf{T}} W GD − a b T = ( U M ) ( V M ) T by idempotence of M M M , the pair ( a , b ) (a,b) ( a , b ) is independent of W G D − a b T W^{\mathrm{GD}}-ab^{\mathsf{T}} W GD − a b T .
Step 4 (two mean-square bounds). For independent a , b ∼ N ( 0 , I d ) a,b\sim\mathcal{N}(0,I_d) a , b ∼ N ( 0 , I d ) and Z Z Z independent of them, E [ ( a T Z b ) 2 ∣ Z ] = ∥ Z ∥ F 2 \mathbb{E}[(a^{\mathsf{T}}Zb)^2\mid Z]=\|Z\|_{\mathrm{F}}^2 E [( a T Z b ) 2 ∣ Z ] = ∥ Z ∥ F 2 ; taking Z = Orth ( W G D − a b T ) Z=\operatorname{Orth}(W^{\mathrm{GD}}-ab^{\mathsf{T}}) Z = Orth ( W GD − a b T ) , whose squared Frobenius norm is its rank, at most d d d , and averaging over i i i ,
( mean i ( a T Orth ( W G D − a b T ) b ) 2 ) 1 / 2 = O p ( d ) . (73) \left(\operatorname{mean}_{i}(a^{\mathsf{T}}\operatorname{Orth}(W^{\mathrm{GD}}-ab^{\mathsf{T}})b)^2\right)^{1/2}=O_p(\sqrt{d}). \tag*{(73)} ( mean i ( a T Orth ( W GD − a b T ) b ) 2 ) 1/2 = O p ( d ) . (73)
Substituting u i = 1 − 1 N a + 1 N U 1 u_i=\sqrt{1-\frac{1}{N}}a+\frac{1}{N}U\mathbf{1} u i = 1 − N 1 a + N 1 U 1 and v i = 1 − 1 N b + 1 N V 1 v_i=\sqrt{1-\frac{1}{N}}b+\frac{1}{N}V\mathbf{1} v i = 1 − N 1 b + N 1 V 1 ,
u i T W M u o n v i − ( 1 − 1 N ) a T W M u o n b = 1 − 1 N a T W M u o n V 1 N + 1 − 1 N ( U 1 ) T N W M u o n b + ( U 1 ) T N W M u o n V 1 N . (74) u_i^{\mathsf{T}}W^{\mathrm{Muon}}v_i-\left(1-\frac{1}{N}\right)a^{\mathsf{T}}W^{\mathrm{Muon}}b
=\sqrt{1-\frac{1}{N}}a^{\mathsf{T}}W^{\mathrm{Muon}}\frac{V\mathbf{1}}{N}
+\sqrt{1-\frac{1}{N}}\frac{(U\mathbf{1})^{\mathsf{T}}}{N}W^{\mathrm{Muon}}b
+\frac{(U\mathbf{1})^{\mathsf{T}}}{N}W^{\mathrm{Muon}}\frac{V\mathbf{1}}{N}. \tag*{(74)} u i T W Muon v i − ( 1 − N 1 ) a T W Muon b = 1 − N 1 a T W Muon N V 1 + 1 − N 1 N ( U 1 ) T W Muon b + N ( U 1 ) T W Muon N V 1 . (74)
so by ( x + y + z ) 2 ≤ 3 ( x 2 + y 2 + z 2 ) (x+y+z)^2\leq3(x^2+y^2+z^2) ( x + y + z ) 2 ≤ 3 ( x 2 + y 2 + z 2 ) , ∥ W M u o n ∥ o p ≤ 1 \|W^{\mathrm{Muon}}\|_{\mathrm{op}}\leq1 ∥ W Muon ∥ op ≤ 1 , E ∥ a ∥ 2 = E ∥ b ∥ 2 = d \mathbb{E}\|a\|^2=\mathbb{E}\|b\|^2=d E ∥ a ∥ 2 = E ∥ b ∥ 2 = d (so that mean i ∥ a ∥ 2 = O p ( d ) \operatorname{mean}_i\|a\|^2=O_p(d) mean i ∥ a ∥ 2 = O p ( d ) by Markov) and Theorem F.3(iii),
mean i ( u i T W M u o n v i − ( 1 − 1 N ) a T W M u o n b ) 2 ≤ 3 [ mean i ∥ a ∥ 2 ∥ V 1 ∥ 2 N 2 + ∥ U 1 ∥ 2 N 2 mean i ∥ b ∥ 2 + ∥ U 1 ∥ 2 ∥ V 1 ∥ 2 N 4 ] = O p ( d 2 N ) , (75) \operatorname{mean}_{i}\left(u_i^{\mathsf{T}}W^{\mathrm{Muon}}v_i-\left(1-\frac{1}{N}\right)a^{\mathsf{T}}W^{\mathrm{Muon}}b\right)^2
\leq3\left[\operatorname{mean}_{i}\|a\|^2\frac{\|V\mathbf{1}\|^2}{N^2}+\frac{\|U\mathbf{1}\|^2}{N^2}\operatorname{mean}_{i}\|b\|^2+\frac{\|U\mathbf{1}\|^2\|V\mathbf{1}\|^2}{N^4}\right]
=O_p\left(\frac{d^2}{N}\right), \tag*{(75)} mean i ( u i T W Muon v i − ( 1 − N 1 ) a T W Muon b ) 2 ≤ 3 [ mean i ∥ a ∥ 2 N 2 ∥ V 1 ∥ 2 + N 2 ∥ U 1 ∥ 2 mean i ∥ b ∥ 2 + N 4 ∥ U 1 ∥ 2 ∥ V 1 ∥ 2 ] = O p ( N d 2 ) , (75)
that is
( mean i ( u i T W M u o n v i − ( 1 − 1 N ) a T W M u o n b ) 2 ) 1 / 2 = O p ( d / N ) . (76) \left(\operatorname{mean}_{i}\left(u_i^{\mathsf{T}}W^{\mathrm{Muon}}v_i-\left(1-\frac{1}{N}\right)a^{\mathsf{T}}W^{\mathrm{Muon}}b\right)^2\right)^{1/2}=O_p(d/\sqrt{N}). \tag*{(76)} ( mean i ( u i T W Muon v i − ( 1 − N 1 ) a T W Muon b ) 2 ) 1/2 = O p ( d / N ) . (76)
Step 5 (combining). By Theorem F.28 and 1 − 1 N > 0 1-\frac{1}{N}>0 1 − N 1 > 0 , u i T W M u o n v i ≥ ( 1 − 1 N ) a T Orth ( W G D − a b T ) b − ∣ u i T W M u o n v i − ( 1 − 1 N ) a T W M u o n b ∣ u_i^{\mathsf{T}}W^{\mathrm{Muon}}v_i\geq(1-\frac{1}{N})a^{\mathsf{T}}\operatorname{Orth}(W^{\mathrm{GD}}-ab^{\mathsf{T}})b-\lvert u_i^{\mathsf{T}}W^{\mathrm{Muon}}v_i-(1-\frac{1}{N})a^{\mathsf{T}}W^{\mathrm{Muon}}b\rvert u i T W Muon v i ≥ ( 1 − N 1 ) a T Orth ( W GD − a b T ) b − ∣ u i T W Muon v i − ( 1 − N 1 ) a T W Muon b ∣ . Using Equation (70) of Step 2 with this lower bound, averaging over i i i , and applying Equation (73) , Equation (76) and ∥ W G D ∥ ∗ ≤ 7 d N d \|W^{\mathrm{GD}}\|_*\leq7d\sqrt{Nd} ∥ W GD ∥ ∗ ≤ 7 d N d from Lemma F.4(iii),
mean i ∣ u i T W M u o n v i ∣ ≤ ∥ W G D ∥ ∗ N + O p ( d / N ) + O p ( d ) + O p ( d / N ) ≤ 7 d 3 / 2 N + O p ( d ) . (77) \operatorname{mean}_{i}\left|u_i^{\mathsf{T}}W^{\mathrm{Muon}}v_i\right|
\leq\frac{\|W^{\mathrm{GD}}\|_*}{N}+O_p(d/N)+O_p(\sqrt{d})+O_p(d/\sqrt{N})
\leq\frac{7d^{3/2}}{\sqrt{N}}+O_p(\sqrt{d}). \tag*{(77)} mean i u i T W Muon v i ≤ N ∥ W GD ∥ ∗ + O p ( d / N ) + O p ( d ) + O p ( d / N ) ≤ N 7 d 3/2 + O p ( d ) . (77)
Substituting into Equation (69) and dividing by ( N − 1 ) d (N-1)d ( N − 1 ) d ,
1 ( N − 1 ) d mean i ( u i T W M u o n v i ) 2 ≤ O p ( d ) ( N − 1 ) d + 2 N − 1 ( 7 d 3 / 2 N + O p ( d ) ) = 14 γ − 3 / 2 ( 1 + 1 N − 1 ) + o p ( 1 ) , (78) \begin{aligned}
\frac{1}{(N-1)d}\operatorname{mean}_{i}(u_i^{\mathsf{T}}W^{\mathrm{Muon}}v_i)^2
\leq\frac{O_p(d)}{(N-1)d}+\frac{2}{N-1}\left(\frac{7d^{3/2}}{\sqrt{N}}+O_p(\sqrt{d})\right) \\
&=14\gamma^{-3/2}\left(1+\frac{1}{N-1}\right)+o_p(1), \tag*{(78)}
\end{aligned} ( N − 1 ) d 1 mean i ( u i T W Muon v i ) 2 ≤ ( N − 1 ) d O p ( d ) + N − 1 2 ( N 7 d 3/2 + O p ( d ) ) = 14 γ − 3/2 ( 1 + N − 1 1 ) + o p ( 1 ) , (78)
which is at most 15 γ − 3 / 2 15\gamma^{-3/2} 15 γ − 3/2 with probability tending to one.
Lemma F.30 (Second moment for Muon). If γ ≥ 16 \gamma\ge16 γ ≥ 16 , then
1 N ( N − 1 ) d ∑ i = 1 N ∑ j ≠ i ( u j T W M u o n v i ) 2 = 1 + O ( γ − 3 / 2 ) + o p ( 1 ) , \frac{1}{N(N-1)d}\sum_{i=1}^{N}\sum_{j\ne i}(u_j^{\mathsf T}W^{\mathrm{Muon}}v_i)^2 = 1 + O(\gamma^{-3/2}) + o_p(1), N ( N − 1 ) d 1 i = 1 ∑ N j = i ∑ ( u j T W Muon v i ) 2 = 1 + O ( γ − 3/2 ) + o p ( 1 ) , where with probability tending to one the O ( γ − 3 / 2 ) O(\gamma^{-3/2}) O ( γ − 3/2 ) term has absolute value at most 52 γ − 3 / 2 52\gamma^{-3/2} 52 γ − 3/2 .
Proof. Divide Theorem F.23 by d d d and subtract 1. By Theorem F.24 the norm term contributes O ( N − 1 ) + O p ( ( N d ) − 1 / 2 ) = o p ( 1 ) O(N^{-1}) + O_p((Nd)^{-1/2}) = o_p(1) O ( N − 1 ) + O p (( N d ) − 1/2 ) = o p ( 1 ) ; by Lemma F.27 the trace term is at most 145 γ − 2 ≤ 145 4 γ − 3 / 2 145\gamma^{-2} \le\frac{145}{4}\gamma^{-3/2} 145 γ − 2 ≤ 4 145 γ − 3/2 in absolute value for γ ≥ 16 \gamma\ge16 γ ≥ 16 ; by Lemma F.29 the diagonal term lies in [ − 15 γ − 3 / 2 , 0 ] [-15\gamma^{-3/2},0] [ − 15 γ − 3/2 , 0 ] . Adding, 145 4 + 15 ≤ 52 \frac{145}{4}+15 \le52 4 145 + 15 ≤ 52 .
Lemma F.31 (κ ˉ \bar{\kappa} κ ˉ for Muon). If γ ≥ 16 \gamma\ge16 γ ≥ 16 , then ( κ ˉ M u o n ) 2 = d ( 1 + O ( γ − 3 / 2 ) + o p ( 1 ) ) (\bar{\kappa}^{\mathrm{Muon}})^2 = d(1+O(\gamma^{-3/2})+o_p(1)) ( κ ˉ Muon ) 2 = d ( 1 + O ( γ − 3/2 ) + o p ( 1 )) with the same constant as in Theorem F.30.
Proof. By Equation (37) with W = W M u o n W = W^{\mathrm{Muon}} W = W Muon , Theorem F.30 and Theorem F.21 , ( κ ˉ M u o n ) 2 = d ( 1 + O ( γ − 3 / 2 ) + o p ( 1 ) ) − O p ( d / N ) = d ( 1 + O ( γ − 3 / 2 ) + o p ( 1 ) ) (\bar{\kappa}^{\mathrm{Muon}})^2 = d(1+O(\gamma^{-3/2})+o_p(1))-O_p(d/N) = d(1+O(\gamma^{-3/2})+o_p(1)) ( κ ˉ Muon ) 2 = d ( 1 + O ( γ − 3/2 ) + o p ( 1 )) − O p ( d / N ) = d ( 1 + O ( γ − 3/2 ) + o p ( 1 )) .
Lemma F.32 (ρ \rho ρ for Muon). If γ ≥ 16 \gamma\ge16 γ ≥ 16 , then
ρ M u o n = d γ [ 8 3 π ( 1 − 1 10 γ ) + O ( γ − 3 / 2 ) + o p ( 1 ) ] . (79) \rho^{\mathrm{Muon}} = \sqrt{\frac{d}{\gamma}}\left[\frac{8}{3\pi}\left(1-\frac{1}{10\gamma}\right)+O(\gamma^{-3/2})+o_p(1)\right]. \tag*{(79)} ρ Muon = γ d [ 3 π 8 ( 1 − 10 γ 1 ) + O ( γ − 3/2 ) + o p ( 1 ) ] . (79)
Unlike for GD, the data budget enters ρ M u o n \rho^{\mathrm{Muon}} ρ Muon a second time, through Φ ( γ ) \Phi(\gamma) Φ ( γ ) .
Proof. By Theorem F.22 , h ˉ M u o n / d = d / γ Φ ( γ ) ( 1 + o p ( 1 ) ) \bar{h}^{\mathrm{Muon}}/\sqrt{d}=\sqrt{d/\gamma}\,\Phi(\gamma)(1+o_p(1)) h ˉ Muon / d = d / γ Φ ( γ ) ( 1 + o p ( 1 )) . Set y : = ( κ ˉ M u o n ) 2 / d y := (\bar{\kappa}^{\mathrm{Muon}})^2/d y := ( κ ˉ Muon ) 2 / d . By Theorem F.31 and γ ≥ 16 \gamma\ge16 γ ≥ 16 , with probability tending to one
∣ y − 1 ∣ ≤ 52 γ − 3 / 2 + o p ( 1 ) ≤ 52 64 + o p ( 1 ) ≤ 9 10 , |y-1| \le52\gamma^{-3/2}+o_p(1) \le\frac{52}{64}+o_p(1) \le\frac{9}{10}, ∣ y − 1∣ ≤ 52 γ − 3/2 + o p ( 1 ) ≤ 64 52 + o p ( 1 ) ≤ 10 9 ,
so y ≥ 1 10 y \ge\frac{1}{10} y ≥ 10 1 . For y ≥ 1 10 y \ge\frac{1}{10} y ≥ 10 1 ,
∣ y − 1 / 2 − 1 ∣ = ∣ 1 − y ∣ y ( 1 + y ) ≤ 10 ∣ y − 1 ∣ , |y^{-1/2}-1| = \frac{|1-y|}{\sqrt{y}(1+\sqrt{y})} \le\sqrt{10}|y-1|, ∣ y − 1/2 − 1∣ = y ( 1 + y ) ∣1 − y ∣ ≤ 10 ∣ y − 1∣ ,
which gives d / κ ˉ M u o n = 1 + O ( γ − 3 / 2 ) + o p ( 1 ) \sqrt{d}/\bar{\kappa}^{\mathrm{Muon}}=1+O(\gamma^{-3/2})+o_p(1) d / κ ˉ Muon = 1 + O ( γ − 3/2 ) + o p ( 1 ) . Multiplying the two,
ρ M u o n = h ˉ M u o n d ⋅ d κ ˉ M u o n = d γ Φ ( γ ) ( 1 + O ( γ − 3 / 2 ) + o p ( 1 ) ) , (80) \rho^{\mathrm{Muon}} = \frac{\bar{h}^{\mathrm{Muon}}}{\sqrt{d}}\cdot\frac{\sqrt{d}}{\bar{\kappa}^{\mathrm{Muon}}} = \sqrt{\frac{d}{\gamma}}\,\Phi(\gamma)(1+O(\gamma^{-3/2})+o_p(1)), \tag*{(80)} ρ Muon = d h ˉ Muon ⋅ κ ˉ Muon d = γ d Φ ( γ ) ( 1 + O ( γ − 3/2 ) + o p ( 1 )) , (80)
and Lemma F.4 (iv) gives Φ ( γ ) = 8 3 π ( 1 − 1 10 γ + r ( γ ) ) \Phi(\gamma)=\frac{8}{3\pi}(1-\frac{1}{10\gamma}+r(\gamma)) Φ ( γ ) = 3 π 8 ( 1 − 10 γ 1 + r ( γ )) with ∣ r ( γ ) ∣ ≤ γ − 2 ≤ 1 4 γ − 3 / 2 |r(\gamma)|\le\gamma^{-2}\le\frac{1}{4}\gamma^{-3/2} ∣ r ( γ ) ∣ ≤ γ − 2 ≤ 4 1 γ − 3/2 , so 0 < Φ ( γ ) ≤ 8 3 π ( 1 + γ − 2 ) < 1 0<\Phi(\gamma)\le\frac{8}{3\pi}(1+\gamma^{-2})<1 0 < Φ ( γ ) ≤ 3 π 8 ( 1 + γ − 2 ) < 1 and
Φ ( γ ) ( 1 + O ( γ − 3 / 2 ) + o p ( 1 ) ) = 8 3 π ( 1 − 1 10 γ ) + O ( γ − 3 / 2 ) + o p ( 1 ) . \Phi(\gamma)(1+O(\gamma^{-3/2})+o_p(1))=\frac{8}{3\pi}(1-\frac{1}{10\gamma})+O(\gamma^{-3/2})+o_p(1). Φ ( γ ) ( 1 + O ( γ − 3/2 ) + o p ( 1 )) = 3 π 8 ( 1 − 10 γ 1 ) + O ( γ − 3/2 ) + o p ( 1 ) .
Two by-products of this computation are used in the proof of Theorem 4.2 . First, the constant implied by the O ( γ − 3 / 2 ) O(\gamma^{-3/2}) O ( γ − 3/2 ) in the displayed multiplicative form is absolute, so there is an absolute γ 1 ≥ 16 \gamma_1\ge16 γ 1 ≥ 16 such that for γ ≥ γ 1 \gamma\ge\gamma_1 γ ≥ γ 1 that factor lies in [ 1 2 , 3 2 ] [\frac{1}{2},\frac{3}{2}] [ 2 1 , 2 3 ] with probability tending to one. Second, for every γ ≥ 16 \gamma\ge16 γ ≥ 16 the same display bounds ρ M u o n \rho^{\mathrm{Muon}} ρ Muon away from zero without any expansion: y ≤ 19 10 y\le\frac{19}{10} y ≤ 10 19 gives d / κ ˉ M u o n ≥ ( 19 / 10 ) − 1 / 2 > 7 10 \sqrt{d}/\bar{\kappa}^{\mathrm{Muon}}\ge(19/10)^{-1/2}>\frac{7}{10} d / κ ˉ Muon ≥ ( 19/10 ) − 1/2 > 10 7 , and Φ ( γ ) ≥ 1 8 \Phi(\gamma)\ge\frac{1}{8} Φ ( γ ) ≥ 8 1 by Lemma F.4 (iii),(iv), so with probability tending to one
ρ M u o n ≥ 7 80 d / γ ( 1 − o p ( 1 ) ) ≥ 1 16 d / γ . (81) \rho^{\mathrm{Muon}}\ge\frac{7}{80}\sqrt{d/\gamma}(1-o_p(1))\ge\frac{1}{16}\sqrt{d/\gamma}. \tag*{(81)} ρ Muon ≥ 80 7 d / γ ( 1 − o p ( 1 )) ≥ 16 1 d / γ . (81)
Combining the GD and Muon estimates
Proof of Equation (8) . Reduction to large γ \gamma γ . By Lemma F.16 and 2 + γ ≥ γ 2+\gamma\ge\gamma 2 + γ ≥ γ , ρ G D ≤ d / γ ( 1 + o p ( 1 ) ) \rho^{\mathrm{GD}}\le\sqrt{d/\gamma}(1+o_p(1)) ρ GD ≤ d / γ ( 1 + o p ( 1 )) , so Equation (81) gives R ≤ 17 R\le17 R ≤ 17 with probability tending to one, for every γ ≥ 16 \gamma\ge16 γ ≥ 16 ; together with 0 ≤ 3 π 8 ( 1 − 9 10 γ ) ≤ 3 π 8 0\le\frac{3\pi}{8}(1-\frac{9}{10\gamma})\le\frac{3\pi}{8} 0 ≤ 8 3 π ( 1 − 10 γ 9 ) ≤ 8 3 π this makes the left-hand side of Equation (8) at most 19 there. On a bounded range 16 ≤ γ ≤ γ 1 16\le\gamma\le\gamma_1 16 ≤ γ ≤ γ 1 we have 19 ≤ 19 γ 1 3 / 2 γ − 3 / 2 19\le19\gamma_1^{3/2}\gamma^{-3/2} 19 ≤ 19 γ 1 3/2 γ − 3/2 , so Equation (8) holds on that range as soon as C 1 ≥ 19 γ 1 3 / 2 C_1\ge19\gamma_1^{3/2} C 1 ≥ 19 γ 1 3/2 . It therefore suffices to prove it for γ ≥ γ 1 \gamma\ge\gamma_1 γ ≥ γ 1 , with γ 1 \gamma_1 γ 1 the absolute constant of Lemma F.32 ; on that range the error factor of Lemma F.32 lies in [ 1 2 , 3 2 ] [\frac{1}{2},\frac{3}{2}] [ 2 1 , 2 3 ] with probability tending to one, so its reciprocal is again 1 + O ( γ − 3 / 2 ) + o p ( 1 ) 1+O(\gamma^{-3/2})+o_p(1) 1 + O ( γ − 3/2 ) + o p ( 1 ) .
Expansion. By Lemma F.16 and Lemma F.32 , and since Φ ( γ ) > 0 \Phi(\gamma)>0 Φ ( γ ) > 0 ,
R = ρ G D ρ M u o n = d 2 + γ ( 1 + o p ( 1 ) ) ⋅ γ d 1 Φ ( γ ) ( 1 + O ( γ − 3 / 2 ) + o p ( 1 ) ) = γ / ( 2 + γ ) Φ ( γ ) ( 1 + O ( γ − 3 / 2 ) + o p ( 1 ) ) . (82) \begin{aligned}
\mathcal{R}=\frac{\rho^{\mathrm{GD}}}{\rho^{\mathrm{Muon}}}=\sqrt{\frac{d}{2+\gamma}\left(1+o_p(1)\right)}\cdot\sqrt{\frac{\gamma}{d}}\frac{1}{\Phi(\gamma)\left(1+O(\gamma^{-3/2})+o_p(1)\right)} \\
&=\frac{\sqrt{\gamma/(2+\gamma)}}{\Phi(\gamma)}\left(1+O(\gamma^{-3/2})+o_p(1)\right). \tag*{(82)}
\end{aligned} R = ρ Muon ρ GD = 2 + γ d ( 1 + o p ( 1 ) ) ⋅ d γ Φ ( γ ) ( 1 + O ( γ − 3/2 ) + o p ( 1 ) ) 1 = Φ ( γ ) γ / ( 2 + γ ) ( 1 + O ( γ − 3/2 ) + o p ( 1 ) ) . (82)
The deterministic prefactor is expanded in 1 / γ 1/\gamma 1/ γ : γ / ( 2 + γ ) = ( 1 + 2 / γ ) − 1 / 2 = 1 − γ − 1 + O ( γ − 2 ) \sqrt{\gamma/(2+\gamma)}=(1+2/\gamma)^{-1/2}=1-\gamma^{-1}+O(\gamma^{-2}) γ / ( 2 + γ ) = ( 1 + 2/ γ ) − 1/2 = 1 − γ − 1 + O ( γ − 2 ) , while Lemma F.4 (iv) gives Φ ( γ ) − 1 = 3 π 8 ( 1 + 1 10 γ + O ( γ − 2 ) ) \Phi(\gamma)^{-1}=\frac{3\pi}{8}\left(1+\frac{1}{10\gamma}+O(\gamma^{-2})\right) Φ ( γ ) − 1 = 8 3 π ( 1 + 10 γ 1 + O ( γ − 2 ) ) , so
γ / ( 2 + γ ) Φ ( γ ) = 3 π 8 ( 1 − 1 γ + O ( γ − 2 ) ) ( 1 + 1 10 γ + O ( γ − 2 ) ) = 3 π 8 ( 1 − 9 10 γ ) + O ( γ − 2 ) . (83) \frac{\sqrt{\gamma/(2+\gamma)}}{\Phi(\gamma)}=\frac{3\pi}{8}\left(1-\frac{1}{\gamma}+O(\gamma^{-2})\right)\left(1+\frac{1}{10\gamma}+O(\gamma^{-2})\right)=\frac{3\pi}{8}\left(1-\frac{9}{10\gamma}\right)+O(\gamma^{-2}). \tag*{(83)} Φ ( γ ) γ / ( 2 + γ ) = 8 3 π ( 1 − γ 1 + O ( γ − 2 ) ) ( 1 + 10 γ 1 + O ( γ − 2 ) ) = 8 3 π ( 1 − 10 γ 9 ) + O ( γ − 2 ) . (83)
For γ ≥ 16 \gamma\ge16 γ ≥ 16 the prefactor is bounded, 0 < γ / ( 2 + γ ) / Φ ( γ ) ≤ 3 π 8 ( 1 − 1 160 − 1 256 ) − 1 0<\sqrt{\gamma/(2+\gamma)}/\Phi(\gamma)\le\frac{3\pi}{8}\left(1-\frac{1}{160}-\frac{1}{256}\right)^{-1} 0 < γ / ( 2 + γ ) /Φ ( γ ) ≤ 8 3 π ( 1 − 160 1 − 256 1 ) − 1 , so multiplying by 1 + O ( γ − 3 / 2 ) + o p ( 1 ) 1+O(\gamma^{-3/2})+o_p(1) 1 + O ( γ − 3/2 ) + o p ( 1 ) leaves the error of the same order; and γ − 2 ≤ 1 4 γ − 3 / 2 \gamma^{-2}\le\frac{1}{4}\gamma^{-3/2} γ − 2 ≤ 4 1 γ − 3/2 in that range absorbs the O ( γ − 2 ) O(\gamma^{-2}) O ( γ − 2 ) term. Hence
R = 3 π 8 ( 1 − 9 10 γ ) + O ( γ − 3 / 2 ) + o p ( 1 ) , (84) \mathcal{R}=\frac{3\pi}{8}\left(1-\frac{9}{10\gamma}\right)+O(\gamma^{-3/2})+o_p(1), \tag*{(84)} R = 8 3 π ( 1 − 10 γ 9 ) + O ( γ − 3/2 ) + o p ( 1 ) , (84)
where, with probability tending to one, the O ( γ − 3 / 2 ) O(\gamma^{-3/2}) O ( γ − 3/2 ) term has absolute value at most C 1 γ − 3 / 2 C_1\gamma^{-3/2} C 1 γ − 3/2 with C 1 C_1 C 1 independent of γ \gamma γ , d d d , N N N . This is Equation (8) .
Monotonicity in the data budget
The expansion Equation (8) has two consequences for how R \mathcal{R} R changes with γ \gamma γ . The first compares a fixed pair of data budgets; the second concerns the limit d → ∞ d\to\infty d → ∞ .
Lemma F.33 (Fixed ratio). Let λ > 0 \lambda>0 λ > 0 and
γ > γ 0 ( λ ) : = max { 16 , ( 160 C 1 ( 1 + λ ) 27 π λ ) 2 } . \gamma>\gamma_0(\lambda):=\max\left\{16,\left(\frac{160C_1(1+\lambda)}{27\pi\lambda}\right)^2\right\}. γ > γ 0 ( λ ) := max { 16 , ( 27 π λ 160 C 1 ( 1 + λ ) ) 2 } . For the same d d d , let N 1 = γ d N_1=\gamma d N 1 = γ d and N 2 = ( 1 + λ ) γ d N_2=(1+\lambda)\gamma d N 2 = ( 1 + λ ) γ d , and let R N 1 , d \mathcal{R}_{N_1,d} R N 1 , d and R N 2 , d \mathcal{R}_{N_2,d} R N 2 , d be the corresponding values of R \mathcal{R} R . Then, with probability tending to one, R N 2 , d > R N 1 , d \mathcal{R}_{N_2,d}>\mathcal{R}_{N_1,d} R N 2 , d > R N 1 , d .
Proof. Apply Equation (8) once at γ \gamma γ and once at ( 1 + λ ) γ (1+\lambda)\gamma ( 1 + λ ) γ , both of which are at least 16: for every ε > 0 \varepsilon>0 ε > 0 , with probability tending to one,
R N 2 , d − R N 1 , d ≥ 3 π 8 ( 1 − 9 10 ( 1 + λ ) γ ) − 3 π 8 ( 1 − 9 10 γ ) − C 1 ( ( 1 + λ ) γ ) − 3 / 2 − C 1 γ − 3 / 2 − 2 ε (85) \mathcal{R}_{N_2,d}-\mathcal{R}_{N_1,d}\ge\frac{3\pi}{8}\left(1-\frac{9}{10(1+\lambda)\gamma}\right)-\frac{3\pi}{8}\left(1-\frac{9}{10\gamma}\right)-C_1((1+\lambda)\gamma)^{-3/2}-C_1\gamma^{-3/2}-2\varepsilon \tag*{(85)} R N 2 , d − R N 1 , d ≥ 8 3 π ( 1 − 10 ( 1 + λ ) γ 9 ) − 8 3 π ( 1 − 10 γ 9 ) − C 1 (( 1 + λ ) γ ) − 3/2 − C 1 γ − 3/2 − 2 ε (85) ≥ 27 π 80 ⋅ λ ( 1 + λ ) γ − 2 C 1 γ − 3 / 2 − 2 ε . \ge\frac{27\pi}{80}\cdot\frac{\lambda}{(1+\lambda)\gamma}-2C_1\gamma^{-3/2}-2\varepsilon. ≥ 80 27 π ⋅ ( 1 + λ ) γ λ − 2 C 1 γ − 3/2 − 2 ε . Moreover 27 π 80 ⋅ λ ( 1 + λ ) γ > 2 C 1 γ − 3 / 2 \frac{27\pi}{80}\cdot\frac{\lambda}{(1+\lambda)\gamma}>2C_1\gamma^{-3/2} 80 27 π ⋅ ( 1 + λ ) γ λ > 2 C 1 γ − 3/2 is equivalent to γ 1 / 2 > 160 C 1 ( 1 + λ ) 27 π λ \gamma^{1/2}>\frac{160C_1(1+\lambda)}{27\pi\lambda} γ 1/2 > 27 π λ 160 C 1 ( 1 + λ ) , which holds by assumption; taking ε = 1 4 ( 27 π 80 ⋅ λ ( 1 + λ ) γ − 2 C 1 γ − 3 / 2 ) \varepsilon=\frac{1}{4}\left(\frac{27\pi}{80}\cdot\frac{\lambda}{(1+\lambda)\gamma}-2C_1\gamma^{-3/2}\right) ε = 4 1 ( 80 27 π ⋅ ( 1 + λ ) γ λ − 2 C 1 γ − 3/2 ) gives the claim.
Deduction of the informal theorem. Write A = 3 π / 8 A=3\pi/8 A = 3 π /8 and B = 27 π / 80 B=27\pi/80 B = 27 π /80 . By Equation (8) , R − A = − B / γ + O ( γ − 3 / 2 ) + o p ( 1 ) \mathcal{R}-A=-B/\gamma+O(\gamma^{-3/2})+o_p(1) R − A = − B / γ + O ( γ − 3/2 ) + o p ( 1 ) . For sufficiently large fixed γ \gamma γ , the O ( γ − 3 / 2 ) O(\gamma^{-3/2}) O ( γ − 3/2 ) term has absolute value at most B / ( 4 γ ) B/(4\gamma) B / ( 4 γ ) with probability tending to one, and the o p ( 1 ) o_p(1) o p ( 1 ) term has absolute value at most B / ( 4 γ ) B/(4\gamma) B / ( 4 γ ) with probability tending to one as d → ∞ d\to\infty d → ∞ . Consequently,
− 3 B 2 γ ≤ R − A ≤ − B 2 γ . -\frac{3B}{2\gamma}\le\mathcal{R}-A\le-\frac{B}{2\gamma}. − 2 γ 3 B ≤ R − A ≤ − 2 γ B .
with probability tending to one. This gives the claimed two-sided 1 / γ 1/\gamma 1/ γ rate, and R > 1 \mathcal{R}>1 R > 1 once γ \gamma γ is sufficiently large because A > 1 A>1 A > 1 . Together with Lemma F.33, this proves all three conclusions of Theorem 4.2 .
Lemma F.34 (Pointwise monotonicity). Suppose that for every sufficiently large γ \gamma γ the limit in probability R ∞ ( γ ) : = lim d → ∞ R \mathcal{R}_{\infty}(\gamma):=\lim_{d\to\infty}\mathcal{R} R ∞ ( γ ) := lim d → ∞ R exists, and that R ∞ \mathcal{R}_{\infty} R ∞ is analytic as a function of 1 / γ 1/\gamma 1/ γ in a neighbourhood of 0 0 0 , i.e. R ∞ ( γ ) = ∑ k ≥ 0 c k γ − k \mathcal{R}_{\infty}(\gamma)=\sum_{k\ge0}c_k\gamma^{-k} R ∞ ( γ ) = ∑ k ≥ 0 c k γ − k for all sufficiently large γ \gamma γ . Then there is γ 2 \gamma_2 γ 2 such that R ∞ ′ ( γ ) > 0 \mathcal{R}'_{\infty}(\gamma)>0 R ∞ ′ ( γ ) > 0 for γ > γ 2 \gamma>\gamma_2 γ > γ 2 .
Proof. By Equation (8) and convergence in probability, ∣ R ∞ ( γ ) − 3 π 8 ( 1 − 9 10 γ ) ∣ ≤ C 1 γ − 3 / 2 \left|\mathcal{R}_{\infty}(\gamma)-\frac{3\pi}{8}\left(1-\frac{9}{10\gamma}\right)\right|\le C_{1}\gamma^{-3/2} R ∞ ( γ ) − 8 3 π ( 1 − 10 γ 9 ) ≤ C 1 γ − 3/2 , that is
∣ ∑ k ≥ 0 c k γ − k − 3 π 8 + 27 π 80 γ − 1 ∣ ≤ C 1 γ − 3 / 2 . (86) \left|\sum_{k\geq0}c_{k}\gamma^{-k}-\frac{3\pi}{8}+\frac{27\pi}{80}\gamma^{-1}\right|\leq C_{1}\gamma^{-3/2}. \tag*{(86)} k ≥ 0 ∑ c k γ − k − 8 3 π + 80 27 π γ − 1 ≤ C 1 γ − 3/2 . (86)
Letting γ → ∞ \gamma\to\infty γ → ∞ gives c 0 = 3 π 8 c_{0}=\frac{3\pi}{8} c 0 = 8 3 π ; dividing both sides by γ − 1 \gamma^{-1} γ − 1 and letting γ → ∞ \gamma\to\infty γ → ∞ gives c 1 = − 27 π 80 c_{1}=-\frac{27\pi}{80} c 1 = − 80 27 π . Hence
R ∞ ′ ( γ ) = − ∑ k ≥ 1 k c k γ − k − 1 = γ − 2 ( 27 π 80 − ∑ k ≥ 2 k c k γ − ( k − 1 ) ) , (87) \mathcal{R}_{\infty}'(\gamma)=-\sum_{k\geq1}kc_{k}\gamma^{-k-1}=\gamma^{-2}\left(\frac{27\pi}{80}-\sum_{k\geq2}kc_{k}\gamma^{-(k-1)}\right), \tag*{(87)} R ∞ ′ ( γ ) = − k ≥ 1 ∑ k c k γ − k − 1 = γ − 2 ( 80 27 π − k ≥ 2 ∑ k c k γ − ( k − 1 ) ) , (87)
and the series in the parentheses tends to 0 as γ → ∞ \gamma\to\infty γ → ∞ , so the parenthesized quantity is positive for all sufficiently large γ \gamma γ . □
Numerical consequences. Since 3 π 8 ≈ 1.178 > 1 \frac{3\pi}{8}\approx1.178>1 8 3 π ≈ 1.178 > 1 , Equation (8) puts R \mathcal{R} R above one for all large γ \gamma γ : past some data budget the GD iterate has the larger margin-to-spread ratio. The leading term 3 π 8 ( 1 − 9 10 γ ) \frac{3\pi}{8}\left(1-\frac{9}{10\gamma}\right) 8 3 π ( 1 − 10 γ 9 ) is increasing in γ \gamma γ ; Theorem F.33 turns this into a statement about R \mathcal{R} R itself, and Theorem F.34 into one about its limit, under the hypotheses discussed in Theorem F.35. The leading term crosses one at γ ≈ 5.95 \gamma\approx5.95 γ ≈ 5.95 , below the range γ ≥ 16 \gamma\geq16 γ ≥ 16 in which Lemma F.32 is proved, so the location of the crossing is established empirically rather than by Theorem 4.2 .