betaFeedback

WorksComputer Science

How Does Orthogonalization Adapt to the Neural Network Hessian Structure? A Gradient Self Outer-Product Analysis at Initialization

Abstract

Muon orthogonalizes a weight matrix's momentum before each step, and on neural networks this simple preconditioner beats AdamW in a wide range of tasks. Existing analyses explain Muon within general optimization frameworks or on a few specific models, and many of them view orthogonalization as a preconditioner. Since the Hessian is the optimal preconditioner under a quadratic approximation of the loss, these works leave a basic question open: what makes orthogonalization suited to the Hessian of a neural network? Prior work shows that the layer-wise Hessian of a neural network is row-block diagonally dominant, and Muon's implicit preconditioner, the Kronecker product (VV⊤)1/2⊗I(\mathbf{V}\mathbf{V}^\top)^{1/2}\otimes\mathbf{I} with V\mathbf{V} the momentum, has the same row-block structure exactly when VV⊤\mathbf{V}\mathbf{V}^\top is diagonally dominant. Diagonal dominance of VV⊤\mathbf{V}\mathbf{V}^\top has been observed in language models, but why a neural network produces it has not been explained. This work provides a quantitative asymptotic theory for neural networks at initialization, and points out the intuition that the overparameterization of the weight matrices drives the diagonal dominance. We analyze E[GG⊤]\mathbb{E}[\mathbf{G}\mathbf{G}^\top], which equals E[VV⊤]\mathbb{E}[\mathbf{V}\mathbf{V}^\top] at initialization, in four canonical models: symmetric matrix factorization, deep linear networks, two-layer ReLU networks, and single-layer linear attention. In each case the diagonal entries outgrow the off-diagonal ones at explicit rates as the hidden dimensions grow; for the query, key and value weights of attention the off-diagonal vanishes altogether. Simulations of the four models and experiments on language models at initialization confirm the predicted scaling.

Introduction

Muon ([32]) optimizes a matrix-shaped weight Wt∈Rm×nW_t \in\mathbb{R}^{m \times n} by orthogonalizing the momentum buffer instead of using it directly. From a stochastic gradient GtG_t,

Vt=βVt−1+(1−β)Gt,Dt=NS⁡5(Vt)≈(VtVtT)−1/2Vt,Wt+1=Wt−ηtDt,(1)\begin{aligned} V_t &= \beta V_{t-1} + (1-\beta)G_t, \qquad D_t = \operatorname{NS}_5(V_t) \approx(V_tV_t^{\mathsf T})^{-1/2}V_t, \qquad W_{t+1} = W_t - \eta_tD_t, \tag*{(1)} \end{aligned}

with V0=G0V_0 = G_0 and NS⁡5\operatorname{NS}_5 a fixed-point iteration for the orthogonal polar factor: if Vt=PΣQTV_t = P\Sigma Q^{\mathsf T} is a thin SVD then Dt=PQT=(VtVtT)−1/2VtD_t = PQ^{\mathsf T} = (V_tV_t^{\mathsf T})^{-1/2}V_t. A growing body of work studies why Muon works within general optimization frameworks: as momentum steepest descent under a chosen operator norm ((Bernstein & Newhouse, 2024b,a; Petrich et al., 2025; Lau et al., 2025)), as a non-Euclidean trust-region step ([34]), under an isotropic curvature model [69], with an implicit spectral-norm constraint on the iterates [10], and with convergence guarantees under smoothness or Hölder smoothness, heavy-tailed noise, inexact orthogonalization, variance reduction, and nonsmooth or convex Lipschitz objectives [39, 65, 1, 35, 49, 63, 23, 27, 12, 66, 33, 59, 16, 8, 60, 40, 83, 36, 54, 42, 43 }). However, those works derive their analyses from abstract assumptions on the loss, such as smoothness or convexity, imposed uniformly over a problem class rather than from any specific model. The limitation of those works is that they cannot explain why Muon, a simple matrix preconditioner, works so well for neural networks specifically. The few theoretical results for specific models, such as matrix factorization and in-context learning with linear transformers {citep}] (Iiduka 2026)[45], linear or bilinear classifiers on separable or imbalanced data {citep}[20, 72], and random-feature and transformer blocks at initialization {citep}[14], sharpen the picture in particular regimes. Nevertheless, since the Hessian is the optimal preconditioner under a quadratic approximation of the loss, those previous works leave a basic structural question for preconditioning methods open: What makes the orthogonalization preconditioner suited to the Hessian of a neural network specifically? This paper gives a theoretical answer that Muon’s implicit preconditioner shares a similar row-block diagonally dominant heterogeneous structure to the neural-network Hessian described by {citet}Zhang et al. (2024a,b)`.

Preconditioner structure versus Hessian structure. Let us first take a view of the equivalent preconditioner form of orthogonalization. The orthogonalized update is Dt=(VtVtT)−1/2VtD_t = (V_tV_t^{\mathsf{T}})^{-1/2}V_t, a left multiplication that mixes the rows of VtV_t. Stacking the rows of DtD_t and of VtV_t into vectors turns it into vec⁡(Dt)=((VtVtT)−1/2⊗In)vec⁡(Vt)\operatorname{vec}(D_t) = ((V_tV_t^{\mathsf{T}})^{-1/2} \otimes I_n)\operatorname{vec}(V_t), so Muon descends in the metric

HMuon=Pt1/2⊗In,Pt:=VtVtT∈Rm×m,(2)H_{\mathrm{Muon}} = P_t^{1/2} \otimes I_n,\qquad P_t := V_tV_t^{\mathsf{T}} \in\mathbb{R}^{m \times m}, \tag*{(2)}

a Kronecker structure determined entirely by PtP_t. At initialization, V0=G0V_0 = G_0 and hence P0=G0G0TP_0 = G_0G_0^{\mathsf{T}}. Previous work has shown that the neural-network Hessian has a special heterogeneous structure: for a weight W∈Rm×nW \in\mathbb{R}^{m \times n}, the m×mm \times m blocks indexed by its rows (output neurons) are dominated by the diagonal ones, that is, the layer-wise Hessian is row-block diagonally dominant (Zhang et al., 2024a,b; Dong et al., 2025). Meanwhile, Deng et al. [15] point out empirically that GtGtTG_tG_t^{\mathsf{T}} is diagonally dominant throughout training, and the larger the model, the stronger the dominance. When PtP_t is close to diagonal, HMuonH_{\mathrm{Muon}} in (2) shares the row-block structure of the Hessian (Figure 1), so the question reduces to a clean property of the gradient.

Hessian and momentum self outer-product for a Transformer weight matrix

Figure 1. Left: Hessian of a Transformer weight matrix at initialization, whose mass concentrates on the diagonal blocks. Right: the momentum self outer-product VtVtTV_tV_t^{\mathsf{T}} of a Transformer weight matrix at initialization (log scale), whose mass concentrates on the diagonal.

Question. How does Neural Networks produce such a diagonally dominant VtVtTV_tV_t^{\mathsf{T}} (equivalently, GtGtTG_tG_t^{\mathsf{T}} at initialization)?

This paper answers this question for neural networks in an asymptotic regime, going a step further than the empirical observation of Deng et al. [15] by turning it into a quantitative asymptotic theoretical result at initialization. We prove that in four canonical neural-network models the diagonal of E[GGT]\mathbb{E}[GG^{\mathsf{T}}] outgrows the off-diagonal as the hidden dimensions of the weight matrices grow, at rates summarized in Table 1. The overparameterization of the weight matrices is thus a key indicator of this diagonalization. Our contributions can be summarized as follows.

ModelDiagonalOff-diagonalAsymptotic dims
Matrix factorizationΘ(k3)\Theta(k^3)Θ(k2)\Theta(k^2)kk
Deep linear, hidden layer iiΘ(di∏j=1,j≠iL−1dj2)\Theta\left(d_i\prod_{j=1,j\ne i}^{L-1}d_j^2\right)00d1,…,dL−1d_1,\ldots,d_{L-1}
Deep linear, output layerΘ(∏j=1L−1dj2)\Theta\left(\prod_{j=1}^{L-1}d_j^2\right)Θ(∏j=1L−1dj)\Theta\left(\prod_{j=1}^{L-1}d_j\right)d1,…,dL−1d_1,\ldots,d_{L-1}
Two-layer ReLU, layer W1W_1Θ(d1)\Theta(d_1)Θ(1)\Theta(1)d1d_1
Two-layer ReLU, layer W2W_2Θ(d12)\Theta(d_1^2)Θ(d1)\Theta(d_1)d1d_1
Linear attention, WQ,WKW_Q,W_KΘ(d22dv2d06/dk)\Theta(d_2^2d_v^2d_0^6/d_k)00dv,d0,d2d_v,d_0,d_2
Linear attention, WVW_VΘ(d2(d2+dv)d06)\Theta(d_2(d_2+d_v)d_0^6)00dv,d0,d2d_v,d_0,d_2
Linear attention, WOW_OΘ(d22d06)\Theta(d_2^2d_0^6)Θ(dvd03)\Theta(d_vd_0^3)dv,d0d_v,d_0

Table 1. Summary of the orders of E[GGT]\mathbb{E}[GG^{\mathsf{T}}] at a Gaussian initialization in the hidden dimensions of each model and, for attention, the token dimensions d0,d2d_0,d_2.

  1. Hidden-dimension asymptotics theory for Muon’s preconditioner. For symmetric matrix factorization, deep linear networks, two-layer ReLU networks and single-layer linear attention, we prove that Muon’s implicit preconditioner becomes diagonally dominant as the hidden dimensions grow, with explicit rates (Theorems 1 to 4 and Table 1).

  1. Empirical evaluation of the predicted scaling, from simulations to language models. Simulations of the four models reproduce the predicted rates (§ 5), and Qwen3.5, Llama-3.2 and Granite-4.0 at initialization show the dominance strengthening with model width (§ 6).

Related Work

Mechanistic analyses of Muon. Muon ([32]) replaces the momentum of each weight matrix by its polar factor. This is the momentum form of steepest descent in the spectral norm, a view developed by Bernstein & Newhouse [6]. Work on why this update suits neural networks follows two routes. The first route models the curvature of the loss. Su [69] introduces an isotropic curvature model and shows that, under a growth condition on the curvature, the optimal update flattens the spectrum of the gradient, as orthogonalization does. Davis & Drusvyatskiy [14] give a layerwise condition, comparing the nuclear-to-Frobenius ratio of the gradient with the stable rank of the incoming activations, under which one spectral step decreases the loss more than one Euclidean step. Ma et al. [45] prove that a simplified Muon converges linearly on matrix factorization and on in-context learning with linear transformers, at a rate independent of the condition number and ahead of gradient descent and Adam. Nguyen et al. [50] recast Muon as a preconditioned gradient method and show, under a Kronecker-factored curvature model, that spectral flattening improves the convergence factor. Wang et al. [77] compare the one-step loss decrease of Muon and Adam under a second-order Taylor approximation and attribute Muon’s advantage to a smaller curvature penalty, and Wu et al. [81] probe the loss landscape along real training runs, estimating the loss-optimal step size along each singular direction of the momentum. Since the isotropic geometry behind Muon only approximates the true curvature, several curvature-aware variants have followed (Zhang et al., 2026c; Du & Su, [18]; Che, [9]; Zhang et al., 2026b). The second route studies the update itself. Beneventano et al. [4] characterize the polar factor as the entropy-maximizing bounded update that keeps the singular directions of the gradient and derive its spectral dynamics. Shumaylov et al. [67] find that random or inverted spectra train equally well, and Wang et al. [76] attribute Muon’s advantage on heavy-tailed data to its more isotropic update spectrum. Zhang (2026) separate spectral control from gauge equivariance, Eschenhagen et al. [19] relate Muon to Shampoo, and Zhang et al. (2026a) ask at which granularity attention weights should be orthogonalized. Both routes take a curvature model or a landscape as given. None of them compares the structure of Muon’s implicit preconditioner with that of the network’s Hessian, which is the question we study.

Block structure of the neural-network Hessian. The Hessian of a neural network is near block diagonal, with most of its mass in dense principal blocks. This observation goes back to Collobert [13], and the same structure is assumed by the block-diagonal curvature approximations of Le Roux et al. [38] and Martens & Grosse [46]. Dong et al. [17] quantify the effect at random initialization: for linear and one-hidden-layer classifiers, the off-diagonal blocks vanish relative to the diagonal ones as the number of classes grows. The dense blocks within a layer are indexed by output neurons. This is the row-block structure we build on, and it is the partition by which Adam-mini assigns one learning rate per block to the value, projection and MLP weights (Zhang et al., 2024b). Beyond being block diagonal, the Hessian is heterogeneous across parameter blocks. Zhang et al. (2024a) show that in Transformers, unlike CNNs, the Hessian spectrum varies strongly from one weight matrix to another, so that no single learning rate serves all of them. Ormaniec et al. [52] derive the Hessian of a self-attention layer and trace this heterogeneity to the softmax and the query-key parameterization, Tomihari & Sato [70] show that gradient heterogeneity across blocks complements it, and Wang et al. [75] exploit the sharpness disparity between blocks to speed up pre-training. Within a block, Zhang et al. (2025) prove that under anisotropic inputs a layer-wise Kronecker-factored preconditioner recovers the feature-learning rate that SGD and Adam miss. Our work builds on the block-diagonal part of this picture: a preconditioner should be block diagonal in the row-block basis, and Muon’s (VVT)1/2⊗I(VV^{\mathsf{T}})^{1/2} \otimes I has this form precisely when VVTVV^{\mathsf{T}} is diagonal. Further preconditioning methods are discussed in Appendix § 9.

Preliminaries

References

  1. [1]Kang An, Yuxing Liu, Rui Pan, Yi Ren, Shiqian Ma, Donald Goldfarb, and Tong Zhang. ASGO: Adaptive structured gradient optimization. arXiv preprint arXiv:2503.20762, 2025.arxiv.org/abs/2503.20762
  2. [2]Rohan Anil, Vineet Gupta, Tomer Koren, and Yoram Singer. Memory-efficient adaptive optimization. In Advances in Neural Information Processing Systems, 2019.arxiv.org/abs/1901.11150
  3. [3]Rohan Anil, Vineet Gupta, Tomer Koren, Kevin Regan, and Yoram Singer. Scalable second order optimization for deep learning. arXiv preprint arXiv:2002.09018, 2020.arxiv.org/abs/2002.09018
  4. [4]Pierfrancesco Beneventano, Mahmoud Abdelmoneum, and Tomaso Poggio. The spectral dynamics and noise geometry of Muon. arXiv preprint arXiv:2606.08388, 2026.arxiv.org/abs/2606.08388
  5. [6]Jeremy Bernstein and Laker Newhouse. Old optimizer, new norm: An anthology. arXiv preprint arXiv:2409.20325, 2024b.arxiv.org/abs/2409.20325
  6. [7]Tom B. Brown et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020.arxiv.org/abs/2005.14165
  7. [8]Da Chang, Yongxiang Liu, and Ganzhao Yuan. On the convergence of Muon and beyond. arXiv preprint arXiv:2509.15816, 2026.arxiv.org/abs/2509.15816
  8. [9]Tong Che. Second-order Muon done right: A principled marriage of spectral geometry and curvature. arXiv preprint arXiv:2608.09763, 2026.arxiv.org/abs/2608.09763
  9. [10]Lizhang Chen, Jikai Li, and Qiang Liu. Muon optimizes under spectral norm constraints. arXiv preprint arXiv:2506.15054, 2025.arxiv.org/abs/2506.15054
  10. [11]Youngmin Cho and Lawrence K. Saul. Kernel methods for deep learning. In Advances in Neural Information Processing Systems, volume 22, pp. 342–350, 2009.
  11. [12]Sayantan Choudhury, Xiaoran Cheng, Martin Takáč, Sen Na, and Mladen Kolar. Muon with Nesterov momentum: Heavy-tailed noise and (randomized) inexact polar decomposition. arXiv preprint arXiv:2605.06884, 2026.arxiv.org/abs/2605.06884
  12. [13]Ronan Collobert. Large Scale Machine Learning. PhD thesis, Université de Paris VI, 2004.
  13. [14]Damek Davis and Dmitry Drusvyatskiy. When do spectral gradient updates help in deep learning? arXiv preprint arXiv:2512.04299, 2025.arxiv.org/abs/2512.04299
  14. [15]Shenyang Deng, Zhuoli Ouyang, Tianyu Pang, Zihang Liu, Ruoch en Jin, Shuhua Yu, and Yaoqing Yang. RMNP: Row-momentum normalized preconditioning for scalable matrix-based optimization. arXiv preprint arXiv:2603.20527, 2026.arxiv.org/abs/2603.20527
  15. [16]Thang Do, Steffen Dereich, and Arnulf Jentzen. On muon optimization: From non-convergence to an error analysis with polar express and the Newton-Schulz polynomial from implementations. arXiv preprint arXiv:2608.04607, 2026.
  16. [17]Zhaorui Dong, Yushun Zhang, Zhi-Quan Luo, Jianfeng Yao, and Rouyu Sun. Towards quantifying the hessian structure of neural networks, 2025. URL https://arxiv.org/abs/2505.02809.
  17. [18]Zhehang Du and Weijie Su. The Newton-Muon optimizer. arXiv preprint arXiv:2604.01472, 2026.arxiv.org/abs/2604.01472
  18. [19]Runa Eschenhagen, Anna Cai, Tsung-Hsien Lee, and Hao-Jun Michael Shi. Clarifying shampoo: Adapting spectral descent to stochasticity and the parameter trajectory. arXiv preprint arXiv:2602.09314, 2026.arxiv.org/abs/2602.09314
  19. [20]Chen Fan, Mark Schmidt, and Christos Thrampoulidis. Implicit bias of spectral descent and Muon on multiclass separable data. arXiv preprint arXiv:2502.04664, 2025.arxiv.org/abs/2502.04664
  20. [21]Athanasios Glentis, Jiaxiang Li, Andi Han, and Mingyi Hong. A minimalist optimizer design for LLM pretraining. arXiv preprint arXiv:2506.16659, 2025.
  21. [22]Aaron Grattafiori et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.arxiv.org/abs/2407.21783
  22. [23]S. Gratton and Ph. L. Toint. A unified convergence theory for adaptive first-order methods in the nonconvex case, including adanorm, full and diagonal adagrad and Muon. arXiv preprint arXiv:2604.17423, 2026.arxiv.org/abs/2604.17423
  23. [25]Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization. arXiv preprint arXiv:1802.09568, 2018.arxiv.org/abs/1802.09568
  24. [26]Florian Hübler, Kai Lion, Antonio Orvieto, and Niao He. Muown implicitly performs angular step-size decay. arXiv preprint arXiv:2606.23637, 2026a.arxiv.org/abs/2606.23637
  25. [27]Florian Hübler, Thomas Pethick, and Suvrit Sra. Free heavy-tailed lunch for Muon: A theoretical justification of empirical success. arXiv preprint arXiv:2606.14560, 2026b.arxiv.org/abs/2606.14560
  26. [28]IBM Granite Team. Granite-4.0-1b-base. https://huggingface.co/ibm-granite/granite-4.0-1b-base, 2025. Model card.
  27. [30]Leon Isserlis. On a formula for the product-moment coefficient of any order of a normal frequency distribution in any number of variables. Biometrika, 12(1–2):134–139, 1918.
  28. [31]Svante Janson. Gaussian Hilbert Spaces. Cambridge University Press, 1997.DOI
  29. [32]Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. https://kellerjordan.github.io/posts/muon/.
  30. [33]Gyu Yeol Kim and Min hwan Oh. Convergence of Muon with Newton-Schulz. arXiv preprint arXiv:2601.19156, 2026.arxiv.org/abs/2601.19156
  31. [34]Dmitry Kovalev. Understanding gradient orthogonalization for deep learning via non-Euclidean trust-region optimization. arXiv preprint arXiv:2503.12645, 2025.arxiv.org/abs/2503.12645
  32. [35]Dmitry Kovalev and Ekaterina Borodich. Non-Euclidean SGD for structured optimization: Unified analysis and improved rates. arXiv preprint arXiv:2511.11466, 2026.arxiv.org/abs/2511.11466
  33. [36]Lexiao Lai, Tianyi Lin, and Jiayu Zhang. Nonsmooth optimization via orthogonalized momentum. arXiv preprint arXiv:2609.13677, 2026.arxiv.org/abs/2609.13677
  34. [37]Tim Tsz-Kit Lau, Qi Long, and Weijie Su. PolarGrad: A class of matrix-gradient optimizers from a unifying preconditioning perspective. arXiv preprint arXiv:2505.21799, 2025.arxiv.org/abs/2505.21799
  35. [38]Nicolas Le Roux, Pierre-Antoine Manzagol, and Yoshua Bengio. Topmoumoute online natural gradient algorithm. In Advances in Neural Information Processing Systems 20, 2008.
  36. [39]Jiaxiang Li and Mingyi Hong. A note on the convergence of Muon. arXiv preprint arXiv:2502.02900, 2025.arxiv.org/abs/2502.02900
  37. [40]Mingyi Li and Taira Tsuchiya. Muon with finite Newton-Schulz: The smoothing benefit in nonsmooth nonconvex optimization. arXiv preprint arXiv:2608.26288, 2026.arxiv.org/abs/2608.26288
  38. [41]Kai Lion, Florian Hübler, Bingcong Li, Antonio Orvieto, and Niao He. Muown: Row-norm control for Muon optimization. arXiv preprint arXiv:2605.10797, 2026.arxiv.org/abs/2605.10797
  39. [42]Jinlin Liu, Song Chen, Jiaxu Liu, and Chao Xu. A continuous-time analysis of smoothed matrix-polar spectral gradient flows for Muon-type optimization. arXiv preprint arXiv:2608.01911, 2026.arxiv.org/abs/2608.01911
  40. [43]Yanlin Long, Yufei Gu, and Zeke Xie. Dimension-free saddle-point escape in Muon. arXiv preprint arXiv:2605.09331, 2026.arxiv.org/abs/2605.09331
  41. [44]Chao Ma, Wenbo Gong, Meyer Scetbon, and Edward Meeds. SWAN: SGD with normalization and whitening enables stateless LLM training. In Proceedings of the 42nd International Conference on Machine Learning (ICML), 2025.arxiv.org/abs/2412.13148
  42. [45]Jianhao Ma, Yu Huang, Yuejie Chi, and Yuxin Chen. Preconditioning benefits of spectral orthogonalization in muon, 2026. URL https://arxiv.org/abs/2601.13474.
  43. [46]James Martens and Roger Grosse. Optimizing neural networks with Kronecker-factored approximate curvature. arXiv preprint arXiv:1503.05671, 2015.arxiv.org/abs/1503.05671
  44. [47]Meta AI. Llama 3.2 1B. https://huggingface.co/meta-llama/Llama-3.2-1B, 2024. Model card.
  45. [48]Depen Morwani, Itai Shapira, Nikhil Vyas, Eran Malach, Sham Kakade, and Lucas Janson. A new perspective on Shampoo’s preconditioner. arXiv preprint arXiv:2406.17748, 2024.arxiv.org/abs/2406.17748
  46. [49]Shuntaro Nagashima and Hideaki Iiduka. Improved convergence rates of Muon optimizer for nonconvex optimization. arXiv preprint arXiv:2601.19400, 2026.arxiv.org/abs/2601.19400
  47. [50]Tien-Phat Nguyen, Truong Nguyen, Minh-Phuc Truong, Tuc Nguyen, James Bailey, and Trung Le. Spectral flattening is all Muon needs: How orthogonalization controls learning rate and convergence. arXiv preprint arXiv:2605.13079, 2026.arxiv.org/abs/2605.13079
  48. [51]Ivan Nourdin and Giovanni Peccati. Normal Approximations with Malliavin Calculus: From Stein’s Method to Universality. Cambridge University Press, 2012.DOI
  49. [52]Weronika Ormaniec, Felix Dangel, and Sidak Pal Singh. What does it mean to be a Transformer? insights from a theoretical Hessian analysis. arXiv preprint arXiv:2410.10986, 2024.
  50. [53]Tianyu Pang, Yujie Fang, Zihang Liu, Shenyang Deng, Lei Hsiung, Shuhua Yu, and Yaoqing Yang. Htmuon: Improving muon via heavy-tailed spectral correction, 2026. URL https://arxiv.org/abs/2603.10067.
  51. [54]Tetiana Parshakova, Ahmed Khaled, Michael Crawshaw, Guillaume Garrigos, and Robert M. Gower. Muon does not converge on convex Lipschitz functions. arXiv preprint arXiv:2605.08980, 2026.
  52. [55]Guilherme Penedo, Hynek Kydlíček, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. In Advances in Neural Information Processing Systems, 2024.arxiv.org/abs/2406.17557
  53. [56]Thomas Pethick. When to use what Schatten-p norm in deep learning? arXiv preprint arXiv:2606.15268, 2026.
  54. [57]Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos, Zhenyu Zhu, Antonio Silveti-Falls, and Volkan Cevher. Training deep learning models with norm-constrained LMOs. In Proceedings of the 42nd International Conference on Machine Learning (ICML), 2025.
  55. [58]Robert Price. A useful theorem for nonlinear devices having Gaussian inputs. IRE Transactions on Information Theory, IT-4(2):69–72, 1958.
  56. [59]Xun Qian and Peter Richtárik. Convergence analysis of Muon-type methods with inexact lmo in the degenerate case. arXiv preprint arXiv:2606.21581, 2026.
  57. [60]Xun Qian, Hussein Rammal, Dmitry Kovalev, and Peter Richtárik. Muon is provably faster with momentum variance reduction. arXiv preprint arXiv:2512.16598, 2025.
  58. [61]Qwen Team. Qwen3.5: Towards native multimodal agents. https://huggingface.co/Qwen/Qwen3.5-0.8B-Base, February 2026. Qwen3.5-0.8B-Base model card.
  59. [62]Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI technical report, 2019.
  60. [63]Naoki Sato, Hiroki Naganuma, and Hideaki Iiduka. Convergence bound and critical batch size of Muon optimizer. arXiv preprint arXiv:2507.01598, 2026.
  61. [64]Meyer Scetbon, Chao Ma, Wenbo Gong, and Edward Meeds. Gradient multi-normalization for stateless and scalable LLM training. arXiv preprint arXiv:2502.06742, 2025.
  62. [65]Wei Shen, Ruichuan Huang, Minhui Huang, Cong Shen, and Jiawei Zhang. On the convergence analysis of Muon. arXiv preprint arXiv:2505.23737, 2025.arxiv.org/abs/2505.23737
  63. [66]Egor Shulgin, Sultan AlRashed, Francesco Orabona, and Peter Richtárik. Beyond the ideal: Analyzing the inexact Muon update. arXiv preprint arXiv:2510.19933, 2025.arxiv.org/abs/2510.19933
  64. [67]Zakhar Shumaylov, Nathaël Da Costa, Peter Zaika, Bálint Mucsányi, Alex Massucco, Yoav Gelberg, Carola-Bibiane Schönlieb, Yarin Gal, and Philipp Hennig. Muon is not that special: Random or inverted spectra work just as well. arXiv preprint arXiv:2605.11181, 2026.arxiv.org/abs/2605.11181
  65. [68]Charles M. Stein. Estimation of the mean of a multivariate normal distribution. The Annals of Statistics, 9(6): 1135–1151, 1981.DOI
  66. [69]Weijie Su. Isotropic curvature model for understanding deep learning optimization: Is gradient orthogonalization optimal? arXiv preprint arXiv:2511.00674, 2025.arxiv.org/abs/2511.00674
  67. [70]Akiyoshi Tomihari and Issei Sato. Gradient heterogeneity complements Hessian heterogeneity in Transformer optimization. arXiv preprint arXiv:2502.00213, 2025.arxiv.org/abs/2502.00213
  68. [71]Hugo Touvron et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.arxiv.org/abs/2302.13971
  69. [72]Bhavya Vasudeva, Puneesh Deora, Yize Zhao, Vatsal Sharan, and Christos Thampoulidis. How Muon’s spectral design benefits generalization: A study on imbalanced data. arXiv preprint arXiv:2510.22980, 2025.arxiv.org/abs/2510.22980
  70. [73]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.arxiv.org/abs/1706.03762
  71. [74]Nikhil Vyas, Depen Morwani, Rosie Zhao, Mujin Kwun, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham Kakade. SOAP: Improving and stabilizing Shampoo using Adam. arXiv preprint arXiv:2409.11321, 2024.arxiv.org/abs/2409.11321
  72. [75]Jinbo Wang, Mingze Wang, Zhanpeng Zhou, Junchi Yan, Weinan E, and Lei Wu. The sharpness disparity principle in Transformers for accelerating language model pre-training. arXiv preprint arXiv:2502.19002, 2025a.arxiv.org/abs/2502.19002
  73. [76]Shuche Wang, Fengzhuo Zhang, Jiaxiang Li, Cunxiao Du, Chao Du, Tianyu Pang, Zhuoran Yang, Mingyi Hong, and Vincent Y. F. Tan. Muon outperforms Adam in tail-end associative memory learning. arXiv preprint arXiv:2509.26030, 2025b.arxiv.org/abs/2509.26030
  74. [77]Shuche Wang, Fengzhuo Zhang, Jiaxiang Li, Dirk Bergemann, and Zhuoran Yang. Why Muon outperforms Adam: A curvature perspective. arXiv preprint arXiv:2606.04662, 2026.arxiv.org/abs/2606.04662
  75. [78]Ziqing Wen, Yanqi Shi, Jiah̄uan Wang, Ping Luo, Linbo Qiao, Dongsheng Li, and Tao Sun. SRON: State-free LLM training via row-wise gradient normalization. In OpenReview preprint, 2025. https://openreview.net/forum?id=BtQLBWr6zI.
  76. [79]Gian-Carlo Wick. The evaluation of the collision matrix. Physical Review, 80(2):268–272, 1950.DOI
  77. [80]Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38–45. Association for Computational Linguistics, 2020. doi: 10.18653/v1/2020.emnlp-demos.6.DOI
  78. [81]Xiaodong Wu, Wenyi Yu, Chao Zhang, and Philip Woodland. Spectral allocation: Why Muon outperforms Adam, and how to improve Muon. arXiv preprint arXiv:2608.25990, 2026.arxiv.org/abs/2608.25990
  79. [82]Ruihan Xu, Jiajin Li, and Yiping Lu. On the width scaling of neural optimizers under matrix operator norms I: Row/column normalization and hyperparameter transfer. arXiv preprint arXiv:2603.09952, 2026.arxiv.org/abs/2603.09952
  80. [83]Yixuan Yang, Yuqing He, and Song Li. Convergence of spectral descent for non-smooth optimization. arXiv preprint arXiv:2605.26977, 2026.arxiv.org/abs/2605.26977

Paper details

vlnc:W_h8mw96tw
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents