Muon, a 2024-born optimizer for language models, differs from the 2015-born spectral gradient descent mainly in its fast, GPU-friendly Newton–Schulz polynomial for orthogonalizing the gradient matrix. Accordingly, a line of research proposes faster polynomial iterations for gradient orthogonalization, such as Polar Express and Gram Newton–Schulz, both built on quintic polynomials. In this paper, we introduce Polar Switch, a new approach to polynomial gradient orthogonalization. Using simple polynomial circuits, Polar Switch reaches a given high polynomial degree at a lower FLOP cost than quintic iterations. We define a circuit as a short sequence of symmetric matrix products on a few square buffers, and circuits with k products form a restricted family of odd polynomials of degree up to 2k+1+1. We choose the sequence of products in each circuit, their coefficients, and how the circuits are composed by numerically minimizing a worst-case bound on the polar approximation error. This minimization is subject to a FLOP budget that accounts for symmetric matrix multiplication kernels and a rounding constraint for numerical stability in half precision. Our two recommended Polar Switch methods have 15.5–18.7% lower runtime than Polar Express on square matrices and 8.6–16.2% lower runtime than Gram Newton–Schulz on rectangular matrices with aspect ratios 3.25, 3.5, and 8, with comparable approximation accuracy. In pretraining Modded-NanoGPT #91, Qwen3-0.6B, and Gemma 3-1B, Polar Switch reaches final validation losses similar to those of established orthogonalization methods. Across the more than 14 million matrices that each of the two recommended methods orthogonalized in these runs, the estimated output spectral norms never exceed the designed bounds by more than 3%, demonstrating their numerical stability.
Introduction
For years, Adam [3] and AdamW [4] have been the go-to optimizers for training large language models (LLMs) [5]. Both scale the momentum coordinatewise, without explicitly accounting for the matrix structure of the parameters. Muon [6] takes a different approach in that it uses the matrix structure of the momentum and approximates its polar factor to form the update. Studies have shown that Muon can reach the same training loss as AdamW with less computation [7, 8], and it has since been used to train several frontier open-weight LLMs, including Kimi K3 [9], GLM-5 [10], Step 3.5 Flash [11], DeepSeek-V4 [12], and Qwen3.8-Flash-Next [13]. This advantage comes with additional computation at each optimizer step, because Muon must approximate the polar factor. The question is therefore how to compute this approximation accurately and efficiently.
Let W∈Rn×m be a matrix-shaped parameter, ∇W its gradient, and M its momentum matrix. Without loss of generality, let n≤m, because matrices with more rows than columns are handled by transposition. With momentum coefficient μ and learning rate η, a simplified Muon update is
M←μM+∇W,W←W−ηpolar(M).
If M=UΣVT is the compact singular value decomposition (SVD), then polar(M)=UVT[14]. However, computing an SVD at every optimizer step is expensive for LLM training. Muon approximates the polar factor using polynomial iterations inspired by Newton–Schulz [6]. First, we normalize M as X←M/(∥M∥F+ϵ) with a small ϵ>0 (10−7 in our implementation), so that all singular values of X lie in [0,1]. At iteration t, we choose a quintic polynomial pt(x)=ctx+btx3+atx5 with coefficients at,bt,ct and compute
A←XXT,Z←btA+atA2,X←ZX+ctX.(1)
Let X=UΣVT be the compact SVD of X. The update is X←U(ctΣ+btΣ3+atΣ5)VT. Suppose that, at each iteration t, pt(σ)≥0 for every singular value σ of the current matrix X. The iteration replaces each singular value σ by pt(σ). After T iterations, each initial singular value σ becomes (pT∘⋯∘p1)(σ). Approximating the polar factor in Muon therefore requires this composition to approximate one on the positive initial singular values.
The classical Newton–Schulz iteration uses pt(x)=(3x−x3)/2, whose repeated application converges to one for 0<x≤1[14]. Muon uses Jordan’s fixed quintic pt(x)=3.4445x−4.775x3+2.0315x5[6]. Polar Express (PE) uses minimax approximation to choose a different quintic polynomial at each iteration, minimizing its maximum deviation from one over the current singular value interval [15]. Jordan’s method and Polar Express differ in their coefficients, but both use quintic iterations. We therefore ask the following question:
Under a fixed FLOP budget for matrix orthogonalization, do repeated quintic iterations yield the most accurate polar approximation?
We answer this question negatively with Polar Switch, which replaces repeated quintic iterations with a few high-degree polynomial circuits. Each circuit is a short sequence of symmetric matrix products on a small number of square buffers. For example, the first circuit of Polar Switch 3-55 (PS3-55) uses three square buffers B1,B2,B3∈Rn×n, with the coefficients aτ,bτ,cτ given in Algorithm A.1:
Each squaring doubles the degree in XXT, to 2, 4, 8, 16, and 32, so the circuit applies an odd polynomial of degree 65 to the singular values. Polar Switch searches numerically for the circuits and their coefficients that approximate the polar factor most accurately under a FLOP budget. Table 1 compares Polar Switch with the quintic baselines. Polar Switch 2-55 (PS2-55) and PS3-55 use fewer FLOPs than Polar Express and Gram Newton–Schulz and less memory than Gram Newton–Schulz.
Table 1. Memory in matrix elements, FLOPs, and final singular value intervals for X∈Rn×m with n≤m and α=m/n. In exact arithmetic, each method maps singular values in [0.001, 1] into its final interval. Jordan and Gram Newton–Schulz multiply the output of their final iteration by a constant that centers this interval at one. Algorithm A.1 gives the iterations of Polar Switch.
We make three contributions. (i) We introduce Polar Switch, a FLOP-efficient orthogonalization method for Muon based on high-degree polynomial circuits. (ii) We develop efficient GPU implementations of Polar Switch with two or three square buffers. Each kernel combines a matrix product and its additions before one float16 store, and we select coefficients that try to keep this half-precision computation numerically stable. (iii) Across three LLM pretraining setups, Polar Switch reaches final validation losses similar to those of established orthogonalization methods.
The remainder of the paper is structured as follows. § 2 reviews related work. § 3 presents Polar Switch, explains how we design its circuits and coefficients under a FLOP budget, and describes its GPU implementation. § 4 compares Polar Switch with quintic baselines in LLM pretraining, runtime, and approximation accuracy. Appendix § 8 gives the theory, the numerical search, and the coefficients of Polar Switch. Appendix § 9 gives the detailed experimental setup for LLM pretraining and the runtime and approximation accuracy experiments.
Related Work
Polar approximation and polynomial design. Polynomial polar iterations [17] and matrix sign methods [14, 18] are used in eigenvalue algorithms [19, 20], control, and model reduction [21]. Rational alternatives include sign iterations [22], QR-based dynamically weighted Halley (QDWH) [23], and Zolotarev approximants [24]. Polar iterations also have backward stability analyses [25]. Broader work covers matrix roots [26, 27, 28, 29, 30], fractional powers [31], general matrix function computation [32, 33, 34, 35, 36], and manifold retractions using projected polynomials [37]. Minimax approximation minimizes the largest error over an interval, with solutions characterized by equioscillation [38] and computed by Remez methods [39]. Related problems include polynomial sign approximation [40], rational minimax [41] and AAA approximation [42], and Parks–McClellan filter design [43, 44]. Paterson–Stockmeyer [45] and related methods [46] reduce the number of matrix multiplications. Other work characterizes which polynomials can be evaluated with a fixed number of products [47]. Chebyshev acceleration [48] constructs odd polynomials with large derivatives at zero to increase small singular values more quickly. Polar Express [15] proves that greedy minimax fitting is optimal for a prescribed degree sequence, and its implementation uses quintics. Dynamic programming has also been used to select minimax compositions for homomorphic comparison under multiplication and depth constraints [49]. Polar Switch selects high-degree circuits under FLOP and buffer budgets that account for symmetric GPU kernels and adds a half-precision rounding constraint for numerical stability.
Efficient Muon orthogonalization. Equivalent formulas can round differently because their intermediate computations differ [50], and small singular values are sensitive to reduced precision [51]. Mixed precision is studied in linear algebra [52, 53, 54] and training [55, 56]. Gram Newton–Schulz [16] accumulates square multipliers between rectangular updates, and Dion3 [57] uses it for distributed training. AMO [58] adapts iteration counts, Dion [59] uses low-rank updates, Dion2 [60] selects rows or columns, Hierarchical Muon [61] uses tiles, and CacheMuon [62] reuses transforms. Turbo-Muon [63] uses an almost-orthogonal preconditioner for faster updates, and PRISM [64] computes matrix functions adaptively. IFNSO [65] replaces the iterations with one polynomial in selected matrix powers and learns its coefficients. A cubic Newton–Schulz schedule [66] uses ten instead of fifteen dominant matrix products and reaches a similar validation loss. MuonBP [67], CMuon [68], and DMuon [69] support distributed computation. Polar Switch builds on polynomial Newton–Schulz iterations with efficient GPU implementations.
Muon and matrix optimization.
References
[1]David Carlson, Volkan Cevher, and Lawrence Carin. Stochastic spectral descent for restricted Boltzmann machines. In Guy Lebanon and S. V. N. Vishwanathan, editors, Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics, volume 38 of Proceedings of Machine Learning Research, pages 111–119. PMLR, 2015.
[2]David Carlson, Edo Collins, Ya-Ping Hsieh, Lawrence Carin, and Volkan Cevher. Preconditioned spectral descent for deep learning. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
[3]Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.arxiv.org/abs/1412.6980
[4]Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
[5]Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, G. S. Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsel, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc., 2020.arxiv.org/abs/2005.14165
[6]Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https://kellerjordan.github.io/posts/muon/.
[8]Essential AI, Ishaan Shah, Anthony M. Polloreno, Karl Stratos, Philip Monk, Adarsh Chaluvaaraju, Andrew Hojel, Andrew Ma, Anil Thomas, Ashish Tanwer, Darsh J Shah, Khoi Nguyen, Kurt Smith, Michael Callahan, Michael Pust, Mohit Parmar, Peter Rushton, Platon Mazarakis, Ritvik Kapila, Saurabh Srivastava, Somanshu Singla, Tim Romanski, Yash Vanjani, and Ashish Vaswani. Practical efficiency of Muon for pretraining. arXiv preprint arXiv:2505.02222, 2025.arxiv.org/abs/2505.02222
[9]Kimi Team. Kimi K3: Open frontier intelligence. arXiv preprint arXiv:2607.24653, 2026.
[10]GLM-5 Team. GLM-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763, 2026.
[11]StepFun Team. Step 3.5 Flash: Open frontier-level intelligence with 11B active parameters. arXiv preprint arXiv:2602.10604, 2026.
[13]Zihan Qiu, Zekun Wang, Xiao Li, Yanpeng Li, Yang Xu, Yixuan Wang, Huaqing Zhang, Rui Men, Bochao Mao, Chengruidong Zhang, Fan Zhou, Hao Luo, Haofeng Huang, Haoran Lian, Haoyan Huang, Hongqing Chen, Jianwei Zhang, Jing Xu, Junjie Wang, Langshi Chen, Liangyu Wang, Linlang Jiang, Man Yuan, Minmin Sun, Peng Jin, Siqi Zhang, Siyu Wang, Xingzhang Ren, Yakai Wang, Yi Zhang, Yiming Dong, Yizhong Cao, Yubo Ma, Yunfei Mao, Bo Zheng, and Dayiheng Liu. On the design of Qwen3.8-Next architecture: Evaluation, efficiency, and training stability. arXiv preprint arXiv:2608.30320, 2026.arxiv.org/abs/2608.30320
[14]Nicholas J. Higham. Functions of Matrices: Theory and Computation. Society for Industrial and Applied Mathematics, 2008. ISBN 9780898717778. doi: 10.1137/1.9780898717778.DOI
[15]Noah Amsel, David Persson, Christopher Musco, and Robert M. Gower. The Polar Express: Optimal matrix sign methods and their application to the Muon algorithm. In C. Vondrick, B. Hariharan, C. Raffel, L. Pinto, D. Yang, and A. Faust, editors, International Conference on Learning Representations, volume 2026, pages 138323–138360, 2026.arxiv.org/abs/2505.16932
[16]Jack Zhang, Noah Amsel, Berlin Chen, and Tri Dao. Gram Newton–Schulz, 2026. URL https://dao-ai-lab.github.io/blog/2026/gram-newton-schulz/.
[17]Å. Björck and C. Bowie. An iterative algorithm for computing the best estimate of an orthogonal matrix. SIAM Journal on Numerical Analysis, 8(2):358–364, 1971. ISSN 1095-7170. doi: 10.1137/0708036.DOI
[18]Nicholas J. Higham. Computing the polar decomposition—with applications. SIAM Journal on Scientific and Statistical Computing, 7(4):1160–1174, 1986. ISSN 2168-3417. doi: 10.1137/0907079.DOI
[19]Eugene D. Denman and Alex N. Beavers. The matrix sign function and computations in systems. Applied Mathematics and Computation, 2(1):63–94, 1976. ISSN 0096-3003. doi: 10.1016/0096-3003(76)90020-5.DOI
[20]Yuji Nakatsukasa and Nicholas J. Higham. Stable and efficient spectral divide and conquer algorithms for the symmetric eigenvalue decomposition and the SVD. SIAM Journal on Scientific Computing, 35(3):A1325–A1349, 2013. ISSN 1095-7197. doi: 10.1137/120876605.DOI
[21]J. D. Roberts. Linear model reduction and solution of the algebraic Riccati equation by use of the sign function. International Journal of Control, 32(4):677–687, 1980. ISSN 1366-5820. doi: 10.1080/00207178008922881.DOI
[22]Charles Kenney and Alan J. Laub. Rational iterative methods for the matrix sign function. SIAM Journal on Matrix Analysis and Applications, 12(2):273–291, 1991. ISSN 1095-7162. doi: 10.1137/0612020.DOI
[23]Yuji Nakatsukasa, Zhaojun Bai, and François Gygi. Optimizing Halley’s iteration for computing the matrix polar decomposition. SIAM Journal on Matrix Analysis and Applications, 31(5):2700–2720, 2010. ISSN 1095-7162. doi: 10.1137/090774999.DOI
[24]Yuji Nakatsukasa and Roland W. Freund. Computing fundamental matrix decompositions accurately via the matrix sign function in two iterations: The power of Zolotarev’s functions. SIAM Review, 58(3):461–493, 2016. ISSN 1095-7200. doi: 10.1137/140990334.DOI
[25]Yuji Nakatsukasa and Nicholas J. Higham. Backward stability of iterations for computing the polar decomposition. SIAM Journal on Matrix Analysis and Applications, 33(2):460–479, 2012. ISSN 1095-7162. doi: 10.1137/110857544.DOI
[26]Nicholas J. Higham. Computing real square roots of a real matrix. Linear Algebra and its Applications, 88-89:405–430, 1987. ISSN 0024-3795. doi: 10.1016/0024-3795(87)90118-2.DOI
[27]Nicholas J. Higham. Stable iterations for the matrix square root. Numerical Algorithms, 15(2):227–242, 1997. ISSN 1572-9265. doi: 10.1023/a:1019150005407.DOI
[28]Edvin Deadman, Nicholas J. Higham, and Rui Ralha. Blocked Schur Algorithms for Computing the Matrix Square Root, pages 171–182. Springer Berlin Heidelberg, 2013. ISBN 9783642368035. doi: 10.1007/978-3-642-36803-5_12.DOI
[29]Dario A. Bini, Nicholas J. Higham, and Beatrice Meini. Algorithms for the matrix pth root. Numerical Algorithms, 39(4):349–378, 2005. ISSN 1572-9265. doi: 10.1007/s11075-004-6709-8.DOI
[30]Bruno Iannazzo. On the Newton method for the matrix pth root. SIAM Journal on Matrix Analysis and Applications, 28(2):503–523, 2006. ISSN 1095-7162. doi: 10.1137/050624790.DOI
[31]Nicholas J. Higham and Lijing Lin. A Schur–Padé algorithm for fractional powers of a matrix. SIAM Journal on Matrix Analysis and Applications, 32(3):1056–1078, 2011. ISSN 1095-7162. doi: 10.1137/10081232x.DOI
[32]Philip I. Davies and Nicholas J. Higham. A Schur–Parlett algorithm for computing matrix functions. SIAM Journal on Matrix Analysis and Applications, 25(2):464–485, 2003. ISSN 1095-7162. doi: 10.1137/S0895479802410815.DOI
[33]Nicholas Hale, Nicholas J. Higham, and Lloyd N. Trefethen. Computing Aᵅ, log(A), and related matrix functions by contour integrals. SIAM Journal on Numerical Analysis, 46(5):2505–2523, 2008. ISSN 1095-7170. doi: 10.1137/070700607.DOI
[34]Stefan Gittel. Rational Krylov approximation of matrix functions: Numerical methods and optimal pole selection. GAMM-Mitteilungen, 36(1):8–31, 2013. ISSN 1522-2608. doi: 10.1002/gamm.201310002.
[35]Nicholas J. Higham. The scaling and squaring method for the matrix exponential revisited. SIAM Journal on Matrix Analysis and Applications, 26(4):1179–1193, 2005. ISSN 1095-7162. doi: 10.1137/04061101x.DOI
[36]Awad H. Al-Mohy and Nicholas J. Higham. Computing the action of the matrix exponential, with an application to exponential integrators. SIAM Journal on Scientific Computing, 33(2):488–511, 2011. ISSN 1095-7197. doi: 10.1137/100788860.DOI
[37]Evan S. Gawlik and Melvin Leok. High-order retractions on matrix manifolds using projected polynomials. SIAM Journal on Matrix Analysis and Applications, 39(2):801–828, 2018. ISSN 1095-7162. doi: 10.1137/17m1130459.DOI
[38]Lloyd N. Trefethen. Approximation Theory and Approximation Practice, Extended Edition. Society for Industrial and Applied Mathematics, 2019. ISBN 9781611975949. doi: 10.1137/1.9781611975949 949.
[39]Ricardo Pachón and Lloyd N. Trefethen. Barycentric-Remez algorithms for best polynomial approximation in the chebfun system. BIT Numerical Mathematics, 49(4):721–741, 2009. ISSN 1572-9125. doi: 10.1007/s10543-009-0240-1.DOI
[40]Alexandre Eremenko and Peter Yuditskii. Uniform approximation of sgn x by polynomials and entire functions. Journal d’Analyse Mathématique, 101(1):313–324, 2007. ISSN 1565-8538. doi: 10.1007/s11854-007-0011-3.DOI
[41]Silviu-Ioan Filip, Yuji Nakatsukasa, Lloyd N. Trefethen, and Bernhard Beckermann. Rational minimax approximation via adaptive barycentric representations. SIAM Journal on Scientific Computing, 40(4):A2427–A2455, 2018. ISSN 1095-7197. doi: 10.1137/17m1132409.DOI
[42]Yuji Nakatsukasa, Olivier Sète, and Lloyd N. Trefethen. The AAA algorithm for rational approximation. SIAM Journal on Scientific Computing, 40(3):A1494–A1522, 2018. ISSN 1095-7197. doi: 10.1137/16m1106122.DOI
[43]T. Parks and J. McClellan. Chebyshev approximation for nonrecursive digital filters with linear phase. IEEE Transactions on Circuit Theory, 19(2):189–194, 1972. ISSN 0018-9324. doi: 10.1109/tct.1972.1083419.DOI
[44]J. McClellan, T. Parks, and L. Rabiner. A computer program for designing optimum FIR linear phase digital filters. IEEE Transactions on Audio and Electroacoustics, 21(6):506–526, 1973. ISSN 0018-9278. doi: 10.1109/tau.1973.1162525.DOI
[45]Michael S. Paterson and Larry J. Stockmeyer. On the number of nonscalar multiplications necessary to evaluate polynomials. SIAM Journal on Computing, 2(1):60–66, 1973. ISSN 1095-7111. doi: 10.1137/0202007.DOI
[46]J. Sastre. Efficient evaluation of matrix polynomials. Linear Algebra and its Applications, 539:229–250, 2018. ISSN 0024-3795. doi: 10.1016/j.laa.2017.11.010.DOI
[47]Elias Jarlebring and Gustaf Lorentzon. The polynomial set associated with a fixed number of matrix-matrix multiplications. SIAM Journal on Applied Algebra and Geometry, 10(3):549–581, 2026. ISSN 2470-6566. doi: 10.1137/25m1779036.DOI
[48]Ekaterina Grishina, Matvey Smirnov, and Maxim Rakhuba. Accelerating Newton–Schulz iteration for orthogonalization via Chebyshev-type polynomials. arXiv preprint arXiv:2506.10935, 2026.arxiv.org/abs/2506.10935
[49]Eunsang Lee, Joon-Woo Lee, Jong-Seon No, and Young-Sik Kim. Minimax approximation of sign function by composite polynomial for homomorphic comparison. IEEE Transactions on Dependable and Secure Computing, 19(6):3711–3727, 2022. ISSN 2160-9209. doi: 10.1109/tdsc.2021.3105111.DOI
[50]Nicholas J. Higham. Accuracy and Stability of Numerical Algorithms: Second Edition. Society for Industrial and Applied Mathematics, 2002. ISBN 9780898718027. doi: 10.1137/1.9780898718027.
[51]Christos Boutsikas, Petros Drineas, and Ilse C. F. Ipsen. Small singular values can increase in lower precision. SIAM Journal on Matrix Analysis and Applications, 45(3):1518–1540, 2024. ISSN 1095-7162. doi: 10.1137/23m1557209.DOI
[52]Marc Baboulin, Alfredo Buttari, Jack Dongarra, Jakub Kurzak, Julie Langou, Julien Langou, Piotr Luszczek, and Stanimire Tomov. Accelerating scientific computations with mixed precision algorithms. Computer Physics Communications, 180(12):2526–2533, 2009. ISSN 0010-4655. doi: 10.1016/j.cpc.2008.11.005.DOI
[53]Erin Carson and Nicholas J. Higham. A new analysis of iterative refinement and its application to accurate solution of ill-conditioned sparse linear systems. SIAM Journal on Scientific Computing, 39(6):A2834–A2856, 2017. ISSN 1095-7197. doi: 10.1137/17m1122918.DOI
[54]Ahmad Abdelfattah, Hartwig Anzt, Erik G Boman, Erin Carson, Terry Cojean, Jack Dongarra, Alyson Fox, Mark Gates, Nicholas J Higham, Xiaoye S Li, Jennifer Loe, Piotr Luszczek, Srikara Pranesh, Siva Rajamanickam, Tobias Ribizel, Barry F Smith, Kasia Szyridowicz, Stephen Thomas, Stanimire Tomov, Yaohung M Tsai, and Ulrike Meier Yang. A survey of numerical linear algebra methods utilizing mixed-precision arithmetic. The International Journal of High Performance Computing Applications, 35(4):344–369, 2021. ISSN 1741-2846. doi: 10.1177/10943420211003313.DOI
[55]Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1737–1746. PMLR, 2015.
[56]Naigang Wang, Jungwook Choi, Daniel Brand, Chia-Yu Chen, and Kailash Gopalakrishnan. Training deep neural networks with 8-bit floating point numbers. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.arxiv.org/abs/1812.08011
[57]Noah Amsel, Jack Zhang, Kwangjun Ahn, Ali Naeimi, Austin Feng, Berlin Chen, Tri Dao, and John Langford. Dion3: Full-stack orthogonal updates. arXiv preprint arXiv:2608.11612, 2026.arxiv.org/abs/2608.11612
[60]Kwangjun Ahn, Noah Amsel, and John Langford. Dion2: A simple method to shrink matrix in Muon. arXiv preprint arXiv:2512.16928, 2025.arxiv.org/abs/2512.16928
[61]Ziyuan Tang, Tianshi Xu, Yousef Saad, and Yuanze Xi. Hierarchical Muon: Tiled Newton–Schulz updates for efficient Muon optimization. arXiv preprint arXiv:2606.27216, 2026.arxiv.org/abs/2606.27216
[62]Bishnu Dev, Sushil Bohara, Martin Takáč, and Samuel Horváth. CacheMuon: Using temporal preconditioning to approximate polar factor. arXiv preprint arXiv:2606.16371, 2026.arxiv.org/abs/2606.16371
[63]Thibaut Boissin, Thomas Massena, Franck Mamalet, and Mathieu Serrurier. Turbo-Muon: Almost-orthogonal pre-conditioning for fast Muon updates. arXiv preprint arXiv:2512.04632, 2026.arxiv.org/abs/2512.04632
[64]Shenghao Yang, Zhichao Wang, Oleg Balabanov, N. Benjamin Erichson, and Michael W. Mahoney. PRISM: Distribution-free adaptive computation of matrix functions for accelerating neural network training. arXiv preprint arXiv:2601.22137, 2026.arxiv.org/abs/2601.22137
[66]Hua Huang. How much orthogonalization does Muon need? arXiv preprint arXiv:2606.00371, 2026.arxiv.org/abs/2606.00371
[67]Ahmed Khaled, Kaan Ozkara, Tao Yu, Mingyi Hong, and Youngsuk Park. MuonBP: Faster Muon via block-periodic orthogonalization. In C. Vondrick, B. Hariharan, C. Raffel, L. Pinto, D. Yang, and A. Faust, editors, International Conference on Learning Representations, volume 2026, pages 17389–17416, 2026.arxiv.org/abs/2510.16981
[68]Chuyan Chen, Peng Sun, and Kun Yuan. CMuon: Accelerating and stabilizing diffusion transformer training via chunked momentum orthogonalization. arXiv preprint arXiv:2608.02502, 2026.arxiv.org/abs/2608.02502
[69]Vincent Chen, Starrick Liu, Regis Cheng, Dance Yang, Shalfun Li, Ryan Yu, Lucy Liang, Hang Su, Roy Gan, Hao Wang, and Qian Wang. DMuon: Efficient distributed Muon training with near-Adam overhead. arXiv preprint arXiv:2606.27153, 2026.arxiv.org/abs/2606.27153
[70]John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(61):2121–2159, 2011.
[71]Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4596–4604. PMLR, 2018.arxiv.org/abs/1804.04235
[72]James Martens and Roger Grosse. Optimizing neural networks with Kronecker-factored approximate curvature. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 2408–2417. PMLR, 2015.
[73]Thomas George, César Laurent, Xavier Bouthillier, Nicolas Ballas, and Pascal Vincent. Fast approximate natural gradient descent in a Kronecker factored eigenbasis. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.arxiv.org/abs/1806.03884
[74]Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1842–1850. PMLR, 2018.
[75]Hao-Jun Michael Shi, Tsung-Hsien Lee, Shintaro Iwasaki, Jose Gallego-Posada, Zhijing Li, Kaushik Rangadurai, Dheevatsa Mudigere, and Michael Rabbat. A distributed data-parallel PyTorch implementation of the distributed Shampoo optimizer for training neural networks at-scale. arXiv preprint arXiv:2309.06497, 2023.arxiv.org/abs/2309.06497
[76]Nikhil Vyas, Depen Morwani, Rosie Zhao, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham Kakade. SOAP: Improving and stabilizing Shampoo using Adam for language modeling. In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu, editors, International Conference on Learning Representations, volume 2025, pages 93423–93444, 2025.arxiv.org/abs/2409.11321
[77]Gyu Yeol Kim and Min-hwan Oh. Convergence of Muon with Newton–Schulz. In C. Vondrick, B. Hariharan, C. Raffel, L. Pinto, D. Yang, and A. Faust, editors, International Conference on Learning Representations, volume 2026, pages 45672–45718, 2026.
[78]Egor Shulgin, Sultan AlRashed, Francesco Orabona, and Peter Richtárik. Beyond the ideal: Analyzing the inexact Muon update. arXiv preprint arXiv:2510.19933, 2025.arxiv.org/abs/2510.19933
[79]Jeremy Bernstein and Laker Newhouse. Modular duality in deep learning. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors, Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 3920–3930. PMLR, 2025.arxiv.org/abs/2410.21265
[80]Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos, Zhenyu Zhu, Antonio Silveti-Falls, and Volkan Cevher. Training deep learning models with norm-constrained LMOs. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors, Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 49069–49104. PMLR, 2025.arxiv.org/abs/2502.07529
[81]Jianhao Ma, Yu Huang, Yuejie Chi, and Yuxin Chen. Preconditioning benefits of spectral orthogonalization in Muon. arXiv preprint arXiv:2601.13474, 2026.arxiv.org/abs/2601.13474
[82]Tim Tsz-Kit Lau, Qi Long, and Weijie Su. PolarGrad: A class of matrix-gradient optimizers from a unifying preconditioning perspective. arXiv preprint arXiv:2505.21799, 2026.arxiv.org/abs/2505.21799
[83]Zhehang Du and Weijie Su. The Newton-Muon optimizer. arXiv preprint arXiv:2604.01472, 2026.arxiv.org/abs/2604.01472
[84]Weijie Su. Isotropic curvature model for understanding deep learning optimization: Is gradient orthogonalization optimal? arXiv preprint arXiv:2511.00674, 2025.arxiv.org/abs/2511.00674
[85]Tien-Phat Nguyen, Truong Nguyen, Minh-Phuc Truong, Tuc Nguyen, James Bailey, and Trung Le. Spectral flattening is all Muon needs: How orthogonalization controls learning rate and convergence. arXiv preprint arXiv:2605.13079, 2026.arxiv.org/abs/2605.13079
[86]Xianbiao Qi, Marco Chen, Jiaquan Ye, Yelin He, and Rong Xiao. Delving into Muon and beyond: Deep analysis and extensions. arXiv preprint arXiv:2602.04669, 2026.arxiv.org/abs/2602.04669
[87]Wei Shen, Ruichuan Huang, Minhui Huang, Cong Shen, and Jiawei Zhang. On the convergence analysis of Muon. arXiv preprint arXiv:2505.23737, 2026.arxiv.org/abs/2505.23737
[88]Jiaxiang Li and Mingyi Hong. A note on the convergence of Muon. arXiv preprint arXiv:2502.02900, 2025.arxiv.org/abs/2502.02900
[89]Naoki Sato, Hiroki Naganuma, and Hideaki Iiduka. Convergence bound and critical batch size of Muon optimizer. arXiv preprint arXiv:2507.01598, 2026.arxiv.org/abs/2507.01598
[90]Greg Yang, James B. Simon, and Jeremy Bernstein. A spectral condition for feature learning. arXiv preprint arXiv:2310.17813, 2024.arxiv.org/abs/2310.17813
[91]Tim Tsz-Kit Lau and Weijie Su. Symmetry-compatible principle for optimizer design: Embeddings, LM heads, SwiGLU MLPs, and MoE routers. arXiv preprint arXiv:2605.18106, 2026.arxiv.org/abs/2605.18106
[92]Zichong Li, Liming Liu, Chen Liang, Weizhu Chen, and Tuo Zhao. NorMuon: Making Muon more efficient and scalable. In Proceedings of the 43rd International Conference on Machine Learning, 2026.arxiv.org/abs/2510.05491
[94]Yehonathan Refael, Guy Smorodinsky, Tom Turer, and Ofir Lindenbaum. SUMO: Subspace-aware moment-orthogonalization for accelerating memory-efficient LLM training. In Advances in Neural Information Processing Systems 38, NeurIPS 2025, pages 163456–163487. Neural Information Processing Systems Foundation, Inc. (NeurIPS), 2025. doi: 10.52202/085713-4926.DOI
[95]Wenbo Gong, Javier Zazo, Qijun Luo, Puqian Wang, James Hensman, and Chao Ma. ARO: A new lens on matrix optimization for large models. arXiv preprint arXiv:2602.09006, 2026.arxiv.org/abs/2602.09006
[96]Eujin Hong and Guannan Qu. Activation-Keyed Momentum: An anisotropic momentum update via the delta rule. arXiv preprint arXiv:2608.19491, 2026.arxiv.org/abs/2608.19491
[97]Chuan He, Shuyi Ren, Jingwei Mao, and Erik G. Larsson. DeMuon: A decentralized Muon for matrix optimization over graphs. arXiv preprint arXiv:2510.01377, 2025.arxiv.org/abs/2510.01377
[98]Chenghao Li, Xiao Han, Xinxin Huang, Wei Liu, Boyang Li, Bing Xiao, Heran Zhang, Juanma Perez Rua, Ke Xu, Kangning Liu, Linjun Kuang, Na Li, Tan Wang, Tian Xie, Wei Peng, Yang Pei, Yifan Xu, Yuanhao Zhai, Yuwei Lin, Zhe Wang, Zihao He, Daniel Li, Junbiao Tang, Ziyang Jiang, and Dake Chen. Scaling Muon for diffusion transformers. arXiv preprint arXiv:2608.20818, 2026.arxiv.org/abs/2608.20818
[99]Zihan Zhu, Zhejiang Du, Xuyang Chen, Tim Tsz-Kit Lau, Jiayuan Wu, X. Y. Han, Qi Long, and Weijie Su. Rank-efficient LoRA via joint tangent-space optimization under isotropic curvature. arXiv preprint arXiv:2609.12123, 2026.arxiv.org/abs/2609.12123
[100]Mikail Khona, Aditya Vavre, Boxiang Wang, Deyu Fu, Hao Wu, Mike Chrzanowski, Bryan Catanzaro, Dheevatsa Mudigere, Jeff Pool, Michael Lightstone, Mohammad Shoeybi, Mostofa Patwary, Nima Tajbakhsh, and Tijmen Blankevoort. SOAP, Muon, and beyond: Pushing LLM pretraining scales. arXiv preprint arXiv:2607.20548, 2026.arxiv.org/abs/2607.20548
[101]Gary L. Miller, Vijaya Ramachandran, and Erich Kaltofen. Efficient parallel evaluation of straight-line code and arithmetic circuits. SIAM Journal on Computing, 17(4):687–695, 1988. doi: 10.1137/0217044. URL https://doi.org/10.1137/0217044.
[102]Keller Jordan, Jeremy Bernstein, Brendan Rappazzo, @fernbear.bsky.social, Vlado Boza, Jiacheng You, Franz Cesista, Braden Kozsarsky, and @Grad62304977. Modded-NanoGPT: Speedrunning the NanoGPT baseline, 2024. URL https://github.com/KellerJordan/modded-nanogpt.
[104]Gemma Team. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025.
[105]Lozko B Milev and Szilárd Gy Révész. Bernstein’s inequality for multivariate polynomials on the standard simplex. Journal of Inequalities and Applications, 2005(2):604086, 2005. ISSN 1029-242X. doi: 10.1155/jia.2005.145. URL http://dx.doi.org/10.1155/JIA.2005.145.
[106]Philippe Tillet, H. T. Kung, and David Cox. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, PLDI ’19, pages 10–19. ACM, 2019. doi: 10.1145/3315508.3329973.DOI
[107]Ted Zadouri, Markus Hoehnerbach, Jay Shah, Timmy Liu, Vijay Thakkar, and Tri Dao. FlashAttention-4: Algorithm and kernel pipelining co-design for asymmetric hardware scaling. arXiv preprint arXiv:2603.05451, 2026.arxiv.org/abs/2603.05451
[108]Guilherme Penedo, Hynek Kydlíček, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The FineWeb datasets: Decanting the web for the finest text data at scale. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 30811–30849. Curran Associates, Inc., 2024. doi: 10.52202/079017-0970.DOI
[109]Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catanzaro, and Evan Shelhamer. cuDNN: Efficient primitives for deep learning. arXiv preprint arXiv:1410.0759, 2014.