betaFeedback

WorksMathematics

Polar Switch: Fast Muon Orthogonalization with Polynomial Circuits

Abstract

Muon, a 2024-born optimizer for language models, differs from the 2015-born spectral gradient descent mainly in its fast, GPU-friendly Newton–Schulz polynomial for orthogonalizing the gradient matrix. Accordingly, a line of research proposes faster polynomial iterations for gradient orthogonalization, such as Polar Express and Gram Newton–Schulz, both built on quintic polynomials. In this paper, we introduce Polar Switch, a new approach to polynomial gradient orthogonalization. Using simple polynomial circuits, Polar Switch reaches a given high polynomial degree at a lower FLOP cost than quintic iterations. We define a circuit as a short sequence of symmetric matrix products on a few square buffers, and circuits with kk products form a restricted family of odd polynomials of degree up to 2k+1+12^{k+1}+1. We choose the sequence of products in each circuit, their coefficients, and how the circuits are composed by numerically minimizing a worst-case bound on the polar approximation error. This minimization is subject to a FLOP budget that accounts for symmetric matrix multiplication kernels and a rounding constraint for numerical stability in half precision. Our two recommended Polar Switch methods have 15.5–18.7% lower runtime than Polar Express on square matrices and 8.6–16.2% lower runtime than Gram Newton–Schulz on rectangular matrices with aspect ratios 3.25, 3.5, and 8, with comparable approximation accuracy. In pretraining Modded-NanoGPT #91, Qwen3-0.6B, and Gemma 3-1B, Polar Switch reaches final validation losses similar to those of established orthogonalization methods. Across the more than 14 million matrices that each of the two recommended methods orthogonalized in these runs, the estimated output spectral norms never exceed the designed bounds by more than 3%, demonstrating their numerical stability.

Introduction

For years, Adam [3] and AdamW [4] have been the go-to optimizers for training large language models (LLMs) [5]. Both scale the momentum coordinatewise, without explicitly accounting for the matrix structure of the parameters. Muon [6] takes a different approach in that it uses the matrix structure of the momentum and approximates its polar factor to form the update. Studies have shown that Muon can reach the same training loss as AdamW with less computation [7, 8], and it has since been used to train several frontier open-weight LLMs, including Kimi K3 [9], GLM-5 [10], Step 3.5 Flash [11], DeepSeek-V4 [12], and Qwen3.8-Flash-Next [13]. This advantage comes with additional computation at each optimizer step, because Muon must approximate the polar factor. The question is therefore how to compute this approximation accurately and efficiently.

Let W∈Rn×mW \in\mathbb{R}^{n \times m} be a matrix-shaped parameter, ∇W\nabla W its gradient, and MM its momentum matrix. Without loss of generality, let n≤mn \le m, because matrices with more rows than columns are handled by transposition. With momentum coefficient μ\mu and learning rate η\eta, a simplified Muon update is

M←μM+∇W,W←W−η polar⁡(M).M \leftarrow\mu M + \nabla W,\qquad W \leftarrow W - \eta\,\operatorname{polar}(M).

If M=UΣVTM = U\Sigma V^{\mathsf{T}} is the compact singular value decomposition (SVD), then polar⁡(M)=UVT\operatorname{polar}(M) = UV^{\mathsf{T}} [14]. However, computing an SVD at every optimizer step is expensive for LLM training. Muon approximates the polar factor using polynomial iterations inspired by Newton–Schulz [6]. First, we normalize MM as X←M/(∥M∥F+ϵ)X \leftarrow M/(\lVert M\rVert_{\mathrm{F}} + \epsilon) with a small ϵ>0\epsilon> 0 (10−710^{-7} in our implementation), so that all singular values of XX lie in [0,1][0,1]. At iteration tt, we choose a quintic polynomial pt(x)=ctx+btx3+atx5p_t(x) = c_t x + b_t x^3 + a_t x^5 with coefficients at,bt,cta_t,b_t,c_t and compute

A←XXT,Z←btA+atA2,X←ZX+ctX.(1)A \leftarrow XX^{\mathsf{T}}, \qquad Z \leftarrow b_tA + a_tA^2, \qquad X \leftarrow ZX + c_tX. \tag*{(1)}

Let X=UΣVTX = U\Sigma V^{\mathsf{T}} be the compact SVD of XX. The update is X←U(ctΣ+btΣ3+atΣ5)VTX \leftarrow U(c_t\Sigma+ b_t\Sigma^3 + a_t\Sigma^5)V^{\mathsf{T}}. Suppose that, at each iteration tt, pt(σ)≥0p_t(\sigma) \ge0 for every singular value σ\sigma of the current matrix XX. The iteration replaces each singular value σ\sigma by pt(σ)p_t(\sigma). After TT iterations, each initial singular value σ\sigma becomes (pT∘⋯∘p1)(σ)(p_T \circ\cdots\circ p_1)(\sigma). Approximating the polar factor in Muon therefore requires this composition to approximate one on the positive initial singular values.

The classical Newton–Schulz iteration uses pt(x)=(3x−x3)/2p_t(x) = (3x-x^3)/2, whose repeated application converges to one for 0<x≤10 < x \le1 [14]. Muon uses Jordan’s fixed quintic pt(x)=3.4445x−4.775x3+2.0315x5p_t(x) = 3.4445x - 4.775x^3 + 2.0315x^5 [6]. Polar Express (PE) uses minimax approximation to choose a different quintic polynomial at each iteration, minimizing its maximum deviation from one over the current singular value interval [15][15]. Jordan’s method and Polar Express differ in their coefficients, but both use quintic iterations. We therefore ask the following question:

Under a fixed FLOP budget for matrix orthogonalization, do repeated quintic iterations yield the most accurate polar approximation?

We answer this question negatively with Polar Switch, which replaces repeated quintic iterations with a few high-degree polynomial circuits. Each circuit is a short sequence of symmetric matrix products on a small number of square buffers. For example, the first circuit of Polar Switch 3-55 (PS3-55) uses three square buffers B1,B2,B3∈Rn×nB_1, B_2, B_3 \in\mathbb{R}^{n \times n}, with the coefficients aτ,bτ,cτa_\tau,b_\tau,c_\tau given in Algorithm A.1:

B1←XXT,B2←a1B12+b1B1+c1I,B3←a2B22+b2B2+c2I,B2←a3B32+b3B3+c3I,B3←a4B22+b4B3+c4I,B1←a5B32+b5B1+c5I,X←B1X.(2)\begin{aligned} B_1 &\leftarrow XX^{\mathsf{T}}, \qquad B_2 \leftarrow a_1B_1^2 + b_1B_1 + c_1I, \qquad B_3 \leftarrow a_2B_2^2 + b_2B_2 + c_2I, \qquad B_2 \leftarrow a_3B_3^2 + b_3B_3 + c_3I,\\ B_3 &\leftarrow a_4B_2^2 + b_4B_3 + c_4I, \qquad B_1 \leftarrow a_5B_3^2 + b_5B_1 + c_5I, \qquad X \leftarrow B_1X. \tag*{(2)} \end{aligned}

Each squaring doubles the degree in XXTXX^{\mathsf{T}}, to 2, 4, 8, 16, and 32, so the circuit applies an odd polynomial of degree 65 to the singular values. Polar Switch searches numerically for the circuits and their coefficients that approximate the polar factor most accurately under a FLOP budget. Table 1 compares Polar Switch with the quintic baselines. Polar Switch 2-55 (PS2-55) and PS3-55 use fewer FLOPs than Polar Express and Gram Newton–Schulz and less memory than Gram Newton–Schulz.

MethodMemoryFLOPsFinal interval
Jordan [6]n2(2α+2)n^2(2\alpha+ 2)(15α+5)n3(15\alpha+ 5)n^3[0.562544, 1.437456]
Polar Express [15]n2(2α+2)n^2(2\alpha+ 2)(15α+5)n3(15\alpha+ 5)n^3[0.858872, 1.141128]
Gram Newton–Schulz [16]n2(2α+4)n^2(2\alpha+ 4)(6α+14)n3(6\alpha+ 14)n^3[0.787021, 1.212979]
Polar Switch 2-55n2(2α+2)n^2(2\alpha+ 2)(6α+10)n3(6\alpha+ 10)n^3[0.872263, 1.106236]
Polar Switch 2-64n2(2α+2)n^2(2\alpha+ 2)(6α+10)n3(6\alpha+ 10)n^3[0.897110, 1.081213]
Polar Switch 2-54n2(2α+2)n^2(2\alpha+ 2)(6α+9)n3(6\alpha+ 9)n^3[0.771715, 1.201363]
Polar Switch 2-512n2(2α+2)n^2(2\alpha+ 2)(9α+8)n3(9\alpha+ 8)n^3[0.894082, 1.091006]
Polar Switch 2-4111n2(2α+2)n^2(2\alpha+ 2)(12α+7)n3(12\alpha+ 7)n^3[0.930600, 1.056149]
Polar Switch 2-11111n2(2α+2)n^2(2\alpha+ 2)(15α+5)n3(15\alpha+ 5)n^3[0.867287, 1.118678]
Polar Switch 3-55n2(2α+3)n^2(2\alpha+ 3)(6α+10)n3(6\alpha+ 10)n^3[0.898309, 1.080954]
Polar Switch 3-64n2(2α+3)n^2(2\alpha+ 3)(6α+10)n3(6\alpha+ 10)n^3[0.913241, 1.070202]
Polar Switch 3-54n2(2α+3)n^2(2\alpha+ 3)(6α+9)n3(6\alpha+ 9)n^3[0.800196, 1.178895]

Table 1. Memory in matrix elements, FLOPs, and final singular value intervals for X∈Rn×mX \in\mathbb{R}^{n \times m} with n≤mn \le m and α=m/n\alpha= m/n. In exact arithmetic, each method maps singular values in [0.001, 1] into its final interval. Jordan and Gram Newton–Schulz multiply the output of their final iteration by a constant that centers this interval at one. Algorithm A.1 gives the iterations of Polar Switch.

We make three contributions. (i) We introduce Polar Switch, a FLOP-efficient orthogonalization method for Muon based on high-degree polynomial circuits. (ii) We develop efficient GPU implementations of Polar Switch with two or three square buffers. Each kernel combines a matrix product and its additions before one float16 store, and we select coefficients that try to keep this half-precision computation numerically stable. (iii) Across three LLM pretraining setups, Polar Switch reaches final validation losses similar to those of established orthogonalization methods.

The remainder of the paper is structured as follows. § 2 reviews related work. § 3 presents Polar Switch, explains how we design its circuits and coefficients under a FLOP budget, and describes its GPU implementation. § 4 compares Polar Switch with quintic baselines in LLM pretraining, runtime, and approximation accuracy. Appendix § 8 gives the theory, the numerical search, and the coefficients of Polar Switch. Appendix § 9 gives the detailed experimental setup for LLM pretraining and the runtime and approximation accuracy experiments.

Related Work

Polar approximation and polynomial design. Polynomial polar iterations [17] and matrix sign methods [14, 18] are used in eigenvalue algorithms [19, 20], control, and model reduction [21]. Rational alternatives include sign iterations [22], QR-based dynamically weighted Halley (QDWH) [23], and Zolotarev approximants [24]. Polar iterations also have backward stability analyses [25]. Broader work covers matrix roots [26, 27, 28, 29, 30], fractional powers [31], general matrix function computation [32, 33, 34, 35, 36], and manifold retractions using projected polynomials [37]. Minimax approximation minimizes the largest error over an interval, with solutions characterized by equioscillation [38] and computed by Remez methods [39]. Related problems include polynomial sign approximation [40], rational minimax [41] and AAA approximation [42], and Parks–McClellan filter design [43, 44]. Paterson–Stockmeyer [45] and related methods [46] reduce the number of matrix multiplications. Other work characterizes which polynomials can be evaluated with a fixed number of products [47]. Chebyshev acceleration [48] constructs odd polynomials with large derivatives at zero to increase small singular values more quickly. Polar Express [15] proves that greedy minimax fitting is optimal for a prescribed degree sequence, and its implementation uses quintics. Dynamic programming has also been used to select minimax compositions for homomorphic comparison under multiplication and depth constraints [49]. Polar Switch selects high-degree circuits under FLOP and buffer budgets that account for symmetric GPU kernels and adds a half-precision rounding constraint for numerical stability.

Efficient Muon orthogonalization. Equivalent formulas can round differently because their intermediate computations differ [50], and small singular values are sensitive to reduced precision [51]. Mixed precision is studied in linear algebra [52, 53, 54] and training [55, 56]. Gram Newton–Schulz [16] accumulates square multipliers between rectangular updates, and Dion3 [57] uses it for distributed training. AMO [58] adapts iteration counts, Dion [59] uses low-rank updates, Dion2 [60] selects rows or columns, Hierarchical Muon [61] uses tiles, and CacheMuon [62] reuses transforms. Turbo-Muon [63] uses an almost-orthogonal preconditioner for faster updates, and PRISM [64] computes matrix functions adaptively. IFNSO [65] replaces the iterations with one polynomial in selected matrix powers and learns its coefficients. A cubic Newton–Schulz schedule [66] uses ten instead of fifteen dominant matrix products and reaches a similar validation loss. MuonBP [67], CMuon [68], and DMuon [69] support distributed computation. Polar Switch builds on polynomial Newton–Schulz iterations with efficient GPU implementations.

Muon and matrix optimization.

References

  1. [1]David Carlson, Volkan Cevher, and Lawrence Carin. Stochastic spectral descent for restricted Boltzmann machines. In Guy Lebanon and S. V. N. Vishwanathan, editors, Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics, volume 38 of Proceedings of Machine Learning Research, pages 111–119. PMLR, 2015.
  2. [2]David Carlson, Edo Collins, Ya-Ping Hsieh, Lawrence Carin, and Volkan Cevher. Preconditioned spectral descent for deep learning. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
  3. [3]Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.arxiv.org/abs/1412.6980
  4. [4]Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
  5. [5]Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, G. S. Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsel, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc., 2020.arxiv.org/abs/2005.14165
  6. [6]Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https://kellerjordan.github.io/posts/muon/.
  7. [7]Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, Yanru Chen, Huabin Zheng, Yibo Liu, Shaowei Liu, Bohong Yin, Weiran He, Han Zhu, Yuzhi Wang, Jianzhou Wang, Mengnan Dong, Zheng Zhang, Yongsheng Kang, Hao Zhang, Xinran Xu, Yutao Zhang, Yuxin Wu, Xinyu Zhou, and Zhilin Yang. Muon is scalable for LLM training. arXiv preprint arXiv:2502.16982, 2025.arxiv.org/abs/2502.16982
  8. [8]Essential AI, Ishaan Shah, Anthony M. Polloreno, Karl Stratos, Philip Monk, Adarsh Chaluvaaraju, Andrew Hojel, Andrew Ma, Anil Thomas, Ashish Tanwer, Darsh J Shah, Khoi Nguyen, Kurt Smith, Michael Callahan, Michael Pust, Mohit Parmar, Peter Rushton, Platon Mazarakis, Ritvik Kapila, Saurabh Srivastava, Somanshu Singla, Tim Romanski, Yash Vanjani, and Ashish Vaswani. Practical efficiency of Muon for pretraining. arXiv preprint arXiv:2505.02222, 2025.arxiv.org/abs/2505.02222
  9. [9]Kimi Team. Kimi K3: Open frontier intelligence. arXiv preprint arXiv:2607.24653, 2026.
  10. [10]GLM-5 Team. GLM-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763, 2026.
  11. [11]StepFun Team. Step 3.5 Flash: Open frontier-level intelligence with 11B active parameters. arXiv preprint arXiv:2602.10604, 2026.
  12. [12]DeepSeek-AI. DeepSeek-V4: Towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348, 2026.arxiv.org/abs/2606.19348
  13. [13]Zihan Qiu, Zekun Wang, Xiao Li, Yanpeng Li, Yang Xu, Yixuan Wang, Huaqing Zhang, Rui Men, Bochao Mao, Chengruidong Zhang, Fan Zhou, Hao Luo, Haofeng Huang, Haoran Lian, Haoyan Huang, Hongqing Chen, Jianwei Zhang, Jing Xu, Junjie Wang, Langshi Chen, Liangyu Wang, Linlang Jiang, Man Yuan, Minmin Sun, Peng Jin, Siqi Zhang, Siyu Wang, Xingzhang Ren, Yakai Wang, Yi Zhang, Yiming Dong, Yizhong Cao, Yubo Ma, Yunfei Mao, Bo Zheng, and Dayiheng Liu. On the design of Qwen3.8-Next architecture: Evaluation, efficiency, and training stability. arXiv preprint arXiv:2608.30320, 2026.arxiv.org/abs/2608.30320
  14. [14]Nicholas J. Higham. Functions of Matrices: Theory and Computation. Society for Industrial and Applied Mathematics, 2008. ISBN 9780898717778. doi: 10.1137/1.9780898717778.DOI
  15. [15]Noah Amsel, David Persson, Christopher Musco, and Robert M. Gower. The Polar Express: Optimal matrix sign methods and their application to the Muon algorithm. In C. Vondrick, B. Hariharan, C. Raffel, L. Pinto, D. Yang, and A. Faust, editors, International Conference on Learning Representations, volume 2026, pages 138323–138360, 2026.arxiv.org/abs/2505.16932
  16. [16]Jack Zhang, Noah Amsel, Berlin Chen, and Tri Dao. Gram Newton–Schulz, 2026. URL https://dao-ai-lab.github.io/blog/2026/gram-newton-schulz/.
  17. [17]Å. Björck and C. Bowie. An iterative algorithm for computing the best estimate of an orthogonal matrix. SIAM Journal on Numerical Analysis, 8(2):358–364, 1971. ISSN 1095-7170. doi: 10.1137/0708036.DOI
  18. [18]Nicholas J. Higham. Computing the polar decomposition—with applications. SIAM Journal on Scientific and Statistical Computing, 7(4):1160–1174, 1986. ISSN 2168-3417. doi: 10.1137/0907079.DOI
  19. [19]Eugene D. Denman and Alex N. Beavers. The matrix sign function and computations in systems. Applied Mathematics and Computation, 2(1):63–94, 1976. ISSN 0096-3003. doi: 10.1016/0096-3003(76)90020-5.DOI
  20. [20]Yuji Nakatsukasa and Nicholas J. Higham. Stable and efficient spectral divide and conquer algorithms for the symmetric eigenvalue decomposition and the SVD. SIAM Journal on Scientific Computing, 35(3):A1325–A1349, 2013. ISSN 1095-7197. doi: 10.1137/120876605.DOI
  21. [21]J. D. Roberts. Linear model reduction and solution of the algebraic Riccati equation by use of the sign function. International Journal of Control, 32(4):677–687, 1980. ISSN 1366-5820. doi: 10.1080/00207178008922881.DOI
  22. [22]Charles Kenney and Alan J. Laub. Rational iterative methods for the matrix sign function. SIAM Journal on Matrix Analysis and Applications, 12(2):273–291, 1991. ISSN 1095-7162. doi: 10.1137/0612020.DOI
  23. [23]Yuji Nakatsukasa, Zhaojun Bai, and François Gygi. Optimizing Halley’s iteration for computing the matrix polar decomposition. SIAM Journal on Matrix Analysis and Applications, 31(5):2700–2720, 2010. ISSN 1095-7162. doi: 10.1137/090774999.DOI
  24. [24]Yuji Nakatsukasa and Roland W. Freund. Computing fundamental matrix decompositions accurately via the matrix sign function in two iterations: The power of Zolotarev’s functions. SIAM Review, 58(3):461–493, 2016. ISSN 1095-7200. doi: 10.1137/140990334.DOI
  25. [25]Yuji Nakatsukasa and Nicholas J. Higham. Backward stability of iterations for computing the polar decomposition. SIAM Journal on Matrix Analysis and Applications, 33(2):460–479, 2012. ISSN 1095-7162. doi: 10.1137/110857544.DOI
  26. [26]Nicholas J. Higham. Computing real square roots of a real matrix. Linear Algebra and its Applications, 88-89:405–430, 1987. ISSN 0024-3795. doi: 10.1016/0024-3795(87)90118-2.DOI
  27. [27]Nicholas J. Higham. Stable iterations for the matrix square root. Numerical Algorithms, 15(2):227–242, 1997. ISSN 1572-9265. doi: 10.1023/a:1019150005407.DOI
  28. [28]Edvin Deadman, Nicholas J. Higham, and Rui Ralha. Blocked Schur Algorithms for Computing the Matrix Square Root, pages 171–182. Springer Berlin Heidelberg, 2013. ISBN 9783642368035. doi: 10.1007/978-3-642-36803-5_12.DOI
  29. [29]Dario A. Bini, Nicholas J. Higham, and Beatrice Meini. Algorithms for the matrix pth root. Numerical Algorithms, 39(4):349–378, 2005. ISSN 1572-9265. doi: 10.1007/s11075-004-6709-8.DOI
  30. [30]Bruno Iannazzo. On the Newton method for the matrix pth root. SIAM Journal on Matrix Analysis and Applications, 28(2):503–523, 2006. ISSN 1095-7162. doi: 10.1137/050624790.DOI
  31. [31]Nicholas J. Higham and Lijing Lin. A Schur–Padé algorithm for fractional powers of a matrix. SIAM Journal on Matrix Analysis and Applications, 32(3):1056–1078, 2011. ISSN 1095-7162. doi: 10.1137/10081232x.DOI
  32. [32]Philip I. Davies and Nicholas J. Higham. A Schur–Parlett algorithm for computing matrix functions. SIAM Journal on Matrix Analysis and Applications, 25(2):464–485, 2003. ISSN 1095-7162. doi: 10.1137/S0895479802410815.DOI
  33. [33]Nicholas Hale, Nicholas J. Higham, and Lloyd N. Trefethen. Computing Aᵅ, log(A), and related matrix functions by contour integrals. SIAM Journal on Numerical Analysis, 46(5):2505–2523, 2008. ISSN 1095-7170. doi: 10.1137/070700607.DOI
  34. [34]Stefan Gittel. Rational Krylov approximation of matrix functions: Numerical methods and optimal pole selection. GAMM-Mitteilungen, 36(1):8–31, 2013. ISSN 1522-2608. doi: 10.1002/gamm.201310002.
  35. [35]Nicholas J. Higham. The scaling and squaring method for the matrix exponential revisited. SIAM Journal on Matrix Analysis and Applications, 26(4):1179–1193, 2005. ISSN 1095-7162. doi: 10.1137/04061101x.DOI
  36. [36]Awad H. Al-Mohy and Nicholas J. Higham. Computing the action of the matrix exponential, with an application to exponential integrators. SIAM Journal on Scientific Computing, 33(2):488–511, 2011. ISSN 1095-7197. doi: 10.1137/100788860.DOI
  37. [37]Evan S. Gawlik and Melvin Leok. High-order retractions on matrix manifolds using projected polynomials. SIAM Journal on Matrix Analysis and Applications, 39(2):801–828, 2018. ISSN 1095-7162. doi: 10.1137/17m1130459.DOI
  38. [38]Lloyd N. Trefethen. Approximation Theory and Approximation Practice, Extended Edition. Society for Industrial and Applied Mathematics, 2019. ISBN 9781611975949. doi: 10.1137/1.9781611975949 949.
  39. [39]Ricardo Pachón and Lloyd N. Trefethen. Barycentric-Remez algorithms for best polynomial approximation in the chebfun system. BIT Numerical Mathematics, 49(4):721–741, 2009. ISSN 1572-9125. doi: 10.1007/s10543-009-0240-1.DOI
  40. [40]Alexandre Eremenko and Peter Yuditskii. Uniform approximation of sgn x by polynomials and entire functions. Journal d’Analyse Mathématique, 101(1):313–324, 2007. ISSN 1565-8538. doi: 10.1007/s11854-007-0011-3.DOI
  41. [41]Silviu-Ioan Filip, Yuji Nakatsukasa, Lloyd N. Trefethen, and Bernhard Beckermann. Rational minimax approximation via adaptive barycentric representations. SIAM Journal on Scientific Computing, 40(4):A2427–A2455, 2018. ISSN 1095-7197. doi: 10.1137/17m1132409.DOI
  42. [42]Yuji Nakatsukasa, Olivier Sète, and Lloyd N. Trefethen. The AAA algorithm for rational approximation. SIAM Journal on Scientific Computing, 40(3):A1494–A1522, 2018. ISSN 1095-7197. doi: 10.1137/16m1106122.DOI
  43. [43]T. Parks and J. McClellan. Chebyshev approximation for nonrecursive digital filters with linear phase. IEEE Transactions on Circuit Theory, 19(2):189–194, 1972. ISSN 0018-9324. doi: 10.1109/tct.1972.1083419.DOI
  44. [44]J. McClellan, T. Parks, and L. Rabiner. A computer program for designing optimum FIR linear phase digital filters. IEEE Transactions on Audio and Electroacoustics, 21(6):506–526, 1973. ISSN 0018-9278. doi: 10.1109/tau.1973.1162525.DOI
  45. [45]Michael S. Paterson and Larry J. Stockmeyer. On the number of nonscalar multiplications necessary to evaluate polynomials. SIAM Journal on Computing, 2(1):60–66, 1973. ISSN 1095-7111. doi: 10.1137/0202007.DOI
  46. [46]J. Sastre. Efficient evaluation of matrix polynomials. Linear Algebra and its Applications, 539:229–250, 2018. ISSN 0024-3795. doi: 10.1016/j.laa.2017.11.010.DOI
  47. [47]Elias Jarlebring and Gustaf Lorentzon. The polynomial set associated with a fixed number of matrix-matrix multiplications. SIAM Journal on Applied Algebra and Geometry, 10(3):549–581, 2026. ISSN 2470-6566. doi: 10.1137/25m1779036.DOI
  48. [48]Ekaterina Grishina, Matvey Smirnov, and Maxim Rakhuba. Accelerating Newton–Schulz iteration for orthogonalization via Chebyshev-type polynomials. arXiv preprint arXiv:2506.10935, 2026.arxiv.org/abs/2506.10935
  49. [49]Eunsang Lee, Joon-Woo Lee, Jong-Seon No, and Young-Sik Kim. Minimax approximation of sign function by composite polynomial for homomorphic comparison. IEEE Transactions on Dependable and Secure Computing, 19(6):3711–3727, 2022. ISSN 2160-9209. doi: 10.1109/tdsc.2021.3105111.DOI
  50. [50]Nicholas J. Higham. Accuracy and Stability of Numerical Algorithms: Second Edition. Society for Industrial and Applied Mathematics, 2002. ISBN 9780898718027. doi: 10.1137/1.9780898718027.
  51. [51]Christos Boutsikas, Petros Drineas, and Ilse C. F. Ipsen. Small singular values can increase in lower precision. SIAM Journal on Matrix Analysis and Applications, 45(3):1518–1540, 2024. ISSN 1095-7162. doi: 10.1137/23m1557209.DOI
  52. [52]Marc Baboulin, Alfredo Buttari, Jack Dongarra, Jakub Kurzak, Julie Langou, Julien Langou, Piotr Luszczek, and Stanimire Tomov. Accelerating scientific computations with mixed precision algorithms. Computer Physics Communications, 180(12):2526–2533, 2009. ISSN 0010-4655. doi: 10.1016/j.cpc.2008.11.005.DOI
  53. [53]Erin Carson and Nicholas J. Higham. A new analysis of iterative refinement and its application to accurate solution of ill-conditioned sparse linear systems. SIAM Journal on Scientific Computing, 39(6):A2834–A2856, 2017. ISSN 1095-7197. doi: 10.1137/17m1122918.DOI
  54. [54]Ahmad Abdelfattah, Hartwig Anzt, Erik G Boman, Erin Carson, Terry Cojean, Jack Dongarra, Alyson Fox, Mark Gates, Nicholas J Higham, Xiaoye S Li, Jennifer Loe, Piotr Luszczek, Srikara Pranesh, Siva Rajamanickam, Tobias Ribizel, Barry F Smith, Kasia Szyridowicz, Stephen Thomas, Stanimire Tomov, Yaohung M Tsai, and Ulrike Meier Yang. A survey of numerical linear algebra methods utilizing mixed-precision arithmetic. The International Journal of High Performance Computing Applications, 35(4):344–369, 2021. ISSN 1741-2846. doi: 10.1177/10943420211003313.DOI
  55. [55]Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1737–1746. PMLR, 2015.
  56. [56]Naigang Wang, Jungwook Choi, Daniel Brand, Chia-Yu Chen, and Kailash Gopalakrishnan. Training deep neural networks with 8-bit floating point numbers. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.arxiv.org/abs/1812.08011
  57. [57]Noah Amsel, Jack Zhang, Kwangjun Ahn, Ali Naeimi, Austin Feng, Berlin Chen, Tri Dao, and John Langford. Dion3: Full-stack orthogonal updates. arXiv preprint arXiv:2608.11612, 2026.arxiv.org/abs/2608.11612
  58. [58]Xinlin Zhuang, Panyi Ouyang, Yichen Li, Jiangming Shi, Yizhang Chen, Shuman Liu, Ying Qian, Weiyang Liu, Haibo Zhang, and Imran Razzak. AMO: Operator-level adaptive Muon orthogonalization. arXiv preprint arXiv:2605.17806, 2026.arxiv.org/abs/2605.17806
  59. [59]Kwangjun Ahn, Byron Xu, Natalie Abreu, Ying Fan, Gagik Magakyan, Pratyusha Sharma, Zheng Zhan, and John Langford. Dion: Distributed orthonormalized updates. arXiv preprint arXiv:2504.05295, 2025.arxiv.org/abs/2504.05295
  60. [60]Kwangjun Ahn, Noah Amsel, and John Langford. Dion2: A simple method to shrink matrix in Muon. arXiv preprint arXiv:2512.16928, 2025.arxiv.org/abs/2512.16928
  61. [61]Ziyuan Tang, Tianshi Xu, Yousef Saad, and Yuanze Xi. Hierarchical Muon: Tiled Newton–Schulz updates for efficient Muon optimization. arXiv preprint arXiv:2606.27216, 2026.arxiv.org/abs/2606.27216
  62. [62]Bishnu Dev, Sushil Bohara, Martin Takáč, and Samuel Horváth. CacheMuon: Using temporal preconditioning to approximate polar factor. arXiv preprint arXiv:2606.16371, 2026.arxiv.org/abs/2606.16371
  63. [63]Thibaut Boissin, Thomas Massena, Franck Mamalet, and Mathieu Serrurier. Turbo-Muon: Almost-orthogonal pre-conditioning for fast Muon updates. arXiv preprint arXiv:2512.04632, 2026.arxiv.org/abs/2512.04632
  64. [64]Shenghao Yang, Zhichao Wang, Oleg Balabanov, N. Benjamin Erichson, and Michael W. Mahoney. PRISM: Distribution-free adaptive computation of matrix functions for accelerating neural network training. arXiv preprint arXiv:2601.22137, 2026.arxiv.org/abs/2601.22137
  65. [65]Chen Hu, Qianxi Zhao, Xiaochen Yuan, Hong Zhang, Ding Yuan, Yanbin Wu, and Xiying Li. IFNSO: Iteration-free Newton-Schulz orthogonalization. arXiv preprint arXiv:2602.02500, 2026.arxiv.org/abs/2602.02500
  66. [66]Hua Huang. How much orthogonalization does Muon need? arXiv preprint arXiv:2606.00371, 2026.arxiv.org/abs/2606.00371
  67. [67]Ahmed Khaled, Kaan Ozkara, Tao Yu, Mingyi Hong, and Youngsuk Park. MuonBP: Faster Muon via block-periodic orthogonalization. In C. Vondrick, B. Hariharan, C. Raffel, L. Pinto, D. Yang, and A. Faust, editors, International Conference on Learning Representations, volume 2026, pages 17389–17416, 2026.arxiv.org/abs/2510.16981
  68. [68]Chuyan Chen, Peng Sun, and Kun Yuan. CMuon: Accelerating and stabilizing diffusion transformer training via chunked momentum orthogonalization. arXiv preprint arXiv:2608.02502, 2026.arxiv.org/abs/2608.02502
  69. [69]Vincent Chen, Starrick Liu, Regis Cheng, Dance Yang, Shalfun Li, Ryan Yu, Lucy Liang, Hang Su, Roy Gan, Hao Wang, and Qian Wang. DMuon: Efficient distributed Muon training with near-Adam overhead. arXiv preprint arXiv:2606.27153, 2026.arxiv.org/abs/2606.27153
  70. [70]John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(61):2121–2159, 2011.
  71. [71]Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4596–4604. PMLR, 2018.arxiv.org/abs/1804.04235
  72. [72]James Martens and Roger Grosse. Optimizing neural networks with Kronecker-factored approximate curvature. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 2408–2417. PMLR, 2015.
  73. [73]Thomas George, César Laurent, Xavier Bouthillier, Nicolas Ballas, and Pascal Vincent. Fast approximate natural gradient descent in a Kronecker factored eigenbasis. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.arxiv.org/abs/1806.03884
  74. [74]Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1842–1850. PMLR, 2018.
  75. [75]Hao-Jun Michael Shi, Tsung-Hsien Lee, Shintaro Iwasaki, Jose Gallego-Posada, Zhijing Li, Kaushik Rangadurai, Dheevatsa Mudigere, and Michael Rabbat. A distributed data-parallel PyTorch implementation of the distributed Shampoo optimizer for training neural networks at-scale. arXiv preprint arXiv:2309.06497, 2023.arxiv.org/abs/2309.06497
  76. [76]Nikhil Vyas, Depen Morwani, Rosie Zhao, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham Kakade. SOAP: Improving and stabilizing Shampoo using Adam for language modeling. In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu, editors, International Conference on Learning Representations, volume 2025, pages 93423–93444, 2025.arxiv.org/abs/2409.11321
  77. [77]Gyu Yeol Kim and Min-hwan Oh. Convergence of Muon with Newton–Schulz. In C. Vondrick, B. Hariharan, C. Raffel, L. Pinto, D. Yang, and A. Faust, editors, International Conference on Learning Representations, volume 2026, pages 45672–45718, 2026.
  78. [78]Egor Shulgin, Sultan AlRashed, Francesco Orabona, and Peter Richtárik. Beyond the ideal: Analyzing the inexact Muon update. arXiv preprint arXiv:2510.19933, 2025.arxiv.org/abs/2510.19933
  79. [79]Jeremy Bernstein and Laker Newhouse. Modular duality in deep learning. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors, Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 3920–3930. PMLR, 2025.arxiv.org/abs/2410.21265
  80. [80]Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos, Zhenyu Zhu, Antonio Silveti-Falls, and Volkan Cevher. Training deep learning models with norm-constrained LMOs. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors, Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 49069–49104. PMLR, 2025.arxiv.org/abs/2502.07529
  81. [81]Jianhao Ma, Yu Huang, Yuejie Chi, and Yuxin Chen. Preconditioning benefits of spectral orthogonalization in Muon. arXiv preprint arXiv:2601.13474, 2026.arxiv.org/abs/2601.13474
  82. [82]Tim Tsz-Kit Lau, Qi Long, and Weijie Su. PolarGrad: A class of matrix-gradient optimizers from a unifying preconditioning perspective. arXiv preprint arXiv:2505.21799, 2026.arxiv.org/abs/2505.21799
  83. [83]Zhehang Du and Weijie Su. The Newton-Muon optimizer. arXiv preprint arXiv:2604.01472, 2026.arxiv.org/abs/2604.01472
  84. [84]Weijie Su. Isotropic curvature model for understanding deep learning optimization: Is gradient orthogonalization optimal? arXiv preprint arXiv:2511.00674, 2025.arxiv.org/abs/2511.00674
  85. [85]Tien-Phat Nguyen, Truong Nguyen, Minh-Phuc Truong, Tuc Nguyen, James Bailey, and Trung Le. Spectral flattening is all Muon needs: How orthogonalization controls learning rate and convergence. arXiv preprint arXiv:2605.13079, 2026.arxiv.org/abs/2605.13079
  86. [86]Xianbiao Qi, Marco Chen, Jiaquan Ye, Yelin He, and Rong Xiao. Delving into Muon and beyond: Deep analysis and extensions. arXiv preprint arXiv:2602.04669, 2026.arxiv.org/abs/2602.04669
  87. [87]Wei Shen, Ruichuan Huang, Minhui Huang, Cong Shen, and Jiawei Zhang. On the convergence analysis of Muon. arXiv preprint arXiv:2505.23737, 2026.arxiv.org/abs/2505.23737
  88. [88]Jiaxiang Li and Mingyi Hong. A note on the convergence of Muon. arXiv preprint arXiv:2502.02900, 2025.arxiv.org/abs/2502.02900
  89. [89]Naoki Sato, Hiroki Naganuma, and Hideaki Iiduka. Convergence bound and critical batch size of Muon optimizer. arXiv preprint arXiv:2507.01598, 2026.arxiv.org/abs/2507.01598
  90. [90]Greg Yang, James B. Simon, and Jeremy Bernstein. A spectral condition for feature learning. arXiv preprint arXiv:2310.17813, 2024.arxiv.org/abs/2310.17813
  91. [91]Tim Tsz-Kit Lau and Weijie Su. Symmetry-compatible principle for optimizer design: Embeddings, LM heads, SwiGLU MLPs, and MoE routers. arXiv preprint arXiv:2605.18106, 2026.arxiv.org/abs/2605.18106
  92. [92]Zichong Li, Liming Liu, Chen Liang, Weizhu Chen, and Tuo Zhao. NorMuon: Making Muon more efficient and scalable. In Proceedings of the 43rd International Conference on Machine Learning, 2026.arxiv.org/abs/2510.05491
  93. [93]Chenrui Xu, Wenjing Yan, and Ying-Jun Angela Zhang. FISMO: Fisher-structured momentum-orthogonalized optimizer. arXiv preprint arXiv:2601.21750, 2026.arxiv.org/abs/2601.21750
  94. [94]Yehonathan Refael, Guy Smorodinsky, Tom Turer, and Ofir Lindenbaum. SUMO: Subspace-aware moment-orthogonalization for accelerating memory-efficient LLM training. In Advances in Neural Information Processing Systems 38, NeurIPS 2025, pages 163456–163487. Neural Information Processing Systems Foundation, Inc. (NeurIPS), 2025. doi: 10.52202/085713-4926.DOI
  95. [95]Wenbo Gong, Javier Zazo, Qijun Luo, Puqian Wang, James Hensman, and Chao Ma. ARO: A new lens on matrix optimization for large models. arXiv preprint arXiv:2602.09006, 2026.arxiv.org/abs/2602.09006
  96. [96]Eujin Hong and Guannan Qu. Activation-Keyed Momentum: An anisotropic momentum update via the delta rule. arXiv preprint arXiv:2608.19491, 2026.arxiv.org/abs/2608.19491
  97. [97]Chuan He, Shuyi Ren, Jingwei Mao, and Erik G. Larsson. DeMuon: A decentralized Muon for matrix optimization over graphs. arXiv preprint arXiv:2510.01377, 2025.arxiv.org/abs/2510.01377
  98. [98]Chenghao Li, Xiao Han, Xinxin Huang, Wei Liu, Boyang Li, Bing Xiao, Heran Zhang, Juanma Perez Rua, Ke Xu, Kangning Liu, Linjun Kuang, Na Li, Tan Wang, Tian Xie, Wei Peng, Yang Pei, Yifan Xu, Yuanhao Zhai, Yuwei Lin, Zhe Wang, Zihao He, Daniel Li, Junbiao Tang, Ziyang Jiang, and Dake Chen. Scaling Muon for diffusion transformers. arXiv preprint arXiv:2608.20818, 2026.arxiv.org/abs/2608.20818
  99. [99]Zihan Zhu, Zhejiang Du, Xuyang Chen, Tim Tsz-Kit Lau, Jiayuan Wu, X. Y. Han, Qi Long, and Weijie Su. Rank-efficient LoRA via joint tangent-space optimization under isotropic curvature. arXiv preprint arXiv:2609.12123, 2026.arxiv.org/abs/2609.12123
  100. [100]Mikail Khona, Aditya Vavre, Boxiang Wang, Deyu Fu, Hao Wu, Mike Chrzanowski, Bryan Catanzaro, Dheevatsa Mudigere, Jeff Pool, Michael Lightstone, Mohammad Shoeybi, Mostofa Patwary, Nima Tajbakhsh, and Tijmen Blankevoort. SOAP, Muon, and beyond: Pushing LLM pretraining scales. arXiv preprint arXiv:2607.20548, 2026.arxiv.org/abs/2607.20548
  101. [101]Gary L. Miller, Vijaya Ramachandran, and Erich Kaltofen. Efficient parallel evaluation of straight-line code and arithmetic circuits. SIAM Journal on Computing, 17(4):687–695, 1988. doi: 10.1137/0217044. URL https://doi.org/10.1137/0217044.
  102. [102]Keller Jordan, Jeremy Bernstein, Brendan Rappazzo, @fernbear.bsky.social, Vlado Boza, Jiacheng You, Franz Cesista, Braden Kozsarsky, and @Grad62304977. Modded-NanoGPT: Speedrunning the NanoGPT baseline, 2024. URL https://github.com/KellerJordan/modded-nanogpt.
  103. [103]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengeng Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.arxiv.org/abs/2505.09388
  104. [104]Gemma Team. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025.
  105. [105]Lozko B Milev and Szilárd Gy Révész. Bernstein’s inequality for multivariate polynomials on the standard simplex. Journal of Inequalities and Applications, 2005(2):604086, 2005. ISSN 1029-242X. doi: 10.1155/jia.2005.145. URL http://dx.doi.org/10.1155/JIA.2005.145.
  106. [106]Philippe Tillet, H. T. Kung, and David Cox. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, PLDI ’19, pages 10–19. ACM, 2019. doi: 10.1145/3315508.3329973.DOI
  107. [107]Ted Zadouri, Markus Hoehnerbach, Jay Shah, Timmy Liu, Vijay Thakkar, and Tri Dao. FlashAttention-4: Algorithm and kernel pipelining co-design for asymmetric hardware scaling. arXiv preprint arXiv:2603.05451, 2026.arxiv.org/abs/2603.05451
  108. [108]Guilherme Penedo, Hynek Kydlíček, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The FineWeb datasets: Decanting the web for the finest text data at scale. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 30811–30849. Curran Associates, Inc., 2024. doi: 10.52202/079017-0970.DOI
  109. [109]Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catanzaro, and Evan Shelhamer. cuDNN: Efficient primitives for deep learning. arXiv preprint arXiv:1410.0759, 2014.

Paper details

vlnc:W_wnd53rrr
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

GPT-6 Astra, GPT-5.6 Sol (bare)

View statement
Looking for related papers…
Contents