betaFeedback

WorksComputer Science

Equal Steps, Unequal Curvature: Where Muon's Update Goes Wrong

Abstract

Muon updates each hidden weight matrix along the polar factor of its momentum, the steepest-descent direction under the spectral norm. That choice fixes two allocations without consulting the loss: every singular direction of the momentum receives the same step, and in a non-square matrix each neuron's step is set by its leverage score. Whether they are good is a question about curvature, which steepest-descent arguments do not answer; we measure it. Using held-out gradients and Hessian-vector products, we decompose Muon's step by singular-value band and by neuron group and compute each group's second-order optimal multiplier. In a 124M-parameter decoder trained on FineWeb-Edu, the multipliers rise monotonically from the top singular directions to the tail: Muon over-steps the high-curvature top of its spectrum. We prove in Lean when such a diagnostic licenses an intervention at Muon's own update norm. On the measured curvature, the criterion predicts, and held-out losses confirm, that moving step length off the top helps mid-training and early in the cooldown, but not at its end. We then test interventions under per-step norm matching, tuned baselines and paired seeds. Orthogonalizing Q, K and V separately rather than as one fused matrix lowers the loss by 0.0250 nats and accounts for all of the gain of block-wise variants. A matmul-only, rotation-equivariant spectral tilt σ−1/4σ^{-1/4} applied in the cooldown lowers the loss of this stronger baseline by a further 0.0120 nats, additively with NorMuon (0.0158 nats together, three seeds; the baseline needs about 4.3% more tokens to match), and by 0.0188 nats at 355M parameters (three seeds). Energy-matched Schatten-1 weight decay adds a further 0.0052 nats. Geometrically principled changes that do not act on the measured miscalibration, namely exact polar factors, steepest-descent input whitening and top-heavy decay, do not help.

Introduction

Muon (Jordan et al., [21]) has become the reference optimizer for the hidden matrices of transformers at small and medium scale (Liu et al., [28]; Essential AI et al. [13]; Jordan et al. [20]). Its theory is clean. For a weight matrix with momentum N=UΣVTN = U\Sigma V^{\mathsf{T}}, the update direction polar⁡(N)=UVT\operatorname{polar}(N) = UV^{\mathsf{T}} is the steepest-descent direction under the spectral norm (Bernstein and Newhouse, [5]; Large et al. [24]; Pethick et al. [35]), and Muon with decoupled weight decay solves a spectral-norm-constrained problem (Chen et al., [8]).

That theory picks the norm in which the step is steepest. It is silent on whether the step’s distribution is right, and UVTUV^{\mathsf{T}} fixes two distributions without asking. (i) Across singular directions, every direction uiviTu_i v_i^{\mathsf{T}} receives a step of the same length, whatever its gradient signal σi\sigma_i and whatever the loss curvature along it. (ii) Across neurons, a tall polar factor has row norms equal to the square roots of the leverage scores of NN’s column space. Each output neuron therefore moves by an amount set by the momentum’s geometry and by how matrices happen to be fused in the implementation.

Recent variants change both distributions. NorMuon, Muon+ and Aurora rebalance neurons (Li et al., [27]; Zhang et al. [54]; Dewulf et al. [11]). DynMuon, Freon/Kaon and Spectrally Targeted Muon reshape the spectrum (Wu et al., [50]; Shumaylov et al. [42]; Goyal and Ramkumar [15]). MuonBP and HiMuon orthogonalize blocks (Khaled et al., [22]; Tang et al. [44]). Some work also measures why. SAMuon probes held-out loss-optimal step sizes along the momentum’s singular directions and builds an optimizer from that profile (Wu et al., [51]). BulkBoost proves when a norm-matched two-band reallocation helps to first order Chaitanya et al., [6]. Curvature analyses explain Muon’s advantage through its actual update Wang et al., [47]; Shumaylov et al. [42]; Shulgin et al. [41]. These measurements treat directions or bands one at a time or compare Muon with Adam, and gains against loosely tuned baselines often shrink against well-tuned ones Wen et al., [49]; Zhao et al. [56].

This paper measures the joint picture. We treat the flat spectrum and the leverage-weighted neuron allocation as hypotheses and test them against the loss’s local geometry. We split Muon’s next step into groups of singular directions or of neurons. On held-out data we measure each group’s gradient alignment gbg_b and the full curvature matrix HH between groups, including the cross-group terms that isolated probes omit. The ratio gb/Hbbg_b/H_{bb} is the step multiplier that would be optimal for group bb on its own to second order, and Muon corresponds to multiplier 1 everywhere. A one-step greedy analysis is known to misjudge the overall step size Wu et al., [52]. We therefore use an exact identity, proved in Lean, that splits Muon’s second-order gap into a scale part (the overall step size) and a shape part (the relative weighting across groups). Our experiments compare steps of equal norm, so we also prove when a same-norm reallocation can beat Muon’s step, and we evaluate that criterion on the measured curvature.

What the anatomy shows (§ 4). The optimal multiplier rises monotonically from the top singular directions to the tail. On ranks 513–768 it is 38–190 times its value on ranks 1–8. Muon thus over-steps the high-curvature top of its spectrum relative to the rest. At Muon’s own update norm, which is the comparison our experiments make, reallocating step length off the top would change the one-step loss by −0.00589 mid-training and −0.00488 at the start of the cooldown. Muon’s own step changes it by −0.00079 and +0.00026. By the end of the cooldown the possible gain is only −0.00009. Measuring the actual held-out loss confirms this: at matched norm, halving only the top eight singular directions improves Muon’s one-step loss by −4.4 and −3.2 (×10−3\times10^{-3}), and not at the end. Per-neuron step sizes vary most in the fused 2304×7682304 \times768 QKV matrix (coefficient of variation 0.43, against 0.18 in the MLP).

What acting on it buys (§ 6). We test interventions under strict controls. Each direction-changing arm is norm-matched to Muon at every step, baselines use tuned (interior-optimum) learning rates, and seeds are paired. Three findings stand out.

  1. The neuron allocation that matters is per projection. Orthogonalizing QQ, KK and VV separately, a known practice Jordan et al., [20], changes the final loss by −0.0250 nats [−0.0290, −0.0210] at 100.7M tokens. It accounts for all of the gain of square-block variants (residual +0.0011 [−0.0029, +0.0051]). Finer rebalancing, per head or per row, does worse. The fused-QKV baseline would need about 6.8% more tokens to match, which confounds comparisons: NorMuon’s gain shrinks from 0.0150 to 0.0038 against the split baseline.

  1. Tilting the spectrum away from its top helps. A cooldown tilt toward small singular values lowers the stronger baseline’s loss by a further 0.0120 nats (−0.0137, −0.0102), better in every seed. We implement it with a Chebyshev filter evaluated by Clenshaw’s recurrence, using matrix products only, and prove that it keeps Muon’s rotational equivariance.

  1. The two corrections add. NorMuon + tilt gives −0.0158 [−0.0178, −0.0138], against −0.0038 and −0.0120 separately. At 355M parameters the combination gains 0.0188 nats [−0.0202, −0.0175] (three seeds). Energy-matched Schatten-1 weight decay adds a further −0.0052 [−0.0066, −0.0038] at 124M.

The negatives are equally informative. Input whitening is provably GL⁡(n)\operatorname{GL}(n)-equivariant and conditioning-free on least squares, yet it hurts (up to +0.06 nats). More accurate polar factors do nothing, and top-heavy decay hurts; none acts on the miscalibration we measured. The diagnostic and the first screens ran concurrently, so the anatomy’s implications are consistency checks, not pre-registered predictions; the factorial ablations and the 355M runs came later.

Contributions. (1) A held-out, curvature-resolved diagnostic for a matrix optimizer’s step, with an exact scale/shape decomposition and a proved criterion for when a same-norm reallocation can improve the step. (2) An anatomy of Muon: monotone over-stepping of the top singular directions, which a same-norm reallocation can exploit until late in the cooldown, and a neuron imbalance concentrated in fused matrices. (3) A controlled attribution, including a 2×22 \times2 factorial with three seeds and an A/A noise check. (4) Machine-checked statements of the geometry used (Section A), and an account of what they did and did not predict (§ 7). Every optimizer ingredient we evaluate has a precedent, and so does probing Muon’s spectrum against held-out loss (§ 8). What is new is the joint-curvature diagnostic with its fixed-norm criterion, the neuron-level and QKV attribution, and the controls that test them.

Two allocations hidden in Muon’s step

For a hidden matrix W∈Rm×nW \in\mathbb{R}^{m \times n} with gradient GtG_t, the Muon variant we study forms the sum-form Nesterov input and steps along a quintic Newton–Schulz map:

Bt=βBt−1+Gt,Nt=Gt+βBt,Wt+1=(1−ηtλ)Wt−ηts NS⁡5(Nt),(1)\begin{aligned} B_t &= \beta B_{t-1} + G_t, \qquad N_t = G_t + \beta B_t,\\ W_{t+1} &= (1-\eta_t\lambda)W_t - \eta_t s\,\operatorname{NS}_5(N_t), \tag*{(1)} \end{aligned}

with shape factor s=max⁡(1,m/n)s = \sqrt{\max(1,m/n)}. NS5_5 normalizes NN by its Frobenius norm and applies five iterations of X↦aX+bXX⊤X+c(XX⊤)2XX \mapsto aX + bXX^{\top}X + c(XX^{\top})^{2}X with (a,b,c)=(3.4445,−4.775,2.0315)(a,b,c)=(3.4445,-4.775,2.0315) [21]. On singular values it acts as a scalar map ϕ5\phi_{5} that is close to 1 on most of the spectrum but well below 1 on near-null directions. Hence NS⁡5(N)≈polar⁡(N)=UV⊤=arg⁡max⁡∥D∥op≤1⟨N,D⟩\operatorname{NS}_{5}(N) \approx\operatorname{polar}(N)=UV^{\top}=\underset{\lVert D\rVert_{\mathrm{op}}\leq1}{\arg\max}\langle N,D\rangle.

Allocation across singular directions. Write any spectral filter as Df=Uf(Σ)V⊤D_f=Uf(\Sigma)V^{\top}. SGD is f(σ)=σf(\sigma)=\sigma, Muon is (approximately) f≡1f\equiv1, and power-law tilts are f(σ)=(σ/σrms)pf(\sigma)=(\sigma/\sigma_{\mathrm{rms}})^{p} [50]. Muon’s choice is optimal for the worst case over a spectral-norm ball. It is not obviously optimal for a loss whose curvature differs across singular directions.

Allocation across neurons. Every theorem and proposition in this paper is proved in Lean 4; Section A states each in full. Row ii of WW is the incoming weight vector of output neuron ii. Muon moves it by ∥(UV⊤)i,:∥\lVert(UV^{\top})_{i,:}\rVert.

Proposition 1 (Polar row norms are leverage scores; Theorem A.10). Let O∈Rm×nO\in\mathbb{R}^{m\times n} with O⊤O=InO^{\top}O=I_n (m≥nm\geq n). Then ∑i∥Oi,:∥2=n\sum_i\lVert O_{i,:}\rVert^2=n, but the individual ∥Oi,:∥2\lVert O_{i,:}\rVert^2, the leverage scores of range⁡(O)\operatorname{range}(O) [12], can be arbitrarily uneven: one row can have norm 1 and another 0. If m=nm=n and OO⊤=IOO^{\top}=I, every row has unit norm.

So Muon’s per-neuron step depends on the shape of the matrix it is applied to. Storing Q, K and V as one fused 3d×d3d\times d matrix (the GPT-2 layout [36]) makes each projection’s step depend on the momenta of the other two. The block-wise alternative is the exact steepest step of a natural norm:

Proposition 2 (Block-wise polar steps; Section A.2). Split W=[W1;…;Wk]W=[W_1;\ldots;W_k] into kk blocks and let ∥D∥blk=max⁡b∥Db∥op\lVert D\rVert_{\mathrm{blk}}=\max_b\lVert D_b\rVert_{\mathrm{op}}. Then [polar⁡(N1);…;polar⁡(Nk)][\operatorname{polar}(N_1);\ldots;\operatorname{polar}(N_k)] maximizes ⟨N,D⟩\langle N,D\rangle over ∥D∥blk≤1\lVert D\rVert_{\mathrm{blk}}\leq1, and the joint polar factor can be strictly worse (for N1=N2=(1,0)N_1=N_2=(1,0) it attains 2\sqrt{2} against 2). For a tall OO split into kk exactly orthogonal n×nn\times n row blocks, Muon’s shape factor s=ks=\sqrt{k} gives the joint step the same Frobenius energy as the block-wise step.

Both propositions concern exact polar factors. NS⁡5\operatorname{NS}_{5} is only approximately orthogonal, so in practice block-wise steps shrink the row spread but do not remove it. The block-wise steepest step for this max-norm is MuonBP’s Block-Muon [22]. The leverage-score reading of tall polar factors is also given by Dewulf et al. [11]. We use both diagnostically, to locate where Muon’s per-neuron steps are uneven, not to show that the unevenness hurts.

A curvature-resolved anatomy of the step

Grouped multipliers. We partition Muon’s next step into groups b=1,…,Bb=1,\ldots,B in one of two ways. Bands: singular-value ranks of the momentum, with Dℓ,b=η sℓUℓ,bVℓ,b⊤D_{\ell,b}=\eta\,s_{\ell}U_{\ell,b}V_{\ell,b}^{\top} for every matrix ℓ\ell. This is the exact-polar idealization of Muon’s step; NS5_5 itself gives near-null directions less than unit length. Neurons: sextiles of neurons ranked by the norm of their row (or column) in Muon’s actual NS5_5 step. Now scale group bb by cbc_b, i.e. Wℓ←Wℓ−∑bcbDℓ,bW_{\ell}\leftarrow W_{\ell}-\sum_b c_bD_{\ell,b}. To second order in cc, the held-out loss changes by

q(c)=−g⊤c+12c⊤Hc,gb=⟨∇L^,Db⟩,Hbb′=⟨Db,∇2L^Db′⟩,(2)q(c)=-g^{\top}c+\frac{1}{2}c^{\top}Hc, \qquad g_b=\langle\nabla\widehat{L},D_b\rangle, \qquad H_{bb'}=\langle D_b,\nabla^2\widehat{L}D_{b'}\rangle, \tag*{(2)}

where L^\widehat{L} is the loss on held-out tokens. Thus gg and HH refer to the held-out loss, not the training minibatch, though they still carry finite-sample noise. Muon is c=1c=1. The stationary point of qq is c∗=H−1gc^*=H^{-1}g. It is the minimizer of qq only when H⪰0H\succeq0, and the unique one when H≻0H\succ0 (Section A.3). Every curvature matrix we estimate is positive definite, with smallest eigenvalue at least 4.6×10−84.6\times10^{-8}. The per-group multiplier cb=gb/Hbbc_b=g_b/H_{bb} minimizes qq for group bb on its own. Muon’s flat filter is second-order optimal up to a common scale exactly when c∗∝1c^*\propto1. All cbc_b scale as 1/n1/n, so only their ratios are comparable across learning rates.

Scale versus shape. A greedy one-step model is a poor guide to the overall step size, because it ignores the step’s effect on future iterates [52, 4]. It can still say which groups get too much or too little of a given step. The following identity separates the two.

Theorem 3 (Exact scale/shape split; Section A.3). Let HH be symmetric positive definite, c∗=H−1gc^*=H^{-1}g the minimizer of qq, and γ∗=1⊤g/1⊤H1\gamma^*=\mathbf{1}^{\top}g/\mathbf{1}^{\top}H\mathbf{1} the minimizer of q(γ1)q(\gamma\mathbf{1}) over γ\gamma, i.e. the best uniform rescaling of Muon’s step. Then

q(1)−q(c∗)=12(1−γ∗)21⊤H1⏟step size+12(γ∗1−c∗)⊤H(γ∗1−c∗)⏟shapeq(\mathbf{1})-q(c^*)= \underbrace{\frac{1}{2}(1-\gamma^*)^2\mathbf{1}^{\top}H\mathbf{1}}_{\text{step size}} + \underbrace{\frac{1}{2}(\gamma^*\mathbf{1}-c^*)^{\top}H(\gamma^*\mathbf{1}-c^*)}_{\text{shape}}

and the shape term is strictly positive unless c∗c^* is flat.

The identity itself needs only symmetry and Hc∗=gHc^*=g. Positive definiteness is what makes c∗c^* and γ∗\gamma^* optima, and both terms losses relative to them. The step-size term carries the known short-horizon bias. Removing it does not make the shape immune: in an explicit example, a one-step shape preference reverses after one more gradient step (Theorem A.28). We therefore treat the shape as a one-step diagnostic whose implications must be tested in training, not as a prescription.

Fixed norm. Theorem 3 compares Muon with c∗c^*, whose norm generally differs from Muon’s. Our experiments instead compare steps of equal Frobenius norm. Band directions are mutually orthogonal, with squared norms proportional to the band sizes ∣Ib∣|I_b|, so equal norm means c⊤Nc=1⊤N1c^{\top}Nc = 1^{\top}N1 with N=diag⁡(∣Ib∣)N = \operatorname{diag}(|I_b|). A positive shape term does not by itself imply that such a step beats Muon:

Proposition 4 (Same-norm reallocation; Section A.3). Let N≻0N \succ0, r=H1−gr = H1 - g the model gradient at Muon’s step, and pp the component of rr orthogonal to N1N1. (i) If p≠0p \ne0, some cc with c⊤Nc=1⊤N1c^{\top}Nc = 1^{\top}N1 has q(c)<q(1)q(c) < q(1). (ii) If r=λN1r = \lambda N1 and H−λN⪰0H - \lambda N \succeq0 for some λ\lambda, no such cc exists. (iii) The shape term does not decide between (i) and (ii). For H=diag⁡(1,3)H = \operatorname{diag}(1,3), g=(0,2)g = (0,2) and N=IN = I, the shape term is 16>0\frac{1}{6} > 0, yet Muon is optimal among all same-norm steps.

We therefore also compute the best same-norm step, an exactly solvable trust-region problem whose solution is certified globally optimal on the ellipsoid (Theorem A.22), and the model’s prediction for the norm-matched filters we train with. Two remarks make this the right comparison. First, every hidden matrix has rank 768 and bands are defined by rank, so each matrix’s step has squared norm proportional to ∑b∣Ib∣cb2\sum_b |I_b|c_b^2; matching each matrix’s norm separately, as the experiments do, is the same constraint for exact-polar bands. Second, the ratios gb/Hbbg_b/H_{bb} are not the fixed-norm criterion: even with diagonal HH, the band with the larger ratio can be the one a same-norm step should shrink (Theorem A.27); only pp, which uses the full HH, decides.

Estimation and validity. We take checkpoints of a Muon training run (§ 6) at steps 1,024 (constant learning rate), 1,536 (start of the cooldown) and 2,048 (end), and form the exact next Nesterov input. Bands are ranks {1–8,9–32,33–96,97–256,257–512,513–768}\{1\text{--}8, 9\text{--}32, 33\text{--}96, 97\text{--}256, 257\text{--}512, 513\text{--}768\} of every matrix. gg, the Hessian–vector products for HH, and realized loss changes use three disjoint held-out sets (393k, 98k and 393k tokens; Section B). Across 12 test steps (Muon and c⋆c^{\star}; bands and neurons; three checkpoints), the predicted and realized changes have the same sign in every case and agree within a factor of three (Figure 3). Most importantly, the c⋆c^{\star} step realizes a much larger decrease than Muon’s step on unseen tokens: −0.00730-0.00730 against −0.00054-0.00054 at step 1,024. HH is ill-conditioned (condition number ∼104\sim10^4), so c⋆c^{\star} is irregular across bands (Section B) and we use it only through realized losses and Theorem 3.

Predicted versus realized held-out loss change

Figure 3. Predicted (quadratic model) versus realized held-out loss change for 12 test steps.

What the anatomy shows

F1: Muon over-steps its top singular directions. At every checkpoint the per-band multiplier rises monotonically from the top band to the tail (Figure 1a). The tail-to-top ratio is 48 at step 1,024, 38 at the start of the cooldown and 190 at the end. Within each of the four layer types separately, the top band has the smallest multiplier in all twelve type–checkpoint cases (Section B). The raw quantities show the mechanism. At step 1,024 the top band’s curvature H11H_{11} is 29×29\times that of the next band, but its gradient alignment g1g_1 is only 9×9\times larger. Curvature falls off faster than signal with rank. Equal step lengths therefore over-step the high-curvature top directions, in which gradients concentrate ([17, 14, 10]). The ratios are isolated-band quantities, however. At fixed norm, the full-curvature analysis (F2) confirms the top as the place to take step length from but gives it to ranks 97–512, not to the extreme tail.

Figure 1. Anatomy of Muon’s step (124M decoder, Muon seed 43; screen schedule with cooldown over steps 1,536–2,048). (a) Per-band second-order optimal multiplier cb=gb/Hbbc_b = g_b/H_{bb} for the exact-polar step restricted to each singular-value band. Muon’s flat filter is cb=1c_b = 1 (dashed). Multipliers scale as 1/η1/\eta, and η\eta is 10× smaller at step 2,048, so compare profiles rather than levels. The tail-to-top ratio is 48, 38 and 190 at the three checkpoints. These isolated-band ratios are not the fixed-norm criterion (Theorem A.27); panel (b) uses the full curvature. (b) Measured change of the held-out loss after one step at Muon’s own update norm (393k tokens; Table 4). Bars show Muon, the exact tilt σ−1/4\sigma^{-1/4}, halving ranks 1–8, and the best same-norm filter from the quadratic model (Theorem 4). Moving step length off the top helps severalfold mid-training and at the start of the cooldown, and not at its end, where Muon’s top band is no longer over-stepped. (c) Spread (coefficient of variation) of per-neuron update norms of Muon’s step, mean over the 12 matrices of each type, at step 1,024. The fused 2304 × 768 QKV matrix carries the largest imbalance. Block-wise polar factors halve the QKV spread and nearly remove the MLP spread.

F2: at Muon’s norm, moving step length off the top helps until late in the cooldown. At step 1,024 and at the start of the cooldown, only 17% and 12% of the gap of Theorem 3 is shape. The rest is the greedy model asking for a smaller step (γ⋆=0.52,0.49\gamma^{\star} = 0.52, 0.49). Even so, most of the gap is reachable without changing the step’s norm. The model gradient is almost entirely tangent to the same-norm ellipsoid (∥p∥/∥r∥=1.00\lVert p\rVert/\lVert r\rVert= 1.00 and 1.00), so by Theorem 4 a same-norm step does better. The best one is predicted to change the one-step loss by −0.00589-0.00589 and −0.00848-0.00848, against −0.00079-0.00079 and +0.00026+0.00026 for Muon; that is 87% and 92% of the full gap. It takes step length from ranks 1–96 and gives it to ranks 97–512 (Section B). We confirmed this by measuring the actual held-out loss after one step at matched norm (Figure 1b, Table 4). Relative to Muon, halving ranks 1–8 changes it by −4.4-4.4 and −3.2-3.2 (×10−3\times10^{-3}), and the best same-norm filter by −4.5-4.5 and −3.4-3.4, which is 77% and 69% of the predicted gain. Along the tilt family σp\sigma^p, dL/dp=+0.090dL/dp = +0.090 and +0.060+0.060 at p=0p = 0, so a negative power helps and a positive one hurts. A proved certificate (Theorem A.26) turns a predicted gain into a guaranteed one once the model’s errors are bounded; its constants are not yet measured (Section E). As a consistency check, the observed errors, at most 5.4×10−45.4 \times10^{-4} at 12 calibration steps and 1.41.4–1.5×10−31.5 \times10^{-3} for the same-norm gains, are well below the predicted gains here and far above the one at the end. At the end of the cooldown the picture changes. The top band is no longer over-stepped (c1=0.98c_1 = 0.98), the model predicts a gain of only −0.00009-0.00009, and the measured one-step losses of all filters agree with Muon’s to within 1.2×10−41.2 \times10^{-4} (dL/dp≈0dL/dp \approx0). Theorem 3 still attributes 97% of the gap to shape there, but that gap is measured against a step of very different norm and is out of reach at Muon’s norm. The headroom a norm-matched tilt can use is therefore largest early in the cooldown and gone by its end.

Stepp=−12p=-\frac{1}{2}−14-\frac{1}{4}−0.1-0.100 (Muon)+0.1+0.1+14+\frac{1}{4}ranks 1–8 ×12\times\frac{1}{2}best same-normdLdp∣0\left.\frac{dL}{dp}\right|_0
1,024−3.07−5.75−5.93−1.76+12.08+80.45−6.14−6.28+0.090
1,536−2.48−4.13−3.99−1.03+7.95+48.44−4.27−4.40+0.060
2,048−0.13−0.26−0.32−0.33−0.38−0.22−0.29−0.21−0.000

Table 4. Measured change of the held-out loss (×10−3\times10^{-3}) after one step at Muon’s own update norm: the exact tilt σp\sigma^p for several pp, halving ranks 1–8, and the best same-norm filter, plus the slope dL/dpdL/dp at p=0p=0.

F3: neuron imbalance is concentrated in fused QKV. Per-neuron update norms of Muon’s step vary most in the fused QKV matrix (CV 0.43, per-layer range 0.14–0.73). The MLP’s expand rows and contract columns have CV 0.18, and the square attention-output matrix has CV 0.02 (Figure 1c). Grouping neurons by Muon’s update size gives the same monotone picture as the bands: the most-moved sextile has multiplier 1.5 and the least-moved 7.1 (Section B). Only 5% of the neuron-group gap at step 1,024 is shape (65% at the end), so any benefit of block-wise steps must come mostly from the new directions they produce, not from reweighting.

Implications. We test three implications in § 6. I1: rebalancing should matter most where the imbalance is largest, in QKV, and little in the MLP. I2: at fixed norm, moving step length off the top of the spectrum (p<0p < 0) should help, independently of I1, because it acts on different structure. By F2 its per-step benefit is largest from mid-training to the start of the cooldown and gone by the end, because the decaying learning rate itself removes the top’s over-stepping. A tilt confined to the cooldown should therefore build an early lead that partly erodes. I3: changes that barely alter the allocation should not help. Making NS5_5 a more exact polar map is one: it removes NS5_5’s small deviations from unit length, mostly by lifting near-null directions, and leaves the top’s share relative to the bulk essentially unchanged.

From measurements to interventions

Neuron allocation: block-wise polar steps. Following Theorem 2, we orthogonalize blocks separately. Split QKV uses three square 768 × 768 blocks, one each for Q, K and V. Per-head uses 36 wide 64 × 768 blocks, whose polar factors have exactly orthonormal rows. Square blocks splits QKV and also splits the MLP’s 3072 × 768 and 768 × 3072 matrices into four square blocks each. Wide matrices’ column blocks are scaled by 1/k1/\sqrt{k} to keep Muon’s update energy. All reported block arms are norm-matched anyway; the factor matters for the deployable version (Section D). The stateful alternative from prior work is NorMuon’s row normalization by an EMA of row second moments ([27]).

Spectral allocation: a matmul-only cooldown tilt

The exact tilt Uϕ5(Σ)(Σ/σrms)pVTU\phi_5(\Sigma)(\Sigma/\sigma_{\mathrm{rms}})^pV^{\mathsf{T}} needs an eigendecomposition per matrix and step. We avoid it as follows. Let Q=NS5(N)=Udiag⁡(fj)VTQ = NS_5(N) = U\operatorname{diag}(f_j)V^{\mathsf{T}} with fj=ϕ5(σ^i)f_j = \phi_5(\hat{\sigma}_i), and S=ONT=Udiag⁡(fiσi)UT⪰0S = ON^{\mathsf{T}} = U\operatorname{diag}(f_i\sigma_i)U^{\mathsf{T}} \succeq0. A polynomial in SS acts in the same singular basis (Theorem 5):

D=P(S/Λ)Q=Udiag⁡(fjP(fiσi/Λ))VT.D = P(S/\Lambda)Q = U\operatorname{diag}\left(f_jP(f_i\sigma_i/\Lambda)\right)V^{\mathsf{T}}.

With P(x)≈xpP(x) \approx x^p the filter is fi1+pσipf_i^{1+p}\sigma_i^p up to scale. On the Newton–Schulz plateau (fi∈[0.68,1.20]f_i \in[0.68, 1.20]) this is the exact tilt fiσipf_i\sigma_i^p up to a factor in [0.96,1.10][0.96, 1.10]; near-null directions are lifted further. We use a degree-12 least-squares Chebyshev fit of xpx^p on [10−3,1][10^{-3}, 1]. Λ\Lambda is 1.01 times a 20-step power-iteration estimate of λmax⁡(S)\lambda_{\max}(S), which is an estimate rather than a proved bound. We then rescale DD to ∥O∥F\lVert O\rVert_F (Algorithm 1).

Proposition 5 (What the tilt computes; Section A.1). For any coefficients and depth kk and any real polynomial PP, let O=ϕk(X)O = \phi^k(X) and D=P(OXT)OD = P(OX^{\mathsf{T}})O. (i) DD has the form f(XTX)f(X^{\mathsf{T}}X), so D(XQ)=D(X)QD(XQ) = D(X)Q for every orthogonal QQ. (ii) If X=Udiag⁡(d)VTX = U\operatorname{diag}(d)V^{\mathsf{T}} with orthonormal columns, then O=Udiag⁡(fi)VTO = U\operatorname{diag}(f_i)V^{\mathsf{T}} and D=Udiag⁡(fiP(fidi))VTD = U\operatorname{diag}(f_iP(f_id_i))V^{\mathsf{T}}, with fif_i the scalar iterate of did_i.*

Part (ii), not (i), is what keeps the singular vectors: X↦AXX \mapsto AX is also a Gram map. The normalization, Λ\Lambda and the final rescaling are scalars. The tilted step therefore changes only how singular directions are weighted, so a comparison with Muon isolates the spectral allocation. One numerical detail decides whether the method works. Evaluating PP in monomials by Horner’s rule in TF32 cancels catastrophically (coefficients reach 4×1044 \times10^{4}) and flipped the filter’s sign on the top direction. With square blocks as the base (−0.0219 alone, against fused-QKV Muon), that version reached −0.0240, against −0.0322 for Clenshaw’s recurrence [9, 45] in FP32 and −0.0332 for the exact tilt; the Clenshaw filter matches the exact tilt to direction cosine 0.998–0.999, so the normalized steps differ by at most 6.3% in Frobenius norm.

Experiments

Setup. We train a 124.5M-parameter pre-norm decoder (12 layers, width 768, 12 heads, tied embeddings) on a 167.8M-token FineWeb-Edu sample [34] with the GPT-2 tokenizer, sequence length 512 and 24,576 tokens per step. Muon (1) acts on the 48 hidden matrices with momentum 0.95 and decoupled weight decay 0.1. Its state is in FP32, and Newton–Schulz runs in FP32 with TF32 matrix products. AdamW [29] (3×10−43 \times10^{-4}) handles embeddings and norms, and gradients are clipped at norm 1.0. The schedule is warmup–constant–cooldown [18]: 128 warmup steps, then linear decay to 0.1× over the last 25% of steps. The screen runs 2,048 steps (50.3M tokens). The confirmation runs 4,096 steps (100.7M tokens). The metric is final loss on 4.18M held-out tokens. A 355M model (24 layers, width 1024, 16 heads) tests scale.

Controls. Norm matching: unless stated otherwise, every direction-changing arm is rescaled, per matrix and per step, to the Frobenius norm of Muon’s NS5 output on the same momentum, so differences reflect direction, not step size. Tuned baselines: on the screen, Muon’s learning rate was swept over {0.0035, 0.005, 0.007, 0.01}, split-QKV Muon’s over {0.0035, 0.005, 0.007}, and the 355M baseline’s over the same three values. 0.005 is an interior optimum in all three cases, and every new arm uses the baseline’s rate. Paired seeds: each seed fixes initialization and data order. Confirmations use seeds 42–44 for every arm, and we report paired 95% tt-intervals. A/A check: before the cooldown the tilt arms run an optimizer mathematically identical to their baselines. Their intermediate evaluations differ by at most 0.009 nats, an empirical bound on numerical non-determinism plus evaluation noise. The screen baseline’s seed spread is 0.0032 nats. Gating: midway through the study we adopted a rule that a full-length run is launched only if its screen beats the baseline by more than 0.005 nats. The factorial ablations and the 355M seeds 43–44 were released under it; the earlier confirmations were not.

OptimizerLossΔ\Delta vs. MuonΔ\Delta vs. split-QKV
Muon (fused QKV)3.9427—+0.0250
NorMuon (fused QKV)3.9277−0.0150+0.0100 [−0.0003, +0.0017]
Square blocks3.9187−0.0239+0.0011 [−0.0043, +0.0051]
Split-QKV Muon3.9176−0.0250—
+ cooldown tilt3.9057−0.0370−0.0120 [−0.0014, −0.0100]
+ NorMuon3.9139−0.0288−0.0038 [−0.0006, −0.0061]
+ NorMuon + tilt3.9018−0.0408−0.0158 [−0.0018, −0.014]
+ NorMuon + tilt + Schatten-1 decay3.8966−0.0460−0.0210 [−0.0024, −0.018]
Sq. blocks + NorMuon + tilt3.9050−0.0376−0.0126 [−0.0015, −0.0100]

Table 1. Confirmation at 100.7M tokens (124M model, three paired seeds per arm). Mean final held-out loss and mean paired differences against the original fused-QKV Muon and against the stronger split-QKV Muon. Brackets give 95% tt-intervals for the latter; intervals against fused-QKV Muon are in the text. Lower is better.

I1: the neuron gain is the QKV split. Splitting Q, K and V changes the final loss by −0.0250 nats [−0.0290, −0.0210] relative to fused-QKV Muon (Table 1). It is better in every seed, and fused-QKV Muon would need about 6.8% more tokens to match. Square blocks everywhere add nothing beyond it (+0.0011 [−0.0029, +0.0051]). Square blocks in the MLP alone, with QKV fused, change the loss by +0.0039 (screen). As I1 implies, the large QKV imbalance is the costly one and the moderate MLP imbalance is harmless. The data also locate the effect at the level of projections rather than rows. Per-head blocks equal-size rows exactly yet do worse than three blocks (−0.0172 against −0.0255, screen). NorMuon’s row normalization on fused QKV is worse than the split (+0.0100 [−0.0031, +0.0170]), and on top of the split it adds only −0.0038 [−0.0062, −0.0013]. One reading is that in a fused matrix Q, K and V share a single spectral budget, so the projection with the strongest momentum claims most of it, whereas splitting gives each projection its own budget. We have not isolated this mechanism. Either way, methods evaluated against fused-QKV Muon can take credit for a gain the split provides at no cost.

I2: the tilt helps, and adds to neuron balance. On the split baseline the cooldown tilt σ−1/4\sigma^{-1/4} changes the loss by −0.0120 nats [−0.0137, −0.0102] and is better in all three seeds (−0.0113, −0.0127, −0.0119). The 2×22 \times2 factorial over {NorMuon, tilt} on split-QKV Muon is additive within noise: −0.0038 and −0.0120 separately, −0.0158 [−0.0178, −0.0138] together. Each component’s effect is the same with or without the other: the tilt given NorMuon gives −0.0120, and NorMuon given the tilt gives −0.0038. The tilt supplies about three quarters of the combined gain. By the split baseline’s own loss-versus-tokens curve (0.260 nats per doubling, a two-point log-linear estimate), split-QKV Muon would need about 4.3% more tokens to match the combination. Other tilts and band filters behave consistently on the screen (Section C); the dose was not optimized. Consistent with F2, timing and dose matter little: on the full stack, starting the tilt at step 1,024 or using σ−1/2\sigma^{-1/2} changes the screen loss by only −0.0007-0.0007 and +0.0009+0.0009 (one seed each, within the 0.003 seed spread). We keep the tilt in the cooldown, where it costs least.

How the gain develops. Figure 2b shows the tilt’s lead peaking at −0.050-0.050 nats 256 steps into the cooldown and narrowing to −0.0120-0.0120 at the end. This is the pattern F2 predicts. At fixed norm the per-step benefit of moving step length off the top is large at the start of the cooldown and gone by its end, because the decaying learning rate itself removes the top’s over-stepping, so the baseline recovers part of the early lead. The baseline’s cooldown length and final learning rate were not tuned, so a different cooldown might recover part of the final gain; we have not tested this.

Figure 2. Controlled results at 100.7M tokens (124M, three paired seeds). (a) Paired difference in final loss against split-QKV Muon. Splitting Q, K and V explains the whole square-block gain; NorMuon on fused QKV is worse than the split; the tilt and NorMuon add on top of the split. (b) Paired difference over training (mean, with the seed range shaded). Intermediate points use 98k held-out tokens; the last point uses 4.18M. Before the cooldown the tilt arm is mathematically identical to its baseline (A/A differences ≤0.009\le0.009). The tilt’s lead peaks at −0.050-0.050 nats 256 steps into the cooldown and settles at −0.0120-0.0120.

Scale. At 355M parameters (GPT-2-medium shape, 96 Muon matrices, same data and tokens), the split-QKV baseline’s learning-rate screen again peaks at 0.005 (4.0642/4.0547/4.0647 for 0.0035/0.005/0.007). Against the baseline at the same seeds, NorMuon + tilt changes the final loss by −0.0188-0.0188 nats [−0.0202-0.0202, −0.0175-0.0175] (per seed −0.0188-0.0188, −0.0183-0.0183, −0.0194-0.0194), comparable to the 124M effect; the baseline’s 3 seeds span only 0.0008 nats.

I3 and other geometric changes. Table 8 (Section C) collects them. Three extra cubic Newton–Schulz iterations drive every singular value of the NS5 output to 1 and change the loss by −0.0020-0.0020, within noise, as I3 implies. Input whitening, steepest descent in the data metric ∥D∥op1/2\lVert D\rVert_{\mathrm{op}}^{1/2}, has the cleanest theory of anything we tried. In its exact form it is GL(nn)-equivariant and conditioning-free on least squares (Section A.4); the damped version we run is approximately so. It is harmful: +0.021+0.021 to +0.060+0.060 when norm-matched, and +0.0524+0.0524 at its own magnitude. A whitened step concentrates its energy in low-variance input directions, so at a fixed norm it moves the layer’s outputs less. The published Toner recipe ([31]) gives −0.0068-0.0068 (one seed), the sign of its reported gain at smaller size. For decay geometry, energy-matched Schatten-4 decay, which mainly shrinks the top of the spectrum, hurts (+0.0177+0.0177). Schatten-1 decay, which removes about the same amount from every singular value and so prunes the bulk, helps (−0.0057-0.0057 [−0.0100-0.0100, −0.0014-0.0014], three seeds). Decay along the polar factor of WW is prior art, used with a fixed coefficient for low rank, where with Muon it costs loss ([3]), and for grokking ([26]). With our energy-matched coefficient ∥W∥F2/∥W∥∗\lVert W\rVert_{\mathrm{F}}^2/\lVert W\rVert_* it is additive with our method: −0.0052-0.0052 [−0.0066-0.0066, −0.0038-0.0038] on top of split QKV + NorMuon + tilt at 100.7M tokens (three seeds, Table 1), −0.0210-0.0210 against split-QKV Muon, at +26%+26\% more wall time. In this regime the useful role of decay is to clean the bulk, not to cap the spectral norm.

ChangeCertified propertyAnat.Δ\Delta
Split QKVsteepest for block max-norm; exact-polar blocks have unit rows; energy identityyes−0.0250-0.02501
MLP square blockssameweak+0.0039+0.0039
Cooldown tiltkeeps Muon’s O(n)O(n)-equivarianceyes−0.0120-0.01202
Accurate polar—no−0.0020-0.0020
Input whiteningGL(n)GL(n)-equivariance; conditioning-free LS (undamped form)—+0.021+0.021–0.0600.060
Toner recipe(prior art)—−0.0068-0.0068
Schatten-4 decay——+0.0177+0.0177
Schatten-1 decay——−0.0057-0.00573

Table 8. Table 8: Theory versus training. What the Lean statements certify, whether the anatomy implies a gain, and the measured paired Δ\Delta against fused-QKV Muon (screen, one seed, unless marked). Formal properties alone did not predict which changes help.

Cost. Splitting QKV adds no FLOPs; its +7%+7\% recorded wall time is mostly the norm-matching scaffold. In our unfused FP32 implementation NorMuon + tilt takes +12%+12\% more wall time than split-QKV Muon at 124M and +12%+12\% at 355M, against a token saving of about 4.3%. The gain is in tokens, not yet in wall clock; folding the tilt into the Newton–Schulz coefficient, as DynMuon does, may remove most of the overhead.

What formal verification did and did not buy

All formal results are stated in Section A. The Newton–Schulz results hold for the iteration Muon actually runs, with any coefficients, depth and normalizer. The block results concern exactly orthogonal blocks. What it bought. (i) Bookkeeping. The energy identity (Theorem 2), carried over to wide matrices, gives the 1/k1/\sqrt{k} factor without which a deployable block step on the MLP contraction is 1.9×\times too large. (ii) A symmetry guarantee. The cheap tilt keeps Muon’s right equivariance (Theorem 5). (iii) *A correction to our own reasoning. Theorem 3’s stationary points are optima only under positive definite curvature, and its shape term compares Muon with a step of a different norm. Formalizing the question our norm-matched experiments ask produced Theorem 4 and a finite-step certificate, and changed our reading of the diagnostic (F2). (iv) A change of direction. Earlier counterexamples (Section A) showed that spatial orientation and curvature, not temporal statistics, decide momentum preferences under NS5_5. What it did not buy was a prediction of training benefit. Whitened Muon is provably equivariant and conditioning-free yet hurts; the joint polar factor is provably suboptimal for the block norm, yet MLP blocks do not help. Whether a certified property is the bottleneck is an empirical question.

Related work

Muon and its geometry. Muon (Jordan et al., [21]) and its scaling studies ([28, 13]) have been analyzed as steepest descent under operator norms ([5, 24]), as linear-minimization-oracle methods ([35, 37]), as non-Euclidean trust regions (Kovalev, [23]) and as constrained optimization ([8, 40]). Prior work also studies polar approximations (Amsel et al., [2]) and the benefit of flat spectra for tail-end memories (Wang et al., [48]). On matrix quadratics, Shulgin et al. [41] show that the polar direction beats the gradient direction, each at its best step size, when large singular directions sit in high curvature, and that finite Newton–Schulz acts as a damped polar update.

Spectra, neurons and blocks. DynMuon schedules a spectral power from +1+1 to −14-\frac{1}{4} (Wu et al., [50]). Freon/Kaon show that random or inverted spectra can work as well as flat ones (Shumaylov et al., [42]), and Spectrally Targeted Muon, AdaMuon and PolarGrad adapt magnitudes ([15, 43, 25]). NorMuon and Muon+ normalize rows ([27, 54]), Aurora gives the leverage-score argument (Dewulf et al., [11]), and row-sphere and Stiefel constraints are compared by Hu et al. [19]. MuonBP and HiMuon orthogonalize blocks ([22, 44]), and separate Q/K/V orthogonalization is standard in modded-nanogpt (Jordan et al., [20]). Closest to our diagnostic, SAMuon measures held-out optimal step sizes along each singular direction of the momentum, finding a volatile head and a tolerant bulk, and builds an optimizer from the profile (Wu et al., [51]). BulkBoost proves when a norm-matched bulk/spike reweighting improves an objective to first order (Chaitanya et al., [6]), and Wang et al. [47] attribute Muon’s advantage over Adam to within- and cross-layer curvature of its update. Our tilt uses DynMuon’s dose and late-training sign. We add the full cross-group curvature, neuron groups, scale versus fixed-norm shape with proved criteria, and a controlled attribution: the QKV split accounts for the block-wise gains and most of NorMuon’s gain over fused-QKV Muon.

Data metric, curvature and evaluation. Shampoo, SOAP and K-FAC precondition with layer statistics ([16, 46, 30]), and whitened Muon has close 2026 relatives ([31, 7, 55, 1, 53]). Gradients concentrate in a few top Hessian directions (Gur-Ari et al., [17]) above a bulk ([38, 14, 32]), near the edge of stability (Cohen et al., [10]). Greedy step sizes suffer short-horizon bias ([52, 4]), and optimizer gains shrink against tuned baselines ([49, 56]). Like Wu et al. [51] and Wang et al. [47], we measure curvature along the optimizer’s own step, here with held-out Hessian–vector products (Pearlmutter, [33]). Formal verification of ML systems goes back at least to Selsam et al. [39].

Novelty. Held-out spectral probing of Muon’s step (Wu et al. [51]) and first-order analysis of norm-matched reallocation (Chaitanya et al. [6]) precede this work. We found no prior work that measures the full cross-group curvature of the step by singular band and by neuron group, separates scale from fixed-norm shape with proved criteria and counterexamples, and uses the result in a controlled attribution of published ingredients.

Limitations and conclusion

Scale and regime. One data distribution; 124M and 355M parameters (three seeds each); at most 100.7M tokens, under one token per parameter. Effect sizes are specific to this regime. The diagnostic is local and noisy. It is second-order, one step ahead and from a single run, without intervals. gg and HH come from separate held-out folds, and its joint optimum is ill-conditioned. It perturbs only the hidden matrices along exact-polar band directions, leaving out the AdamW and decay parts of the update. The measured one-step losses use one checkpoint seed and one validation subset, without intervals, and the certificate’s constants are not yet measured. We used the diagnostic for direction, not dose; the tilt power comes from prior work. The fixed-norm analysis covers bands only. Design and statistics. The anatomy was not pre-registered; intervals are nominal (three paired seeds); most screen arms are single-seed. Alternatives. The baseline’s cooldown was not tuned; the QKV mechanism is a hypothesis. Provenance. Comparisons with fused-QKV or square-block arms cross code versions; split-QKV arms share one (Section E).

Conclusion. Measured against curvature, both of Muon’s hidden allocations are miscalibrated. Its spectrum oversteps the top, which a same-norm reallocation can exploit until late in the cooldown, and its per-neuron steps are most uneven in fused QKV matrices. Corrections aimed at these help additively; principled changes that miss them do not.

Formal results

This appendix states, in ordinary mathematical language, every formal result the paper uses. All of them are proved in Lean 4 with Mathlib, in exact real arithmetic. They certify the algebra and geometry behind each method. They say nothing about floating-point behaviour, convergence or training benefit.

Notation. ⟨A,B⟩=tr⁡(ATB)\langle A, B\rangle= \operatorname{tr}(A^{\mathsf{T}}B). For real coefficients a,b,ca,b,c let φ(X)=aX+bXXTX+c(XXT)2X\varphi(X) = aX + bXX^{\mathsf{T}}X + c(XX^{\mathsf{T}})^2X, and let φk\varphi^k be its kk-fold composition. For any function ν\nu of an m×mm \times m matrix, the Newton–Schulz map is

NS⁡(X)=φk(ν(XXT)X)\operatorname{NS}(X) = \varphi^k\left(\nu(XX^{\mathsf{T}})X\right)

Muon’s NS⁡5\operatorname{NS}_5 is the case k=5k=5, (a,b,c)=(3.4445,−4.775,2.0315)(a,b,c)=(3.4445,-4.775,2.0315), ν(XXT)=1/tr⁡XXT\nu(XX^{\mathsf{T}})=1/\sqrt{\operatorname{tr}XX^{\mathsf{T}}}. Results stated for NS hold for every choice of a,b,c,ka,b,c,k and ν\nu, and so apply to the iteration Muon actually runs, not only to an exact polar factor. A matrix QQ is orthogonal if QQT=IQQ^{\mathsf{T}}=I.

Newton–Schulz maps and the matmul-only tilt

Definition A.1 (Gram map). A map G:Rm×n→Rm×nG:\mathbb{R}^{m \times n}\to\mathbb{R}^{m \times n} is a Gram map if there is a function F:Rm×m→Rm×mF:\mathbb{R}^{m \times m}\to\mathbb{R}^{m \times m} such that G(X)=F(XXT)XG(X)=F(XX^{\mathsf{T}})X for every XX.

Theorem A.2 (Gram maps commute with right rotations). If GG is a Gram map and Q∈Rn×nQ\in\mathbb{R}^{n \times n} is orthogonal, then G(XQ)=G(X)QG(XQ)=G(X)Q for every XX.

Proof idea. (XQ)(XQ)T=XXT(XQ)(XQ)^{\mathsf{T}}=XX^{\mathsf{T}}, so G(XQ)=F(XXT)XQG(XQ)=F(XX^{\mathsf{T}})XQ.

Theorem A.3 (Newton–Schulz iterates are Gram maps). For all a,b,ca,b,c and kk, the map X↦φk(X)X\mapsto\varphi^k(X) is a Gram map. Consequently NS⁡(XQ)=NS⁡(X)Q\operatorname{NS}(XQ)=\operatorname{NS}(X)Q for every orthogonal QQ, for any normalizer ν\nu.

Proof idea. Each step multiplies XX on the left by the matrix aI+bXXT+c(XXT)2aI+bXX^{\mathsf{T}}+c(XX^{\mathsf{T}})^2. That matrix depends only on the Gram matrix, and the class of Gram maps is closed under composition.

Theorem A.4 (The polynomial tilt preserves Muon’s symmetry). Let PP be any real polynomial and O=φk(X)O=\varphi^k(X). The tilt TP(X)=P(OXT)OT_P(X)=P(OX^{\mathsf{T}})O is a Gram map. Hence TP(XQ)=TP(X)QT_P(XQ)=T_P(X)Q for every orthogonal QQ.

Right equivariance alone does not mean singular vectors are kept: X↦AXX\mapsto AX for a fixed non-scalar AA is also a Gram map. The next theorem shows that the Newton–Schulz iterates and the tilt do keep them.

Theorem A.5 (Singular-basis action of Newton–Schulz and of the tilt). Let X=Udiag⁡(d)VTX=U\operatorname{diag}(d)V^{\mathsf{T}}, where UU and VV have orthonormal columns (UTU=IU^{\mathsf{T}}U=I, VTV=IV^{\mathsf{T}}V=I; any shapes, e.g. a thin SVD). Let sks_k be the scalar map x↦ax+bx3+cx5x\mapsto ax+bx^3+cx^5 iterated kk times, and fi=sk(di)f_i=s_k(d_i). (i) φk(X)=Udiag⁡(fi)VT\varphi^k(X)=U\operatorname{diag}(f_i)V^{\mathsf{T}}. (ii) For every real polynomial PP, P(φk(X)XT)φk(X)=Udiag⁡(fiP(fidi))VTP(\varphi^k(X)X^{\mathsf{T}})\varphi^k(X)=U\operatorname{diag}\left(f_iP(f_id_i)\right)V^{\mathsf{T}}.

Proof idea. XXT=Udiag⁡(di2)UTXX^{\mathsf{T}}=U\operatorname{diag}(d_i^2)U^{\mathsf{T}} and UTU=IU^{\mathsf{T}}U=I make every product collapse onto the diagonal. Induct on kk for (i); for (ii), induct on the powers of SS, then use linearity in PP.

In the implementation XX is first normalized and SS is divided by an estimate Λ\Lambda of its largest eigenvalue. Both are scalar factors, absorbed into dd and PP. Both singular frames are therefore kept, and the implemented filter is fiP(fiσi/Λ)f_iP(f_i\sigma_i/\Lambda), not fiP(σi)f_iP(\sigma_i). With P(x)≈xpP(x)\approx x^p it is fi1+pσipf_i^{1+p}\sigma_i^p up to scale. On the Newton–Schulz plateau (fi∈[0.68,1.20]f_i\in[0.68,1.20]) this differs from the exact tilt fiσipf_i\sigma_i^p by the factor fipf_i^p, which lies in [0.96,1.10][0.96,1.10] for p=−14p=-\frac14.

Theorem A.6 (From filter error to step error). (i) For non-zero Y,Y^Y,\widehat{Y} in a real normed space and ρ≥0\rho\geq0, ∥ρY^/∥Y^∥−ρY/∥Y∥∥≤2ρ∥Y^−Y∥/∥Y∥\left\|\rho\widehat{Y}/\|\widehat{Y}\|-\rho Y/\|Y\|\right\|\leq2\rho\|\widehat{Y}-Y\|/\|Y\|. (ii) For unit vectors u,vu,v in a real inner-product space, ∥u−v∥2=2(1−⟨u,v⟩)\|u-v\|^2=2(1-\langle u,v\rangle).

By (i), a raw filter error of ϵ∥Y∥\epsilon\|Y\| moves the norm-matched step by at most 2ϵ2\epsilon of its norm. By (ii), the measured direction cosine of 0.998 between the matmul and exact tilts means their normalized steps are within 6.3%.

Block geometry and per-neuron step sizes

Definition A.7 (Steepest direction). For a norm-like function N\mathcal{N} and a gradient GG, a matrix DD is a steepest direction if N(D)≤1\mathcal{N}(D) \le1 and ⟨G,D′⟩≤⟨G,D⟩\langle G,D'\rangle\le\langle G,D\rangle for every D′D' with $\mathcal{N}(D') \le 1.

Theorem A.8 (Block-wise steepest descent). Let G=(G1,…,Gp)G = (G_1,\ldots,G_p) be a block gradient and N1,…,Np\mathcal{N}_1,\ldots,\mathcal{N}_p block norms. If each DiD_i is a steepest direction for Ni\mathcal{N}_i at GiG_i, then (D1,…,Dp)(D_1,\ldots,D_p) is a steepest direction at GG for the max-norm ∥D∥max⁡=max⁡iNi(Di)\lVert D\rVert_{\max} = \max_i \mathcal{N}_i(D_i), whose unit ball is the product of the block unit balls. With every Ni\mathcal{N}_i the spectral norm, the block-wise polar factors are therefore steepest for ∥⋅∥blk\lVert\cdot\rVert_{\mathrm{blk}}.

Proposition A.9 (The joint polar step can be strictly worse). Take two 1×21 \times2 blocks G1=G2=(1,0)G_1 = G_2 = (1,0). The polar factor of the stacked matrix has blocks D1=D2=(1/2,0)D_1 = D_2 = (1/\sqrt{2},0), a partial isometry with D1TD1+D2TD2=e1e1TD_1^{\mathsf{T}}D_1 + D_2^{\mathsf{T}}D_2 = e_1e_1^{\mathsf{T}}, and attains ⟨G,D⟩=2\langle G,D\rangle= \sqrt{2}. The block-wise step D1=D2=(1,0)D_1 = D_2 = (1,0) is feasible for the max-norm and attains 2>22 > \sqrt{2}.

Proposition A.10 (Row norms of polar factors). (i) If OO is square and OOT=IOO^{\mathsf{T}} = I, every row of OO has unit norm. (ii) If O∈Rm×kO \in\mathbb{R}^{m \times k} has orthonormal columns (OTO=Ik)(O^{\mathsf{T}}O = I_k), its squared row norms sum to kk. (iii) They need not be equal: O=(1;0)∈R2×1O = (1;0) \in\mathbb{R}^{2 \times1} satisfies OTO=1O^{\mathsf{T}}O = 1 and has row norms 11 and 00.

The squared row norms in (ii) are the leverage scores of range⁡(O)\operatorname{range}(O). In Muon’s step they are the per-neuron step sizes of a tall matrix.

Proposition A.11 (Energy identity for block-wise steps). Let O∈Rm×kO \in\mathbb{R}^{m \times k} with OTO=IkO^{\mathsf{T}}O = I_k, let B1,…,Bp∈Rk×kB_1,\ldots,B_p \in\mathbb{R}^{k \times k} be orthogonal, and let c2=pc^2 = p. Then

∥cO∥F2=∑b=1p∥Bb∥F2  (=pk).\lVert cO\rVert_{\mathrm{F}}^2 = \sum_{b=1}^{p}\lVert B_b\rVert_{\mathrm{F}}^2 \;(= pk).

For a tall matrix with m=pkm = pk rows, Muon’s shape factor s=m/k=ps = \sqrt{m/k} = \sqrt{p} therefore gives the joint step the same Frobenius energy as pp orthogonal k×kk \times k blocks, and the learning rate transfers.

Second-order model of a filtered step

For group multipliers c∈RBc \in\mathbb{R}^{B}, a gradient-alignment vector gg and a symmetric curvature matrix HH, let q(c)=−gTc+12cTHcq(c) = -g^{\mathsf{T}}c + \frac{1}{2}c^{\mathsf{T}}Hc (Equation (2)).

Theorem A.12 (Gap formula). If HH is symmetric and Hc⋆=gHc^\star= g, then for every cc,

q(c)−q(c⋆)=12(c−c⋆)TH(c−c⋆).q(c) - q(c^\star) = \frac{1}{2}(c-c^\star)^{\mathsf{T}}H(c-c^\star).

Theorem A.13 (Scale/shape split). Let HH be symmetric with Hc⋆=gHc^\star= g, let uu be a direction (Muon’s step is u=1u = 1), and let γ\gamma satisfy uTH(γu−c⋆)=0u^{\mathsf{T}}H(\gamma u-c^\star) = 0. Then

q(u)−q(c⋆)=12(1−γ)2uTHu+12(γu−c⋆)TH(γu−c⋆).q(u) - q(c^\star) = \frac{1}{2}(1-\gamma)^2u^{\mathsf{T}}Hu + \frac{1}{2}(\gamma u-c^\star)^{\mathsf{T}}H(\gamma u-c^\star).

When HH is positive definite (Theorem A.16), the first term is the loss due to the overall step size along uu and the second the loss due to the shape of uu.

Proof idea. Write u−c⋆=(1−γ)u+(γu−c⋆)u-c^\star= (1-\gamma)u + (\gamma u-c^\star) and apply Theorem A.12. The cross term vanishes by the condition on γ\gamma, so the identity is Pythagoras in the HH-inner product.

Lemma A.14 (The best uniform rescaling is an HH-projection). If uTHu≠0u^{\mathsf{T}}Hu \ne0, then γ⋆=uTg/uTHu\gamma^\star= u^{\mathsf{T}}g/u^{\mathsf{T}}Hu satisfies uTH(γ⋆u−c⋆)=0u^{\mathsf{T}}H(\gamma^\star u-c^\star) = 0. So Theorem A.13 applies with γ=γ⋆\gamma= \gamma^\star, the best scalar multiple of uu.

Lemma A.15 (The shape term is strictly positive). If HH is positive definite and γu≠c⋆\gamma u \ne c^\star, then 12(γu−c⋆)TH(γu−c⋆)>0\frac{1}{2}(\gamma u-c^\star)^{\mathsf{T}}H(\gamma u-c^\star) > 0. Muon’s flat step therefore loses something to shape unless the second-order optimum is itself flat.

When the stationary points are optima. Theorems A.12 and A.13 are identities that hold for any symmetric HH. Calling c⋆c^\star and γ⋆\gamma^\star optima, and the two terms of Theorem A.13 losses, needs curvature assumptions:

Theorem A.16 (Stationary points are minimizers under positive curvature). Let HH be symmetric and Hc⋆=gHc^\star= g. (i) If HH is positive semidefinite, then q(c⋆)≤q(c)q(c^\star) \le q(c) for every cc. (ii) If HH is positive definite, then q(c⋆)<q(c)q(c^\star) < q(c) for every c≠c⋆c \ne c^\star.

Lemma A.17 (The best uniform rescaling minimizes along the ray). If uTHu>0u^{\mathsf{T}}Hu > 0, then q(γ⋆u)≤q(γu)q(\gamma^\star u) \le q(\gamma u) for every γ\gamma, where γ⋆=uTg/uTHu\gamma^\star= u^{\mathsf{T}}g/u^{\mathsf{T}}Hu. Indeed q(γu)−q(γ⋆u)=12uTHu(γ−γ⋆)2q(\gamma u) - q(\gamma^\star u) = \frac{1}{2}u^{\mathsf{T}}Hu(\gamma-\gamma^\star)^2.

Comparison at a fixed update norm. The shape term of Theorem A.13 compares uu with c⋆c^\star, which generally has a different norm. Experiments that norm-match every arm compare steps on the ellipsoid {c:cTNc=uTNu}\{c : c^{\mathsf T}Nc = u^{\mathsf T}Nu\} for a symmetric positive definite NN. Write r=Hu−gr = Hu - g for the gradient of qq at uu, w=Nuw = Nu, and

p=r−rTwwTwwp = r - \frac{r^{\mathsf T}w}{w^{\mathsf T}w}w

for the component of rr orthogonal to ww.

Theorem A.18 (Lagrange identity on the same-norm ellipsoid). Let HH and NN be symmetric. For every cc with cTNc=uTNuc^{\mathsf T}Nc = u^{\mathsf T}Nu and every real λ\lambda,

q(c)−q(u)=(r−λNu)T(c−u)+12(c−u)T(H−λN)(c−u).q(c)-q(u)=(r-\lambda Nu)^{\mathsf T}(c-u)+\frac{1}{2}(c-u)^{\mathsf T}(H-\lambda N)(c-u).

Proof idea. Expand q(u+d)−q(u)q(u+d)-q(u), and subtract λ2\frac{\lambda}{2} times the constraint, which in terms of d=c−ud=c-u reads $2u^{\mathsf T}Nd+d^{\mathsf T}Nd=0.

Corollary A.19 (Certificate that no same-norm step improves). If r=λNur=\lambda Nu and H−λNH-\lambda N is positive semidefinite for some λ\lambda, then q(u)≤q(c)q(u)\leq q(c) for every cc with cTNc=uTNuc^{\mathsf T}Nc=u^{\mathsf T}Nu.

Theorem A.20 (A same-norm improvement exists when the tangential gradient is non-zero). Let NN be symmetric positive definite, u≠0u\neq0, and p≠0p\neq0. Then there is cc with cTNc=uTNuc^{\mathsf T}Nc=u^{\mathsf T}Nu and q(c)<q(u)q(c)<q(u).

Proof idea. The vector v=−pv=-p satisfies wTv=0w^{\mathsf T}v=0 and rTv=−∥p∥2<0r^{\mathsf T}v=-\lVert p\rVert^2<0. Rescale it to e=κve=\kappa v with eTNe=uTNue^{\mathsf T}Ne=u^{\mathsf T}Nu; then uTNe=0u^{\mathsf T}Ne=0. Every c=xu+yec=xu+ye with x2+y2=1x^2+y^2=1 lies on the ellipsoid. Along the rational parametrization x=(1−s2)/(1+s2)x=(1-s^2)/(1+s^2), y=2s/(1+s2)y=2s/(1+s^2),

q(c)−q(u)=2s[rTe+sα1+s2α2+s3α3](1+s2)2q(c)-q(u)=\frac{2s\left[r^{\mathsf T}e+s\alpha_1+s^2\alpha_2+s^3\alpha_3\right]}{(1+s^2)^2}

for constants αi\alpha_i. Choosing s=min⁡{1,−rTe/(2K)}s=\min\{1,-r^{\mathsf T}e/(2K)\} with K=∣α1∣+∣α2∣+∣α3∣+1K=|\alpha_1|+|\alpha_2|+|\alpha_3|+1 makes the bracket at most rTe/2<0r^{\mathsf T}e/2<0.

Proposition A.21 (A positive shape term does not imply a same-norm improvement). Let H=diag⁡(1,3)H=\operatorname{diag}(1,3), g=(0,2)g=(0,2), u=(1,1)u=(1,1) and N=IN=I. The minimizer is c⋆=(0,23)c^\star=(0,\frac{2}{3}), the best uniform rescaling is γ⋆=12\gamma^\star=\frac{1}{2}, and the shape term of Theorem A.13 equals 16>0\frac{1}{6}>0. Yet q(u)≤q(c)q(u)\leq q(c) for every cc with ∥c∥=∥u∥\lVert c\rVert=\lVert u\rVert. Indeed, on that circle q(c)−q(u)=(c2−1)2q(c)-q(u)=(c_2-1)^2.

Here r=(1,1)=1⋅Nur=(1,1)=1\cdot Nu and H−N=diag⁡(0,2)⪰0H-N=\operatorname{diag}(0,2)\succeq0, so Theorem A.19 applies. Theorems A.19 and A.20 together give the decision rule used in § 4: compute pp, and if it is non-zero, solve for the best same-norm step.

Certifying the best same-norm step and a finite-step gain.

Theorem A.22 (Certificate for the best same-norm step). Let HH and NN be symmetric. If cc satisfies cTNc=Rc^{\mathsf T}Nc=R, (H+λN)c=g(H+\lambda N)c=g and H+λN⪰0H+\lambda N\succeq0, then q(c)≤q(c′)q(c)\leq q(c^\prime) for every c′c^\prime with c′TNc′=Rc^{\prime\mathsf T}Nc^\prime=R.

The best same-norm steps reported in § 4 satisfy these conditions at all three checkpoints: λ>0\lambda>0, so H+λN≻0H+\lambda N\succ0, and the stationarity residual is below 10−1510^{-15}. They are therefore the global optima of the model on the ellipsoid.

Lemma A.23 (Model-error budget). For estimates (g^,H^)(\hat{g},\hat{H}) of (g,H)(g,H) and any cc, ∣q^(c)−q(c)∣≤∣(g^−g)Tc∣+12∣cT(H^−H)c∣|\hat{q}(c)-q(c)|\leq|(\hat{g}-g)^{\mathsf T}c|+\frac{1}{2}|c^{\mathsf T}(\hat{H}-H)c|.

With ∥g^−g∥≤δg\lVert\hat{g}-g\rVert\leq\delta_g and ∥H^−H∥op≤δH\lVert\hat{H}-H\rVert_{\mathrm{op}}\leq\delta_H this is at most δg∥c∥+12δH∥c∥2\delta_g\lVert c\rVert+\frac{1}{2}\delta_H\lVert c\rVert^2.

Theorem A.24 (A-posteriori finite-step certificate). Let F(x)F(x) be the true loss change of an implemented step xx and q^(x)\hat{q}(x) its predicted change, with ∣F(x)−q^(x)∣≤E(x)|F(x)-\hat{q}(x)|\leq E(x) for a candidate x1x_1 and every comparator xx in a set SS. If q^(x1)+E(x1)<q^(x)−E(x)\hat{q}(x_1)+E(x_1)<\hat{q}(x)-E(x) for every x∈Sx\in S, then F(x1)<F(x)F(x_1)<F(x) for every x∈Sx\in S.

The certificate needs no inverse of HH. Each part of the error EE is bounded by a proved result:

Theorem A.25 (The three error terms). (i) Taylor remainder. Let ϕ(t)\phi(t) be the loss after a fraction t∈[0,1]t \in[0,1] of a step, twice continuously differentiable with ∣ϕ′′′∣≤M|\phi'''| \le M on (0,1)(0,1). Then ∣ϕ(1)−ϕ(0)−ϕ′(0)−12ϕ′′(0)∣≤M/6|\phi(1)-\phi(0)-\phi'(0)-\frac{1}{2}\phi''(0)| \le M/6. (ii) Estimation. ∣xTy∣≤∥x∥∥y∥|x^{\mathsf{T}}y| \le\|x\|\|y\|. Hence, if ∥g^−g∥≤δg\|\hat{g}-g\| \le\delta_g and ∣xT(H^−H)x∣≤δH∥x∥2|x^{\mathsf{T}}(\hat{H}-H)x| \le\delta_H\|x\|^2 for all xx, then ∣q^(c)−q(c)∣≤δg∥c∥+12δH∥c∥2|\hat{q}(c)-q(c)| \le\delta_g\|c\|+\frac{1}{2}\delta_H\|c\|^2. (iii) Implementation. If ∥∇L∥≤G\|\nabla L\| \le G on a convex set containing the endpoints of the exact and the implemented step, and both steps are norm-matched to radius ρ\rho from raw outputs YY and Y^\hat{Y}, then their losses differ by at most 2Gρ∥Y^−Y∥/∥Y∥2G\rho\|\hat{Y}-Y\|/\|Y\|.

In (i), ϕ′(0)=−gTc\phi'(0)=-g^{\mathsf{T}}c and ϕ′′(0)=cTHc\phi''(0)=c^{\mathsf{T}}Hc are the first and second directional derivatives of the held-out loss along the step, which the diagnostic measures. Part (iii) combines the mean-value inequality with Theorem A.6.

Corollary A.26 (End-to-end certificate). For an implemented candidate step and a comparator, let II, TT and EE bound the implementation, Taylor and estimation errors of each. If q^1+(I1+T1+E1)<q^0−(I0+T0+E0)\hat{q}_1+(I_1+T_1+E_1)<\hat{q}_0-(I_0+T_0+E_0), then the implemented candidate truly lowers the loss more than the comparator.

Which inputs are established. The theorems are unconditional; their inputs differ in status. Established for our measurements: the trust-region conditions of Theorem A.22 (λ>0\lambda>0, residual below 10−1510^{-15}), so the reported best same-norm steps are certified optima of the model; the tangential gradient p≠0p\ne0 of Theorem A.20 (∥p∥/∥r∥=1.00,1.00,0.79\|p\|/\|r\|=1.00, 1.00, 0.79), so a same-norm improvement of the model exists at every checkpoint; and the hypotheses of Theorems A.5 and A.6, which hold by construction. Not established: the constants MM, δg\delta_g, δH\delta_H and GG of Theorem A.25, and sampling intervals for the measured one-step losses. Theorem A.26 is therefore not instantiated as a guarantee. The comparison in F2 with observed prediction errors is a consistency check only. Section E specifies the measurements that would instantiate it.

Limits of the diagnostic. Two further facts bound what the measurements can say.

Proposition A.27 (Per-band ratios are not the fixed-norm criterion). Let H=diag⁡(1,3)H=\operatorname{diag}(1,3), g=(12,2)g=(\frac{1}{2},2), u=(1,1)u=(1,1) and N=IN=I. The isolated ratio gb/Hbbg_b/H_{bb} is larger for band 2 (23\frac{2}{3} against 12\frac{1}{2}). Yet every same-norm step that gives band 2 more weight is strictly worse than uu: ∥c∥=∥u∥\|c\|=\|u\| and c2>1c_2>1 imply q(c)>q(u)q(c)>q(u). The same-norm step c=(3125,1725)c=(\frac{31}{25},\frac{17}{25}), which gives band 1 more weight, is strictly better.

On the circle, q(c)−q(u)=12(1−c1)+(c2−1)2q(c)-q(u)=\frac{1}{2}(1-c_1)+(c_2-1)^2. No cross-curvature is involved: at fixed norm what matters is the tangential part of Hu−gHu-g (Theorem A.20), not the ratio gb/Hbbg_b/H_{bb}.

Proposition A.28 (A one-step shape preference can reverse over two steps). Let F(x)=12x12+14x22F(x)=\frac{1}{2}x_1^2+\frac{1}{4}x_2^2, whose gradient step of size 11 maps xx to (0,12x2)(0,\frac{1}{2}x_2), and let V(x)=F(0,12x2)V(x)=F(0,\frac{1}{2}x_2) be the loss after one more such step. The points A=(14261,9861)A=(\frac{142}{61},\frac{98}{61}) and B=(9861,14261)B=(\frac{98}{61},\frac{142}{61}) have the same norm as x=(2,2)x=(2,2). Then F(B)<F(x)<F(A)F(B)<F(x)<F(A) but V(A)<V(x)<V(B)V(A)<V(x)<V(B).

Removing the step-size term therefore does not protect a one-step shape preference from horizon bias. This is why we treat the measured shape as a hypothesis for training experiments rather than as a prescription.

Input-whitened Muon

For a whitening factor P∈Rn×nP\in\mathbb{R}^{n\times n}, the whitened direction is W(G;P)=NS⁡(GP)PT\mathcal{W}(G;P)=\operatorname{NS}(GP)P^{\mathsf{T}}. With PPT=C−1PP^{\mathsf{T}}=C^{-1} it is the steepest direction for the data-dependent norm ∥DC1/2∥op\|DC^{1/2}\|_{\mathrm{op}}.

Theorem A.29 (Only the metric matters). If P1P1T=P2P2TP_1P_1^{\mathsf{T}}=P_2P_2^{\mathsf{T}} and P2P_2 is invertible, then W(G;P1)=W(G;P2)\mathcal{W}(G;P_1)=\mathcal{W}(G;P_2). The whitened step depends on CC only, not on the square root chosen.

Proof idea. P1=P2QP_1=P_2Q with QQ orthogonal, and Theorem A.3 moves QQ through NS⁡\operatorname{NS}.

Theorem A.30 (Exact GL⁡(n)\operatorname{GL}(n)-equivariance). Reparametrize the inputs by an invertible MM: x′=Mxx'=Mx, W′=WM−1W'=WM^{-1}, G′=GMTG'=GM^{\mathsf{T}}, C′=MCMTC'=MCM^{\mathsf{T}}. (i) If PPT=C−1PP^{\mathsf{T}}=C^{-1} and P′P' is invertible with P′P′T=C′−1P'P'^{\mathsf{T}}=C'^{-1}, then W(G′;P′)=W(G;P)M−1\mathcal{W}(G';P')=\mathcal{W}(G;P)M^{-1}. (ii) The full step with decoupled weight decay is equivariant too:

(1−ηλ)WM−1−ηW(GMT;(MT)−1P)=[(1−ηλ)W−ηW(G;P)]M−1.(1-\eta\lambda)WM^{-1}-\eta\mathcal{W}(GM^{\mathsf{T}};(M^{\mathsf{T}})^{-1}P)=[(1-\eta\lambda)W-\eta\mathcal{W}(G;P)]M^{-1}.

The trained function is therefore independent of the input coordinates.

Theorem A.31 (Steepest directions transfer through a change of metric). Let PP be invertible and N\mathcal{N} any norm-like function. If DD is a steepest direction for N\mathcal{N} at GPGP, then DPTDP^{\mathsf{T}} is a steepest direction at GG for the pulled-back function Δ↦N(Δ(PT)−1)\Delta\mapsto\mathcal{N}\left(\Delta(P^{\mathsf{T}})^{-1}\right).

Theorem A.32 (Whitened Muon is conditioning-free on least squares). Let L(W)=12tr⁡((W−W⋆)C(W−W⋆)T)L(W) = \frac{1}{2}\operatorname{tr}\left((W-W_{\star})C(W-W_{\star})^{\mathsf{T}}\right) with C=RRTC=RR^{\mathsf{T}} and RR invertible. Run whitened Muon with P=(RT)−1P=(R^{\mathsf{T}})^{-1} and any step sizes η0,η1,…\eta_0,\eta_1,\ldots. Then the whitened error Et=(Wt−W⋆)RE_t=(W_t-W_{\star})R satisfies

Et+1=Et−ηtNS⁡(Et),L(Wt)=12∥Et∥F2,E_{t+1}=E_t-\eta_t\operatorname{NS}(E_t), \qquad L(W_t)=\frac{1}{2}\lVert E_t\rVert_{\mathrm{F}}^2,

which is plain Muon on the isotropic loss 12∥E∥F2\frac{1}{2}\lVert E\rVert_{\mathrm{F}}^2, for every CC, however ill-conditioned.

Proposition A.33 (Plain Muon is not GL⁡(n)\operatorname{GL}(n)-equivariant). For G=(1,1)G=(1,1) and M=diag⁡(2,1)M=\operatorname{diag}(2,1): if plain Muon’s step were the same in both coordinate systems, NS⁡(GMT)M=NS⁡(G)\operatorname{NS}(GM^{\mathsf{T}})M=\operatorname{NS}(G), then NS⁡(G)=0\operatorname{NS}(G)=0.

Energy-matched decay

Ordinary decoupled decay W←W−ηλWW\leftarrow W-\eta\lambda W shrinks every singular value by the same factor. Energy-matched Schatten-1 decay replaces λW\lambda W by λ(∥W∥F2/⟨W,P⟩)P\lambda\left(\lVert W\rVert_{\mathrm{F}}^2/\langle W,P\rangle\right)P with PP the polar factor of WW. Since ⟨W,P⟩=∥W∥∗=∑iσi\langle W,P\rangle=\lVert W\rVert_{\ast}=\sum_i\sigma_i, every singular value then loses the same absolute amount c1=λ∑jσj2/∑jσjc_1=\lambda\sum_j\sigma_j^2/\sum_j\sigma_j.

Theorem A.34 (Energy match). (i) If ∑jσj≠0\sum_j\sigma_j\ne0, then ∑iσic1=∑iσi(λσi)\sum_i\sigma_i c_1=\sum_i\sigma_i(\lambda\sigma_i): subtracting c1c_1 from every singular value removes the same first-order energy as ordinary decay. (ii) More generally, for matrices WW and DD with ⟨W,D⟩≠0\langle W,D\rangle\ne0, the rescaled direction D~=(∥W∥F2/⟨W,D⟩)D\widetilde{D}=(\lVert W\rVert_{\mathrm{F}}^2/\langle W,D\rangle)D satisfies ⟨W,D~⟩=⟨W,W⟩\langle W,\widetilde{D}\rangle=\langle W,W\rangle and ⟨W,D~−W⟩=0\langle W,\widetilde{D}-W\rangle=0. The energy-matched step is ordinary decay plus a step orthogonal to WW.

Proposition A.35 (Which singular values shrink more). Let λ>0\lambda>0 and σi>0\sigma_i>0. Under energy-matched Schatten-1 decay, σi\sigma_i shrinks by a larger fraction than under ordinary decay (c1/σi>λc_1/\sigma_i>\lambda) if and only if σi<∑jσj2/∑jσj\sigma_i<\sum_j\sigma_j^2/\sum_j\sigma_j. If c1>0c_1>0 and 0<σi<σj0<\sigma_i<\sigma_j, then c1/σj<c1/σic_1/\sigma_j<c_1/\sigma_i: smaller singular values always lose a larger fraction.

The decay therefore prunes the bulk of the spectrum and spares its top. That is the opposite of Schatten-4 decay, which hurt in our experiments while Schatten-1 helped.

Results from earlier formal work cited in Section 7

These results come from the earlier, temporal-filter part of the project. Most are explicit counterexamples or elementary identities, and we state them for what they rule out.

Proposition A.36 (Moments do not determine the momentum preference under NS5). (i) Any risk that is affine in the first two moments of a finite noise law takes the same value on two laws with equal mass, mean and second moment. (ii) There are two explicit laws for the amplitude of rank-two gradient noise with the following properties: equal means, equal second moments, identical temporal covariance of the noise streams, and identical linearized risk curves β↦Rlin(β)\beta\mapsto R_{\mathrm{lin}}(\beta) for every momentum β\beta. Yet under the actual five-step NS5 map with stabilized normalization, the first law strictly prefers β=0.95\beta=0.95 to β=0.85\beta=0.85 and the second strictly prefers 0.850.85 to $0.95. (iii) Hence no rule that observes only the linearized risk curve can choose the better momentum for both laws.

Proposition A.37 (Spatial orientation decides the momentum preference). In an explicit full-rank construction with two consecutive updates, the signal path, drift, noise energy and time dependence are all matched. The Euclidean block risk is identical for every β\beta whether the noise is tangential or radial. After Newton–Schulz with any finite depth (including five), tangential noise makes β=0.95\beta=0.95 strictly better than β=0.85\beta=0.85, and radial noise makes β=0.85\beta=0.85 strictly better than β=0.95\beta=0.95.

Proposition A.38 (First-order alignment can mis-rank finite steps). Let L(x,y)=x+2y+12(x2+10y2)L(x,y)=x+2y+\frac{1}{2}(x^2+10y^2). The unit steps A=(−1,0)A=(-1,0) and B=(0,−1)B=(0,-1) have equal length. At the origin, first-order analysis prefers BB (directional decrease 22 against 11), yet L(A)−L(B)=−72L(A)-L(B)=-\frac{7}{2}, so AA is the better finite step.

Proposition A.39 (Curvature decides between the polar and gradient directions). Let a step dd change a diagonal quadratic by Δ(d)=−⟨g,d⟩+12∑ihidi2\Delta(d)=-\langle g,d\rangle+\frac{1}{2}\sum_i h_i d_i^2, with gradient g=(7,1)g=(7,1). The polar direction dP=(1,1)d_P=(1,1) and the norm-matched gradient direction dG=(7/5,1/5)d_G=(7/5,1/5) both have ∥d∥2=2\lVert d\rVert^2=2. (i) With curvature h=(1,1)h=(1,1), dGd_G is better. (ii) With h=(6,1)h=(6,1), dPd_P is better. (iii) With h=(4,1)h=(4,1), the norm-matched intermediate direction (17/13,7/13)(17/13,7/13) is strictly better than both.

This is the two-direction version of Figure 1a: which spectral allocation is best depends on how curvature compares with gradient signal across directions.

Proposition A.40 (Greedy step-size facts). For a quadratic loss with gradient W↦HW−bW \mapsto HW-b and a step W−ηOW-\eta O: (i) ⟨∇L(W−ηO),O⟩=⟨∇L(W),O⟩−η⟨HO,O⟩\langle\nabla L(W-\eta O),O\rangle=\langle\nabla L(W),O\rangle-\eta\langle HO,O\rangle, which is affine in η\eta. (ii) With a=⟨∇L(W),O⟩a=\langle\nabla L(W),O\rangle and q=⟨HO,O⟩>0q=\langle HO,O\rangle>0, the one-step loss L(W−ηO)L(W-\eta O) decreases with η\eta exactly when η<a/q\eta<a/q. (iii) With no aligned signal (a=0a=0), every positive step looks too large to a greedy criterion. (iv) A bias added equally to the marginal value of every group does not change their comparison.

Diagnostic details and additional results

Scope. The diagnostic perturbs only the 48 hidden matrices, along exact-polar band directions or the actual NS5_5 neuron groups. The AdamW update of embeddings and norms and the weight decay are left out. gg and HH are estimated on separate held-out folds, so they are estimates of a common local model with separate errors, not derivatives of one empirical loss. The realized-loss checks use a third fold. The inverse-based quantities (c⋆c^\star and the shape shares) are sensitive because HH is ill-conditioned. The fixed-norm band-filter predictions evaluate qq at a given vector and need no inverse.

Procedure. For a checkpoint at step tt (Muon, seed 43, screen schedule), we recompute the next training minibatch’s gradient with the same clipping, form N=G+β(βB+G)N=G+\beta(\beta B+G) exactly as the optimizer would, and take the full SVD of NN for each of the 48 matrices. Band directions use the exact polar factor UIbVIbTU_{I_b}V_{I_b}^{\mathsf{T}}, scaled by the learning rate at step tt and Muon’s shape factor. Neuron groups use Muon’s actual NS5_5 step, split into six equal-count sextiles by row norm (tall or square matrices) or column norm (wide matrices). The held-out gradient uses 768 validation sequences (393k tokens). Each Hessian–vector product uses 192 further sequences (98k tokens), computed by double backward with the math attention kernel. Realized loss changes use 768 further sequences. The three sets are disjoint. For B=6B=6 groups and five group sets (all matrices, and each layer type), the diagnostic needs 30 Hessian–vector products per checkpoint.

Table 2: Band diagnostic over all 48 matrices. Top: per-band multiplier cb=gb/Hbbc_b=g_b/H_{bb}, best uniform rescaling γ⋆\gamma^\star, second-order gap q(1)−q(c⋆)q(1)-q(c^\star) (nats) and its shape share (Theorem 3). Bottom: the joint optimum c⋆=H−1gc^\star=H^{-1}g and the condition number of HH. c⋆c^\star is irregular because HH is ill-conditioned, but it also puts nearly all its weight on the lower half of the spectrum. All multipliers scale as 1/η1/\eta; η\eta at step 2,048 is 10×10\times smaller.

Step1–89–3233–9697–256257–512513–768γ⋆\gamma^\stargapshape share
1,0240.591.962.464.2712.028.30.520.0067417%
1,5360.551.902.394.219.5221.00.490.0053212%
2,0480.987.6716.431.675.8186.31.520.0013797%
1,024 (c⋆c^\star)0.530.18−0.10-0.100.315.9216.4cond⁡(H)≈1×104\operatorname{cond}(H)\approx1\times10^4
1,536 (c⋆c^\star)0.480.34−0.15-0.150.884.0111.5cond⁡(H)≈8×103\operatorname{cond}(H)\approx8\times10^3
2,048 (c⋆c^\star)0.400.123.168.0534.8115.9cond⁡(H)≈4×103\operatorname{cond}(H)\approx4\times10^3

Table 2.

Table 3: Quadratic-model calibration (bands, all matrices): predicted versus realized change in held-out loss (393k disjoint tokens) for Muon’s step (c=1c=1) and the model-optimal band shape (c=c⋆c=c^\star).

CheckpointMuon, predictedMuon, realizedc⋆c^\star, predictedc⋆c^\star, realized
1,024−0.00079-0.00079−0.00054-0.00054−0.00753-0.00753−0.00730-0.00730
1,536+0.00026+0.00026+0.00073+0.00073−0.00506-0.00506−0.00471-0.00471
2,048−0.00028-0.00028−0.00046-0.00046−0.00165-0.00165−0.00183-0.00183

Table 3.

Measured one-step losses at matched norm

To test the fixed-norm predictions with the actual loss rather than the quadratic model, we start from the same checkpoints and form each matrix’s next Nesterov input. We take the exact tilted direction ϕ5(σ/σrms)p\phi_5(\sigma/\sigma_{\mathrm{rms}})^p, rescaled per matrix to the Frobenius norm of Muon’s NS5_5 step (p=0p=0 is Muon), or a norm-matched band filter. We then measure the held-out loss after one step on a fixed 393k-token set (Table 4). The slope dL/dpdL/dp at p=0p=0 is a central finite difference. The best same-norm filter is the step-1,536 model optimum. One checkpoint seed and one validation subset are used, without intervals.

Fixed-norm analysis. Band directions Dℓ,b=ηsℓUIbVIbTD_{\ell,b}=\eta s_{\ell}U_{I_b}V_{I_b}^{\mathsf{T}} are mutually Frobenius-orthogonal, with ∥Db∥F2∝∣Ib∣\lVert D_b\rVert_{\mathrm{F}}^2\propto\lvert I_b\rvert (every hidden matrix has rank 768). Every matrix’s step therefore has the same norm as Muon’s exactly when cTNc=1TN1c^{\mathsf{T}}Nc=\mathbf{1}^{\mathsf{T}}N\mathbf{1} with N=diag⁡(8,24,64,160,256,256)N=\operatorname{diag}(8,24,64,160,256,256). The best same-norm step minimizes qq on this ellipsoid. With H~=N−1/2HN−1/2\widetilde{H}=N^{-1/2}HN^{-1/2} and g~=N−1/2g\widetilde{g}=N^{-1/2}g, it is c=N−1/2(H~+λI)−1g~c=N^{-1/2}(\widetilde{H}+\lambda I)^{-1}\widetilde{g}, with λ>−λmin⁡(H~)\lambda>-\lambda_{\min}(\widetilde{H}) chosen by a one-dimensional root search so that the constraint holds. Norm-matched filters are rescaled onto the ellipsoid before evaluation. Table 5 gives the predictions.

CheckpointMuon’s q(1)q(1)best same-norm ccband filterranks 1–8 halved∥p∥/∥r∥\lVert p\rVert/\lVert r\rVert
1,024−0.00079−0.00589−0.00570−0.005271.00
1,536+0.00026−0.00488−0.00478−0.004431.00
2,048−0.00028−0.00009+0.00005+0.000010.79
Best same-norm cc at step 1,024(0.54, 0.22, 0.21, 1.26, 1.26, 0.62)
Best same-norm cc at step 1,536(0.49, 0.35, 0.17, 1.38, 1.17, 0.63)
Best same-norm cc at step 2,048(0.64, 2.92, 2.30, 1.05, 0.39, 0.17)

Table 5. Second-order predictions at Muon’s own update norm (bands, all 48 matrices). Column 2 is Muon’s predicted one-step change q(1)q(1). Columns 3–5 are q(c)−q(1)q(c)-q(1) for the best same-norm step, the norm-matched band filter used in training (mb∝cb1/4m_b\propto c_b^{1/4} at step 1,024), and halving ranks 1–8 with norm matching. The last column is the relative size of the tangential gradient pp (Theorem A.20). Bottom: the best same-norm step’s band multipliers.

Greedy multiplier per neuron sextile

Figure 4. Greedy multiplier per neuron sextile, ranked by Muon’s per-neuron update norm. The most-moved neurons sit in higher-curvature directions.

Implied spectral exponent. To relate the band multipliers to a power law f(σ)∝σpf(\sigma)\propto\sigma^p, we take each matrix’s singular values at ranks 1, 8, 32, 96 and 256, normalize by the value at rank 96, and average them in log space over the 48 matrices. Each of the top four bands is represented by the geometric mean of its two edges, and we regress log⁡cb\log c_b on log⁡σb\log\sigma_b. The slopes are −0.8, −0.8 and −1.4 at the three checkpoints. The estimate is crude: the band representatives are coarse, and the tail bands, whose lower edges we did not log, are excluded. It is also a greedy quantity, so it overstates how strong a tilt to use. We report it only to show that the prescribed sign is negative and of order one in magnitude. The deployed power, −14-\frac{1}{4}, is deliberately conservative.

Figure 5. Band multipliers computed separately for each layer type. The top band has the smallest multiplier in every case.

Per-neuron spread. Table 6 reports the coefficient of variation (CV) of per-neuron update norms at the three checkpoints. Each entry is the mean over the 12 matrices of a type, along the axis that indexes neurons: rows for qkv, expand and attention-out, columns for contract. For comparison, the raw momentum’s QKV row-norm CV is 0.93. Muon reduces the spread but does not remove it. Block-wise NS5_5 steps leave a residual QKV spread (CV 0.22) although exact orthogonal blocks would have none (Theorem A.10). A plausible cause is that NS5_5 maps the near-null singular directions of low-rank blocks to values well below 1; we have not verified this.

Layer (neuron axis)Muon, step 1,024Muon, step 2,048Block-wise, step 1,024Muon max/median, step 1,024
qkv (rows)0.430.470.221.84
expand (rows)0.180.180.031.72
contract (columns)0.180.170.021.67
attention-out (rows)0.020.020.021.05

Table 6. Coefficient of variation of per-neuron update norms of Muon’s step and of the block-wise polar step.

All screen runs

Table 7 and Figure 6 list every completed screen arm (2,048 steps, 50.3M tokens). Differences are paired against fused-QKV Muon at LR 0.005: the seed-42 difference for every arm, and the mean over seeds 42–44 with a 95% tt-interval where three seeds exist. The arms were selected adaptively during the study; no multiplicity correction is applied.

Table 7: Screen results. Final held-out loss (4.18M tokens) at seed 42 and paired differences from fused-QKV Muon.

FamilyArmFinal (s42)Δ\Delta (s42)Δ\Delta, 3 seeds [95% CI]
Muon, LR 0.005 (seeds 42/43/44: 4.2043, 4.2066, 4.2035)4.2043——
Learning rateMuon, LR 0.00354.2179+0.0136
Learning rateMuon, LR 0.0074.2077+0.0034
Learning rateMuon, LR 0.014.2376+0.0332
Learning rateSplit QKV, LR 0.00354.1931−0.0112
Learning rateSplit QKV, LR 0.0074.1847−0.0196
Learning rateSquare blocks, LR 0.0074.1840−0.0203
Learning rateSquare blocks, LR 0.014.2045+0.0001
Neuron balanceSplit QKV (Q, K, V orthogonalized separately)4.1788−0.0255−0.0267 [−0.0304, −0.0230]
Neuron balancePer-head Q/K/V blocks (36 blocks of 64×76864 \times768)4.1871−0.0172
Neuron balanceSquare blocks (QKV + MLP)4.1824−0.0219−0.0238 [−0.0282, −0.0194]
Neuron balanceSquare blocks, not norm-matched (deployable)4.1843−0.0200
Neuron balanceSquare blocks in the MLP only (QKV fused)4.2082+0.0039
Neuron balanceNorMuon row normalization4.1854−0.0189−0.0188 [−0.0193, −0.0183]
Neuron balanceRow normalization without EMA4.1852−0.0192
Spectral shapeCooldown tilt σ−1/4\sigma^{-1/4} (exact)4.1910−0.0133
Spectral shapeCooldown tilt σ−1/2\sigma^{-1/2} (exact)4.1888−0.0155
Spectral shapeTilt σ−1/4\sigma^{-1/4} from step 1,0244.1896−0.0147
Spectral shapeCooldown band filter, mb∝cb1/4m_b \propto c_b^{1/4} from step 1,0244.1927−0.0116
Spectral shapeAccurate polar factor (NS5 + 3 cubic steps)4.2023−0.0020
Spectral shapeSquare blocks + cooldown tilt (Chebyshev, monomial/TF32; faulty)4.1804−0.0240
Spectral shapeSquare blocks + cooldown tilt (Chebyshev, Clenshaw/FP32)4.1721−0.0322
Spectral shapeSquare blocks + cooldown tilt (exact)4.1711−0.0332
Decay geometrySchatten-1 decay (bulk shrinkage)4.1991−0.0052−0.0057 [−0.0100, −0.0014]
Decay geometrySchatten-4 decay (top-heavy)4.2220+0.0177
Data metric (whitening)Input whitening (14,14)\left(\frac{1}{4},\frac{1}{4}\right), norm-matched4.2254+0.0211
Data metric (whitening)Input whitening (12,12)\left(\frac{1}{2},\frac{1}{2}\right), norm-matched4.2642+0.0599
Data metric (whitening)Input whitening (12,12)\left(\frac{1}{2},\frac{1}{2}\right), δ=0.3\delta=0.3, norm-matched4.2474+0.0431
Data metric (whitening)Input whitening (12,12)\left(\frac{1}{2},\frac{1}{2}\right), data-metric magnitude4.2567+0.0524
Data metric (whitening)Input whitening (12,12)\left(\frac{1}{2},\frac{1}{2}\right) + shaped decay4.2638+0.0595
Data metric (whitening)Toner recipe (12,18)\left(\frac{1}{2},\frac{1}{8}\right) + shaped decay4.1975−0.0068
CombinationsTilt σ−1/4\sigma^{-1/4} + row normalization (no EMA)4.1737−0.0306
CombinationsBand filter (step-1,024 multipliers) + row normalization (no EMA)4.1742−0.0302
CombinationsSquare blocks + NorMuon4.1780−0.0263
CombinationsSquare blocks + NorMuon + cooldown tilt (exact)4.1661−0.0382
CombinationsSquare blocks + NorMuon + cooldown tilt4.1665−0.0378
Combinations... + Schatten-1 decay4.1648−0.0396
CombinationsSplit QKV + cooldown tilt4.1667−0.0376
CombinationsSplit QKV + NorMuon4.1722−0.0321
CombinationsSplit QKV + NorMuon + cooldown tilt4.1632−0.0411−0.0436 [−0.0511, −0.0362]
CombinationsSplit QKV + NorMuon + tilt from step 1,0244.1625−0.0418
CombinationsSplit QKV + NorMuon + cooldown tilt σ−1/2\sigma^{-1/2}4.1641−0.0402
CombinationsSplit QKV + NorMuon + cooldown tilt + Schatten-1 decay4.1581−0.0462−0.0487 [−0.0555, −0.0420]

Table 7. Screen results. Final held-out loss (4.18M tokens) at seed 42 and paired differences from fused-QKV Muon.

Figure 6. Screen arms sorted by their mean paired difference from fused-QKV Muon (seed 42, or the three-seed mean with 95% tt-interval and open circles for seeds where three seeds exist).

Learning-rate robustness of block-wise steps. With square blocks, raising the learning rate from 0.005 to 0.007 and 0.01 gives final losses of 4.1840 and 4.2045. Muon at the same rates gives 4.2077 and 4.2376 (+0.0034 and +0.0332 relative to its optimum). Doubling the learning rate leaves square-block Muon level with Muon’s tuned optimum. MuonBP reported that block-only training needed smaller learning rates at scale ([22]); in our setting block-wise steps are, if anything, more tolerant of large rates.

Implementation details

Norm matching. For every matrix and step, the candidate direction DD is rescaled to ∥NS5(N)∥F\lVert\mathrm{NS}_5(N)\rVert_F, with NN the same momentum Muon would use, and then enters (1) with Muon’s shape factor and learning rate. Norm-matched whitening arms are matched to ∥NS5(NP)∥F\lVert\mathrm{NS}_5(NP)\rVert_F, the norm of their own whitened polar factor. The trainer enables TF32 matrix products globally, so NS5_5 runs in FP32 storage with TF32 products; only the Chebyshev filter disables TF32. Matching costs one extra joint NS5_5 per matrix for block-wise arms, used only to read its norm. The deployable square-block step omits it: it scales wide-matrix column blocks by 1/k1/\sqrt{k} and runs all 144 square blocks in one batched NS5_5 call. It gives −0.0200 on the screen, against −0.0219 norm-matched.

NorMuon. Following Li et al. [27], the orthogonalized update is divided row-wise by v+10−8\sqrt{v}+10^{-8}, where vv is an EMA (β2=0.95\beta_2 = 0.95) of the row mean of D⊙DD \odot D. It is then norm-matched. A stateless variant with β2=0\beta_2 = 0 gives −0.0192, against −0.0189 for the EMA version (screen), so the EMA is not what helps.

Spectral tilt. The exact tilt maps singular value σi\sigma_i of NN to ϕ5(σi/∥N∥F)⋅(σi/σrms)p\phi_5(\sigma_i/\lVert N\rVert_{\mathrm{F}})\cdot(\sigma_i/\sigma_{\mathrm{rms}})^p via an eigendecomposition of NNTNN^{\mathsf{T}}. With p=0p=0 it reproduces NS5_5 to 2×10−42\times10^{-4} relative error. The matmul-only tilt (Algorithm 1) uses a degree-12 Chebyshev least-squares fit of xpx^p on [10−3,1][10^{-3},1] (4,000 Chebyshev nodes) in the variable t=2(x−xl0)/(1−xl0)−1t=2(x-x_{l0})/(1-x_{l0})-1. It is evaluated by Clenshaw’s recurrence with TF32 disabled. S=ONTS=ON^{\mathsf{T}} is symmetrized and scaled by Λ=1.01\Lambda=1.01 times a 20-step power-iteration estimate of λmax⁡(S)\lambda_{\max}(S). That is not a proved upper bound: eigenvalues above Λ\Lambda, or below 10−3Λ10^{-3}\Lambda, fall outside the fitted interval, where the Chebyshev polynomial is not controlled. By Theorem A.5, the implemented filter is fi1+pσipf_i^{1+p}\sigma_i^p rather than the exact fiσipf_i\sigma_i^p; on the Newton–Schulz plateau they differ by a factor in [0.96,1.10][0.96,1.10]. On real momenta the matmul tilt matches the exact tilt to direction cosine 0.9980.998–0.9990.999; enabling TF32 lowers the dot product to 0.9920.992. By Theorem A.6(ii) the normalized steps differ by at most 6.3% in Frobenius norm. Turning that into a loss bound would need a curvature bound along the difference (Theorem A.24), which we do not have. When Q, K and V are split, the three 768×768768\times768 blocks of all 12 layers and the square attention-output matrices are filtered in one batched chain. The MLP matrices are filtered whole. With TF32 off the tilt costs about 0.17 s per step on an A6000, during the cooldown only.

Input whitening. CC is an EMA of the uncentered input second moment, trace-normalized with damping, C^=C/λ(C)+δI\widehat{C}=C/\lambda(C)+\delta I (δ=0.1\delta=0.1 unless stated). It is refreshed from 4,096-token activation samples. The step is polar (NC^−qin)C^−qout(N\widehat{C}^{-q_{\mathrm{in}}})\widehat{C}^{-q_{\mathrm{out}}}: (14,14)\left(\frac{1}{4},\frac{1}{4}\right) and (12,12)\left(\frac{1}{2},\frac{1}{2}\right) norm-matched, (12,12)\left(\frac{1}{2},\frac{1}{2}\right) at its own magnitude, and the Toner recipe (12,18)\left(\frac{1}{2},\frac{1}{8}\right) with shaped decay λWC^−1/4\lambda W\widehat{C}^{-1/4}.

Schatten-pp decay. We replace λW\lambda W by λ∥W∥F2⟨W,W(WTW)(p−2)/2⟩W(WTW)(p−2)/2\lambda\frac{\lVert W\rVert_{\mathrm{F}}^{2}}{\langle W, W(W^{\mathsf{T}}W)^{(p-2)/2}\rangle}W(W^{\mathsf{T}}W)^{(p-2)/2}. This removes the same Frobenius energy per step as uniform decay, with relative rate λσip−2∑σ2/∑σp\lambda\sigma_i^{p-2}\sum\sigma^2/\sum\sigma^p on singular value σi\sigma_i. For p=1p=1 we use NS5⁡(W)\operatorname{NS5}(W) for the polar factor.

Cost. Measured seconds per step on one RTX A6000 (100.7M-token runs): Muon 0.343; NorMuon 0.329; norm-matched square blocks 0.415; square blocks + NorMuon + tilt 0.418 before and 0.588 during the cooldown. Mean wall time per 100.7M-token run, including evaluations: Muon 23.3 min, split-QKV 24.9, split-QKV + tilt 26.4, split-QKV + NorMuon 25.1, split-QKV + NorMuon + tilt 27.8. Three runs shared a host, so these timings are indicative.

Reproducibility and provenance

Logging. Each run writes an event log with its recipe, optimizer variant, seed, software versions and a hash of the optimizer source. The results tables, figures and run-derived numbers in the results text are generated automatically from these logs. A few descriptive quantities (per-step timings, curvature ratios, some leverage-table entries and filter cosines) were transcribed and checked in an independent pass against the logs. Training uses torch.compile and BF16 autocast for the forward pass; the optimizer state is FP32, and Newton–Schulz runs in FP32 with TF32 matrix products. The trainer reproduces an earlier, slower harness’s Muon loss (3.9445 against 3.9463 at seed 42).

Order of experiments. The screen arms, the diagnostic and the first confirmations ran in parallel on three GPUs over one day. The anatomy was not pre-registered: the square-block, whitening and accurate-polar screens started before the curvature diagnostics, and the neuron diagnostic postdated the first block arms. The square-block arm was motivated by the leverage analysis; the attribution runs later showed that the QKV split carries its gain. Two confirmations, square blocks + NorMuon + tilt and split QKV + NorMuon + tilt, were launched before their own screens had finished. The 355M seed-42 pair was launched after the positive 124M confirmation. The gating rule of Section 6 was adopted after that; it released the full-length factorial ablations (split QKV + tilt; split QKV + NorMuon), whose screens beat the baseline by 0.0121 and

Algorithm 1 One step for a hidden matrix: split-QKV Muon with NorMuon row normalization and the cooldown tilt (cooldown is true in the last 25% of steps).

  1. B←βB+GB \leftarrow\beta B + G; N←G+βBN \leftarrow G + \beta B

  2. Split NN into blocks NjN_j (Q, K, V for the attention input matrix; one block otherwise)

  3. for each block NjN_j do

  4. Oj←NS5(Nj)O_j \leftarrow\mathrm{NS}_5(N_j)

  5. if cooldown then

  6. S←sym⁡(OjNjT)S \leftarrow\operatorname{sym}(O_jN_j^{\mathsf{T}}); λ←1.01λmax⁡(S)\lambda\leftarrow1.01\lambda_{\max}(S) (20 power iters)

  7. Dj←ClenshawCheb⁡x−1/4(S/λ)OjD_j \leftarrow\operatorname{ClenshawCheb}_{x^{-1/4}}(S/\lambda)O_j (FP32)

  8. else

  9. Dj←OjD_j \leftarrow O_j

  10. end if

  11. end for

  12. D←[D1;…]D \leftarrow[D_1;\ldots]; v←β2v+(1−β2)rowmean⁡(D⊙D)v \leftarrow\beta_2v + (1-\beta_2)\operatorname{rowmean}(D \odot D); D←D/(v+ϵ)D \leftarrow D/(\sqrt{v}+\epsilon)

  13. D←D⋅∥NS5(N)∥F/∥D∥FD \leftarrow D \cdot\lVert\mathrm{NS}_5(N)\rVert_F/\lVert D\rVert_F (norm match)

  14. W←(1−ηλ)W−ηsDW \leftarrow(1-\eta\lambda)W-\eta sD

0.0066, and the 355M seeds 43–44.

Source versions. The optimizer file changed during the study. The original fused-QKV Muon, NorMuon and square-block confirmations span four source hashes, and a read-only audit of the raw logs could not certify, from the retained files, that the changes touched only inactive branches. All split-QKV arms (split-QKV Muon, the factorial and the 355M runs) ran on a single source version, so the factorial comparisons are unaffected. Every comparison involving fused-QKV Muon, NorMuon on fused QKV, square blocks or square blocks + NorMuon + tilt crosses source versions; this includes the attribution results “square blocks vs. split QKV” and “NorMuon vs. split QKV”. The A/A bound and the reproduction of the earlier harness’s loss suggest the effect is small, but we cannot certify it.

Measurements needed to instantiate the certificate. Experiments were paused before these could run. Each uses saved checkpoints and no training: (1) Sampling intervals: record per-sequence held-out losses for every step in Table 4 and bootstrap paired differences over sequences, across several checkpoint seeds. (2) Estimation errors δg,δH\delta_g,\delta_H: recompute gg and HH on disjoint folds and take the spread of g^\hat{g} and of H^\hat{H} over folds. (3) Third-derivative bound MM: evaluate the loss at t∈{0,±12,±1}t \in\{0,\pm\frac{1}{2},\pm1\} along each step and take third finite differences. (4) Gradient bound GG: the held-out gradient norm at both endpoints and the midpoint of each step pair.

Statistics. Intervals are two-sided 95% Student-tt intervals over three paired seeds (2 degrees of freedom). They are nominal: the same seeds were used in the screen, and no correction is made for the number of arms examined. The A/A bound (0.009 nats) uses 98k-token intermediate evaluations; final evaluations use 4.18M tokens and are less noisy.

References

  1. [1]Zhehang Du and Weijie Su. The Newton-Muon optimizer. arXiv preprint arXiv:2604.01472, 2026.arxiv.org/abs/2604.01472
  2. [2]Noah Amsel, David Persson, Christopher Musco, and Robert M. Gower. The polar express: Optimal matrix sign methods and their application to the Muon algorithm. In International Conference on Learning Representations (ICLR), 2026. arXiv:2505.16932.arxiv.org/abs/2505.16932
  3. [3]Dmitrii Andrianov, Andrey Veprekov, and Aleksandr Beznosikov. Spectral weight decay: Inducing low-rank structure in neural network weights. arXiv preprint arXiv:2610.11730, 2026.
  4. [4]Atilim Gunes Baydin, Robert Cornish, David Martinez Rubio, Mark Schmidt, and Frank Wood. Online learning rate adaptation with hypergradient descent. In International Conference on Learning Representations (ICLR), 2018.arxiv.org/abs/1703.04782
  5. [5]Jeremy Bernstein and Laker Newhouse. Old optimizer, new norm: An anthology. arXiv preprint arXiv:2409.20325, 2024.arxiv.org/abs/2409.20325
  6. [6]Meher Chaitanya, Tianyi Zhou, and Aristides Gionis. Does Muon need fine-grained spectral shaping? arXiv preprint arXiv:2610.07497, 2026.arxiv.org/abs/2610.07497
  7. [7]Tong Che. Second-order Muon done right: A principled marriage of spectral geometry and curvature. arXiv preprint arXiv:2608.09763, 2026.arxiv.org/abs/2608.09763
  8. [8]Lizhang Chen, Jonathan Li, and Qiang Liu. Muon optimizes under spectral norm constraints. Transactions on Machine Learning Research, 2026. ISSN 2835-8856. URL https://openreview.net/forum?id=BLz4hjxIWu.arxiv.org/abs/2506.15054
  9. [9]C. W. Clenshaw. A note on the summation of Chebyshev series. Mathematical Tables and Other Aids to Computation, 9(51):118–120, 1955. doi: 10.1090/S0025-5718-1955-0071856-0.DOI
  10. [10]Jeremy M. Cohen, Simran Kaur, Yuanzhi Li, J. Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. In International Conference on Learning Representations (ICLR), 2021.
  11. [11]Alec Dewulf, Dhruv Pai, Li Yang, Ashley Zhang, and Ben Keigwin. Aurora: A leverage-aware spectral optimizer. arXiv preprint arXiv:2606.27715, 2026.arxiv.org/abs/2606.27715
  12. [12]Petros Drineas, Malik Magdon-Ismail, Michael W. Mahoney, and David P. Woodruff. Fast approximation of matrix coherence and statistical leverage. Journal of Machine Learning Research, 13(111):3475–3506, 2012.arxiv.org/abs/1109.3843
  13. [13]Essential AI, Ishaan Shah, et al. Practical efficiency of Muon for pretraining. arXiv preprint arXiv:2505.02222, 2025.
  14. [14]Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via Hessian eigenvalue density. In International Conference on Machine Learning (ICML), volume 97 of Proceedings of Machine Learning Research, pages 2232–2241, 2019.arxiv.org/abs/1901.10159
  15. [15]Vishrut Goyal and Rohan Ramkumar. Spectrally targeted Muon. arXiv preprint arXiv:2610.10965, 2026.arxiv.org/abs/2610.10965
  16. [16]Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization. In International Conference on Machine Learning (ICML), volume 80 of Proceedings of Machine Learning Research, pages 1842–1850, 2018.arxiv.org/abs/1802.09568
  17. [17]Guy Gur-Ari, Daniel A. Roberts, and Ethan Dyer. Gradient descent happens in a tiny subspace. arXiv preprint arXiv:1812.04754, 2018.
  18. [18]Alexander Hägele, Elie Bakouch, Atli Kosson, Loubna Ben Allal, Leandro Von Werra, and Martin Jaggi. Scaling laws and compute-optimal training beyond fixed training durations. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, 2024.arxiv.org/abs/2405.18392
  19. [19]Kai Hu, Li Zhang, Weichen Yu, Matt Fredrikson, and Haoqi Hu. Do orthogonality constraints help language-model training beyond sphere optimization? arXiv preprint arXiv:2611.00593, 2026.arxiv.org/abs/2611.00593
  20. [20]Keller Jordan et al. modded-nanogpt: Speedrunning the NanoGPT baseline. GitHub repository, https://github.com/KellerJordan/modded-nanogpt, 2024a.
  21. [21]Keller Jordan et al. Muon: An optimizer for hidden layers in neural networks. Blog post, https://kellerjordan.github.io/posts/muon/, December 2024b.
  22. [22]Ahmed Khaled, Kaan Ozkara, Tao Yu, Mingyi Hong, and Youngsuk Park. MuonBP: Faster Muon via block-periodic orthogonalization. In International Conference on Learning Representations (ICLR), 2026. arXiv:2510.16981.arxiv.org/abs/2510.16981
  23. [23]Dmitry Kovalev. Understanding gradient orthogonalization for deep learning via non-Euclidean trust-region optimization. arXiv preprint arXiv:2503.12645, 2025.arxiv.org/abs/2503.12645
  24. [24]Tim Large, Yang Liu, Minyoung Huh, Hyojin Bahng, Phillip Isola, and Jeremy Bernstein. Scalable optimization in the modular norm. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, 2024. doi: 10.52202/079017-2338.DOI
  25. [25]Tim Tsz-Kit Lau, Qi Long, and Weijie Su. PolarGrad: A class of matrix-gradient optimizers from a unifying preconditioning perspective. arXiv preprint arXiv:2505.21799, 2025.arxiv.org/abs/2505.21799
  26. [26]Mingyu Li. Low-rank decay for grokking in scale-invariant transformers: A spectral-geometric view. arXiv preprint arXiv:2606.04405, 2026.arxiv.org/abs/2606.04405
  27. [27]Zichong Li, Liming Liu, Chen Liang, Weizhu Chen, and Tuo Zhao. NorMuon: Making Muon more efficient and scalable. In International Conference on Machine Learning (ICML), 2026. arXiv:2510.05491.arxiv.org/abs/2510.05491
  28. [28]Jingyuan Liu et al. Muon is scalable for LLM training. arXiv preprint arXiv:2502.16982, 2025.arxiv.org/abs/2502.16982
  29. [29]Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019.
  30. [30]James Martens and Roger Grosse. Optimizing neural networks with Kronecker-factored approximate curvature. In International Conference on Machine Learning (ICML), volume 37 of Proceedings of Machine Learning Research, pages 2408–2417, 2015.arxiv.org/abs/1503.05671
  31. [31]M. Mohamadi, K. Wen, T. Wang, and Z. Li. Toner / MuonT: input-whitened Muon. Blog post and code, November 2026.
  32. [32]Vardan Papyan. Traces of class/cross-class structure pervade deep learning spectra. Journal of Machine Learning Research, 21(252):1–64, 2020.arxiv.org/abs/2008.11865
  33. [33]Barak A. Pearlmutter. Fast exact multiplication by the Hessian. Neural Computation, 6(1):147–160, 1994. doi: 10.1162/neco.1994.6.1.147.DOI
  34. [34]Guilherme Penedo et al. The FineWeb datasets: Decanting the web for the finest text data at scale. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, volume 37, 2024. doi: 10.52202/079017-0970.DOI
  35. [35]Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos, Zhenyu Zhu, Antonio Silvetti-Falls, and Volkan Cevher. Training deep learning models with norm-constrained LMOs. In International Conference on Machine Learning (ICML), volume 267 of Proceedings of Machine Learning Research, pages 49069–49104, 2025.arxiv.org/abs/2502.07529
  36. [36]Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. Technical report, OpenAI, 2019.
  37. [37]Artem Riabinin, Egor Shulgin, Kaja Gruntkowska, and Peter Richtárik. Gluon: Making Muon & Scion great again! (Bridging theory and practice of LMO-based optimizers for LLMs). arXiv preprint arXiv:2505.13416, 2025.arxiv.org/abs/2505.13416
  38. [38]Levent Sagun, Utku Evci, V. Ugur Guney, Yann Dauphin, and Léon Bottou. Empirical analysis of the Hessian of over-parametrized neural networks. arXiv preprint arXiv:1706.04545, 2017.
  39. [39]Daniel Selsam, Percy Liang, and David L. Dill. Developing bug-free machine learning systems with formal mathematics. In International Conference on Machine Learning (ICML), volume 70 of Proceedings of Machine Learning Research, pages 3047–3056, 2017.arxiv.org/abs/1706.08605
  40. [40]Wei Shen, Ruichuan Huang, Minhui Huang, Cong Shen, and Jiawei Zhang. On the convergence analysis of Muon. Transactions on Machine Learning Research, 2026. ISSN 2835-8856. URL https://openreview.net/forum?id=4nH4CulGaP.arxiv.org/abs/2505.23737
  41. [41]Egor Shulgin, Sam Laing, Antonio Orvieto, and Peter Richtárik. A quadratic lens on Muon’s polar update: Benefits, pitfalls, and the role of Newton-Schulz. arXiv preprint arXiv:2610.30280, 2026.arxiv.org/abs/2610.30280
  42. [42]Zakhar Shumaylov et al. Muon is not that special: Random or inverted spectra work just as well. arXiv preprint arXiv:2605.11181, 2026.arxiv.org/abs/2605.11181
  43. [43]Chongjie Si, Debing Zhang, and Wei Shen. AdaMuon: Adaptive Muon optimizer. arXiv preprint arXiv:2507.11005, 2025.arxiv.org/abs/2507.11005
  44. [44]Ziyuan Tang, Tianshi Xu, Yousef Saad, and Yuanzhe Xi. Hierarchical Muon: Tiled Newton-Schulz updates for efficient Muon optimization. arXiv preprint arXiv:2606.27216, 2026.arxiv.org/abs/2606.27216
  45. [45]Lloyd N. Trefethen. Approximation Theory and Approximation Practice, Extended Edition. Society for Industrial and Applied Mathematics (SIAM), 2019. doi: 10.1137/1.9781611975949.DOI
  46. [46]Nikhil Vyas et al. SOAP: Improving and stabilizing Shampoo using Adam for language modeling. In International Conference on Learning Representations (ICLR), 2025. arXiv:2409.11321.arxiv.org/abs/2409.11321
  47. [47]Shuche Wang, Fengzhou Zhang, Jiaxiang Li, Dirk Bergemann, and Zhuoran Yang. Why Muon outperforms Adam: A curvature perspective. arXiv preprint arXiv:2606.04662, 2026a.arxiv.org/abs/2606.04662
  48. [48]Shuche Wang et al. Muon outperforms Adam in tail-end associative memory learning. In International Conference on Learning Representations (ICLR), 2026b. arXiv:2509.26030.arxiv.org/abs/2509.26030
  49. [49]Kaiyue Wen, David Hall, Tengyu Ma, and Percy Jiang. Fantastic pretraining optimizers and where to find them. In International Conference on Learning Representations (ICLR), 2026. arXiv:2509.02046.arxiv.org/abs/2509.02046
  50. [50]Fangzhou Wu, Rikhav Shah, Sandeep Silwal, and Qiuyi Zhang. DynMuon: A dynamic spectral shaping view of Muon. arXiv preprint arXiv:2605.17109, 2026a. ICML 2026 Workshop on High-dimensional Learning Dynamics (HiLD).arxiv.org/abs/2605.17109
  51. [51]Xiaodong Wu, Wenyi Yu, Chao Zhang, and Philip Woodland. Spectral allocation: Why Muon outperforms Adam, and how to improve Muon. arXiv preprint arXiv:2608.25990, 2026b.arxiv.org/abs/2608.25990
  52. [52]Yuhuai Wu, Mengye Ren, Renjie Liao, and Roger Grosse. Understanding short-horizon bias in stochastic meta-optimization. In International Conference on Learning Representations (ICLR), 2018.arxiv.org/abs/1803.02021
  53. [53]Chenrui Xu, Wenjing Yan, and Ying-Jun Angela Zhang. FISMO: Fisher-structured momentum-orthogonalized optimizer. arXiv preprint arXiv:2601.21750, 2026.arxiv.org/abs/2601.21750
  54. [54]Ruijie Zhang et al. MUON+: Towards more effective Muon via one additional normalization step for LLM pre-training. arXiv preprint arXiv:2602.21545, 2026a.arxiv.org/abs/2602.21545
  55. [55]Yechen Zhang et al. Mousse: Rectifying the geometry of Muon with curvature-aware preconditioning. arXiv preprint arXiv:2603.09697, 2026b.arxiv.org/abs/2603.09697
  56. [56]Rosie Zhao, Depen Morwani, David Brandfonbrener, Nikhil Vyas, and Sham Kakade. Deconstructing what makes a good optimizer for language models. In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2407.07972

Paper details

vlnc:W_m9t2tvq9
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents