WorksComputer Science
Equal Steps, Unequal Curvature: Where Muon's Update Goes Wrong
Abstract
Muon updates each hidden weight matrix along the polar factor of its momentum, the steepest-descent direction under the spectral norm. That choice fixes two allocations without consulting the loss: every singular direction of the momentum receives the same step, and in a non-square matrix each neuron's step is set by its leverage score. Whether they are good is a question about curvature, which steepest-descent arguments do not answer; we measure it. Using held-out gradients and Hessian-vector products, we decompose Muon's step by singular-value band and by neuron group and compute each group's second-order optimal multiplier. In a 124M-parameter decoder trained on FineWeb-Edu, the multipliers rise monotonically from the top singular directions to the tail: Muon over-steps the high-curvature top of its spectrum. We prove in Lean when such a diagnostic licenses an intervention at Muon's own update norm. On the measured curvature, the criterion predicts, and held-out losses confirm, that moving step length off the top helps mid-training and early in the cooldown, but not at its end. We then test interventions under per-step norm matching, tuned baselines and paired seeds. Orthogonalizing Q, K and V separately rather than as one fused matrix lowers the loss by 0.0250 nats and accounts for all of the gain of block-wise variants. A matmul-only, rotation-equivariant spectral tilt applied in the cooldown lowers the loss of this stronger baseline by a further 0.0120 nats, additively with NorMuon (0.0158 nats together, three seeds; the baseline needs about 4.3% more tokens to match), and by 0.0188 nats at 355M parameters (three seeds). Energy-matched Schatten-1 weight decay adds a further 0.0052 nats. Geometrically principled changes that do not act on the measured miscalibration, namely exact polar factors, steepest-descent input whitening and top-heavy decay, do not help.
Introduction
Muon (Jordan et al., [21]) has become the reference optimizer for the hidden matrices of transformers at small and medium scale (Liu et al., [28]; Essential AI et al. [13]; Jordan et al. [20]). Its theory is clean. For a weight matrix with momentum , the update direction is the steepest-descent direction under the spectral norm (Bernstein and Newhouse, [5]; Large et al. [24]; Pethick et al. [35]), and Muon with decoupled weight decay solves a spectral-norm-constrained problem (Chen et al., [8]).
That theory picks the norm in which the step is steepest. It is silent on whether the step’s distribution is right, and fixes two distributions without asking. (i) Across singular directions, every direction receives a step of the same length, whatever its gradient signal and whatever the loss curvature along it. (ii) Across neurons, a tall polar factor has row norms equal to the square roots of the leverage scores of ’s column space. Each output neuron therefore moves by an amount set by the momentum’s geometry and by how matrices happen to be fused in the implementation.
Recent variants change both distributions. NorMuon, Muon+ and Aurora rebalance neurons (Li et al., [27]; Zhang et al. [54]; Dewulf et al. [11]). DynMuon, Freon/Kaon and Spectrally Targeted Muon reshape the spectrum (Wu et al., [50]; Shumaylov et al. [42]; Goyal and Ramkumar [15]). MuonBP and HiMuon orthogonalize blocks (Khaled et al., [22]; Tang et al. [44]). Some work also measures why. SAMuon probes held-out loss-optimal step sizes along the momentum’s singular directions and builds an optimizer from that profile (Wu et al., [51]). BulkBoost proves when a norm-matched two-band reallocation helps to first order Chaitanya et al., [6]. Curvature analyses explain Muon’s advantage through its actual update Wang et al., [47]; Shumaylov et al. [42]; Shulgin et al. [41]. These measurements treat directions or bands one at a time or compare Muon with Adam, and gains against loosely tuned baselines often shrink against well-tuned ones Wen et al., [49]; Zhao et al. [56].
This paper measures the joint picture. We treat the flat spectrum and the leverage-weighted neuron allocation as hypotheses and test them against the loss’s local geometry. We split Muon’s next step into groups of singular directions or of neurons. On held-out data we measure each group’s gradient alignment and the full curvature matrix between groups, including the cross-group terms that isolated probes omit. The ratio is the step multiplier that would be optimal for group on its own to second order, and Muon corresponds to multiplier 1 everywhere. A one-step greedy analysis is known to misjudge the overall step size Wu et al., [52]. We therefore use an exact identity, proved in Lean, that splits Muon’s second-order gap into a scale part (the overall step size) and a shape part (the relative weighting across groups). Our experiments compare steps of equal norm, so we also prove when a same-norm reallocation can beat Muon’s step, and we evaluate that criterion on the measured curvature.
What the anatomy shows (§ 4). The optimal multiplier rises monotonically from the top singular directions to the tail. On ranks 513–768 it is 38–190 times its value on ranks 1–8. Muon thus over-steps the high-curvature top of its spectrum relative to the rest. At Muon’s own update norm, which is the comparison our experiments make, reallocating step length off the top would change the one-step loss by −0.00589 mid-training and −0.00488 at the start of the cooldown. Muon’s own step changes it by −0.00079 and +0.00026. By the end of the cooldown the possible gain is only −0.00009. Measuring the actual held-out loss confirms this: at matched norm, halving only the top eight singular directions improves Muon’s one-step loss by −4.4 and −3.2 (), and not at the end. Per-neuron step sizes vary most in the fused QKV matrix (coefficient of variation 0.43, against 0.18 in the MLP).
What acting on it buys (§ 6). We test interventions under strict controls. Each direction-changing arm is norm-matched to Muon at every step, baselines use tuned (interior-optimum) learning rates, and seeds are paired. Three findings stand out.
The neuron allocation that matters is per projection. Orthogonalizing , and separately, a known practice Jordan et al., [20], changes the final loss by −0.0250 nats [−0.0290, −0.0210] at 100.7M tokens. It accounts for all of the gain of square-block variants (residual +0.0011 [−0.0029, +0.0051]). Finer rebalancing, per head or per row, does worse. The fused-QKV baseline would need about 6.8% more tokens to match, which confounds comparisons: NorMuon’s gain shrinks from 0.0150 to 0.0038 against the split baseline.
Tilting the spectrum away from its top helps. A cooldown tilt toward small singular values lowers the stronger baseline’s loss by a further 0.0120 nats (−0.0137, −0.0102), better in every seed. We implement it with a Chebyshev filter evaluated by Clenshaw’s recurrence, using matrix products only, and prove that it keeps Muon’s rotational equivariance.
The two corrections add. NorMuon + tilt gives −0.0158 [−0.0178, −0.0138], against −0.0038 and −0.0120 separately. At 355M parameters the combination gains 0.0188 nats [−0.0202, −0.0175] (three seeds). Energy-matched Schatten-1 weight decay adds a further −0.0052 [−0.0066, −0.0038] at 124M.
The negatives are equally informative. Input whitening is provably -equivariant and conditioning-free on least squares, yet it hurts (up to +0.06 nats). More accurate polar factors do nothing, and top-heavy decay hurts; none acts on the miscalibration we measured. The diagnostic and the first screens ran concurrently, so the anatomy’s implications are consistency checks, not pre-registered predictions; the factorial ablations and the 355M runs came later.
Contributions. (1) A held-out, curvature-resolved diagnostic for a matrix optimizer’s step, with an exact scale/shape decomposition and a proved criterion for when a same-norm reallocation can improve the step. (2) An anatomy of Muon: monotone over-stepping of the top singular directions, which a same-norm reallocation can exploit until late in the cooldown, and a neuron imbalance concentrated in fused matrices. (3) A controlled attribution, including a factorial with three seeds and an A/A noise check. (4) Machine-checked statements of the geometry used (Section A), and an account of what they did and did not predict (§ 7). Every optimizer ingredient we evaluate has a precedent, and so does probing Muon’s spectrum against held-out loss (§ 8). What is new is the joint-curvature diagnostic with its fixed-norm criterion, the neuron-level and QKV attribution, and the controls that test them.
Two allocations hidden in Muon’s step
For a hidden matrix with gradient , the Muon variant we study forms the sum-form Nesterov input and steps along a quintic Newton–Schulz map:
with shape factor . NS normalizes by its Frobenius norm and applies five iterations of with [21]. On singular values it acts as a scalar map that is close to 1 on most of the spectrum but well below 1 on near-null directions. Hence .
Allocation across singular directions. Write any spectral filter as . SGD is , Muon is (approximately) , and power-law tilts are [50]. Muon’s choice is optimal for the worst case over a spectral-norm ball. It is not obviously optimal for a loss whose curvature differs across singular directions.
Allocation across neurons. Every theorem and proposition in this paper is proved in Lean 4; Section A states each in full. Row of is the incoming weight vector of output neuron . Muon moves it by .
Proposition 1 (Polar row norms are leverage scores; Theorem A.10). Let with (). Then , but the individual , the leverage scores of [12], can be arbitrarily uneven: one row can have norm 1 and another 0. If and , every row has unit norm.
So Muon’s per-neuron step depends on the shape of the matrix it is applied to. Storing Q, K and V as one fused matrix (the GPT-2 layout [36]) makes each projection’s step depend on the momenta of the other two. The block-wise alternative is the exact steepest step of a natural norm:
Proposition 2 (Block-wise polar steps; Section A.2). Split into blocks and let . Then maximizes over , and the joint polar factor can be strictly worse (for it attains against 2). For a tall split into exactly orthogonal row blocks, Muon’s shape factor gives the joint step the same Frobenius energy as the block-wise step.
Both propositions concern exact polar factors. is only approximately orthogonal, so in practice block-wise steps shrink the row spread but do not remove it. The block-wise steepest step for this max-norm is MuonBP’s Block-Muon [22]. The leverage-score reading of tall polar factors is also given by Dewulf et al. [11]. We use both diagnostically, to locate where Muon’s per-neuron steps are uneven, not to show that the unevenness hurts.
A curvature-resolved anatomy of the step
Grouped multipliers. We partition Muon’s next step into groups in one of two ways. Bands: singular-value ranks of the momentum, with for every matrix . This is the exact-polar idealization of Muon’s step; NS itself gives near-null directions less than unit length. Neurons: sextiles of neurons ranked by the norm of their row (or column) in Muon’s actual NS step. Now scale group by , i.e. . To second order in , the held-out loss changes by
where is the loss on held-out tokens. Thus and refer to the held-out loss, not the training minibatch, though they still carry finite-sample noise. Muon is . The stationary point of is . It is the minimizer of only when , and the unique one when (Section A.3). Every curvature matrix we estimate is positive definite, with smallest eigenvalue at least . The per-group multiplier minimizes for group on its own. Muon’s flat filter is second-order optimal up to a common scale exactly when . All scale as , so only their ratios are comparable across learning rates.
Scale versus shape. A greedy one-step model is a poor guide to the overall step size, because it ignores the step’s effect on future iterates [52, 4]. It can still say which groups get too much or too little of a given step. The following identity separates the two.
Theorem 3 (Exact scale/shape split; Section A.3). Let be symmetric positive definite, the minimizer of , and the minimizer of over , i.e. the best uniform rescaling of Muon’s step. Then
and the shape term is strictly positive unless is flat.
The identity itself needs only symmetry and . Positive definiteness is what makes and optima, and both terms losses relative to them. The step-size term carries the known short-horizon bias. Removing it does not make the shape immune: in an explicit example, a one-step shape preference reverses after one more gradient step (Theorem A.28). We therefore treat the shape as a one-step diagnostic whose implications must be tested in training, not as a prescription.
Fixed norm. Theorem 3 compares Muon with , whose norm generally differs from Muon’s. Our experiments instead compare steps of equal Frobenius norm. Band directions are mutually orthogonal, with squared norms proportional to the band sizes , so equal norm means with . A positive shape term does not by itself imply that such a step beats Muon:
Proposition 4 (Same-norm reallocation; Section A.3). Let , the model gradient at Muon’s step, and the component of orthogonal to . (i) If , some with has . (ii) If and for some , no such exists. (iii) The shape term does not decide between (i) and (ii). For , and , the shape term is , yet Muon is optimal among all same-norm steps.
We therefore also compute the best same-norm step, an exactly solvable trust-region problem whose solution is certified globally optimal on the ellipsoid (Theorem A.22), and the model’s prediction for the norm-matched filters we train with. Two remarks make this the right comparison. First, every hidden matrix has rank 768 and bands are defined by rank, so each matrix’s step has squared norm proportional to ; matching each matrix’s norm separately, as the experiments do, is the same constraint for exact-polar bands. Second, the ratios are not the fixed-norm criterion: even with diagonal , the band with the larger ratio can be the one a same-norm step should shrink (Theorem A.27); only , which uses the full , decides.
Estimation and validity. We take checkpoints of a Muon training run (§ 6) at steps 1,024 (constant learning rate), 1,536 (start of the cooldown) and 2,048 (end), and form the exact next Nesterov input. Bands are ranks of every matrix. , the Hessian–vector products for , and realized loss changes use three disjoint held-out sets (393k, 98k and 393k tokens; Section B). Across 12 test steps (Muon and ; bands and neurons; three checkpoints), the predicted and realized changes have the same sign in every case and agree within a factor of three (Figure 3). Most importantly, the step realizes a much larger decrease than Muon’s step on unseen tokens: against at step 1,024. is ill-conditioned (condition number ), so is irregular across bands (Section B) and we use it only through realized losses and Theorem 3.

Figure 3. Predicted (quadratic model) versus realized held-out loss change for 12 test steps.
What the anatomy shows
F1: Muon over-steps its top singular directions. At every checkpoint the per-band multiplier rises monotonically from the top band to the tail (Figure 1a). The tail-to-top ratio is 48 at step 1,024, 38 at the start of the cooldown and 190 at the end. Within each of the four layer types separately, the top band has the smallest multiplier in all twelve type–checkpoint cases (Section B). The raw quantities show the mechanism. At step 1,024 the top band’s curvature is that of the next band, but its gradient alignment is only larger. Curvature falls off faster than signal with rank. Equal step lengths therefore over-step the high-curvature top directions, in which gradients concentrate ([17, 14, 10]). The ratios are isolated-band quantities, however. At fixed norm, the full-curvature analysis (F2) confirms the top as the place to take step length from but gives it to ranks 97–512, not to the extreme tail.
Figure 1. Anatomy of Muon’s step (124M decoder, Muon seed 43; screen schedule with cooldown over steps 1,536–2,048). (a) Per-band second-order optimal multiplier for the exact-polar step restricted to each singular-value band. Muon’s flat filter is (dashed). Multipliers scale as , and is 10× smaller at step 2,048, so compare profiles rather than levels. The tail-to-top ratio is 48, 38 and 190 at the three checkpoints. These isolated-band ratios are not the fixed-norm criterion (Theorem A.27); panel (b) uses the full curvature. (b) Measured change of the held-out loss after one step at Muon’s own update norm (393k tokens; Table 4). Bars show Muon, the exact tilt , halving ranks 1–8, and the best same-norm filter from the quadratic model (Theorem 4). Moving step length off the top helps severalfold mid-training and at the start of the cooldown, and not at its end, where Muon’s top band is no longer over-stepped. (c) Spread (coefficient of variation) of per-neuron update norms of Muon’s step, mean over the 12 matrices of each type, at step 1,024. The fused 2304 × 768 QKV matrix carries the largest imbalance. Block-wise polar factors halve the QKV spread and nearly remove the MLP spread.
F2: at Muon’s norm, moving step length off the top helps until late in the cooldown. At step 1,024 and at the start of the cooldown, only 17% and 12% of the gap of Theorem 3 is shape. The rest is the greedy model asking for a smaller step (). Even so, most of the gap is reachable without changing the step’s norm. The model gradient is almost entirely tangent to the same-norm ellipsoid ( and 1.00), so by Theorem 4 a same-norm step does better. The best one is predicted to change the one-step loss by and , against and for Muon; that is 87% and 92% of the full gap. It takes step length from ranks 1–96 and gives it to ranks 97–512 (Section B). We confirmed this by measuring the actual held-out loss after one step at matched norm (Figure 1b, Table 4). Relative to Muon, halving ranks 1–8 changes it by and (), and the best same-norm filter by and , which is 77% and 69% of the predicted gain. Along the tilt family , and at , so a negative power helps and a positive one hurts. A proved certificate (Theorem A.26) turns a predicted gain into a guaranteed one once the model’s errors are bounded; its constants are not yet measured (Section E). As a consistency check, the observed errors, at most at 12 calibration steps and – for the same-norm gains, are well below the predicted gains here and far above the one at the end. At the end of the cooldown the picture changes. The top band is no longer over-stepped (), the model predicts a gain of only , and the measured one-step losses of all filters agree with Muon’s to within (). Theorem 3 still attributes 97% of the gap to shape there, but that gap is measured against a step of very different norm and is out of reach at Muon’s norm. The headroom a norm-matched tilt can use is therefore largest early in the cooldown and gone by its end.
| Step | (Muon) | ranks 1–8 | best same-norm | ||||||
| 1,024 | −3.07 | −5.75 | −5.93 | −1.76 | +12.08 | +80.45 | −6.14 | −6.28 | +0.090 |
| 1,536 | −2.48 | −4.13 | −3.99 | −1.03 | +7.95 | +48.44 | −4.27 | −4.40 | +0.060 |
| 2,048 | −0.13 | −0.26 | −0.32 | −0.33 | −0.38 | −0.22 | −0.29 | −0.21 | −0.000 |
Table 4. Measured change of the held-out loss () after one step at Muon’s own update norm: the exact tilt for several , halving ranks 1–8, and the best same-norm filter, plus the slope at .
F3: neuron imbalance is concentrated in fused QKV. Per-neuron update norms of Muon’s step vary most in the fused QKV matrix (CV 0.43, per-layer range 0.14–0.73). The MLP’s expand rows and contract columns have CV 0.18, and the square attention-output matrix has CV 0.02 (Figure 1c). Grouping neurons by Muon’s update size gives the same monotone picture as the bands: the most-moved sextile has multiplier 1.5 and the least-moved 7.1 (Section B). Only 5% of the neuron-group gap at step 1,024 is shape (65% at the end), so any benefit of block-wise steps must come mostly from the new directions they produce, not from reweighting.
Implications. We test three implications in § 6. I1: rebalancing should matter most where the imbalance is largest, in QKV, and little in the MLP. I2: at fixed norm, moving step length off the top of the spectrum () should help, independently of I1, because it acts on different structure. By F2 its per-step benefit is largest from mid-training to the start of the cooldown and gone by the end, because the decaying learning rate itself removes the top’s over-stepping. A tilt confined to the cooldown should therefore build an early lead that partly erodes. I3: changes that barely alter the allocation should not help. Making NS a more exact polar map is one: it removes NS’s small deviations from unit length, mostly by lifting near-null directions, and leaves the top’s share relative to the bulk essentially unchanged.
From measurements to interventions
Neuron allocation: block-wise polar steps. Following Theorem 2, we orthogonalize blocks separately. Split QKV uses three square 768 × 768 blocks, one each for Q, K and V. Per-head uses 36 wide 64 × 768 blocks, whose polar factors have exactly orthonormal rows. Square blocks splits QKV and also splits the MLP’s 3072 × 768 and 768 × 3072 matrices into four square blocks each. Wide matrices’ column blocks are scaled by to keep Muon’s update energy. All reported block arms are norm-matched anyway; the factor matters for the deployable version (Section D). The stateful alternative from prior work is NorMuon’s row normalization by an EMA of row second moments ([27]).
Spectral allocation: a matmul-only cooldown tilt
The exact tilt needs an eigendecomposition per matrix and step. We avoid it as follows. Let with , and . A polynomial in acts in the same singular basis (Theorem 5):
With the filter is up to scale. On the Newton–Schulz plateau () this is the exact tilt up to a factor in ; near-null directions are lifted further. We use a degree-12 least-squares Chebyshev fit of on . is 1.01 times a 20-step power-iteration estimate of , which is an estimate rather than a proved bound. We then rescale to (Algorithm 1).
Proposition 5 (What the tilt computes; Section A.1). For any coefficients and depth and any real polynomial , let and . (i) has the form , so for every orthogonal . (ii) If with orthonormal columns, then and , with the scalar iterate of .*
Part (ii), not (i), is what keeps the singular vectors: is also a Gram map. The normalization, and the final rescaling are scalars. The tilted step therefore changes only how singular directions are weighted, so a comparison with Muon isolates the spectral allocation. One numerical detail decides whether the method works. Evaluating in monomials by Horner’s rule in TF32 cancels catastrophically (coefficients reach ) and flipped the filter’s sign on the top direction. With square blocks as the base (−0.0219 alone, against fused-QKV Muon), that version reached −0.0240, against −0.0322 for Clenshaw’s recurrence [9, 45] in FP32 and −0.0332 for the exact tilt; the Clenshaw filter matches the exact tilt to direction cosine 0.998–0.999, so the normalized steps differ by at most 6.3% in Frobenius norm.
Experiments
Setup. We train a 124.5M-parameter pre-norm decoder (12 layers, width 768, 12 heads, tied embeddings) on a 167.8M-token FineWeb-Edu sample [34] with the GPT-2 tokenizer, sequence length 512 and 24,576 tokens per step. Muon (1) acts on the 48 hidden matrices with momentum 0.95 and decoupled weight decay 0.1. Its state is in FP32, and Newton–Schulz runs in FP32 with TF32 matrix products. AdamW [29] () handles embeddings and norms, and gradients are clipped at norm 1.0. The schedule is warmup–constant–cooldown [18]: 128 warmup steps, then linear decay to 0.1× over the last 25% of steps. The screen runs 2,048 steps (50.3M tokens). The confirmation runs 4,096 steps (100.7M tokens). The metric is final loss on 4.18M held-out tokens. A 355M model (24 layers, width 1024, 16 heads) tests scale.
Controls. Norm matching: unless stated otherwise, every direction-changing arm is rescaled, per matrix and per step, to the Frobenius norm of Muon’s NS5 output on the same momentum, so differences reflect direction, not step size. Tuned baselines: on the screen, Muon’s learning rate was swept over {0.0035, 0.005, 0.007, 0.01}, split-QKV Muon’s over {0.0035, 0.005, 0.007}, and the 355M baseline’s over the same three values. 0.005 is an interior optimum in all three cases, and every new arm uses the baseline’s rate. Paired seeds: each seed fixes initialization and data order. Confirmations use seeds 42–44 for every arm, and we report paired 95% -intervals. A/A check: before the cooldown the tilt arms run an optimizer mathematically identical to their baselines. Their intermediate evaluations differ by at most 0.009 nats, an empirical bound on numerical non-determinism plus evaluation noise. The screen baseline’s seed spread is 0.0032 nats. Gating: midway through the study we adopted a rule that a full-length run is launched only if its screen beats the baseline by more than 0.005 nats. The factorial ablations and the 355M seeds 43–44 were released under it; the earlier confirmations were not.
| Optimizer | Loss | vs. Muon | vs. split-QKV |
| Muon (fused QKV) | 3.9427 | — | +0.0250 |
| NorMuon (fused QKV) | 3.9277 | −0.0150 | +0.0100 [−0.0003, +0.0017] |
| Square blocks | 3.9187 | −0.0239 | +0.0011 [−0.0043, +0.0051] |
| Split-QKV Muon | 3.9176 | −0.0250 | — |
| + cooldown tilt | 3.9057 | −0.0370 | −0.0120 [−0.0014, −0.0100] |
| + NorMuon | 3.9139 | −0.0288 | −0.0038 [−0.0006, −0.0061] |
| + NorMuon + tilt | 3.9018 | −0.0408 | −0.0158 [−0.0018, −0.014] |
| + NorMuon + tilt + Schatten-1 decay | 3.8966 | −0.0460 | −0.0210 [−0.0024, −0.018] |
| Sq. blocks + NorMuon + tilt | 3.9050 | −0.0376 | −0.0126 [−0.0015, −0.0100] |
Table 1. Confirmation at 100.7M tokens (124M model, three paired seeds per arm). Mean final held-out loss and mean paired differences against the original fused-QKV Muon and against the stronger split-QKV Muon. Brackets give 95% -intervals for the latter; intervals against fused-QKV Muon are in the text. Lower is better.
I1: the neuron gain is the QKV split. Splitting Q, K and V changes the final loss by −0.0250 nats [−0.0290, −0.0210] relative to fused-QKV Muon (Table 1). It is better in every seed, and fused-QKV Muon would need about 6.8% more tokens to match. Square blocks everywhere add nothing beyond it (+0.0011 [−0.0029, +0.0051]). Square blocks in the MLP alone, with QKV fused, change the loss by +0.0039 (screen). As I1 implies, the large QKV imbalance is the costly one and the moderate MLP imbalance is harmless. The data also locate the effect at the level of projections rather than rows. Per-head blocks equal-size rows exactly yet do worse than three blocks (−0.0172 against −0.0255, screen). NorMuon’s row normalization on fused QKV is worse than the split (+0.0100 [−0.0031, +0.0170]), and on top of the split it adds only −0.0038 [−0.0062, −0.0013]. One reading is that in a fused matrix Q, K and V share a single spectral budget, so the projection with the strongest momentum claims most of it, whereas splitting gives each projection its own budget. We have not isolated this mechanism. Either way, methods evaluated against fused-QKV Muon can take credit for a gain the split provides at no cost.
I2: the tilt helps, and adds to neuron balance. On the split baseline the cooldown tilt changes the loss by −0.0120 nats [−0.0137, −0.0102] and is better in all three seeds (−0.0113, −0.0127, −0.0119). The factorial over {NorMuon, tilt} on split-QKV Muon is additive within noise: −0.0038 and −0.0120 separately, −0.0158 [−0.0178, −0.0138] together. Each component’s effect is the same with or without the other: the tilt given NorMuon gives −0.0120, and NorMuon given the tilt gives −0.0038. The tilt supplies about three quarters of the combined gain. By the split baseline’s own loss-versus-tokens curve (0.260 nats per doubling, a two-point log-linear estimate), split-QKV Muon would need about 4.3% more tokens to match the combination. Other tilts and band filters behave consistently on the screen (Section C); the dose was not optimized. Consistent with F2, timing and dose matter little: on the full stack, starting the tilt at step 1,024 or using changes the screen loss by only and (one seed each, within the 0.003 seed spread). We keep the tilt in the cooldown, where it costs least.
How the gain develops. Figure 2b shows the tilt’s lead peaking at nats 256 steps into the cooldown and narrowing to at the end. This is the pattern F2 predicts. At fixed norm the per-step benefit of moving step length off the top is large at the start of the cooldown and gone by its end, because the decaying learning rate itself removes the top’s over-stepping, so the baseline recovers part of the early lead. The baseline’s cooldown length and final learning rate were not tuned, so a different cooldown might recover part of the final gain; we have not tested this.
Figure 2. Controlled results at 100.7M tokens (124M, three paired seeds). (a) Paired difference in final loss against split-QKV Muon. Splitting Q, K and V explains the whole square-block gain; NorMuon on fused QKV is worse than the split; the tilt and NorMuon add on top of the split. (b) Paired difference over training (mean, with the seed range shaded). Intermediate points use 98k held-out tokens; the last point uses 4.18M. Before the cooldown the tilt arm is mathematically identical to its baseline (A/A differences ). The tilt’s lead peaks at nats 256 steps into the cooldown and settles at .
Scale. At 355M parameters (GPT-2-medium shape, 96 Muon matrices, same data and tokens), the split-QKV baseline’s learning-rate screen again peaks at 0.005 (4.0642/4.0547/4.0647 for 0.0035/0.005/0.007). Against the baseline at the same seeds, NorMuon + tilt changes the final loss by nats [, ] (per seed , , ), comparable to the 124M effect; the baseline’s 3 seeds span only 0.0008 nats.
I3 and other geometric changes. Table 8 (Section C) collects them. Three extra cubic Newton–Schulz iterations drive every singular value of the NS5 output to 1 and change the loss by , within noise, as I3 implies. Input whitening, steepest descent in the data metric , has the cleanest theory of anything we tried. In its exact form it is GL()-equivariant and conditioning-free on least squares (Section A.4); the damped version we run is approximately so. It is harmful: to when norm-matched, and at its own magnitude. A whitened step concentrates its energy in low-variance input directions, so at a fixed norm it moves the layer’s outputs less. The published Toner recipe ([31]) gives (one seed), the sign of its reported gain at smaller size. For decay geometry, energy-matched Schatten-4 decay, which mainly shrinks the top of the spectrum, hurts (). Schatten-1 decay, which removes about the same amount from every singular value and so prunes the bulk, helps ( [, ], three seeds). Decay along the polar factor of is prior art, used with a fixed coefficient for low rank, where with Muon it costs loss ([3]), and for grokking ([26]). With our energy-matched coefficient it is additive with our method: [, ] on top of split QKV + NorMuon + tilt at 100.7M tokens (three seeds, Table 1), against split-QKV Muon, at more wall time. In this regime the useful role of decay is to clean the bulk, not to cap the spectral norm.
| Change | Certified property | Anat. | |
| Split QKV | steepest for block max-norm; exact-polar blocks have unit rows; energy identity | yes | 1 |
| MLP square blocks | same | weak | |
| Cooldown tilt | keeps Muon’s -equivariance | yes | 2 |
| Accurate polar | — | no | |
| Input whitening | -equivariance; conditioning-free LS (undamped form) | — | – |
| Toner recipe | (prior art) | — | |
| Schatten-4 decay | — | — | |
| Schatten-1 decay | — | — | 3 |
Table 8. Table 8: Theory versus training. What the Lean statements certify, whether the anatomy implies a gain, and the measured paired against fused-QKV Muon (screen, one seed, unless marked). Formal properties alone did not predict which changes help.
Cost. Splitting QKV adds no FLOPs; its recorded wall time is mostly the norm-matching scaffold. In our unfused FP32 implementation NorMuon + tilt takes more wall time than split-QKV Muon at 124M and at 355M, against a token saving of about 4.3%. The gain is in tokens, not yet in wall clock; folding the tilt into the Newton–Schulz coefficient, as DynMuon does, may remove most of the overhead.
What formal verification did and did not buy
All formal results are stated in Section A. The Newton–Schulz results hold for the iteration Muon actually runs, with any coefficients, depth and normalizer. The block results concern exactly orthogonal blocks. What it bought. (i) Bookkeeping. The energy identity (Theorem 2), carried over to wide matrices, gives the factor without which a deployable block step on the MLP contraction is 1.9 too large. (ii) A symmetry guarantee. The cheap tilt keeps Muon’s right equivariance (Theorem 5). (iii) *A correction to our own reasoning. Theorem 3’s stationary points are optima only under positive definite curvature, and its shape term compares Muon with a step of a different norm. Formalizing the question our norm-matched experiments ask produced Theorem 4 and a finite-step certificate, and changed our reading of the diagnostic (F2). (iv) A change of direction. Earlier counterexamples (Section A) showed that spatial orientation and curvature, not temporal statistics, decide momentum preferences under NS. What it did not buy was a prediction of training benefit. Whitened Muon is provably equivariant and conditioning-free yet hurts; the joint polar factor is provably suboptimal for the block norm, yet MLP blocks do not help. Whether a certified property is the bottleneck is an empirical question.
Related work
Muon and its geometry. Muon (Jordan et al., [21]) and its scaling studies ([28, 13]) have been analyzed as steepest descent under operator norms ([5, 24]), as linear-minimization-oracle methods ([35, 37]), as non-Euclidean trust regions (Kovalev, [23]) and as constrained optimization ([8, 40]). Prior work also studies polar approximations (Amsel et al., [2]) and the benefit of flat spectra for tail-end memories (Wang et al., [48]). On matrix quadratics, Shulgin et al. [41] show that the polar direction beats the gradient direction, each at its best step size, when large singular directions sit in high curvature, and that finite Newton–Schulz acts as a damped polar update.
Spectra, neurons and blocks. DynMuon schedules a spectral power from to (Wu et al., [50]). Freon/Kaon show that random or inverted spectra can work as well as flat ones (Shumaylov et al., [42]), and Spectrally Targeted Muon, AdaMuon and PolarGrad adapt magnitudes ([15, 43, 25]). NorMuon and Muon+ normalize rows ([27, 54]), Aurora gives the leverage-score argument (Dewulf et al., [11]), and row-sphere and Stiefel constraints are compared by Hu et al. [19]. MuonBP and HiMuon orthogonalize blocks ([22, 44]), and separate Q/K/V orthogonalization is standard in modded-nanogpt (Jordan et al., [20]). Closest to our diagnostic, SAMuon measures held-out optimal step sizes along each singular direction of the momentum, finding a volatile head and a tolerant bulk, and builds an optimizer from the profile (Wu et al., [51]). BulkBoost proves when a norm-matched bulk/spike reweighting improves an objective to first order (Chaitanya et al., [6]), and Wang et al. [47] attribute Muon’s advantage over Adam to within- and cross-layer curvature of its update. Our tilt uses DynMuon’s dose and late-training sign. We add the full cross-group curvature, neuron groups, scale versus fixed-norm shape with proved criteria, and a controlled attribution: the QKV split accounts for the block-wise gains and most of NorMuon’s gain over fused-QKV Muon.
Data metric, curvature and evaluation. Shampoo, SOAP and K-FAC precondition with layer statistics ([16, 46, 30]), and whitened Muon has close 2026 relatives ([31, 7, 55, 1, 53]). Gradients concentrate in a few top Hessian directions (Gur-Ari et al., [17]) above a bulk ([38, 14, 32]), near the edge of stability (Cohen et al., [10]). Greedy step sizes suffer short-horizon bias ([52, 4]), and optimizer gains shrink against tuned baselines ([49, 56]). Like Wu et al. [51] and Wang et al. [47], we measure curvature along the optimizer’s own step, here with held-out Hessian–vector products (Pearlmutter, [33]). Formal verification of ML systems goes back at least to Selsam et al. [39].
Novelty. Held-out spectral probing of Muon’s step (Wu et al. [51]) and first-order analysis of norm-matched reallocation (Chaitanya et al. [6]) precede this work. We found no prior work that measures the full cross-group curvature of the step by singular band and by neuron group, separates scale from fixed-norm shape with proved criteria and counterexamples, and uses the result in a controlled attribution of published ingredients.
Limitations and conclusion
Scale and regime. One data distribution; 124M and 355M parameters (three seeds each); at most 100.7M tokens, under one token per parameter. Effect sizes are specific to this regime. The diagnostic is local and noisy. It is second-order, one step ahead and from a single run, without intervals. and come from separate held-out folds, and its joint optimum is ill-conditioned. It perturbs only the hidden matrices along exact-polar band directions, leaving out the AdamW and decay parts of the update. The measured one-step losses use one checkpoint seed and one validation subset, without intervals, and the certificate’s constants are not yet measured. We used the diagnostic for direction, not dose; the tilt power comes from prior work. The fixed-norm analysis covers bands only. Design and statistics. The anatomy was not pre-registered; intervals are nominal (three paired seeds); most screen arms are single-seed. Alternatives. The baseline’s cooldown was not tuned; the QKV mechanism is a hypothesis. Provenance. Comparisons with fused-QKV or square-block arms cross code versions; split-QKV arms share one (Section E).
Conclusion. Measured against curvature, both of Muon’s hidden allocations are miscalibrated. Its spectrum oversteps the top, which a same-norm reallocation can exploit until late in the cooldown, and its per-neuron steps are most uneven in fused QKV matrices. Corrections aimed at these help additively; principled changes that miss them do not.
Formal results
This appendix states, in ordinary mathematical language, every formal result the paper uses. All of them are proved in Lean 4 with Mathlib, in exact real arithmetic. They certify the algebra and geometry behind each method. They say nothing about floating-point behaviour, convergence or training benefit.
Notation. . For real coefficients let , and let be its -fold composition. For any function of an matrix, the Newton–Schulz map is
Muon’s is the case , , . Results stated for NS hold for every choice of and , and so apply to the iteration Muon actually runs, not only to an exact polar factor. A matrix is orthogonal if .
Newton–Schulz maps and the matmul-only tilt
Definition A.1 (Gram map). A map is a Gram map if there is a function such that for every .
Theorem A.2 (Gram maps commute with right rotations). If is a Gram map and is orthogonal, then for every .
Proof idea. , so .
Theorem A.3 (Newton–Schulz iterates are Gram maps). For all and , the map is a Gram map. Consequently for every orthogonal , for any normalizer .
Proof idea. Each step multiplies on the left by the matrix . That matrix depends only on the Gram matrix, and the class of Gram maps is closed under composition.
Theorem A.4 (The polynomial tilt preserves Muon’s symmetry). Let be any real polynomial and . The tilt is a Gram map. Hence for every orthogonal .
Right equivariance alone does not mean singular vectors are kept: for a fixed non-scalar is also a Gram map. The next theorem shows that the Newton–Schulz iterates and the tilt do keep them.
Theorem A.5 (Singular-basis action of Newton–Schulz and of the tilt). Let , where and have orthonormal columns (, ; any shapes, e.g. a thin SVD). Let be the scalar map iterated times, and . (i) . (ii) For every real polynomial , .
Proof idea. and make every product collapse onto the diagonal. Induct on for (i); for (ii), induct on the powers of , then use linearity in .
In the implementation is first normalized and is divided by an estimate of its largest eigenvalue. Both are scalar factors, absorbed into and . Both singular frames are therefore kept, and the implemented filter is , not . With it is up to scale. On the Newton–Schulz plateau () this differs from the exact tilt by the factor , which lies in for .
Theorem A.6 (From filter error to step error). (i) For non-zero in a real normed space and , . (ii) For unit vectors in a real inner-product space, .
By (i), a raw filter error of moves the norm-matched step by at most of its norm. By (ii), the measured direction cosine of 0.998 between the matmul and exact tilts means their normalized steps are within 6.3%.
Block geometry and per-neuron step sizes
Definition A.7 (Steepest direction). For a norm-like function and a gradient , a matrix is a steepest direction if and for every with $\mathcal{N}(D') \le 1.
Theorem A.8 (Block-wise steepest descent). Let be a block gradient and block norms. If each is a steepest direction for at , then is a steepest direction at for the max-norm , whose unit ball is the product of the block unit balls. With every the spectral norm, the block-wise polar factors are therefore steepest for .
Proposition A.9 (The joint polar step can be strictly worse). Take two blocks . The polar factor of the stacked matrix has blocks , a partial isometry with , and attains . The block-wise step is feasible for the max-norm and attains .
Proposition A.10 (Row norms of polar factors). (i) If is square and , every row of has unit norm. (ii) If has orthonormal columns , its squared row norms sum to . (iii) They need not be equal: satisfies and has row norms and .
The squared row norms in (ii) are the leverage scores of . In Muon’s step they are the per-neuron step sizes of a tall matrix.
Proposition A.11 (Energy identity for block-wise steps). Let with , let be orthogonal, and let . Then
For a tall matrix with rows, Muon’s shape factor therefore gives the joint step the same Frobenius energy as orthogonal blocks, and the learning rate transfers.
Second-order model of a filtered step
For group multipliers , a gradient-alignment vector and a symmetric curvature matrix , let (Equation (2)).
Theorem A.12 (Gap formula). If is symmetric and , then for every ,
Theorem A.13 (Scale/shape split). Let be symmetric with , let be a direction (Muon’s step is ), and let satisfy . Then
When is positive definite (Theorem A.16), the first term is the loss due to the overall step size along and the second the loss due to the shape of .
Proof idea. Write and apply Theorem A.12. The cross term vanishes by the condition on , so the identity is Pythagoras in the -inner product.
Lemma A.14 (The best uniform rescaling is an -projection). If , then satisfies . So Theorem A.13 applies with , the best scalar multiple of .
Lemma A.15 (The shape term is strictly positive). If is positive definite and , then . Muon’s flat step therefore loses something to shape unless the second-order optimum is itself flat.
When the stationary points are optima. Theorems A.12 and A.13 are identities that hold for any symmetric . Calling and optima, and the two terms of Theorem A.13 losses, needs curvature assumptions:
Theorem A.16 (Stationary points are minimizers under positive curvature). Let be symmetric and . (i) If is positive semidefinite, then for every . (ii) If is positive definite, then for every .
Lemma A.17 (The best uniform rescaling minimizes along the ray). If , then for every , where . Indeed .
Comparison at a fixed update norm. The shape term of Theorem A.13 compares with , which generally has a different norm. Experiments that norm-match every arm compare steps on the ellipsoid for a symmetric positive definite . Write for the gradient of at , , and
for the component of orthogonal to .
Theorem A.18 (Lagrange identity on the same-norm ellipsoid). Let and be symmetric. For every with and every real ,
Proof idea. Expand , and subtract times the constraint, which in terms of reads $2u^{\mathsf T}Nd+d^{\mathsf T}Nd=0.
Corollary A.19 (Certificate that no same-norm step improves). If and is positive semidefinite for some , then for every with .
Theorem A.20 (A same-norm improvement exists when the tangential gradient is non-zero). Let be symmetric positive definite, , and . Then there is with and .
Proof idea. The vector satisfies and . Rescale it to with ; then . Every with lies on the ellipsoid. Along the rational parametrization , ,
for constants . Choosing with makes the bracket at most .
Proposition A.21 (A positive shape term does not imply a same-norm improvement). Let , , and . The minimizer is , the best uniform rescaling is , and the shape term of Theorem A.13 equals . Yet for every with . Indeed, on that circle .
Here and , so Theorem A.19 applies. Theorems A.19 and A.20 together give the decision rule used in § 4: compute , and if it is non-zero, solve for the best same-norm step.
Certifying the best same-norm step and a finite-step gain.
Theorem A.22 (Certificate for the best same-norm step). Let and be symmetric. If satisfies , and , then for every with .
The best same-norm steps reported in § 4 satisfy these conditions at all three checkpoints: , so , and the stationarity residual is below . They are therefore the global optima of the model on the ellipsoid.
Lemma A.23 (Model-error budget). For estimates of and any , .
With and this is at most .
Theorem A.24 (A-posteriori finite-step certificate). Let be the true loss change of an implemented step and its predicted change, with for a candidate and every comparator in a set . If for every , then for every .
The certificate needs no inverse of . Each part of the error is bounded by a proved result:
Theorem A.25 (The three error terms). (i) Taylor remainder. Let be the loss after a fraction of a step, twice continuously differentiable with on . Then . (ii) Estimation. . Hence, if and for all , then . (iii) Implementation. If on a convex set containing the endpoints of the exact and the implemented step, and both steps are norm-matched to radius from raw outputs and , then their losses differ by at most .
In (i), and are the first and second directional derivatives of the held-out loss along the step, which the diagnostic measures. Part (iii) combines the mean-value inequality with Theorem A.6.
Corollary A.26 (End-to-end certificate). For an implemented candidate step and a comparator, let , and bound the implementation, Taylor and estimation errors of each. If , then the implemented candidate truly lowers the loss more than the comparator.
Which inputs are established. The theorems are unconditional; their inputs differ in status. Established for our measurements: the trust-region conditions of Theorem A.22 (, residual below ), so the reported best same-norm steps are certified optima of the model; the tangential gradient of Theorem A.20 (), so a same-norm improvement of the model exists at every checkpoint; and the hypotheses of Theorems A.5 and A.6, which hold by construction. Not established: the constants , , and of Theorem A.25, and sampling intervals for the measured one-step losses. Theorem A.26 is therefore not instantiated as a guarantee. The comparison in F2 with observed prediction errors is a consistency check only. Section E specifies the measurements that would instantiate it.
Limits of the diagnostic. Two further facts bound what the measurements can say.
Proposition A.27 (Per-band ratios are not the fixed-norm criterion). Let , , and . The isolated ratio is larger for band 2 ( against ). Yet every same-norm step that gives band 2 more weight is strictly worse than : and imply . The same-norm step , which gives band 1 more weight, is strictly better.
On the circle, . No cross-curvature is involved: at fixed norm what matters is the tangential part of (Theorem A.20), not the ratio .
Proposition A.28 (A one-step shape preference can reverse over two steps). Let , whose gradient step of size maps to , and let be the loss after one more such step. The points and have the same norm as . Then but .
Removing the step-size term therefore does not protect a one-step shape preference from horizon bias. This is why we treat the measured shape as a hypothesis for training experiments rather than as a prescription.
Input-whitened Muon
For a whitening factor , the whitened direction is . With it is the steepest direction for the data-dependent norm .
Theorem A.29 (Only the metric matters). If and is invertible, then . The whitened step depends on only, not on the square root chosen.
Proof idea. with orthogonal, and Theorem A.3 moves through .
Theorem A.30 (Exact -equivariance). Reparametrize the inputs by an invertible : , , , . (i) If and is invertible with , then . (ii) The full step with decoupled weight decay is equivariant too:
The trained function is therefore independent of the input coordinates.
Theorem A.31 (Steepest directions transfer through a change of metric). Let be invertible and any norm-like function. If is a steepest direction for at , then is a steepest direction at for the pulled-back function .
Theorem A.32 (Whitened Muon is conditioning-free on least squares). Let with and invertible. Run whitened Muon with and any step sizes . Then the whitened error satisfies
which is plain Muon on the isotropic loss , for every , however ill-conditioned.
Proposition A.33 (Plain Muon is not -equivariant). For and : if plain Muon’s step were the same in both coordinate systems, , then .
Energy-matched decay
Ordinary decoupled decay shrinks every singular value by the same factor. Energy-matched Schatten-1 decay replaces by with the polar factor of . Since , every singular value then loses the same absolute amount .
Theorem A.34 (Energy match). (i) If , then : subtracting from every singular value removes the same first-order energy as ordinary decay. (ii) More generally, for matrices and with , the rescaled direction satisfies and . The energy-matched step is ordinary decay plus a step orthogonal to .
Proposition A.35 (Which singular values shrink more). Let and . Under energy-matched Schatten-1 decay, shrinks by a larger fraction than under ordinary decay () if and only if . If and , then : smaller singular values always lose a larger fraction.
The decay therefore prunes the bulk of the spectrum and spares its top. That is the opposite of Schatten-4 decay, which hurt in our experiments while Schatten-1 helped.
Results from earlier formal work cited in Section 7
These results come from the earlier, temporal-filter part of the project. Most are explicit counterexamples or elementary identities, and we state them for what they rule out.
Proposition A.36 (Moments do not determine the momentum preference under NS5). (i) Any risk that is affine in the first two moments of a finite noise law takes the same value on two laws with equal mass, mean and second moment. (ii) There are two explicit laws for the amplitude of rank-two gradient noise with the following properties: equal means, equal second moments, identical temporal covariance of the noise streams, and identical linearized risk curves for every momentum . Yet under the actual five-step NS5 map with stabilized normalization, the first law strictly prefers to and the second strictly prefers to $0.95. (iii) Hence no rule that observes only the linearized risk curve can choose the better momentum for both laws.
Proposition A.37 (Spatial orientation decides the momentum preference). In an explicit full-rank construction with two consecutive updates, the signal path, drift, noise energy and time dependence are all matched. The Euclidean block risk is identical for every whether the noise is tangential or radial. After Newton–Schulz with any finite depth (including five), tangential noise makes strictly better than , and radial noise makes strictly better than .
Proposition A.38 (First-order alignment can mis-rank finite steps). Let . The unit steps and have equal length. At the origin, first-order analysis prefers (directional decrease against ), yet , so is the better finite step.
Proposition A.39 (Curvature decides between the polar and gradient directions). Let a step change a diagonal quadratic by , with gradient . The polar direction and the norm-matched gradient direction both have . (i) With curvature , is better. (ii) With , is better. (iii) With , the norm-matched intermediate direction is strictly better than both.
This is the two-direction version of Figure 1a: which spectral allocation is best depends on how curvature compares with gradient signal across directions.
Proposition A.40 (Greedy step-size facts). For a quadratic loss with gradient and a step : (i) , which is affine in . (ii) With and , the one-step loss decreases with exactly when . (iii) With no aligned signal (), every positive step looks too large to a greedy criterion. (iv) A bias added equally to the marginal value of every group does not change their comparison.
Diagnostic details and additional results
Scope. The diagnostic perturbs only the 48 hidden matrices, along exact-polar band directions or the actual NS neuron groups. The AdamW update of embeddings and norms and the weight decay are left out. and are estimated on separate held-out folds, so they are estimates of a common local model with separate errors, not derivatives of one empirical loss. The realized-loss checks use a third fold. The inverse-based quantities ( and the shape shares) are sensitive because is ill-conditioned. The fixed-norm band-filter predictions evaluate at a given vector and need no inverse.
Procedure. For a checkpoint at step (Muon, seed 43, screen schedule), we recompute the next training minibatch’s gradient with the same clipping, form exactly as the optimizer would, and take the full SVD of for each of the 48 matrices. Band directions use the exact polar factor , scaled by the learning rate at step and Muon’s shape factor. Neuron groups use Muon’s actual NS step, split into six equal-count sextiles by row norm (tall or square matrices) or column norm (wide matrices). The held-out gradient uses 768 validation sequences (393k tokens). Each Hessian–vector product uses 192 further sequences (98k tokens), computed by double backward with the math attention kernel. Realized loss changes use 768 further sequences. The three sets are disjoint. For groups and five group sets (all matrices, and each layer type), the diagnostic needs 30 Hessian–vector products per checkpoint.
Table 2: Band diagnostic over all 48 matrices. Top: per-band multiplier , best uniform rescaling , second-order gap (nats) and its shape share (Theorem 3). Bottom: the joint optimum and the condition number of . is irregular because is ill-conditioned, but it also puts nearly all its weight on the lower half of the spectrum. All multipliers scale as ; at step 2,048 is smaller.
| Step | 1–8 | 9–32 | 33–96 | 97–256 | 257–512 | 513–768 | gap | shape share | |
| 1,024 | 0.59 | 1.96 | 2.46 | 4.27 | 12.0 | 28.3 | 0.52 | 0.00674 | 17% |
| 1,536 | 0.55 | 1.90 | 2.39 | 4.21 | 9.52 | 21.0 | 0.49 | 0.00532 | 12% |
| 2,048 | 0.98 | 7.67 | 16.4 | 31.6 | 75.8 | 186.3 | 1.52 | 0.00137 | 97% |
| 1,024 () | 0.53 | 0.18 | 0.31 | 5.92 | 16.4 | ||||
| 1,536 () | 0.48 | 0.34 | 0.88 | 4.01 | 11.5 | ||||
| 2,048 () | 0.40 | 0.12 | 3.16 | 8.05 | 34.8 | 115.9 |
Table 2.
Table 3: Quadratic-model calibration (bands, all matrices): predicted versus realized change in held-out loss (393k disjoint tokens) for Muon’s step () and the model-optimal band shape ().
| Checkpoint | Muon, predicted | Muon, realized | , predicted | , realized |
| 1,024 | ||||
| 1,536 | ||||
| 2,048 |
Table 3.
Measured one-step losses at matched norm
To test the fixed-norm predictions with the actual loss rather than the quadratic model, we start from the same checkpoints and form each matrix’s next Nesterov input. We take the exact tilted direction , rescaled per matrix to the Frobenius norm of Muon’s NS step ( is Muon), or a norm-matched band filter. We then measure the held-out loss after one step on a fixed 393k-token set (Table 4). The slope at is a central finite difference. The best same-norm filter is the step-1,536 model optimum. One checkpoint seed and one validation subset are used, without intervals.
Fixed-norm analysis. Band directions are mutually Frobenius-orthogonal, with (every hidden matrix has rank 768). Every matrix’s step therefore has the same norm as Muon’s exactly when with . The best same-norm step minimizes on this ellipsoid. With and , it is , with chosen by a one-dimensional root search so that the constraint holds. Norm-matched filters are rescaled onto the ellipsoid before evaluation. Table 5 gives the predictions.
| Checkpoint | Muon’s | best same-norm | band filter | ranks 1–8 halved | |
| 1,024 | −0.00079 | −0.00589 | −0.00570 | −0.00527 | 1.00 |
| 1,536 | +0.00026 | −0.00488 | −0.00478 | −0.00443 | 1.00 |
| 2,048 | −0.00028 | −0.00009 | +0.00005 | +0.00001 | 0.79 |
| Best same-norm at step 1,024 | (0.54, 0.22, 0.21, 1.26, 1.26, 0.62) | ||||
| Best same-norm at step 1,536 | (0.49, 0.35, 0.17, 1.38, 1.17, 0.63) | ||||
| Best same-norm at step 2,048 | (0.64, 2.92, 2.30, 1.05, 0.39, 0.17) |
Table 5. Second-order predictions at Muon’s own update norm (bands, all 48 matrices). Column 2 is Muon’s predicted one-step change . Columns 3–5 are for the best same-norm step, the norm-matched band filter used in training ( at step 1,024), and halving ranks 1–8 with norm matching. The last column is the relative size of the tangential gradient (Theorem A.20). Bottom: the best same-norm step’s band multipliers.

Figure 4. Greedy multiplier per neuron sextile, ranked by Muon’s per-neuron update norm. The most-moved neurons sit in higher-curvature directions.
Implied spectral exponent. To relate the band multipliers to a power law , we take each matrix’s singular values at ranks 1, 8, 32, 96 and 256, normalize by the value at rank 96, and average them in log space over the 48 matrices. Each of the top four bands is represented by the geometric mean of its two edges, and we regress on . The slopes are −0.8, −0.8 and −1.4 at the three checkpoints. The estimate is crude: the band representatives are coarse, and the tail bands, whose lower edges we did not log, are excluded. It is also a greedy quantity, so it overstates how strong a tilt to use. We report it only to show that the prescribed sign is negative and of order one in magnitude. The deployed power, , is deliberately conservative.
Figure 5. Band multipliers computed separately for each layer type. The top band has the smallest multiplier in every case.
Per-neuron spread. Table 6 reports the coefficient of variation (CV) of per-neuron update norms at the three checkpoints. Each entry is the mean over the 12 matrices of a type, along the axis that indexes neurons: rows for qkv, expand and attention-out, columns for contract. For comparison, the raw momentum’s QKV row-norm CV is 0.93. Muon reduces the spread but does not remove it. Block-wise NS steps leave a residual QKV spread (CV 0.22) although exact orthogonal blocks would have none (Theorem A.10). A plausible cause is that NS maps the near-null singular directions of low-rank blocks to values well below 1; we have not verified this.
| Layer (neuron axis) | Muon, step 1,024 | Muon, step 2,048 | Block-wise, step 1,024 | Muon max/median, step 1,024 |
| qkv (rows) | 0.43 | 0.47 | 0.22 | 1.84 |
| expand (rows) | 0.18 | 0.18 | 0.03 | 1.72 |
| contract (columns) | 0.18 | 0.17 | 0.02 | 1.67 |
| attention-out (rows) | 0.02 | 0.02 | 0.02 | 1.05 |
Table 6. Coefficient of variation of per-neuron update norms of Muon’s step and of the block-wise polar step.
All screen runs
Table 7 and Figure 6 list every completed screen arm (2,048 steps, 50.3M tokens). Differences are paired against fused-QKV Muon at LR 0.005: the seed-42 difference for every arm, and the mean over seeds 42–44 with a 95% -interval where three seeds exist. The arms were selected adaptively during the study; no multiplicity correction is applied.
Table 7: Screen results. Final held-out loss (4.18M tokens) at seed 42 and paired differences from fused-QKV Muon.
| Family | Arm | Final (s42) | (s42) | , 3 seeds [95% CI] |
| Muon, LR 0.005 (seeds 42/43/44: 4.2043, 4.2066, 4.2035) | 4.2043 | — | — | |
| Learning rate | Muon, LR 0.0035 | 4.2179 | +0.0136 | |
| Learning rate | Muon, LR 0.007 | 4.2077 | +0.0034 | |
| Learning rate | Muon, LR 0.01 | 4.2376 | +0.0332 | |
| Learning rate | Split QKV, LR 0.0035 | 4.1931 | −0.0112 | |
| Learning rate | Split QKV, LR 0.007 | 4.1847 | −0.0196 | |
| Learning rate | Square blocks, LR 0.007 | 4.1840 | −0.0203 | |
| Learning rate | Square blocks, LR 0.01 | 4.2045 | +0.0001 | |
| Neuron balance | Split QKV (Q, K, V orthogonalized separately) | 4.1788 | −0.0255 | −0.0267 [−0.0304, −0.0230] |
| Neuron balance | Per-head Q/K/V blocks (36 blocks of ) | 4.1871 | −0.0172 | |
| Neuron balance | Square blocks (QKV + MLP) | 4.1824 | −0.0219 | −0.0238 [−0.0282, −0.0194] |
| Neuron balance | Square blocks, not norm-matched (deployable) | 4.1843 | −0.0200 | |
| Neuron balance | Square blocks in the MLP only (QKV fused) | 4.2082 | +0.0039 | |
| Neuron balance | NorMuon row normalization | 4.1854 | −0.0189 | −0.0188 [−0.0193, −0.0183] |
| Neuron balance | Row normalization without EMA | 4.1852 | −0.0192 | |
| Spectral shape | Cooldown tilt (exact) | 4.1910 | −0.0133 | |
| Spectral shape | Cooldown tilt (exact) | 4.1888 | −0.0155 | |
| Spectral shape | Tilt from step 1,024 | 4.1896 | −0.0147 | |
| Spectral shape | Cooldown band filter, from step 1,024 | 4.1927 | −0.0116 | |
| Spectral shape | Accurate polar factor (NS5 + 3 cubic steps) | 4.2023 | −0.0020 | |
| Spectral shape | Square blocks + cooldown tilt (Chebyshev, monomial/TF32; faulty) | 4.1804 | −0.0240 | |
| Spectral shape | Square blocks + cooldown tilt (Chebyshev, Clenshaw/FP32) | 4.1721 | −0.0322 | |
| Spectral shape | Square blocks + cooldown tilt (exact) | 4.1711 | −0.0332 | |
| Decay geometry | Schatten-1 decay (bulk shrinkage) | 4.1991 | −0.0052 | −0.0057 [−0.0100, −0.0014] |
| Decay geometry | Schatten-4 decay (top-heavy) | 4.2220 | +0.0177 | |
| Data metric (whitening) | Input whitening , norm-matched | 4.2254 | +0.0211 | |
| Data metric (whitening) | Input whitening , norm-matched | 4.2642 | +0.0599 | |
| Data metric (whitening) | Input whitening , , norm-matched | 4.2474 | +0.0431 | |
| Data metric (whitening) | Input whitening , data-metric magnitude | 4.2567 | +0.0524 | |
| Data metric (whitening) | Input whitening + shaped decay | 4.2638 | +0.0595 | |
| Data metric (whitening) | Toner recipe + shaped decay | 4.1975 | −0.0068 | |
| Combinations | Tilt + row normalization (no EMA) | 4.1737 | −0.0306 | |
| Combinations | Band filter (step-1,024 multipliers) + row normalization (no EMA) | 4.1742 | −0.0302 | |
| Combinations | Square blocks + NorMuon | 4.1780 | −0.0263 | |
| Combinations | Square blocks + NorMuon + cooldown tilt (exact) | 4.1661 | −0.0382 | |
| Combinations | Square blocks + NorMuon + cooldown tilt | 4.1665 | −0.0378 | |
| Combinations | ... + Schatten-1 decay | 4.1648 | −0.0396 | |
| Combinations | Split QKV + cooldown tilt | 4.1667 | −0.0376 | |
| Combinations | Split QKV + NorMuon | 4.1722 | −0.0321 | |
| Combinations | Split QKV + NorMuon + cooldown tilt | 4.1632 | −0.0411 | −0.0436 [−0.0511, −0.0362] |
| Combinations | Split QKV + NorMuon + tilt from step 1,024 | 4.1625 | −0.0418 | |
| Combinations | Split QKV + NorMuon + cooldown tilt | 4.1641 | −0.0402 | |
| Combinations | Split QKV + NorMuon + cooldown tilt + Schatten-1 decay | 4.1581 | −0.0462 | −0.0487 [−0.0555, −0.0420] |
Table 7. Screen results. Final held-out loss (4.18M tokens) at seed 42 and paired differences from fused-QKV Muon.
Figure 6. Screen arms sorted by their mean paired difference from fused-QKV Muon (seed 42, or the three-seed mean with 95% -interval and open circles for seeds where three seeds exist).
Learning-rate robustness of block-wise steps. With square blocks, raising the learning rate from 0.005 to 0.007 and 0.01 gives final losses of 4.1840 and 4.2045. Muon at the same rates gives 4.2077 and 4.2376 (+0.0034 and +0.0332 relative to its optimum). Doubling the learning rate leaves square-block Muon level with Muon’s tuned optimum. MuonBP reported that block-only training needed smaller learning rates at scale ([22]); in our setting block-wise steps are, if anything, more tolerant of large rates.
Implementation details
Norm matching. For every matrix and step, the candidate direction is rescaled to , with the same momentum Muon would use, and then enters (1) with Muon’s shape factor and learning rate. Norm-matched whitening arms are matched to , the norm of their own whitened polar factor. The trainer enables TF32 matrix products globally, so NS runs in FP32 storage with TF32 products; only the Chebyshev filter disables TF32. Matching costs one extra joint NS per matrix for block-wise arms, used only to read its norm. The deployable square-block step omits it: it scales wide-matrix column blocks by and runs all 144 square blocks in one batched NS call. It gives −0.0200 on the screen, against −0.0219 norm-matched.
NorMuon. Following Li et al. [27], the orthogonalized update is divided row-wise by , where is an EMA () of the row mean of . It is then norm-matched. A stateless variant with gives −0.0192, against −0.0189 for the EMA version (screen), so the EMA is not what helps.
Spectral tilt. The exact tilt maps singular value of to via an eigendecomposition of . With it reproduces NS to relative error. The matmul-only tilt (Algorithm 1) uses a degree-12 Chebyshev least-squares fit of on (4,000 Chebyshev nodes) in the variable . It is evaluated by Clenshaw’s recurrence with TF32 disabled. is symmetrized and scaled by times a 20-step power-iteration estimate of . That is not a proved upper bound: eigenvalues above , or below , fall outside the fitted interval, where the Chebyshev polynomial is not controlled. By Theorem A.5, the implemented filter is rather than the exact ; on the Newton–Schulz plateau they differ by a factor in . On real momenta the matmul tilt matches the exact tilt to direction cosine –; enabling TF32 lowers the dot product to . By Theorem A.6(ii) the normalized steps differ by at most 6.3% in Frobenius norm. Turning that into a loss bound would need a curvature bound along the difference (Theorem A.24), which we do not have. When Q, K and V are split, the three blocks of all 12 layers and the square attention-output matrices are filtered in one batched chain. The MLP matrices are filtered whole. With TF32 off the tilt costs about 0.17 s per step on an A6000, during the cooldown only.
Input whitening. is an EMA of the uncentered input second moment, trace-normalized with damping, ( unless stated). It is refreshed from 4,096-token activation samples. The step is polar : and norm-matched, at its own magnitude, and the Toner recipe with shaped decay .
Schatten- decay. We replace by . This removes the same Frobenius energy per step as uniform decay, with relative rate on singular value . For we use for the polar factor.
Cost. Measured seconds per step on one RTX A6000 (100.7M-token runs): Muon 0.343; NorMuon 0.329; norm-matched square blocks 0.415; square blocks + NorMuon + tilt 0.418 before and 0.588 during the cooldown. Mean wall time per 100.7M-token run, including evaluations: Muon 23.3 min, split-QKV 24.9, split-QKV + tilt 26.4, split-QKV + NorMuon 25.1, split-QKV + NorMuon + tilt 27.8. Three runs shared a host, so these timings are indicative.
Reproducibility and provenance
Logging. Each run writes an event log with its recipe, optimizer variant, seed, software versions and a hash of the optimizer source. The results tables, figures and run-derived numbers in the results text are generated automatically from these logs. A few descriptive quantities (per-step timings, curvature ratios, some leverage-table entries and filter cosines) were transcribed and checked in an independent pass against the logs. Training uses torch.compile and BF16 autocast for the forward pass; the optimizer state is FP32, and Newton–Schulz runs in FP32 with TF32 matrix products. The trainer reproduces an earlier, slower harness’s Muon loss (3.9445 against 3.9463 at seed 42).
Order of experiments. The screen arms, the diagnostic and the first confirmations ran in parallel on three GPUs over one day. The anatomy was not pre-registered: the square-block, whitening and accurate-polar screens started before the curvature diagnostics, and the neuron diagnostic postdated the first block arms. The square-block arm was motivated by the leverage analysis; the attribution runs later showed that the QKV split carries its gain. Two confirmations, square blocks + NorMuon + tilt and split QKV + NorMuon + tilt, were launched before their own screens had finished. The 355M seed-42 pair was launched after the positive 124M confirmation. The gating rule of Section 6 was adopted after that; it released the full-length factorial ablations (split QKV + tilt; split QKV + NorMuon), whose screens beat the baseline by 0.0121 and
Algorithm 1 One step for a hidden matrix: split-QKV Muon with NorMuon row normalization and the cooldown tilt (cooldown is true in the last 25% of steps).
;
Split into blocks (Q, K, V for the attention input matrix; one block otherwise)
for each block do
if cooldown then
; (20 power iters)
(FP32)
else
end if
end for
; ;
(norm match)
0.0066, and the 355M seeds 43–44.
Source versions. The optimizer file changed during the study. The original fused-QKV Muon, NorMuon and square-block confirmations span four source hashes, and a read-only audit of the raw logs could not certify, from the retained files, that the changes touched only inactive branches. All split-QKV arms (split-QKV Muon, the factorial and the 355M runs) ran on a single source version, so the factorial comparisons are unaffected. Every comparison involving fused-QKV Muon, NorMuon on fused QKV, square blocks or square blocks + NorMuon + tilt crosses source versions; this includes the attribution results “square blocks vs. split QKV” and “NorMuon vs. split QKV”. The A/A bound and the reproduction of the earlier harness’s loss suggest the effect is small, but we cannot certify it.
Measurements needed to instantiate the certificate. Experiments were paused before these could run. Each uses saved checkpoints and no training: (1) Sampling intervals: record per-sequence held-out losses for every step in Table 4 and bootstrap paired differences over sequences, across several checkpoint seeds. (2) Estimation errors : recompute and on disjoint folds and take the spread of and of over folds. (3) Third-derivative bound : evaluate the loss at along each step and take third finite differences. (4) Gradient bound : the held-out gradient norm at both endpoints and the midpoint of each step pair.
Statistics. Intervals are two-sided 95% Student- intervals over three paired seeds (2 degrees of freedom). They are nominal: the same seeds were used in the screen, and no correction is made for the number of arms examined. The A/A bound (0.009 nats) uses 98k-token intermediate evaluations; final evaluations use 4.18M tokens and are less noisy.
References
- [1]Zhehang Du and Weijie Su. The Newton-Muon optimizer. arXiv preprint arXiv:2604.01472, 2026.arxiv.org/abs/2604.01472
- [2]Noah Amsel, David Persson, Christopher Musco, and Robert M. Gower. The polar express: Optimal matrix sign methods and their application to the Muon algorithm. In International Conference on Learning Representations (ICLR), 2026. arXiv:2505.16932.arxiv.org/abs/2505.16932
- [3]Dmitrii Andrianov, Andrey Veprekov, and Aleksandr Beznosikov. Spectral weight decay: Inducing low-rank structure in neural network weights. arXiv preprint arXiv:2610.11730, 2026.
- [4]Atilim Gunes Baydin, Robert Cornish, David Martinez Rubio, Mark Schmidt, and Frank Wood. Online learning rate adaptation with hypergradient descent. In International Conference on Learning Representations (ICLR), 2018.arxiv.org/abs/1703.04782
- [5]Jeremy Bernstein and Laker Newhouse. Old optimizer, new norm: An anthology. arXiv preprint arXiv:2409.20325, 2024.arxiv.org/abs/2409.20325
- [6]Meher Chaitanya, Tianyi Zhou, and Aristides Gionis. Does Muon need fine-grained spectral shaping? arXiv preprint arXiv:2610.07497, 2026.arxiv.org/abs/2610.07497
- [7]Tong Che. Second-order Muon done right: A principled marriage of spectral geometry and curvature. arXiv preprint arXiv:2608.09763, 2026.arxiv.org/abs/2608.09763
- [8]Lizhang Chen, Jonathan Li, and Qiang Liu. Muon optimizes under spectral norm constraints. Transactions on Machine Learning Research, 2026. ISSN 2835-8856. URL https://openreview.net/forum?id=BLz4hjxIWu.arxiv.org/abs/2506.15054
- [9]C. W. Clenshaw. A note on the summation of Chebyshev series. Mathematical Tables and Other Aids to Computation, 9(51):118–120, 1955. doi: 10.1090/S0025-5718-1955-0071856-0.DOI
- [10]Jeremy M. Cohen, Simran Kaur, Yuanzhi Li, J. Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. In International Conference on Learning Representations (ICLR), 2021.
- [11]Alec Dewulf, Dhruv Pai, Li Yang, Ashley Zhang, and Ben Keigwin. Aurora: A leverage-aware spectral optimizer. arXiv preprint arXiv:2606.27715, 2026.arxiv.org/abs/2606.27715
- [12]Petros Drineas, Malik Magdon-Ismail, Michael W. Mahoney, and David P. Woodruff. Fast approximation of matrix coherence and statistical leverage. Journal of Machine Learning Research, 13(111):3475–3506, 2012.arxiv.org/abs/1109.3843
- [13]Essential AI, Ishaan Shah, et al. Practical efficiency of Muon for pretraining. arXiv preprint arXiv:2505.02222, 2025.
- [14]Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via Hessian eigenvalue density. In International Conference on Machine Learning (ICML), volume 97 of Proceedings of Machine Learning Research, pages 2232–2241, 2019.arxiv.org/abs/1901.10159
- [15]Vishrut Goyal and Rohan Ramkumar. Spectrally targeted Muon. arXiv preprint arXiv:2610.10965, 2026.arxiv.org/abs/2610.10965
- [16]Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization. In International Conference on Machine Learning (ICML), volume 80 of Proceedings of Machine Learning Research, pages 1842–1850, 2018.arxiv.org/abs/1802.09568
- [17]Guy Gur-Ari, Daniel A. Roberts, and Ethan Dyer. Gradient descent happens in a tiny subspace. arXiv preprint arXiv:1812.04754, 2018.
- [18]Alexander Hägele, Elie Bakouch, Atli Kosson, Loubna Ben Allal, Leandro Von Werra, and Martin Jaggi. Scaling laws and compute-optimal training beyond fixed training durations. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, 2024.arxiv.org/abs/2405.18392
- [19]Kai Hu, Li Zhang, Weichen Yu, Matt Fredrikson, and Haoqi Hu. Do orthogonality constraints help language-model training beyond sphere optimization? arXiv preprint arXiv:2611.00593, 2026.arxiv.org/abs/2611.00593
- [20]Keller Jordan et al. modded-nanogpt: Speedrunning the NanoGPT baseline. GitHub repository, https://github.com/KellerJordan/modded-nanogpt, 2024a.
- [21]Keller Jordan et al. Muon: An optimizer for hidden layers in neural networks. Blog post, https://kellerjordan.github.io/posts/muon/, December 2024b.
- [22]Ahmed Khaled, Kaan Ozkara, Tao Yu, Mingyi Hong, and Youngsuk Park. MuonBP: Faster Muon via block-periodic orthogonalization. In International Conference on Learning Representations (ICLR), 2026. arXiv:2510.16981.arxiv.org/abs/2510.16981
- [23]Dmitry Kovalev. Understanding gradient orthogonalization for deep learning via non-Euclidean trust-region optimization. arXiv preprint arXiv:2503.12645, 2025.arxiv.org/abs/2503.12645
- [24]Tim Large, Yang Liu, Minyoung Huh, Hyojin Bahng, Phillip Isola, and Jeremy Bernstein. Scalable optimization in the modular norm. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, 2024. doi: 10.52202/079017-2338.DOI
- [25]Tim Tsz-Kit Lau, Qi Long, and Weijie Su. PolarGrad: A class of matrix-gradient optimizers from a unifying preconditioning perspective. arXiv preprint arXiv:2505.21799, 2025.arxiv.org/abs/2505.21799
- [26]Mingyu Li. Low-rank decay for grokking in scale-invariant transformers: A spectral-geometric view. arXiv preprint arXiv:2606.04405, 2026.arxiv.org/abs/2606.04405
- [27]Zichong Li, Liming Liu, Chen Liang, Weizhu Chen, and Tuo Zhao. NorMuon: Making Muon more efficient and scalable. In International Conference on Machine Learning (ICML), 2026. arXiv:2510.05491.arxiv.org/abs/2510.05491
- [28]Jingyuan Liu et al. Muon is scalable for LLM training. arXiv preprint arXiv:2502.16982, 2025.arxiv.org/abs/2502.16982
- [29]Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019.
- [30]James Martens and Roger Grosse. Optimizing neural networks with Kronecker-factored approximate curvature. In International Conference on Machine Learning (ICML), volume 37 of Proceedings of Machine Learning Research, pages 2408–2417, 2015.arxiv.org/abs/1503.05671
- [31]M. Mohamadi, K. Wen, T. Wang, and Z. Li. Toner / MuonT: input-whitened Muon. Blog post and code, November 2026.
- [32]Vardan Papyan. Traces of class/cross-class structure pervade deep learning spectra. Journal of Machine Learning Research, 21(252):1–64, 2020.arxiv.org/abs/2008.11865
- [33]Barak A. Pearlmutter. Fast exact multiplication by the Hessian. Neural Computation, 6(1):147–160, 1994. doi: 10.1162/neco.1994.6.1.147.DOI
- [34]Guilherme Penedo et al. The FineWeb datasets: Decanting the web for the finest text data at scale. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, volume 37, 2024. doi: 10.52202/079017-0970.DOI
- [35]Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos, Zhenyu Zhu, Antonio Silvetti-Falls, and Volkan Cevher. Training deep learning models with norm-constrained LMOs. In International Conference on Machine Learning (ICML), volume 267 of Proceedings of Machine Learning Research, pages 49069–49104, 2025.arxiv.org/abs/2502.07529
- [36]Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. Technical report, OpenAI, 2019.
- [37]Artem Riabinin, Egor Shulgin, Kaja Gruntkowska, and Peter Richtárik. Gluon: Making Muon & Scion great again! (Bridging theory and practice of LMO-based optimizers for LLMs). arXiv preprint arXiv:2505.13416, 2025.arxiv.org/abs/2505.13416
- [38]Levent Sagun, Utku Evci, V. Ugur Guney, Yann Dauphin, and Léon Bottou. Empirical analysis of the Hessian of over-parametrized neural networks. arXiv preprint arXiv:1706.04545, 2017.
- [39]Daniel Selsam, Percy Liang, and David L. Dill. Developing bug-free machine learning systems with formal mathematics. In International Conference on Machine Learning (ICML), volume 70 of Proceedings of Machine Learning Research, pages 3047–3056, 2017.arxiv.org/abs/1706.08605
- [40]Wei Shen, Ruichuan Huang, Minhui Huang, Cong Shen, and Jiawei Zhang. On the convergence analysis of Muon. Transactions on Machine Learning Research, 2026. ISSN 2835-8856. URL https://openreview.net/forum?id=4nH4CulGaP.arxiv.org/abs/2505.23737
- [41]Egor Shulgin, Sam Laing, Antonio Orvieto, and Peter Richtárik. A quadratic lens on Muon’s polar update: Benefits, pitfalls, and the role of Newton-Schulz. arXiv preprint arXiv:2610.30280, 2026.arxiv.org/abs/2610.30280
- [42]Zakhar Shumaylov et al. Muon is not that special: Random or inverted spectra work just as well. arXiv preprint arXiv:2605.11181, 2026.arxiv.org/abs/2605.11181
- [43]Chongjie Si, Debing Zhang, and Wei Shen. AdaMuon: Adaptive Muon optimizer. arXiv preprint arXiv:2507.11005, 2025.arxiv.org/abs/2507.11005
- [44]Ziyuan Tang, Tianshi Xu, Yousef Saad, and Yuanzhe Xi. Hierarchical Muon: Tiled Newton-Schulz updates for efficient Muon optimization. arXiv preprint arXiv:2606.27216, 2026.arxiv.org/abs/2606.27216
- [45]Lloyd N. Trefethen. Approximation Theory and Approximation Practice, Extended Edition. Society for Industrial and Applied Mathematics (SIAM), 2019. doi: 10.1137/1.9781611975949.DOI
- [46]Nikhil Vyas et al. SOAP: Improving and stabilizing Shampoo using Adam for language modeling. In International Conference on Learning Representations (ICLR), 2025. arXiv:2409.11321.arxiv.org/abs/2409.11321
- [47]Shuche Wang, Fengzhou Zhang, Jiaxiang Li, Dirk Bergemann, and Zhuoran Yang. Why Muon outperforms Adam: A curvature perspective. arXiv preprint arXiv:2606.04662, 2026a.arxiv.org/abs/2606.04662
- [48]Shuche Wang et al. Muon outperforms Adam in tail-end associative memory learning. In International Conference on Learning Representations (ICLR), 2026b. arXiv:2509.26030.arxiv.org/abs/2509.26030
- [49]Kaiyue Wen, David Hall, Tengyu Ma, and Percy Jiang. Fantastic pretraining optimizers and where to find them. In International Conference on Learning Representations (ICLR), 2026. arXiv:2509.02046.arxiv.org/abs/2509.02046
- [50]Fangzhou Wu, Rikhav Shah, Sandeep Silwal, and Qiuyi Zhang. DynMuon: A dynamic spectral shaping view of Muon. arXiv preprint arXiv:2605.17109, 2026a. ICML 2026 Workshop on High-dimensional Learning Dynamics (HiLD).arxiv.org/abs/2605.17109
- [51]Xiaodong Wu, Wenyi Yu, Chao Zhang, and Philip Woodland. Spectral allocation: Why Muon outperforms Adam, and how to improve Muon. arXiv preprint arXiv:2608.25990, 2026b.arxiv.org/abs/2608.25990
- [52]Yuhuai Wu, Mengye Ren, Renjie Liao, and Roger Grosse. Understanding short-horizon bias in stochastic meta-optimization. In International Conference on Learning Representations (ICLR), 2018.arxiv.org/abs/1803.02021
- [53]Chenrui Xu, Wenjing Yan, and Ying-Jun Angela Zhang. FISMO: Fisher-structured momentum-orthogonalized optimizer. arXiv preprint arXiv:2601.21750, 2026.arxiv.org/abs/2601.21750
- [54]Ruijie Zhang et al. MUON+: Towards more effective Muon via one additional normalization step for LLM pre-training. arXiv preprint arXiv:2602.21545, 2026a.arxiv.org/abs/2602.21545
- [55]Yechen Zhang et al. Mousse: Rectifying the geometry of Muon with curvature-aware preconditioning. arXiv preprint arXiv:2603.09697, 2026b.arxiv.org/abs/2603.09697
- [56]Rosie Zhao, Depen Morwani, David Brandfonbrener, Nikhil Vyas, and Sham Kakade. Deconstructing what makes a good optimizer for language models. In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2407.07972



