WorksComputer Science
Factored One-sided Adam-Moment for Practical and Scalable SOAP
Abstract
Second-order optimizers (e.g., Shampoo, SOAP) have emerged as compelling alternatives to Adam/AdamW and promise faster per-step convergence than AdamW in transformer training. However, they still remain costly in practice, and even SOAP still pays substantial optimizer-state memory and eigendecomposition overhead. We ask whether the benefits of SOAP can be retained at nearly the efficiency of AdamW. In this paper, we answer yes with a lightweight SOAP variant, Factored One-sided Adam-Moment (FOAM). The path to practical impact is twofold: first, FOAM employs input-only preconditioning, dropping the output-side Kronecker factor to reduce memory and remove one eigendecomposition. The Kronecker structure of the Hessian shows that this is exact for a diagonal output-side curvature and nearly lossless for a near-diagonal one, as we observe empirically. Second, FOAM replaces SOAP's dense rotated second moment with a factored Adam-moment estimator based on row-by-column statistics. Under the K-FAC Fisher assumption with a diagonal output curvature, this estimator exactly recovers SOAP's dense second moment, and we measure only a 15-20% K-FAC violation, with negligible effect on the update. Our empirical results support the claim that FOAM matches strong SOAP and Muon baselines in validation loss across scales of large language model (LLM) pre-training, uses half of SOAP's optimizer state, runs at Muon-class wall-clock time, and retains robustness across a three-fold learning-rate range. The results hold from 124M to 760M parameters, demonstrating that SOAP's gains need not require SOAP's cost. Code: https://github.com/naver-ai/foam
Introduction
Adam [23] and AdamW [32] have long been the default for pre-training transformer architectures, including large language models (LLMs). Their successors, including memory-efficient adaptive optimizers [50, 51, 53, 15], show that comparable optimization quality can be achieved with substantially less optimizer memory. In another direction, Shampoo [16], a foundational method in the family of Kronecker-factored second-order optimizers, has shown that the optimization quality of adaptive first-order methods such as Adam can be surpassed. Its most practical variant, SOAP [44], the Adam-in-eigenbasis version of Shampoo, reports 35–40% improvements in wall-clock time and step count over AdamW at large batch sizes, building upon the Distributed Shampoo baselines [3, 40] that won the external-tuning track of the first AlgoPerf competition [6, 21].
Still, practitioners often default to AdamW or Muon [20], because SOAP achieves its gains at the expense of computational efficiency, requiring about the optimizer state of Adam and a significantly slower per-step wall-clock time. More specifically, SOAP’s memory is dominated by the first moment, the second moment in the eigenbasis-rotated space, and the two Kronecker factor matrices () and () of a gradient . Its runtime is further occupied by the periodic eigendecomposition of these factor matrices, which makes SOAP substantially more expensive than AdamW despite its strong optimization performance.
This paper aims to retain SOAP-level quality while approaching AdamW-level cost. Our motivation is simple. If SOAP’s computational overhead can be sufficiently reduced, its optimization advantages could make it a practical successor to AdamW. To this end, we revisit SOAP’s major cost components and show that several can be relaxed or replaced without sacrificing performance. The original SOAP paper [44] already evaluated one-sided and factored variants, but as cost-reducing ablations without a criterion for when they preserve quality. Frans et al. [12] then found that input-only rotation and a factored variance each retain most of SOAP’s quality, and other works [30, 11] improved SOAP along other axes. Our contribution is thus not one-sided preconditioning or factorization itself, but explaining which side to retain, when factorization is justified, and why their combination is effective.
Concretely, our approach is built on two structural simplifications: input-only preconditioning and a factored Adam-Moment update. The first simplification drops the output-side factor and retains only the input-side factor , eliminating its memory and one of the two eigendecompositions. Its justification follows from the Kronecker structure of the Hessian, in which all output rows of an affine layer share the same input-side outer product [25], while the Hessian of transformer layers is empirically near-block-diagonal along output neurons or heads [50], making the input-only form the natural one-sided approximation. The second simplification replaces the dense rotated second moment with a row-and-column factored estimator, and , with trace normalization, reducing the buffer size from to . Under the K-FAC Fisher assumption [35] with a diagonal output curvature, this estimator exactly recovers SOAP’s dense second moment, and the K-FAC part holds to a 15–20% relative error in practice. Together, these simplifications reduce SOAP’s optimizer state from to and the amortized eigendecomposition count from two to one. We call the resulting method FOAM (Factored One-sided Adam-Moment).
Figure 1 highlights FOAM’s main advantages. On pre-training, FOAM matches the strong Muon and SOAP baselines on validation loss with half of SOAP’s optimizer state and reaches AdamW’s final loss within 7% of AdamW’s own wall-clock time, while SOAP needs more than twice as long at 760M, and FOAM matches AdamW’s hyperparameter robustness over a three-fold learning-rate range. The advantage holds from 124M to 760M parameters with learning rates re-tuned at every size from a μP [46] anchor. In summary, our contributions are: (i) a structural justification of input-only preconditioning, with the Kronecker structure of the Hessian and empirical verifications with LLMs with recent architectural elements; (ii) a factored Adam-Moment update with trace normalization, exact under K-FAC with a diagonal output curvature and empirically validated through K-FAC-residual measurements during training; and (iii) the resulting algorithm FOAM, a scalable and practical SOAP variant that delivers SOAP-class quality with half of SOAP’s optimizer state (+7.4% measured peak training memory over AdamW) and Muon-class wall-clock time.
Figure 1. FOAM is a Pareto-efficient second-order optimizer. GPT-2 large (760M), 29.55K steps, each optimizer at its tuned learning rate. Curves are 3-seed means smoothed over 1K steps, and grey boxes mark the regions magnified in the insets. (a) Per-step. Validation loss against training step. In the last 4K steps (inset) SOAP ends slightly below FOAM (2.791 vs. 2.796, within about one seed std), both below Muon, while AdamW stays above the inset range. (b) Wall-clock. Same runs vs. wall time, normalized so AdamW finishes at 1.0× (Muon, FOAM, and SOAP finish at 1.15×, 1.17×, and 2.31×). Within AdamW’s own budget, FOAM is already 0.04 nats below AdamW’s final loss (inset), and it reaches SOAP’s quality band at 51% of SOAP’s wall-clock time. (c) Time to AdamW’s final loss (2.847; Table 2). FOAM gets there as fast as AdamW itself (10.53 vs. 10.55 h) and slightly ahead of Muon, while bilateral SOAP needs 2.15× as long.
Related Work
Adaptive optimizers can be categorized by how aggressively they approximate the empirical Fisher , which all of Adam [23]/AdamW [32], Shampoo [16], and SOAP [44] implicitly use as a curvature surrogate in place of the true Fisher [24].
Empirical Fisher and Kronecker-factored Optimizers
AdaGrad [8] introduced per-coordinate scaling by an empirical second-moment statistic, an early diagonal approximation of the inverse empirical Fisher. K-FAC [35] replaces the diagonal with a Kronecker product, where is the layer input and the gradient with respect to the layer’s pre-activation output, the assumption under which our factored Adam-Moment update is exact. Eschenhagen et al. [10] adapt K-FAC to transformer architectures via the expand and reduce flavors. Shampoo [16] carries this idea to the matrix parameters. It maintains EMAs of and and steps in , i.e., it preconditions by ; Morwani et al. [36] show that equals one power-iteration step toward the optimal Kronecker-product approximation of the empirical Fisher and is empirically close to it. SOAP [44] eigendecomposes and and runs Adam in the resulting rotated coordinates, an EKFAC-style construction [14] that Lu et al. [33] interpret as gradient whitening. Variants modify specific pieces: CASPR [9] combines the per-axis factors through a Kronecker sum instead of a Kronecker product. KL-Shampoo and KL-SOAP [30] recast the family via KL-divergence minimization, and Purifying Shampoo [11] shows that trace scaling of the preconditioner can replace learning-rate grafting, which corroborates our trace-norm choice in § 4.2. The one-sided fallback in distributed Shampoo [3] drops a factor on memory grounds, whereas we give a structural argument for keeping the input side (§ 4.1). Purifying Shampoo’s adaptive tolerance criterion for skipping the periodic eigendecomposition is adopted unchanged as FOAM’s amortized eigenbasis-refresh policy (§ 4.2).
Optimizers with Coarser Approximations
A second line of work adopts coarser approximations to the empirical Fisher in exchange for lower computational cost. Adam [23, 32] keeps only the diagonal of , sacrificing all matrix structure. Signum [4] keeps only the sign of the momentum. Muon [20] replaces the second-moment buffer with Newton-Schulz orthogonalization , which can be read as a square-root natural-gradient step under a structured Fisher with an input-side factor shared across output rows [34, 15]. AdaMuon [41] adds an element-wise Adam-style second moment on top of the orthogonalized update. SPlus [13], a related SOAP variant, replaces the rotated second moment with sign normalization in the Shampoo eigenbasis and adds iterate averaging. Recent Muon variants (NorMuon [28], Muon [31], NAMO [48], Polar Express [2]) refine specific components of Muon. A parallel branch targets memory-efficient AdamW rather than matching SOAP-class quality: Adam-mini [50], GaLore [51], APOLLO [53], Alice [15], and SWAN [34]. We exclude these methods from the main comparison since they mainly target a smaller memory footprint than AdamW, whereas we target SOAP-level performance at near-Adam cost.
Positioning our method. FOAM lies at the intersection of these two branches. It stays within the Kronecker-factored Fisher framework but borrows the factored-second-moment idea from Adafactor [39]’s row-by-column estimator, applied here in the rotated coordinates of SOAP. Both ingredients appear in the SOAP paper as cost-reducing ablations, a one-sided variant and an Adafactor-style factored second moment in the eigenbasis, and Frans et al. [12] found empirically that input-only rotation and a factored variant each preserve most of SOAP’s quality; the distinction lies in the criteria. SOAP’s one-sided variant keeps the side with the smaller dimension, whereas we keep the input side irrespective of size (§ 4.1), so the two rules choose opposite sides for the MLP down-projection; likewise, we factorize only after the input-side rotation, which makes the rotated second moment separable into row and column factors (Lemma 1, § 4.2).
Preliminary: Shampoo and SOAP
We consider a transformer with matrix parameters , where and denote the input and output dimensions, respectively. Let denote the mini-batch gradient at step . Following the convention of SOAP [44], we define the two Kronecker factors and as the second-moment matrices of the gradient:
acts on the left side (output) of , and acts on the right side (input). In practice, both are estimated by exponential moving averages (EMAs), for which we reuse the same symbols, and we reserve the term estimator for such EMA buffers. Shampoo [16] takes a fractional inverse of each factor and applies the preconditioned update:
with learning rate . SOAP [44] instead eigendecomposes and to obtain orthogonal rotations and , and runs Adam in the rotated coordinates:
following the EKFAC tradition [14] that tracks diagonal second moments in the eigenbasis of a Kronecker-factored preconditioner. The per-step update maintains a dense second moment in the rotated space. The rotations and are refreshed every (precondition frequency) steps via two eigendecompositions of and . and require memory, the rotated second moment requires an additional memory, and the periodic eigendecompositions incur further computational overhead. These costs constitute the overhead structure that FOAM addresses.
Factored One-sided Adam-Moment (FOAM)
We propose FOAM, Factored One-sided Adam-Moment, which combines two structural simplifications of SOAP into a single optimizer. FOAM applies an input-side K-FAC factor combined with a factored output diagonal. It maintains only the input-side factor , runs Adam in the eigenbasis , and replaces SOAP’s dense rotated second moment with a row-column factored estimator . Here and are bias-corrected EMAs of the row and column means of the squared rotated gradient, and is a trace-matching scalar. The two simplifications together give FOAM the implicit Fisher , a full input-side factor with a diagonal output side, derived in Algorithm 1 and the following sections. Figure 2 compares this preconditioner structure with those of Adam, Muon, and SOAP against the near-block-diagonal Hessian of a transformer layer, where FOAM keeps SOAP’s input factor with a diagonal output side.
Figure 2. Hessian and its approximation across optimizers. For with , , we visualize the Hessian alongside the preconditioner of each optimizer; shaded cells are non-zero and darker means larger; coordinates are grouped by output row ( blocks), while labels follow the column-major convention of Lemma 1. The Hessian is near-block-diagonal. Adam (): fully diagonal approximation with no Kronecker structure. Muon ((): identical outer-diagonal blocks, built statelessly from the current momentum via . SOAP (): full Kronecker fit with EMAs on both sides, with all outer blocks populated. FOAM (, ours): replaces Muon’s identity output with a per-block diagonal scaling, and its stateless input factor with the EMA of .
Derivation overview. The steps below form a single chain from , , to the two stored vectors. (i) Under K-FAC, , and is near-diagonal (§ 4.1), so the off-diagonal structure lives on the input side, which is the side to keep. (ii) Rotating by turns the covariance into (Lemma 1). Like SOAP running Adam in its eigenbasis, FOAM keeps only the diagonal of this covariance, a modeling choice that the rotation makes accurate. (iii) This diagonal, reshaped to , is the rank-one matrix , so storing and keeps every coordinate’s own variance and loses nothing relative to the dense buffer, as formalized in § 4.2.
Algorithm 1 FOAM update at step for . is the all-ones vector in , and , , and act element-wise. is the off-diagonal ratio of a matrix (§ 4.2). is initialized to the identity, and decoupled weight decay is applied as in AdamW and omitted here.
Require: gradient ; hyperparameters ; state , , , ,
(input-side factor)
if and then
(refresh the eigenbasis)
end if
(first moment)
(rotate into the eigenbasis)
(factored second moment)
(bias correction)
with (rank-one second moment at ’s scale)
(Adam step in the eigenbasis)
(rotate back and update)
Input-only Preconditioning
Our idea is to remove the output-side factor and maintain only with its eigenbasis . The gradient rotation is applied only to the input axis, i.e., . This reduces the factor-state memory from to and the number of eigendecompositions from two to one, halving this cost for square matrices.
A conceptual justification comes from the curvature of an affine layer with cross-entropy loss, the standard loss for language modeling and classification. For and , the Hessian of the loss for a target class with respect to decomposes exactly as
The output-side factor has diagonal entries and off-diagonal entries . Each off-diagonal entry is smaller than the diagonal entry by a factor of , which is small for most classes of a large vocabulary even far from a one-hot prediction, in line with the dominant per-class diagonal blocks of softmax linear models [25]. Larger entries arise only where a class takes much of the mass, and each pair receives them only from examples in which or is such a class, whereas the diagonal accumulates over all examples, so with a large vocabulary the expected output-side curvature is dominated by its diagonal, and under the K-FAC independence approximation the batch expectation becomes
This has the structure of FOAM’s implicit Fisher, with the input-side curvature tracked by and the per-output curvature scale absorbed into . The output side need not be homogeneous across rows. FOAM only requires that it is dominated by its diagonal, which this mechanism suggests and the Gauss-Newton measurements of § 5.4 support empirically.
Beyond the affine setting. For a hidden affine layer , the per-example Hessian keeps the same form , since is linear in , and only the output-side factor changes, from to the curvature with respect to . The vanishing-off-diagonal mechanism is specific to the output layer, but near-block-diagonal Hessians have been observed in MLPs throughout training [5, 50] and, with blocks along output neurons or heads, in a small transformer [50]. We provide additional indirect evidence specific to our setting through Gauss-Newton alignment measurements in § 5.4.
Factored Eigenvalue Under Input-only Preconditioning
Even with the input-only simplification, SOAP still has overheads due to the dense rotated second-moment buffer . Our next step is to replace it with two factored estimators and , EMAs of the row- and column-means of , and reconstruct the rotated second moment as an outer product with trace normalization:
The state shrinks from to per matrix parameter. The trace-matching scalar makes the entries of sum to , fixing the overall scale of the reconstructed second moment to that of the input-side factor. We now justify why this rank-one factorization is sufficient.
Lemma 1 (Covariance of the input-rotated gradient). Let and assume the K-FAC factorization . Let be the input-rotated gradient in column-major form. Then
where is the diagonal eigenvalue matrix of .
Proof. Using the column-major identity with and , we have . Therefore,
The Kronecker mixed-product identity , applied twice, gives
Lemma 1 tells us two things about the rotated gradient. First, the input-side covariance becomes diagonal, because rotating by decorrelates the input directions and replaces with its eigenvalues . Second, the output-side factor is unchanged, so when is diagonal (§ 4.1) the rotated covariance is itself diagonal, and its diagonal, the part FOAM keeps, factorizes as
with the -th diagonal entry of . The right-hand side is a per-row factor depending only on times a per-column factor depending only on , so the entire second-moment surface is captured by numbers rather than . The row- and column-means of are the natural estimators of these two factors, up to a global scalar that the trace normalization removes, so recovers the dense buffer exactly under K-FAC, and SOAP’s two-sided one when is also diagonal. In our runs, the K-FAC assumption is violated by a 15–20% relative error, mostly at the key projection, yet this costs only about 0.002 nats in final validation loss (Table 3, Appendix § 9).
| input | output | factored | val loss | state (GiB) | wall (h) |
| ✓ | ✓ | 2.92 | 0.48 | ||
| ✓ | ✓ | ✓ | 2.59 | 0.49 | |
| ✓ | 1.78 | 0.44 | |||
| ✓ | 1.78 | 0.46 | |||
| ✓ | ✓ | 1.46 | 0.46 |
Table 3. FOAM-construction ablation on GPT-2 small (124M). Starting from SOAP, input and output mark which Kronecker factor (with its rotation) is kept, and factored replaces the dense second moment by . Each variant uses its own best LR/WD cell; mean std over 3 seeds of the ablation runs’ final evaluation. state is the matrix-parameter optimizer state (GiB) and wall is the total training wall-clock on a single H100 node.
Amortized eigenbasis update. The eigenbasis is refreshed every pf steps via the adaptive skip rule of Purifying Shampoo [11]. We form and skip the eigendecomposition whenever its off-diagonal ratio , keeping the previous (tol refreshes at every pf-th step). With a small toll, an eigendecomposition is invoked only when has drifted from diagonal in the current basis, which becomes rare late in training.
Complexity. Table 1 compares per-parameter optimizer state and compute. SOAP stores values for the first and rotated second moments, two factor EMAs, and two eigenbases. FOAM reduces this to for one first moment, the estimators , , and the factor with its eigenbasis . Per-step cost is from the rotation , matching Muon, and the amortized eigendecomposition cost drops from to .
| AdamW | Muon | SOAP | FOAM | |
| Optimizer state | ||||
| Per-step cost | ||||
| Amortized (per pf) | – | – |
Table 1. Per-parameter complexity. State, per-step compute, and amortized eigendecomposition cost for with output-side and input-side . FOAM keeps Muon’s per-step cost.
Experiment
This section asks whether FOAM matches strong Muon and SOAP baselines in pre-training across scales (§ 5.2), and whether its design components are well-founded and each contributes (§ 5.4).
Experimental Setup
Setup. We pre-train the models following a modern LLM architectural setup (RoPE [42], RM-SNorm [47], untied embeddings, MLP with GELU [17], no bias) at three scales following GPT-2 [38]: small (124M), medium (350M), and large (760M), on FineWeb [37] at a 1 Chinchilla compute budget ( tokens per parameter) [19], giving 2.49B, 7.10B, and 15.49B training tokens over 4,750, 13,550, and 29,550 optimizer steps, respectively, at a fixed batch of 0.5M tokens (1k sequence length). All runs use bfloat16 mixed precision on 8H100 (FSDP [52] at medium and large), with cosine decay to 10% of the peak LR after 2% warmup, gradient clipping at 1.0, and .
Parameter groups. Following the setup of Jordan et al. [20], Frans et al. [12], only the matrix parameters (Q/K/V/O attention projections and MLP linears) are preconditioned by the optimizer under study, while embeddings, the lm-head, and 1D and norm parameters are routed to a fixed AdamW group with LR , , no weight decay on embeddings, and weight decay on the lm-head. Unless noted, all groups use , and SOAP variants use precondition frequency pf with the Purifying-Shampoo skip tolerance tol [11].
Hyperparameter search. At small we run a LR/WD grid (factor on LR as in Frans et al. [12], and on WD) per optimizer at , re-centering on grid edges, and re-validate the inner neighborhood over three seeds. For medium and large, dense LR/WD grids are infeasible, so we transfer the small inner LR via a P [46] () anchor, sweep 5 LR points (factor ) around it with WD held at the small inner optimum, refine the best point with a LR/WD grid extended at grid edges, and report the selected cell as the per-size point estimate. Validation loss is reported as mean std over 11 seeds at small and medium and 3 seeds at large.
Baselines. The headline comparison spans AdamW [32], Muon [20], AdaMuon [41], SOAP [44] (the bilateral, dense- variant), and FOAM (ours) at all three scales. At GPT-2 small we additionally evaluate SPlus [13], Shampoo [16], and the SOAP variants of our construction ablation (§ 5.4).
Main Results
Table 2 reports the main pre-training results across three model scales. SOAP attains the lowest validation loss at every scale, with FOAM behind by 0.004, 0.008, and 0.005 nats at small, medium, and large, each within about one seed std. FOAM is ahead of Muon, AdaMuon, and AdamW at every scale. The cost gap to SOAP widens at scale. FOAM uses half of SOAP’s matrix-parameter state at every size, and its measured peak memory at small size is +7.4% over AdamW against +12.9% for SOAP. On wall-clock time, FOAM needs fewer steps than AdamW to reach AdamW’s final loss, which offsets its higher per-step cost: its time to that loss, , is within 7% of AdamW’s own at every scale and equal at large (10.53 vs. 10.55 h), whereas SOAP needs 1.35 at small and 2.15 at large, as its two eigendecompositions grow cubically with width.
| Optimizer | | small (124M) | | | medium (350M) | | | large (760M) | | |||||||||
| AdamW | 0.63 | 311 | 23.8 | 2.25 | 856 | 3.04 | 5.27 | 1449 | 10.55 | |||
| Muon | 0.32 | 361 | 22.9 | 1.12 | 983 | 2.95 | 2.64 | 1668 | 11.12 | |||
| AdaMuon | 0.63 | 366 | 23.2 | 2.25 | 991 | 2.97 | 5.27 | 1684 | 10.85 | |||
| SPlus | 2.85 | 525 | 35.0 | – | – | – | – | – | – | – | – | |
| SOAP | 2.85 | 536 | 32.1 | 10.12 | 1521 | 4.56 | 23.73 | 3343 | 22.66 | |||
| FOAM (ours) | 1.46 | 393 | 24.9 | 5.06 | 1083 | 3.25 | 11.87 | 1692 | 10.53 |
Table 2. Pre-training results. For each method and scale at the best LR/WD (details in Appendix § 7) of its own search, we report validation loss (, mean std over 11 seeds and 3 at large), matrix-group optimizer state in GiB (), per-step wall time in ms on one 8H100 node (), and the wall-clock (minutes at small, hours otherwise) to reach AdamW’s per-size final loss, i.e., the steps to reach it times .
Hyperparameter Robustness
A practical optimizer must also tolerate imprecise tuning. Figure 3 sweeps LR, WD, and around each optimizer’s inner best on GPT-2 small. FOAM’s bowl shape and width track AdamW and SOAP across all four axes within 0.05 nats. The one notable asymmetry is AdamW’s catastrophic collapse to 3.86 at , where SOAP and FOAM degrade by only 0.02 nats, so FOAM inherits SOAP’s wider safe range on the momentum axis, as panel (c) shows.
Figure 3. Hyperparameter robustness around each optimizer’s best cell on GPT-2 small. For AdamW, SOAP, and FOAM we sweep 5 LR points (factor ), 5 WD points (factor ) at the best LR, and 3 values each of and at the best LR/WD. All other HPs are held fixed. Panel (c) breaks the -axis at the top: AdamW’s point at is 3.86, far above the other values.
Analysis
We first measure what each FOAM simplification contributes to quality and cost, then ask whether input-only preconditioning is structurally justified on actual GPT-2 layers, and finally whether the factored update stays close to the dense SOAP update in practice.
FOAM-construction ablation. Table 3 ablates, starting from SOAP, the two structural simplifications of FOAM on GPT-2 small, namely input-only preconditioning, which drops the output-side rotation , and the factored second moment, which replaces the dense with the row and column estimators . Each variant is evaluated at its own best HP cell with 3 seeds. Input-only preconditioning substantially reduces both optimizer state and wall-clock while keeping validation loss within 0.01 nats of full SOAP, whereas output-only preconditioning, at the same state, loses about 0.02 nats, in line with the input-output asymmetry analyzed below. Factorization further trades a small wall-clock increase for a meaningful reduction in optimizer state with no measurable loss in validation. Stacked, the two simplifications yield FOAM, which retains SOAP’s validation while halving the matrix-parameter optimizer state. The same holds at medium and large, where input-only SOAP and FOAM differ by at most 0.01 nats while the factored second moment saves 18% of the state.
Gauss-Newton fit. Figure 4 compares six implicit preconditioners against the per-layer Gauss-Newton matrix on all 72 matrix parameters of the 12-layer GPT-2 small backbone. SOAP’s full bilateral preconditioner is the closest Kronecker approximation of , about 6 above Adam-diag and 2 above Muon. The asymmetry between input and output rotations is sharp and persistent. Dropping the output-side rotation costs only a quarter of the alignment, so input-only keeps 75% of SOAP-full at 0.38, whereas dropping the input-side rotation costs 80% and leaves output-only at 0.10. FOAM, which is input-only and factored, tracks SOAP-input-only within 1–2% at every checkpoint, indicating that factoring the eigenvalue tensor preserves nearly all of input-only’s curvature alignment. This asymmetry, in which the input-feature axis carries the dominant curvature structure, is the empirical justification for the input-only recipe used in the main results. It also holds per module, since on the exact layer-wise Gauss-Newton matrix input-only exceeds output-only in all six module types at both 124M and 350M, by 1.5× to 11.5× (Appendix § 10).
Figure 4. Layer-wise Gauss-Newton alignment. For each of 72 matrix parameters, we estimate via 10 to 30 Hutchinson probes with the closed-form softmax-CE output Hessian, and compute the Frobenius cosine in sqrt-power form for six implicit preconditioners. Curves are means over the 72 matrices along one SOAP run on GPT-2 small.
Factored update validity. Figure 5 reports two diagnostics that explain why the factored variants of Table 3 preserve quality. The rank-1 share in panel (a) is the key diagnostic. In Adam’s raw basis, is only 0.54–0.63 rank-1 separable and drifts down during training, while in SOAP’s eigenbasis the rotated stays near 0.90 for the bilateral rotation and 0.84 for the input-only rotation of FOAM. The cosine in panel (b) follows accordingly, at about 0.97, 0.96, and 0.94 in the bilateral, input-only, and raw bases. Consequently, factoring the SOAP-rotated loses essentially nothing, costing less than 0.005 nats in validation, whereas Adafactor-style factoring [39] of Adam’s raw is far less accurate. With the alignment of Figure 4, this completes FOAM’s rationale, which rotates into the input-side eigenbasis, where becomes nearly rank-1 separable, and then factors.
Figure 5. Validity of the factored second moment. On the dense run of each regime, we form the rank-1 approximation from the row and column sums of the second-moment estimate , and report (a) the rank-1 share and (b) the Frobenius cosine between and , averaged across all matrix parameters.
Beyond the Main Setting
Scale. At GPT-2 XL, with 1.56B parameters and 31B tokens, FOAM and SOAP are within 0.001 nats over three seeds and both ahead of Muon and AdamW, which are the other two optimizers run at this scale (Appendix § 11). Modality. On ViT-Tiny trained on Tiny-ImageNet, the matrix-preconditioned optimizers beat AdamW by 6.5, 8.5 points of top-1 accuracy, with SOAP first and FOAM second, on par with Muon and AdaMuon (Appendix § 12). Batch size. At a fixed token budget, growing the batch from 0.5M to 2M tokens per step raises AdamW’s loss from 3.370 to 3.956 but FOAM’s only from 3.309 to 3.364. FOAM also stays ahead of Muon in final loss at every batch size, and at 2M tokens per step only FOAM reaches AdamW’s 0.5M-batch quality. Larger batches thus stay usable with FOAM, which at 2M tokens per step reaches AdamW’s 0.5M-batch quality in about a third of its own 0.5M-batch wall-clock time (Appendix § 13).
Discussion and Conclusion
Limitations and future work. Our experiments stop at GPT-2 scale, up to 1.56B parameters, so whether FOAM’s advantages hold for frontier-scale pre-training remains the main open question. Our implementation does not yet shard and across devices, which matters at the B scale, although the layer-wise distributed optimizer of Khona et al. [22] suggests that this is an engineering rather than a fundamental obstacle. The factored update is least accurate at the key projection, whose gradients pass through RoPE and softmax (Appendix § 9). Finally, the embeddings and the lm-head stay on AdamW because their vocabulary dimension rules out a dense factor on that side, although a FOAM-style factor on their small hidden dimension would be affordable. Extending FOAM to these layers, and attention-aware factorizations for the key projection, are natural next steps.
When to use FOAM. FOAM fits when SOAP-class quality is needed but SOAP’s memory or wall-clock is binding, since it stays within 0.004–0.008 nats of SOAP at half its optimizer state and, at 760M, about half its wall-clock. Against Muon and AdaMuon, quality slightly favors FOAM at every scale and speed is near parity, while AdaMuon keeps less state. Muon and AdaMuon thus remain preferable when optimizer state is the hardest constraint. Against AdamW, FOAM, like other matrix-preconditioned methods, is most valuable at large effective batch, where AdamW degrades (Appendix § 13) and, at fixed , the optimizer’s compute share shrinks. FOAM is least attractive when memory is dominated by modules with a large input dimension, such as the MLP down-projection, whose factor carries most of FOAM’s residual state over AdamW.
Conclusion. We presented FOAM, a scalable and practical SOAP variant built on two structural simplifications: input-only preconditioning, justified by the Kronecker structure of the Hessian and a near-diagonal output curvature, and a factored Adam-Moment update, exact under K-FAC, with Purifying Shampoo’s skip rule for the amortized eigenbasis update. Where memory-efficient variants mainly shrink AdamW’s memory, FOAM addresses the complementary problem and brings SOAP-level quality close to Adam-level cost. FOAM matches strong Muon and SOAP baselines on LLM pre-training with modern architectures while using half of SOAP’s optimizer state and Muon-level wall-clock time, from 124M to 760M, with supporting evidence at 1.56B and on ViT.
Selected Hyperparameters
Table A lists the matrix-group LR and WD that each optimizer selects in the search of § 5.1.
| Optimizer | small (124M) | medium (350M) | large (760M) |
| AdamW | / 0.316 | / 0.316 | / 0.178 |
| Muon | / 0.0562 | / 0.0316 | / 0.0178 |
| AdaMuon | / 3.16 | / 1.0 | / 1.0 |
| SPlus | 0.333 / 0.00562 | – | – |
| Shampoo | / 0.1 | – | – |
| SOAP | / 0.562 | / 0.1 | / 0.1 |
| FOAM (ours) | / 0.316 | / 0.178 | / 0.1 |
Table A. Selected matrix-group LR / WD behind Table 2, for each optimizer and scale; all use and , and the tail group is fixed as in § 5.1.
Shampoo on GPT-2 Small
We run Shampoo [16] at small only, as the Distributed Shampoo preconditioner with exponent per factor and no grafting, and otherwise with the recipe and tuning of the other optimizers (Table A). Shampoo lands 0.07 nats behind FOAM and even slightly behind AdamW (Table B), whereas SOAP, which runs Adam in the same Kronecker eigenbasis, closes this gap.
| Optimizer | val |
| AdamW | |
| Shampoo | |
| SOAP | |
| FOAM (ours) |
Table B. Shampoo on GPT-2 small (124M). Validation loss at each optimizer’s selected cell.
Validity of the K-FAC Assumption
The factored update is exact under the K-FAC factorization (Lemma 1); the diagonal output side matters only for matching SOAP’s two-sided rotation (Appendix § 10). For each instrumented matrix we measure on checkpoints (i) the fidelity , the relative Frobenius error of the row-column factorization of the rotated second moment ( under K-FAC), and (ii) the sensitivity over random probes , where is the layer’s exact Gauss-Newton matrix (labels sampled from the model) and , are FOAM’s preconditioner with the dense and the factored second moment, so that isolates what the violation costs the update. The 18 instrumented matrices are the six module types at three depths, at 124M and 350M. Table C reports every matrix at step 2,000.
Findings. The violation is moderate and scale-stable: averages 0.162 at 124M and 0.147 at 350M, and the row-column marginals nearly reach the best rank-one fit (rank-one energy 0.975). The attention key projection is the outlier ( up to 0.79 in middle and late layers; its gradients pass through RoPE and softmax), while and are almost exact. The consequence for the update is small: averages +0.005 and +0.004, and stays below +0.05 even for , consistent with the 0.002-nat end-to-end difference between input-only SOAP and FOAM (Table 3). Across six training stages at 124M (steps 200–4,749), the 18-matrix mean of stays within 0.16–0.19 and that of within +0.001–+0.005, and at the three stages measured at 350M within 0.15–0.17 and +0.003–+0.004, so no stage of training makes the factored update meaningfully worse.
| module | 124M early | 124M mid | 124M late | 350M early | 350M mid | 350M late |
| attention | .209 / +.001 | .059 / +.000 | .067 / +.001 | .350 / −.001 | .070 / −.001 | .062 / −.001 |
| attention | .372 / +.005 | .785 / +.044 | .769 / +.046 | .352 / +.004 | .706 / +.036 | .508 / +.030 |
| attention | .017 / +.000 | .026 / −.000 | .011 / −.000 | .027 / +.001 | .028 / −.000 | .012 / −.000 |
| attention | .013 / −.000 | .027 / −.001 | .023 / −.001 | .014 / −.000 | .029 / −.001 | .020 / −.001 |
| MLP up | .035 / −.000 | .077 / −.001 | .122 / +.002 | .039 / −.000 | .087 / −.000 | .060 / +.004 |
| MLP down | .050 / −.000 | .145 / −.002 | .112 / +.000 | .084 / −.000 | .154 / −.001 | .047 / +.001 |
| mean over 18 | .162 / +.005 | .147 / +.004 |
Table C. Per-layer K-FAC violation and its cost (shown as ) at step 2,000 for every instrumented matrix: blocks 0, 6, 11 of 124M and 0, 12, 23 of 350M.
Per-module Input/Output Asymmetry
Table D breaks the Gauss-Newton alignment down by module type, measured with the exact layer-wise Gauss-Newton matrix of Appendix C at step 2,000 (Figure 4 uses a Hutchinson estimate over all 72 matrices of a separate run). Removing the output-side rotation from the two-sided preconditioner costs 0.160 of alignment on average, whereas removing the input-side rotation costs 0.375; averaged over depth, input-only exceeds output-only in every module type at both scales (1.5× to 10.9× at 124M and 1.9× to 11.5× at 350M), although a few individual layers reverse the order. Output-only preconditioning (0.132) is barely above diagonal Adam (0.090). The two factors differ in conditioning: after normalizing each by its own diagonal, the ratio of the largest to the 20th eigenvalue is 5.0 on the output side against 67.9 on the input side (7.4 against 213 on the EMA factors SOAP maintains), so on the output side the coordinate axes already capture most of the structure.
| module | output only | input only | both | input / output |
| attention | 0.176 | 0.322 | 0.499 | 1.8× |
| attention | 0.118 | 0.178 | 0.314 | 1.5× |
| attention | 0.209 | 0.453 | 0.614 | 2.2× |
| attention | 0.045 | 0.493 | 0.629 | 10.9× |
| MLP up | 0.152 | 0.332 | 0.522 | 2.2× |
| MLP down | 0.091 | 0.300 | 0.466 | 3.3× |
| mean over 18 matrices | 0.132 | 0.347 | 0.507 | 2.6× |
Table D. Per-module input/output asymmetry at 124M. Gauss-Newton alignment per module (mean over early, mid, and late depth) for output-only, input-only, and two-sided rotations; diagonal Adam scores 0.090 on the same measure, and the 350M values give the same ordering.
GPT-2 XL (1.56B)
We test whether the ordering of the three smaller sizes holds at GPT-2 XL with 1.56B parameters (Table E). SOAP and FOAM are tied (SOAP − FOAM = −0.0008 ± 0.0002, paired over seeds), and both are 0.012–0.013 nats ahead of Muon and 0.04 ahead of AdamW.
The model has 48 layers and width 1600 and otherwise follows §5.1. It is trained for 59,450 steps of 0.5M tokens (31B tokens, 1× Chinchilla) on the FineWeb 100B-token sample with 20% warmup and cosine decay to 10%, and evaluated on a fixed 20M-token validation set. Each optimizer is tuned on 11 to 14 LR and WD cells with seed 42, centered by the rule LR · WD · const fitted on the smaller sizes, and its best cell is re-run on seeds 1339 and 2024. SOAP and FOAM use pf = 10.
ViT-Tiny on Tiny-ImageNet
We test whether the comparison transfers across modality, task, and loss by training ViT-Tiny [7, 43] for Tiny-ImageNet [26] classification (Table F). The matrix-preconditioned optimizers beat AdamW (58.6) by 6.5 to 8.5 points of top-1 accuracy, SOAP leads (67.1), and FOAM is second (65.4), tied with Muon and AdaMuon (both 65.1) within noise.
| Optimizer | LR / WD | val |
| AdamW | / 0.423 | |
| Muon | / 0.0089 | |
| SOAP | / 0.0529 | |
| FOAM (ours) | / 0.043 |
Table E. GPT-2 XL (1.56B, 31B tokens). Validation loss on a fixed 20M-token set, mean std over seeds at each optimizer’s coarse-grid optimum.
We follow the plain ViT-Tiny recipe of Zhang et al. [49], with 300 epochs, a cosine schedule with 20-epoch warmup, and its augmentation and regularization, but train at the native image size 64 with patch size 4 and batch 256. Only the attention and MLP weight matrices use the optimizer under study, and all other parameters run AdamW with LR and WD 0.05. Each optimizer’s matrix LR and WD are tuned on a geometric grid with ratios and on one seed.
| Optimizer | matrix LR / WD | top-1 (%) |
| AdamW | / 0.158 | 58.6 |
| Muon | / 0.0158 | 65.1 |
| AdaMuon | / 0.158 | 65.1 |
| SOAP | / 0.05 | 67.1 |
| FOAM (ours) | / 0.158 | 65.4 |
Table F. ViT-Tiny on Tiny-ImageNet. Top-1 is the best evaluation over 300 epochs (final-epoch values are 0.1–0.4 points lower, with SOAP first, FOAM second, and AdamW last); single seed. AdamW’s matrix group uses , the others .
Batch-size Scaling
A fixed token budget can be spent at different batch sizes, and under data-parallel weak scaling each batch doubling roughly halves the wall-clock time, so we test whether that speed is usable (Table G). AdamW degrades sharply (3.370 3.465 3.956) and never recovers its 0.5M-batch quality at larger batches, whose wall-clock saving thus comes with a worse model. FOAM holds (3.309 3.315 3.364) and stays ahead of Muon at every batch size, turning the same scaling into about a reduction in the time to AdamW’s quality (24.9 13.8 8.2 minutes).
GPT-2 small follows the recipe of § 5.1 with a fixed 2.49B-token budget, i.e., 4,750 steps at 0.5M tokens, and each optimizer’s LR is re-tuned at 0.5M, 1M, and 2M tokens per step.
| tokens/step (steps) | AdamW | Muon | FOAM | AdamW | FOAM |
| 0.5M (4,750) | 3.370 | 3.313 | 3.309 | 23.8 min | 24.9 min |
| 1M (2,375) | 3.465 | 3.323 | 3.315 | N/R | 13.8 min |
| 2M (1,188) | 3.956 | 3.384 | 3.364 | N/R | 8.2 min |
Table G. Batch-size scaling on GPT-2 small (under the fixed 2.49B-token budget, LR re-tuned at every batch size). wall 3.370 is the wall-clock time to reach AdamW’s own 0.5M-batch quality under data-parallel weak scaling (total wall steps); N/R (not reached) marks runs that do not reach it, and the 1M and 2M rows are from a single seed each.
LLM Post-training
We test whether FOAM can replace AdamW in two post-training recipes on Qwen3 [45], changing only the optimizer (Table H). FOAM is best in SFT and tied with AdamW in OPD, while Muon is 0.013 below both in OPD. The SFT differences among the three optimizers are within 0.002 NLL, so FOAM serves as a drop-in replacement for AdamW in both recipes.
Only the 2D weight matrices of the transformer blocks use the optimizer under study, and the rest run AdamW. One LR, shared by all groups, is swept per optimizer. FOAM uses pf = 10 and tol = 0, and Muon scales its update by . All runs use no weight decay, gradient clipping at 1.0, a cosine schedule to 10%, bfloat16 with fp32 master weights, and a single seed. Supervised fine-tuning (SFT). Qwen3-1.7B-Base is fine-tuned for one epoch on 100K NuminaMath-CoT [27] examples (782 steps of batch 128, up to 4,096 tokens, ) and evaluated by the NLL on 1,000 held-out examples. On-policy distillation (OPD). [1] Qwen3-1.7B is distilled from Qwen3-4B-Instruct-2507 in non-thinking mode with a per-token reverse-KL loss on its own rollouts for open reasoning prompts (100 steps of 256 prompts, up to 8,192 tokens, ) and scored by average accuracy on MATH500 [18, 29], AIME24, AIME25, and AMC23.
| Recipe | base | AdamW | Muon | FOAM |
| SFT | 0.625 | 0.4677 () | 0.4668 () | 0.4657 () |
| OPD | 0.338 | 0.4719 () | 0.4591 () | 0.4719 () |
Table H. Post-training of Qwen3-1.7B with each optimizer at its best LR (in parentheses). SFT reports held-out NLL (lower is better) and OPD average math accuracy (higher is better), single seed.
References
- [1]Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In International Conference on Learning Representations (ICLR), 2024. arXiv:2306.13649.arxiv.org/abs/2306.13649
- [2]Noah Amsel, David Persson, Christopher Musco, and Robert M. Gower. The Polar Express: Optimal matrix sign methods and their application to the Muon algorithm. In International Conference on Learning Representations (ICLR), 2026. arXiv:2505.16932.arxiv.org/abs/2505.16932
- [3]Rohan Anil, Vineet Gupta, Tomer Koren, Kevin Regan, and Yoram Singer. Scalable second order optimization for deep learning. arXiv preprint arXiv:2002.09018, 2020.arxiv.org/abs/2002.09018
- [4]Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizdaneshesheli, and Animashree Anandkumar. signSGD: Compressed optimisation for non-convex problems. In International Conference on Machine Learning (ICML), pages 560–569, 2018.arxiv.org/abs/1802.04434
- [5]Ronan Collobert. Large Scale Machine Learning. PhD thesis, Université Paris VI, Paris, France, 2004.
- [6]George E. Dahl, Frank Schneider, Zachary Nado, Naman Agarwal, Chandramouli Shama Sastry, Philipp Hennig, Sourabh Medapati, Runa Eschenhagen, Priya Kasimbeg, Daniel Suo, Juhan Bae, Justin Gilmer, Abel L. Peirson, Bilal Khan, Rohan Anil, Mike Rabbat, Shankar Krishnan, Daniel Snider, Ehsan Amid, Kongtao Chen, Chris J. Maddison, Rakshith Vasudev, Michal Badura, Ankush Garg, and Peter Mattson. Benchmarking neural network training algorithms. arXiv preprint arXiv:2306.07179, 2023.arxiv.org/abs/2306.07179
- [7]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021. arXiv:2010.11929.arxiv.org/abs/2010.11929
- [8]John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(61):2121–2159, 2011.
- [9]Sai Surya Duvvuri, Fnu Devvrit, Rohan Anil, Cho-Jui Hsieh, and Inderjit S. Dhillon. Combining axes preconditioners through Kronecker approximation for deep learning. In International Conference on Learning Representations (ICLR), 2024.
- [10]Runa Eschenhagen, Alexander Immer, Richard E. Turner, Frank Schneider, and Philipp Hennig. Kronecker-factored approximate curvature for modern neural network architectures. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 2023. arXiv:2311.00636.arxiv.org/abs/2311.00636
- [11]Runa Eschenhagen, Aaron Defazio, Tsung-Hsien Lee, Richard E. Turner, and Hao-Jun Michael Shi. Purifying Shampoo: Investigating Shampoo’s heuristics by decomposing its preconditioner. In Advances in Neural Information Processing Systems (NeurIPS), 2025. arXiv:2506.03595.arxiv.org/abs/2506.03595
- [12]Kevin Frans, Pieter Abbeel, and Sergey Levine. What really matters in matrix-whitening optimizers? arXiv preprint arXiv:2510.25000, 2025.arxiv.org/abs/2510.25000
- [13]Kevin Frans, Sergey Levine, and Pieter Abbeel. A stable whitening optimizer for efficient neural network training. In Advances in Neural Information Processing Systems (NeurIPS), volume 38, 2025. arXiv:2506.07254.arxiv.org/abs/2506.07254
- [14]Thomas George, César Laurent, Xavier Bouthillier, Nicolas Ballas, and Pascal Vincent. Fast approximate natural gradient descent in a Kronecker-factored eigenbasis. In Advances in Neural Information Processing Systems (NeurIPS), volume 31, 2018. arXiv:1806.03884.arxiv.org/abs/1806.03884
- [15]Wenbo Gong, Meyer Scetbon, Chao Ma, and Edward Meeds. Towards efficient optimizer design for LLM via structured Fisher approximation with a low-rank extension. In International Conference on Learning Representations (ICLR), 2026. arXiv:2502.07752.arxiv.org/abs/2502.07752
- [16]Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization. In International Conference on Machine Learning (ICML), pages 1842–1850, 2018. arXiv:1802.09568.arxiv.org/abs/1802.09568
- [17]Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs). arXiv preprint arXiv:1606.08415, 2016.
- [18]Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, volume 1, 2021. arXiv:2103.03874.
- [19]Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Thomas Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae, and Laurent Sifre. An empirical analysis of compute-optimal large language model training. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, 2022. arXiv:2203.15556 (arXiv version titled “Training Compute-Optimal Large Language Models”).
- [20]Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks. https://kellerjordan.github.io/posts/muon/, 2024. Blog post, December 2024. Accessed: 2026-05-07.
- [21]Priya Kasimbeg, Frank Schneider, Runa Eschenhagen, Juhan Bae, Chandramouli Shama Sastry, Mark Saroufim, Boyuan Feng, Less Wright, Edward Z. Yang, Zachary Nado, Sourabh Medapati, Philipp Hennig, Michael Rabbat, and George E. Dahl. Accelerating neural network training: An analysis of the AlgoPerf competition. In International Conference on Learning Representations (ICLR), 2025. arXiv:2502.15015.arxiv.org/abs/2502.15015
- [22]Mikhail Khona, Aditya Vavre, Boxiang Wang, Deyu Fu, Hao Wu, Mike Chrzanowski, Bryan Catanzaro, Dheevatsa Mudigere, Jeff Pool, Michael Lightstone, Mohammad Shoeybi, Mostofa Patwary, Nima Tajbakhsh, and Tijmen Blankevoort. SOAP, Muon, and beyond: Pushing LLM pretraining scales. arXiv preprint arXiv:2607.20548, 2026.arxiv.org/abs/2607.20548
- [23]Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015. arXiv:1412.6980.arxiv.org/abs/1412.6980
- [24]Frederik Kunstner, Philipp Hennig, and Lukas Balles. Limitations of the empirical Fisher approximation for natural gradient descent. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019. arXiv:1905.12558.arxiv.org/abs/1905.12558
- [25]Frederik Kunstner, Alan Milligan, Robin Yadav, Mark Schmidt, and Alberto Bietti. Heavy-tailed class imbalance and why Adam outperforms gradient descent on language models. Advances in Neural Information Processing Systems (NeurIPS), volume 37, pages 30106–30148, 2024. arXiv:2402.19449.arxiv.org/abs/2402.19449
- [26]Ya Le and Xuan Yang. Tiny ImageNet visual recognition challenge. CS231N course project report, Stanford University, 2015. URL http://cs231n.stanford.edu/reports/2015/pdfs/yle_project.pdf.
- [27]Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. NuminaMath. https://github.com/project-numina/aimo-progress-prize/blob/main/report/numina_dataset.pdf, 2024. Hugging Face dataset: https://huggingface.co/datasets/AI-MO/NuminaMath-CoT.arxiv.org/abs/2510.05491
- [28]Zichong Li, Liming Liu, Chen Liang, Weizhuo Chen, and Tuo Zhao. NorMuon: Making Muon more efficient and scalable. In International Conference on Machine Learning (ICML), pages 68514–68533, 2026. arXiv:2510.05491.arxiv.org/abs/2510.05491
- [29]Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In International Conference on Learning Representations (ICLR), 2024. arXiv:2305.20050.arxiv.org/abs/2305.20050
- [30]Wu Lin, Scott C. Lowe, Felix Dangel, Runa Eschenhagen, Zikun Xu, and Roger B. Grosse. Understanding and improving Shampoo and SOAP via Kullback-Leibler minimization. In International Conference on Learning Representations (ICLR), 2026. arXiv:2509.03378.
- [31]Ziyue Liu, Ruijie Zhang, Zhengyang Wang, Yequan Zhao, Yupeng Su, Zi Yang, and Zheng Zhang. Muon²: Boosting Muon via adaptive second-moment preconditioning. arXiv preprint arXiv:2604.09967, 2026.
- [32]Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019. arXiv:1711.05101.arxiv.org/abs/1711.05101
- [33]Yanqing Lu, Leitao Wang, and Jinbo Liu. Understanding SOAP from the perspective of gradient whitening. arXiv preprint arXiv:2509.22938, 2025.
- [34]Chao Ma, Wenbo Gong, Meyer Scetbon, and Edward Meeds. SWAN: SGD with normalization and whitening enables stateless LLM training. In International Conference on Machine Learning (ICML), pages 41907–41942, 2025. arXiv:2412.13148.
- [35]James Martens and Roger Grosse. Optimizing neural networks with Kronecker-factored approximate curvature. In International Conference on Machine Learning (ICML), pages 2408–2417, 2015.
- [36]Depen Morwani, Itai Shapira, Nikhil Vyas, Eran Malach, Sham Kakade, and Lucas Janson. A new perspective on Shampoo’s preconditioner. In International Conference on Learning Representations (ICLR), 2025. arXiv:2406.17748.arxiv.org/abs/2406.17748
- [37]Guilherme Penedo, Hynek Kydlíček, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The FineWeb datasets: Decanting the web for the finest text data at scale. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, volume 37, pages 30811–30849, 2024. arXiv:2406.17557.
- [38]Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
- [39]Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning (ICML), pages 4596–4604, 2018. arXiv:1804.04235.
- [40]Hao-Jun Michael Shi, Tsung-Hsien Lee, Shintaro Iwasaki, Jose Gallego-Posada, Zhijing Li, Kaushik Rangadurai, Dheeraj Mudigere, and Michael Rabbat. A distributed data-parallel PyTorch implementation of the distributed Shampoo optimizer for training neural networks at-scale. arXiv preprint arXiv:2309.06497, 2023.
- [41]Chongjie Si, Debing Zhang, and Wei Shen. AdaMuon: Adaptive Muon optimizer. arXiv preprint arXiv:2507.11005, 2025.arxiv.org/abs/2507.11005
- [42]Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. RoFormer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. arXiv:2104.09864.
- [43]Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347–10357. PMLR, 2021.
- [44]Nikhil Vyas, Depen Morwani, Rosie Zhao, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham Kakade. SOAP: Improving and stabilizing Shampoo using Adam for language modeling. In International Conference on Learning Representations (ICLR), 2025. arXiv:2409.11321.arxiv.org/abs/2409.11321
- [45]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.arxiv.org/abs/2505.09388
- [46]Ge Yang, Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tuning large neural networks via zero-shot hyperparameter transfer. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pages 17084–17097, 2021. arXiv:2203.03466 (extended arXiv version titled “Tensor Programs V: ...”).arxiv.org/abs/2203.03466
- [47]Biao Zhang and Rico Sennrich. Root mean square layer normalization. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019. arXiv:1910.07467.arxiv.org/abs/1910.07467
- [48]Minxin Zhang, Yuxuan Liu, and Hayden Schaeffer. Adam improves Muon: Adaptive moment estimation with orthogonalized momentum. arXiv preprint arXiv:2602.17080, 2026.arxiv.org/abs/2602.17080
- [49]Tianxiao Zhang, Wenju Xu, Bo Luo, and Guanghui Wang. Depth-wise convolutions in vision transformers for efficient training on small datasets. Neurocomputing, 617:128998, 2025. arXiv:2407.19394.DOI
- [50]Yushun Zhang, Congliang Chen, Ziniu Li, Tian Ding, Chenwei Wu, Diederik P. Kingma, Yinyu Ye, Zhi-Quan Luo, and Ruoyu Sun. Adam-mini: Use fewer learning rates to gain more. In International Conference on Learning Representations (ICLR), 2025. arXiv:2406.16793.arxiv.org/abs/2406.16793
- [51]Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. GaLore: Memory-efficient LLM training by gradient low-rank projection. In International Conference on Machine Learning (ICML), pages 61121–61143, 2024.arxiv.org/abs/2403.03507
- [52]Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. PyTorch FSDP: Experiences on scaling fully sharded data parallel. Proceedings of the VLDB Endowment, 16(12):3848–3860, 2023. arXiv:2304.11277.arxiv.org/abs/2304.11277
- [53]Hanqing Zhu, Zhenyu Zhang, Wenyan Cong, Xi Liu, Sem Park, Vikas Chandra, Bo Long, David Z. Pan, Zhangyang Wang, and Jinwon Lee. APOLLO: SGD-like memory saving. AdamW-level performance. In Proceedings of Machine Learning and Systems (MLSys), volume 7, 2025. arXiv:2412.05270.




