betaFeedback

WorksComputer Science

Factored One-sided Adam-Moment for Practical and Scalable SOAP

Abstract

Second-order optimizers (e.g., Shampoo, SOAP) have emerged as compelling alternatives to Adam/AdamW and promise faster per-step convergence than AdamW in transformer training. However, they still remain costly in practice, and even SOAP still pays substantial optimizer-state memory and eigendecomposition overhead. We ask whether the benefits of SOAP can be retained at nearly the efficiency of AdamW. In this paper, we answer yes with a lightweight SOAP variant, Factored One-sided Adam-Moment (FOAM). The path to practical impact is twofold: first, FOAM employs input-only preconditioning, dropping the output-side Kronecker factor to reduce memory and remove one eigendecomposition. The Kronecker structure of the Hessian shows that this is exact for a diagonal output-side curvature and nearly lossless for a near-diagonal one, as we observe empirically. Second, FOAM replaces SOAP's dense rotated second moment with a factored Adam-moment estimator based on row-by-column statistics. Under the K-FAC Fisher assumption with a diagonal output curvature, this estimator exactly recovers SOAP's dense second moment, and we measure only a 15-20% K-FAC violation, with negligible effect on the update. Our empirical results support the claim that FOAM matches strong SOAP and Muon baselines in validation loss across scales of large language model (LLM) pre-training, uses half of SOAP's optimizer state, runs at Muon-class wall-clock time, and retains robustness across a three-fold learning-rate range. The results hold from 124M to 760M parameters, demonstrating that SOAP's gains need not require SOAP's cost. Code: https://github.com/naver-ai/foam

Introduction

Adam [23] and AdamW [32] have long been the default for pre-training transformer architectures, including large language models (LLMs). Their successors, including memory-efficient adaptive optimizers [50, 51, 53, 15], show that comparable optimization quality can be achieved with substantially less optimizer memory. In another direction, Shampoo [16], a foundational method in the family of Kronecker-factored second-order optimizers, has shown that the optimization quality of adaptive first-order methods such as Adam can be surpassed. Its most practical variant, SOAP [44], the Adam-in-eigenbasis version of Shampoo, reports 35–40% improvements in wall-clock time and step count over AdamW at large batch sizes, building upon the Distributed Shampoo baselines [3, 40] that won the external-tuning track of the first AlgoPerf competition [6, 21].

Still, practitioners often default to AdamW or Muon [20], because SOAP achieves its gains at the expense of computational efficiency, requiring about 4.5×4.5\times the optimizer state of Adam and a significantly slower per-step wall-clock time. More specifically, SOAP’s memory is dominated by the first moment, the second moment in the eigenbasis-rotated space, and the two Kronecker factor matrices L=GGTL = GG^{\mathsf{T}} (m2m^2) and R=GTGR = G^{\mathsf{T}}G (n2n^2) of a gradient G∈Rm×nG \in\mathbb{R}^{m \times n}. Its runtime is further occupied by the periodic eigendecomposition of these factor matrices, which makes SOAP substantially more expensive than AdamW despite its strong optimization performance.

This paper aims to retain SOAP-level quality while approaching AdamW-level cost. Our motivation is simple. If SOAP’s computational overhead can be sufficiently reduced, its optimization advantages could make it a practical successor to AdamW. To this end, we revisit SOAP’s major cost components and show that several can be relaxed or replaced without sacrificing performance. The original SOAP paper [44] already evaluated one-sided and factored variants, but as cost-reducing ablations without a criterion for when they preserve quality. Frans et al. [12] then found that input-only rotation and a factored variance each retain most of SOAP’s quality, and other works [30, 11] improved SOAP along other axes. Our contribution is thus not one-sided preconditioning or factorization itself, but explaining which side to retain, when factorization is justified, and why their combination is effective.

Concretely, our approach is built on two structural simplifications: input-only preconditioning and a factored Adam-Moment update. The first simplification drops the output-side factor LL and retains only the input-side factor RR, eliminating its m2m^2 memory and one of the two eigendecompositions. Its justification follows from the Kronecker structure of the Hessian, in which all output rows of an affine layer share the same input-side outer product xxTxx^{\mathsf{T}} [25], while the Hessian of transformer layers is empirically near-block-diagonal along output neurons or heads [50], making the input-only form the natural one-sided approximation. The second simplification replaces the dense rotated second moment with a row-and-column factored estimator, r∈Rmr \in\mathbb{R}^{m} and c∈Rnc \in\mathbb{R}^{n}, with trace normalization, reducing the buffer size from mnmn to m+nm+n. Under the K-FAC Fisher assumption [35] with a diagonal output curvature, this estimator exactly recovers SOAP’s dense second moment, and the K-FAC part holds to a 15–20% relative error in practice. Together, these simplifications reduce SOAP’s optimizer state from 2mn+2m2+2n22mn + 2m^2 + 2n^2 to mn+2n2+m+nmn + 2n^2 + m + n and the amortized eigendecomposition count from two to one. We call the resulting method FOAM (Factored One-sided Adam-Moment).

Figure 1 highlights FOAM’s main advantages. On pre-training, FOAM matches the strong Muon and SOAP baselines on validation loss with half of SOAP’s optimizer state and reaches AdamW’s final loss within 7% of AdamW’s own wall-clock time, while SOAP needs more than twice as long at 760M, and FOAM matches AdamW’s hyperparameter robustness over a three-fold learning-rate range. The advantage holds from 124M to 760M parameters with learning rates re-tuned at every size from a μP [46] anchor. In summary, our contributions are: (i) a structural justification of input-only preconditioning, with the Kronecker structure of the Hessian and empirical verifications with LLMs with recent architectural elements; (ii) a factored Adam-Moment update with trace normalization, exact under K-FAC with a diagonal output curvature and empirically validated through K-FAC-residual measurements during training; and (iii) the resulting algorithm FOAM, a scalable and practical SOAP variant that delivers SOAP-class quality with half of SOAP’s optimizer state (+7.4% measured peak training memory over AdamW) and Muon-class wall-clock time.

Figure 1. FOAM is a Pareto-efficient second-order optimizer. GPT-2 large (760M), 29.55K steps, each optimizer at its tuned learning rate. Curves are 3-seed means smoothed over 1K steps, and grey boxes mark the regions magnified in the insets. (a) Per-step. Validation loss against training step. In the last 4K steps (inset) SOAP ends slightly below FOAM (2.791 vs. 2.796, within about one seed std), both below Muon, while AdamW stays above the inset range. (b) Wall-clock. Same runs vs. wall time, normalized so AdamW finishes at 1.0× (Muon, FOAM, and SOAP finish at 1.15×, 1.17×, and 2.31×). Within AdamW’s own budget, FOAM is already 0.04 nats below AdamW’s final loss (inset), and it reaches SOAP’s quality band at 51% of SOAP’s wall-clock time. (c) Time to AdamW’s final loss (2.847; Table 2). FOAM gets there as fast as AdamW itself (10.53 vs. 10.55 h) and slightly ahead of Muon, while bilateral SOAP needs 2.15× as long.

Related Work

Adaptive optimizers can be categorized by how aggressively they approximate the empirical Fisher F=E[vec⁡(G)vec⁡(G)⊤]F = \mathbb{E}[\operatorname{vec}(G)\operatorname{vec}(G)^{\top}], which all of Adam [23]/AdamW [32], Shampoo [16], and SOAP [44] implicitly use as a curvature surrogate in place of the true Fisher [24].

Empirical Fisher and Kronecker-factored Optimizers

AdaGrad [8] introduced per-coordinate scaling by an empirical second-moment statistic, an early diagonal approximation of the inverse empirical Fisher. K-FAC [35] replaces the diagonal with a Kronecker product, F≈E[aa⊤]⊗E[ss⊤]F \approx\mathbb{E}[aa^{\top}] \otimes\mathbb{E}[ss^{\top}] where aa is the layer input and ss the gradient with respect to the layer’s pre-activation output, the assumption under which our factored Adam-Moment update is exact. Eschenhagen et al. [10] adapt K-FAC to transformer architectures via the expand and reduce flavors. Shampoo [16] carries this idea to the matrix parameters. It maintains EMAs of L=GG⊤L = GG^{\top} and R=G⊤GR = G^{\top}G and steps in L−1/4GR−1/4L^{-1/4} G R^{-1/4}, i.e., it preconditions by (R⊗L)−1/4(R \otimes L)^{-1/4}; Morwani et al. [36] show that R⊗LR \otimes L equals one power-iteration step toward the optimal Kronecker-product approximation of the empirical Fisher and is empirically close to it. SOAP [44] eigendecomposes LL and RR and runs Adam in the resulting rotated coordinates, an EKFAC-style construction [14] that Lu et al. [33] interpret as gradient whitening. Variants modify specific pieces: CASPR [9] combines the per-axis factors through a Kronecker sum instead of a Kronecker product. KL-Shampoo and KL-SOAP [30] recast the family via KL-divergence minimization, and Purifying Shampoo [11] shows that trace scaling of the preconditioner can replace learning-rate grafting, which corroborates our trace-norm choice in § 4.2. The one-sided fallback in distributed Shampoo [3] drops a factor on memory grounds, whereas we give a structural argument for keeping the input side (§ 4.1). Purifying Shampoo’s adaptive tolerance criterion for skipping the periodic eigendecomposition is adopted unchanged as FOAM’s amortized eigenbasis-refresh policy (§ 4.2).

Optimizers with Coarser Approximations

A second line of work adopts coarser approximations to the empirical Fisher in exchange for lower computational cost. Adam [23, 32] keeps only the diagonal of FF, sacrificing all matrix structure. Signum [4] keeps only the sign of the momentum. Muon [20] replaces the second-moment buffer with Newton-Schulz orthogonalization M(M⊤M)−1/2M(M^{\top}M)^{-1/2}, which can be read as a square-root natural-gradient step under a structured Fisher R~⊗Im\widetilde{R} \otimes I_m with an input-side factor R~∈Rn×n\widetilde{R} \in\mathbb{R}^{n \times n} shared across output rows [34, 15]. AdaMuon [41] adds an element-wise Adam-style second moment on top of the orthogonalized update. SPlus [13], a related SOAP variant, replaces the rotated second moment with sign normalization in the Shampoo eigenbasis and adds iterate averaging. Recent Muon variants (NorMuon [28], Muon2^2 [31], NAMO [48], Polar Express [2]) refine specific components of Muon. A parallel branch targets memory-efficient AdamW rather than matching SOAP-class quality: Adam-mini [50], GaLore [51], APOLLO [53], Alice [15], and SWAN [34]. We exclude these methods from the main comparison since they mainly target a smaller memory footprint than AdamW, whereas we target SOAP-level performance at near-Adam cost.

Positioning our method. FOAM lies at the intersection of these two branches. It stays within the Kronecker-factored Fisher framework but borrows the factored-second-moment idea from Adafactor [39]’s row-by-column estimator, applied here in the rotated coordinates of SOAP. Both ingredients appear in the SOAP paper as cost-reducing ablations, a one-sided variant and an Adafactor-style factored second moment in the eigenbasis, and Frans et al. [12] found empirically that input-only rotation and a factored variant each preserve most of SOAP’s quality; the distinction lies in the criteria. SOAP’s one-sided variant keeps the side with the smaller dimension, whereas we keep the input side irrespective of size (§ 4.1), so the two rules choose opposite sides for the MLP down-projection; likewise, we factorize only after the input-side rotation, which makes the rotated second moment separable into row and column factors (Lemma 1, § 4.2).

Preliminary: Shampoo and SOAP

We consider a transformer with matrix parameters W∈Rm×nW \in\mathbb{R}^{m \times n}, where nn and mm denote the input and output dimensions, respectively. Let Gt∈Rm×nG_t \in\mathbb{R}^{m \times n} denote the mini-batch gradient at step tt. Following the convention of SOAP [44], we define the two Kronecker factors LtL_t and RtR_t as the second-moment matrices of the gradient:

Lt=E[GtGtT]∈Rm×m,Rt=E[GtTGt]∈Rn×n,(1)L_t = \mathbb{E}[G_tG_t^{\mathsf{T}}] \in\mathbb{R}^{m \times m}, \qquad R_t = \mathbb{E}[G_t^{\mathsf{T}}G_t] \in\mathbb{R}^{n \times n}, \tag*{(1)}

LtL_t acts on the left side (output) of GtG_t, and RtR_t acts on the right side (input). In practice, both are estimated by exponential moving averages (EMAs), for which we reuse the same symbols, and we reserve the term estimator for such EMA buffers. Shampoo [16] takes a fractional inverse of each factor and applies the preconditioned update:

Wt+1=Wt−ηLt−1/4GtRt−1/4,(2)W_{t+1} = W_t - \eta L_t^{-1/4}G_tR_t^{-1/4}, \tag*{(2)}

with learning rate η\eta. SOAP [44] instead eigendecomposes LtL_t and RtR_t to obtain orthogonal rotations QL∈Rm×mQ_L \in\mathbb{R}^{m \times m} and QR∈Rn×nQ_R \in\mathbb{R}^{n \times n}, and runs Adam in the rotated coordinates:

Grot=QLTGQR∈Rm×n,(3)G_{\mathrm{rot}} = Q_L^{\mathsf{T}} G Q_R \in\mathbb{R}^{m \times n}, \tag*{(3)}

following the EKFAC tradition [14] that tracks diagonal second moments in the eigenbasis of a Kronecker-factored preconditioner. The per-step update maintains a dense second moment vrot∈Rm×nv_{\mathrm{rot}} \in\mathbb{R}^{m \times n} in the rotated space. The rotations QLQ_L and QRQ_R are refreshed every pfp_f (precondition frequency) steps via two eigendecompositions of LtL_t and RtR_t. LtL_t and RtR_t require m2+n2m^2+n^2 memory, the rotated second moment vrotv_{\mathrm{rot}} requires an additional mnmn memory, and the periodic eigendecompositions incur further computational overhead. These costs constitute the overhead structure that FOAM addresses.

Factored One-sided Adam-Moment (FOAM)

We propose FOAM, Factored One-sided Adam-Moment, which combines two structural simplifications of SOAP into a single optimizer. FOAM applies an input-side K-FAC factor combined with a factored output diagonal. It maintains only the input-side factor RtR_t, runs Adam in the eigenbasis QR=eigh⁡(Rt)Q_R = \operatorname{eigh}(R_t), and replaces SOAP’s dense rotated second moment vrot∈Rm×nv_{\mathrm{rot}} \in\mathbb{R}^{m \times n} with a row-column factored estimator v^rot=r^c^T\hat{v}_{\mathrm{rot}} = \hat{r}\hat{c}^{\mathsf{T}}. Here r^∈Rm\hat{r} \in\mathbb{R}^{m} and c^∈Rn\hat{c} \in\mathbb{R}^{n} are bias-corrected EMAs of the row and column means of the squared rotated gradient, and r^\hat{r} is a trace-matching scalar. The two simplifications together give FOAM the implicit Fisher Rt⊗diag⁡(r^t)R_t \otimes\operatorname{diag}(\hat{r}_t), a full input-side factor with a diagonal output side, derived in Algorithm 1 and the following sections. Figure 2 compares this preconditioner structure with those of Adam, Muon, and SOAP against the near-block-diagonal Hessian of a transformer layer, where FOAM keeps SOAP’s input factor with a diagonal output side.

Figure 2. Hessian and its approximation across optimizers. For W∈Rm×nW \in\mathbb{R}^{m \times n} with m=3m=3, n=2n=2, we visualize the mn×mnmn \times mn Hessian alongside the preconditioner of each optimizer; shaded cells are non-zero and darker means larger; coordinates are grouped by output row (n×nn \times n blocks), while labels follow the column-major convention of Lemma 1. The Hessian is near-block-diagonal. Adam (DmnD_{mn}): fully diagonal approximation with no Kronecker structure. Muon ((MtTMt)1/2⊗ImM_t^{\mathsf{T}} M_t)^{1/2} \otimes I_m): mm identical outer-diagonal blocks, built statelessly from the current momentum via Mt(MtTMt)−1/2M_t(M_t^{\mathsf{T}} M_t)^{-1/2}. SOAP (Rt⊗LtR_t \otimes L_t): full Kronecker fit with EMAs on both sides, with all m2m^2 outer blocks populated. FOAM (Rt⊗diag⁡(r^t)R_t \otimes\operatorname{diag}(\hat{r}_t), ours): replaces Muon’s identity output with a per-block diagonal scaling, and its stateless input factor with the EMA of GtTGtG_t^{\mathsf{T}}G_t.

Derivation overview. The steps below form a single chain from E[ggT]\mathbb{E}[gg^{\mathsf{T}}], g=vec⁡(G)g=\operatorname{vec}(G), to the two stored vectors. (i) Under K-FAC, E[ggT]=R⊗L\mathbb{E}[gg^{\mathsf{T}}] = R \otimes L, and LL is near-diagonal (§ 4.1), so the off-diagonal structure lives on the input side, which is the side to keep. (ii) Rotating by QRQ_R turns the covariance into ΛR⊗L\Lambda_R \otimes L (Lemma 1). Like SOAP running Adam in its eigenbasis, FOAM keeps only the diagonal of this covariance, a modeling choice that the rotation makes accurate. (iii) This diagonal, reshaped to m×nm \times n, is the rank-one matrix diag⁡(L)r^T\operatorname{diag}(L)\hat{r}^{\mathsf{T}}, so storing r^\hat{r} and c^\hat{c} keeps every coordinate’s own variance and loses nothing relative to the dense mnmn buffer, as formalized in § 4.2.

Algorithm 1 FOAM update at step tt for W∈Rm×nW \in\mathbb{R}^{m \times n}. 1d\mathbf{1}_d is the all-ones vector in Rd\mathbb{R}^d, and G∘2G^{\circ2}, ⊘\oslash, and ⋅\sqrt{\cdot} act element-wise. off⁡(⋅)\operatorname{off}(\cdot) is the off-diagonal ratio of a matrix (§ 4.2). QRQ_R is initialized to the identity, and decoupled weight decay is applied as in AdamW and omitted here.

Require: gradient GG; hyperparameters (η,β1,β2,ε,pf,tol)(\eta, \beta_1, \beta_2, \varepsilon, \mathrm{pf}, \mathrm{tol}); state MM, rr, cc, RR, QRQ_R

  1. R←β2R+(1−β2)GTGR \leftarrow\beta_2 R + (1-\beta_2)G^{\mathsf{T}}G (input-side factor)

  2. if t mod pf=0t \bmod\mathrm{pf}=0 and off⁡(QRTRQR)>tol\operatorname{off}(Q_R^{\mathsf{T}}R Q_R)>\mathrm{tol} then

  3. QR←eigh⁡(R)Q_R \leftarrow\operatorname{eigh}(R) (refresh the eigenbasis)

  4. end if

  5. M←β1M+(1−β1)GM \leftarrow\beta_1 M + (1-\beta_1)G (first moment)

  6. Grot←GQR, Mrot←MQRG_{\mathrm{rot}} \leftarrow GQ_R,\ M_{\mathrm{rot}} \leftarrow MQ_R (rotate into the eigenbasis)

  7. r←β2r+(1−β2)Grot∘21n/n, c←β2c+(1−β2)1mTGrot∘2/mr \leftarrow\beta_2r+(1-\beta_2)G_{\mathrm{rot}}^{\circ2}\mathbf{1}_n/n,\ c \leftarrow\beta_2c+(1-\beta_2)\mathbf{1}_m^{\mathsf{T}}G_{\mathrm{rot}}^{\circ2}/m (factored second moment)

  8. M^rot←Mrot/(1−β1t), r^←r/(1−β2t), c^←c/(1−β2t)\widehat{M}_{\mathrm{rot}} \leftarrow M_{\mathrm{rot}}/(1-\beta_1^t),\ \widehat{r} \leftarrow r/(1-\beta_2^t),\ \widehat{c} \leftarrow c/(1-\beta_2^t) (bias correction)

  9. v^rot←τr^c^T\widehat{v}_{\mathrm{rot}} \leftarrow\tau\widehat{r}\widehat{c}^{\mathsf{T}} with τ=tr⁡(R)/(∥r^∥1∥c^∥1)\tau=\operatorname{tr}(R)/(\lVert\widehat{r}\rVert_1\lVert\widehat{c}\rVert_1) (rank-one second moment at RR’s scale)

  10. Urot←M^rot⊘(v^rot+ε)U_{\mathrm{rot}} \leftarrow\widehat{M}_{\mathrm{rot}}\oslash(\sqrt{\widehat{v}_{\mathrm{rot}}}+\varepsilon) (Adam step in the eigenbasis)

  11. W←W−ηUrotQRTW \leftarrow W-\eta U_{\mathrm{rot}}Q_R^{\mathsf{T}} (rotate back and update)

Input-only Preconditioning

Our idea is to remove the output-side factor LtL_t and maintain only Rt←β2Rt−1+(1−β2)GtTGtR_t \leftarrow\beta_2R_{t-1}+(1-\beta_2)G_t^{\mathsf{T}}G_t with its eigenbasis QR=eigh⁡(Rt)Q_R=\operatorname{eigh}(R_t). The gradient rotation is applied only to the input axis, i.e., Grot=GQR∈Rm×nG_{\mathrm{rot}}=GQ_R\in\mathbb{R}^{m\times n}. This reduces the factor-state memory from O(m2+n2)O(m^2+n^2) to O(n2)O(n^2) and the number of eigendecompositions from two to one, halving this cost for square matrices.

A conceptual justification comes from the curvature of an affine layer with cross-entropy loss, the standard loss for language modeling and classification. For y=Wxy=Wx and p=softmax⁡(y)p=\operatorname{softmax}(y), the Hessian of the loss ℓ=−log⁡pk⋆\ell=-\log p_{k^\star} for a target class k⋆k^\star with respect to WW decomposes exactly as

H=xxT⊗(diag⁡(p)−ppT).(4)H=xx^{\mathsf{T}}\otimes\left(\operatorname{diag}(p)-pp^{\mathsf{T}}\right). \tag*{(4)}

The output-side factor diag⁡(p)−ppT\operatorname{diag}(p)-pp^{\mathsf{T}} has diagonal entries pi(1−pi)p_i(1-p_i) and off-diagonal entries −pipk-p_ip_k. Each off-diagonal entry pipkp_ip_k is smaller than the diagonal entry pi(1−pi)p_i(1-p_i) by a factor of pk/(1−pi)p_k/(1-p_i), which is small for most classes of a large vocabulary even far from a one-hot prediction, in line with the dominant per-class diagonal blocks of softmax linear models [25]. Larger entries arise only where a class takes much of the mass, and each pair (i,k)(i,k) receives them only from examples in which ii or kk is such a class, whereas the diagonal accumulates over all examples, so with a large vocabulary the expected output-side curvature is dominated by its diagonal, and under the K-FAC independence approximation the batch expectation becomes

E[H]≈E[xxT]⊗diag⁡(κ),κi=E[pi(1−pi)].(5)\mathbb{E}[H]\approx\mathbb{E}[xx^{\mathsf{T}}]\otimes\operatorname{diag}(\kappa),\qquad\kappa_i=\mathbb{E}[p_i(1-p_i)]. \tag*{(5)}

This has the structure of FOAM’s implicit Fisher, with the input-side curvature E[xxT]\mathbb{E}[xx^{\mathsf{T}}] tracked by RtR_t and the per-output curvature scale κi\kappa_i absorbed into r^t\widehat{r}_t. The output side need not be homogeneous across rows. FOAM only requires that it is dominated by its diagonal, which this mechanism suggests and the Gauss-Newton measurements of § 5.4 support empirically.

Beyond the affine setting. For a hidden affine layer h=Wxh=Wx, the per-example Hessian keeps the same form xxT⊗∇h2ℓxx^{\mathsf{T}}\otimes\nabla_h^2\ell, since hh is linear in WW, and only the output-side factor changes, from diag⁡(p)−ppT\operatorname{diag}(p)-pp^{\mathsf{T}} to the curvature with respect to hh. The vanishing-off-diagonal mechanism is specific to the output layer, but near-block-diagonal Hessians have been observed in MLPs throughout training [5, 50] and, with blocks along output neurons or heads, in a small transformer [50]. We provide additional indirect evidence specific to our setting through Gauss-Newton alignment measurements in § 5.4.

Factored Eigenvalue Under Input-only Preconditioning

Even with the input-only simplification, SOAP still has overheads due to the dense rotated second-moment buffer vrot∈Rm×nv_{\mathrm{rot}}\in\mathbb{R}^{m\times n}. Our next step is to replace it with two factored estimators r∈Rmr\in\mathbb{R}^m and c∈Rnc\in\mathbb{R}^n, EMAs of the row- and column-means of Grot∘2=Grot∘GrotG_{\mathrm{rot}}^{\circ2}=G_{\mathrm{rot}}\circ G_{\mathrm{rot}}, and reconstruct the rotated second moment as an outer product with trace normalization:

v^rot[i,j]=τ⋅r^[i]⋅c^[j],τ=tr⁡(R)∥r^∥1∥c^∥1.(6)\hat{v}_{\mathrm{rot}}[i,j] = \tau\cdot\hat{r}[i] \cdot\hat{c}[j], \qquad\tau= \frac{\operatorname{tr}(R)}{\lVert\hat{r}\rVert_1 \lVert\hat{c}\rVert_1}. \tag*{(6)}

The state shrinks from mnmn to m+nm+n per matrix parameter. The trace-matching scalar τ\tau makes the entries of v^rot\hat{v}_{\mathrm{rot}} sum to tr⁡(R)\operatorname{tr}(R), fixing the overall scale of the reconstructed second moment to that of the input-side factor. We now justify why this rank-one factorization is sufficient.

Lemma 1 (Covariance of the input-rotated gradient). Let g=vec⁡(G)∈Rmng = \operatorname{vec}(G) \in\mathbb{R}^{mn} and assume the K-FAC factorization E[gg⊤]=R⊗L\mathbb{E}[gg^{\top}] = R \otimes L. Let g′=vec⁡(GQR)g^{\prime} = \operatorname{vec}(GQ_R) be the input-rotated gradient in column-major form. Then

E[g′g′⊤]=ΛR⊗L,(7)\mathbb{E}[g^{\prime}g^{\prime\top}] = \Lambda_R \otimes L, \tag*{(7)}

where ΛR=QR⊤RQR\Lambda_R = Q_R^{\top} R Q_R is the diagonal eigenvalue matrix of RR.

Proof. Using the column-major identity vec⁡(AGB)=(B⊤⊗A)vec⁡(G)\operatorname{vec}(AGB) = (B^{\top} \otimes A)\operatorname{vec}(G) with A=ImA = I_m and B=QRB = Q_R, we have g′=(QR⊤⊗Im)gg^{\prime} = (Q_R^{\top} \otimes I_m)g. Therefore,

E[g′g′⊤]=(QR⊤⊗Im)E[gg⊤](QR⊗Im)=(QR⊤⊗Im)(R⊗L)(QR⊗Im).(8)\mathbb{E}[g^{\prime}g^{\prime\top}] = (Q_R^{\top} \otimes I_m)\mathbb{E}[gg^{\top}](Q_R \otimes I_m) = (Q_R^{\top} \otimes I_m)(R \otimes L)(Q_R \otimes I_m). \tag*{(8)}

The Kronecker mixed-product identity (A⊗B)(C⊗D)=(AC)⊗(BD)(A \otimes B)(C \otimes D) = (AC) \otimes(BD), applied twice, gives

(QR⊤⊗Im)(R⊗L)(QR⊗Im)=(QR⊤RQR)⊗(ImLIm)=ΛR⊗L.(9)(Q_R^{\top} \otimes I_m)(R \otimes L)(Q_R \otimes I_m) = (Q_R^{\top}RQ_R) \otimes(I_m L I_m) = \Lambda_R \otimes L. \tag*{(9)}

Lemma 1 tells us two things about the rotated gradient. First, the input-side covariance becomes diagonal, because rotating by QRQ_R decorrelates the input directions and replaces RR with its eigenvalues ΛR\Lambda_R. Second, the output-side factor LL is unchanged, so when LL is diagonal (§ 4.1) the rotated covariance ΛR⊗L\Lambda_R \otimes L is itself diagonal, and its diagonal, the part FOAM keeps, factorizes as

E[(GQR)[i,j]2]=λR[j]⋅L[i,i],(10)\mathbb{E}[(GQ_R)[i,j]^2] = \lambda_R[j] \cdot L[i,i], \tag*{(10)}

with λR[j]\lambda_R[j] the jj-th diagonal entry of ΛR\Lambda_R. The right-hand side is a per-row factor depending only on ii times a per-column factor depending only on jj, so the entire m×nm \times n second-moment surface is captured by m+nm+n numbers rather than mnmn. The row- and column-means of Grot∘2G_{\mathrm{rot}}^{\circ2} are the natural estimators of these two factors, up to a global scalar that the trace normalization τ\tau removes, so τr^c^⊤\tau\hat{r}\hat{c}^{\top} recovers the dense mnmn buffer exactly under K-FAC, and SOAP’s two-sided one when LL is also diagonal. In our runs, the K-FAC assumption is violated by a 15–20% relative error, mostly at the key projection, yet this costs only about 0.002 nats in final validation loss (Table 3, Appendix § 9).

input RRoutput LLfactored vvval lossstate (GiB)wall (h)
✓✓3.295±.0113.295 \pm.0112.920.48
✓✓✓3.293±.0083.293 \pm.0082.590.49
✓3.302±.0083.302 \pm.0081.780.44
✓3.322±.0103.322 \pm.0101.780.46
✓✓3.304±.0073.304 \pm.0071.460.46

Table 3. FOAM-construction ablation on GPT-2 small (124M). Starting from SOAP, input RR and output LL mark which Kronecker factor (with its rotation) is kept, and factored vv replaces the dense second moment by r,cr,c. Each variant uses its own best LR/WD cell; mean ±\pm std over 3 seeds of the ablation runs’ final evaluation. state is the matrix-parameter optimizer state (GiB) and wall is the total training wall-clock on a single 8×8 \timesH100 node.

Amortized eigenbasis update. The eigenbasis QRQ_R is refreshed every pf steps via the adaptive skip rule of Purifying Shampoo [11]. We form B=QR⊤RtQRB = Q_R^{\top}R_tQ_R and skip the eigendecomposition whenever its off-diagonal ratio off⁡(B)=∥B−diag⁡(B)∥F/∥B∥F≤τ\operatorname{off}(B) = \lVert B-\operatorname{diag}(B)\rVert_F/\lVert B\rVert_F \leq\tau, keeping the previous QRQ_R (tol =0= 0 refreshes at every pf-th step). With a small toll, an eigendecomposition is invoked only when RtR_t has drifted from diagonal in the current basis, which becomes rare late in training.

Complexity. Table 1 compares per-parameter optimizer state and compute. SOAP stores 2mn+2m2+2n22mn + 2m^2 + 2n^2 values for the first and rotated second moments, two factor EMAs, and two eigenbases. FOAM reduces this to mn+2n2+(m+n)mn + 2n^2 + (m+n) for one first moment, the estimators rr, cc, and the factor RR with its eigenbasis QRQ_R. Per-step cost is O(mn2)\mathcal{O}(mn^2) from the rotation GQRGQ_R, matching Muon, and the amortized eigendecomposition cost drops from O(m3+n3)\mathcal{O}(m^3+n^3) to O(n3)\mathcal{O}(n^3).

AdamWMuonSOAPFOAM
Optimizer state2mn2mnmnmn2mn+2m2+2n22mn + 2m^2 + 2n^2mn+2n2+(m+n)mn + 2n^2 + (m + n)
Per-step costO(mn)\mathcal{O}(mn)O(mn2)\mathcal{O}(mn^2)O(mn(m+n))\mathcal{O}(mn(m+n))O((mn2))\mathcal{O}((mn^2))
Amortized (per pf)––O(m3+n3)\mathcal{O}(m^3+n^3)O(n3)\mathcal{O}(n^3)

Table 1. Per-parameter complexity. State, per-step compute, and amortized eigendecomposition cost for W∈Rm×nW \in\mathbb{R}^{m \times n} with output-side L∈Rm×mL \in\mathbb{R}^{m \times m} and input-side R∈Rn×nR \in\mathbb{R}^{n \times n}. FOAM keeps Muon’s per-step cost.

Experiment

This section asks whether FOAM matches strong Muon and SOAP baselines in pre-training across scales (§ 5.2), and whether its design components are well-founded and each contributes (§ 5.4).

Experimental Setup

Setup. We pre-train the models following a modern LLM architectural setup (RoPE [42], RM-SNorm [47], untied embeddings, MLP with GELU [17], no bias) at three scales following GPT-2 [38]: small (124M), medium (350M), and large (760M), on FineWeb [37] at a 1×\times Chinchilla compute budget (≈20\approx20 tokens per parameter) [19], giving 2.49B, 7.10B, and 15.49B training tokens over 4,750, 13,550, and 29,550 optimizer steps, respectively, at a fixed batch of 0.5M tokens (1k sequence length). All runs use bfloat16 mixed precision on 8×\timesH100 (FSDP [52] at medium and large), with cosine decay to 10% of the peak LR after 2% warmup, gradient clipping at 1.0, and ε=10−8\varepsilon=10^{-8}.

Parameter groups. Following the setup of Jordan et al. [20], Frans et al. [12], only the matrix parameters (Q/K/V/O attention projections and MLP linears) are preconditioned by the optimizer under study, while embeddings, the lm-head, and 1D and norm parameters are routed to a fixed AdamW group with LR 10−210^{-2}, β=(0.95,0.95)\beta=(0.95,0.95), no weight decay on embeddings, and weight decay 10−310^{-3} on the lm-head. Unless noted, all groups use β=(0.95,0.95)\beta=(0.95,0.95), and SOAP variants use precondition frequency pf =10=10 with the Purifying-Shampoo skip tolerance tol =0.1=0.1 [11].

Hyperparameter search. At small we run a 5×55\times5 LR/WD grid (factor 101/8≈1.3310^{1/8}\approx1.33 on LR as in Frans et al. [12], and 101/4≈1.7810^{1/4}\approx1.78 on WD) per optimizer at β=(0.95,0.95)\beta=(0.95,0.95), re-centering on grid edges, and re-validate the inner 3×33\times3 neighborhood over three seeds. For medium and large, dense LR/WD grids are infeasible, so we transfer the small inner LR via a μ\muP [46] (∝1/width\propto1/\text{width}) anchor, sweep 5 LR points (factor 101/810^{1/8}) around it with WD held at the small inner optimum, refine the best point with a 3×33\times3 LR/WD grid extended at grid edges, and report the selected cell as the per-size point estimate. Validation loss is reported as mean ±\pm std over 11 seeds at small and medium and 3 seeds at large.

Baselines. The headline comparison spans AdamW [32], Muon [20], AdaMuon [41], SOAP [44] (the bilateral, dense-ν\nu variant), and FOAM (ours) at all three scales. At GPT-2 small we additionally evaluate SPlus [13], Shampoo [16], and the SOAP variants of our construction ablation (§ 5.4).

Main Results

Table 2 reports the main pre-training results across three model scales. SOAP attains the lowest validation loss at every scale, with FOAM behind by 0.004, 0.008, and 0.005 nats at small, medium, and large, each within about one seed std. FOAM is ahead of Muon, AdaMuon, and AdamW at every scale. The cost gap to SOAP widens at scale. FOAM uses half of SOAP’s matrix-parameter state at every size, and its measured peak memory at small size is +7.4% over AdamW against +12.9% for SOAP. On wall-clock time, FOAM needs fewer steps than AdamW to reach AdamW’s final loss, which offsets its higher per-step cost: its time to that loss, tAt_A, is within 7% of AdamW’s own at every scale and equal at large (10.53 vs. 10.55 h), whereas SOAP needs 1.35×\times at small and 2.15×\times at large, as its two eigendecompositions grow cubically with width.

Optimizer| small (124M) || medium (350M) || large (760M) |
valvalstatestateiteritertAt_AvalvalstatestateiteritertAt_AvalvalstatestateiteritertAt_A
AdamW3.370±.0093.370\pm.0090.6331123.83.040±.0123.040\pm.0122.258563.042.847±.0042.847\pm.0045.27144910.55
Muon3.313±.0083.313\pm.0080.3236122.92.998±.0102.998\pm.0101.129832.952.806±.0052.806\pm.0052.64166811.12
AdaMuon3.311±.0093.311\pm.0090.6336623.22.992±.0122.992\pm.0122.259912.972.799±.0042.799\pm.0045.27168410.85
SPlus3.325±.0073.325\pm.0072.8552535.0––––––––
SOAP3.305±.0093.305\pm.0092.8553632.12.983±.0122.983\pm.01210.1215214.562.791±.0052.791\pm.00523.73334322.66
FOAM (ours)3.309±.0083.309\pm.0081.4639324.92.991±.0122.991\pm.0125.0610833.252.796±.0052.796\pm.00511.87169210.53

Table 2. Pre-training results. For each method and scale at the best LR/WD (details in Appendix § 7) of its own search, we report validation loss (valval, mean ±\pm std over 11 seeds and 3 at large), matrix-group optimizer state in GiB (statestate), per-step wall time in ms on one 8×\timesH100 node (iteriter), and the wall-clock tAt_A (minutes at small, hours otherwise) to reach AdamW’s per-size final loss, i.e., the steps to reach it times iteriter.

Hyperparameter Robustness

A practical optimizer must also tolerate imprecise tuning. Figure 3 sweeps LR, WD, and β1,β2\beta_1, \beta_2 around each optimizer’s inner best on GPT-2 small. FOAM’s bowl shape and width track AdamW and SOAP across all four axes within 0.05 nats. The one notable asymmetry is AdamW’s catastrophic collapse to 3.86 at β1=0.99\beta_1 = 0.99, where SOAP and FOAM degrade by only 0.02 nats, so FOAM inherits SOAP’s wider safe range on the momentum axis, as panel (c) shows.

Figure 3. Hyperparameter robustness around each optimizer’s best cell on GPT-2 small. For AdamW, SOAP, and FOAM we sweep 5 LR points (factor 101/810^{1/8}), 5 WD points (factor 101/410^{1/4}) at the best LR, and 3 values each of β1\beta_1 and β2\beta_2 at the best LR/WD. All other HPs are held fixed. Panel (c) breaks the yy-axis at the top: AdamW’s point at β1=0.99\beta_1 = 0.99 is 3.86, far above the other values.

Analysis

We first measure what each FOAM simplification contributes to quality and cost, then ask whether input-only preconditioning is structurally justified on actual GPT-2 layers, and finally whether the factored update stays close to the dense SOAP update in practice.

FOAM-construction ablation. Table 3 ablates, starting from SOAP, the two structural simplifications of FOAM on GPT-2 small, namely input-only preconditioning, which drops the output-side rotation QLQ_L, and the factored second moment, which replaces the dense vv with the row and column estimators r,cr,c. Each variant is evaluated at its own best HP cell with 3 seeds. Input-only preconditioning substantially reduces both optimizer state and wall-clock while keeping validation loss within 0.01 nats of full SOAP, whereas output-only preconditioning, at the same state, loses about 0.02 nats, in line with the input-output asymmetry analyzed below. Factorization further trades a small wall-clock increase for a meaningful reduction in optimizer state with no measurable loss in validation. Stacked, the two simplifications yield FOAM, which retains SOAP’s validation while halving the matrix-parameter optimizer state. The same holds at medium and large, where input-only SOAP and FOAM differ by at most 0.01 nats while the factored second moment saves 18% of the state.

Gauss-Newton fit. Figure 4 compares six implicit preconditioners against the per-layer Gauss-Newton matrix HGNH_{\mathrm{GN}} on all 72 matrix parameters of the 12-layer GPT-2 small backbone. SOAP’s full bilateral preconditioner is the closest Kronecker approximation of HGNH_{\mathrm{GN}}, about 6×\times above Adam-diag and 2×\times above Muon. The asymmetry between input and output rotations is sharp and persistent. Dropping the output-side rotation costs only a quarter of the alignment, so input-only keeps 75% of SOAP-full at 0.38, whereas dropping the input-side rotation costs 80% and leaves output-only at 0.10. FOAM, which is input-only and factored, tracks SOAP-input-only within 1–2% at every checkpoint, indicating that factoring the eigenvalue tensor preserves nearly all of input-only’s curvature alignment. This asymmetry, in which the input-feature axis carries the dominant curvature structure, is the empirical justification for the input-only recipe used in the main results. It also holds per module, since on the exact layer-wise Gauss-Newton matrix input-only exceeds output-only in all six module types at both 124M and 350M, by 1.5× to 11.5× (Appendix § 10).

Figure 4. Layer-wise Gauss-Newton alignment. For each of 72 matrix parameters, we estimate HGN=JℓTGyJℓH_{\mathrm{GN}} = J_{\ell}^{\mathsf{T}} G_y J_{\ell} via 10 to 30 Hutchinson probes with the closed-form softmax-CE output Hessian, and compute the Frobenius cosine cos⁡(P,HGN)\cos(P, H_{\mathrm{GN}}) in sqrt-power form for six implicit preconditioners. Curves are means over the 72 matrices along one SOAP run on GPT-2 small.

Factored update validity. Figure 5 reports two diagnostics that explain why the factored variants of Table 3 preserve quality. The rank-1 share in panel (a) is the key diagnostic. In Adam’s raw g2g^2 basis, VV is only 0.54–0.63 rank-1 separable and drifts down during training, while in SOAP’s eigenbasis the rotated VV stays near 0.90 for the bilateral rotation and 0.84 for the input-only rotation of FOAM. The cosine in panel (b) follows accordingly, at about 0.97, 0.96, and 0.94 in the bilateral, input-only, and raw bases. Consequently, factoring the SOAP-rotated VV loses essentially nothing, costing less than 0.005 nats in validation, whereas Adafactor-style factoring [39] of Adam’s raw VV is far less accurate. With the HGNH_{\mathrm{GN}} alignment of Figure 4, this completes FOAM’s rationale, which rotates into the input-side eigenbasis, where VV becomes nearly rank-1 separable, and then factors.

Figure 5. Validity of the factored second moment. On the dense run of each regime, we form the rank-1 approximation Vfac=(V1)(1TV)/(1TV1)V_{\mathrm{fac}} = (V\mathbf{1})(\mathbf{1}^{\mathsf{T}}V)/(\mathbf{1}^{\mathsf{T}}V\mathbf{1}) from the row and column sums of the second-moment estimate VV, and report (a) the rank-1 share σ1(V)/∑iσi(V)\sigma_1(V)/\sum_i \sigma_i(V) and (b) the Frobenius cosine between VfacV_{\mathrm{fac}} and VV, averaged across all matrix parameters.

Beyond the Main Setting

Scale. At GPT-2 XL, with 1.56B parameters and 31B tokens, FOAM and SOAP are within 0.001 nats over three seeds and both ahead of Muon and AdamW, which are the other two optimizers run at this scale (Appendix § 11). Modality. On ViT-Tiny trained on Tiny-ImageNet, the matrix-preconditioned optimizers beat AdamW by 6.5, 8.5 points of top-1 accuracy, with SOAP first and FOAM second, on par with Muon and AdaMuon (Appendix § 12). Batch size. At a fixed token budget, growing the batch from 0.5M to 2M tokens per step raises AdamW’s loss from 3.370 to 3.956 but FOAM’s only from 3.309 to 3.364. FOAM also stays ahead of Muon in final loss at every batch size, and at 2M tokens per step only FOAM reaches AdamW’s 0.5M-batch quality. Larger batches thus stay usable with FOAM, which at 2M tokens per step reaches AdamW’s 0.5M-batch quality in about a third of its own 0.5M-batch wall-clock time (Appendix § 13).

Discussion and Conclusion

Limitations and future work. Our experiments stop at GPT-2 scale, up to 1.56B parameters, so whether FOAM’s advantages hold for frontier-scale pre-training remains the main open question. Our implementation does not yet shard RR and QRQ_R across devices, which matters at the ≥7\geq7B scale, although the layer-wise distributed optimizer of Khona et al. [22] suggests that this is an engineering rather than a fundamental obstacle. The factored update is least accurate at the key projection, whose gradients pass through RoPE and softmax (Appendix § 9). Finally, the embeddings and the lm-head stay on AdamW because their vocabulary dimension rules out a dense factor on that side, although a FOAM-style factor on their small hidden dimension would be affordable. Extending FOAM to these layers, and attention-aware factorizations for the key projection, are natural next steps.

When to use FOAM. FOAM fits when SOAP-class quality is needed but SOAP’s memory or wall-clock is binding, since it stays within 0.004–0.008 nats of SOAP at half its optimizer state and, at 760M, about half its wall-clock. Against Muon and AdaMuon, quality slightly favors FOAM at every scale and speed is near parity, while AdaMuon keeps less state. Muon and AdaMuon thus remain preferable when optimizer state is the hardest constraint. Against AdamW, FOAM, like other matrix-preconditioned methods, is most valuable at large effective batch, where AdamW degrades (Appendix § 13) and, at fixed pfp_f, the optimizer’s compute share shrinks. FOAM is least attractive when memory is dominated by modules with a large input dimension, such as the MLP down-projection, whose n2n^2 factor carries most of FOAM’s residual state over AdamW.

Conclusion. We presented FOAM, a scalable and practical SOAP variant built on two structural simplifications: input-only preconditioning, justified by the Kronecker structure of the Hessian and a near-diagonal output curvature, and a factored Adam-Moment update, exact under K-FAC, with Purifying Shampoo’s skip rule for the amortized eigenbasis update. Where memory-efficient variants mainly shrink AdamW’s memory, FOAM addresses the complementary problem and brings SOAP-level quality close to Adam-level cost. FOAM matches strong Muon and SOAP baselines on LLM pre-training with modern architectures while using half of SOAP’s optimizer state and Muon-level wall-clock time, from 124M to 760M, with supporting evidence at 1.56B and on ViT.

Selected Hyperparameters

Table A lists the matrix-group LR and WD that each optimizer selects in the search of § 5.1.

Optimizersmall (124M)medium (350M)large (760M)
AdamW1.9e−31.9\mathrm{e}{-3} / 0.3161.06e−31.06\mathrm{e}{-3} / 0.3168.5e−48.5\mathrm{e}{-4} / 0.178
Muon1.92e−21.92\mathrm{e}{-2} / 0.05621.09e−21.09\mathrm{e}{-2} / 0.03168.76e−38.76\mathrm{e}{-3} / 0.0178
AdaMuon4.41e−44.41\mathrm{e}{-4} / 3.163.33e−43.33\mathrm{e}{-4} / 1.02.0e−42.0\mathrm{e}{-4} / 1.0
SPlus0.333 / 0.00562––
Shampoo1.4e−21.4\mathrm{e}{-2} / 0.1––
SOAP1.86e−31.86\mathrm{e}{-3} / 0.5623.32e−33.32\mathrm{e}{-3} / 0.11.99e−31.99\mathrm{e}{-3} / 0.1
FOAM (ours)2.5e−32.5\mathrm{e}{-3} / 0.3161.41e−31.41\mathrm{e}{-3} / 0.1781.51e−31.51\mathrm{e}{-3} / 0.1

Table A. Selected matrix-group LR / WD behind Table 2, for each optimizer and scale; all use β=(0.95,0.95)\beta= (0.95, 0.95) and ϵ=10−8\epsilon= 10^{-8}, and the tail group is fixed as in § 5.1.

Shampoo on GPT-2 Small

We run Shampoo [16] at small only, as the Distributed Shampoo preconditioner with exponent −1/4-1/4 per factor and no grafting, and otherwise with the recipe and tuning of the other optimizers (Table A). Shampoo lands 0.07 nats behind FOAM and even slightly behind AdamW (Table B), whereas SOAP, which runs Adam in the same Kronecker eigenbasis, closes this gap.

Optimizerval
AdamW3.370±.0093.370 \pm.009
Shampoo3.379±.0063.379 \pm.006
SOAP3.305±.0093.305 \pm.009
FOAM (ours)3.309±.0083.309 \pm.008

Table B. Shampoo on GPT-2 small (124M). Validation loss at each optimizer’s selected cell.

Validity of the K-FAC Assumption

The factored update is exact under the K-FAC factorization (Lemma 1); the diagonal output side matters only for matching SOAP’s two-sided rotation (Appendix § 10). For each instrumented matrix we measure on checkpoints (i) the fidelity ρ(W)=∥Λ−rc⊤∥F/∥Λ∥F\rho(W) = \lVert\Lambda- rc^{\top} \rVert_{F}/\lVert\Lambda\rVert_{F}, the relative Frobenius error of the row-column factorization of the rotated second moment Λ\Lambda (ρ=0\rho= 0 under K-FAC), and (ii) the sensitivity Δ(W)=cos⁡(PdenseV,HGNV)−cos⁡(PfacV,HGNV)\Delta(W) = \cos(P_{\mathrm{dense}}V, H_{\mathrm{GN}}V) - \cos(P_{\mathrm{fac}}V, H_{\mathrm{GN}}V) over random probes VV, where HGNH_{\mathrm{GN}} is the layer’s exact Gauss-Newton matrix (labels sampled from the model) and PdenseP_{\mathrm{dense}}, PfacP_{\mathrm{fac}} are FOAM’s preconditioner with the dense and the factored second moment, so that Δ\Delta isolates what the violation costs the update. The 18 instrumented matrices are the six module types at three depths, at 124M and 350M. Table C reports every matrix at step 2,000.

Findings. The violation is moderate and scale-stable: ρ\rho averages 0.162 at 124M and 0.147 at 350M, and the row-column marginals nearly reach the best rank-one fit (rank-one energy 0.975). The attention key projection is the outlier (ρ\rho up to 0.79 in middle and late layers; its gradients pass through RoPE and softmax), while WvW_v and WoW_o are almost exact. The consequence for the update is small: Δ\Delta averages +0.005 and +0.004, and stays below +0.05 even for WkW_k, consistent with the 0.002-nat end-to-end difference between input-only SOAP and FOAM (Table 3). Across six training stages at 124M (steps 200–4,749), the 18-matrix mean of ρ\rho stays within 0.16–0.19 and that of Δ\Delta within +0.001–+0.005, and at the three stages measured at 350M within 0.15–0.17 and +0.003–+0.004, so no stage of training makes the factored update meaningfully worse.

module124M early124M mid124M late350M early350M mid350M late
attention WqW_q.209 / +.001.059 / +.000.067 / +.001.350 / −.001.070 / −.001.062 / −.001
attention WkW_k.372 / +.005.785 / +.044.769 / +.046.352 / +.004.706 / +.036.508 / +.030
attention WvW_v.017 / +.000.026 / −.000.011 / −.000.027 / +.001.028 / −.000.012 / −.000
attention WoW_o.013 / −.000.027 / −.001.023 / −.001.014 / −.000.029 / −.001.020 / −.001
MLP up.035 / −.000.077 / −.001.122 / +.002.039 / −.000.087 / −.000.060 / +.004
MLP down.050 / −.000.145 / −.002.112 / +.000.084 / −.000.154 / −.001.047 / +.001
mean over 18.162 / +.005.147 / +.004

Table C. Per-layer K-FAC violation ρ\rho and its cost Δ\Delta (shown as ρ/Δ\rho/ \Delta) at step 2,000 for every instrumented matrix: blocks 0, 6, 11 of 124M and 0, 12, 23 of 350M.

Per-module Input/Output Asymmetry

Table D breaks the Gauss-Newton alignment down by module type, measured with the exact layer-wise Gauss-Newton matrix of Appendix C at step 2,000 (Figure 4 uses a Hutchinson estimate over all 72 matrices of a separate run). Removing the output-side rotation from the two-sided preconditioner costs 0.160 of alignment on average, whereas removing the input-side rotation costs 0.375; averaged over depth, input-only exceeds output-only in every module type at both scales (1.5× to 10.9× at 124M and 1.9× to 11.5× at 350M), although a few individual layers reverse the order. Output-only preconditioning (0.132) is barely above diagonal Adam (0.090). The two factors differ in conditioning: after normalizing each by its own diagonal, the ratio of the largest to the 20th eigenvalue is 5.0 on the output side against 67.9 on the input side (7.4 against 213 on the EMA factors SOAP maintains), so on the output side the coordinate axes already capture most of the structure.

moduleoutput onlyinput onlybothinput / output
attention WqW_q0.1760.3220.4991.8×
attention WkW_k0.1180.1780.3141.5×
attention WvW_v0.2090.4530.6142.2×
attention WoW_o0.0450.4930.62910.9×
MLP up0.1520.3320.5222.2×
MLP down0.0910.3000.4663.3×
mean over 18 matrices0.1320.3470.5072.6×

Table D. Per-module input/output asymmetry at 124M. Gauss-Newton alignment per module (mean over early, mid, and late depth) for output-only, input-only, and two-sided rotations; diagonal Adam scores 0.090 on the same measure, and the 350M values give the same ordering.

GPT-2 XL (1.56B)

We test whether the ordering of the three smaller sizes holds at GPT-2 XL with 1.56B parameters (Table E). SOAP and FOAM are tied (SOAP − FOAM = −0.0008 ± 0.0002, paired over seeds), and both are 0.012–0.013 nats ahead of Muon and 0.04 ahead of AdamW.

The model has 48 layers and width 1600 and otherwise follows §5.1. It is trained for 59,450 steps of 0.5M tokens (31B tokens, 1× Chinchilla) on the FineWeb 100B-token sample with 20% warmup and cosine decay to 10%, and evaluated on a fixed 20M-token validation set. Each optimizer is tuned on 11 to 14 LR and WD cells with seed 42, centered by the rule LR · WD · T≈T \approx const fitted on the smaller sizes, and its best cell is re-run on seeds 1339 and 2024. SOAP and FOAM use pf = 10.

ViT-Tiny on Tiny-ImageNet

We test whether the comparison transfers across modality, task, and loss by training ViT-Tiny [7, 43] for Tiny-ImageNet [26] classification (Table F). The matrix-preconditioned optimizers beat AdamW (58.6) by 6.5 to 8.5 points of top-1 accuracy, SOAP leads (67.1), and FOAM is second (65.4), tied with Muon and AdaMuon (both 65.1) within noise.

OptimizerLR / WDval
AdamW2.4e−42.4\mathrm{e}{-4} / 0.4232.6365±.00022.6365\pm.0002
Muon6.78e−36.78\mathrm{e}{-3} / 0.00892.6081±.00052.6081\pm.0005
SOAP1.17e−31.17\mathrm{e}{-3} / 0.05292.5951±.00072.5951\pm.0007
FOAM (ours)1.48e−31.48\mathrm{e}{-3} / 0.0432.5959±.00032.5959\pm.0003

Table E. GPT-2 XL (1.56B, 31B tokens). Validation loss on a fixed 20M-token set, mean ±\pm std over seeds {42,1339,2024}\{42, 1339, 2024\} at each optimizer’s coarse-grid optimum.

We follow the plain ViT-Tiny recipe of Zhang et al. [49], with 300 epochs, a cosine schedule with 20-epoch warmup, and its augmentation and regularization, but train at the native image size 64 with patch size 4 and batch 256. Only the attention and MLP weight matrices use the optimizer under study, and all other parameters run AdamW with LR 5⋅10−45\cdot10^{-4} and WD 0.05. Each optimizer’s matrix LR and WD are tuned on a geometric grid with ratios 101/410^{1/4} and 101/210^{1/2} on one seed.

Optimizermatrix LR / WDtop-1 (%)
AdamW1.0e−31.0\mathrm{e}{-3} / 0.15858.6
Muon1.0e−21.0\mathrm{e}{-2} / 0.015865.1
AdaMuon1.0e−31.0\mathrm{e}{-3} / 0.15865.1
SOAP5.6e−35.6\mathrm{e}{-3} / 0.0567.1
FOAM (ours)1.8e−31.8\mathrm{e}{-3} / 0.15865.4

Table F. ViT-Tiny on Tiny-ImageNet. Top-1 is the best evaluation over 300 epochs (final-epoch values are 0.1–0.4 points lower, with SOAP first, FOAM second, and AdamW last); single seed. AdamW’s matrix group uses β=(0.9,0.999)\beta=(0.9, 0.999), the others (0.95,0.95)(0.95, 0.95).

Batch-size Scaling

A fixed token budget can be spent at different batch sizes, and under data-parallel weak scaling each batch doubling roughly halves the wall-clock time, so we test whether that speed is usable (Table G). AdamW degrades sharply (3.370 →\to 3.465 →\to 3.956) and never recovers its 0.5M-batch quality at larger batches, whose wall-clock saving thus comes with a worse model. FOAM holds (3.309 →\to 3.315 →\to 3.364) and stays ahead of Muon at every batch size, turning the same scaling into about a 3×3\times reduction in the time to AdamW’s quality (24.9 →\to 13.8 →\to 8.2 minutes).

GPT-2 small follows the recipe of § 5.1 with a fixed 2.49B-token budget, i.e., 4,750 steps at 0.5M tokens, and each optimizer’s LR is re-tuned at 0.5M, 1M, and 2M tokens per step.

tokens/step (steps)AdamWMuonFOAMAdamWFOAM
0.5M (4,750)3.3703.3133.30923.8 min24.9 min
1M (2,375)3.4653.3233.315N/R13.8 min
2M (1,188)3.9563.3843.364N/R8.2 min

Table G. Batch-size scaling on GPT-2 small (under the fixed 2.49B-token budget, LR re-tuned at every batch size). wall →\to 3.370 is the wall-clock time to reach AdamW’s own 0.5M-batch quality under data-parallel weak scaling (total wall ∝\propto steps); N/R (not reached) marks runs that do not reach it, and the 1M and 2M rows are from a single seed each.

LLM Post-training

We test whether FOAM can replace AdamW in two post-training recipes on Qwen3 [45], changing only the optimizer (Table H). FOAM is best in SFT and tied with AdamW in OPD, while Muon is 0.013 below both in OPD. The SFT differences among the three optimizers are within 0.002 NLL, so FOAM serves as a drop-in replacement for AdamW in both recipes.

Only the 2D weight matrices of the transformer blocks use the optimizer under study, and the rest run AdamW. One LR, shared by all groups, is swept per optimizer. FOAM uses pf = 10 and tol = 0, and Muon scales its update by 0.2max⁡(m,n)0.2\sqrt{\max(m,n)}. All runs use no weight decay, gradient clipping at 1.0, a cosine schedule to 10%, bfloat16 with fp32 master weights, and a single seed. Supervised fine-tuning (SFT). Qwen3-1.7B-Base is fine-tuned for one epoch on 100K NuminaMath-CoT [27] examples (782 steps of batch 128, up to 4,096 tokens, β=(0.9,0.95)\beta= (0.9, 0.95)) and evaluated by the NLL on 1,000 held-out examples. On-policy distillation (OPD). [1] Qwen3-1.7B is distilled from Qwen3-4B-Instruct-2507 in non-thinking mode with a per-token reverse-KL loss on its own rollouts for open reasoning prompts (100 steps of 256 prompts, up to 8,192 tokens, β=(0.9,0.999)\beta= (0.9, 0.999)) and scored by average accuracy on MATH500 [18, 29], AIME24, AIME25, and AMC23.

RecipebaseAdamWMuonFOAM
SFT0.6250.4677 (3e−53\mathrm{e}{-5})0.4668 (3e−53\mathrm{e}{-5})0.4657 (3e−53\mathrm{e}{-5})
OPD0.3380.4719 (1e−51\mathrm{e}{-5})0.4591 (1e−51\mathrm{e}{-5})0.4719 (1e−51\mathrm{e}{-5})

Table H. Post-training of Qwen3-1.7B with each optimizer at its best LR (in parentheses). SFT reports held-out NLL (lower is better) and OPD average math accuracy (higher is better), single seed.

References

  1. [1]Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In International Conference on Learning Representations (ICLR), 2024. arXiv:2306.13649.arxiv.org/abs/2306.13649
  2. [2]Noah Amsel, David Persson, Christopher Musco, and Robert M. Gower. The Polar Express: Optimal matrix sign methods and their application to the Muon algorithm. In International Conference on Learning Representations (ICLR), 2026. arXiv:2505.16932.arxiv.org/abs/2505.16932
  3. [3]Rohan Anil, Vineet Gupta, Tomer Koren, Kevin Regan, and Yoram Singer. Scalable second order optimization for deep learning. arXiv preprint arXiv:2002.09018, 2020.arxiv.org/abs/2002.09018
  4. [4]Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizdaneshesheli, and Animashree Anandkumar. signSGD: Compressed optimisation for non-convex problems. In International Conference on Machine Learning (ICML), pages 560–569, 2018.arxiv.org/abs/1802.04434
  5. [5]Ronan Collobert. Large Scale Machine Learning. PhD thesis, Université Paris VI, Paris, France, 2004.
  6. [6]George E. Dahl, Frank Schneider, Zachary Nado, Naman Agarwal, Chandramouli Shama Sastry, Philipp Hennig, Sourabh Medapati, Runa Eschenhagen, Priya Kasimbeg, Daniel Suo, Juhan Bae, Justin Gilmer, Abel L. Peirson, Bilal Khan, Rohan Anil, Mike Rabbat, Shankar Krishnan, Daniel Snider, Ehsan Amid, Kongtao Chen, Chris J. Maddison, Rakshith Vasudev, Michal Badura, Ankush Garg, and Peter Mattson. Benchmarking neural network training algorithms. arXiv preprint arXiv:2306.07179, 2023.arxiv.org/abs/2306.07179
  7. [7]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021. arXiv:2010.11929.arxiv.org/abs/2010.11929
  8. [8]John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(61):2121–2159, 2011.
  9. [9]Sai Surya Duvvuri, Fnu Devvrit, Rohan Anil, Cho-Jui Hsieh, and Inderjit S. Dhillon. Combining axes preconditioners through Kronecker approximation for deep learning. In International Conference on Learning Representations (ICLR), 2024.
  10. [10]Runa Eschenhagen, Alexander Immer, Richard E. Turner, Frank Schneider, and Philipp Hennig. Kronecker-factored approximate curvature for modern neural network architectures. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 2023. arXiv:2311.00636.arxiv.org/abs/2311.00636
  11. [11]Runa Eschenhagen, Aaron Defazio, Tsung-Hsien Lee, Richard E. Turner, and Hao-Jun Michael Shi. Purifying Shampoo: Investigating Shampoo’s heuristics by decomposing its preconditioner. In Advances in Neural Information Processing Systems (NeurIPS), 2025. arXiv:2506.03595.arxiv.org/abs/2506.03595
  12. [12]Kevin Frans, Pieter Abbeel, and Sergey Levine. What really matters in matrix-whitening optimizers? arXiv preprint arXiv:2510.25000, 2025.arxiv.org/abs/2510.25000
  13. [13]Kevin Frans, Sergey Levine, and Pieter Abbeel. A stable whitening optimizer for efficient neural network training. In Advances in Neural Information Processing Systems (NeurIPS), volume 38, 2025. arXiv:2506.07254.arxiv.org/abs/2506.07254
  14. [14]Thomas George, César Laurent, Xavier Bouthillier, Nicolas Ballas, and Pascal Vincent. Fast approximate natural gradient descent in a Kronecker-factored eigenbasis. In Advances in Neural Information Processing Systems (NeurIPS), volume 31, 2018. arXiv:1806.03884.arxiv.org/abs/1806.03884
  15. [15]Wenbo Gong, Meyer Scetbon, Chao Ma, and Edward Meeds. Towards efficient optimizer design for LLM via structured Fisher approximation with a low-rank extension. In International Conference on Learning Representations (ICLR), 2026. arXiv:2502.07752.arxiv.org/abs/2502.07752
  16. [16]Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization. In International Conference on Machine Learning (ICML), pages 1842–1850, 2018. arXiv:1802.09568.arxiv.org/abs/1802.09568
  17. [17]Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs). arXiv preprint arXiv:1606.08415, 2016.
  18. [18]Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, volume 1, 2021. arXiv:2103.03874.
  19. [19]Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Thomas Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae, and Laurent Sifre. An empirical analysis of compute-optimal large language model training. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, 2022. arXiv:2203.15556 (arXiv version titled “Training Compute-Optimal Large Language Models”).
  20. [20]Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks. https://kellerjordan.github.io/posts/muon/, 2024. Blog post, December 2024. Accessed: 2026-05-07.
  21. [21]Priya Kasimbeg, Frank Schneider, Runa Eschenhagen, Juhan Bae, Chandramouli Shama Sastry, Mark Saroufim, Boyuan Feng, Less Wright, Edward Z. Yang, Zachary Nado, Sourabh Medapati, Philipp Hennig, Michael Rabbat, and George E. Dahl. Accelerating neural network training: An analysis of the AlgoPerf competition. In International Conference on Learning Representations (ICLR), 2025. arXiv:2502.15015.arxiv.org/abs/2502.15015
  22. [22]Mikhail Khona, Aditya Vavre, Boxiang Wang, Deyu Fu, Hao Wu, Mike Chrzanowski, Bryan Catanzaro, Dhee­vatsa Mudigere, Jeff Pool, Michael Lightstone, Mohammad Shoeybi, Mostofa Patwary, Nima Tajbakhsh, and Tijmen Blankevoort. SOAP, Muon, and beyond: Pushing LLM pretraining scales. arXiv preprint arXiv:2607.20548, 2026.arxiv.org/abs/2607.20548
  23. [23]Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015. arXiv:1412.6980.arxiv.org/abs/1412.6980
  24. [24]Frederik Kunstner, Philipp Hennig, and Lukas Balles. Limitations of the empirical Fisher approximation for natural gradient descent. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019. arXiv:1905.12558.arxiv.org/abs/1905.12558
  25. [25]Frederik Kunstner, Alan Milligan, Robin Yadav, Mark Schmidt, and Alberto Bietti. Heavy-tailed class imbalance and why Adam outperforms gradient descent on language models. Advances in Neural Information Processing Systems (NeurIPS), volume 37, pages 30106–30148, 2024. arXiv:2402.19449.arxiv.org/abs/2402.19449
  26. [26]Ya Le and Xuan Yang. Tiny ImageNet visual recognition challenge. CS231N course project report, Stanford University, 2015. URL http://cs231n.stanford.edu/reports/2015/pdfs/yle_project.pdf.
  27. [27]Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. NuminaMath. https://github.com/project-numina/aimo-progress-prize/blob/main/report/numina_dataset.pdf, 2024. Hugging Face dataset: https://huggingface.co/datasets/AI-MO/NuminaMath-CoT.arxiv.org/abs/2510.05491
  28. [28]Zichong Li, Liming Liu, Chen Liang, Weizhuo Chen, and Tuo Zhao. NorMuon: Making Muon more efficient and scalable. In International Conference on Machine Learning (ICML), pages 68514–68533, 2026. arXiv:2510.05491.arxiv.org/abs/2510.05491
  29. [29]Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In International Conference on Learning Representations (ICLR), 2024. arXiv:2305.20050.arxiv.org/abs/2305.20050
  30. [30]Wu Lin, Scott C. Lowe, Felix Dangel, Runa Eschenhagen, Zikun Xu, and Roger B. Grosse. Understanding and improving Shampoo and SOAP via Kullback-Leibler minimization. In International Conference on Learning Representations (ICLR), 2026. arXiv:2509.03378.
  31. [31]Ziyue Liu, Ruijie Zhang, Zhengyang Wang, Yequan Zhao, Yupeng Su, Zi Yang, and Zheng Zhang. Muon²: Boosting Muon via adaptive second-moment preconditioning. arXiv preprint arXiv:2604.09967, 2026.
  32. [32]Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019. arXiv:1711.05101.arxiv.org/abs/1711.05101
  33. [33]Yanqing Lu, Leitao Wang, and Jinbo Liu. Understanding SOAP from the perspective of gradient whitening. arXiv preprint arXiv:2509.22938, 2025.
  34. [34]Chao Ma, Wenbo Gong, Meyer Scetbon, and Edward Meeds. SWAN: SGD with normalization and whitening enables stateless LLM training. In International Conference on Machine Learning (ICML), pages 41907–41942, 2025. arXiv:2412.13148.
  35. [35]James Martens and Roger Grosse. Optimizing neural networks with Kronecker-factored approximate curvature. In International Conference on Machine Learning (ICML), pages 2408–2417, 2015.
  36. [36]Depen Morwani, Itai Shapira, Nikhil Vyas, Eran Malach, Sham Kakade, and Lucas Janson. A new perspective on Shampoo’s preconditioner. In International Conference on Learning Representations (ICLR), 2025. arXiv:2406.17748.arxiv.org/abs/2406.17748
  37. [37]Guilherme Penedo, Hynek Kydlíček, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The FineWeb datasets: Decanting the web for the finest text data at scale. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, volume 37, pages 30811–30849, 2024. arXiv:2406.17557.
  38. [38]Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
  39. [39]Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning (ICML), pages 4596–4604, 2018. arXiv:1804.04235.
  40. [40]Hao-Jun Michael Shi, Tsung-Hsien Lee, Shintaro Iwasaki, Jose Gallego-Posada, Zhijing Li, Kaushik Rangadurai, Dheeraj Mudigere, and Michael Rabbat. A distributed data-parallel PyTorch implementation of the distributed Shampoo optimizer for training neural networks at-scale. arXiv preprint arXiv:2309.06497, 2023.
  41. [41]Chongjie Si, Debing Zhang, and Wei Shen. AdaMuon: Adaptive Muon optimizer. arXiv preprint arXiv:2507.11005, 2025.arxiv.org/abs/2507.11005
  42. [42]Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. RoFormer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. arXiv:2104.09864.
  43. [43]Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347–10357. PMLR, 2021.
  44. [44]Nikhil Vyas, Depen Morwani, Rosie Zhao, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham Kakade. SOAP: Improving and stabilizing Shampoo using Adam for language modeling. In International Conference on Learning Representations (ICLR), 2025. arXiv:2409.11321.arxiv.org/abs/2409.11321
  45. [45]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Ying­er Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.arxiv.org/abs/2505.09388
  46. [46]Ge Yang, Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tuning large neural networks via zero-shot hyperparameter transfer. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pages 17084–17097, 2021. arXiv:2203.03466 (extended arXiv version titled “Tensor Programs V: ...”).arxiv.org/abs/2203.03466
  47. [47]Biao Zhang and Rico Sennrich. Root mean square layer normalization. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019. arXiv:1910.07467.arxiv.org/abs/1910.07467
  48. [48]Minxin Zhang, Yuxuan Liu, and Hayden Schaeffer. Adam improves Muon: Adaptive moment estimation with orthogonalized momentum. arXiv preprint arXiv:2602.17080, 2026.arxiv.org/abs/2602.17080
  49. [49]Tianxiao Zhang, Wenju Xu, Bo Luo, and Guanghui Wang. Depth-wise convolutions in vision transformers for efficient training on small datasets. Neurocomputing, 617:128998, 2025. arXiv:2407.19394.DOI
  50. [50]Yushun Zhang, Congliang Chen, Ziniu Li, Tian Ding, Chenwei Wu, Diederik P. Kingma, Yinyu Ye, Zhi-Quan Luo, and Ruoyu Sun. Adam-mini: Use fewer learning rates to gain more. In International Conference on Learning Representations (ICLR), 2025. arXiv:2406.16793.arxiv.org/abs/2406.16793
  51. [51]Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. GaLore: Memory-efficient LLM training by gradient low-rank projection. In International Conference on Machine Learning (ICML), pages 61121–61143, 2024.arxiv.org/abs/2403.03507
  52. [52]Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. PyTorch FSDP: Experiences on scaling fully sharded data parallel. Proceedings of the VLDB Endowment, 16(12):3848–3860, 2023. arXiv:2304.11277.arxiv.org/abs/2304.11277
  53. [53]Hanqing Zhu, Zhenyu Zhang, Weny­an Cong, Xi Liu, Sem Park, Vikas Chandra, Bo Long, David Z. Pan, Zhangyang Wang, and Jinwon Lee. APOLLO: SGD-like memory saving. AdamW-level performance. In Proceedings of Machine Learning and Systems (MLSys), volume 7, 2025. arXiv:2412.05270.

Paper details

vlnc:W_5g85num5
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents