betaFeedback

WorksComputer Science

Do Pretraining Optimizer Gains Transfer to KL Distillation?

Abstract

Matrix-structured optimizers can improve language-model pretraining, but it is unclear whether these gains transfer to short-horizon KL distillation from an instruction-tuned initialization. We compare AdamW and diagonal AdaGrad with two tractable Shampoo reductions, SOAP, and All-2D Muon across four dense Qwen student-teacher pairs under a common forward-KL objective and fixed training budget. Across scales, none of the evaluated matrix-structured recipes consistently improves the quality versus trainer-time trade-off over the elementwise references: AdamW generally attains the lowest validation KL, while AdaGrad can be slightly faster at larger scale. SOAP often tracks AdamW in KL but incurs substantial trainer-time overhead, and the two Shampoo reductions reverse ordering across model pairs, showing that conclusions depend on the feasible recipe. These results suggest that pretraining optimizer gains do not transfer automatically to KL distillation and should be re-established for the target objective and systems regime.

Introduction

Prior work on Shampoo, SOAP, Muon, and related matrix-structured optimizers reports data-efficiency or wall-clock gains over AdamW in language-model pretraining [1, 6, 9, 10, 11]. Those comparisons concern long-horizon next-token training from random initialization, with outcomes sensitive to model scale, batch size, parameterization, and hyperparameter transfer [3, 10, 11]. KL distillation instead starts from an instruction-tuned student and uses the teacher’s full next-token probability distribution, rather than a one-hot target, at every supervised token. It also runs for a comparatively short horizon, while online teacher inference reduces the optimizer’s share of trainer time. Consequently, an optimizer advantage in pretraining does not necessarily imply lower validation KL per training step or per unit of trainer time in short-horizon distillation.

Within each student–teacher pair, all recipes share the tokenizer, data, forward-KL objective, batch size, and training budget; we compare minimum validation KL within budget, validation KL at matched steps, and trainer time. Canonical two-sided Shampoo [1] cannot be applied uniformly to the evaluated matrix geometries under the factor-dimension cap. We therefore derive a scalar-row-mean quarter-power reduction and compare it with a right-only half-power reduction based on Scalable Shampoo’s one-sided construction [3], treating both as specific feasible recipes rather than family-wide representatives.

The rest of the paper is structured as follows: Section 2 defines the distillation objective and optimizer recipes, including the Shampoo reductions. Section 3 describes the experimental design, and Section 4 reports fixed-budget quality, optimization trajectories, and trainer time. Section 5 discusses interpretation and limitations, and Section 6 concludes. Appendix A documents implementation fidelity, Appendix B grafting, and Appendix C hyperparameter selection.

Objective and optimizer recipes

KL distillation

Knowledge distillation matches a student distribution pθp_{\theta} to a frozen teacher distribution qq by minimizing forward KL at temperature one [13]. Let qn,tq_{n,t} and pθ,n,tp_{\theta,n,t} denote their distributions over the shared vocabulary at token position tt in example nn, and let mn,t∈{0,1}m_{n,t} \in\{0,1\} indicate a supervised assistant token. We report the validation token mean

Lval(θ)=1∑n,tmn,t∑n,tmn,tDKL(qn,t ∥ pθ,n,t).(1)\mathcal{L}_{\mathrm{val}}(\theta) = \frac{1}{\sum_{n,t} m_{n,t}} \sum_{n,t} m_{n,t} D_{\mathrm{KL}}(q_{n,t} \,\|\, p_{\theta,n,t}) . \tag*{(1)}

Validation weights tokens equally, so region contributions scale with token count and longer reasoning regions generally dominate. A smaller value indicates closer teacher matching. The blended training loss [14] avoids this between-region length dependence by weighting the answer and reasoning means separately within each example. For example nn, let AnA_n and CnC_n denote its nonempty sets of answer and reasoning tokens, and let ℓn,t=DKL(qn,t∥pθ,n,t)\ell_{n,t}=D_{\mathrm{KL}}(q_{n,t}\Vert p_{\theta,n,t}). Its loss is

Ln=λ1∣An∣∑t∈Anℓn,t+(1−λ)1∣Cn∣∑t∈Cnℓn,t(2)\mathcal{L}_n=\lambda\frac{1}{\lvert A_n\rvert}\sum_{t\in A_n}\ell_{n,t}+(1-\lambda)\frac{1}{\lvert C_n\rvert}\sum_{t\in C_n}\ell_{n,t} \tag*{(2)}

and training averages Ln\mathcal{L}_n across examples. Thus, λ\lambda weights the answer–region mean independently of region length. Prior evaluation identified λ=0.5\lambda=0.5 as the most stable answer–reasoning weighting [14]; we use it throughout. If either region is empty, the other receives unit weight. The objective contains no hard-label language-modeling term.

Matrix-structured pretraining optimizers

We use AdamW and diagonal AdaGrad as complementary elementwise references. AdamW reflects standard language-model practice, while AdaGrad connects directly to the structured preconditioners studied here; a different distillation formulation on Nemotron reasoning traces also found slightly higher MMLU-Pro with AdaGrad than with its best AdamW configuration [15]. AdaGrad accumulates a coordinatewise sum of squared gradients [2], while AdamW decouples weight decay from Adam’s adaptive update [21]. The comparison includes three matrix-structured optimizer families used in language-model pretraining. Across Shampoo, SOAP, and Muon, structured updates act on two-dimensional parameters as matrices rather than treating their entries as independent scalar coordinates.

Shampoo [1] constructs left- and right-side second-moment accumulators for a matrix gradient and applies inverse matrix roots on both sides, providing a tractable structured alternative to full-matrix AdaGrad [2]. Maintaining these accumulators and computing their inverse roots adds cost. SOAP [6] periodically derives eigenbases from Shampoo-style left and right accumulators and runs Adam in the resulting rotated coordinates; between basis refreshes, it continues updating coordinatewise second moments.

Muon [8] instead orthogonalizes a momentum matrix with a Newton–Schulz iteration. Its published recipe applies Muon to hidden-layer matrices and uses AdamW for embeddings and the output head [8]. Our All-2D Muon recipe instead applies the spectral update to every two-dimensional parameter, including embeddings and the output head, and uses Adam only for one-dimensional parameters. This assignment is part of the evaluated recipe rather than an implementation detail. We choose it to test the spectral update uniformly across matrix-valued student parameters, without routing the embedding and output matrices through a separate Adam-family optimizer. Published language-model studies report improved data efficiency or lower wall-clock cost for SOAP [6] and Muon [9, 10, 11] under their respective pretraining protocols. KL-Shampoo and KL-SOAP [7] use KL divergence to derive structured covariance estimators; that optimizer-internal construction is distinct from the teacher–student output KL studied here, and we do not evaluate those variants.

Shampoo variants and matrix geometry

Elementwise methods scale parameters independently. Full-matrix AdaGrad [2] captures correlations among all mnmn entries of a weight matrix but requires a preconditioner with (mn)2(mn)^2 entries. Shampoo [1] instead maintains one factor per matrix axis. For the gradient Gt∈Rm×nG_t\in\mathbb{R}^{m\times n} of a weight with mm row coordinates and nn column coordinates, define

Lt=ϵIm+∑τ=1tGτGτT,Rt=ϵIn+∑τ=1tGτTGτ.(3)L_t=\epsilon I_m+\sum_{\tau=1}^{t}G_\tau G_\tau^{\mathsf{T}},\qquad R_t=\epsilon I_n+\sum_{\tau=1}^{t}G_\tau^{\mathsf{T}}G_\tau. \tag*{(3)}

Lt∈Rm×mL_t\in\mathbb{R}^{m\times m} acts in row-coordinate space and Rt∈Rn×nR_t\in\mathbb{R}^{n\times n} acts in column-coordinate space. Storage falls from m2n2m^2n^2 entries to m2+n2m^2+n^2, while the full factors mix coordinates and can change the gradient direction rather than merely rescale entries independently. Here ϵ>0\epsilon>0 denotes regularization; the equations suppress the additional numerical damping used by the inverse-root routine.

Canonical matrix Shampoo applies both factors:

G~t=Lt−1/4GtRt−1/4.(4)\widetilde{G}_t=L_t^{-1/4}G_tR_t^{-1/4}. \tag*{(4)}

For row-major vectorization gt=vec⁡r(Gt)g_t=\operatorname{vec}_{r}(G_t), Equation (4) is equivalent to

vec⁡r(G~t)=(Lt1/4⊗Rt1/4)−1gt.(5)\operatorname{vec}_{r}(\widetilde{G}_t)=\left(L_t^{1/4}\otimes R_t^{1/4}\right)^{-1}g_t. \tag*{(5)}

If every GτG_{\tau} has rank at most rr, the Scalable Shampoo analysis [3] relates the full-matrix AdaGrad accumulator At=ϵImn+∑τ=1tgτgτTA_t=\epsilon I_{mn}+\sum_{\tau=1}^{t}g_{\tau}g_{\tau}^{\mathsf{T}} to these factors through the positive-semidefinite bound

At⪯rLt1/2⊗Rt1/2.(6)A_t\preceq rL_t^{1/2}\otimes R_t^{1/2}. \tag*{(6)}

Thus, the two quarter powers form a rank-dependent structured preconditioner motivated by a positive-semidefinite bound, not an exact factorization of the full accumulator. If both factors grow proportionally to tt, they give the usual heuristic decay t−1/4t−1/4=t−1/2t^{-1/4}t^{-1/4}=t^{-1/2} [1]. Equation (4) is the reference point below.

Under our fixed factor-dimension cap of 8192, the students’ matrix geometries [19, 20] permit right factors for more parameters than left factors. The embedding and output head have shape (vocabulary, hidden), making their left factor at least 151,936 square and their right factor at most 5120 square. The MLP gate and up projections have shape (intermediate, hidden), also placing the smaller hidden dimension on the right; the down projection reverses these dimensions, and every attention projection satisfies the cap. A right factor therefore captures correlations in the hidden-feature or input-column space for almost every two-dimensional weight without constructing vocabulary- or intermediate-width matrix roots. We select right-sided preconditioning for computational feasibility, not because column-space structure is assumed to be intrinsically more informative. Because transposition exchanges LtL_t and RtR_t, the left/right label is layout-dependent. Here right-only refers to the checkpoint’s PyTorch layout, in which linear weights are stored as (output, input) and embedding weights as (vocabulary, hidden).

The first variant replaces the left matrix factor by a scalar summary for every matrix-eligible two-dimensional weight, including weights whose full left factor would fit. Define that scalar row mean as

st=tr⁡(Lt)m=ϵ+1m∑τ=1t∥Gτ∥F2.(7)s_t=\frac{\operatorname{tr}(L_t)}{m}=\epsilon+\frac{1}{m}\sum_{\tau=1}^{t}\lVert G_{\tau}\rVert_{\mathrm{F}}^2. \tag*{(7)}

Replacing LtL_t by stIms_tI_m in Equation (4) gives

G~t=st−1/4GtRt−1/4.(8)\widetilde{G}_t=s_t^{-1/4}G_tR_t^{-1/4}. \tag*{(8)}

We call this scalar-row-mean quarter-power Shampoo. It preserves the mean eigenvalue of LtL_t and canonical Shampoo’s quarter-power structure [1] but removes every directional distinction in output-row space; it reproduces Lt−1/4L_t^{-1/4} as an operator only when Lt=stImL_t=s_tI_m. It has a right matrix factor, but it is not a pure right-only method.

The second variant follows the one-sided construction of Scalable Shampoo [3]. For p,q∈[1,∞]p,q\in[1,\infty] satisfying 1/p+1/q=11/p+1/q=1, with 1/∞=01/\infty=0, that construction uses

G~t=Lt−1/(2p)GtRt−1/(2q).(9)\widetilde{G}_t=L_t^{-1/(2p)}G_tR_t^{-1/(2q)}. \tag*{(9)}

The symmetric choice p=q=2p=q=2 recovers Equation (4). Taking p→∞p\to\infty, with q=p/(p−1)→1q=p/(p-1)\to1, reaches the endpoint (p,q)=(∞,1)(p,q)=(\infty,1), removes the left factor, and gives

G~t=GtRt−1/2.(10)\widetilde{G}_t=G_tR_t^{-1/2}. \tag*{(10)}

The half power is therefore the endpoint of the same Kronecker-factored full-matrix-AdaGrad surrogate, rather than an independent exponent choice. Scalable Shampoo introduces these one-sided endpoints for large layers, applies them to embedding, softmax, and projection layers, and proves an online-convex regret bound under uniformly rank-bounded gradients [3]. Omitting LtL_t discards directional information in output-row coordinate space, so this right-only half-power Shampoo is not merely a lower-cost implementation of the canonical preconditioner.

Thus, scalar-row-mean quarter-power Shampoo retains an isotropic left-side scale and a right quarter-power direction, whereas right-only half-power Shampoo removes the left factor and strengthens the right exponent.

For ideal, unstabilized Frobenius-norm grafting to a donor direction DD [12], write Q=GtRt−1/4≠0Q=G_tR_t^{-1/4}\ne0 and P=st−1/4QP=s_t^{-1/4}Q. Then

P∥D∥F∥P∥F=Q∥D∥F∥Q∥F.(11)P\frac{\lVert D\rVert_{\mathrm{F}}}{\lVert P\rVert_{\mathrm{F}}} = Q\frac{\lVert D\rVert_{\mathrm{F}}}{\lVert Q\rVert_{\mathrm{F}}}. \tag*{(11)}

so, up to the implementation’s 10−1610^{-16} denominator stabilizer, the scalar factor cancels and no scalar-row-mean information survives. We therefore exclude this grafted variant as equivalent to the corresponding normalized right-quarter direction. For right-only half-power Shampoo, normalization preserves the non-scalar Rt−1/2R_t^{-1/2} direction while replacing its norm with the donor norm, so the grafted and ungrafted recipes remain distinct. The main comparison uses the ungrafted update; Appendix § 8 discusses its grafted variant.

With blocking disabled, either evaluated recipe forms a right matrix factor only when its dimension is at most 8192. At 0.6B, every two-dimensional parameter satisfies this cap and receives the right factor. At 4B, 14B, and 32B, the MLP down projection has shape (hidden, intermediate), so its right factor exceeds the cap and the parameter is instead updated with diagonal AdaGrad. Embeddings, the output head, attention, and the gate and up projections remain right-matrix-preconditioned at every scale; one-dimensional parameters always use the diagonal update. Both recipes use this right-factor/fallback rule; Scalable Shampoo instead combines two-sided factors, selected one-sided factors, and blocking according to layer geometry [3].

With no momentum, cumulative statistics are updated before preconditioning on every optimizer step. On the first preconditioner step, the resulting roots are used for that same update. Subsequent refreshes are measured relative to this step, and the roots are held fixed between refreshes. Before activation, matrix-preconditioned parameters use the raw gradient; fallback parameters use the diagonal update throughout. The learning rate and both schedule values are selected separately for each method and student–teacher pair. Appendix § 9 gives the selected values, and Appendix § 7 gives the numerical root procedure.

Experimental design

Models and data

The first three pairs use the instruction-tuned Qwen3-0.6B, Qwen3-1.7B, Qwen3-4B, Qwen3-8B, Qwen3-14B, and Qwen3-32B checkpoints [19]: 0.6B ← 1.7B, 4B ← 8B, 14B ← 32B. The fourth pair uses Qwen2.5-32B-Instruct and Qwen2.5-72B-Instruct [20]. We use Qwen2.5 for the largest pair because its released dense 32B and 72B checkpoints have a student–teacher size ratio comparable to those of the Qwen3 pairs. The Qwen3 checkpoints share a tokenizer and vocabulary; the two Qwen2.5 checkpoints likewise have byte-identical tokenizer artifacts. Although the tokenizer and vocabulary differ between Qwen3 and Qwen2.5, the student and teacher within each pair receive identical token IDs.

Each student is a dense decoder-only transformer. The hidden and MLP intermediate sizes are respectively (1024, 3072), (2560, 9728), (5120, 17408), and (5120, 27468) for the 0.6B, 4B, 14B, and Qwen2.5 32B students. The Qwen3 and Qwen2.5 vocabularies contain 151,936 and 152,064 entries, respectively. The 0.6B and 4B checkpoints tie the token embedding and output head; the 14B and Qwen2.5 32B checkpoints use separate matrices of the same shape. These dimensions determine the factor coverage described in § 2.3.

The examples are chain-of-thought traces selected from NVIDIA’s Nemotron-Post-Training-Dataset-v2{2} [22, 23]. We retain reasoning-on SFT traces with complete <think> and </think> markup and create training and validation splits stratified by the dataset’s category field. The retained <think> and </think> delimiters define the reasoning and answer regions in Equation (2). We apply a maximum sequence length of 8192. Under both Qwen3 and Qwen2.5 tokenization, removal of overlength traces leaves 79,535 training examples and 1,006 unique validation examples.

Comparison and evaluation protocol

We compare six distillation recipes: AdamW [21] and diagonal AdaGrad [2] as references; scalar-row-mean quarter-power and right-only half-power Shampoo; SOAP [6]; and All-2D Muon [8]. All-2D Muon applies its spectral update to every two-dimensional parameter and Adam to one-dimensional parameters.

AdamW and AdaGrad use native PyTorch implementations; the remaining methods use PyTorch/FSDP2 adaptations. SOAP matches the authors’ implementation in fixed-gradient parity tests. Trainer times are specific to these adapters.

Every full-budget final configuration is evaluated with three seeds. Qwen2.5 SOAP is limited to one matched 561-step run because of its measured trainer-time overhead and is excluded from full-budget summaries. Appendices A–C document implementation details, grafting controls, selection, and final settings.

Within each student–teacher pair, every method uses the forward-KL training loss in Equation (2), with λ=0.5\lambda= 0.5, and global batch 144. Weight decay is zero, no learning-rate scheduler or warmup is used, and gradients are clipped to unit norm. Training uses bfloat16 FSDP2 with float32 gradient reduction. Because weight decay is zero, the PyTorch AdamW update is equivalent here to Adam; we retain the AdamW label.

All methods use the same fixed budget of 1683 optimizer updates. One effective epoch is 552 updates, the number of complete batches of 144 in the 79,535-example training split, so the budget runs slightly past three effective epochs. Pilot runs showed diminishing validation-KL improvements toward the end of this horizon.

The three Qwen3 pairs use 72 NVIDIA GB200 accelerators across 18 four-accelerator nodes. The larger Qwen2.5 32B ←\leftarrow 72B pair uses 144 accelerators across 36 such nodes to accommodate the dense teacher and student. We use trainer time for elapsed time spent in training, validation, and improvement-triggered checkpointing; it excludes model loading, queueing, and final model export.

We evaluate quality using two criteria. The within-budget criterion is the three-seed mean of each run’s minimum validation KL observed during 1683 steps, corresponding to validation-based checkpoint selection. The matched-step criterion compares the three-seed mean trajectories at common evaluation steps. Absolute KL is compared only within a student–teacher pair. We report means with one standard deviation and interpret both quality views together with trainer time rather than combine them into a single score. Comparisons are descriptive and use neither hypothesis tests nor a fixed threshold for small KL differences. We describe an advantage as consistent only when the same qualitative ordering recurs across model pairs.

Results

Validation KL within a fixed budget

Within the fixed budget, the evaluated matrix-structured methods show no validation-KL advantage over AdamW that persists across model pairs (Figure 1).

Figure 1. Minimum validation KL within 1683 steps: three-seed means of per-seed minima, with one-standard-deviation error bars. Lower is better. Qwen3 pairs use 72 GPUs and Qwen2.5 uses 144; Qwen2.5 SOAP was not extended beyond one matched 561-step run.

AdamW, SOAP, and All-2D Muon are closely grouped at 0.6B; AdamW is lower than SOAP and All-2D Muon at 4B and 14B, and lower than AdaGrad on all four pairs. The Shampoo comparison is pair-specific. Each Shampoo point is a pair-specific, validation-selected configuration (Appendix § 9), not a family-wide optimum under an equal tuning budget. Right-only half-power Shampoo has lower validation KL at 0.6B, whereas scalar-row-mean quarter-power is lower at 4B; their nearly identical trainer-time ratios indicate recipe dependence rather than a quality–trainer-time trade-off. Scalar-row-mean quarter-power remains above AdamW at 14B and Qwen2.5 32B. The selected k=10k = 10 right-only half-power configuration is likewise above AdamW at 14B, and the Qwen2.5 32B configuration is above both AdamW and scalar-row-mean quarter-power. Given that the teacher, student architecture, matrix geometry, and factor coverage change together, these cross-pair differences do not establish a scale trend.

Optimization trajectories across model pairs

Matched-step trajectories show that optimizer ordering can depend on training horizon, but no evaluated matrix-structured method shows a stable early-step advantage across model pairs (Figure 2). Across methods, validation KL declines most steeply during the first effective epoch and more slowly during the next two.

Validation KL over training for four model pairs

Figure 2. Validation KL over training (three-seed means). Dashed lines mark effective epochs of 552 updates; training ends at step 1683. Vertical scales differ. Qwen3 pairs use 72 GPUs and Qwen2.5 uses 144; Qwen2.5 SOAP was not extended beyond one matched 561-step run.

At 0.6B, AdaGrad and right-only half-power Shampoo are below AdamW after roughly one effective epoch, but these advantages disappear by the end of the budget. Scalar-row-mean quarter-power Shampoo remains above AdamW at every recorded validation step; most of its initial gap closes during the first effective epoch, but a positive gap remains at the final update.

At 4B and 14B, AdamW has lower validation KL than every other completed method at each effective-epoch boundary. SOAP approaches AdamW over training but does not overtake it at any recorded step on those pairs. AdamW likewise remains lowest at each boundary on the Qwen2.5 pair, where both Shampoo reductions remain above it. Thus, the trajectories do not identify an advantage for matrix-structured methods that is hidden by the fixed-budget summary.

Trainer-time overhead

Trainer-time results sharpen the quality comparison: every matrix-structured method with completed results adds trainer time relative to AdamW, whereas AdaGrad becomes slightly faster than AdamW at the larger evaluated student sizes (Figure 3). This is consistent with AdaGrad’s lighter elementwise update: it maintains one accumulator rather than AdamW’s first- and second-moment states, reducing optimizer-state traffic and pointwise work. The Qwen2.5 experiments use 144 GPUs, however, and therefore do not provide a pure scale continuation of the fixed 72-GPU regime used for Qwen3.

Figure 3. Trainer time normalized per pair and seed to AdamW (1.0); bars show three-seed means with one-standard-deviation error bars. Qwen3 pairs use 72 GPUs and Qwen2.5 uses 144; ratios are within-pair. The matched 561-step Qwen2.5 SOAP run took 4.31×\times AdamW trainer time and was not extended.

Among the evaluated matrix-structured methods, the two Shampoo reductions have the smallest overhead and nearly identical trainer time on their shared completed pairs. Their reversal in validation KL therefore reflects the optimizer recipe rather than different compute budgets. All-2D Muon becomes more expensive on the larger pairs. SOAP is qualitatively different: it requires roughly two to three times AdamW’s trainer time on the Qwen3 pairs.

Online teacher inference, validation, and checkpointing reduce the optimizer’s share of total time. These ratios therefore measure systems-level protocol overhead, not student-only optimizer-kernel overhead, and may understate the latter.

At Qwen2.5 32B ←\leftarrow 72B, SOAP incurred substantially greater trainer time than AdamW without a clear improvement in validation KL at the matched 561-step horizon. Its unblocked FSDP2 adapter gathers full gradients and moments, replicates matrix factors, and refreshes bases frequently (Appendix § 7). Model sharding does not distribute this optimizer work, so we did not extend SOAP.

Discussion and limitations

The transfer gap may reflect the shift from long-horizon pretraining to short-horizon distillation from an instruction-tuned initialization. Matrix-structured methods add estimation and refresh costs, leaving less opportunity for quality gains to offset trainer-time overhead; the study does not isolate these mechanisms causally.

That reversal comes from approximations that retain different information and exponents, with factor coverage and selected hyperparameters that also vary across pairs.

The SOAP and Muon conclusions are similarly tied to routing and implementation choices. All-2D Muon differs from the hidden-layer-only reference recipe, and unblocked SOAP differs algorithmically and operationally from blocked distributed alternatives. Moreover, Shampoo and SOAP use observed-gradient moments, which need not approximate the model’s Fisher information matrix [17]. The findings therefore do not rank optimizer families independently of factorization, routing, and implementation.

Limitations. The empirical scope is four dense Qwen student–teacher pairs from two model generations and one chain-of-thought corpus. The study does not test longer schedules, other distillation objectives or data, or sparse model architectures. Validation KL measures teacher matching rather than downstream performance.

The evaluated Shampoo recipes are unblocked and use at most one full right matrix factor.

Trainer times are specific to the 72- and 144-accelerator GB200 allocations and depend on the evaluated adapters. Qwen2.5 SOAP is deliberately limited to one matched 561-step run and is excluded from full-budget summaries. The study also includes no matched controls initialized from checkpoints pretrained with each evaluated optimizer.

Conclusion

Simple elementwise optimizers remain strong baselines for short-horizon online KL distillation. Across four Qwen student–teacher pairs, AdamW generally delivered the lowest validation KL, while AdaGrad provided comparable efficiency and could be slightly faster for the larger LLMs evaluated. The evaluated matrix-structured methods did not produce a persistent quality–trainer-time advantage, and the changing order of the two Shampoo reductions showed that the exact feasible formulation matters. For short-horizon online KL distillation, AdamW and AdaGrad should therefore be the starting point, with matrix structure added only when it yields a clear quality or trainer-time benefit in the target setting.

Optimizer implementations used

AdamW and AdaGrad use torch.optim; the matrix-structured methods use the PyTorch/FSDP2 implementations below. Method labels denote these specific implementations, not entire optimizer families. Code for these experiments will be released. Our main model-distillation-recipes repository3 documents the surrounding distillation protocol, including data formatting and token masking. Alternatives prefixed by “Meta” use the public Meta Distributed Shampoo repository4 [5]. The implementation controls cover the three Qwen3 model pairs: 0.6B ← 1.7B, 4B ← 8B, and 14B ← 32B. These controls identify faithful, practical implementations rather than estimate scaling trends, so we do not repeat them on the Qwen2.5 pair.

Shampoo

The two Shampoo recipes in the main comparison use our PyTorch/FSDP2 Scalable Shampoo backend. It ports selected equations and state-machine behavior from the Google Research implementation5, written in JAX [4] and released with Scalable Shampoo [3]. The port retains epsilon-initialized Kronecker statistics, cumulative or exponential-moving-average updates, separate statistic and inverse-root schedules, delayed activation, SGD/AdaGrad norm grafting, separate donor and Shampoo momentum, and the coupled-Newton inverse-root equations. It is not a line-by-line translation: optimizer calls are indexed from one, only two-dimensional parameters use Kronecker factors, and the largest-eigenvalue estimate is deterministic across ranks.

The pinned Google implementation does not expose omission of an individual Kronecker factor. Following the one-sided construction in the paper, we add the right-only half-power update. The same backend also supports a right-only quarter-power ablation and an AdaGrad-grafted right-only half-power update, as well as scalar-sum and scalar-row-mean quarter-power variants. Each scalar variant maintains one full-parameter scalar left statistic even when the right factor is partitioned into blocks. For an m×nm \times n gradient, the sum variant accumulates ∥Gt∥F2\lVert G_t\rVert_F^2 and the row-mean variant accumulates ∥Gt∥F2/m\lVert G_t\rVert_F^2/m. Non-matrix parameters and matrices whose active factor exceeds the configured cap instead use diagonal AdaGrad and maintain no scalar left statistic. The port further adds FSDP2 execution, distributed checkpointing, and recipe fingerprints that reject incompatible optimizer state. The pinned revision contains only the JAX implementation. Its exponent override assigns one power to every Kronecker factor, so the one-sided updates above are part of the port, as are the scalar statistics and the distributed interfaces. A later public PyTorch port of the same implementation6 retains the single-exponent interface and does not provide distributed execution. Meta Distributed Shampoo provides a right-only half-power update and distributed execution in PyTorch, with the recipe differences recorded below.

For the reported runs, each matrix statistic is initialized with 10−6I10^{-6}I, and each inverse root is computed after adding 10−6λ^max⁡(A)I10^{-6}\widehat{\lambda}_{\max}(A)I to the statistic AA. The largest-eigenvalue estimate uses a fixed seed and a relative convergence test. FP32 coupled Newton is used for the inverse root. A nonfinite candidate or a final coupled-iteration error of at least 0.1 is rejected, leaving the preceding valid root in place. The scalar statistic is initialized at 10−610^{-6}, and its inverse fourth root follows the matrix-root schedule. We use ϵ=10−10\epsilon= 10^{-10} for diagonal AdaGrad, both as the fallback update and as the grafting donor.

Two execution paths use this backend. Replicated Scalable Shampoo reconstructs each full gradient and repeats the optimizer computation on every rank. Parameter-owner Scalable Shampoo assigns each full parameter to one deterministic, load-balanced owner rank, where its statistics, roots, grafting, and update are computed once. It reuses the pinned Meta repository’s FSDP2 communication contexts, including persistent buffers and coalesced all-to-all exchanges, but retains our Scalable Shampoo mathematics and checkpoint schema. Updated parameter shards are returned to their FSDP2 ranks; owner-only factors are replicated only while saving or loading a checkpoint. In a matched single-seed, 561-update control, parameter-owner Scalable Shampoo was approximately 2.24×2.24\times faster in trainer time than replicated Scalable Shampoo, while validation KL changed by less than 2×10−42 \times10^{-4}.

Meta Distributed Shampoo can also construct right-only half-power Shampoo and its AdaGrad-grafted variant by combining per-dimension exponent overrides with a graft donor; our Meta adapter configures these lower-level options. They are not equivalent off-the-shelf presets for the recipes used in the main comparison.

In particular, the pinned Meta backend does not expose our full-parameter scalar-row-mean construction. Moreover, when blocking is enabled, Meta applies grafting per block; our Meta adapter couples first activation to the inverse-root refresh interval; and its default inverse-root implementation uses eigendecomposition rather than coupled Newton. We therefore use parameter-owner Scalable Shampoo for the main recipes and reserve Meta Distributed Shampoo as an implementation alternative.

Figure 4 compares a grafted right-only configuration of replicated Scalable Shampoo, parameter-owner Scalable Shampoo, and Meta Distributed Shampoo. Parameter-owner Scalable Shampoo and Meta Distributed Shampoo substantially reduce trainer time relative to replicated Scalable Shampoo. Their validation behavior is recipe-and pair-dependent rather than uniformly worse for Meta Distributed Shampoo. Because root schedules and selected hyperparameters differ among these historical controls, the figure documents implementation selection rather than an isolated backend benchmark.

Figure 4. Validation-KL trajectories for grafted right-only Shampoo. Replicated Scalable Shampoo and parameter-owner Scalable Shampoo are three-seed means; Meta Distributed Shampoo is a single-seed control with additional recipe differences.

SOAP

SOAP rotates gradients and moments into Shampoo-derived eigenbases before applying Adam along the resulting matrix directions [6]. The main SOAP implementation is an unblocked FSDP2 adaptation of the authors’ implementation7 and matches it in fixed-gradient parity tests. It constructs a factor for each matrix axis of dimension at most 8192, retains the identity basis along larger axes, and initializes the basis on its first optimization step without updating the parameters. Meta blockwise SOAP instead uses Meta’s DefaultSOAPConfig8, partitions matrix dimensions into blocks of size at most 1024, begins preconditioning at the first eigenbasis refresh on update 10, and distributes the blocked optimizer state across ranks. Its blocking and delayed use of the learned eigenbasis change the optimizer sequence, not only its systems implementation.

We use the unblocked implementation because it more closely follows the authors’ implementation, retaining Meta blockwise SOAP as an alternative. Under FSDP2, its full-matrix gradients, moments, factors, and bases are replicated rather than partitioned. Across the three controls in Figure 5, Meta blockwise SOAP achieved a 1.61.6–1.9×1.9\times trainer-time speedup over the unblocked implementation but had best validation KL values 0.006–0.013 higher. Because blocking and delayed preconditioning alter the update sequence, this is a comparison of alternative recipes rather than an isolated backend benchmark.

Validation-KL trajectories for unblocked SOAP and Meta blockwise SOAP

Figure 5. Validation-KL trajectories for unblocked SOAP (three-seed mean) and Meta blockwise SOAP (single-seed control). The comparison includes backend and algorithmic differences, including Meta’s matrix blocking and delayed basis use.

Muon

Muon approximately orthogonalizes matrix-valued momentum updates with Newton–Schulz iterations. A shared distributed adapter reconstructs each sharded two-dimensional gradient across ranks and redistributes the update. All-2D Muon applies Muon to embeddings, the output head, and transformer matrices; only one-dimensional parameters use Adam. Reference Muon instead follows the parameter assignment of the reference implementation9: Muon is restricted to hidden transformer matrices, while embeddings, the output head, and one-dimensional parameters use Adam. Meta Muon implements the same hidden-only assignment with Meta’s SpectralDescentPreconditionerConfig10 and distributes optimizer state across ranks. The three implementations otherwise share the momentum coefficient, Nesterov ordering, Newton–Schulz coefficients, and aspect-ratio scaling.

In matched single-seed controls, Meta Muon achieved a 1.2–1.7× trainer-time speedup over Reference Muon, while their best validation KL values differed by at most 0.00031 (Figure 6). This supports agreement between the hidden-only implementations; it does not validate All-2D Muon, whose distinct routing means that comparisons involving its curves reflect both recipe and implementation differences.

Validation-KL trajectories for All-2D Muon, Reference Muon, and Meta Muon

Figure 6. Validation-KL trajectories for All-2D Muon (three-seed mean) and the single-seed Reference and Meta controls. The controls apply Muon only to hidden-layer matrices; All-2D also includes embeddings and the output head.

AdaGrad-grafted right-only half-power Shampoo

Norm grafting takes the update direction from one optimizer and rescales it to match the update norm of another [12]. Here, Pt=GtRt−1/2P_t = G_t R_t^{-1/2} is the right-only half-power direction for an eligible matrix, and DtD_t is the diagonal AdaGrad-preconditioned direction before the outer learning-rate multiplier. After preconditioner activation, the grafted variant uses

G~tgraft=Pt∥Dt∥F∥Pt∥F+10−16.(12)\widetilde{G}^{\mathrm{graft}}_t = P_t \frac{\lVert D_t\rVert_F}{\lVert P_t\rVert_F + 10^{-16}}. \tag*{(12)}

Thus grafting retains the matrix-preconditioned direction while matching its Frobenius norm to the donor up to the stabilizer. Before activation, eligible matrices use the donor update; parameters outside the matrix recipe always use diagonal AdaGrad. Unlike scalar-row-mean quarter-power grafting, this operation does not cancel the matrix preconditioner.

Grafting changes the recipe, not the backend: grafted and ungrafted variants share the parameter-owner Scalable Shampoo execution path, cumulative statistics, coupled-Newton roots, and parameter routing. Grafting changes update scale, so its learning rate and preconditioner schedule are selected independently; the comparison is therefore between tuned recipes, not an isolated fixed-hyperparameter grafting effect. The main comparison reports the ungrafted variant. Completed single-seed grafted runs are exploratory and cannot characterize seed variability or support scale-dependent conclusions.

Hyperparameter selection and final configurations

Selection protocol

AdamW, AdaGrad, SOAP, and All-2D Muon were screened in short single-seed pilots before promising settings were promoted to longer runs. Learning rates were searched on logarithmically spaced grids spanning 3×10−63 \times10^{-6}–3×10−43 \times10^{-4} for AdamW, 3×10−53 \times10^{-5}–3×10−33 \times10^{-3} for AdaGrad, 3×10−63 \times10^{-6}–3×10−43 \times10^{-4} for SOAP, and 3×10−53 \times10^{-5}–3×10−33 \times10^{-3} for All-2D Muon, with boundary refinement when a pilot optimum approached a grid limit. AdamW additionally compared β1∈{0.8,0.9,0.95}\beta_{1} \in\{0.8, 0.9, 0.95\} and β2∈{0.95,0.99}\beta_{2} \in\{0.95, 0.99\}, while SOAP compared β1∈{0.8,0.9,0.95}\beta_{1} \in\{0.8, 0.9, 0.95\}. Candidate configurations were ranked by the minimum validation KL observed within their common stage budget. The initial 0.6B ←\leftarrow 1.7B screens used global batch 96; final runs use global batch 144. Subsequent pair-specific pilots searched within the same ranges. These searches select credible settings within each recipe rather than impose an equal tuning budget across methods.

The Shampoo recipes use pair-specific staged searches over learning rate, first preconditioner step, and refresh interval. After the 561-update screens, the three best valid configurations are each run for 1683 updates with three seeds. For each recipe and pair, the configuration with the lowest three-seed mean validation KL is selected; results are not pooled across hyperparameters.

Exploratory single-seed pilots also screened Sophia-G [18], motivated by its reported pretraining efficiency, and K-FAC [16], motivated by the relationship between KL and Fisher geometry. Sophia-G did not improve the observed validation-KL–trainer-time trade-off over the retained recipes, and K-FAC did not outperform AdamW in its control. Neither method was promoted to the main comparison.

Final configurations

Table 1 records the final pair-specific values.

MethodQwen3 0.6B←\leftarrow1.7BQwen3 4B←\leftarrow8BQwen3 14B←\leftarrow32BQwen2.5 32B←\leftarrow72B
Learning rate
AdamW3×10−53 \times10^{-5}3×10−53 \times10^{-5}3×10−53 \times10^{-5}3×10−53 \times10^{-5}
AdaGrad3×10−43 \times10^{-4}3×10−43 \times10^{-4}3×10−43 \times10^{-4}1.5×10−41.5 \times10^{-4}
Scalar-row-mean quarter-power Shampoo7×10−37 \times10^{-3}5×10−45 \times10^{-4}5×10−45 \times10^{-4}5×10−45 \times10^{-4}
Right-only half-power Shampoo10−210^{-2}3×10−43 \times10^{-4}3×10−43 \times10^{-4}3×10−43 \times10^{-4}
SOAP3×10−53 \times10^{-5}3×10−53 \times10^{-5}10−410^{-4}10−410^{-4}†
All-2D Muon3×10−43 \times10^{-4}3×10−43 \times10^{-4}3×10−43 \times10^{-4}5×10−45 \times10^{-4}
Shampoo first step/refresh interval
Scalar-row-mean quarter-power Shampoo20/20050/10020/10020/100
Right-only half-power Shampoo10/10100/10010/1050/50

Table 1. Pair-specific learning rates and, for each Shampoo recipe, the first preconditioner step and refresh interval (in updates; reported as first/refresh).

The remaining fixed optimizer-specific settings are:

  • AdamW: (β1,β2)=(0.8,0.99)(\beta_{1}, \beta_{2}) = (0.8, 0.99) and ϵ=10−8\epsilon= 10^{-8}.

  • AdaGrad: ϵ=10−10\epsilon= 10^{-10} and zero initial accumulator. Unlisted AdamW and AdaGrad arguments use PyTorch defaults.

  • Both Shampoo recipes: cumulative statistics updated every step, no momentum or grafting, maximum matrix-factor dimension 8192, matrix-statistic initialization 10−610^{-6}, and diagonal-fallback ϵ=10−10\epsilon= 10^{-10}. Scalar-row-mean quarter-power Shampoo also initializes its scalar statistic to 10−610^{-6}.

  • SOAP: (β1,β2)=(0.8,0.95)(\beta_{1}, \beta_{2}) = (0.8, 0.95), Adam-denominator ϵ=10−6\epsilon= 10^{-6}, factor EMA coefficient 0.95, and eigenbasis refresh every 10 updates.

  • All-2D Muon: momentum 0.95 with Nesterov acceleration, five bfloat16 Newton–Schulz iterations, and one-dimensional Adam with (β1,β2)=(0.9,0.95)(\beta_{1}, \beta_{2}) = (0.9, 0.95) and ϵ=10−10\epsilon= 10^{-10}.

References

  1. [1]V. Gupta, T. Koren, and Y. Singer. Shampoo: Preconditioned stochastic tensor optimization. ICML, 2018.arxiv.org/abs/1802.09568
  2. [2]J. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 2011.
  3. [3]R. Anil, V. Gupta, T. Koren, K. Regan, and Y. Singer. Scalable second order optimization for deep learning. arXiv:2002.09018, 2020.arxiv.org/abs/2002.09018
  4. [4]J. Bradbury, R. Frostig, P. Hawkins, et al. JAX: Composable transformations of Python+NumPy programs. Software repository, 2018.
  5. [5]H.-J. M. Shi, T.-H. Lee, S. Iwasaki, J. Gallego-Posada, Z. Li, K. Rangadurai, D. Mudigere, and M. Rabbat. A distributed data-parallel PyTorch implementation of the Distributed Shampoo optimizer for training neural networks at-scale. arXiv:2309.06497, 2023.arxiv.org/abs/2309.06497
  6. [6]N. Vyas, D. Morwani, R. Zhao, M. Kwun, I. Shapira, D. Brandfonbrener, L. Janson, and S. Kakade. SOAP: Improving and stabilizing Shampoo using Adam for language modeling. ICLR, 2025.arxiv.org/abs/2409.11321
  7. [7]W. Lin, S. C. Lowe, F. Dangel, R. Eschenhagen, Z. Xu, and R. B. Grosse. Understanding and improving Shampoo and SOAP via Kullback–Leibler minimization. ICLR, 2026.arxiv.org/abs/2509.03378
  8. [8]K. Jordan, Y. Jin, V. Boza, J. You, F. Cesista, L. Newhouse, and J. Bernstein. Muon: An optimizer for hidden layers in neural networks. Technical note, 2024.
  9. [9]J. Liu, J. Su, X. Yao, et al. Muon is scalable for LLM training. arXiv:2502.16982, 2025.arxiv.org/abs/2502.16982
  10. [10]I. Shah, A. M. Polloreno, K. Stratos, et al. Practical efficiency of Muon for pretraining. arXiv:2505.02222, 2025.arxiv.org/abs/2505.02222
  11. [11]S. Qiu, Z. Chen, H. Phan, Q. Lei, and A. G. Wilson. Hyperparameter transfer enables consistent gains of matrix-preconditioned optimizers across scales. NeurIPS, 2025.arxiv.org/abs/2512.05620
  12. [12]N. Agarwal, R. Anil, E. Hazan, T. Koren, and C. Zhang. Disentangling adaptive gradient methods from learning rates. arXiv:2002.11803, 2020.arxiv.org/abs/2002.11803
  13. [13]G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. arXiv:1503.02531, 2015.arxiv.org/abs/1503.02531
  14. [14]L. Ghita, D. Desai, and I. Boier. Scaling laws for task-specific LLM distillation. arXiv:2606.24747, 2026.arxiv.org/abs/2606.24747
  15. [15]L. Ghita and D. Desai. Effective teacher support for reasoning distillation. arXiv:2611.14850, 2026.arxiv.org/abs/2611.14850
  16. [16]J. Martens and R. Grosse. Optimizing neural networks with Kronecker-factored approximate curvature. ICML, 2015.arxiv.org/abs/1503.05671
  17. [17]F. Kunstner, L. Balles, and P. Hennig. Limitations of the empirical Fisher approximation for natural gradient descent. NeurIPS, 2019.arxiv.org/abs/1905.12558
  18. [18]H. Liu, Z. Li, D. Hall, P. Liang, and T. Ma. Sophia: A scalable stochastic second-order optimizer for language model pre-training. ICLR, 2024.arxiv.org/abs/2305.14342
  19. [19]Qwen Team. Qwen3 technical report. arXiv:2505.09388, 2025.
  20. [20]Qwen Team. Qwen2.5 technical report. arXiv:2412.15115, 2024.
  21. [21]I. Loshchilov and F. Hutter. Decoupled weight decay regularization. ICLR, 2019.
  22. [22]D. Nathawani, S. Ding, V. Lavrukhin, I. Gitman, S. Majumdar, E. Bakhturina, B. Ginsburg, and J. Polak Scowcroft. Nemotron-Post-Training-Dataset-v2. Version 2.0, NVIDIA, 2025.
  23. [23]NVIDIA. NVIDIA Nemotron Nano 2: An accurate and efficient hybrid Mamba–Transformer reasoning model. arXiv:2508.14444, 2025.

Paper details

vlnc:W_9mc6sn2f
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents