betaFeedback

WorksComputer Science

Your LR Schedule May Not Be Principled: Effective Learning Rate Scheduling in Weight-Normalized Training

Abstract

Weight-normalized (WN) training is emerging as a compelling alternative to conventional weight decay (WD), with recent studies reporting substantial pretraining loss gains. We show that these gains are driven primarily by the induced effective learning rate (ELR) trajectory rather than by normalization alone. Matching ELR trajectories largely collapses the training loss dynamics of WN and WD. Schedule design should therefore target ELR rather than the nominal learning rate (LR) alone. Building on this principle, we propose AIR (Affine Inverse-square-Root decay), which combines a theoretically derived 1/t1/\sqrt{t} decay with an affine adjustment to satisfy prescribed peak and terminal LRs over a finite training horizon. We further identify late-stage ELR scheduling as a key source of its gains, because a shared late schedule attenuates the effect of earlier LR choices. Experiments on 128 B-token Mixture-of-Experts pretraining demonstrate that AIR outperforms cosine and other candidate schedules in training loss, downstream performance, and hyperparameter robustness, reducing terminal training loss by approximately 0.01 relative to cosine at matched cumulative LR. Applying the same ELR trajectory to WD reproduces the corresponding training dynamics, showing that the benefits extend beyond fixed-norm training and establishing ELR as the principled scheduling object.

Introduction

In Large Language Model (LLM) pretraining, weight-normalized (WN) training [23, 33] has emerged as a promising alternative to conventional weight decay (WD) [20]. It keeps parameter norms fixed after each update, with recent work reporting lower pretraining loss, improved token efficiency, greater stability, and better hyperparameter transfer [1, 18, 33]. Yet WN and WD exhibit markedly different training dynamics under the same learning rate (LR) schedule. As shown in Figure 1(a), under cosine decay [19], WN initially trails WD but eventually outperforms it. However, under warmup-stable-decay (WSD) [10], WN remains behind WD throughout training and ends with a substantially higher loss. Therefore, understanding this contrast is essential to explaining when and why WN outperforms WD.

Training loss, all-parameter norm, and all-parameter effective learning rate under cosine and WSD schedules

Figure 1. WN achieves lower final loss than WD under cosine, but significantly higher under WSD. Effective LR (nominal LR / all-parameter norm) is the key driver of loss dynamics.

We attribute this contrast to the effective learning rate (ELR) [5, 29], defined as ηeff=η/r\eta^{\mathrm{eff}} = \eta/ r, where η\eta is nominal LR and rr is the norm of a parameter block. In our study, we discover that the loss dynamics is strongly related to the decay of ELR. When WN and WD share the same WSD LR schedule, their ELR is governed entirely by the parameter norm rr, which evolves very differently in the two regimes (Figure 1(b,Figure 1c)). WN fixes the parameter norm, holding its ELR constant at peak throughout the entire stable phase and leading to a slow decrease in loss. By contrast, WD’s ELR keeps falling steadily, owing to the continuing growth of its parameter norm. This sustained ELR decay keeps its loss dropping much faster than WN.

A more interesting finding is that the loss dynamics of WN and WD largely collapse onto a common trajectory once we align their ELR trajectories, whether by replaying WD’s norms or by rescaling one’s blockwise LRs. (see Figure 2 and § 2.2 for details). Concurrent studies [17, 34, 38] also report similar results, corroborating our findings.

Training loss relative to the WD baseline under cosine and AIR

Figure 2. ELR matching collapses WD–WN training loss dynamics. Training loss minus the WD baseline under (a) cosine or (b) AIR, paired at the same step; zero denotes the reference. Arrows indicate the ELR target, matched by replaying WD norms (via norm) or rescaling blockwise LR (via LR). WD→WN also replays WD norms.

Taken together, these findings lead us to our core conclusion: ELR is the key driver of the loss dynamics. WN’s core advantage actually comes from a better ELR schedule, not normalization alone. Principled LR schedule design should therefore target ELR rather than the nominal LR alone. This core idea naturally inspires an essential research question:

What should a better ELR schedule look like?

To address this question, we study ELR scheduling under WN, a clean setting in which each block’s ELR is proportional to its nominal LR. Here, we find that 1/t1/\sqrt{t} decay further improves training. A theoretical analysis of WN dynamics shows that, in the unconstrained setting, 1/t1/\sqrt{t} decay maximizes alignment with an oracle direction at every step. To make this practical, we introduce AIR (Affine Inverse-square-Root decay), which adapts this 1/t1/\sqrt{t} decay through an affine transformation to meet prescribed peak and terminal LRs over a finite horizon. A further analysis shows how a shared late-stage schedule attenuates the alignment differences induced by earlier LR choices. This helps explain our finding that late-stage ELR scheduling is a key source of AIR’s gains.

Empirically, we evaluate AIR against cosine decay and alternative LR schedule families. In 128 B-token Mixture-of-Experts (MoE) [25] pretraining, AIR reduces terminal training loss by approximately 0.01 relative to cosine decay. Among these families, it achieves the lowest loss, the best downstream performance, and the greatest robustness to hyperparameter choices. Consistent with our theoretical analysis, controlled experiments show that the final training loss gaps between schedules narrow substantially when they share the same late-stage AIR schedule and differ only in their earlier LR trajectories. Transferring the induced ELR trajectory to WD also reproduces the corresponding training dynamics, showing that the benefits extend beyond fixed-norm training.

In summary, our contributions are:

  • We identify the induced ELR trajectory as the key driver of WN’s loss advantage over WD. Matching ELR trajectories largely aligns their training loss dynamics, highlighting ELR, rather than nominal LR alone, as the target for schedule design.

  • We propose AIR, an affine 1/t1/\sqrt{t} schedule for finite-horizon pretraining. Its decay shape follows from a theoretical analysis of directional alignment under WN, which also helps explain the role of late-stage ELR scheduling in AIR’s gains.

  • We empirically validate AIR against cosine decay and alternative schedule families in 128 B-token MoE pretraining, demonstrating improvements in training loss, downstream performance, and hyperparameter robustness. It reduces terminal training loss by approximately 0.01 relative to cosine at matched cumulative LR. Applying the same ELR trajectory to WD reproduces the corresponding training loss dynamics.

Effective Learning Rate as the Scheduling Principle

Effective learning rate under WN and WD

Let θk\theta_k denote a nonzero, vectorized parameter block at optimizer step kk, ηk\eta_k the nominal learning rate, and uku_k the optimizer update before LR scaling. With decay coefficient λ≥0\lambda\ge0 and prescribed norm R>0R > 0, decoupled WD [20] and fixed-norm WN [33] use the following update rules, respectively:

WD:θk+1=(1−ληk)θk+ηkuk(1)\mathrm{WD:}\quad\theta_{k+1} = (1-\lambda\eta_k)\theta_k + \eta_k u_k \tag*{(1)}
WN:θk+1=Rθk+ηkuk∥θk+ηkuk∥(2)\mathrm{WN:}\quad\theta_{k+1} = R\frac{\theta_k+\eta_k u_k}{\lVert\theta_k+\eta_k u_k\rVert} \tag*{(2)}

WD shrinks the current parameters before adding the optimizer update, while WN rescales the updated block to norm RR, preserving its direction.

The same nominal LR can produce different relative updates when parameter norms differ. Writing rk=∥θk∥r_k = \lVert\theta_k\rVert for the block’s norm at step kk, we define the blockwise norm-based effective learning rate [5, 13, 29] as

ηkeff≔ηkrk.(3)\eta_k^{\mathrm{eff}} \coloneqq\frac{\eta_k}{r_k}. \tag*{(3)}

It captures how much a given optimizer update can affect the parameter direction, providing a more informative measure of its impact than the nominal LR alone (see Appendices § 9.§ 1 and § 9.§ 2).

Aligning the training dynamics of WN and WD

Experimental setup. To validate the claim that matching ELRs in Eq. (3) aligns the training dynamics of WN and WD, we match WN’s ELR to WD’s on each parameter block in two ways: (i) Via norm. We rescale each WN block at each step to the corresponding recorded WD norm while retaining the same nominal LR as WD. (ii) Via LR. We keep each WN block at its fixed norm RR and adjust its blockwise nominal LR to satisfy

ηkWNR=ηkWDrkWD  ⟹  ηkWN=RηkWDrkWD(4)\frac{\eta_k^{\mathrm{WN}}}{R}=\frac{\eta_k^{\mathrm{WD}}}{r_k^{\mathrm{WD}}} \;\Longrightarrow\; \eta_k^{\mathrm{WN}}=R\frac{\eta_k^{\mathrm{WD}}}{r_k^{\mathrm{WD}}} \tag*{(4)}

where rkWDr_k^{\mathrm{WD}} is the recorded WD norm before the current update, and ηkWN\eta_k^{\mathrm{WN}} and ηkWD\eta_k^{\mathrm{WD}} are the respective nominal LRs.

For the reverse matching, we replay the WD norm trajectories and adjust blockwise LRs to match WN’s ELR. Details are given in Appendix § 9.§ 3. We conduct this reverse comparison under both cosine and AIR, alongside the corresponding WD and WN baselines.

Results. Figure 2 shows that ELR matching collapses the training loss dynamics between WD and WN. Under cosine, norm replay precisely reproduces WD dynamics, with a gap less than 2×10−42 \times10^{-4}, while LR matching at fixed WN norms also closely tracks the WD baseline, leaving a residual loss gap of several ×10−3\times10^{-3}. Under both cosine and AIR, matching WN’s ELR along recorded WD norms yields a loss dynamics close to the WN baseline. The consistent collapse across both schedules supports ELR as a key driver of training dynamics and thus a principled target for schedule design.

WN as a setting for ELR schedule design

Under WD, changing the nominal LR also affects the subsequent parameter norms, coupling the two quantities in Eq. (3), so that rescaling the nominal LR alone does not reproduce a target ELR trajectory [13]. WN removes this coupling by fixing rk=Rr_k = R, giving ηkeff=ηk/R\eta_k^{\mathrm{eff}} = \eta_k/R, making each block’s ELR proportional to its nominal LR [1, 33]. This direct proportionality makes WN a clean setting. Therefore, we study ELR scheduling under WN, using its update rule as the starting point for the theory.

Designing a Better ELR Schedule under WN: AIR

Directional-alignment dynamics under WN

Starting from the WN update in Eq. (2), which fixes a parameter block’s norm and thus leaves only its direction to evolve, we analyze how the LR schedule affects this evolution. Let vv denote an idealized unit target direction, fixed throughout the analysis. Given the training history F0\mathcal{F}_0 at the analysis starting point, define

mk:=E[vTθkR∣F0].(5)m_k := \mathbb{E}\left[\left.\frac{v^\mathsf{T}\theta_k}{R}\right|\mathcal{F}_0\right]. \tag*{(5)}

This is the expected cosine alignment after kk updates from the same starting point. The expectation averages over minibatch sampling and other randomness in these kk updates, while the initial history stays fixed. For this analysis, we consider an LR schedule specified in advance and assume tangential updates of fixed magnitude, θkTuk=0\theta_k^\mathsf{T}u_k = 0 and ∥uk∥=U>0\lVert u_k\rVert= U > 0.

Deriving the alignment dynamics. Taking the conditional expectation of the inner product with vv in Eq. (2) gives

mk+1=mk+aηk1+2bηk2.(6)m_{k+1} = \frac{m_k + a\eta_k}{\sqrt{1 + 2b\eta_k^2}}. \tag*{(6)}

where

a:=E[vTukR∣F0]>0,b:=U22R2.(7)a := \mathbb{E}\left[\left.\frac{v^\mathsf{T}u_k}{R}\right|\mathcal{F}_0\right] > 0,\qquad b := \frac{U^2}{2R^2}. \tag*{(7)}

We assume that aa and bb are constant across steps and schedules. The optimization below concerns these constant-coefficient scalar dynamics. For 2bηk2≪12b\eta_k^2 \ll1, a second-order expansion yields the continuous approximation, with time measured in optimizer steps,

m˙(t)=aη(t)−bη(t)2m(t).(8)\dot{m}(t) = a\eta(t) - b\eta(t)^2m(t). \tag*{(8)}

A larger LR increases new alignment linearly but strengthens the retraction of existing alignment quadratically. Detailed derivations are provided in Appendix § 10.§ 1.

Exact integral solution. For a nonnegative, square-integrable schedule on [0,h][0,h] and initial alignment m(0)=m0>0m(0) = m_0 > 0, Eq. (8) has the exact solution

m(h)=m0H(h,0)+a∫0hH(h,t)η(t) dt,H(h,t):=exp⁡[−b∫thη(s)2 ds].(9)m(h) = m_0 H(h,0) + a\int_0^h H(h,t)\eta(t)\,\mathrm{d}t,\qquad H(h,t) := \exp\left[-b\int_t^h \eta(s)^2\,\mathrm{d}s\right]. \tag*{(9)}

The solution decomposes endpoint alignment into retained initial alignment and retained contributions from subsequent updates. Each contribution is weighted by H(h,t)H(h,t), which represents the fraction of alignment present at time tt that remains at the endpoint hh. Detailed derivations are provided in Appendix § 10.§ 2.

Unconstrained optimal schedule: inverse-square-root decay

We first maximize terminal alignment m(h)m(h) in Eq. (9) without prescribing a peak, a terminal LR, or an LR budget. For fixed aa, bb, m0m_0, h>0h>0, completing the square in the alignment dynamics gives

ddtm(t)2=a22b−2b(m(t)η(t)−a2b)2.(10)\frac{\mathrm{d}}{\mathrm{d}t}m(t)^2=\frac{a^2}{2b}-2b\left(m(t)\eta(t)-\frac{a}{2b}\right)^2. \tag*{(10)}

Since m(t)>0m(t)>0, integration yields the bound m(h)≤m02+a2h/(2b)m(h)\le\sqrt{m_0^2+a^2h/(2b)}, with equality if and only if m(t)η(t)=a/(2b)m(t)\eta(t)=a/(2b) almost everywhere on [0,h][0,h]. Under this condition, m(t)2m(t)^2 grows linearly, giving m⋆(t)=m02+a2t/(2b)m^\star(t)=\sqrt{m_0^2+a^2t/(2b)} and hence the optimal time-shifted schedule:

η⋆(t)=η01+2bη02t,η0=a2bm0.(11)\eta^\star(t)=\frac{\eta_0}{\sqrt{1+2b\eta_0^2t}},\qquad\eta_0=\frac{a}{2bm_0}. \tag*{(11)}

Because the same schedule attains the bound for every endpoint hh, it maximizes expected alignment at every time under these dynamics. The 1/t1/\sqrt{t} ELR profiles have also been observed to arise implicitly: without weight decay, growing parameter norms make the realized relative update decay as t−1/2t^{-1/2} [13]. Here the same shape instead follows from maximizing directional alignment under WN. Appendix § 10.§ 3 gives the full proof and shows that its integer samples are also optimal for the discrete recurrence in Eq. (6). Appendix § 10.§ 4 derives the geometric restrictions needed to interpret these states as expected cosine alignments.

AIR: A practical schedule design

Building on the 1/t1/\sqrt{t} decay in Eq. (11), we propose AIR (Affine Inverse-square-Root decay) for pretraining with prescribed peak and terminal LRs. The decay schedule controls how rapidly the LR decreases at different stages of training. To preserve the relative magnitudes of LR decay, we use an affine transformation that scales them by a common positive factor. Let k=0,…,Kk=0,\ldots,K index the decay stage, with K>0K>0 its horizon. For a decay time scale α>0\alpha>0, define

q(k)=(1+kα)−1/2,f(k)=q(k)−q(K)1−q(K).(12)q(k)=\left(1+\frac{k}{\alpha}\right)^{-1/2},\qquad f(k)=\frac{q(k)-q(K)}{1-q(K)}. \tag*{(12)}

Here, f(k)f(k) rescales the 1/t1/\sqrt{t} curve q(k)q(k) from 1 to 0. For a peak LR pp and a terminal LR 0<ηmin⁡<p0<\eta_{\min}<p, the resulting schedule is

ηAIR(k)=ηmin⁡+(p−ηmin⁡)f(k),k=0,…,K.(13)\eta_{\mathrm{AIR}}(k)=\eta_{\min}+(p-\eta_{\min})f(k),\qquad k=0,\ldots,K. \tag*{(13)}

This schedule decreases monotonically from pp at k=0k=0 to ηmin⁡\eta_{\min} at k=Kk=K. Appendix § 10.§ 5 derives this affine construction and shows that it preserves the ratios between decay rates. We use α=K/8\alpha=K/8 in the main comparisons and evaluate sensitivity to this choice in § 4.3.

When an LR budget is prescribed instead of the peak, we determine pp from the complete discrete LR sum, including warmup, with the other settings fixed in Appendix § 10.§ 6.

Empirical Evaluation

Experimental setup

Pretraining setting. We pretrain a mixture-of-experts (MoE) model with 110 M activated parameters and 4.7 B total parameters on 128 B tokens using Muon [11] under WN. Each complete run takes 48 hours on 16 GPUs. To further test scalability, we apply the same scheduler design to a larger MoE with 790 M activated and 35.7 B total parameters, trained on 500 B tokens, with each run taking 168 hours on 32 GPUs. Additional experimental settings and reporting conventions are given in Appendix § 11.

Comparison controls. We perform a hyperparameter search on the cosine baseline and select the best peak LR and warmup ratio. For a fair comparison across all schedule families, we use linear warmup with the same duration and match the terminal LR and cumulative LR, defined as the sum of LRs over all training steps. Each schedule’s peak LR is calibrated to match this sum, including warmup, as described in Appendix § 10.§ 6. All schedules are compared at the end of training.

Figure 3a shows the cosine baseline, raw 1/t1/\sqrt{t} decay [30], and its affine transformation AIR, defined by Eqs. (12)–(13). The raw curve is shown as a reference with the same peak as AIR, without matching its terminal or cumulative LR. We also compare other decay shapes under the same controls and affine construction, including e−te^{-t} [34], 1−log⁡(1+t)1-\log(1+t) [18], and 1−t1-\sqrt{t} [7, 38]. Appendix § 11.§ 2 lists the complete set of decay shapes.

Figure 3. Learning rate schedules and training loss under WN. With the same terminal and cumulative LRs, AIR achieves lower final training loss than cosine.

Pretraining loss and downstream performance

Experimental setup. Following § 4.1, all schedules share the same warmup, terminal LR, and cumulative LR, and each decay shape enters the comparison at its best shape parameter selected during evaluation, listed in Table 4. We report terminal training and evaluation loss together with performance on the downstream benchmarks listed in Appendix § 11.§ 4. We repeat the comparison between cosine and AIR at a larger model scale.

Shapeq(k)q(k), with τ=k/K\tau= k/KSelected
Cosine12[1+cos⁡(πτ)]\frac{1}{2}[1 + \cos(\pi\tau)]–
Linear1−τ1 - \tau–
Half-cosinecos⁡(πτ/2)\cos(\pi\tau/2)–
Rear-cosine1−sin⁡(πτ/2)1 - \sin(\pi\tau/2)–
Cycloidcos⁡θ,θ−sin⁡θ=τ(θ1−sin⁡θ1)\cos\theta,\quad\theta- \sin\theta= \tau(\theta_1 - \sin\theta_1)θ1=3π/4\theta_1 = 3\pi/4
Double-cosineqdc(τ)q_{\mathrm{dc}}(\tau), defined below–
Power of 1−τ1 - \tau(1−τ)β(1 - \tau)^\betaβ=2\beta= 2
Power of τ\tau1−τβ1 - \tau^\betaβ=0.2\beta= 0.2
Logarithmic1−log⁡(1+k/α)1 - \log(1 + k/\alpha)α=K/32\alpha= K/32
Exponentialexp⁡(−ρτ)\exp(-\rho\tau)ρ=3.594\rho= 3.594
Inverse square(1+k/α)−2(1 + k/\alpha)^{-2}α=K/2\alpha= K/2
Inverse time(1+k/α)−1(1 + k/\alpha)^{-1}α=0.243K\alpha= 0.243K
Variable power(1+k/α)−[1/2+(βend−1/2)τ](1 + k/\alpha)^{-[1/2+(\beta_{\mathrm{end}}-1/2)\tau]}α=K/8, βend=1.771\alpha= K/8,\ \beta_{\mathrm{end}} = 1.771
Inverse square root (AIR)(1+k/α)−1/2(1 + k/\alpha)^{-1/2}α=K/8\alpha= K/8

Table 4. Raw decay shapes before the common affine transformation, and the shape parameter selected for the smaller-scale rows of Table 1. The time scale α>0\alpha> 0, decay exponent β>0\beta> 0, which is fixed for the two power families and rises to βend\beta_{\mathrm{end}} for variable power, the terminal angle 0<θ1≤π0 < \theta_1 \le\pi, and the exponential decay parameter ρ>0\rho> 0 control the respective curve shapes. A dash marks a shape with no free parameter.

Results. Figure 3b and Table 1 show that AIR lowers terminal training and evaluation loss by about 0.01 relative to cosine at both model scales and achieves the lowest reported losses among the compared schedulers. It also attains the best downstream average among all compared shapes, and at the larger scale it improves over cosine on most benchmarks, highlighting the scale-up ability of our scheduler design.

ScheduleTrain Loss (↓)Eval Loss (↓)MMLU (↑)MMLU-Pro (↑)AGIEval (↑)WinoGrande (↑)BBH (↑)DROP (↑)GSM8k (↑)MATH (↑)HumanEval (↑)MBPP+ (↑)TriviaQA (↑)Avg. (↑)
110 M/4.7 B model
Cosine1.7402.18549.018.229.149.530.922.017.011.128.735.230.929.24
Linear1.7422.18748.916.426.349.631.122.613.411.326.235.431.228.40
Half-cosine1.7492.19548.615.528.250.732.523.915.011.724.434.729.528.61
Rear-cosine1.7362.18048.016.228.751.231.622.314.310.525.036.230.728.61
Cycloid1.7342.17847.316.027.549.930.722.615.110.026.233.631.128.18
(1−t)2(1-t)^21.7362.18148.116.827.650.730.622.116.512.027.436.829.528.92
1−tβ1-t^\beta1.7322.17448.518.130.250.931.822.316.410.931.135.232.329.79
1−log⁡t1-\log t1.7312.17548.115.629.648.832.021.516.811.623.834.433.228.67
e−te^{-t}1.7322.17648.916.029.250.631.621.216.810.827.436.831.729.18
1/t21/t^21.7302.17550.518.429.050.832.122.217.511.629.933.130.329.58
1/t1/t1.7302.17449.716.929.650.830.823.016.712.226.832.829.729.00
t−β(t)t^{-\beta(t)}1.7332.17648.816.730.150.730.022.914.210.628.034.730.628.85
1t\frac{1}{\sqrt{t}} (AIR)1.7302.17449.718.629.650.131.123.815.813.029.936.531.629.97
790 M/35.7 B model
Cosine1.4311.84370.740.654.250.456.256.059.936.867.760.165.856.22
1t\frac{1}{\sqrt{t}} (AIR)1.4211.83272.142.455.751.859.451.264.237.270.165.366.257.78

Table 1. Terminal losses and downstream performance of matched-budget decay shapes under WN. Each row is labeled by its raw decay shape before the common affine transformation and uses that shape’s best parameter selected during evaluation (Appendix C.2). AIR achieves the lowest terminal losses and attains the best downstream average at both model scales.

Robustness to hyperparameter choices

Experimental setup. Under the same matched controls, we sweep each family’s shape parameter, which also moves its peak LR. For each family we report its LR sensitivity SS, the average loss above the family’s best configuration per fold of peak LR, defined formally in Appendix § 11.§ 5 (Eq. (31)).

Results. Figure 4 plots terminal training loss against peak LR with family-wise polynomial fits, and Table 5 in Appendix § 11.§ 5 reports the resulting LR sensitivity. AIR has the lowest LR sensitivity over the widest range of peak LR. Its region of low loss is therefore wide enough that the gain does not depend on a narrowly tuned peak LR and thus facilitates LR transfer [21].

Figure 4. AIR stays near its best loss over the widest peak LR range. Circles are measured configurations, stars the best per family, and dashed curves family-wise convex fits.

Table 5. AIR is the least sensitive to the peak LR on training loss. Sensitivity is the mean loss above a family’s best per e-fold of peak LR (Eq. (31)).

ScheduleConfigsLR range (e-folds)LR sensitivity (↓)
Cycloid30.3461Train: 0.0022Eval: 0.0017
(1−t)β(1-t)^{\beta}30.4219Train: 0.0160Eval: 0.0126
1−tβ1-t^{\beta}30.5630Train: 0.0023Eval: 0.0039
1−log⁡t1-\log t30.2182Train: 0.0047Eval: 0.0054
e−te^{-t}40.7081Train: 0.0113Eval: 0.0110
1/t21/t^{2}30.5630Train: 0.0032Eval: 0.0021
1/t1/t50.5874Train: 0.0019Eval: 0.0027
t−β(t)t^{-\beta(t)}20.0427Train: 0.0230Eval: 0.0234
1/t1/\sqrt{t} (AIR)50.7506Train: 0.0018Eval: 0.0018

Late-Stage Scheduling Matters More

Having established the gains of AIR in § 4, we further explore where in training this advantage is created. We find that late-stage scheduling drives the final loss gap (§ 5.1) and that sharing the late schedule substantially reduces gaps created by distinct early schedules (Section 5.2). Further comparisons in Section 5.3 reveal a trade-off between sustained annealing and gradual LR changes, with AIR naturally striking this balance. Finally, we provide a brief theoretical explanation in Section 5.4 of how early and late schedules affect terminal alignment.

Late-stage scheduling drives the final loss gap

Among the candidate schedules matched to the same cumulative LR, a Cycloid schedule with θ1=π\theta_1 = \pi happens to take exactly the same peak LR as AIR, so the two runs share their warmup and terminal LR as well and differ only in the shape of the decay in between (Figure 5(a)). However, their final training losses still differ by approximately 0.005, a substantial gap at this scale. Notice that the two intersections of their decay curves naturally divide the training horizon into early, middle, and late stages, when we swap one stage at a time between them while leaving the other two unchanged.

As shown in Figure 5(b,c), swaps of the early and middle stages leave the final loss nearly unchanged, whereas swapping the late stage closes about 90% of the gap for either schedule. For this matched pair, the gap is almost entirely attributable to the late stage. This motivates us to examine whether a shared late schedule can reduce the final loss gap created by distinct early schedules. A related effect was reported for scale-invariant networks trained with weight decay, where reaching the equilibrium of a late phase makes the outcome largely independent of earlier phases [15].

Figure 5. Late-stage scheduling dominates the final loss difference. Starting from two full schedules with matched peak, terminal, and cumulative LRs, swapping the late stage between AIR and Cycloid nearly transfers the full final loss gap to either schedule, whereas swaps of the early and middle stages have little effect. The bars use the terminal exponential moving average (EMA-0.99) training loss, as in Table 1; the intervention and reporting details are given in Appendix C.6.

Shared late scheduling reduces the final loss gap

Experimental setup. To address this question, we consider six decay schedules, Cosine, Linear, Half-cosine, Double-cosine, (1−t)2(1-t)^2, and 1−t1-\sqrt{t}, whose original runs end with training losses 0.005–0.018 higher than AIR. For each schedule, we retain its decay shape over the first 70% of training steps and concatenate it with the same late-stage schedule of AIR. The early segment and its warmup target are rescaled to match the cumulative LR under the same comparison constraints as in § 4.1. The two runs of each schedule are denoted original and shared tail, respectively.

Results. Table 2 shows that the shared late schedule lowers terminal training and evaluation loss for all six schedules, closing 42%–83% and 45%–94% of their respective gaps to AIR. Thus, a shared late schedule substantially reduces the performance differences arising from different early schedules.

ScheduleTraining loss (relative to AIR)Eval loss (relative to AIR)
OriginalShared tailGap closed (%) ↑OriginalShared tailGap closed (%) ↑
Cosine1.7399 (+0.0096)1.7358 (+0.0055)42%2.1846 (+0.0106)2.1799 (+0.0059)45%
Linear1.7415 (+0.0112)1.7341 (+0.0038)66%2.1875 (+0.0135)2.1787 (+0.0047)65%
Half-cosine1.7487 (+0.0184)1.7374 (+0.0071)62%2.1950 (+0.0210)2.1815 (+0.0076)64%
Double-cosine1.7446 (+0.0143)1.7328 (+0.0025)83%2.1901 (+0.0161)2.1769 (+0.0029)82%
(1−t)2(1-t)^21.7361 (+0.0058)1.7319 (+0.0016)72%2.1805 (+0.0065)2.1753 (+0.0013)80%
1−t1-\sqrt{t}1.7354 (+0.0050)1.7316 (+0.0012)75%2.1787 (+0.0047)2.1743 (+0.0003)94%
1/t1/\sqrt{t} (AIR)1.7303––2.1740––

Table 2. A shared late schedule narrows the endpoint gap for every early decay shape. Each schedule is trained twice, once in its original form and once with the common late schedule. Parentheses give the gap of each loss to the AIR anchor in the last row, and Gap is the fraction of that gap the shared tail removes. Training loss is the terminal EMA-0.99 value and evaluation loss the terminal raw value, as in Table 1.

A trade-off in late-stage scheduling

Experimental setup. Having established the importance of late-stage scheduling, we next examine what makes a late schedule effective. Based on the design of AIR in § 3.3, we vary the final 20% of its decay stage, starting at Ks≈0.8KK_s \approx0.8K, as

ηγ(k)=ηmin⁡+[ηAIR(Ks)−ηmin⁡](K−kK−Ks)γ,Ks≤k≤K,(14)\eta_\gamma(k) = \eta_{\min} + \left[\eta_{\mathrm{AIR}}(K_s) - \eta_{\min}\right]\left(\frac{K-k}{K-K_s}\right)^\gamma,\quad K_s \le k \le K, \tag*{(14)}

where γ∈{0.5,0.75,1,2,3}\gamma\in\{0.5, 0.75, 1, 2, 3\}. Smaller exponents postpone more of the LR reduction to the endpoint, whereas larger exponents reduce the LR earlier and then flatten out. The junction and terminal LRs remain fixed, while the area under the recorded LR traces varies by less than 1% (Appendix § 11.§ 7).

Results. Figure 6 shows that both the abrupt decay at γ=0.5\gamma= 0.5 and the early flattening at γ∈{2,3}\gamma\in\{2,3\} increase final loss, while AIR and the variants with γ∈{0.75,1}\gamma\in\{0.75,1\} perform comparably, achieving the lowest final losses. These results suggest a trade-off: late-stage scheduling should sustain annealing while avoiding overly abrupt LR changes. The design of AIR naturally balances these requirements and performs close to the best variants in this sweep.

Figure 6. Late-stage decay exhibits a two-sided trade-off. With the junction and terminal LRs fixed, both overly abrupt decay and early flattening increase final loss.

A theoretical explanation of schedule dependence

To interpret these results, we use the alignment dynamics of § 3 to separate the effects of early- and late-stage schedule changes. Consider bounded nonnegative schedules η1,η2\eta_1,\eta_2 on the decay interval [0,K][0,K], with common aa, bb, m0m_0, F0\mathcal{F}_0 and a fixed stage boundary KsK_s. Write Δm=m1−m2\Delta m = m_1 - m_2 and Δη=η1−η2\Delta\eta= \eta_1 - \eta_2, where the expected alignments mim_i retain the conditioning in Eq. (5) throughout. For a shared late schedule, η1=η2=η\eta_1 = \eta_2 = \eta on [Ks,K][K_s, K], Eq. (9) gives

Δm(K)=H(K,Ks)Δm(Ks),H(K,Ks)=exp⁡[−b∫KsKη(k)2 dk].(15)\Delta m(K) = H(K,K_s)\Delta m(K_s), \qquad H(K,K_s) = \exp\left[-b\int_{K_s}^{K}\eta(k)^2\,\mathrm{d}k\right]. \tag*{(15)}

Since 0<H(K,Ks)≤10 < H(K,K_s) \le1, the shared late stage attenuates the entering difference without reversing its sign, ignoring stochastic fluctuations around the mean. Over a fixed late interval, higher LRs increase the cumulative squared LR and reduce H(K,Ks)H(K,K_s), thereby weakening the effect of early-stage scheduling on endpoint alignment. The derivation is given in Appendix § 12.§ 2.

For different late-stage schedules, a shared early schedule gives Δm(Ks)=0\Delta m(K_s) = 0. For bounds 0≤ηi≤P0 \le\eta_i \le P and 0≤mi≤M0 \le m_i \le M over the late-stage interval, let L12=max⁡{a,∣a−2bPM∣}L_{12} = \max\{a, |a - 2bPM|\} and H12=H1H2H_{12} = \sqrt{H_1H_2}, with HiH_i the retention factor for ηi\eta_i. This gives

∣Δm(K)∣≤L12∫KsKH12(K,k)∣Δη(k)∣ dk.(16)|\Delta m(K)| \le L_{12}\int_{K_s}^{K} H_{12}(K,k)|\Delta\eta(k)|\,\mathrm{d}k. \tag*{(16)}

Smaller weighted LR differences give a tighter bound on the terminal alignment difference. Within this bound, differences nearer the endpoint receive greater weight because less subsequent attenuation remains. Details are given in Appendix § 12.§ 3. Further, if the schedules differ in both stages,

∣Δm(K)∣≤H12(K,Ks)∣Δm(Ks)∣+L12∫KsKH12(K,k)∣Δη(k)∣ dk.(17)|\Delta m(K)| \le H_{12}(K,K_s)|\Delta m(K_s)| + L_{12}\int_{K_s}^{K} H_{12}(K,k)|\Delta\eta(k)|\,\mathrm{d}k. \tag*{(17)}

The first term retains the early-stage difference, while the second bounds the late-stage contribution. Slope-based refinements are in Appendix § 12.§ 4. These bounds on expected alignment provide a qualitative explanation for the observed loss gaps.

Related Work

WN and related constraints provide explicit parameter-scale control as an alternative to WD, with reported efficiency and stability gains [1, 33]. Methods differ in whether they separate magnitude from direction [23] or fix weight and update norms [33]. The choice of matrix norm further shapes the weight constraints and update geometry [35, 36]. For manifold constraints, Su [26, 27] and Bernste[in] [2] optimize tangent-space descent directions before retracting weights, so that the update respects the imposed geometry.

Beyond norm control, scheduling determines how the effective step size evolves during training. River-valley analyses explain WSD [32], while convex bounds and functional scaling laws motivate cooldown and power decay [14, 24]. However, these theories focus on nominal LR alone, which is not principled when parameter norms evolve [5]. Concurrent work corroborates our finding that matching ELR largely aligns WN and WD loss dynamics [17, 34, 38], but does not derive a decay law from an optimization objective. We address this gap by optimizing directional alignment under WN to derive 1/t1/\sqrt{t} decay, then adapting it to finite horizons through an affine transformation, yielding AIR as an effective ELR schedule.

Conclusion and Future Work

We identify the induced effective learning rate (ELR) trajectory as the key driver of WN’s advantage over WD, with matched ELRs largely aligning their training loss dynamics. This finding motivates schedule design that targets ELR directly. Our theoretical analysis of directional alignment under WN, together with an affine adaptation of the unconstrained optimal 1/t1/\sqrt{t} schedule, yields AIR for finite-horizon training. Experiments on MoE models at two scales show that, at matched cumulative LR, AIR reduces terminal loss by about 0.01 relative to cosine and improves downstream performance, while schedule sweeps demonstrate hyperparameter robustness. Further stage-wise interventions show that sharing the late schedule substantially narrows final loss gaps from different early schedules, identifying late-stage scheduling as a key source of AIR’s effectiveness. Transferring its ELR trajectory to WD reproduces WN training loss dynamics, demonstrating that our design extends beyond fixed-norm training.

Future work includes extending our directional-alignment theory to account for evolving update statistics and quantitatively connect alignment to loss. We also aim to investigate the ELR principle and AIR across more architectures, optimizers, and larger training scales, and to study ELR scheduling in continued training (CT), supervised fine-tuning (SFT), and reinforcement learning (RL).

Appendix

Directional Geometry and ELR Matching

We derive the geometric interpretation of ELR in § 2.1 and the blockwise matching rule used in § 2.2.

Exact normalized updates

For a nonzero parameter block, retain rk=∥θk∥r_k = \lVert\theta_k\rVert and ηkeff=ηk/rk\eta_k^{\mathrm{eff}} = \eta_k/r_k from § 2.1, and write θ^k=θk/rk\hat{\theta}_k = \theta_k/r_k for its unit direction. Throughout, ηk≥0\eta_k \ge0, and norms are Euclidean on vectorized blocks. Under fixed-norm WN, rk=Rr_k = R. Dividing the updates in Eqs. (1) and (2) by their pre-update parameter norms and normalizing gives

WD:θ^k+1=(1−ληk)θ^k+ηkeffuk∥(1−ληk)θ^k+ηkeffuk∥(18)\mathrm{WD:}\quad\hat{\theta}_{k+1} = \frac{(1-\lambda\eta_k)\hat{\theta}_k+\eta_k^{\mathrm{eff}}u_k}{\lVert(1-\lambda\eta_k)\hat{\theta}_k+\eta_k^{\mathrm{eff}}u_k\rVert} \tag*{(18)}
WN:θ^k+1=θ^k+ηkeffuk∥θ^k+ηkeffuk∥,ηkeff=ηk/R.(19)\mathrm{WN:}\quad\hat{\theta}_{k+1} = \frac{\hat{\theta}_k+\eta_k^{\mathrm{eff}}u_k}{\lVert\hat{\theta}_k+\eta_k^{\mathrm{eff}}u_k\rVert},\qquad\eta_k^{\mathrm{eff}}=\eta_k/R. \tag*{(19)}

These identities hold whenever the vector being normalized is nonzero. They keep the realized optimizer update explicit. The ELR multiplies that update relative to the current parameter norm, while WD also contributes radial shrinkage.

Radial and tangential components

Define

u∥,k=θ^kTuk,uk⊥=uk−u∥,kθ^k.u_{\parallel,k}=\hat{\theta}_k^{\mathsf{T}}u_k,\qquad u_k^\perp=u_k-u_{\parallel,k}\hat{\theta}_k.

The scalar u∥,ku_{\parallel,k} is the radial component, and uk⊥u_k^\perp is the component orthogonal to the parameter direction. Under WD, the updated vector is

[(1−ληk)rk+ηku∥,k]θ^k+ηkuk⊥.\left[(1-\lambda\eta_k)r_k+\eta_k u_{\parallel,k}\right]\hat{\theta}_k+\eta_k u_k^\perp.

Orthogonality gives its squared norm as the sum of the squared radial and tangential terms. The unsigned angle ϕk∈[0,π]\phi_k\in[0,\pi] between the old and updated directions is therefore

ϕk=atan2⁡(ηk∥uk⊥∥,(1−ληk)rk+ηku∥,k).(20)\phi_k=\operatorname{atan2}\left(\eta_k\lVert u_k^\perp\rVert,(1-\lambda\eta_k)r_k+\eta_k u_{\parallel,k}\right). \tag*{(20)}

For WN, the same expression applies with λ=0\lambda=0 and rk=Rr_k=R, because the final normalization preserves direction.

To obtain the first-order change, the derivative of x↦x/∥x∥x\mapsto x/\lVert x\rVert at a unit vector θ^k\hat{\theta}_k is I−θ^kθ^kTI-\hat{\theta}_k\hat{\theta}_k^{\mathsf{T}}. Applied to the increment −ληkθ^k+ηkeffuk-\lambda\eta_k\hat{\theta}_k+\eta_k^{\mathrm{eff}}u_k, it removes the radial term and gives

θ^k+1−θ^k=ηkeffuk⊥+O((ληk+ηkeff∥uk∥)2).(21)\hat{\theta}_{k+1}-\hat{\theta}_k=\eta_k^{\mathrm{eff}}u_k^\perp+O\left((\lambda\eta_k+\eta_k^{\mathrm{eff}}\lVert u_k\rVert)^2\right). \tag*{(21)}

The expansion requires ληk+ηkeff∥uk∥≪1\lambda\eta_k+\eta_k^{\mathrm{eff}}\lVert u_k\rVert\ll1. The WN expression again sets λ=0\lambda=0. For a given tangential optimizer update, the ELR thus sets the first-order scale of the directional change.

Blockwise ELR transfer

For a target trajectory ηkeff,tar\eta_k^{\mathrm{eff,tar}}, choosing ηkWD=rkWDηkeff,tar\eta_k^{\mathrm{WD}}=r_k^{\mathrm{WD}}\eta_k^{\mathrm{eff,tar}} gives ηkWD/rkWD=ηkeff,tar\eta_k^{\mathrm{WD}}/r_k^{\mathrm{WD}}=\eta_k^{\mathrm{eff,tar}} by substitution. A WN schedule supplies the target ηkeff,tar=ηkWN/R\eta_k^{\mathrm{eff,tar}}=\eta_k^{\mathrm{WN}}/R, giving

ηkWD=rkWDRηkWN.(22)\eta_k^{\mathrm{WD}}=\frac{r_k^{\mathrm{WD}}}{R}\eta_k^{\mathrm{WN}}. \tag*{(22)}

This is the reverse direction of the coefficient matching in Eq. (4). We apply this rule separately to each controlled block, using the recorded WD norm rkWDr_k^{\mathrm{WD}} replayed at step kk in the reverse-matching experiments.

WN Schedule Design and Calibration

We first derive the alignment dynamics and their unconstrained optimum from § 3. We then establish the affine endpoint construction and discrete LR sum calibration used for finite-horizon schedules.

From the WN update to alignment dynamics

Under the assumptions in § 3.1, ∥θk∥=R\lVert\theta_k\rVert= R, θkTuk=0\theta_k^{\mathsf{T}}u_k = 0, and ∥uk∥=U\lVert u_k\rVert= U. Hence

∥θk+ηkuk∥2=R2+ηk2U2.\lVert\theta_k+\eta_k u_k\rVert^2 = R^2+\eta_k^2U^2.

Taking the inner product of Eq. (2) with v/Rv/R gives

vTθk+1R=vTθk/R+ηkvTuk/R1+2bηk2.\frac{v^{\mathsf{T}}\theta_{k+1}}{R} = \frac{v^{\mathsf{T}}\theta_k/R+\eta_k v^{\mathsf{T}}u_k/R}{\sqrt{1+2b\eta_k^2}}.

We condition on a fixed initial history F0\mathcal{F}_0 and a prescribed LR schedule, so ηk\eta_k and the denominator are deterministic under this conditioning. We average over the first kk updates that generate θk\theta_k and the randomness in uku_k. No independence between θk\theta_k and uku_k is needed. For an LR selected adaptively from later random observations, this factorization would in general fail. The definitions of mkm_k and aa then give Eq. (6). For 2bηk2≪12b\eta_k^2 \ll1, the binomial expansion gives

(1+2bηk2)−1/2=1−bηk2+O(b2ηk4).(1+2b\eta_k^2)^{-1/2}=1-b\eta_k^2+O(b^2\eta_k^4).

Multiplying by mk+aηkm_k+a\eta_k and subtracting mkm_k yields

mk+1−mk=aηk−bmkηk2+O(abηk3+b2∣mk∣ηk4).(23)m_{k+1}-m_k=a\eta_k-bm_k\eta_k^2+O\left(ab\eta_k^3+b^2\lvert m_k\rvert\eta_k^4\right). \tag*{(23)}

Approximating this increment by a derivative with one optimizer step as the time unit gives

m˙(t)=aη(t)−bη(t)2m(t).\dot{m}(t)=a\eta(t)-b\eta(t)^2m(t).

This establishes the second-order discrete increment underlying Eq. (8). Replacing that increment by a derivative is a further continuous approximation, requiring small relative updates, slow variation of the LR and alignment over individual optimizer steps, and approximately constant update statistics.

Derivation of the integral solution

To solve Eq. (8) on [0,h][0,h] with m(0)=m0>0m(0)=m_0>0, take a nonnegative η∈L2([0,h])\eta\in L^2([0,h]). Both η\eta and η2\eta^2 are integrable on the finite interval, so the initial-value problem has a unique absolutely continuous solution. Multiplying the ODE by exp⁡[b∫0tη(s)2 ds]\exp\left[b\int_0^t\eta(s)^2\,\mathrm{d}s\right] gives

ddt(m(t)exp⁡[b∫0tη(s)2 ds])=aη(t)exp⁡[b∫0tη(s)2 ds].\frac{\mathrm{d}}{\mathrm{d}t}\left(m(t)\exp\left[b\int_0^t\eta(s)^2\,\mathrm{d}s\right]\right) = a\eta(t)\exp\left[b\int_0^t\eta(s)^2\,\mathrm{d}s\right].

Integration from 00 to hh, followed by division by the terminal integrating factor, multiplies each contribution introduced at time tt by

exp⁡[b∫0tη(s)2 ds]exp⁡[b∫0hη(s)2 ds]=exp⁡[−b∫thη(s)2 ds]=H(h,t).\frac{\exp\left[b\int_0^t\eta(s)^2\,\mathrm{d}s\right]}{\exp\left[b\int_0^h\eta(s)^2\,\mathrm{d}s\right]} = \exp\left[-b\int_t^h\eta(s)^2\,\mathrm{d}s\right] = H(h,t).

This is the fraction retained from tt to hh, as introduced in Eq. (9). Combining the initial and accumulated contributions gives

m(h)=m0H(h,0)+a∫0hH(h,t)η(t) dt.(24)m(h)=m_0H(h,0)+a\int_0^h H(h,t)\eta(t)\,\mathrm{d}t. \tag*{(24)}

This proves Eq. (9). Since m0>0m_0>0, H(h,t)>0H(h,t)>0, and the integral term is nonnegative, the solution is strictly positive at every time in the interval.

Proof of the unconstrained optimal schedule

Proof of the unconstrained optimal schedule

For any nonnegative η∈L2([0,h])\eta\in L^2([0,h]), with fixed aa, bb, m0m_0, h>0h > 0, the positive solution satisfies

ddtm(t)2=2am(t)η(t)−2bm(t)2η(t)2=a22b−2b(m(t)η(t)−a2b)2\frac{\mathrm{d}}{\mathrm{d}t}m(t)^2 = 2am(t)\eta(t) - 2bm(t)^2\eta(t)^2 = \frac{a^2}{2b} - 2b\left(m(t)\eta(t) - \frac{a}{2b}\right)^2

almost everywhere. Integrating establishes m(h)2≤m02+a2h/(2b)m(h)^2 \le m_0^2 + a^2h/(2b). Equality requires m(t)η(t)=a/(2b)m(t)\eta(t) = a/(2b) almost everywhere. Under this condition, d(m2)/dt=a2/(2b)\mathrm{d}(m^2)/\mathrm{d}t = a^2/(2b), so

m⋆(t)=m02+a22bt,η⋆(t)=a2bm02+a2t/(2b).m^\star(t) = \sqrt{m_0^2 + \frac{a^2}{2b}t}, \qquad\eta^\star(t) = \frac{a}{2b\sqrt{m_0^2 + a^2t/(2b)}}.

The resulting LR is nonnegative, continuous, and square-integrable on any finite interval. With η0=a/(2bm0)\eta_0 = a/(2bm_0), it is exactly Eq. (11). Substitution verifies that it solves the ODE with the prescribed initial state and attains the bound. For every candidate, the precise terminal gap is

m⋆(h)2−m(h)2=2b∫0h(m(t)η(t)−a2b)2dt.(25)m^\star(h)^2 - m(h)^2 = 2b\int_0^h\left(m(t)\eta(t) - \frac{a}{2b}\right)^2\mathrm{d}t. \tag*{(25)}

The nonnegative right-hand side proves global optimality. A zero gap uniquely determines m⋆m^\star at every time and η⋆\eta^\star almost everywhere. Since the endpoint hh was arbitrary and the optimal curve does not depend on it, this same curve attains the alignment bound at every time. This completes the proof of the result in § 3.2. The squared states here are squares of expected alignments, not expectations of squared alignments.

Exact discrete counterpart. For the recurrence in Eq. (6), direct algebra gives, for mk>0m_k > 0,

mk2+a22b−mk+12=2b(mkηk−a/(2b))21+2bηk2≥0.m_k^2 + \frac{a^2}{2b} - m_{k+1}^2 = \frac{2b\left(m_k\eta_k - a/(2b)\right)^2}{1 + 2b\eta_k^2} \ge0.

After NN updates, this implies mN2≤m02+Na2/(2b)m_N^2 \le m_0^2 + Na^2/(2b). Choosing ηk=a/(2bmk)\eta_k = a/(2bm_k) makes every single-step gap zero and yields

mk⋆=m02+ka22b,ηk⋆=η01+2bη02k.(26)m_k^\star= \sqrt{m_0^2 + \frac{ka^2}{2b}}, \qquad\eta_k^\star= \frac{\eta_0}{\sqrt{1 + 2b\eta_0^2k}}. \tag*{(26)}

Thus the integer samples of the continuous optimum exactly maximize terminal alignment in this discrete recurrence. The NN updates use η0,…,ηN−1\eta_0,\ldots,\eta_{N-1} and produce mNm_N.

Geometric range of the constant-coefficient assumptions

The scalar recurrence and its continuous approximation have a broader algebraic domain than an expected cosine alignment. To identify the necessary geometric restriction, tangency and the Cauchy–Schwarz inequality give, pathwise,

∣v⊤ukR∣=∣[v−(v⊤θk/R)θk/R]⊤ukR∣≤UR1−(v⊤θkR)2.\left|\frac{v^\top u_k}{R}\right| = \left|\frac{\left[v - (v^\top\theta_k/R)\theta_k/R\right]^\top u_k}{R}\right| \le\frac{U}{R}\sqrt{1-\left(\frac{v^\top\theta_k}{R}\right)^2}.

The square-root function here is concave as a function of the cosine on [−1,1][-1,1]. Taking the conditional expectation and using Jensen’s inequality therefore yields

a≤2b E[1−(v⊤θkR)2∣F0]≤2b1−mk2.(27)a \le\sqrt{2b}\,\mathbb{E}\left[\left.\sqrt{1-\left(\frac{v^\top\theta_k}{R}\right)^2}\right|\mathcal{F}_0\right] \le\sqrt{2b}\sqrt{1-m_k^2}. \tag*{(27)}

Thus mk2+a2/(2b)≤1m_k^2 + a^2/(2b) \le1 is necessary at every state from which an update with the assumed positive drift is taken. For the exact discrete optimum over NN updates, applying it at k=N−1k = N - 1 requires

m02+Na22b≤1.m_0^2 + \frac{Na^2}{2b} \le1.

A constant positive drift cannot therefore describe an indefinitely growing physical alignment. The optimization proof is global over the prescribed nonnegative schedules of the scalar dynamics, while its interpretation as WN alignment is restricted to intervals on which the geometric assumptions remain realizable. Likewise, the continuous optimum need not satisfy the small-step condition automatically: its largest relative squared step is 2bη02=a2/(2bm02)2b\eta_0^2 = a^2/(2bm_0^2). This ratio must be small when the ODE optimum is used through the continuous approximation, although the discrete optimality identity itself does not require it.

Affine endpoint construction

Let K≥1K \ge1 be the integer decay horizon from § 3.3. For α>0\alpha> 0, the inverse-square-root curve in Eq. (12) satisfies q(0)=1>q(K)>0q(0) = 1 > q(K) > 0. Write an affine transformation of this curve as sq(k)+dsq(k) + d, where ss sets its scale and dd its offset. An initial decay LR pp and final LR ηmin⁡\eta_{\min} require

s+d=p,sq(K)+d=ηmin⁡.s + d = p,\qquad sq(K) + d = \eta_{\min}.

Since q(K)<1q(K) < 1, these equations uniquely determine

s=p−ηmin⁡1−q(K),d=ηmin⁡−sq(K).s = \frac{p-\eta_{\min}}{1-q(K)},\qquad d = \eta_{\min}-sq(K).

Substitution gives the decay sequence

ηmin⁡+(p−ηmin⁡)q(k)−q(K)1−q(K).\eta_{\min} + (p-\eta_{\min})\frac{q(k)-q(K)}{1-q(K)}.

For p>ηmin⁡p > \eta_{\min}, we have s>0s > 0, so the schedule decreases monotonically between its prescribed endpoints. For any $0 \le i < j \le K$, subtracting the two LR values gives

ηAIR(i)−ηAIR(j)=s(q(i)−q(j)).\eta_{\mathrm{AIR}}(i)-\eta_{\mathrm{AIR}}(j)=s\bigl(q(i)-q(j)\bigr).

Since p−ηmin⁡=s[1−q(K)]p-\eta_{\min}=s[1-q(K)], it follows that

ηAIR(i)−ηAIR(j)p−ηmin⁡=q(i)−q(j)1−q(K).(28)\frac{\eta_{\mathrm{AIR}}(i)-\eta_{\mathrm{AIR}}(j)}{p-\eta_{\min}}=\frac{q(i)-q(j)}{1-q(K)}. \tag*{(28)}

Thus, each interval accounts for the same fraction of the total LR reduction as in the original curve. In particular, all per-step decreases are scaled by the same positive factor. For the continuous interpolation, ηAIR(t)=sq′(t)\eta_{\mathrm{AIR}}(t)=sq'(t), so all decay rates are scaled by the same positive factor and their relative magnitudes are preserved. These properties establish the endpoint calibration and preservation of the relative magnitudes of decay rates. The unconstrained optimum in § 3.2 thereby inspires the practical affine schedule.

Matching the discrete LR sum

Fix the warmup rule, decay horizon KK, time scale α\alpha, terminal LR ηmin⁡\eta_{\min}, and the indices at which LR values are executed. Write the warmup values as cip+dic_i p + d_i, where the coefficients ci,dic_i,d_i are fixed independently of the peak pp. This includes linear warmup and fixed multiplicative adjustments to it. Let I⊆{0,…,K}\mathcal{I} \subseteq\{0,\ldots,K\} be the executed decay indices, and let nd=∣I∣n_d = |\mathcal{I}|. For a target sum SdS_d over all executed LR values, define

CW:=∑i∈warmupci,BW:=∑i∈warmupdi,FI:=∑k∈If(k).C_W := \sum_{i\in\mathrm{warmup}} c_i,\qquad B_W := \sum_{i\in\mathrm{warmup}} d_i,\qquad F_{\mathcal{I}} := \sum_{k\in\mathcal{I}} f(k).

The first two quantities collect the peak-dependent and constant warmup contributions, and the third collects the peak coefficient from decay. Summing the warmup and Eq. (13) gives

Sd=BW+p(CW+FI)+ηmin⁡(nd−FI).S_d = B_W + p(C_W + F_{\mathcal{I}}) + \eta_{\min}(n_d - F_{\mathcal{I}}).

If CW+FI>0C_W + F_I > 0, the peak has the unique solution

p=Sd−BW−ηmin⁡(nd−FI)CW+FI.(29)p = \frac{S_d - B_W - \eta_{\min}(n_d - F_I)}{C_W + F_I}. \tag*{(29)}

The calibrated peak must still exceed ηmin⁡\eta_{\min} and satisfy the prescribed warmup constraints. Under the positive-coefficient condition above, a prescribed peak fixes the sum, whereas a prescribed feasible sum determines the peak. The endpoint k=Kk = K contributes ηmin⁡\eta_{\min} exactly when it belongs to II. Thus the sum must use the executed LR values rather than infer endpoint inclusion from a logging convention.

For example, consider W≥1W \ge1 linear warmup values ηinit+(p−ηinit)i/W\eta_{\mathrm{init}} + (p - \eta_{\mathrm{init}})i/W at i=0,…,W−1i = 0,\ldots,W - 1, followed by all decay values k=0,…,Kk = 0,\ldots,K. For this convention,

CW=W−12,BW=ηinit(W−CW),nd=K+1.C_W = \frac{W - 1}{2}, \qquad B_W = \eta_{\mathrm{init}}(W - C_W), \qquad n_d = K + 1.

There are W+K+1W + K + 1 values, the peak occurs at the first decay step, and both decay endpoints are included. The warmup and decay regime requires p≥ηinitp \ge\eta_{\mathrm{init}} and p>ηmin⁡p > \eta_{\min}. For any other execution convention, the same general formula applies with its actual coefficients and index set.

Experimental Settings and Evaluation Protocol

This appendix details the experimental setup in § 4 and § 5.

Training settings and comparison controls

Table 3 summarizes the training settings for the smaller-scale scheduler experiments. The larger-scale comparison uses 790 M activated and 35.7 B total parameters, 500 B training tokens, and 168 hours per run on 32 GPUs. Within each fixed-norm scheduler comparison, the training setting is held fixed and the LR schedule is varied.

ConfigurationValue
Activated parameters110 M
Total parameters4.7 B
Training tokens128 B
OptimizerMuon
Global batch size128
NormalizationFrobenius norm
Normalization epsilon10−810^{-8}
Scheduled optimizer updates32,047
Warmup updates3,040
Minimum LR10−510^{-5}
Cosine peak LR1.07×10−31.07 \times10^{-3}
AIR peak LR1.946367×10−31.946367 \times10^{-3}
AIR decay time scaleα=K/8\alpha= K/8
GPUs per run16
Wall-clock per complete run48 hours

Table 3. Training settings for the smaller-scale scheduler comparisons. The peak LRs specify the main comparison between cosine and AIR; other schedule peaks are calibrated to the same cumulative LR.

The LR curves in Figure 3(a) show a linear warmup illustration; the comparison controls concern the complete training schedule, including its batch-size ramp. The observed terminal training loss step for these runs is 32,046. Logging indices and scheduled endpoints are therefore distinguished when pairing losses or evaluating a schedule sum.

Within a fixed-norm comparison, the model, token budget, optimizer settings, batch-size schedule, constrained parameter blocks, and their radii are held fixed. The complete optimizer-step LR sequence is the object being compared. Warmup and the last executed update are included in its LR sum. For matched-sum comparisons with cosine, peak calibration follows Appendix § 10.§ 6. This leaves peak LR free to differ between schedules while holding the specified sum, horizon, warmup, and terminal LR fixed.

Stage replacements require a separate accounting of the resulting sequence. Replacing only one interval can alter its total LR sum or create a jump at the boundary. These properties follow from the actual combined curve and cannot be inferred from the controls of the original full schedules. A shared late schedule means equal LR values at every shared optimizer step, not merely equal endpoints or equal integrated LR. Likewise, varying an entire decay exponent is a different intervention from fixing the early schedule and replacing only the late part.

Compared decay shapes

Let k=0,…,Kk = 0, \ldots, K index the decay stage as in Eq. (12), and write τ=k/K\tau= k/K. Thus τ∈[0,1]\tau\in[0, 1] is the fraction of the decay stage completed. Table 4 gives the raw shape q(k)q(k) for each schedule. All shapes satisfy q(0)=1q(0) = 1 and use the same affine endpoint transformation and LR interpolation in Eqs. (12)–(13), with their own qq substituted for the inverse-square-root curve. The peak is then calibrated to match the cosine baseline’s cumulative LR, including warmup, following Appendix § 10.§ 6. This holds the warmup duration and terminal LR fixed while allowing the peak to differ. The raw inverse-square-root reference in Figure 3(a) is excluded from these controls.

The square-root shape used outside Table 1 is 1−τβ1 - \tau^\beta with β=1/2\beta= 1/2, and the quadratic shape is (1−τ)β(1 - \tau)^\beta with β=2\beta= 2. In Table 1 each schedule is named by the shape in this table, written up to the affine transformation: the time variable tt absorbs an affine rescaling, so t=τt = \tau for the polynomial shapes and t=1+k/αt = 1 + k/\alpha for the inverse-power and logarithmic ones, and t=ρτt = \rho\tau for the exponential shape, and the endpoint normalization is left implicit. Cosine, Half-cosine, Rear-cosine, and Cycloid are not separated by such a formula and keep their names there.

For the symmetric double-cosine schedule, the two cosine segments join at τ=1/2\tau= 1/2 and qdc=1/2q_{\mathrm{dc}} = 1/2:

qdc(τ)={34+14cos⁡(2πτ),0≤τ≤12,14+14cos⁡(2πτ−π),12<τ≤1.q_{\mathrm{dc}}(\tau) = \begin{cases} \frac{3}{4} + \frac{1}{4}\cos(2\pi\tau), & 0 \le\tau\le\frac{1}{2}, \\ \frac{1}{4} + \frac{1}{4}\cos(2\pi\tau- \pi), & \frac{1}{2} < \tau\le1. \end{cases}

For Cycloid, θ∈[0,θ1]\theta\in[0,\theta_1] is the unique solution of the relation in the table. For variable power, the exponent varies linearly from 1/21/2 to βend\beta_{\mathrm{end}}.

Loss summaries

Tables 1 and 2, and the robustness comparison, use the same terminal loss convention. For training loss, take the last 100 finite logged values in step order, denoted x1,…,x100x_1,\ldots,x_{100}, initialize z1=x1z_1=x_1, and recursively compute

zj=0.99zj−1+0.01xj,j=2,…,100.(30)z_j = 0.99z_{j-1} + 0.01x_j,\qquad j=2,\ldots,100. \tag*{(30)}

The reported value is z100z_{100}; evaluation loss is the last finite logged evaluation value without smoothing. The last 100 observations need not span exactly 100 optimizer steps if a log has gaps. The smaller-scale comparison ends with training loss at step 32,046 and evaluation loss at step 32,000; the corresponding larger-scale steps are 62,602 and 62,000. These are terminal summaries of individual trajectories, not averages across independent training seeds.

Training restarts are stitched by optimizer step, with the later-started segment taking precedence at duplicate steps. For reference loss curves, we first pair raw losses at common optimizer steps and subtract the specified reference, then apply a debiased EMA with decay 0.99 to the complete paired sequence. For paired differences d1,…,djd_1,\ldots,d_j, the plotted value is

∑i=1j0.99j−idi∑i=1j0.99j−i.\frac{\sum_{i=1}^{j}0.99^{j-i}d_i}{\sum_{i=1}^{j}0.99^{j-i}}.

Smoothing advances once per retained paired observation; missing steps are not filled by interpolation. The terminal reference loss points in Figure 6 use the final value of this curve. Figure 5(c) does not use this curve: for each of the eight runs it reports the terminal training loss summary of Eq. (30), and its stage-swap gaps are differences of these terminal values. No independent-seed confidence interval is inferred from adjacent training steps.

Downstream benchmarks

Table 1 reports performance on MMLU [8], MMLU-Pro [31], AGIEval [37], WinoGrande [22], BBH [28], DROP [6], GSM8k [4], MATH [9], HumanEval [3], MBPP+ [16], and TriviaQA [12].

Hyperparameter sweeps and LR sensitivity

For the robustness sweep in § 4.3, we report each family’s LR sensitivity. For a family with configurations i=1,…,ni=1,\ldots,n of terminal loss LiL_i and peak LR ηi\eta_i, the LR sensitivity is

S=1Wlog⁡(1n∑i=1nLi−min⁡iLi),Wlog⁡=ln⁡max⁡iηimin⁡iηi.(31)S=\frac{1}{W_{\log}}\left(\frac{1}{n}\sum_{i=1}^{n}L_i-\min_i L_i\right),\qquad W_{\log}=\ln\frac{\max_i\eta_i}{\min_i\eta_i}. \tag*{(31)}

the average loss above the family’s best configuration per e-fold of peak LR: the numerator is the mean terminal loss above the family’s best measured configuration, and dividing by the log-span Wlog⁡W_{\log} of the swept peak LR normalizes for search width. This is a descriptive statistic over each family’s measured range and is not extrapolated beyond it. Table 5 reports SS on training and evaluation loss.

Table 5 gives the configuration counts and logarithmic spans. Search ranges and configuration counts differ across families; the normalization by Wlog⁡W_{\log} does not make their tuning budgets identical. The fitted curves visualize the measured points, whereas SS is computed directly from those points. For a family with nn configurations, the fit has degree min⁡(3,n−1)\min(3,n-1); fits with at least three points are constrained to be convex over the observed peak-LR interval.

Stage swaps

Figure 5 compares AIR with Cycloid at θ1=π\theta_1=\pi, which shares its peak LR, warmup, terminal LR, and full-schedule cumulative LR. This comparator differs from the evaluation-selected Cycloid at θ1=3π/4\theta_1=3\pi/4 in

Table 6. Measured shape settings and peak-LR ranges for the smaller-scale sweep. Decimal shape settings are rounded to three places; the other values are exact. Counts and logarithmic spans appear in Table 5.

FamilyShape settingsPeak LR (10−310^{-3})
Cycloidθ1∈{π/2,3π/4,π}\theta_{1} \in\{\pi/2,3\pi/4,\pi\}1.377–1.946
(1−t)β(1-t)^{\beta}β∈{2,3,4}\beta\in\{2,3,4\}1.528–2.330
1−tβ1-t^{\beta}β∈{1/2,1/3,1/5}\beta\in\{1/2,1/3,1/5\}1.528–2.683
Logarithmicα/K∈{1/8,1/16,1/32}\alpha/K \in\{1/8,1/16,1/32\}1.541–1.916
Exponentialρ∈{2,3.594,5.549,7}\rho\in\{2,3.594,5.549,7\}1.489–3.023
Inverse squareα/K∈{1,1/2,1/4}\alpha/K \in\{1,1/2,1/4\}1.528–2.683
Inverse timeα/K∈{0.864,0.400,0.243,0.166,0.125}\alpha/K \in\{0.864,0.400,0.243,0.166,0.125\}1.382–2.487
Variable powerβend∈{1.771,2}\beta_{\mathrm{end}} \in\{1.771,2\}, α=K/8\alpha=K/81.946–2.031
AIRα/K∈{0.557,1/4,1/8,1/16,1/32}\alpha/K \in\{0.557,1/4,1/8,1/16,1/32\}1.382–2.928

Table 1. The two intersections occur at approximately global steps 4,346 and 16,219 and define the early, middle, and late intervals. Each replacement follows the donor schedule only within the named interval and the recipient schedule outside it. Matching the cumulative LR of the two original schedules does not imply that an individual stage replacement preserves that sum. The terminal summary and reference loss smoothing are specified in Appendix § 11.§ 3.

Late decay exponents

The exponent interventions in Eq. (14) switch at global step 25,955, corresponding to Ks/K≃0.79K_s/K \simeq0.79 of the decay stage after warmup. They use γ∈{0.5,0.75,1,2,3}\gamma\in\{0.5, 0.75, 1, 2, 3\}, with the same earlier schedule, junction LR, and configured terminal LR. Their cumulative LR is not exactly matched. Trapezoidal integration of the recorded LR traces gives deviations below 1 % relative to the AIR reference. Figure 6 reports same-step loss differences to the full-run AIR reference with the full-sequence EMA convention in Appendix § 11.§ 3.

Bounds for Early- and Late-Schedule Differences

Common setting

We use the continuous relaxation of the decay-step coordinate k∈[0,K]k \in[0, K], with a fixed split 0≤Ks<K0 \le K_s < K. For i∈{1,2}i \in\{1, 2\}, the expected alignment mim_i follows the dynamics from § 3,

m˙i(k)=aηi(k)−bηi(k)2mi(k),mi(0)=m0≥0,\dot{m}_i(k) = a\eta_i(k) - b\eta_i(k)^2m_i(k), \qquad m_i(0) = m_0 \ge0,

with common a,b>0a, b > 0 and bounded, measurable, nonnegative schedules ηi\eta_i. Dots denote derivatives with respect to this continuous step coordinate. The expectations retain the same initial conditioning F0\mathcal{F}_0 as Eq. (5); the conditioning is not reset at KsK_s. Write Δm=m1−m2\Delta m = m_1 - m_2 and Δη=η1−η2\Delta\eta= \eta_1 - \eta_2. A shared early schedule and the common initial alignment give Δm(Ks)=0\Delta m(K_s) = 0 by uniqueness of the linear ODE.

For each schedule, the retention factor from Eq. (9) is

Hi(K,k)=exp⁡[−b∫kKηi(s)2 ds],H_i(K,k) = \exp\left[-b\int_k^K \eta_i(s)^2\,\mathrm{d}s\right],
H12(K,k)=H1(K,k)H2(K,k).(32)H_{12}(K,k) = \sqrt{H_1(K,k)H_2(K,k)}. \tag*{(32)}

The second factor retains a difference under the average retraction coefficient b(η12+η22)/2b(\eta_1^2+\eta_2^2)/2. When the late schedules coincide, H1=H2=H12=HH_1 = H_2 = H_{12} = H on that interval.

Varying early-stage schedules

If η1=η2=η\eta_1 = \eta_2 = \eta on [Ks,K][K_s,K], subtracting the state equations gives

dΔm(k)dk=−bη(k)2Δm(k).\frac{\mathrm{d}\Delta m(k)}{\mathrm{d}k} = -b\eta(k)^2\Delta m(k).

Multiplication by exp⁡[b∫Kskη(s)2 ds]\exp\left[b\int_{K_s}^{k}\eta(s)^2\,\mathrm{d}s\right] makes the derivative of the product zero. Evaluating this constant at KsK_s and KK yields

Δm(K)=H(K,Ks)Δm(Ks).(33)\Delta m(K) = H(K,K_s)\Delta m(K_s). \tag*{(33)}

The argument never divides by Δm\Delta m, so it also covers zero or negative entering differences. Since 0<H(K,Ks)≤10 < H(K,K_s) \le1, the sign is preserved and the magnitude cannot increase, proving Eq. (15). Ignoring the effects of stochastic noise, the expected-alignment ordering is therefore preserved along the shared tail.

Early schedules with equal integrated LR. Assume additionally 0≤mi(k)≤10 \le m_i(k) \le1 on [0,Ks][0,K_s] and ∫0Ksη1(k) dk=∫0Ksη2(k) dk\int_0^{K_s}\eta_1(k)\,\mathrm{d}k = \int_0^{K_s}\eta_2(k)\,\mathrm{d}k. Integrating the two state equations cancels the common initial alignment and the cumulative new-alignment terms:

Δm(Ks)=−b∫0Ks[η1(k)2m1(k)−η2(k)2m2(k)]dk.\Delta m(K_s) = -b\int_0^{K_s}\left[\eta_1(k)^2m_1(k)-\eta_2(k)^2m_2(k)\right]\mathrm{d}k.

Each retraction integral is nonnegative and at most ∫0Ksηi(k)2 dk\int_{0}^{K_s} \eta_i(k)^2\,\mathrm{d}k. Their absolute difference is therefore bounded by the larger integral, giving

∣Δm(K)∣≤bH(K,Ks)max⁡i∈{1,2}∫0Ksηi(k)2 dk.(34)|\Delta m(K)| \le bH(K,K_s)\max_{i\in\{1,2\}}\int_{0}^{K_s}\eta_i(k)^2\,\mathrm{d}k. \tag*{(34)}

Equal integrated LR cancels the cumulative new-alignment difference but does not equate the retraction corrections.

Discrete counterpart. Take Ks,KK_s,K to be integers and let Δmk=m1,k−m2,k\Delta m_k=m_{1,k}-m_{2,k} for the recurrences in Eq. (6). If they share ηk\eta_k for k=Ks,…,K−1k=K_s,\ldots,K-1, their difference obeys Δmk+1=Δmk/1+2bηk2\Delta m_{k+1}=\Delta m_k/\sqrt{1+2b\eta_k^2}. Iteration gives

ΔmK=ΔmKs∏k=KsK−1(1+2bηk2)−1/2.\Delta m_K=\Delta m_{K_s}\prod_{k=K_s}^{K-1}(1+2b\eta_k^2)^{-1/2}.

This exact discrete relation has the same sign-preserving attenuation property without replacing the product by a continuous exponential.

Varying late-stage schedules

On [Ks,K][K_s,K], the algebraic identity

η12m1−η22m2=η12+η222Δm+(η1+η2)(m1+m2)2Δη\eta_1^2m_1-\eta_2^2m_2=\frac{\eta_1^2+\eta_2^2}{2}\Delta m+\frac{(\eta_1+\eta_2)(m_1+m_2)}{2}\Delta\eta

gives the exact difference equation

dΔmdt=−b2(η12+η22)Δm+[a−b2(η1+η2)(m1+m2)]Δη.(35)\frac{\mathrm{d}\Delta m}{\mathrm{d}t}=-\frac{b}{2}(\eta_1^2+\eta_2^2)\Delta m+\left[a-\frac{b}{2}(\eta_1+\eta_2)(m_1+m_2)\right]\Delta\eta. \tag*{(35)}

Using the integrating factor associated with the first term yields

Δm(K)=H12(K,Ks)Δm(Ks)+∫KsKH12(K,k)[a−b2(η1(k)+η2(k))(m1(k)+m2(k))]Δη(k) dk.(36)\begin{aligned} \Delta m(K)={}&H_{12}(K,K_s)\Delta m(K_s)\\ &+\int_{K_s}^{K}H_{12}(K,k)\left[a-\frac{b}{2}(\eta_1(k)+\eta_2(k))(m_1(k)+m_2(k))\right]\Delta\eta(k)\,\mathrm{d}k. \tag*{(36)} \end{aligned}

The first term propagates the entering difference, and the second accumulates differences caused by the later LR choices. This identity is exact within the ODE and does not assume a small schedule perturbation.

Choose common bounds 0≤ηi≤P0\le\eta_i\le P and 0≤mi≤M0\le m_i\le M on the late interval. Here PP bounds both schedules; it does not require them to have equal peaks. The integral solution preserves nonnegative states, and m˙i≤aP\dot{m}_i\le aP, so M=max⁡imi(Ks)+aP(K−Ks)M=\max_i m_i(K_s)+aP(K-K_s) is a valid finite choice. When the alignments stay in their physical range, M=1M=1 also suffices. Since

0≤b2(η1+η2)(m1+m2)≤2bPM,0\le\frac{b}{2}(\eta_1+\eta_2)(m_1+m_2)\le2bPM,

the response coefficient in brackets lies between a−2bPMa-2bPM and aa. Its absolute value is bounded by

L12=max⁡{a,∣a−2bPM∣}.L_{12}=\max\{a,|a-2bPM|\}.

This preserves the cancellation between alignment gain and retraction before taking absolute values. When only the late-stage schedule varies, the shared early schedule gives Δm(Ks)=0\Delta m(K_s)=0. Applying the triangle inequality to Eq. (36) therefore gives

∣Δm(K)∣≤L12∫KsKH12(K,k)∣Δη(k)∣ dk,|\Delta m(K)|\le L_{12}\int_{K_s}^{K}H_{12}(K,k)|\Delta\eta(k)|\,\mathrm{d}k,

which proves Eq. (16).

If both stages vary, retain the first term in Eq. (36) and apply the same bound to its integral. Since H12>0H_{12}>0, this gives

∣Δm(K)∣≤H12(K,Ks)∣Δm(Ks)∣+L12∫KsKH12(K,k)∣Δη(k)∣ dk,\lvert\Delta m(K)\rvert\le H_{12}(K,K_s)\lvert\Delta m(K_s)\rvert+ L_{12}\int_{K_s}^{K} H_{12}(K,k)\lvert\Delta\eta(k)\rvert\,\mathrm{d}k,

proving Eq. (17) without requiring a common entering alignment. Together, these comparisons characterize how LR schedules promote expected directional alignment under common coefficients, thereby providing a mechanism for improved training loss.

Slope bounds and endpoint information

Assume Δm(Ks)=0\Delta m(K_s)=0, absolutely continuous late schedules, and a common terminal LR η1(K)=η2(K)\eta_1(K)=\eta_2(K). Let ε1=ess sup⁡[Ks,K]∣η˙1−η˙2∣<∞\varepsilon_1=\operatorname{ess\,sup}_{[K_s,K]}\lvert\dot{\eta}_1-\dot{\eta}_2\rvert<\infty. Subtracting the terminal values and integrating the derivative difference gives

Δη(k)=−∫kK(η˙1(s)−η˙2(s)) ds,∣Δη(k)∣≤ε1(K−k).\Delta\eta(k)=-\int_k^K\bigl(\dot{\eta}_1(s)-\dot{\eta}_2(s)\bigr)\,\mathrm{d}s,\qquad\lvert\Delta\eta(k)\rvert\le\varepsilon_1(K-k).

Substitution into Eq. (16), followed by H12≤1H_{12}\le1, yields

∣Δm(K)∣≤L12ε1∫KsKH12(K,k)(K−k) dk(37)\lvert\Delta m(K)\rvert\le L_{12}\varepsilon_1\int_{K_s}^{K}H_{12}(K,k)(K-k)\,\mathrm{d}k \tag*{(37)}
≤L12ε1(K−Ks)22.(38)\le\frac{L_{12}\varepsilon_1(K-K_s)^2}{2}. \tag*{(38)}

If the schedules also share their LR at KsK_s, integration from the left endpoint gives ∣Δη(k)∣≤ε1min⁡{k−Ks,K−k}\lvert\Delta\eta(k)\rvert\le\varepsilon_1\min\{k-K_s,K-k\}. The integral of this envelope is ε1(K−Ks)2/4\varepsilon_1(K-K_s)^2/4, hence

∣Δm(K)∣≤L12ε1(K−Ks)24.\lvert\Delta m(K)\rvert\le\frac{L_{12}\varepsilon_1(K-K_s)^2}{4}.

A shared early schedule alone does not impose this extra endpoint condition if a jump is allowed at the switch.

Applicability to the late-stage ablation. The schedules in Eq. (14) are bounded and absolutely continuous for every γ>0\gamma>0, so the LR-difference bound in Eq. (16) applies under the stated alignment dynamics. For 0<γ<10<\gamma<1, however, the continuous terminal derivative is unbounded. Indeed, writing the positive junction-to-floor difference directly, differentiation gives

η˙γ(k)=−γ[ηAIR(Ks)−ηmin⁡]K−Ks(K−kK−Ks)γ−1,k<K.\dot{\eta}_{\gamma}(k)=-\frac{\gamma[\eta_{\mathrm{AIR}}(K_s)-\eta_{\min}]}{K-K_s}\left(\frac{K-k}{K-K_s}\right)^{\gamma-1},\qquad k<K.

Its magnitude diverges at KK when γ<1\gamma<1, is constant when γ=1\gamma=1, and vanishes there when γ>1\gamma>1. A comparison with the smooth AIR tail, or with a distinct exponent, therefore need not satisfy the finite ε1\varepsilon_1 condition. Accordingly, comparisons involving 0<γ<10<\gamma<1 are governed by the general LR-difference bound rather than its slope-based corollary. This bound connects the exponent intervention to expected directional alignment, complementing the empirical loss comparison in § 5.3.

Knowing only the terminal slope. For twice continuously differentiable schedules with a common terminal LR and Δm(Ks)=0\Delta m(K_s)=0, define δK=∣η˙1(K)−η˙2(K)∣\delta_K=\lvert\dot{\eta}_1(K)-\dot{\eta}_2(K)\rvert and C2=sup⁡[Ks,K]∣η¨1−η¨2∣C_2=\sup_{[K_s,K]}\lvert\ddot{\eta}_1-\ddot{\eta}_2\rvert. These measure the terminal slope difference and the curvature difference over the full interval. Taylor expansion with integral remainder gives

∣Δη(k)∣≤δK(K−k)+C22(K−k)2.\lvert\Delta\eta(k)\rvert\le\delta_K(K-k)+\frac{C_2}{2}(K-k)^2.

Substitution into the LR-difference bound yields

∣Δm(K)∣≤L12(δK(K−Ks)22+C2(K−Ks)36).\lvert\Delta m(K)\rvert\le L_{12}\left(\frac{\delta_K(K-K_s)^2}{2}+\frac{C_2(K-K_s)^3}{6}\right).

Equal terminal slopes remove the first contribution, while the curvature term captures differences across the interior of the schedules.

Affine power curves and the scope of the bounds

As an auxiliary example on [Ks,K][K_s,K], fix a common initial LR P≥ηmin⁡≥0P \ge\eta_{\min} \ge0 and a time scale αtail>0\alpha_{\mathrm{tail}} > 0. For exponent β>0\beta> 0, define

qβ(k)=(1+k−Ksαtail)−β,ηβ(k)=ηmin⁡+(P−ηmin⁡)qβ(k)−qβ(K)1−qβ(K).q_\beta(k)=\left(1+\frac{k-K_s}{\alpha_{\mathrm{tail}}}\right)^{-\beta}, \qquad\eta_\beta(k)=\eta_{\min}+(P-\eta_{\min})\frac{q_\beta(k)-q_\beta(K)}{1-q_\beta(K)}.

This affine inverse-power family shares its initial and terminal LRs; β=1/2\beta=1/2 gives an inverse-square-root shape. It is distinct from the terminal-power variants indexed by γ\gamma in Eq. (14). Direct differentiation gives

∣η˙β(k)∣=P−ηmin⁡αtailβ(1+(k−Ks)/αtail)−β−11−(1+(K−Ks)/αtail)−β.\left|\dot{\eta}_\beta(k)\right|=\frac{P-\eta_{\min}}{\alpha_{\mathrm{tail}}}\frac{\beta\left(1+(k-K_s)/\alpha_{\mathrm{tail}}\right)^{-\beta-1}}{1-\left(1+(K-K_s)/\alpha_{\mathrm{tail}}\right)^{-\beta}}.

The terminal slope follows by setting k=Kk=K. On a compact interval of positive exponents, the slope is jointly continuous in β\beta and kk, so close exponents give uniformly close slopes. Combined with the preceding bounds, this continuity shows that nearby exponents produce nearby terminal expected alignments from a common entering state. When the early stage also varies, Eq. (36) incorporates the resulting entering-state difference.

Infinitesimal sensitivity to the time of an LR change

For an admissible family ηε=η+εδη\eta_\varepsilon=\eta+\varepsilon\delta\eta with bounded perturbation δη\delta\eta and a common initial state, let δm=∂εmε∣ε=0\delta m=\left.\partial_\varepsilon m_\varepsilon\right|_{\varepsilon=0}. It is the first-order change in expected alignment induced by the LR perturbation. Differentiating the state equation gives

dδmdt=−bη2δm+(a−2bηm)δη,δm(0)=0.\frac{\mathrm{d}\delta m}{\mathrm{d}t}=-b\eta^2\delta m+(a-2b\eta m)\delta\eta,\qquad\delta m(0)=0.

This differentiation is justified directly from the integral solution: bounded η\eta and δη\delta\eta on the finite interval uniformly bound both the integrand and its derivative for sufficiently small admissible ε\varepsilon. Differentiation under the integral sign therefore gives an absolutely continuous δm\delta m satisfying this linear equation. Applying the same integrating factor gives

δm(K)=∫0KH(K,k)[a−2bη(k)m(k)]δη(k) dk.\delta m(K)=\int_0^K H(K,k)\left[a-2b\eta(k)m(k)\right]\delta\eta(k)\,\mathrm{d}k.

Later perturbations are retained more strongly through H(K,k)H(K,k), while the factor a−2bη(k)m(k)a-2b\eta(k)m(k) determines their local alignment gain. The resulting sensitivity kernel identifies when and where an LR change most strongly influences terminal alignment.

References

  1. [1]Kang An, Jiaxiang Li, Donald Goldfarb, and Shiqian Ma. Demystifying Manifold Constraints in LLM Pre-training, 2026. URL https://arxiv.org/abs/2605.04418.
  2. [2]Jeremy Bernstein. Modular manifolds. *Thinking Machines Lab: Connectionism*, 2025. doi: 10.64434/tml.20250926. https://thinkingmachines.ai/blog/modular-manifolds/.DOI
  3. [3]Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating Large Language Models Trained on Code, 2021. URL https://arxiv.org/abs/2107.03374.
  4. [4]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training Verifiers to Solve Math Word Problems, 2021. URL https://arxiv.org/abs/2110.14168.
  5. [5]Francesco D’Angelo, Maksym Andriushchenko, Aditya Varre, and Nicolas Flammarion. Why Do We Need Weight Decay in Modern Deep Learning? In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, *Advances in Neural Information Processing Systems*, volume 37, pages 23191–23223. Curran Associates, Inc., 2024. doi: 10.52202/079017-0730. URL https://proceedings.neurips.cc/paper_files/paper/2024/file/29496c9249d6e08cec469f451ebff0-Paper-Conference.pdf.
  6. [6]Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)*, pages 2368–2378, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1246. URL https://aclanthology.org/N19-1246/.DOI
  7. [7]Alexander Hägele, Elie Bakouch, Atli Kosson, Loubna Ben Allal, Leandro Von Werra, and Martin Jaggi. Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, *Advances in Neural Information Processing Systems*, volume 37, pages 76232–76264. Curran Associates, Inc., 2024. doi: 10.52202/079017-7427. URL https://proceedings.neurips.cc/paper_files/paper/2024/file/8b970e15a89bf5d12542810df8eae8fc-Paper-Conference.pdf.arxiv.org/abs/2405.18392
  8. [8]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring Massive Multitask Language Understanding. In *9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021*. OpenReview.net, 2021. URL https://openreview.net/forum?id=d7KBjmI3mGm.
  9. [9]Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring Mathematical Problem Solving With the MATH Dataset. In J. Vanschoren and S. Yeung, editors, *Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks*, volume 1, 2021. URL https://datasets-benchmarks-proceedings.neurips.cc/paper_files/paper/2021/file/be833ac3d0db773eb2dcb10a17836a1-Paper-round2.pdf.
  10. [10]Shengding Hu, Yuge Tu, Xu Han, Ganqu Cui, Chaoqun He, Weilin Zhao, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Xinrong Zhang, Zhen Leng Thai, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, dahai li, Zhiyuan Liu, and Maosong Sun. MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies. In *First Conference on Language Modeling*, 2024. URL https://openreview.net/forum?id=3X2L2TrFrOf.arxiv.org/abs/2404.06395
  11. [11]Keller Jordan, Yuchen Jin, Vlado Boza, You Jiacheng, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https://kellerjordan.github.io/posts/muon/.
  12. [12]Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. In Regina Barzilay and Min-Yen Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v1/P17-1147. URL https://aclanthology.org/P17-1147/.
  13. [13]Atli Kosson, Jeremy Welborn, Yang Liu, Martin Jaggi, and Xi Chen. Weight Decay may matter more than μP for Learning Rate Transfer in Practice. In C. VonDrick, B. Hariharan, C. Raffel, L. Pinto, D. Yang, and A. Faust, editors, International Conference on Learning Representations, volume 2026, pages 67012–67048, 2026. URL https://proceedings.iclr.cc/paper_files/paper/2026/file/6dc63b4063c978cf195bc151782e8152a-Paper-Conference.pdf.arxiv.org/abs/2510.19093
  14. [14]Binghui Li, Zilin Wang, Fengling Chen, Shiyang Zhao, Ruiheng Zheng, and Lei Wu. Optimal Learning Rate Schedules under Functional Scaling Laws: Power Decay and Warmup–Stable–Decay (Extended Abstract). In Steve Hanneke and Tor Lattimore, editors, Proceedings of Thirty Ninth Conference on Learning Theory, volume 336 of Proceedings of Machine Learning Research, pages 4722–4723. PMLR, 29 Jun–03 Jul 2026. URL https://proceedings.mlr.press/v336/li26b.html.arxiv.org/abs/2602.06797
  15. [15]Zhiyuan Li, Kaifeng Lyu, and Sanjeev Arora. Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Rate. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 14544–14555. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/a7453a5f026fbf6831d68bdc9cb0edca-Paper.pdf.arxiv.org/abs/2010.02916
  16. [16]Jiawei Liu, Chunqiu Steven Xia, Yutao Wang, and LINGMING ZHANG. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 21558–21572. Curran Associates, Inc., 2023. doi: 10.5220/075280-0943. URL https://proceedings.neurips.cc/paper_files/paper/2023/file/43e9d647ccd3e47b55baab53f0368686-Paper-Conference.pdf.arxiv.org/abs/2305.01210
  17. [17]Zihan Liu, Ruiheng Zheng, Shaobo Zhang, Changxin Tian, Kunlong Chen, Zhiqiang Zhang, and Lei Wu. Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining, 2026. URL https://arxiv.org/abs/2608.24814.
  18. [18]Ilya Loshchilov and Boris Ginsburg. Training nGPT, 2026. URL https://arxiv.org/abs/2608.01284.
  19. [19]Ilya Loshchilov and Frank Hutter. SGDR: Stochastic Gradient Descent with Warm Restarts. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24–26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https://openreview.net/forum?id=Skq89Scxx.arxiv.org/abs/1608.03983
  20. [20]Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.
  21. [21]Liliang Ren, Yang Liu, Yelong Shen, and Weizhu Chen. Rethinking Language Model Scaling under Transferable Hypersphere Optimization, 2026. URL https://arxiv.org/abs/2603.28743.
  22. [22]Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An Adversarial Winograd Schema Challenge at Scale. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8732–8740, April 2020. ISSN 2159-5399. doi: 10.1609/aaai.v34i05.6399. URL http://dx.doi.org/10.1609/aaai.v34i05.6399.
  23. [23]Tim Salimans and D.P. Kingma. Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL https://proceedings.neurips.cc/paper_files/paper/2016/file/ed265bc903a5a097f161d3ec064d96d2e-Paper.pdf.
  24. [24]Fabian Schaipp, Alexander Hägele, Adrien Taylor, Umut Simsekli, and Francis Bach. The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Training. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors, Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 53267–53294. PMLR, 13–19 Jul 2025. URL https://proceedings.mlr.press/v267/schaipp25a.html.arxiv.org/abs/2501.18965
  25. [25]Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. In International Conference on Learning Representations, 2017. URL https://mlanthology.org/iclr/2017/shazeer2017iclr-outrageously/.
  26. [26]Jianlin Su. Steepest Descent on Manifolds: 1. SGD + Hypersphere, Aug 2025. URL https://kexue.fm/archives/11196.
  27. [27]Jianlin Su. Steepest Descent on Manifolds: 3. Muon + Stiefel, Aug 2025. URL https://kexue.fm/archives/11221.
  28. [28]Mirac Suzgun, Nathan Scales, Nathanael Schä rli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, pages 13003–13051, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-acl.824. URL https://aclanthology.org/2023.findings-acl.824/.DOI
  29. [29]Twan van Laarhoven. L2 Regularization versus Batch and Weight Normalization, 2017. URL https://arxiv.org/abs/1706.05350.
  30. [30]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.arxiv.org/abs/1706.03762
  31. [31]Yubo Wang, Xueguang Ma, Ge Zhang, Yusheng Ni, A Bhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Alrujaj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 95266–95290. Curran Associates, Inc., 2024. doi: 10.52202/079017-3018. URL https://proceedings.neurips.cc/paper_files/paper/2024/file/ad236edc564f3e3156e1b2feafb99a24-Paper-Datasets_and_Benchmarks_Track.pdf.DOI
  32. [32]Kaiyue Wen, Zhiyuan Li, Jason Wang, David Hall, Percy Liang, and Tengyu Ma. Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape View. In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu, editors, International Conference on Learning Representations, volume 2025, pages 42840–42885, 2025. URL https://proceedings.iclr.cc/paper_files/paper/2025/file/6a1fe80a9e2dcda0b3e5fd0fd87eb097-Paper-Conference.pdf.arxiv.org/abs/2410.05192
  33. [33]Kaiyue Wen, Xingyu Dang, Kaifeng Lyu, Tengyu Ma, and Percy Liang. Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization, 2026. URL https://arxiv.org/abs/2606.16899.
  34. [34]Yihao Xiao, Jialong Sun, Zitian Gao, Zeming Wei, Chutian Wang, Ran Tao, Jiaye Teng, and Bryan Dai. Hyperball May Not Be a Free Lunch, 2026. URL https://arxiv.org/abs/2607.22444.
  35. [35]Tian Xie, Haoming Luo, Haoyu Tang, Yiwen Hu, Jason Klein Liu, Qingnan Ren, Yang Wang, Wayne Xin Zhao, Rui Yan, Bing Su, Chong Luo, and Baining Guo. Controlled LLM Training on Spectral Sphere, 2026. URL https://arxiv.org/abs/2601.08393.
  36. [36]Ruihan Xu, Jiajin Li, and Yiping Lu. On the Width Scaling of Neural Optimizers Under Matrix Operator Norms I: Row/Column Normalization and Hyperparameter Transfer, 2026. URL https://arxiv.org/abs/2603.09952.
  37. [37]Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Findings of the Association for Computational Linguistics: NAACL 2024, pages 2299–2314, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-naacl.149. URL https://aclanthology.org/2024.findings-naacl.149/.DOI
  38. [38]Zhanpeng Zhou, Wenjie Zhou, Yufei Gu, Juqiu Wang, Guanwei Zhang, Ruobing Xie, Xianbiao Qi, Yushi Bai, Kaifeng Lyu, and Shunyu Yao. From LR to ELR: A Better Heuristic for Pretraining Dynamics, 2026. URL https://hy.tencent.ai/research/elr.

Paper details

vlnc:W_2a3v9dz8
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents