WorksComputer Science
Your LR Schedule May Not Be Principled: Effective Learning Rate Scheduling in Weight-Normalized Training
Abstract
Weight-normalized (WN) training is emerging as a compelling alternative to conventional weight decay (WD), with recent studies reporting substantial pretraining loss gains. We show that these gains are driven primarily by the induced effective learning rate (ELR) trajectory rather than by normalization alone. Matching ELR trajectories largely collapses the training loss dynamics of WN and WD. Schedule design should therefore target ELR rather than the nominal learning rate (LR) alone. Building on this principle, we propose AIR (Affine Inverse-square-Root decay), which combines a theoretically derived decay with an affine adjustment to satisfy prescribed peak and terminal LRs over a finite training horizon. We further identify late-stage ELR scheduling as a key source of its gains, because a shared late schedule attenuates the effect of earlier LR choices. Experiments on 128 B-token Mixture-of-Experts pretraining demonstrate that AIR outperforms cosine and other candidate schedules in training loss, downstream performance, and hyperparameter robustness, reducing terminal training loss by approximately 0.01 relative to cosine at matched cumulative LR. Applying the same ELR trajectory to WD reproduces the corresponding training dynamics, showing that the benefits extend beyond fixed-norm training and establishing ELR as the principled scheduling object.
Introduction
In Large Language Model (LLM) pretraining, weight-normalized (WN) training [23, 33] has emerged as a promising alternative to conventional weight decay (WD) [20]. It keeps parameter norms fixed after each update, with recent work reporting lower pretraining loss, improved token efficiency, greater stability, and better hyperparameter transfer [1, 18, 33]. Yet WN and WD exhibit markedly different training dynamics under the same learning rate (LR) schedule. As shown in Figure 1(a), under cosine decay [19], WN initially trails WD but eventually outperforms it. However, under warmup-stable-decay (WSD) [10], WN remains behind WD throughout training and ends with a substantially higher loss. Therefore, understanding this contrast is essential to explaining when and why WN outperforms WD.

Figure 1. WN achieves lower final loss than WD under cosine, but significantly higher under WSD. Effective LR (nominal LR / all-parameter norm) is the key driver of loss dynamics.
We attribute this contrast to the effective learning rate (ELR) [5, 29], defined as , where is nominal LR and is the norm of a parameter block. In our study, we discover that the loss dynamics is strongly related to the decay of ELR. When WN and WD share the same WSD LR schedule, their ELR is governed entirely by the parameter norm , which evolves very differently in the two regimes (Figure 1(b,Figure 1c)). WN fixes the parameter norm, holding its ELR constant at peak throughout the entire stable phase and leading to a slow decrease in loss. By contrast, WD’s ELR keeps falling steadily, owing to the continuing growth of its parameter norm. This sustained ELR decay keeps its loss dropping much faster than WN.
A more interesting finding is that the loss dynamics of WN and WD largely collapse onto a common trajectory once we align their ELR trajectories, whether by replaying WD’s norms or by rescaling one’s blockwise LRs. (see Figure 2 and § 2.2 for details). Concurrent studies [17, 34, 38] also report similar results, corroborating our findings.

Figure 2. ELR matching collapses WD–WN training loss dynamics. Training loss minus the WD baseline under (a) cosine or (b) AIR, paired at the same step; zero denotes the reference. Arrows indicate the ELR target, matched by replaying WD norms (via norm) or rescaling blockwise LR (via LR). WD→WN also replays WD norms.
Taken together, these findings lead us to our core conclusion: ELR is the key driver of the loss dynamics. WN’s core advantage actually comes from a better ELR schedule, not normalization alone. Principled LR schedule design should therefore target ELR rather than the nominal LR alone. This core idea naturally inspires an essential research question:
What should a better ELR schedule look like?
To address this question, we study ELR scheduling under WN, a clean setting in which each block’s ELR is proportional to its nominal LR. Here, we find that decay further improves training. A theoretical analysis of WN dynamics shows that, in the unconstrained setting, decay maximizes alignment with an oracle direction at every step. To make this practical, we introduce AIR (Affine Inverse-square-Root decay), which adapts this decay through an affine transformation to meet prescribed peak and terminal LRs over a finite horizon. A further analysis shows how a shared late-stage schedule attenuates the alignment differences induced by earlier LR choices. This helps explain our finding that late-stage ELR scheduling is a key source of AIR’s gains.
Empirically, we evaluate AIR against cosine decay and alternative LR schedule families. In 128 B-token Mixture-of-Experts (MoE) [25] pretraining, AIR reduces terminal training loss by approximately 0.01 relative to cosine decay. Among these families, it achieves the lowest loss, the best downstream performance, and the greatest robustness to hyperparameter choices. Consistent with our theoretical analysis, controlled experiments show that the final training loss gaps between schedules narrow substantially when they share the same late-stage AIR schedule and differ only in their earlier LR trajectories. Transferring the induced ELR trajectory to WD also reproduces the corresponding training dynamics, showing that the benefits extend beyond fixed-norm training.
In summary, our contributions are:
We identify the induced ELR trajectory as the key driver of WN’s loss advantage over WD. Matching ELR trajectories largely aligns their training loss dynamics, highlighting ELR, rather than nominal LR alone, as the target for schedule design.
We propose AIR, an affine schedule for finite-horizon pretraining. Its decay shape follows from a theoretical analysis of directional alignment under WN, which also helps explain the role of late-stage ELR scheduling in AIR’s gains.
We empirically validate AIR against cosine decay and alternative schedule families in 128 B-token MoE pretraining, demonstrating improvements in training loss, downstream performance, and hyperparameter robustness. It reduces terminal training loss by approximately 0.01 relative to cosine at matched cumulative LR. Applying the same ELR trajectory to WD reproduces the corresponding training loss dynamics.
Effective Learning Rate as the Scheduling Principle
Effective learning rate under WN and WD
Let denote a nonzero, vectorized parameter block at optimizer step , the nominal learning rate, and the optimizer update before LR scaling. With decay coefficient and prescribed norm , decoupled WD [20] and fixed-norm WN [33] use the following update rules, respectively:
WD shrinks the current parameters before adding the optimizer update, while WN rescales the updated block to norm , preserving its direction.
The same nominal LR can produce different relative updates when parameter norms differ. Writing for the block’s norm at step , we define the blockwise norm-based effective learning rate [5, 13, 29] as
It captures how much a given optimizer update can affect the parameter direction, providing a more informative measure of its impact than the nominal LR alone (see Appendices § 9.§ 1 and § 9.§ 2).
Aligning the training dynamics of WN and WD
Experimental setup. To validate the claim that matching ELRs in Eq. (3) aligns the training dynamics of WN and WD, we match WN’s ELR to WD’s on each parameter block in two ways: (i) Via norm. We rescale each WN block at each step to the corresponding recorded WD norm while retaining the same nominal LR as WD. (ii) Via LR. We keep each WN block at its fixed norm and adjust its blockwise nominal LR to satisfy
where is the recorded WD norm before the current update, and and are the respective nominal LRs.
For the reverse matching, we replay the WD norm trajectories and adjust blockwise LRs to match WN’s ELR. Details are given in Appendix § 9.§ 3. We conduct this reverse comparison under both cosine and AIR, alongside the corresponding WD and WN baselines.
Results. Figure 2 shows that ELR matching collapses the training loss dynamics between WD and WN. Under cosine, norm replay precisely reproduces WD dynamics, with a gap less than , while LR matching at fixed WN norms also closely tracks the WD baseline, leaving a residual loss gap of several . Under both cosine and AIR, matching WN’s ELR along recorded WD norms yields a loss dynamics close to the WN baseline. The consistent collapse across both schedules supports ELR as a key driver of training dynamics and thus a principled target for schedule design.
WN as a setting for ELR schedule design
Under WD, changing the nominal LR also affects the subsequent parameter norms, coupling the two quantities in Eq. (3), so that rescaling the nominal LR alone does not reproduce a target ELR trajectory [13]. WN removes this coupling by fixing , giving , making each block’s ELR proportional to its nominal LR [1, 33]. This direct proportionality makes WN a clean setting. Therefore, we study ELR scheduling under WN, using its update rule as the starting point for the theory.
Designing a Better ELR Schedule under WN: AIR
Directional-alignment dynamics under WN
Starting from the WN update in Eq. (2), which fixes a parameter block’s norm and thus leaves only its direction to evolve, we analyze how the LR schedule affects this evolution. Let denote an idealized unit target direction, fixed throughout the analysis. Given the training history at the analysis starting point, define
This is the expected cosine alignment after updates from the same starting point. The expectation averages over minibatch sampling and other randomness in these updates, while the initial history stays fixed. For this analysis, we consider an LR schedule specified in advance and assume tangential updates of fixed magnitude, and .
Deriving the alignment dynamics. Taking the conditional expectation of the inner product with in Eq. (2) gives
where
We assume that and are constant across steps and schedules. The optimization below concerns these constant-coefficient scalar dynamics. For , a second-order expansion yields the continuous approximation, with time measured in optimizer steps,
A larger LR increases new alignment linearly but strengthens the retraction of existing alignment quadratically. Detailed derivations are provided in Appendix § 10.§ 1.
Exact integral solution. For a nonnegative, square-integrable schedule on and initial alignment , Eq. (8) has the exact solution
The solution decomposes endpoint alignment into retained initial alignment and retained contributions from subsequent updates. Each contribution is weighted by , which represents the fraction of alignment present at time that remains at the endpoint . Detailed derivations are provided in Appendix § 10.§ 2.
Unconstrained optimal schedule: inverse-square-root decay
We first maximize terminal alignment in Eq. (9) without prescribing a peak, a terminal LR, or an LR budget. For fixed , , , , completing the square in the alignment dynamics gives
Since , integration yields the bound , with equality if and only if almost everywhere on . Under this condition, grows linearly, giving and hence the optimal time-shifted schedule:
Because the same schedule attains the bound for every endpoint , it maximizes expected alignment at every time under these dynamics. The ELR profiles have also been observed to arise implicitly: without weight decay, growing parameter norms make the realized relative update decay as [13]. Here the same shape instead follows from maximizing directional alignment under WN. Appendix § 10.§ 3 gives the full proof and shows that its integer samples are also optimal for the discrete recurrence in Eq. (6). Appendix § 10.§ 4 derives the geometric restrictions needed to interpret these states as expected cosine alignments.
AIR: A practical schedule design
Building on the decay in Eq. (11), we propose AIR (Affine Inverse-square-Root decay) for pretraining with prescribed peak and terminal LRs. The decay schedule controls how rapidly the LR decreases at different stages of training. To preserve the relative magnitudes of LR decay, we use an affine transformation that scales them by a common positive factor. Let index the decay stage, with its horizon. For a decay time scale , define
Here, rescales the curve from 1 to 0. For a peak LR and a terminal LR , the resulting schedule is
This schedule decreases monotonically from at to at . Appendix § 10.§ 5 derives this affine construction and shows that it preserves the ratios between decay rates. We use in the main comparisons and evaluate sensitivity to this choice in § 4.3.
When an LR budget is prescribed instead of the peak, we determine from the complete discrete LR sum, including warmup, with the other settings fixed in Appendix § 10.§ 6.
Empirical Evaluation
Experimental setup
Pretraining setting. We pretrain a mixture-of-experts (MoE) model with 110 M activated parameters and 4.7 B total parameters on 128 B tokens using Muon [11] under WN. Each complete run takes 48 hours on 16 GPUs. To further test scalability, we apply the same scheduler design to a larger MoE with 790 M activated and 35.7 B total parameters, trained on 500 B tokens, with each run taking 168 hours on 32 GPUs. Additional experimental settings and reporting conventions are given in Appendix § 11.
Comparison controls. We perform a hyperparameter search on the cosine baseline and select the best peak LR and warmup ratio. For a fair comparison across all schedule families, we use linear warmup with the same duration and match the terminal LR and cumulative LR, defined as the sum of LRs over all training steps. Each schedule’s peak LR is calibrated to match this sum, including warmup, as described in Appendix § 10.§ 6. All schedules are compared at the end of training.
Figure 3a shows the cosine baseline, raw decay [30], and its affine transformation AIR, defined by Eqs. (12)–(13). The raw curve is shown as a reference with the same peak as AIR, without matching its terminal or cumulative LR. We also compare other decay shapes under the same controls and affine construction, including [34], [18], and [7, 38]. Appendix § 11.§ 2 lists the complete set of decay shapes.
Figure 3. Learning rate schedules and training loss under WN. With the same terminal and cumulative LRs, AIR achieves lower final training loss than cosine.
Pretraining loss and downstream performance
Experimental setup. Following § 4.1, all schedules share the same warmup, terminal LR, and cumulative LR, and each decay shape enters the comparison at its best shape parameter selected during evaluation, listed in Table 4. We report terminal training and evaluation loss together with performance on the downstream benchmarks listed in Appendix § 11.§ 4. We repeat the comparison between cosine and AIR at a larger model scale.
| Shape | , with | Selected |
| Cosine | – | |
| Linear | – | |
| Half-cosine | – | |
| Rear-cosine | – | |
| Cycloid | ||
| Double-cosine | , defined below | – |
| Power of | ||
| Power of | ||
| Logarithmic | ||
| Exponential | ||
| Inverse square | ||
| Inverse time | ||
| Variable power | ||
| Inverse square root (AIR) |
Table 4. Raw decay shapes before the common affine transformation, and the shape parameter selected for the smaller-scale rows of Table 1. The time scale , decay exponent , which is fixed for the two power families and rises to for variable power, the terminal angle , and the exponential decay parameter control the respective curve shapes. A dash marks a shape with no free parameter.
Results. Figure 3b and Table 1 show that AIR lowers terminal training and evaluation loss by about 0.01 relative to cosine at both model scales and achieves the lowest reported losses among the compared schedulers. It also attains the best downstream average among all compared shapes, and at the larger scale it improves over cosine on most benchmarks, highlighting the scale-up ability of our scheduler design.
| Schedule | Train Loss (↓) | Eval Loss (↓) | MMLU (↑) | MMLU-Pro (↑) | AGIEval (↑) | WinoGrande (↑) | BBH (↑) | DROP (↑) | GSM8k (↑) | MATH (↑) | HumanEval (↑) | MBPP+ (↑) | TriviaQA (↑) | Avg. (↑) |
| 110 M/4.7 B model | ||||||||||||||
| Cosine | 1.740 | 2.185 | 49.0 | 18.2 | 29.1 | 49.5 | 30.9 | 22.0 | 17.0 | 11.1 | 28.7 | 35.2 | 30.9 | 29.24 |
| Linear | 1.742 | 2.187 | 48.9 | 16.4 | 26.3 | 49.6 | 31.1 | 22.6 | 13.4 | 11.3 | 26.2 | 35.4 | 31.2 | 28.40 |
| Half-cosine | 1.749 | 2.195 | 48.6 | 15.5 | 28.2 | 50.7 | 32.5 | 23.9 | 15.0 | 11.7 | 24.4 | 34.7 | 29.5 | 28.61 |
| Rear-cosine | 1.736 | 2.180 | 48.0 | 16.2 | 28.7 | 51.2 | 31.6 | 22.3 | 14.3 | 10.5 | 25.0 | 36.2 | 30.7 | 28.61 |
| Cycloid | 1.734 | 2.178 | 47.3 | 16.0 | 27.5 | 49.9 | 30.7 | 22.6 | 15.1 | 10.0 | 26.2 | 33.6 | 31.1 | 28.18 |
| 1.736 | 2.181 | 48.1 | 16.8 | 27.6 | 50.7 | 30.6 | 22.1 | 16.5 | 12.0 | 27.4 | 36.8 | 29.5 | 28.92 | |
| 1.732 | 2.174 | 48.5 | 18.1 | 30.2 | 50.9 | 31.8 | 22.3 | 16.4 | 10.9 | 31.1 | 35.2 | 32.3 | 29.79 | |
| 1.731 | 2.175 | 48.1 | 15.6 | 29.6 | 48.8 | 32.0 | 21.5 | 16.8 | 11.6 | 23.8 | 34.4 | 33.2 | 28.67 | |
| 1.732 | 2.176 | 48.9 | 16.0 | 29.2 | 50.6 | 31.6 | 21.2 | 16.8 | 10.8 | 27.4 | 36.8 | 31.7 | 29.18 | |
| 1.730 | 2.175 | 50.5 | 18.4 | 29.0 | 50.8 | 32.1 | 22.2 | 17.5 | 11.6 | 29.9 | 33.1 | 30.3 | 29.58 | |
| 1.730 | 2.174 | 49.7 | 16.9 | 29.6 | 50.8 | 30.8 | 23.0 | 16.7 | 12.2 | 26.8 | 32.8 | 29.7 | 29.00 | |
| 1.733 | 2.176 | 48.8 | 16.7 | 30.1 | 50.7 | 30.0 | 22.9 | 14.2 | 10.6 | 28.0 | 34.7 | 30.6 | 28.85 | |
| (AIR) | 1.730 | 2.174 | 49.7 | 18.6 | 29.6 | 50.1 | 31.1 | 23.8 | 15.8 | 13.0 | 29.9 | 36.5 | 31.6 | 29.97 |
| 790 M/35.7 B model | ||||||||||||||
| Cosine | 1.431 | 1.843 | 70.7 | 40.6 | 54.2 | 50.4 | 56.2 | 56.0 | 59.9 | 36.8 | 67.7 | 60.1 | 65.8 | 56.22 |
| (AIR) | 1.421 | 1.832 | 72.1 | 42.4 | 55.7 | 51.8 | 59.4 | 51.2 | 64.2 | 37.2 | 70.1 | 65.3 | 66.2 | 57.78 |
Table 1. Terminal losses and downstream performance of matched-budget decay shapes under WN. Each row is labeled by its raw decay shape before the common affine transformation and uses that shape’s best parameter selected during evaluation (Appendix C.2). AIR achieves the lowest terminal losses and attains the best downstream average at both model scales.
Robustness to hyperparameter choices
Experimental setup. Under the same matched controls, we sweep each family’s shape parameter, which also moves its peak LR. For each family we report its LR sensitivity , the average loss above the family’s best configuration per fold of peak LR, defined formally in Appendix § 11.§ 5 (Eq. (31)).
Results. Figure 4 plots terminal training loss against peak LR with family-wise polynomial fits, and Table 5 in Appendix § 11.§ 5 reports the resulting LR sensitivity. AIR has the lowest LR sensitivity over the widest range of peak LR. Its region of low loss is therefore wide enough that the gain does not depend on a narrowly tuned peak LR and thus facilitates LR transfer [21].
Figure 4. AIR stays near its best loss over the widest peak LR range. Circles are measured configurations, stars the best per family, and dashed curves family-wise convex fits.
Table 5. AIR is the least sensitive to the peak LR on training loss. Sensitivity is the mean loss above a family’s best per e-fold of peak LR (Eq. (31)).
| Schedule | Configs | LR range (e-folds) | LR sensitivity (↓) | |
| Cycloid | 3 | 0.3461 | Train: 0.0022 | Eval: 0.0017 |
| 3 | 0.4219 | Train: 0.0160 | Eval: 0.0126 | |
| 3 | 0.5630 | Train: 0.0023 | Eval: 0.0039 | |
| 3 | 0.2182 | Train: 0.0047 | Eval: 0.0054 | |
| 4 | 0.7081 | Train: 0.0113 | Eval: 0.0110 | |
| 3 | 0.5630 | Train: 0.0032 | Eval: 0.0021 | |
| 5 | 0.5874 | Train: 0.0019 | Eval: 0.0027 | |
| 2 | 0.0427 | Train: 0.0230 | Eval: 0.0234 | |
| (AIR) | 5 | 0.7506 | Train: 0.0018 | Eval: 0.0018 |
Late-Stage Scheduling Matters More
Having established the gains of AIR in § 4, we further explore where in training this advantage is created. We find that late-stage scheduling drives the final loss gap (§ 5.1) and that sharing the late schedule substantially reduces gaps created by distinct early schedules (Section 5.2). Further comparisons in Section 5.3 reveal a trade-off between sustained annealing and gradual LR changes, with AIR naturally striking this balance. Finally, we provide a brief theoretical explanation in Section 5.4 of how early and late schedules affect terminal alignment.
Late-stage scheduling drives the final loss gap
Among the candidate schedules matched to the same cumulative LR, a Cycloid schedule with happens to take exactly the same peak LR as AIR, so the two runs share their warmup and terminal LR as well and differ only in the shape of the decay in between (Figure 5(a)). However, their final training losses still differ by approximately 0.005, a substantial gap at this scale. Notice that the two intersections of their decay curves naturally divide the training horizon into early, middle, and late stages, when we swap one stage at a time between them while leaving the other two unchanged.
As shown in Figure 5(b,c), swaps of the early and middle stages leave the final loss nearly unchanged, whereas swapping the late stage closes about 90% of the gap for either schedule. For this matched pair, the gap is almost entirely attributable to the late stage. This motivates us to examine whether a shared late schedule can reduce the final loss gap created by distinct early schedules. A related effect was reported for scale-invariant networks trained with weight decay, where reaching the equilibrium of a late phase makes the outcome largely independent of earlier phases [15].
Figure 5. Late-stage scheduling dominates the final loss difference. Starting from two full schedules with matched peak, terminal, and cumulative LRs, swapping the late stage between AIR and Cycloid nearly transfers the full final loss gap to either schedule, whereas swaps of the early and middle stages have little effect. The bars use the terminal exponential moving average (EMA-0.99) training loss, as in Table 1; the intervention and reporting details are given in Appendix C.6.
Shared late scheduling reduces the final loss gap
Experimental setup. To address this question, we consider six decay schedules, Cosine, Linear, Half-cosine, Double-cosine, , and , whose original runs end with training losses 0.005–0.018 higher than AIR. For each schedule, we retain its decay shape over the first 70% of training steps and concatenate it with the same late-stage schedule of AIR. The early segment and its warmup target are rescaled to match the cumulative LR under the same comparison constraints as in § 4.1. The two runs of each schedule are denoted original and shared tail, respectively.
Results. Table 2 shows that the shared late schedule lowers terminal training and evaluation loss for all six schedules, closing 42%–83% and 45%–94% of their respective gaps to AIR. Thus, a shared late schedule substantially reduces the performance differences arising from different early schedules.
| Schedule | Training loss (relative to AIR) | Eval loss (relative to AIR) | ||||
| Original | Shared tail | Gap closed (%) ↑ | Original | Shared tail | Gap closed (%) ↑ | |
| Cosine | 1.7399 (+0.0096) | 1.7358 (+0.0055) | 42% | 2.1846 (+0.0106) | 2.1799 (+0.0059) | 45% |
| Linear | 1.7415 (+0.0112) | 1.7341 (+0.0038) | 66% | 2.1875 (+0.0135) | 2.1787 (+0.0047) | 65% |
| Half-cosine | 1.7487 (+0.0184) | 1.7374 (+0.0071) | 62% | 2.1950 (+0.0210) | 2.1815 (+0.0076) | 64% |
| Double-cosine | 1.7446 (+0.0143) | 1.7328 (+0.0025) | 83% | 2.1901 (+0.0161) | 2.1769 (+0.0029) | 82% |
| 1.7361 (+0.0058) | 1.7319 (+0.0016) | 72% | 2.1805 (+0.0065) | 2.1753 (+0.0013) | 80% | |
| 1.7354 (+0.0050) | 1.7316 (+0.0012) | 75% | 2.1787 (+0.0047) | 2.1743 (+0.0003) | 94% | |
| (AIR) | 1.7303 | – | – | 2.1740 | – | – |
Table 2. A shared late schedule narrows the endpoint gap for every early decay shape. Each schedule is trained twice, once in its original form and once with the common late schedule. Parentheses give the gap of each loss to the AIR anchor in the last row, and Gap is the fraction of that gap the shared tail removes. Training loss is the terminal EMA-0.99 value and evaluation loss the terminal raw value, as in Table 1.
A trade-off in late-stage scheduling
Experimental setup. Having established the importance of late-stage scheduling, we next examine what makes a late schedule effective. Based on the design of AIR in § 3.3, we vary the final 20% of its decay stage, starting at , as
where . Smaller exponents postpone more of the LR reduction to the endpoint, whereas larger exponents reduce the LR earlier and then flatten out. The junction and terminal LRs remain fixed, while the area under the recorded LR traces varies by less than 1% (Appendix § 11.§ 7).
Results. Figure 6 shows that both the abrupt decay at and the early flattening at increase final loss, while AIR and the variants with perform comparably, achieving the lowest final losses. These results suggest a trade-off: late-stage scheduling should sustain annealing while avoiding overly abrupt LR changes. The design of AIR naturally balances these requirements and performs close to the best variants in this sweep.
Figure 6. Late-stage decay exhibits a two-sided trade-off. With the junction and terminal LRs fixed, both overly abrupt decay and early flattening increase final loss.
A theoretical explanation of schedule dependence
To interpret these results, we use the alignment dynamics of § 3 to separate the effects of early- and late-stage schedule changes. Consider bounded nonnegative schedules on the decay interval , with common , , , and a fixed stage boundary . Write and , where the expected alignments retain the conditioning in Eq. (5) throughout. For a shared late schedule, on , Eq. (9) gives
Since , the shared late stage attenuates the entering difference without reversing its sign, ignoring stochastic fluctuations around the mean. Over a fixed late interval, higher LRs increase the cumulative squared LR and reduce , thereby weakening the effect of early-stage scheduling on endpoint alignment. The derivation is given in Appendix § 12.§ 2.
For different late-stage schedules, a shared early schedule gives . For bounds and over the late-stage interval, let and , with the retention factor for . This gives
Smaller weighted LR differences give a tighter bound on the terminal alignment difference. Within this bound, differences nearer the endpoint receive greater weight because less subsequent attenuation remains. Details are given in Appendix § 12.§ 3. Further, if the schedules differ in both stages,
The first term retains the early-stage difference, while the second bounds the late-stage contribution. Slope-based refinements are in Appendix § 12.§ 4. These bounds on expected alignment provide a qualitative explanation for the observed loss gaps.
Related Work
WN and related constraints provide explicit parameter-scale control as an alternative to WD, with reported efficiency and stability gains [1, 33]. Methods differ in whether they separate magnitude from direction [23] or fix weight and update norms [33]. The choice of matrix norm further shapes the weight constraints and update geometry [35, 36]. For manifold constraints, Su [26, 27] and Bernste[in] [2] optimize tangent-space descent directions before retracting weights, so that the update respects the imposed geometry.
Beyond norm control, scheduling determines how the effective step size evolves during training. River-valley analyses explain WSD [32], while convex bounds and functional scaling laws motivate cooldown and power decay [14, 24]. However, these theories focus on nominal LR alone, which is not principled when parameter norms evolve [5]. Concurrent work corroborates our finding that matching ELR largely aligns WN and WD loss dynamics [17, 34, 38], but does not derive a decay law from an optimization objective. We address this gap by optimizing directional alignment under WN to derive decay, then adapting it to finite horizons through an affine transformation, yielding AIR as an effective ELR schedule.
Conclusion and Future Work
We identify the induced effective learning rate (ELR) trajectory as the key driver of WN’s advantage over WD, with matched ELRs largely aligning their training loss dynamics. This finding motivates schedule design that targets ELR directly. Our theoretical analysis of directional alignment under WN, together with an affine adaptation of the unconstrained optimal schedule, yields AIR for finite-horizon training. Experiments on MoE models at two scales show that, at matched cumulative LR, AIR reduces terminal loss by about 0.01 relative to cosine and improves downstream performance, while schedule sweeps demonstrate hyperparameter robustness. Further stage-wise interventions show that sharing the late schedule substantially narrows final loss gaps from different early schedules, identifying late-stage scheduling as a key source of AIR’s effectiveness. Transferring its ELR trajectory to WD reproduces WN training loss dynamics, demonstrating that our design extends beyond fixed-norm training.
Future work includes extending our directional-alignment theory to account for evolving update statistics and quantitatively connect alignment to loss. We also aim to investigate the ELR principle and AIR across more architectures, optimizers, and larger training scales, and to study ELR scheduling in continued training (CT), supervised fine-tuning (SFT), and reinforcement learning (RL).
Appendix
Directional Geometry and ELR Matching
We derive the geometric interpretation of ELR in § 2.1 and the blockwise matching rule used in § 2.2.
Exact normalized updates
For a nonzero parameter block, retain and from § 2.1, and write for its unit direction. Throughout, , and norms are Euclidean on vectorized blocks. Under fixed-norm WN, . Dividing the updates in Eqs. (1) and (2) by their pre-update parameter norms and normalizing gives
These identities hold whenever the vector being normalized is nonzero. They keep the realized optimizer update explicit. The ELR multiplies that update relative to the current parameter norm, while WD also contributes radial shrinkage.
Radial and tangential components
Define
The scalar is the radial component, and is the component orthogonal to the parameter direction. Under WD, the updated vector is
Orthogonality gives its squared norm as the sum of the squared radial and tangential terms. The unsigned angle between the old and updated directions is therefore
For WN, the same expression applies with and , because the final normalization preserves direction.
To obtain the first-order change, the derivative of at a unit vector is . Applied to the increment , it removes the radial term and gives
The expansion requires . The WN expression again sets . For a given tangential optimizer update, the ELR thus sets the first-order scale of the directional change.
Blockwise ELR transfer
For a target trajectory , choosing gives by substitution. A WN schedule supplies the target , giving
This is the reverse direction of the coefficient matching in Eq. (4). We apply this rule separately to each controlled block, using the recorded WD norm replayed at step in the reverse-matching experiments.
WN Schedule Design and Calibration
We first derive the alignment dynamics and their unconstrained optimum from § 3. We then establish the affine endpoint construction and discrete LR sum calibration used for finite-horizon schedules.
From the WN update to alignment dynamics
Under the assumptions in § 3.1, , , and . Hence
Taking the inner product of Eq. (2) with gives
We condition on a fixed initial history and a prescribed LR schedule, so and the denominator are deterministic under this conditioning. We average over the first updates that generate and the randomness in . No independence between and is needed. For an LR selected adaptively from later random observations, this factorization would in general fail. The definitions of and then give Eq. (6). For , the binomial expansion gives
Multiplying by and subtracting yields
Approximating this increment by a derivative with one optimizer step as the time unit gives
This establishes the second-order discrete increment underlying Eq. (8). Replacing that increment by a derivative is a further continuous approximation, requiring small relative updates, slow variation of the LR and alignment over individual optimizer steps, and approximately constant update statistics.
Derivation of the integral solution
To solve Eq. (8) on with , take a nonnegative . Both and are integrable on the finite interval, so the initial-value problem has a unique absolutely continuous solution. Multiplying the ODE by gives
Integration from to , followed by division by the terminal integrating factor, multiplies each contribution introduced at time by
This is the fraction retained from to , as introduced in Eq. (9). Combining the initial and accumulated contributions gives
This proves Eq. (9). Since , , and the integral term is nonnegative, the solution is strictly positive at every time in the interval.
Proof of the unconstrained optimal schedule
Proof of the unconstrained optimal schedule
For any nonnegative , with fixed , , , , the positive solution satisfies
almost everywhere. Integrating establishes . Equality requires almost everywhere. Under this condition, , so
The resulting LR is nonnegative, continuous, and square-integrable on any finite interval. With , it is exactly Eq. (11). Substitution verifies that it solves the ODE with the prescribed initial state and attains the bound. For every candidate, the precise terminal gap is
The nonnegative right-hand side proves global optimality. A zero gap uniquely determines at every time and almost everywhere. Since the endpoint was arbitrary and the optimal curve does not depend on it, this same curve attains the alignment bound at every time. This completes the proof of the result in § 3.2. The squared states here are squares of expected alignments, not expectations of squared alignments.
Exact discrete counterpart. For the recurrence in Eq. (6), direct algebra gives, for ,
After updates, this implies . Choosing makes every single-step gap zero and yields
Thus the integer samples of the continuous optimum exactly maximize terminal alignment in this discrete recurrence. The updates use and produce .
Geometric range of the constant-coefficient assumptions
The scalar recurrence and its continuous approximation have a broader algebraic domain than an expected cosine alignment. To identify the necessary geometric restriction, tangency and the Cauchy–Schwarz inequality give, pathwise,
The square-root function here is concave as a function of the cosine on . Taking the conditional expectation and using Jensen’s inequality therefore yields
Thus is necessary at every state from which an update with the assumed positive drift is taken. For the exact discrete optimum over updates, applying it at requires
A constant positive drift cannot therefore describe an indefinitely growing physical alignment. The optimization proof is global over the prescribed nonnegative schedules of the scalar dynamics, while its interpretation as WN alignment is restricted to intervals on which the geometric assumptions remain realizable. Likewise, the continuous optimum need not satisfy the small-step condition automatically: its largest relative squared step is . This ratio must be small when the ODE optimum is used through the continuous approximation, although the discrete optimality identity itself does not require it.
Affine endpoint construction
Let be the integer decay horizon from § 3.3. For , the inverse-square-root curve in Eq. (12) satisfies . Write an affine transformation of this curve as , where sets its scale and its offset. An initial decay LR and final LR require
Since , these equations uniquely determine
Substitution gives the decay sequence
For , we have , so the schedule decreases monotonically between its prescribed endpoints. For any $0 \le i < j \le K$, subtracting the two LR values gives
Since , it follows that
Thus, each interval accounts for the same fraction of the total LR reduction as in the original curve. In particular, all per-step decreases are scaled by the same positive factor. For the continuous interpolation, , so all decay rates are scaled by the same positive factor and their relative magnitudes are preserved. These properties establish the endpoint calibration and preservation of the relative magnitudes of decay rates. The unconstrained optimum in § 3.2 thereby inspires the practical affine schedule.
Matching the discrete LR sum
Fix the warmup rule, decay horizon , time scale , terminal LR , and the indices at which LR values are executed. Write the warmup values as , where the coefficients are fixed independently of the peak . This includes linear warmup and fixed multiplicative adjustments to it. Let be the executed decay indices, and let . For a target sum over all executed LR values, define
The first two quantities collect the peak-dependent and constant warmup contributions, and the third collects the peak coefficient from decay. Summing the warmup and Eq. (13) gives
If , the peak has the unique solution
The calibrated peak must still exceed and satisfy the prescribed warmup constraints. Under the positive-coefficient condition above, a prescribed peak fixes the sum, whereas a prescribed feasible sum determines the peak. The endpoint contributes exactly when it belongs to . Thus the sum must use the executed LR values rather than infer endpoint inclusion from a logging convention.
For example, consider linear warmup values at , followed by all decay values . For this convention,
There are values, the peak occurs at the first decay step, and both decay endpoints are included. The warmup and decay regime requires and . For any other execution convention, the same general formula applies with its actual coefficients and index set.
Experimental Settings and Evaluation Protocol
This appendix details the experimental setup in § 4 and § 5.
Training settings and comparison controls
Table 3 summarizes the training settings for the smaller-scale scheduler experiments. The larger-scale comparison uses 790 M activated and 35.7 B total parameters, 500 B training tokens, and 168 hours per run on 32 GPUs. Within each fixed-norm scheduler comparison, the training setting is held fixed and the LR schedule is varied.
| Configuration | Value |
| Activated parameters | 110 M |
| Total parameters | 4.7 B |
| Training tokens | 128 B |
| Optimizer | Muon |
| Global batch size | 128 |
| Normalization | Frobenius norm |
| Normalization epsilon | |
| Scheduled optimizer updates | 32,047 |
| Warmup updates | 3,040 |
| Minimum LR | |
| Cosine peak LR | |
| AIR peak LR | |
| AIR decay time scale | |
| GPUs per run | 16 |
| Wall-clock per complete run | 48 hours |
Table 3. Training settings for the smaller-scale scheduler comparisons. The peak LRs specify the main comparison between cosine and AIR; other schedule peaks are calibrated to the same cumulative LR.
The LR curves in Figure 3(a) show a linear warmup illustration; the comparison controls concern the complete training schedule, including its batch-size ramp. The observed terminal training loss step for these runs is 32,046. Logging indices and scheduled endpoints are therefore distinguished when pairing losses or evaluating a schedule sum.
Within a fixed-norm comparison, the model, token budget, optimizer settings, batch-size schedule, constrained parameter blocks, and their radii are held fixed. The complete optimizer-step LR sequence is the object being compared. Warmup and the last executed update are included in its LR sum. For matched-sum comparisons with cosine, peak calibration follows Appendix § 10.§ 6. This leaves peak LR free to differ between schedules while holding the specified sum, horizon, warmup, and terminal LR fixed.
Stage replacements require a separate accounting of the resulting sequence. Replacing only one interval can alter its total LR sum or create a jump at the boundary. These properties follow from the actual combined curve and cannot be inferred from the controls of the original full schedules. A shared late schedule means equal LR values at every shared optimizer step, not merely equal endpoints or equal integrated LR. Likewise, varying an entire decay exponent is a different intervention from fixing the early schedule and replacing only the late part.
Compared decay shapes
Let index the decay stage as in Eq. (12), and write . Thus is the fraction of the decay stage completed. Table 4 gives the raw shape for each schedule. All shapes satisfy and use the same affine endpoint transformation and LR interpolation in Eqs. (12)–(13), with their own substituted for the inverse-square-root curve. The peak is then calibrated to match the cosine baseline’s cumulative LR, including warmup, following Appendix § 10.§ 6. This holds the warmup duration and terminal LR fixed while allowing the peak to differ. The raw inverse-square-root reference in Figure 3(a) is excluded from these controls.
The square-root shape used outside Table 1 is with , and the quadratic shape is with . In Table 1 each schedule is named by the shape in this table, written up to the affine transformation: the time variable absorbs an affine rescaling, so for the polynomial shapes and for the inverse-power and logarithmic ones, and for the exponential shape, and the endpoint normalization is left implicit. Cosine, Half-cosine, Rear-cosine, and Cycloid are not separated by such a formula and keep their names there.
For the symmetric double-cosine schedule, the two cosine segments join at and :
For Cycloid, is the unique solution of the relation in the table. For variable power, the exponent varies linearly from to .
Loss summaries
Tables 1 and 2, and the robustness comparison, use the same terminal loss convention. For training loss, take the last 100 finite logged values in step order, denoted , initialize , and recursively compute
The reported value is ; evaluation loss is the last finite logged evaluation value without smoothing. The last 100 observations need not span exactly 100 optimizer steps if a log has gaps. The smaller-scale comparison ends with training loss at step 32,046 and evaluation loss at step 32,000; the corresponding larger-scale steps are 62,602 and 62,000. These are terminal summaries of individual trajectories, not averages across independent training seeds.
Training restarts are stitched by optimizer step, with the later-started segment taking precedence at duplicate steps. For reference loss curves, we first pair raw losses at common optimizer steps and subtract the specified reference, then apply a debiased EMA with decay 0.99 to the complete paired sequence. For paired differences , the plotted value is
Smoothing advances once per retained paired observation; missing steps are not filled by interpolation. The terminal reference loss points in Figure 6 use the final value of this curve. Figure 5(c) does not use this curve: for each of the eight runs it reports the terminal training loss summary of Eq. (30), and its stage-swap gaps are differences of these terminal values. No independent-seed confidence interval is inferred from adjacent training steps.
Downstream benchmarks
Table 1 reports performance on MMLU [8], MMLU-Pro [31], AGIEval [37], WinoGrande [22], BBH [28], DROP [6], GSM8k [4], MATH [9], HumanEval [3], MBPP+ [16], and TriviaQA [12].
Hyperparameter sweeps and LR sensitivity
For the robustness sweep in § 4.3, we report each family’s LR sensitivity. For a family with configurations of terminal loss and peak LR , the LR sensitivity is
the average loss above the family’s best configuration per e-fold of peak LR: the numerator is the mean terminal loss above the family’s best measured configuration, and dividing by the log-span of the swept peak LR normalizes for search width. This is a descriptive statistic over each family’s measured range and is not extrapolated beyond it. Table 5 reports on training and evaluation loss.
Table 5 gives the configuration counts and logarithmic spans. Search ranges and configuration counts differ across families; the normalization by does not make their tuning budgets identical. The fitted curves visualize the measured points, whereas is computed directly from those points. For a family with configurations, the fit has degree ; fits with at least three points are constrained to be convex over the observed peak-LR interval.
Stage swaps
Figure 5 compares AIR with Cycloid at , which shares its peak LR, warmup, terminal LR, and full-schedule cumulative LR. This comparator differs from the evaluation-selected Cycloid at in
Table 6. Measured shape settings and peak-LR ranges for the smaller-scale sweep. Decimal shape settings are rounded to three places; the other values are exact. Counts and logarithmic spans appear in Table 5.
| Family | Shape settings | Peak LR () |
| Cycloid | 1.377–1.946 | |
| 1.528–2.330 | ||
| 1.528–2.683 | ||
| Logarithmic | 1.541–1.916 | |
| Exponential | 1.489–3.023 | |
| Inverse square | 1.528–2.683 | |
| Inverse time | 1.382–2.487 | |
| Variable power | , | 1.946–2.031 |
| AIR | 1.382–2.928 |
Table 1. The two intersections occur at approximately global steps 4,346 and 16,219 and define the early, middle, and late intervals. Each replacement follows the donor schedule only within the named interval and the recipient schedule outside it. Matching the cumulative LR of the two original schedules does not imply that an individual stage replacement preserves that sum. The terminal summary and reference loss smoothing are specified in Appendix § 11.§ 3.
Late decay exponents
The exponent interventions in Eq. (14) switch at global step 25,955, corresponding to of the decay stage after warmup. They use , with the same earlier schedule, junction LR, and configured terminal LR. Their cumulative LR is not exactly matched. Trapezoidal integration of the recorded LR traces gives deviations below 1 % relative to the AIR reference. Figure 6 reports same-step loss differences to the full-run AIR reference with the full-sequence EMA convention in Appendix § 11.§ 3.
Bounds for Early- and Late-Schedule Differences
Common setting
We use the continuous relaxation of the decay-step coordinate , with a fixed split . For , the expected alignment follows the dynamics from § 3,
with common and bounded, measurable, nonnegative schedules . Dots denote derivatives with respect to this continuous step coordinate. The expectations retain the same initial conditioning as Eq. (5); the conditioning is not reset at . Write and . A shared early schedule and the common initial alignment give by uniqueness of the linear ODE.
For each schedule, the retention factor from Eq. (9) is
The second factor retains a difference under the average retraction coefficient . When the late schedules coincide, on that interval.
Varying early-stage schedules
If on , subtracting the state equations gives
Multiplication by makes the derivative of the product zero. Evaluating this constant at and yields
The argument never divides by , so it also covers zero or negative entering differences. Since , the sign is preserved and the magnitude cannot increase, proving Eq. (15). Ignoring the effects of stochastic noise, the expected-alignment ordering is therefore preserved along the shared tail.
Early schedules with equal integrated LR. Assume additionally on and . Integrating the two state equations cancels the common initial alignment and the cumulative new-alignment terms:
Each retraction integral is nonnegative and at most . Their absolute difference is therefore bounded by the larger integral, giving
Equal integrated LR cancels the cumulative new-alignment difference but does not equate the retraction corrections.
Discrete counterpart. Take to be integers and let for the recurrences in Eq. (6). If they share for , their difference obeys . Iteration gives
This exact discrete relation has the same sign-preserving attenuation property without replacing the product by a continuous exponential.
Varying late-stage schedules
On , the algebraic identity
gives the exact difference equation
Using the integrating factor associated with the first term yields
The first term propagates the entering difference, and the second accumulates differences caused by the later LR choices. This identity is exact within the ODE and does not assume a small schedule perturbation.
Choose common bounds and on the late interval. Here bounds both schedules; it does not require them to have equal peaks. The integral solution preserves nonnegative states, and , so is a valid finite choice. When the alignments stay in their physical range, also suffices. Since
the response coefficient in brackets lies between and . Its absolute value is bounded by
This preserves the cancellation between alignment gain and retraction before taking absolute values. When only the late-stage schedule varies, the shared early schedule gives . Applying the triangle inequality to Eq. (36) therefore gives
which proves Eq. (16).
If both stages vary, retain the first term in Eq. (36) and apply the same bound to its integral. Since , this gives
proving Eq. (17) without requiring a common entering alignment. Together, these comparisons characterize how LR schedules promote expected directional alignment under common coefficients, thereby providing a mechanism for improved training loss.
Slope bounds and endpoint information
Assume , absolutely continuous late schedules, and a common terminal LR . Let . Subtracting the terminal values and integrating the derivative difference gives
Substitution into Eq. (16), followed by , yields
If the schedules also share their LR at , integration from the left endpoint gives . The integral of this envelope is , hence
A shared early schedule alone does not impose this extra endpoint condition if a jump is allowed at the switch.
Applicability to the late-stage ablation. The schedules in Eq. (14) are bounded and absolutely continuous for every , so the LR-difference bound in Eq. (16) applies under the stated alignment dynamics. For , however, the continuous terminal derivative is unbounded. Indeed, writing the positive junction-to-floor difference directly, differentiation gives
Its magnitude diverges at when , is constant when , and vanishes there when . A comparison with the smooth AIR tail, or with a distinct exponent, therefore need not satisfy the finite condition. Accordingly, comparisons involving are governed by the general LR-difference bound rather than its slope-based corollary. This bound connects the exponent intervention to expected directional alignment, complementing the empirical loss comparison in § 5.3.
Knowing only the terminal slope. For twice continuously differentiable schedules with a common terminal LR and , define and . These measure the terminal slope difference and the curvature difference over the full interval. Taylor expansion with integral remainder gives
Substitution into the LR-difference bound yields
Equal terminal slopes remove the first contribution, while the curvature term captures differences across the interior of the schedules.
Affine power curves and the scope of the bounds
As an auxiliary example on , fix a common initial LR and a time scale . For exponent , define
This affine inverse-power family shares its initial and terminal LRs; gives an inverse-square-root shape. It is distinct from the terminal-power variants indexed by in Eq. (14). Direct differentiation gives
The terminal slope follows by setting . On a compact interval of positive exponents, the slope is jointly continuous in and , so close exponents give uniformly close slopes. Combined with the preceding bounds, this continuity shows that nearby exponents produce nearby terminal expected alignments from a common entering state. When the early stage also varies, Eq. (36) incorporates the resulting entering-state difference.
Infinitesimal sensitivity to the time of an LR change
For an admissible family with bounded perturbation and a common initial state, let . It is the first-order change in expected alignment induced by the LR perturbation. Differentiating the state equation gives
This differentiation is justified directly from the integral solution: bounded and on the finite interval uniformly bound both the integrand and its derivative for sufficiently small admissible . Differentiation under the integral sign therefore gives an absolutely continuous satisfying this linear equation. Applying the same integrating factor gives
Later perturbations are retained more strongly through , while the factor determines their local alignment gain. The resulting sensitivity kernel identifies when and where an LR change most strongly influences terminal alignment.
References
- [1]Kang An, Jiaxiang Li, Donald Goldfarb, and Shiqian Ma. Demystifying Manifold Constraints in LLM Pre-training, 2026. URL https://arxiv.org/abs/2605.04418.
- [2]Jeremy Bernstein. Modular manifolds. *Thinking Machines Lab: Connectionism*, 2025. doi: 10.64434/tml.20250926. https://thinkingmachines.ai/blog/modular-manifolds/.DOI
- [3]Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating Large Language Models Trained on Code, 2021. URL https://arxiv.org/abs/2107.03374.
- [4]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training Verifiers to Solve Math Word Problems, 2021. URL https://arxiv.org/abs/2110.14168.
- [5]Francesco D’Angelo, Maksym Andriushchenko, Aditya Varre, and Nicolas Flammarion. Why Do We Need Weight Decay in Modern Deep Learning? In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, *Advances in Neural Information Processing Systems*, volume 37, pages 23191–23223. Curran Associates, Inc., 2024. doi: 10.52202/079017-0730. URL https://proceedings.neurips.cc/paper_files/paper/2024/file/29496c9249d6e08cec469f451ebff0-Paper-Conference.pdf.
- [6]Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)*, pages 2368–2378, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1246. URL https://aclanthology.org/N19-1246/.DOI
- [7]Alexander Hägele, Elie Bakouch, Atli Kosson, Loubna Ben Allal, Leandro Von Werra, and Martin Jaggi. Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, *Advances in Neural Information Processing Systems*, volume 37, pages 76232–76264. Curran Associates, Inc., 2024. doi: 10.52202/079017-7427. URL https://proceedings.neurips.cc/paper_files/paper/2024/file/8b970e15a89bf5d12542810df8eae8fc-Paper-Conference.pdf.arxiv.org/abs/2405.18392
- [8]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring Massive Multitask Language Understanding. In *9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021*. OpenReview.net, 2021. URL https://openreview.net/forum?id=d7KBjmI3mGm.
- [9]Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring Mathematical Problem Solving With the MATH Dataset. In J. Vanschoren and S. Yeung, editors, *Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks*, volume 1, 2021. URL https://datasets-benchmarks-proceedings.neurips.cc/paper_files/paper/2021/file/be833ac3d0db773eb2dcb10a17836a1-Paper-round2.pdf.
- [10]Shengding Hu, Yuge Tu, Xu Han, Ganqu Cui, Chaoqun He, Weilin Zhao, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Xinrong Zhang, Zhen Leng Thai, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, dahai li, Zhiyuan Liu, and Maosong Sun. MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies. In *First Conference on Language Modeling*, 2024. URL https://openreview.net/forum?id=3X2L2TrFrOf.arxiv.org/abs/2404.06395
- [11]Keller Jordan, Yuchen Jin, Vlado Boza, You Jiacheng, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https://kellerjordan.github.io/posts/muon/.
- [12]Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. In Regina Barzilay and Min-Yen Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v1/P17-1147. URL https://aclanthology.org/P17-1147/.
- [13]Atli Kosson, Jeremy Welborn, Yang Liu, Martin Jaggi, and Xi Chen. Weight Decay may matter more than μP for Learning Rate Transfer in Practice. In C. VonDrick, B. Hariharan, C. Raffel, L. Pinto, D. Yang, and A. Faust, editors, International Conference on Learning Representations, volume 2026, pages 67012–67048, 2026. URL https://proceedings.iclr.cc/paper_files/paper/2026/file/6dc63b4063c978cf195bc151782e8152a-Paper-Conference.pdf.arxiv.org/abs/2510.19093
- [14]Binghui Li, Zilin Wang, Fengling Chen, Shiyang Zhao, Ruiheng Zheng, and Lei Wu. Optimal Learning Rate Schedules under Functional Scaling Laws: Power Decay and Warmup–Stable–Decay (Extended Abstract). In Steve Hanneke and Tor Lattimore, editors, Proceedings of Thirty Ninth Conference on Learning Theory, volume 336 of Proceedings of Machine Learning Research, pages 4722–4723. PMLR, 29 Jun–03 Jul 2026. URL https://proceedings.mlr.press/v336/li26b.html.arxiv.org/abs/2602.06797
- [15]Zhiyuan Li, Kaifeng Lyu, and Sanjeev Arora. Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Rate. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 14544–14555. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/a7453a5f026fbf6831d68bdc9cb0edca-Paper.pdf.arxiv.org/abs/2010.02916
- [16]Jiawei Liu, Chunqiu Steven Xia, Yutao Wang, and LINGMING ZHANG. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 21558–21572. Curran Associates, Inc., 2023. doi: 10.5220/075280-0943. URL https://proceedings.neurips.cc/paper_files/paper/2023/file/43e9d647ccd3e47b55baab53f0368686-Paper-Conference.pdf.arxiv.org/abs/2305.01210
- [17]Zihan Liu, Ruiheng Zheng, Shaobo Zhang, Changxin Tian, Kunlong Chen, Zhiqiang Zhang, and Lei Wu. Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining, 2026. URL https://arxiv.org/abs/2608.24814.
- [18]Ilya Loshchilov and Boris Ginsburg. Training nGPT, 2026. URL https://arxiv.org/abs/2608.01284.
- [19]Ilya Loshchilov and Frank Hutter. SGDR: Stochastic Gradient Descent with Warm Restarts. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24–26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https://openreview.net/forum?id=Skq89Scxx.arxiv.org/abs/1608.03983
- [20]Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.
- [21]Liliang Ren, Yang Liu, Yelong Shen, and Weizhu Chen. Rethinking Language Model Scaling under Transferable Hypersphere Optimization, 2026. URL https://arxiv.org/abs/2603.28743.
- [22]Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An Adversarial Winograd Schema Challenge at Scale. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8732–8740, April 2020. ISSN 2159-5399. doi: 10.1609/aaai.v34i05.6399. URL http://dx.doi.org/10.1609/aaai.v34i05.6399.
- [23]Tim Salimans and D.P. Kingma. Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL https://proceedings.neurips.cc/paper_files/paper/2016/file/ed265bc903a5a097f161d3ec064d96d2e-Paper.pdf.
- [24]Fabian Schaipp, Alexander Hägele, Adrien Taylor, Umut Simsekli, and Francis Bach. The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Training. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors, Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 53267–53294. PMLR, 13–19 Jul 2025. URL https://proceedings.mlr.press/v267/schaipp25a.html.arxiv.org/abs/2501.18965
- [25]Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. In International Conference on Learning Representations, 2017. URL https://mlanthology.org/iclr/2017/shazeer2017iclr-outrageously/.
- [26]Jianlin Su. Steepest Descent on Manifolds: 1. SGD + Hypersphere, Aug 2025. URL https://kexue.fm/archives/11196.
- [27]Jianlin Su. Steepest Descent on Manifolds: 3. Muon + Stiefel, Aug 2025. URL https://kexue.fm/archives/11221.
- [28]Mirac Suzgun, Nathan Scales, Nathanael Schä rli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, pages 13003–13051, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-acl.824. URL https://aclanthology.org/2023.findings-acl.824/.DOI
- [29]Twan van Laarhoven. L2 Regularization versus Batch and Weight Normalization, 2017. URL https://arxiv.org/abs/1706.05350.
- [30]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.arxiv.org/abs/1706.03762
- [31]Yubo Wang, Xueguang Ma, Ge Zhang, Yusheng Ni, A Bhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Alrujaj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 95266–95290. Curran Associates, Inc., 2024. doi: 10.52202/079017-3018. URL https://proceedings.neurips.cc/paper_files/paper/2024/file/ad236edc564f3e3156e1b2feafb99a24-Paper-Datasets_and_Benchmarks_Track.pdf.DOI
- [32]Kaiyue Wen, Zhiyuan Li, Jason Wang, David Hall, Percy Liang, and Tengyu Ma. Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape View. In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu, editors, International Conference on Learning Representations, volume 2025, pages 42840–42885, 2025. URL https://proceedings.iclr.cc/paper_files/paper/2025/file/6a1fe80a9e2dcda0b3e5fd0fd87eb097-Paper-Conference.pdf.arxiv.org/abs/2410.05192
- [33]Kaiyue Wen, Xingyu Dang, Kaifeng Lyu, Tengyu Ma, and Percy Liang. Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization, 2026. URL https://arxiv.org/abs/2606.16899.
- [34]Yihao Xiao, Jialong Sun, Zitian Gao, Zeming Wei, Chutian Wang, Ran Tao, Jiaye Teng, and Bryan Dai. Hyperball May Not Be a Free Lunch, 2026. URL https://arxiv.org/abs/2607.22444.
- [35]Tian Xie, Haoming Luo, Haoyu Tang, Yiwen Hu, Jason Klein Liu, Qingnan Ren, Yang Wang, Wayne Xin Zhao, Rui Yan, Bing Su, Chong Luo, and Baining Guo. Controlled LLM Training on Spectral Sphere, 2026. URL https://arxiv.org/abs/2601.08393.
- [36]Ruihan Xu, Jiajin Li, and Yiping Lu. On the Width Scaling of Neural Optimizers Under Matrix Operator Norms I: Row/Column Normalization and Hyperparameter Transfer, 2026. URL https://arxiv.org/abs/2603.09952.
- [37]Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Findings of the Association for Computational Linguistics: NAACL 2024, pages 2299–2314, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-naacl.149. URL https://aclanthology.org/2024.findings-naacl.149/.DOI
- [38]Zhanpeng Zhou, Wenjie Zhou, Yufei Gu, Juqiu Wang, Guanwei Zhang, Ruobing Xie, Xianbiao Qi, Yushi Bai, Kaifeng Lyu, and Shunyu Yao. From LR to ELR: A Better Heuristic for Pretraining Dynamics, 2026. URL https://hy.tencent.ai/research/elr.



