WorksComputer Science
Reactive Optimizers
Abstract
We study a reactive optimizer that uses a pretrained language-model agent to inspect training dynamics and rewrite executable optimizer code during a single training run. The agent can change update rules and schedules for individual parameter groups, without optimizer-specific meta-training or memory transferred from other runs. We evaluate this approach, OPAL, on language modeling and image generation when a recipe has not been tuned for the target setting. Across the evaluated settings, it matches or improves on selected fixed recipes. Analysis of its decisions shows changes to learning-rate schedules, optimizer structure, numerical precision, and momentum. Controlled experiments also show adaptation to repeated data corruption and reduced overfitting. We report these behaviors together with the agent's computational cost and implementation requirements.
This document is a static version of our blog post. We recommend the original interactive version at https://xingjianbai.com/reactive-optimizers/. We release the code at https://github.com/xingjian-bai/opal.
Introduction
Today, the recipe used to optimize a deep neural network is typically completely determined before training even begins. A great deal of human ingenuity has gone into priors regarding how training should progress: adaptive step sizes [19], preconditioning [12, 38], orthogonalized updates [17, 23], schedules tied to the training step [24, 14], and learning rates scaled with model size (Yang et al., 2022). Automated discovery [25, 30, 6, 28] searches across many runs for better optimizers. Whether hand-crafted or automatically discovered, the optimizer then runs unchanged through the end of training, adapting, if at all, according to rules written in advance. This scope leaves the following questions unanswered: How should an optimizer respond to an unexpected loss spike or stall? What if one module’s gradients start to oscillate while the others keep improving smoothly? And when a new type of architecture trains poorly with Adam or Muon, is the architecture worse, or has it not yet met an optimizer suited to it?
In this work, we attempt to expand the scope of conventional optimizers and make them reactive to the training itself, leading to an ever-adapting optimizer inside a single training run. This reactive optimizer reasons on the spot and writes the weight update rule for each parameter group. Directly writing optimization code, it has the freedom to implement complicated schedules, respond to training accidents, and customize the optimizer for the single training episode in which it exists. Building on earlier attempts to adapt optimizers during training [28, 9, 16] and to let a model direct its own fine-tuning (Zweiger et al., 2025), the reactive optimizer is entirely zero-shot, with nothing learned or retained from repeated runs, and puts the optimizer’s entire algorithm up for rewriting.
We start with a practical situation, with tasks from image generation and language modeling, in which a reactive optimizer can do much better than static ones. We then turn to the emergent properties of the reactive optimizer, including sophisticated scheduling, suppressing loss spikes, and escaping overfitting. Lastly, we ask what all this costs and how practical it is at scale. We expect the reactive optimizer to grow only stronger with each generation of agents, to be most useful exactly where no recipe exists yet (for a new architecture or a new dataset), and to be one step toward solving the puzzles of deep learning optimization.
Training without a recipe
We first consider a practical setting in which the training budget permits one substantial run but not a broad search over optimization recipes. For example, suppose a PhD student has designed a new model or selected a new dataset for a transformer, with one GPU for a day and a modest budget for LLM API calls. A conventional approach is to adopt a standard recipe or one from a related task; if that recipe is unsuitable, training may exhibit loss spikes or stalled progress.
The underlying question is broader: when bounded by GPU time, researcher time, or wall-clock time, how can a deep learning model be trained to its best performance without knowing the optimal optimization recipe in advance? New architectures, modalities, languages, and scientific data all fall into this setting, where few people can afford a sweep at scale.
A reactive optimizer offers another way to use that budget: it spends LLM tokens in place of GPU/human hours. With optimizers reacting to training incidents and adjusting themselves, much of the burden of recipe tuning can be reduced.
Results in brief
We start from the optimization track (Track 3) of modded-nanogpt (Jordan, [18]), a language-model pre-training task (training a 162M GPT on FineWeb) with many recipes tuned specifically for this one setting. From this task, we create two slightly differing natural settings: (1) the same model trained on an alternate language dataset (we select Chinese, Thai, and Kyrgyz), and (2) the original setup with a quarter of the batch size. Neither comes with a recipe tuned for that setting, so we compare the reactive optimizer with the natural options one would have if they wanted to one-shot the new training: adopting five leaderboard recipes with natural adjustments. In all these new settings, the reactive optimizer outperforms or matches the best baseline (Figure 1).
Figure 1. OPAL (OPtimizer with an Agent in the Loop), our reactive optimizer, across different settings. In most settings without heavily tuned optimization hyperparameters, reactive optimizers improve the training loss and test performance upon the canonical choices. In modded-nanogpt, #2 is AdamW, #21 is Shampoo, #36 is a tuned Muon, and #30 and #46 are more elaborate Muon-based recipes.
We also apply the reactive optimizer directly to Goldfish (Chang et al., [4]), which proposes a dataset of 350 languages and trains one small GPT per dataset with a standard recipe not tuned for each language. This makes it an ideal ground to test the optimizer’s adaptiveness in one shot. On the Amharic dataset, the official recipe ends at 83.48 bits per sentence on held-out FLORES-200 text (lower is better); the reactive optimizer, initialized with this same recipe, ends at 81.11 in one shot (on Swahili, we obtain a similar gain). Sweeping the learning rate of the standard recipe finds a static recipe that ends ahead of the reactive optimizer, at 80.89, but the reactive optimizer initialized from that swept recipe ends ahead again, at 80.73.
We also try the reactive optimizer on image generation, a direction where the literature usually does not sweep optimization hyperparameters, since diffusion-model design and optimizer design are viewed as mostly orthogonal1. It ends ahead of the official recipe on LightningDiT-XL/1 ((Yao et al. (2025); FID-50K 1.83 against 2.18 with guidance, 4.33 against 5.15 without), whose official recipe trains with AdamW, and on RAEv2-B (3.19 against 3.39), whose recipe trains with a Muon variant.
Figure 1 shows these runs, each with Claude Opus 5.5 powering the agent; we name our reactive optimizer OPAL (OPtimizer with an Agent in the Loop). Together, these results show that in one-shot settings, a reactive optimizer can lead to a better solution than adhering to a predetermined recipe.
Beyond the practical benefit, we are intrigued by the spontaneous decisions made by the zero-shot agent, after observing and analyzing the training dynamics in its own way. In the next two sections, we explore what the reactive optimizer chooses to watch and what it changes in response.
et al., 2026); Li et al. [21]; RAEv2’s GMuon (Singh et al. [34]) and a recent controlled benchmark of optimizers for diffusion models (Bolatov et al. [3]).
Updating with spontaneous analysis
Figure 2 shows how an agent updates the optimizer code on the fly. Concretely, we allow the agent to rewrite a file, updater.py, used in every step of optimization. We use code as the entire interface since this allows the agent to easily alter any part of the optimization. Yet, before the agent can make changes, it must observe, on its own, how training is progressing, leading to the second program it can rewrite, analysis.py.
Figure 2. The two programs the agent writes and rewrites iteratively during training. With code as the entire interface, we give the agent maximal freedom to choose what to watch and what to change.
What information should the agent use to make its decisions? Raw training states would be impractical: the weights, activations, and gradients would swamp its context. Instead, the statistics and geometry of training dynamics could be helpful for decisions, such as the gradient norm of each parameter group, the size of an update next to the weights it changes, the cosine between consecutive gradients, and which tensors exhibit unusually large gradient spikes. Therefore, as with the optimizer, we give the agent the most freedom: it writes a program, analysis.py, to extract whatever it wants to know from the training dynamics. Its self-written probes run throughout training, taking from the state of training whatever the agent asks for and handing it to the agent for its next decision.
We complete the bidirectional loop. Through analysis.py, training reaches the agent in whatever form the agent chooses to see it, and through updater.py, the agent’s decisions reach every step of training. The two programs let the agent devise new optimization algorithms along the way with maximum flexibility.
What is dynamically changing during optimization?
In this section, we look at what the reactive optimizer does throughout training, examining the decisions behind these results. As a summary, Figure 3 shows when, how, and why each agent edits its code.
Figure 3. Training trajectories and code edits in four OPAL runs. Blue curves show OPAL’s loss alongside the corresponding fixed-recipe baselines; ticks below each curve mark decisions and distinguish code edits. The edit patterns differ across runs, reflecting module-specific changes made at different stages of training. The curves show the outcomes of the complete runs, rather than isolating the effect of each edit.
The rest of this section examines these adaptations type by type.
Learning-rate schedule
Frontier training runs come with elaborate hyperparameter schedules, such as the five stages of DeepSeek-V3’s learning rate (DeepSeek-AI et al. [10]; Figure 4, upper left), with such schedules sometimes being written by humans during the training process (Zhang et al., 2022). Engineers can make only so many decisions, each weighing only so many signals. A reactive optimizer can take over this work and has no such limit: it decides frequently, and at each decision, it can access the entire history of the run. As Figure 4 shows, the schedules our agents built are intricate, with separate multi-stage schedules for each parameter group, and differ from run to run. Several of them raise a learning rate mid-run, a move rarely seen in modern schedulers, yet naturally helpful in many scenarios ([24, 35]). Moreover, the four examples in Appendix § 10 show the evidence they cited for their decisions.
Figure 4. DeepSeek-V3’s learning rate, designed by hand (upper left), and learning rates OPAL set in four runs, as multiples of each recipe’s rate. The schedules vary across tasks and parameter groups and include mid-run rate increases as well as cooldowns. These are observed trajectories, rather than schedules specified before training.
We observe that the agents base these decisions on the diagnostics from analysis.py. Some examples are: the cosine between a module’s consecutive gradients as a sign of overshooting (“Deduced from clues” in Appendix § 10); A/B testing between two learning rates every few hundred updates, comparing how fast the loss falls under each (“A-B-A test”); trial and error, with the learning rate raised, lowered, and raised again (“Trial and error”); and the trend of the loss, which makes agents start their cooldown earlier than planned (“Re-planned cooldown”). Each of these changes of schedule involves observation of the training dynamics and reasoning upon said observations.
Optimizer design
The reactive optimizers also change which rule updates which tensors. In our experiments, many switch from the initially given AdamW to the well-known hybrid, Muon, for the attention and MLP matrices while remaining with AdamW for the rest. Later, they make more specific moves (Figure 5).
Figure 5. Changes of optimizer type and of precision in four OPAL runs. In some runs, the weights are simply moved to standard Muon, while in others, OPAL moves them gradually by module type and chooses high precision for a very small part of the weights, citing the signals it observed.
The optimizer can also decide where precision matters. On modded-nanogpt Track 3, it measured that the entries of the token embedding are about 15 in size, where adjacent bfloat16 values are 0.0625 apart, so as the learning rate decayed, rounding would throw away a growing share of each update. From update 1,040, it kept that specific matrix’s Adam moments in float32 and wrote its weights back with compensated rounding and left every other tensor as it was.
Momentum
We investigate the reactive optimizer’s ability to manipulate momentum on D toy landscapes, where we give the agent new abilities: resetting its momentum, returning to its best point so far, and jumping to a random nearby point. In these toy settings, standard OPAL would simply default recipe with a second-order method within a few decisions and reach perfect performance very fast. We therefore restrict OPAL to the heavy-ball algorithm, while letting it control the hyperparameters. On the landscape in Figure 6, the start sits beside a nearly flat saddle where common optimizers stall. The reactive optimizer raises its momentum to cross the flat stretch, lowers it as the gradient falls near the minimum, and each time it overshoots, returns to its best point
Figure 6. A variant of the tilted double-well potential, one of the 34 landscapes we test optimizers on. The dashed line uses the blog’s static Preset A, while the blue line traces the reactive optimizer. The reactive optimizer raises its momentum along the gentle slope, lowers it as the gradient falls, and clears it when it overshoots, reaching the optimum where common optimizers at their default settings stall.
and restarts its momentum, reaching the minimum by the end of its 300-step budget. Over all 34 landscapes we ran, it achieves an average score of (, lower is better), while any single static setting does worse: for the best of twelve common recipes, and for the best of 382,080 swept recipes (see the aggregate results in the interactive blog).
Loss spikes
In our large-scale training, OPAL also limits the damage of gradient spikes with module-specific clipping: another SiT agent traced 17 of the 18 gradient spikes it logged to the timestep embedder, and added a clipping threshold for each tensor, set from that tensor’s own recent gradient norm, so that a spike in the embedder no longer cuts the updates of every other tensor; its old global threshold stays only as a backstop. Neither move is standard practice, but both have precedents: clipping each tensor against its own history [39], and damping the one layer that spikes, as GLM-130B did for its embedding layer (Zeng et al., 2022).
Spikes can also come from the data, through the interaction of particular batches with the model state [7], and clipping does not in general protect against noisy labels [27]. In a controlled experiment, we insert four stretches of random images and labels into CIFAR-10, and train a ResNet-18 with OPAL, which is not told about the corruption, and with static optimizers (Figure 7). The first two stretches cost OPAL 50 and 66 points of test accuracy, about as much as the baselines. In between, it added loss damping, replaced it with clipping, and then wrote a gate that holds back batches whose loss is far above recent clean ones. After it tuned that gate, the third stretch cost 12 points and the fourth none, while the clipped baselines and Stable-SPAM [13] still lost 28 to 76 points in each.
Figure 7. ResNet-18 on CIFAR-10 with four stretches of random images and labels, shaded. Left: test loss; markers indicate changes to OPAL’s safeguards. Right: the change in test accuracy in each stretch. Reactive agents can defend against the designed type of loss spikes, induced by data corruption, after seeing it a few times. Clean segments use Gaussian smoothing in log loss ( updates); the first 1,000 updates, corruption intervals and two evaluations after each interval remain unsmoothed.
Overfitting
Training on data with wrong labels is a standard way of testing overfitting [29]. We design such a setting (Appendix § 11.§ 2) and ask if the optimizer can overcome overfitting by itself, reacting to signals from the training set and the validation set. Trained straight through, the AdamW baseline ends at test loss 2.49 and accuracy 73.5% (stopped early on the validation set, at 1.01 and 72.9%; with its weights averaged and the average chosen there, at 0.83 and 77.5%). The reactive optimizer ends at loss 0.69 and accuracy 78.6%. As a direct measure of memorization, when asked to classify the 4,500 mislabeled training images, the baseline gives the wrong label it was trained on for all of them, its early-stopped version for 19% and its weight-averaged version for 14%. The reactive optimizer, on the other hand, gives that label for only 4%, and the true label for 78%. Its strategy combines progressively higher learning rates with weight averaging, and it chooses its final average on the validation set, which its own analysis program calls “extended SWA” [15].
Summary of analysis
Above, we show that reactive optimizers can adapt their recipe to the dataset, to individual modules, and to the behavior of each run, a trait unseen in conventional optimizers. Such customization and self-adaptiveness allow them to craft sophisticated schedules and navigate complex training landscapes.
Still, tuning hyperparameters, suppressing loss spikes, and watching out for overfitting are things human experts can do, albeit less conveniently and at a higher cost. As agents get smarter, with hundreds of papers in their memory, thousands of tokens in their chain of thought, millions of training signals in their context, and every step in their hands, we are convinced that reactive optimizers will only become better, developing mechanisms during training more arcane than the ones demonstrated here, beyond what human experts could tune by hand. Such mechanisms may include tracking the second-order curvature of the loss landscape, auditing the variance of each data batch’s signal, or implementing never-before-seen optimization schemes.
Figure 8. ResNet-18 trained for 120 epochs on 45,000 CIFAR-10 images, 10% with a wrong label. At epochs 24 and 28, validation loss climbs and the agent reacts. OPAL counters the overfitting with progressively higher learning rates and a weight average it chooses on the validation set. OPAL ends at a test loss of 0.685, compared with 2.493 for the final AdamW weights and 0.832 for its validation-selected weight average; the test set is not used by the agent.
| Test loss ↓ | Test acc. ↑ | Mislabeled training subset | ||
| True label ↑ | Wrong label ↓ | |||
| OPAL | 0.685 | 78.6% | 77.5% | 3.7% |
| AdamW, final | 2.493 | 73.5% | 0.0% | 100.0% |
| AdamW, early stopping on validation | 1.010 | 72.9% | 59.7% | 19.4% |
| AdamW, validation-selected EMA | 0.832 | 77.5% | 70.0% | 14.2% |
Table 1. Test performance and predictions on the mislabeled training subset. Early stopping and weight averaging use the validation set; details are in Appendix § 11.§ 2.
Real-world applicability
Is a reactive optimizer practical? The very first concern is the time spent waiting for reasoning decisions. In its simplest design, training is stopped while the agent reasons, inducing significant overhead. In our OPAL implementation, we instead provide the option to allow training to continue while the agent thinks, and swap in the agent’s edits at a later update (e.g., a median of 65 updates later on Goldfish). This should have little impact on performance, since most optimizer decisions are not especially time-critical. With this option, the runs in Figure 9 add only a little time on top of the static run ( to ). At a larger scale, the current implementation still needs work to support sharded training and optimizer compilation; we leave these practical concerns to later work.
Figure 9. Wall-clock time relative to the same task trained without an agent. Training that waits for the agent at every decision takes much longer ( to ), while training that does not wait adds only a little time on top of the static run ( to ). That remaining overhead is the cost of running what the agent wrote: an update rule that costs more per step than the recipe’s (such as Muon’s orthogonalization, or keeping a tensor in float32) and the measurement cost of its analysis program.
When comparing API tokens to GPU hours, the reasoning of a reactive optimizer costs little compared with the training compute. At about 100 decisions in a single training run, the LightningDiT agent costs $6.42 at Claude Opus 5.5 list price, on top of about 74 H100-hours of training (about $510 at AWS’s current on-demand price); equipping it on larger training would only make this relative cost smaller. Admittedly, counted in FLOPs, any reactive optimizer would be impractical given the size of frontier agents. Here we simply want to point out that, for most researchers, GPU hours and token access to closed-source models are two different commodities and as the frontier continues to advance, the price of a given capability will continue to fall.
Another limitation is that the reactive optimizer needs room to observe and react; its gains shrink when a run is short and its recipe is fine-tuned. The modded-nanogpt track is such a setting: 3,250 updates of a recipe that many people have tuned for nearly this exact task; our gains there are smaller than in the other tasks we tried, which train for about 8 to 120 times as many updates.
Moreover, the reactive behavior introduces another source of uncertainty, potentially making training less reproducible. Most experiments in this work are run once, a few three times; reruns on the released package are in the OPAL repository.
Fine-tuning platforms are bringing training to everyone. Tinker [36] and Prime Intellect’s Hosted Training [31] let users fine-tune open-weight models for their own applications without managing the GPUs behind them. On these platforms, however, the mental burden of a good training recipe still falls on people. A reactive optimizer fits naturally on top of these platforms, offloading that burden to agents and helping each user personalize a model to their own needs.
Conclusion
Until now, optimizers have been written before training. Agents, as a cheap form of reasoner with a long context window and expert knowledge, make something new possible: an optimizer that keeps deciding while the network trains in reaction to optimization events. We have demonstrated that such a reactive optimizer watched the training dynamics and rewrote its own code as training unfolded, for different groups of parameters, at the step it deemed proper. On toy and scaled-up experiments, most of these runs ended ahead of the recipes from which they started.
The analysis and update mechanism we design is meant to be as simple and general as possible. Wrapping a standard optimizer, it assumes nothing about the model, the data, or the training dynamics. It is useful today wherever no recipe exists yet. For example, a model on a biological or other scientific dataset, an architecture for which no recipe was written, or a language for which no one has tuned a recipe are all excellent use cases for such a reactive optimizer. This is because the optimizer’s agent takes the role of a human expert at countless training steps, where no expert has ever had the time to sit, making similar decisions in response to events in the run. Seen this way, a reactive optimizer is a natural thing to build, and its gains over static ones are to be expected. Furthermore, a training run holds far more information than humans can use, almost all of which is ignored today. A reactive optimizer has all this information within reach. Imagine a world where the notion of a monolithic optimizer is gone; what replaces it is an optimization intelligence that adapts to every specific run, and it might not be gradient-based, but zeroth-order [8], language-based or more. Perhaps, in the future, we should delegate optimization decisions to agents, not humans.
In the past decade, deep learning models have come to possess unreasonably effective power through training. Yet the very heart of their making, the optimizer, is still mechanistic. This work is an attempt to breathe the magic that training has given us into the core of training itself, forming a loop that we hope will create more capability in the future.
Acknowledgements
We sincerely thank George Cazenavette, David Charatan, Shivam Duggal, Jyo Pari and Qianwei Jia for their invaluable advice on this write-up; thank Zhenting Qi and Zeqing Yuan for their discussion. We are also grateful to Professor Phillip Isola and the Isola lab for their insightful feedback on an earlier version of this work.
Related work
Learning optimization rules and adapting them to training feedback have a long history. Early work already considered improving learning strategies themselves and learning context-dependent weight updates, through self-referential learning systems and neural fast-weight controllers (Schmidhuber [32, 33]). Andrychowicz et al. [1] train recurrent networks to produce parameter updates, while Li and Malik [22] formulate optimization as a policy learned through reinforcement learning. Daniel et al. [9] learn step-size controllers, and VeLO scales meta-training across optimization tasks ([28]). These methods already make updates depend on the evolving training state. For reinforcement learning, Optim4RL and OPEN learn update rules tailored to its optimization difficulties ([20, 11]). AdaQN instead selects a shared target online from critics trained with different hyperparameters ([37]). Population-based training discovers hyperparameter schedules while jointly training a population of models ([16]). SEAL uses reinforcement learning to train a language model to generate its own fine-tuning data and update directives, including optimization hyperparameters (Zweiger et al., 2025). Our setting uses a pretrained general-purpose agent to inspect one ongoing run and rewrite its executable optimizer code, without optimizer-specific meta-training. The distinction lies in adapting executable update code within the run, rather than in the ability to respond to training feedback itself.
Learning-rate decisions
Deduced from clues
The SiT agent’s analysis program measured the cosine between consecutive gradients of each parameter group. Where it was negative, the agent lowered the learning rate: the timestep embedder’s went to and then to of the official rate, with this stated reason:
Step 7,506, Decision 6
Timestep-embedder learning rate goes from to () after update 7506, because its cosine is still around .
Where the cosine stayed slightly positive, for the Muon matrices, it raised the learning rate.
A-B-A test
The RAEv2 agent alternated two learning rates over consecutive stretches of a few hundred updates (A-B-A) and compared the training loss under each. After one such test, it switched to from update 3,100, with this stated reason:
Update 3,000, Decision 12
The A-B-A test points the other way from my provisional read at update 2750: gives faster real progress than .
Trial and error
The same agent set every group’s learning rate to , returned the backbone’s learning rate to when the backbone’s gradients began to change sign at each step, and at the next decision raised every group’s learning rate to :
Update 1,000, Decision 4
My explanation from last time was wrong, so I’m raising the learning rate in every group to ().
It then ran learning rates of , and , each for about 650 updates, and kept . Below, the learning rate of its backbone attention matrices, as a multiple of the official rate; the two deep dips are 75-update reductions of the learning rate used as tests.

Figure 10 (PDF p. 16).
Re-planned cooldown
At step 190K, the SiT agent moved its planned cooldown from 320K to 280K and lowered its learning-rate floor from 0.1 to 0.05 of its peak (the bend in the SiT panel of Figure 4). Its stated reason:
Step 190,152, Decision 53
the constant-rate phase is now mostly holding the weights at a noise floor set by the learning rate, rather than making progress.
The RAEv2 agent started its cooldown at update 11,250, 45% of the run instead of the planned 75%, citing the trend of its loss and a paper on decaying the learning rate [2].
Experimental settings and decision records
Overfitting experiment
A ResNet-18 trained on CIFAR-10 for 120 epochs without data augmentation. Among the 50,000 training images, 10% are held out with their correct labels as a validation set; among the rest, 10% get a wrong label, drawn uniformly from the wrong classes and fixed for the run. The agent saw training statistics, along with validation loss and validation accuracy; it was not told about the specific setting (that some labels were wrong), and it never saw the test performance. We make the network normalize its logits (a centered form of LogitNorm, [40]) to prevent loss-hacking behavior. AdamW runs from learning rate , which is also the starting recipe for the reactive optimizer.
OPAL’s final weights are an average of its weights from epochs 96–117, chosen on the validation set. The AdamW EMA baseline uses a moving average of its weights, also chosen on the validation set.
Settings for Figure 1
| Model / task | Field | Setting |
| modded-nanogpt Track 3 | Task | the 162M GPT of the modded-nanogpt optimization track (Track 3), the same runs in each setting: official NanoGPT, the track’s own (FineWeb at the official batch of 524,288 tokens per update); NanoGPT, ¼ batch (FineWeb, 13,000 updates of 131,072 tokens); NanoGPT on Chinese, Thai and Kyrgyz (the 1 GB Goldfish Chinese, Thai and Kyrgyz sets, 3,000 updates of 524,288 tokens) |
| modded-nanogpt Track 3 | Metric | official NanoGPT: the first update at which the validation loss reaches 3.28, the track’s target (validation every 25 updates); ¼ batch: validation loss after 13,000 updates; Chinese, Thai and Kyrgyz: validation loss after 3,000 updates, for #46 and OPAL from #36 with #46’s own readout (a moving average of the weights over the last updates), for the other records with the last weights; in Chinese and Thai, OPAL’s bar is the mean of three runs. The loss bars start at c, the largest multiple of 0.1 at least 0.05 below the setting’s lowest value: 3.2 (¼ batch), 4.0 (Chinese), 4.5 (Thai) and 3.7 (Kyrgyz) |
| modded-nanogpt Track 3 | Baseline | five records from the leaderboard, #46, #30, #21 (Shampoo), #36 (the tuned Muon baseline) and #2 (AdamW), and OPAL from #36. The leaderboard lists 2,690, 2,930, 4,100 and 3,250 updates for #46, #30, #21 and #36; #2’s bar in official NanoGPT is the leaderboard’s 5,625: our own run of #2 ended at 3.28449 after 5,625 updates and never reached 3.28; at ¼ batch every record ran with all its learning rates ×¼. |
| Goldfish, Amharic | Task | a 125M GPT trained from scratch on 1 GB of Amharic (Goldfish), 10 epochs (64,626 updates) |
| Goldfish, Amharic | Metric | bits per sentence on FLORES-200 (dev + devtest) |
| Goldfish, Amharic | Baseline | the official Goldfish recipe (AdamW, learning rate 1e-4, weight decay 0.01, 10% warmup, then linear decay; gradient clipping at 1.0); nine more static recipes, each for all 10 epochs: AdamW at learning rates 6e-4, 1e-3, 2e-3 and 4e-3 with weight decay 0.1 (a four-value sweep), AdamW at 3e-4 with weight decay 0.1 or 0.01 and at 1e-4 with weight decay 0.1, and Muon on the hidden block matrices in the official recipe and at learning rate 3e-4, weight decay 0.1. OPAL ran twice: from the official recipe (at its defaults, 50 decisions; shown by default) and from the sweep’s best point, AdamW at learning rate 2e-3 (200 decisions; shown with the sweep) |
| LightningDiT-XL/1 | Task | image generation on ImageNet 256×256 in a VA-VAE latent space; LightningDiT-XL/1, 64 epochs |
| LightningDiT-XL/1 | Metric | FID-50K at the end, with and without guidance; the curve shows FID-10K during training |
| LightningDiT-XL/1 | Baseline | the official recipe, retrained by us, and the authors’ released checkpoint (FID-50K 5.23 without guidance, 2.16 with) |
| RAEv2-B | Task | image generation on ImageNet 256×256 in a representation-autoencoder latent space; RAEv2-B, 20 epochs |
| RAEv2-B | Metric | FID-50K |
| RAEv2-B | Baseline | the official recipe, which already trains with GMuon, a Muon variant |
Table 1 (PDF p. 17).
Program changes in Figure 3
All decisions that changed a program
LightningDiT-XL/1
| Step | File | What changed | Tags |
| 800 | updater.py + analysis.py | Moved the attention and MLP weights to Muon, 196 matrices once the fused tensors are split, each step matched to its AdamW size; kept adaLN and the rest on AdamW; planned a cosine cooldown from 60K | block-specific, time-step-specific |
| 1,600 | updater.py + analysis.py | Ramped Muon’s steps to 1.25× their matched size over updates 1,600–2,000; added a guard that cuts a step’s rate when the gradient norm passes 3× its mean | observation-originated, time-step-specific |
| 2,400 | updater.py + analysis.py | Fixed a reporting bug; split its gradient–momentum cosine by matrix type and depth, and added it for the AdamW tensors | analysis only, fix or revert |
| 3,200 | updater.py | Added a controller that multiplies each AdamW tensor’s rate by 0.9 every 100 updates while its gradient opposes its momentum (down to 0.2) | niche practice, block-specific, observation-originated |
| 4,000 | updater.py | Removed that controller; moved the adaLN weights to Muon, each split into its six 1152×1152 chunks | fix or revert, niche practice, block-specific, observation-originated |
| 4,800 | updater.py | Ramped Muon’s steps to 1.5× their matched size for attention and MLP and 1.25× for adaLN, over updates 4,800–5,600 | block-specific, observation-originated, time-step-specific |
| 6,400 | updater.py | Moved three more matrices to Muon: the timestep embedder’s second layer and the final adaLN’s shift and scale | block-specific, observation-originated |
| 13,600 | updater.py | Replaced the planned cooldown with a decay from 54K to 0.03× at 76K (was cosine from 60K) | time-step-specific |
| 16,000 | analysis.py | Added per-tensor gradient norms outside the blocks and a spike log; never applied (its answer reached the host too late) | analysis only, observation-originated |
| 16,800 | analysis.py | Wrote the same diagnostics again; also never applied | analysis only, observation-originated |
| 17,600 | analysis.py | Wrote them a third time; these ran | analysis only, observation-originated |
| 18,400 | updater.py + analysis.py | Made the spike guard ignore the timestep embedder’s first layer, whose noisy gradient had caused nearly all its cuts | fix or revert, block-specific, observation-originated |
| 39,200 | updater.py | Moved the cooldown start from 54K to 50K, as progress at the constant rate had nearly stopped | observation-originated, time-step-specific |
Table 3.
RAEv2-B
| Step | File | What changed | Tags |
| 250 | updater.py + analysis.py | Re-implemented the official GMuon and AdamW with a rate multiplier per parameter group (all ); planned a linear cooldown from update 18,765 to | time-step-specific |
| 500 | updater.py + analysis.py | Ramped every rate to over updates 500–750; added a clip at twice the running gradient norm | time-step-specific |
| 750 | updater.py + analysis.py | Put the backbone back to and kept the head at , after the backbone’s gradients began to flip sign each step | block-specific, observation-originated |
| 1,000 | updater.py + analysis.py | Raised every group to over updates 1,000–1,250; its analysis now multiplies the rate by when the loss rises 3% above its recent best | observation-originated, time-step-specific |
| 1,500 | updater.py + analysis.py | Scheduled a probe: the rate at of its value for updates 1,576–1,650, to measure the step-size noise in the loss | niche practice, time-step-specific |
| 1,750 | updater.py | Scheduled an A-B-A test: for updates 1,850–2,500, between stretches at | niche practice, observation-originated, time-step-specific |
| 3,000 | updater.py | Chose from update 3,100, after the test showed faster progress at than at | observation-originated, time-step-specific |
| 3,500 | updater.py | Scheduled a second test: for updates 4,000–4,750, then back to | observation-originated, time-step-specific |
| 5,000 | updater.py | Scheduled a test of Muon momentum 0.98, instead of 0.95, for updates 5,500–6,250 | observation-originated, time-step-specific |
| 5,750 | updater.py | Cut the momentum test short, back to 0.95 from update 6,001, as the loss had risen | fix or revert, observation-originated, time-step-specific |
| 6,500 | updater.py + analysis.py | Moved the two input embedders ( convolutions) from AdamW to Muon | block-specific, observation-originated |
| 7,000 | updater.py | Moved the input embedders back to AdamW, after no gain | fix or revert, block-specific, observation-originated |
| 7,250 | updater.py | Scheduled a test: the AdamW groups and the timestep embedder at for updates 7,500–8,250, the rest at | block-specific, observation-originated, time-step-specific |
| 8,500 | updater.py | Scheduled a second dip, to of the rate for updates 8,576–8,650, after the loss had stalled | observation-originated, time-step-specific |
| 8,750 | updater.py | Scheduled a third test, for updates 9,000–9,750; fixed a bug that would have broken a rerun of its schedule | fix or revert, observation-originated, time-step-specific |
| 11,250 | updater.py | Started the cooldown at update 11,250 instead of 18,765, after the loss stalled at | observation-originated, time-step-specific |
Table 4.
Goldfish (Amharic)
| Step | File | What changed | Tags |
| 323 | updater.py + analysis.py | Moved the 48 block matrices to Muon at 0.02 (Q, K and V orthogonalized apart) and kept AdamW at 2e-3 on the rest; ramp to the peak at update 1,600, then a linear decay to 0 | block-specific, time-step-specific |
| 646 | analysis.py | Added a wake-up: an early decision when a window’s mean loss rises 0.25 above the best recent one | analysis only |
| 1,615 | updater.py | Halved both peak rates (Muon 0.02 to 0.01, AdamW 2e-3 to 1e-3) and doubled Muon’s weight decay to 0.02 | block-specific, observation-originated |
| 6,460 | analysis.py | Fixed an off-by-one in the same-batch comparison: an epoch is 6,462 updates, not 6,463 | analysis only, fix or revert |
| 6,783 | updater.py | Raised Muon momentum from 0.95 to 0.975 | block-specific, observation-originated |
| 7,106 | updater.py | Put Muon momentum back to 0.95, after the loss rose and the weights grew | fix or revert, block-specific, observation-originated |
| 8,075 | updater.py | Lowered the Muon peak rate from 0.01 to 0.007 | block-specific, observation-originated |
| 9,044 | updater.py | Cut the AdamW rate (embeddings, LayerNorms, biases) to 0.7 | block-specific, observation-originated |
| 11,628 | updater.py | Raised that AdamW multiplier from 0.7 to 0.85, by the rule it had set at decision 28 | fix or revert, block-specific, observation-originated |
| 12,597 | analysis.py | Added the loss over the last 323 updates minus the same updates one epoch earlier, as batch difficulty repeats each epoch | analysis only, observation-originated |
Table 1 (PDF p. 20).
NanoGPT, ¼ batch
| Step | File | What changed | Tags |
| 260 | updater.py + analysis.py | Rewrote the optimizer: Muon on the 72 block matrices on every rank, momentum restarted; rates, momenta and Muon weight decay moved part way to small-batch values | |
| 1,560 | updater.py | Gave the final norm gain, which scales the logits, its own rate, cut from 0.009 to 0.003 | block-specific, observation-originated |
| 1,820 | updater.py + analysis.py | Put that gain’s rate back to 0.009; added gradient-consistency measures from the optimizer state | fix or revert, block-specific, observation-originated |
| 2,080 | updater.py | Lowered Muon’s rate on the attention value matrices to 0.75 of the rest | block-specific, observation-originated |
| 8,060 | updater.py | Made Muon’s weight decay follow the learning-rate schedule from here on | niche practice, block-specific, observation-originated, time-step-specific |
Table 2.
References
- [1]Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W. Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando de Freitas. Learning to learn by gradient descent by gradient descent. In Advances in Neural Information Processing Systems, volume 29, 2016. URL https://arxiv.org/abs/1606.04474.
- [2]Shane Bergsma, Nolan Dey, Gurpreet Gosal, Gavia Gray, Daria Soboleva, and Joel Hestness. Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs, 2025. URL https://arxiv.org/abs/2502.15938. ICLR.
- [3]Arman Bolatov, Egor Shulgin, David Li, Abduragim Shtanchaev, Sebastian U. Stich, Maxim Panov, Eric Moulines, Peter Richtárik, and Martin Takáč. Optimizers for Diffusion Models: A Controlled Benchmark, 2026. URL https://arxiv.org/abs/2609.23055.
- [4]Tyler A. Chang, Catherine Arnett, Zhuowen Tu, and Benjamin K. Bergen. Goldfish: Monolingual Language Models for 350 Languages, 2026. URL https://arxiv.org/abs/2408.10441. LREC.
- [6]Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Yao Liu, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, and Quoc V. Le. Symbolic Discovery of Optimization Algorithms, 2023. URL https://arxiv.org/abs/2302.06675.
- [7]Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsyvashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. PaLM: Scaling Language Modeling with Pathways, 2022. URL https://arxiv.org/abs/2204.02311.
- [8]Samip Dahal, Bishwas Mandal, Serdar Gülbahar, and Akshay Vegesna. Dust: Pretraining Transformers Without Backpropagation, 2026. URL https://qlabs.sh/research/dust.
- [9]Christian Daniel, Jonathan Taylor, and Sebastian Nowozin. Learning Step Size Controllers for Robust Neural Network Training, 2016. URL https://doi.org/10.1609/aaai.v30i1.10187. AAAI.
- [10]DeepSeek-AI et al. DeepSeek-V3 Technical Report, 2024. URL https://arxiv.org/abs/2412.19437.
- [11]Alexander David Goldie, Chris Lu, Matthew Thomas Jackson, Shimon Whiteson, and Jakob Nicolaus Foerster. Can learned optimization make reinforcement learning less difficult? In Advances in Neural Information Processing Systems, volume 37, 2024. URL https://arxiv.org/abs/2407.07082.
- [12]Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned Stochastic Tensor Optimization, 2018. URL https://arxiv.org/abs/1802.09568. ICML.
- [13]Tianjin Huang, Haotian Hu, Zhenyu Zhang, Gaojie Jin, Xiang Li, Li Shen, Tianlong Chen, Lu Liu, Qingsong Wen, Zhangyang Wang, and Shiwei Liu. Stable-SPAM: How to Train in 4-Bit More Stably than 16-Bit Adam, 2025. URL https://arxiv.org/abs/2502.17055v2.
- [14]Alexander Hägele, Elie Bakouch, Atli Kosson, Loubna Ben Allal, Leandro Von Werra, and Martin Jaggi. Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations, 2024. URL https://arxiv.org/abs/2405.18392. NeurIPS.
- [15]Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging Weights Leads to Wider Optima and Better Generalization, 2018. URL https://arxiv.org/abs/1803.05407. UAI.
- [16]Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M. Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, Iain Dunning, Karen Simonyan, Chrisantha Fernando, and Koray Kavukcuoglu. Population Based Training of Neural Networks, 2017. URL https://arxiv.org/abs/1711.09846.
- [17]Keller Jordan. Muon: An optimizer for hidden layers in neural networks, 2024. URL https://kellerjordan.github.io/posts/muon/.
- [18]Keller Jordan. Modded-NanoGPT Optimization Benchmark, 2026. URL https://github.com/KellerJordan/modded-nanogpt/tree/master/records/track_3_optimization. Online benchmark version accessed October 24, 2026.
- [19]Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization, 2015. URL https://arxiv.org/abs/1412.6980. ICLR.
- [20]Qingfeng Lan, A. Rupam Mahmood, Shuicheng Yan, and Zhongwen Xu. Learning to optimize for reinforcement learning. In Reinforcement Learning Conference, 2024. URL https://arxiv.org/abs/2302.01470.
- [21]Chenghao Li, Xiao Han, Xinxin Huang, Wei Liu, Boyang Li, Bing Xiao, Heran Zhang, Juanma Perez Rua, Ke Xu, Kangning Liu, Linjun Kuang, Na Li, Tan Wang, Tian Xie, Wei Peng, Yang Pei, Yifan Xu, Yuanhao Zhai, Yuwei Lin, Zhe Wang, Zihao He, Daniel Li, Junbiao Tang, Ziyang Jiang, and Dake Chen. Scaling Muon for Diffusion Transformers, 2026. URL https://arxiv.org/abs/2608.20818.
- [22]Ke Li and Jitendra Malik. Learning to optimize, 2016. URL https://arxiv.org/abs/1606.01885. arXiv:1606.01885.
- [23]Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, Yanru Chen, Huabin Zheng, Yibo Liu, Shaowei Liu, Bohong Yin, Weiran He, Han Zhu, Yuzhi Wang, Jianzhou Wang, Mengnan Dong, Zheng Zhang, Yongsheng Kang, Hao Zhang, Xinran Xu, Yutao Zhang, Yuxin Wu, Xinyu Zhou, and Zhilin Yang. Muon is Scalable for LLM Training, 2025. URL https://arxiv.org/abs/2502.16982.
- [24]Ilya Loshchilov and Frank Hutter. SGDR: Stochastic Gradient Descent with Warm Restarts, 2017. URL https://arxiv.org/abs/1608.03983. ICLR.
- [25]Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery, 2024. URL https://arxiv.org/abs/2408.06292.
- [26]Nanye Ma, Mark Goldstein, Michael S. Albergo, Nicholas M. Boffi, Eric Vanden-Eijnden, and Saining Xie. SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers, 2024. URL https://arxiv.org/abs/2401.08740.
- [27]Aditya Krishna Menon, Ankit Singh Rawat, Sashank J. Reddi, and Sanjiv Kumar. Can Gradient Clipping Mitigate Label Noise?, 2020. URL https://openreview.net/forum?id=rk1B76EKPr. ICLR.
- [28]Luke Metz, James Harrison, C. Daniel Freeman, Amil Merchant, Lucas Beyer, James Bradbury, Naman Agrawal, Ben Poole, Igor Mordatch, Adam Roberts, and Jascha Sohl-Dickstein. VeLO: Training Versatile Learned Optimizers by Scaling Up, 2022. URL https://arxiv.org/abs/2211.09760.
- [29]Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep Double Descent: Where Bigger Models and More Data Hurt, 2020. URL https://arxiv.org/abs/1912.02292. ICLR.
- [30]Prime Intellect. Autonomous Speedrunning Experiment, 2026a. URL https://github.com/PrimeIntellect-ai/experiments-autonomous-speedrunning.
- [31]Prime Intellect. Getting Started (Hosted Training), 2026b. URL https://docs.primeintellect.ai/hosted-training/getting-started. Accessed October 21, 2026; publication date unspecified.
- [32]Jürgen Schmidhuber. Evolutionary principles in self-referential learning. Diploma thesis, Technische Universität München, May 1987. URL https://people.idsia.ch/~juergen/diploma1987ocr.pdf.
- [33]Jürgen Schmidhuber. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation, 4(1):131–139, 1992. doi: 10.1162/neco.1992.4.1.131. URL https://people.idsia.ch/~juergen/fastweights/.
- [34]Jaskirat Singh, Boyang Zheng, Zongze Wu, Richard Zhang, Eli Shechtman, and Saining Xie. Improved Baselines with Representation Autoencoders, 2026. URL https://arxiv.org/abs/2605.18324.
- [35]Leslie N. Smith. Cyclical Learning Rates for Training Neural Networks, 2017. URL https://arxiv.org/abs/1506.01186. WACV.
- [36]Thinking Machines Lab. Announcing Tinker, 2025. URL https://thinkingmachines.ai/news/announcing-tinker/.
- [37]Théo Vincent, Fabian Wahren, Jan Peters, Boris Belousov, and Carlo D’Eramo. Adaptive Q-network: On-the-fly target selection for deep reinforcement learning. In International Conference on Learning Representations, 2025. URL https://arxiv.org/abs/2405.16195.
- [38]Nikhil Vyas, Depen Morwani, Rosie Zhao, Mujin Kwun, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham Kakade. SOAP: Improving and Stabilizing Shampoo using Adam, 2024. URL https://arxiv.org/abs/2409.11321.
- [39]Guox ia Wang, Shuai Li, Congliang Chen, Jinle Zeng, Jiabin Yang, Dianhai Yu, Yanjun Ma, and Li Shen. AdaGC: Enhancing LLM Pretraining Stability via Adaptive Gradient Clipping, 2026. URL https://arxiv.org/abs/2502.11034. ICML.
- [40]Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng, Bo An, and Yixuan Li. Mitigating Neural Network Overconfidence with Logit Normalization, 2022. URL https://arxiv.org/abs/2205.09310. ICML.








