WorksComputer Science
The Role of Quantization in Loss Spikes
Abstract
Loss spikes in language model pretraining can waste substantial compute and derail otherwise healthy training runs. A primary driver of these instabilities is the growth and divergence of output logits in the final language modeling (LM) head. In this work, we study a particularly costly form of this instability associated with output logit growth: delayed, terminal divergences that emerge only after prolonged training, which we call time-bomb spikes. We find that these instabilities are preceded by rapidly growing quantization error in the LM head, which eventually culminates in a loss spike. Crucially, reducing numerical precision accelerates time-bomb spikes, while increasing precision improves stability. We then propose a simple yet effective mitigation that replaces AdamW on the LM head with SGD with momentum, suppressing logit growth and improving stability over long training horizons. Finally, we find that existing mitigation techniques from the literature (along with our own) all reduce quantization error, suggesting a shared numerical mechanism behind their effectiveness. Together, our results identify finite precision error as a hidden driver of time-bomb spikes and show that suppressing this error is key to stable long-horizon pretraining.
Introduction
Pretraining Transformer-based language models at scale is notoriously difficult due to the instability of the objective and its propensity to diverge or spike during training [8] (Zhang et al. 2022; Zeng et al. 2023; OLMo et al. 2025). Large-scale training efforts have relied on heuristic procedures in order to recover from these spikes, including reverting to earlier checkpoints, skipping data near observed spikes, and adjusting learning rates [8, 2] (Zhang et al. 2022). While such interventions allow training to continue, they treat the symptoms of instability rather than its underlying cause.
A common class of loss spikes is associated with the growth and divergence of output logits in the language modeling (LM) head. Chowdhery et al. [8] originally identified this behavior and introduced -loss as a mitigation, an auxiliary loss penalty that discourages output logits from growing too large in magnitude. This mitigation has since become a standard safeguard in many training pipelines (Alial et al., 2025) and remains important in recent open-source, frontier-style training recipes. The latest Marin recipe includes -loss alongside norm-preserving hyperball optimization, with leave-one-out ablations showing substantial degradation when -loss is removed (Marin Community, 2026a;b).
Despite the widespread adoption of methods for controlling output logit growth, our understanding of why such unmitigated growth leads to loss spikes remains incomplete. Recent work has traced the growth of logits to the optimizer, showing that Adam’s adaptive preconditioning induces persistent drift in the mean output embedding of the LM head, which in turn causes the mean output logit to drift towards extreme values (Stollenwerk & Stollenwerk, 2025; Stollenwerk et al., 2026). Yet, as we show in § 3, this drift alone cannot substantively affect the forward or backward pass if we assume exact arithmetic. If this is the case, how can an apparently harmless drift ultimately destabilize training?
Code available at: https://github.com/tingtang2/quantization-loss-spikes
Figure 1. Reduced precision accelerates time-bomb spikes, while LM head SGD-M prevents them. Lowering LM head precision brings the failure earlier, with BF16 true diverging first, BF16 mixed later, and FP32 remaining stable throughout training (left). Output logit norm grows under every AdamW baseline, including FP32 (middle), but it destabilizes training only in reduced precision, where LM head quantization error grows by orders of magnitude before each failure (right). Quantization error is measured as the RMS difference between native BF16 logits and FP32 logits computed from the same hidden states. Replacing AdamW with SGD-M on the LM head keeps both logit norm and quantization error nearly flat, preventing the time bomb even in BF16 mixed precision. Results are for 28M parameter models trained at peak learning rate 0.1 for the Chinchilla optimal token horizon, using AdamW throughout (baselines) or SGD with momentum on the LM head (SGD-M).
To unravel this mystery we focus our study on a particularly costly form of instability driven by output logit growth: slow, unrecoverable loss divergences that emerge only after prolonged training, which we call time-bomb spikes. A run affected by a time-bomb spike can appear healthy for most of training before deteriorating irreversibly, after much of the compute budget has already been spent. While other kinds of fast, transient loss spikes exist, we find these are largely suppressed by learnable RMSNorm gains (§ 2.4). Time-bomb spikes, in contrast, persist in these architectures and become more severe with longer training horizons, making them increasingly relevant as models are trained far beyond the compute-optimal token budget [12] (Yang et al. 2025).
In this work, we argue that the link between output logit growth and catastrophic loss spikes stems from how finite precision computation in the LM head leads to quantization error. Using the time-bomb spike as a testbed, we explore simple interventions that act separately on each element of the puzzle, the drift itself and the precision of the LM head. First, we show that optimizing the LM head using SGD with momentum, rather than AdamW, eliminates these spikes by removing drift directly, indicating that the drift induced by preconditioning impacts stability in practice more than theory would predict. Second, we confirm that reducing quantization error by simply computing and optimizing just the LM head in higher precision is similarly sufficient for preventing time-bomb spikes. Finally, we tie together our LM head SGD-M mitigation, the effectiveness of increased precision, and other proposed methods in the literature by demonstrating that they all seem to work by suppressing quantization error through different mechanisms. In summary, our findings are:
LM head SGD-M effectively squashes spikes. We propose replacing AdamW with carefully tuned SGD with momentum exclusively on the LM head. This targeted intervention suppresses logit growth and improves stability over long training horizons.
Finite precision is a hidden driver of time-bomb spikes. Quantization error in the LM head grows rapidly before each failure, and loss diverges soon after. Precision directly controls when this happens. Lowering LM head precision triggers time-bomb spikes earlier, while computing the LM head in FP32 keeps runs stable over the horizons we study.
Existing mitigations all share a common numerical mechanism. Even though our proposed LM head SGD-M and other mitigations from the literature intervene in different ways, they all keep quantization error bounded, whereas it grows by orders of magnitude in unmitigated runs. Suppressing quantization error thus emerges as a shared reason these seemingly unrelated methods prevent time-bomb spikes.
Preliminaries
In this work, we investigate loss spikes stemming from output logit divergence in the language modeling head ([8] (Wortsman et al. 2024)). This type of training instability was identified during the training of PaLM ([8]), where the authors proposed z-loss as a mitigation technique.
Language modeling head
Consider a vocabulary of size and a final hidden state . Let denote the one-hot encoded ground truth labels for a single example such that . The output embedding matrix maps the final hidden state to output logits . The next-token distribution and cross entropy objective are computed as
Baseline Mitigations for Logit Divergence
z-loss. Output logit divergence occurs when the absolute value of the output logits grows excessively large. This can lead to both fast spikes, where loss increases quickly but eventually recovers, as well as slow loss divergences as studied in Wortsman et al. (2024). z-loss penalizes the logits for drifting too far from zero by introducing a regularization term based on whereby the complete loss becomes .
-centering. More recently, Stollenwerk et al. (2026) introduced -centering to precisely suppress output logit divergence. Unlike z-loss, which regularizes the loss objective, -centering is a structural intervention as it centers the output embeddings around zero. Let be the output embedding matrix. After each optimization step, the mean output embedding is subtracted from each individual output embedding , creating new output embeddings . By enforcing , this procedure ensures the mean output logit remains zero, regardless of the hidden state magnitude.
Experimental Setup
To reproduce the loss spikes seen in larger models, we train 28M–1.4B parameter models at elevated learning rates following Wortsman et al. (2024); Zhao et al. (2025); Stollenwerk et al. (2026); Su et al. (2025); Li et al. (2025) (fast and slow spike examples in Figure 3 and Figure 8). Models are implemented in JAX (Bradbury et al. [6]) and Flax (Heek et al. (2024)) and, unless otherwise noted, trained on up to 32 Google Cloud v6e or v4 TPUs in bfloat16 mixed precision. We use a modernized GPT-2 (Radford et al. (2019)) architecture with rotary positional embeddings (Su et al. (2024)), GELU activations (Hendrycks & Gimpel (2023)), RMSNorm (Zhang & Sennrich (2019)), untied embeddings, and no biases. We optimize with AdamW (Loshchilov & Hutter (2019)) (, , ) using linear warmup, cosine decay, and gradient clipping at a global norm of 1. Unless otherwise specified, weight decay is and batch size is 262,144 tokens. For z-loss runs, following Chowdhery et al. [8].
Figure 3. Long horizon training can result in time-bomb instability. Two 404M parameter models train similarly for most of pretraining, but the higher LR run exhibits delayed, unrecoverable loss increases accompanied by accelerating output logit growth.
Figure 8. Learning rates can control the type of training instability elicited. Example training runs at 1.4B scale trained for 10B tokens. We focus on the learning rates similar to the middle one in order to elicit fast loss spikes to analyze.
All runs use QK-Norm (Dehghani et al. [10]), the recommended intervention against attention logit growth (Wortsman et al. (2024)). This removes that source of spikes, so we can study instabilities from output logit divergence in isolation. All models are trained on FineWeb (Penedo et al. (2024)) with the GPT-2 tokenizer and, unless otherwise noted, standard cross entropy loss.
Warmup: Fast, Transient spikes
We begin by briefly examining fast loss spikes, which cause abrupt increases in loss before recovering. Unless otherwise stated, these experiments use parameter-less norms—RMSNorm layers without learnable gain parameters. This design choice is used in several large language model suites (Groeneveld et al., 2024; Wang et al., 2026) and is also shared by prior work that studies or reports training instabilities during language model pretraining (Geiping et al., 2025; Filatov et al., 2026).
We first investigate whether interventions targeting individual spike-inducing tokens or batches can resolve fast loss spikes. We find that masking as little as 0.2% of high-loss tokens suppresses visible spikes but degrades convergence, while skipping triggering batches and reverting the optimizer state merely postpones subsequent spikes (Section A.2). These findings suggest that avoiding individual spike-inducing examples is insufficient to eliminate the model’s susceptibility to fast spikes. These spikes stem not only from data but from the parameters and/or optimizer state.
Fast spikes depend strongly on normalization type. To test how strongly RMSNorm parameterization affects fast spikes, we repeat a representative fast spike configuration with learnable RMSNorm gains, holding all other hyperparameters and random seeds fixed. Across five matched seed pairs, parameterized RMSNorms substantially reduce the fast loss spikes and the associated output logit mean and norm growth. Figure 2 shows a representative matched pair, with all five comparisons reported in Figure 13.
Figure 2. Learnable gain parameters in normalization layers stabilize training and can prevent fast loss spikes. Comparing a 162M spiking run with the same seed and configuration only differing in the inclusion of learnable RMSNorm parameters, the parameterized norm configuration exhibits no fast spikes, output logit swings, or large logit norm growth.

Figure 13. Parameterized RMSNorms effectively mitigate fast loss spikes. The five parameterized runs are highly consistent and overlap closely, indicating low seed-to-seed variability relative to the parameter-less runs.
These results suggest that the abrupt fast spike regime studied above is strongly affected by RMSNorm parameterization and is substantially less prominent with the learnable gains used in many modern language models (Yang et al., 2025; Shoeybi et al., 2019). Importantly, however, LM head instability does not disappear altogether. In the next section, we show that models with learnable RMSNorm gains can still develop a slower, delayed form of instability during long horizon training.
Defusing Time-Bomb Loss Spikes
In this section, we turn our attention to a type of training instability that persists even when using parameterized norms: slow time-bomb spikes. Experiments in this section use RMSNorm layers with learnable gain parameters. All models use a batch size of 131,072 tokens and no weight decay; all other training hyperparameters follow the shared setup in § 2. We use this configuration because preliminary experiments indicated that smaller batch sizes lead to earlier time-bomb onset, while weight decay delays their emergence (as also noted in Wortsman et al. (2024)). All models use bfloat16 mixed precision (BF16 Mixed) by default unless otherwise noted.
Time-bomb loss spikes emerge during overtraining
To study time-bomb spikes, we overtrain 404M parameter models for 96B tokens across the same learning rate range as in Wortsman et al. (2024): . This corresponds to approximately 240 training tokens per parameter, or roughly 12× the Chinchilla-optimal token budget. Unless otherwise specified, these experiments are conducted on four NVIDIA B200 GPUs. Figure 3 shows two runs that follow nearly identical loss trajectories through most of training, but where the higher-learning rate run eventually stops improving and undergoes a sustained, monotonic increase in evaluation loss—a time-bomb. Full loss and output logit norm curves for the learning rate sweep are shown in Figure 16. Longer training also leads to greater late stage output logit norm growth (Appendix Figure 17), suggesting that time-bomb spikes become more of an issue in the overtraining regime.
Figure 16. Time-bomb instability emerges across the long-horizon learning rate sweep alongside accelerating output logit growth. Evaluation loss (left), output logit norm (middle), and output logit mean (right) for 404M-parameter models trained for 96B tokens across peak learning rates from to . Runs train similarly for much of pretraining, while higher-learning rate settings exhibit substantially greater late stage logit norm growth and, in several cases, deterioration in evaluation loss.
Adam's preconditioner induces LM head mean drift
Prior work has shown that the cross entropy gradient of the LM head has a zero-sum structure along the vocabulary axis, while Adam’s coordinate-wise preconditioning breaks this property ([4] (Stollenwerk & Stollenwerk 2025)). This can induce persistent drift in the mean output embedding and, by extension, in the mean output logits, directly linking the logit growth dynamics observed in the previous experiments to an interaction between the LM head gradients and Adam’s preconditioner ((Stollenwerk & Stollenwerk, 2025)).
Let denote the output embedding matrix and the incoming hidden activations, such that . The cross entropy gradient admits the following zero-sum property:
because softmax probabilities and one-hot labels both independently sum to one. Thus, any optimizer update that preserves this zero-sum structure leaves the mean output embedding over the vocabulary axis unchanged. Although Adam’s first moment estimate remains zero-sum, division by the coordinate-dependent second moment reweights vocabulary rows non-uniformly. Because these second moments vary strongly with token frequency ((Stollenwerk & Stollenwerk, 2025)), the resulting Adam update can have a non-zero vocabulary mean and induce drift in .
Larger learning rates and longer training horizons amplify logit growth. Because Adam’s preconditioned updates need not have zero mean across each word embedding along the vocabulary axis, larger learning rates can amplify their contribution to the mean output embedding at each optimization step. Over longer training horizons, these updates have more opportunities to accumulate. This provides a possible explanation for the greater output logit growth observed at higher learning rates (Figure 16) and over longer training horizons (Figure 17).
Figure 17. Longer training horizons are associated with increasingly severe late stage LM head growth. Evaluation loss (left), output logit norm (middle), and output logit mean (right) for 28M parameter models with parameterized RMSNorm, and peak LR of 0.1 trained with horizons of 4B, 8B, 12B, and 16B tokens. Extending training exposes progressively larger output logit norm growth and late stage loss deterioration, consistent with time-bomb instability becoming more prominent during prolonged training.
Mitigating Drift in the LM Head
The zero-sum structure motivates a simple optimizer level intervention: train the LM head with SGD with momentum while retaining AdamW for the remainder of the model. Unlike Adam, SGD with momentum (SGD-M) does not apply coordinate-wise adaptive rescaling and therefore preserves the zero-sum structure of the LM head gradient. We compare this approach against AdamW, -centering, and -loss.
Comparing mitigation methods. For LM head SGD-M, we tune the head learning rate separately and use increased momentum of 0.99. To ensure a fair comparison, we also allocate the same hyperparameter tuning budget to the AdamW based methods, as discussed below and shown in Figure 5.
Figure 5. Increasing LM head is more effective than tuning its learning rate alone. Left: Final evaluation loss across trunk learning rates. Head learning rate tuning yields modest gains, while closes much of the gap to LM head SGD-M. Adding -centering or -loss improves some settings, but all AdamW variants diverge or time-bomb at the largest trunk learning rate, whereas SGD-M remains stable. Dashed lines denote tuned head learning rates. Right: output logit norm trajectories, showing how each method constrains LM head growth.
We first compare the methods without head-specific retuning of the AdamW variants in Figure 4, which shows results across the 404M-parameter learning rate sweep. All three methods improve stability relative to the AdamW baseline and substantially reduce output logit growth. In particular, LM head SGD-M strongly constrains logit norm growth while remaining competitive in final evaluation loss across the sweep. In Appendix Section A.6, we also compare these mitigations in the parameter-less norm, fast spikes regime from § 2.4 and find that they are similarly effective against this class of instability.
Figure 4. SGD-M is competitive in both optimization quality and control of LM head growth. Across trunk learning rates, -centering, -loss, and LM head SGD-M improve stability relative to the AdamW baseline, while differing in both final evaluation loss (left) and the degree of LM head growth suppression (right). Across high learning rates, -centering and LM head SGD-M strongly suppress output logit norm growth, whereas the baseline develops the largest late growth and corresponding time-bomb spikes (right). Faint curves show individual learning rates; solid curves and shaded regions show the median and interquartile range across learning rates.
LM head specific tuning. For LM head SGD-M, the highest head learning rate in our grid achieves the best performance across all trunk learning rates and extending the search to larger values yields no meaningful further improvement. In order to measure whether head specific hyperparameter tuning gives SGD-M a disproportionate advantage, in Section B.3 and Figure 5 we explore the results of tuning the head LR and head specific value for AdamW. Tuning the AdamW head-specific learning rate alone provides only modest gains, while increasing the first moment coefficient for the LM head to closes much of the optimization gap between AdamW and SGD-M. This is consistent with recent work showing that longer training horizons can require retuning optimizer hyperparameters across the overtraining axis (Everett & Qiu, [12]). We also explore combining the higher- AdamW head with -centering and -loss. However, despite their improved performance over the tuned AdamW at some learning rate settings, those combinations both also diverge at the largest trunk learning rate we test.
SGD-M on the LM head is an effective stabilizer. Replacing AdamW with SGD with momentum exclusively on the LM head suppresses logit growth and improves long-horizon training stability. Increasing AdamW’s first moment coefficient recovers much of the optimization gap, but does not consistently prevent time-bomb instability.
Finite Precision's Role in Loss Spikes
Mean preservation alone cannot explain training stabilization. Although the analysis and experiments above connect Adam’s preconditioner to drift in the vocabulary-mean output embedding, controlling this component alone cannot affect the cross entropy computation in exact arithmetic.
To see why, let . For a fixed hidden state, this transformation shifts every logit by the same scalar and therefore leaves the softmax distribution unchanged (Stollenwerk et al., 2026). Because is unchanged, the cross entropy gradient with respect to the LM head is consequently unchanged as well. Centering also leaves the gradient propagated into the hidden state unchanged:
Therefore, in settings without weight decay, and assuming exact computation, removing the vocabulary-mean component changes neither the model predictions nor the cross entropy gradients through the LM head. Thus, in theory, growth in this mean component alone should not affect the model’s predictions or optimization trajectory, nor should the centering performed by interventions like -centering meaningfully stabilize training.
However, in finite precision, a large logit mean implies an increase in the absolute numerical scale of the individual logit values, potentially reducing the precision with which the differences between logits that determine the predictive distribution can be represented. In our final set of experiments, we explore how the effect of limited precision in the LM head can help explain both the sensitivity to logit mean drift that real language models exhibit as well as the apparent effectiveness of mitigations like -centering and -loss by accounting for changes in arithmetic representation error.
Reduced precision in the LM head accelerates time-bombs. For these experiments, we overtrain 28M parameter models for 16B tokens using the same learning rate range as before. These models are thus training for approximately 570 tokens per parameter, or roughly 28 the Chinchilla-optimal token budget. We vary LM head precision across three settings: FP32, BF16 mixed, and BF16 true, ordered from highest to lowest numerical precision. BF16 mixed, the standard setting for large-scale pretraining, stores FP32 master weights but casts to BF16 at runtime for faster matrix multiplications and lower activation memory; BF16 true additionally stores the LM head weights and optimizer states in BF16 (see Section C for details).
To directly measure numerical error in the LM head, at each logging step we recompute the LM head logits in FP32 from the same hidden states used by the native forward pass, yielding FP32 reference logits . For these reference logits, the LM head GEMM is also performed in FP32. We compare these reference logits against the native BF16 logits , measuring logit quantization error as the RMS distance, , analogous to the commonly used reconstruction error in quantization literature ([14]). We also measure the predictive-distribution quantization error as the KL divergence between the corresponding post-softmax distributions, , following Lotfi et al. (2026).
Figure 6 shows that BF16 true runs develop substantial numerical error and time-bomb first, BF16 mixed runs fail a bit later, but FP32 runs remain stable over the training horizon. In both reduced-precision settings, increasing logit quantization error is accompanied by increasing predictive-distribution quantization error and an eventual loss spike. We present a deeper analysis of the precision-error correspondence in Section C.1.
Figure 6. Reduced precision in the LM head accelerates time-bomb instability and increases quantization error as logits grow. For 28M parameter models trained at peak LR 0.1, BF16 true runs fail earliest, BF16 mixed runs deteriorate later, and FP32 remains stable across five seeds (left). The same ordering appears in the quantization error between BF16 native and FP32 reference logits (middle) and in the KL divergence between the corresponding predictive distributions (right), indicating that loss spikes coincide with increasingly large finite precision error. Solid lines show the median across seeds and faint lines show individual runs.
Finite precision drives time-bomb spikes
Although mean output logit drift is harmless in exact arithmetic, it amplifies numerical error under finite precision. Reducing LM head precision accelerates time-bomb spikes, while increasing precision improves stability.
All mitigation methods suppress quantization error. Finally, we test the hypothesis that different mitigation methods like -loss and -centering may work at least in part because they suppress the growth in numerical errors associated with time-bomb instabilities. Figure 7 evaluates some of the methods from the previous section under BF16 mixed training. We observe that while the unmitigated baseline exhibits time-bomb loss spikes accompanied by a significant rise in quantization error in both logits and predictive distribution, in contrast, -centering, -loss, and LM head SGD-M all prevent time-bomb failure and keep both error metrics bounded across five seeds. Thus, methods that stabilize LM head growth and prevent time-bomb spikes also suppress runaway growth in finite precision error, suggesting an underlying mechanistic connection between these independently proposed approaches.
Figure 7. Methods that prevent time-bomb failure suppress quantization error. For 28M-parameter models trained in BF16 mixed at peak LR 0.1, the unmitigated baseline develops time-bomb spikes (left) together with orders-of-magnitude growth in logit quantization error (middle) and predictive distribution quantization error (right). All mitigation methods prevent the time bomb and keep both numerical-error metrics bounded across five seeds. Solid lines show medians and faint lines show individual runs. The same effect is more pronounced under BF16 true training; the unmitigated baseline fails catastrophically after significant growth in numerical error (Figure 20).
All roads lead to quantization error control
-loss, -centering, and LM head SGD-M act on the loss objective, the output embeddings, and the optimizer, respectively. Yet all three prevent time-bomb spikes and keep quantization error bounded, while error in the unmitigated baseline grows by orders of magnitude. Controlling finite precision error thus emerges as a common thread behind these otherwise distinct methods.
Practical considerations. The interventions studied differ significantly in their computational and memory costs. Computing the LM head in FP32 is the most direct fix but also the most expensive. FP32 GEMMs run at a fraction of BF16 throughput on modern accelerators, and the LM head is one of the largest matrix multiplications in the model, with the output dimension equal to the vocabulary size. Materializing FP32 logits also doubles the memory of what is often the largest activation tensor in training, a cost that grows with batch size, sequence length, and vocabulary size. By contrast, the other mitigations retain BF16 mixed precision and add little overhead. LM head SGD-M replaces AdamW only for the output embeddings and stores one fewer optimizer moment buffer, reducing memory; z-loss adds an auxiliary loss term; and -centering requires a reduction across vocabulary rows after each update. These methods therefore keep the LM head in a regime where reduced precision suffices, without paying the cost of higher precision.
Related Work
Loss spikes in language model pretraining. Loss spikes have been widely documented in large-scale language model pretraining ([8, 9] (Zhang et al. 2022; Workshop et al. 2022; Zeng et al. 2023; Team et al. 2026)), motivating a variety of empirical stabilization strategies. These include skipping problematic batches or rewinding to earlier checkpoints ([8] (Zhang et al. 2022)), modifying gradient clipping or optimizer states ((Pascanu et al., 2013; Huang et al., 2025)), and architectural interventions targeting embedding or attention-layer instabilities ([11, 10] (Zeng et al. 2023; Workshop et al. 2022; Henry et al. 2020; Team et al. 2026)). While these approaches can improve training stability, they address different potential sources of instability without necessarily identifying the mechanisms actually responsible for spikes. Notably, Chowdhery et al. [8] found that replaying the same batches from different checkpoints did not reproduce loss spikes, suggesting that problematic training examples alone cannot explain their occurrence. We extend this observation by distinguishing the immediate triggers of loss spikes from the underlying instability that makes models susceptible to them. We find that fast spikes are highly sensitive to individual tokens and their contexts, yet skipping spike-inducing batches merely postpones subsequent failures a small number of iterations. We further uncover the cause of slow, terminal “time-bomb” spikes that emerge only after prolonged training, even in architectures where fast spikes are largely suppressed. This regime is likely to become more relevant as models are trained further beyond compute-optimal budgets.
Optimization dynamics of the LM head. A growing body of work identifies the LM head as particularly sensitive to the interaction between language modeling objectives and adaptive optimization. The highly imbalanced token frequencies in language modeling induce heterogeneous gradient statistics across vocabulary rows, and Adam’s coordinate-wise adaptivity has been shown to play an important role in optimizing these parameters ((Kunstner et al., 2024; Zhao et al., 2025)). However, this same adaptivity can introduce undesirable behavior in the output embeddings. Biś et al. [4] study common shifts in Transformer embeddings, while Stollenwerk & Stollenwerk (2025) connect output embedding drift to Adam’s preconditioner, which breaks the zero-sum structure of the cross entropy gradient along the vocabulary axis. This creates an apparent tension: the adaptivity that benefits LM head optimization can also induce persistent growth in the mean output embedding. Our work builds on this observation by examining the stability consequences of this drift over long training horizons. Crucially, because shifting all output logits by a common value leaves the softmax distribution and cross entropy gradients unchanged in exact arithmetic, mean drift alone cannot explain the resulting instability. We show that finite precision computation provides a mechanism connecting this otherwise harmless growth to training failure.
Finite precision and training instability. Concurrent work has identified several ways that numerical errors in softmax gradients can destabilize training. Chen et al. [7] show that BF16 rounding in the backward pass of fused attention kernels can violate the zero-sum structure of attention softmax gradients, causing delayed training instability. In contrast, our work focuses on quantization errors in the forward computation of the LM head caused by logit drift induced by Adam’s preconditioner.
Liu et al. (2026) and Barron et al. [3] show that numerical errors in softmax cross entropy gradients underlie the Slingshot phenomenon (Thilak et al. (2022)), a pattern of abrupt, recoverable loss spikes observed primarily in grokking rather than language modeling. Barron et al. [3] evaluate on synthetic modular arithmetic tasks, while Liu et al. (2026) also examine language modeling, finding abnormal logit growth during GPT training but no Slingshot spikes. We instead study slow, unrecoverable time-bomb spikes in the LM head, which arise when Adam induced logit drift, harmless in exact arithmetic, is amplified by quantization error in the forward logit computation. Consistent with this distinction, replacing standard cross entropy with the fixed softmax of Barron et al. [3] does not prevent the underlying logit growth and quantization error, and delays but does not prevent time-bomb spikes under BF16 true (Section C.4).
Mitigating output logit divergence. Several methods have been proposed to stabilize training by constraining output logit growth, although they intervene at different points in the optimization process. The -loss introduced by Chowdhery et al. [8] penalizes large log-partition values and has become a common component of language model training recipes (Allal et al. [1]; Marin Community (2026a,b)). Similarly, Gemma 2 (Team et al. (2024)) employs logit soft-capping, applying a scaled tanh transformation to directly bound the output logits before the softmax operation.
Other approaches instead intervene on the output embeddings or their optimization dynamics. Stollenwerk et al. (2026) propose -centering, which explicitly removes the vocabulary-mean component of the output embedding after each optimization step, while Stollenwerk & Stollenwerk (2025) propose modifying Adam’s second-moment preconditioning to prevent this drift from arising. These approaches constrain different aspects of LM head growth, ranging from directly bounding logits to modifying the objective, output embeddings, or optimizer updates.
Motivated by the zero-sum structure of the LM head gradient, we revisit the use of SGD with momentum specifically on the LM head, which preserves this structure without requiring an explicit centering operation. Beyond comparing the effectiveness of the interventions we evaluate, we investigate why they improve stability despite acting through different mechanisms. Our experiments show that -loss, -centering, and LM head SGD-M all suppress the growth of quantization error in the LM head, and that higher precision LM heads do not require these interventions, suggesting that controlling finite precision error is an important shared mechanism underlying their effectiveness against time-bomb spikes.
Conclusion
We study time-bomb spikes, a deleterious form of loss divergence that emerges in standard language model architectures after prolonged pretraining. Prior work traces the underlying logit growth to Adam’s preconditioner, which breaks the zero-sum structure of the cross entropy gradient and causes the mean output embedding to drift. Yet this drift leaves both the forward and backward pass unchanged in exact arithmetic. We show that finite precision resolves this puzzle. As the logit mean grows, quantization error in the LM head grows by orders of magnitude before each failure, and the precision of the LM head computation directly controls when failures occur. Lowering precision brings time-bomb spikes earlier, while computing the LM head in FP32 keeps runs stable over the horizons we study.
This perspective also clarifies why our proposed application of SGD-M to the LM head and existing mitigations seem to work so well. Replacing AdamW with carefully tuned SGD with momentum on just the LM head prevents mean drift at its source and stabilizes training across learning rates where unmitigated training runs eventually diverge. Similarly, while -loss, -centering, and LM head SGD-M intervene at different points in the optimization process, we show that all approaches keep quantization error bounded. Rather than competing explanations of instability, these methods can be understood as different ways of keeping the LM head within a regime where finite precision suffices.
Future Work. We simulate spiking settings using higher than normal learning rates (as is common in Zhao et al. (2025); Stollenwerk et al. (2026); Su et al. (2025); Li et al. (2025)) but we acknowledge that our findings may not fully transfer to spikes that arise in more natural settings in larger models, and it remains to be seen whether LM head SGD-M similarly outperforms standard optimizers there. Other avenues for future work include building specialized optimizers for the LM head and evaluating these mitigations in even lower precision formats such as FP8.
AI Use Statement
We used generative AI tools (Gemini, ChatGPT, Claude, Codex, and ARIA) at several stages of this work. During research, they assisted with brainstorming, discovering research topics and identifying gaps, refining hypotheses, suggesting experimental parameters, providing feedback on experiments, interpreting results, and supporting qualitative and thematic data analysis. During implementation, they assisted with implementing methods and with creating and editing software code. During writing, they assisted with creating scientific figures and editing the paper to improve readability.
We did not use generative AI tools for cleaning or formatting datasets, helping develop theoretical models, or formulating mathematical claims. Providing critical ingredients for proving mathematical claims, assisting in the writing of proofs, generating synthetic datasets, and translation are not applicable to this work.
All experiments were designed and run by the authors, and all reported results come from those experiments. We reviewed all AI-assisted work. LLM-generated code was verified and tested for correctness, and LLM-retrieved papers were checked to confirm that they are relevant and correctly cited. The authors take full responsibility for the final content of this work, including any text, claims, or artifacts produced with the aid of generative AI.
Acknowledgments
We gratefully acknowledge use of the research computing resources of the Empire AI Consortium, Inc (Bloom et al., [5]), with support from Empire State Development of the State of New York, the Simons Foundation, and the Secunda Family Foundation. This work was also supported by Google’s TPU Research Cloud (TRC) program. MG and ZC received generous support from the Columbia University Stabilization Award and from the Center for AI and Responsible Financial Innovation. Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean-Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette-Choo, Danila Sinopalnikov, David Weinberger, Dimple Vijaykumar, Dominika Rogozínska, Dustin Herbison, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Klimczak-Plucinska, Harleen Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz, Josh Newlan, Ju yeong Ji, Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui, Laurent Sifre, Lena Heuermann, Leticia Lago, Lilly McNealus, Livio Baldini Soares, Logan Kilpatrick, Lucas Dixon, Luciano Martins, Machel Reid, Manvinder Singh, Mark Iverson, Martin Görner, Mat Velloso, Mateo Wirth, Matt Davidow, Matt Miller, Matthew Rahtz, Matthew Watson, Meg Risdal, Mehran Kazemi, Michael Moynihan, Ming Zhang, Minsuk Kahng, Minwoo Park, Mofi Rahman, Mohit Khatwani, Natalie Dao, Nenshad Bardoliwalla, Nesh Devanathan, Neta Dumai, Nilay Chauhan, Oscar Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengchong Jin, Petko Georgiev, Phil Culliton, Pradeep Kuppala, Ramona Comanescu, Ramona Merhej, Reena Jana, Reza Ardeshir Rokni, Rishabh Agarwal, Ryan Mullins, Samaneh Saadat, Sara Mc Carthy, Sarah Cogan, Sarah Perrin, Sébastien M. R. Arnold, Sebastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Ronstrom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles, Tom Hennigan, Tomas Kocisky, Tulsee Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wenming Ye, Woohyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Victor Cotruta, Phoebe Kirk, Anand Rao, Minh Giang, Ludovic Peran, Tris Warkentin, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, D. Sculley, Jeanine Banks, Anca Dragan, Slav Petrov, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenealy, Robert Dadashi, and Alek Andreev. Gemma 2: Improving open language models at a practical size, 2024. URL https://arxiv.org/abs/2408.00118.
Kimi Team, Yifan Bai, Yiping Bao, Y. Charles, Cheng Chen, Guanduo Chen, Haiting Chen, Huarong Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, Zhuofu Chen, Jialei Cui, Hao Ding, Mengnan Dong, Angang Du, Chenzhuang Du, Dikang Du, Yulun Du, Yu Fan, Yichen Feng, Kelin Fu, Bofei Gao, Chenxiao Gao, Hongcheng Gao, Peijia Gao, Tong Gao, Yuyao Ge, Shangyi Geng, Qizheng Gu, Xinran Gu, Longyu Guan, Haiqing Guo, Jianhang Guo, Xiaoru Hao, Tianhong He, Weiran He, Wenyang He, Yunjia He, Chao Hong, Hao Hu, Yangyang Hu, Zhenxing Hu, Weixiao Huang, Zhiqi Huang, Zihao Huang, Tao Jiang, Zhejun Jiang, Xinyi Jin, Yongsheng Kang, Guokun Lai, Cheng Li, Fang Li, Haoyang Li, Ming Li, Wentao Li, Yang Li, Yanhao Li, Yiwei Li, Zhaowei Li, Zheming Li, Hongzhan Lin, Xiaohan Lin, Zongyu Lin, Chengyin Liu, Chenyu Liu, Hongzhang Liu, Jingyuan Liu, Junqi Liu, Liang Liu, Shaowei Liu, T. Y. Liu, Tianwei Liu, Weizhou Liu, Yangyang Liu, Yibo Liu, Yiping Liu, Yue Liu, Zhengying Liu, Enzhe Lu, Haoyu Lu, Lijun Lu, Yashuo Luo, Shengling Ma, Xinyu Ma, Yingwei Ma, Shaoguang Mao, Jie Mei, Xin Men, Yibo Miao, Siyuan Pan, Yebo Peng, Ruoyu Qin, Zeyu Qin, Bowen Qu, Zeyu Shang, Lidong Shi, Shengyuan Shi, Feifan Song, Jianlin Su, Zhengyuan Su, Lin Sui, Xinjie Sun, Flood Sung, Yunpeng Tai, Heyi Tang, Jiawen Tao, Qifeng Teng, Chaoran Tian, Chensi Wang, Dinglu Wang, Feng Wang, Hailong Wang, Haiming Wang, Jianzhou Wang, Jiaxing Wang, Jinhong Wang, Shengjie Wang, Shuyi Wang, Si Wang, Xinyuan Wang, Yao Wang, Yejie Wang, Yiqin Wang, Yuxin Wang, Yuzhi Wang, Zhaoji Wang, Zhentao Wang, Zhengtao Wang, Zhexu Wang, Chu Wei, Qianqian Wei, Haoning Wu, Wenhao Wu, Xingzhe Wu, Yuxin Wu, Chenjun Xiao, Jin Xie, Xiaotong Xie, Weimin Xiong, Boyu Xu, Jinjing Xu, L. H. Xu, Lin Xu, Suting Xu, Weixin Xu, Xinran Xu, Yangchuan Xu, Ziyao Xu, Jing Xu, Jing Xu, Junjie Yan, Yuzi Yan, Hao Yang, Xiaofei Yang, Yi Yang, Ying Yang, Zhen Yang, Zhilin Yang, Zonghan Yang, Haotian Yao, Xingcheng Yao, Wenjie Ye, Zhuorui Ye, Bohong Yin, Longhui Yu, Enming Yuan, Hongbang Yuan, Mengjie Yuan, Siyu Yuan, Haobing Zhan, Dehao Zhang, Hao Zhang, Wanlu Zhang, Xiaobin Zhang, Yadong Zhang, Yangkun Zhang, Yichi Zhang, Yizhi Zhang, Yongting Zhang, Yu Zhang, Yutao Zhang, Yutong Zhang, Zheng Zhang, Haotian Zhao, Yikai Zhao, Zijia Zhao, Huabin Zheng, Shaojie Zheng, Longguang Zhong, Jianren Zhou, Xinyu Zhou, Zaida Zhou,
Additional fast spike results
Learning rate controls the type of instability
Fast spikes concentrate in outlier tokens
During a fast loss spike, we find that a small subset of tokens incur extremely large cross entropy losses (100+) and therefore are responsible for the majority of the spike’s apparent magnitude. To quantify these outliers, we compute a robust -score using a rolling window of token losses from the preceding 256 steps (after an initial warmup period at the beginning of the run). This is similar in spirit to Karpathy (2024), but uses a robust -score as we find token loss distributions to be heavy-tailed. As shown in Figure 9, masking as little as 0.2% of tokens is sufficient to eliminate major spikes. This masking, however, consistently hurts overall convergence in training loss.

Figure 9. Skipping a small fraction of high-loss tokens eliminates loss spikes but hurts convergence. A 162M model trained at the Chinchilla-optimal token budget with peak learning rate 0.15. One run is the baseline; the other skips high-loss tokens identified via a rolling window threshold.
Upon observing that skipping tokens can be effective at eliminating fast spikes, one might conclude that loss spikes stem from exposure to bad training examples and could be eliminated by filtering the data. We interrogate this hypothesis further in Section A.3 by identifying extreme outlier tokens and perturbing their contexts to gain a more mechanistic understanding of the role they play in causing a spike. While these token-level interventions are also effective at reducing the loss during a spike, the token types and context-sensitive perturbations required to achieve stability in this manner appear to be non-systematic and therefore do not admit a practical train-time strategy that one could deploy during a real training run.
Spike events are highly sensitive to context
To understand what happens at the token level during a fast spike, we rank validation-set predictions by cross entropy loss at the peak of a spike in the 162M run from Figure 11, aggregating across all batches, sequences, and positions. We refer to the highest-loss prediction as the Rank-1 event. In this event, the model receives the context “...the commencement of the 50th Commemoration of that War. The Command” and must predict the target token ery, which completes the word “Commandery”. The model instead assigns a logit of 782.05 to the unrelated token ible, incurring a loss of 610.0 nats (Table 1).
| Original context (spiking) | Swapped context (recovered) | |||
| Rank | Token | Logit | Token | Logit |
| 1 | ible | 782.05 | ▁the | 95.01 |
| 2 | ments | 669.86 | ▁in | 94.12 |
| 3 | ured | 640.67 | ▁however | 93.80 |
| 4 | ive | 603.56 | ▁until | 92.99 |
| 5 | ivism | 595.26 | ▁and | 92.96 |
| Target | ery | — | ery | — |
| Loss | 610.00 | 12.73 |
Table 1. A single-token context swap collapses the spike. Top-5 predicted tokens at the Rank-1 spike event before and after replacing the final context token Command with a comma. Before the swap, the model assigns extreme logits to unrelated subword tokens; after the swap, the top predictions are ordinary function words with logits in a normal range, and the loss falls from 610.00 to 12.73. ▁ denotes a leading space.
Figure 11. Fast loss spikes coincide with sharp swings in the output logit mean. A spiking 162M run compared with a non-spiking run using the same configuration and differing only in random seed.
This spike is remarkably fragile. Replacing only the final context token, Command, with a comma while keeping the target fixed reduces the loss from 610.0 to 12.7 nats, and the largest logit falls from 782.05 to 95.01. Figure 10 shows that the swap significantly changes the logit distribution. Before the swap, the logits have a standard deviation of 121.87, so even low-probability tokens receive extreme values. After the swap, the standard deviation shrinks to 6.87, which is typical of the model’s behavior during non-spiking steps.

Figure 10. A single-token context swap collapses the logit distribution at a spike. Full-vocabulary logits () at the Rank-1 spike event before (blue) and after (orange) replacing the final context token Command with a comma, with the target token ery held fixed. The standard deviation contracts 17×, from 121.87 to 6.87, recovering the range typical of non-spiking steps.
We observe similar recoveries when swapping tokens at other positions in the context and when using other replacement tokens, including the single space ’ ’ and empty space ” tokens. Spike events are therefore highly sensitive to context. Small perturbations to the preceding tokens are often enough to restore normal loss values, and the spike itself corresponds to the model placing nearly all of its probability mass on a single wrong token.
Skipping triggering batches only delays spikes
Output logit mean dynamics during fast spikes. Fast spikes coincide with sharp swings in the mean output logit. As shown in Figure 11, these changes occur alongside the onset of a spike and are absent in an otherwise matched non-spiking run differing only in random seed.
Removing triggering batches only delays fast spikes. We also observe that fast loss spikes frequently coincide with sharp changes in the mean output logit, as shown in Figure 11. We therefore ask whether these events can be treated in the same way as the high-loss outlier tokens by skipping the batch that induces a loss spike. Following the batch-skipping procedure of Karpathy (2024), we find that skipping high-loss batches merely defers the instability rather than resolving it. As shown in Figure 12, after reverting to an earlier optimizer state, subsequent training again produces large changes in the output logit mean and further loss spike events.
Figure 12. Skipping high-loss batches defers rather than eliminates fast spike instability. Following Karpathy (2024), we skip any batch whose loss exceeds 7 standard deviations from a rolling 1000-step window and revert the optimizer state. Reverting to either the previous or second-to-previous optimizer state does not prevent subsequent fluctuations in output logit mean characteristic of spike events.
Together with the context perturbations in Section A.3, this suggests that individual batches do not trigger loss spikes, but rather an underlying instability in the model’s parameters and optimizer states. Consequently, we hypothesize that data filtering of this kind actually targets the symptom rather than the underlying source of the instability.
Learnable RMSNorm gains across seeds
Figure 13 extends the matched comparison in Figure 2 to five seed pairs. Without learnable gains, every seed exhibits fast spikes accompanied by large swings in the output logit mean. With learnable gains, all five runs train smoothly, with much less growth/drift in the output logit norm and mean, and their trajectories nearly overlap.
Mitigations in the fast spike regime
We also evaluate the mitigations from § 3.3 in the fast spike regime, following the setup of § 2.4. For LM head SGD-M, we use momentum and apply no weight decay to the LM head. We additionally evaluate row-centering, which subtracts each token’s mean across the hidden dimension from its update (). Unlike -centering, this operation is local to each vocabulary row and does not restore the zero-sum structure across the vocabulary.
Why SGD-M requires higher momentum. On the LM head, Adam’s coordinate-wise rescaling amplifies updates to rows of rare tokens, whose second-moment estimates are small, relative to the updates SGD would produce (Stollenwerk & Stollenwerk, 2025). This adaptivity has been identified as a key reason Adam outperforms gradient descent on language modeling (Kunstner et al., 2024). Replacing AdamW with SGD-M removes it, which substantially reduces the effective step size for most rows. Increasing the momentum coefficient from 0.9 to 0.99 raises the steady-state amplification factor from to , and we find that the higher value is important in practice: outperforms at every learning rate and both model sizes in Table 2. This may explain why Stollenwerk & Stollenwerk (2025), who evaluate SGD-M on the LM head with , find it slightly worse than their proposed coupled Adam.
| Method | |||||||
| 162M | |||||||
| AdamW | 3.642 | 3.679 | 3.733 | 3.733 | |||
| -centering | 3.637 | 3.656 | 3.674 | 3.701 | |||
| -loss | 3.636 | 3.655 | 3.690 | 3.718 | |||
| row-centering | 3.617 | 3.631 | 3.633 | 3.718 | 3.697 | 3.793 | 3.863 |
| SGD-M, | 3.926 | 3.891 | 3.869 | 3.849 | 3.821 | 3.803 | 3.789 |
| SGD-M, | 3.683 | 3.674 | 3.659 | 3.649 | 3.642 | 3.666 | 3.736 |
| 404M | |||||||
| AdamW | 3.295 | 3.323 | 3.248 | ||||
| -centering | 3.172 | 3.197 | 3.221 | 3.231 | |||
| -loss | 3.161 | 3.183 | 3.213 | 3.249 | 3.289 | ||
| row-centering | 3.257 | 3.166 | 3.550 | 3.310 | 3.452 | ||
| SGD-M, | 3.335 | 3.319 | 3.304 | 3.280 | 3.268 | 3.255 | 3.244 |
| SGD-M, | 3.186 | 3.183 | 3.180 | 3.173 | 3.169 | 3.167 | 3.167 |
Table 2. Learning rate sweep across the spike-prone regime () for 162M and 404M models. Final evaluation loss ( lower is better) is reported. marks runs whose loss spiked during training and did not recover.
Learning rate sweep. We compare the mitigation methods across learning rates , spanning the regime in which fast spikes emerge. As shown in Figure 14, the methods differ substantially in both the range over which they suppress spikes and their final evaluation loss. -centering and -loss prevent fast spikes over much of the sweep but become unstable at higher learning rates, and row-centering remains stable at 162M but diverges inconsistently at 404M. LM head SGD-M with remains stable across the full range at both model sizes and frequently attains the lowest final evaluation loss at higher learning rates. Full numerical results are reported in Table 2.
Figure 14. LM head SGD-M is the only intervention stable across the spike-prone learning rate regime at both model sizes. Final evaluation loss versus peak learning rate for 162M and 404M models. -centering and -loss diverge at some at both sizes, and row-centering, while stable at 162M, diverges inconsistently at 404M and is omitted there. LM head SGD-M remains stable throughout and achieves the lowest loss at the high end.
Spike frequency across seeds. To compare stability across runs, we follow OLMo et al. (2025) and define the spike score as the percentage of training steps whose loss exceeds seven standard deviations from a trailing 1000-step window. Table 3 reports both spike score and final evaluation loss across seeds and model scales, allowing us to distinguish suppression of visible spikes from differences in final evaluation loss. The mitigation methods reduce fast spike incidence to differing degrees, with LM head SGD-M exhibiting low spike scores across the settings considered.
Table 3: Tuned SGD-M on the LM head suppresses spikes and remains competitive on evaluation loss. Final evaluation loss and training-loss spike score (defined in Section A.6) at peak learning rate for 162M and 404M models. Values are mean standard error across 5 seeds; bold marks the lowest evaluation loss for each model scale.
| Method | 162M Eval Loss | 162M Spike Score | 404M Eval Loss | 404M Spike Score |
| Baseline (AdamW) | 3.702 0.014 | 0.088 0.029 | 3.916 0.278 | 0.085 0.029 |
| -loss | 3.644 0.006 | 0.064 0.027 | 3.214 0.008 | 0.106 0.040 |
| -centering | 3.646 0.009 | 0.066 0.043 | 3.208 0.004 | 0.064 0.002 |
| Row-centering | 3.621 0.016 | 0.038 0.007 | 4.858 1.438 | 0.056 0.011 |
| SGD-M, | 3.855 0.003 | 0.038 0.004 | 3.301 0.003 | 0.060 0.002 |
| SGD-M, | 3.644 0.005 | 0.038 0.004 | 3.178 0.002 | 0.069 0.003 |
Table 3.
Scaling to 1.4B parameters. We additionally repeat the comparison at 1.4B scale. As shown in Figure 15, the methods again differ in their behavior at larger scale; in this experiment, LM head SGD-M reaches a lower training loss than AdamW, -centering, and -loss. We report the corresponding results with a Muon (Jordan et al., 2024) backbone in the same figure.
Figure 15. LM head SGD-M also stabilizes training at 1.4B scale and pairs well with Muon. Training loss curves for 1.4B models at peak learning rate (single seed). Replacing AdamW with SGD-M () on the LM head reaches a lower final training loss than AdamW, -centering, and -loss, and combining a Muon backbone with LM head SGD-M reaches the lowest loss of any configuration shown.
Additional time-bomb results
Full learning rate sweep
Figure 16 shows the complete 404M learning rate sweep previewed in Figure 3. Runs at all learning rates follow similar loss trajectories for much of training, but higher learning rates produce substantially greater growth/drift in both the output logit norm and mean, accompanied in several cases by time-bomb spikes.
Effect of training horizon
Figure 17 examines how the training horizon affects time-bomb instability in 28M-parameter models. Shorter runs end before severe late stage behavior appears, while longer runs show increasingly large output logit norms, substantial drift in the mean logit, and eventual deterioration in evaluation loss. Time-bomb instabilities can therefore remain hidden in shorter training runs and emerge only as training is extended.
LM head tuning details
Because LM head SGD-M uses a separately tuned learning rate and momentum coefficient, we additionally tune LM head specific AdamW hyperparameters. We consider tuning the LM head learning rate independently of the trunk learning rate (using the grid detailed in § 3.1), increasing the LM head first moment coefficient from the default to , and combining these changes with -centering and -loss. As shown in Figure 5, tuning the LM head learning rate alone provides only modest improvement, whereas increasing produces a substantially larger gain in final evaluation loss.
These runs were performed across three hardware configurations: NVIDIA B200, NVIDIA H100, and NVIDIA A100 GPUs.
Precision analysis
Precision settings. In BF16 mixed precision, parameters and optimizer states are stored in FP32, while weights and activations are cast to BF16 at runtime so that matrix multiplications run on BF16 hardware units. In Flax, this corresponds to param_dtype=float32 with dtype=bfloat16, analogous to torch.autocast in PyTorch. BF16 true stores the LM head weights and optimizer states in BF16, removing the FP32 master copies. In the FP32 setting, both the LM head weights and the LM head GEMM are kept in full FP32 precision. All three settings use the default GEMM routine for their respective precision, and all non-LM-head layers use BF16 mixed precision throughout.
Why error grows with the logit mean
In § 3.4, we showed that quantization error in the LM head grows alongside the output logits before time-bomb spikes. Here we sketch why this happens, based on the fixed relative precision of BF16.
BF16 has seven stored fraction bits and an implicit leading bit, giving eight bits of significand precision (Kalamkar et al., 2019). For normal values , the spacing between consecutive representable values is
This spacing grows with magnitude; between 1 and 2, adjacent BF16 values are approximately 0.008 apart, whereas between 1024 and 2048, they are 8 apart.
A large logit mean can increase the magnitude of the logits without changing the differences that determine softmax probabilities. Because BF16 spacing grows with magnitude, these differences can become too small to represent accurately.
As a concrete example, the logits yield probabilities of approximately . Shifting both by leaves the exact probabilities unchanged, but because adjacent BF16 values near 1024 are 8 apart, both and round to , collapsing the distribution to . Numerically stable softmax implementations, which subtract the maximum logit, cannot recover this difference once it has been lost in the computation that produced the logits.
Because rounding error scales with the magnitude of each logit, while the softmax depends only on the differences between logits, the relevant quantity is the logit mean relative to the spread of the logits,
When , typical logit magnitudes are comparable to , so the BF16 rounding resolution is small relative to the spread of the logits. When , the logit magnitudes are instead dominated by , and numerical errors can become much larger relative to the differences that determine the softmax distribution.
As shown in Figure 18, the normalized numerical error
Figure 18. Normalized logit error grows as the logit mean becomes large relative to variation across logits. Across BF16 true and BF16 mixed baseline runs, the normalized RMS error between native BF16 logits and FP32 reference logits, , increases with the relative logit mean, . Colors indicate precision, markers indicate peak learning rate, and lines trace individual training trajectories.
increases as grows across both BF16 precision settings and the learning rates tested. This trend is consistent with the loss of numerical resolution predicted by BF16’s fixed relative precision.
Lower precision triggers time bombs at smaller learning rates
Figure 19 shows the fraction of seeds that develop time-bomb spikes across the full learning rate sweep for the 28M precision experiments in § 3.4. Reducing precision causes time-bomb spikes to appear at smaller learning rates, and FP32 remains stable even at the largest learning rate, where every BF16 run fails.

Figure 19. Lower precision triggers time-bomb spikes at smaller learning rates. Fraction of seeds exhibiting a time-bomb instability across peak learning rates and numerical precision settings. As precision decreases from FP32 to BF16 mixed to BF16 true, instability emerges at progressively smaller learning rates. A time-bomb is defined as three consecutive post-midpoint evaluations at least 0.01 above the running-best evaluation loss.
Mitigations under BF16 true
Figure 20 repeats the mitigation comparison of Figure 7 under BF16 true, where the LM head weights and optimizer states are also stored in BF16. The same pattern holds and is more pronounced: the unmitigated baseline fails after quantization error grows by several orders of magnitude, while -centering, -loss, and LM head SGD-M all keep the error bounded and prevent the time bomb. In this setting, however, LM head SGD-M converges to a noticeably higher loss than the other mitigations, even though its quantization error remains low. The gap appears early in training rather than as a late failure, suggesting slower optimization rather than instability.
Figure 20. Methods that prevent time-bomb spikes also suppress quantization error. For 28M-parameter models trained in BF16 true at peak learning rate 0.1, the unmitigated baseline develops severe unrecoverable loss spikes (left) together with orders-of-magnitude growth in logit quantization error (middle) and predictive distribution quantization error (right). -centering, -loss, and LM head SGD-M prevent the time bomb and keep both numerical-error metrics bounded across five seeds. Solid lines show medians and faint lines show individual runs.
Comparison with the fixed softmax of Barron et al. [3]
Concurrent work shows that standard softmax cross entropy implementations can round the target logit’s gradient to zero at high confidence, breaking the zero-sum structure of the cross entropy gradient with respect to the logits ([3] (Liu et al. 2026)). To test whether this Softmax Collapse contributes to time-bomb spikes, we replace standard cross entropy with the fixed implementation of Barron et al. [3], which preserves the zero-sum structure at large logit margins. Importantly, this correction addresses numerical errors in the cross entropy computation and its gradients, but does not directly eliminate the mean drift induced by Adam’s preconditioner or recover logit differences lost during the BF16 forward computation of the LM head.
We evaluate the corrected cross entropy against standard cross entropy and -loss in the 28M-parameter time-bomb setting, using a peak learning rate of 0.1 and five seeds per method. We consider both BF16 mixed and BF16 true precision, as defined in Section C. Results are shown in Figure 21 and Figure 22. Under BF16 true, the fixed softmax delays failure, but all five seeds still develop time-bomb spikes. Under BF16 mixed, none of the fixed softmax runs develops result in a time bomb spike over the 16B token training horizon. Nevertheless, output logit norms and quantization error continue to grow and remain substantially higher than runs with -loss.

Figure 21. Under BF16 mixed, the fixed softmax avoids divergence within the training horizon, but logit growth and quantization error persist. For 28M-parameter models trained at peak learning rate 0.1, we compare standard cross entropy, the fixed softmax of Barron et al. [3], and -loss. The fixed softmax tracks the baseline for most of training, and its output logit norm (middle) and quantization error (right) remain significantly higher than -loss runs. Solid lines show medians across five seeds and faint lines show individual runs.
Figure 22. Under BF16 true, the fixed softmax delays but does not prevent time-bomb spikes. Same setup as Figure 21, with the LM head weights and optimizer states stored in BF16. The output logit norm and quantization error with the fixed softmax closely track the baseline until late in training, with five seeds still diverging, while -loss keeps both bounded. Solid lines show medians across five seeds and faint lines show individual runs.
Together, these results show that correcting cross entropy can substantially improve training stability without eliminating LM head logit growth or its associated quantization error. However, the correction alone is insufficient to prevent time-bomb instability under BF16 true precision.
References
- [1]Loubna Ben Allal, Lewis Tunstall, Nouamane Tazi, Elie Bakouch, Ed Beeching, Carlos Miguel Patiño, Clémentine Fourrier, Thibaud Frere, Anton Lozhkov, Colin Raffel, Leandro von Werra, and Thomas Wolf. The smol training playbook: The secrets to building world-class llms, 2025.
- [2]Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. The falcon series of open language models, 2023. URL https://arxiv.org/abs/2311.16867.
- [3]Jonathan T. Barron, Dan Gnapragasam, and Michael Tschannen. A fix for softmax and cross-entropy, 2026. URL https://arxiv.org/abs/2610.31642.
- [4]Daniel Biš, Maksim Podkorytov, and Xiuwen Liu. Too much in common: Shifting of embeddings in transformer language models and its implications. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemover, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5117–5130, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.naacl-main.403. URL https://aclanthology.org/2021.naacl-main.403/.
- [5]Stacie Bloom, Joshua C. Brumberg, Ian Fisk, Robert J. Harrison, Robert Hull, Melur Ramasubramanian, Krystyn Van Vliet, and Jeannette Wing. Empire AI: A new model for provisioning AI and HPC for academic research in the public good. In Practice and Experience in Advanced Research Computing (PEARC ’25), pp. 4, Columbus, OH, USA, July 2025. ACM. doi: 10.1145/3708035.3736070. URL https://doi.org/10.1145/3708035.3736070.
- [6]James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github.com/jax-ml/jax.
- [7]Junlin Chen, Daize Dong, Huanwei Di, Haolong Jia, Jiawei Wu, Haotian Xie, Mingkai Zheng, Yang Li, Leshang Chen, Huishui Wang, Eric P. Xing, and Hongyi Wang. Broken symmetry in bf16 attention: Why flashattention gradients blow up late in training, 2026. URL https://arxiv.org/abs/2609.34272.
- [8]Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sashank Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. PaLM: scaling language modeling with pathways. J. Mach. Learn. Res., 24(1), January 2023. ISSN 1532-4435.arxiv.org/abs/2204.02311
- [9]DeepSeek-AI. Deepseek-v4: Towards highly efficient million-token context intelligence, 2026.arxiv.org/abs/2606.19348
- [10]Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, Rodolphe Jenatton, Lucas Beyer, Michael Tschannen, Anurag Arnab, Xiao Wang, Carlos Riquelme Ruiz, Matthias Minderer, Joan Puigcerver, Utku Evci, Manoj Kumar, Sjoerd Van Steenkiste, Gamaleldin Fathy Elsayed, Aravindh Mahendran, Fisher Yu, Avital Oliver, Fantine Huot, Jasmijn Bastings, Mark Collier, Alexey A. Gritsenko, Vighnesh Birodkar, Cristina Nader Vasconcelos, Yi Tay, Thomas Mensink, Alexander Kolesnikov, Filip Pavetic, Dustin Tran, Thomas Kipf, Mario Lucic, Xiaohua Zhai, Daniel Keysers, Jeremiah J. Harmsen, and Neil Houlsby. Scaling vision transformers to 22 billion parameters. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 7480–7512. PMLR, 23–29 Jul 2023. URL https://proceedings.mlr.press/v202/dehghani23a.html.
- [11]Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 8-bit optimizers via block-wise quantization. arXiv preprint arXiv:2110.02861, 2021.arxiv.org/abs/2110.02861
- [12]Katie Everett and Shikai Qiu. Optimizer memory schedules for outscaling the overtraining axis, 2026. URL https://arxiv.org/abs/2609.04577.
- [14]Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. OPTQ: Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=tcbPmfpxs.arxiv.org/abs/2210.17323
















