WorksComputer Science
Closing the BF16-FP16 Convergence Speed Gap in RL Post-Training with Stochastic Rounding
Abstract
BF16 is widely used to pretrain large language models because it has the same dynamic range as FP32 and converges similarly to FP32 training. However, in reinforcement learning with verifiable rewards, where learning rates are typically low, we find that standard mixed-precision BF16 training converges more slowly than FP16 training at the same learning rate. We attribute this gap to weight sticking: when FP32 shadow weights are quantized to BF16 using round-to-nearest, small updates often leave the BF16 weights used for rollout generation and training unchanged. Consequently, several optimizer steps may be needed before updates to the shadow weights are reflected in the BF16 weights. Raising the BF16 learning rate reduces weight sticking and matches FP16's convergence speed, but it also increases the KL divergence from the reference model. We instead apply stochastic rounding when quantizing the FP32 shadow weights to BF16. This approach matches FP16's convergence speed across different models and datasets while reducing the KL divergence relative to BF16 training with a larger learning rate. Unlike FP16, it keeps BF16's wider dynamic range and needs no loss scaling.
Introduction
Mixed-precision BF16 training is widely used for pretraining large language models [9] (Bi et al. 2024). BF16 has the same dynamic range as FP32, and mixed-precision BF16 training converges similarly to FP32 training [6]. BF16 is also broadly supported by modern AI accelerators, which typically offer substantially higher peak throughput for BF16 matrix multiplications than for FP32. These properties make BF16 a natural choice for reinforcement learning (RL) post-training as well.
However, at the low learning rates (LRs) common in reinforcement learning with verifiable rewards (RLVR), such as [Yu et al., 2025, Liu et al., 2025], we find that BF16 needs more RL iterations than FP16 to reach the same reward or downstream task score. This gap persists even when the rollout and training backends produce bit-exact log-probabilities for the same weights and inputs, so training-inference mismatch [Qi et al., 2025] alone cannot explain it. Throughout this paper, we measure convergence speed by the number of RL iterations needed to reach a given reward or score. We attribute the slower convergence to weight sticking: when FP32 shadow weights are quantized to BF16 using round-to-nearest, small updates often leave the BF16 weights used for rollout generation and training unchanged for several optimizer steps. Formats with fewer mantissa bits, such as BF16, are more prone to sticking [4] (Miahi and Belilovsky 2026).
Two simple fixes can reduce weight sticking: switching to FP16 or increasing the BF16 LR. Because of its narrower dynamic range, FP16 typically requires loss scaling to avoid gradient underflow. In preliminary FP16 RLVR experiments on agentic tasks, we observed frequent overflows even with loss scales much smaller than those common in pretraining. Each overflow skips the corresponding optimizer step, so the rollouts used for that step are wasted. Increasing the BF16 LR can recover FP16’s convergence speed, but it also increases the policy’s drift from the reference model, which we measure by KL divergence. Such drift is commonly penalized in RL post-training [2], and it has been shown to predict how much a post-trained model forgets [14].
We instead apply stochastic rounding (SR) when quantizing the FP32 shadow weights to BF16. SR rounds each shadow weight to one of its neighboring BF16 values, with probabilities set by the shadow weight’s position between them. Even a small update to a shadow weight therefore changes the rounding probabilities, so the expected value of each BF16 weight moves with its FP32 shadow weight. BF16 with SR (BF16+SR) matches FP16’s convergence speed (Fig. 1) with less drift than BF16 training at a higher LR. It also benefits from BF16’s broad accelerator support and avoids the need for FP16 loss scaling.
Our contributions are as follows:
We show that, at typical RLVR LRs, BF16 converges more slowly than FP16 even when the rollout and training backends produce bit-exact log-probabilities for the same weights and inputs.
We identify weight sticking as a contributor to this slower convergence and show that increasing the number of mantissa bits in simulated BF16-like formats speeds up convergence.
We show that applying SR when quantizing the FP32 shadow weights to BF16 matches FP16’s convergence speed with less drift than raising the BF16 LR, and that applying SR in the forward passes alone is sufficient.
Weight Sticking and Stochastic Rounding
Standard mixed-precision training applies optimizer updates to FP32 shadow weights and quantizes weights to lower-precision formats, such as BF16 or FP16, for the forward and backward passes [11]. In RL post-training, the rollout backend also uses the low-precision weights, so this weight quantization affects the policy that produces training data.
Let denote an FP32 shadow weight, and let and be adjacent representable values in the target format such that . Their spacing, , is one unit in the last place (ULP) of the target format at . We consider quantizers that map to either or .
Round-to-nearest (RN) maps to
where depends on the tie-breaking rule.
Under RN quantization, weight sticking occurs when a nonzero update to an FP32 shadow weight leaves its quantized value unchanged, i.e., . Here, may be a single optimizer update or the net update over several consecutive steps.
When post-training starts from a BF16 checkpoint, each initial shadow weight lies on the BF16 grid, and, ignoring special cases, its quantized value stays at as long as the net update is smaller than half a ULP:
BF16 has 7 mantissa bits compared to FP16’s 10, so for values in the normal range of both formats, the BF16 ULP is the FP16 ULP, and a BF16 weight needs an larger net update to move. Equivalently, since the ULP grows with , a given update can move FP16 weights up to about larger in magnitude than the largest BF16 weights it can move.
For a stuck weight, the RN quantization error exactly cancels the optimizer’s net update to :
To remove this systematic error in expectation, SR quantizes a shadow weight according to
Consequently, , where the expectation is over the rounding randomness. SR replaces RN’s systematic cancellation of updates in stuck weights with a rounding error that has zero mean but higher variance. We hypothesize that the added variance harms RLVR convergence less than RN’s systematic error does, even when rollout generation and training use a single random realization of the quantized weights rather than their expectation. We test this hypothesis in § 3 by applying SR when quantizing the FP32 shadow weights to BF16 while leaving the rest of the training recipe unchanged.
Experiments
Experimental Setup
Models and datasets. Most experiments train DeepSeek-R1-Distill-Qwen-1.5B (DeepSeek-R1-1.5B) [3] on MATH-1460 [Qi et al., 2025], a curated set of 1,460 competition math problems with binary correctness rewards. Larger-scale experiments train DeepSeek-R1-Distill-Qwen-14B (DeepSeek-R1-14B) [3] on MATH-1460 or DAPO-Math-17k [Yu et al., 2025], and Qwen3-30B-A3B [13] on MATH-1460. The TorchInductor study in § 3.2 also trains Qwen3-1.7B [13] on MATH-1460. We measure downstream accuracy on the AIME 2024 and AIME 2025 datasets and report mean@32, the accuracy averaged over 32 sampled responses per problem.
Training configuration. Unless noted otherwise, experiments use Dr. GRPO [Liu et al., 2025] in the verl framework [15], with vLLM 0.12 [7] for rollout generation and Hugging Face Transformers [16] as the training backend. We use asymmetric clip ratios and [Yu et al., 2025], a dual-clip lower bound [Ye et al., 2020], and a constant token-budget loss normalizer [Liu et al., 2025]. We use AdamW with FP32 shadow weights and a default LR of . Each RL iteration samples 8 responses for each of 512 problems and splits the resulting 4,096 sequences into 4 mini-batches of 1,024, so each RL iteration takes 4 optimizer steps. The maximum response length is 8,192 tokens. FP16 runs use dynamic loss scaling with PyTorch’s ShardedGradScaler and a growth interval of 400 steps.
Numerical parity and hardware. For tightly controlled experiments, we add run-to-run determinism and training-inference numerical parity for selected models in verl. Determinism makes runs reproducible, and parity removes training-inference mismatch as a confounder, so numerical formats and rounding schemes can be compared fairly. We achieve parity by using batch-invariant kernels [He and Lab, 2025] and by aligning the numerics of the rollout and training backends so that they produce bit-exact log-probabilities for the same weights and inputs. Enforcing parity reduces throughput, so we reserve it for the BF16–FP16 parity comparison and the weight sticking measurements in § 3.2, which run on 64 NVIDIA H100 GPUs. All other experiments run on 64 NVIDIA B200 GPUs.
Data types. Regardless of the numerical format or rounding scheme, attention accumulation, RMS normalization, log-softmax, optimizer states, and gradient all-reduce use FP32. Matrix multiplications and their inputs (weights, activations, and gradients), attention outputs, and residual additions use BF16 or FP16.
Results and Discussion
BF16 converges more slowly than FP16 even after removing training-inference mismatch. Qi et al. [2025] reported reward collapse in BF16 RLVR and attributed it to training-inference mismatch. In our setup, the collapse occurs only when vLLM’s TorchInductor compiler optimizations are enabled; disabling them prevents it (Figure 2) and increases end-to-end step time by less than 2%. We observed this behavior with DeepSeek-R1-1.5B and Qwen3-1.7B on all vLLM releases we used during this work (0.12 to 0.17), but we have not tested later releases (the latest is 0.33 at the time of writing). We therefore disable TorchInductor optimizations in all remaining experiments. With TorchInductor disabled and parity enforced, FP16 still reaches a given reward level in fewer RL iterations than BF16 at the same LR (Figure 3). In the FP16 run, ShardedGradScaler never skipped an optimizer step.
Figure 2. Disabling TorchInductor compiler optimizations in vLLM prevents BF16 reward collapse.
Figure 3. Under training-inference numerical parity, FP16 reaches a given mean reward and AIME score (mean@32) in fewer RL iterations than BF16 (DeepSeek-R1-1.5B, MATH-1460).
BF16 has a higher weight sticking rate than FP16 at RLVR’s low LRs. Table 1 reports the percentage of weights that still equal their values in the BF16 checkpoint after each of the four optimizer steps in the first RL iteration. Across the transformer blocks of the DeepSeek-R1-1.5B model, the median weight magnitude is , and the 25th and 75th percentiles are and . At the median magnitude, half a ULP is about in BF16 and in FP16, both of which are well above the Adam update of roughly . Most weights therefore remain unchanged in both formats. However, because FP16’s half-ULP threshold is smaller, the same update changes a larger fraction of the small-magnitude weights, and FP16 changes roughly as many weights as BF16 by the end of the first RL iteration (9.21% versus 1.16%).
| Optimizer Step | BF16 | FP16 |
| 1 | 99.42% | 95.35% |
| 2 | 99.23% | 93.86% |
| 3 | 98.93% | 91.51% |
| 4 | 98.84% | 90.79% |
Table 1. Percentage of quantized weights equal to their initial checkpoint values after each optimizer step in the first RL iteration (DeepSeek-R1-1.5B, MATH-1460, ).
Equation (2) suggests two ways to reduce weight sticking: shrinking the ULP, e.g., by increasing the number of mantissa bits, and increasing the optimizer’s update size, e.g., by increasing the LR. Figure 4a tests the first with simulated formats that keep BF16’s 8 exponent bits but have mantissa bits. Each added mantissa bit speeds up convergence, with the format tracks FP16 closely throughout training. These results support limited mantissa precision as a contributor to the convergence speed gap. Figure 4b tests the second with LRs from to
Figure 4. Effect of mantissa bit-width and LR on convergence (DeepSeek-R1-1.5B, MATH-1460). (a) A simulated BF16-like format with 10 mantissa bits tracks FP16. (b) A higher LR speeds up convergence for both formats, but FP16 stays ahead at every LR.
. A higher LR speeds up convergence for both formats, but FP16 stays ahead of BF16 at every LR.
Stochastic rounding closes the convergence speed gap. BF16+SR matches the FP16 reward trajectory across the evaluated model–dataset combinations, spanning 1.5B, 14B, and 30B-A3B models and two training datasets (Figure 1). On AIME 2024 and AIME 2025, BF16 initially trails FP16 and BF16+SR but usually closes the gap later in training (Figure 5). Early in training, FP16 and BF16+SR produce shorter responses than BF16 for all models and datasets, but all three configurations eventually converge to similar response lengths (Figure 6). The only exception is the Qwen3-30B-A3B model, for which BF16+SR produces responses about 25% shorter than the other two configurations. We leave the study of this behavior to future work.
Figure 1. Applying stochastic rounding when quantizing the FP32 shadow weights to BF16 (BF16+SR) closes the BF16–FP16 convergence speed gap across different models and datasets. The FP16 and BF16+SR reward trajectories closely overlap.
Figure 5. AIME 2024 and AIME 2025 scores (mean@32). BF16 initially trails FP16 and BF16+SR on downstream task scores but usually closes the gap as training progresses.

Figure 6. Mean response length during training. After an initial phase in which FP16 and BF16+SR produce shorter responses, all three configurations converge to similar lengths, except on Qwen3-30B-A3B.
A higher BF16 LR closes the gap but increases drift. BF16 at follows a reward trajectory similar to FP16 at (Figure 4b), so tuning the LR is a viable way to close the gap.
However, a higher LR also changes the optimization dynamics, so we compare the resulting drift from the reference model using the estimator of the KL divergence. Figure 7 plots the lowest measured drift at each reward level. For mean rewards above 0.7, BF16 at drifts about 1.5× as much as FP16, whereas BF16+SR stays close to FP16. Higher LRs can therefore close the convergence speed gap, but at the cost of more drift from the reference model.
Figure 7. Drift from the reference model at matched reward (DeepSeek-R1-1.5B, MATH-1460). BF16 with a higher LR closes the reward gap but incurs larger drift, while BF16+SR at the default LR stays close to FP16.
Stochastic rounding of forward-pass weights is sufficient. We compared several SR variants on DeepSeek-R1-1.5B and MATH-1460. Drawing a fresh SR realization in every forward pass gives the same reward trajectory as drawing one realization after each optimizer step and using it for all forward passes until the next optimizer step, including rollout generation after weight synchronization. Also applying SR to the weights in the backward pass, or to the gradients, does not speed up convergence further. Applying SR only in the rollout backend, however, is not enough: the mean reward plateaus at a value well below FP16’s. We therefore recommend the simplest variant: apply SR once after each optimizer step, use the resulting BF16 weights for all forward passes in both backends until the next optimizer step, and use round-to-nearest for all other conversions.
Conclusions and Future Work
We showed that, at the low LRs typical of RLVR, BF16 converges more slowly than FP16, and that this convergence speed gap persists under training-inference numerical parity. We attribute the gap to weight sticking: with RN and only 7 mantissa bits, a BF16 weight reflects small updates to its shadow weight only after the net update crosses a rounding boundary.
We compared three remedies. Switching to FP16 works well for the models and datasets in this paper, but in preliminary RLVR experiments with Qwen3.5-397B-A17B on an agentic software engineering task, FP16 training overflowed frequently even at a loss scale of 1,024, which is much smaller than the loss scales common in pretraining. Raising the BF16 LR matches FP16’s convergence speed but increases drift from the reference model, which has been shown to predict how much a post-trained model forgets. Applying stochastic rounding when quantizing the FP32 shadow weights to BF16 closes the convergence speed gap with drift close to FP16’s.
We expect weight sticking to worsen in formats with even fewer mantissa bits than BF16. AI accelerators typically provide higher throughput for matrix multiplications in lower-precision formats, so weight sticking could limit their use for RLVR. Preliminary asynchronous RLVR experiments on Qwen3.5-397B-A17B, in which rollouts lag the trained policy by at most four policy versions, suggest that SR speeds up convergence in BF16 as well as in low-precision formats such as MXFP8. We leave a detailed study of RLVR using low-precision formats to future work.
References
- [2]DeepSeek-AI, “DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning,” CoRR, vol. abs/2501.12948, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2501.12948
- [3]DeepSeek-AI, “DeepSeek-V3.2: Pushing the frontier of open large language models,” CoRR, vol. abs/2512.02556, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2512.02556
- [4]A. Dirhoussi, Q. Gallouédec, E. Beeching, L. Tunstall, K. Rasul, and L. von Werra, “Defeating the trainer-generator precision mismatch in TRL,” 2026. [Online]. Available: https://huggingface.co/spaces/aminediroHF/trainer-generator-bf16-mismatch
- [6]D. D. Kalamkar, D. Mudigere, N. Mellempudi et al., “A study of BFLOAT16 for deep learning training,” CoRR, vol. abs/1905.12322, 2019. [Online]. Available: http://arxiv.org/abs/1905.12322
- [7]W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with PagedAttention,” in Proceedings of the 29th Symposium on Operating Systems Principles, SOSP 2023, Koblenz, Germany, October 23-26, 2023. ACM, 2023, pp. 611–626. [Online]. Available: https://doi.org/10.1145/3600006.3613165
- [9]Llama Team, “The Llama 3 herd of models,” CoRR, vol. abs/2407.21783, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2407.21783
- [11]P. Mickevicius, S. Narang, J. Alben et al., “Mixed precision training,” in 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. [Online]. Available: https://openreview.net/forum?id=r1gs9JgRZ
- [13]Qwen Team, “Qwen3 technical report,” CoRR, vol. abs/2505.09388, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2505.09388
- [14]I. Shenfeld, J. Pari, and P. Agrawal, “RL’s razor: Why online reinforcement learning forgets less,” CoRR, vol. abs/2509.04259, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2509.04259
- [15]G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu, “HybridFlow: A flexible and efficient RLHF framework,” in Proceedings of the Twentieth European Conference on Computer Systems, EuroSys 2025, Rotterdam, The Netherlands, 30 March 2025 - 3 April 2025. ACM, 2025, pp. 1279–1297. [Online]. Available: https://doi.org/10.1145/3698903.3696075
- [16]T. Wolf, L. Debut, V. Sanh et al., “Transformers: State-of-the-art natural language processing,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, EMNLP 2020 - Demos, Online, November 16-20, 2020. Association for Computational Linguistics, 2020, pp. 38–45. [Online]. Available: https://doi.org/10.18653/v1/2020.emnlp-demos.6





