betaFeedback

WorksComputer Science

A Theory of Policy Optimization in Post-Training: Coverage Collapse and Reference Resets

Abstract

Policy optimization is central to language model post-training, yet principled guidance remains limited. We revisit its theoretical foundations to identify algorithms that learn near-optimal policies using total samples and gradient steps governed by the base model's response coverage, a measure of how readily it generates optimal responses. We call such algorithms coverage-preserving. This guarantee captures the goal of amplifying capabilities accessible through sampling from the base model. (1) Coverage collapse. REINFORCE, the basis for PPO and GRPO, can drive the probability of optimal responses to near zero, even as reward improves. This coverage collapse stalls optimization, and convergence can require exponentially many steps despite good initial coverage. The failure arises even with exact gradients and persists under KL regularization to the base model. (2) Reference resets. We show that repeatedly resetting the KL reference to the most recent policy mitigates coverage collapse. We formalize this intervention in Approximate Policy Mirror Descent (APMD), a meta-algorithm that approximately solves a sequence of KL-regularized RL subproblems. APMD enables stronger regularization, making each subproblem easier to solve, while reference resets allow convergence to global optimality. (3) Coverage-preserving algorithms. For softmax linear policies over pretrained features, we use APMD to establish the first global convergence guarantees showing that widely used families of post-training algorithms are coverage-preserving. These include stylized PPO and GRPO variants with explicit KL regularization and reference resets, and squared-loss algorithms that implement them implicitly, including those used to train the Kimi models. These results clarify how regularization, clipping, and objective choice affect the sampling and optimization costs of post-training.

Introduction

Policy optimization algorithms are the reinforcement learning (RL) methods of choice in practice, with successes spanning dexterous robotic manipulation and control ([47, 84, 66, 87]), superhuman performance in strategic games ([97, 108, 80, 82, 4]), and, most notably, frontier reasoning in language models (LMs) ([81, 79, 33, 50]). Across these applications, post-training has emerged as a successful paradigm in which policy optimization uses reward feedback to finetune an already capable base model, developed through large-scale supervised pretraining. The base model provides a foundation for learning complex tasks, while reward feedback offers a way to improve beyond the limits of available demonstrations.

Despite these successes, principled guidance for policy optimization methodology remains limited [44, 2], while the range of interventions continues to grow [96, 116, 64, 120, 75, 125]. Tuning any given method requires extensive trial and error [42, 23, 65], making it difficult to evaluate the effects of specific interventions [81, 48, 99]. Even training recipes for leading open models show little consensus on core algorithmic choices, such as which objective to optimize [33, 104, 125, 49]. Principled insights are therefore increasingly important for focusing limited resources on promising interventions and hyperparameter regimes.

Theory has, in the past, provided key design principles for policy optimization methods. TRPO [92], followed by PPO [93], directly drew from the proximal updates and policy improvement guarantees established for early theoretical algorithms [45, 46]. In post-training, however, the central goal is to learn a performant policy while making the most of the base model, a goal that existing theory has left largely unaddressed. Most guarantees either require a uniform initial policy, sidestepping the role of the base model altogether [70, 72, 55], or cannot preclude poor performance even with a high-quality base model [1, 72, 73, 109]. They therefore offer little guidance for understanding and designing policy optimization methods in post-training.

Motivated by this gap, we revisit the theory of policy optimization in post-training, and ask:

Which policy optimization algorithms effectively leverage the base model for learning?

To capture this goal, we will seek algorithms whose performance is governed by the quality of the base model. This is especially important given the substantial resources invested in developing pretrained base models with strong post-training performance [32, 20]. We formalize our question and summarize our contributions below.

Contributions

We ask which policy optimization algorithms learn near-optimal policies using total samples and gradient steps governed by the base model’s coverage over optimal responses, a guarantee that we call coverage-preserving (Section 2.2). This captures the basic post-training goal of leveraging the base model for learning and is achieved by methods such as Best-of-N sampling (Section 1.3), but has not been established in policy optimization. By designing algorithms that achieve this guarantee, we recover key interventions in modern post-training and help explain their practical efficacy.

  1. Coverage collapse with a fixed reference policy or no regularization (Section 2.1). When initialized from a base model with good (constant) coverage, REINFORCE may require exponentially many gradient steps to converge, even with KL regularization to the base model. The cause is coverage collapse, in which coverage over optimal responses deteriorates even as reward improves, causing the gradient signal to vanish (Figure 2). This failure is intrinsic to the RL objective—it occurs with exact gradients and without parameter sharing—motivating interventions that preserve coverage for efficient learning.

  1. Approximate Policy Mirror Descent (APMD) preserves coverage (Section 3.1). We show that algorithms built around a single intervention—resetting the reference policy—are coverage-preserving. We formalize this in APMD, a meta-algorithm that approximates mirror descent by iteratively setting the reference to the most recent policy, then running an inner optimization loop for a regularized RL subproblem. APMD algorithms can heavily regularize the learner to the reference, making each subproblem easy to solve. While the learner improves only locally, over successive resets it reaches global optimality (Figure 1).

  1. Applying APMD: Coverage-preserving algorithms. APMD is a template for algorithm design and analysis, and can be combined with existing methods or directly applied to analyze them. We give end-to-end convergence guarantees for softmax linear policies, a model of learning with frozen pretrained features.

(a) PPO, GRPO, and variants (Sections 3.2 and 3.3). We analyze variants of PPO and GRPO with reference resets, as in ProRL [62] and DeepSeek [96, 33]. In PPO-NoClip (Algorithm 1), which does not clip the importance weights, regularization alone enables efficient learning by controlling the policy ratios (Corollary 3.1). We then add clipping to obtain PPO-Lite (Algorithm 2). Our analysis shows that clipping complements regularization by limiting the statistical cost of large importance weights, rather than directly improving optimization (Corollary 3.2).

(b) Squared-loss algorithms (§ 4). These algorithms fit log policy ratios to rewards using squared-loss objectives. We first show that Kimi-style algorithms (Kimi Team et al. [50]), which shift the rewards by an estimated baseline function, are coverage-preserving for two common baseline choices, the mean reward in KimiPO (Corollary 4.1) and the normalization function in KimiPO-Norm (Corollary K.1). Lastly, we show that Rebel-OnPolicy, an on-policy variant of Rebel that eliminates the baseline using paired responses, is also coverage-preserving (Corollary 4.2).

Implications of our theory (§ 5). Our results show that optimization in a local neighborhood of the reference policy makes improvement tractable, whether regularization is explicitly used or implicit. This role of the reference in anchoring optimization is distinct from its use in importance weighting or batch reuse. Compared to fixed-reference (or unregularized) policy optimization, algorithms that regularize the learner to a periodically updated reference converge using substantially fewer gradient steps and even fewer rounds of batch generation.

Background: Policy Optimization in Post-Training

We study policy optimization in a contextual bandit setting motivated by RL post-training, where reward feedback is used to finetune the weights of a pretrained base model. A language model (LM) is a policy πθ:X→Δ(Y)\pi_\theta: \mathcal{X} \to\Delta(\mathcal{Y}) that maps a prompt x∈Xx \in\mathcal{X} to a distribution over complete responses, denoted yy. Given a prompt distribution μ∈Δ(X)\mu\in\Delta(\mathcal{X}) and bounded reward function r:X×Y→[0,Rmax⁡]r : \mathcal{X} \times\mathcal{Y} \to[0,R_{\max}], the goal of policy optimization is to find

arg max⁡πθ∈ΠΘ J(πθ),whereJ(π):=Ex∼μ,y∼π(⋅∣x)[r(x,y)].\underset{\pi_\theta\in\Pi_\Theta}{\operatorname{arg\,max}}\ J(\pi_\theta), \qquad\text{where}\quad J(\pi) := \mathbb{E}_{x\sim\mu,y\sim\pi(\cdot|x)}[r(x,y)].

The objective J(π)J(\pi) is the expected reward, and ΠΘ={πθ:θ∈Θ}\Pi_\Theta= \{\pi_\theta: \theta\in\Theta\} denotes the policy class induced by a fixed architecture with weights in Θ⊆Rd\Theta\subseteq\mathbb{R}^d. Reward feedback is assigned to a complete response yy, the sequence of tokens generated for prompt xx. This abstraction captures RL for one-shot reasoning.

In post-training, policy optimization starts from a pretrained base model π0=πbase=πθbase∈ΠΘ\pi_0 = \pi_{\mathrm{base}} = \pi_{\theta_{\mathrm{base}}} \in\Pi_{\Theta} and produces the policy iterates {πt=πθt}t=0T\{\pi_t = \pi_{\theta_t}\}_{t=0}^{T} over TT iterations. Objectives and update rules vary across algorithms, but we focus on first-order methods that update πt\pi_t to πt+1\pi_{t+1} using gradients of reward-based objectives. Regardless of the algorithm, we evaluate the learned policy by the suboptimality of its expected reward. For any ε∈(0,Rmax⁡]\varepsilon\in(0, R_{\max}], we say that a policy π\pi is ε\varepsilon-optimal if J⋆−J(π)≤εJ^\star- J(\pi) \le\varepsilon, where J⋆:=max⁡π:X→Δ(Y)J(π)J^\star:= \max_{\pi:\mathcal{X}\to\Delta(\mathcal{Y})} J(\pi) is the maximum reward achievable by any conditional distribution.

Example: REINFORCE. REINFORCE [111, 102] is a foundational policy optimization algorithm that performs stochastic gradient ascent on the expected reward J(πθ)J(\pi_\theta). For a fixed learning rate α>0\alpha> 0, the parameter update with exact gradients is θt+1=θt+α∇J(πt)\theta_{t+1} = \theta_t + \alpha\nabla J(\pi_t), where ∇\nabla denotes differentiation with respect to the policy parameters and ∇J(πt)\nabla J(\pi_t) is evaluated at θ=θt\theta= \theta_t,

∇J(πt)=Ex∼μ,y∼πt(⋅∣x)[∇log⁡πt(y∣x)r(x,y)].\nabla J(\pi_t) = \mathbb{E}_{x\sim\mu,y\sim\pi_t(\cdot\mid x)}[\nabla\log\pi_t(y\mid x)r(x,y)].

In practice, REINFORCE estimates ∇J(πt)\nabla J(\pi_t) using responses sampled from πt\pi_t. Modern policy gradient methods build on this update with additions such as KL regularization to the base model [81, 96], estimated advantage functions, and clipped importance weights for batch reuse [93, 96]. In § 2.1, we study REINFORCE to identify optimization challenges specific to RL. These challenges motivate from first principles the design of modern post-training methods such as PPO and GRPO (§ 3).

Background: Coverage and the Role of the Base Model

Policy optimization in post-training begins from a base model, and we measure the quality of the base model by its response coverage, which quantifies how often it generates optimal responses.

Definition 1.1 (Response coverage). For any prompt x∈Xx \in\mathcal{X}, let Yx⋆:={y∈Y:r(x,y)=max⁡y′r(x,y′)}\mathcal{Y}^{\star}_x := \{y \in\mathcal{Y}: r(x,y) = \max_{y'} r(x,y')\} denote the set of optimal responses. Then the response coverage of a policy π\pi is measured by

Cπ:=Ex∼μ[1π(Yx⋆∣x)].C_\pi:= \mathbb{E}_{x\sim\mu}\left[\frac{1}{\pi(\mathcal{Y}^{\star}_x\mid x)}\right].

For the base model, we use Cbase≡CπbaseC_{\mathrm{base}} \equiv C_{\pi_{\mathrm{base}}}.

For a policy π\pi and prompt xx, the expected number of generations needed to observe an optimal response is π(Yx⋆∣x)−1\pi(\mathcal{Y}^{\star}_x\mid x)^{-1}, and CπC_\pi averages this cost over prompts. Smaller values of CπC_\pi indicate better response coverage, meaning that fewer generations are needed to find an optimal response. A central premise of post-training is that pretraining imbues the base model with favorable coverage (a moderate CbaseC_{\mathrm{base}}). This has helped explain the success of inference-time sampling [9, 40] and supervised finetuning on filtered generations [98, 56, 39].

Coverage and Best-of-N. The base model’s response coverage governs the performance of Best-of-N sampling, a widely used inference-time method for improving response quality without training. Given a prompt xx, Best-of-N draws NN independent responses from πbase(⋅∣x)\pi_{\mathrm{base}}(\cdot\mid x) and returns the one with the highest reward r(x,y)r(x,y). It fails to return an optimal response only when none of the NN candidates is optimal, an event with probability Ex[(1−πbase(Yx⋆∣x))N]≤Cbase/N\mathbb{E}_x[(1-\pi_{\mathrm{base}}(\mathcal{Y}^{\star}_x\mid x))^N] \le C_{\mathrm{base}}/N. Thus, by using N=Θ~(Cbase/ε)N = \widetilde{\Theta}(C_{\mathrm{base}}/\varepsilon) responses per prompt, Best-of-N returns an optimal response with probability at least 1−ε1-\varepsilon [40, 16].

Coverage and policy optimization. The Best-of-N guarantee above suggests that policy optimization algorithms should learn near-optimal policies using total samples controlled by CbaseC_{\mathrm{base}}. However, unlike Best-of-N, which always samples from πbase\pi_{\mathrm{base}}, in policy optimization responses are sampled from the learner, whose coverage evolves during training. For example, with binary rewards r(x,y)∈{0,1}r(x,y) \in\{0,1\}, the estimated REINFORCE gradient ∇J(πt)=Eπt[∇log⁡πt(y∣x)r(x,y)]\nabla J(\pi_t) = \mathbb{E}_{\pi_t}[\nabla\log\pi_t(y\mid x)r(x,y)] is zero unless the batch contains a correct response. It is therefore the coverage of πt\pi_t that governs convergence in policy optimization. While the initial policy inherits the base model’s coverage, it is unclear whether this coverage, which supports the learning signal, persists during training.

Motivation: The Coverage Collapse Phenomenon

We therefore ask whether policy optimization algorithms can learn efficiently when initialized from a high-quality base model, the basic goal of post-training. We first show that REINFORCE can suffer coverage collapse, causing exponentially slow convergence despite favorable initial coverage (§ 2.1). This failure motivates our central question in Section 2.2, where we ask which policy optimization methods learn near-optimal policies with total samples and gradient steps controlled by the base model’s coverage, a guarantee we call coverage-preserving.

Lower Bound: Coverage Collapse in Naïve Policy Optimization Methods

We now investigate whether REINFORCE converges efficiently when initialized from a base model with good coverage. Throughout this section, we use exact rather than estimated gradients to isolate the effect of coverage on the dynamics of policy optimization. We also allow KL regularization to the base model, with regularization parameter η∈(0,∞]\eta\in(0,\infty], giving the objective

Jη(πθ):=J(πθ)−1ηDKL(πθ∥πbase),whereDKL(π∥πbase)=Ex∼μ, y∼π(⋅∣x)[log⁡π(y∣x)πbase(y∣x)].(1)J_\eta(\pi_\theta) := J(\pi_\theta) - \frac{1}{\eta}D_{\mathrm{KL}}(\pi_\theta\mathbin\Vert\pi_{\mathrm{base}}), \qquad\text{where}\quad D_{\mathrm{KL}}(\pi\mathbin\Vert\pi_{\mathrm{base}}) = \mathbb{E}_{x\sim\mu,\,y\sim\pi(\cdot\mid x)}\left[\log\frac{\pi(y\mid x)}{\pi_{\mathrm{base}}(y\mid x)}\right]. \tag*{(1)}

where η=∞\eta= \infty recovers the unregularized objective JJ. Starting from π0=πbase\pi_0 = \pi_{\mathrm{base}}, REINFORCE performs gradient ascent with a fixed learning rate α>0\alpha> 0,

θt+1=θt+α∇Jη(πt),where∇Jη(πt)=Eπt[∇log⁡πt(y∣x)(r(x,y)−1ηlog⁡πt(y∣x)πbase(y∣x))].(2)\theta_{t+1} = \theta_t + \alpha\nabla J_\eta(\pi_t), \qquad\text{where}\quad\nabla J_\eta(\pi_t) = \mathbb{E}_{\pi_t}\left[\nabla\log\pi_t(y\mid x)\left(r(x,y)-\frac{1}{\eta}\log\frac{\pi_t(y\mid x)}{\pi_{\mathrm{base}}(y\mid x)}\right)\right]. \tag*{(2)}

KL regularization is widely used in post-training to stabilize optimization and help preserve pre-existing behavior ([96, 52, 62]). Because it discourages deviations from the base model, one might expect πt\pi_t to stay close to πbase\pi_{\mathrm{base}} in distribution and retain the base model’s coverage, especially since improving performance involves assigning greater probability to higher-reward responses.

However, Theorem 2.1 shows that the policy’s coverage can deteriorate until it is exponentially worse than the base model’s, causing convergence to be exponentially slow. Favorable coverage in the base model therefore does not guarantee efficient convergence of REINFORCE, with or without KL regularization.

References

  1. [1]Alekh Agarwal, Sham M. Kakade, Jason D. Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22(98): 1–76, 2021. URL https://www.jmlr.org/papers/v22/19-736.html.
  2. [2]Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting REINFORCE-style optimization for learning from human feedback in LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12248–12267. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.acl-long.662. URL https://aclanthology.org/2024.acl-long.662/.DOI
  3. [3]Reza Asad, Reza Babanezhad Harikandeh, Issam H. Laradji, Nicolas Le Roux, and Sharan Vaswani. Fast convergence of softmax policy mirror ascent. In Proceedings of the 28th International Conference on Artificial Intelligence and Statistics, volume 258 of Proceedings of Machine Learning Research, pages 3943–3951, 2025. URL https://proceedings.mlr.press/v258/asad25a.html.arxiv.org/abs/2411.12042
  4. [4]Bowen Baker, Ilge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video PreTraining (VPT): Learning to act by watching unlabeled online videos. In Advances in Neural Information Processing Systems, volume 35, 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/hash/9c7008aff45b5d8f0973b23e1a22ada0-Abstract-Conference.html.arxiv.org/abs/2206.11795
  5. [5]Dorian Baudry, Emmeran Johnson, Simon Vary, Ciara Pike-Burke, and Patrick Rebeschini. Does stochastic gradient really succeed for bandits? In Advances in Neural Information Processing Systems, volume 38, 2025. doi: 10.52202/085713-3787. URL https://papers.nips.cc/paper_files/paper/2025/hash/a4e683f0ce6b91e7fbdae9d32642d88f-Abstract-Conference.html.DOI
  6. [6]Ahmad Beirami, Alekh Agarwal, Jonathan Berant, Alexander D’Amour, Jacob Eisenstein, Chirag Nagpal, and Ananda Theertha Suresh. Theoretical guarantees on the best-of-n alignment policy. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 3580–3602, 2025.arxiv.org/abs/2401.01879
  7. [7]Jalaj Bhandari and Daniel Russo. Global Optimality Guarantees for Policy Gradient Methods. Operations Research, 2024. URL https://arxiv.org/abs/1906.01786.
  8. [8]Kianté Brantley, Mingyu Chen, Zhaolin Gao, Jason Lee, Wen Sun, Wenhao Zhan, and Xuezhou Zhang. Accelerating rl for llm reasoning with optimal advantage regression. Advances in Neural Information Processing Systems, 38:151492–151531, 2025.arxiv.org/abs/2505.20686
  9. [9]Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787, 2024. doi: 10.48550/arXiv.2407.21787.DOI
  10. [10]Dake Bu, Wei Huang, Andi Han, Atsushi Nitanda, Hau-San Wong, Qingfu Zhang, and Taiji Suzuki. Provable Sample Efficiency of Curriculum Post-Training for Transformer Reasoning. In International Conference on Machine Learning, 2026. URL https://arxiv.org/abs/2511.07372. Earlier arXiv title: Provable Benefit of Curriculum in Transformer Tree-Reasoning Post-Training.
  11. [11]Sébastien Bubeck. Convex optimization: Algorithms and complexity. Foundations and Trends in Machine Learning, 8(3–4):231–357, 2015. URL https://arxiv.org/abs/1405.4980.
  12. [12]Daniele Calandriello, Zhaohan Daniel Guo, Remi Munos, Mark Rowland, Yunhao Tang, Bernardo Avila Pires, Pierre Harvey Richmond, Charline Le Lan, Michal Valko, Tianqi Liu, Rishabh Joshi, Zeyu Zheng, and Bilal Piot. Human alignment of large language models through online preference optimisation. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 5409–5435, 2024. URL https://proceedings.mlr.press/v235/calandriello24a.html.arxiv.org/abs/2403.08635
  13. [13]Semih Cayci, Niao He, and R. Srikant. Convergence of entropy-regularized natural policy gradient with linear function approximation. SIAM Journal on Optimization, 34(3):2729–2755, 2024. doi: 10.1137/22M1540156. URL https://doi.org/10.1137/22M1540156.
  14. [14]Shicong Cen, Chen Cheng, Yuxin Chen, Yuting Wei, and Yuejie Chi. Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularization. Operations Research, 2022. URL https://arxiv.org/abs/2007.06558.
  15. [15]Jonathan D. Chang, Andrew Drozdov, Shubham Toshniwal, Owen Oertell, Alexander Trott, Jacob Portes, Abhay Gupta, Pallavi Koppol, Ashutosh Baheti, Sean Kulinski, Ivan Zhou, Irene Dea, Krista Opsahl-Ong, Simon Favreau-Lessard, Sean Owen, Jose Javier Gonzalez Ortiz, Arnav Singhvi, Xabi Andrade, Cindy Wang, Kartik Sreenivasan, Sam Havens, Jialu Liu, Peyton DeNiro, Wen Sun, Michael Bendersky, and Jonathan Frankle. KARL: Knowledge agents via reinforcement learning. arXiv preprint arXiv:2603.05218, 2026. doi: 10.48550/arXiv.2603.05218. URL https://arxiv.org/abs/2603.05218.
  16. [16]Fan Chen, Audrey Huang, Noah Golowich, Sadhika Malladi, Adam Block, Jordan T. Ash, Akshay Krishnamurthy, and Dylan J. Foster. The coverage principle: How pre-training enables post-training. arXiv preprint arXiv:2510.15020, 2025. doi: 10.48550/arXiv.2510.15020. URL https://arxiv.org/abs/2510.15020.
  17. [17]Wesley Chung, Valentin Thomas, Marlos C. Machado, and Nicolas Le Roux. Beyond variance reduction: Understanding the true impact of baselines on policy optimization. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 1999–2009, 2021. URL https://proceedings.mlr.press/v139/chung21a.html.
  18. [18]Taco Cohen, David W. Zhang, Kunhao Zheng, Yunhao Tang, Remi Munos, and Gabriel Synnaeve. Soft policy optimization: Online off-policy RL for sequence models. arXiv preprint arXiv:2503.05453, 2025. doi: 10.48550/arXiv.2503.05453. URL https://arxiv.org/abs/2503.05453.
  19. [19]Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, Zhiyuan Liu, Hao Peng, Lei Bai, Wanli Ouyang, Yu Cheng, Bowen Zhou, and Ning Ding. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.22617, 2025. doi: 10.48550/arXiv.2505.22617.
  20. [20]DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, et al. DeepSeek-V3 technical report. arXiv preprint arXiv:2412.19437, 2024. doi: 10.48550/arXiv.2412.19437. URL https://arxiv.org/abs/2412.19437.
  21. [21]Wenhao Deng, Long Wei, Chenglei Yu, and Tailin Wu. Unlocking reasoning capabilities in LLMs via reinforcement learning exploration. arXiv preprint arXiv:2510.03865, 2025. doi: 10.48550/arXiv.2510.03865. URL https://arxiv.org/abs/2510.03865.
  22. [22]Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. RAFT: Reward ranked finetuning for generative foundation model alignment. Transactions on Machine Learning Research, 2023.
  23. [23]Theresa Eimer, Marius Lindauer, and Roberta Raileanu. Hyperparameters in reinforcement learning and how to tune them. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 9104–9149. PMLR, 2023. URL https://proceedings.mlr.press/v202/eimer23a.html.
  24. [24]Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep policy gradients: A case study on PPO and TRPO. arXiv preprint arXiv:2005.12729, 2020. doi: 10.48550/arXiv.2005.12729.
  25. [25]Dylan J. Foster, Zakaria Mhammedi, and Dhruv Rohatgi. Is a good foundation necessary for efficient reinforcement learning? the computational role of the base model in exploration. arXiv preprint arXiv:2503.07453, 2025.
  26. [26]Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 10835–10866, 2023.arxiv.org/abs/2210.10760
  27. [27]Zhaolin Gao, Jonathan D. Chang, Wenhao Zhan, Owen Oertell, Gokul Swamy, Kianté Brantley, Thorsten Joachims, J. Andrew Bagnell, Jason D. Lee, and Wen Sun. REBEL: Reinforcement learning via regressing relative rewards. arXiv preprint arXiv:2404.16767, 2024.
  28. [28]Zhaolin Gao, Wenhao Zhan, Jonathan D. Chang, Gokul Swamy, Kianté Brantley, Jason D. Lee, and Wen Sun. Regressing the relative future: Efficient policy optimization for multi-turn RLHF. International Conference on Learning Representations, 2025.arxiv.org/abs/2410.04612
  29. [29]Saurabh Garg, Joshua Zhanson, Emilio Parisotto, Adarsh Prasad, J. Zico Kolter, Zachary C. Lipton, Sivaram an Balakrishnan, Ruslan Salakhutdinov, and Pradeep Ravikumar. On proximal policy optimization’s heavy-tailed gradients. Proceedings of the 38th International Conference on Machine Learning, 2021. doi: 10.48550/arXiv:2102.10264.arxiv.org/abs/2102.10264
  30. [30]Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. A Theory of Regularized Markov Decision Processes. In International Conference on Machine Learning, 2019. URL https://arxiv.org/abs/1901.11275.
  31. [31]Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In Proceedings of the 27th International Conference on Artificial Intelligence and Statistics, volume 238 of Proceedings of Machine Learning Research, pages 4447–4455, 2024. URL https://proceedings.mlr.press/v238/gheshlaghi-azar24a.html.arxiv.org/abs/2310.12036
  32. [32]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. doi: 10.48550/arXiv.2407.21783. URL https://arxiv.org/abs/2407.21783.
  33. [33]Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Xiao Bi, et al. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. Nature, 645:633–638, 2025. doi: 10.1038/s41586-025-09422-z.DOI
  34. [34]Jingkai He, Tianjian Li, Erhu Feng, Dong Du, Qian Liu, Tao Liu, Yubin Xia, and Haibo Chen. History doesn’t repeat itself but rollouts rhyme: Accelerating reinforcement learning with RhymeRL. In Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (ASPLOS ’26), pages 929–945. ACM, 2026. doi: 10.1145/3779212.3790172.DOI
  35. [35]Jian Hu, Mingjie Liu, Shizhe Diao, Ximing Lu, Xin Dong, Pavlo Molchanov, Yejin Choi, Jan Kautz, and Yi Dong. ProRL V2: Prolonged training validates RL scaling laws. Online technical report, August 2025a. URL https://hijkzzz.notion.site/prorl-v2.
  36. [36]Jian Hu, Xibin Wu, Wei Shen, Jason Klein Liu, Weixun Wang, Songlin Jiang, Haoran Wang, Hao Chen, Bin Chen, Wenkai Fang, Xianyu, Yu Cao, Haotian Xu, and Yiming Liu. OpenRLHF: A ray-based easy-to-use, scalable and high-performance RLHF framework. In Ivan Habernal, Peter Schlum, and Joerg Tiedemann, editors, Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 656–666, Suzhou, China, November 2025b. Association for Computational Linguistics. doi: 10.18653/v1/2025.emnlp-demos.48. URL https://aclanthology.org/2025.emnlp-demos.48/.DOI
  37. [38]Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. Open-Reasoner-Zero: An open source approach to scaling up reinforcement learning on the base model. arXiv preprint arXiv:2503.24290, 2025c. doi: 10.48550/arXiv.2503.24290.arxiv.org/abs/2503.24290
  38. [39]Audrey Huang, Adam Block, Dylan J. Foster, Dhruv Rohatgi, Cyril Zhang, Max Simchowitz, Jordan T. Ash, and Akshay Krishnamurthy. Self-improvement in language models: The sharpening mechanism. In The Thirteenth International Conference on Learning Representations, 2025a.arxiv.org/abs/2412.01951
  39. [40]Audrey Huang, Adam Block, Qinghua Liu, Nan Jiang, Akshay Krishnamurthy, and Dylan J. Foster. Is best-of-n the best of them? coverage, scaling, and optimality in inference-time alignment. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 25075–25126, 2025b.arxiv.org/abs/2503.21878
  40. [41]Guanhua Huang, Tingqiang Xu, Mingze Wang, Qi Yi, Xue Gong, Siheng Li, Ruibin Xiong, Kejiao Li, Yuhao Jiang, and Bo Zhou. Low-probability tokens sustain exploration in reinforcement learning with verifiable reward. arXiv preprint arXiv:2510.03222, 2025c. doi: 10.48550/arXiv.2510.03222. URL https://arxiv.org/abs/2510.03222.
  41. [42]Shengyi Huang, Michael Noukhovitch, Arian Hosseini, Kashif Rasul, Weixun Wang, and Lewis Tunstall. The N+ implementation details of RLHF with PPO: A case study on TL;DR summarization. arXiv preprint arXiv:2403.17031, 2024. doi: 10.48550/arXiv.2403.17031. URL https://arxiv.org/abs/2403.17031.
  42. [43]Yu Huang, Zixin Wen, Yuejie Chi, Yuting Wei, Aarti Singh, Yingbin Liang, and Yuxin Chen. On the Emergence of Implicit Curriculum in RLVR Learning Dynamics. arXiv preprint arXiv:2602.14872, 2026. URL https://arxiv.org/abs/2602.14872.
  43. [44]Andrew Ilyas, Logan Engstrom, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. A closer look at deep policy gradients. In International Conference on Learning Representations, 2020. URL https://arxiv.org/abs/1811.02553.
  44. [45]Sham M. Kakade. A natural policy gradient. In Advances in Neural Information Processing Systems 14, pages 1531–1538, 2001.
  45. [46]Sham M. Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning, pages 267–274, 2002.
  46. [47]Elia Kaufmann, Leonard Bauersfeld, Antonio Loquercio, Matthias Müller, Vladlen Koltun, and Davide Scaramuzza. Champion-level drone racing using deep reinforcement learning. Nature, 620(7976):982–987, 2023. doi: 10.1038/s41586-023-06419-4. URL https://www.nature.com/articles/s41586-023-06419-4.DOI
  47. [48]Devvrit Khatri, Lovish Madaan, Rishabh Tiwari, Rachit Bansal, Sai Surya Duvvuri, Manzil Zaheer, Inderjit S. Dhillon, David Brandfonbrener, and Rishabh Agarwal. The art of scaling reinforcement learning compute for LLMs. arXiv preprint arXiv:2510.13786, 2025. doi: 10.48550/arXiv.2510.13786. URL https://arxiv.org/abs/2510.13786.
  48. [49]Kimi Team, Yifan Bai, Yiping Bao, Y Charles, Cheng Chen, Guanduo Chen, Haiting Chen, Huarong Chen, Jiahao Chen, Ningxin Chen, et al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534, 2025a.
  49. [50]Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, et al. Kimi k1.5: Scaling reinforcement learning with LLMs. arXiv preprint arXiv:2501.12599, 2025b. doi: 10.48550/arXiv.2501.12599.
  50. [51]Sara Klein, Simon Weissmann, and Leif Döring. Beyond stationarity: Convergence analysis of stochastic softmax policy gradient methods. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=1VeQ6VBbev.arxiv.org/abs/2310.02671
  51. [52]Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124, 2024. doi: 10.48550/arXiv.2411.15124.DOI
  52. [53]Guanghui Lan. Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classes. Mathematical Programming, 2023. URL https://arxiv.org/abs/2102.00135.
  53. [54]Tor Lattimore. A diffusion analysis of policy gradient for stochastic bandits. arXiv preprint arXiv:2603.10219, 2026a. doi: 10.48550/arXiv.2603.10219. URL https://arxiv.org/abs/2603.10219.
  54. [55]Tor Lattimore. A Lyapunov Analysis of Softmax Policy Gradient for Stochastic Bandits. arXiv preprint arXiv:2603.26547, 2026b. URL https://arxiv.org/abs/2603.26547.
  55. [56]Nayoung Lee, Ziyang Cai, Avi Schwarzschild, Kangwook Lee, and Dimitris Papailiopoulos. Self-improving transformers overcome easy-to-hard and length generalization challenges. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 32930–32964, 2025. URL https://proceedings.mlr.press/v267/lee25d.html.arxiv.org/abs/2502.01612
  56. [57]Gen Li, Yuting Wei, Yuejie Chi, Yuantao Gu, and Yuxin Chen. Softmax policy gradient methods can take exponential time to converge. In Proceedings of Thirty Fourth Conference on Learning Theory, volume 134 of Proceedings of Machine Learning Research, pages 3107–3110, 2021. URL https://proceedings.mlr.press/v134/li21b.html.arxiv.org/abs/2102.11270
  57. [58]Yan Li and Guanghui Lan. Policy Mirror Descent Inherently Explores Action Space. SIAM Journal on Optimization, 2025. URL https://arxiv.org/abs/2303.04386.
  58. [59]Max Qiushi Lin, Jincheng Mei, Matin Aghaee, Michael Lu, Bo Dai, Alekh Agarwal, Dale Schuurmans, Csaba Szepesvari, and Sharan Vaswani. Rethinking the global convergence of softmax policy gradient with linear function approximation. arXiv preprint arXiv:2505.03155, 2025. doi: 10.48550/arXiv.2505.03155.
  59. [60]Boyi Liu, Qi Cai, Zhuoran Yang, and Zhaoran Wang. Neural Proximal/Trust Region Policy Optimization Attains Globally Optimal Policy. In Advances in Neural Information Processing Systems, 2019. URL https://arxiv.org/abs/1906.10306.
  60. [61]Chenruo Liu, Yijun Dong, Yiqun Shen, and Qi Lei. A task-centric theory for iterative self-improvement with easy-to-hard curricula. arXiv preprint arXiv:2602.10014, 2026. URL https://arxiv.org/abs/2602.10014.
  61. [62]Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. ProRL: Prolonged reinforcement learning expands reasoning boundaries in large language models. arXiv preprint arXiv:2505.24864, 2025a. doi: 10.48550/arXiv.2505.24864.arxiv.org/abs/2505.24864
  62. [63]Mingyang Liu, Gabriele Farina, and Asuman Ozdaglar. UFT: Unifying Supervised and Reinforcement Fine-Tuning. arXiv preprint arXiv:2505.16984, 2025b. URL https://arxiv.org/abs/2505.16984.
  63. [64]Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding R1-Zero-Like training: A critical perspective. In Second Conference on Language Modeling, 2025c. URL https://openreview.net/forum?id=5PAF7PAY2Y.arxiv.org/abs/2503.20783
  64. [65]Zihe Liu, Jiashun Liu, Yancheng He, Weixun Wang, Jiaheng Liu, Ling Pan, Xinyu Hu, Shaopan Xiong, Ju Huang, Jian Hu, Shengyi Huang, Johan Obando-Ceron, Siran Yang, Jiamang Wang, Wenbo Su, and Bo Zheng. Part I: Tricks or traps? a deep dive into RL for LLM reasoning. arXiv preprint arXiv:2508.08221, 2025d. doi: 10.48550/arXiv.2508.08221. URL https://arxiv.org/abs/2508.08221.
  65. [66]Jianlan Luo, Charles Xu, Jeffrey Wu, and Sergey Levine. Precise and dexterous robotic manipulation via human-in-the-loop reinforcement learning. Science Robotics, 10(105):eads5033, 2025. doi: 10.1126/scirobotics.ads5033. URL https://arxiv.org/abs/2410.21845.
  66. [67]Martin Marek and Max Ryabinin. Score centering stabilizes off-policy reinforcement learning. arXiv preprint arXiv:2609.20807, 2026. doi: 10.48550/arXiv.2609.20807. URL https://arxiv.org/abs/2609.20807.
  67. [68]Simon Matrenok, Skander Moalla, and Caglar Gulcehre. Quantile reward policy optimization: Alignment with pointwise regression and exact partition functions. In Advances in Neural Information Processing Systems, volume 38, 2025. doi: 10.52202/085713-3434. URL https://proceedings.neurips.cc/paper_files/paper/2025/hash/94840c41497ead6a84f493f029eba7fa-Abstract-Conference.html.DOI
  68. [69]Jincheng Mei, Chenjun Xiao, Bo Dai, Lihong Li, Csaba Szepesvari, and Dale Schuurmans. Escaping the gravitational pull of softmax. In Advances in Neural Information Processing Systems, volume 33, 2020a. URL https://proceedings.neurips.cc/paper/2020/hash/f1cf2a082126bf02de0b30778ce73a7-Abstract.html.
  69. [70]Jincheng Mei, Chenjun Xiao, Csaba Szepesvari, and Dale Schuurmans. On the global convergence rates of softmax policy gradient methods. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 6820–6829, 2020b.arxiv.org/abs/2005.06392
  70. [71]Jincheng Mei, Bo Dai, Chenjun Xiao, Csaba Szepesvari, and Dale Schuurmans. Understanding the effect of stochasticity in policy optimization. In Advances in Neural Information Processing Systems, volume 34, 2021a. URL https://proceedings.neurips.cc/paper_files/paper/2021/hash/a12f69495f41bb3b637ba1b6238884d6-Abstract.html.arxiv.org/abs/2110.15572
  71. [72]Jincheng Mei, Yue Gao, Bo Dai, Csaba Szepesvari, and Dale Schuurmans. Leveraging non-uniformity in first-order non-convex optimization. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 7555–7564, 2021b. URL https://proceedings.mlr.press/v139/mei21a.html.arxiv.org/abs/2105.06072
  72. [73]Jincheng Mei, Zixin Zhong, Bo Dai, Alekh Agarwal, Csaba Szepesvari, and Dale Schuurmans. Stochastic gradient succeeds for bandits. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 24325–24360, 2023.arxiv.org/abs/2402.17235
  73. [74]Jincheng Mei, Bo Dai, Alekh Agarwal, Sharan Vaswani, Anant Raj, Csaba Szepesvari, and Dale Schuurmans. Small steps no more: Global convergence of stochastic gradient bandits for arbitrary learning rates. In Advances in Neural Information Processing Systems, volume 37, 2024. doi: 10.52202/079017-2371.DOI
  74. [75]MiniMax. MiniMax-M1: Scaling test-time compute efficiently with lightning attention. arXiv preprint arXiv:2506.13585, 2025. doi: 10.48550/arXiv.2506.13585. URL https://arxiv.org/abs/2506.13585.
  75. [76]Alireza Mousavi-Hosseini and Murat A. Erdogdu. Post-training with policy gradients: Optimality and the base model barrier. arXiv preprint arXiv:2603.06957, 2026.
  76. [77]Gergely Neu, Anders Jonsson, and Vicenç Gómez. A unified view of entropy-regularized Markov decision processes. arXiv preprint arXiv:1705.07798, 2017. URL https://arxiv.org/abs/1705.07798.
  77. [78]Phuc Minh Nguyen, Chinh D. La, Duy M. H. Nguyen, Nitesh V. Chawla, Binh T. Nguyen, and Khoa D. Doan. The reasoning boundary paradox: How reinforcement learning constrains language models. arXiv preprint arXiv:2510.02230, 2025. doi: 10.48550/arXiv.2510.02230.DOI
  78. [79]OpenAI. Learning to reason with LLMs. OpenAI, September 2024. URL https://openai.com/index/learning-to-reason-with-llms/.
  79. [80]OpenAI, Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680, 2019. URL https://arxiv.org/abs/1912.06680.
  80. [81]Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022. doi: 10.48550/arXiv.2203.02155.
  81. [82]Julien Perolat, Bart De Vylder, Daniel Hennes, Eugene Tarassov, Florian Strub, et al. Mastering the game of Stratego with model-free multiagent reinforcement learning. Science, 378(6623):990–996, 2022. doi: 10.1126/science.add4679. URL https://arxiv.org/abs/2206.15378.
  82. [83]Ruoyu Qin, Weiran He, Weixiao Huang, Yangkun Zhang, Yikai Zhao, Bo Pang, Xinran Xu, Yingdi Shan, Yongwei Wu, and Mingxing Zhang. Seer: Online context learning for fast synchronous LLM reinforcement learning. In 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI 26), pages 883–901. USENIX Association, 2026. URL https://www.usenix.org/conference/osdi26/presentation/qin.arxiv.org/abs/2511.14617
  83. [84]Ilija Radosavovic, Tete Xiao, Bike Zhang, Trevor Darrell, Jitendra Malik, and Koushil Sreenath. Real-world humanoid locomotion with reinforcement learning. Science Robotics, 9(89):eadi9579, 2024. doi: 10.1126/scirobotics.adi9579. URL https://arxiv.org/abs/2303.03381.
  84. [85]Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290, 2023. doi: 10.48550/arXiv.2305.18290.DOI
  85. [86]Noam Razin, Hattie Zhou, Omid Saremi, Vimal Thilak, Arwen Bradley, Preetum Nakkiran, Joshua Susskind, and Etai Littwin. Vanishing gradients in reinforcement finetuning of language models. In International Conference on Learning Representations, 2024. URL https://proceedings.iclr.cc/paper_files/paper/2024/hash/2b09bb02b90584e2be94ff3ae09289bc-Abstract-Conference.html.arxiv.org/abs/2310.20703
  86. [87]Allen Z. Ren, Justin Lidard, Lars L. Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majumdar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffusion policy policy optimization. In International Conference on Learning Representations, 2025. URL https://proceedings.iclr.cc/paper_files/paper/2025/hash/c0749c39aaff9e9e4c91f718bf21b1e-Abstract-Conference.html.
  87. [88]Pierre Harvey Richemon​​d, Yunhao Tang, Daniel Guo, Daniele Calandriello, Mohammad Gheshlaghi Azar, Rafael Rafailov, Bernardo Avila Pires, Eugene Tarassov, Lucas Spangher, Will Ellsworth, Aliaksei Severyn, Jonathan Mallinson, Lior Shani, Gil Shamir, Rishabh Joshi, Tianqi Liu, Remi Munos, and Bilal Piot. Offline regularised reinforcement learning for large language models alignment. arXiv preprint arXiv:2405.19107, 2024. doi: 10.48550/arXiv.2405.19107. URL https://arxiv.org/abs/2405.19107.
  88. [89]Daniel Ritter, Owen Oertell, Bradley Guo, Jonathan Chang, Kiante Brantley, and Wen Sun. Llms can learn to reason via off-policy rl. arXiv preprint arXiv:2602.19362, 2026.arxiv.org/abs/2602.19362
  89. [90]Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Direct nash optimization: Teaching language models to self-improve with general preferences. arXiv preprint arXiv:2404.03715, 2024. doi: 10.48550/arXiv.2404.03715.DOI
  90. [91]Tom Schaul, Diana Borsa, Joseph Modayil, and Razvan Pascanu. Ray interference: A source of plateaus in deep reinforcement learning. arXiv preprint arXiv:1904.11455, 2019. doi: 10.48550/arXiv.1904.11455. URL https://arxiv.org/abs/1904.11455.
  91. [92]John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1889–1897, 2015.arxiv.org/abs/1502.05477
  92. [93]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. doi: 10.48550/arXiv.1707.06347.arxiv.org/abs/1707.06347
  93. [94]Vedant Shah, Johan Obando-Ceron, Vineet Jain, Brian Bartoldson, Bhavya Kailkhura, et al. A comedy of estimators: On KL regularization in RL training of LLMs. arXiv preprint arXiv:2512.21852, 2025. doi: 10.48550/arXiv.2512.21852.DOI
  94. [95]Lior Shani, Yonathan Efroni, and Shie Mannor. Adaptive Trust Region Policy Optimization: Global Convergence and Faster Rates for Regularized MDPs. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020. URL https://arxiv.org/abs/1909.02769.
  95. [96]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, et al. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. doi: 10.48550/arXiv.2402.03300.
  96. [97]David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, et al. Mastering the game of Go with deep neural networks and tree search. Nature, 529:484–489, 2016. doi: 10.1038/nature16961. URL https://www.nature.com/articles/nature16961.DOI
  97. [98]Avi Singh, John D. Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J. Liu, James Harrison, Jaehoon Lee, et al. Beyond human data: Scaling self-training for problem-solving with language models. Transactions on Machine Learning Research, 2024.arxiv.org/abs/2312.06585
  98. [99]Lucas Spangher, Rama Kumar Pasumarthi, Nick Masiewicki, William F. Arnold, Aditi Kaushal, Dale Johnson, Peter Grabowski, and Eugene Ie. RLHF algorithms ranked: An extensive evaluation across diverse tasks, rewards, and hyperparameters. In Saloni Potdar, Lina Rojas-Barahona, and Sebastien Montella, editors, Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 518–529, Suzhou (China), November 2025. Association for Computational Linguistics. doi: 10.18653/v1/2025.emnlp-industry.35. URL https://aclanthology.org/2025.emnlp-industry.35/.DOI
  99. [100]Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. In Advances in Neural Information Processing Systems, volume 33, 2020.arxiv.org/abs/2009.01325
  100. [101]Mingfei Sun, Vitaly Kurin, Guoqing Liu, Sam Devlin, Tao Qin, Katja Hofmann, and Shimon Whiteson. You may not need ratio clipping in PPO. arXiv preprint arXiv:2202.00079, 2022. doi: 10.48550/arXiv.2202.00079.DOI
  101. [102]Richard S. Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems, volume 12, pages 1057–1063, 1999.
  102. [103]Yunhao Tang and Rémi Munos. On a few pitfalls in KL divergence gradient estimation for RL. arXiv preprint arXiv:2506.09477, 2025. doi: 10.48550/arXiv.2506.09477.DOI
  103. [104]Team Olmo, Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groeneveld, et al. OLMo 3. arXiv preprint arXiv:2512.13961, 2025. doi: 10.48550/arXiv.2512.13961.DOI
  104. [105]Manan Tomar, Lior Shani, Yonathan Efroni, and Mohammad Ghavamzadeh. Mirror Descent Policy Optimization. In International Conference on Learning Representations, 2022. URL https://arxiv.org/abs/2005.09814.
  105. [106]Joel A. Tropp. An introduction to matrix concentration inequalities. Foundations and Trends in Machine Learning, 8(1–2):1–230, 2015. doi: 10.1561/2200000048. URL https://tropp.caltech.edu/books/Tro14-Introduction-Matrix-preprint.pdf.DOI
  106. [107]Roman Vershynin. High-Dimensional Probability: An Introduction with Applications in Data Science, volume 47 of Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2018. doi: 10.1017/9781108231599. URL https://webapps.math.uci.edu/~rvershyn/papers/HDP-book/HDP-1.pdf.
  107. [108]Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575:350–354, 2019. doi: 10.1038/s41586-019-1724-z. URL https://www.nature.com/articles/s41586-019-1724-z.
  108. [109]Lingxiao Wang, Qi Cai, Zhuoran Yang, and Zhaoran Wang. Neural policy gradient methods: Global optimality and rates of convergence. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=BJgQfkSYDS.arxiv.org/abs/1909.01150
  109. [110]Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for LLM reasoning. arXiv preprint arXiv:2506.01939, 2025. doi: 10.48550/arXiv.2506.01939.arxiv.org/abs/2506.01939
  110. [111]Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8:229–256, 1992. doi: 10.1007/BF00992696. URL https://doi.org/10.1007/BF00992696.
  111. [112]Violet Xiang, Amrith Setlur, Chase Blagden, Nick Haber, and Aviral Kumar. ExpRL: Exploratory RL for LLM mid-training. arXiv preprint arXiv:2606.17024, 2026. doi: 10.48550/arXiv.2606.17024. URL https://arxiv.org/abs/2606.17024.
  112. [113]Lin Xiao. On the Convergence Rates of Policy Gradient Methods. Journal of Machine Learning Research, 2022. URL https://arxiv.org/abs/2201.07443.
  113. [114]Wei Xiong, Jiarui Yao, Yuhui Xu, Bo Pang, Lei Wang, Doyen Sahoo, Junnan Li, Nan Jiang, Tong Zhang, Caiming Xiong, and Hanze Dong. A minimalist approach to LLM reasoning: From rejection sampling to REINFORCE. arXiv preprint arXiv:2504.11343, 2025. doi: 10.48550/arXiv.2504.11343.DOI
  114. [115]Zhenghao Xu, Qin Lu, Changlong Yu, and Tuo Zhao. Approximation of log-partition function in policy mirror descent induces implicit regularization for LLM post-training. arXiv preprint arXiv:2602.05933, 2026. doi: 10.48550/arXiv.2602.05933. URL https://arxiv.org/abs/2602.05933.
  115. [116]Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaoch en Zuo, Yu Yue, et al. DAPO: An open-source LLM reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025. doi: 10.48550/arXiv.2503.14476.DOI
  116. [117]Lifan Yuan, Weize Chen, Yuchen Zhang, Ganqu Cui, Hanbin Wang, Ziming You, Ning Ding, Zhiyuan Liu, Maosong Sun, and Hao Peng. From f(x) and g(x) to f(g(x)): LLMs learn new skills in RL by composing old ones. arXiv preprint arXiv:2509.25123, 2025. doi: 10.48550/arXiv.2509.25123.DOI
  117. [118]Rui Yuan, Simon S. Du, Robert M. Gower, Alessandro Lazaric, and Lin Xiao. Linear convergence of natural policy gradient methods with log-linear policies. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=-z9hdsyUwVQ.arxiv.org/abs/2210.01400
  118. [119]Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model? arXiv preprint arXiv:2504.13837, 2025a. doi: 10.48550/arXiv.2504.13837.DOI
  119. [120]Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, Tiantian Fan, Zhengyin Du, et al. VAPO: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:2504.05118, 2025b. doi: 10.48550/arXiv.2504.05118.DOI
  120. [121]Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. STaR: Bootstrapping reasoning with reasoning. In Advances in Neural Information Processing Systems, volume 35, 2022.arxiv.org/abs/2203.14465
  121. [122]Lunjun Zhang and Jimmy Ba. EMA policy gradient: Taming reinforcement learning for LLMs with EMA anchor and top-k KL. arXiv preprint arXiv:2602.04417, 2026. doi: 10.48550/arXiv.2602.04417. URL https://arxiv.org/abs/2602.04417.
  122. [123]Heyang Zhao, Chenlu He, Qunquan Gu, and Tong Zhang. Sharp Analysis for KL-Regularized Contextual Bandits and RLHF. In Advances in Neural Information Processing Systems, 2025a. URL https://arxiv.org/abs/2411.04625.
  123. [124]Rosie Zhao, Alexandru Meterez, Sham M. Kakade, Cengiz Pehlevan, Samy Jelassi, and Eran Malach. Echo chamber: RL post-training amplifies behaviors learned in pretraining. In Second Conference on Language Modeling, 2025b. URL https://openreview.net/forum?id=dp4KWuSDzj.arxiv.org/abs/2504.07912
  124. [125]Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507.18071, 2025. doi: 10.48550/arXiv.2507.18071.
  125. [126]Jin Peng Zhou, Kaiwen Wang, Jonathan Chang, Zhaolin Gao, Nathan Kallus, Kilian Q. Weinberger, Kianté Brantley, and Wen Sun. Q‡: Provably Optimal Distributional RL for LLM Post-Training. arXiv preprint arXiv:2502.20548, 2025. doi: 10.48550/arXiv.2502.20548.arxiv.org/abs/2502.20548

Paper details

vlnc:W_53r8mtqy
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents