betaFeedback

WorksComputer Science

Understanding R1-Zero-Like Training: A Critical Perspective

Abstract

DeepSeek-R1-Zero has shown that reinforcement learning (RL) at scale can directly enhance the reasoning capabilities of LLMs without supervised fine-tuning. In this work, we critically examine R1-Zero-like training by analyzing its two core components: base models and RL. We investigate a wide range of base models, including DeepSeek-V3-Base, to understand how pretraining characteristics influence RL performance. Our analysis reveals that DeepSeek-V3-Base already exhibit ''Aha moment'', while Qwen2.5 base models demonstrate strong reasoning capabilities even without prompt templates, suggesting potential pretraining biases. Additionally, we identify an optimization bias in Group Relative Policy Optimization (GRPO), which artificially increases response length (especially for incorrect outputs) during training. To address this, we introduce Dr. GRPO, an unbiased optimization method that improves token efficiency while maintaining reasoning performance. Leveraging these insights, we present a minimalist R1-Zero recipe that achieves 43.3% accuracy on AIME 2024 with a 7B base model, establishing a new state-of-the-art. Our code is available at https://github.com/sail-sg/understand-r1-zero.

https://github.com/sail-sg/understand-r1-zero1

Introduction

DeepSeek-R1-Zero (Guo et al., [8]) revolutionizes the pipeline of large language model (LLM) post-training by introducing the R1-Zero-like training paradigm: directly applying RL to base LLMs without relying on supervised fine-tuning (SFT) as a preliminary step. This new paradigm is appealing due to its simplicity and the demonstrated RL scaling phenomenon: the model reasoning capabilities improve along with a continual increase in model’s response length. This phenomenon is also accompanied by the “Aha moment”, at which the model learns emergent skills such as self-reflections.

In this paper, we aim to understand R1-Zero-like training by studying two essential components: base models and RL. In the first part, we investigate various attributes of base models, with the focus on the Qwen2.5 model family (Yang et al., [33]), which has been used in recent attempts to reproduce R1-Zero (Pan et al., [23]; Zeng et al., [36]; Liu et al., [21]; Hu et al., [12]), as well as DeepSeek-V3-Base (Liu et al., [19]), from which the real R1-Zero model was RL-tuned. In the second part, we identify the bias in optimization of GRPO (Shao et al., [27]), which may lead to progressively longer incorrect responses. To this end, we propose a simple modification to eliminate the bias, i.e., to get GRPO Done Right (Dr. GRPO), which leads to better token efficiency (highlighted in Figure 1).

GRPO, Dr. GRPO, and token efficiency comparison

Figure 1. Left: Dr. GRPO introduces simple yet significant modifications to address the biases in GRPO (Shao et al., [27]), by removing the length and std normalization terms. Right: Our unbiased optimizer effectively prevents the model from generating progressively longer incorrect responses, thereby enhancing token efficiency.

Our analysis on base models and RL suggests a minimalist recipe for R1-Zero-like training: we RL-tune Qwen2.5-Math-7B using the (unbiased) Dr. GRPO algorithm on MATH (Hendrycks et al., [10]) level 3-5 questions with the Qwen-Math template, and achieve state-of-the-art performance (Figure 2) with only 27 hours compute on 8×8 \times A100 GPUs. We hope our findings presented in this paper, models released, and the codebase open-sourced could benefit future research in the field. As an overview, we summarize the takeaways of this paper below:

Figure 2. Model performance comparison. Oat-Zero-7B is RL-tuned with our minimalist recipe described in § 1 (third paragraph). Please see Sec. B for more results.

Overview of takeaways

  • (§ 2.1) Template is crucial to make base models answer questions instead of completing sentences. In addition, all base models already possess math-solving capability prior to RL.

  • (§ 2.2) Intriguingly, Qwen-2.5 base models get an immediate ∼60%\sim60\% improvement by not using template, making us hypothesize that they may pretrain on concatenated question-answer texts when cooking the models.

  • (§ 2.3) Nearly all base models already exhibit the “Aha moment”, including DeepSeek-V3-Base.

  • (§ 3.1, § 3.2) Dr. GRPO effectively fixes GRPO’s bias in optimization, achieving better token efficiency.

  • (§ 3.3) Model-template mismatch can destroy reasoning capabilities before RL reconstructs it.

  • (§ 3.4) Math pretraining on Llama-3.2-3B improves its RL ceiling.

Analysis on Base Models

In this section, we scrutinize a wide range of base models, including the Qwen-2.5 family (Yang et al., [33]), Llama-3.1 (Grattafiori et al., [7]) and DeepSeek series (Liu et al., [19]; Shao et al., [27]; Guo et al., [8]), asking them 500 questions sampled from the MATH (Hendrycks et al., [10]) training set and analyzing their responses.

R1-Zero Trainability: Templates Construct Exploratory Base Policies

Since training from a base model is a fundamental setting of the R1-Zero-like paradigm, we first investigate whether widely used open-source base models, which are typically trained for sentence completion (i.e., pθ(x)p_\theta(x)), can have their question-answering capabilities effectively elicited through appropriate templates, thereby functioning as a question-answering base policy πθ(⋅∣q)\pi_\theta(\cdot|q). In addition to the R1 template (Template 1) in Guo et al. [8], we consider the Qwen-Math template (Template 2) used by Zeng et al. [36], as well as No template (Template 3):

Template 1 (R1 template). A conversation between User and Assistant. The User asks a question, and the Assistant solves it. The Assistant first thinks about the reasoning process in the mind and then provides the User with the answer. The reasoning process is enclosed within <think> </think> and answer is enclosed within <answer> </answer> tags, respectively, i.e., <think> reasoning process here </think> <answer> answer here </answer>.\nUser: {question}\nAssistant: <think>

Template 2 (Qwen-Math template). <|im_start|>system\nPlease reason step by step, and put your final answer within \boxed{}.<|im_end|>\n<|im_start|>user\n{question}\n<|im_end|>\n<|im_start|>assistant\n

Template 3 (No template). {question}

Experimental settings. We include Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, Qwen2.5-7B, Llama-3.1-8B, DeepSeek-Math-7B and DeepSeek-V3-Base-685B for experiments. For each model, we first apply No template to get the model responses, then let GPT-4o-mini to judge whether the model responses are in an answering format (regardless of quality) or in a sentence-completion pattern. We record the percentage of responses that tend to answer the question as the metric. We then apply both R1 template and Qwen-Math template to obtain model responses, and determine the most suitable template for each model based on the metric. Finally, we evaluate the pass@8 accuracy of each model with the corresponding template to assess whether the base policies can explore rewarding trajectories for RL improvement.

Results. The left plot of Figure 3 shows how well base models (with or without templates) answer the provided questions. We observe that Llama and DeepSeek models all improve the answering ability by employing the proper template (R1 template). However, Qwen2.5 models work best (with 100% answering rate) when no template is used. This intriguing property motivates further investigation which will be discussed in § 2.2. Meanwhile, the lowest answering rate with no template suggests that DeepSeek-V3-Base is a nearly pure base model. This observation motivates us to explore whether a pure base model like DeepSeek-V3-Base demonstrates the Aha moment (§ 2.3). The middle plot of Figure 3 shows the pass@8 accuracy of different base models (with template) at different sampling temperatures. This metric can serve as an indicator of base policy’s exploration ability. For example, if a base policy cannot even sample a single trajectory that leads to the correct final answer, it is impossible for RL to improve the policy because there is no reward signal. Our results demonstrate that all tested models are exploratory (thus ready for RL), with Qwen2.5 models performing the best (even surpassing DeepSeek-V3-Base). This might partially explain that most R1-Zero projects ([36, 12]) are based on Qwen2.5 models.

Figure 3. Model attributes across three aspects. Question-Answering Ability: the extent to which a pretrained language model provides a direct answer to a question rather than continuing or expanding upon it; Exploration Ability: pass@8 measures how well base models explore; Self-Reflection: counts are obtained through cross-validation between keyword-based detection and LLM-based detection, as detailed in Appendix § 8.

Qwen-2.5 Models Unlock the Best Performance When Discarding Template

We next dig into the intriguing observation (c.f. Figure 3(Left)) that all Qwen2.5 base models readily serve as chat models even without any template. We take a step further to evaluate the reasoning ability of Qwen2.5-Math models on five standard benchmarks: AIME 2024 ([17]), AMC ([17]), MATH500 ([10]), Minerva Math ([16]), and OlympiadBench ([9]). Following common practice, we use greedy decoding and limit the sampling budget to 3000 tokens.

As shown in Table 1, not using any template can drastically boost the average performance, resulting in an improvement of about 60% compared to the traditional 4-shot prompting. Since Qwen2.5-Math ([34]) uses chat model’s data (question-answer pairs) during the pretraining stage, we hypothesize that they might pretrain on the concatenated text to maximize log⁡pθ(q;o)\log p_{\theta}(q; o) directly. If our hypothesis turns out true, we shall be more careful about using Qwen2.5 models to reproduce DeepSeek-R1-Zero, since the base models are already SFT-like without templates.

Base model + TemplateAIME24AMCMATH500MinervaOlympiadBenchAvg.
Qwen2.5-Math-1.5B
(4-shot prompting)0.020.050.412.115.919.7
R1 template0.09.621.26.62.27.9
Qwen template20.032.533.012.522.824.2
No template16.743.461.815.128.433.1
Qwen2.5-Math-7B
(4-shot prompting)3.322.561.610.720.923.8
R1 template0.00.00.00.00.10.0
Qwen template16.738.650.69.916.626.5
No template0.245.869.021.334.738.2

Table 1. Qwen2.5-Math models might be pretrained on concatenated question-answer text, resulting in peak performance when no template is applied.

Aha Moment Already Appears in Base Models Including DeepSeek-V3-Base

One of the most inspiring results of DeepSeek-R1-Zero is the emergence of self-reflection behaviors, a.k.a., Aha moment, through pure RL training. A few prior studies ([21, 35]) have suggested that there may not be Aha moment in open-source R1 replications because the base models they use already exhibit self-reflection keywords. However, they have not tested DeepSeek-V3-Base, on which the real R1-Zero model was RL-tuned. We complete this missing piece by hosting DeepSeek-V3-Base-685B ourselves and investigating its responses to the 500 MATH questions with the R1 template. From the right plot of Figure 3, we can observe that DeepSeek-V3-Base also generates a decent amount of self-reflections, further validating the claims of Liu et al. [21]. We also show examples in Sec. E (Figure 13) where DeepSeek-V3-Base generates keywords such as “Aha” and “wait”.

Two example cases showing reasoning responses from DeepSeek-V3-Base

Figure 13. Cases showing that DeepSeek-V3-Base already exhibits “Aha moment” even before RL tunning. Question(a): An element is randomly chosen from among the first 15 rows ... Response. In Pascal’s Triangle, every row starts and ends with 1, ... ... This can be calculated as: awkward silence Wait, I’m overthinking. Let’s try again. The number of elements in the first nn rows of Pascal’s Triangle is (n+1)!/[2!∗(n−1)!](n+1)!/[2!*(n-1)!]. Question(b): Find the least positive integer nn such that 1/sin⁡45∘sin⁡46∘+⋯1/\sin45^\circ\sin46^\circ+\cdots Response. First, I notice that the left-hand side of the equation has the form ... Simplifying, I get ... Aha! I can use this to get sin⁡(a+2b)=sin⁡((a+b)+b)=3sin⁡(a+b)+b\sin(a+2b)=\sin((a+b)+b)=3\sin(a+b)+b ... This looks messy, but I notice that it has the form sin⁡acos⁡b+cos⁡asin⁡b\sin a\cos b+\cos a\sin b again. This gives me an idea. Maybe I can ... ...

An additional important question is whether self-reflection behaviors are associated with improved model performance after RL training. To investigate this, we host DeepSeek-R1-Zero and analyze its responses to the same questions from the MATH dataset. Although self-reflection behaviors occur more frequently in R1-Zero, we observe that these behaviors are not positively correlated with higher accuracy. Detailed analysis can be found in Sec. F.

Analysis on Reinforcement Learning

Language model generation can be formulated as a token-level Markov Decision Process (MDP) M=(S,A,r,pQ)M = (\mathcal{S}, \mathcal{A}, r, p_Q). At each generation step tt, the state st∈Ss_t \in\mathcal{S} is the concatenation of the input question and the output response generated so far: st=q;o<t=[q1,…,qM,o1,…,ot−1]s_t = \mathbf{q};\mathbf{o}_{<t} = [q_1,\ldots,q_M,o_1,\ldots,o_{t-1}]. The policy πθ(⋅∣st)\pi_\theta(\cdot|s_t) will select the next token oto_t from the vocabulary A\mathcal{A}, resulting in a deterministic transition to the next state st+1=st;[ot]s_{t+1} = s_t;[o_t]. The generation process starts from sampling an initial state s1=q∼pQs_1 = \mathbf{q} \sim p_Q from a set of questions, and stops when the autoregressive policy generates the [eos] token or exhausts the budget.

Typically, we maximize the entropy-regularized objective [25]:

J(πθ)=Eq∼pQEo∼πθ(⋅∣q)[R(q,o)−βDKL[πθ(⋅∣q)∥πref(⋅∣q)]](1)\mathcal{J}(\pi_\theta)=\mathbb{E}_{\mathbf{q}\sim p_Q}\mathbb{E}_{\mathbf{o}\sim\pi_\theta(\cdot|\mathbf{q})}\left[R(\mathbf{q},\mathbf{o})-\beta D_{\mathrm{KL}}[\pi_\theta(\cdot|\mathbf{q})\Vert\pi_{\mathrm{ref}}(\cdot|\mathbf{q})]\right] \tag*{(1)}

where R(q,o)=∑t=1∣o∣r(st,ot)R(\mathbf{q},\mathbf{o}) = \sum_{t=1}^{|\mathbf{o}|} r(s_t,o_t) is the return [31] of the trajectory q;o\mathbf{q};\mathbf{o}, and πref\pi_{\mathrm{ref}} is a reference policy. The KL regularization term is usually adopted (β>0\beta> 0) for reinforcement learning from human feedback [5], where rr is a reward model learned from data collected by πref\pi_{\mathrm{ref}}. In this case, regularization helps prevent πθ\pi_\theta from deviating too far from the distribution where the reward model is accurate [13, 30]. However, RL-tuning reasoning models typically employs rule-based verifiers as rr [15], eliminating the concerns of distributional shift. This allows us to remove the KL term, which not only saves the memory and computation required by πref\pi_{\mathrm{ref}} during training, but also potentially leads to better performance for R1-Zero-like training [12]. We will assume β=0\beta= 0 throughout this paper.

Policy optimization algorithms. To optimize πθ\pi_\theta with the above objective (Eq. (1) with β=0\beta= 0), Proximal Policy Optimization (PPO) [26] maximizes the following surrogate objective:

JPPO(πθ)=Eq∼pQ,o∼πθold(⋅∣q)∑t=1∣o∣{min⁡[πθ(ot∣q,o<t)πθold(ot∣q,o<t)A^t,clip⁡(πθ(ot∣q,o<t)πθold(ot∣q,o<t),1−ϵ,1+ϵ)A^t]}(2)\mathcal{J}_{\mathrm{PPO}}(\pi_\theta)=\mathbb{E}_{\mathbf{q}\sim p_Q,\mathbf{o}\sim\pi_{\theta_{\mathrm{old}}}(\cdot|\mathbf{q})}\sum_{t=1}^{|\mathbf{o}|}\left\{\min\left[\frac{\pi_\theta(o_t|\mathbf{q},\mathbf{o}_{<t})}{\pi_{\theta_{\mathrm{old}}}(o_t|\mathbf{q},\mathbf{o}_{<t})}\hat{A}_t,\operatorname{clip}\left(\frac{\pi_\theta(o_t|\mathbf{q},\mathbf{o}_{<t})}{\pi_{\theta_{\mathrm{old}}}(o_t|\mathbf{q},\mathbf{o}_{<t})},1-\epsilon,1+\epsilon\right)\hat{A}_t\right]\right\} \tag*{(2)}

where πθold\pi_{\theta_{\mathrm{old}}} is the policy before the update, ϵ\epsilon is the clipping hyperparameter, and A^t\hat{A}_t is an estimator of the advantage function of the tt-th token. A standard way to estimate A^t\hat{A}_t is to compute the Generalized Advantage Estimation (GAE) [24] with a learned value model VϕV_\phi. However, in the context of LLM RL-tuning, learning the value model is computationally expensive, so methods that estimate A^t\hat{A}_t without VϕV_\phi are practically preferred. For example, Shao et al. [27] proposed GRPO, which first samples a group of responses {o1,…,oG}\{o_1,\ldots,o_G\} per question and computes their returns R={R1,…,RG}\mathbf{R}=\{R_1,\ldots,R_G\}, then sets the advantage of all tokens from oio_i as A^t=Ri−mean⁡(R)std⁡(R)\hat{A}_t=\frac{R_i-\operatorname{mean}(\mathbf{R})}{\operatorname{std}(\mathbf{R})}.

GRPO Leads to Biased Optimization

In Deepseek-R1-Zero [8], a notable trend is the consistent increase in response length throughout the training process. This is frequently interpreted as an indication of the development of advanced reasoning abilities such as self-reflection. Recent studies [23, 36, 12] have replicated this phenomenon using various algorithms and implementations. However, we argue that the observed increase in response length may also be attributed to a bias inherent in the GRPO [27] objective function:

JGRPO(πθ)=Eq∼pQ,{oi}i=1G∼πθold(⋅∣q)1G∑i=1G1∣oi∣∑t=1∣oi∣{min⁡[πθ(oi,t∣q,oi,<t)πθold(oi,t∣q,oi,<t)A^i,t,clip⁡(πθ(oi,t∣q,oi,<t)πθold(oi,t∣q,oi,<t),1−ϵ,1+ϵ)A^i,t]}(3)\mathcal{J}_{\mathrm{GRPO}}(\pi_\theta)=\mathbb{E}_{\mathbf{q}\sim p_Q,\{o_i\}_{i=1}^{G}\sim\pi_{\theta_{\mathrm{old}}}(\cdot|\mathbf{q})}\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_i|}\sum_{t=1}^{|o_i|}\left\{\min\left[\frac{\pi_\theta(o_{i,t}|\mathbf{q},\mathbf{o}_{i,<t})}{\pi_{\theta_{\mathrm{old}}}(o_{i,t}|\mathbf{q},\mathbf{o}_{i,<t})}\hat{A}_{i,t},\operatorname{clip}\left(\frac{\pi_\theta(o_{i,t}|\mathbf{q},\mathbf{o}_{i,<t})}{\pi_{\theta_{\mathrm{old}}}(o_{i,t}|\mathbf{q},\mathbf{o}_{i,<t})},1-\epsilon,1+\epsilon\right)\hat{A}_{i,t}\right]\right\} \tag*{(3)}

where

A^i,t=R(q,oi)−mean⁡({R(q,o1),…,R(q,oG)})std⁡({R(q,o1),…,R(q,oG)}),\hat{A}_{i,t}=\frac{R(\mathbf{q},o_i)-\operatorname{mean}(\{R(\mathbf{q},o_1),\ldots,R(\mathbf{q},o_G)\})}{\operatorname{std}(\{R(\mathbf{q},o_1),\ldots,R(\mathbf{q},o_G)\})},

with the return R(q,oi)R(q,\mathbf{o}_i) typically only including the outcome verifiable reward in LLM reasoning (the analysis also applies to process reward cases).

Compared to the objective function in Eq. (2), GRPO introduces two biases (see also Figure 4):

Figure 4. Illustration of the biases in GRPO. Note that the effective advantage of GRPO ai,ta_{i,t} is equivalent to a reweighted version of the unbiased advantage A~i,t=R(q,oi)−mean⁡(R)\tilde{A}_{i,t}=R(q,o_i)-\operatorname{mean}(R). The terms std⁡(R)\operatorname{std}(R) and ∣oi∣|o_i| could bias the optimization by assigning different weights to different questions and responses, as denoted by the sizes of the blue circles and the lengths of the orange arrows. Upward arrows indicate positive advantages, and vice versa.

  • Response-level length bias: This arises from dividing by ∣oi∣|o_i|. For positive advantages (A^i,t>0\hat{A}_{i,t}>0, indicating a correct response), this bias results in greater gradient updates for shorter responses, leading the policy to favor brevity in correct answers. Conversely, for negative advantages (A^i,t<0\hat{A}_{i,t}<0, indicating an incorrect response), longer responses are penalized less due to their larger ∣oi∣|o_i|, causing the policy to prefer lengthier responses among incorrect ones.

  • Question-level difficulty bias: This is caused by dividing the centered outcome reward by std⁡({R(q,o1),…,R(q,oG)})\operatorname{std}(\{R(q,o_1),\ldots,R(q,o_G)\}). Questions with lower standard deviations (e.g., those that are too easy or too hard, with the outcome rewards being almost all 1 or 0) are given higher weights during policy updates. While advantage normalization is a common trick in RL ([3]), it is typically computed across an entire batch. In contrast, question-level normalization results in varying weights in the objective for different questions, leading to a difficulty bias in optimization.

Length Bias Also Exists in Open-Source PPO Implementations. We also examined several popular open-source implementations of vanilla PPO algorithms for LLM post-training. To our surprise, all of these implementations normalize the loss by response length (see Listing 1 and Table 2), which misaligns with the PPO objective as defined in Eq. (2). This formulation-implementation misalignment was present even before the publication of GRPO. We speculate that the misalignment might originate from the pretraining stage ([29]), where all tokens are packed into a fixed-length context and normalizing the loss by the context length (i.e., computing loss⁡⋅mean⁡(−1)\operatorname{loss}\cdot\operatorname{mean}(-1)) improves the numerical stability. However, in the RL-tuning stage, typical implementations ([32]) normalize the loss by the response length, which is not a constant, introducing an unintended length bias.

RepositoryCode LinkUnbiased?
trl ([32])PPO Loss×\times
OpenRLHF ([11])PPO Loss×\times
verl ([28])PPO Loss×\times
SimpleRL-Zero ([36])PPO Loss×\times
Open-Reasoner-Zero ([12])PPO Loss×\times

Table 2. Many open-sourced PPO implementations contain length bias.

Listing 1: Comparison between a typical open-source PPO loss implementation that is biased (red) and our implementation (green). MAX_TOKENS is a global constant during the entire training (unless budget curriculum is enabled), which specifies the maximum number of generation tokens. Other constants also work with differences in gradient norm.

def masked_mean(tensor, mask, dim):
-    return (tensor * mask).sum(axis=dim) / mask.sum(axis=dim)
+    return (tensor * mask).sum(axis=-1) / MAX_TOKENS

ppo_loss = ...        # compute per-token ppo loss
response_mask = ...   # per-token response mask
# per-response length normalization (e.g., OpenRLHF)
loss_variant1 = masked_mean(ppo_loss, response_mask, dim=-1).mean()
# OR per-batch length normalization (e.g., trl, verl)
loss_variant2 = masked_mean(ppo_loss, response_mask, dim=None).mean()

Dr. GRPO: Group Relative Policy Optimization Done Right

To avoid the aforementioned optimization bias in GRPO, we propose to simply remove the 1∣oi∣\frac{1}{|o_i|} and std⁡({R(q,o1),…,R(q,oG)})\operatorname{std}(\{R(q,o_1),\ldots,R(q,o_G)\}) normalization terms. Meanwhile, to faithfully implement the unbiased optimization objective, we could replace the mask.sum(axis=dim) with a constant value (e.g., generation budget) in the masked mean function in listing 1, as highlighted by the line in green. Notably, these simple modifications recover the PPO objective in Eq. (2), with the advantage estimated by Monte Carlo return with an unbiased baseline [31]. We give detailed derivations in Sec. A. We refer to our new optimization algorithm as Dr. GRPO. We next experimentally validate its effectiveness.

Experimental settings. We implement our algorithm using Oat ([20]), a modular, research-friendly and efficient LLM RL framework. We adopt the Qwen2.5-1.5B base model and the R1 template (Template 1) for online RL-tuning. We implement the verification-based reward function using Math-Verify4, with the following minimalistic rule:

R(q,o)={1if o contains the correct final answer to q0otherwiseR(\mathbf{q}, \mathbf{o}) = \begin{cases} 1 & \text{if } \mathbf{o}\text{ contains the correct final answer to }\mathbf{q} \\ 0 & \text{otherwise} \end{cases}

We run RL on questions sampled from the MATH ([10]) training dataset, and compare the vanilla GRPO with the proposed Dr. GRPO. We evaluate the online model on five benchmarks: AIME2024, AMC, MATH500, Minerva Math and OlympiadBench. More experimental details including hyperparameters can be found in Sec. G.

Results. We report various metrics in Figure 5 to demonstrate that Dr. GRPO can effectively mitigate the optimization bias and lead to better token efficiency. In particular, we first note that both GRPO and Dr. GRPO exhibit similar trend to DeepSeek-R1-Zero ([8]), namely their response length increases along with training reward (Plots 1 & 2). However, we observe that GRPO tends to continually generate longer responses even when the reward improvement slows down (Plot 2). Although such a phenomenon is often referred to as the “emergence” of long-CoT through RL ([36, 12]), we argue that it is also confounded by the response-level length bias (§ 3.1) during optimization5. In contrast, by computing the unbiased policy gradients, Dr. GRPO prevents the response length from growing wildly during training (Plot 2). Moreover, on evaluation benchmarks, the length of incorrect responses is substantially reduced by Dr. GRPO compared to the baseline (Plot 4), suggesting that an unbiased optimizer also mitigates overthinking [4].

Figure 5. Comparison of Dr. GRPO and GRPO in terms of training dynamics (Top) and evaluation results (Bottom).

A Duet of Template and Question Set Coverage in RL dynamics

Recall that the Qwen2.5-Math base models can readily answer questions with high accuracy without any prompt template (§ 2.2). Based on this intriguing observation, we are interested in how different templates affect the RL training. Furthermore, given the general belief that larger question set coverage leads to better performance [22, 12], we also study the interaction between different templates and different levels of question coverage.

Experimental settings. Starting from the Qwen2.5-Math-1.5B base model, we apply R1 template, Qwen-Math template and No template respectively to run RL using Dr. GRPO. All experiments are repeated for different question sets that are detailed in Table 3.

Question set#Description
ORZ57kCombining AIME, Numina-Math, Tulu3 MATH; diverse and large amount
MATH12kHigh-school math competition questions
GSM8kSimpler grade-school math questions
ASDiv2kBasic algebra (+−×÷+ - \times\div) questions

Table 3. Different question sets that have different levels of difficulty and coverage.

Results. Figure 6 shows the RL curves of different runs, from which we can make several interesting observations: 1) Templates determine the performance of the initial policies, but RL can improve all policies to a comparable performance of ∼40%\sim40\% (given a proper question set); 2) When using the R1 template, question sets have a significant impact on the dynamics of RL, with too narrow coverage leading to lower plateau performance. However, when using the Qwen-Math template, the best final performance is attained by RL on GSM-8K, demonstrating that training on much simpler (and o.o.d.) questions can largely improve (nearly double) the test accuracy on harder questions. From these observations, we draw the following insights:

Figure 6. The average benchmark accuracy of different {template, question set} combinations during RL training.

  • The Qwen2.5-Math-1.5B base model already possesses strong math-solving capabilities (see the starting point in the right plot of Figure 6). Applying templates in fact destroys the capability before RL reconstructs it. This implies that we should be more conservative in claiming the huge gains brought about by pure RL.

  • When there is a large mismatch between base models and templates (e.g., R1 template mismatches Qwen2.5-Math-1.5B), the policy improvement mainly comes from RL-tuning, thus requiring question set to have good coverage (left plot of Figure 6). Otherwise, even a small and completely o.o.d. question set could induce the reasoning ability equally well, by reinforcing useful reasoning behaviors instead of infusing new knowledge.

Domain-Specific Pretraining Improves RL Ceiling

Recent successful R1-Zero-like replications of math reasoners mostly employ Qwen2.5 base models as the initial policies ([36, 6, 12]), which are already strong math solvers and exhibit self-reflection patterns (§ 2.2 and 2.3). In this section we hope to explore the other side: can R1-Zero-like training succeed on originally weak (in terms of math reasoning) base models? We answer this question affirmatively, with the observation that math pretraining would improve the ceiling of RL.

Experimental settings. We adopt the Llama-3.2-3B base model as our starting point, and use the unbiased Dr. GRPO algorithm for RL-tuning with the R1 template. We hypothesize that domain-specific pretraining would help RL, hence we adopt the Llama-3.2-3B-FineMath6, which is continual pretrained on the FineMath dataset ([2]). Moreover, as we hypothesize that Qwen2.5 models are likely to be pretrained on concatenated question-response texts (§ 2.2), we similarly prepare a concatenated dataset from NuminaMath-1.5 ([18]), and continual pretrain Llama-3.2-3B-FineMath for 2 epochs with learning rate 1e-5. We refer to the concatenated continual pretrained model as Llama-3.2-3B-NuminaQA.

Results. We present the RL curves of different base models in the left plot of Figure 7. We observe that RL can even improve the vanilla Llama base model, but the gain is minimal. After continual pretraining (and concatenated continual pretraining) to embed math domain knowledge, Llama models can show much stronger RL performance, validating our hypothesis. We also revisit the GRPO’s optimization bias with the Llama base model. The right plot of Figure 7 compares the model performance and response length trained with GRPO and Dr. GRPO. We can clearly see that GRPO can produce the “double-increase” phenomenon, potentially leading to a misperception that long-CoT can also emerge on Llama models after math pretraining. Unfortunately, the increase of length might be due to the optimization bias (§ 3.1), which can be effectively mitigated by the proposed Dr. GRPO (§ 3.2 & right plot of Figure 7).

Figure 7. Left: The average benchmark performance curves of different base models. Right: The comparison between Dr. GRPO and GRPO with respect to reasoning accuracy (solid lines) and model response length (dashed lines).

Closing Remarks

We have taken a critical perspective to examine base models used for R1-Zero-like training, as well as algorithms used for RL. Through the analysis, we demystified how pretraining biases influence RL outcomes and how optimization choices, like GRPO, can unintentionally shape model behavior. With the proposed Dr. GRPO, we offer a simple fix that improves token efficiency while preserving reasoning performance. Our results show that scaling RL can be both effective and efficient—sometimes, less really is more.

Policy Gradient Derivations

In the context of RL for LLM post-training, we typically maximize the value of

J(πθ)=Eq∼pQ[Eo∼πθ(⋅∣q)[R(q,o)]](4)\mathcal{J}(\pi_\theta)=\mathbb{E}_{q\sim p_Q}\left[\mathbb{E}_{o\sim\pi_\theta(\cdot\mid q)}[R(q,o)]\right] \tag*{(4)}

where R(q,o)=∑t=1∣o∣r(q,o≤t)R(q,o)=\sum_{t=1}^{|o|}r(q,o_{\leq t}) is the return ([31]) of the trajectory q;oq;o, and r(q,o≤t)r(q,o_{\leq t}) represents the token-level reward for tt-th token in response oo.

The Monte Carlo policy gradient ([31]) of Eq. (4) is

∇θJ(πθ)=Eq∼pQ[Eo∼πθ(⋅∣q)[∇θlog⁡πθ(o∣q)R(q,o)]]=Eq∼pQ[Eo∼πθ(⋅∣q)[∇θ∑t=1∣o∣log⁡πθ(ot∣q,o<t)R(q,o)]]=Eq∼pQ[Eo∼πθ(⋅∣q)[∑t=1∣o∣∇θlog⁡πθ(ot∣q,o<t)∑t′=t∣o∣r(q,o≤t′)]]=Eq∼pQ[Eo∼πθ(⋅∣q)[∑t=1∣o∣∇θlog⁡πθ(ot∣q,o<t)(∑t′=t∣o∣r(q,o≤t′)−B(q,o<t))]].(5)\begin{aligned} \nabla_\theta\mathcal{J}(\pi_\theta) &=\mathbb{E}_{q\sim p_Q}\left[\mathbb{E}_{o\sim\pi_\theta(\cdot\mid q)}[\nabla_\theta\log\pi_\theta(o\mid q)R(q,o)]\right]\\ &=\mathbb{E}_{q\sim p_Q}\left[\mathbb{E}_{o\sim\pi_\theta(\cdot\mid q)}\left[\nabla_\theta\sum_{t=1}^{|o|}\log\pi_\theta(o_t\mid q,o_{<t})R(q,o)\right]\right]\\ &=\mathbb{E}_{q\sim p_Q}\left[\mathbb{E}_{o\sim\pi_\theta(\cdot\mid q)}\left[\sum_{t=1}^{|o|}\nabla_\theta\log\pi_\theta(o_t\mid q,o_{<t})\sum_{t'=t}^{|o|}r(q,o_{\leq t'})\right]\right]\\ &=\mathbb{E}_{q\sim p_Q}\left[\mathbb{E}_{o\sim\pi_\theta(\cdot\mid q)}\left[\sum_{t=1}^{|o|}\nabla_\theta\log\pi_\theta(o_t\mid q,o_{<t})\left(\sum_{t'=t}^{|o|}r(q,o_{\leq t'})-B(q,o_{<t})\right)\right]\right]. \tag*{(5)} \end{aligned}

where B(q,o<t)B(q,o_{<t}) is a variance reduction term, which is invariant with respect to oto_t so that

Eot∼πθ(⋅∣q,o<t)[∇θlog⁡πθ(ot∣q,o<t)B(q,o<t)]=Eot∼πθ(⋅∣q,o<t)[∇θlog⁡πθ(ot∣q,o<t)]B(q,o<t)=[∑otπθ(ot∣q,o<t)∇θlog⁡πθ(ot∣q,o<t)]B(q,o<t)=[∑ot∇θπθ(ot∣q,o<t)]B(q,o<t)=[∇θ∑otπθ(ot∣q,o<t)]B(q,o<t)=[∇θ1]B(q,o<t)=0.\begin{aligned} \mathbb{E}_{o_t\sim\pi_\theta(\cdot\mid q,o_{<t})}\left[\nabla_\theta\log\pi_\theta(o_t\mid q,o_{<t})B(q,o_{<t})\right] &=\mathbb{E}_{o_t\sim\pi_\theta(\cdot\mid q,o_{<t})}\left[\nabla_\theta\log\pi_\theta(o_t\mid q,o_{<t})\right]B(q,o_{<t})\\ &=\left[\sum_{o_t}\pi_\theta(o_t\mid q,o_{<t})\nabla_\theta\log\pi_\theta(o_t\mid q,o_{<t})\right]B(q,o_{<t})\\ &=\left[\sum_{o_t}\nabla_\theta\pi_\theta(o_t\mid q,o_{<t})\right]B(q,o_{<t})\\ &=\left[\nabla_\theta\sum_{o_t}\pi_\theta(o_t\mid q,o_{<t})\right]B(q,o_{<t})\\ &=[\nabla_\theta1]B(q,o_{<t})=0. \end{aligned}

Typically, we set B(q,o<t)=Eo≥t∼πθ(⋅∣q,o<t)[∑t′=t∣o∣r(q,o≤t′)]B(q,o_{<t})=\mathbb{E}_{o_{\geq t}\sim\pi_\theta(\cdot\mid q,o_{<t})}\left[\sum_{t'=t}^{|o|}r(q,o_{\leq t'})\right], which is the expected cumulative reward in the future (also known as the value of the current state), and denote A(ot∣q,o<t)=∑t′=t∣o∣r(q,o≤t′)−B(q,o<t)A(o_t\mid q,o_{<t})=\sum_{t'=t}^{|o|}r(q,o_{\leq t'})-B(q,o_{<t}) as the advantage. In the case of outcome reward, ∑t′=t∣o∣r(q,o≤t′)=∑t′=1∣o∣r(q,o≤t′)=R(q,o)\sum_{t'=t}^{|o|}r(q,o_{\leq t'})=\sum_{t'=1}^{|o|}r(q,o_{\leq t'})=R(q,o).

By setting B(q,o<t)=mean⁡({R(q,o1),…,R(q,oG)})B(q,o_{<t})=\operatorname{mean}(\{R(q,o_1),\ldots,R(q,o_G)\}), the policy gradient of Eq. (5) becomes

∇θJ(πθ)=Eq∼pQ[E{oi}i=1G∼πθ(⋅∣q)[1G∑i=1G∑t=1∣oi∣∇θlog⁡πθ(oi,t∣q,oi,<t)A~i,t]](6)\nabla_\theta\mathcal{J}(\pi_\theta)=\mathbb{E}_{q\sim p_Q}\left[\mathbb{E}_{\{o_i\}_{i=1}^{G}\sim\pi_\theta(\cdot\mid q)}\left[\frac{1}{G}\sum_{i=1}^{G}\sum_{t=1}^{|o_i|}\nabla_\theta\log\pi_\theta(o_{i,t}\mid q,o_{i,<t})\widetilde{A}_{i,t}\right]\right] \tag*{(6)}

where

A~i,t=R(q,oi)−mean⁡({R(q,o1),…,R(q,oG)})std⁡({R(q,o1),…,R(q,oG)}).\widetilde{A}_{i,t}=\frac{R(q,o_i)-\operatorname{mean}(\{R(q,o_1),\ldots,R(q,o_G)\})}{\cancel{\operatorname{std}(\{R(q,o_1),\ldots,R(q,o_G)\})}}.

We adopt the PPO ([26]) objective to compute Eq. (6):

J(πθ)=E[q∼pQ,{oi}i=1G∼πθold(⋅∣q)][1G∑i=1G1∣oi∣∑t=1∣oi∣{min⁡[πθ(oi,t∣q,oi,<t)πθold(oi,t∣q,oi,<t)A~i,clip⁡(πθ(oi,t∣q,oi,<t)πθold(oi,t∣q,oi,<t),1−ϵ,1+ϵ)A~i]}].\mathcal{J}(\pi_\theta)=\mathbb{E}\left[q\sim p_Q,\{o_i\}_{i=1}^{G}\sim\pi_{\theta_{\mathrm{old}}}(\cdot\mid q)\right]\left[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{\cancel{|o_i|}}\sum_{t=1}^{|o_i|}\left\{\min\left[\frac{\pi_\theta(o_{i,t}\mid q,o_{i,<t})}{\pi_{\theta_{\mathrm{old}}}(o_{i,t}\mid q,o_{i,<t})}\widetilde{A}_i,\operatorname{clip}\left(\frac{\pi_\theta(o_{i,t}\mid q,o_{i,<t})}{\pi_{\theta_{\mathrm{old}}}(o_{i,t}\mid q,o_{i,<t})},1-\epsilon,1+\epsilon\right)\widetilde{A}_i\right]\right\}\right].

from which we conclude that both std and ∣o∣|o| should not appear in the RL objective.

Unbiasedness of A~i,t\widetilde{A}_{i,t}. We note that A~i,t\widetilde{A}_{i,t} computed above is equivalent to that of REINFORCE Leave-One-Out (RLOO) [1, 14] up to a scaling factor, which can be subsumed into the learning rate without affecting the RL dynamics. Specifically,

GG−1⋅A~i,t=GG−1R(q,oi)−GG−11G∑j=1GR(q,oj)=GG−1R(q,oi)−1G−1∑j=1,j≠iGR(q,oj)−1G−1R(q,oi)=A^i,tRLOO.\frac{G}{G-1}\cdot\widetilde{A}_{i,t}=\frac{G}{G-1}R(\mathbf{q},o_i)-\frac{G}{G-1}\frac{1}{G}\sum_{j=1}^{G}R(\mathbf{q},o_j) =\frac{G}{G-1}R(\mathbf{q},o_i)-\frac{1}{G-1}\sum_{j=1,j\ne i}^{G}R(\mathbf{q},o_j)-\frac{1}{G-1}R(\mathbf{q},o_i) =\widehat{A}^{\mathrm{RLOO}}_{i,t}.

Detailed Benchmark Results

We show the detailed benchmark results for three scales (1.5B, 3B and 7B) in Table 4. We also include the instruct models at the same scale and R1-Distill models for comparison. Note that since we employ the Qwen2.5-Math base models, which have a context length of 4k, we thus limit the generation budget at 3k for all baselines compared. For models that are trained for a longer context (OpenReasoner-Zero and R1-Distill-Qwen), we also report their performance at 8k generation budget.

Base model + MethodAIME24AMCMATH500MinervaOlympiadBenchAvg.
Qwen2.5-Math-1.5B20.032.533.012.522.824.2
Qwen2.5-Math-1.5B*16.743.461.815.128.433.1
Oat-Zero-1.5B20.053.074.225.737.642.1
R1-Distill-Qwen-1.5B @ 3k2.521.752.216.317.322.0
R1-Distill-Qwen-1.5B @ 8k20.049.477.425.035.841.5
Qwen2.5-Math-1.5B-Instruct10.048.274.226.540.239.8
Llama-3.2-3B0.02.46.46.31.33.3
+ RL w. Dr. GRPO3.37.210.011.02.26.8
Llama-3.2-3B-FineMath0.03.618.45.92.26.0
+ RL w. Dr. GRPO3.310.838.012.99.014.8
Llama-3.2-3B-NuminaQA0.00.00.60.00.10.14
+ RL w. Dr. GRPO (Oat-Zero-3B)6.718.150.014.314.720.7
Llama-3.2-3B-Instruct6.715.738.811.812.617.1
Qwen2.5-Math-7B16.738.650.69.916.626.5
Qwen2.5-Math-7B*0.245.869.021.334.738.2
SimpleRL-Zero-7B26.760.278.227.640.346.6
PRIME-Zero-7B16.762.783.836.040.948.0
OpenReasoner-Zero-7B @ 3k13.347.079.231.644.043.0
OpenReasoner-Zero-7B @ 8k13.354.282.431.647.945.9
Oat-Zero-7B43.362.780.030.141.051.4
R1-Distill-Qwen-7B @ 3k10.026.260.123.023.128.5
R1-Distill-Qwen-7B @ 8k33.368.488.135.947.754.7
Qwen2.5-Math-7B-Instruct16.753.083.629.842.745.1

Table 4. A comparison on benchmark scores. Ours models are RL-tuned by our minimalist recipe (Sec. 1). * means we employ the best template (no template) to generate answers, such that the test scores are highest and can faithfully reflect the capabilities of the base models.

Extended Empirical Results

In this section we present two extended empirical results for (1) the ablation of different bias terms in GRPO and (2) statistical significance of Dr. GRPO’s results. We RL-tune the Qwen2.5-1.5B base model on a mixture of 3K diverse math questions drawn from ASDiv, MATH, and AIME (pre-2023).

Figure 8 shows the training and evaluation curves for the following variants: Dr. GRPO, GRPO w/o length normalization, GRPO w/o standard deviation (std) normalization and Vanilla GRPO. From the middle subplot, we observe that both Dr. GRPO and the variant without length normalization generate shorter responses compared to the other two. This confirms that the length bias term has a more significant influence on response length–consistent with our expectations.

Figure 8. Ablation results on the two bias terms in GRPO.

In terms of performance, Dr. GRPO and the other ablated variants consistently outperform vanilla GRPO in both training rewards and evaluation accuracy. This indicates that removing bias terms (either length or std) improves policy learning, validating our motivation for Dr. GRPO.

Figure 9 compares GRPO and Dr. GRPO across three independent runs. We observe that Dr. GRPO consistently demonstrates statistically significant improvements–both in token efficiency and final accuracy–across different random seeds.

Figure 9. Evaluation results of 3 independent RL runs. The mean curves are drawn in solid lines and the standard deviation is plotted in the shaded areas.

Keyword-based Detection and LLM-Based Identification of Self-Reflection Behaviors

We construct a pool of carefully selected keywords and phrases that signal self-reflection behaviors in the LLM’s responses. However, LLM-generated responses often contain hallucinations and off-topic content, leading to the presence of simple, ambiguous keywords that do not necessarily indicate genuine self-reflection. For instance, terms like “wait” and “try again” frequently result in false positive detections. To reduce false positives, we maintain a small, highly selective keyword pool consisting of terms that are strongly indicative of self-reflection. In our experiment, the keyword pool is limited to: recheck, rethink, reassess, reevaluate, re-evaluate, reevaluation, re-examine, reexamine, reconsider, reanalyze, double-check, check again, think again, verify again, and go over the steps.

We present the occurrences of various keywords in the responses generated by different models in Figure 10. Interestingly, different model families emphasize different keywords. For instance, phrases such as “check again”, “double-check”, “re-evaluate”, “re-examine”, “recheck”, “reconsider”, and “verify again” appear most frequently in the Qwen2.5 family. In contrast, “re-evaluate”, “re-examine”, and “verify again” do not appear in the responses of the DeepSeek family, while Llama models frequently use the phrase “think again.” We hypothesize that this phenomenon results from differences in the pretraining data, particularly in relation to reasoning and mathematics.

Count of self-reflection keyword occurrences by model and keyword

Figure 10. Count of keyword occurrences out of 40,000 responses (500 questions ×\times 8 responses per question ×\times 10 temperatures). y is in log scale.

Although we meticulously select the keyword pool, it may still be insufficient to identify some implicit behaviors of self-reflection that do not contain a specific keyword. Additionally, it can lead to false positives, as illustrated in Case (a) of Figure 11. To address these limitations and more accurately assess the self-reflection capability of base models, we leverage stronger LLMs (GPT-4o-mini in our experiments) to analyze the responses and determine whether they exhibit explicit self-reflection (e.g., keywords like ”recheck” and ”reevaluate”) or implicit self-reflection (e.g., more sophisticated patterns that cannot be easily captured through keyword matching). This approach helps distinguish true self-reflection behaviors from superficial or incidental use of related terms.

Two side-by-side cases showing false positives in keyword-based and LLM-based detection

Figure 11. Case (a): a false positive in keyword-based detection. Case (b): a false positive in LLM-based detection.

While LLM-based detection effectively filters out false positives from keyword-based detection and identifies implicit self-reflection behaviors, it can still misclassify responses, particularly when they are lengthy and complex. For instance, Case (b) in Figure 11 shows a false positive in LLM-based detection, where the response is categorized as self-reflection by the LLM but does not actually exhibit self-reflection. This type of error can be filtered out by keyword-based detection. To enhance robustness, we integrate keyword-based and LLM-based detection through cross-validation. The combined detection results, along with the individual results from keyword-based and LLM-based methods, are presented in Figure 12.

Figure 12. Comparison of keyword-based detection, LLM-based detection, and cross detection. Self-reflections are counted at the question level across 500 questions, where a question is marked as having self-reflection if at least one of its eight responses exhibits self-reflection.

Examples of Aha Moment in DeepSeek-V3-Base

Figure 13 shows two examples to demonstrate that the DeepSeek-V3-Base model already exhibits the so-called “aha moment” even before the RL-tuning.

Comparison Between DeepSeek-V3-Base and DeepSeek-R1-Zero

We analyze DeepSeek-V3-Base and DeepSeek-R1-Zero to understand changes in model behavior during R1-Zero training. In Figure 14, we present the breakdown of response categories across difficulty levels for 500 MATH questions evaluated on both models. The results indicate that most incorrect responses are corrected after RL training, demonstrating substantial performance gains from R1-Zero training. Meanwhile, we find an increase in unformatted responses, which aligns with the observation in Liu et al. [21].

Breakdown of response categories across difficulty levels for DeepSeek-V3-Base and DeepSeek-R1-Zero

Figure 14. Breakdown of response categories across difficulty levels in the MATH dataset for DeepSeek-V3-Base and DeepSeek-R1-Zero.

In Table 5, we report the average response lengths across categories. Note that truncated responses would fall into any of the other three categories if a larger context size were used; thus, we exclude them from the table. The results show a substantial increase in response lengths across all categories, including correct responses, consistent with the results in the

CategoryBaseR1-Zero
Unformatted880.77870.3
Correct621.34965.4
Incorrect1038.98206.1

Table 5. Average response string lengths across categories for DeepSeek-V3-Base (Base) and DeepSeek-R1-Zero (R1-Zero).

Fig. 3 of Guo et al. [8]. However, the average length of incorrect responses is notably longer than that of correct responses. We hypothesize this is because more challenging questions generally require longer responses due to increased reasoning complexity, and incorrect responses are more likely to originate from harder questions, resulting in a longer average length.

Self-reflection does not necessarily imply higher accuracy. To investigate whether self-reflection behaviors are associated with model performance during the inference (acknowledging that self-reflection may improve exploration during training—a potential positive effect outside this section’s scope), we analyze questions that elicit at least one response with self-reflection from DeepSeek-R1-Zero across eight trials. For each question, we sample 100 responses and divide them into two groups: those with self-reflection and those without. We then compute the accuracy difference between these two groups for each question. As shown in Figure 15, the results indicate that nearly half responses with self-reflection do not achieve higher accuracy than those without self-reflection, suggesting that self-reflection does not necessarily imply higher inference-stage accuracy for DeepSeek-R1-Zero.

Accuracy difference between responses with and without self-reflection for each question

Figure 15. Accuracy difference between responses with and without self-reflection for each question (responses sampled from DeepSeek-R1-Zero).

Detailed Experimental Settings

All our experiments are performed on 8 ×\times A100 GPUs and finished in about one day. We enable the actor-learner collocation supported by Oat [20] to optimize the training efficiency. We show the experimental configurations in Table 6.

ParameterValue
ACTOR
Maximum response length3000 tokens
Sampling temperature1.0
(top P, top k)(1.0, -1)
Number of responses per question8
LEARNER
OptimizerAdamW
Adam parameters (β1,β2\beta_1, \beta_2)(0.9, 0.95)
Weight decay0.0
Gradient norm clipping1.0
Learning rate schedulerConstant
Learning rate1×10−61 \times10^{-6}
Inner proximal update epoch1
KL loss coefficient0.0
KL penalty coefficient0.0
Policy clipping parameter0.2

Table 6. Hyperparameter configurations used in all experiments.

Prompts Used for GPT-As-A-Judge

Prompt for checking the model’s question-answering ability.

Prompt for Checking Question-Answering Ability

I will send you a question and a long response generated by an LLM. Your task is to determine whether the output attempts to answer the question or not. The output may sometimes include irrelevant content, hallucinations, or random, off-topic responses.

Please classify the output into one of the following categories:

Output Format:

Your response must start with a single integer (0 or 1), followed by a brief explanation.

  • Return 0: → The output is not trying to answer the question (e.g., irrelevant content, random talking, hallucinations). Example output: ‘0: The response is off-topic and does not address the question.‘

  • Return 1: → The output attempts to answer the question, regardless of how complete or accurate the answer is. Example output: ‘1: The response engages with the question, even if the answer is incomplete or incorrect.‘

Question: {question}

Response: {response}

Prompt for LLM-based detection to determine whether a response contains self-reflection behaviors.

LLM-based Detection for Self-Reflection

I will send you a mathematical question along with a detailed response. Your task is to determine whether the response is attempting to answer the question. If the response is off-topic, hallucinated, random talk, or otherwise irrelevant, mark it as 0. Otherwise, assess whether the response exhibits self-reflection.

Categorization Rules:

  1. Category 0: The response is off-topic, nonsensical, incoherent, overly repetitive, or lacks logical reasoning.

    • Example cases:

      • The response does not relate to the question.

      • It contains meaningless or hallucinated content.

      • It consists of excessive repetition without coherence.

  1. Category 1: The response attempts to answer the question but does not exhibit self-reflection.

    • Example cases:

      • The response directly solves the problem without revisiting steps.

      • No attempt is made to verify the correctness of the answer or explore alternative solutions.

  1. Category 2: The response demonstrates self-reflection at any level.

    • This may include:

      • Explicit self-reflection keywords, such as: *recheck, rethink, reassess, reevaluate, re-evaluate, reevaluation, re-examine, reexamine, reconsider, reanalyze, double-check, check again, think again, verify again, go over the steps*, etc.

      • Implicit self-reflection behaviors, such as revisiting the solution, questioning assumptions, or considering alternative approaches without explicit keywords.

    • If any form of self-reflection is present, always categorize it as 2, regardless of correctness or answer quality.

  1. Category 3: The response consists solely of Python code for calculations without exhibiting self-reflection.

    • Example cases:

      • The response only provides a Python script to compute the solution without any verification, re-evaluation, or alternative considerations.

Output Format:

Your response should first provide a very brief explanation of your analysis, followed by a single category number (0, 1, 2, or 3) at the end. You must include the category number at the end of your response.

Example outputs:

  • ‘The response is off-topic and does not attempt to answer the question. 0.’

  • ‘The response provides a direct solution without self-reflection. 1.’

  • ‘The response demonstrates self-reflection. 2.’

  • ‘The response consists solely of Python code without any self-reflection. 3.’

Question: {question}

Response: {response}

References

  1. [1]Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce optimization for learning from human feedback in llms. arXiv preprint arXiv:2402.14740, 2024.arxiv.org/abs/2402.14740
  2. [2]Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíček, Agustín Piqueres Lajarín, Vaibhav Srivastav, et al. SmolLM2: When small models go big—data-centric training of a small language model. arXiv preprint arXiv:2502.02737, 2025.arxiv.org/abs/2502.02737
  3. [3]Marcin Andrychowicz, Anton Raichuk, Piotr Stańczyk, Manu Orsini, S er tan Girgin, Raphaël Marinier, Leonhard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, et al. What matters for on-policy deep actor-critic methods? a large-scale study. In International conference on learning representations, 2021.
  4. [4]Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, et al. Do not think that much for 2+ 3=? on the overthinking of o1-like llms. arXiv preprint arXiv:2412.21187, 2024.arxiv.org/abs/2412.21187
  5. [5]Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.arxiv.org/abs/1706.03741
  6. [6]Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456, 2025.arxiv.org/abs/2502.01456
  7. [7]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil M athur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.arxiv.org/abs/2407.21783
  8. [8]Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.DOI
  9. [9]Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402.14008, 2024.arxiv.org/abs/2402.14008
  10. [10]Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021.
  11. [11]Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. Open-rlhf: An easy-to-use, scalable and high-performance rlhf framework. arXiv preprint arXiv:2405.11143, 2024.arxiv.org/abs/2405.11143
  12. [12]Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, and Heung-Yeung Shum Xiangyu Zhang. Open-reasoner-zero: An open source approach to scaling reinforcement learning on the base model. https://github.com/Open-Reasoner-Zero/Open-Reasoner-Zero, 2025.arxiv.org/abs/2503.24290
  13. [13]Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv:1907.00456, 2019.arxiv.org/abs/1907.00456
  14. [14]Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 reinforce samples, get a baseline for free!, 2019.
  15. [15]Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tülu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124, 2024.arxiv.org/abs/2411.15124
  16. [16]Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843–3857, 2022.arxiv.org/abs/2206.14858
  17. [17]Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletsky, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numanimath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13:9, 2024a.
  18. [18]Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletsky, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numinamath, 2024b.
  19. [19]Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bo­chao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyuan Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024.arxiv.org/abs/2412.19437
  20. [20]Zichen Liu, Changyu Chen, Chao Du, Wee Sun Lee, and Min Lin. Oat: A research-friendly framework for llm online alignment. https://github.com/sail-sg/oat, 2025a.
  21. [21]Zichen Liu, Changyu Chen, Wenjun Li, Tianyu Pang, Chao Du, and Min Lin. There may not be aha moment in r1-zero-like training — a pilot study. https://oatllm.notion.site/oat-zero, 2025b. Notion Blog.
  22. [22]Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl. https://github.com/agentica-project/deepscaler, 2025.
  23. [23]Jiayi Pan, Junjie Zhang, Xingyao Wang, Lifan Yuan, Hao Peng, and Alane Suhr. Tinyzero. https://github.com/Jiayi-Pan/TinyZero, 2025. Accessed: 2025-01-24.
  24. [24]John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015.arxiv.org/abs/1506.02438
  25. [25]John Schulman, Xi Chen, and Pieter Abbeel. Equivalence between policy gradients and soft q-learning. arXiv preprint arXiv:1704.06440, 2017a.arxiv.org/abs/1704.06440
  26. [26]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017b.arxiv.org/abs/1707.06347
  27. [27]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.arxiv.org/abs/2402.03300
  28. [28]Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv:2409.19256, 2024.DOI
  29. [29]Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019.arxiv.org/abs/1909.08053
  30. [30]Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008–3021, 2020.arxiv.org/abs/2009.01325
  31. [31]Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018.
  32. [32]Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement learning. https://github.com/huggingface/trl, 2020.
  33. [33]An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024a.arxiv.org/abs/2412.15115
  34. [34]An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2.5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122, 2024b.arxiv.org/abs/2409.12122
  35. [35]Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long chain-of-thought reasoning in llms. arXiv preprint arXiv:2502.03373, 2025.arxiv.org/abs/2502.03373
  36. [36]Weihao Zeng, Yuzhen Huang, Wei Liu, Keqing He, Qian Liu, Zejun Ma, and Junxian He. 7b model and 8k examples: Emerging reasoning with reinforcement learning is both effective and efficient. https://hkust-nlp.notion.site/simplerl-reason, 2025. Notion Blog.

Paper details

vlnc:W_jn64zpw4
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents