betaFeedback

WorksComputer Science

What Drives the Zero-to-One Jump? Diagnosing and Steering Hard-Prompt Learning in RLVR

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has substantially improved mathematical reasoning, yet progress on hard prompts often stalls: a prompt can remain unsolved for many training steps before abruptly transitioning to solved–a discrete "zero-to-one" jump whose underlying dynamics and steerability remain poorly understood. We investigate two questions: what drives this transition, and can it be steered? Through event-based analysis measuring the time-to-first-positive (TTFP), we find that negative rollouts preceding successful transitions consistently retain higher early-stage generation entropy, distinguishing near-solvable failures from dead ends. This signal motivates NAPO (Negative-Augmented Policy Optimization), a method integrated with Group Relative Policy Optimization (GRPO) that ranks negatives in unsolved groups via an early-branching driver and assigns graded advantages, injecting informative gradients precisely where standard group-relative updates vanish. Under fixed rollout budgets, NAPO accelerates early solution discovery and improves Pass@256 by 10.0 points on AIME24 and 6.6 points on AIME2025, demonstrating that zero-to-one transitions are diagnosable and steerable through driver-guided negative learning. Our code is available at https://github.com/jameszhu-pixel-time/Z2O.

Introduction

Reinforcement Learning with Verifiable Rewards (RLVR) ([7]) has rapidly emerged as a dominant paradigm for post-training reasoning models, particularly in mathematical problem solving where correctness can be automatically verified ([5] (Yang et al. 2025a)). At each training step, the policy samples multiple rollouts

per prompt and receives binary rewards indicating whether each final answer is correct. When some rollouts in a group succeed, the resulting contrast between positive and negative samples provides a clear gradient that drives steady improvement. However, on sufficiently hard prompts the on-policy pass rate is so low that entire rollout groups contain only incorrect samples for many consecutive steps. In group-relative methods such as GRPO ([15, 5]), this eliminates reward contrast and yields near-zero advantages on precisely the prompts that most need exploration. As a result, hard prompts often remain stagnant for many steps and then transition abruptly from unsolved to solved — a discrete “zero-to-one” jump rather than smooth refinement. The dynamics that drive this transition, and whether they can be steered, remain largely unexplored.

Answering these questions, however, requires confronting two intertwined obstacles that current RLVR pipelines leave unresolved. The first obstacle is diagnosing zero-to-one events under binary supervision. Standard RLVR diagnostics offer little purchase on what happens inside a hard prompt before it is first solved: pass@k ([2]) aggregates end-of-training capability and cannot localize the moment a previously unsolved prompt becomes solvable, while group-level rewards remain identically zero across long stretches of training and therefore carry no information about which unsolved prompts are close to a breakthrough. What is missing is (i) an event-based view of training that pinpoints when transitions occur and (ii) an internal, rollout-level signal that distinguishes near-solvable failures from true dead ends using only the information already available at sampling time.

The second obstacle is turning heterogeneous negatives into informative gradient on fully-negative groups. A majority of RLVR methods build on grouped objectives such as GRPO and its variants [15, 20] (Yu et al. 2025; Zheng et al. 2025a), where all-negative groups produce vanishing advantages and provide little learning signal [10] (Yu et al. 2025). Recent variants restore a gradient through weighted negative learning, virtual rewards, or entropy-aware updates [12, 8] (Zhu et al. 2025), but they apply roughly uniform pressure across negatives. The open problem is therefore how to assign graded credit among them under only binary rewards, in a way that injects gradient into unsolved groups without destabilizing the rest of training.

Figure 1. Overview of NAPO pipeline

We analyze token-level generation entropy during RLVR training and find that negative rollouts from prompts that later undergo a Negative-to-Positive (N→P) transition exhibit front-loaded uncertainty, with early-response entropy patterns that distinguish them from rollouts on prompts that remain negative (N→N). Based on this signal, we propose NAPO (Negative-Augmented Policy Optimization), a GRPO-integrated method that ranks negative samples by front-loaded entropy and injects informative gradients into unsolved groups.

Our contributions are threefold:

  • We study zero-to-one learning dynamics in RLVR through event-based metrics, especially N→P transitions and TTFP, to characterize how hard prompts first become solvable.

  • We identify a token-entropy pattern associated with N→P transitions and propose early-branching drivers that rank negative rollouts by their latent learning potential.

  • We introduce NAPO, a GRPO-integrated algorithm that assigns graded advantages to negative samples according to their driver-based ranking while maintaining stable updates via in-group normalization, improving token-efficient exploration on hard prompts as reflected by better TTFP and higher pass@k.

Preliminaries

Related Work. Reinforcement learning (RL) [17] has been shown to improve LLM reasoning via both exploration and exploitation [1, 3]. Approaches such as RLHF [13], Test-Time RL (Zuo et al., 2025), and RLVR [5] train policies using verifiable rewards or reward models, often with policy-gradient methods like PPO, GRPO, and REINFORCE [14, 15, 19]. These methods can stall on very hard prompts [10]. Recent work addresses this by reallocating rollouts toward learnable prompts and by extracting informative signals from negative samples, including token-level entropy to encourage exploration and reduce overfitting (Zhu et al., 2025; Le et al., [8]; Fang et al. [4]. We further introduce the negative-to-positive transition to predict and leverage latent prompt learnability to overcome this challenge.

Group-Relative Policy Optimization for RLVR. Following recent RLVR practice for reasoning models, we sample a group of GG rollouts {yi}i=1G\{y_i\}_{i=1}^{G} for each prompt and optimize the policy with Group Relative Policy Optimization (GRPO) [5, 15]. Let πθold\pi_{\theta_{\mathrm{old}}} be the snapshot policy and define the token-level likelihood ratio ρi,t(θ)=πθ(ai,t∣si,t)πθold(ai,t∣si,t)\rho_{i,t}(\theta)=\frac{\pi_\theta(a_{i,t}\mid s_{i,t})}{\pi_{\theta_{\mathrm{old}}}(a_{i,t}\mid s_{i,t})}. GRPO then applies the clipped surrogate objective

max⁡θExEyi∼πθold[∑i=1G∑tℓi,t−βKL⁡(πθ∥πref)]\max_\theta\mathbb{E}_{x}\mathbb{E}_{y_i\sim\pi_{\theta_{\mathrm{old}}}}\left[\sum_{i=1}^{G}\sum_t \ell_{i,t}-\beta\operatorname{KL}(\pi_\theta\Vert\pi_{\mathrm{ref}})\right]
ℓi,t=min⁡(ρi,t(θ)A^i,t,clip⁡(ρi,t(θ),1−ϵ,1+ϵ)A^i,t)(1)\ell_{i,t}=\min\left(\rho_{i,t}(\theta)\widehat{A}_{i,t},\operatorname{clip}(\rho_{i,t}(\theta),1-\epsilon,1+\epsilon)\widehat{A}_{i,t}\right) \tag*{(1)}

where A^i,t\widehat{A}_{i,t} is the group-relative advantage, πref\pi_{\mathrm{ref}} is a reference policy, and β\beta controls KL regularization. For brevity, we use πθ\pi_\theta and πref\pi_{\mathrm{ref}} to denote the distributions conditioned on a given prompt xx, i.e., πθ(⋅∣x)\pi_\theta(\cdot\mid x) and πref(⋅∣x)\pi_{\mathrm{ref}}(\cdot\mid x), respectively.

PSR/NSR decomposition. For verifiable-reward post-training, each rollout receives a binary outcome. We view the learning signal through the positive-sample reward (PSR) and negative-sample reward (NSR) decomposition (Zhu et al., 2025). Let r(x,y)∈{−1,+1}r(x,y)\in\{-1,+1\} be the verifiable reward for completion yy on prompt xx. A simplified RLVR objective is

LRLVR(θ):=−Ex∼DEy∼πθ(⋅∣x)[r(x,y)].(2)\mathcal{L}_{\mathrm{RLVR}}(\theta):=-\mathbb{E}_{x\sim\mathcal{D}}\mathbb{E}_{y\sim\pi_\theta(\cdot\mid x)}[r(x,y)]. \tag*{(2)}

Using r=I[r=+1]−I[r=−1]r=\mathbb{I}[r=+1]-\mathbb{I}[r=-1], we can write

LRLVR(θ)=−Ex∼DEy∼πθ(⋅∣x)[I[r(x,y)=+1]]+Ex∼DEy∼πθ(⋅∣x)[I[r(x,y)=−1]].(3)\mathcal{L}_{\mathrm{RLVR}}(\theta)=-\mathbb{E}_{x\sim\mathcal{D}}\mathbb{E}_{y\sim\pi_\theta(\cdot\mid x)}[\mathbb{I}[r(x,y)=+1]]+\mathbb{E}_{x\sim\mathcal{D}}\mathbb{E}_{y\sim\pi_\theta(\cdot\mid x)}[\mathbb{I}[r(x,y)=-1]]. \tag*{(3)}

PSR reinforces verified rollouts, whereas NSR suppresses failed ones. Both sides can provide useful supervision on reasoning tasks (Zhu et al., 2025).

Token-level uncertainty and rollout-level drivers. To analyze how N→P happens, we analyze the patterns of negative rollouts before N→P happens. We measure token-level uncertainty with the entropy of the next-token distribution [18] (Yang et al. 2025b). For each rollout, let pt(k)p_t^{(k)} denote the top-kk truncated next-token distribution at position tt after renormalization, with invalid and padding positions masked out. We define the token-level entropy sequence

Ht(k)=−∑vpt(k)(v)log⁡pt(k)(v),(4)H_t^{(k)}=-\sum_v p_t^{(k)}(v)\log p_t^{(k)}(v), \tag*{(4)}

and its exponential form

ρt(k)=exp⁡(Ht(k)),(5)\rho_t^{(k)}=\exp(H_t^{(k)}), \tag*{(5)}

which we use as a token-level perplexity / effective branching statistic. Our goal is not to measure absolute early-stage uncertainty in isolation, but to summarize the relative prominence of early uncertainty within each rollout.2

N→P Transition. To localize progress on hard prompts, let qt(x)q_t(x) indicate whether evaluation at training step tt contains at least one positive rollout for prompt xx. We define the time-to-first-positive (TTFP) as

TTFP⁡(x)=min⁡{t:qt(x)=1},\operatorname{TTFP}(x)=\min\{t:q_t(x)=1\},

with TTFP⁡(x)=∞\operatorname{TTFP}(x)=\infty if no positive appears within the training horizon. We define an N→P transition as

qt−1(x)=0∧qt(x)=1.q_{t-1}(x)=0\land q_t(x)=1.

This implies that the model solves a problem on which all previous attempts failed.

Signals of Near-Solvable Failures

This section answers the question of what drives the N→P transition.

Experimental Setup

We examine two axes of variation in this section: (i) within-batch, comparing rollouts inside all-negative groups, and (ii) between-replay, comparing two training continuations that share a base state but differ in sampling stochasticity.

Fixed-batch protocol. RLVR on hard prompts exhibits strong temporal correlations across training steps [9] (Zheng et al. 2025b). To expose this structure directly, we train Qwen2.5-Math-7B on a fixed set of 1024 DAPO-17k prompts for 10 consecutive gradient steps, sampling G=8G = 8 rollouts per prompt per step. Holding the prompt batch fixed across steps lets us align rollout-internal uncertainty patterns with later N→P events using the metrics of § 2. Unless otherwise noted, we use temperature T=1.0T = 1.0, maximum context length 4096, learning rate 10−610^{-6}, and zero entropy/KL penalties.3

CategorySetting
Model
ModelQwen2.5-Math-7B
Data
Train batch size1024
Max prompt length1024
Max response length3072
RL Algorithm
KL coefficient0.0
Entropy coefficient0.0
Critic warmup0
Optimization
Learning rate1×10−61 \times10^{-6}
PPO minibatch size256
Sampling
top_k-1
top_p1
temperature1.0

Table 4. Detailed training hyperparameters.

Critical-step replay. Across the 10-step window, we identify the step t∗t^* at which N→P events peak and treat it as the critical checkpoint. Starting from this shared state, we replay training with the same base policy, optimizer state, data order, and hyperparameters, varying only rollout stochasticity. This yields two representative continuations, denoted n1n_1 and n2n_2. We do not label either replay as “good” or “bad” at this point.

Outcome measures. (i) Prompt-level transition label. Among prompts that are negative at the base checkpoint t∗t^*, we call a prompt N→P if at least one later checkpoint within the replay window produces a positive rollout for it, and N→N otherwise. Each negative rollout inherits its prompt’s label. At training time the binary reward gives both groups identical labels; the partition into N→P and N→N is only observable after the replay window, which is precisely what makes the within-batch driver signal of § 3.2.2 non-trivial.

(ii) Replay-level transition counts. For each replay we report the number of N→P and P→N transitions under 64 rollouts per prompt, together with the mean TTFP over prompts within the replay window. Smaller first-positive indices indicate earlier solution discovery.

(iii) Validation sets. We evaluate replay checkpoints on AMC23, AIME 2025, and Math-Beyond (Mayilvahanan et al., 2025) under the same 64-rollout budget.

Findings and Implications

We first define a rollout-time driver that compresses the entropy–position pattern into a single scalar (§ 3.2.1), then validate it at each level in turn (§ 3.2.2 and § 3.2.3).

An early-branching driver

Why a ratio, not an absolute level. A naive metric would be the average entropy of a rollout, but the contrast we care about is not “more entropy” in absolute terms; it is a difference in the timing of uncertainty—how much of a rollout’s total uncertainty is allocated to its opening tokens [4].

Definition. Let Tα=⌈αT⌉T_\alpha= \lceil\alpha T \rceil denote the number of tokens in the first α\alpha fraction of a response of length TT. The early mean-perplexity ratio is

dearly-ppl=1Tα∑t=1Tαρt(k)1T∑t=1Tρt(k),(6)d_{\mathrm{early\text{-}ppl}} = \frac{\frac{1}{T_\alpha}\sum_{t=1}^{T_\alpha}\rho_t^{(k)}} {\frac{1}{T}\sum_{t=1}^{T}\rho_t^{(k)}} , \tag*{(6)}

where ρt(k)=exp⁡(Ht(k))\rho_t^{(k)} = \exp(H_t^{(k)}) is the exponential of the top-kk truncated entropy defined in Eq. (5). Values of dearly-ppl>1d_{\mathrm{early\text{-}ppl}} > 1 indicate that uncertainty is concentrated early in the response.4

Choice of the early fraction. We introduce the early-fraction parameter α\alpha and compare 0.1, 0.2, and 0.5 on the n1n_1–n2n_2 replay pair (Fig. 5).5 these candidates, 0.2 yields the clearest and most stable separation between N→\toP and N→\toN rollouts while remaining robust across both replays, and we use it throughout the remainder of the paper.

Within-batch evidence: negatives are not homogeneous

Entropy–position pattern. Figure 2a plots mean token entropy as a function of position for negative rollouts in replay n2n_2. The N→\toP group maintains noticeably higher uncertainty at the beginning of the response, whereas the N→\toN group collapses earlier to lower-entropy continuations. The two groups converge later in the response, confirming that the discriminative signal is localized in the early stage.

Entropy-position plots

Figure 2. (a) Entropy-position curves for the bad replay n2n_2. (b) Comparison of the N→\toP group between n1n_1 and n2n_2. Figure 2: Entropy-position plots. The main separation is localized in the early part of generation. Prompts that later undergo N→\toP transitions preserve more early-stage uncertainty than prompts that remain N→\toN, and this pattern is sharper in the good replay n1n_1 than in the bad replay n2n_2.

Driver separation. Figure 3a shows the distribution of dearly-ppld_{\mathrm{early\text{-}ppl}} for the two groups in replay n1n_1. Negatives associated with future N→\toP prompts are systematically shifted toward larger driver values than negatives associated with N→\toN prompts, mirroring the qualitative pattern of Figure 2a under the single scalar of Eq. (6).

Driver distribution for the good replay $n1$ and driver distribution for the bad replay $n2$ and good replay $n1$

Figure 3. (a) Driver distribution for the good replay n1n1. See Appendix § 8 for statistics’ demonstration. (b) Driver distribution for the bad replay n2n2 and good replay n1n1.

Claim A (within-batch). The driver provides a rollout-level ordering signal that separates near-solvable failures from dead ends inside an otherwise indistinguishable negative pool.

Between-replay evidence: the driver tracks aggregate transition tendency

Entropy–position pattern. Figure 2b restricts attention to N→\toP negatives and compares n1n_1 against n2n_2. The good replay n1n_1 preserves more early-stage uncertainty than the bad replay n2n_2, sharpening the front-loaded pattern of Claim A but now across replays rather than within one. This is consistent with the relevant contrast being a matter of timing rather than magnitude: the same shape of distribution is shifted earlier in n1n_1 than in n2n_2.

Driver distribution. Figure 3b overlays all four (replay ×\times label) groups. n1n_1 allocates more probability mass to high-driver rollouts than n2n_2 does in both groups.6

Realized transitions. Table 1 reports the downstream outcome. Starting from an identical base state, the good replay n1n_1 realizes more N→\toP transitions on the training set and attains a smaller mean first-positive index than n2n_2. The ranking of realized transitions matches the ranking of right-tail driver mass, indicating that the batch-level driver shift is not merely a cosmetic difference but is related to which replay will subsequently break more prompts out of the all-negative regime. This is consistent with recent RLVR observations that low-success prompts exhibit structured temporal dynamics rather than noise-like fluctuations throughout training (Zheng et al., 2025b).

ReplayAMC23 N→PAMC23 P→NAIME2025 N→PAIME2025 P→NMath-Beyond N→PMath-Beyond P→NTraining Set N→PTraining Set P→NTTFP
n1n_12/30.332.67/20212/1345.331201161.213
n2n_21.67/32.672/202.338.33/1345.33117.67102.671.225

Table 1. Transition statistics at t∗=1t^* = 1 for the n1n_1–n2n_2 replay pair. A prompt is labeled as PP if at least one rollout is correct under the fixed rollout budget, and as NN otherwise. In the validation columns, a/ba/b reports a N→P transitions among bb prompts that are negative at the base checkpoint. Results are averaged over three runs, and TTFP is computed over the first three steps. Best value is shown in bold.

Claim B (between-replay). The driver also functions as a batch-level diagnostic: batches whose driver distribution has a heavier right-tail mass go on to realize more N→\toP events.

Figure 3: Within-batch separation under the practical driver dearly−ppld_{\mathrm{early-ppl}}. Rollouts associated with prompts that later undergo N→P transitions are systematically right-shifted relative to rollouts from prompts that remain N→N. The separation is present in both replays, but is cleaner and more pronounced in the good replay n1n1.

Takeaway

Prompts approaching the N→P boundary exhibit a distinctive form of front-loaded uncertainty in their negative rollouts. The early-branching driver dearly−ppld_{\mathrm{early-ppl}} captures this pattern at two levels of aggregation: it separates N→P from N→N negatives within a batch (Claim A), and it separates higher-yield from lower-yield batches across replays (Claim B).

Algorithm

We now turn the empirical findings above into a training design. Our goal is to translate the driver into a practical update rule for RLVR.

Theoretical Interpretation

We provide a view of Fisher information to theoretically interpret the driver on the negative rollouts.

Fisher information as a measure of policy plasticity. Larger local Fisher means that a failed continuation is easier to reshape under a small parameter update, so the policy remains more plastic around that rollout. Let pt=πθ(⋅∣x,y<t)\mathbf{p}_t = \pi_\theta(\cdot\mid x, y_{<t}) be the next-token distribution at step tt, and let Ft\mathbf{F}_t denote the corresponding token-level Fisher contribution.

Lemma 4.1 (Token-level Fisher information and entropy). For token distribution pt\mathbf{p}_t, the token-level Fisher trace is

Tr⁡(Ft)=1−∑k∈Vpt,k2.\operatorname{Tr}(\mathbf{F}_t) = 1 - \sum_{k \in\mathcal{V}} p_{t,k}^{2}.

Accordingly, Shannon entropy serves as a useful proxy for local Fisher information: higher-entropy token distributions typically correspond to more policy-plastic states, although the two quantities are not identical in general.

Implication for negative learning. Lemma 4.1 suggests that uniform penalties are wasteful: low-entropy failures are already rigid, so pushing them down mostly reinforces dead ends. By contrast, failures generated in higher-entropy states are more plastic and retain nearby alternatives, making negative updates more likely to redirect the policy toward useful exploration. Empirically, the informative plasticity is concentrated early in the rollout, which is why NAPO uses the early-branching driver rather than generic response-level entropy. In short, we want to penalize failures more strongly where the policy is still movable.

Takeaway

The driver indicates that failures arising in more plastic regions of the policy. The N→P phenomenon suggests that failures with smaller driver values are not random or hopeless: they come from prompts whose trajectories can cross into correctness under nearby policy shifts.

This leads to a design intuition that larger penalties can be assigned to negative trajectories with larger drivers where the policy still retains meaningful uncertainty and plausible alternatives.

Negative-Augmented Policy Optimization

Motivated by the heterogeneity of negative rollouts, we propose NAPO, a GRPO-integrated algorithm that augments standard policy optimization

Number of solved problems across training steps

Figure 5. Number of solved problems across training steps. Setting the early fraction gate to 0.2 leads to the fastest exploration of correct solutions, consistent with our inference-time findings.

Algorithm 1 NAPO: Negative-Augmented Policy Optimization

  1. Sample prompts xx; for each xx, sample rollouts {yi}\{y_i\} and rewards r(x,yi)r(x,y_i);

  2. if ∃i:r(x,yi)=+1\exists i : r(x,y_i)=+1 then

  3. Compute standard GRPO advantages AGRPO(x,yi)A_{\mathrm{GRPO}}(x,y_i);

  4. Update the policy with GRPO loss.

  5. else

  6. Compute driver scores for all negative rollouts by (6);

  7. Re-normalize scores into [0,1][0,1] by (7);

  8. Assign rollout-level advantages by (8);

  9. Update the policy with GRPO loss.

  10. end if

with driver-weighted negative learning on unsolved groups.

Driver-weighted negative advantage. NAPO uses the driver to modulate negative learning strength inside unsolved groups. Given driver scores d(x,yi)d(x,y_i) for rollouts in a fully negative group, we first re-normalize them into [0,1][0,1] to avoid exploding gradient while preserving a relevant advantage magnitude:

d~(x,yi)=d(x,yi)−min⁡jd(x,yj)max⁡jd(x,yj)−min⁡jd(x,yj)+ϵ.(7)\tilde{d}(x,y_i)=\frac{d(x,y_i)-\min_j d(x,y_j)}{\max_j d(x,y_j)-\min_j d(x,y_j)+\epsilon}. \tag*{(7)}

We then assign rollout-level advantages as

A(x,y)={AGRPO(x,y),if p(x)=1,−d~(x,y),if p(x)=0,(8)A(x,y)= \begin{cases} A_{\mathrm{GRPO}}(x,y), & \text{if } p(x)=1,\\ -\tilde{d}(x,y), & \text{if } p(x)=0, \end{cases} \tag*{(8)}

where p(x)=I[∃i:r(x,yi)=+1]p(x)=\mathbb{I}[\exists i:r(x,y_i)=+1] indicates whether the current group is solved.

Number of solved problems across training steps

Figure 5 (PDF p. 7). Number of solved problems across training steps. Setting the early fraction gate to 0.2 leads to the fastest exploration of correct solutions, consistent with our inference-time findings.

Objective: Grouped advantage on solved groups and weighted NSR on unsolved groups. Let p(x)=I[∃i:r(x,yi)=+1]p(x)=\mathbb{I}[\exists i:r(x,y_i)=+1] denote whether the current group is solved. Equivalently, the rollout-level advantage in Eq. (8) can be written as

A(x,yi)=p(x)AGRPO(x,yi)−(1−p(x))d~(x,yi).(9)A(x,y_i)=p(x)A_{\mathrm{GRPO}}(x,y_i) -(1-p(x))\tilde{d}(x,y_i). \tag*{(9)}

When p(x)=1p(x)=1, Eq. (9) reduces to the standard GRPO clipped update. When p(x)=0p(x)=0, we have A(x,yi)=−d~(x,yi)A(x,y_i)=-\tilde{d}(x,y_i), so the same clipped surrogate becomes a weighted NSR loss in which larger-driver negatives receive larger-magnitude penalties while clipping preserves the usual PPO/GRPO trust-region effect.

Evaluation

We empirically evaluate the effectiveness of NAPO on hard prompts.

Setup. We evaluate NAPO under a fixed rollout budget and compare against GRPO and negative-sample-based baselines, including NSR’s weighted variant (w-REINFORCE). We also evaluate the impact of early fraction ratio, where NAPO02 refers to taking the first 20% of the response in the driver, while NAPO05 takes 50% and NAPO01 takes 10%. We report (i) Pass@k on validation sets and (ii) TFFP. Experiments are conducted on multiple validation sets (AMC23, AIME2025, AIME24, and Math-Beyond), covering a wide range of difficulty levels.7

MethodPass@k
kk1248163264128256
AMC23
Base0.28470.45120.62370.75460.84000.90100.94950.98381.0000
GRPO0.73520.79100.82110.84430.86810.89720.93060.95790.9750
W-REINFORCE0.45140.57000.65700.72450.79000.85550.90860.94500.9750
NAPO020.71410.77180.80510.83280.86280.89400.92220.94670.9750
NAPO050.71910.78280.82470.85560.88450.91470.94380.96540.9750
AIME2025
Base0.03050.05610.09640.14960.20740.26570.32160.36840.4000
GRPO0.13460.17530.21650.25630.29100.32710.37400.42940.4667
W-REINFORCE0.02540.04390.06890.09700.13000.17670.24250.32780.4333
NAPO020.12200.16480.21130.26310.31750.36950.41950.47450.5333
NAPO050.11380.15760.20080.24750.30060.35550.41300.47410.5333
AIME24
Base0.08570.15080.24150.34250.43430.51120.58190.65750.7333
GRPO0.28320.35740.42420.48600.53910.58690.63300.67190.7000
W-REINFORCE0.15160.23220.31050.37060.41520.45000.48210.52090.5667
NAPO020.29880.38780.46480.53140.59170.65040.70480.75640.8000
NAPO050.28980.37620.44900.51330.57340.63010.68100.72480.7667
Math-Beyond
Base0.04260.07020.10510.14460.18790.23570.28660.33910.3978
GRPO0.13370.15740.17800.19840.22050.24540.27230.30140.3370
W-REINFORCE0.05870.08660.11470.14000.16390.18990.21970.25640.3094
NAPO020.13480.16120.18820.21300.23340.25270.27630.31050.3591
NAPO050.13830.16800.19480.21640.23370.25080.27040.29260.3204

Table 2. Pass@k comparison across datasets. The row corresponding to our method is highlighted in gray, the best value within each dataset block and column is shown in bold, and the second-best value is underlined. For W-REINFORCE method, we set the positive advantage weight to 0.5 to handle the much harder prompts.

Training protocol. We use the verl framework [16] and train on DAPO-17k (Yu et al., 2025), a challenging dataset for Qwen2.5-Math-7B (Yang et al., 2024), where the initial pass rate is below 0.2. For each prompt, we sample G=8G = 8 rollouts per step. Unless otherwise specified, we use temperature T=1.0T = 1.0 during training, a maximum context length of 4096, and train for one epoch.8 We report both large-scale validation results and controlled experiments under identical training batches (as in § 3.2).

Results. (1) Improved sample efficiency on hard prompts. As shown in Figure 4, NAPO discovers first-positive rollouts earlier across datasets, solving more prompts during early training under the same rollout budget. This indicates improved sample efficiency, particularly in low-success regimes where standard GRPO suffers from weak learning signals. These gains require little additional computation: NAPO uses no extra rollout, verifier call, auxiliary model, or policy forward pass. We compute the uncertainty statistic from logits already produced during the old-policy forward pass and retain one scalar per valid response token rather than a vocabulary-sized tensor; the remaining operations are masking, sequence-wise reductions, and within-group min–max normalization. Across common logged steps, the advantage calculation time is 0.935±0.0300.935 \pm0.030 s/step for NAPO and 0.525±0.0160.525 \pm0.016 s/step for GRPO (mean ±\pm standard deviation). The resulting increment is only 0.410 s/step, approximately 0.03% of the observed end-to-end step time. Thus, the trace shows no material training-time overhead.

Cumulative solved prompts on validation sets

Figure 4. Cumulative solved prompts on validation sets. NAPO achieves faster early progress, solving more prompts within the first 50 steps, while maintaining comparable performance at later stages.

(2) Effect of negative scaling. Comparing NAPO and NAPO05 reveals a clear trade-off rather than a uniform winner. The milder NAPO05 variant performs best on AMC23, on the highest-kk region of AIME2025, and on low-kk Math-Beyond, whereas full NAPO is clearly strongest on AIME24 and in the mid-kk regime of AIME2025. This pattern suggests that negative updates should be tuned to benchmark difficulty: stronger shaping helps when there is substantial room for zero-to-one improvement, while milder scaling is more robust once exploration is less constrained.

(3) Within-group differentiation versus uniform negative feedback. To isolate the effect of within-group differentiation, we add a Uniform(−0.1-0.1) control that assigns the same small negative advantage to every rollout in an all-negative group. All training and evaluation settings are otherwise matched. Table 3 reports the mean paired improvement of NAPO over this control across four independent evaluations.

Datasetk=1k = 1k=8k = 8k=256k = 256
AMC23+14.40+7.95+1.88
AIME2025+10.48+15.31+8.33
AIME24+20.15+27.28+17.50
Math-Beyond+7.43+8.73+0.41

Table 3. Pass@kk gains of NAPO over Uniform(−0.1-0.1) across datasets at T=1.0T = 1.0, averaged over four rollout reevaluations with 256 generations per question. Values are percentage points; positive values favor NAPO.

Among the evaluation results above, the separation is strongest on the AIME benchmarks—for example, +27.28 points at Pass@8 on AIME24—and the mean remains positive at Pass@256 on all four datasets. In a separate single-evaluation control with Uniform(−1-1), NAPO improves Pass@256 by 17.50, 23.33, 30.00, and 17.13 percentage points on AMC23, AIME2025, AIME24, and Math-Beyond, respectively. Thus, simply activating constant negative feedback does not reproduce NAPO in these runs; these controls isolate the benefit of driver-based within-group differentiation over a constant update. Because the four reevaluations use fixed checkpoints rather than independent training runs, they establish evaluation-level consistency but not training-seed significance.

Conclusion

We studied the zero-to-one problem in RLVR, motivated by two obstacles: diagnosing N→PN \to P transitions under binary supervision, and extracting informative gradients from heterogeneous negatives in fully-unsolved groups. Using event-based metrics (TTFP and N→PN \to P events), we showed that failures preceding successful transitions retain higher early-stage uncertainty, providing a practical signal to separate near-solvable rollouts from dead ends. Building on this, we proposed NAPO, a GRPO-integrated method that assigns graded negative advantages according to driver scores under a Fisher-information view, thereby injecting structured gradients into unsolved groups while preserving stable updates on solved ones. NAPO accelerates early solution discovery and improves Pass@256 by 10.0 points on AIME24 and 6.6 points on AIME2025 compared with GRPO, confirming that driver-guided negative learning is an effective way to broaden reasoning boundaries in low-success RLVR.

Limitations

Our analysis and method are currently focused on math-oriented RLVR settings with binary verifiers. Although these tasks provide a clean testbed for studying zero-to-one transitions, the same entropy-based driver may behave differently in broader domains, longer-horizon reasoning tasks, or environments with partial-credit, noisy, or subjective feedback. Extending NAPO to such settings may require more robust driver definitions and calibration procedures.

NAPO also relies on rollout-level statistics collected during training. Its benefits may therefore depend on the rollout budget, group size, decoding temperature, and the quality of the base policy. When the model rarely approaches a correct solution, early uncertainty patterns may be insufficient to identify useful negative samples; conversely, when success rates are already high, the additional weighting of negatives may provide limited gains. Future work should evaluate the method under more diverse model scales, verifier designs, and compute budgets.

Acknowledgments

This work was supported in part by China NSF grant No.U2268204, 62322206, 62432007, 62441236, 62132018, 62025204, 62272307, 62372296, U25A6024. The opinions, findings, conclusions, and recommendations expressed in this paper are those of the authors and do not necessarily reflect the views of the funding agencies or the government.

Experiment Settings

Chat template

Chat template for Qwen

System: You are a helpful assistant.

User: {input} Please reason step by step, and put your final answer within \boxed{}.

Assistant: Model generates a step-by-step solution; the final answer is enclosed in \boxed{}.

Hyperparameter Settings

All experiments are conducted on a single node equipped with eight NVIDIA H100 GPUs. We report final evaluation results using the checkpoint at step 200 for Qwen2.5-Math-7B and the checkpoint at step 100 for Llama-3.1-8B-Instruct.

Definitions of Batch-Level Driver Statistics

In this appendix, we define the batch-level summary statistics used in Table 5 and Figure 3 to characterize the distribution of rollout-time drivers within the N→\toP group. For Figure 3 we use kernel density estimation (KDE) only for visualization, to provide a smoothed view of the empirical distribution of the driver scores. KDE is a standard non-parametric density estimation method that places a smooth kernel around each observed sample and aggregates these kernels to estimate a continuous density. It does not affect training, evaluation, or the computation of the proposed driver; all quantitative analyses are performed on the original empirical scores.

ReplayTraining N→\toPmskrt
n1n11200.9240.039
n2n2117.670.8120.026

Table 5. Representative batch-level driver statistics for a good(n1n1)/bad(n2n2) replay comparison. Larger msk and rt indicate more mass in the high-driver region.

Let {di}i=1n\{d_i\}_{i=1}^{n} denote the driver values computed for the rollouts in a given subset (e.g., the N→\toP rollout group of a batch), where nn is the number of rollouts in that subset.

Mean-shift statistic (msk). The statistic msk summarizes the average magnitude of the driver values within the selected rollout group:

msk=1n∑i=1ndi.(10)\mathrm{msk} = \frac{1}{n}\sum_{i=1}^{n} d_i. \tag*{(10)}

In our analysis, larger msk indicates that the corresponding batch tends to allocate relatively more uncertainty to the early stage of generation.

Right-tail mass statistic (rt). The statistic rt summarizes the heaviness of the right tail of the driver distribution. Let τ\tau be a fixed upper-tail threshold (we set τ=1.95\tau= 1.95 in practice). We define

rt=1n∑i=1n1[di>τ],(11)\mathrm{rt} = \frac{1}{n}\sum_{i=1}^{n} \mathbf{1}[d_i > \tau], \tag*{(11)}

where 1[⋅]\mathbf{1}[\cdot] is the indicator function. Larger rt indicates that a larger fraction of rollouts fall into the high-driver region, i.e., the distribution has a heavier right tail.

Interpretation. These statistics are not intended to serve as calibrated predictors of future success or as exact estimates of future reward. In the main text, we use them only to support a weaker but more practically relevant claim: larger driver values, especially a heavier right tail, are enriched in rollout groups associated with stronger N→\toP tendency. This is the only property required by our method design, where the driver is used as a surrogate ordering signal rather than as a direct estimator of future pass rate.

Additional results

Generalization beyond Qwen2.5-Math-7B

Transition-level evidence. Starting from a shared Llama-3.1-8B-Instruct checkpoint, we run five stochastic GRPO continuations. Across the 696 prompts that are negative at the shared checkpoint, the mean N→\toP rate is (14.86±0.44)%(14.86 \pm0.44)\%

(mean ±\pm standard error across continuations). Thus, delayed first-solution discovery is not confined to Qwen2.5-Math-7B.

Training-level evidence. We apply NAPO using the same early-fraction window α=0.2\alpha= 0.2 selected on Qwen2.5-Math-7B (Figure 5), without Llama-specific tuning. Table 6 compares the resulting step-100 checkpoint against GRPO.

Method (Llama)Pass@k (%)
kk128256
AMC23
NAPO0240.5047.6857.8280.83
GRPO37.5547.2959.8980.00
AIME2025
NAPO020.060.110.4513.33
GRPO0.170.331.2520.00
AIME24
NAPO024.756.6711.9637.78
GRPO7.1110.4116.1533.33
Math-Beyond
NAPO025.907.329.6817.50
GRPO4.936.398.9216.21

Table 6. Pass@k comparison across datasets for Llama-3.1-8B-Instruct at temperature T=1.0T = 1.0. Values are percentages. The row corresponding to NAPO02 is highlighted in gray, and the best value between GRPO and NAPO02 within each dataset block and column is shown in bold. Due to limited computational resources, we evaluate the checkpoint at step 100.

The results are mixed, where NAPO performs best on Math-Beyond and high-budget AIME24, while degrading on AIME2025. Thus, the evaluation indicates that the driver window and resulting advantage magnitudes may require calibration across model families, while remaining robust to a certain degree.

Beyond mathematics: binary-verifiable code

Setup. We examine whether the transition-level entropy signal extends beyond mathematical reasoning. We train Qwen3-4B-Instruct (Yang et al., 2025a) on 512 distinct coding problems from LiveCodeBench v5 (Jain et al., [6]) using execution-based binary rewards, and evaluate on 100 held-out problems from LiveCodeBench v6. Five consecutive GRPO steps each visit all 512 training prompts and sample G=8G = 8 rollouts per prompt, yielding four adjacent transitions for the analysis below. All other settings follow § 3.1.

Code-domain entropy statistic. On the fixed sensitivity grid α∈{0.1,0.2,0.5}\alpha\in\{0.1, 0.2, 0.5\} evaluated on the same transitions, α=0.5\alpha= 0.5 gives the clearest observed separation, consistent with the different output structure and prompt formatting of coding tasks. For this experiment, msk denotes the prompt-level mean early-to-global plain-entropy ratio across the eight source-step rollouts. Unlike the NAPO training driver dearly−ppld_{\mathrm{early-ppl}}, this analysis-only statistic uses entropy directly rather than its exponential.

Pooling the four transitions gives 83 N→\toP and 550 N→\toN prompt–transition observations, whose mean msk values are 1.0985 and 1.0709, respec-

TTFP distribution across AIME2025, AMC23, Math-Beyond, and ALL

Figure 6. TTFP distribution

Cumulative solved number across AIME2025, AMC23, Math-Beyond, and ALL

Figure 7. Cumulative solved number

TransitionN→P / N→Nmsk N→Pmsk N→NΔmsk
1 → 220/1331.10441.0740+0.0304
2 → 320/1351.09971.0710+0.0287
3 → 418/1441.09681.0679+0.0289
4 → 525/1381.09411.0710+0.0231

Table 7. Transition-level entropy statistics for Qwen3-4B-Instruct on LiveCodeBench at α=0.5\alpha= 0.5. The second column reports the N→P and N→N prompt counts, and the remaining columns report their mean msk values and difference; positive differences indicate higher early relative entropy for N→P.

tively. The difference is +0.0276, with a 95% prompt-clustered bootstrap confidence interval of [0.0066, 0.0484]. The step-stratified AUC is 0.608, with a 95% confidence interval of [0.532, 0.681]. The difference has the same sign at all four transitions rather than being driven by a single favorable step. These results provide preliminary evidence for an entropy–exploration signal in the coding tasks.

Theoretical Derivation: Fisher Information and Negative-Augmented Policy Optimization

In this section, we provide a theoretical justification for the proposed driver-based weighting scheme. We demonstrate that penalizing negative samples with uniform advantage (standard GRPO) wastes optimization budget on samples that provide negligible Fisher Information. We then derive Negative-Augmented Policy Optimization (NAPO), where the advantage is proportional to the Fisher Information of the trajectory, effectively maximizing the information gain of the policy update.

Fisher Information of Negative Samples

Consider a prompt xx with a rollout group G={yi}i=1G\mathcal{G} = \{y_i\}_{i=1}^{G} where all rollouts are negative, i.e., r(x,yi)=−1r(x, y_i) = -1. In policy optimization, the Fisher Information Matrix (FIM) F(θ)\mathbf{F}(\theta) measures the curvature of the KL divergence and the informativeness of a sample gradient. For a specific trajectory yy, the contribution to the FIM is given by the outer product of the score function:

Fy(θ)=∇θlog⁡πθ(y∣x)∇θlog⁡πθ(y∣x)⊤.(12)\mathbf{F}_y(\theta) = \nabla_\theta\log\pi_\theta(y|x)\nabla_\theta\log\pi_\theta(y|x)^\top. \tag*{(12)}

A sample yy is considered informative for optimization if its gradient vector has a large magnitude in the parameter space, indicating a significant potential to reshape the policy distribution.

We analyze the Fisher Information at the token level. Let pt=πθ(⋅∣x,y<t)p_t = \pi_\theta(\cdot|x, y_{<t}) be the probability vector over the vocabulary V\mathcal{V} at step tt. Let Ft\mathbf{F}_t denote the Fisher Information contribution from position tt.

Lemma D.1 (Fisher Information and Entropy). Let Ht=−∑kpklog⁡pkH_t=-\sum_k p_k\log p_k be the entropy of the distribution at step tt. The trace of the token-level Fisher Information matrix satisfies:

Tr⁡(Ft)=1−∑k∈Vpk2.(13)\operatorname{Tr}(F_t)=1-\sum_{k\in V}p_k^2. \tag*{(13)}

Furthermore, for a fixed vocabulary size VV, the quantity 1−∑kpk21-\sum_k p_k^2 and the entropy HtH_t are positively associated: both are maximized by the uniform distribution (at V−1V\frac{V-1}{V} and log⁡V\log V, respectively) and both equal zero only for deterministic distributions. In particular, Ht→0H_t\to0 implies Tr⁡(Ft)→0\operatorname{Tr}(F_t)\to0.

Proof. The score function for a sampled token yty_t is ∇θlog⁡πθ(yt∣x,y<t)\nabla_\theta\log\pi_\theta(y_t|x,y_{<t}). Under the common softmax parameterization pk=ezk∑jezjp_k=\frac{e^{z_k}}{\sum_j e^{z_j}}, the gradient of the log-probability with respect to the logits zz is:

∇zlog⁡p(yt)=eyt−pt,(14)\nabla_z\log p(y_t)=e_{y_t}-p_t, \tag*{(14)}

where eyte_{y_t} is the one-hot vector for the sampled token. The Fisher Information matrix is the covariance of this gradient:

Ft=Eyt[(eyt−pt)(eyt−pt)⊤]F_t=\mathbb{E}_{y_t}\left[(e_{y_t}-p_t)(e_{y_t}-p_t)^\top\right]
=E[eyteyt⊤]−ptpt⊤.(15)=\mathbb{E}[e_{y_t}e_{y_t}^\top]-p_tp_t^\top. \tag*{(15)}

Since E[eyteyt⊤]=diag⁡(pt)\mathbb{E}[e_{y_t}e_{y_t}^\top]=\operatorname{diag}(p_t), the trace of the FIM is:

Tr⁡(Ft)=Tr⁡(diag⁡(pt))−Tr⁡(ptpt⊤)\operatorname{Tr}(F_t)=\operatorname{Tr}(\operatorname{diag}(p_t))-\operatorname{Tr}(p_tp_t^\top)
=∑kpk−∑kpk2=1−∑kpk2.(16)=\sum_k p_k-\sum_k p_k^2=1-\sum_k p_k^2. \tag*{(16)}

The quantity G:=1−∑kpk2G:=1-\sum_k p_k^2 is the Gini impurity (equivalently, the Tsallis entropy of order 2). Both GG and the Shannon entropy H=−∑kpklog⁡pkH=-\sum_k p_k\log p_k are Schur-concave functions of the probability vector: they are both uniquely maximized by the uniform distribution and both equal zero if and only if the distribution is deterministic. While GG and HH are not in general strict monotonic functions of each other for V≥3V\ge3, they are tightly coupled via the Rényi entropy ordering H1≥H2H_1\ge H_2, which gives:

1−∑kpk2≤1−e−H.1-\sum_k p_k^2\le1-e^{-H}.

In particular, H→0H\to0 forces Tr⁡(Ft)→0\operatorname{Tr}(F_t)\to0, and conversely the uniform distribution uniquely maximizes both quantities. Hence, higher-entropy distributions correspond, in a strong qualitative sense, to larger local Fisher information.

Analysis of Standard NSR on Unsolved Groups

Standard NSR assigns uniform negative advantages A(y)≈−1A(y)\approx-1 to all samples in an unsolved group. From the perspective of Fisher Information, this implies that the optimization update treats samples with zero information (confident errors) and high information (uncertain errors) equally.

Proposition D.2 (Gradient Saturation and Fisher Information). Consider a negative sample yy generated from a low-entropy (confident) policy state.

  1. Zero Information: If Ht→0H_t\to0, then Tr⁡(Ft)→0\operatorname{Tr}(F_t)\to0. The gradient ∇θlog⁡πθ(yt)\nabla_\theta\log\pi_\theta(y_t) vanishes.

  1. Saddle Points: Negative samples generated with high confidence correspond to “saddle points” or “dead ends” in the likelihood landscape. Uniform penalization (A=−1A=-1) attempts to push probability mass away from these points, but the vanishing Fisher Information implies that Δθ≈0\Delta\theta\approx0 regardless of the advantage magnitude.

Thus, uniform advantage assignment wastes gradient budget on samples that lie in low-information regions of the parameter space.

Proof. The gradient vector for the sampled token yty_t is gt=eyt−ptg_t=e_{y_t}-p_t, where eyte_{y_t} is the one-hot encoding of the sampled token and ptp_t is the probability vector πθ(⋅∣x,y<t)\pi_\theta(\cdot|x,y_{<t}). The squared norm of this gradient is:

∥gt∥22=∥eyt−pt∥22=(1−π(yt))2+∑v≠ytπ(v)2.(17)\begin{aligned} \lVert g_t\rVert_2^2=\lVert e_{y_t}-p_t\rVert_2^2 \\ &=(1-\pi(y_t))^2+\sum_{v\ne y_t}\pi(v)^2. \tag*{(17)} \end{aligned}

As π(yt)→1\pi(y_t)\to1 (confidence approaches certainty, entropy Ht→0H_t\to0), the first term approaches 0. Simultaneously, the mass of the probability vector concentrates on yty_t, forcing π(v)→0\pi(v)\to0 for all other vv. Thus, ∥gt∥2→0\lVert g_t\rVert_2\to0.

This implies that confidently generated tokens (which characterize the “dead-end” negative rollouts in § 3) contribute negligible gradient signal. Uniformly weighting such samples wastes optimization budget on saturated gradients that cannot effectively redirect the policy.

Derivation of NAPO

To maximize the efficiency of the update, we should allocate the “negative learning budget” to the samples whose penalization best reduces the total likelihood of failed trajectories. We formalize this as a constrained optimization over per-rollout weights and show that the solution naturally connects to Fisher Information and, through it, to trajectory entropy.

Exact formulation via the Gram matrix. Let G−={yi}i=1G\mathcal{G}^{-} = \{y_i\}_{i=1}^{G} be a fully negative rollout group for prompt xx. Write gy=∇θlog⁡πθ(y∣x)g_y = \nabla_\theta\log\pi_\theta(y \mid x) for the trajectory-level score vector, and define the Gram matrix G∈RG×G\mathbf{G} \in\mathbb{R}^{G \times G} with entries

Gij=gyiTgyj.(18)G_{ij} = g_{y_i}^{\mathsf{T}} g_{y_j}. \tag*{(18)}

Given a weight vector w=(w1,…,wG)Tw = (w_1, \ldots, w_G)^{\mathsf{T}}, the parameter update is δ=−η∑iwigyi\delta= -\eta\sum_i w_i g_{y_i}. The total first-order reduction in the log-likelihood of every negative rollout in the group is

R(w)=−∑j=1GgyjTδ=η1TGw.(19)R(w) = -\sum_{j=1}^{G} g_{y_j}^{\mathsf{T}}\delta= \eta\mathbf{1}^{\mathsf{T}}\mathbf{G}w. \tag*{(19)}

We seek the weight vector that maximizes R(w)R(w) subject to a variance budget ∥w∥2≤B\lVert w\rVert^2 \le B (which controls the variance of the stochastic gradient and ensures stable updates):

max⁡w 1TGws.t.∥w∥2≤B.(20)\max_w \ \mathbf{1}^{\mathsf{T}}\mathbf{G}w \quad\text{s.t.} \quad\lVert w\rVert^2 \le B. \tag*{(20)}

By the Cauchy–Schwarz inequality, 1TGw≤∥GT1∥∥w∥\mathbf{1}^{\mathsf{T}}\mathbf{G}w \le\lVert\mathbf{G}^{\mathsf{T}}\mathbf{1}\rVert\lVert w\rVert, with equality when w∝GT1w \propto\mathbf{G}^{\mathsf{T}}\mathbf{1}. Since G\mathbf{G} is symmetric, the exact solution is

w∗∝G1,w^* \propto\mathbf{G}\mathbf{1},
w∗(yi)∝[G1]i=∑j=1GgyiTgyj.(21)w^*(y_i) \propto[\mathbf{G}\mathbf{1}]_i = \sum_{j=1}^{G} g_{y_i}^{\mathsf{T}}g_{y_j}. \tag*{(21)}

Computing G1\mathbf{G}\mathbf{1} exactly is impractical (it requires full parameter-space inner products between all pairs of rollout gradients in a multibillion-parameter model). We therefore seek a tractable diagonal proxy and state a structural condition under which the ranking induced by G1\mathbf{G}\mathbf{1} coincides with the ranking induced by the diagonal alone.

Uniform-correlation condition. We decompose each entry of G1\mathbf{G}\mathbf{1}:

[G1]i=∥gyi∥2+∑j≠igyiTgyj=∥gyi∥2⏟=Tr⁡(Fyi)+∑j≠igyiTgyj.[\mathbf{G}\mathbf{1}]_i = \lVert g_{y_i}\rVert^2 + \sum_{j\ne i} g_{y_i}^{\mathsf{T}}g_{y_j} = \underbrace{\lVert g_{y_i}\rVert^2}_{=\operatorname{Tr}(\mathbf{F}_{y_i})} + \sum_{j\ne i} g_{y_i}^{\mathsf{T}}g_{y_j}.

The diagonal term is exactly the trajectory-level Fisher Information; the off-diagonal sum captures cross-sample interference. Directly computing the latter is as expensive as the full Gram matrix. We therefore introduce the following structural assumption.

Assumption D.3 (Uniform pairwise correlation). There exists a constant c≥0c \ge0 such that, for all distinct rollouts yi,yj∈G−y_i, y_j \in\mathcal{G}^{-},

gyiTgyj≈c∥gyi∥∥gyj∥.g_{y_i}^{\mathsf{T}}g_{y_j} \approx c \lVert g_{y_i}\rVert\lVert g_{y_j}\rVert.

It states that all same-prompt rollout gradients are roughly equally correlated with each other—a plausible condition because they share the same prompt, model architecture, and sampling distribution, so no single pair of rollouts is systematically more aligned than any other. The discriminating factor across rollouts is then their gradient magnitude, not their mutual orientation.

Theorem D.4 (Optimality of Fisher-Weighted Advantage). Under Assumption D.3, the optimal weight w∗(y)w^*(y) from Eq. (21) is a monotonically increasing function of Tr⁡(Fy)=∥gy∥2\operatorname{Tr}(\mathbf{F}_y) = \lVert g_y\rVert^2. By Lemma D.1 and the token-independence approximation, this quantity is positively associated with the trajectory entropy ∑tHt\sum_t H_t. Therefore, we define the Entropy-Weighted Advantage:

ANAPO(y)=−d(y)1∣G∣∑y′∈Gd(y′).(22)A_{\mathrm{NAPO}}(y) = -\frac{d(y)}{\frac{1}{|\mathcal{G}|}\sum_{y'\in\mathcal{G}} d(y')}. \tag*{(22)}

where d(y)d(y) is the driver score (normalized entropy) defined in Eq. (6) of the main text.

Proof. Step 1: Exact solution (Gram matrix). The constrained optimum of Eq. (20) is w∗∝G1w^* \propto\mathbf{G}\mathbf{1} as derived in Eq. (21).

Step 2: Reduction under uniform correlation. Let S=∑j=1G∥gyj∥S = \sum_{j=1}^{G} \lVert g_{y_j}\rVert denote the total gradient norm across the group. Under Assumption D.3:

[G1]i=∥gyi∥2+c∥gyi∥∑j≠i∥gyj∥=∥gyi∥2+c∥gyi∥(S−∥gyi∥)=(1−c)∥gyi∥2+cS∥gyi∥.(23)[\mathbf{G}\mathbf{1}]_i = \lVert g_{y_i}\rVert^2 + c\lVert g_{y_i}\rVert\sum_{j\ne i}\lVert g_{y_j}\rVert = \lVert g_{y_i}\rVert^2 + c\lVert g_{y_i}\rVert(S-\lVert g_{y_i}\rVert) = (1-c)\lVert g_{y_i}\rVert^2 + cS\lVert g_{y_i}\rVert. \tag*{(23)}

Define f(u)=(1−c)u2+cSuf(u) = (1-c)u^2 + cSu for u≥0u \ge0. Its derivative is f′(u)=2(1−c)u+cSf'(u) = 2(1-c)u + cS. Since 0≤c≤10 \le c \le1 (cosine similarity is bounded by one) and S≥0S \ge0, we have f′(u)≥0f'(u) \ge0 for all u≥0u \ge0. Hence ff is monotonically increasing in u=∥gyi∥u = \lVert g_{y_i}\rVert, and therefore also in ∥gyi∥2=Tr⁡(Fyi)\lVert g_{y_i}\rVert^2 = \operatorname{Tr}(\mathbf{F}_{y_i}).

The ranking of w∗(y)w^*(y) across the group thus coincides with the ranking of Tr⁡(Fy)\operatorname{Tr}(F_y).

Step 3: Connection to trajectory entropy. Since gy=∑t=1T∇θlog⁡πθ(yt∣x,y<t)g_y = \sum_{t=1}^{T} \nabla_\theta\log\pi_\theta(y_t \mid x, y_{<t}), under the approximation that token-level score vectors are approximately uncorrelated across positions we have:

∥gy∥2≈∑t=1T∥∇θlog⁡πθ(yt∣x,y<t)∥2.\lVert g_y\rVert^2 \approx\sum_{t=1}^{T} \left\lVert\nabla_\theta\log\pi_\theta(y_t \mid x, y_{<t}) \right\rVert^2.

Taking the expectation over the sampled token at each position and applying Lemma D.1:

E[∥gy∥2]≈∑t=1TTr⁡(Ft)=∑t=1T(1−∑kpt,k2),\mathbb{E}\left[\lVert g_y\rVert^2\right] \approx\sum_{t=1}^{T} \operatorname{Tr}(F_t) = \sum_{t=1}^{T} \left(1-\sum_k p_{t,k}^2\right),

which is positively associated with ∑tHt\sum_t H_t by Lemma D.1.

The driver d(y)d(y) in Eq. (6) of the main text is a normalized measure of early-stage entropy and thus serves as a tractable proxy for Tr⁡(Fy)\operatorname{Tr}(F_y). The advantage assignment ANAPOA_{\mathrm{NAPO}} therefore implements an approximation to the optimal Fisher-weighted update, prioritizing samples where the policy is “plastic” (high entropy) over samples where it is rigid (low entropy).

Remark D.5 (Interpretation as a diagonal preconditioner). The uniform-correlation reduction plays the role of a diagonal preconditioner for the Gram matrix. Just as diagonal preconditioning of a full Newton step retains the correct coordinate-wise scaling while ignoring off-diagonal curvature, ranking by Tr⁡(Fy)\operatorname{Tr}(F_y) retains the correct per-rollout scaling while treating cross-rollout interactions as uniform. When the uniform-correlation assumption is violated (e.g., if two rollouts share unusually long common prefixes), the proxy may over- or under-weight individual rollouts, but the overall ranking remains a principled first-order approximation.

Remark D.6 (Relation to orthogonality). Setting c=0c=0 recovers the special case of orthogonal score vectors, for which w∗(y)∝∥gy∥2w^*(y) \propto\lVert g_y\rVert^2 exactly. However, same-prompt rollouts typically share early tokens and pass through shared model parameters, making c>0c>0 the realistic regime. The uniform-correlation condition accommodates this shared structure while still yielding a tractable ranking criterion.

Remark D.7 (Practical implementation). In practice, we approximate the theoretically optimal weighting via monotonic min–max rescaling, which preserves the sample ordering, and retain the PPO/GRPO clipping mechanism for stable policy updates.

References

  1. [1]Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer. 2002. Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47(2–3):235–256.DOI
  2. [2]Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 39 others. 2021. Evaluating large language models trained on code. CoRR, abs/2107.03374.arxiv.org/abs/2107.03374
  3. [3]Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Xin Zhao, Zhenliang Zhang, and Furu Wei. 2026. Reasoning with exploration: An entropy perspective. Proceedings of the AAAI Conference on Artificial Intelligence, 40(36):30377–30385.arxiv.org/abs/2506.14758
  4. [4]Yangyi Fang, Jiaye Lin, Xiaoliang Fu, Cong Qin, Haolin Shi, Chaowen Hu, Lu Pan, Ke Zeng, and Xunliang Cai. 2026. How to allocate, how to learn? dynamic rollout allocation and advantage modulation for policy optimization. Preprint, arXiv:2602.19208.
  5. [5]Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, and 175 others. 2025. Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. Nature, 645(8081):633–638.DOI
  6. [6]Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. Live-codebench: Holistic and contamination free evaluation of large language models for code. Preprint, arXiv:2403.07974.
  7. [7]Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James Validad Miranda, Alisa Liu, Nouha Dziri, Xinxi Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Christopher Wilhelm, Luca Soldaini, and 4 others. 2025. Tulu 3: Pushing frontiers in open language model post-training. In Second Conference on Language Modeling.
  8. [8]Thanh-Long V. Le, Myeongho Jeon, Kim Vu, Viet Dac Lai, and Eunho Yang. 2026. No prompt left behind: Exploiting zero-variance prompts in LLM reinforcement learning via entropy-guided advantage shaping. In The Fourteenth International Conference on Learning Representations.
  9. [9]Xuefeng Li, Haoyang Zou, and Pengfei Liu. 2025. Limer: Less is more for rl scaling. Preprint, arXiv:2502.11886.arxiv.org/abs/2502.11886
  10. [10]Shichao Ma, Zhiyuan Ma, Ming Yang, Xiaofan Li, Xing Wu, Jintao Du, Yu Cheng, Weiqiang Wang, Qiliang Liu, Zhengyang Zhou, and Yang Wang. 2026. Tspo: Breaking the double homogenization dilemma in multi-turn search policy optimization. Preprint, arXiv:2601.22776.
  11. [12]Gongrui Nan, Siye Chen, Jing Huang, Mengyu Lu, Dexun Wang, Chunmei Xie, Weiqi Xiong, Xianzhuo Zeng, Qixuan Zhou, Yadong Li, and Xingzhong Xu. 2025. Ngrpo: Negative-enhanced group relative policy optimization. Preprint, arXiv:2509.18851.
  12. [13]Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 27730–27744. Curran Associates, Inc.
  13. [14]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. Preprint, arXiv:1707.06347.arxiv.org/abs/1707.06347
  14. [15]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. Preprint, arXiv:2402.03300.
  15. [16]Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanguang Peng, Haibin Lin, and Chuan Wu. 2025. Hybridflow: A flexible and efficient rlhf framework. In EuroSys, pages 1279–1297.
  16. [17]Richard S Sutton and Andrew G Barto. 1998. Reinforcement learning: An introduction, volume 1. MIT press.DOI
  17. [18]Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xiong-Hui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin. 2025. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning. In Advances in Neural Information Processing Systems, volume 38, pages 115452–115486. Curran Associates, Inc.
  18. [19]Ronald J. Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3–4):229–256.
  19. [20]Junkang Wu, Kexin Huang, Jiancan Wu, An Zhang, Xiang Wang, and Xiangnan He. 2026. Quantile advantage estimation: Stabilizing RLVR for LLM reasoning. In The Fourteenth International Conference on Learning Representations.

Paper details

vlnc:W_upya6dh5
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents