betaFeedback

WorksComputer Science

Not All Entropy Is Equal: Reward-Conditioned Entropy Allocation in Reinforcement Learning with Verifiable Rewards

Abstract

Policy entropy is widely used to diagnose exploration and collapse in reinforcement learning with verifiable rewards (RLVR), yet its magnitude does not consistently track reasoning performance across optimization objectives. We argue that this ambiguity arises because global token entropy conflates uncertainty in successful and failed trajectories, diversity across correct solution strategies, and variation among realizations of the same strategy. We introduce Reward-Conditioned Entropy Allocation (RCEA), a hierarchical diagnostic framework that separates complete-trajectory entropy from average token entropy, conditions uncertainty on verifier outcome and process validity, measures correct-solution-mode retention, and connects these population-level quantities to local covariance-driven updates. In controlled sweeps over six RLVR objectives at two model scales, entropy trajectories do not induce a stable ordering of reasoning accuracy. Token entropy is higher for verifier-failed than successful trajectories under both broad separately aggregated estimates and a common-question paired analysis, whose confidence intervals exclude equality at 10 of 12 displayed endpoints. A local covariance approximation predicts the direction of adjacent-checkpoint entropy change in 95.5% of nonfailed transitions; a reward-stratified endpoint proxy further shows that positive chosen-token covariance is more tail-concentrated within failed prefixes. Every reportable trained endpoint exhibits some contraction in independently audited correct-solution-mode breadth, whose positive association with large-kk coverage remains exploratory. These results show that entropy collapse should be diagnosed by which uncertainty is removed, rather than by global entropy alone.

Introduction

Reinforcement learning with verifiable rewards (RLVR) has become a central approach for improving the mathematical and programmatic reasoning of large language models. Given a problem, the policy generates a complete reasoning trajectory, while a rule-based verifier assigns a binary reward according to the final answer [22, 13]. This sparse but reliable supervision has enabled substantial gains without process annotations. However, when the policy becomes more certain during training, has it learned to avoid errors, or has it merely lost the ability to explore alternative correct solutions?

Recent work suggests that this concern is not merely hypothetical. At the sequence level, Yue et al. [10] find that current RLVR primarily improves sampling efficiency over reasoning paths already accessible to the base model and can reduce large-kk problem coverage, narrowing the model’s operational reasoning boundary. Average token-level policy entropy is commonly used to diagnose this concentration, but recent analyses reveal that the underlying changes are highly non-uniform. Wang et al. [6] identify a small high-entropy minority that acts as critical forks in reasoning trajectories and accounts for most policy-gradient gains, while Cui et al. [12] trace entropy reduction to a small tail of token updates with large positive covariance contributions. Together, these findings suggest that RLVR reallocates probability mass within a pretrained reasoning prior through highly non-uniform local updates. It remains unresolved whether the uncertainty being removed is that of erroneous exploration, redundant realizations, or distinct correct solution strategies.

Figure 1 provides a compact example of the entropy–performance inconsistency. GTPO Simoni et al. [1] drives entropy close to zero by 25% progress while retaining nontrivial MATH500 accuracy. Among the covariance-aware objectives Cui et al. [12], Clip-Cov finishes with one of the lowest entropy values and one of the highest accuracies, whereas KL-Cov preserves more late-stage entropy without attaining the best accuracy. Our full two-scale evaluation below yields the same qualitative conclusion: endpoint leaders change with model scale and metric, and their performance ordering does not follow their entropy ordering. Higher global entropy is therefore neither necessary nor sufficient for better RLVR. This challenges the premise that the magnitude of global entropy is itself a reliable measure of useful exploration.

Two-panel line plots showing training entropy and MATH500 accuracy across training progress

Figure 1. Global entropy does not monotonically track reasoning performance in the formal Qwen2.5-3B sweep.

The contradiction arises from conflating distinct quantities within a single scalar. Average token entropy, the statistic commonly logged during training, mixes successful and failed trajectories, reasoning and formatting tokens, and length effects. It is not the entropy of the complete trajectory distribution. Even when restricted to successful trajectories, diversity can reflect either genuinely different mathematical strategies or merely alternative realizations of the same strategy, a distinction formalized by semantic-uncertainty methods [17, 14]. Consequently, a decrease in global entropy may signal the elimination of errors, the removal of redundant expression, or the collapse of valuable correct-solution modes; an increase may preserve alternative strategies or merely retain failure-region uncertainty. Global entropy alone cannot distinguish among these cases.

Our central claim is that the magnitude of entropy is not the relevant diagnostic; what matters is how uncertainty is allocated across reward outcomes and semantic solution modes. We formalize this as Reward-Conditioned Entropy Allocation (RCEA), a three-level diagnostic hierarchy. First, we decompose complete-trajectory and average token entropy by eventual verifier outcome, separating success-region from failure-region uncertainty. Second, within successful trajectories, we group generations by semantic solution strategy, distinguishing diversity across strategies from within-strategy realization variability. Third, we connect these components to local covariance-driven entropy updates, which predict the direction of entropy change while reward conditioning identifies the affected outcome stratum. We evaluate semantic-mode retention as a separate diagnostic and do not estimate mode-stratified covariance. We further introduce a stricter notion of solution-mode collapse, defined through contraction of the conditional distribution over semantically distinct successful modes, and reserve the term harmful collapse for mode contraction that coincides with reduced large-kk coverage or out-of-distribution robustness. We evaluate RCEA through controlled comparisons of objectives that induce different entropy trajectories, examining whether independently audited successful-mode retention is exploratorily associated with large-kk coverage.

Our contributions are threefold:

  • We identify and systematically study an entropy–performance inconsistency in RLVR: substantially different global entropy trajectories do not induce a consistent ordering in reasoning accuracy.

  • We develop RCEA, a hierarchical diagnostic framework that distinguishes complete-trajectory entropy from logged average token entropy, decomposes uncertainty across verifier outcomes and semantic solution modes, and defines solution-mode collapse through contraction of the conditional successful-mode distribution.

  • We operationalize the framework with fixed-probe estimators, effective mode counts, and reward-stratified covariance attribution, and use them to study greedy performance and, exploratorily, large-kk coverage beyond what global entropy reveals.

RLVR uses rule-based outcome verification, and GRPO converts sparse outcomes into group-relative advantages without a learned critic Shao et al. [22]; DeepSeek-AI [13]; Hu et al. (2025); Zeng et al. [11]. DAPO refines this pipeline through dynamic sampling, asymmetric clipping, token-level loss normalization, and overlong-sample control (Yu et al. 2025). Because outcome-level acceptance certifies the extracted answer rather than every intermediate step, process-supervision work instead evaluates the reasoning steps themselves Uesato et al. [3]; Lightman et al. [19]. Complementing optimizer design, RCEA keeps verifier success and process validity distinct and diagnoses how these objectives redistribute uncertainty.

Large-kk evaluations suggest that RLVR mostly sharpens paths accessible to the base policy and may narrow problem coverage Yue et al. [10]. At the token level, a small high-entropy minority marks reasoning forks and accounts for much of the policy-gradient gain [6, 4, 5, 20]. EP-GRPO gates outcome advantages by predictive entropy and adds an outcome-anchored policy-reference log-probability signal for implicit process guidance (Yu et al. 2026). Covariance analysis attributes entropy reduction to a small tail of token updates; Clip-Cov and KL-Cov suppress these updates Cui et al. [12], whereas GTPO controls entropy growth from negative updates to confident trajectories Simoni et al. [1]. Collectively, these studies connect entropy dynamics to reasoning coverage, influential token updates, and entropy-aware optimization, but do not decompose the affected uncertainty by verifier outcome or determine whether successful trajectories instantiate the same semantic strategy. Self-consistency benefits from diverse reasoning paths Wang et al. [7], while semantic-uncertainty methods group meaning-equivalent generations [17, 14]. Their primary object is answer-level semantic uncertainty rather than a problem-local partition of the mathematical operations used by successful derivations.

Reward-Conditioned Entropy Allocation

RCEA separates outcome uncertainty, diversity within successful and failed regions, process validity within verifier successes, and diversity among successful semantic strategies. We then connect these global quantities to the local covariance mechanism that changes token entropy. Together, these levels ask where an observed entropy decrease occurs and what kind of behavior it removes. Figure 2 illustrates the resulting distinction.

Figure 2. Conceptual overview of RCEA. The same direction of global entropy change can reflect error removal with successful-mode preservation or correct-solution mode contraction. Here RR is verifier reward, VV is process validity, and ZZ is a semantic solution mode; the panels illustrate limiting cases rather than empirically assigned collapse labels.

Preliminary

Let xx be a reasoning problem and Y=(Y1,…,YT)∼πθ(⋅∣x)Y = (Y_1,\ldots,Y_T) \sim\pi_\theta(\cdot\mid x) a complete autoregressive trajectory, including its final answer and end-of-sequence token. An answer extractor AxA_x and rule-based verifier VxV_x define

R=fx(Y):=Vx(Ax(Y))∈{0,1}.(1)R = f_x(Y) := V_x(A_x(Y)) \in\{0,1\}. \tag*{(1)}

Both are deterministic, and the verifier evaluates only the extracted final answer, thus H(R∣Y,x)=0H(R \mid Y,x) = 0. Importantly, R=1R = 1 certifies verifier acceptance, not the logical validity of every intermediate step. For verifier-successful trajectories, let V=ux(Y)∈{0,1}V = u_x(Y) \in\{0,1\} denote process validity and v+(x)=Pθ(V=1∣R=1,x)v_+(x) = P_\theta(V = 1 \mid R = 1,x). We model VV as a deterministic target property of the trajectory; experimentally, it is approximated by a frozen judge audited against human labels. We therefore use successful and failed for reward outcomes, reserving valid strategy for modes additionally satisfying V=1V = 1.

Let p+(x)=Pθ(R=1∣x)p_+(x) = P_\theta(R = 1 \mid x) and p−(x)=1−p+(x)p_-(x) = 1 - p_+(x). We distinguish two quantities that are often both called policy entropy. The sequence entropy can be written as

Hθ(Y∣x)=−∑yπθ(y∣x)log⁡πθ(y∣x).(2)H_\theta(Y \mid x) = -\sum_y \pi_\theta(y \mid x)\log\pi_\theta(y \mid x). \tag*{(2)}

It measures uncertainty over complete trajectories. By contrast, training systems usually report a length-normalized average of local next-token entropies.

Sequence-Level Reward Decomposition

Proposition 1 (exact reward decomposition). For the deterministic verifier in Eq. (1),

Hθ(Y∣x)=Hθ(R∣x)+p+(x)Hθ(Y∣R=1,x)+p−(x)Hθ(Y∣R=0,x).(3)H_\theta(Y \mid x) = H_\theta(R \mid x) + p_+(x)H_\theta(Y \mid R = 1,x) + p_-(x)H_\theta(Y \mid R = 0,x). \tag*{(3)}

To see this, expand the joint entropy in two ways:

H(Y,R∣x)=H(Y∣x)+H(R∣Y,x)=H(R∣x)+H(Y∣R,x).\begin{aligned} H(Y,R \mid x) &= H(Y \mid x) + H(R \mid Y,x)\\ &= H(R \mid x) + H(Y \mid R,x). \end{aligned}

The first conditional term vanishes because R=fx(Y)R = f_x(Y), and expanding the second over the two reward values gives Eq. (3). The first term is binary reward-outcome entropy,

H(R∣x)=hb(p+)=−p+log⁡p+−p−log⁡p−.H(R \mid x) = h_b(p_+) = -p_+\log p_+ - p_-\log p_-.

It quantifies uncertainty over success versus failure. It is maximal at p+=1/2p_+ = 1/2 and approaches zero for both certain success and certain failure; consequently, its decrease is not intrinsically desirable. The remaining terms describe diversity within the successful and failed regions. Equation (3) already shows why total entropy is ambiguous: the same decrease can result from resolving reward uncertainty, eliminating failed trajectories, or concentrating the successful distribution. When pr(x)=0p_r(x) = 0, the weighted contribution is zero by the usual convention 0log⁡0=00\log0 = 0; reliable estimation of the conditional term requires sufficient observations of outcome rr.

Reward-Conditioned Token Entropy

For a sampled trajectory yy, define the local entropy at prefix st=(x,y<t)s_t = (x,y_{<t}) and its trajectory-normalized average as

hθ(st)=H(πθ(⋅∣st)),h_\theta(s_t) = H(\pi_\theta(\cdot\mid s_t)),
hˉθ(y,x)=1T(y)∑t=1T(y)hθ(x,y<t).(4)\bar h_\theta(y,x) = \frac{1}{T(y)}\sum_{t=1}^{T(y)} h_\theta(x,y_{<t}). \tag*{(4)}

The on-policy average token entropy is Htok(x)=EY∼πθ[hˉθ(Y,x)]\mathcal{H}_{\mathrm{tok}}(x) = \mathbb{E}_{Y\sim\pi_\theta}[\bar h_\theta(Y,x)]. By the law of total expectation,

Htok(x)=p+Htok,+(x)+p−Htok,−(x),(5)\mathcal{H}_{\mathrm{tok}}(x) = p_+\mathcal{H}_{\mathrm{tok},+}(x) + p_-\mathcal{H}_{\mathrm{tok},-}(x), \tag*{(5)}

where Htok,r(x)=E[hˉθ(Y,x)∣R=r,x]\mathcal{H}_{\mathrm{tok},r}(x) = \mathbb{E}[\bar h_\theta(Y,x) \mid R = r,x]. Unlike Eq. (3), this decomposition has no H(R∣x)H(R \mid x) term, as it partitions an expected local statistic rather than the Shannon entropy of the trajectory distribution.

Semantic Solution Modes and Collapse

Even after process-validity filtering, successful trajectory entropy conflates different mathematical strategies with different realizations of the same strategy. Let Z=gx(Y)Z = g_x(Y) be a deterministic semantic partition of trajectories satisfying R=1,V=1R = 1,V = 1, where two trajectories share a mode when they use the same core solution strategy. Write Cx=(R=1,V=1,x)C_x = (R = 1,V = 1,x). Because ZZ is a function of YY, another application of the chain rule gives

H(Y∣Cx)=H(Z∣Cx)+EZ∣Cx[H(Y∣Z,Cx)].(6)\begin{aligned} H(Y \mid C_x) &= H(Z \mid C_x)\\ &\quad+ \mathbb{E}_{Z\mid C_x}[H(Y \mid Z,C_x)]. \tag*{(6)} \end{aligned}

The first term is valid solution-mode entropy; the second is within-mode realization entropy. If gxg_x is implemented by a stochastic classifier, the right-hand side of Eq. (6) additionally subtracts H(Z∣Y,Cx)H(Z \mid Y,C_x).

Combining Eqs. (3) and (6) produces the hierarchy

H(Y∣x)=hb(p+)+p−H(Y∣R=0,x)+p+{hb(v+)+v+H(Z∣R=1,V=1,x)+v+EZ[H(Y∣Z,R=1,V=1,x)]+(1−v+)H(Y∣R=1,V=0,x)}.(7)\begin{aligned} H(Y \mid x) &= h_b(p_+)\\ &\quad+ p_-H(Y \mid R = 0,x)\\ &\quad+ p_+\{h_b(v_+)\\ &\qquad+ v_+H(Z \mid R = 1,V = 1,x)\\ &\qquad+ v_+\mathbb{E}_Z[H(Y \mid Z,R = 1,V = 1,x)]\\ &\qquad+ (1-v_+)H(Y \mid R = 1,V = 0,x)\}. \tag*{(7)} \end{aligned}

It separates reward-outcome uncertainty, failed-region diversity, validity uncertainty within verifier successes, valid strategy diversity, within-mode realization diversity, and diversity among verifier-successful but process-invalid trajectories. This is an exact decomposition under the stated deterministic partitions, not an assertion that every component should be maximized.

Definition 1 (correct-solution mode collapse). Let

qθ,t(z∣x)=Pπθt(Z=z∣R=1,V=1,x)q_{\theta,t}(z \mid x)=P_{\pi_{\theta_t}}(Z=z \mid R=1,V=1,x)

denote the successful mode distribution at training checkpoint tt. Its effective number of modes is

Neff,t+(x)=exp⁡(Hπθt(Z∣R=1,V=1,x)).(8)N_{\mathrm{eff},t}^{+}(x)=\exp\left(H_{\pi_{\theta_t}}(Z \mid R=1,V=1,x)\right). \tag*{(8)}

We say that correct-solution mode collapse occurs when training concentrates qθ,tq_{\theta,t} onto a substantially smaller effective set of semantically distinct, valid modes, suppressing modes that had non-negligible probability under the initial policy. Complete collapse is the limiting case qθ,t→δz⋆q_{\theta,t}\to\delta_{z^\star}, for which H(Z∣R=1,V=1,x)→0H(Z \mid R=1,V=1,x)\to0 and Neff,t+(x)→1N_{\mathrm{eff},t}^{+}(x)\to1. A finite decrease is reported as mode contraction and called partial collapse only relative to a predeclared retention threshold. This definition concerns effective probability concentration, not literal support removal, because a softmax policy generally assigns nonzero probability to nominally reachable trajectories.

The definition distinguishes three superficially similar events. Increasing p+p_{+} by moving mass from failed to successful trajectories is successful learning, not collapse. Rejecting process-invalid verifier successes changes v+v_{+}, not the conditional mode distribution. Reducing H(Y∣Z,R=1,V=1,x)H(Y \mid Z,R=1,V=1,x) while preserving H(Z∣R=1,V=1,x)H(Z \mid R=1,V=1,x) is realization-level standardization. Only contraction of the conditional semantic distribution removes distinct valid strategies.

We summarize contraction by effective-mode retention relative to the initial policy,

Ret⁡t+(x)=Neff,t+(x)Neff,0+(x).(9)\operatorname{Ret}_{t}^{+}(x)=\frac{N_{\mathrm{eff},t}^{+}(x)}{N_{\mathrm{eff},0}^{+}(x)}. \tag*{(9)}

From Local Covariance to Entropy Allocation

The preceding decompositions describe where uncertainty resides, while covariance analysis explains how an optimizer changes local token entropy. For a prefix ss, let z(v)z(v) be the logit of token vv and Δz(v)\Delta z(v) a small update. A first-order expansion gives [12]

Δhθ(s)≈−Cov⁡v∼πθ(⋅∣s)(log⁡πθ(v∣s),Δz(v)).(10)\Delta h_{\theta}(s)\approx-\operatorname{Cov}_{v\sim\pi_{\theta}(\cdot\mid s)}\left(\log\pi_{\theta}(v\mid s),\Delta z(v)\right). \tag*{(10)}

Positive covariance therefore decreases local entropy. Under simplified policy-gradient and natural-gradient updates, the corresponding terms take the forms −η Cov⁡(log⁡π,πA)-\eta\,\operatorname{Cov}(\log\pi,\pi A) and −η Cov⁡(log⁡π,A)-\eta\,\operatorname{Cov}(\log\pi,A), respectively. Large positive contributions are not restricted to high-probability, positive-advantage tokens: low-probability tokens receiving relatively negative updates can also sharpen the distribution. Covariance-aware objectives act on this local mechanism by clipping or regularizing its extreme contributions.

Equation (10), however, does not determine whether the resulting concentration is useful. It does not identify whether prefix ss belongs to an eventual success or failure, nor whether two successful prefixes instantiate the same semantic strategy. Reward conditioning can group prefix-level entropy and covariance statistics by eventual verifier outcome, while the mode decomposition can in principle separate between-mode from within-mode variation. Our experiments test the reward-stratified covariance bridge and mode retention as separate diagnostics; they do not estimate within-mode realization entropy or mode-stratified covariance. Covariance therefore predicts the direction of local entropy change, while conditioning specifies progressively finer questions about its allocation.

This bridge motivates directly testable hypotheses. Lower global entropy can be beneficial when failed-region or within-mode entropy falls while successful mode entropy is retained. Higher global entropy can be unhelpful when it primarily preserves failed-region uncertainty. Harmful collapse is the narrower case in which successful mode entropy and effective mode count contract together, potentially reducing large-kk coverage or out-of-distribution robustness.

Experiments

The RCEA decomposes global entropy into reward-conditioned, semantic-mode, and covariance-attributed components. We now operationalize this hierarchy to ask four questions.

Experimental Setup

Training Details. We train Qwen2.5-3B and Qwen2.5-7B [21] for one epoch and one seed on the same 8,000-example subset of Skywork-OR1-RL-Data [2]. Each prompt receives eight on-policy completions at temperature 1.0 and top-p=0.95p = 0.95; reward is determined solely by the normalized equivalence of the final boxed answer. All objectives share the same LoRA (Hu et al. 2022), optimizer, and sampling setup within each scale, and training curves use normalized one-epoch progress.

Baseline Methods. We compare the untrained base policy with six representative RLVR objectives: GRPO [22], DAPO (Yu et al. 2025), the covariance-aware Clip-Cov and KL-Cov objectives [12], EP-GRPO (Yu et al. 2026), and GTPO [1]. All non-objective-specific choices are held fixed within each model scale.

Evaluation Strategies. We report greedy accuracy on AMC23, AIME24, AIME25, MATH500 [19], and Minerva Math [18], together with MATH500 pass@64 using the standard unbiased estimator (Chen et al. 2021). A 200-prompt fixed probe sampled at the base policy and four training milestones supports reward-conditioned entropy estimates and correct-solution-mode retention. We report both broad outcome-wise aggregates and a post-hoc common-question comparison restricted to prompts with at least eight samples from each reward outcome. The frozen semantic partition passes an independent adjudicated human audit. We additionally evaluate the exact full-vocabulary covariance prediction across adjacent checkpoints and a reward-stratified endpoint proxy.

Main Results

RQ1: Does Global Entropy Reliably Indicate Reasoning Performance?

Table 1 and Figure 3 extend the comparison to benchmark metrics and complete training traces at both scales. Table 1 uses final checkpoints, and no endpoint ranking is stable across scale and metric. Clip-Cov has the highest 3B five-benchmark macro accuracy (27.75%), whereas KL-Cov leads at 7B (35.72%). DAPO has the highest 7B MATH500 greedy accuracy (77.2%), but EP-GRPO and KL-Cov have higher pass@64 (94.8% and 94.0% versus 92.6%). These rankings do not follow the entropy ordering in Figure 3. Most sharply, GTPO drives training entropy close to zero early, yet retains 61.0% and 71.0% MATH500 greedy accuracy at 3B and 7B. Conversely, preserving more entropy does not uniformly improve either greedy accuracy or macro accuracy. Global entropy magnitude is therefore an ambiguous diagnostic rather than an objective to maximize.

MethodAIME’24AIME’25MATH 500AMC’23MinervaAveragePass@64Mode Retention
Qwen2.5-3B13.33.348.635.015.023.0592.8100.0
+ GRPO ([22])3.30.063.835.020.024.4393.094.2
+ DAPO ((Yu et al. 2025))10.00.064.427.520.024.3891.291.1
+ Clip-Cov ([12])10.03.365.442.517.527.7593.296.0
+ KL-Cov ([12])3.33.362.632.515.023.3593.694.8
+ EP-GRPO ((Yu et al. 2026))0.00.061.437.515.022.7893.299.9
+ GTPO ([1])6.70.061.037.515.024.0385.683.7
Qwen2.5-7B10.06.736.025.020.019.5394.8100.0
+ GRPO ([22])10.06.771.845.025.031.6993.089.6
+ DAPO ((Yu et al. 2025))13.36.777.245.025.033.4492.688.8
+ Clip-Cov ([12])6.73.372.842.517.528.56–93.5
+ KL-Cov ([12])16.713.368.657.522.535.7294.092.3
+ EP-GRPO ((Yu et al. 2026))20.06.773.237.517.530.9794.896.2
+ GTPO ([1])16.76.771.047.515.031.3789.685.9

Table 1. Reasoning accuracy, MATH 500 pass@64, and audited correct-solution-mode retention (%). Macro Average weights the five benchmarks equally. Bold and underline mark the best and second-best reportable results within each scale. Clip-Cov-7B: step 3,600 (pre-failure).

Figure 3. Training token-entropy trajectories. Thin lines are raw traces, thick lines are rolling summaries.

Notably, Clip-Cov-7B is a distinct training failure: its MATH500 accuracy drops from 60.8% at step 3,600 to 0.2% at step 4,000 and remains near zero thereafter. This transition coincides with completion lengths saturating the token limit and the boxed-answer rate collapsing from 95.2% to 1.0%. During the surrounding failure window, most updates activate neither reward-variance learning signals nor the covariance mask, suggesting a self-reinforcing near-all-failure regime in which the sparse Clip-Cov intervention provides little recovery signal.

RQ2: How Is Entropy Allocated Across Successful and Failed Trajectories?

Figure 4 summarizes the broad reward-stratified ordering through ρH=Htok,−/Htok,+\rho_{\mathrm{H}} = \mathcal{H}_{\mathrm{tok},-}/\mathcal{H}_{\mathrm{tok},+}. All 12 displayed endpoints have ρH>1\rho_{\mathrm{H}} > 1. However, successful and failed coverage ranges from 40.5–53.0% and 72.5–85.5%, respectively, so the two aggregates need not use the same problems. Near-zero reconstruction residuals verify the decomposition’s accounting but not this substantive ordering.

Figure 4. Failed-to-successful token-entropy ratios. Values above one indicate higher entropy in verifier-failed trajectories; outcome strata are aggregated separately.

To control for prompt-set composition, Figure 5 compares both outcomes within the same 40–57 eligible questions per endpoint. All 12 paired estimates exceed one, and bootstrap intervals exclude equality for 10; only GTPO-3B and GTPO-7B overlap one. The broad ordering therefore persists after controlling for question identity, although the paired result does not cover prompts dominated by a single outcome. Together, the analyses show that verifier-failed trajectories generally retain more token-level uncertainty than successful trajectories, motivating future investigation of process-aware rewards.

Figure 5. Common-question paired token-entropy ratios. Points show geometric means and bars denote paired-bootstrap 95% confidence intervals.

RQ3: Is Correct-Solution-Mode Retention Associated with Large-kk Coverage?

Table 1 shows that audited effective-mode retention ranges from 83.7% to 99.9% at 3B and from 85.9% to 96.2% at 7B. All algorithms exhibited a reduction in the patterns of correct solutions. This indicates that, at the current stage, RLVR algorithms narrow the scope of reasoning. Across trained methods with reportable coverage, its association with MATH500 pass@64 is positive (Spearman ρ=0.812\rho= 0.812, n=6n = 6, at 3B; ρ=1.000\rho= 1.000, n=5n = 5, at 7B). EP-GRPO has the highest retention at both scales but not the highest greedy accuracy, while DAPO-7B retains fewer modes than GRPO-7B yet has substantially higher greedy MATH500 accuracy (77.2% versus 71.8%). Thus, the association is consistent with strategy breadth benefiting repeated-sampling coverage.

RQ4: Does Local Covariance Explain Entropy Change and Its Reward-Stratified Concentration?

Predicting entropy change. Across the 44 adjacent-checkpoint transitions from 11 nonfailed runs, the covariance approximation predicts the sign of the observed entropy change in 42 cases (95.5%), with Spearman ρ=0.986\rho= 0.986, Pearson r=0.968r = 0.968, slope 0.993, and R2=0.937R^{2} = 0.937 (Figure 6). The median relative error is 12.7%. The points closely follow the identity line y=xy = x, indicating that the approximation captures not only the direction but also the magnitude of the observed entropy change. This near-identity fit provides strong consistency evidence for the local mechanism in Eq. (10).

Figure 6. Predicted versus observed entropy changes across adjacent checkpoints. y=xy=x indicates perfect prediction.

Locating reward-stratified tail concentration. Table 2 shows the same separation at both scales and for every objective. Across trained, nonfailed methods, the q99 tail accounts for 4.6–18.2% of positive chosen-token covariance within successful-prefix strata but 70.4–99.3% within failed-prefix strata, gaps of 65.8–83.7 percentage points. Because cutoffs are estimated separately by reward stratum, the values compare concentration rather than absolute covariance mass. Ties are retained; for GTPO failed-prefix strata they select roughly 58–59% of tokens, so this is not a strict top-1% attribution. Figure 6 tests how covariance predicts entropy change, whereas Table 2 identifies where its positive endpoint contribution is most tail-concentrated. Together they support the local-to-reward-stratified diagnostic bridge.

MethodQwen2.5-3B successful → failedQwen2.5-7B successful → failed
Base2.4 → 61.2<br>+58.8 pp2.7 → 43.2<br>+40.4 pp
GRPO9.5 → 82.7<br>+73.2 pp9.9 → 91.8<br>+81.9 pp
DAPO12.5 → 83.9<br>+71.4 pp10.8 → 93.2<br>+82.4 pp
Clip-Cov10.0 → 84.5<br>+74.5 pp1.5 → 83.0<br>+81.4 pp
KL-Cov4.6 → 70.4<br>+65.8 pp10.2 → 86.2<br>+76.0 pp
EP-GRPO7.1 → 82.9<br>+75.8 pp7.6 → 81.3<br>+73.6 pp
GTPO18.2 → 98.6<br>+80.4 pp15.6 → 99.3<br>+83.7 pp

Table 2. Within-stratum q99 share of positive chosen-token covariance (%), successful → failed.

Discussion

Together, the four RQs explain why global entropy is an ambiguous RLVR diagnostic. RQ1 shows that it does not stably order reasoning performance, while RQ2 localizes part of this ambiguity to different uncertainty levels across verifier outcomes. RQ4 links the observed changes to local covariance and its concentration in failed prefixes; RQ3 asks the distinct semantic question of whether correct strategies are retained. Covariance thus explains how a distribution sharpens locally, reward conditioning identifies where this occurs, and mode analysis asks what behavior is removed.

Entropy should be neither universally maximized nor minimized. A decrease may remove failed-region uncertainty or redundant realizations, but becomes harmful when it eliminates distinct correct strategies. The Clip-Cov-7B failure likewise shows why entropy must be interpreted alongside reward variance, response length, and termination behavior. Although diagnostic rather than causal, this pattern motivates adaptive controls based on outcome composition instead of a single global entropy target. RCEA accordingly pairs global entropy with reward-conditioned summaries, length diagnostics, audited mode retention, covariance attribution.

Conclusion and Limitations

Global token entropy conflates uncertainty across reward outcomes and solution strategies, and therefore does not consistently order RLVR reasoning performance. We introduce RCEA to resolve this ambiguity through trajectory-level decomposition, reward conditioning, independently audited solution modes, and local covariance attribution. Across six

objectives and two model scales, failed trajectories retain higher token entropy, covariance closely predicts local entropy change and is most tail-concentrated in failed prefixes, and all reportable trained endpoints show estimated contraction in correct-solution-mode breadth. The exploratory association between mode retention and pass@64 further suggests that strategy breadth may matter more for repeated-sampling coverage than for greedy accuracy. RCEA thus replaces a one-dimensional collapse signal with a structured diagnosis of where uncertainty changes and whether correct reasoning diversity is preserved.

The study uses a single seed, one model family at two scales. Process validity remains model-judged, the paired and mode analyses cover restricted prompt sets, and we do not estimate within-mode realization entropy or mode-stratified covariance. The finite-difference covariance analysis is diagnostic, leaving causal validation to future work.

References

  1. [1]Simoni, M.; Fontana, A.; Rossolini, G.; Saracino, A.; and Mori, P. 2025. GTPO: Stabilizing Group Relative Policy Optimization via Gradient and Entropy Control. arXiv:2508.03772.arxiv.org/abs/2508.03772
  2. [2]Skywork AI. 2025. Skywork-OR1-RL-Data. Hugging Face dataset.
  3. [3]Uesato, J.; Kushman, N.; Kumar, R.; Song, F.; Siegel, N.; Wang, L.; Creswell, A.; Irving, G.; and Higgins, I. 2022. Solving math word problems with process- and outcome-based feedback. arXiv:2211.14275.arxiv.org/abs/2211.14275
  4. [4]Vassoyan, J.; Beau, N.; and Plaud, R. 2025. Ignore the KL Penalty! Boosting Exploration on Critical Tokens to Enhance RL Fine-Tuning. arXiv:2502.06533.arxiv.org/abs/2502.06533
  5. [5]Wang, J.; Liu, R.; Zhang, F.; Li, X.; Zhou, G.; et al. 2025a. Stabilizing Knowledge, Promoting Reasoning: Dual-Token Constraints for RLVR. arXiv:2507.15778.arxiv.org/abs/2507.15778
  6. [6]Wang, S.; Yu, L.; Gao, C.; Zheng, C.; Liu, S.; Lu, R.; Dang, K.; Chen, X.; Yang, J.; Zhang, Z.; et al. 2025b. Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning. arXiv:2506.01939.arxiv.org/abs/2506.01939
  7. [7]Wang, X.; Wei, J.; Schuurmans, D.; Le, Q. V.; Chi, E. H.; Narang, S.; Chowdhery, A.; and Zhou, D. 2023. Self-Consistency Improves Chain-of-Thought Reasoning in Language Models. In The Eleventh International Conference on Learning Representations.
  8. [10]Yue, Y.; Chen, Z.; Lu, R.; Zhao, A.; Wang, Z.; Yue, Y.; Song, S.; and Huang, G. 2025. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? arXiv:2504.13837.arxiv.org/abs/2504.13837
  9. [11]Zeng, W.; Huang, Y.; Liu, Q.; Liu, W.; He, K.; Ma, Z.; and He, J. 2025. SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild. arXiv:2503.18892.arxiv.org/abs/2503.18892
  10. [12]Cui, G.; Zhang, Y.; Chen, J.; Yuan, L.; Wang, Z.; Zuo, Y.; Li, H.; Fan, Y.; Chen, H.; Chen, W.; et al. 2025. The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models. arXiv:2505.22617.arxiv.org/abs/2505.22617
  11. [13]DeepSeek-AI. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948.DOI
  12. [14]Farquhar, S.; Kossen, J.; Kuhn, L.; and Gal, Y. 2024. Detecting Hallucinations in Large Language Models Using Semantic Entropy. Nature, 630(8017): 625–630.DOI
  13. [17]Kuhn, L.; Gal, Y.; and Farquhar, S. 2023. Semantic Uncertainty: Linguistic Invariants for Uncertainty Estimation in Natural Language Generation. In The Eleventh International Conference on Learning Representations.arxiv.org/abs/2302.09664
  14. [18]Lewkowycz, A.; Andreassen, A.; Dohan, D.; Dyer, E.; Michalewski, H.; Ramasesh, V.; Slone, A.; Anil, C.; Schlag, I.; Gutman-Solo, T.; et al. 2022. Solving Quantitative Reasoning Problems with Language Models. In Advances in Neural Information Processing Systems.arxiv.org/abs/2206.14858
  15. [19]Lightman, H.; Kosaraju, V.; Burda, Y.; Edwards, H.; Baker, B.; Lee, T.; Leike, J.; Schulman, J.; Sutskever, I.; and Cobbe, K. 2024. Let’s Verify Step by Step. In The Twelfth International Conference on Learning Representations.arxiv.org/abs/2305.20050
  16. [20]Liu, Z.; Liu, M.; Wen, S.; Cai, M.; Cui, B.; He, C.; and Zhang, W. 2025. From Uniform to Heterogeneous: Tailoring Policy Optimization to Every Token’s Nature. arXiv:2509.16591.arxiv.org/abs/2509.16591
  17. [21]Qwen Team. 2024. Qwen2.5 Technical Report. arXiv:2412.15115.
  18. [22]Shao, Z.; Wang, P.; Zhu, Q.; Xu, R.; Song, J.; Bi, X.; Zhang, H.; Zhang, M.; Li, Y. K.; Wu, Y.; and Guo, D. 2024. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300.arxiv.org/abs/2402.03300

Paper details

vlnc:W_55cdpg65
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents