betaFeedback

WorksComputer Science

MGE-PO: Entropy-Aware Credit Assignment for Mathematical Reasoning

Abstract

GRPO assigns the same group-relative advantage to equally rewarded responses within a question, leaving their generation uncertainty unused. We introduce MGE-PO, centered on four-quadrant (4Q) advantage shaping that combines verified correctness with rollout generation entropy. Correctness determines the sign of an additional correction, while entropy controls its magnitude: lower-entropy successes receive larger bonuses and higher-entropy failures receive larger penalties. An online entropy-based question curriculum supports this credit rule using cached rollout measurements. At the final checkpoint after eight training epochs on Qwen3-8B-Base, adding 4Q across three policy optimization settings yields absolute gains of 3.23-3.69% in six-benchmark macro avg@32 and 4.24-11.06% in macro pass@32. On Qwen2.5-7B-Base, the complete configuration raises four-benchmark macro avg@32 from 41.5% to 45.4%.

Introduction

Reinforcement learning from verifiable rewards (RLVR) improves language-model reasoning using outcome feedback. For mathematical problems, an answer grader supplies a correctness reward without a learned reward model or step-level annotation. Group Relative Policy Optimization (GRPO) estimates advantages from groups of sampled answers and avoids a separate value network ([9]). DeepSeek-R1 and DAPO demonstrate the potential of this training paradigm ([14, 10]).

Binary outcome feedback leaves a credit-assignment decision unresolved. In GRPO, responses to the same question with the same reward receive the same base advantage, even when the policy is concentrated along one response and dispersed along another. Entropy-conditioned updates can supplement outcome feedback, but differ in the uncertainty measure, unit of credit, and mapping from uncertainty to update strength ([3, 2, 6 v1]). This motivates a specific question: how should verified outcomes and rollout generation uncertainty jointly determine additional credit?

We introduce MGE-PO, centered on entropy-aware advantage shaping (EA-Adv), which we call the four-quadrant (4Q) rule. The rule combines verified correctness with normalized rollout entropy in a continuous correction. Correctness selects its sign; entropy sets its magnitude. Lower-entropy successes receive larger additional bonuses, while higher-entropy failures receive larger additional penalties. Entropy measures predictive concentration along sampled prefixes, and the verifier supplies correctness. Conditioning on the outcome therefore allows concentration to guide credit even when a low-entropy response is incorrect. Cached token entropies, pooled over the most uncertain positions, provide the rollout statistic.

DynZPD supports 4Q by allocating questions through an online intermediate-entropy window with a positive sampling floor. It changes rollout opportunities, while 4Q changes rollout credit. Entropy-weighted reference KL is an actor optimization setting specified in Section 4.4 and Appendix B.4.

Our analysis characterizes the correction’s sign and the within-outcome coefficient ordering preserved by batch recentering under binary-reward GRPO. A separate mixed-reward calculation motivates attention to partially mastered questions, with the entropy window assessed empirically. On Qwen3-8B-Base, adding 4Q to GRPO yields absolute gains of 3.69% in six-benchmark macro avg@32 and 11.06% in macro pass@32 at the final checkpoint after eight epochs. Adding 4Q to DAPO and Clip-Cov also improves both macro metrics. The complete MGE-PO configuration raises four-benchmark macro avg@32 from 41.5% to 45.4% on Qwen2.5-7B-Base, with gains across the evaluated Qwen2.5 base initializations. The paired Qwen3 comparisons evaluate the credit rule, and the Qwen2.5 comparisons establish the performance of the complete configuration.

Our contributions are:

  1. An outcome-conditioned rollout advantage correction that uses generation entropy to distinguish responses with the same verified outcome, with relative-credit ordering characterized under binary-reward GRPO and batch recentering. An online entropy-based question curriculum supports this credit rule using cached training rollouts and a positive sampling floor.

  1. Improvements in both average response accuracy and sampled question coverage from adding 4Q to three optimization settings on Qwen3-8B-Base, together with complete-configuration gains across the evaluated Qwen2.5 base initializations.

Related Work

Uncertainty and supervision in mathematical generation. Malinin and Gales [21] study token- and sequence-level uncertainty; Kuhn et al. [22] group meaning-equivalent answers to estimate semantic uncertainty. Token predictive entropy measures concentration at generated prefixes, whereas semantic entropy measures uncertainty across answers. MGE-PO pools token entropy for rollout credit and question sampling, supervised by verified final answers. Unlike human step-level reward supervision (Lightman et al. [25]) or Math-Shepherd’s automatic process supervision (Wang et al. [26]), 4Q needs neither step-correctness labels nor a process reward model.

Policy optimization and entropy-conditioned credit. GRPO derives advantages from within-question rewards (Shao et al. [9]), DAPO adds asymmetric clipping, dynamic sampling of nondegenerate groups, token-level loss aggregation, and overlong reward shaping [10]. EntroAdv adds a clipped, detached, nonnegative token-entropy bonus [3]; SEED-GRPO attenuates question-level updates using answer semantic entropy [2]. EAPO v1 crosses advantage polarity with token entropy and amplifies high-entropy updates using group-normalized token entropy [6 v1]. MGE-PO instead uses verified correctness to select a rollout correction’s sign: lower entropy increases the success bonus, and higher entropy increases the failure penalty. This coefficient enters the rollout advantage. The distinctions concern the credit unit, outcome signal, and within-outcome entropy mapping.

Adaptive curricula. Competence-based curricula expand the difficulty range during learning [24]; uncertainty-aware translation curricula combine difficulty with model uncertainty [23]. SENT uses initial-policy semantic entropy for easy-to-hard reasoning stages [1]. EDCO periodically estimates entropy from short prefixes and selects high-entropy subsets, also testing a moderate-entropy window [7]. DynZPD instead updates question statistics from training rollouts and samples through a bell-shaped distribution with a positive floor, supplying opportunities for rollout credit assignment.

Reference regularization in RLVR. Selective KL penalties vary reference costs by token properties [13, 4, 1], while DAPO omits frozen-reference KL [10]. MGE-PO uses bounded entropy-dependent token weights. Appendices B.4 and D.7 specify this optimization setting and its diagnostics; Appendix E compares intervention rules.

Preliminaries

Group-relative outcome supervision

For question qq, policy πk\pi_k samples yi=(ai,1,…,ai,Ti)y_i = (a_{i,1}, \ldots, a_{i,T_i}). A deterministic final-answer grader gives 4Q4Q binary correctness ri∈{0,1}r_i \in\{0, 1\}; host reward shaping may separately determine the base advantage.

Binary-reward GRPO normalizes GG completions of qq:

AiGRPO=ri−μqσq+ϵ,A_i^{\mathrm{GRPO}} = \frac{r_i-\mu_q}{\sigma_q+\epsilon},
μq=1G∑j=1Grj,\mu_q = \frac{1}{G}\sum_{j=1}^{G} r_j,

Here σq\sigma_q is the empirical reward standard deviation and ϵ>0\epsilon> 0. Equal rewards give equal coefficients; homogeneous groups give zero.

At prefix ci,t=(q,ai,<t)c_{i,t} = (q, a_{i,<t}), the likelihood ratio and clipped token surrogate are

ρi,t(θ)=πθ(ai,t∣ci,t)πk(ai,t∣ci,t),\rho_{i,t}(\theta) = \frac{\pi_\theta(a_{i,t}\mid c_{i,t})}{\pi_k(a_{i,t}\mid c_{i,t})},
ℓi,tclip(θ,Ai)=min⁡[ρi,t(θ)Ai, clip⁡(ρi,t(θ),1−ϵℓ,1+ϵh)Ai].\ell_{i,t}^{\mathrm{clip}}(\theta,A_i) = \min\left[\rho_{i,t}(\theta)A_i,\ \operatorname{clip}\left(\rho_{i,t}(\theta),1-\epsilon_\ell,1+\epsilon_h\right)A_i\right].

Matched component comparisons use the same masked actor-batch aggregation Lclip\mathcal{L}_{\mathrm{clip}}. The rollout policy πk\pi_k is refreshed between rounds; fixed πref\pi_{\mathrm{ref}} defines the reference-deviation cost.

Generation entropy

Generation entropy is the next-token Shannon entropy:

Hi,t=−∑v∈Vπk(v∣ci,t)log⁡πk(v∣ci,t).H_{i,t} = -\sum_{v\in\mathcal{V}} \pi_k(v\mid c_{i,t})\log\pi_k(v\mid c_{i,t}).

Here πk\pi_k is the pre-decoding policy distribution; entropy is in nats and excludes padding. Cached entropy statistics and coefficients are detached during actor updates. Entropy measures concentration, separately from correctness; Appendix C.1 lists the information needed to reproduce the cached entropy measurements.

MGE-PO

MGE-PO uses 4Q4Q to assign rollout credit, DynZPD to sample questions, and reference regularization in the actor objective (Figure 1).

GRPO-based MGE-PO centered on 4Q advantage shaping

Figure 1. GRPO-based MGE-PO centered on 4Q advantage shaping. Verified outcomes and cached token entropy determine continuous corrections: lower-entropy successes receive larger bonuses, higher-entropy failures larger penalties. Top-KK pooling and running normalization produce uiu_i. Corrections precede batch recentering and the clipped update; gi=0g_i = 0 disables them for truncated rollouts. High-entropy successes and low-entropy failures remain active. DynZPD and EA-KL support the complete configuration.

Entropy summaries for rollout and question decisions

Rollout credit pools the highest-entropy positions:

hitop=1Ki∑t∈TopK⁡(Hi,⋅,Ki)Hi,t,h_i^{\mathrm{top}}=\frac{1}{K_i}\sum_{t\in\operatorname{TopK}(H_{i,\cdot},K_i)}H_{i,t},
Ki=min⁡(Ti,max⁡(1,min⁡(⌊rKTi⌋,Kmax⁡))).K_i=\min\left(T_i,\max\left(1,\min\left(\lfloor r_K T_i\rfloor,K_{\max}\right)\right)\right).

With rK=0.1r_K=0.1 and Kmax⁡=4096K_{\max}=4096, running mean μH\mu_H and scale τH\tau_H normalize this uncertainty summary:

zi=hitop−μHτH+ϵu,z_i=\frac{h_i^{\mathrm{top}}-\mu_H}{\tau_H+\epsilon_u},
ui=1−sigmoid⁡(zi).u_i=1-\operatorname{sigmoid}(z_i).

Higher uiu_i means lower relative pooled entropy. Appendix B.1 specifies the EMA with λH=0.99\lambda_H=0.99 and ϵu=10−6\epsilon_u=10^{-6}.

Question sampling uses all active generated tokens:

himean=1Ti∑t=1TiHi,t.h_i^{\mathrm{mean}}=\frac{1}{T_i}\sum_{t=1}^{T_i}H_{i,t}.

Both summaries reuse cached token entropies: top-entropy pooling targets uncertain prefixes, while the full-token mean describes overall generation uncertainty.

4Q: outcome-conditioned advantage shaping

The 4Q rule, also denoted EA-Adv in the component experiments, adds an outcome-conditioned correction to a base advantage coefficient. Its mathematical interface is

A^i=Aibase+δi,\widehat{A}_i=A_i^{\mathrm{base}}+\delta_i,
δi=gi[riγ+ui−(1−ri)γ−(1−ui)],\delta_i=g_i\left[r_i\gamma_+u_i-(1-r_i)\gamma_-(1-u_i)\right],
gi=1−1[yi is truncated].g_i=1-\mathbb{1}[y_i\text{ is truncated}].

For the GRPO instantiation, Aibase=AiGRPOA_i^{\mathrm{base}}=A_i^{\mathrm{GRPO}}, giving A^i=AiGRPO+δi\widehat{A}_i=A_i^{\mathrm{GRPO}}+\delta_i. Other host optimizers supply their own base coefficients, while 4Q uses the binary verification outcome independently of any shaped reward. Appendix C.4 records the integration scope for the DAPO and Clip-Cov combinations.

Correctness determines the sign of the direct correction, and normalized rollout entropy determines its magnitude. Lower-entropy successes receive larger additional bonuses; higher-entropy

failures receive larger additional penalties. This preference emphasizes successful behavior with concentrated generation distributions and downweights failed behavior with dispersed distributions. The Qwen2.5 configuration uses γ+=0.5\gamma_+=0.5 and γ−=0.3\gamma_-=0.3; Appendix C.4 describes the separate Qwen3 training and evaluation protocol.

The four-quadrant terminology crosses verified correctness with normalized rollout entropy: PLR denotes low-entropy correct rollouts, PHR high-entropy correct rollouts, NLR low-entropy incorrect rollouts, and NHR high-entropy incorrect rollouts. These labels describe rollout regions; the sigmoid interpolates continuously within each outcome class. The default combines a concentration-weighted positive term and an uncertainty-weighted negative term, so high-entropy successes and low-entropy failures still receive continuous corrections. Appendix B.2 introduces separately weighted PHR and NLR basis terms for the coefficient comparisons.

Figure 2 visualizes this continuous correction within each outcome class.

Figure 2. Continuous 4Q corrections for gi=1g_i=1, γ+=0.5\gamma_+=0.5, and γ−=0.3\gamma_-=0.3 (Qwen2.5). The horizontal axis is standardized rollout entropy ziz_i (§ 4.1). Lower entropy increases the success bonus; higher entropy increases the failure penalty. PLR/PHR and NLR/NHR mark within-outcome entropy regions; dashed lines mark zero. Corrections precede batch recentering, and PHR/NLR remain active.

A fixed outcome-dependent correction changes the separation between successes and failures. Entropy additionally distinguishes equally rewarded responses within a question, giving the relative-credit ordering analyzed in § 5.2.

The specified GRPO-based MGE-PO actor uses batch-recentered advantages:

A~i=A^i−1M∑j=1MA^j,\widetilde{A}_i = \widehat{A}_i - \frac{1}{M}\sum_{j=1}^{M}\widehat{A}_j,

where MM counts the rollouts included in recentering. This places the batch at zero mean and expresses each coefficient relative to the other rollouts. The sign interpretation above applies to δi\delta_i; A~i\widetilde{A}_i includes the base advantage and shared centering shift. The truncation gate removes the direct correction before centering. Appendix § 9.§ 3 describes truncation handling and the resulting behavior in homogeneous reward groups.

DynZPD: online question sampling

For each question’s rollout group Bq\mathcal{B}_q, DynZPD computes

eq(k)=1∣Bq∣∑i∈Bqhimean.e_q^{(k)} = \frac{1}{\lvert\mathcal{B}_q\rvert}\sum_{i\in\mathcal{B}_q} h_i^{\mathrm{mean}}.

Its stored statistic follows the EMA

ξq(k)=λqξq(k−1)+(1−λq)eq(k).\xi_q^{(k)} = \lambda_q \xi_q^{(k-1)} + (1-\lambda_q)e_q^{(k)}.

Unobserved questions retain their statistics. Round kk samples using values from round k−1k-1:

vq(k)=η+(1−η)exp⁡[−(ξq(k−1)−μZPD)22σZPD2],v_q^{(k)} = \eta+ (1-\eta)\exp\left[-\frac{\left(\xi_q^{(k-1)}-\mu_{\mathrm{ZPD}}\right)^2}{2\sigma_{\mathrm{ZPD}}^2}\right],
pk(q)=vq(k)∑q′∈Dvq′(k).p_k(q) = \frac{v_q^{(k)}}{\sum_{q'\in\mathcal{D}}v_{q'}^{(k)}}.

The Gaussian favors the target entropy; η\eta retains positive dataset-wide support. Settings are μZPD=0.30\mu_{\mathrm{ZPD}} = 0.30, σZPD=0.15\sigma_{\mathrm{ZPD}} = 0.15, η=0.10\eta= 0.10, and λq=0.9\lambda_q = 0.9. Initializing ξq(0)=μZPD\xi_q^{(0)} = \mu_{\mathrm{ZPD}} gives uniform sampling.

This model-relative curriculum changes training emphasis. § 5 relates partial mastery to learning opportunities; § 6 reports allocation diagnostics. Appendix § 9.§ 5 details the sampling floor and statistic updates.

Actor optimization

The GRPO-based actor optimizes recentered advantages on curriculum-sampled questions:

Jk(θ)=EBk∼(pk,πk)[Lclip(θ;A~)−βK^w(θ,πref)].J_k(\theta) = \mathbb{E}_{B_k\sim(p_k,\pi_k)}\left[\mathcal{L}_{\mathrm{clip}}(\theta;\widetilde{A})-\beta\widehat{\mathcal{K}}_w(\theta,\pi_{\mathrm{ref}})\right].

EA-KL uses detached token weights: reference deviation costs more below the batch mean entropy and less above it. The complete Qwen2.5 configuration uses β=0.001\beta= 0.001, α=0.20\alpha= 0.20; its GRPO baseline uses β=0.04\beta= 0.04 (Appendix § 11.10). Setting α=0\alpha= 0 retains uniform KL, while β=0\beta= 0 removes it. Appendix § 9.§ 4 specifies weights, masks, and the sampled penalty.

Entropy coefficients stay fixed within each actor update; the rollout policy refreshes between rounds. Appendices § 9.§ 6 and § 10 give the procedure and configurations.

Properties of Outcome-Conditioned Entropy Shaping

Entropy statistics are frozen within each update. § 5.1 concerns direct corrections; § 5.2 assumes binary-reward GRPO normalization and shared batch recentering. Appendix § 8 gives the sampling proof and idealized optimization interpretation.

Outcome-conditioned direction and magnitude

For γ+,γ−≥0\gamma_+, \gamma_- \ge0, ui∈[0,1]u_i \in[0,1], and gi∈{0,1}g_i \in\{0,1\}, the direct correction satisfies

−giγ−≤δi≤giγ+,ri=1⇒δi≥0,ri=0⇒δi≤0.(1)-g_i\gamma_- \le\delta_i \le g_i\gamma_+, \qquad r_i = 1 \Rightarrow\delta_i \ge0, \qquad r_i = 0 \Rightarrow\delta_i \le0. \tag*{(1)}

At fixed running statistics, ui=1−sigmoid⁡(zi)u_i = 1-\operatorname{sigmoid}(z_i) decreases with entropy:

∂δi∂zi=−gi[riγ++(1−ri)γ−]ui(1−ui)≤0.\frac{\partial\delta_i}{\partial z_i} = -g_i\left[r_i\gamma_+ + (1-r_i)\gamma_-\right]u_i(1-u_i) \le0.

For untruncated responses and positive default coefficients, entropy reduces the success bonus or increases the failure penalty. This bounded, continuous rule assigns sign by correctness and strength by concentration. The derivative concerns cached inputs; entropy remains detached during actor updates.

Relative credit survives batch recentering

Property 1 (Relative-credit ordering). Consider two untruncated responses i,ji, j from the same binary-reward GRPO normalization group and actor batch, with the same outcome and shared running entropy statistics. Their base GRPO advantages are equal. With a1=γ+a_1=\gamma_+ and a0=γ−a_0=\gamma_-, batch recentering gives

A~i−A~j=δi−δj=ari(ui−uj).(2)\widetilde{A}_i-\widetilde{A}_j=\delta_i-\delta_j=a r_i(u_i-u_j). \tag*{(2)}

Equal base advantages and the shared centering constant cancel. For positive coefficients, lower entropy gives a larger success bonus or a smaller failure penalty within the same outcome class.

Replacing uiu_i by u0∈[0,1]u_0 \in[0,1] gives the untruncated outcome-only correction δiconst=riγ+u0−(1−ri)γ−(1−u0)\delta_i^{\mathrm{const}}=r_i\gamma_+u_0-(1-r_i)\gamma_-(1-u_0) and decomposition

δi=δiconst+ari(ui−u0).(3)\delta_i=\delta_i^{\mathrm{const}}+a r_i(u_i-u_0). \tag*{(3)}

For the same-outcome pair in Property 1, the constant correction gives identical additional credit and zero coefficient difference after recentering. (3) identifies the additional contribution of entropy: response-dependent coefficient variation with the ordering in (S2). A correction determined only by the outcome and shared group statistics changes the separation between outcome classes while leaving this same-outcome pair undifferentiated.

For a correct response cc and an incorrect response ff in the same binary-reward GRPO normalization group and actor batch, the same cancellation yields

A~c−A~f=1σq+ϵ+gcγ+uc+gfγ−(1−uf)≥AcGRPO−AfGRPO.(4)\widetilde{A}_c-\widetilde{A}_f=\frac{1}{\sigma_q+\epsilon}+g_c\gamma_+u_c+g_f\gamma_-(1-u_f)\geq A_c^{\mathrm{GRPO}}-A_f^{\mathrm{GRPO}}. \tag*{(4)}

Under the specified binary-reward GRPO normalization, 4Q preserves within-outcome entropy ordering and weakly increases the correct-versus-incorrect coefficient gap. These are properties of relative credit; an individual centered advantage can have a different sign from its direct correction (Appendix B.3). The resulting 4Q training rule is evaluated in Section 6.2, with coefficient and component comparisons in Appendices D.6 and D.10.

Sampling opportunities for outcome-relative learning

For success probability sqs_q, G≥2G \geq2 independent completions give mixed-reward probability

mG(sq)=1−sqG−(1−sq)G.(5)m_G(s_q)=1-s_q^G-(1-s_q)^G. \tag*{(5)}

Proposition 1 (Mixed-reward opportunities). mGm_G is uniquely maximized at sq=1/2s_q=1/2 and vanishes at sq∈{0,1}s_q \in\{0,1\} (proof: Appendix A.1).

Mixed groups give nonzero base GRPO coefficients, motivating partially mastered questions. DynZPD’s intermediate-entropy window is an empirical proxy for this success-probability regime, tested through component comparisons. 4Q can also differentiate responses within reward-homogeneous groups (Appendix B.3).

Shared parameters and a finite update budget make question allocation consequential. DynZPD changes gradient frequencies without inverse-probability correction (Appendix A.1); Section 6.4 reports sampling concentration.

Experiments

Qwen3 comparisons test 4Q across three optimization settings; Qwen2.5 comparisons evaluate complete configurations. Diagnostics examine predictive concentration, training, and sampling.

Experimental setup

Qwen3-8B-Base {(citep)}(Yang et al., 2025) uses a random 10,000-question Skywork-OR1 mathematics subset {(citep)}(He et al., 2025), subsampled with seed 1; filtering leaves 9,998 questions. Batches contain 128 questions with eight rollouts each, capped at 20,480 tokens. All seven configurations are evaluated at final controller step 624 after eight epochs (Appendix C.4).

Learning rate, benchmark sets, and decoding settings are matched. Evaluation samples 32 responses per question at temperature 0.6 on AIME2025, AIME2024, AMC2023, MATH500, Minerva, and OlympiadBench. We denote mean@32 by avg@32; macro scores weight the six benchmarks equally.

The original series uses Qwen2.5-7B-Base, Qwen2.5-Math-1.5B-Base, Qwen2.5-Math-7B-Base, and an additional Math-7B-Instruct initialization {(citep)}(Yang et al., 2024a,b), trained on 17,398 DAPO-Math-17k questions {(citep)}(Yu et al., 2025). Base runs use eight epochs, batch size 128, 16 rollouts per question, and a 12,288-token maximum. Under adaptive sampling, epochs measure dataset-equivalent budgets, not guaranteed visits.

Qwen2.5 evaluation uses avg@32/pass@32 on MATH-500, AMC23, AIME24, and AIME25, with temperature 1.0, top-pp 0.7, and approximately 12k tokens. Supplementary evaluation uses avg@64 on MATH-500, AMC23, Minerva-Math, and Olympiad, plus AIME24/AIME25 pass@256; Minerva/Olympiad responses are capped at 8,192 tokens. Protocols remain separate because inference budgets and checkpoint metadata differ. Table 9 gives sizes and metrics; MATH and Minerva follow Hendrycks et al. [15] and Lewkowycz et al. [16].

BenchmarkQuestionsOriginal protocolSupplementary protocol
MATH-500500avg@32, pass@32avg@64
AMC 202340avg@32, pass@32avg@64
AIME 202430avg@32, pass@32pass@256
AIME 202530avg@32, pass@32pass@256
Minerva-Math / OCW272Not evaluatedavg@64
BenchmarkQuestionsOriginal protocolSupplementary protocol
Olympiad, text-only English exact-answer subset674Not evaluatedavg@64

Table 9. Qwen2.5 evaluation datasets and reported metrics. Competition subset sizes and the text-only English exact-answer Olympiad subset follow the experimental record. Appendix C lists the reproducibility requirements for exact manifests, grader versions, and overlap filtering.

For NN questions with nn responses each and cjc_j correct responses on question jj,

avg⁡@n=100N∑j=1Ncjn.\operatorname{avg}@n = \frac{100}{N}\sum_{j=1}^{N}\frac{c_j}{n}.

Pass@KK is the proportion of questions with at least one correct response among KK samples (Chen et al., [20]), averaged over RKR_K evaluations of the same checkpoint:

pass⁡@K=100NRK∑ℓ=1RK∑j=1N1[cj,ℓ(K)>0].\operatorname{pass}@K = \frac{100}{NR_K}\sum_{\ell=1}^{R_K}\sum_{j=1}^{N}\mathbf{1}\left[c_{j,\ell}^{(K)}>0\right].

Here cj,ℓ(K)c_{j,\ell}^{(K)} counts correct responses among KK samples in repeat ℓ\ell; a single evaluation has RK=1R_K=1. Qwen2.5 averages inference repeats of one training run per model-method cell; Qwen3 reports final-step point estimates. Neither measures training-seed variability (Appendix § 10).

Scores use percentages (%); changes are absolute differences, ΔS=Smethod−Sbaseline\Delta S=S_{\mathrm{method}}-S_{\mathrm{baseline}}, also denoted %.

Qwen2.5 baselines are GRPO, DAPO, and KL-Cov, plus supplementary Clip-Cov on Minerva/Olympiad. Qwen3 compares GRPO, DAPO, and Clip-Cov on all six benchmarks. Appendix § 11.§ 1 includes pre-RL Qwen2.5 scores.

4Q across optimization settings on Qwen3-8B-Base

Adding 4Q improves macro accuracy and coverage across all three optimization settings (Table 1; Figure 3). The paired configurations are GRPO versus 4Q, DAPO versus 4Q + DAPO, and Clip-Cov versus Clip-Cov + 4Q; Table 1 also reports complete MGE-PO.

MethodAIME2025 (%)AIME2024 (%)AMC2023 (%)MATH500 (%)Minerva (%)Olympiad Bench (%)Macro (%)
GRPO13.44 / 33.335.52 / 13.3361.02 / 85.0076.45 / 89.4046.66 / 60.2944.78 / 66.4741.31 / 57.97
DAPO16.35 / 40.0010.73 / 23.3365.47 / 92.5078.79 / 92.6044.47 / 74.6345.87 / 75.2243.61 / 66.38
Clip-Cov15.52 / 43.3310.83 / 33.3363.59 / 92.5078.40 / 93.2045.23 / 72.7945.92 / 72.8543.25 / 68.00
MGE-PO23.96 / 53.3313.02 / 40.0064.53 / 97.5081.31 / 93.0047.48 / 68.0152.47 / 74.3347.13 / 71.03
Clip-Cov + 4Q20.42 / 56.6716.25 / 43.3367.81 / 92.5081.11 / 94.0044.16 / 70.9649.11 / 75.9646.48 / 72.24
4Q18.85 / 46.6712.60 / 36.6762.66 / 92.5079.66 / 92.6046.90 / 69.4949.33 / 76.2645.00 / 69.03
4Q + DAPO20.31 / 50.0016.67 / 40.0063.20 / 97.5082.54 / 93.2046.61 / 70.9652.62 / 78.4946.99 / 71.69

Table 1. Qwen3-8B-Base at final controller step 624 (eight epochs). Cells report avg@32 / pass@32 (%) using 32 responses per question, temperature 0.6, and matched test sets and decoding. Avg@32 equals mean@32. Macro scores weight six benchmarks equally. Bold marks each metric’s maximum across seven configurations, including ties. Displayed scores use two decimals; macro averages retain reported precision before rounding.

Figure 3. Adding 4Q to GRPO, DAPO, and Clip-Cov on Qwen3-8B-Base at step 624. Bars show absolute macro mean@32 (avg@32) and pass@32 changes (%), computed before rounding. Pairings are those in § 6.2; values are point estimates without uncertainty intervals.

GRPO’s macro avg@32 rises from 41.31% to 45.00% and pass@32 from 57.97% to 69.03%. DAPO gains 3.38%/5.31%; Clip-Cov gains 3.23%/4.24%. Standalone 4Q also exceeds unaugmented DAPO and Clip-Cov on both macro metrics (paired differences: Appendix § 11.11).

Complete MGE-PO leads macro avg@32 at 47.13% (+5.82% over GRPO), with pass@32 of 71.03% (+13.06%). Clip-Cov + 4Q leads pass@32 at 72.24%, followed by 4Q + DAPO at 71.69%. Accuracy and coverage identify different leading configurations.

The macro gains include benchmark-specific tradeoffs. Adding 4Q to DAPO lowers AMC2023 avg@32 by 2.27% and Minerva pass@32 by 3.67%; Clip-Cov’s Minerva avg@32/pass@32 falls by 1.07%/1.83%. Figure 5 shows all 36 contrasts, including unchanged Clip-Cov AMC2023 pass@32.

Figure 5. Per-benchmark effect of adding 4Q on Qwen3-8B-Base at the final controller step 624. Each cell gives the augmented score minus its baseline score as an absolute difference (%), for mean@32 (top) and pass@32 (bottom). Rows identify GRPO, DAPO, and Clip-Cov; the corresponding augmented configurations are 4Q, 4Q + DAPO, and Clip-Cov + 4Q. All 36 contrasts share a zero-centered color scale, including negative and zero changes. Differences use the reported two-decimal benchmark scores. Evaluation samples 32 responses per question at temperature 0.6 on identical test sets. Cells describe point estimates without significance claims.

Complete-configuration results across Qwen2.5 initializations

Table 2 evaluates the complete Qwen2.5 configuration across model size and mathematical pre-training under the original four-benchmark protocol (full scores: Appendix § 11.§ 1).

InitializationMethodMacro avg@32 (%)Macro pass@32 (%)AIME24 pass@32 (%)AIME25 pass@32 (%)
Qwen2.5-Math-1.5B-BaseGRPO33.760.130.330.7
DAPO35.359.232.726.7
KL-Cov35.861.531.835.7
MGE-PO36.063.333.338.3
Qwen2.5-Math-7B-BaseGRPO41.463.236.735.4
DAPO42.765.343.336.7
KL-Cov43.666.244.438.3
MGE-PO43.570.050.043.3
Qwen2.5-7B-BaseGRPO41.558.630.323.3
DAPO42.560.136.725.7
KL-Cov44.462.239.528.3
MGE-PO45.462.840.030.3

Table 2. Four-benchmark summary (%): macro scores are unweighted means over MATH-500, AMC23, AIME24, and AIME25 (full scores: Appendix § 11.§ 1). Each model-method cell uses one training run. MGE-PO-GRPO comparisons include a global KL-coefficient difference (Appendices § 10.§ 3 and § 11.10).

MGE-PO improves both macro metrics over GRPO across all three initializations. On Qwen2.5-7B-Base, avg@32 rises 41.5% to 45.4% and pass@32 58.6% to 62.8%, with AIME24/AIME25 gains of 9.7%/7.0%. Math-7B pass@32 rises 63.2% to 70.0%, with AIME gains of 13.3%/7.9%.

At fixed β=0.001\beta= 0.001, Figure 4 shows 4Q gains of 2.1%, 2.3%, and 2.9% with EA-KL, DynZPD, and both. These single-run conditional contrasts retain supporting configurations; fixed-correction and shuffled-entropy controls remain needed (Appendix § 11.10). Appendix § 11.§ 6 tests quadrant coefficients.

Figure 4. Conditional 4Q gains on Qwen2.5-7B-Base at β=0.001\beta= 0.001 (Table 15). Pairs share their supporting configuration; EA-KL off retains uniform KL. Scores are four-benchmark macro avg@32; annotations give percentage-point gains. Each configuration uses one training run; uncertainty intervals are unavailable.

Tradeoffs remain: Math-7B KL-Cov exceeds MGE-PO on macro avg@32 (43.6% versus 43.5%) and MATH-500 avg@32 (75.8% versus 73.1%). On plain 7B, AMC23 avg@32 rises from GRPO’s 58.7% to 65.5%, while pass@32 falls from 92.5% to 90.0%.

MGE-PO leads five of six supplementary metrics, including Minerva/Olympiad gains of 3.1%/3.9% over GRPO; DAPO leads AMC23 (Appendix § 11.§ 2, including Clip-Cov). The Instruct comparison improves all six metrics over GRPO (Appendix § 11.§ 5).

Outcome, training, and sampling diagnostics

Low-entropy incorrect responses account for 45.5% of initialization rollouts and 37.2% after GRPO (Table 8). Concentration can coexist with failure, motivating outcome-conditioned entropy for rollout credit.

Diagnostic quadrantInitialization countInitialization share (%)GRPO countGRPO share (%)
Low-entropy correct1734.549212.8
High-entropy correct741.9962.5
Low-entropy incorrect174745.5142837.2
High-entropy incorrect184648.1182447.5

Table 8. Aggregate quadrant counts in the original manuscript. Each column summarizes 3,840 rollouts, with a reporting threshold at ui=0.5u_i = 0.5. The comparison is initialization versus ordinary GRPO, not GRPO versus MGE-PO.

Appendix § 13.§ 3 retains the original full-configuration training curves as supplementary diagnostics. Their checkpoint correspondence with Table 2 is undocumented. Appendix § 11.§ 7 reports reference-weight diagnostics.

For sampling distribution pp, ESS(p)=1/∑qp(q)2\mathrm{ESS}(p) = 1/\sum_q p(q)^2, with relative ESS 100ESS(p)/∣D∣100\mathrm{ESS}(p)/\lvert D\rvert. Over 21 steps and 43,008 rollouts, MGE-PO samples 2,440 unique questions: relative ESS is 91.5%, top-decile mass 18.3%, and coverage 14.0% versus uniform expectation 14.3% (Table 14). These are window-specific diagnostics; the positive floor retains dataset-wide support.

DiagnosticGRPO, uniformKL-Cov, uniformMGE-PO, DynZPD
Relative ESS100%100%91.5%
Top-10% question probability mass10.0%10.0%18.3%
Sampling-distribution Gini000.085
Maximum question share (definition unreported)Not reportedNot reported0.4%
In-window unique-question coverage14.3%, expected14.3%, expected14.0%, observed

Table 14.

Appendix § 11.8 reports complete-run timing (Table 13).

InitializationMethodWall-clock hours
Base settingGRPO22.8
Base settingKL-Cov29.6
Base settingMGE-PO22.6
Qwen2.5-Math-7B-InstructGRPO23.6
Qwen2.5-Math-7B-InstructMGE-PO22.8

Table 13. Table 13: End-to-end wall-clock time for complete single runs with matched training budgets. Matching refers to training budget rather than KL coefficients. The timing record identifies a base setting without a separate model-checkpoint or hardware identifier. These are wall-clock measurements; utilization and GPU-hour accounting are unavailable.

Conclusion

MGE-PO uses verified outcomes and generation entropy to refine rollout credit in mathematical RLVR. Its 4Q rule gives lower-entropy successes larger additional bonuses and higher-entropy failures larger additional penalties. Under binary-reward GRPO, shared batch recentering preserves the entropy ordering of equally rewarded responses. An online entropy-based question curriculum supports this rule using cached rollout measurements.

On Qwen3-8B-Base at the final checkpoint after eight epochs, adding 4Q to GRPO, DAPO, and Clip-Cov yields absolute gains of 3.23–3.69% in six-benchmark macro avg@32 and 4.24–11.06% in macro pass@32. The complete configuration also improves mathematical accuracy and AIME coverage over GRPO across the evaluated Qwen2.5 base initializations. These results support outcome-conditioned entropy as a credit signal across the evaluated model and optimization settings.

Limitations

Evaluation covers mathematical reasoning on Qwen models. The comparisons evaluate the specified 4Q rule; fixed-correction and shuffled-entropy controls remain needed to isolate entropy-dependent ordering from outcome-only shaping and arbitrary response-dependent variation.

The Qwen2.5 component study uses a different global KL coefficient for GRPO, so its complete-method comparison includes that change. Shared-coefficient component contrasts provide the conditional comparisons described in Appendix § 11.10.

The quadrant-coefficient sweep uses AIME24, also a final evaluation benchmark, without a documented independent development set.

Training-seed variability is unmeasured. The Qwen2.5 scores average repeated inference evaluations within trained checkpoints. Appendix § 10 records the available protocol metadata and the run-specific settings that remain unspecified.

Ethical Considerations

This study uses existing mathematical benchmarks and pretrained model checkpoints. Improved answer generation can support research and educational assistance, and can also facilitate misuse in assessments. Final-answer verification and predictive concentration do not validate every intermediate reasoning step. Applications should evaluate generated reasoning independently before presenting it as trustworthy. Reproducibility artifacts should identify the exact model and dataset releases, their licenses, and the training and evaluation configurations.

Proofs and Additional Optimization Interpretation

Section 5 characterizes 4Q’s credit coefficients directly. We first prove the sampling proposition, then provide a standard reference-constrained interpretation of a frozen outcome-conditioned utility. The exact-optimum results assume a finite completion space, finite fixed utilities, a full-support reference distribution, and fixed β>0\beta> 0. They explain preference and deviation cost under exact distributional optimization; the empirical actor uses sampled, clipped updates with shared neural parameters.

Mixed-reward opportunities and finite-budget allocation

Proof of Proposition 1. Independence gives probability sGs^{G} for an all-success group and (1−s)G(1-s)^{G} for an all-failure group. These events are disjoint, so mG(s)=1−sG−(1−s)Gm_G(s)=1-s^{G}-(1-s)^{G}. For s∈(0,1)s \in(0,1),

mG′(s)=G[(1−s)G−1−sG−1].m'_G(s)=G\left[(1-s)^{G-1}-s^{G-1}\right].

For G≥2G \ge2, this derivative is positive below 1/21/2, zero at $1/2, and negative above 1/21/2. Thus 1/21/2 is the unique maximizer, with value 1−21−G1-2^{1-G}; both endpoint values are zero. Mixed groups supply nonzero base GRPO coefficients, although their aggregate gradient depends on the sampled actions and model parameters.

Sampling frequency matters for a policy whose parameters are shared across questions. Let V(θ)V(\theta) be a locally smooth held-out objective and gqg_q an ascent update estimated from question qq, including the learning signal and regularization. For a sufficiently small step λstep\lambda_{\mathrm{step}},

V(θ+λstepgq)=V(θ)+λstep∇V(θ)Tgq+O(λstep2∥gq∥2).V(\theta+\lambda_{\mathrm{step}}g_q)=V(\theta)+\lambda_{\mathrm{step}}\nabla V(\theta)^{\mathsf{T}}g_q+O\left(\lambda_{\mathrm{step}}^2\lVert g_q\rVert^2\right).

Averaging over update noise and question draws gives first-order contribution proportional to

∑qpk(q)∇V(θ)TE[gq].\sum_q p_k(q)\nabla V(\theta)^{\mathsf{T}}\mathbb{E}[g_q].

This expression describes immediate learning value through gradient alignment. DynZPD uses online entropy as an empirical proxy for useful training regimes; neither the alignment expression nor Proposition 1 establishes an optimal entropy target. For the independent distributional model in Equation (8), let Jq(πq)J_q(\pi_q) denote its per-question objective. Positive question weights in ∑qpk(q)Jq(πq)\sum_q p_k(q)J_q(\pi_q) do not change the per-question exact optima. A shared neural policy and a finite update budget make allocation consequential. The weight floor preserves positive sampling support; realized coverage depends on the finite sequence of draws.

Frozen-utility preferences and outcome-sign consistency

For a question qq and completion yy, let f0,k(q,y)f_{0,k}(q,y) be a fixed baseline utility, and freeze the entropy and truncation signals. Retain r(q,y)∈{0,1}r(q,y) \in\{0,1\}, uk(q,y)∈[0,1]u_k(q,y) \in[0,1], g(q,y)∈{0,1}g(q,y) \in\{0,1\}, and γ+,γ−≥0\gamma_+,\gamma_- \ge0. Define

δk(q,y)=g(q,y)[r(q,y)γ+uk(q,y)−(1−r(q,y))γ−(1−uk(q,y))].(6)\delta_k(q,y)=g(q,y)\left[r(q,y)\gamma_+u_k(q,y)-(1-r(q,y))\gamma_-(1-u_k(q,y))\right]. \tag*{(6)}
fk(q,y)=f0,k(q,y)+δk(q,y).(7)f_k(q,y)=f_{0,k}(q,y)+\delta_k(q,y). \tag*{(7)}

Let νq(y)=πref(y∣q)>0\nu_q(y)=\pi_{\mathrm{ref}}(y\mid q)>0. At fixed β>0\beta>0, consider

max⁡π(⋅∣q){Ey∼π(⋅∣q)[fk(q,y)]−βDKL(π(⋅∣q)∥νq)}.(8)\max_{\pi(\cdot\mid q)} \left\{\mathbb{E}_{y\sim\pi(\cdot\mid q)}[f_k(q,y)]-\beta D_{\mathrm{KL}}\left(\pi(\cdot\mid q)\Vert\nu_q\right)\right\}. \tag*{(8)}

Proposition A.1 (Regularized trajectory preference). The unique maximizer is

πk∗(y∣q)=νq(y)exp⁡(fk(q,y)/β)Zk(q),(9)\pi_k^*(y \mid q) = \frac{\nu_q(y)\exp(f_k(q,y)/\beta)}{Z_k(q)}, \tag*{(9)}
Zk(q)=∑yνq(y)exp⁡(fk(q,y)/β).Z_k(q) = \sum_y \nu_q(y)\exp(f_k(q,y)/\beta).

This standard KL-regularized solution ([11, 12]) implies

log⁡πk∗(y1∣q)πk∗(y2∣q)=log⁡νq(y1)νq(y2)+fk(q,y1)−fk(q,y2)β.(10)\log\frac{\pi_k^*(y_1 \mid q)}{\pi_k^*(y_2 \mid q)} = \log\frac{\nu_q(y_1)}{\nu_q(y_2)} + \frac{f_k(q,y_1)-f_k(q,y_2)}{\beta}. \tag*{(10)}

Proof. Suppressing q,kq,k, substitute π∗(y)=ν(y)ef(y)/β/Z\pi^*(y)=\nu(y)e^{f(y)/\beta}/Z to obtain

Eπ[f]−βDKL(π∥ν)=βlog⁡Z−βDKL(π∥π∗).\mathbb{E}_{\pi}[f]-\beta D_{\mathrm{KL}}(\pi\Vert\nu)=\beta\log Z-\beta D_{\mathrm{KL}}(\pi\Vert\pi^*).

KL nonnegativity gives the unique maximum at π∗\pi^*. The stated assumptions ensure that ZZ is finite and positive and π∗\pi^* has full support. Subtracting a common frozen constant from ff multiplies both numerator and normalizer by the same factor, leaving the optimum unchanged. This includes batch recentering after its common shift is frozen.

Proposition A.2 (Outcome-sign consistency at an exact optimum). Holding the reference, baseline utility, and positive β\beta fixed, adding a correction that is nonnegative on successes and nonpositive on failures weakly increases total success probability at the exact optimum.

Proof. Let C={y:r(q,y)=1}\mathcal{C}=\{y:r(q,y)=1\} and I={y:r(q,y)=0}\mathcal{I}=\{y:r(q,y)=0\}. If either set is empty, success probability is fixed at zero or one. Otherwise both baseline masses

MC=∑y∈Cν(y)ef0(y)/β,MI=∑y∈Iν(y)ef0(y)/βM_{\mathcal{C}}=\sum_{y\in\mathcal{C}}\nu(y)e^{f^0(y)/\beta}, \qquad M_{\mathcal{I}}=\sum_{y\in\mathcal{I}}\nu(y)e^{f^0(y)/\beta}

are positive. The shaped masses satisfy

MC′=∑y∈Cν(y)ef0(y)/βeδ(y)/β≥MC,M'_{\mathcal{C}}=\sum_{y\in\mathcal{C}}\nu(y)e^{f^0(y)/\beta}e^{\delta(y)/\beta}\ge M_{\mathcal{C}},
MI′=∑y∈Iν(y)ef0(y)/βeδ(y)/β≤MI.M'_{\mathcal{I}}=\sum_{y\in\mathcal{I}}\nu(y)e^{f^0(y)/\beta}e^{\delta(y)/\beta}\le M_{\mathcal{I}}.

Consequently MC′/MI′≥MC/MIM'_{\mathcal{C}}/M'_{\mathcal{I}}\ge M_{\mathcal{C}}/M_{\mathcal{I}}, and monotonicity of x/(1+x)x/(1+x) gives the claim. The inequality is strict if a success has δ>0\delta>0 or a failure has δ<0\delta<0. Equation (6) satisfies these signs before recentering; a common frozen shift cancels by Proposition A.1. This comparison supports the outcome-conditioned signs, while several magnitude rules satisfy the same condition. The entropy ordering and the sampled update’s performance remain empirical questions.

Reference costs and the sampled actor objective

At a fixed prefix cc, let Qc(v)Q_c(v) be a fixed finite next-token utility, νc(v)>0\nu_c(v)>0 the reference distribution, and wk(c)>0w_k(c)>0 a detached weight. The local problem

max⁡p∈Δ(V){∑vp(v)Qc(v)−βwk(c)DKL(p∥νc)}(11)\max_{p\in\Delta(\mathcal{V})}\left\{\sum_v p(v)Q_c(v)-\beta w_k(c)D_{\mathrm{KL}}(p\Vert\nu_c)\right\} \tag*{(11)}

has the solution

pc∗(v)=νc(v)exp⁡(Qc(v)/(βwk(c)))∑v′νc(v′)exp⁡(Qc(v′)/(βwk(c))).(12)p_c^*(v)=\frac{\nu_c(v)\exp(Q_c(v)/(\beta w_k(c)))}{\sum_{v'}\nu_c(v')\exp(Q_c(v')/(\beta w_k(c)))}. \tag*{(12)}

This follows from Proposition A.1 with temperature βwk(c)\beta w_k(c). In particular,

log⁡pc∗(v1)pc∗(v2)−log⁡νc(v1)νc(v2)=Qc(v1)−Qc(v2)βwk(c).\log\frac{p_c^*(v_1)}{p_c^*(v_2)}-\log\frac{\nu_c(v_1)}{\nu_c(v_2)}=\frac{Q_c(v_1)-Q_c(v_2)}{\beta w_k(c)}.

Increasing the weight reduces the utility-induced shift in reference-relative log odds. It need not increase Shannon entropy, which also depends on the reference and utility ordering. At wk(c)=0w_k(c)=0, the local objective is linear and any distribution supported on utility maximizers is optimal. As positive ww approaches zero, the Gibbs solution concentrates on these maximizers with their relative reference weights.

For consistently treated termination, full-sequence KL obeys the autoregressive chain rule

DKL(π(⋅∣q)∥πref(⋅∣q))=Ey∼π(⋅∣q)[∑t=1∣y∣DKL(π(⋅∣q,y<t)∥πref(⋅∣q,y<t))].D_{\mathrm{KL}}\left(\pi(\cdot\mid q)\Vert\pi_{\mathrm{ref}}(\cdot\mid q)\right) = \mathbb{E}_{y\sim\pi(\cdot\mid q)} \left[ \sum_{t=1}^{\lvert y\rvert} D_{\mathrm{KL}}\left(\pi(\cdot\mid q,y_{<t})\Vert\pi_{\mathrm{ref}}(\cdot\mid q,y_{<t})\right) \right].

This identity uses candidate-policy prefix occupancy and a sum over sequence positions. The actor evaluates rollout-policy samples with masks, normalization, entropy weights, and a clipped likelihood-ratio surrogate. Appendix B.4 gives the implemented token proxy and its sampling convention. The exact-optimum statements describe frozen preferences and deviation costs; the component experiments assess the sampled neural update. Uniform-weight controls and entropy-weighted controls retain their stated global coefficients, while complete KL removal changes the reference constraint.

Implementation Details

Running rollout-entropy statistics

For an actor batch with MM rollouts, define

hˉb=1M∑i=1Mhitop,\bar{h}_b=\frac{1}{M}\sum_{i=1}^{M}h_i^{\mathrm{top}},
tb=1M−1∑i=1M(hitop−hˉb)2.t_b=\sqrt{\frac{1}{M-1}\sum_{i=1}^{M}\left(h_i^{\mathrm{top}}-\bar{h}_b\right)^2}.

The running statistics are updated as

μH←λHμH+(1−λH)hˉb,\mu_H\leftarrow\lambda_H\mu_H+(1-\lambda_H)\bar{h}_b,
τH←λHτH+(1−λH)tb.\tau_H\leftarrow\lambda_H\tau_H+(1-\lambda_H)t_b.

Both statistics are initialized from the first valid batch. The reported momentum is λH=0.99\lambda_H=0.99 and the denominator stabilizer is ϵu=10−6\epsilon_u=10^{-6}. The running scale is an EMA of batch standard deviations. Algorithm 1 updates these statistics before computing uiu_i.

Generalized continuous four-quadrant rule

For sensitivity comparisons, define

δigeneral=gi{ri[γPLRui+γPHR(1−ui)]−(1−ri)[γNHR(1−ui)+γNLRui]}.\delta_i^{\mathrm{general}}=g_i\left\{r_i\left[\gamma_{\mathrm{PLR}}u_i+\gamma_{\mathrm{PHR}}(1-u_i)\right]-(1-r_i)\left[\gamma_{\mathrm{NHR}}(1-u_i)+\gamma_{\mathrm{NLR}}u_i\right]\right\}.

The historical default sets (γPLR,γNHR,γPHR,γNLR)=(0.5,0.3,0,0)(\gamma_{\mathrm{PLR}},\gamma_{\mathrm{NHR}},\gamma_{\mathrm{PHR}},\gamma_{\mathrm{NLR}})=(0.5,0.3,0,0). The named terms form a continuous interpolation within each outcome class. The ui=0.5u_i=0.5 threshold defines the diagnostic count bins.

Degenerate groups, truncation, and recentering

If all outcomes in a group are identical, the base GRPO advantages are zero. Entropy corrections may still vary across rollouts, so the modified advantages need not be zero. After batch recentering, an incorrect rollout can receive a positive effective advantage relative to the batch, even when its direct correction is nonpositive. Effective advantages therefore need to be interpreted after centering.

For example, in an all-incorrect, untruncated batch, the default direct correction is −γ−(1−ui)-\gamma_{-}(1-u_i). Recentering gives

A~i=γ−(ui−1M∑juj).\widetilde{A}_i=\gamma_{-}\left(u_i-\frac{1}{M}\sum_j u_j\right).

A rollout with above-average uiu_i can therefore receive positive relative credit without being correct. The example concerns a finite empirical batch, while Proposition A.2 compares exact optima with a frozen baseline. Effective-advantage diagnostics can quantify this behavior during training.

The gate gi=1−1[yi is truncated]g_i=1-\mathbb{1}[y_i\text{ is truncated}] suppresses only the direct entropy correction. The gate tests truncation; zero-variance groups follow the shaping and centering rules above. Grading and normalization conventions determine which truncated rollouts participate in the batch. Padding masks exclude invalid token positions from actor and KL reductions.

Implemented entropy-weighted reference regularization

EA-KL varies the token-level price of reference deviation according to cached entropy. With active-token mask mi,tm_{i,t}, define a stabilized mean and the corresponding weights:

H‾=max⁡(∑i,tmi,tHi,t∑i,tmi,t,Hmin⁡),\overline{H}=\max\left(\frac{\sum_{i,t}m_{i,t}H_{i,t}}{\sum_{i,t}m_{i,t}},H_{\min}\right),
wi,t=clip⁡[1+α(1−Hi,tH‾),0,wmax⁡].w_{i,t}=\operatorname{clip}\left[1+\alpha\left(1-\frac{H_{i,t}}{\overline{H}}\right),0,w_{\max}\right].

For α≥0\alpha\ge0, tokens below H‾\overline{H} receive a larger relative deviation cost and those above it a smaller one. Uniform weights are recovered at α=0\alpha=0. The reported configuration uses α=0.20\alpha=0.20, Hmin⁡=5×10−3H_{\min}=5\times10^{-3}, and wmax⁡=1.5w_{\max}=1.5; the realized upper bound and zero-weight cases are described below.

The actor penalty uses the sampled nonnegative proxy

ki,t(θ)=exp⁡(di,t)−di,t−1,k_{i,t}(\theta)=\exp(d_{i,t})-d_{i,t}-1,
di,t=log⁡πref(ai,t∣ci,t)−log⁡πθ(ai,t∣ci,t),d_{i,t}=\log\pi_{\mathrm{ref}}(a_{i,t}\mid c_{i,t})-\log\pi_{\theta}(a_{i,t}\mid c_{i,t}),

aggregated as

K^w=∑i,tmi,twi,tki,t(θ)∑i,tmi,t.\widehat{K}_{w}=\frac{\sum_{i,t}m_{i,t}w_{i,t}k_{i,t}(\theta)}{\sum_{i,t}m_{i,t}}.

This proxy equals forward KL in expectation when the action is sampled from the current policy. The implemented actor update evaluates rollout-policy samples, and Appendix § 8.§ 3 distinguishes this sampled penalty from its exact distributional model. Gradient treatment, reduction, and any importance correction are part of the configuration metadata in Appendix § 10.

Weights and entropy statistics are detached. For the historical α=0.20\alpha=0.20, the raw expression satisfies wi,t≤1.20w_{i,t}\le1.20 for nonnegative entropy before upper clipping. The lower clipping threshold is reached when Hi,t/H‾>6H_{i,t}/\overline{H}>6. Appendix § 11.§ 7 supplies the PLR-stratified weight summaries. Overall mean weight and zero-weight fraction are unreported.

Adaptive weights can correlate with token KL values, so their arithmetic mean alone does not determine aggregate penalty strength.

Curriculum normalization and support

For a finite dataset with NDN_D questions, vq∈[η,1]v_q \in[\eta,1] implies

pk(q)>0,p_k(q) > 0,
pk(q)≥ηND.p_k(q) \ge\frac{\eta}{N_D}.

This bound preserves support at each draw. The normalized distribution can be expressed as a mixture with uniform sampling, whose mixture coefficient depends on the sum of Gaussian weights. Finite-run coverage depends on the realized draws.

Stored statistics are refreshed when their questions are sampled; their update ages are not recorded in the reported diagnostics.

Update procedure

Algorithm 1: GRPO-based MGE-PO. This procedure specifies the complete GRPO-based configuration. Each rollout round freezes the policy used for generation and its entropy measurements during the actor update. Question-statistic updates use the recorded rollout entropies and affect subsequent draws.

Input: training questions D, initial policy pi, fixed reference pi_ref,
       G completions per question, actor-update budget,
       shaping, KL-weight, and curriculum parameters.
Initialize xi_q = mu_ZPD for every question q in D.
For each rollout round k:
  1. Normalize stored question weights to obtain p_k; sample questions.
  2. Generate G completions per question with rollout policy pi_k.
     Cache valid-token entropy, rollout log probabilities, and masks.
  3. Grade final answers and compute group-relative base advantages.
  4. Compute top-entropy summaries h_top and full-token means h_mean.
     On the first valid batch, initialize mu_H and tau_H from its
     top-entropy statistics; on later batches, update their EMAs.
  5. Compute detached concentration scores u_i and gated corrections.
     Recenter the adjusted advantages over the actor batch.
  6. Compute detached token KL weights from cached entropy.
  7. Optimize the clipped actor surrogate minus beta times the
     weighted reference-deviation proxy; keep cached signals fixed.
  8. Update xi_q for sampled questions using their rollout mean entropy.
Return: final policy and training/evaluation configuration metadata.

Reproducibility Configuration

Reported settings

Table 3: Reported hyperparameters and configuration metadata. Values describe the original Qwen2.5 experiments and their supplementary settings. Appendix C.4 describes the separate Qwen3-8B-Base training and evaluation protocol.

CategoryQuantityReported value or status
TrainingReported global batch size128; question/response unit unspecified
TrainingCompletions per question16
TrainingDataset-equivalent epochs8
TrainingSampling temperature1.0
CategoryQuantityReported value or status
TrainingMaximum generation length12,288 tokens
TrainingTop-pp during rolloutNot reported
TrainingExact actor/optimizer update scheduleNot reported
TrainingActor-loss reductionNot reported
TrainingOptimizer, learning rate, schedulerNot reported
TrainingCheckpoint identifiersSupplementary Instruct: final step 1080; supplementary Base step unreported
EA-Advγ+\gamma_{+}0.5
EA-Advγ−\gamma_{-}0.3
EA-AdvSeparate PHR/NLR coefficients0, 0
EA-AdvrKr_{K}, Kmax⁡K_{\max}0.1, 4096
EA-AdvλH\lambda_{H}, ϵu\epsilon_{u}0.99, 10−610^{-6}
EA-AdvRecentering and masksBatch recentering specified; exact implementation masks unreported
EntropyProbability distributionPolicy distribution in the formulation; decoding-transform convention unreported
EA-KLα\alpha0.20
EA-KLHmin⁡H_{\min}5×10−35 \times10^{-3}
EA-KLConfigured weight bounds0, 1.5
KLOriginal MGE-PO coefficient10−310^{-3}
KLOriginal GRPO coefficient0.04
KLSeven nonempty component configurations10−310^{-3}; uniform weights when EA-KL is off
DynZPDμZPD\mu_{\mathrm{ZPD}}, σZPD\sigma_{\mathrm{ZPD}}0.30, 0.15
DynZPDWeight floor η\eta0.10
DynZPDQuestion EMA λq\lambda_{q}0.9
DynZPDInitial question statisticμZPD\mu_{\mathrm{ZPD}}
EvaluationOriginal temperature, top-pp1.0, 0.7; supplementary values unspecified
EvaluationSupplementary Minerva/Olympiad maximum length8,192 tokens
EvaluationSupplementary accuracy samples64 per question
EvaluationSupplementary AIME coveragepass@256, averaged across repeated inference evaluations
EvaluationPass@K repeat counts and inference seedsNot reported for either protocol
ComputeHistorical training hardware2×2 \times NVIDIA H200, 141 GB each
StatisticsTraining runs per RL method cellOne

Table 3. Table 3 (continued)

The entropy equation in § 3.2 defines the policy distribution before decoding transformations. Reproducing cached entropy measurements additionally requires the extraction stage relative to temperature and sampling transformations, together with the active-token masks. The run-specific decoding-transform convention is not specified in the current configuration record.

Dataset and evaluation metadata

The original Qwen2.5 evaluation uses MATH-500, AMC23, and AIME 2024/2025; its supplementary evaluation adds Minerva-Math/OCW and a text-only English exact-answer Olympiad subset. Benchmark names and counts are reported. Exact question manifests, model revisions, prompt templates, grader versions, extraction rules, and training–test overlap checks are not documented in the current record.

In the Qwen2.5 series, pass@KK averages repeated inference evaluations of a trained checkpoint. The repeat counts and inference seeds are unreported. The supplementary base comparison omits its checkpoint step. The Instruct comparison identifies final step 1080 after eight epochs and describes a “4k” setting without specifying its role.

Measurement scope

Each original Qwen2.5 RL model–method cell uses one training run. Its reported macro scores retain the original rounding, and pass@KK repeats are inference repetitions. The Qwen3-8B-Base table reports one point estimate per method and metric at the final controller step 624; its inference-repeat count and number of training seeds are unspecified. Training-seed standard deviations and paired problem-level intervals are unavailable.

Runtime values refer to complete single runs with matched training budgets. Peak-memory and detailed GPU-hour accounting are unavailable. Curriculum statistics cover a 21-step window containing 43,008 rollouts and 2,440 observed questions; full-run coverage and stored-statistic ages are unreported. The reported maximum question share of 0.4% has no documented denominator. For the default sampler with ND=17,398N_D = 17{,}398 and η=0.10\eta= 0.10, vq∈[η,1]v_q \in[\eta, 1] implies pk(q)≤1/[1+(ND−1)η]≈0.0574%p_k(q) \le1/[1 + (N_D - 1)\eta] \approx0.0574\%. The 0.4% entry therefore cannot be interpreted as the maximum single-question probability under this distribution; its measurement definition remains unresolved.

Qwen3-8B-Base training and evaluation protocol

The subset is sampled before filtering, leaving 9,998 training questions. The reported 624 controller steps constitute the full eight-epoch schedule. Table 1 uses the final checkpoint for every configuration; the periodic checkpoint interval is 50 controller steps. Data-subsampling seed 1 does not specify the model-training or inference seeds.

Evaluation uses identical test question sets and decoding settings across all seven configurations, sampling 32 responses per question at temperature 0.6. The reported mean@32 is the sample-average accuracy defined as avg@32 in § 6.1; pass@32 measures question coverage within the 32 responses. The six benchmarks receive equal weight in each macro score. The 20,480-token limit in Table 4 applies to training responses; the maximum evaluation response length, numerical learning rate, and remaining decoding parameter values are unspecified.

ConfigurationValue
Base modelQwen3-8B-Base
Training dataRandom 10,000-question subset of the Skywork-OR1 mathematical training set
Data-subsampling seed1
Training questions after filtering9,998
Train batch size128 questions
Training rollouts per question8
Maximum training response length20,480 tokens
Training epochs8
Complete training schedule624 controller steps
Periodic checkpoint intervalEvery 50 controller steps
Learning rateSame across all seven configurations
Evaluation benchmarksAIME2024, AIME2025, AMC2023, MATH500, Minerva, OlympiadBench; identical question sets across configurations
Evaluation responses per question32
Evaluation temperature0.6
ConfigurationValue
Evaluation decoding settingsSame across all seven configurations
Evaluation metricsmean@32 (denoted avg@32 in this paper), pass@32
Macro aggregationEqual-weight mean over the six benchmarks

Table 4. Training and evaluation protocol for the Qwen3-8B-Base experiments. The seed below controls dataset subsampling. All seven configurations share the learning rate, evaluation question sets, and decoding settings, and are evaluated at the final controller step 624.

Table 1 retains the per-benchmark scores and macro averages at their reported precision. The manuscript displays two decimal places; Table 16 computes differences before rounding the macro averages to that precision. Small differences between reported macro scores and means recomputed from the displayed per-benchmark values reflect their different display precision. The 4Q comparisons are GRPO versus 4Q, DAPO versus 4Q + DAPO, and Clip-Cov versus Clip-Cov + 4Q. These final-checkpoint comparisons do not estimate training-seed uncertainty.

Optimization settingavg@32 without 4Q (%)avg@32 with 4Q (%)Absolute Δ\Delta avg@32 (%)pass@32 without 4Q (%)pass@32 with 4Q (%)Absolute Δ\Delta pass@32 (%)
GRPO41.3145.00+3.6957.9769.03+11.06
DAPO43.6146.99+3.3866.3871.69+5.31
Clip-Cov43.2546.48+3.2368.0072.24+4.24

Table 16. 4Q ablation across three Qwen3-8B-Base optimization settings at the final controller step 624. Each row compares a method with its 4Q-augmented configuration in Table 1. The GRPO comparison uses the row labeled 4Q. Scores are six-benchmark macro averages (%); differences are absolute score changes (%) computed before rounding the macro averages to two decimals.

Qwen3 inherits the Qwen2.5 component defaults in Table 3, except for the training and evaluation settings in Table 4. For 4Q, γ+=0.5\gamma_{+} = 0.5 and γ−=0.3\gamma_{-} = 0.3, with separate PHR/NLR coefficients set to zero. Entropy pooling uses rK=0.1r_{K} = 0.1, Kmax⁡=4096K_{\max} = 4096, λH=0.99\lambda_{H} = 0.99, and ϵu=10−6\epsilon_{u} = 10^{-6} (Appendix § 9.§ 1). Where DynZPD is active, it uses the question statistic and update rule in § 4.3 with μZPD=0.30\mu_{\mathrm{ZPD}} = 0.30, σZPD=0.15\sigma_{\mathrm{ZPD}} = 0.15, η=0.10\eta= 0.10, and λq=0.9\lambda_{q} = 0.9. EA-KL uses α=0.20\alpha= 0.20, Hmin⁡=5×10−3H_{\min} = 5 \times10^{-3}, and weight bounds [0,1.5][0, 1.5]; the complete MGE-PO configuration retains β=0.001\beta= 0.001.

Component-default inheritance does not specify each host optimizer’s KL setting or its integration details. For the 4Q-host combinations, the run-specific host reward and base-advantage computation, correction insertion point relative to normalization, recentering population, truncation and token-selection statistic derived from the adjusted advantage remain undocumented. § 4.2 separates binary verification from the host’s base reward; § 5.2 retains its binary-reward GRPO conditions. These remaining record gaps limit exact reproduction of the host integrations.

Complete Reported Results and Additional Comparisons

Original per-benchmark scores

InitializationMethodA25 avg (%)A25 pass (%)A24 avg (%)A24 pass (%)AMC avg (%)AMC pass (%)M500 avg (%)M500 pass (%)Macro avg (%)Macro pass (%)
Qwen2.5-Math-1.5B BaseBase3.328.76.910.332.572.532.877.818.947.3
GRPO6.330.79.530.352.790.066.589.433.760.1
DAPO7.126.711.032.754.587.568.790.035.359.2
KL-Cov8.335.710.931.855.187.568.890.835.861.5
MGE-PO8.938.310.233.355.392.569.489.236.063.3
Qwen2.5-Math-7B BaseBase3.526.79.030.038.385.347.186.424.557.1
GRPO10.835.420.636.761.990.072.490.841.463.2
DAPO11.836.721.343.363.990.073.791.042.765.3
KL-Cov13.138.322.944.462.491.375.890.643.666.2
MGE-PO13.643.323.150.064.395.073.191.643.570.0
Qwen2.5-7B BaseBase1.36.72.316.720.070.061.875.421.342.2
GRPO9.623.320.830.358.792.576.788.241.558.6
DAPO10.125.722.136.759.587.578.390.442.560.1
KL-Cov12.728.322.639.561.491.280.889.844.462.2
MGE-PO12.930.323.240.065.590.079.890.845.462.8

Table 5. Full original results under avg@32/pass@32. Each RL method cell is based on one training run; Base rows evaluate the pre-RL initialization. Pass@32 averages repeated inference evaluations of that checkpoint. Macro averages are retained as reported; rounding of displayed per-benchmark values can produce small differences from a recomputed mean. A25/A24 denote AIME 2025/2024, AMC denotes AMC23, and M500 denotes MATH-500; avg/pass denote avg@32/pass@32. All scores use %.

The original generation limit is approximately 12k tokens. Table 6 reports the separate supplementary protocol; Appendix § 10 identifies the missing repeat and configuration metadata.

MethodMATH-500 avg@64 (%)AMC23 avg@64 (%)Minerva avg@64 (%)Olympiad avg@64 (%)AIME24 pass@256 (%)AIME25 pass@256 (%)
GRPO66.350.035.437.436.753.3
DAPO68.357.722.730.130.053.3
KL-Cov67.153.536.339.938.957.8
MGE-PO68.655.638.541.340.060.0

Table 6. Supplementary Qwen2.5-7B-Base evaluation. Accuracy uses 64 samples per question and AIME coverage uses 256. Methods share the initialization, reported training budget, and evaluation protocol. The KL-coefficient distinction is specified in Appendix § 11.10. All scores use %.

Supplementary Qwen2.5 evaluation and Clip-Cov comparison

The supplementary protocol evaluates Qwen2.5-7B-Base with 64 samples per question for accuracy and 256 samples for AIME coverage. It is reported separately from the original avg@32/pass@32 protocol and the six-benchmark Qwen3 protocol.

MGE-PO leads five of the six supplementary metrics. On Minerva, its absolute margins over GRPO, DAPO, and KL-Cov are 3.1%, 15.8%, and 2.2%; on Olympiad they are 3.9%, 11.2%, and 1.4%. These 272- and 674-question evaluations broaden the comparison beyond the smaller competition sets. AIME24/AIME25 coverage shows absolute gains of 3.3%/6.7% over GRPO and 1.1%/2.2% over KL-Cov. DAPO leads AMC23 by an absolute margin of 2.1%, while MGE-PO’s MATH-500 margin over DAPO is 0.3%.

The available supplementary Clip-Cov comparison covers Minerva and Olympiad under the same 64-sample accuracy protocol:

MethodMinerva-Math avg@64 (%)Olympiad avg@64 (%)
GRPO35.437.4
DAPO22.730.1
KL-Cov36.339.9
Clip-Cov35.138.4
MGE-PO38.541.3

Table 7. Available supplementary larger-benchmark comparisons. Qwen2.5-7B-Base; 64 responses per question, maximum generation length 8,192 tokens. The available Clip-Cov comparison covers these two benchmarks. All scores use %.

Existing aggregate quadrant diagnostic

Low-entropy incorrect responses account for 45.5% of initialization rollouts and 37.2% after GRPO. These aggregate counts establish the coexistence of predictive concentration and incorrectness; they do not identify paired trajectory transitions or MGE-PO error categories.

Evaluation datasets and protocols

The original Qwen2.5 protocol uses temperature 1.0 and top-pp 0.7. The supplementary Minerva/Olympiad generation limit is 8,192 tokens; Appendix § 10.§ 2 records the unresolved checkpoint and subset metadata. The separate Qwen3-8B-Base protocol uses avg@32 and pass@32 for all six benchmarks (Table 1; Appendix § 10.§ 4).

Instruct initialization

Table 10: Qwen2.5-Math-7B-Instruct at the final reported checkpoint. Both methods start from the same Instruct initialization. The first four columns use avg@64; AIME columns use pass@256. The record identifies step 1080 after eight epochs in the setting described as “4k”; the role of this setting is unspecified. All scores and absolute differences use %.

MethodMATH-500 (%)AMC23 (%)Minerva (%)Olympiad (%)AIME24 (%)AIME25 (%)
GRPO80.362.432.434.643.353.3
MGE-PO81.063.934.536.346.763.3
Absolute difference: MGE-PO minus GRPO (%)+0.7+1.5+2.1+1.7+3.4+10.0

Table 10.

The Instruct comparison yields absolute gains over GRPO of 0.7%, 1.5%, 2.1%, 1.7%, 3.4%, and 10.0% in Table 10’s benchmark order, extending the two-method comparison to one instruction-tuned initialization. Pre-RL Instruct scores and Instruct DAPO/KL-Cov comparisons are unavailable.

Quadrant-coefficient sensitivity

Table 11: Quadrant-coefficient ablation on Qwen2.5-Math-7B-Base. AIME24 uses the original avg@32/pass@32 protocol. Coefficients are nonnegative magnitudes in the generalized shaping rule: PLR/PHR terms are added and NHR/NLR terms are subtracted. Test-set selection scope is discussed below. Accuracy and coverage use %; quadrant coefficients are unitless.

The quadrant sensitivity study tests the two continuous correction terms and their relative strengths (Table 11). On Math-7B AIME24, removing either default term lowers pass@32 from 50.0% to 36.7%; swapping their magnitudes gives 40.0%. The default (γPLR,γNHR,γPHR,γNLR)=(0.5,0.3,0,0)(\gamma_{\mathrm{PLR}}, \gamma_{\mathrm{NHR}}, \gamma_{\mathrm{PHR}}, \gamma_{\mathrm{NLR}}) = (0.5, 0.3, 0, 0) performs best among the seven tested settings, and is shared across the three base models. The tested additional PHR/NLR terms also reduce performance. This comparison measures local sensitivity on AIME24, which is also used for final evaluation; an independent development-set selection procedure is not established in the experimental record.

ConfigurationPLRNHRPHRNLRavg@32 (%)pass@32 (%)
Default0.50.30023.150.0
PLR-weighted term removed00.30021.336.7
NHR-weighted term removed0.500022.236.7
Magnitudes swapped0.30.50022.640.0
Additional PHR term0.50.30.3021.836.7
NHR-weighted term replaced by NLR-weighted term0.5000.321.633.3
Additional NLR term0.50.300.122.436.7

Table 11.

Reference regularization and full-method dynamics

DiagnosticGRPO / comparison valueMGE-PO / EA-KL valueScope
Mean EA-KL weight in lower-entropy PLR token stratumNot applicable1.19Active PLR tokens, nine updates
Mean EA-KL weight in higher-entropy PLR token stratumNot applicable0.35Same token window
Lower-/higher-entropy weight ratioNot applicable3.1–3.4 across updatesSame token window
Late-training policy entropy (nats/token)Approximately 0.001Approximately 0.06Original Qwen2.5-7B-Base training curves
Late-training response length (tokens)Approximately 6,000Approximately 4,500Original Qwen2.5-7B-Base training curves

Table 12. Table 12: Token-weight and training-dynamics diagnostics. The token analysis covers approximately 1.78 million active PLR tokens over nine actor updates. Entropy and response-length entries summarize the original plain-7B training curves and have a different scope from the token window.

The PLR token window records lower-entropy mean weights of 1.19 and higher-entropy mean weights of 0.35, with an update-wise ratio of 3.1–3.4. These measurements characterize the implemented allocation of reference cost; the entropy-stratum boundaries are unspecified. Full-method curves combine EA-KL, 4Q, sampling, and the global KL coefficient.

Complete removal of reference KL is a separate intervention from disabling adaptive weights. In the reported no-KL comparison (β=0\beta= 0), training-end reference KL rises from approximately 0.5 to 3 nats and all four benchmark scores show absolute decreases of 4–7%. This supports retaining reference regularization in the tested setting. Detailed per-benchmark values and the score metric are unreported.

Complete-run timing

The original experimental hardware is two NVIDIA H200 GPUs with 141 GB each. Entropy is cached as one scalar per generated token. Separate memory overhead was not measured and depends on datatype and residency.

Curriculum concentration

Table 14: Curriculum diagnostics in a 21-step window. The window contains 43,008 rollouts and 2,440 unique MGE-PO questions from the 17,398-question training dataset. Relative ESS, mass, and Gini describe the sampling distribution; unique-question coverage distinguishes uniform expectations from observed draws.

For question distribution pp, ESS⁡(p)=1/∑qp(q)2\operatorname{ESS}(p) = 1/\sum_q p(q)^2 and relative ESS is 100ESS⁡(p)/∣D∣100\operatorname{ESS}(p)/|D| in %. The averaging convention over the window is unspecified. Appendix § 10.§ 3 explains the unresolved definition of the 0.4% maximum-share entry. The finite window characterizes observed concentration and coverage in § 6.4.

Qwen2.5 component configurations

Table 15: Component configurations on Qwen2.5-7B-Base. The score is four-benchmark macro avg@32. EA-KL off retains uniform reference KL.

EA-AdvEA-KLDynZPDReference coefficient β\betaMacro avg@32 (%)Absolute difference from GRPO (%)
OffOffOff0.0441.50.0
OnOffOff0.00143.3+1.8
OffOnOff0.00142.2+0.7
OffOffOn0.00141.6+0.1
OnOnOff0.00144.3+2.8
OnOffOn0.00143.9+2.4
OffOnOn0.00142.5+1.0
OnOnOn0.00145.4+3.9

Table 15.

4Q improves all three available shared-coefficient comparisons that add it to a supporting configuration. With β=10−3\beta= 10^{-3}, adding 4Q to EA-KL raises macro avg@32 from 42.2% to 44.3% (+2.1%), and adding it to DynZPD raises the score from 41.6% to 43.9% (+2.3%). Adding 4Q to EA-KL+DynZPD yields 42.5% to 45.4% (+2.9%), the largest conditional gain among removals from the complete configuration. These observed contrasts support outcome-conditioned entropy credit as the central method component.

The complete score also depends on its supporting settings. Adding EA-KL to 4Q+DynZPD gives 43.9% to 45.4% (+1.5%), while adding DynZPD to 4Q+EA-KL gives 44.3% to 45.4% (+1.1%). Thus 45.4% belongs to the complete implemented configuration. EA-KL off retains uniform reference KL with α=0\alpha= 0; on uses α=0.20\alpha= 0.20. The GRPO row uses β=0.04\beta= 0.04, while all seven nonempty configurations use 0.001. Full-method differences from GRPO include this coefficient change; the shared-coefficient contrasts report conditional component differences in single training runs. Adaptive weighting can additionally alter realized aggregate penalty strength. Appendix § 11.§ 7 retains the separate complete-KL-removal observation.

Qwen3-8B paired 4Q comparisons

The paired values below supplement Figure 3. All methods use the final controller step 624 and the common evaluation protocol in Appendix § 10.§ 4.

Both macro metrics improve in all three settings, while individual benchmarks can decrease (Figure 5).

Formulation Comparison

Advantage and allocation formulas in Table 17. The following expressions summarize the interventions compared in Table 17.

MethodSignal and granularityAdvantage interventionKL interventionData selection
DAPO ([10])Group outcomesGroup-relative advantageFrozen-reference KL omittedRetains nondegenerate reward groups
EntroAdv ([3])Token predictive entropyDetached entropy bonus, capped by advantage magnitude; Equation (E1)——
SEED-GRPO ([2])Question semantic entropy SE⁡(q)\operatorname{SE}(q)Semantic-entropy attenuation of advantage magnitude; Equation (E2)——
EAPO v1 ([6])Group-normalized token entropy zi,tz_{i,t} and advantage polaritySigned, capped token-entropy correction; Equation (E3)——
KL-Cov ([4])Token covariance between log probability and advantageBase policy-gradient termβDKL(πold∥πθ)\beta D_{\mathrm{KL}}(\pi_{\mathrm{old}}\Vert\pi_{\theta}) on top-covariance tokens—
SENT ([1])Question semantic entropy; token entropy and covarianceBase GRPO advantagePiecewise coefficients: zero on high-entropy tokens; larger on high-covariance low-entropy tokensInitial-policy semantic-entropy sorting, easy-to-hard stages
EDCO ([7])Prefix-based inference-entropy estimateUnderlying SFT or RL objectiveUnderlying training objectivePeriodic top-NN high-entropy subset selection
MGE-PORollout top-KK entropy; relative token entropy; question EMAOutcome-conditioned rollout correction; Equation (E4)Continuous entropy weights multiply reference KL; Equation (E5)Online intermediate-entropy sampling with a positive floor; Equation (E6)

Table 17. Comparison of intervention rules. Hi,tH_{i,t} denotes token predictive entropy, hitoph_i^{\mathrm{top}} the top-KK pooled rollout statistic, uiu_i its decreasing logistic transform, rir_i the binary outcome, gig_i the truncation gate, and ξq\xi_q the question EMA. AiA_i is the base GRPO advantage. ΔQ\Delta Q is MGE-PO’s central credit-assignment rule; its curriculum and entropy-weighted reference penalty are included to specify the complete implemented configuration. The MGE-PO advantage expression precedes batch re-centering. An em dash indicates that the cited method introduces no separate intervention in that column.

At′=At+min⁡(αHtdetach,∣At∣κ),κ>1.(13)A'_t = A_t + \min\left(\alpha H_t^{\mathrm{detach}}, \frac{\lvert A_t\rvert}{\kappa}\right), \qquad\kappa> 1. \tag*{(13)}
Ai′=Aif(αSE⁡(q)SE⁡max⁡(q)).(14)A'_i = A_i f\left(\frac{\alpha\operatorname{SE}(q)}{\operatorname{SE}_{\max}(q)}\right). \tag*{(14)}

In Equation (E2), ff is the nonnegative semantic-entropy attenuation function and SE⁡max⁡(q)=log⁡G\operatorname{SE}_{\max}(q)=\log G for GG sampled responses to question qq. Chen et al. [2] consider linear, exponential, and focal attenuation functions.

Ai,t′=Ai,t+αsign⁡(Ai,t)detach⁡[clip⁡(zi,t,−∣Ai,t∣/φ,∣Ai,t∣/φ)].(15)A'_{i,t}=A_{i,t}+\alpha\operatorname{sign}(A_{i,t})\operatorname{detach}\left[\operatorname{clip}\left(z_{i,t},-\lvert A_{i,t}\rvert/\varphi,\lvert A_{i,t}\rvert/\varphi\right)\right]. \tag*{(15)}
Ai′=Ai+gi[riγPLRui−(1−ri)γNHR(1−ui)].(16)A'_i=A_i+g_i\left[r_i\gamma_{\mathrm{PLR}}u_i-(1-r_i)\gamma_{\mathrm{NHR}}(1-u_i)\right]. \tag*{(16)}
wi,t=clip⁡(1+α(1−Hi,t/H‾),0,wmax⁡).(17)w_{i,t}=\operatorname{clip}\left(1+\alpha\left(1-H_{i,t}/\overline{H}\right),0,w_{\max}\right). \tag*{(17)}
Pt(q)∝η+(1−η)exp⁡[−(ξq−μ)2/(2σ2)].(18)P_t(q)\propto\eta+(1-\eta)\exp\left[-(\xi q-\mu)^2/(2\sigma^2)\right]. \tag*{(18)}

The EAPO formulation in this comparison is pinned to arXiv:2604.11056v1. The second version uses the title Where Hindsight Credit Can Reside: A Signed-Capacity View of Token Updates in RLVR and the method name HAPO.

Supplementary Figures

Per-benchmark 4Q effects on Qwen3-8B-Base

Figure 5 expands the macro comparison in Figure 3 into its six benchmark contributions. Adding 4Q to GRPO improves both metrics on all six benchmarks. The DAPO and Clip-Cov comparisons include local decreases alongside their positive macro gains.

Original entropy-stratified comparison

Figure 6 groups questions by the initial model’s generation entropy and compares GRPO with the complete MGE-PO configuration. The highest-entropy bucket has lower reported scores under MGE-PO. These are full-configuration differences across entropy strata.

Paired bar charts comparing GRPO and MGE-PO mean@32 and pass@32 across five base-model entropy buckets

Figure 6. Original scores stratified by initial-model entropy. The pooled AIME24, AIME25, and AMC23 set contains 100 questions, divided into five equal-frequency buckets of 20 using top-KK pooled entropy from 64 base-model completions per question (pooling fraction 0.1). Paired bars show GRPO and complete MGE-PO mean@32 (left) and pass@32 (right), both in %; annotations give absolute score differences (%). The source does not specify the backbone. This is a full-configuration diagnostic and is separate from the Qwen3-8B 4Q comparisons.

Original full-configuration training trajectories

Figure 7 retains the original Qwen2.5-7B-Base training curves. Late entropy is approximately 0.06 nats/token for MGE-PO versus 0.001 for GRPO, and response length approximately 4,500 versus 6,000 tokens. The early pass@32 advantage contracts later in training. The plotted endpoints differ from Table 2’s summary scores; their checkpoint correspondence is undocumented. These curves describe complete configurations and do not isolate 4Q. Figure 4 presents the available shared-coefficient conditional comparisons.

Figure 7. Original complete-configuration training trajectories on Qwen2.5-7B-Base over 1080 steps. Panels show four-benchmark mean@32 (avg@32), pass@32, response length, and token entropy for GRPO, DAPO, and MGE-PO. These trajectories are separate from Table 2’s summary scores; their checkpoint correspondence is undocumented. The curves are retained as supplementary diagnostics of complete configurations, not as a 4Q component comparison.

References

  1. [1]Hongye Cao, Zhixin Bai, Ziyue Peng, Boyan Wang, Tianpei Yang, Jing Huo, Yuyao Zhang, and Yang Gao. 2025. Efficient Reinforcement Learning with Semantic and Token Entropy for LLM Reasoning. arXiv:2512.04359. Paper.arxiv.org/abs/2512.04359
  2. [2]Minghan Chen, Guikun Chen, Wenguan Wang, and Yi Yang. 2025. SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization. arXiv:2505.12346. Paper.arxiv.org/abs/2505.12346
  3. [3]Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Xin Zhao, Zhenliang Zhang, and Furu Wei. 2026. Reasoning with Exploration: An Entropy Perspective. Proceedings of the AAAI Conference on Artificial Intelligence, 40(36):30377–30385. DOI:10.1609/aaai.v40i36.40290. Paper.DOI
  4. [4]Ganqiu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, Zheyuan Liu, Hao Peng, Lei Bai, Wanli Ouyang, Yu Cheng, Bowen Zhou, and Ning Ding. 2025. The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models. arXiv:2505.22617. Paper.arxiv.org/abs/2505.22617
  5. [6]Yuhang He, Haodong Wu, Siyi Liu, Hongyu Ge, Hange Zhou, Keyi Wu, Zhuo Zheng, Qihong Lin, Zixin Zhong, and Yongqi Zhang. 2026. Rethinking Token-Level Credit Assignment in RLVR: A Polarity-Entropy Analysis. arXiv:2604.11056v1, April 13, 2026. Paper.
  6. [7]Jing-Cheng Pang, Liu Sun, Chang Zhou, Xian Tang, Haichuan Ma, Kun Jiang, Jianlong Wang, Kai Zhang, Shijie Wu, Haoran Cai, Chenwei Wu, Xubin Li, and Xin Chen. 2026. EDCO: Dynamic Curriculum Orchestration for Domain-specific Large Language Model Fine-tuning. arXiv:2601.03725. Paper.arxiv.org/abs/2601.03725
  7. [9]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. W. Yu, and Daya Guo. 2024. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300. Paper.arxiv.org/abs/2402.03300
  8. [10]Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mof an Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. 2025. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476. Paper.arxiv.org/abs/2503.14476
  9. [11]Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. 2019. A Theory of Regularized Markov Decision Processes. Proceedings of the 36th International Conference on Machine Learning, PMLR 97:2160–2169. Paper.arxiv.org/abs/1901.11275
  10. [12]Youssef Mroueh. 2026. Reinforcement Learning with Verifiable Rewards: GRPO’s Loss, Dynamics, and Success Amplification. Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:90821–90847. Paper.arxiv.org/abs/2503.06639
  11. [13]Jean Vassoyan, Nathanaël Beau, and Roman Plaud. 2025. Ignore the KL Penalty: Boosting Exploration on Critical Tokens to Enhance RL Fine-Tuning. Findings of the Association for Computational Linguistics: NAACL 2025, pages 6123–6133. Paper.
  12. [14]Daya Guo et al. 2025. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature, 645:633–638. Paper.DOI
  13. [15]Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring Mathematical Problem Solving With the MATH Dataset. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks. Paper.
  14. [16]Aitor Lewkowycz et al. 2022. Solving Quantitative Reasoning Problems with Language Models. Advances in Neural Information Processing Systems. Paper.arxiv.org/abs/2206.14858
  15. [20]Mark Chen et al. 2021. Evaluating Large Language Models Trained on Code. arXiv:2107.03374. Paper.arxiv.org/abs/2107.03374
  16. [21]Andrey Malinin and Mark Gales. 2021. Uncertainty Estimation in Autoregressive Structured Prediction. International Conference on Learning Representations. Paper.arxiv.org/abs/2002.07650
  17. [22]Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation. International Conference on Learning Representations. Paper.arxiv.org/abs/2302.09664
  18. [23]Yikai Zhou, Baosong Yang, Derek F. Wong, Yu Wan, and Lidia S. Chao. 2020. Uncertainty-Aware Curriculum Learning for Neural Machine Translation. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6934–6944. Paper.DOI
  19. [24]Emmanouil Antonios Platanios, Otilia Stretcu, Graham Neubig, Barnabas Poczos, and Tom Mitchell. 2019. Competence-based Curriculum Learning for Neural Machine Translation. Proceedings of NAACL-HLT, Volume 1, pages 1162–1172. Paper.
  20. [25]Hunter Lightman et al. 2024. Let’s Verify Step by Step. International Conference on Learning Representations. Paper.arxiv.org/abs/2305.20050
  21. [26]Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. 2024. Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, Volume 1: Long Papers, pages 9426–9439. Paper.

Paper details

vlnc:W_6wvzctz7
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents