WorksComputer Science
Are Dense Rewards Worth Their Tokens in GRPO?
Abstract
RLVR trains language models to reason by rewarding correct final answers. A widely used algorithm, group-relative policy optimization (GRPO), gives every token of a solution the same credit, so dense process rewards have been proposed to credit intermediate steps, for example with a Monte-Carlo reward that scores each step by sampling continuations from it. Comparing such a reward with GRPO per training update, or vetting it with a gradient test on a single batch, leaves out what it costs: in our setting, the continuations alone take 91% of all generated tokens. We introduce a matched-compute protocol that gives every training intervention the same budget, measured in generated tokens and training GPU-hours, and tests it in preregistered studies with controls for side effects. Across 1,668 runs on six models, matching compute reverses the ranking. On Qwen2.5-1.5B, the Monte-Carlo reward leads standard GRPO per update, yet ends 14.3 percentage points lower at equal tokens and 7.8 points lower at equal training time, and none of four dense-reward variants outperforms GRPO even with as much training time. A per-token gradient test orders these rewards as training does but overestimates each by one to two points, as much as the gains it is meant to detect. In contrast, training on the prompts whose predicted success is nearest 1/2 improves accuracy at matched compute, by 1.7 points on Qwen3-0.6B and 0.8 on Qwen2.5-1.5B. These gains replicate on fresh seeds and unseen problems, and a simple rule, that the candidate prompts are predicted to fail more often than not, predicted where they appear in five of six preregistered tests. Both gains come early in training, and standard GRPO with twice the initial learning rate comes within half a point of them. We argue that training interventions should be validated at matched compute, which overturns some apparent gains and confirms others.
Introduction
Reinforcement learning with verifiable rewards (RLVR) has become a standard way to improve the reasoning of large language models [26, 5]. The model samples solutions to problems whose final answers a verifier can check, and training makes the correct solutions more likely. A widely used algorithm, group-relative policy optimization (GRPO), weights every token of a sampled solution, or completion, by the same advantage, the completion’s reward relative to the others sampled for the same prompt. A solution that sets up a problem correctly but slips in its last step is penalized for its good steps as well as its bad one. Dense rewards, also known as process rewards, give different steps different credit, derived from human step labels [16], from a prover’s progress [24], or from rewards learned on outcome labels [4]. A step can also be scored by the probability of reaching the correct answer from it, estimated
by sampling continuations from the partial solution, giving a Monte-Carlo reward [31, 15], or predicted by a learned value model (a critic).
The hidden cost of dense rewards. How should we decide whether a dense reward helps? Two checks are cheap enough to run on any candidate reward, and a reward that passes both looks worth using. A gradient test takes a fixed batch of samples and asks whether the reward’s update points in the same direction as an independent estimate of the gradient of expected accuracy (Section 2.4). A per-update comparison trains one model with the reward and one with outcome-only GRPO, the algorithm above, which credits only the final answer, and asks which is more accurate after the same number of updates. However, neither check considers what a dense reward costs. First, it consumes generated tokens. In our setting, the Monte-Carlo reward’s continuations account for 91% of all generated tokens, so with the same tokens, outcome-only GRPO makes 584 updates and the Monte-Carlo reward 50 (Figure 1). Second, it consumes GPU time, though not in proportion to its tokens. The continuations are only generated and scored, never trained on, and an inference engine generates tokens in large batches more cheaply than a training step processes them. With the same tokens, outcome-only GRPO therefore trains three times as long. A reward that wins per update can lose per token, and a comparison that counts only tokens can mislead about GPU-hours.
Even without counting cost, the gradient test can mislead. In our probes, the Monte-Carlo reward’s update aligns closely with a reference gradient (mean cosine 0.84). Yet even with the same number of updates as outcome-only GRPO, adding it with a small weight is equivalent to outcome-only GRPO within points of greedy accuracy, and adding it with a large weight loses 12.4 points (Section 3.1). We therefore ask whether a dense reward that passes the gradient test still improves GRPO once its compute is counted, and if not, where the generated tokens should go instead.
Our approach. We compare training interventions at matched compute: every method gets the same budget, measured on the same runs in generated tokens, including continuations and discarded samples, and in training GPU-hours, broken down by training stage. We apply this protocol to two interventions that spend compute in opposite ways. Dense rewards spend generated tokens to improve each update, whereas prompt selection spends one forward pass over candidate prompts to choose those whose predicted success is near , since a group whose completions are all correct or all wrong produces no update. We reimplemented the GRPO trainer and ran 1,668 runs (1,468 GPU-hours) on six Qwen and Llama models. Because the gains we expected were small, comparable to the variation between random seeds, we preregistered each study, i.e., fixed its hypotheses, primary comparisons, and equivalence margins in a written plan before running it. Configurations compared within a study share prompts and evaluation samples under each seed, so each seed yields a paired difference. Controls separate an intervention’s intended effect from its side effects: random choice among the same candidate prompts (for a faster pass through the prompt stream), a baseline with a retuned learning rate (for an under-tuned baseline), and a strict verifier (for drift in what the reward accepts).
Findings. Our metric is the accuracy of sampled solutions to held-out MATH problems (pass @1), and we report differences in percentage points.
Matching compute reverses the ranking. The Monte-Carlo reward leads outcome-only GRPO at equal updates but ends 14.3 points behind at equal tokens and 7.8 behind at equal training time, in every seed. Even given as much training time as outcome-only GRPO, none of the four dense-reward variants we tested shows a gain (Figure 3a).
Figure 3. At equal training time, no tested dense-reward variant showed a gain, and prompt selection keeps most of its lead. (a) A preregistered study ran every configuration beyond the training time that outcome-only GRPO needs to reach the token budget (dashed line). Curves show each configuration minus outcome-only GRPO at equal training hours (MATH L4–5, mean over seeds); open markers show where each configuration reaches . Bars give the preregistered contrasts at the dashed line with 95% CIs, and the legend gives their means. (b) The prompt filters, and random choice among the filter’s candidates, minus outcome-only GRPO at equal training hours, up to the time the faster configuration takes to reach the budget (exploratory, H200 GPUs). Bands are 95% CIs; the gray bands mark .
Gradient tests rank but cannot certify. Per generated token, the efficiency test orders the Monte-Carlo variants as training does, yet it overestimates the accuracy reached with every reward we checked by one to two points, as large as the gains we measured.
Choosing which prompts to train on yields a gain that the protocol confirms and explains. Training on the candidate prompts whose predicted success is nearest improves accuracy by 1.7 points on Qwen3-0.6B and 0.8 on Qwen2.5-1.5B at equal tokens, and by 1.0 and 0.5 at equal training time (exploratory). Both gains replicate on fresh seeds and on 1,543 unseen MATH test problems, and random choice among the same candidates does not reproduce them. The gain appears where the candidates are predicted to fail more often than not, a rule that held in five of six preregistered tests, and outcome-only GRPO with twice the initial learning rate
comes within half a point of it.
(4) A lenient verifier changes what training optimizes. With a verifier that also accepts answers outside , runs gradually stopped boxing their answers. A strict verifier removed this drift without lowering accuracy.
Our contributions are a matched-compute protocol for evaluating RL training interventions, whose trainer, preregistrations, and run records we release (§ 2), and two case studies in which the protocol rejects one intervention and confirms another (§ 3 and § 4). We also characterize exactly when a dense reward’s update on a fixed batch equals some reweighting of whole completions (§ 22), and we draw lessons for validating training interventions (§ 6).
Background and Protocol
GRPO and Credit
For a prompt , GRPO samples a group of completions (rollouts) from the policy , and a verifier gives each completion a reward , which is 1 if its final answer is correct. Each completion’s advantage is its reward relative to the rest of the group:
A policy-gradient update multiplies the log-probability gradient of each sampled token by a scalar weight, the token’s credit :
where is the log-probability gradient, or score, of the -th token of completion . Outcome-only GRPO gives every token of a completion the same credit, . Its update, , is therefore a weighted sum of the completions’ summed scores . We write for the set of all such weighted sums, that is, of all updates that weight each completion as a whole. Any update in could also come from reweighting whole completions, so what a dense reward adds beyond such a reweighting lies outside (§ 2.4). A group whose completions all succeed or all fail has for every and produces no update.
A dense reward makes the credit vary within a completion. It adds a token-level term, , that raises the credit of steps estimated to increase the chance of success and lowers the credit of steps estimated to decrease it. Its update can therefore leave . Ratio clipping, KL penalties, and the optimizer can transform .
Interventions and What They Cost
We test two kinds of intervention (Figure 1, Table 2): dense rewards, which change the credit , and prompt selection, which changes which prompts are sampled. Their costs fall on generated tokens, on training time, or on both. Section 3 also tests a learned critic and two variants that reuse samples, and Section 4 tests simpler selection rules; we describe each where we test it.

Figure 1. Counting every token changes which intervention wins. Four ways to train Qwen2.5-1.5B-Instruct with GRPO on MATH, compared at a matched budget of tokens generated during training (12.25M). Bars show accuracy minus outcome-only GRPO, to scale (5 seeds for the Monte-Carlo reward, 24 for the critics, and 60 for prompt selection, 54 on Qwen3-0.6B); light bars give the same comparisons at equal training GPU-hours on H200s (exploratory).
| Intervention | Qwen 2.5-1.5B | Qwen 3-0.6B | Qwen 3-1.7B | Qwen 3-4B | Llama-3.2 1B | Llama-3.2 3B |
| Completion limit | 380 | 768 | 768 | 768 | 768 | 768 |
| Monte-Carlo reward | • | ◦ | ◦ | ◦ | – | – |
| Learned-critic reward | • | ◦ | ◦ | ◦ | – | – |
| Rollout reuse, VinePPO-style | • | – | – | – | – | – |
| Continuation reuse | • | – | – | – | – | – |
| Critic filter | • | • | • | • | • | • |
| Random choice of candidates | • | • | • | – | – | – |
| History, level, dynamic sampling | • | ◦ | ◦ | – | – | – |
| Retuned learning rate | • | • | – | – | – | – |
| Strict verifier as reward | – | • | – | • | • | • |
Table 2. Which interventions were tested at which model. •: preregistered study with 8 or more seeds; ◦: pilot or pooled estimate with 3–9 seeds per model; –: not tested. Completion limit in tokens; on Qwen2.5-1.5B, a preregistered study also compared the Monte-Carlo reward and both kinds of reuse at 768 tokens, to equal training time.
Monte-Carlo reward. Following [31, 15], we cut each completion at five fixed token positions, which splits it into six segments, and sample continuations from the prefix that ends at each cut. The fraction of continuations that end with a correct answer estimates the prefix’s value, i.e., the probability that it leads to a correct answer. Each segment’s credit is the change in estimated value from its start to its end. The continuations are only scored and then discarded, so this reward costs generated tokens: at , it uses 91% of all generated tokens.
Prompt selection. Because uniform groups produce no update (§ 2.1), several methods aim training at prompts whose success rate is near ([9, 28] (Qu et al. 2026)). In each training iteration, our critic filter draws 24 candidate prompts and predicts each one’s success with a ridge critic on the policy’s hidden state at the end of the prompt. It trains on the 6 candidates whose predictions are nearest , plus 2 others chosen at random. Scoring the candidates takes one forward (prefill) pass and generates no tokens. As a control, random choice trains on 8 of the same 24 candidates chosen at random.
Matched-Compute Protocol
Models and recipe. Resolving small effects takes many paired seeds, so we train six small models: Qwen2.5-1.5B-Instruct (Qwen Team, 2024; [33]) with completions of up to 380 tokens, and Qwen3 0.6B, 1.7B, and 4B in non-thinking mode ([34]) and Llama-3.2-1B- and 3B-Instruct (Graffiti et al., 2024) with completions of up to 768 tokens. The training pool consists of MATH problems with numeric answers ([13]), so that the verifier’s answer check is well defined. It uses levels 4–5 in the dense-reward studies and levels 3–5 in the prompt-selection studies, except those that test level 5 or levels 1–2 alone. Each training iteration uses 8 prompts with 8 completions each and takes one on-policy Adam step at learning rate , without ratio clipping, sample reuse (except in the reuse variants), or a KL penalty.
Two budgets. Generated tokens and training time can rank methods differently (§ 1), so we set both. The token budget counts every token generated during training, including continuations and the groups that dynamic sampling discards; a run ends once it has generated tokens. A preregistered rule set for each model to the number of tokens that the Monte-Carlo reward generates in 48 iterations ( for Qwen2.5-1.5B; Section C.1 lists the others). Even with the most expensive reward, training thus makes about 50 updates. Training time, in GPU-hours, counts every training stage of a run: prompt scoring, generation, credit computation, the update, and the weight sync, but not evaluation or logging. To compare at equal training time, we compare the runs of one study, on one GPU type, at the time the faster configuration takes to reach . We also report every comparison at an equal number of updates, the comparison we argue against.
Metric. We measure accuracy, the quantity that training and the gradient tests target: the fraction of 8 completions, sampled at temperature 1, that the verifier marks correct (pass@1), averaged over 200 held-out numeric-answer MATH test problems. We take it from the first evaluation at or after the budget , without selecting checkpoints. We report differences in accuracy in percentage points; figure axes show them as fractions (0.01 is one point).
Preregistration and statistics. Before its runs, each study fixed in a plan file its hypotheses, its primary contrasts, its Holm families, and its equivalence margin ( points, or in four studies on Qwen2.5-1.5B). We report means with 95% paired- intervals over seeds in brackets. We label each claim primary (preregistered, with -values adjusted for multiple comparisons by Holm’s method), secondary (preregistered, unadjusted), or exploratory (not preregistered, marked †). To show that two configurations are equivalent within a margin, we use two one-sided tests (TOST). A crossed bootstrap over seeds and test problems checks every primary contrast. Tables 3 and 4 list every contrast, and Section C reports every secondary and exploratory analysis.
Verifier. The verifier defines the reward, so a lenient one can change what training optimizes. The default verifier credits the boxed answer, or the last number in the completion when no answer is boxed; the strict verifier credits only boxed answers. Runs trained with the default verifier can drift away from boxing their answers, which can bias a comparison in either direction. The strict verifier gives no reward for unboxed answers and so rules out this drift; studies marked “strict” train with it.
Gradient Tests
Gradient tests are a cheap way to vet a dense reward before training it (Section 1), so we measure two and check in Section 3.3 whether they predict training. Both compare a method’s update, on fixed batches at one policy, with an estimate of the gradient of expected accuracy from separately sampled continuations. Alignment is the cosine between the components of a dense update and of that lie outside , the only part of a dense update that can improve, to first order, on reweighting whole completions. Efficiency, for an update direction , also accounts for how noisy an update is, and for a smooth objective it is proportional to the best expected gain of one SGD step along . We compute it over 96 fixed batches at the initial policy and report it relative to outcome-only GRPO, per update and per generated token; the per-token value divides by the tokens each update consumes.
Section A gives further details of the recipe, the statistics, the verifier, and the gradient tests.
Case Study: Dense Rewards
Of the interventions we test, dense rewards cost the most compute, and they are the ones that per-update comparisons and gradient tests favor. We examine them in three ways: per update, where they lead (Section 3.1); per generated token and per GPU-hour, where they lose (Section 3.2); and through the gradient efficiency test, which anticipates the reversal but overestimates every reward it rates (Section 3.3).
Does a Dense Reward Help per Update?
The gradient test favor it. We start with a per-update comparison, an equal-iteration study. It compares configurations after the same number of iterations and uses its own recipe: Qwen2.5-1.5B-Instruct, fully fine-tuned at learning rate for 32 iterations and scored by greedy validation accuracy (Section G). In that study, the Monte-Carlo reward is centered within each completion, normalized, and added with weight : . On four full-parameter probes with disjoint rollouts, the component of its update outside has mean cosine 0.84 with that of the reference gradient (Section G). To check that alignment is not automatic for any reward whose update leaves , we include two control rewards, whose updates lie outside to a similar extent (the part outside carries 0.86–0.95 of the update’s norm for all three rewards) but do not align with the reference. One is an implicit reward log in the style of PRIME; the other shuffles the Monte-Carlo reward within each completion. Their cosines are and . In the matched-compute recipe, the Monte-Carlo reward is 1.56× [1.28, 2.03] as efficient as outcome-only GRPO per update, and with the same number of updates it leads outcome-only GRPO by 1.5 [1.0, 2.1] points† (Figure 2a).

Figure 2. The gradient test ranks the Monte-Carlo reward first per update and last per generated token, the same order training follows at matched tokens; the critics’ higher per-token efficiency brings at most 0.7 points. Qwen2.5-1.5B on MATH L4–5. Horizontal: gradient efficiency relative to outcome-only GRPO § 2.4, per update (a) or per generated token (b). Vertical: accuracy minus outcome-only GRPO on the same seeds at equal updates (a) or equal tokens (b). Red: the Monte-Carlo reward with (triangle), 2 (square), and 4 (circle) continuations per cut (12 fresh seeds); green: the two critic forms (24 fresh seeds). In (a), outcome-only GRPO is interpolated to each reward’s update count (exploratory). Dashed line in (b): the heuristic extrapolation for a reward of that efficiency. Bars are 95% CIs; the band marks .
Training does not follow, even at an equal number of iterations. In the equal-iteration study, outcome-only GRPO improves greedy accuracy by 5.6 [4.1, 7.1] points. In a prespecified expansion to 14 paired seeds, the Monte-Carlo reward with a low weight (, 0.5, or 1) is equivalent to outcome-only GRPO within ±3 points (Holm-corrected bootstrap and paired- TOST), and with the high weight it loses 12.4 [8.7, 16.1] points (). Other recipes point the same way. On Qwen2.5-7B, no dense-reward configuration in a 43-run sweep exceeds the outcome baseline’s mean gain, SmolLM3-3B and a NuminaMath study show the same pattern, and only single-seed Phi-4 pilots are positive (Section H.6).
Alignment stays high even as training fails. In checkpoints taken every four iterations, the high-weight reward keeps an alignment of 0.969 to 1.000 with the reference on disjoint samples, while greedy accuracy falls from 25% to 12% and response length stays within 1% of the low-weight runs. Entropy instead rises from 0.21 to 0.58 (low weight: 0.18 to 0.20), and the KL from the initial policy is about 30 times that of the low-weight runs. Because alignment does not decay, the loss is not explained by the reward pointing the wrong way; the rising entropy and KL are consistent with over-optimizing it (Section H.5).
Answer. The gradient tests say yes, and the Monte-Carlo reward leads at equal updates, but in the equal-iteration study it does not beat outcome-only GRPO.
Does a Dense Reward Pay for Its Compute?
Per generated token, the Monte-Carlo reward is far behind. At , continuations take 91% of the generated tokens, so by the time the token budget is used up, training with it has made 50 updates and outcome-only GRPO 584. Its accuracy at is 14.3 [12.7, 16.0] points lower (, Holm ), and 14.9 [14.0, 15.8] points lower on twelve fresh seeds (Figure 2b). The deficit recurs on Qwen3 0.6B, 1.7B, and 4B (8.0, 9.1, and 10.0 points; each), on GSM8K (9.8), and with 768-token completions (6.6, ).
Per GPU-hour, it is behind too. Not all tokens cost the same. With the same generated tokens, training takes 0.14 hours on an H200 with the reward and 0.43 with outcome-only GRPO. Outcome-only GRPO spends 61% of its time on updates, whereas the Monte-Carlo reward spends 73% of its time generating continuations, which the inference engine generates in large batches (Section D). Even when it stops at the reward’s training time, outcome-only GRPO leads by 7.8 [6.8, 8.8] points, in all twelve seeds†.
Cheaper rewards lose less. The randomness of continuation sampling contributes only 4% of the second moment of the update, so using fewer continuations should cut the cost while adding little noise. A preregistered study tests , 2, and 4 on twelve fresh seeds. With one continuation per cut, which still spends 70% of the budget on continuations, the reward trails outcome-only GRPO by 6.8 [6.0, 7.6] points and leads the reward by 8.1 [7.1, 9.0] (both Holm , in all twelve seeds); falls in between ( [, ]). Compared at their own training times, the and rewards trail by 5.1 [4.0, 6.2] and 2.2 [1.7, 2.8] points, again in every seed†. Fewer continuations narrow the gap but do not close it.
Reusing samples helps the reward but not the baseline. Another way to lower the cost is to get more out of each sample. A VinePPO-style variant [15] uses value differences as advantages and, as in PPO, takes four clipped optimizer steps on each batch, and continuation reuse, adapted from SPO-tree and TreeRL [11, 14], also trains on the continuations instead of discarding them (Table 16). In three-seed pilots, this reuse raises the reward’s accuracy at matched tokens by 7.7 points (Holm ) but lowers outcome-only GRPO’s by 5.9 (Section E).
| VinePPO | Our VinePPO-style variant | SPO-tree; TreeRL | Our continuation reuse | |
| Credit | for each reasoning step; advantages normalized over the batch | at a segment’s end minus at its start, with the leave-one-out group mean and ; not normalized | SPO-tree: a node’s value minus its siblings’ mean, on tokens with probability below 0.9. TreeRL: , over leaves below | Rollouts: group advantage plus segment credit (the reward). Continuations: their outcome minus the other draw’s at their cut |
| Branch points | Every reasoning step (newline and punctuation boundaries) | 5 fixed offsets (128, 256, …, 640 tokens) | SPO-tree: fixed token intervals, 3 levels. TreeRL: the 2 most uncertain tokens in each of 6 trees | The same 5 offsets; one level |
| Samples per point | 9 rollouts | 4 continuations | SPO-tree: 6 children per node (216 paths). TreeRL: 2 branches per fork (30 responses) | 2 continuations |
| Trains on continuations | No | No | SPO-tree: segments with nonzero credit. TreeRL: every root-to-leaf path | Every continuation, with the prompt and prefix as context |
| Shared prefixes | – | – | SPO-tree: one training segment per node. TreeRL: in every path through them, downweighted | Trained once, in the rollout |
| Update | Clipped (0.2), 2 epochs, minibatch 64, 512 episodes per step; KL penalty | Clipped (0.2), 2 epochs 2 minibatches of 32, 64 completions per iteration; no KL penalty | SPO-tree: clipped, 1 epoch, KL . TreeRL: policy gradient, one step per 16 prompts, KL | One on-policy step over rollouts and continuations, the loss divided by their number; no clipping or KL penalty |
| Models and length | RhoMath 1.1B and DeepSeekMath 7B fine-tuned on the task; 1,024 tokens; temperature 0.6 | Qwen2.5-1.5B-Instruct; 768 tokens; temperature 1 | SPO-tree: RhoMath 1.1B; R1-Distill-Qwen-1.5B at 2K–4K tokens. TreeRL: 7B–14B models; 8,192 tokens | Qwen2.5-1.5B-Instruct; 768 tokens; temperature 1 |
Table 16. What our dense-reward variants keep from the methods they adapt. Published settings are the main ones of each paper [15, 11, 14]. Our configurations share one trainer and recipe (MATH L4–5, 8 prompts 8 completions per iteration, Adam at a constant , the strict reward in S35). The Monte-Carlo rewards () are our construction: segment credit from the same kind of continuations, with a fixed (prompt) = 0.5, added to the group advantage, with one on-policy step. A shared prefix is the part of a completion before a branch point.
Even with as much training time as outcome-only GRPO, no variant shows a significant gain. The comparisons above stop when the faster configuration reaches . A preregistered study with 768-token completions therefore trains four variants beyond , the training time that outcome-only GRPO needs to reach (1.29 A100 GPU-hours; 8 seeds, strict verifier), and compares each with outcome-only GRPO at (Holm over the four; Figure 3a, Section F). Two variants trail in every seed: the Monte-Carlo reward by 3.3 [2.4, 4.3] points (Holm ), and continuation reuse by 5.2 [4.1, 6.3] (), because its updates, which include the continuations, take 78% of its training time (45 seconds per iteration, against 6 for outcome-only GRPO). The VinePPO-style variant trails by 1.1 [0.4, 1.7] (). This deficit lies within points (TOST ), its crossed interval reaches zero, and at the variant’s own, shorter training time it is even with outcome-only GRPO (, secondary). The reward’s deficit, 1.1 [, ] (), is neither significant nor shown to lie within . Given still more time, the reward reaches outcome-only GRPO’s accuracy at after on the mean curves, and the other three do not within their runs, which end at to (Section F).
Answer. No. Per generated token, every Monte-Carlo variant loses 3 to 15 points. Given as much training time as outcome-only GRPO, none outperforms it, and the two closest trail by 1.1 points.
Can a Gradient Test Predict the Result?
Per generated token, the efficiency test anticipates the reversal. The reward is as efficient as outcome-only GRPO per update but only [0.120, 0.190] per generated token. Per token, the test ranks the three Monte-Carlo rewards as (), (), and (), the same order as training (Figure 2b). We had preregistered this order, together with the prediction that every Monte-Carlo reward would trail outcome-only GRPO, and both predictions held. Because the three are about equally efficient per update (1.55 to 1.63), this order follows their token costs, and what the test adds is the size of each deficit.
But it overestimates every reward it rates. Whether a reward is worth training depends on the size of its effect, not only its rank. To turn efficiency into a predicted accuracy, we use a heuristic extrapolation, not a derived prediction: we suppose that a reward with per-token efficiency reaches, after tokens, the accuracy that outcome-only GRPO reaches after tokens. This predicts 35.9%, 31.9%, and 27.5% accuracy at for , 2, and 4; the rewards actually reach 34.3%, 29.9%, and 26.3%†. A learned critic tests the extrapolation where it is most favorable to dense rewards. Instead of sampling continuations, which dominate the cost of the Monte-Carlo reward, it predicts each prefix’s value with a ridge regression on the policy’s hidden states, refit each iteration on earlier rollouts; its forward passes and fits still add training time. Its temporal-difference (TD) form credits each segment with the change in predicted value, and its baseline form subtracts the predicted value from the final reward. Per token, the critic is 1.36 [1.20, 1.62] as efficient as outcome-only GRPO in its TD form and 1.15 [0.97, 1.39] in its baseline form. On 24 fresh Qwen2.5-1.5B seeds, both forms are equivalent to outcome-only GRPO within points (TOST, Holm ). The TD form changes accuracy by [−1.2, +0.3] points, and the baseline form ends 0.7 [0.2, 1.2] points ahead (Holm ), a lead that appears only at the final evaluation. Fitting the critic adds 23 to 24% to training time, so at equal training
time both forms trail outcome-only GRPO, by 1.2 [0.7, 1.7] and 0.7 [0.1, 1.2] points†. Where the extrapolation predicts average leads of 2.0 and 0.9 points, the critics lead by 0.2 and 0.0, shortfalls we had preregistered ( each). Across all five rewards, the extrapolation overestimates accuracy by 0.8 to 1.8 points on average, and every interval excludes zero (exploratory for the Monte-Carlo rewards). The test therefore cannot certify a gain of a point or two. It describes a single SGD step, with its step size tuned to each update, at the initial policy, whereas training takes many Adam steps at a fixed learning rate as the policy changes.
Answer. It predicts the order, not the size. The test ranks the Monte-Carlo rewards as training does but overestimates all five rewards by 0.8 to 1.8 points, as large as the gains we measured.
Case Study: Prompt Selection
Dense rewards show what the protocol rejects; prompt selection shows what it confirms. Rather than spending generated tokens to improve the update on prompts already sampled, prompt selection spends one forward pass to decide which prompts to sample § 2.2. The protocol confirms its gain, shows where the gain comes from § 4.1, predicts where it appears Section 4.2, and measures what it is worth § 4.3.
Does Prompt Selection Help?
It gains on two of six models with the level 3–5 pool, and both gains replicate. On Qwen3-0.6B, the critic filter raises accuracy by 1.7 [1.2, 2.1] points over 54 seeds from four studies (and is higher in 42 of them), including a gain of 1.2 [0.5, 1.9] points in a preregistered replication on 24 fresh seeds (Holm ). On Qwen2.5-1.5B, the gain is smaller but holds in three preregistered studies (12, 24, and 24 seeds; Holm , .006, and .034); over all 60 seeds, it is 0.8 [0.4, 1.1] points. The other four models show no significant gain on this pool (Figure 4a), and Section 4.2 explains where the filter helps.
Figure 4. Where prompt selection helps is predicted by how hard its candidates are, not by how many informative groups it adds. At matched generated tokens: (a) Accuracy at the budget minus outcome-only GRPO on the same seeds, for the critic filter and for random choice of 8 of its 24 candidates (dots: seeds; bold: the filter’s mean). The pool is MATH levels 3–5 unless labeled; Llama models train on the strict verifier. (b) Change in the share of mixed-outcome groups trained on before against the change in accuracy, both relative to outcome-only GRPO; the diamond is the filter aimed at predicted success 0.85. (c) The critic’s mean predicted success of the candidates against the change in accuracy; the dashed line marks 1/2. Hollow points are the first studies, on a level 4–5 pool, and rings mark the four settings with a preregistered prediction. Bars are 95% CIs over seeds; the gray band marks ±0.02.
It carries over to problems that no study had evaluated on. Every study evaluated on the same 200 test problems, and on them the crossed bootstrap interval for the Qwen2.5-1.5B gain reached zero each time, so we tested transfer directly. The last study on each model also scored, at , the 1,543 numeric-answer MATH test problems of levels 3–5 that lie outside all evaluation sets, and the 1,319 GSM8K test problems. On these MATH problems, the filter gains 0.8 [0.5, 1.1] points on Qwen2.5-1.5B and 1.1 [0.6, 1.5] on Qwen3-0.6B (Holm each), with crossed intervals excluding zero. On GSM8K, which neither configuration trains on, it gains 1.1 and 1.3 points (secondary).
It comes from which prompts are chosen. Because it draws 24 candidates per iteration and trains on 8, the filter also moves through the prompt stream three times as fast as outcome-only GRPO, which could help on its own. Random choice among the same candidates isolates this side effect. In preregistered primary contrasts, random choice trails the filter on both models where the filter helps: by 1.7 [0.5, 2.9] points on Qwen3-0.6B (, twelve fresh seeds), and on Qwen2.5-1.5B by 1.0 [0.4, 1.6] (Holm ) and again by 0.9 [0.3, 1.5] on 24 fresh seeds (Holm ). Random choice itself is equivalent to outcome-only GRPO within points on all three models tested.
Most of it survives when training time is counted. Scoring the candidates costs a forward pass, so the filter’s runs take 9 to 14% longer to reach . At equal training time, the filter still leads, by 1.0 [0.6, 1.5] points on Qwen3-0.6B and 0.5 [0.2, 0.8] on Qwen2.5-1.5B‡ (Figure 3b); on the seeds of the Qwen3-0.6B replication alone, this lead is 0.4 [−0.1, +1.0].
A per-prompt success estimate carries the gain, and cheaper signals do not. Simpler rules test which part of the filter carries the gain. A history filter, close to GRESO (Zheng et al., [39]), uses the reward of each prompt’s most recent group and needs no forward pass. On Qwen2.5-1.5B, it gains 0.6 [0.1, 1.1] points (Holm , 36 seeds), also at equal training time, while the learned critic adds an early lead of 1.9 [1.6, 2.1] points over it during the first half of the budget (secondary). On Qwen3-0.6B and 1.7B, filtering by MATH difficulty level gains nothing (+0.2, TOST ), and DAPO’s dynamic sampling (Yu et al., 2025) loses 1.8 [0.8, 2.7] points, because the groups it discards after generating them count toward the budget.
Answer. Yes, on two of six models: by 1.7 and 0.8 points at equal tokens and by 1.0 and 0.5 at equal training time (exploratory). The gains replicate on fresh seeds and unseen problems and come from the choice of prompts.
When Does Prompt Selection Help?
Not where it adds the most informative groups. The rationale for training on prompts near 1/2, and for dynamic sampling (Gao et al., [9]; Yu et al., 2025), is that only mixed-outcome groups have nonzero advantages. The filter raises the share of these groups by 8 to 19 points in every setting at learning rate . Accuracy, however, rises in only some settings, and not in those where the share rises most (Figure 4b), so this rationale does not predict it.
It helps where the candidates are predicted to fail more often than not. We formed this account post hoc from five settings. In them, the critic’s mean predicted success over the 24 candidates separates the two settings where the filter helped (0.42 and 0.40) from the rest (0.54 to 0.70), with no overlap between seeds, and the predictions are calibrated (Table 9). Random choice, which trains on a uniform draw of the candidates, realizes the predicted success to within 0.2 points on all three models. The rule follows from how the filter chooses prompts. When the mean prediction is below 1/2, keeping the candidates nearest 1/2 shifts training toward easier prompts and away from groups that are all wrong; when it is above 1/2, the filter shifts training toward harder prompts. Outcome-only GRPO’s own training reward separates the first eleven settings in the same way (Table 8), so a short pilot run is enough to apply the rule.
| Setting | candidates | filter | outcome-only | iteration 0 | start | filter − outcome | |
| Qwen3-0.6B, L4–5 | 6 | 0.34 (0.32–0.35) | 0.43 | 0.30 (0.30–0.31) | 0.18 | 0.36 | +0.028 [+0.007, +0.050] |
| Qwen3-1.7B, level 5 | 24 | 0.39 (0.38–0.41) | 0.45 | 0.35 (0.34–0.36) | 0.28 | 0.29 | +0.011 [+0.002, +0.020] |
| Qwen2.5-1.5B, L3–5 | 60 | 0.40 (0.38–0.41) | 0.46 | 0.37 (0.35–0.38) | 0.21 | 0.22 | +0.008 [+0.004, +0.011] |
| Qwen3-0.6B, L3–5 | 54 | 0.42 (0.40–0.43) | 0.47 | 0.39 (0.37–0.40) | 0.35 | 0.35 | +0.017 [+0.012, +0.021] |
| Llama-3.2-1B, L3–5 | 24 | 0.42 (0.39–0.44) | 0.46 | 0.39 (0.36–0.42) | 0.13 | 0.15 | +0.004 [−0.001, +0.010] |
| Qwen3-1.7B, L4–5 | 6 | 0.52 (0.51–0.52) | 0.51 | 0.50 (0.49–0.51) | 0.32 | 0.56 | +0.017 [−0.007, +0.040] |
| Qwen3-4B, level 5 | 12 | 0.54 (0.53–0.56) | 0.51 | 0.52 (0.49–0.54) | 0.30 | 0.35 | +0.002 [−0.028, +0.032] |
| Qwen3-1.7B, L3–5 | 24 | 0.59 (0.58–0.60) | 0.53 | 0.58 (0.56–0.59) | 0.51 | 0.55 | +0.002 [−0.006, +0.011] |
| Llama-3.2-3B, L3–5 | 24 | 0.68 (0.64–0.72) | 0.59 | 0.70 (0.66–0.72) | 0.39 | 0.29 | −0.004 [−0.010, +0.002] |
| Qwen3-4B, L4–5 | 8 | 0.68 (0.67–0.70) | 0.58 | 0.66 (0.65–0.67) | 0.40 | 0.62 | +0.002 [−0.013, +0.017] |
| Qwen3-4B, L3–5 | 12 | 0.70 (0.69–0.72) | 0.58 | 0.71 (0.69–0.72) | 0.49 | 0.62 | −0.002 [−0.015, +0.010] |
| Qwen3-0.6B, L1–2 | 24 | 0.79 (0.77–0.80) | 0.67 | 0.81 (0.80–0.83) | 0.74 | 0.71 | −0.005 [−0.011, +0.000] |
Table 8.
| Model | configuration | pred. | real. | real. − pred. | ||
| Qwen3-0.6B | random | 18 | 0.394 | 0.393 | −0.14 [−0.39, +0.10] | 0.67 |
| filter | 18 | 0.469 | 0.468 | −0.05 [−0.25, +0.14] | 0.47 | |
| Qwen3-1.7B | random | 24 | 0.579 | 0.578 | −0.13 [−0.26, 0.00] | 0.72 |
| filter | 24 | 0.533 | 0.530 | −0.24 [−0.39, −0.09] | 0.50 | |
| Qwen2.5-1.5B | random | 36 | 0.366 | 0.368 | +0.17 [+0.03, +0.31] | 0.68 |
| filter | 36 | 0.458 | 0.459 | +0.08 [−0.05, +0.22] | 0.48 | |
| Qwen3-0.6B, L1–2 | filter | 24 | 0.656 | 0.668 | +1.21 [+1.11, +1.32] | 0.56 |
| Llama-3.2-3B | filter | 24 | 0.585 | 0.594 | +0.92 [+0.79, +1.06] | 0.46 |
| Llama-3.2-1B | filter | 24 | 0.467 | 0.468 | +0.08 [−0.05, +0.21] | 0.44 |
Table 9.
**Preregistered tests confirm the account, including a direct
manipulation. Five of six preregistered tests agree with it (Table 1). The two most direct tests hold the model and pool fixed and change only the filter’s target, to predicted success 0.25, so that the filter moves training toward prompts harder than its candidates: on both models where the gain had replicated, it disappears. Two tests change the training pool: the filter helps on Qwen3-1.7B with level 5 (mean prediction 0.39), although its crossed interval reaches zero, and not on Qwen3-0.6B with levels 1–2 (0.79). Two change the model family: it does not help on Llama-3.2-3B (0.68), as predicted, nor on Llama-3.2-1B (0.42), the one failed prediction. Because four of the five confirmed predictions are of no gain, the more common outcome, the two direct manipulations weigh most (Section B).
| Setting | Cand. | Predicted | Result |
| Qwen3-1.7B, L5 | 0.39 | gain ✓ | +1.1 [+0.2, +2.0] |
| Qwen3-0.6B, L1–2 | 0.79 | none ✓ | −3.0 [−3.9, −2.0] |
| Llama-3.2-3B | 0.68 | none ✓ | −0.4 [−1.0, +0.2] |
| Llama-3.2-1B | 0.42 | gain ✗ | +0.4 [−0.1, +1.0] |
| Qwen3-0.6B, =0.25 | 0.42 | loss ✓ | −2.8 [−3.6, −2.0] |
| Qwen2.5-1.5B, =0.25 | 0.40 | loss ✓ | −1.0 [−1.6, −0.4] |
Table 1. Preregistered tests of the account that prompt selection helps when the candidates’ mean predicted success (Cand.) is below 1/2. Result: accuracy at minus outcome-only GRPO, in points (24 seeds, 95% CIs), except the gain on levels 1–2 minus the gain on levels 3–5 (24 seeds) and the filter aimed at minus the filter aimed at 1/2. L5, L1–2: level-5 and level 1–2 pools. A check mark is a prediction that held.
Answer. Where candidates are predicted to fail more often than not. The filter helps significantly in four of the five such settings and in none of the seven others (Figure 4c); on a fixed model and pool, aiming it at harder prompts removes the gain.
What Is the Gain Worth?
It is a head start that saves about a fifth of the token budget. The lead forms early: on Qwen2.5-1.5B, it peaks at 2.6 [2.2, 3.0] points at three-sixteenths of the budget () and falls to 0.8 at (Figure 5). On both models, the filter reaches outcome-only GRPO’s final accuracy with 0.78 of the token budget, and with 0.85 and 0.87 of its training time, including the time spent scoring candidates†. On Qwen3-0.6B, the gain also holds at pass@8, by 0.9 [0.2, 1.7] points, and at every test level, whereas on Qwen2.5-1.5B it shrinks as grows† (Table 10). With longer training, outcome-only GRPO catches up. A preregistered study trained both Qwen3-0.6B configurations to twice the budget (strict verifier, 24 seeds): the filter leads by 1.9 [1.3, 2.6] points at but by only 0.5 [−0.5, +1.6] at , which is equivalent to zero within ±2 points (TOST = .004).
Figure 5. The prompt filter leads early, then holds or shrinks. (d) The differences of Figure 4a at every evaluation: the critic filter (blue) and random choice (gray) minus outcome-only GRPO on the same seeds. In the last panel, Qwen3-0.6B runs trained for twice the budget with the strict verifier as the reward of both configurations (24 seeds). Bands are 95% CIs over seeds; the gray band marks ±0.02.
| Model | Configuration () | pass@ | pass@ | pass@ | pass@ | base |
| Qwen3-0.6B | filter () | [, ] | [, ] | |||
| random () | [, ] | [, ] | ||||
| filter, default reward, () | [, ] | [, ] | ||||
| filter, strict reward () | [, ] | [, ] | ||||
| same, () | [, ] | [, ] | ||||
| Qwen2.5-1.5B | filter () | [, ] | [, ] | |||
| random () | [, ] | [, ] | ||||
| Qwen3-1.7B | filter () | [, ] | [, ] | |||
| Qwen3-4B | filter () | [, ] | [, ] | |||
| Qwen3-4B, L5 | filter () | [, ] | [, ] | |||
| Qwen3-1.7B, L5 | filter () | [, ] | [, ] | |||
| Qwen3-0.6B, L1–2 | filter () | [, ] | [, ] | |||
| Llama-3.2-3B | filter () | [, ] | [, ] | |||
| Llama-3.2-1B | filter () | [, ] | [, ] |
Table 10. Pass@ at the token budget (exploratory): differences against outcome-only GRPO on the same seeds, with 95% paired- CIs for pass@ and pass@; the last column is outcome-only GRPO’s pass@.
A retuned baseline gains about as much, so the baseline sets the benchmark. A head start could also reflect a baseline that learns too slowly, so we retuned its learning rate. On Qwen3-0.6B, a preregistered study (strict verifier, 24 seeds) doubled the constant learning rate of , which raised outcome-only GRPO’s accuracy by 1.9 [0.5, 3.3] points (Holm = .018); the faster baseline minus the filter is −0.03 [−1.3, +1.2] points, equivalent within ±2 (TOST = .002; Figure 6a). On Qwen2.5-1.5B, a doubled rate lowers both by about three points, and the filter’s gain holds (1.1, Holm = .018). A rate that decays linearly from to 0 at keeps the faster early rise without that loss (Figure 6b). With it, outcome-only GRPO ends 0.5 [−0.3, +1.2] points below the filter at the constant rate (Holm = .41; equivalent within ±2 but not shown within ±1), and the filter’s own gain under the schedule, 0.2 [−0.5, +0.9], is equivalent to zero within ±1 point (TOST = .015; Figure 6c).
Figure 6. A retuned learning rate brings outcome-only GRPO within half a point of the critic filter in mean accuracy. Accuracy at every evaluation, paired by seed (preregistered studies on the level 3–5 pool, 24 seeds each). (a, b) Outcome-only GRPO at each learning rate minus the filter at the default constant rate ; on Qwen3-0.6B every configuration trains on the strict verifier. (c) On Qwen2.5-1.5B, the filter minus outcome-only GRPO at the same rate: under the decaying rate its gain fades by the budget. Bands are 95% CIs; the gray band marks ±0.02.
Answer. A head start worth about a fifth of the budget. It fades by , and a retuned baseline comes within half a point of it.
Related Work
The structure of policy-gradient updates. Writing an outcome update as a weighted sum of the completions’ summed scores follows from the score-function estimator [32, 29], and it applies to the completion-level advantages of GRPO and RLOO [26, 2]; Mishra et al. [17] study this factorization for GRPO. Analyses of GRPO’s dynamics in expectation [18] and of which credit schemes give proportional expected gradients (Fu et al., 2026) leave open whether the updates computed on a finite batch are equal, which Section I answers exactly. The low-dimensional gradient subspace measured by Schneider et al. [23] also differ from the span of one batch’s completion scores. HAPO and EAPO redistribute advantages within completions while preserving their signs (He et al., 2026; Yeo et al., 2026), so their updates, too, can leave . The sampled gradients of potential-based shaping (Ng et al., 1999) need not lie in either, and work on reward hacking [27, 8] shows that optimizing a proxy reward too hard can lower the true objective.
Process rewards and how they are combined with outcomes. Step-level supervision [16, 31], progress-based advantages [24], Monte-Carlo value estimates [15], and implicit reward models trained online from outcome labels [37, 4] report gains from feedback on intermediate steps. VinePPO reaches PPO’s peak accuracy in up to 3× less wall-clock time and outperforms GRPO and RLOO, with 9 rollouts at every reasoning step, models fine-tuned on the task, and KL-regularized PPO steps; our adaptation keeps its value-difference advantage and clipped reuse but not its recipe (Table 16). SPO-tree (Guo et al., 2026) and TreeRL [14] also train on the sampled continuations, so that their tokens are not spent on value estimates alone. How advantages are normalized, the level at which credit is assigned, and how uniform groups are handled affect how process and outcome rewards combine (Wang et al., 2026; Agrawal et al., 2026).
Choosing training prompts. Because a uniform group gives GRPO no advantage, several methods aim training at prompts with success near 1/2. Some measure success with rollouts first: DAPO discards uniform groups after generating them (Yu et al., 2025), online difficulty filtering keeps prompts whose pass rates fall in a band around 1/2 [3], and LILO ranks prompts by learnability [6]. Others decide before generating: GRESO skips prompts whose recent groups were uniform [39], MoPPS keeps a Bayesian estimate of each prompt’s success (Qu et al., 2026), DOTS predicts difficulty from similar questions [28], and PCL trains on the prompts whose value-model prediction is nearest 1/2 [9]. Our critic filter uses PCL’s rule and our history filter resembles GRESO and MoPPS; rather than propose a new rule, we test such rules at matched compute.
Lessons and Conclusion
Each lesson comes from a result that a cheaper protocol would have missed. (1) Count every generated token and every GPU-hour, and report updates alongside them. The Monte-Carlo reward ranked first at equal updates and last at equal tokens, and a critic’s small gain at equal tokens disappeared at equal training time. (2) Treat gradient tests as heuristics, not certificates. The efficiency test overestimated every reward it rated by one to two points, as large as the gains we measured. (3) Control for side effects. Random choice among the same candidates separated which prompts were chosen from how fast training moved through the prompt stream. (4) Test the mechanism, not only the method. The share of informative groups did not predict where prompt selection helped; the candidates’ predicted success did, in five of six preregistered tests. (5) Retune the baseline. Doubling the initial learning rate brought outcome-only GRPO within half a point of the critic filter on both models where its gain replicated. (6) Fix the verifier first. A lenient verifier let runs in either configuration drift away from boxing their answers. (7) Replicate on fresh seeds and on more than one model, and preregister. A six-seed gain on Qwen3-1.7B did not replicate on eighteen fresh seeds, and the one gain we predicted outside the Qwen family did not appear. Section B discusses the scope of these lessons and the limitations of our studies.
Dense rewards can produce update directions that no reweighting of whole completions can, but only training at matched compute can show whether they help. None of our dense-reward variants beat outcome-only GRPO by more than a point, and the Monte-Carlo reward fell 14.3 points behind at equal tokens, while choosing which prompts to train on beat it by up to 1.7 points. Because that gain comes early, a faster-learning baseline nearly matches it: outcome-only GRPO with twice the initial learning rate comes within half a point. A claim that a training intervention helps should state its token and GPU-hour budgets, its verifier, the models on which it replicated, and how its baseline was tuned.
AI Usage Disclosure
We used generative AI tools for grammar check, spelling mistakes and help us to visualize our results efficiently as well as debugging help during codebase development.
Reproducibility and Responsible Reporting
The supplementary material gives the proofs, training and evaluation settings, seed counts, and statistical methods. The artifact at https://github.com/vicky157/grpo-validation contains the trainer and analysis code, every configuration and preregistration file with its dated amendments, each study’s report and submission record, and every run’s configuration, evaluation log (with training time), seconds per training stage, and compute record, from which every contrast at matched tokens or matched training time can be recomputed; one script rebuilds the training-time tables and figures. The per-iteration logs are omitted for size. We use public mathematical-reasoning benchmarks and released language-model checkpoints. The study involves no human participants or new personal-data collection. We report accuracy losses and inconclusive comparisons alongside positive results.
Conventions. The appendix uses the terms defined in § 2, and reports accuracy and differences in accuracy as fractions, as the figures do (0.01 is one percentage point of the main text).
Additional Background
This section gives the details of the protocol and of the gradient tests that § 2 summarizes.
Recipe. A preregistered stability rule set the learning rate to , after degraded every configuration over the matched horizon. The completion limit matters: at 380 tokens, 73% of the initial samples are cut off, so part of what every run learns is to finish within the limit; at 768 tokens, 20% are. Accuracy is averaged over held-out test problems of levels 3–5, or of the training pool’s own levels for the level-5 and level 1–2 pools. Both reuse variants keep the published form of credit but not the published step boundaries, sample counts, or recipes (Table 16).
Statistics. A study’s Holm families are the groups of tests corrected together. The plan files are internal rather than in a public registry; Section C audits their timing. Because every study evaluates on the same 200 test problems, a crossed bootstrap, which resamples seeds and test problems together, checks every primary contrast.
Verifier drift. Some runs trained with the default verifier gradually stop boxing their answers, which shows up as accuracy under the default verifier exceeding that under the strict one by more than 10 points. This drift occurred in up to 49 of a study’s 408 evaluations and in either configuration. Preregistered studies that retrained the drifting configurations with the strict verifier found no drift (0 of 2,196 evaluations) and no loss in accuracy (Section C.7).
Alignment. Let be the expected accuracy of , its gradient, and an estimate of from separately sampled continuations. Every update that weights whole completions lies in . A dense update can therefore improve on its projection onto , to first order, only through its component outside :
(Proposition 1; § 22 characterizes exactly which dense rewards produce updates outside .) The alignment test reports the cosine between and , the components of the dense update and of the estimated gradient that lie outside .
Efficiency. Alignment ignores how noisy an update is and what it costs; efficiency accounts for both. For an objective with smoothness constant , the gradient efficiency of an update direction equals times the best expected one-step gain of an SGD step along , ; the expected inner product was positive for every method we measured (Section C).
Scope and Open Validation Questions
This section expands on the limitations in § 6. It first covers the matched-compute studies of § 3 and § 4, and then the equal-iteration study of Section 3.1 and the fixed-batch theory of § 22.
Matched-compute studies
Scope. These studies use an independent reimplementation of the trainer and a learning rate of , rather than the equal-iteration study’s , because the faster rate degraded every configuration over the matched horizon. They cover numeric-answer MATH problems, evaluation sets of 200 problems (on Qwen2.5-1.5B and Qwen3-0.6B, also 1,543 further MATH test problems and the GSM8K test set), models of up to 4B parameters, and budgets of 8–35M generated tokens. Most contrasts pool 6–60 seed pairs, but some, including primary contrasts of the first studies, have only –5. Four of the six models are from the Qwen family, in which RLVR can behave differently from other families [25]. We tested the other two, Llama-3.2-1B and 3B, only with prompt selection and on one pool. On 3B, the candidates’ mean predicted success was above . On 1B it was below , but the predicted gain did not appear, so we have shown no gain outside the Qwen family.
Learning rate. Most runs use a constant learning rate of . Three studies double it, and one decays it linearly from to zero over the budget; we did not tune it for each model or configuration. On Qwen3-0.6B, doubling the rate gave outcome-only GRPO as large a gain as the filter, also without drift, so a baseline tuned for speed may leave less for prompt selection to add. On Qwen2.5-1.5B, doubling the rate lowered both configurations at the budget, and the filter’s gain held. With the decaying rate, however, outcome-only GRPO ended 0.005 below the filter at , a difference that is neither significant nor shown to lie within , and the filter’s gain under that schedule was equivalent to zero within , a margin wider than the gain at the constant rate. We tried no warmup, no other peak rates, and no other schedules, and none of the schedules we tried was tuned for the filter. A schedule tuned for the filter might restore its gain, and one tuned for the baseline might close the remaining half point.
Where prompt selection helps. Prompt selection helped Qwen3-0.6B (on levels 3–5 and 4–5), Qwen2.5-1.5B on levels 3–5, and Qwen3-1.7B on level 5. It did not help Qwen3-1.7B on levels 3–5, Qwen3-4B, Qwen3-0.6B on levels 1–2, or Llama-3.2-3B, and it did not help Llama-3.2-1B significantly. We formed the account that it helps when the candidates’ mean predicted success is below 1/2 post hoc, from five settings.
Tests of the account. Two preregistered studies agreed with the account, one in each direction: they found a small gain (+0.011) whose crossed interval reaches zero, and no gain where the candidates averaged 0.79. Both moved the candidates across 1/2 by changing the training pool, which also changes the test set, so they do not separate the candidates’ difficulty from other properties of a pool. A third study kept the pool and changed the model. It agreed with the prediction of no gain, but the candidates fell on the other side of 1/2 from the one its design anticipated. Two of these three predictions were of no gain, which is the more common outcome here (seven of twelve settings), so they put the account at less risk than a predicted gain does. A fourth study tested a predicted gain on the smaller Llama-3.2-1B with the level 3–5 pool, chosen by a pilot because outcome-only GRPO’s training reward on it was below 0.45. That prediction failed (+0.004 [−0.001, +0.010], , equivalent to zero within ), although the filter’s mean lead was positive at every evaluation after the start. Of the account’s two predicted gains, therefore, one was found. A fifth study kept the model and pool fixed (Qwen3-0.6B on levels 3–5) and changed only the filter’s target, to 0.25. As predicted, the filter then trained on prompts harder than its candidates and lost its gain. A sixth study repeated this on Qwen2.5-1.5B with the same pool and found the same result, although the deficit there (−0.010, as large as the gain with target 1/2) has a crossed interval that just reaches zero, and the filter kept a small head start that its preregistration did not predict.
not predict. These two studies separate the direction in which the filter moves training from the pool, but only on one pool and on the two models where the filter gained.
Applying the account. Both of the account’s predictors come from training runs. The base model’s accuracy on its first training prompts, which is known before training, does not separate the settings (Table 8), so applying the account requires at least a short pilot run of outcome-only GRPO. On Llama-3.2-3B, a training reward far above held-out accuracy may reflect exposure to MATH training problems during pretraining; we did not test for contamination. The first studies, on a level 4–5 pool, were not used to form the account. They agree with it on Qwen3-0.6B and 4B (6 and 8 seeds) and are inconclusive on Qwen3-1.7B, whose candidates averaged 0.52. Starting accuracy alone does not predict a gain: training on level 5 gives Qwen3-4B the starting accuracy of 0.6B, but no gain. The composition of the trained groups does not predict a gain either.
Compared methods. Prior work also uses the selection rule itself. We compared it at matched tokens with dynamic sampling, the history filter, the level filter, and random choice, but not with methods that rank prompts by first sampling rollouts for them, such as LILO or online difficulty filtering [6, 3]. For dense rewards, we tested at matched tokens Monte-Carlo rewards with one, two, and four continuations at each of five cuts, whose continuations are scored and discarded; a ridge critic; and, with 768-token completions, a variant that also trains on its continuations and a VinePPO-style advantage with rollout reuse. These variants adapt published methods to our trainer and recipe rather than reproduce them (Table 16). VinePPO estimates values at every reasoning step with nine rollouts under a KL-regularized PPO recipe; SPoT-tree trains on a three-level tree with a probability mask; and TreeRL branches at uncertain tokens and trains on every path from the root to a leaf. Faithful reproductions, fewer cuts, other tree-structured estimators, or a separately trained implicit reward model may trade off cost differently.
Gradient test. The per-token gradient test ranked the three Monte-Carlo rewards in the same order as training, but only on one model. When we extrapolated it to training as a heuristic, it overestimated all five rewards we checked by about 0.01 to 0.02 (exploratory for the Monte-Carlo rewards and preregistered for the two critics; every interval excludes zero).
Duration and transfer of the gain. We trained beyond the budget only on Qwen3-0.6B, where most of the gain was gone by (+0.005 [−0.005, +0.016], compared with +0.019 at ). On Qwen2.5-1.5B, the gain replicated twice on fresh seeds, but in all three studies the crossed seed-by-problem bootstrap intervals on the 200 problems include zero. The third study therefore also scored the 1,543 MATH test problems of levels 3–5 that lie outside every other evaluation set. There, the crossed interval excludes zero (+0.008 [0.004, 0.012]), so the gain carries over to other problems from the same distribution. On GSM8K, an easier task that neither configuration trains on, the gain was +0.011 [0.006, 0.016] (secondary); we tested no other task. Measured with the strict verifier, both contrasts are unresolved, because one filter run (two on GSM8K) had drifted at ; without the drifted runs, both are positive (exploratory). On Qwen3-0.6B, a fourth study on fresh seeds replicated the gain on the 200 problems, at +0.012 compared with the first studies’ +0.021, and found it on the further MATH problems (crossed interval [0.006, 0.016]) and on GSM8K (+0.013). At equal training time, however, its lead on those seeds alone was not significant (+0.004 [−0.001, +0.010]).
Multiplicity. Holm’s correction applies within each study, not across studies. Under a correction across studies, the replicated gains (Qwen3-0.6B and Qwen2.5-1.5B on levels 3–5) remain stronger evidence than single-study results with between .01 and .05, such as the level-5 gain on Qwen3-1.7B, the history filter’s gain, and the baseline-form critic’s gain at the budget.
Truncation. On Qwen2.5-1.5B, with a 380-token limit, 73% of the initial samples are cut off, so part of what every run learns there is to finish within the limit. The Monte-Carlo reward’s deficit is smaller with 768-token completions and a budget calibrated for them (−0.061 on three seeds and −0.066 on eight, compared with −0.143). With 768-token completions and as much training time as outcome-only GRPO, the reward and continuation reuse still trailed (−0.033 and −0.052), and the VinePPO-style variant trailed within ±0.02 (−0.011). The reward’s deficit (−0.011 [−0.028, +0.005]), however, was neither significant nor shown to lie within ±0.02. We therefore claim neither that a cheap Monte-Carlo reward loses per GPU-hour in that setting nor that it matches outcome-only GRPO.
Hardware and accounting. Every equal-time contrast compares runs on one GPU type. On other hardware, the ratio of generation time to update time may differ, and the equal-time results with it. Behavior at larger scales and on other tasks remains open. The token budget counts generated tokens. It does not count the prefill pass of the prompt critic, which we report as training time instead. Measured in training time, the head start in the three settings with a gain shrinks from 22–23% to 13–17% (Table 11).
| Setting () | target | tokens | training time |
| Qwen3-0.6B, L3–5 () | [, ] | [, ] | |
| strict reward () | [, ] | – | |
| strict reward, target at () | [, n.r.] | – | |
| Qwen2.5-1.5B, L3–5 () | [, ] | [, ] | |
| history filter () | [, ] | [, ] | |
| Qwen3-1.7B, level 5 () | [, ] | [, ] | |
| Qwen3-1.7B, L3–5 () | [, n.r.] | [, n.r.] | |
| Qwen3-4B, L3–5 () | n.r. [, n.r.] | n.r. [, n.r.] | |
| Qwen3-4B, level 5 () | [, n.r.] | [, n.r.] | |
| Qwen3-0.6B, L1–2 () | n.r. [n.r., n.r.] | n.r. [n.r., n.r.] | |
| Llama-3.2-3B, L3–5 () | n.r. [, n.r.] | n.r. [, n.r.] | |
| Llama-3.2-1B, L3–5 () | [, ] | [, ] |
Table 11. Share of the budget, and of outcome-only GRPO’s training time, that the critic filter needs to reach outcome-only GRPO’s accuracy at (exploratory; 1 means no saving). The history-filter row uses the critic-free filter of S22 on the same seeds. Training time includes the critic’s scoring and excludes evaluation.
Equal-iteration study and theory
Sampled geometry. The score map describes one policy and one sampled batch, and the scores and the span of whole-completion updates change during training. Also, contains every reweighting of whole completions, a larger set than the one coefficient vector that a single outcome update uses. The geometry describes dense rewards used as training coefficients. Using process rewards to rank or search among solutions at inference time is a separate use, which it does not cover.
Optimizer and controls. The neural runs use Adam with gradient-norm clipping at 1.0. Per-coordinate scaling and optimizer state can change the direction of a raw gradient, so the geometric decomposition describes the gradient before these transformations. Permuting token weights preserves their magnitudes, but the norm of the projected gradient, the norm of the total update, and the policy KL can still change. To separate the effect of coherent credit from the effect of update size, controls must therefore be matched at the stage of the update being studied.
Reference and evaluation target. The progress reference relies on a finite number of Monte-Carlo continuations, a segmentation of each prefix, a finite continuation horizon, and a verifier. Splitting samples avoids reusing them directly, but the reference keeps these estimator choices. The strongest check, with disjoint samples and full-parameter gradients, uses four Qwen2.5-1.5B probes with mean alignment 0.84; we report references that partially overlap with the training samples separately. Training samples are at temperature 1.0 with top- 0.95, whereas validation uses greedy decoding. Evaluating accuracy on the same fixed prompts ties the local objective more directly to measured performance, which is what the matched-compute studies do (§ 2.3).
Statistical resolution. The primary comparison, with , establishes equivalence within ±0.03 greedy accuracy, so smaller benefits and harms remain compatible with the data. These seed intervals are conditional on the chosen validation set. A larger fixed test set, separate sampled and greedy evaluations, and replication with independent seeds would each address a different source of uncertainty.
Breadth and horizon. The amount of replication varies across the five-family probe panel, and most secondary training comparisons use . The SmolLM3 endpoints and the positive single-seed Phi-4 pilots are descriptive evidence only, and the expansion of the gate study to includes eight exploratory seeds. The experiments use six segments, a constant dense weight, and short training horizons. Other segmentations, schedules, datasets, initial policies, and longer horizons remain open.
Reward integration. Our implicit-reward control uses the log ratio between the actor and the reference policy, whereas PRIME uses a separately trained reward model in the numerator. The learned-PRM study uses one particular segmentation and a reconstructed scoring head. Finally, masking tokens and then centering credit within each completion redistributes credit to the tokens that were not masked. Gates should therefore be evaluated after every transformation the trainer applies, and a controlled allocation experiment must also match the chosen credit or update budget.
Matched-Compute Studies
This appendix documents the matched-compute studies of Sections 3 and 4. Section C.1 describes the setup that all studies share, and Section C.2 lists every contrast and every study. Sections C.3 to C.7 then follow the questions of the main text. Each opens with what the studies show together and continues with one record per study: its design, its preregistered contrasts, its checks, and the secondary results that the tables do not show.
Setup
Why a separate implementation. The equal-iteration comparisons of Section 3.1 match seeds and training iterations, so the Monte-Carlo reward uses more compute there than outcome-only GRPO: every scored prefix needs continuation rollouts. To compare the methods at the compute they consume, we reimplemented the training pipeline independently of the equal-iteration code, counted every generated token toward a matched budget (in this appendix, denotes this budget, not the matrix of Section I), and extended the comparison to a learned critic and to prompt selection on six models. Together the studies comprise 1,668 runs and 1,468 GPU-hours, with one GPU per run. All ran on H200 GPUs except S35 (105 GPU-hours) and 22 GPU-hours of task studies, which ran on A100 GPUs.
Implementation. Rollouts are sampled with vLLM in bf16; the trainer keeps float32 master weights and Adam states with bf16 autocast compute. With pure bf16 weights, Adam’s per-element step at these learning rates is smaller than half a bf16 unit in the last place for typical weights, so most weights would not change: in a check on Qwen2.5-0.5B, 87% of parameters were unchanged after one step. The equal-iteration study’s recipe at learning rate degraded or collapsed every configuration, outcome-only included, over the matched horizon. A preregistered stability probe, on a development set disjoint from the training pools and test sets, selected ; a rate of failed the probe. For Qwen3, a preregistered truncation rule raised the generation limit from 380 to 768 tokens. Qwen3-4B and Llama-3.2 runs use a microbatch size of 1 to fit in memory, which changes neither the update nor the token count. Llama-3.2 runs also fix the date that the chat template inserts. Each iteration trains on 8 prompts with 8 completions each, sampled at temperature 1 and top- 1, and takes one optimizer step on them, with no ratio clipping or reuse of rollouts. The group advantage is the reward minus the group mean, divided by the group’s population standard deviation; the loss sums the token terms and divides by the number of completions in the iteration. The optimizer is Adam (, ) without weight decay, with gradient clipping at norm 1, no KL penalty, and a constant learning rate without warmup, so a run’s result at does not depend on whether it continues past (only S32 decays the rate, linearly to zero at ). The token budget counts all tokens generated during training: rollouts, continuations, and the discarded groups of dynamic sampling. A preregistered calibration rule set it for each model and task before the studies: 48 times the mean number of tokens per iteration at the Monte-Carlo reward generates over iterations 0–2 of one calibration run, rounded up to a multiple of 250,000. The densest reward thus gets 50 to 57 updates, and outcome-only GRPO gets 584 on Qwen2.5-1.5B and 755 on Qwen3-4B.
Models and budgets. The models are Qwen2.5-1.5B-Instruct (380-token completions, M); Qwen3 0.6B, 1.7B, and 4B in non-thinking mode (768 tokens, M, 21.0M, and 22.25M); and, for prompt selection only, Llama-3.2-1B- and 3B-Instruct (768 tokens, M and 8.25M). Each iteration trains on 8 prompts with 8 completions each.
Pools and evaluation. All problems have plain numeric answers, so the last-number rule is well defined. Training uses MATH train problems of levels 4–5 (562 prompts after removing overlaps with the evaluation sets) or levels 3–5 (3,486 prompts, 3,458 used). Evaluation uses 200 MATH test problems of levels 3–5, every tokens. The level-5 (S14, S18, S19) and level 1–2 (S21) studies train and evaluate on those levels only, with their own 200-problem test sets. S33 and S34 also score two further test sets, only at the initial evaluation and at the first evaluation at or after . These scores are computed after the primary evaluation and with their own common random numbers, so they cannot affect training. The first holds 1,543 MATH test problems of levels 3–5 with numeric answers of at most 1,000 characters that are in no other evaluation set. Of 2,295 such problems of any length, 623 are in earlier evaluation sets, 12 are longer, and 117 are near-duplicates of the level 3–5 training pool or the 200 (exact match after normalization, or 8-gram Jaccard similarity of at least 0.6, the check used to clean the training pools). The second is the GSM8K test set (1,319 problems), on which neither configuration trains. At each evaluation the policy draws 8 completions per problem at temperature 1 and top- 1, with common random numbers across configurations of the same seed. The primary metric is accuracy at the first evaluation at or after , with no checkpoint selection. Greedy accuracy and strict-verifier accuracy, which counts only boxed answers, are secondary metrics.
Statistics. The unit is a seed, or a (model size, seed) pair when sizes are pooled. We use paired -tests, Holm adjustment within each preregistered primary family, and two one-sided tests for equivalence within ( in S28 and S31–S33, where the gain under test is 0.008). A crossed seed-by-problem bootstrap, which resamples seeds and test problems together, serves as a robustness check. Secondary contrasts are not corrected for multiple comparisons. Non-inferiority means that the lower bound of the 90% interval exceeds . Studies S15, S25, S26, and S32–S35 ran a smoke stage before their main runs, which checked that each configuration differed from its comparator only in the intended setting and that paired configurations shared their first iteration. Each preregistration also fixed how each possible outcome would be read, and the last sentence of each study below gives that reading. Because some of the gains we find are smaller than these margins, we report intervals throughout, alongside the equivalence tests.
Preregistration. Before its runs, each study preregistered its hypotheses, primary contrasts, Holm families, equivalence margin, budgets, and seeds. The preregistration files are kept in the project repository, so their timing rests on file-system records. For 27 of the 34 files whose studies have main training runs, the last modification precedes the study’s first main training run. Of the other seven, six changed only in comments, and S2’s two dated amendments added rules for parts of the study before those parts ran. Calibration, pilot, and smoke stages may start before a file is final (S24, S27, and S35), and dated amendments, such as S1’s change of learning rate after its pilot, are recorded in the files.
Monte-Carlo reward (S1, S25). The completion is cut into six segments at fixed token offsets. Continuations start at the five interior cuts, with draws each ( and 2 in two S25 configurations), and each continuation may use the rest of the completion’s token limit. The segment credit is added, without centering, to the group advantage. Because the cuts are at fixed offsets and the continuations run to the token limit, the credit targets the same reward as outcome-only GRPO, accuracy at the end of the completion: its expected contribution for a prompt is the gradient of that prompt’s success probability. Normalized GRPO’s expected update weights that gradient by , which varies across prompts (Proposition 4), so adding the credit can change how the gradients of different prompts are weighted relative to each other. The equal-iteration study’s rule, which centers the credit and normalizes it by its maximum, serves as a diagnostic configuration with the same continuations.
Learned critic (S2–S5, S26). A ridge regression on the policy’s hidden states predicts the success probability of a prefix. It is refit every iteration on the rollouts of earlier iterations, so each iteration is scored by a critic fit on other iterations. The temporal-difference form credits a segment with the change in predicted value across it. The baseline form subtracts the predicted value at the start of the segment from the outcome. Because the prediction depends only on the prefix and is fit on earlier iterations, this baseline does not change the expectation of the score-function estimator for the final reward. Both forms are added to the group advantage, which, as for the Monte-Carlo reward, can change how the gradients of different prompts are weighted relative to each other. At the initial Qwen2.5-1.5B policy, over 29,875 prefix cuts, the cross-fitted critic’s squared error against the prefix value is 0.057, where the value’s variance across cuts is 0.076 and the Monte-Carlo estimate’s error is 0.010; its Brier score against outcomes is 0.101, against 0.122 for a constant predictor. The critic is thus far noisier than the Monte-Carlo reward, but it needs no continuations.
Prompt filters (S6–S24, S27, S28, S30, S31). Each iteration draws 24 candidates from the seed’s prompt stream, 3 times the 8 prompts it trains on. The stream is the pool shuffled once per epoch and consumed in order, so a filter passes through it three times as fast as outcome-only GRPO. A candidate’s predicted success comes from one of three sources. The critic filter uses a ridge critic on the hidden state at the end of the prompt, refit on the group mean rewards of the previous 32 iterations. The history filter uses the prompt’s own mean group reward from the last iteration in which it was trained, or the recent mean reward for prompts not yet trained. The level filter uses the recent mean reward of the prompt’s MATH level. The trainer keeps the 6 candidates with nearest the target, by default, plus 2 random others. Until 32 prompts have been trained, it takes the first 8 candidates. The random-choice control (S11–S13, S15, S20) fits and logs the critic filter but trains on 8 of the 24 candidates drawn uniformly at random. Because the stream depends only on the seed and the iteration, the control and the filter of a seed score the same candidates in every iteration. DAPO-style dynamic sampling generates groups in stream order and keeps those with mixed outcomes until it has 8; every discarded group still counts toward .
Length reward (S6–S9). Some early configurations added a length term relative to the group. Every token of a correct completion has its credit changed by , where is the completion’s length, the mean length of the correct completions in its group, and the token limit. Wrong completions are unchanged, and the term sums to zero over each group. We report this term only because of the drift it caused.
Verifiers. The default verifier follows the equal-iteration study’s rule. It reads the last number in the final boxed expression, or, if there is no boxed expression, the last number in the completion. The strict verifier requires a boxed answer. An evaluation counts as drifted when defaultverifier accuracy exceeds strict-verifier accuracy by more than 0.1.
Overview
Tables 3 and 4 give every matched-compute contrast, Figure 7 and Figure 8 show the same contrasts with their per-seed values, and Table 5 lists the studies. The studies are numbered in the order they were preregistered; the artifact names each by its original label.
Figure 7. Figure 7. Every matched-compute comparison, part 1: dense rewards, and the choice of prompts on Qwen3-0.6B and 1.7B. Held-out accuracy at the token budget minus outcome-only GRPO on the same seeds and evaluation draws, on the 200 level 3–5 test problems (level 5 or level 1–2 problems for those pools), except in the rows marked “further test problems” and “GSM8K” (part 2). Gray dots are seeds; bars are 95% paired- CIs, blue when above zero and red when below; the band marks . “Fresh seeds” rows come from studies that repeated an earlier comparison on new seeds. “Default reward” and “strict reward” rows train both configurations for twice the budget with that verifier as the reward. Table 3 and Table 4 mark which rows are preregistered primary contrasts.
Figure 8. Every matched-compute comparison, part 2: the choice of prompts on Qwen2.5-1.5B, Qwen3-4B, and the Llama models, the learning rate, and the verifier. As in Figure 7. In the last two blocks, a run twice the learning rate or with a learning rate decaying linearly from to 0 at the budget, or trained on the strict verifier, minus the same configuration at or trained on the default verifier.
Training time per run. Scoring the 24 candidates adds one prefill pass per iteration, and the configurations complete different numbers of iterations within the budget because their completions differ in length. The mean training time per run, in hours, for the critic filter and for outcome-only GRPO is: 0.57 vs. 0.52 (Qwen3-0.6B, eighteen seeds), 0.93 vs. 0.89 (1.7B, 24), 2.45 vs. 2.52 (4B, level 3–5 pool, 12), 2.29 vs. 2.14 (4B, level 5, 12), 0.85 vs. 0.78 (1.7B, level 5, 24), 0.67 vs. 0.70 (0.6B, level 1–2, 24), 0.52 vs. 0.45 (Qwen2.5-1.5B, 36), 0.73 vs. 0.70 (Llama-3.2-3B, 24), and 0.525 vs. 0.450 (Llama-3.2-1B, 24), a change of −5% to +17%. The random-choice control, which also scores every candidate, takes 0.56, 0.96, and 0.50 hours on Qwen3-0.6B, Qwen3-1.7B, and Qwen2.5-1.5B.
Dense rewards at matched compute
These studies ask whether the per-update advantage of dense rewards survives once their continuations are paid for, and, if not, whether a cheaper estimate closes the gap. The Monte-Carlo reward spends 91% of its generated tokens on continuations, so by the time it reaches it has made 50 updates to outcome-only GRPO’s 584. That cost decides the comparison on every model we trained (Table 6). Fewer continuations per cut shrink the deficit, in the order the per-token gradient test predicts, because continuation sampling adds little noise to the update (S25). The heuristic extrapolation from that test overestimates all five rewards we checked by 0.008 to 0.018 on average over the evaluations, and every interval excludes zero. This error is small next to the Monte-Carlo rewards’ extrapolated deficits, so their order survives, but it is as large as the critics’ extrapolated leads. A learned critic, which needs no continuations, matches outcome-only GRPO within ±0.02 (S26); its temporal-difference form shortens completions, by 16% [6, 24] on Qwen3 and 3% [1, 6] on Qwen2.5-1.5B. The equal-iteration study’s rule for combining credit, which centers it and normalizes it by its maximum, is equivalent to ours with the same continuations (Table 6), so the result does not hinge on that choice. No Monte-Carlo reward we can afford, with and six segments, matches outcome-only GRPO at this budget, and the designs that reuse samples or train past the budget leave the ranking unchanged (Sections E and F).
| Comparison | Model, pool | Estimate | Test | ||
| S1 | Monte-Carlo reward, | Qwen2.5-1.5B, L4–5 | 5 | −0.143 [−0.160, −0.127] | Holm |
| at matched updates (50 vs. 584) | 5 | +0.015 [+0.010, +0.021] | exploratory | ||
| centered, max-normalized rule minus it | 5 | −0.007 [−0.013, +0.000] | TOST | ||
| Monte-Carlo reward, | Qwen3-0.6B, L4–5 | 3 | −0.080 [−0.093, −0.067] | ||
| Qwen3-1.7B, L4–5 | 3 | −0.091 [−0.118, −0.065] | |||
| Qwen3-4B, L4–5 | 3 | −0.100 [−0.142, −0.057] | |||
| Qwen2.5-1.5B, GSM8K | 3 | −0.098 [−0.134, −0.062] | |||
| S2 | Monte-Carlo reward, , 768 tokens | Qwen2.5-1.5B, L4–5 | 3 | −0.061 [−0.074, −0.048] | Holm |
| efficiency, reward, per update | Qwen2.5-1.5B | – | 1.56 [1.28, 2.03] | – | |
| per generated token | – | 0.145 [0.120, 0.190] | – | ||
| efficiency per token, TD critic | – | 1.36 [1.20, 1.62] | – | ||
| baseline critic | – | 1.15 [0.97, 1.39] | – | ||
| S4 | learned critic, TD form | Qwen2.5-1.5B, L4–5 | 5 | −0.008 [−0.028, +0.012] | TOST |
| baseline form | 5 | +0.006 [−0.014, +0.026] | TOST | ||
| S5 | learned critic, TD form | Qwen3, 3 sizes, L4–5 | 9 | −0.0003 [−0.016, +0.016] | TOST |
| baseline form | 9 | +0.010 [−0.004, +0.023] | , non-inferior | ||
| TD form, log length (16% shorter) | 9 | −0.169 [−0.280, −0.058] | Holm | ||
| S6 | critic filter | Qwen3 0.6B+1.7B, L4–5 | 12 | +0.023 [+0.009, +0.036] | Holm |
| per size | Qwen3-0.6B, L4–5 | 6 | +0.028 [+0.007, +0.050] | secondary | |
| Qwen3-1.7B, L4–5 | 6 | +0.017 [−0.007, +0.040] | secondary | ||
| S7 | critic minus history filter | Qwen3 0.6B+1.7B, L4–5 | 12 | +0.005 [−0.004, +0.015] | TOST |
| critic filter | Qwen3-4B, L4–5 | 8 | +0.002 [−0.013, +0.017] | Holm | |
| DAPO dynamic sampling | Qwen3 0.6B+1.7B, L4–5 | 12 | −0.018 [−0.027, −0.008] | secondary | |
| S8 | critic filter | Qwen3 0.6B+1.7B, L3–5 | 12 | +0.017 [+0.006, +0.028] | Holm |
| critic minus history filter | 12 | +0.003 [−0.007, +0.014] | TOST | ||
| history filter | 12 | +0.014 [+0.004, +0.024] | secondary | ||
| S9 | critic minus level filter | Qwen3 0.6B+1.7B, L3–5 | 12 | +0.015 [+0.004, +0.027] | Holm |
| level filter | 12 | +0.002 [−0.011, +0.015] | TOST | ||
| strict verifier as reward | Qwen3-4B, L4–5 | 3 | +0.008 [−0.020, +0.036] | non-inferior | |
| S10 | critic filter, target 0.7 | Qwen3, 3 sizes, L3–5 | 18 | +0.016 [+0.005, +0.027] | secondary |
Table 6. Table 6. Results of the first matched-compute studies quoted in § 3 and § 4. Differences are in held-out accuracy at against outcome-only GRPO on the same seeds unless stated, with 95% paired- CIs; efficiencies are ratios to outcome-only GRPO with 95% bootstrap CIs. “Holm” marks a preregistered primary contrast; other rows are secondary or, where marked, exploratory.
First studies (S1–S10)
Table 6 gives the source of every number from S1–S10 that Sections 3 and 4 quote and that later sections of this appendix do not report. Gradient efficiencies were measured on 96 fixed batches of 8 × 8 rollouts at the initial Qwen2.5-1.5B policy, relative to outcome-only GRPO. The expected alignment was positive for every method in every screen, at least three standard errors above zero, so squaring it in the efficiency hides no sign. Of these studies, the GSM8K study ran on A100 GPUs and the others on H200 GPUs.
Fewer continuations per cut (S25)
S25 repeats the S1 protocol (Qwen2.5-1.5B, level 4–5 pool, ) on fresh seeds 5–16 with four configurations: outcome-only GRPO and Monte-Carlo reward with , 2, and 1 continuations per interior cut.
The prediction. A second gradient measurement scored rewards with to 16 on 96 batches at the initial policy, against the same reference gradient as the first measurement but with separately sampled rollouts; its values for (1.63× per update and 0.15× per token) differ from the first measurement’s (1.56× and 0.145×) because of this resampling. Per update, the rewards with , 2, 4, 8, and 16 are 1.55×, 1.60×, 1.63×, 1.67×, and 1.69× as efficient as outcome-only GRPO; per generated token, they are 0.45× [0.35, 0.63], 0.27× [0.21, 0.38], 0.15× [0.12, 0.21], 0.08×, and 0.04× [0.03, 0.05]. Fitting the update’s second moment as attributes 3.8% [3.4, 4.2] of it at to continuation sampling. Therefore, no allocation of the continuations across cuts can raise the per-update efficiency by more than a factor of 1.04. The preregistered prediction was that every Monte-Carlo reward trails outcome-only GRPO and that the rewards order as .
Primary contrasts (Holm over two; ).
| Study | Artifact | Question | Model, pool | Where | |
| S1 | P3 | Does the Monte-Carlo reward pay for its continuations? | Qwen2.5-1.5B, L4–5; Qwen3, three sizes; GSM8K | 5; 3 each | C.3 |
| S2 | P4 | Longer completions; gradient efficiency per update and per token | Qwen2.5-1.5B, L4–5 | 3 | C.3 |
| S3 | P5 | Pilot: reusing samples | Qwen2.5-1.5B, L4–5 | 3 | E |
| S4 | P6 | Learned critic; pilot of other estimators | Qwen2.5-1.5B, L4–5 | 5; 3 | C.3 |
| S5 | P9 | Learned critic on other models | Qwen3, three sizes, L4–5 | 9 | C.3 |
| S6 | P10 | Does the critic filter help? | Qwen3 0.6B+1.7B, L4–5 | 12 | C.4 |
| S7 | P11 | History filter, dynamic sampling, a larger model | Qwen3, L4–5 | 12; 8 | C.4 |
| S8 | P12 | Critic filter and history filter on a wider pool | Qwen3 0.6B+1.7B, L3–5 | 12 | C.4 |
| S9 | P13 | Level filter; strict verifier as reward | Qwen3 0.6B+1.7B, L3–5; Qwen3-4B | 12; 3 | C.4 |
| S10 | P14 | Filter aimed at predicted success 0.7 | Qwen3, three sizes, L3–5 | 18 | C.4 |
| S11 | P15 | Random choice among the candidates; 4B replication | Qwen3-0.6B, Qwen3-4B, L3–5 | 6; 6 | C.4 |
| S12 | P16 | Random choice on fresh seeds | Qwen3-0.6B, L3–5 | 12 | C.4 |
| S13 | P17 | Replication with random choice | Qwen3-1.7B, L3–5 | 18 | C.4 |
| S14 | P18 | Is the 4B null a ceiling? | Qwen3-4B, level 5 | 12 | C.4 |
| S15 | P19 | Critic filter and random choice | Qwen2.5-1.5B, L3–5 | 12 | C.4 |
| S16 | P20 | Twice the budget | Qwen3-0.6B, L3–5 | 12 | C.6 |
| S17 | P21 | Twice the budget, strict reward | Qwen3-0.6B, L3–5 | 24 | C.6 |
| S18 | P22 | Strict verifier as reward | Qwen3-4B, level 5 | 12 | C.7 |
| S19 | P23 | Account: a gain predicted | Qwen3-1.7B, level 5 | 24 | C.5 |
| S20 | P24 | Replication on fresh seeds | Qwen2.5-1.5B, L3–5 | 24 | C.4 |
| S21 | P25 | Account: no gain predicted | Qwen3-0.6B, levels 1–2 | 24 | C.5 |
| S22 | P26 | A filter without a critic | Qwen2.5-1.5B, L3–5 | 36 | C.4 |
| S23 | P27 | Twice the learning rate | Qwen3-0.6B, L3–5 | 24 | C.6 |
| S24 | P28 | Account outside the Qwen family: no gain predicted | Llama-3.2-3B, L3–5 | 24 | C.5 |
| S25 | P29 | Fewer continuations per cut | Qwen2.5-1.5B, L4–5 | 12 | C.3 |
| S26 | P30 | Learned critic on fresh seeds | Qwen2.5-1.5B, L4–5 | 24 | C.3 |
| S27 | P31 | Account outside the Qwen family: a gain predicted | Llama-3.2-1B, L3–5 | 24 | C.5 |
| S28 | P32 | Twice the learning rate | Qwen2.5-1.5B, L3–5 | 24 | C.6 |
| S29 | P33 | Twice the learning rate without drift | Qwen3-0.6B, L3–5 | 24 | C.6 |
| S30 | P34 | Filter aimed at harder prompts | Qwen3-0.6B, L3–5 | 24 | C.5 |
| S31 | P35 | Filter aimed at harder prompts | Qwen2.5-1.5B, L3–5 | 24 | C.5 |
| S32 | P36 | Decaying learning rate | Qwen2.5-1.5B, L3–5 | 24 | C.6 |
| S33 | P37 | Further test problems | Qwen2.5-1.5B, L3–5 | 24 | C.4 |
| S34 | P38 | Further test problems | Qwen3-0.6B, L3–5 | 24 | C.4 |
| S35 | P39 | Equal training time past the budget | Qwen2.5-1.5B, L4–5, 768 tokens | 8 | F |
Table 5. The matched-compute studies, in the order they were preregistered. Artifact: the study’s label in the artifact, whose configuration files are named after it (S35, for example, is P39, configuration phase39). : seeds per configuration in the study’s primary contrasts (several values for several models or configurations). Where: the section that reports the study.
minus outcome-only GRPO: [, ], Holm , lower in all 12 seeds; crossed bootstrap [, ].
minus : [, ], Holm , higher in all 12; crossed [, ].
Secondary contrasts
minus outcome-only GRPO: [, ], which replicates S1’s . minus outcome-only: [, ]. Between rewards, leads by [, ] and leads by [, ], each in all twelve seeds. On strict-verifier accuracy, trails outcome-only GRPO by [, ], and averaged over the first half of the budget it trails by [, ].
Descriptive. At , the configurations have made 599, 163, 92, and 50 updates (outcome-only, , 2, 4). Continuations take 70%, 83%, and 91% of the Monte-Carlo rewards’ tokens. The share of cut-off samples falls from 74% at the start to 22%, 46%, 58%, and 67%, so the rewards with fewer updates keep more of the initial policy’s long completions. Training takes 0.43, 0.20, 0.16, and 0.14 hours per run. The deficit grows to 0.094 at and is 0.068 at . Its lead over grows to 0.081 at .
A heuristic extrapolation (exploratory). If a reward’s per-token efficiency held throughout training, training with it would reach, after tokens, the accuracy that outcome-only GRPO reaches after tokens. This is a heuristic, not a consequence of the analysis. Read off outcome-only GRPO’s curve on the same seed, it gives 0.359, 0.319, and 0.275 at for , 2, and 4. The rewards reached 0.343, 0.299, and 0.263, against outcome-only GRPO’s 0.412. Averaged over all 16 evaluations, observed minus extrapolated accuracy is [, ], [, ], and [, ]. Applied to the critics of Section 3.3 (seeds 0–4), the same extrapolation uses only the evaluations at which stays within the length of the outcome-only run: up to for the temporal-difference form and up to for the baseline form. It gives leads of and ; the critics led by [, ] and . Observed minus extrapolated is [, ] and [, ]. The extrapolation thus overestimates all five rewards by similar amounts. S26 then tested the critics’ shortfall as a preregistered prediction on 24 fresh seeds (next subsection), and it held. The efficiency is measured at the initial policy and describes the first-order gain of an SGD step tuned to each reward, whereas training uses Adam at one fixed learning rate.
Learned critic on fresh seeds (S26)
S26 repeated the S4 critic comparison on Qwen2.5-1.5B (level 4–5 pool, 380 tokens, M, the same configuration files) on seeds 5–28, after S25 suggested that the extrapolation overestimates the critics. Outcome-only seeds 5–16 are the S25 runs, finished before S26 was designed; seeds 17–28 were trained in S26. The primary family was equivalence within ±0.02 for both forms (TOST, Holm over the two). The preregistration also specified each primary contrast’s two-sided paired , Holm-adjusted over the two, and that a form significantly above zero would count as a gain at matched tokens.
Primary contrasts. Both forms are equivalent to outcome-only GRPO.
Temporal-difference form minus outcome-only: −0.0044 [−0.0115, +0.0027], TOST Holm , higher in 11 of 24 seeds; crossed bootstrap [−0.016, +0.007].
Baseline form minus outcome-only: +0.0067 [+0.0020, +0.0115], TOST Holm ; two-sided (Holm .016), higher in 17 of 24; crossed bootstrap [−0.002, +0.016].
By that rule, the baseline form gains. Its lead appears only at , however. At and it was −0.003 [−0.009, +0.003] and +0.005 [−0.003, +0.012].
Secondary contrasts. On strict-verifier accuracy, the temporal-difference form minus outcome-only GRPO is −0.021 [−0.049, +0.007]. Two of its runs drifted: seed 5 from on, by 0.29 to 0.36, and seed 15 at , by 0.14. Without them it is −0.004 [−0.011, +0.004]. On strict-verifier accuracy the baseline form leads by 0.004 [−0.002, +0.011]. Averaged over the first half of the budget, the temporal-difference form leads by 0.004 [0.001, 0.008] (, predicted no difference) and the baseline form by 0.000 [−0.003, +0.004]. The temporal-difference form trails the baseline form by 0.011 [0.005, 0.017] at (, predicted no difference).
Shortfall from the extrapolation (preregistered, F1 and F2). The extrapolation gave the temporal-difference form () a lead of +0.0205 [0.0195, 0.0216] on average over the evaluations up to . It led by +0.0023 [−0.0013, +0.0059], a shortfall of 0.018 [0.015, 0.022] (). It gave the baseline form () +0.0088 up to . That form led by +0.0001 [−0.003, +0.003], a shortfall of 0.009 [0.006, 0.012] (). Pooled with seeds 0–4 (, estimates), the forms differ from outcome-only GRPO by −0.005 [−0.011, +0.001] and +0.007 [0.002, 0.011] at , and fall short of the extrapolation by 0.018 and 0.009.
Descriptive. At , accuracy is 0.411 (outcome-only), 0.407 (temporal-difference), and 0.418 (baseline). The temporal-difference form’s greedy completions are 3% [1, 6] shorter, and 18.7% of its samples are truncated against 22.4%. Both critics are in use from iteration 4 on.
Does prompt selection help?
Prompt selection spends compute differently from dense rewards. One forward pass over 24 candidate prompts per iteration decides which 8 prompts to sample, and scoring generates no tokens. Curriculum methods that train on prompts with predicted success near 1/2 use the same rule (Gao et al., [9]; Qu et al. (2026); Sun et al. [28]), so we test the rule rather than propose it. It gains on two of six models with the level 3–5 pool, Qwen3-0.6B and Qwen2.5-1.5B, and both gains replicate on fresh seeds (Table 3). The studies below probe the gain from four sides. Fresh seeds and further test sets ask whether it is real (S12, S20, S33, S34). Random choice among the same candidates asks whether it comes from the choice of prompts or from moving faster through the prompt stream (S11–S13, S15, S20). Simpler estimates of a prompt’s success ask whether the learned critic is needed (S7–S9, S22). Other models and pools ask how far the gain extends (S13, S14, and Section C.5).
| Method | Model, training pool | accuracy [95% CI] | Test | |
| Dense rewards within a completion | ||||
| Monte-Carlo reward, | Qwen2.5-1.5B, L4–5 | 5 | −0.143 [−0.160, −0.127] | Holm |
| same, at matched updates | Qwen2.5-1.5B, L4–5 | 5 | +0.015 [+0.010, +0.021] | exploratory |
| fresh seeds | Qwen2.5-1.5B, L4–5 | 12 | −0.149 [−0.158, −0.140] | secondary |
| Monte-Carlo reward, | Qwen2.5-1.5B, L4–5 | 12 | −0.112 [−0.124, −0.101] | secondary |
| Monte-Carlo reward, | Qwen2.5-1.5B, L4–5 | 12 | −0.068 [−0.076, −0.060] | Holm |
| minus | Qwen2.5-1.5B, L4–5 | 12 | +0.081 [+0.071, +0.090] | Holm |
| , at matched updates | Qwen2.5-1.5B, L4–5 | 12 | +0.014 / +0.018 / +0.010 | exploratory |
| Monte-Carlo reward, | Qwen3 0.6B/1.7B/4B, L4–5 | 3 each | −0.080 / −0.091 / −0.100 | each |
| Learned critic, TD form | Qwen2.5-1.5B, L4–5 | 24 | −0.004 [−0.012, +0.003] | Holm TOST |
| first study | Qwen2.5-1.5B, L4–5 | 5 | −0.008 [−0.028, +0.012] | TOST |
| Learned critic, baseline form | Qwen2.5-1.5B, L4–5 | 24 | +0.007 [+0.002, +0.012] | Holm , TOST |
| first study | Qwen2.5-1.5B, L4–5 | 5 | +0.006 [−0.014, +0.026] | TOST |
| Learned critic, TD form | Qwen3, 3 sizes, L4–5 | 9 | −0.0003 [−0.016, +0.016] | TOST |
| Learned critic, baseline form | Qwen3, 3 sizes, L4–5 | 9 | +0.010 [−0.004, +0.023] | , non-inferior |
| At outcome-only GRPO’s training time at (768 tokens, strict reward, A100) | ||||
| Monte-Carlo reward, | Qwen2.5-1.5B, L4–5 | 8 | −0.033 [−0.043, −0.024] | Holm |
| at matched tokens | Qwen2.5-1.5B, L4–5 | 8 | −0.066 [−0.079, −0.052] | secondary |
| Monte-Carlo reward, | Qwen2.5-1.5B, L4–5 | 8 | −0.011 [−0.028, +0.005] | Holm |
| VinePPO-style, 4 steps per batch | Qwen2.5-1.5B, L4–5 | 8 | −0.011 [−0.017, −0.004] | Holm , TOST |
| Continuation reuse, | Qwen2.5-1.5B, L4–5 | 8 | −0.052 [−0.063, −0.041] | Holm |
| Critic filter on Qwen3-0.6B and Qwen2.5-1.5B | ||||
| Critic filter | Qwen3-0.6B, L3–5 | 54 | +0.017 [+0.012, +0.021] | estimate, 42 of 54 higher |
| fresh seeds of one study | Qwen3-0.6B, L3–5 | 12 | +0.025 [+0.015, +0.034] | secondary |
| fresh seeds of a fourth study | Qwen3-0.6B, L3–5 | 24 | +0.012 [+0.005, +0.019] | Holm |
| same, 1,543 further test problems | Qwen3-0.6B, L3–5 | 24 | +0.011 [+0.006, +0.015] | Holm , 21 of 24 higher |
| same, GSM8K test set | Qwen3-0.6B, L3–5 | 24 | +0.013 [+0.007, +0.019] | secondary |
| default reward, at | Qwen3-0.6B, L3–5 | 12 | +0.021 [−0.001, +0.042] | Holm |
| strict reward, at | Qwen3-0.6B, L3–5 | 24 | +0.019 [+0.013, +0.026] | secondary, 22 of 24 higher |
| strict reward, at | Qwen3-0.6B, L3–5 | 24 | +0.005 [−0.005, +0.016] | Holm , TOST |
| both at learning rate | Qwen3-0.6B, L3–5 | 24 | +0.009 [−0.006, +0.024] | Holm |
| at vs. outcome at | Qwen3-0.6B, L3–5 | 24 | −0.005 [−0.018, +0.007] | Holm , TOST |
| same, strict reward | Qwen3-0.6B, L3–5 | 24 | +0.000 [−0.012, +0.013] | Holm , TOST |
| Critic filter | Qwen3-0.6B, levels 1–2 | 24 | −0.005 [−0.011, +0.000] | secondary, TOST |
| minus its gain on L3–5 | Qwen3-0.6B | 24, 18 | −0.030 [−0.039, −0.020] | Holm (Welch) |
| Critic filter | Qwen2.5-1.5B, L3–5 | 60 | +0.008 [+0.004, +0.011] | estimate, 42 of 60 higher |
| first study | Qwen2.5-1.5B, L3–5 | 12 | +0.009 [+0.001, +0.017] | Holm |
| fresh seeds of one study | Qwen2.5-1.5B, L3–5 | 24 | +0.008 [+0.003, +0.014] | Holm |
| fresh seeds of a third study | Qwen2.5-1.5B, L3–5 | 24 | +0.007 [+0.001, +0.013] | Holm |
| same, 1,543 further test problems | Qwen2.5-1.5B, L3–5 | 24 | +0.008 [+0.005, +0.011] | Holm , 23 of 24 higher |
| same, GSM8K test set | Qwen2.5-1.5B, L3–5 | 24 | +0.011 [+0.006, +0.016] | secondary |
| both at learning rate | Qwen2.5-1.5B, L3–5 | 24 | +0.011 [+0.002, +0.020] | Holm |
| at vs. outcome at | Qwen2.5-1.5B, L3–5 | 24 | +0.040 [+0.033, +0.047] | Holm |
| both at decaying learning rate | Qwen2.5-1.5B, L3–5 | 24 | +0.002 [−0.005, +0.009] | Holm , TOST |
| at vs. outcome, decaying rate | Qwen2.5-1.5B, L3–5 | 24 | +0.005 [−0.003, +0.012] | Holm , TOST |
Table 3. Table 3. Every matched-compute contrast, part 1: dense rewards within a completion, and the critic filter on the two models where it gained. Held-out accuracy at the first evaluation at or after the token budget , or after or at outcome-only GRPO’s training time at where marked, minus outcome-only GRPO on the same seeds and evaluation draws. Evaluation uses the setting’s 200 test problems (8 samples each), except in the rows marked “further test problems” (the 1,543 numeric-answer MATH test problems of levels 3–5 outside every other evaluation set) and “GSM8K” (its 1,319 test problems). Intervals are 95% paired- CIs over seeds; the bars draw them on the scale above each block, over a gray band at , blue when the interval lies above zero and red when below. The per-size Monte-Carlo rows give three means without intervals. “Holm” marks a preregistered primary contrast with its Holm-adjusted ; “TOST” gives the equivalence at (studies on Qwen2.5-1.5B that compare against its gain of 0.008 preregistered , reported in the text). Secondary, estimate (seeds pooled over several studies of the same configurations), and exploratory rows carry no multiplicity correction. The Monte-Carlo reward scores six segments from continuations at each cut; learned critics are ridge regressions on hidden states that credit the change in predicted value (TD form) or subtract it from the outcome (baseline form); the critic filter keeps the 6 of 24 candidates whose predicted success is nearest 1/2, plus 2 at random. Table 4 continues the table.
The answers are consistent. On Qwen2.5-1.5B, the gain carries over to 1,543 MATH test problems that no study had evaluated on, with an interval that also resamples the problems, and to GSM8K, which neither configuration trains on; the same holds on Qwen3-0.6B. On both models, the choice of prompts carries the gain: random choice among the same candidates is equivalent to outcome-only GRPO (on Qwen3-0.6B, +0.008 [−0.003, +0.019] over 18 seeds from two studies, TOST ), and it trails the filter in every preregistered primary contrast. A per-prompt success estimate near 1/2 carries most of the gain at even without a learned critic, and the critic adds an early lead. A filter built from the MATH difficulty level alone gains nothing, and dynamic sampling loses at matched tokens, because the groups it discards still count toward the budget.
Random choice among the candidates (S11, S12) and a Qwen3-4B replication (S11)
Table 7 lists the Qwen3-0.6B configurations that share a seed’s prompt stream and evaluation draws. Random choice among the filter’s candidates reproduces outcome-only GRPO’s mix of uniform and mixed groups, while training on fewer distinct prompts. The three critic-filter configurations train on fewer all-wrong groups than either. Target 0.85 trains on the same share of mixed-outcome groups as random choice and still leads it by 0.016 [0.009, 0.024] over eighteen seeds (higher in 15). The share of mixed-outcome groups, which dynamic sampling maximizes, therefore does not by itself account for the differences in accuracy.
| Configuration | accuracy | wrong | correct | mixed | prompts |
| Outcome only | 0.440 | 0.359 | 0.140 | 0.500 | 3,458 |
| Random choice | 0.447 | 0.353 | 0.144 | 0.503 | 2,557 |
| Filter, | 0.464 | 0.235 | 0.153 | 0.612 | 2,248 |
| Filter, | 0.466 | 0.167 | 0.267 | 0.566 | 2,215 |
| Filter, | 0.464 | 0.151 | 0.344 | 0.505 | 2,130 |
Table 7. Qwen3-0.6B configurations on one prompt stream (L3–5 pool; seeds 0–17, , except target 0.7, seeds 0–5, ). Accuracy at ; shares of groups that were all-wrong, all-correct, or mixed; and distinct prompts trained, over iterations before . Filter rows are the critic filter at target ; random choice draws from the same candidates.
The first random-choice study (S11, seeds 0–5) left its primary contrast unresolved: random choice minus the filter was −0.017 [−0.037, +0.003] (Holm ). Adding seeds to that contrast after seeing the result would have been optional stopping. S12 therefore preregistered the same contrast as its only primary test on twelve fresh seeds (6–17), repeating random choice, the filter at targets 1/2 and 0.85, and outcome-only GRPO. Pooled values over seeds 0–17 are reported as estimates.
The manipulation check passed in every run. The control’s chosen prompts matched the candidates’ predicted success to within 0.005, and it trained on 0.118 [0.109, 0.127] more all-wrong groups than the filter of the same seed. The control and the filter scored identical candidates in every iteration that both runs reached. On the fresh seeds, random choice minus the filter is −0.017 [−0.029, −0.005] (; per seed −0.029, −0.027, −0.008, −0.038, +0.002, −0.020, +0.023, −0.039, −0.014, −0.001, −0.028, −0.024). The crossed seed-by-problem bootstrap gives [−0.032, −0.002]. Strict-verifier accuracy moves the same way (−0.017 [−0.036, +0.002]). The secondary contrasts on the fresh seeds are: filter minus outcome +0.025 [0.015, 0.034] (); random choice minus outcome +0.008 [−0.007, +0.023] (TOST ); target 0.85 minus random choice +0.015 [0.005, 0.024] ( = .007); target 0.85 minus target 1/2 −0.002 (TOST = .005); and target 0.85 minus outcome +0.022 [0.007, 0.038] ( = .010).
Per seed, the filter’s gain over outcome-only GRPO splits into the critic’s choice (filter minus random choice) and the stream (random choice minus outcome). Over the eighteen seeds, the gain is +0.024 [0.016, 0.033], the critic’s choice +0.017 [0.008, 0.026], and the stream +0.008 [−0.003, +0.019]; the parts sum to the gain up to rounding.
On Qwen3-4B, fresh seeds 6–11 repeated target 0.7, target 1/2, and outcome-only GRPO. Per seed, target 0.7 minus target 1/2 was −0.017, −0.016, −0.040, +0.016, +0.012, and −0.022. Pooled with seeds 0–5 ( = 12, secondary), the three pairwise differences are +0.003 (target 0.7 minus 1/2), +0.0001 (target 0.7 minus outcome), and −0.002 (1/2 minus outcome). Each is equivalent within ±0.02 (TOST = .008, .004, .006). One fresh filter run stopped boxing its answers late in training and ended with strict-verifier accuracy 0.004 at default-verifier accuracy 0.814. Over the twelve seeds, one outcome-only run (strict 0.312) and no target-0.7 run ended drifted.
Qwen3-1.7B replication with random choice (S13)
On seeds 0–5 (S8), the Qwen3-1.7B filter led outcome-only GRPO by +0.010 [0.001, 0.019] ( = .033). S13 preregistered random choice minus the filter as its only primary contrast on eighteen fresh seeds (6–23), with the filter and outcome-only configurations repeated on those seeds and random choice added on seeds 0–5. Random choice minus the filter was −0.005 [−0.013, +0.003] ( = .184; crossed bootstrap [−0.017, +0.006]), equivalent to zero within ±0.02 (TOST < .001). The filter’s gain did not replicate: filter minus outcome was −0.0004 [−0.012, +0.011] (TOST = .001), and random choice minus outcome −0.005 [−0.018, +0.007]. Over all 24 seeds, the three differences are +0.002 [−0.006, +0.011] (filter minus outcome), −0.004 [−0.011, +0.003] (random minus filter), and −0.002 [−0.013, +0.009] (random minus outcome). The manipulation check passed in every run. The control’s chosen prompts matched the candidates’ predicted success to within 0.007. It trained on 2.5 [2.1, 2.9] points more all-wrong groups than the filter, and the two configurations scored identical candidates in all 11,102 shared iterations before . On this size the filter removes mainly all-correct groups. Over the fresh seeds, the all-correct share falls from 38% to 27% and the all-wrong share from 25% to 22%, while the mixed share rises from 38% to 51%.
Qwen3-4B on level-5 problems (S14)
On Qwen3-4B, outcome-only accuracy on the level 3–5 test set rises from 0.62 to 0.76 within and then plateaus near 0.81. S14 asked whether the 4B null reflects this ceiling. Both configurations trained on the level-5 problems of the training pool (1,460 after removing near-duplicates of the new test set) and were evaluated on 200 level-5 test problems disjoint from the level 3–5 set. There, 4B starts at 0.350, and outcome-only GRPO reaches 0.629 at , below the preregistered ceiling of 0.80. The primary contrast, filter minus outcome at , was +0.002 [−0.028, +0.032] ( = .889; crossed [−0.027, +0.033]; TOST = .109; per-seed SD 0.048). Secondary results:
The filter led early: +0.018 at , +0.042 at ( < ), and +0.016 at . Its mean lead over the first half of the budget was +0.013 [0.001, 0.025] ( = .032).
Strict-verifier accuracy differed by +0.099 [−0.051, +0.250], carried by two drifted outcome-only runs.
The filter shortened completions (log length −0.134 [−0.222, −0.046]), with 11% of samples truncated at against 17%.
It trained on 10.4 [9.0, 11.7] points more mixed groups.
Its chosen prompts were nearer 1/2 than its candidates in all twelve runs (0.514 against 0.544).
Qwen2.5-1.5B with random choice (S15)
S15 repeated the random-choice design on the equal-iteration study’s Qwen2.5-1.5B-Instruct. It used the level 3–5 pool, 380-token completions, = 12.25M, and seeds 0–11. The filter and random-choice configurations used the Qwen3 filter settings. Holm adjustment covered two primary contrasts:
Filter minus outcome: +0.0086 [0.0005, 0.0168] (Holm = .039; higher in 9 of 12; crossed bootstrap [−0.004, +0.022]).
Random choice minus the filter: −0.0101 [−0.0165, −0.0036] (Holm = .011; lower in 10 of 12; crossed [−0.023, +0.002]).
Random choice minus outcome was −0.001 [−0.008, +0.005] (TOST < .001). Strict-verifier accuracy moves the same way: +0.008 [0.001, 0.015] for filter minus outcome and −0.019 [−0.034, −0.003] for random choice minus the filter. Per seed, the gain splits into the critic’s choice, +0.010 [0.004, 0.016], and the stream, −0.001 [−0.008, +0.005]. The manipulation check passed. The control’s chosen prompts matched the candidates’ predicted success to within 0.006, and it trained on 12.3 points more all-wrong groups than the filter. The two configurations scored identical candidates in all 7,527 shared iterations. The filter’s lead was +0.030 at and and declined to +0.009 at .
Replication on Qwen2.5-1.5B (S20)
S20 repeated the three S15 configurations on fresh seeds 12–35, with no other change. Holm adjustment covered two primary contrasts:
Filter minus outcome: +0.0083 [0.0031, 0.0135] (Holm = .006; higher in 18 of 24; crossed bootstrap [−0.001, +0.018]).
Random choice minus the filter: −0.0087 [−0.0148, −0.0026] (Holm = .007; lower in 18 of 24; crossed [−0.019, +0.001]).
Random choice minus outcome was −0.0004 [−0.006, +0.005] (TOST < .001). Per seed, the gain splits into the critic’s choice, +0.009 [0.003, 0.015], and the stream, −0.000 [−0.006, +0.005]. The manipulation check passed. The control’s chosen prompts matched the candidates’ predicted success to within 0.011, it trained on 12.6 points more all-wrong groups than the filter, the filter’s chosen prompts were nearer than its candidates in every run, and the two configurations scored identical candidates in all 14,885 shared iterations. The candidates’ mean predicted success was 0.395 (S15: 0.398), and the filter raised mixed groups by 9.7 points [9.1, 10.3]. Over the 36 seeds of S15 and S20, filter minus outcome is +0.0084 [0.0043, 0.0126] (higher in 27), random choice minus the filter −0.0091 [−0.0135, −0.0048] (lower in 28), and random choice minus outcome −0.0007 [−0.0049, +0.0034]. Two outcome-only runs (seeds 30 and 32) drifted, the first drift on this model. In strict-verifier accuracy, those two seeds put the filter ahead by +0.247 and +0.223, which makes the strict-verifier difference +0.028 [0.000, 0.055] ( = .049, secondary). In default-verifier accuracy, which the primary contrast uses, the same seeds differ by +0.006 and +0.003.
Further test problems on Qwen2.5-1.5B (S33)
In S15 and S20 the crossed bootstrap intervals of the gain reached zero, so S33 asked whether the gain carries over to other test problems. It repeated the S20 filter and outcome-only configurations on fresh seeds 36–59 with one addition: at the initial evaluation and at the first evaluation at or after , each run also scored the two further test sets described under Pools and evaluation. The margin was ±0.01. Holm adjustment covered two primary contrasts, filter minus outcome-only GRPO at :
On the 1,543 further MATH test problems: +0.0084 [0.0054, 0.0114] (Holm = ; higher in 23 of 24 seeds; crossed bootstrap [0.004, 0.012]).
On the 200 problems: +0.0067 [0.0006, 0.0129] (Holm = .034; higher in 15 of 24; crossed [−0.004, +0.017]).
Neither is equivalent to zero within ±0.01 (TOST = .14 for each). Secondary results:
GSM8K, with no directional prediction: +0.0112 [0.0059, 0.0164] ( < .001; higher in 19 of 24; crossed [0.005, 0.017]). From 0.596 for both configurations at the start, the filter reached 0.717 and outcome-only GRPO 0.706.
Strict-verifier accuracy: −0.001 [−0.021, +0.018] on the further MATH problems, −0.004 [−0.024, +0.017] on the 200, and −0.020 [−0.066, +0.027] on the GSM8K. One filter run (seed 57) was drifted at on both MATH sets, with default-verifier accuracy 0.40 and strict-verifier accuracy 0.19 on the further problems, and on GSM8K so was a second (seed 40); no outcome-only run drifted. Without the drifted pairs (exploratory), the strict contrasts are +0.008 [0.003, 0.013], +0.006 [−0.002, +0.014], and +0.012 [0.006, 0.017], and the default ones +0.009, +0.007, and +0.012.
Over the first half of the budget, the filter led by 0.020 [0.017, 0.023] on the 200, in every seed, and it raised mixed groups by 9.7 points [9.4, 10.1], in every seed.
Descriptive: at the initial evaluation, both configurations of every seed scored the same on every set. On the further MATH problems, accuracy rose from 0.212 to 0.407 for outcome-only GRPO and 0.415 for the filter. The gain was +0.009 [0.004, 0.013] on level 3 (586 problems), +0.009 [0.005, 0.013] on level 4 (555), and +0.007 [0.003, 0.011] on level 5 (402). Per seed, the gain on the further problems minus that on the 200 was +0.002 [−0.005, +0.008]. The candidates’ mean predicted success was 0.398, and the chosen prompts’ 0.460. Over the 66 seeds of S15, S20, and S33, the filter minus outcome-only GRPO on the 200 is +0.0078 [0.0043, 0.0112] (higher in 42; exploratory). The gain therefore carries over to other test problems from the same distribution.
Further test problems on Qwen3-0.6B (S34)
S34 is the companion of S33 on the other model with a replicated gain. It repeated the S12 Qwen3-0.6B filter and outcome-only configurations on the level 3–5 pool, on fresh seeds 42–65 (earlier Qwen3-0.6B studies used seeds 0–41), with S33’s one addition: at the initial evaluation and at the first evaluation at or after (17.25M), each run also scored the two further test sets. The margin was ±0.02. Holm adjustment covered two primary contrasts, filter minus outcome-only GRPO at :
On the 1,543 further MATH test problems: +0.0108 [0.0064, 0.0152] (Holm = ; higher in 21 of 24 seeds and lower in 2; crossed bootstrap [0.006, 0.016]).
On the 200 problems: +0.0117 [0.0045, 0.0188] (Holm = .003; higher in 16 of 24 and lower in 7; crossed [0.0022, 0.0215]).
Both predictions held (Figure 9). Secondary results:
Figure 9. The prompt filter’s gain carries over to further test sets. Critic filter minus outcome-only GRPO at the token budget, on the same runs: the 200 test problems of every study, the 1,543 numeric-answer MATH test problems of levels 3–5 that no study had evaluated on, and the 1,319 GSM8K test problems, on Qwen3-0.6B (S34) and Qwen2.5-1.5B (S33), 24 fresh seeds each. Dots are seeds, bars 95% paired- CIs, and the gray band marks ±0.02.
GSM8K, with no directional prediction: +0.0131 [0.0067, 0.0194] (; higher in 20 of 24; crossed [0.006, 0.019]). From 0.579 for both configurations at the start, the filter reached 0.673 and outcome-only GRPO 0.660.
Strict-verifier accuracy: +0.017 [0.000, 0.034] on the further MATH problems (), +0.017 [0.002, 0.031] on the 200 (), and +0.041 [−0.017, +0.098] on GSM8K (). One run of each configuration was drifted at on both MATH sets (outcome-only seed 46 and filter seed 52), and on GSM8K two of each.
Over the first half of the budget, the filter led by 0.009 [0.006, 0.013] on the 200 (higher in 20 of 24), and it raised mixed groups by 11.1 points [10.6, 11.6], in every seed.
Descriptive: at the initial evaluation, both configurations of every seed scored the same on every set. On the further MATH problems, accuracy rose from 0.375 to 0.469 for outcome-only GRPO and 0.480 for the filter. The gain was +0.008 [0.003, 0.013] on level 3 (586 problems), +0.012 [0.008, 0.017] on level 4 (555), and +0.013 [0.007, 0.018] on level 5 (402). Per seed, the gain on the further problems minus that on the 200 was −0.001 [−0.007, +0.005]. The candidates’ mean predicted success was 0.42, and the chosen prompts’ 0.47. Pooled with seeds 0–23 as preregistered, filter minus outcome-only GRPO on the 200 is +0.017 [0.012, 0.022] (48 seeds, higher in 38; exploratory), and over all 54 seeds on this model, +0.017 [0.012, 0.021] (higher in 42). The replication’s gain is smaller than the first studies’ +0.021 on seeds 0–29, and at equal training time it is +0.004 [−0.001, +0.010], not significant, against +0.015 [0.008, 0.022] on seeds 0–29 (exploratory; Table 13). The gain on this model also carries over to the further problems.
| Configuration | Base | Equal tokens | Equal training time | Hours to (configuration / base) | |
| Monte-Carlo, | outcome-only | 12 | −0.149 | −0.078 [−0.088, −0.068] | 0.136 / 0.431 |
| Monte-Carlo, | outcome-only | 12 | −0.112 | −0.051 [−0.062, −0.040] | 0.160 / 0.431 |
| Monte-Carlo, | outcome-only | 12 | −0.068 | −0.022 [−0.028, −0.017] | 0.201 / 0.431 |
| Critic, TD form | outcome-only | 24 | −0.004 | −0.012 [−0.017, −0.007] | 0.536 / 0.431 |
| Critic, baseline form | outcome-only | 24 | +0.007 | −0.007 [−0.012, −0.001] | 0.531 / 0.431 |
| Monte-Carlo, , 768 tokens | outcome-only | 3 | −0.061 | −0.016 [−0.045, +0.013] | 0.202 / 0.556 |
| Critic, TD form, 768 tokens | outcome-only | 3 | −0.011 | −0.018 [−0.050, +0.015] | 0.786 / 0.556 |
| Critic, baseline form, 768 tokens | outcome-only | 3 | +0.008 | +0.006 [−0.021, +0.033] | 0.653 / 0.556 |
| Monte-Carlo, , 4 steps per batch | outcome-only | 3 | −0.071 | −0.019 [−0.079, +0.042] | 0.165 / 0.424 |
| Outcome-only, 4 steps per batch | outcome-only | 3 | −0.059 | −0.032 [−0.082, +0.018] | 0.819 / 0.424 |
| VinePPO-style | outcome-only | 3 | −0.150 | −0.080 [−0.104, −0.057] | 0.136 / 0.424 |
| VinePPO-style, 4 steps per batch | outcome-only | 3 | −0.070 | −0.023 [−0.048, +0.003] | 0.172 / 0.424 |
| Outcome-only, lr | outcome-only | 3 | −0.034 | −0.036 [−0.095, +0.022] | 0.486 / 0.424 |
| Outcome-only, lr | outcome-only | 3 | −0.111 | −0.096 [−0.184, −0.009] | 0.477 / 0.424 |
| Critic filter | outcome-only | 60 | +0.008 | +0.005 [+0.002, +0.008] | 0.516 / 0.451 |
| seeds of the third study | outcome-only | 24 | +0.007 | +0.006 [+0.001, +0.010] | 0.517 / 0.450 |
| History prompt filter | outcome-only | 36 | +0.006 | +0.007 [+0.002, +0.011] | 0.452 / 0.452 |
| Random choice of candidates | outcome-only | 36 | −0.001 | −0.002 [−0.006, +0.002] | 0.496 / 0.452 |
| Outcome-only, decaying lr | filter at | 24 | −0.005 | −0.003 [−0.010, +0.004] | 0.474 / 0.514 |
| Filter, decaying lr | outcome-only, decaying lr | 24 | +0.002 | +0.002 [−0.005, +0.009] | 0.544 / 0.474 |
| Critic filter, Qwen3-0.6B | outcome-only | 54 | +0.017 | +0.010 [+0.006, +0.015] | 0.570 / 0.522 |
| seeds of the fourth study | outcome-only | 24 | +0.012 | +0.004 [−0.001, +0.010] | 0.568 / 0.525 |
Table 13. Matched generated tokens versus matched training time, on the same runs. Configuration minus base in held-out accuracy at the first evaluation at or after (equal tokens) and at equal training time (exploratory), with 95% paired- CIs over seeds; training hours before the evaluation at on H200 GPUs. Qwen2.5-1.5B on the level 4–5 pool at 380 tokens unless marked; prompt-selection rows on the level 3–5 pool.
A filter without a critic on Qwen2.5-1.5B (S22)
S22 asked whether the Qwen2.5-1.5B gain needs a learned critic. Its one new configuration is the S15/S20 filter configuration with the history estimate in place of the critic: the same candidates, rule, warm-up, and random picks. It ran on the same 36 seeds, paired with the S15 and S20 runs of each seed. The history filter has no per-prompt estimate until a prompt has been trained. A filter run scores about 15,200 candidates before , 4.4 passes through the 3,458 prompts of the stream, so its first pass ends near 0.23. Holm adjustment covered two primary contrasts:
History filter minus outcome, with no directional prediction: +0.0060 [0.0011, 0.0110] (Holm = .035; higher in 22 of 36; crossed bootstrap [−0.002, +0.014]).
Critic filter minus history filter, predicted positive: +0.0024 [−0.0020, +0.0068] (Holm = .27; higher in 21 of 36; crossed [−0.006, +0.010]), equivalent to zero within ±0.02 (TOST ).
Secondary results:
Over the first half of the budget, the critic filter led the history filter by 0.019 [0.016, 0.021] on average, in all 36 seeds. By evaluation, its lead was +0.020 at , peaked at +0.026 at , and fell to +0.002 at . Over the same half, the history filter led outcome-only GRPO by +0.002 [−0.001, +0.004].
The history filter led random choice from its candidates by +0.007 [0.002, 0.012] ($p = .010).
Strict-verifier accuracy: +0.017 [−0.001, +0.035] over outcome-only, carried by the two drifted outcome-only runs (+0.236 and +0.204 on seeds 30 and 32).
The history filter raised mixed groups by 4.8 points [4.4, 5.2], higher in every seed, about half the critic filter’s 9.8 on the same seeds.
Descriptive: its candidates’ estimated success averaged 0.33, and its chosen prompts’ 0.37. Its mean training reward before was 0.40, against 0.37 for outcome-only GRPO and 0.46 for the critic filter. It trained on 3,444 distinct prompts per run on average, against the critic filter’s 2,605. A per-prompt success estimate near 1/2 thus carries the gain at without a learned critic. The interval on the critic filter’s advantage at , [−0.002, +0.007], does not exclude a difference nearly as large as the critic filter’s own gain (+0.008). On this model, the critic’s clearest contribution is the early lead.
Where does prompt selection help?
The filter changes the trained groups in every setting: it raises the share of mixed-outcome groups, the only groups with a nonzero GRPO advantage, by 8 to 19 points. Accuracy, however, rises in only some settings, so the composition of the groups does not predict the gain. On Qwen3-0.6B, the filter aimed at predicted success 0.85 trains on about the same share of mixed groups as outcome-only GRPO (+0.5 points, compared with +0.3 for random choice) and still gains as much as the filter aimed at 1/2 (Table 4). The share of all-wrong groups does not predict the gain either: on Qwen3-0.6B the target 0.85 cuts it from 36% to 15% and gains, while on Qwen3-1.7B it cuts it from 25% to 10% and gains nothing (+0.001, ).
| Method | Model, training pool | accuracy [95% CI] | Test | |
| Critic filter at the other four models | ||||
| Critic filter | Qwen3-1.7B, L3–5 | 24 | +0.002 [−0.006, +0.011] | estimate, TOST |
| fresh seeds of one study | Qwen3-1.7B, L3–5 | 18 | −0.0004 [−0.012, +0.011] | secondary, TOST |
| Critic filter | Qwen3-1.7B, level 5 | 24 | +0.011 [+0.002, +0.020] | Holm |
| Critic filter | Qwen3-4B, L4–5 | 8 | +0.002 [−0.013, +0.017] | Holm |
| Critic filter | Qwen3-4B, L3–5 | 12 | −0.002 [−0.015, +0.010] | TOST |
| Critic filter | Qwen3-4B, level 5 | 12 | +0.002 [−0.028, +0.032] | Holm |
| Critic filter | Llama-3.2-3B, L3–5 | 24 | −0.004 [−0.010, +0.002] | Holm , TOST |
| Critic filter | Llama-3.2-1B, L3–5 | 24 | +0.004 [−0.001, +0.010] | Holm , TOST |
| Random choice of the filter’s candidates | ||||
| Random choice | Qwen3-0.6B, L3–5 | 18 | +0.008 [−0.003, +0.019] | Holm (6), .008 (12) |
| Random choice | Qwen2.5-1.5B, L3–5 | 36 | −0.001 [−0.005, +0.003] | Holm (12), .007 (24) |
| Random choice | Qwen3-1.7B, L3–5 | 24 | −0.002 [−0.013, +0.009] | Holm (18) |
| Other selection rules | ||||
| Critic filter | Qwen3 0.6B+1.7B, L4–5 | 12 | +0.023 [+0.009, +0.036] | Holm |
| Critic filter | Qwen3 0.6B+1.7B, L3–5 | 12 | +0.017 [+0.006, +0.028] | Holm |
| History prompt filter | Qwen3 0.6B+1.7B, L3–5 | 12 | +0.014 [+0.004, +0.024] | secondary |
| History prompt filter | Qwen2.5-1.5B, L3–5 | 36 | +0.006 [+0.001, +0.011] | Holm |
| critic filter minus history filter | Qwen2.5-1.5B, L3–5 | 36 | +0.002 [−0.002, +0.007] | Holm , TOST |
| Difficulty-level filter | Qwen3 0.6B+1.7B, L3–5 | 12 | +0.002 [−0.011, +0.015] | TOST |
| critic filter minus level filter | Qwen3 0.6B+1.7B, L3–5 | 12 | +0.015 [+0.004, +0.027] | Holm |
| DAPO dynamic sampling | Qwen3 0.6B+1.7B, L4–5 | 12 | −0.018 [−0.027, −0.008] | secondary |
| Critic filter, target | Qwen3, three sizes, L3–5 | 18 | +0.016 [+0.005, +0.027] | secondary |
| Critic filter, target | Qwen3-4B, L3–5 | 12 | +0.0001 [−0.014, +0.014] | TOST |
| Critic filter, target | Qwen3-0.6B, L3–5 | 18 | +0.024 [+0.013, +0.035] | estimate |
| Critic filter, target | Qwen3-0.6B, L3–5 | 24 | −0.006 [−0.015, +0.003] | Holm , TOST |
| minus target 1/2 | Qwen3-0.6B, L3–5 | 24 | −0.028 [−0.036, −0.020] | Holm |
| Critic filter, target | Qwen2.5-1.5B, L3–5 | 24 | −0.001 [−0.010, +0.007] | Holm , TOST |
| minus target 1/2 | Qwen2.5-1.5B, L3–5 | 24 | −0.010 [−0.016, −0.004] | Holm |
| Learning rate , or decaying from it to 0 at , minus , same configuration and seeds | ||||
| Outcome-only GRPO | Qwen3-0.6B, L3–5 | 24 | +0.027 [+0.014, +0.040] | secondary |
| Outcome-only GRPO, strict reward | Qwen3-0.6B, L3–5 | 24 | +0.019 [+0.005, +0.033] | Holm |
| Outcome-only GRPO | Qwen2.5-1.5B, L3–5 | 24 | −0.032 [−0.039, −0.024] | secondary |
| Critic filter | Qwen2.5-1.5B, L3–5 | 24 | −0.029 [−0.038, −0.020] | secondary |
| Outcome-only GRPO, decaying rate | Qwen2.5-1.5B, L3–5 | 24 | +0.004 [−0.005, +0.012] | secondary |
| Critic filter, decaying rate | Qwen2.5-1.5B, L3–5 | 24 | −0.003 [−0.007, +0.002] | exploratory |
| Reward verification | ||||
| Strict verifier as training reward | Qwen3-4B, L4–5 | 3 | +0.008 [−0.020, +0.036] | non-inferior |
| Strict verifier as training reward | Qwen3-4B, level 5 | 12 | +0.008 [−0.019, +0.036] | Holm , NI |
| Strict verifier as training reward | Qwen3-0.6B, L3–5, | 12 | +0.023 [+0.009, +0.036] | Holm , NI |
| with the critic filter | Qwen3-0.6B, L3–5, | 12 | +0.003 [−0.012, +0.019] | secondary, NI |
Table 4. Table 4. Every matched-compute contrast, part 2: the critic filter on the other models, controls, other selection rules, learning rate, and verifier. As in Table 3; both Llama models train on the strict verifier. Random-choice rows give the difference from outcome-only GRPO over all seeds, and their test column gives the preregistered primary contrast against the critic filter, one per study with its number of seeds; random choice trains on 8 of the filter’s 24 candidates drawn at random. History and level filters predict success from a prompt’s last group reward or MATH level; is the filter’s target. Learning-rate rows give a configuration at , or at a rate decaying linearly with generated tokens from to 0 at , minus the same configuration and seeds at . Verifier rows give a configuration trained on the strict verifier, which requires a boxed answer, minus the same configuration and seed trained on the default verifier, which falls back to the last number; both are evaluated with the default verifier, and “NI” marks a one-sided non-inferiority test at −0.02. All runs in both parts used H200 GPUs (Section C), except those of the 768-token rows at outcome-only GRPO’s training time, which used A100 GPUs (Section F).
What separates the settings is how hard the candidates are. In the five settings from which we formed this account, post hoc, the critic’s mean predicted success over the 24 candidates was 0.42 and 0.40 where the filter helped and 0.54 to 0.70 where it did not, with no overlap between seeds (Table 8). When the candidates’ mean prediction is below 1/2, keeping those nearest 1/2 moves training toward easier prompts and away from all-wrong groups; when it is above 1/2, the filter moves training toward harder prompts. The training reward shows the move. Wherever the filter helped, its mean training reward exceeded outcome-only GRPO’s (0.43–0.47 against 0.30–0.39), and in the six settings without a gain whose candidates averaged above 0.53, it fell below. Outcome-only GRPO’s own training reward, which needs no critic, separates the first eleven settings in the same way, whereas the base model’s accuracy on its first training prompts, known before training, does not.
Six preregistered studies then tested the account (Table 1).
Two moved the candidates across 1/2 by changing the pool (S19, S21), two changed the model family (S24, S27), and two kept model and pool fixed and reversed only the direction of the move, by aiming the filter at 0.25 (S30, S31). Five of the six predictions held. The gain predicted on Llama-3.2-1B did not appear at , although the filter led at every evaluation after the first. Across all twelve settings, the filter gained significantly in four of the five settings whose candidates averaged below 1/2 and in none of the seven above. Candidates below 1/2 thus mark where a gain is possible, and on a fixed model and pool the direction of the move decides whether it appears.
Candidate difficulty and a test on Qwen3-1.7B with level 5 (S19)
Averaged over the iterations before in which the critic was in use, the critic’s mean predicted success over the 24 candidates separated, in the five settings from which the account was formed, the settings in which the filter helped from those in which it did not (S27 later found a setting below 1/2 without a significant gain). It is 0.419 (Qwen3-0.6B, 54 seeds, range 0.404–0.430) and 0.397 (Qwen2.5-1.5B, 60 seeds, 0.383–0.409) where the filter helped, against 0.587 (Qwen3-1.7B, 0.578–0.596), 0.544 (Qwen3-4B on level 5, 0.533–0.558), and 0.701 (Qwen3-4B on levels 3–5, 0.688–0.717) where it did not. Below 1/2, the filter moves training toward easier prompts and away from all-wrong groups (36% → 24% on Qwen3-0.6B, 39% → 27% on Qwen2.5-1.5B). Above 1/2, it moves training toward harder prompts and mainly removes all-correct groups. The level-5 study on 4B did not test this account, because 4B’s candidates there were still above 1/2 on average. S19, preregistered after this observation, trains Qwen3-1.7B, the model without a gain on levels 3–5, on the level-5 pool and test set (24 seeds). Its precondition is a candidate mean below 1/2, and its primary prediction is a gain.
The prediction held: the candidates’ mean predicted success was 0.389 (seed range 0.377–0.407), and the chosen prompts’ 0.457. The single primary contrast, filter minus outcome-only at , was +0.0112 [0.0020, 0.0205] (; higher in 17 of 24; crossed bootstrap [−0.0003, +0.024]), its 90% interval, [0.004, 0.019], also lies inside ±0.02 (TOST ), so the gain is significant but smaller than two points. Strict-verifier accuracy moved the same way (+0.010 [0.001, 0.019], , secondary). As on Qwen3-0.6B, the filter removed mainly all-wrong groups (45% → 30%; all-correct 17% → 19%) and raised mixed groups by 12.7 points [12.1, 13.3], higher in every seed. The lead rose to +0.018 at and stayed between +0.007 and +0.014 from to . Training started from accuracy 0.294 and reached 0.409 (outcome-only) and 0.420 (filter) at . The account thus predicted a gain in a model that had none on levels 3–5, a confirmatory test of an account formed post hoc on five settings.
A test in the other direction: Qwen3-0.6B with levels 1–2 (S21)
S21, preregistered after S19 and S20, tests the account where it predicts no gain, at the model where the filter helped most. Both Qwen3-0.6B configurations trained on the 1,367 MATH train problems of levels 1–2 with numeric answers (after removing 26 that overlap an evaluation set) and were evaluated on 200 level 1–2 test problems, with the same budget as on levels 3–5 () and seeds 0–23. The precondition was a candidate mean above 1/2. The single primary contrast compares the filter’s gain on levels 1–2 with its gain on levels 3–5 (seeds 0–17 of S8 and S12, Welch , unpaired because the two sides use different runs, pools, and test sets); the account predicts it to be negative.
The precondition held: the candidates’ mean predicted success was 0.791 (seed range 0.771–0.803), and the chosen prompts’ 0.656. Outcome-only GRPO rose from 0.706 to 0.809 at , below the preregistered headroom bound of 0.9. The primary contrast was −0.030 [−0.039, −0.020] (Welch , 32 df): the level 1–2 gain, −0.005, against +0.024 on levels 3–5. Secondary results:
Filter minus outcome-only on levels 1–2: −0.005 [−0.011, +0.000] (; lower in 14 of 24; crossed bootstrap [−0.014, +0.004]), equivalent to zero within ±0.02 (TOST ). Strict-verifier accuracy: −0.008 [−0.021, +0.006].
The filter raised mixed groups by 19.1 points [18.4, 19.8], higher in every seed, by removing mainly all-correct groups (57% → 33%; all-wrong 4% → 9%). Its mean training reward was 0.67 against outcome-only’s 0.81.
Exploratory: the filter trailed from the first evaluation; over the first half of the budget its mean difference was −0.010 [−0.014, −0.006] (higher in 3 of 24 seeds), and at it had not reached outcome-only GRPO’s accuracy (Table 11). Pass@8 differed by +0.005 [−0.003, +0.012]. The critic’s predictions on the trained prompts were 1.2 points below their realized success (Table 9), which does not change the side of 1/2 on which the candidates fall. The account’s prediction thus held in its second direction: the model that gains on levels 3–5 does not gain where its candidates average above 1/2. With S19, this gives two confirmatory tests, one in each direction, of an account formed post hoc.
Outside the Qwen family: Llama-3.2-3B (S24)
S24 trains Llama-3.2-3B-Instruct on the level 3–5 pool and evaluates it on the level 3–5 test set (seeds 0–23). A calibration stage, which ran before the preregistration and computed no contrast, applied the S1 rules: 33% of the base model’s samples were truncated at 380 tokens, so the limit is 768 tokens, and the budget rule gives . Because the default verifier’s fallback had let outcome-only GRPO drift on three of the four Qwen models, both configurations train with the strict verifier, as S17 and S18 support; held-out evaluation is unchanged. The preregistered prediction depends on the candidates’ mean predicted success. Below 1/2, the account predicts a gain, tested by the single primary contrast, filter minus outcome-only at . Above 1/2, it predicts none, and a gain would contradict it.
The candidates averaged 0.678 (seed range 0.638–0.718), and the chosen prompts 0.585, so the account predicts no gain. The model did much better on the training pool than on the test set. Outcome-only GRPO’s training reward was 0.378 in iterations 0–2 and averaged 0.698 (0.663–0.722) over the iterations before , while its held-out accuracy rose from 0.288 (strict 0.238) to 0.434. On the four Qwen models with the same pool, the training reward of iterations 0–2 was 0.01 to 0.11 below the initial held-out accuracy. The primary contrast was −0.0037 [−0.0098, +0.0025] (; higher in 10 of 24 seeds; crossed bootstrap [−0.013, +0.006]), equivalent to zero within ±0.02 (TOST ). Secondary results:
Strict-verifier accuracy: −0.003 [−0.009, +0.003]. Mean over the first half of the budget: +0.004 [−0.001, +0.008] ().
The filter raised mixed groups by 9.1 points [8.3, 9.8], higher in every seed, mainly by removing all-correct groups (35% → 22%; all-wrong 8% → 12%). Its mean training reward was 0.591.
Exploratory: the difference peaked at +0.007 at and was negative from on. Pass@8 was 0.011 [0.0003, 0.023] lower (Table 10). The critic’s predictions on the trained prompts were 0.9 points below their realized success (Table 9). The account predicted no gain on this side of 1/2, and none was found; S27 tests whether the filter can help outside the Qwen family.
The gain prediction outside the Qwen family: Llama-3.2-1B (S27)
S27 trains Llama-3.2-1B-Instruct, the smaller model of the same family, with both configurations on the strict verifier as in S24 (seeds 0–23). Two stages ran before the preregistration and computed no contrast. A calibration stage applied the S1 rules: 37.5% of the base model’s samples were truncated at 380 tokens, so the limit is 768 tokens, and the budget rule gives . At 768 tokens the base model’s accuracy on the level 3–5 test set was 0.147 (strict 0.075). A pilot then chose the training pool by a rule fixed with the calibration. Outcome-only GRPO was trained to with a seed outside the study, on the level 3–5 pool and on the level-5 pool. The study would use the level 3–5 pool if that pilot’s mean training reward over the iterations before was below 0.45, and otherwise the level-5 pool under the same condition. The level 3–5 pilot’s training reward was 0.387, so the study uses that pool and its test set. The precondition is a candidate mean below 1/2; the single primary contrast, filter minus outcome-only at , is then predicted to be positive, and a contrast equivalent to zero or negative counts against the account.
The candidates averaged 0.420 (seed range 0.395–0.443), and the chosen prompts 0.467, so the precondition held. Outcome-only GRPO’s training reward, 0.394 (0.364–0.423), was also below 1/2. The primary contrast was +0.0044 [−0.0008, +0.0096] (; higher in 16 of 24 seeds; crossed bootstrap [−0.0045, +0.0136]), equivalent to zero within ±0.02 (TOST ). At the per-pair spread observed (0.012), the study had power 0.86 for a gain of +0.008, the size found on Qwen2.5-1.5B. Secondary results:
Strict-verifier accuracy: +0.003 [−0.003, +0.009]. Mean over the first half of the budget: +0.005 [0.003, 0.008] (; higher in 18 of 24), as predicted.
The filter raised mixed groups by 8.8 points [8.2, 9.5], higher in every seed, mainly by removing all-wrong groups (28% → 18%; all-correct 10% → 11%). Its mean training reward was 0.464.
Exploratory: the difference lay between +0.001 and +0.009 at every evaluation after the first. Averaged over the second half of the budget it was +0.007 [0.002, 0.011] (), and over all sixteen evaluations +0.006 [0.003, 0.009]. Outcome-only GRPO rose from 0.148 to 0.261 at , by only 0.009 in the second half, so the filter reached its accuracy at after 0.59 of the budget (Table 11). Pass@8 differed by +0.003 [−0.011, +0.016] (Table 10). The critic’s predictions on the trained prompts were 0.08 points below their realized success (Table 9). The account’s prediction of a gain at thus failed outside the Qwen family. The filter’s mean lead was small but positive at every evaluation, significant only when averaged over half the budget, and the interval at includes the Qwen2.5-1.5B gain.
The filter aimed at harder prompts on a fixed pool (S30)
The tests of the account above changed the pool (S19, S21) or the model (S24, S27). S30 keeps the model and pool where the filter gains most, Qwen3-0.6B on levels 3–5, and changes only the filter’s target, from 1/2 to 0.25, so that the filter trains on prompts harder than its candidates while still avoiding those predicted near 0 (seeds 0–23, , default reward, as the comparators). The comparators are the filter at 1/2 and outcome-only GRPO on the same seeds (S8, S12, and S16 read at ). Two primary contrasts, Holm-adjusted, were preregistered: T1, the filter at 0.25 minus the filter at 1/2, predicted negative, and T2, the filter at 0.25 minus outcome-only GRPO, predicted to show no gain. A manipulation check required the chosen prompts’ predicted success to lie at least 0.05 below the candidates’ on average and the filter’s training reward to lie below outcome-only GRPO’s.
The manipulation check passed. The chosen prompts’ predicted success was 0.287 against the candidates’ 0.359, between 0.064 and 0.080 lower in every seed (at target 1/2: 0.468 against 0.419). The training reward was 0.287 (0.279–0.292), against 0.386 for outcome-only GRPO and 0.468 for the filter at 1/2, lower in every seed. All-wrong groups rose from 35.9% to 41.5% and all-correct groups fell from 14.1% to 6.8%, so mixed groups rose by only 1.7 points [1.3, 2.2]. Both primary predictions held. T1 was −0.0280 [−0.0359, −0.0201] (Holm ; lower in 22 of 24 seeds; crossed bootstrap [−0.041, −0.015]). T2 was −0.0060 [−0.0154, +0.0033] (Holm ; crossed bootstrap [−0.019, +0.007]), equivalent to zero within ±0.02 (TOST ). Secondary results: strict-verifier accuracy −0.003 [−0.019, +0.012] against outcome-only GRPO; over the first half of the budget, −0.002 [−0.004, +0.001] against outcome-only GRPO, so there was no head start. In strict-verifier accuracy the filter at 0.25 trailed the filter at 1/2 by 0.028 [0.013, 0.043] (exploratory). The filter at 0.25 drifted in 9 of 408 evaluations, in two runs, against 1 of 414 (outcome-only GRPO) and 0 of 414 (filter at 1/2) up to on the same seeds; one run (seed 4) was drifted at . Without it, T1 is −0.028 [−0.036, −0.020] and T2 −0.005 [−0.015, +0.005] (exploratory). Training took 0.54 hours per run on H200 GPUs. On a fixed model and pool, the gain thus depended on the direction of the move, as the account says.
The filter aimed at harder prompts on Qwen2.5-1.5B (S31)
S31 repeats S30 on the other model with a replicated gain, Qwen2.5-1.5B on levels 3–5 (candidates 0.40, gain +0.008). The configuration is the S20 filter with the target changed from 1/2 to 0.25 and nothing else; the comparators are the S20 filter and outcome-only runs of the same seeds (12–35). The primary contrasts, Holm-adjusted, were those of S30: T1, the filter at 0.25 minus the filter at 1/2, predicted negative, and T2, the filter at 0.25 minus outcome-only GRPO, predicted to show no gain. Because the gain under test is 0.008, the equivalence margin was ±0.01, as in S28. The manipulation check was that of S30, and a secondary contrast predicted no head start over outcome-only GRPO.
The manipulation check passed, by a smaller margin than in S30. The chosen prompts’ predicted success was 0.278 against the candidates’ 0.331, 0.053 lower on average and between 0.039 and 0.065 lower per seed (at target 1/2: 0.458 against 0.395). The training reward was 0.282 (0.274–0.289), against 0.367 for outcome-only GRPO and 0.457 for the filter at 1/2, lower in every seed. All-wrong groups rose from 39.1% to 43.2%, but all-correct groups fell from 15.8% to 8.0%, so mixed groups rose by 3.9 points [3.4, 4.4], in every seed. Both primary predictions held. was −0.0097 [−0.0159, −0.0035] (Holm ; lower in 18 of 24 seeds; crossed bootstrap [−0.020, +0.0003]), against a gain of the filter at 1/2 of +0.0083 on these seeds. was −0.0014 [−0.0095, +0.0067] (Holm ; crossed bootstrap [−0.012, +0.009]), equivalent to zero within ±0.01 (TOST ) and within ±0.02 (). The secondary prediction of no head start failed: over the first half of the budget the filter at 0.25 led outcome-only GRPO by 0.0047 [0.0007, 0.0087] (), a quarter of the filter’s lead at 1/2 (0.019), and trailed the filter at 1/2 by 0.014 [0.011, 0.017], in every seed. In strict-verifier accuracy, was +0.010 [−0.027, +0.047] and −0.018 [−0.040, +0.004], both widened by drift. The filter at 0.25 drifted in 5 of 408 evaluations, against 16 (outcome-only GRPO) and 1 (filter at 1/2) of 408 on the same seeds; one of its runs (seed 21) and two outcome-only runs (seeds 30 and 32) were drifted at . Without the filter’s drifted run, is −0.0086 [−0.0146, −0.0025] and +0.0007 [−0.0064, +0.0079]; without all three, −0.0073 [−0.0137, −0.0010] and +0.0024 [−0.0050, +0.0099] (exploratory). Training took 0.49 hours per run on H200 GPUs. The direction of the move thus mattered on a second model; the early lead shows that training on prompts harder than the candidates, while still avoiding those predicted near 0, kept a small part of the filter’s head start on this model, but none of its gain at .
Candidates, calibration, and training reward (exploratory)
Table 8 lists every setting with a critic-filter configuration and an outcome-only configuration on the same seeds, ordered by the candidates’ mean predicted success. The first studies on the level 4–5 pool (S6, S7; 6–8 seeds) were not among the five settings on which the account of Section 4.2 was formed, and S19, S21, S24, and S27 tested it; we add all of them here. Training rewards are means over the iterations started before , under the verifier that trained the configuration (strict for the Llama models, default otherwise). In all five settings below 1/2 the filter raised the training reward; it gained significantly in four, and on Llama-3.2-1B its +0.004 [−0.001, +0.010] was not significant. In the six settings above 0.53 it lowered the training reward (by 0.01 on Qwen3-4B on level 5, by 0.05 to 0.15 in the others) and did not gain; Qwen3-1.7B on levels 4–5 sits at 0.52, with the filter’s training reward 0.01 above outcome-only GRPO’s and an inconclusive estimate. Outcome-only GRPO’s training reward, which needs no critic, orders the Qwen settings as the candidates’ predicted success does. In the first eleven settings it separated the seeds of the settings with and without a gain without overlap; on Llama-3.2-1B it fell in the range of the settings with a gain, so it, too, predicted the gain that S27 did not find. The rewards of the first iteration, before any update, do not separate the settings: Qwen3-4B on level 5 (0.30) falls inside the range of the settings with a gain (0.18–0.35), Llama-3.2-3B (0.39) just above it, and Llama-3.2-1B (0.13) below it. Over a whole budget, outcome-only GRPO’s training reward reflects how quickly the model learns the pool, which its first iteration does not.
Table 9 checks the critic’s predictions against the groups it trained on. Random choice trains on a uniform draw of the candidates, so its realized minus predicted success measures calibration on the candidate distribution: it is within 0.2 points on all three models. On level 1–2 problems and on Llama-3.2-3B, which have no random-choice configuration, the filter’s trained prompts succeeded 1.2 and 0.9 points more often than predicted; there the critic, if anything, underestimated how far above 1/2 they lay. On Llama-3.2-1B they succeeded as often as predicted (+0.08 points [−0.05, +0.21]), so its candidates’ side of 1/2 is not an artifact of calibration. A prompt’s prediction correlates with its group’s success at 0.67 to 0.72, although a group has only eight rollouts. On the filter’s chosen prompts, which lie near 1/2, the correlation is lower because their range is narrower. The threshold of 1/2 in the account is therefore a threshold on realized success, not only on the critic’s scale.
Does the gain last, and can retuning match it?
Where we followed the gain over time, it was a head start (Figure 5). On Qwen3-0.6B, it peaked at 5/8 and had mostly faded by 2 (S16, S17). On Qwen2.5-1.5B (60 seeds), it peaked at +0.026 [0.022, 0.030] at 3/16 and fell to +0.008 at . On Qwen3-4B with level 5, it peaked at +0.042 [0.026, 0.057] at /8 (S14). A head start is a saving in compute: where the filter gains, it reaches outcome-only GRPO’s accuracy at with 22 to 23% fewer generated tokens and 13 to 17% less training time (Table 11). The Qwen3-0.6B gain also holds at pass@8 (+0.009 [0.002, 0.017], ), which cuts the share of test problems that no sample solves from 27.1% to 26.2%, so the filter does more than sharpen sampling on problems the policy already solves (Table 10).
Because the gain is a head start, a faster baseline is its natural rival, and a doubled or decaying learning rate tests it directly. On Qwen3-0.6B, doubling outcome-only GRPO’s
Table 8. Where the critic filter helps (exploratory). Candidates: the critic’s mean predicted success of the 24 candidates per iteration (seed mean, with the range over seeds). Training reward: mean group reward over the iterations started before (strict verifier for the Llama models, whose configurations train on it). Iteration 0: training reward of the first iteration, before any update, the same in both configurations (the base model on its first 8 training prompts per seed). Start: outcome-only GRPO’s accuracy on the test set before training. Last column: filter minus outcome-only accuracy at (95% paired- CI).
Table 9. Calibration of the prompt critic (exploratory), on the prompts trained in the iterations before in which the critic was in use: mean predicted and realized group success, their difference per seed in points (95% CI), and the correlation over groups.
learning rate buys as much as the filter, with and without drift (S23, S29). On Qwen2.5-1.5B, a doubled constant rate lowers both configurations and the filter’s gain holds (S28), while a rate decaying from to 0 at brings outcome-only GRPO within half a point of the filter and removes the filter’s own gain at (S32). The filter’s gain is thus about as large as what tuning the baseline’s learning rate buys, which is why a claimed gain should be measured against a tuned baseline.
Twice the budget (S16)
On Qwen3-0.6B, the filter’s lead grew over the budget (+0.011 at , +0.020 at , and +0.024 at over seeds 0–17), and outcome-only accuracy was still rising at . S16 trained both configurations to on fresh seeds 18–29. Because the learning rate is constant, the first tokens of each run are trained exactly as in the configurations with budget . The primary contrast at was +0.021 [−0.001, +0.042] (; crossed [−0.001, +0.044]; higher in 8 of 12). At it was +0.015 [0.003, 0.026] (), and the change between and was +0.006 [−0.009, +0.021]. From to , outcome-only accuracy rose from 0.444 to 0.487 and the filter’s from 0.459 to 0.507. Strict-verifier accuracy at differed by +0.032 [−0.065, +0.130], because drifted runs occurred in both configurations. One outcome-only run (seed 25) drifted from and ended with 99% of its samples truncated. Its default-verifier accuracy at was 0.425, against 0.519 for the filter on the same seed. Pooling the evaluations at of all thirty Qwen3-0.6B seeds (0–29), the filter leads by +0.021 [0.014, 0.027], higher in 26 of 30; adding the 24 seeds of S34 (above), it leads by +0.017 [0.012, 0.021] over 54 seeds.
Twice the budget with the strict reward (S17)
Because drift affected S16’s outcome-only configuration more often than its filter configuration, S17 repeated both S16 configurations with the strict verifier as the training reward. Seeds 18–29 pair with the S16 runs (same prompt stream and evaluation draws), and seeds 30–41 are fresh. Holm adjustment covered two primary contrasts:
Filter minus outcome at , both on the strict reward (): +0.0053 [−0.0051, +0.0157] (; crossed bootstrap [−0.007, +0.018]; higher in 16 of 24; TOST ). Seeds 18–29 give +0.001 and seeds 30–41 +0.010.
Outcome-only on the strict minus on the default reward at (, non-inferiority at −0.02): +0.0228 [0.0092, 0.0363] (Holm ; crossed [0.006, 0.040]; higher in 11 of 12).
The preregistered secondary contrasts, without correction for multiple comparisons, were as follows. At , the filter led by +0.019 [0.013, 0.026] (, higher in 22 of 24; crossed [0.008, 0.031]), and the lead shrank by 0.014 [0.004, 0.024] from to (, lower in 17 of 24). Over the 33 evaluations, the lead peaked at +0.021 [0.014, 0.028] at . The filter configuration on the strict reward was non-inferior to the same configuration on the default reward (+0.003 [−0.012, +0.019], ). The filter’s lead under the strict reward minus its lead under the default reward, on seeds 18–29, was −0.020 [−0.043, +0.004] (). Strict-verifier accuracy of outcome-only GRPO at rose from 0.420 to 0.497 (+0.077 [0.001, 0.152], ), with +0.412 and +0.207 on seeds 21 and 25, both of which drifted under the default reward. The filter raised the share of mixed-outcome groups trained on before by 10.2 points [9.6, 10.8], higher in all 24 seeds. In strict-verifier accuracy, the filter led by +0.018 [0.011, 0.024] at and +0.004 [−0.006, +0.014] at . No strict-reward run drifted (0 of 792 evaluations per configuration), and all 49,006 training iterations of the two configurations used the strict reward. Truncated samples at were 28% (filter) and 23% (outcome-only) over the 24 seeds. On seeds 18–29, outcome-only GRPO’s fell from 36% under the default reward to 24% under the strict reward, and both configurations had 36% under the default reward. Training took 1.17 hours per filter run and 1.07 per outcome-only run on H200 GPUs. The gain is therefore not shown to persist to ; Table 3 and Figure 7 give S16 and S17 side by side.
Twice the learning rate on Qwen3-0.6B (S23)
S23 repeats the Qwen3-0.6B level 3–5 configurations, the critic filter and outcome-only GRPO on the default reward, at learning rate on seeds 0–23. Each configuration differs from its counterpart only in the learning rate. The runs of the same seeds (S8, S12, and S16 read at ) share the prompt streams and evaluation draws. Primary contrasts (Holm over two; ):
Filter minus outcome-only, both at : +0.009 [−0.006, +0.024], Holm , higher in 17 of 24 seeds; crossed bootstrap [−0.0076, +0.0259]; TOST .
Outcome-only at minus the filter at : +0.005 [−0.007, +0.018], Holm , higher in 14; crossed [−0.0105, +0.0206]; equivalent within ±0.02 (TOST ).
Secondary contrasts. Doubling the learning rate raised outcome-only GRPO’s accuracy by 0.027 [0.014, 0.040] () and the filter’s by 0.014 [0.003, 0.025] (). The filter’s gain changed by −0.013 [−0.029, +0.003] (). Over the first half of the budget, the filter led by 0.014 [0.006, 0.023] (). It raised mixed groups by 9.5 points [8.7, 10.3], higher in every seed. In strict-verifier accuracy, the filter minus outcome-only at was +0.043 [−0.036, +0.121].
Stability (descriptive and exploratory). Up to , 49 of 408 outcome-only evaluations and 36 of 408 filter evaluations at drifted, in 14 and 8 runs; 5 and 3 runs were drifted at . At , the same seeds had one drifted evaluation up to (outcome-only) and no run drifted at . Truncated samples at were 43% and 42%, against 36% and 35% at . Strict-verifier accuracy at was 0.366 (outcome-only) and 0.409 (filter) at , against 0.410 and 0.435 at . In strict-verifier accuracy, outcome-only GRPO at trails the filter at by 0.069 [0.005, 0.132] (). On the 17 seeds where none of the four runs was drifted at , outcome-only GRPO at minus the filter at is +0.003 [−0.010, +0.016] in default and −0.000 [−0.020, +0.019] in strict-verifier accuracy. On the same seeds, the filter’s gain at is +0.011 [−0.004, +0.027], and doubling the learning rate raised outcome-only GRPO’s strict-verifier accuracy by 0.026 [0.007, 0.045]. Training took 0.56 hours per filter run and 0.51 per outcome-only run on H200 GPUs. Doubling the learning rate thus buys as much as the filter on this model.
Twice the learning rate without drift (S29)
S23’s match rested on default-verifier accuracy, under which most of its faster outcome-only runs drifted, and its drift-free comparison rested on a subset of seeds chosen after the fact. S29 trains outcome-only GRPO at with the strict verifier as the reward, to , on the seeds of the S17 strict-reward runs (18–41), and compares it with both S17 configurations at read at . Because the learning rate is constant, a run to read at is equivalent to a run to . Its two primary contrasts, Holm-adjusted, are M1, outcome-only GRPO at minus the filter at , and M2, the same minus outcome-only GRPO at ; both are two-sided. No run drifted (0 of 408 evaluations). was [, ] (Holm ; crossed bootstrap [, ]), equivalent to zero within (TOST ) but not within (). was [0.005, 0.033] (Holm , higher in 18 of 24 seeds; crossed bootstrap [0.001, 0.037]), as large as the filter’s gain on the same seeds ( [0.013, 0.026]). The per-seed spread was larger at the faster rate (0.029 for , against 0.016 for the filter’s gain). Secondary results: in strict-verifier accuracy, was [, ] and [0.003, 0.034]; over the first half of the budget, outcome-only GRPO at led the filter at by 0.015 [0.010, 0.019] (higher in 22 of 24 seeds), so the faster rate gives a larger head start than the filter does. It trained on slightly fewer mixed groups than outcome-only GRPO at (46% against 47%, and 59% for the filter). Truncated samples at were 32%, against 27% (outcome-only, ) and 32% (filter). Training took 0.53 hours per run on H200 GPUs. On Qwen3-0.6B, doubling the learning rate thus buys as much as the filter without drift.
Twice the learning rate on Qwen2.5-1.5B (S28)
S28 repeats S23 on Qwen2.5-1.5B: it runs both S20 configurations at , with nothing else changed, on the S20 seeds (12–35), so that each run shares its prompt stream and evaluation draws with the S20 runs of its seed at . On this model the filter’s gain is 0.008, so the preregistered equivalence margin is . The two primary contrasts, Holm-adjusted and two-sided, are L1, the filter minus outcome-only GRPO at , and L2, outcome-only GRPO at minus the filter at .
L1 was [0.0021, 0.0201] (Holm ; higher in 19 of 24 seeds; crossed bootstrap [, ]), and L2 was [, ] (Holm ; lower in every seed; crossed bootstrap [, ]). Secondary results:
Doubling the rate lowered accuracy at by 0.032 [0.024, 0.039] for outcome-only GRPO and by 0.029 [0.020, 0.038] for the filter, lower in 23 of 24 seeds each. The filter’s gain did not change with the rate: [, ], equivalent to zero within (TOST ).
Strict-verifier accuracy: L1 [, ], L2 [, ]. Over the first half of the budget, the filter led outcome-only GRPO at by 0.014 [0.010, 0.019], and it raised mixed groups by 7.5 points [7.0, 8.1], higher in every seed.
Exploratory: at both configurations rose faster at first. Outcome-only GRPO at led the filter at by 0.033 to 0.039 from to , reached 0.39 by , and then stayed between 0.39 and 0.40, falling behind from on. Even its best evaluation (seed mean 0.40, at ) stayed below outcome-only GRPO at at (0.42), so selecting its best checkpoint would not have closed the gap. Both configurations drifted at , outcome-only GRPO in 11 and the filter in 12 of 408 evaluations, one run each at ; at the same seeds gave 16 and 1. Truncated samples at were 16% and 17%, against 20% and 21% at . The candidates averaged 0.420 at , against 0.395 at . Training took 0.55 hours per filter run and 0.48 per outcome-only run on H200 GPUs. On Qwen2.5-1.5B, the filter’s gain is thus not reached by a faster baseline, and it holds at twice the learning rate.
A decaying learning rate on Qwen2.5-1.5B (S32)
S28’s faster runs rose quickly and then stalled. S32 gives both S28 configurations a learning rate that decays linearly with generated tokens. An iteration that starts after a fraction of the budget steps at , so the rate falls from to 0 at and averages . Nothing else changes. The seeds are S20’s (12–35), so each run shares its prompt stream and evaluation draws with the S20 and S28 runs of its seed. The margin is . The two primary contrasts, Holm-adjusted and two-sided, are V1, outcome-only GRPO with the decaying rate minus the filter at a constant , and V2, the filter minus outcome-only GRPO, both with the decaying rate.
V1 was [, ] (Holm ; lower in 14 and higher in 9 of 24 seeds; crossed bootstrap [, ]). It is not equivalent to zero within (TOST ) but is within (). V2 was [, ] (Holm ; higher in 11 of 24 seeds; crossed bootstrap [, ]), equivalent to zero within (TOST ). Secondary results:
The schedule changed outcome-only GRPO by [, ] against a constant () and by [0.027, 0.044] against a constant (higher in 23 of 24 seeds). The filter’s gain changed by [, ] ().
Over the first half of the budget, outcome-only GRPO with the decaying rate led the filter at by 0.020 [0.017, 0.024], in every seed. Under the same schedule, the filter led outcome-only GRPO by 0.014 [0.010, 0.018] (22 of 24 seeds) and raised mixed groups by 8.3 points [7.8, 8.7], in every seed.
Strict-verifier accuracy: V1 [, ], V2 [, ].
Exploratory results:
The filter with the decaying rate minus the filter at was [, ], equivalent within (TOST ). Seed means at were 0.429 for the filter at , 0.426 for the filter with the decaying rate, 0.424 for outcome-only GRPO with it, and 0.420 for outcome-only GRPO at .
Outcome-only GRPO with the decaying rate led the filter at by up to 0.033, at , and trailed it from on (Figure 6b). Under the schedule, the filter’s lead over outcome-only GRPO peaked at 0.024, at and , and stayed between 0.000 and 0.006 from on.
Drift: outcome-only GRPO drifted in 16 and the filter in 5 of 408 evaluations, with one run of each drifted at . Without those two seeds, V1 is and V2 is (TOST ).
Truncated samples at : 20% for each configuration. Candidates: 0.425 on average, against 0.395 at . Training time on H200 GPUs: 0.54 hours per filter run and 0.47 per outcome-only run.
Because V1 is neither significant nor equivalent to zero within , it remains unresolved whether this baseline reaches the filter at a constant rate; and because V2 is equivalent to zero, the schedule removes the filter’s gain at . The margin is wider than the gain at , however, and the V2 interval does not exclude a gain of that size.
Pass@ and test levels (exploratory)
Each evaluation stores, per test problem, the number of correct samples out of . Table 10 gives the unbiased estimate of pass@, averaged over problems, as paired differences at . Pass@8 equals one minus the share of problems that no sample solves. On Qwen3-0.6B (54 seeds), the filter’s gain over outcome-only holds at every level of the test set. It is +0.015 [0.008, 0.022] on the 79 level-3 problems, +0.024 [0.016, 0.032] on the 62 level-4 problems, and +0.011 [0.006, 0.017] on the 59 level-5 problems. With both configurations on the strict reward, the gain at again holds at pass@8, and by it is gone at every . On Qwen2.5-1.5B, the gain shrinks as grows, and pass@8 does not differ. On Qwen3-0.6B on levels 1–2, where the filter trains on harder prompts, its small pass@1 deficit becomes a small pass@8 surplus; neither is significant. On Llama-3.2-3B, where it also trains on harder prompts, the deficit instead grows with , and at pass@8 its interval just excludes zero. On Llama-3.2-1B, as on Qwen2.5-1.5B, the small pass@1 lead shrinks at larger .
Tokens to reach outcome-only accuracy (exploratory)
To express the head start in tokens, we take outcome-only GRPO’s mean accuracy over seeds at as the target and interpolate the filter’s mean curve over the same seeds (evaluations every ) to its first crossing of the target. Table 11 gives that crossing as a fraction of the budget, with 95% intervals from 10,000 resamples of seeds that keep each seed’s two configurations together. A resample whose curve never reaches the target counts as not reached. In the three settings of Table 11 with a gain, the filter reaches outcome-only GRPO’s accuracy at with 22 to 23% fewer generated tokens. The same crossing on the training-time axis, which counts the critic’s prefill and refit, saves 13 to 17% of outcome-only GRPO’s training time. At under the strict reward, the saving on Qwen3-0.6B is 8% and its interval includes no saving. The measure depends on the slope of outcome-only GRPO’s curve near , not only on the lead: on Llama-3.2-1B, whose outcome-only curve had nearly flattened, a lead that was not significant at gives the largest saving in the table, 41% of the tokens.
Reward verification and drift
When no answer is boxed, the default verifier falls back to the last number in a completion, so it also credits unboxed answers. Some runs trained on it drift away from boxing their answers; we count an evaluation as drifted when default-verifier accuracy exceeds strict-verifier accuracy by more than 0.1. Table 12 counts drift by study and configuration. Drift appeared mainly in long or fast training: on Qwen3-4B with level 5, on Qwen3-0.6B over twice the budget or at twice the learning rate, and on Qwen2.5-1.5B in the replication and under both faster learning rates. It affected outcome-only GRPO more often than the filter, which may have inflated the filter’s lead at in S16. Elsewhere it was rare. The history filter of S22 drifted in 1 of 612 evaluations; on the Qwen2.5-1.5B level 4–5 pool, no Monte-Carlo, outcome-only, or baseline-critic run drifted (S25, S26), while the temporal-difference critic drifted at 5 of 408 evaluations, and 2 of its 24 runs ended drifted. On Qwen3-1.7B, drift appeared only under a length-reward term (5 of 3,717 evaluations over all finished runs), and under that term two of three Qwen3-4B runs stopped boxing their answers, with strict-verifier accuracy of 0.165 and 0.077 while default-verifier accuracy stayed high. One filter run and one outcome-only run of the S11 Qwen3-4B replication also ended drifted.
| Study | Model, pool, horizon | outcome-only | critic filter | random choice |
| S14 | Qwen3-4B, level 5, | 16/204 (2/12) | 1/204 (0/12) | – |
| S18 | same, strict training reward | 0/204 (0/12) | – | – |
| S16 | Qwen3-0.6B, L3–5, | 33/396 (2/12) | 4/396 (1/12) | – |
| S17 | same, strict training reward | 0/792 (0/24) | 0/792 (0/24) | – |
| S23 | Qwen3-0.6B, L3–5, , learning rate | 49/408 (5/24) | 36/408 (3/24) | – |
| S34 | Qwen3-0.6B, L3–5, , fresh seeds | 5/408 (1/24) | 1/408 (1/24) | – |
| S13 | Qwen3-1.7B, L3–5, | 0/306 | 0/306 | 0/306 |
| S19 | Qwen3-1.7B, level 5, | 0/408 | 0/408 | – |
| S15 | Qwen2.5-1.5B, L3–5, | 0/204 | 0/204 | 0/204 |
| S20 | same, fresh seeds | 16/408 (2/24) | 1/408 (0/24) | 0/408 (0/24) |
| S33 | same, further fresh seeds | 0/408 (0/24) | 8/408 (1/24) | – |
| S21 | Qwen3-0.6B, levels 1–2, | 1/408 (0/24) | 2/408 (1/24) | – |
| S24 | Llama-3.2-3B, L3–5, , strict training reward | 0/408 | 0/408 | – |
| S27 | Llama-3.2-1B, L3–5, , strict training reward | 0/408 | 1/408 (0/24) | – |
| S28 | Qwen2.5-1.5B, L3–5, , learning rate | 11/408 (1/24) | 12/408 (1/24) | – |
| S29 | Qwen3-0.6B, L3–5, , , strict training reward | 0/408 (0/24) | – | – |
| S30 | Qwen3-0.6B, L3–5, , filter target 0.25 | – | 9/408 (1/24) | – |
| S31 | Qwen2.5-1.5B, L3–5, , filter target 0.25 | – | 5/408 (1/24) | – |
| S32 | Qwen2.5-1.5B, L3–5, , learning rate decaying from | 16/408 (1/24) | 5/408 (1/24) | – |
Table 12. Drifted evaluations by configuration (default minus strict-verifier accuracy ), with the runs drifted at the last evaluation in parentheses.
Training on the strict verifier removes drift without lowering accuracy. In the studies that retrained drifting configurations on it (S17, S18), no run drifted (0 of 204 evaluations on Qwen3-4B and 0 of 1,584 on Qwen3-0.6B), and neither did outcome-only GRPO at twice the learning rate (S29, 0 of n.r.: not reached within the run.
408). On the two Llama models, which trained on the strict reward throughout, 1 of 1,632 evaluations drifted. Default-verifier accuracy was non-inferior in both preregistered tests, and strict-verifier accuracy rose, mostly on the seeds whose default-reward runs had drifted. We therefore report strict-verifier accuracy alongside default-verifier accuracy and recommend the strict verifier as the training reward. Held-out evaluation still used the default verifier, with strict-verifier accuracy logged. In every one of the 8,384 training iterations, the logged training reward equals the strict reward. The preregistered primary contrast, default-verifier accuracy at under the strict reward minus under the default reward, was [, ] (non-inferiority at , ; crossed bootstrap [, ]; higher in 6 of 12). No strict-reward run drifted (0 of 204 evaluations, against 16 of 204 under the default reward). Strict-verifier accuracy at was 0.631 against 0.520 ( [, ], , secondary). The difference comes from the two seeds whose default-reward runs had drifted ( and ); on the other ten seeds it lies between and . Truncated samples at fell from 17% to 13%, and training time per run was 2.17 against 2.14 hours.
Strict verifier as the training reward on Qwen3-4B with level 5 (S18)
A first check, in S9, trained three Qwen3-4B outcome-only runs on levels 4–5 with the strict verifier as the reward: against the default-reward runs of the same seeds, accuracy changed by [, ] (non-inferior at ), and none of their 51 evaluations drifted. S18 repeated the S14 outcome-only configuration with the strict verifier as the training reward, on the same twelve seeds, so that each run shares its prompt stream and evaluation draws with the S14 run of its seed.
Summary
Matching compute reverses the ranking that the per-update gradient test gives: the densest reward ranks first per update and last per token, and a learned critic, which needs no continuations, matches outcome-only GRPO within two points.
The same protocol confirms a real gain where one exists.
Choosing prompts whose predicted success is nearest improves accuracy on two models, replicates on fresh seeds and on test problems that no study had seen, comes from the choice of prompts, and follows a rule that predicted the outcome of five of six preregistered tests. The protocol also measures that gain precisely: it is a head start worth about a fifth of the budget, comparable to what a retuned learning rate buys. A claim that a dense reward or a prompt-selection rule helps should therefore state the budget it matches, the verifier that defines its reward, how its baseline was tuned, and the models on which it replicated.
Training-Time Accounting
Generated tokens are not the only cost of an intervention. An inference engine generates continuations more cheaply per token than a training step processes rollouts, and a critic adds forward passes and fits that generate no tokens at all. We therefore also measure training time, stage by stage, and compare configurations at equal training time.
Every run logs, for each iteration, the wall time of prompt selection (including the prompt critic’s prefill pass), generation of rollouts, credit (including Monte-Carlo continuations and the learned critic’s features and fits), the optimizer update, and the copy of the updated weights into the generation engine. Training time is the sum of these over the iterations before an evaluation; evaluation, logging, and checkpointing are timed separately and excluded. Each run uses one GPU (an H200, for every run in this section) on nodes shared with other jobs. We compare configurations only within a study or across studies run on the same GPU type, paired by seed.
Equal-time contrasts (exploratory reanalyses of preregistered runs). For each seed, is the shorter of the two runs’ training times up to their evaluation at . Each run’s accuracy is interpolated linearly in training time at , so the configuration that reaches first is read at , and the other is read between two of its evaluations. These contrasts stop when the faster configuration reaches its budget. A Monte-Carlo reward’s runs end before outcome-only GRPO has trained as long as it needs to reach , so Section F gives every configuration that much time in a preregistered study. Table 13 lists every contrast, Table 14 gives the time per stage, and Figure 10 shows the training curves of the Monte-Carlo rewards and critics.
Figure 10. At 380 tokens, the Monte-Carlo reward trails outcome-only GRPO at its own training time. Exploratory reanalysis of preregistered runs on H200 GPUs: accuracy against training hours for Qwen2.5-1.5B on MATH L4–5 (seeds 5–16, the same for every configuration), each run up to its token budget. The Monte-Carlo rewards reach the budget sooner because their continuations take less time per token than training does; the arrow marks the reward’s deficit at its own training time (per-seed mean). Figure 3a and Section F run every configuration past outcome-only GRPO’s training time, at 768 tokens.
| Configuration | Select | Generate | Credit | Update | Total |
| Outcome-only (L4–5) | 0 | 604 | 0 | 940 | 1,552 |
| Monte-Carlo, | 0 | 53 | 357 | 80 | 491 |
| Monte-Carlo, | 0 | 97 | 331 | 147 | 576 |
| Monte-Carlo, | 0 | 168 | 292 | 261 | 723 |
| Critic, TD form | 0 | 607 | 366 | 948 | 1,929 |
| Critic, baseline form | 0 | 601 | 362 | 939 | 1,910 |
| Outcome-only (L3–5) | 0 | 627 | 0 | 988 | 1,624 |
| Critic filter | 186 | 642 | 0 | 1,019 | 1,857 |
| History prompt filter | 1 | 627 | 0 | 990 | 1,627 |
Table 14. Where the training time goes. Mean seconds per run in each stage before the evaluation at , on H200 GPUs; Qwen2.5-1.5B, seeds of Table 13.
Totals include the weight sync (1 to 9 seconds per run). The Monte-Carlo rewards spend most of their time on continuations (the credit stage), which the generation engine samples in large batches, whereas outcome-only GRPO spends most of its time on its many updates. A continuation token therefore costs less time than a rollout token that is also trained on, which is why the token budget charges the Monte-Carlo rewards more heavily than the time budget does. The learned critic’s features and fits add about a quarter to outcome-only GRPO’s training time, and the prompt critic’s prefill pass adds about a tenth.
Pilot Studies of Sample Reuse and Other Estimators (S3, S4)
Two preregistered pilot studies tested ways to make dense rewards cheaper, as well as alternative estimators. They used Qwen2.5-1.5B with the protocol of S1 (MATH L4–5 pool, 380 tokens, , learning rate ) and seeds 0–2 on H200 GPUs, and compared against the outcome-only and Monte-Carlo runs of S2 on the same seeds. With three seeds, these are pilots: their intervals are wide, and only large effects are resolved.
Continuation reuse at the gradient level
Training on the continuations themselves uses every generated token for training. Each continuation becomes a training sequence, with the prompt and the rollout prefix as context, and receives leave-one-out credit among the draws at its cut, in the style of SPO-tree and TreeRL. On 64 fixed batches, measured with the gradient efficiency of § 3.3, this was as efficient as outcome-only GRPO per update with continuations per cut and with , but and per generated token, against for the Monte-Carlo reward without reuse on the same batches. The preregistered condition for training it, a per-token efficiency of at least 0.8, was not met.
Rollout reuse and the VinePPO-style advantage
Accuracy at : outcome-only GRPO 0.408, the Monte-Carlo reward 0.260. Four clipped optimizer steps per batch (two epochs of two minibatches, clip 0.2) gave 0.337 for the reward ( [0.049, 0.105], Holm ) and 0.349 for outcome-only GRPO ( [, ], ); the preregistered interaction, the change in outcome-only GRPO’s lead, was [, ] (Holm ). The VinePPO-style value advantage (Monte-Carlo value differences alone, with the leaveone-out group mean) gave 0.258, equivalent to the reward (−0.002 [−0.024, +0.019], TOST p = .036), and 0.337 with the same reuse (+0.000 [−0.042, +0.043]) against the reused reward. REINFORCE with a leave-one-out baseline instead of the group-normalized advantage changed neither configuration materially (outcome-only 0.418, Monte-Carlo reward 0.259).
Process rewards from a frozen PRM. Process-supervised GRPO (adapted from DeepSeekMath) with the frozen Qwen2.5-Math-PRM-7B and without the verifier reward collapsed: the policy stopped boxing its answers (0.005 boxes per completion), and accuracy fell to 0.043 (−0.365 [−0.389, −0.341]). Adding the outcome reward to the process rewards did not prevent the collapse (0.040). Used instead as a baseline for the outcome reward that depends only on the state, the same PRM was equivalent to outcome-only GRPO (−0.002 [−0.023, +0.018], TOST p = .033).
Learning rate and group size. Outcome-only GRPO at and ended 0.034 [0.004, 0.064] and 0.111 [−0.003, +0.224] below its run at on this pool. Neither critic form gained over outcome-only GRPO at the same higher rate, with four optimizer steps per batch, or with groups of 2 or 4 completions (every contrast within [−0.019, +0.021], none significant). Table 13 gives these pilots on the training-time axis.
Equal Training Time Past the Token Budget (S35)
The equal-time contrasts of Section D stop at a Monte-Carlo reward’s own training time at , which is shorter than outcome-only GRPO’s. This study asks whether a reward catches up when given as much training time as outcome-only GRPO. A preregistered study (fixed before any of its training runs) answers this question with 768-token completions, where 20% of the initial samples are truncated instead of 73%. It trains Qwen2.5-1.5B on the level 4–5 pool, with the strict verifier as every configuration’s training reward, learning rate , 8 prompts × 8 completions per iteration, M generated tokens, an evaluation every , seeds 200–207, and one A100 GPU per run (time axes are compared only within the study). Every configuration trains past , with budgets set from the time ratios of Table 13 so that its training time passes , the training time that outcome-only GRPO needs to reach on the same seed. Outcome-only GRPO trains to , the Monte-Carlo reward to , the reward to , continuation reuse with (every continuation is also trained on, as in Section E) to , and the VinePPO-style value advantage with four clipped optimizer steps per batch to . The primary contrasts T1–T4 compare each configuration with outcome-only GRPO at , with the configuration’s accuracy interpolated linearly in training time (paired over seeds, two-sided, Holm over the four). They were predicted to be negative for both Monte-Carlo rewards, with no predicted direction for the two variants with reuse. The same contrasts at equal generated tokens, and at the shorter of the two runs’ training times at , are secondary.
Primary. Every configuration’s training time passed on every seed, by a factor of at least 1.23, so no budget had to be extended. was 1.29 hours (1.27–1.31 across seeds), at which outcome-only GRPO’s accuracy was 0.467, up from 0.361 at the start. The reward trailed by 0.033 [0.024, 0.043] (T1, Holm , lower in all 8 seeds; crossed seed-by-problem interval [−0.048, −0.019]), as predicted. The reward trailed by 0.011 [−0.005, +0.028] (T2, Holm , lower in 7 seeds), a difference neither shown below zero nor within ±0.02 (TOST ). Its predicted deficit was therefore not confirmed, so our claim about the reward in training time is limited to its own training time. Continuation reuse trailed by 0.052 [0.041, 0.063] (T3, Holm , lower in all 8; crossed [−0.069, −0.036]), and the VinePPO-style advantage with four steps per batch by 0.011 [0.004, 0.017] (T4, Holm , lower in 7), a deficit within ±0.02 (TOST ) whose crossed interval reaches zero ([−0.025, +0.003]). This configuration therefore matches outcome-only GRPO per GPU-hour within ±0.02, while trailing it.
Secondary. At equal generated tokens, all four configurations trailed, by 0.066 (), 0.042 (), 0.046 (continuation reuse), and 0.030 (VinePPO-style), each with and lower in all 8 seeds. At the shorter of the two runs’ training times at , which for every configuration except continuation reuse is the configuration’s own, the and rewards trailed by 0.024 [0.014, 0.034] () and 0.011 [0.001, 0.021] (), and the VinePPO-style variant was even with outcome-only GRPO, at +0.003 [−0.008, +0.015] (TOST ). Compared only when the faster configuration reaches its budget, as in Table 13, the VinePPO-style variant would have tied outcome-only GRPO; it rose fastest at first and then levelled off near 0.45 (Figure 3a, Figure 11a).
Figure 11. Equal training time past the token budget (S35). Qwen2.5-1.5B on MATH L4–5 at 768 tokens, strict training reward, 8 seeds on A100 GPUs. (a) Mean accuracy against tokens generated in training, over each configuration’s whole run; markers show where each configuration’s training time reaches , the time outcome-only GRPO needs to reach . Matching that time gives the reward 3.2B tokens and continuation reuse 0.82B. (b) Each configuration minus outcome-only GRPO at (filled; preregistered primary) and at equal generated tokens (open; secondary), with 95% paired- CIs over seeds; the gray band marks .
Descriptive. Training hours to were 0.40 for the reward, 0.52 for the VinePPO-style variant, 0.63 for the reward, and 1.58 for continuation reuse, against 1.29 for outcome-only GRPO, so in outcome-only GRPO’s training time they could generate 3.2, 2.4, 2.0, and 0.82 tokens. Before , outcome-only GRPO spent 60% of its training time on updates and 39% on generation; the reward spent 66% on its credit stage, which includes generating its continuations; and continuation reuse spent 78% on updates, which include training on the continuations (45 seconds per iteration, against outcome-only GRPO’s 6). On the mean curves, the reward reached outcome-only GRPO’s accuracy at after ; the other three did not within their runs, which ended at (), (VinePPO-style), and (continuation reuse). The share of truncated samples fell from 20% at the start to 9% for outcome-only GRPO at and to between 9% and 16% for the other configurations at . With the strict reward, no evaluation drifted (default minus strict-verifier accuracy above 0.1), and strict-verifier accuracy at was 0.004 to 0.012 below default-verifier accuracy. Peak reserved GPU memory was 68–73 GiB per run; the study took 105 A100 GPU-hours, including a smoke test of every configuration.
Adaptations, not reproductions. The two configurations with reuse adapt published methods to our trainer and recipe. Table 16 lists what each keeps and what it changes; the differences in branch points, samples per branch point, and update rule change both the variance of the credit and its cost. These results therefore describe the adaptations, not the published methods in their own settings.
Equal-Iteration Study: Full Results
This section gives the complete account of the equal-iteration study that Section 3.1 summarizes.
Figure 13 traces the dense reward from the gradient probe to the training result. This experiment tests whether favorable local measurements pick out a useful dense reward in a short GRPO recipe. We fully fine-tune Qwen2.5-1.5B-Instruct ([22, 33]) on MATH problems of levels 4–5 ([13]) for 32 iterations, with 8 completions for each of 8 prompts per iteration. Separate probes, with LoRA and with full-parameter gradients, measure the gradient geometry on fixed batches. Training samples at temperature 1.0 with top- 0.95 and uses Adam with gradient clipping, whereas the reported accuracy uses greedy decoding. Greedy accuracy is therefore a different target from the expected accuracy under sampling used in the local analysis. The probes also compare against a reference gradient of progress estimated from a finite number of Monte-Carlo continuations, denoted , rather than against the exact population gradient (Sections H.1 and H.3).
Figure 13. How a dense reward goes from its estimate to the training result. In the primary full-fine-tuning recipe, the Monte-Carlo reward is estimated, optionally gated, centered, normalized, and combined with outcome-only GRPO before global gradient-norm clipping and one Adam step per iteration. Gating comes before centering, which can put credit back on cleared tokens. The MC-within configuration uses co-equal standardization; the other primary weights and shuffled controls use the max-normalized rule. Training comparisons use the same seed, validation prompts, and scoring procedure. The diagnostic branch measures the fixed-batch score map and agreement with a finite-budget Monte-Carlo reference. The training branch measures same-seed greedy-accuracy differences after Adam updates (Table 19 and section H.1).
| Setting | Value | Setting | Value |
| policy | Qwen2.5-1.5B-Instruct | optimizer | Adam, lr = |
| trained parameters | all (1.5B, full FT) | grad-norm clip | 1.0 |
| iterations | 32 | inner epochs per iteration | 1 (single step) |
| prompts per iteration | 8 | microbatch (completions) | 4 |
| group size | 8 | ratio clipping | none (on-policy) |
| max new tokens | 380 | 0 (full FT) | |
| MC continuations | 4 | MC continuation length | 200 tokens |
| segments per completion | 6 | schedule | constant |
| train data | MATH L4–5 | val data | MATH L3–5, |
| eval cadence | every 2 iterations | gain | final − start val. acc. |
Table 19. End-task training recipe and evaluation target. Settings for the primary full-fine-tuning runs in Table 17, Table 18 and Table 22, with exceptions below and details for each comparison in Section G.5. Each iteration takes one on-policy gradient step, with gradient-norm clipping but no importance ratio or ratio clipping. Training and Monte-Carlo continuations use temperature 1.0 and top- 0.95; reported validation accuracy uses greedy decoding, which measures a different target from the accuracy of the theory. The LoRA probe uses rank-16 query/value adapters, 12 iterations, , , and MATH L3–5 training, with other shared settings listed in Section H.3.
Terms. The Monte-Carlo reward assigns each of six segments of a completion the change in success rate across it, as estimated from continuations. In this study, the credit is centered within each completion (MC-within in the tables), normalized, and combined with the outcome advantage as . There are two normalizations: dividing by the credit’s largest absolute value in the batch (max-normalized, the default), or dividing by its standard deviation (co-equal standardization). The probes split the continuation samples into two halves: half A defines the progress reference , and half B gives a candidate credit from disjoint samples. Gates act on the credit before centering. The sign-opposition gate keeps only the entries whose sign is opposite to ; the rollout-disagreement gate weights each segment by , where is the fraction of continuations that succeed from the segment’s end. The implicit reward is , the log ratio between the current policy and the reference policy.
| Configuration | Equal training time (primary) | Equal tokens | Earlier time at | Hours to | |||||
| [95% CI] | Holm | TOST | Lower | [95% CI] | [95% CI] | ||||
| Outcome-only GRPO | – | – | – | – | – | – | – | 1.29 | |
| Monte-Carlo, | −0.033 [−0.043, −0.024] | .99 | 8/8 | −0.066 [−0.079, −0.052] | −0.024 [−0.034, −0.014] | 0.40 | |||
| Monte-Carlo, | −0.011 [−0.028, +0.005] | .15 | .13 | 7/8 | −0.042 [−0.054, −0.029] | −0.011 [−0.021, −0.001] | 0.63 | ||
| VinePPO-style, 4 steps per batch | −0.011 [−0.017, −0.004] | .013 | .005 | 7/8 | −0.030 [−0.041, −0.019] | +0.003 [−0.008, +0.015] | 0.52 | ||
| Continuation reuse, | −0.052 [−0.063, −0.041] | 1.00 | 8/8 | −0.046 [−0.053, −0.039] | −0.052 [−0.063, −0.041] | 1.58 |
Table 15. Equal training time past the token budget (S35). Qwen2.5-1.5B on MATH L4–5 at 768 tokens, strict training reward, 8 seeds, A100 GPUs. Configuration minus outcome-only GRPO in held-out accuracy: at , outcome-only GRPO’s training time at (preregistered primary, Holm over the four configurations; TOST at ; seeds in which the configuration was lower), at equal generated tokens , and at the earlier of the two runs’ training times at (both secondary), with 95% paired- CIs over seeds. Continuation reuse reaches after outcome-only GRPO, so its earlier time is . Last column: mean training hours before the evaluation at .
What the paired comparisons measure. For each seed, we compare the change in accuracy from start to end of the dense-reward configuration with that of the outcome-only configuration, using the same validation prompts and scoring procedure. Seed-bootstrap intervals describe uncertainty across seeds on these fixed prompts. Probes and training configurations are separate measurements, so Figure 12 sets them side by side rather than run by run. Matching seeds and training iterations also leaves a difference in compute, because the Monte-Carlo reward needs extra continuations from each prefix. Section 3.2 repeats the comparison at matched generated tokens; measuring the resources used by each stage is a separate validation step (Sections H.1 and H.2).
Figure 12. Strong local agreement can coexist with weak training results. Qwen2.5-1.5B, Monte-Carlo reward, equal iterations. (a) The three low-weight configurations satisfy the equivalence criterion (14 paired seeds; thick and thin bars: 90% and 95% seed-bootstrap intervals). (b) High-weight configurations against their own matched outcome controls (95% intervals); the deficit survives only the bootstrap Holm correction, and the starred configuration uses co-equal standardization, the others the max-normalized rule. (c) Four full-parameter probes with disjoint rollouts (mean projected reference cosine 0.84), measured separately from the training runs.
Does agreement with the reference survive disjoint sampling?
The probe computes the summed scores of each completion, the outcome gradient, and the dense-reward gradient, and then projects the gradients onto (Figure 15). outcome gradient’s residual outside is in full-parameter space, a numerical check of the fixed-batch identity. For the MC-within reward, the strongest check uses four full-parameter probes with disjoint rollouts. Their alignments with an independently sampled half-A reference are 0.97, 0.66, 0.89, 0.82, with mean 0.84. The larger values we also report, 0.95 in pooled LoRA probes and 0.96 in full-parameter probes, compare a credit computed from all rollouts with a reference that partly overlaps with it. Disjoint sampling removes the direct reuse of rollouts, but the reference still depends on the estimator’s finite horizon and segmentation.
Does the credit add a useful direction? The Monte-Carlo reward, the implicit reward, and a permutation of the Monte-Carlo reward within each completion all produce updates that lie largely outside : the part outside carries 0.86–0.95 of the update’s norm (Table 20). Only the Monte-Carlo reward consistently aligns with the estimated reference. The pooled alignments of the implicit-reward and shuffled rewards are and . Shuffling keeps the magnitudes of the token credits unchanged, but it can change the norm of the projected gradient and the Adam step. Interpolating between an independent half-B credit and noise also links the alignment of the projected components to the cosine between the full gradient and the reference (Figure 14). Similar descriptive patterns appear across 8 checkpoints from five model families (Table 21). These probes measure raw credits before the optimizer acts on them. The next question is whether this well-aligned reward improves training.
| Dense reward | |||
| implicit reward | |||
| MC-within | |||
| MC-shuffled control |
Table 20. Monte-Carlo reward aligns with the measured progress reference. MeanSD over probe rows pooled across drifted iterations of five runs; rows within a run are correlated. The MC-within row compares the full-rollout credit with its overlapping half-A reference , so its 0.95 projected cosine uses overlapping samples. A separate full-parameter half-B/half-A check averages 0.84 over four probes (Section H.3). The final column measures cosine with the reference gradient. The shuffled control keeps token-credit magnitudes unchanged, but can change projected-gradient norms and Adam steps.
| Model | (residual) | MC | shuf | (MCshuf) [95% CI] | family [95% CI] | seeds/rows |
| Qwen2.5-1.5B | 0.010 | +0.97 | −0.05 | +1.01 [+0.92, +1.12] | 0.98 [0.97, 0.99] | 2/5 |
| Qwen2.5-Math-1.5B | 0.009 | +0.97 | −0.07 | +1.04 [+0.93, +1.18] | 1.00 [1.00, 1.00] | 2/6 |
| SmolLM2-1.7B | 0.010 | +0.95 | −0.04 | +0.99 [+0.94, +1.03] | 1.00 [1.00, 1.00] | 2/5 |
| Phi-3.5-mini-3.8B | 0.011 | +0.93 | −0.02 | +0.95 [+0.89, +1.01] | 0.98 [0.95, 0.99] | 2/6 |
| Qwen2.5-Math-7B | +0.97 | −0.08 | +1.04 [+1.01, +1.09] | 0.99 [0.99, 1.00] | 1/4 | |
| gpt-oss-20B | 0.009 | +0.86 | −0.03 | +0.90 [+0.67, +1.08] | 1.00 [0.99, 1.00] | 1/3 |
| Qwen3.6-27B (dense) | 0.010 | +0.93 | +0.06 | +0.87 [+0.71, +0.98] | 0.89 [0.76, 0.96] | 1/3 |
| Qwen3.6-35B-A3B (MoE) | 0.009 | +0.98 | +0.02 | +0.96 [+0.93, +1.02] | 0.98 [0.96, 0.99] | 1/3 |
Table 21. Descriptive gradient-reference measurements across eight checkpoints. Means and 95% intervals are computed over each model’s pooled probe rows. Rows from the same run are correlated, so these intervals are descriptive. The cosine columns compare raw gradients with the measured reference . is the paired MC-minus-shuffled difference. “Family ” correlates projected alignment and full-gradient reference cosine within a model’s mixing family. Confinement residuals check the projector numerically. Single-seed rows are descriptive. Shuffling keeps token magnitudes unchanged, but can change projected-gradient norms (Section H).
Figure 14. Projected and full-gradient reference agreement rise together. Mixing weight moves the credit from noise toward half-B Monte-Carlo reward. Both measurements use the disjoint half-A reference . The horizontal axis is ; the vertical axis is . Their means rise from 0.02 to 0.83 and from 0.01 to 0.85, respectively. The inset shows the mean and range of the fraction of the update’s norm outside , which falls from 0.96 to 0.87. Each of five weights has 25 probe rows from five runs and five drifted iterations; rows from the same run are correlated. Self-log-ratio () and shuffled () controls are also shown; two of 63 implicit-reward rows lie outside the window at alignments and . Both axes are descriptive gradient measurements against the same reference.
Does the gradient test pick out a better training reward?
The outcome-only baseline improves greedy validation accuracy by [] (Table 17). At , a prespecified expansion to paired seeds supports equivalence with the baseline within : the 90% intervals lie inside this margin. The result holds with Holm correction over the three configurations, under percentile-bootstrap, paired- two one-sided (TOST), and studentized-bootstrap tests. Smaller benefits and harms remain unresolved. At , the paired deficit is [] (, ), which survives Holm correction over the nine Monte-Carlo weight configurations of the primary harm analysis (Section H.2) under both bootstrap and paired- tests. The deficit at survives only the bootstrap test. In this short training recipe, therefore, strong agreement with the reference coexists with equivalence at low weights and harm at high weights.
| Training reward | gain (final−start) | vs. outcome | regime | ||
| GRPO (outcome) | 14 | +0.056 [+0.041, +0.071] | – | – | baseline |
| Monte-Carlo, | 14 | +0.051 [+0.037, +0.067] | −0.005 [−0.026, +0.018] | −0.12 | equivalent |
| Monte-Carlo, | 14 | +0.052 [+0.038, +0.066] | −0.004 [−0.023, +0.016] | −0.09 | equivalent |
| Monte-Carlo, | 14 | +0.053 [+0.041, +0.065] | −0.003 [−0.018, +0.012] | −0.09 | equivalent |
| Monte-Carlo, | 6 | −0.049 [−0.067, −0.029] | −0.105 [−0.144, −0.061] | −1.83 | harm (bootstrap only) |
| Monte-Carlo, | 8 | −0.071 [−0.096, −0.048] | −0.124 [−0.161, −0.087] | −2.15 | harm (all tests) |
| Shuffled Monte-Carlo, | 5 | +0.038 [+0.022, +0.054] | −0.022 [−0.061, +0.025] | −0.41 | inconclusive |
Table 17. Table 17. Low-weight equivalence and high-weight harm in the primary full-fine-tuning recipe. Qwen2.5-1.5B trained on MATH L4–5. Gain is final−start greedy validation accuracy; “vs. outcome” is the paired same-seed difference, with 95% seed-bootstrap CIs and effect size . Only the three low-weight configurations establish equivalence within ±0.03, under bootstrap TOST (), paired- TOST, and studentized-bootstrap checks. Effects smaller than this margin remain unresolved. Harm at survives bootstrap and paired- Holm correction over nine configurations; survives only the bootstrap construction. The shuffled reward keeps token-credit magnitudes unchanged, but can change projected-gradient norms and Adam steps. Its comparison also changes the rule for combining credit with the outcome. The comparison in Table 22 uses the same rule. These controls show differences between the rewards (Sections H.2 and H.5).
How do the choices for combining credit with the outcome affect the comparison?
We next test choices made between estimating the reward and applying the optimizer. The coherent and shuffled rewards behave differently at high weight. Their paired difference-in-differences is [] at , but those configurations use different rules for combining credit with the outcome. At , where the rules match, it is [] over five common seeds (Section H.5). These controls match the magnitudes
of the token credits, but the magnitudes of the projected gradients and the actual Adam steps can still differ. Along two paired training runs, agreement with the reference remains high both in the low-weight configuration and in the degrading high-weight configuration (minimum 0.969 against the strict half-B reference; Figure 16). Agreement thus stays high while accuracy falls, which points to how the credit is combined and how the optimizer acts, rather than to a decay in alignment, as questions for further study.
Figure 16. Reference agreement stays high as greedy accuracy falls. (left) Cosine between the components outside of the full-parameter update and of the Monte-Carlo progress reference, during Qwen2.5-1.5B training. The minimum is 0.992 for the full credit against the half-A reference, and 0.969 for the disjoint half-B credit (long-dashed). Solid and dotted curves show seeds 0 and 1. (right) At , greedy accuracy peaks and then falls, while at it remains stable or improves. Peak-to-final drops of 0.100 and 0.115 stay below the descriptive 0.15 collapse threshold. Seed 0 uses a 100-prompt checkpoint re-evaluation and seed 1 the training loop’s 200-prompt evaluation, so accuracy levels across seeds use different evaluation sets. These two runs give a descriptive comparison of reference cosine and greedy accuracy (Section H.5).
Can gating help? A prespecified expansion of the sign-opposition gate at gives [] against its matched outcome-only control (, one-sided paired- , ; Table 18). The expansion includes eight exploratory seeds, and its interval includes a gain of 2.1 percentage points. The smaller gate comparisons are descriptive. The sign-opposition configuration at gives [], on the same validation prompts and scoring procedure as its outcome-only control; no matched ungated comparison is available. Rollout-disagreement gating improves on the ungated reward with the same rule by [] at (), but the paired- test does not exclude zero (). Masking changes the total amount of credit as well as where it is placed, and centering afterward can put credit back on the masked positions (Section G.5).
| Dense reward | vs. outcome () | vs. ungated, same | ||
| Sign-opposition gate, 150-problem evaluation: | ||||
| sign-opposition, expanded | 1 | 32 | +0.008 [−0.004, +0.021] (+0.23) | – |
| sign-opposition, exploratory | 3 | 3 | +0.036 [−0.013, +0.080] (+0.76) | – |
| Disagreement gate and ungated reward, 200-problem evaluation: | ||||
| rollout-disagreement | 3 | 5 | −0.007 [−0.044, +0.031] (−0.14) | +0.056 [+0.007, +0.094] |
| ungated (ungated) | 3 | 6 | −0.064 [−0.107, +0.002] (−0.82) | – |
Table 18. Table 18. Gating results depend on the matched control. Qwen2.5-1.5B full fine-tuning; paired greedy-accuracy differences with 95% seed-bootstrap intervals. The expansion includes eight pilot seeds. The opposition row uses its matched 150-problem outcome control; no ungated configuration was run on that evaluation set. The disagreement and ungated rows use the same 200-problem evaluation set and combination rule. Masking changes both placement and total credit. Rows with small are descriptive; the disagreement-versus-ungated paired- test gives .
How does the result vary across recipes?
Finally, we check how far the pattern extends. The Qwen2.5-7B sweep has 43 dense runs, with 3–5 seeds per configuration. tion. No dense-reward configuration’s mean gain exceeds that of the outcome-only baseline, which itself gains 0.21, and the instability at high weights varies by seed (Table 23). SmolLM3-3B ( for the outcome-only configuration and the configurations at and 5) and a small NuminaMath study show similar point estimates at lower statistical resolution (Table 24 and Table 26). The single-seed Phi-4 pilots are positive, with paired gains of +0.093 and +0.080 at and 5 (Section H.4). These positive pilots show why each recipe needs its own replication.
| Training reward | gain (final−start) | vs. outcome () | collapse events | |
| GRPO (outcome) | 5 | +0.209 ± 0.048 | – | 0/5 |
| MC-within, | 5 | +0.133 ± 0.090 | −0.076 [−0.167, −0.013] (−0.75) | 1/5 |
| MC-within, | 5 | +0.204 ± 0.071 | −0.005 [−0.091, +0.079] (−0.05) | 0/5 |
| MC-within, | 5 | +0.147 ± 0.091 | −0.063 [−0.159, +0.036] (−0.49) | 1/5 |
| MC-within, | 3 | +0.124 ± 0.047 | −0.100 [−0.180, 0.000] (−1.09) | 0/3 |
| MC-within, | 5 | −0.008 ± 0.220 | −0.217 [−0.411, −0.051] (−0.96) | 3/5 |
| MC-within, | 5 | −0.063 ± 0.178 | −0.272 [−0.385, −0.151] (−1.80) | 3/5 |
| MC-within, | 5 | +0.157 ± 0.053 | −0.052 [−0.107, −0.003] (−0.79) | 0/5 |
| MC-within, | 5 | +0.144 ± 0.069 | −0.065 [−0.143, +0.009] (−0.66) | 0/5 |
| MC-within, | 5 | +0.077 ± 0.173 | −0.132 [−0.288, +0.021] (−0.64) | 2/5 |
Table 23. Accuracy losses in the 7B sweep vary across seeds. Full fine-tuning of Qwen2.5-7B on MATH L4–5. Gain is final−start greedy validation accuracy (mean±SD); “vs. outcome” gives the paired difference, 95% seed-bootstrap CI, and paired effect size . All rows are descriptive (); tests are not adjusted for multiple comparisons. A collapse event is a final accuracy at least 0.15 below that run’s peak; this threshold was selected after inspecting baseline drops. The outcome-only configuration has 0/5 events, compared with 3/5 at each of and 8. These counts describe instability in this recipe. The comparison uses only three shared seeds, so subtracting the displayed configuration means does not give its paired difference.
| Family / reward | gain | 95% CI | vs. outcome | regime | |
| SmolLM3-3B, outcome | 10 | +0.153 | [+0.143, +0.167] | – | baseline |
| 10 | +0.139 | [+0.114, +0.162] | −0.015 | inconclusive | |
| 3 | +0.087 | [−0.093, +0.193] | −0.062 | deficit (desc.) | |
| 10 | +0.061 | [−0.024, +0.123] | −0.092 | deficit (desc.) |
Table 24. SmolLM3 reproduces the primary pattern at lower resolution. Greedy validation-accuracy gain after full fine-tuning on MATH L4–5, with seed-bootstrap 95% CIs and paired same-seed differences from outcome-only. The outcome, , and 5 configurations use a preregistered expansion. At , the paired 90% interval [−0.034, +0.005] falls just outside equivalence at ±0.03. At , the mean paired difference is −0.092 (one-sided , against a prespecified harm threshold of 0.005); two of ten runs meet the descriptive collapse-event criterion. The row is descriptive ().
| Dataset / reward | gain | 95% CI | vs. outcome | regime | |
| NuminaMath, outcome | 3 | +0.098 | [+0.075, +0.115] | – | baseline |
| 3 | +0.112 | [+0.105, +0.115] | +0.013 | gain (desc.) | |
| 3 | +0.068 | [+0.050, +0.100] | −0.030 | deficit (desc.) |
Table 26. A small NuminaMath study points the same way. Qwen2.5-1.5B full fine-tuning on a numeric-answer NuminaMath slice, using the primary recipe and verifier. Gain is final−start greedy validation accuracy (mean and 95% seed-bootstrap CI); “vs. outcome” is the paired same-seed difference. At , the estimate is +0.013 and the estimate is −0.030, with all three high-weight runs below their paired baselines. These observations motivate larger comparisons.
Do the reward source and the rollout budget matter? With co-equal standardization, the Monte-Carlo reward also gives a small difference at low weight and a deficit at high weight (; Table 22). An ablation of the implicit reward with a frozen reference shows a large loss in accuracy (, , descriptive). The implicit reward uses the current policy in its numerator, whereas PRIME uses a separately trained implicit PRM (Cui et al., [4]). A frozen PRM trained on step labels also fails to improve accuracy when converted to credit centered within each completion, but its segmentation differs from the PRM’s native setup and its scoring head was rebuilt, which limits this comparison (Table 25). Finally, a 20-iteration sweep over continuations at () detects no improvement at those budgets. Noise in the estimator remains a possible contributor, although in the matched-compute recipe continuation sampling contributes only 4% of the second moment of the update (§ 3.2). In the primary short-horizon recipe, then, high agreement between the gradient and the reference did not by itself pick out a reward that beat the declared margin, which is why the matched-compute studies of Section C measure training outcomes directly.
| Training reward | gain (final−start) | vs. outcome () | regime | |
| MC-within, | 6 | −0.009 [−0.048, +0.026] | −0.065 [−0.101, −0.016] (−1.08) | transition |
| MC-within, | 6 | −0.008 [−0.040, +0.038] | −0.064 [−0.107, +0.002] (−0.82) | transition |
| MC-shuffled control, | 5 | +0.035 [+0.018, +0.063] | −0.025 [−0.056, +0.006] (−0.62) | uncertain |
| MC-shuffled control, | 5 | +0.040 [+0.023, +0.065] | −0.020 [−0.058, +0.027] (−0.37) | uncertain |
| MC-shuffled control, | 5 | +0.022 [−0.026, +0.070] | −0.038 [−0.094, +0.023] (−0.49) | uncertain |
| Co-equal standardization (same MC reward, , segments, seeds 0–4): | ||||
| MC-within, | 5 | +0.056 [+0.031, +0.077] | −0.004 [−0.032, +0.034] (−0.09) | tie (n.c.) |
| MC-within, | 5 | −0.080 [−0.122, −0.029] | −0.140 [−0.186, −0.066] (−1.74) | degrades |
| Self-log-ratio (co-equal standardization, frozen reference; , descriptive): | ||||
| Self-log-ratio () | 3 | −0.260 [−0.285, −0.220] | −0.313 [−0.355, −0.275] (−7.81) | collapse |
Table 22. Changing reward weight and the credit combination rule. Qwen2.5-1.5B full fine-tuning; gains and paired differences use greedy accuracy and the 200-problem evaluation set. Intervals and follow Table 17. The transition rows do not survive Holm correction. The middle block applies co-equal standardization to the same Monte-Carlo reward; the bottom block uses the policy’s implicit reward against a frozen reference. These ablations use rewards that differ from PRIME’s learned implicit PRM. Shuffled-control intervals do not establish equivalence at ±0.03. Estimates with small are descriptive (Section H.2).
| Training reward | gain (final−start) | vs. outcome () | regime | |
| GRPO (outcome) | 14 | +0.056 [+0.041, +0.071] | – | baseline |
| PRM, | 8 | +0.044 [+0.007, +0.076] | −0.009 [−0.049, +0.024] (−0.15) | inconclusive |
| PRM, | 3 | −0.028 [−0.035, −0.020] | −0.090 [−0.125, −0.040] (−2.03) | deficit (desc.) |
| PRM, | 5 | −0.073 [−0.123, −0.023] | −0.133 [−0.188, −0.077] (−1.89) | deficit (desc.) |
Table 25. A frozen step-labeled PRM gives inconclusive low-weight and negative high-weight estimates. Qwen2.5-1.5B full fine-tuning on MATH L4–5 using differences between Qwen2.5-Math-PRM-7B scores as within-completion credit. Gain is final−start greedy accuracy; intervals are 95% seed-bootstrap CIs and contrasts are paired against outcome-only. All 16 planned runs completed. At , equivalence at ±0.03 is not established; the interval still permits harm beyond the margin. Higher-weight deficits are descriptive, and the one-sided misses the prespecified 0.005 harm threshold. The step boundaries differ from the native PRM setup, and the scoring head was rebuilt. These changes may weaken the reward and limit conclusions about the released PRM (Section H.6).
Compute. The equal-iteration experiments of Section 3.1 used a node with eight H100-80GB GPUs. A 12-iteration probe on one GPU takes about 20 minutes, a 32-iteration training run for the 1.5B model takes 1–2 GPU-hours, and a 7B run on two GPUs takes 4–6 GPU-hours. The 27B/35B Qwen3.6 and 20B probes use two-GPU sharding for inference, and the equal-iteration study as a whole, with its ablations of weight, continuation budget, combination rule, and model family, used a few hundred GPU-hours. The matched-compute studies of § 2.3 used one GPU per run (an H200 for all but 22 of about 1,320 GPU-hours) and report training time per run (Section C). A systems-performance evaluation would also need the runtime and memory of each stage.
Gate protocols and comparisons
Gates test whether concentrating the credit on the segments where it should matter most helps: the opposition gate keeps the credit that opposes the outcome advantage, and the disagreement gate favors segments whose continuations disagree.
The Monte-Carlo gates use the Qwen2.5-1.5B full-finetuning recipe: 32 iterations, 8 prompts, group size 8, learning rate , no KL penalty, six segments, and continuations. The opposition gate keeps the entries whose sign is opposite to , and the disagreement gate weights each segment by , where is the fraction of continuations that succeed. Both are applied before centering and normalization. We normalize the peak credit within each configuration, which does not match the total credit mass, the norm of the projected gradient, or the size of the actual optimizer step across configurations. Centering also changes which positions have nonzero credit. We trained no control that keeps the entries agreeing with the outcome’s sign while preserving the mask.
Expanded opposition gate. The pilot at gave a paired mean of +0.013 (), with four of eight differences negative. We fixed the analysis plan before adding seeds and then analyzed the combined result, with , once. All 64 dense and control runs reached iteration 31. The paired mean is +0.008, with 95% bootstrap interval [−0.004, +0.021], one-sided , , and ; 13/32 differences are negative. No gated run meets the collapse criterion, a drop of 0.15 from peak to final accuracy; the largest drop is 0.067. The expansion includes the eight pilot seeds. Its post hoc power at the observed effect is 0.37, and the interval remains compatible with an improvement of 2.1 points.
Other opposition weights. On the same 150-problem evaluation set and against matched outcome controls, give respectively +0.015 [−0.043, +0.073] (), +0.013 [−0.048, +0.075] (), and +0.036 [−0.013, +0.080] (, ). A fourth seed stopped at iteration 8 and is excluded. All eleven completed runs remain below the collapse-event threshold (largest drop 0.080). These exploratory comparisons are not corrected for multiple comparisons.
Disagreement versus ungated reward. At , both rewards use the same max-normalized combination rule, six segments, , 200-problem evaluation, and seeds 0–4. Their paired difference is +0.056 [+0.007, +0.094], . The reported paired differences are −0.035, +0.050, +0.070, +0.080, +0.115; a paired- test gives , . The difference grows from +0.006 at iteration 15 to +0.029 at 23 and +0.056 at 31. At , the matched difference is +0.021 [+0.005, +0.033] (all 5⁵ bootstrap resamples), , four of five differences positive, with , . That configuration’s difference against outcome is +0.009 [−0.022, +0.047], . Both comparisons are exploratory, use five seeds, and have no correction for multiple comparisons. They change the mask while keeping the recipe fixed, but they do not hold the size of the resulting gradient or optimizer step fixed.
Seeds per configuration (contiguous from 0; the column of each table gives the count). Outcome baseline and , 0.5, 1: after the pre-registered expansion. , 3, 4: . : . Shuffled control: . Prover and PRM configurations use their own pre-registered seed lists (Section H.6).
Deviations. 7B (Table 23): Qwen2.5-7B, swept to 24, 3–5 seeds per configuration. Cross-family (Table 24): SmolLM3-3B. NuminaMath (Table 26): train split swapped, . Validation set: the gate, prover and cross-family configurations use a 150-problem L4–5 split rather than the primary 200-problem L3–5 split; valid contrasts require the same evaluation instrument. Coequal-standardization ablations (Table 22) standardize the outcome advantage and the dense credit separately. The implicit-reward configurations use a frozen policy reference; they are not PRIME replications.
Language-model setup and implementation
Policy, data, and training recipe
Training and evaluation. The main policy is Qwen2.5-1.5B-Instruct. LoRA probes train rank-16 adapters (adapter scale 32) on query and value projections; the main training study updates all parameters. Each iteration samples eight completions for each of eight prompts, using temperature 1.0, top- 0.95, and at most 380 new tokens. Adam uses , , no weight decay, constant learning rate, global gradient-norm clipping at 1.0, and bfloat16 weights. There is one on-policy gradient step per iteration, no inner-epoch loop, and no importance-ratio clipping. Dense weights stay constant throughout each run. The six-segment end-task credits use up to 200 continuation tokens per prefix.
The verifier extracts the last number inside the final \boxed{}, then falls back to the last number after ### or the last number in the completion. It strips a <think> prefix and uses numerical tolerance , or exact string matching otherwise. The same verifier scores continuations. Validation uses greedy decoding. The reported greedy accuracy and the expected accuracy under sampling therefore measure different quantities. Because of top- sampling, finite continuation length, fixed segmentation, and the initial success prior, the probe reference also differs from an exact population gradient.
Probe and full-fine-tuning configurations. Five geometry runs train LoRA for 12 iterations at learning rate on MATH L3–5 with a KL-to-initial-policy penalty 0.04 ( estimator) and continuations, split three per half. We measure every reward gradient at the same checkpoints from outcome-only training. Three-seed LoRA accuracy runs use the same horizon, learning rate, and KL penalty, with 120 validation prompts. Full-parameter probes and full fine-tuning omit the KL penalty. Main full fine-tuning uses 32 iterations, learning rate , a 568-problem MATH L4–5 training pool, and a 200-problem MATH L3–5 evaluation set. Gains are the recorded final-minus-start accuracy (iteration 31 minus iteration 0). An initial recipe check gave outcome gain +0.052 [+0.031, +0.074]; the larger baseline gives +0.056 [+0.041, +0.071]. At learning rate and in the reported GSM8K control, the outcome baseline does not measurably improve.
Rules for combining credit with the outcome. Co-equal standardization scales the outcome and dense terms separately. The main Monte-Carlo configuration at uses co-equal standardization, whereas the other primary weights and the shuffled controls use the max-normalized rule. Comparisons with outcome-only GRPO remain well defined, because it has no dense term to combine, but comparisons between dense-reward configurations must account for this difference in rules.
Model identifiers. The eight checkpoint probe panel uses Qwen/Qwen2.5-1.5B-Instruct, Qwen/Qwen2.5-Math-1.5B-Instruct, Qwen/Qwen2.5-Math-7B-Instruct, HuggingFaceTB/SmolLM2-1.7B-Instruct, microsoft/Phi-3.5-mini-instruct, openai/gpt-oss-20b, Qwen/Qwen3.6-27B, and Qwen/Qwen3.6-35B-A3B. The final two rows use one seed and are descriptive. Larger models use two-GPU sharding. The panel reports measurements at these checkpoints.
Statistical reporting
Unless stated otherwise, end-task intervals are 95% percentile-bootstrap intervals with seed resamples. A gain row resamples per-seed final-minus-start gains; a paired row resamples same-seed dense-minus-outcome differences. These intervals describe uncertainty across seeds on the fixed validation examples. The standardized paired effect is , without a small-sample correction. Reported bootstrap values are twice the proportion of resampled means on the opposite side of zero. Because such values can overstate the evidence when is small, we give paired- sensitivity checks for the principal claims. Brackets denote intervals; denotes the sample SD unless explicitly labeled SE. SD uses the variance denominator; SE = SD/.
Small samples. At , a 95% percentile-bootstrap interval equals the observed minimum and maximum (an all-one-seed resample has probability ), so these rows describe the observed range. For the , transition, exhaustive enumeration of all resamples gives , and the paired- test gives , , with five of six differences negative. For the other paired rows with , exhaustive enumeration does not change whether the interval crosses zero. Probe rows pooled across checkpoints are correlated, so their spread is descriptive.
Groups of comparisons and registration. The primary harm analysis includes nine Monte-Carlo weight configurations on the Qwen2.5-1.5B, 200-problem, iteration-31 recipe: five in Table 17, two transition weights, and two co-equal-standardization configurations in Table 22. The three low-weight equivalence tests form a separate Holm family. This group of comparisons excludes other models, datasets, evaluation sets, and the 20-iteration estimator sweep; the source study has 31 Monte-Carlo configurations at iteration 31 (30 after merging two with the same configuration), 22 of them outside the nine-configuration family. Analysis plans were locked before adding data for the low-weight seed expansion, complementary-prover confirmation, SmolLM3 expansion, and gate expansion. The primary low-weight expansion adds seeds 8–13 or 5–13 to existing configurations; the gate expansion retains its eight pilot seeds. Because the expansions add seeds to existing configurations, they extend those configurations rather than replicate them independently.
Additional drift seeds. Two completed drift-probe seeds extend the paired estimate from [, , , to [, , . One eligible co-equal-standardization seed changes [ to [; its second drift run stops at iteration 20 and is excluded under the stated iteration-28 admission rule. Drift seeds use their own outcome controls run at the same time. These runs serve as a sensitivity analysis; the nine-configuration family and the seed sets of the main tables are unchanged.
Validation sets. The main 200-problem MATH L3–5 set is used for the weight sweep, low-weight equivalence, PRM, implicit-reward, and disagreement-versus-ungated comparisons. A 150-problem L4–5 set is used for opposition gates, prover, and cross-family configurations. A separate 115-problem set is used only for the degenerate-slice ablation. Starting accuracies differ: Qwen2.5-1.5B/OLMo-2 means are 0.24–0.26 on the 200-problem set and 0.15–0.20 on the 150-problem set; SmolLM3 starts at 0.32–0.34, Phi-4-mini at 0.25, NuminaMath at 0.28–0.29, and the degenerate slice at 0.03–0.10. We therefore compare configurations only within an evaluation set. Table 18 uses the available opposition estimate from the same evaluation set and omits the unmatched ungated comparison.
Deficits and collapse events. A negative paired configuration mean is different from a collapse within one run. We define a collapse event descriptively as final accuracy at least 0.15 below that run’s peak. We chose this threshold after inspecting variation in baseline drops. At 1.5B, the Monte-Carlo and 4 configurations have 0/8 and 0/6 such events; their largest drops are 0.14 and 0.11, versus 0.055 for outcome-only. Still, all eight and six seeds have negative absolute gains and paired differences. One of five co-equal-standardization seeds has a drop exactly 0.150. One and two seeds peak at iteration 0, so a rise-then-decay trajectory does not describe every deficit. Across 85 eligible gate/control runs (86 launched, one stopping at iteration 8), no drop reaches 0.15; the largest is 0.080.
Per-configuration bootstrap values. For Table 17, the two-sided bootstrap values are 0.638, 0.740, 0.714 at , below at 4, 5, and 0.304 for the shuffled control. For Table 22, they are 0.014, 0.060 for weights 2, 3; 0.747, 0.0004 for co-equal-standardization weights 0.5, 3; 0.134, 0.390, 0.248 for shuffled weights 1, 3, 4; and below for the three-seed implicit-reward row.
Gradient probes
This subsection describes how the gradient tests of § 2.4 are computed at language-model scale.
Score-gradient map. At a fixed checkpoint, we compute a credit gradient with one weighted backward pass:
We accumulate gradients over completion microbatches using logsumexp token probabilities. The probe builds from at most 24 completions and forms from the same subsample, so the least-squares projection uses this fixed, subsampled . It costs forward passes, with the backward graphs kept in memory. The reported outcome residual 0.012 0.008 checks the numerical projection rather than the theorem itself; with full-parameter gradients, the residuals are as small as .
Progress reference and sample overlap. Following Monte-Carlo prefix-value methods [15, 31], the probe uses three interior cuts and the endpoint, with up to 200 continuation tokens, temperature 1.0, top- 0.95, and initial success prior 0.5. Each segment’s credit is the change in success rate across it. Half A of the continuations defines , and half B, where labeled, provides a candidate credit from disjoint samples. The credit computed from all rollouts, which the primary pooled measurement uses, overlaps with half A. After projection, it differs from the centered full credit only by the projection residual (agreement within over 135 rows), so the reported agreement with the reference, 0.95, uses overlapping samples. The full-parameter alignments using disjoint samples are 0.97, 0.66, 0.89, 0.82, mean 0.84. The overlapping full-parameter mean is 0.96, with mean1.96SE interval [0.92, 0.99] (percentile bootstrap [0.92, 0.98]), shuffled mean , and off- fraction 0.93.
Controls and interpolation. Permuting values within each completion preserves the distribution of token-credit magnitudes. It does not, in general, preserve , because the Gram matrix of the scores is not isotropic, and it need not preserve the actual Adam step. The interpolation family is , , with unit-normalized, zero-sum half-B and Gaussian rewards. The fractions of their updates that lie outside are similar, but their budgets are not exactly matched. The reported within-bin and magnitude-adjusted correlations are 0.88 [0.84, 0.92] and 0.90 [0.85, 0.94]; the aligned-minus-shuffled reference-cosine gap is +0.97 [+0.93, +1.01]. These measurements describe agreement with the measured reference at fixed probes.
Implicit reward. This credit is , with the actor itself in the numerator and the initial policy in the denominator. We do not scale it in the probes, because the cosine and the fraction outside do not depend on scale. Averaged over each run, the cosines of the five runs stabilized by a KL penalty are , , , , ; without the KL penalty, values at single checkpoints reach , so alignment with the estimated reference can change sign. At initialization, the credit is zero because the actor and the reference are the same, so these summaries use only iterations after the first. This implicit reward differs from PRIME’s separate implicit PRM trained with outcome labels [4].
End-task results and controls
This subsection gives the end-task results behind Section 3.1, with the checks behind each claim.
Choice of recipe. A short LoRA recipe and a GSM8K recipe barely improved the outcome-only baseline, so they could not separate the rewards; the main study therefore uses full fine-tuning on MATH.
Low-weight equivalence. At paired seeds, the 90% percentile-bootstrap intervals for are respectively , , and , with TOST . All pass Holm correction over the three configurations at the margin. Paired- TOST gives , also passing Holm, and studentized-bootstrap 90% intervals also lie inside the margin. The reported minimum effects detectable by superiority tests are 0.025–0.035 at 80% power; estimated power at an effect equal to the margin ranges from 0.67 to 0.92. These results support equivalence at the stated resolution.
High-weight deficits. The deficit remains significant after bootstrap and paired- Holm correction over all nine primary weight configurations. The deficit passes bootstrap Holm correction but misses the paired- step ( versus threshold 0.0063). Unadjusted one-sided paired- values are 0.0033 and 0.00025 for ; studentized-bootstrap 95% intervals are and . Using only seven configurations for correction would change the conclusion, but the reported claim uses all nine. The co-equal-standardization configuration passes the bootstrap correction, while the transition passes only the uncorrected test. The implicit-reward row has and , so it remains descriptive despite its large deficit.
Figure 15. Gradient-reference measurements at a fixed policy and batch. Weighted backward passes compute completion-score sums , outcome gradient , dense gradients , and Monte-Carlo reference in LoRA parameter space. Projecting onto gives the confinement residual, projected alignment, and full-gradient reference cosine. The lower axis illustrates : positive rescaling preserves its sign, while a direction change can change it. This inner product measures agreement with an estimated reference. The population directional derivative and actual Adam update are separate quantities.
Curve summaries. We reanalyze the low-weight runs with three measures: iterations to first reach the baseline’s mean final accuracy 0.308 (baseline median 20/32 iterations), area under the accuracy curve relative to start (baseline +0.038), and per-run peak (baseline mean 0.324). The paired dense-minus-outcome estimates, in that order, are:
: −0.1 [−4.0, +3.3] iterations, −0.001 [−0.014, +0.014] area, and −0.008 [−0.018, +0.003] peak.
: −0.2 [−6.3, +6.1], +0.000 [−0.013, +0.013], and +0.002 [−0.008, +0.013].
: +4.4 [−0.7, +9.4] (), −0.007 [−0.019, +0.006], and −0.004 [−0.014, +0.005].
These post hoc summaries detect no speed advantage. Runs that never reach the target are coded as 32 iterations, so the time-to-target statistic is censored.
Continuation-budget sweep. At , the study has four seeds per setting and ends at iteration 19 for all twelve runs. No 32-iteration comparison is available. It detects no difference at that horizon. A separate co-equal-standardization set also ends before the primary iteration-31 horizon; neither short study is included in the nine-configuration group of comparisons.
Positive pilots. The Phi-4 outcome-only configuration has three seeds and mean gain +0.142. Its dense and 5 configurations each have one seed, with same-seed paired gains +0.093 and +0.080. These positive pilots exceed the primary equivalence margin but provide no seed-level uncertainty estimate. They remain unconfirmed, but they limit any broad negative claim.
Qwen2.5-7B sweep. Full fine-tuning uses two GPUs, six completions for each of six prompts, 32 iterations, learning rate , and 150 validation prompts. Seeds 0–4 are used except at , which has seeds 0–2. The outcome baseline gains +0.209, while no dense-reward configuration mean exceeds it across 43 dense runs and weights up to 24. The descriptive collapse-event count is 3/5 at each of , versus 0/5 for outcome-only (largest baseline drop 0.11). One seed of each weight ends at zero accuracy after peaks 0.46 and 0.43. High-weight seeds that do not collapse stay close enough to the baseline that configurations means at remain positive. Expanding the high-weight configurations from three to five seeds increased event counts and changed the mean gain from positive to negative.
Controls for interpreting performance loss
The high-weight deficits raise a question: why does a reward that aligns well with the reference hurt training? These controls check the most direct explanations: a decay of alignment during training, where the credit falls, the coherence of the credit, and changes in length and entropy.
Alignment along training. Paired full-fine-tuning runs at , two seeds each, save checkpoints every four iterations. Full-parameter probes use , . The alignment between the full credit and the overlapping half-A reference stays at or above 0.992 (from 0.993 to 1.000 for seed 1). With disjoint half-B samples, it ranges from 0.969 to 1.000; the shuffled reward’s alignment remains near zero; and the fraction of the update outside ranges from 0.62 to 0.98. In seed 1, the configuration declines from peak 0.27 to 0.155, while the low-weight configuration gains +0.07. The two high-weight probe runs’ drops, 0.100 and 0.115, are below the descriptive 0.15 event threshold. These observations show that agreement with the reference persists even as greedy accuracy falls.
Credit on segments where continuations disagree. At initialization, 12 prompts with four completions each are cut at eight interior points plus the endpoint, yielding 432 segments and 2,592 continuations at . Credit magnitude is , with . A continuation-success variance above 0.1 means at least one success and one failure: the smallest nonzero variance at is . Such segments account for 19.7% of segments and carry 38.4% of credit; the remaining 61.6% lies on unanimous segments. For disagreement segments, the ratio of credit share to segment share is 1.95×, with correlation 0.30 and mean magnitude 0.22 versus 0.085.
Shuffled-credit comparison. At , the coherent and shuffled configurations use different rules for combining credit with the outcome; their paired difference-in-differences is −0.128 [−0.155, −0.098], , over five common seeds. At , both use the max-normalized rule, and the difference is −0.074 [−0.111, −0.040], , , with gains −0.052 and +0.022 on that common-seed subset. Matching the rule removes one confound; the norms of the projected gradients, the clipping, the policy KL, and the Adam steps can still differ between the two configurations.
Length, entropy, and drift. On a fixed 100-prompt subsample, response length stays within 1% of the low-weight configuration through 28 iterations while high-weight accuracy moves from 0.25 to 0.12; the fraction of degenerate completions does not rise. High-weight entropy rises from 0.21 to 0.58, versus 0.18–0.20 for low weight, and KL from the initial policy is about 30 times larger. These changes occur together and are consistent with several optimization explanations, such as rising entropy or over-optimization of the proxy reward.
Other families, provers, reward sources, and datasets
To see whether the equal-iteration pattern depends on the model family, on where the progress estimates come from, or on the dataset, we repeat the comparison with each of them changed.
SmolLM3-3B. The same trainer uses HuggingFaceTB/SmolLM3-3B, with extended thinking disabled in the chat template. Outcome, , and configurations were expanded to seeds 0–9 under a fixed paired- TOST analysis plan at and one-sided harm test at 0.005. The low-weight 90% interval excludes a gain of 0.03 (reported ) and extends 0.004 beyond the lower equivalence boundary. The high-weight paired effect is , , one-sided , short of the specified 0.005 criterion. Nine of ten differences are negative; two seeds drop 0.34 and 0.17 from peak, one ends at gain , and a third drop is 0.14. At , one of three runs has a negative final gain. These results reproduce the qualitative pattern of the primary study at lower statistical resolution.
Complementary prover. Qwen2.5-Math-1.5B-Instruct supplies continuations from the actor’s prefixes, following the complementary-prover motivation of Setlur et al. [24]; the actor receives the resulting dense credit. At , three paired differences are , , , giving . A three-seed pilot gives . This was one of six exploratory comparisons. It motivated an confirmation, which required a paired interval excluding zero and a result that passed Holm correction. The expanded estimate is , , from differences , , , , , , , . It falls short of the first criterion, and its interval still allows meaningful gains.
Step-labeled PRM. The frozen Qwen2.5-Math-PRM-7B (Zhang et al. [38]) scores actor-generated completions at the Monte-Carlo configuration’s interior cuts, using the step separator, canonical system prompt, and boxed-answer format. Differences in the PRM’s positive-class probabilities form a credit, with initial prior 0.5, combined with the outcome by the same rule. These differ from the model’s native double-newline step boundaries, which may weaken the reward. We rebuilt the scoring head from the released weights on a standard backbone; a convention check gives approximately 1.0 for correct chains and 0.0 for a corrupted step.
All 16 planned PRM runs completed. At , the paired effect is −0.009 [−0.049, +0.024], , superiority one-sided ; TOST fails (). At , it is −0.090 [−0.125, −0.040], . At , it is −0.133 [−0.188, −0.077], , one-sided , above the specified 0.005 harm criterion. The latter two comparisons remain descriptive. These results apply to this conversion and scoring pipeline.
Degenerate-group channel ablation
Groups whose completions all fail give outcome-only GRPO no signal. Dense rewards are often motivated as a way to learn from such groups, whereas prompt selection (Section 4) avoids them, so we ask whether dense rewards help where they dominate. On a curated MATH slice, the fraction of groups with identical outcomes is 0.894 (per-iteration range 0.50–1.00). Qwen2.5-1.5B is full-fine-tuned for 32 iterations with group size eight, coequal standardization, , and seeds 0–2. The dense credit is zero-sum within each completion, which removes its completion-mean term but generally leaves a component in ; at language-model scale, the measured component in carries about 0.26 of the squared norm. Outcome gains are +0.017, +0.009, +0.052 (mean +0.026), versus −0.070, +0.009, +0.009 (mean −0.017) for the dense-reward configuration. The paired effect is −0.043 [−0.087, 0.000] over three seeds (descriptive).
All completions in the degenerate groups are wrong to the reported precision, and training accuracy is about 0.02. Coequal standardization amplifies a low-signal progress reward, and a frozen math-specialized prover succeeds on 13% of 150 tested prompts with , so progress is hard to estimate in this setting: where the outcome gradient is zero, a dense signal can exist without helping. Attributing the gains of other degenerate-group methods [20, 1] to a particular channel would need a control that trains only the completion-mean term.
NuminaMath. A 1,500/200 train/validation slice of NuminaMath-CoT uses the cn_k12 and synthetic-AMC sources, filtered to numeric answers supported by the verifier. It uses the same trainer for 32 iterations and seeds 0–2; the outcome baseline gains +0.098 (Table 26). These rows use small samples and are descriptive.
What a Dense Reward Adds to One Update
This section states the fixed-batch results behind the alignment test of Section 2.4. They need only linear algebra and apply to any differentiable policy on a fixed batch. Proofs are in Section K, and Table 27 collects the notation.
| Symbol | Meaning |
| Group size, completion length and parameter dimension | |
| Group-relative completion coefficient on the sampled batch equation (1) | |
| Token score and completion score sum | |
| Sampled score matrix, completion-sum operator and score Gram matrix | |
| Conditional score contribution of token coefficients | |
| Completion-constant and within-completion zero-sum parts of | |
| Comparison span allowing arbitrary completion coefficients | |
| Projectors onto and its orthogonal complement | |
| Within-completion credits mapped into parameter space | |
| Unclipped sampled outcome score contribution | |
| Population expected accuracy and its full gradient | |
| Estimated Monte-Carlo progress-gradient reference | |
| Experimental dense weight and total continuation budget |
Table 27. Notation and scope. All projectors use the Euclidean parameter-space metric unless stated otherwise.
Notation. We use the notation of Section 2.1, with completions of length and parameters . Stacking the token scores into , a token-level credit (one coefficient per sampled token) contributes
before token-specific clipping, importance weighting, KL terms, and optimizer state, which determine the effective update. Let sum tokens within each completion (in this section and in Section K, denotes this matrix, not the token budget), , , proj, and . Any reweighting of whole completions updates within ; this is broader than the outcome baseline, whose centered coefficients span only . We write , where and , and for the expected accuracy, with gradient .
What can a dense reward add to one update?
Outcome coefficients and completion means produce updates in (Propositions 2 and 3). The next result gives the exact condition under which a token-level credit stays in this space, accounting for the score map’s kernel: different token credits can produce the same gradient.
Theorem 1 (Sample-conditional equivalence). *For every
,
Moreover, . If , equivalence holds exactly for completion-constant credits, and the residual dimension is .
Centering credit within each completion does not generally make its gradient orthogonal to (Theorem 3). Once a credit adds a direction outside , we can ask whether that direction helps a local target.
Proposition 1 (First-order projection identity). For arbitrary and dense direction , the first-order advantage of over its matched projection is
Only the components of and outside decide whether improves on its projection to first order. This is the alignment test of § 2.4, which Section 3.1 measures against a sampled reference. It compares a direction with its own projection at one policy, whereas an outcome-only training run follows its own sequence of policies and sampled batches.
Notation
Table 27 collects the notation for the sampled score map, the population gradient, and the experiments. These objects describe different parts of the analysis and need not coincide.
Proofs and Supporting Geometry
All vector spaces below are finite-dimensional and use the Euclidean inner product. We flatten credits before multiplying them by or . The results allow rank-deficient score matrices, including .
The sampled score map
Proposition 2 (Decomposition). For every ,
and
Proof. Apply the linear map to the decomposition of token coefficients. A completion-constant credit is , so its image is .
Proposition 3 (Outcome confinement on the realized batch). The set of score contributions with arbitrary real completion coefficients equals , and . Restricting coefficients to gives , of dimension at most .
Proof. The unrestricted set is the image of . Restricting its domain gives the second image and rank bound. The sampled rewards further constrain the normalized, zero-sum advantages. They select one update from this comparison space.
Proof of Theorem 1. The condition holds if and only if there is a with . This is exactly . If has full row rank, its transpose is injective, so the kernel is zero. The dimension claim follows from the next result.
Figure 17. Reported weight sweeps in four training settings. Paired greedy-accuracy differences versus outcome-only, with asymmetric 95% seed-bootstrap intervals and a reference band. Color labels summarize the plotted estimates; only the primary low-weight 1.5B configurations have certified equivalence. The 7B and NuminaMath configurations have 3–5 and 3 seeds; SmolLM3 endpoints have 10. Carets show where the 7B lower tails extend to and . The positive single-seed Phi-4 pilots appear in Section H.4.
Theorem 2 (Reachable residual dimension). Let and . Then and
Equality with the right side holds when .
Proof. The coefficient space decomposes as . Applying gives . On , projection onto has kernel . Therefore, . It is at most . If is injective, and . At both sides of the exact identity are zero; at the within space is zero.
Theorem 3 (Projection through the score Gram matrix). For , the projection of onto is , where
The Moore–Penrose solution gives the minimum-norm coefficient vector if the matrix is singular. If , the fraction of the squared norm that lies in is
Orthogonality holds exactly when .
Proof. Least squares for and gives . These normal equations are consistent and give a unique projected vector even when is not unique. Substituting gives equation 5. The numerator is the squared norm of that vector, and the denominator is . If , the fraction is undefined. If for , every within-completion credit is orthogonal to ; positive overlap therefore does not hold in every case.
Proof of Proposition 1. The complementary orthogonal projectors are self-adjoint and satisfy . Thus . Subtracting the rate of proves the identity. We do not need to represent through the finite sampled matrix. If we use an estimator , the rate error satisfies
An empirical alignment measurement alone does not bound this estimation error.
Telescoping is not confinement
Take one two-token completion with linearly independent scores . The credit has zero total, yet . It can represent the potential increments along a path with values 0, 1, 0. Broadcasting the total instead produces the zero credit in . Potential-based shaping results concern particular reward, return and baseline constructions (Ng et al., 1999); a telescoping total alone does not constrain the sampled score sum weighted by token coefficients. Likewise, when all group rewards agree, and the outcome update is zero, but this does not guarantee that a nonzero dense update improves .
The Expected GRPO Update
The results of § 22 describe one sampled batch. This section gives the expected outcome-only GRPO update over fresh samples, which the matched-compute methods use to explain how adding a dense reward can change the relative weight of prompts (Section C).
The population gradient is not confined to a sampled span
Proposition 4 (Expected binary group-relative score). Fix a prompt and a differentiable distribution over completions. Assume its support is independent of the parameters and that differentiation can pass under the expectation. Let be a fixed verifier, , , and sample completions iid. Define , , and let be the group reward standard deviation when . With advantages equation (1),
Groups with contribute zero.
Proof. The score identity gives and , hence and . Condition on . By exchangeability, the expected sums of successful and failed completion scores are and . Their advantage coefficients are and . Substituting these values and applying the outer gives
Averaging proves the formula. Since and , a mixed group has positive probability. The multiplier is therefore positive. At or , all sampled advantages are zero and the displayed ratio is undefined.
This score-function identity does not identify a new obstacle to population learning. Population GRPO analyses also study the normalization and success dynamics [18]. Across prompts, can vary and reweight their gradients. Bias from clipping, off-policy sampling or other transformations needs a separate analysis. An update can belong to its sampled without its expectation belonging to any one realized .
Figure 18. Which part of a dense update is new? (a) Completion-constant coefficients produce updates in the span of completion score sums. A within-completion credit can add a residual direction , while also having a component in . (b) For a target gradient , the residual directional derivative is positive in the shaded half-space. The dashed line is the zero-rate boundary perpendicular to the target residual. This local comparison uses the projected direction as its baseline. It does not guarantee a gain over a separate outcome-only training run.
References
- [1]Agrawal, A., Liu, M., and Huang, F. VeriGate: Verifier-gated step-level supervision for GRPO, 2026. URL https://arxiv.org/abs/2605.30451.
- [2]Ahmadian, A., Cremer, C., Gallé, M., Fadaee, M., Kreutzer, J., Pietquin, O., Üstün, A., and Hooker, S. Back to basics: Revisiting REINFORCE-style optimization for learning from human feedback in LLMs. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12248–12267, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.662. URL https://aclanthology.org/2024.acl-long.662/.DOI
- [3]Bae, S., Hong, J., Lee, M. Y., Kim, H., Nam, J., and Kwak, D. Online difficulty filtering for reasoning oriented reinforcement learning. In Demberg, V., Inui, K., and Marquez, L. (eds.), Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 700–719, Rabat, Morocco, March 2026. Association for Computational Linguistics. ISBN 979-8-89176-380-7. doi: 10.18653/v1/2026.eacl-long.30. URL https://aclanthology.org/2026.eacl-long.30/.DOI
- [4]Cui, G., Yuan, L., Wang, Z., Wang, H., Zhang, Y., Chen, J., Li, W., He, B., Fan, Y., Yu, T., Xu, Q., Chen, W., Yuan, J., Chen, H., Zhang, K., Lv, X., Wang, S., Yao, Y., Han, X., Peng, H., Cheng, Y., Liu, Z., Sun, M., Zhou, B., and Ding, N. Process reinforcement through implicit rewards. Transactions on Machine Learning Research, 2026. ISSN 2835-8856. URL https://openreview.net/forum?id=9SkkiflopZ.arxiv.org/abs/2502.01456
- [5]DeepSeek-AI et al. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning, 2025. URL https://arxiv.org/abs/2501.12948.
- [6]Foster, T., Sims, A., Forkel, J., and Foerster, J. N. LILO: Learning to reason at the frontier of learnability. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=8HYeWMf0W3.DOI
- [8]Gao, L., Schulman, J., and Hilton, J. Scaling laws for reward model overoptimization. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org, 2023.arxiv.org/abs/2210.10760
- [9]Gao, Z., Kim, J., Sun, W., Joachims, T., Wang, S., Pang, R. Y., and Tan, L. Prompt curriculum learning for efficient LLM post-training. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=zqQCaBD3P.arxiv.org/abs/2510.01135
- [11]Guo, Y., Xu, L., Liu, J., Ye, D., and Qiu, S. Segment policy optimization: Effective segment-level credit assignment in RL for large language models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=9osvTOYbT4.arxiv.org/abs/2505.23564
- [13]Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL https://openreview.net/forum?id=7Bywt2mQsCe.
- [14]Hou, Z., Hu, Z., Li, Y., Lu, R., Tang, J., and Dong, Y. TreeRL: LLM reinforcement learning with on-policy tree search. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025.arxiv.org/abs/2506.11902
- [15]Kazemnejad, A., Aghaohari, M., Portelance, E., Sordoni, A., Reddy, S., Courville, A., and Le Roux, N. VinePPO: Refining credit assignment in RL training of LLMs. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=Myx2kJfZzn.arxiv.org/abs/2410.01679
- [16]Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let’s verify step by step. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=v8L0pN6EOi.arxiv.org/abs/2305.20050
- [17]Mishra, A., Chakraborty, S., and Kapusuzoglu, B. On the policy gradient foundations of group relative policy optimization: Credit assignment, gradient sparsity, and rank collapse, 2026. URL https://arxiv.org/abs/2606.29238.
- [18]Mroueh, Y. Reinforcement learning with verifiable rewards: GRPO’s loss, dynamics, and success amplification. In Forty-third International Conference on Machine Learning, 2026. URL https://openreview.net/forum?id=z9Bo4eXSpW.arxiv.org/abs/2503.06639
- [20]Palandye, A. R., Glick, R., and Kaul, O. Reward granularity in RLVR: Comparing process and outcome reward structures for mathematical reasoning in small language models, 2026. URL https://arxiv.org/abs/2607.02869.
- [22]Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL https://qwenlm.github.io/blog/qwen2.5/.
- [23]Schneider, J., Schumacher, P., Guist, S., Chen, L., Haeufle, D., Schölkopf, B., and Büchler, D. Identifying policy gradient subspaces. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=iPWxqnt2ke.arxiv.org/abs/2401.06604
- [24]Setlur, A., Nagpal, C., Fisch, A., Geng, X., Eisenstein, J., Agarwal, R., Agarwal, A., Berant, J., and Kumar, A. Rewarding progress: Scaling automated process verifiers for LLM reasoning. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=A6Y7AqIzLW.arxiv.org/abs/2410.08146
- [25]Shao, R., Li, S. S., Xin, R., Geng, S., Wang, Y., Oh, S., Du, S. S., Lambert, N., Min, S., Krishna, R., Tsvetkov, Y., Hajishirzi, H., Koh, P. W., and Zettlemoyer, L. Spurious rewards: Rethinking training signals in RLVR. In Forty-third International Conference on Machine Learning, 2026. URL https://openreview.net/forum?id=tqTNOpkP5j.arxiv.org/abs/2506.10947
- [26]Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y. K., Wu, Y., and Guo, D. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402.03300.
- [27]Skalse, J. M. V., Howe, N. H. R., Krasheninnikov, D., and Krueger, D. Defining and characterizing reward gaming. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=yb3HOXQ3lX2.arxiv.org/abs/2209.13085
- [28]Sun, Y., Shen, J., Wang, Y., Chen, T., Wang, Z., Zhou, M., and Zhang, H. Improving data efficiency for LLM reinforcement fine-tuning through difficulty-targeted online data selection and rollout replay. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=uwUkETPIJN.arxiv.org/abs/2506.05316
- [29]Sutton, R. S., McAllester, D., Singh, S., and Mansour, Y. Policy gradient methods for reinforcement learning with function approximation. In Solla, S., Leen, T., and Müller, K. (eds.), Advances in Neural Information Processing Systems, volume 12. MIT Press, 1999. URL https://proceedings.neurips.cc/paper_files/paper/1999/file/464d8288b85b0bed98e8 0ade0a5c43b0f-Paper.pdf.
- [31]Wang, P., Li, L., Shao, Z., Xu, R., Dai, D., Li, Y., Chen, D., Wu, Y., and Sui, Z. Math-Shepherd: Verify and reinforce LLMs step-by-step without human annotations. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9426–9439, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.510. URL https://aclanthology.org/2024.acl-long.510/.DOI
- [32]Williams, R. J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3–4):229–256, May 1992. ISSN 0885-6125. doi: 10.1007/BF00992696. URL https://doi.org/10.1007/BF00992696.
- [33]Yang, A. et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024.arxiv.org/abs/2407.10671
- [34]Yang, A. et al. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388.
- [37]Yuan, L., Li, W., Chen, H., Cui, G., Ding, N., Zhang, K., Zhou, B., Liu, Z., and Peng, H. Free process rewards without process labels. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=8ThnPFhGm8.arxiv.org/abs/2412.01981
- [38]Zhang, Z., Zheng, C., Wu, Y., Zhang, B., Lin, R., Yu, B., Liu, D., Zhou, J., and Lin, J. The lessons of developing process reward models in mathematical reasoning. In Che, W., Nabende, J., Shutova, E., and Pilehvar, M. T. (eds.), Findings of the Association for Computational Linguistics: ACL 2025, pp. 10495–10516, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-256-5. doi: 10.18653/v1/2025.findings-acl.547. URL https://aclanthology.org/2025.findings-acl.547/.DOI
- [39]Zheng, H., Zhou, Y., Bartoldson, B. R., Kailkhura, B., Lai, F., Zhao, J., and Chen, B. Act only when it pays: Efficient reinforcement learning for LLM reasoning via selective rollouts. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=x5lTYXmW2.arxiv.org/abs/2506.02177















