betaFeedback

WorksComputer Science

Calibrated Decision Rewards: Training Language Models with a Judge That Knows How Sure It Is

Abstract

Reinforcement learning (RL) can only teach a language model what its reward can recognize. Answer checkers are exact but need a reference answer; learned reward models and LLM judges work everywhere, but their scores are not probabilities. We study RLCDR (Reinforcement Learning from Calibrated Decision Rewards). An LLM converted into a decision model answers a typed question about a response ("is the final answer correct?", "which step is the first error?") with a probability distribution, read in one forward pass, debiased over option orders and calibrated with about a hundred labels. Training Qwen3-1.7B with GRPO on MATH, this reference-free reward raises MATH500 accuracy from 72.2% to 76.8% (+4.65 points, p < 1e-4), matching training on the answer key (76.9%), while a random reward gains only +0.65. The gain grows with difficulty, from 44.8% to 51.7% on the hardest level. On the policy's own changing samples the reward stays calibrated: its expected calibration error is half that of a softmax judge on the same 27B backbone (0.074 vs. 0.141), and it shows no sign of reward hacking. Without step training, the same model finds the first wrong step better than a dedicated 7B process reward model (ProcessBench F1 0.772 vs. 0.722, 20/20 splits). We also map where the approach stops being enough: trained only on problems the base model mostly fails, the reference-free reward still improves the policy (+1.9 MATH500 points, p = 0.03) but trails the answer key, because its probabilities become overconfident on that distribution. We state six hypotheses in advance and test each with a control designed to falsify it.

Key results at a glance
The model gets betterMATH500 72.2% →\to 76.8% (+4.65, p<10−4p < 10^{-4}); hardest level 44.8% →\to 51.7% (p=0.002p = 0.002); pass@4 85.2% →\to 88.4%.
Because of the rewardRandom reward: +0.65 (n.s.). RLCDR vs. random: +4.0 (p=10−4p = 10^{-4}).
As good as the answer key76.8% vs. 76.9% with ground-truth reward (difference −0.1-0.1, 95% CI [−1.8-1.8, 1.5]), with no access to reference answers.
Where it stops being enoughTraining only on problems the base model mostly fails, the reference-free reward still helps (MATH500 +1.9, p=0.03p = 0.03) but trails the answer key (74.1% vs. 75.3%, n.s.). Here its step credit works best added to the answer key (75.7%).
A more honest rewardOn-policy ECE 0.074 vs. 0.141 (softmax judge, same backbone); Brier 0.077 vs. 0.106; better ranking at every difficulty level.
Finds reasoning errorsProcessBench F1 0.772 vs. 0.722 (Qwen2.5-Math-PRM-7B) and 0.682 (softmax), winning 20/20 splits.
CheapOne forward pass per question; no generated critique, no sampling.

Table 1.

Introduction

RL from verifiable rewards has driven much of the recent progress in LLM reasoning [10, 26, 50, 71]. Its strength and its limit are the same thing: an exact answer checker. Where a checker exists (final answers in math, unit tests in code), RL works. Where none exists (proofs, partial credit, open-ended answers, most of alignment), practitioners use learned rewards, and learned rewards are fragile. Bradley–Terry reward models output scores that are only meaningful within one prompt [43, 57]. They are overoptimized as training proceeds [8, 17] and pick up spurious features such as length [55]. LLM judges read through their next-token softmax are badly overconfident [20, 42, 62], sensitive to option order [45, 54, 83] and foolable by single tokens [81]. The strongest judges buy accuracy by generating long critiques sampled many times [35, 68, 77], which makes them a large share of the cost of each RL step [39, 86]. DeepSeek-R1 explicitly avoided neural reward models for reasoning because of hacking and cost [10].

We argue that the missing ingredient is calibration: a reward whose numbers mean what they say. If a reward says “0.8”, then about 80% of the responses it scores that way should be correct, and this should keep holding as the policy changes. Calibration is what lets a reward be compared across prompts. It lets the reward express doubt instead of confident error, and it makes the KL-regularized RL objective a principled Bayesian update [24].

The research gap

G1. No reward is both reference-free and trustworthy. Answer checkers need a reference answer; reward models and judges do not, but their scores are not probabilities and are known to be overconfident, position-biased and hackable (§ 2).

G2. Calibration of rewards is measured offline, if at all. Prior work reports judge calibration on fixed benchmarks. Nobody has measured whether a reward stays calibrated on the policy’s own, shifting samples during RL, which is when it matters.

G3. Reward quality is rarely tied to model quality. Reward benchmarks predict RL outcomes poorly [67], and RL gains on Qwen models can appear with random rewards [51]. Few papers show, with a random-reward floor and an answer-key ceiling, that their reward is what improved the model.

G4. Hard problems give a 0/1 checker nothing to learn from. When all sampled answers fail, the answer key returns all zeros and GRPO receives no gradient. A reward that can grade how far a failed attempt got has not been tested as a calibrated, single-pass signal.

Objective. Our objective is to show that an LLM converted into a decision model, debiased and calibrated with about a hundred labels, is a reward that (i) gives better probabilities than the same LLM read through its softmax, (ii) stays calibrated on-policy, (iii) rewards reasoning steps as well as final answers, and above all (iv) makes a real model more accurate, as much as the answer key does, without seeing a reference answer.

Our approach. We use a decision model: an LLM converted, via a trained readout head, into a model that answers a typed multiple-choice question with a probability distribution over the options. The reward is that distribution, after (i) averaging over option orders to remove position bias and (ii) fitting two calibration parameters on a small labeled set (§ 3). A single forward pass gives an outcome reward (“is the final answer correct?”) or a step-level reward (“which step is the first error?”).

Research questions and hypotheses. We fixed six hypotheses before running the RL experiments. Table 1 lists each one, the experiment that tests it, the control that could falsify it, and the verdict. § 5.9 returns to this table with the evidence.

What we show. This paper makes one central claim and supports it with controls designed to rule out the usual alternative explanations:

C1. A real model improves, and the reward is the cause (§ 5.2). With every other setting identical, RL with our reference-free reward improves Qwen3-1.7B by +4.65 MATH500 points. That equals training on the answer key. A random-reward control rules out “RL alone helps”, which is a known confound for Qwen models [51].

HypothesisTest (falsifying control)Verdict
H1RL with the calibrated decision reward makes the policy more accurate on held-out problems, and the reward, not RL itself, is the cause.E6: GRPO, ground-truth eval (random-reward floor)Supported
H2The reference-free reward trains the policy as well as the answer key.E6 (answer-key ceiling); E6bSupported on MATH; Partly on hard problems
H3The reward stays calibrated and unhacked on the policy’s own samples, better than a softmax judge on the same backbone.E6 on-policy logs (same-backbone softmax)Supported
H4The decision model locates reasoning errors at least as well as a dedicated process reward model, with no step training.E4 ProcessBench; E6b prefix restarts (random step)Supported
H5With equal calibration labels, decision probabilities are better reward probabilities for correctness than softmax probabilities and scalar-RM scores.E1, E2 (same labels, same backbone, head refit)Supported vs. softmax; Partly vs. scalar RM
H6On hard problems, step-level decision credit gives a better policy than the answer key alone, and a policy better than the softmax judge on the full set.E6b; E6 (softmax judge)Partly

Table 1 (PDF p. 4). Table 1. Hypotheses, how each is tested, and the outcome. Supported: significant in the predicted direction. Partly: the predicted direction, not yet significant, or significant only on some measures.

C2. The reward stays calibrated under policy shift (§ 5.3). On 25,600 on-policy samples, its calibration error is half that of the same 27B backbone read through softmax. It orders correct above incorrect answers more often in every difficulty bucket, and its reward–truth gap stays stable across training: no hacking.

C3. It rewards reasoning, not just answers (§ 5.5). Without any step-level training, it beats a dedicated 7B process reward model at locating the first error (ProcessBench, 20/20 splits). On our own policy’s failed attempts, re-solving from the step it flags shows that it finds real errors (a random step shows none).

C4. Decision probabilities are better reward probabilities than softmax ones, given equal labels (§ 5.6). We compare with proper scoring rules (Brier, log-loss) after giving every method the same calibration labels. The decision readout is better on 3 of 4 reward benchmarks in all 20 splits and reaches its best quality with about 100 labels.

C5. We map where the reference-free reward stops being enough (§ 5.4). On the 1,758 training problems the base model mostly fails, the reference-free reward still improves the policy but trails the answer key. On-policy, its probabilities become overconfident there (mean 0.28 against a true accuracy of 0.17). Its step credit is most useful added to the answer key.

Figure 3 shows how the experiments fit together.

The experimental program

Figure 3. The experimental program. Stage 1 asks whether the decision model is a good reward on fixed data (E1, E2, E4). Stage 2 asks whether training with it improves a real model, with controls that isolate the reward (E6). Stage 3 tests the regime where the answer key gives no signal (E6b). The boxes at the bottom name the hypotheses (Table 1) that each experiment tests.

Related Work

We cover the four lines of work closest to ours.

LLM judges and generative reward models. LLM judges [32, 82] and AI feedback [3, 27, 74] made model-based rewards cheap to build. Generative verifiers [2, 5, 35, 68, 77], rubric rewards [18, 64, 86] and model-based answer checkers [39, 58, 72] carry RL beyond tasks with an exact checker. Frontier post-training now relies on them. DeepSeek-GRM writes principles and critiques and votes over several samples [35]. DeepSeek-V4 drops scalar reward models for a rubric-guided generative reward model that the policy itself serves as [11]. Kimi K2 uses a self-critique rubric reward based on pairwise comparisons [22], and Kimi K3 an agentic judge that writes a rubric and ranks candidates in a tournament of binary comparisons [23]. None of these reports whether its judge’s scores are calibrated, and the Kimi K2 report names finer handling of calibrated uncertainty as future work [22]. They also generate text, often several samples per answer; our reward is read in one forward pass.

Scalar reward models and overoptimization. Bradley–Terry reward models [43, 57] remain the default, with strong recent instances [36, 66]. Overoptimization [17], length bias [4, 55] and imperfect ensembles [8, 13] motivated uncertainty-aware and distributional reward models [6, 12, 37, 59, 70, 76]. Reward benchmarks measure accuracy [14, 25, 33, 41], but accuracy alone predicts RL outcomes poorly [48, 67], and RL gains on Qwen models can appear even with random rewards [51]. We therefore evaluate rewards by training with them, against a random-reward floor and an answer-key ceiling.

Process reward models. Process supervision began with step-labelled reward models [31, 63] and moved to automatic labels from Monte Carlo roll-outs [38, 49, 65] and implicit process rewards [9, 73]. ProcessBench and PRMBench showed that PRMs often miss the first error [56, 84], and Monte Carlo labels bias them [78]. Generative “thinking” verifiers are more accurate at the price of long generations [21, 52, 80]. Used as RL rewards, PRMs are hacked [7, 10]; recent fixes co-train or gate the verifier [1, 75, 87]. Our step reward comes from a general decision model with no step training.

Judge calibration. LLM judges read through their softmax are overconfident [62], sensitive to option order [54, 83] and attackable [81]. Order effects can be corrected [19, 79, 85], and confidence can be improved by verbalized estimates [61], linear probes [47] or distribution matching [29]. Reward miscalibration passes into the trained policy [28]. The only calibrated PRMs we know of are calibrated post hoc for one policy and used at inference time [40, 44]. Probability readouts have been used as RL rewards before [6, 16, 27, 30, 53, 58, 69, 77], but none measures calibration on the policy’s own samples during RL (Table 2). As far as we can determine, the combination is new: a debiased, label-calibrated, typed multi-option decision readout used as an online RL reward for both outcomes and steps, evaluated with equal-label, on-policy and random-reward controls.

WorkCalibratedDebiasedTyped / multi-optionStep-levelOnline LLM RL
d-RLAIF [27]✗labels onlyscore scale✗✓
GenRM [77]✗✗binary✗✗
Reward Bridge [58]✗✗binary✗✓
CCR [53]✗✗binary✗✓
Probe judges [69]not measuredn/abinary✗✓
OPRM [6]✓✗score scale✗✗
Calibrated PRM [44]✓ (post hoc)✗binary✓✗
RLJF [16]✗✗partly✗✓
RLCDR (ours)✓ (on-policy)✓✓✓✓

Table 2. Probability readouts used as RL rewards. ✓ = present, ✗ = absent. Compiled from the cited papers.

RLCDR in one picture

Figure 2. RLCDR in one picture. Each policy answer is wrapped in a typed question. The decision model returns a distribution over the options in one forward pass. Averaging over option orders removes position bias, and a two-parameter fit on a small labeled set makes the probability calibrated. The calibrated probability is the reward.

Method

Decision model. The decision model (FxAI-DM, built by FuturixAI) is a 26.1B-parameter model on a Qwen3.8-27B backbone with a trained readout head. Given a state (e.g. a problem and a response), an instruction and up to 255 named options, the head reads the last hidden state and outputs a categorical distribution p(⋅∣x)p(\cdot\mid x) over the options. No text is generated. The readout was trained for general decision making, not on reward-model data.

Typed questions. We use two questions, given verbatim in Section A. The outcome question asks whether the response’s final answer is correct, with options {correct, incorrect}. It gives pok(x)p_{\mathrm{ok}}(x). The critic question presents the response split into steps and asks which step contains the first error, with options {none, step 1, …\ldots, step nn}. It gives a distribution qq over the location of the first error.

Debiasing. Readouts of multiple-choice questions depend on option order [45, 83]. We ask every question in each option order (forward and reversed for the questions here) and average the probability assigned to each option.

Calibration. We map the debiased probability through a Platt fit, r=σ(a⋅logit⁡(pok)+b)r = \sigma(a \cdot\operatorname{logit}(p_{\mathrm{ok}}) + b). The two parameters are fitted once, by maximum likelihood, on 1,024 labelled policy responses to MATH problems from PPE [14], collected from other models. They are then frozen during RL. The resulting reward is reference-free: it never sees the answer key.

Outcome reward and step reward. The outcome reward is rout=rr_{\mathrm{out}} = r. From the critic distribution qq, the progress of a response with nn steps is the expected fraction of steps that come before the first error:

prog⁡(x)=q(none)+∑k=1nq(step k)k−1n.(1)\operatorname{prog}(x) = q(\mathrm{none}) + \sum_{k=1}^{n} q(\mathrm{step}\ k)\frac{k-1}{n}. \tag*{(1)}

The hard-problem experiments (§ 5.4) combine the two. The reference-free step reward replaces the answer key by the decision model’s own outcome probability,

rstep(x)=rout(x)+(1−rout(x))λprog⁡(x),λ=0.5,(2)r_{\mathrm{step}}(x) = r_{\mathrm{out}}(x) + (1-r_{\mathrm{out}}(x))\lambda\operatorname{prog}(x), \qquad\lambda= 0.5, \tag*{(2)}
Exp.Data (size)ComparedLabelsMain metric
E1PPE best-of-K (5,120), JudgeBench (620), RM-Bench (3,981), RewardBench 2 (1,723)FxAI-DM, softmax, Skywork-V2-8B0–1,600Brier, NLL, AUROC
E2Same as E1, hidden states saved6 head variants25–1,600Brier
E4ProcessBench (3,400 solutions, 16,548 steps)FxAI-DM, softmax, Qwen2.5-Math-PRM-7B0–400First-error F1
E6MATH train (5,584) → MATH500, AIME24/255 rewards (§ 4)1,024 (frozen)pass@1, pass@kk
E6b1,758 hard MATH problems; 4,451 + 800 policy answersFxAI-DM only; answer key ± step credit1,024 (frozen)pass@1, re-solve drop

Table 3. Every experiment at a glance. “Labels” is the number of labelled examples any method may use for calibration. All splits are by question; nothing is fitted on evaluation items.

so an answer judged correct scores near 1 and an answer judged wrong is graded by how far it got. The hybrid reward gives 1 to a correct final answer according to the answer key and λ⋅prog⁡(x)\lambda\cdot\operatorname{prog}(x) otherwise; it tests step credit alone. A 0/1 checker gives exactly zero gradient whenever all answers in a group fail.

Policy optimization. We use GRPO [50] with the Dr. GRPO loss [34]: group-mean baseline, no division by the group standard deviation and no length normalization. The advantage of each sample is therefore ri−rˉr_i-\bar{r}, so a calibrated reward enters the gradient on its own scale.

Experimental Setup

Offline protocol (E1, E2, E4). We use 20 random half splits by question. From the fit half we draw NN labelled items, fit each method’s calibration on them, and evaluate on the other half. Every method gets the same NN items. We report Brier and log-loss (NLL), which are proper scoring rules: they reward a probability that is both calibrated and informative. Expected calibration error (ECE, 10 equal-width bins) is reported but never decides, because after calibration even a constant base-rate predictor has ECE 0. AUROC measures ranking alone.

Policy and data. The policy is Qwen3-1.7B [46] in non-thinking mode. It trains on the MATH training split (levels 3–5, 5,584 problems), with the prompt “Please reason step by step, and put your final answer within \boxed{}.” We removed every training problem whose normalized text matches an evaluation problem.

RL recipe (identical for every run). The recipe uses TRL’s GRPO trainer with vLLM generation:

  • 32 prompts × 8 samples per step; temperature 1.0; up to 3,072 completion tokens.

  • Learning rate 2×10−62 \times10^{-6} (constant); KL coefficient β=10−3\beta= 10^{-3}; 100 steps.

  • fp32 master weights with bf16 compute.

Each run used one NVIDIA RTX PRO 6000 (96 GB) for the policy, plus a second one for the reward model when needed. Section B lists every setting.

Rewards compared. We compare five rewards, and only the reward differs between runs:

  • RLCDR: the calibrated FxAI-DM outcome reward.

  • Softmax judge: the same Qwen3.8-27B backbone, the byte-identical prompt, the same debiasing and the same calibration procedure, but reading the next-token softmax over the option letters (the G-Eval / d-RLAIF readout [27, 32]). It has the same size and knowledge as our reward and differs only in the readout, which isolates the contribution of decision training.

  • Answer key: a math-verify check against the reference answer. This is the ceiling.

  • Random: Bernoulli(0.5). This is the floor, and it catches gains that come from RL itself rather than from the reward [51].

  • Base: no training.

The answer-key check is computed and logged for every run, but has zero weight except in the answer-key run. This lets us measure the reward’s faithfulness on-policy.

Evaluation. Evaluation uses reference answers only, never a reward model:

  • MATH500 with 4 samples per problem; AIME 2024 and AIME 2025 with 16 samples each.

  • Temperature 0.6, top-pp 0.95.

  • We report pass@1 (mean accuracy) and pass@kk. “MATH500-clean” drops the 8 problems that also appear in the calibration set.

Differences are paired by problem. We report 95% bootstrap intervals (10,000 resamples) and paired tt-test pp-values. Evaluation itself is noisy: re-evaluating the same checkpoint gave 74.85 and 73.90 on MATH500, because vLLM batching is not deterministic. We therefore treat differences of 1–2 points as unresolved unless their paired interval excludes zero, and we do not claim single significant results on the 30-problem AIME sets, where we run many comparisons.

Results

H1–H2: The policy improves, and the reward is the cause

H1–H2: The policy improves, and the reward is the cause

Figure 1 and Table 4 and Table 5 answer the question that matters most: does a real model get better? It does. Three findings attribute the gain to the reward.

Two-panel plot showing MATH500 pass@1 after 100 RL steps and accuracy during training

Figure 1. The policy improves because of the reward. Qwen3-1.7B trained with GRPO on MATH (training split), changing only the reward. Every evaluation uses reference answers, never the reward model. Left: accuracy after 100 steps, with 95% paired-bootstrap intervals for the gain over the base model. Right: our reward improves the policy at every checkpoint and ends level with the answer key. Random reward does not help. Each evaluated checkpoint carries about ±1.8\pm1.8 points of 95% uncertainty.

Reward used for RLMATH500M500-cleanAIME24AIME25M500 pass@4
None (base model)72.272.311.39.485.2
Random (floor)72.872.812.38.585.8
Softmax judge (same backbone)75.575.512.910.088.4
RLCDR (ours)76.876.916.510.888.4
Answer key (uses reference answers)76.977.114.413.189.2

Table 4. Ground-truth evaluation after 100 GRPO steps (pass@1, %; MATH500 pass@4 in the last column). Only the reward differs between rows. Bold: best reward that does not use reference answers.

ComparisonΔ\Delta95% CIpp
Gain over base model
RLCDR − base+4.65[+2.80,+6.50][+2.80, +6.50]<10−4< 10^{-4}
Answer key − base+4.75[+3.00,+6.55][+3.00, +6.55]<10−4< 10^{-4}
Softmax − base+3.35[+1.55,+5.15][+1.55, +5.15]3×10−43 \times10^{-4}
Random − base+0.65[−1.30,+2.50][-1.30, +2.50]0.50
What causes the gain
RLCDR − random+4.00[+2.05,+6.05][+2.05, +6.05]10−410^{-4}
RLCDR − answer key−0.10[−1.75,+1.50][-1.75, +1.50]0.90
RLCDR − softmax+1.30[−0.50,+3.10][-0.50, +3.10]0.16

Table 5. Paired differences on MATH500 after 100 steps (points; 500 problems ×\times 4 samples). The second block isolates the causes.

(1) The gain is real and large. RLCDR improves MATH500 by +4.65 points (p<10−4p < 10^{-4}). The gain is the same on the clean subset (+4.62), so it does not come from overlap with the calibration data. Accuracy rises at each checkpoint we evaluated (72.2 →\to 73.9 →\to 73.9 →\to 75.6 →\to 76.8; Figure 1, right). pass@4 also rises (85.2 →\to 88.4): the model solves more distinct problems, rather than merely sharpening answers it could already find.

(2) The reward causes it, not RL itself. The random-reward run uses the same data, steps and optimizer and gains only +0.65 (n.s.). Our reward beats it by +4.0 points (p=10−4p = 10^{-4}). This rules out the explanation that RL alone improves Qwen math models [51].

(3) It matches the answer key without using it. RLCDR and the answer-key run are statistically identical (−0.10, 95% CI [−1.75,1.50][-1.75, 1.50]). The decision model never sees the reference answer, so this level of training signal is available on problems that have no reference answer, which is where RL most needs it.

Against the softmax judge, the policy-level comparison is unresolved. Both judges improve the policy. Our reward is ahead on MATH500 (+1.30, 95% CI [−0.50,+3.10][-0.50, +3.10], p=0.16p = 0.16), which is within evaluation noise for one seed, so we make no claim about which of the two trains a better policy. As rewards, the two differ clearly: ours is better calibrated and ranks answers better on the policy’s own samples (§ 5.3) and with equal labels offline (§ 5.6). § 5.3 explains why GRPO turns only part of that difference into policy accuracy.

Our reward teaches what the answer key teaches. The two runs that improved most also changed the policy in the same way (Table 6). Both the answer-key run and RLCDR learned to reason at greater length, raising mean MATH500 response length from 837 to about 1,050 tokens. The softmax and random runs did not. Because the behaviour appears under the ground-truth reward too, it reflects learning, not exploitation of our reward.

BaseRandomSoftmaxRLCDRAnswer key
MATH500 mean tokens8378638561,0561,049
MATH500 truncated4.3%5.0%4.0%9.7%9.2%
AIME24 truncated37%32%37%62%64%

Table 6. Response length after 100 steps. Mean MATH500 tokens; share of responses truncated at 3,072 tokens.

Where the gain comes from. Figure 4 shows every paired gain over the base model with its 95% interval, and Figure 5 splits MATH500 by the dataset’s difficulty level. Three things stand out.

Gains over the base model for MATH500, MATH500 levels 4–5, and AIME24+25

Figure 4. Every gain over the base model, with 95% paired-bootstrap intervals (after 100 steps; number of problems in brackets). An interval that does not cross the black zero line is a significant gain. Our reward and the answer key are indistinguishable on every slice; random reward is indistinguishable from no training on every slice.

MATH500 gain over base by difficulty level, with bars for Random, Softmax judge, RLCDR (ours), and Answer key across Levels 1–5

Figure 5. MATH500 gain over base by difficulty level (after 100 steps). Our reward’s gain rises with difficulty, from about +3 points on levels 2–3 to +6.9 on level 5 (44.8% → 51.7%), tracking the answer key. Level 1 has only 43 problems that the base model already solves 96% of the time, so its differences (±2 points) are noise.

  • The gain grows with difficulty. Our reward adds +2.6 points on levels 1–3 (p=0.02p = 0.02), +6.5 on levels 4–5 (p<10−4p < 10^{-4}) and +6.9 on level 5 alone (p=0.002p = 0.002). The answer-key run shows the same pattern. Gains on easy problems are capped, since the base model already solves 88% of them.

  • Random reward does not produce this pattern. Its gains are inside the noise on every slice (levels 4–5: +2.6, p=0.08p = 0.08; levels 1–3: −1.5).

  • AIME moves in the same direction but is noisier. Pooled over the 60 AIME problems, our reward gains +3.3 points (p=0.09p = 0.09) and the answer key +3.4 (p=0.02p = 0.02); the difference between them is −0.1 (p=0.94p = 0.94). The softmax judge gains +1.2 (p=0.50p = 0.50), random +0.1.

The model also solves new problems rather than only sharpening old ones: with our reward, 31 MATH500 problems that the base model never solved in 4 attempts become solved, and 15 are lost (answer key: 32 and 12; random: 25 and 22).

Training dynamics. Each run logs the answer-key correctness of every training sample (with zero weight, except in the answer-key run). Figure 6 shows this training accuracy over the 100 steps. All runs see the same prompts in the same order, so the batch-to-batch ups and downs are shared. The three informative rewards climb together, while the random-reward run falls 2–5 points below them from step 20 onward. The policy is learning from the reward signal throughout training, not only at the end.

Training-sample accuracy over RL steps for RLCDR (ours), Answer key, Softmax judge, and Random

Figure 6. Accuracy of the policy’s own training samples during RL (256 samples per step, averaged over 10 steps, labelled by the answer key but never shown to the non-answer-key rewards). Every informative reward pulls away from the random-reward run after about 20 steps. Note: training accuracy is measured on training problems; Table 4 gives held-out results.

The reward stays honest while the policy changes

A reward that is accurate on a fixed benchmark can still fail during RL: the policy drifts to new kinds of answers and may find ones the reward over-scores. Because every run logs the true correctness of every sample, we can measure the reward’s quality on-policy, over all 25,600 samples per run.

Calibration survives policy shift. On-policy, RLCDR has ECE 0.074 and Brier 0.077, against 0.141 and 0.106 for the softmax judge on the same backbone (Figure 7a). It also separates correct from incorrect answers better (AUROC 0.956 vs. 0.943). This is not circular. Both rewards were calibrated once, on PPE answers written by four other models (Gemma-2-9B, Llama-3-8B, GPT-4o-mini, Claude-3-Haiku), and then frozen. The on-policy numbers are measured on Qwen3-1.7B’s own, changing answers, which neither reward saw during calibration. The decision readout carried its calibration onto that new policy.

Three-panel plots showing on-policy calibration, no drift or hacking, and better ranking at every level

Figure 7. The reward measured on the policy’s own samples during RL (25,600 samples per run, labelled with the answer key, which neither reward saw). (a) Our reward’s probabilities track the true accuracy of the policy’s answers about twice as closely. (b) The gap between reward and truth stays flat over training. Our reward is slightly cautious (never overpaying), while the softmax judge undervalues correct answers about twice as much. (c) Within each GRPO group, our reward ranks a correct answer above a wrong one more often at every difficulty level. This within-group ordering is what drives GRPO’s update.

No reward hacking. The reward–truth gap of RLCDR stays between −0.04 and −0.06 for all 100 steps (Figure 7b). Its average reward never exceeded the policy’s true accuracy, and response length grew without any rise in reward. The softmax judge stays about twice as far from the truth (−0.09 to −0.13).

Better within-group ranking. GRPO learns from differences within a group of answers to the same prompt. In every group size where some answers are right and some wrong, RLCDR ranks right above wrong more often (Figure 7c). This includes the hardest case, groups with only one correct answer.

There, RLCDR ranks that answer highest 59.3% of the time, against 52.3% for the softmax judge (12.5% by chance), and pushes 47% more advantage mass toward it.

Why a better reward only partly becomes a better policy. 57% of all training groups had either all 8 answers right or all 8 wrong. In those groups both rewards give the policy almost no usable signal. The ranking advantage in Figure 7c therefore applies to only a minority of updates. This points to the regime where calibrated decision rewards should matter most: hard problems.

H2, H4, H6: Hard problems, where the answer key is silent

On a problem where every sampled answer is wrong, a 0/1 answer checker returns all zeros and GRPO learns nothing. On the 5,584 MATH training problems, the base policy solves 0 of 8 samples on 823 problems and at most 3 of 8 on 1,758. A step-level reward can still tell a nearly-right attempt from a hopeless one.

Does the critic find real errors in our policy’s answers? We asked both 27B judges the critic question on 4,451 policy answers to these hard problems. On these answers the decision readout separates correct from incorrect solutions much better (AUROC of q(none)q(\text{none}): 0.781 vs. 0.698). It also wrongly declares “no error” in a failed solution less often (9.6% vs. 14.4%).

To check localization without human labels, we use roll-outs in the style of Math-Shepherd [65]. For 800 failed answers, the base policy re-solves the problem four times from just before the step the judge flags, and four times from just after it. If the flagged step really is the error, success should drop once it is included.

Figure 8 shows that it does. Re-solving succeeds about half as often once the step FxAI-DM flags is included (0.133 → 0.069), while a random step makes no difference. The effect is also larger than for the softmax judge (paired difference +0.016, 95% CI [0.001, 0.031]). On these failed attempts the critic’s progress score is spread out (mean 0.51, s.d. 0.19), so it gives GRPO a usable signal exactly where the answer key gives none.

Bar chart showing drop in re-solve success for random step, softmax judge, and RLCDR (ours), with a comparison table of before, after, and drop values

Figure 8. The decision model finds the step where a failed solution goes wrong. Including the step it flags roughly halves the chance of recovering, whereas a random step has no effect. The effect is 45% larger than for the softmax judge, and the prefix our model keeps before the flagged step is more useful (13.3% vs. 10.6% recovery).

RL on hard problems. We trained three policies on the 1,758 hard problems for 50 steps with the recipe of § 5.2, changing only the reward (Table 7). The first uses the answer key alone. The second uses the reference-free step reward of § 3, which never sees the answer key. The third, the hybrid, adds FxAI-DM step credit to the answer key for failed attempts. The second run asks whether the decision model can replace the answer key where the answer key is least informative. The third asks whether step credit adds signal that the answer key lacks.

Answer keyFxAI-DM only (reference-free)Answer key + FxAI-DM step credit
Score of an answer1 if correct, else 0rout+(1−rout)0.5 progr_{\mathrm{out}} + (1-r_{\mathrm{out}}) 0.5\ \mathrm{prog}1 if correct, else 0.5 prog
Group with all 8 answers wrongno gradientgradedgraded
Uses the reference answeryesnoyes (0/1 part only)

Table 7. What each hard-problem reward sees. routr_{\mathrm{out}} is the calibrated FxAI-DM probability that the final answer is correct; prog is the critic’s progress score (§ 3).

Policy (50 steps, hard problems only)MATH500Level 5AIME24AIME25
Base (Qwen3-1.7B)72.244.811.39.4
Answer key75.350.614.89.2
FxAI-DM only (reference-free)74.148.913.110.0
Answer key + FxAI-DM step credit75.752.112.913.1
FxAI-DM only − base+1.9 [+0.2, +3.8]+4.1 [−0.2, +8.4]+1.9+0.6
FxAI-DM only − answer key−1.2 [−2.9, +0.5]−1.7 [−5.8, +2.4]−1.7+0.8
Hybrid − answer key+0.4 [−1.3, +2.1]+1.5 [−2.4, +5.4]−1.9+4.0 [−0.2, +9.0]

Table 8. RL on hard problems (pass@1, %, after 50 steps; paired 95% bootstrap CIs; level 5 has 134 MATH500 problems). Pooled over both AIME sets, only the hybrid policy improves significantly on base (+2.7 [+1.0, +4.7], p=0.006p = 0.006; answer key +1.7, p=0.26p = 0.26; FxAI-DM only +1.2, p=0.24p = 0.24).

Without the answer key, the policy improves but trails. Trained without any reference answer, the policy still improves: +1.9 MATH500 points over base ([+0.2, +3.8], p=0.03p = 0.03) and +4.1 on level 5 (p=0.07p = 0.07). It trails the answer-key policy by 1.2 points on MATH500 ([−2.9, +0.5], p=0.17p = 0.17) and by 0.4 on pooled AIME (p=0.72p = 0.72). Neither gap is significant with one seed. But the direction is consistent, and accuracy on the training problems agrees (Figure 9b: 19.6% vs. 23.0% over the last 10 steps). We therefore read it as a real shortfall: on hard problems, the reference-free reward is weaker than the answer key, whereas on the full MATH training set it matched it (§ 5.2).

Held-out accuracy by difficulty and accuracy on hard training problems

Figure 9. Hard-problem RL (50 steps on the 1,758 problems the base model mostly fails). (a) On held-out MATH500, the reference-free reward (dark blue) is above base at every level but below the answer key on levels 2, 4 and 5. Answer key + step credit (light blue) gives the best level-5 accuracy (52.1% vs. 50.6%; base 44.8%). (b) On the training problems, the reference-free run learns more slowly and rises only in the last 20 steps. The hybrid is slightly behind the answer key here, so its held-out advantage is not from fitting the training set better.

Why: on hard problems the reward becomes overconfident. On the policy’s own samples in this run, the decision model still separates right from wrong answers well (AUROC 0.89). Its probabilities, however, are too high: the mean reward probability is 0.28 against a true accuracy of 0.17 (ECE 0.11, against 0.074 on the full training set, where it was slightly under-confident). At a 0.5 threshold it recognized only 65% of correct answers. Correct answers therefore received an average reward of 0.73 instead of 1, and wrong answers 0.41. That margin is far smaller than the answer key’s 1 against 0. The two Platt parameters were fitted on PPE answers of mixed difficulty, and they do not transfer to a distribution in which 83% of answers are wrong. Recalibrating on about a hundred labelled hard answers is the direct fix (§ 6).

No hacking, and the gain is in reliability. The share of wrong answers judged correct stayed at 9–10% throughout training. The reward on wrong answers was negatively correlated with their length (−0.48), and answers did not grow (833 MATH500 tokens, base 837). pass@4 was unchanged (85.0% vs. 85.2%). The gain is in pass@1: the policy became more reliable on problems it could already solve sometimes, rather than solving new ones. At step 25 it was still at base level (72.3%), so all of the gain came in the second half, and training accuracy was still rising when we stopped.

Step credit works best added to the answer key. The hybrid reward gives the best hard-problem policy. It reaches 75.7% on MATH500 and 52.1% on level 5, +7.3 over base ([+3.0, +11.9], p=0.002p = 0.002) against +5.8 for the answer key (p=0.01p = 0.01). It is also the only run with a significant pooled-AIME gain over base. Head-to-head against the answer key, the differences are not significant (+0.4 MATH500, p=0.65p = 0.65; +1.5 level 5, p=0.46p = 0.46; +1.0 pooled AIME, p=0.50p = 0.50), so we claim parity with a positive trend. During training, 42–51% of the hybrid run’s groups had all answers wrong; these received graded credit instead of zero gradient. Partial credit was negatively correlated with length (−0.4), and the hybrid’s answers were shorter (921 vs. 981 tokens; 5.7% vs. 7.8% truncated). On the training problems themselves the answer-key run is slightly ahead (23.0% vs. 21.6% over the last 10 steps), so the held-out advantage does not come from fitting the training set better.

H4: Step-level rewards, better than a dedicated process reward model

Table 9 evaluates step rewards on ProcessBench [84]. Our reproduction of Qwen2.5-Math-PRM-7B [78] scores 73.0 on average (published: 73.5). The decision model asks “is step kk correct?” over the prefix, as a PRM would during RL. It outperforms the PRM, which was trained for exactly this task, by +5 F1 on average and in 20/20 splits. The margin is largest on the hardest sets (Olympiad +7.7, Omni-MATH +9.2). Its step probabilities are also better: AUROC 0.972 vs. 0.954, and Brier 0.041 vs. 0.051 after calibration. Calibration lowers the decision model’s step-level ECE from 0.073 to 0.011 with 100 labels. The one-question critic used for our RL reward reaches 0.764 zero-shot in a single pass per solution, against 0.696 for the same question read through softmax.

MethodLabelsGSM8KMATHOlympiadOmni-MATHAverage
Softmax judge (same 27B backbone)1000.7250.7210.6440.6370.682
Qwen2.5-Math-PRM-7B (step-trained)1000.8110.7730.6590.6450.722
FxAI-DM decision (ours)1000.8160.7990.7360.7370.772
FxAI-DM decision, zero-shot00.8580.7990.7050.7160.770
FxAI-DM one-question critic, zero-shot00.8610.7890.7190.6870.764
Softmax one-question critic, zero-shot00.7900.7310.6550.6100.696

Table 9. Finding the first wrong step (ProcessBench F1, 3,400 human-labelled solutions; 20 random half splits; “labels” = solutions used to fit the calibration and the cutoff). FxAI-DM was never trained on step labels.

H5: Calibrated decisions vs. softmax and scalar rewards, given equal labels

Figure 11 compares reward probabilities on four benchmarks [14, 33, 41, 60], giving every method the same calibration labels and judging with proper scoring rules. After calibration any method can reach low ECE (even a constant base-rate predictor has ECE 0), so ECE alone cannot decide. Brier and log loss reward a probability that is both calibrated and informative.

Equal-label comparison of reward probabilities

Figure 11. Equal-label comparison of reward probabilities. Left: on PPE (one sampled policy answer per item, the RL reward case), the decision readout with zero labels is already better than either baseline with 1,600 labels. Right: the decision readout wins where the question is whether an answer is correct. A preference-trained scalar RM wins on preference-pair benchmarks that match its training format.

The decision readout is the better reward probability for judging correctness. On PPE it reaches AUROC 0.809 vs. 0.709 for softmax and 0.708 for the scalar RM, with the largest margins on MATH (0.877) and IFEval (0.829). It is also about 5× less position-biased than the softmax readout. Without debiasing, the softmax judge loses 5.4 accuracy points on PPE and 8.8 on RM-Bench, while the decision readout loses at most 0.8. The scalar RM’s raw σ(r)\sigma(r) is far from a probability: it predicts 0.90 where the true rate is 0.47 (zero-label ECE 0.455).

One calibration transfers across domains, except code. The MATH fit is almost a pure shift for FxAI-DM (a=1.04a = 1.04), but compresses the softmax judge’s scores by a=0.72a = 0.72. Carried to the other four domains without new labels, it lowers FxAI-DM’s ECE in every one, while it makes the softmax judge worse in all four (Table 10). When FxAI-DM gives more than 0.9, it is right 87–94% of the time on MATH, MMLU-Pro and IFEval. The exceptions are GPQA and especially code, where answers scored above 0.9 are right only 74–76% of the time. We read this as a limit of what the judge can know, not of the readout: a model that reads code without running it cannot see an edge case that fails a hidden test, and graduate-level science questions exceed what a 27B model knows. This is the same overconfidence we find on hard math problems (§ 5.4). It is cheap to fix where it occurs: 100 in-domain labels bring code ECE from 0.142 to 0.090 (better in 19 of 20 splits).

DomainAcc.FxAI-DM ECE rawFxAI-DM ECE MATH-fitFxAI-DM >0.9Softmax judge ECE rawSoftmax judge ECE MATH-fitFxAI-DM 100-label refit
MATH0.360.0840.0480.87 (128)0.1330.062–
MMLU-Pro0.480.0960.0740.94 (224)0.0940.115–
IFEval0.500.0860.0750.91 (197)0.1990.300–
GPQA0.390.1180.0880.76 (49)0.1000.1100.075
MBPP+ (code)0.600.1620.1420.74 (487)0.2110.2330.090

Table 10. One calibration, five domains. PPE, 1,024 policy answers per domain. ECE before calibration (raw) and after the two Platt parameters fitted on MATH only, the fit used for RL (MATH is in-sample). “>0.9”: accuracy among answers the raw readout scores above 0.9 (nn in parentheses). Last column: Platt refit on 100 in-domain labels, mean over 20 splits.

The advantage is in the representation. One might ask whether the decision model wins only because its readout head is better tuned. A head-refit control (E2, Section C) retrains heads with the same labels on the hidden states of both models. A new head on the base Qwen3.8-27B features still stays well behind the decision-model features on PPE, JudgeBench and RM-Bench.

Cost

The decision reward needs one forward pass per question and option order, and generates no text. In E6, scoring 256 answers (2 option orders each) took 146 s of a 340 s training step on a separate GPU, run sequentially after generation; on hard problems the step critic for failed attempts roughly doubled step time (about 8 vs. 4 minutes for the answer key alone), because failed attempts are long, and the reference-free version, which asks both questions of every answer, took about 13.5 minutes per step. In E4 the decision model processed 39,896 step questions in 5,220 s on one GPU (7.6 requests/s). A generative judge that samples 8–32 critiques of several hundred tokens per answer would need one to two orders of magnitude more decoding. Overlapping reward scoring with generation, or scoring one option order, would cut our reward time further; we have not optimized it.

Bar charts comparing AUROC for policy-answer judging by domain and raw first-slot position bias before debiasing

Figure 12. Why the decision readout is the better reward for correctness. (a) It separates correct from wrong policy answers best on MATH, MMLU-Pro and IFEval; the scalar RM is better on code and GPQA, our weakest domains. (b) Before debiasing, the softmax readout’s bias against the first option slot is 4–6× larger than the decision readout’s. Without debiasing the softmax judge loses 5.4 accuracy points on PPE; the decision readout loses at most 0.8.

Summary: each hypothesis against its evidence

Summary: each hypothesis against its evidence

VerdictEvidence forEvidence against / still missing
H1 Supported+4.65 MATH500 (p<10−4p < 10^{-4}); +4.0 over random reward (p=10−4p = 10^{-4}); gains grow with difficulty (Figure 4 and Figure 5)One seed; one policy family; AIME gain +3.3 not significant alone (p=0.09p = 0.09)
H2 Supported on MATH / Partly on hard problems−0.1 vs. answer key on MATH500 (CI [−1.8, 1.5]), −0.1 on AIME; same length changeOn hard problems only, trails the answer key (−1.2 MATH500, p=0.17p = 0.17; lower training accuracy); parity, not superiority
H3 SupportedOn-policy ECE 0.074 vs. 0.141, Brier 0.077 vs. 0.106; flat reward–truth gap; better ranking in all 7 group types (Figure 7)Measured on math only
H4 SupportedProcessBench 0.772 vs. 0.722 (PRM), 20/20 splits; re-solve drop 0.064 vs. 0.044 (softmax), random step −0.006 (Figure 8 and Figure 10)A 72B PRM would likely score higher; math only
H5 Supported / PartlyBetter Brier than same-backbone softmax on 3/4 benchmarks in 20/20 splits, tie on 1; better than scalar RM on PPE and JudgeBench; edge survives head refit (Table 11 and Table 13)Scalar RM wins RM-Bench and RewardBench 2 (preference pairs), code and GPQA
H6 PartlyHybrid: best level-5 accuracy (52.1 vs. 50.6) and only significant pooled-AIME gain over base (+2.7, p=0.006p = 0.006)Head-to-head vs. answer key not significant (+0.4); vs. softmax judge on the full set not significant (+1.3, p=0.16p = 0.16); one seed

Table 12. Verdicts. The strongest piece of evidence for and against each hypothesis of Table 1.

ProcessBench F1 comparison and step probability metrics

Figure 10. Step-level rewards on ProcessBench (100 calibration labels, mean of 20 splits). The decision model, never trained on step labels, beats the dedicated step-trained PRM on every subset, and its lead is largest on the hardest ones (Olympiad +7.7, Omni-MATH +9.2 F1). Its step probabilities also rank better (AUROC) and, after calibration, are the most accurate (Brier).

Limitations and Next Steps

  • One seed, one policy, math only. The RL results use a single seed of Qwen3-1.7B on math. Our largest effects (+4 to +7 points over base and random) are well outside evaluation noise. The smaller head-to-head differences, between our reward and the softmax judge as RL rewards and between the hard-problem rewards, are unresolved and need three seeds per arm. Until then we make no policy-level claim between them.

  • The reference-free reward is weaker on hard problems. Trained only on problems the policy mostly fails, it trails the answer key, and its probabilities are overconfident (mean 0.28 against a true accuracy of 0.17). Recalibrating on a hundred labelled hard answers, and training beyond 50 steps (accuracy was still rising), are the next experiments.

  • A large judge trains a small policy. The reward model (27B) is much larger than the policy (1.7B). It never generates text or sees a reference answer, and the same-size softmax judge controls for size and knowledge. It cannot rule out that much of the benefit comes from a strong judge teaching a weak policy, though. Larger and non-Qwen policies are needed.

  • Missing RL baselines and domains. We have not yet trained with a scalar reward model or a generative judge as the reward, or on a second domain such as instruction following (where the decision model already leads offline: IFEval AUROC 0.83 vs. 0.69). On preference-pair benchmarks the scalar RM is better (Table 11), so fine-tuning the decision model on reward data is a natural next step.

BenchmarkMethodAcc↑AUROC↑Brier↓NLL↓ECEWins
PPE (policy answers)FxAI-DM decision0.7350.8090.1840.5620.058–
Softmax (same backbone)0.6580.7090.2220.6420.0590/20
Scalar RM (Skywork-V2-8B)0.6680.7080.2170.6360.0730/20
JudgeBenchFxAI-DM decision0.780–0.2970.4460.066–
Softmax (same backbone)0.708–0.3730.5560.0670/20
Scalar RM0.744–0.3520.5230.0631/20
RM-BenchFxAI-DM decision0.891–0.1600.2650.030–
Softmax (same backbone)0.848–0.2190.3420.0360/20
Scalar RM0.951–0.0790.1440.02220/20
RewardBench 2FxAI-DM decision0.852–0.2140.4270.041–
Softmax (same backbone)0.847–0.2150.4180.0337/20
Scalar RM0.894–0.1530.2970.03020/20

Table 11. Table 11. Full equal-label comparison (N=100N = 100 calibration labels, mean over 20 held-out splits). “Wins” is the number of splits in which the method’s Brier beats FxAI-DM. AUROC applies to pointwise PPE only.

  • Truncation. With the 3,072-token limit, over 60% of AIME answers from the two best full-set runs were cut off. Longer budgets would likely raise their AIME scores.

  • Context length. FxAI-DM accepts at most 8,192 tokens, and we make no long-context claims.

Conclusion

A reward is only as useful as the trust one can place in its numbers. We showed that an LLM converted into a decision model, debiased and calibrated with about a hundred labels, gives a reward whose probabilities stay honest while the policy trains on it. Without ever seeing a reference answer, it improved a real model by +4.65 MATH500 points, the same as the answer key. Random-reward and same-backbone controls tie that gain to the reward. The gain grows with problem difficulty, reaching +6.9 points on the hardest MATH500 level. The same model locates reasoning errors better than a dedicated process reward model. On problems the policy mostly fails, the reference-free reward still helps but trails the answer key, because its calibration does not hold there; its step credit is then most useful added to the answer key. Calibrating for hard problems is the clearest next step. Calibrated decision rewards are a practical, cheap and auditable way to bring the success of RL beyond the tasks that come with an answer key.

Availability

FxAI-DM is public under the Apache-2.0 license [15]. All benchmarks, the policy (Qwen3-1.7B) and the training data (MATH) are public. The typed questions are given verbatim in Section A and all hyperparameters in Section B. Our training and evaluation code, the reward server, the hard-problem split, and every rollout, reward and per-problem evaluation result will be released at https://github.com/rudransh2004/rlcdr.

Typed Questions (Verbatim)

Outcome question (RL reward). The state is “Task given to an AI assistant: {problem} \n\n The assistant’s response: {response}”. The instruction is “Is the assistant’s final answer to this math problem correct?”. The options are correct: “The final answer is correct” and incorrect: “The final answer is wrong, missing, or not stated”. Both option orders are asked, and the reward is the calibrated mean of P(correct).

Critic question (step reward). The response is split at blank lines (merged to at most 30 steps) and shown as “Step 1: …”. The instruction is “Find the earliest step that contains an error (a calculation error, a logical error, a misreading of the problem, or an unjustified or false claim). If every step is correct, choose that no step contains an error.” The options are none: “No step contains an error” and step k: “Step k is the first step with an error”. They are asked in forward and reversed order.

Hyperparameters

PolicyQwen3-1.7B, non-thinking chat template
AlgorithmGRPO, Dr. GRPO loss, no reward standardization (TRL)
Prompts ×\times samples per step32×832 \times8
Steps100 (checkpoints every 25)
Learning rate2×10−62 \times10^{-6}, constant, no warm-up
KL coefficient β\beta10−310^{-3}
Max completion3,072 tokens; sampling temperature 1.0
Precisionfp32 master weights, bf16 autocast, gradient checkpointing
GenerationvLLM (colocated), memory fraction 0.35, max length 4,096
Calibration (RLCDR)Platt a=1.036a = 1.036, b=−0.556b = -0.556 (1,024 PPE-MATH items)
Calibration (softmax judge)Platt a=0.718a = 0.718, b=−0.799b = -0.799 (same items)
EvaluationMATH500 n=4n=4; AIME24/25 n=16n=16; T=0.6T=0.6, top-pp 0.95; math-verify
HardwareNVIDIA RTX PRO 6000 Blackwell (96 GB): one GPU for the policy, one for the reward

Table 16.

Head-Refit Control (E2)

One might ask whether the decision model only wins because its readout head is better tuned. We therefore retrained heads with the same labels on both models’ hidden states (Table 13). The base Qwen3.8-27B features improve with a new head (PPE Brier 0.217 → 0.198) but stay well behind the decision-model features (0.177) on PPE, JudgeBench and RM-Bench. The two tie on RewardBench 2 (0.185 vs. 0.191). The shipped decision head with two Platt parameters is already within 0.01 Brier of a fully refitted head on correctness tasks, so two parameters are enough there. A refitted head does not close the gap to the scalar RM on preference benchmarks; that needs fine-tuning on preference data.

Features + headPPEJudgeBenchRM-BenchRewardBench 2
FxAI-DM + Platt (shipped head, 2 parameters)0.1790.3280.1570.212
FxAI-DM head refit0.1780.3180.1550.185
FxAI-DM new head on features0.1770.3220.1540.193
Base Qwen3.8-27B, new head (probe judge)0.1980.3660.1860.191
Skywork-Reward-V2-8B + Platt0.2100.3720.0780.153

Table 13. Head refit with equal labels (held-out Brier↓\downarrow; 1,600 labels for PPE and RM-Bench, 400 for JudgeBench and RewardBench 2).

Full RL Results

RewardStep 50Step 100
M500AIME24AIME25M500AIME24AIME25AIME24 p@16
Base72.211.39.472.211.39.430.0
Random71.214.08.572.812.38.526.7
Softmax74.812.710.075.512.910.036.7
RLCDR73.914.28.876.816.510.840.0
Answer key74.412.913.376.914.413.133.3

Table 14.

At step 50 no reward-based method is significantly different from another. A separate evaluation of the same RLCDR step-50 checkpoint gave 74.85 on MATH500, which shows evaluation noise of about ±1\pm1 point.

Hard-problem runs (E6b) at both checkpoints.

RewardStep 25Step 50
M500AIME24AIME25M500AIME24AIME25
Base72.211.39.472.211.39.4
Answer key75.012.511.775.314.89.2
FxAI-DM only (reference-free)72.311.39.474.113.110.0
Answer key + FxAI-DM step credit71.911.711.775.712.913.1

Table 15.

Both runs that use FxAI-DM start more slowly than the answer key and gain between steps 25 and 50. The reference-free step-25 checkpoint matches base on average, but its per-problem results differ from base.

References

  1. [1]Agrawal et al. VeriGate: Verifier-gated step-level supervision for GRPO, 2026.arxiv.org/abs/2605.30451
  2. [2]Ankner et al. Critique-out-loud reward models, 2024.arxiv.org/abs/2408.11791
  3. [3]Bai et al. Constitutional AI: Harmlessness from AI feedback, 2022.arxiv.org/abs/2212.08073
  4. [4]Chen et al. ODIN: Disentangled reward mitigates hacking in RLHF, 2024.arxiv.org/abs/2402.07319
  5. [5]Chen et al. RM-R1: Reward modeling as reasoning, 2025.arxiv.org/abs/2505.02387
  6. [6]Chen et al. Learning ordinal probabilistic reward from preferences, 2026.arxiv.org/abs/2602.12660
  7. [7]Cheng et al. Stop summation: Min-form credit assignment is all process reward model needs for reasoning, 2025.arxiv.org/abs/2504.15275
  8. [8]Coste, Anwar, Kirk, and Krueger. Reward model ensembles help mitigate overoptimization, 2023.arxiv.org/abs/2310.02743
  9. [9]Cui et al. Process reinforcement through implicit rewards, 2025.arxiv.org/abs/2502.01456
  10. [10]DeepSeek-AI. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning, 2025.DOI
  11. [11]DeepSeek-AI. DeepSeek-V4: Towards highly efficient million-token context intelligence, 2026.arxiv.org/abs/2606.19348
  12. [12]Dorka. Quantile regression for distributional reward models in RLHF, 2024.arxiv.org/abs/2409.10164
  13. [13]Eisenstein et al. Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking, 2023.arxiv.org/abs/2312.09244
  14. [14]Frick et al. How to evaluate reward models for RLHF, 2024.arxiv.org/abs/2410.14872
  15. [15]FuturixAI. FxAI Decision Model. Hugging Face model card, https://huggingface.co/Futurix-AI/FxAI-decision-model, 2026.
  16. [16]Gallego. Reinforcement learning from Jev feedback. Hugging Face community article, https://huggingface.co/blog/jgallego/rljf, 2026. Not peer-reviewed.
  17. [17]Gao, Schulman, and Hilton. Scaling laws for reward model overoptimization, 2022.arxiv.org/abs/2210.10760
  18. [18]Gunjal et al. Rubrics as rewards: Reinforcement learning beyond verifiable domains, 2025.arxiv.org/abs/2507.17746
  19. [19]Holtzman et al. Surface form competition: Why the highest probability answer isn’t always right, 2021.arxiv.org/abs/2104.08315
  20. [20]Kadavath et al. Language models (mostly) know what they know, 2022.arxiv.org/abs/2207.05221
  21. [21]Khalifa et al. Process reward models that think, 2025.arxiv.org/abs/2504.16828
  22. [22]Kimi Team. Kimi K2: Open agentic intelligence, 2025.
  23. [23]Kimi Team. Kimi K3: Open frontier intelligence, 2026.arxiv.org/abs/2607.24653
  24. [24]Korbak, Perez, and Buckley. RL with KL penalties is better viewed as bayesian inference, 2022.arxiv.org/abs/2205.11275
  25. [25]Lambert et al. RewardBench: Evaluating reward models for language modeling, 2024.
  26. [26]Lambert et al. Tulu 3: Pushing frontiers in open language model post-training, 2024.arxiv.org/abs/2411.15124
  27. [27]Lee et al. RLAIF vs. RLHF: Scaling reinforcement learning from human feedback with AI feedback, 2023.arxiv.org/abs/2309.00267
  28. [28]Leng et al. Taming overconfidence in LLMs: Reward calibration in RLHF, 2024.arxiv.org/abs/2410.09724
  29. [29]Li et al. Judging with confidence: Calibrating autoraters to preference distributions, 2025.arxiv.org/abs/2510.00263
  30. [30]Li et al. Reference-bound verification for multi-subject image generation, 2026.arxiv.org/abs/2610.09328
  31. [31]Lightman et al. Let’s verify step by step, 2023.arxiv.org/abs/2305.20050
  32. [32]Liu et al. G-Eval: NLG evaluation using GPT-4 with better human alignment, 2023.arxiv.org/abs/2303.16634
  33. [33]Liu et al. RM-Bench: Benchmarking reward models of language models with subtlety and style, 2024.arxiv.org/abs/2410.16184
  34. [34]Liu et al. Understanding R1-Zero-like training: A critical perspective, 2025.arxiv.org/abs/2503.20783
  35. [35]Liu et al. Inference-time scaling for generalist reward modeling, 2025.arxiv.org/abs/2504.02495
  36. [36]Liu et al. Skywork-Reward-V2: Scaling preference data curation via human-AI synergy, 2025.arxiv.org/abs/2507.01352
  37. [37]Lou et al. Uncertainty-aware reward model: Teaching reward models to know what is unknown, 2024.arxiv.org/abs/2410.00847
  38. [38]Luo et al. Improve mathematical reasoning in language models by automated process supervision, 2024.arxiv.org/abs/2406.06592
  39. [39]Ma et al. General-Reasoner: Advancing LLM reasoning across all domains, 2025.arxiv.org/abs/2505.14652
  40. [40]Ma et al. Calibrated prediction of future rewards via conditional optimal transport, 2026.arxiv.org/abs/2605.06785
  41. [41]Malik et al. RewardBench 2: Advancing reward model evaluation, 2025.arxiv.org/abs/2506.01937
  42. [42]OpenAI. GPT-4 technical report, 2023.arxiv.org/abs/2303.08774
  43. [43]Ouyang et al. Training language models to follow instructions with human feedback, 2022.arxiv.org/abs/2203.02155
  44. [44]Park et al. Know what you don’t know: Uncertainty calibration of process reward models, 2025.arxiv.org/abs/2506.09338
  45. [45]Pezeshkpour and Hruschka. Large language models sensitivity to the order of options in multiple-choice questions, 2023.
  46. [46]Qwen Team. Qwen3 technical report, 2025.
  47. [47]Radharapu et al. Calibrating LLM judges: Linear probes for fast and reliable uncertainty estimation, 2025.
  48. [48]Razin et al. What makes a reward model a good teacher? an optimization perspective, 2025.arxiv.org/abs/2503.15477
  49. [49]Setlur et al. Rewarding progress: Scaling automated process verifiers for LLM reasoning, 2024.arxiv.org/abs/2410.08146
  50. [50]Shao et al. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models, 2024.arxiv.org/abs/2402.03300
  51. [51]Shao et al. Spurious rewards: Rethinking training signals in RLVR, 2025.arxiv.org/abs/2506.10947
  52. [52]She et al. R-PRM: Reasoning-driven process reward modeling, 2025.arxiv.org/abs/2503.21295
  53. [53]Shen et al. RL-based semi-supervised knowledge distillation with LLM-as-a-judge, 2026.DOI
  54. [54]Shi et al. Judging the judges: A systematic study of position bias in LLM-as-a-judge, 2024.
  55. [55]Singhal et al. A long way to go: Investigating length correlations in RLHF, 2023.arxiv.org/abs/2310.03716
  56. [56]Song et al. PRMBench: A fine-grained and challenging benchmark for process-level reward models, 2025.
  57. [57]Stiennon et al. Learning to summarize from human feedback, 2020.arxiv.org/abs/2009.01325
  58. [58]Su et al. Crossing the reward bridge: Expanding RL with verifiable rewards across diverse domains, 2025.arxiv.org/abs/2503.23829
  59. [59]Sun et al. Probabilistic uncertain reward model, 2025.arxiv.org/abs/2503.22480
  60. [60]Tan et al. JudgeBench: A benchmark for evaluating LLM-based judges, 2024.arxiv.org/abs/2410.12784
  61. [61]Tian et al. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback, 2023.
  62. [62]Tian et al. Overconfidence in LLM-as-a-judge: Diagnosis and confidence-driven solution, 2025.arxiv.org/abs/2508.06225
  63. [63]Uesato et al. Solving math word problems with process- and outcome-based feedback, 2022.arxiv.org/abs/2211.14275
  64. [64]Viswanathan et al. Checklists are better than reward models for aligning language models, 2025.arxiv.org/abs/2507.18624
  65. [65]Wang et al. Math-shepherd: Verify and reinforce LLMs step-by-step without human annotations, 2023.
  66. [66]Wang et al. Interpretable preferences via multi-objective reward modeling and mixture-of-experts, 2024.arxiv.org/abs/2406.12845
  67. [67]Wen et al. Rethinking reward model evaluation: Are we barking up the wrong tree?, 2024.arxiv.org/abs/2410.05584
  68. [68]Whitehouse et al. J1: Incentivizing thinking in LLM-as-a-judge via reinforcement learning, 2025.arxiv.org/abs/2505.10320
  69. [69]Xie et al. Small language models as judges for rubric-based reinforcement learning, 2026.arxiv.org/abs/2608.30005
  70. [70]Yang et al. Bayesian reward models for LLM alignment, 2024.arxiv.org/abs/2402.13210
  71. [71]Yu et al. DAPO: An open-source LLM reinforcement learning system at scale, 2025.arxiv.org/abs/2503.14476
  72. [72]Yu et al. RLPR: Extrapolating RLVR to general domains without verifiers, 2025.arxiv.org/abs/2506.18254
  73. [73]Yuan et al. Free process rewards without process labels, 2024.arxiv.org/abs/2412.01981
  74. [74]Yuan et al. Self-rewarding language models, 2024.arxiv.org/abs/2401.10020
  75. [75]Zha et al. RL tango: Reinforcing generator and verifier together for language reasoning, 2025.arxiv.org/abs/2505.15034
  76. [76]Zhai et al. Uncertainty-penalized reinforcement learning from human feedback with diverse reward LoRA ensembles, 2023.arxiv.org/abs/2401.00243
  77. [77]Zhang et al. Generative verifiers: Reward modeling as next-token prediction, 2024.arxiv.org/abs/2408.15240
  78. [78]Zhang et al. The lessons of developing process reward models in mathematical reasoning, 2025.
  79. [79]Zhao et al. Calibrate before use: Improving few-shot performance of language models, 2021.arxiv.org/abs/2102.09690
  80. [80]Zhao et al. GenPRM: Scaling test-time compute of process reward models via generative reasoning, 2025.
  81. [81]Zhao et al. One token to fool LLM-as-a-judge, 2025.arxiv.org/abs/2507.08794
  82. [82]Zheng et al. Judging LLM-as-a-judge with MT-Bench and chatbot arena, 2023.arxiv.org/abs/2306.05685
  83. [83]Zheng et al. Large language models are not robust multiple choice selectors, 2023.arxiv.org/abs/2309.03882
  84. [84]Zheng et al. ProcessBench: Identifying process errors in mathematical reasoning, 2024.arxiv.org/abs/2412.06559
  85. [85]Zhou et al. Batch calibration: Rethinking calibration for in-context learning and prompt engineering, 2023.arxiv.org/abs/2309.17249
  86. [86]Zhou et al. Breaking the exploration bottleneck: Rubric-scaffolded reinforcement learning for general LLM reasoning, 2025.arxiv.org/abs/2508.16949
  87. [87]Zou et al. ReasonFlux-PRM: Trajectory-aware PRMs for long chain-of-thought reasoning in LLMs, 2025.

Paper details

vlnc:W_yfeu8r3v
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents