betaFeedback

WorksComputer Science

Reinforcement Learning Done Right for Explicit Confidence Targets

Abstract

Large language models (LLMs) can produce plausible but incorrect answers when they are uncertain, which can mislead users and undermine model reliability. Training models to abstain with reinforcement learning is a natural approach, but most existing methods use a fixed error penalty and therefore cannot adapt their behavior to different confidence requirements at inference time. We study explicit confidence targets, where a user specifies a target pp in the prompt and the model directly chooses whether to answer or abstain, with the goal of achieving an accuracy of at least pp among answered questions. However, learning to respond to different confidence targets with a single model has proven difficult, and prior attempts have failed to distinguish different targets. We show that this failure comes from two problems: the reward signal that distinguishes different values of pp is often lost during optimization, while the original error penalty grows without bound as pp approaches one. We propose RLEC (Reinforcement Learning with Explicit Confidence Targets), which preserves the learning signal needed to distinguish confidence targets, introduces a stable reward family that keeps the desired answer condition unchanged, and uses reward scheduling to balance effective learning and training stability. Together, these designs enable the first successful training of a single LLM for explicit confidence target control. Extensive experiments across diverse benchmarks and model scales show that RLEC largely preserves general capabilities and achieves competitive calibration performance. RLEC also supports fine-grained target control without an external decision system or threshold calibration after training. By making explicit confidence targets part of the trained policy, RLEC opens a new and more direct path toward behavioral calibration.

Introduction

Large language models (LLMs) [1, 6, 29] have shown strong abilities in knowledge-intensive and reasoning tasks, but they can still generate plausible yet incorrect responses, commonly known as hallucinations [13, 17]. Such errors reduce the reliability of LLMs and may mislead users, posing particular risks when incorrect answers are costly. Rather than guessing when uncertain, a reliable model should know when not to answer [18, 19, 20].

Kalai et al. [19] argue that common training objectives reward guessing over acknowledging uncertainty. A natural solution is to train abstention with

reinforcement learning [16, 24, 25, 35], rewarding correct answers, penalizing incorrect answers, and assigning no reward to abstentions. For example, TruthRL [35] uses such a reward with a fixed error penalty. However, a fixed penalty represents only one risk preference, while different applications may require different levels of reliability. To represent these requirements, we study an explicit confidence target: the user specifies a target pp in the prompt, and the model directly chooses whether to answer or abstain. The desired behavior is

Acc⁡(p)=Pr⁡(correct∣answer,p)≥p,(1)\operatorname{Acc}(p) = \operatorname{Pr}(\text{correct} \mid\text{answer}, p) \ge p, \tag*{(1)}

and a single model should respond to different values of pp without retraining or an external decision system. However, learning such behavior has proved difficult. Wu et al. [36] propose Explicit Risk Thresholding (ERT), which trains on randomly sampled values of pp using GRPO [10, 27], assigning rewards of 1, −p/(1−p)-p/(1-p), and 0 to correct answers, incorrect answers, and abstentions, respectively. However, they report that the model’s hallucination rates remain insensitive to the input target. They attribute this failure to the noisy optimization problem created by training across many thresholds. Other work [24] further argues that limited model capacity may force a policy to specialize in one risk preference. These findings leave it unclear whether the difficulty is inherent to a single model or caused by the training method.

We show that this failure does not mean that the task is beyond the capacity of a single model. Instead, it can be explained by two problems in the training setup. First, ERT, like many recent RL methods for LLMs, uses GRPO. We observe that most of its rollout groups contain only one or two distinct reward values. Groups with a single reward value yield zero advantages, while in groups with two values, standard deviation normalization removes the effect of the reward scale induced by pp on the advantages. The resulting advantages depend only on the fraction of the two values, rather than their magnitudes. As a result, reward magnitudes provide little direct signal for distinguishing confidence targets. Second, the original error reward grows without bound as pp approaches one. Once the reward magnitudes directly affect the advantages, samples with large pp can dominate the gradient and destabilize training. Based on these findings, we introduce RLEC (Reinforcement Learning with Explicit Confidence Targets). RLEC removes division by the group standard deviation while retaining mean centering, thereby preserving the effect of the reward scale induced by pp on the advantages. It further introduces a reward family controlled by a stability coefficient bb, which controls reward magnitudes at large pp without changing the desired decision rule. Finally, we observe that fixed values of bb tend to trade off the strength of the learning signal against training stability. We therefore introduce a schedule for bb that combines a strong signal for learning different confidence targets with stable training.

We evaluate RLEC using Qwen3-4B-Base and Qwen3-30B-A3B-Base [37] across eight benchmarks covering mathematical reasoning, knowledge and hallucination, and scientific reasoning. Our results show that a single model can successfully learn different behaviors for different explicit confidence targets, as shown in Figure 1(a). We also compare RLEC with a standard RLVR [10] baseline, showing that RLEC largely preserves the baseline’s general capabilities at both model scales. For additional context, we further compare RLEC with RLCR [8], which focuses on confidence reporting rather than direct abstention control. RLEC achieves calibration performance comparable to RLCR at 4B and obtains a lower overconfidence rate and a higher overall score at 30B. RLEC also provides several practical advantages. Methods based on verbalized confidence [8, 30] often require an external rule to convert their confidence estimates into answer or abstention decisions, while other selective prediction methods [20, 32] further rely on separate calibrators or held-out calibration sets. In contrast, RLEC receives pp in the prompt and directly produces the final answer or abstention without external processing. It also supports fine-grained control over answer or abstention, with smooth changes across densely spaced values of pp, as illustrated in Figure 1(b).

Figure 1. Explicit Confidence Target control. (a) RLEC’s average accuracy across eight benchmarks increases with pp specified in the prompt. (b) RLEC’s fine-grained control on AIME with pp.

Our main contributions are as follows:

  • We identify how the optimization algorithm and reward design used in prior work weaken the confidence target signal, providing a concrete explanation for the previously reported failure.

  • We propose RLEC, which preserves the confidence target signal, introduces a stable reward family that keeps the desired answer condition unchanged, and uses reward scheduling to balance effective learning and training stability. To our knowledge, RLEC thereby enables the first successful training of a single LLM for explicit confidence target control.

  • Extensive experiments across diverse benchmarks and model scales show that RLEC largely preserves general capabilities and achieves competitive calibration performance. It also requires no external decision system or calibration after training and supports fine-grained target control.

Background

Fixed-Penalty Abstention

To mitigate hallucinations [13, 17], a reliable model should know when to answer and when to abstain. Given a question xx, the model chooses an action a∈{ANS,ABS}a \in\{\mathrm{ANS}, \mathrm{ABS}\}, where ANS denotes answering and ABS denotes abstaining. We primarily evaluate on verifiable question answering tasks following prior work [8, 35]. This setup yields three outcomes: a correct answer, an incorrect answer, or an abstention. We treat an incorrect answer as a hallucination. We define accuracy among answered questions as

Acc⁡=Pr⁡(correct∣a=ANS).(2)\operatorname{Acc} = \operatorname{Pr}(\text{correct} \mid a = \mathrm{ANS}). \tag*{(2)}

Recent work [16, 24, 35] has used reinforcement learning with verifiable rewards (RLVR) [10, 27] to directly train LLMs to decide whether to answer or abstain. A common formulation assigns a positive reward to a correct answer, a neutral reward to abstention, and a fixed negative reward to an incorrect answer:

Rλ(a,y)={1,a=ANS and y is correct,0,a=ABS,−λ,a=ANS and y is incorrect,λ≥0.(3)R_{\lambda}(a,y) = \begin{cases} 1, & a = \mathrm{ANS}\ \text{and}\ y\ \text{is correct},\\ 0, & a = \mathrm{ABS},\\ -\lambda, & a = \mathrm{ANS}\ \text{and}\ y\ \text{is incorrect}, \end{cases} \qquad\lambda\ge0. \tag*{(3)}

TruthRL [35] optimizes this reward with λ=1\lambda= 1 using Group Relative Policy Optimization (GRPO) [27], whereas Reinforced Hesitation [24] trains separate models for different values of λ\lambda. Let q(x)q(x) be the true conditional probability that the model’s intended answer to xx is correct. Under Equation (3), the expected reward of answering is

E[Rλ∣x,a=ANS]=q(x)−λ(1−q(x))=(1+λ)q(x)−λ.(4)\begin{aligned} \mathbb{E}[R_{\lambda} \mid x, a = \mathrm{ANS}] &= q(x) - \lambda(1-q(x))\\ &= (1+\lambda)q(x) - \lambda. \tag*{(4)} \end{aligned}

Since abstention yields zero reward, the policy answers when

q(x)>λ1+λ,(5)q(x) > \frac{\lambda}{1+\lambda}, \tag*{(5)}

and abstains when the inequality is reversed. Equivalently, λ\lambda determines the fixed confidence threshold

pλ=λ1+λ.(6)p_{\lambda} = \frac{\lambda}{1+\lambda}. \tag*{(6)}

Under the idealized assumptions that q(x)q(x) gives the true probability of correctness and that the policy exactly follows Equation (5), its selective accuracy satisfies

Pr⁡(correct∣a=ANS)=E[q(x)∣a=ANS]=E[q(x)∣q(x)>pλ]≥pλ.(7)\begin{aligned} \operatorname{Pr}(\text{correct} \mid a = \mathrm{ANS}) &= \mathbb{E}[q(x) \mid a = \mathrm{ANS}]\\ &= \mathbb{E}[q(x) \mid q(x) > p_{\lambda}] \ge p_{\lambda}. \tag*{(7)} \end{aligned}

This characterizes idealized behavior, not an empirical guarantee, since the learned policy may not optimize or generalize perfectly and its uncertainty may not match the true probability of correctness.

Training with a fixed penalty provides an interpretable trade-off between accuracy and coverage, but each λ\lambda represents only one operating point. Adapting to different error costs therefore generally requires retraining the model [16, 24] or selecting a different model.

Explicit Confidence Targets

Explicit confidence targets [19] extend fixed-penalty abstention by allowing users to specify the required accuracy in the prompt. Let p∈[0,1)p \in[0,1) denote this target. Setting

λ(p)=p1−p(8)\lambda(p) = \frac{p}{1-p} \tag*{(8)}

in Equation (3) gives

Rp={1,correct answer,−p1−p,incorrect answer,0,abstention.(9)R_p = \begin{cases} 1, & \text{correct answer},\\ -\frac{p}{1-p}, & \text{incorrect answer},\\ 0, & \text{abstention}. \end{cases} \tag*{(9)}

Suppose the probability of answering correctly is q(x)q(x). The expected reward for answering is

E[Rp∣x,ANS]=q(x)−p1−p(1−q(x))=q(x)−p1−p.(10)\mathbb{E}[R_p \mid x,\mathrm{ANS}] = q(x) - \frac{p}{1-p}(1-q(x)) = \frac{q(x)-p}{1-p}. \tag*{(10)}

Abstention receives zero reward. A policy that maximizes expected reward therefore answers when q(x)>pq(x)>p, abstains when q(x)<pq(x)<p, and is indifferent when q(x)=pq(x)=p. Thus, pp directly controls the decision between answering and abstaining. When p=0p=0, answering is at least as good as abstaining for every question and is strictly better whenever q(x)>0q(x)>0. We use this endpoint to represent the setting in which the user requires the model to answer, with ties resolved in favor of answering.

Our goal is to train a single policy πθ(⋅∣x,p)\pi_\theta(\cdot\mid x,p) that follows arbitrary confidence targets without retraining. For every pp at which the model answers with nonzero probability, the desired behavior is

Acc⁡(p):=Pr⁡(correct∣ANS,p)≥p.(11)\operatorname{Acc}(p) := \Pr(\mathrm{correct} \mid\mathrm{ANS},p) \ge p. \tag*{(11)}

The model should answer when doing so yields positive expected reward, rather than defaulting to abstention on every question. Importantly, pp is a target specified by the user in the prompt, and the model directly chooses whether to answer or abstain. Inference therefore requires no external system to compare a reported confidence score [8] with a threshold.

Wu et al. [36] propose Explicit Risk Thresholding (ERT) to study this approach by sampling pp during training, including it in the prompt, and using the reward in Equation (9). However, their hallucination rates remain largely unchanged across different pp, showing that the model does not adapt its behavior to the target. They attribute this failure to a noisy optimization problem caused by sampling targets across the full range, making it difficult to learn a consistent strategy for different pp. Mohamadi et al. [24] also argue that a single model cannot support all risk preferences because limited model capacity prevents it from optimizing every error penalty simultaneously. Together, these findings illustrate the difficulty of training one model to follow arbitrary confidence targets.

Method

Signal Preservation

We first identify a key problem in prior work: most of the direct reward signal distinguishing different confidence targets is lost during training. ERT uses GRPO to optimize the reward in Equation (9). For each prompt, GRPO samples GG responses and estimates their advantages from relative rewards. Let RiR_i denote the reward of response ii. The corresponding advantage is

Ai=Ri−R‾σR,R‾=1G∑j=1GRj,σR=1G∑j=1G(Rj−R‾)2.(12)A_i = \frac{R_i-\overline{R}}{\sigma_R}, \qquad \overline{R} = \frac{1}{G}\sum_{j=1}^{G} R_j, \qquad \sigma_R = \sqrt{\frac{1}{G}\sum_{j=1}^{G}(R_j-\overline{R})^2}. \tag*{(12)}

However, this standardization can remove the effect of the reward scale induced by pp on the advantages. At a fixed pp, consider a rollout group with two distinct reward values rH>rLr_{\mathrm{H}} > r_{\mathrm{L}}. Let ρ∈(0,1)\rho\in(0,1) denote the fraction of responses receiving rHr_{\mathrm{H}}. The group mean and standard deviation are

R‾=ρrH+(1−ρ)rL,σR=(rH−rL)ρ(1−ρ).(13)\overline{R} = \rho r_{\mathrm{H}} + (1-\rho)r_{\mathrm{L}}, \qquad\sigma_R = (r_{\mathrm{H}} - r_{\mathrm{L}})\sqrt{\rho(1-\rho)}. \tag*{(13)}

Substituting Equation (13) into Equation (12) gives

AH=1−ρρ,AL=−ρ1−ρ.(14)A_{\mathrm{H}} = \sqrt{\frac{1-\rho}{\rho}}, \qquad A_{\mathrm{L}} = -\sqrt{\frac{\rho}{1-\rho}}. \tag*{(14)}

The resulting advantages are independent of pp, so the training signal carried by the reward scale of pp disappears. For example, in a group with two distinct reward values, a higher pp imposes a larger reward penalty on an incorrect answer, but does not make its standardized advantage more negative. Also, for a group with one reward value, all advantages in Equation (12) are computed as zero in our implementation, so the group provides no reward signal. Only groups containing all three outcomes can retain the learning signal from changes in the reward scale induced by pp.

Figure Figure 2 shows how the three group types evolve during training. Groups with three reward values are rare and become less frequent, while groups with one value eventually dominate and most others contain two. Thus, only a small fraction of groups retains a direct signal from reward magnitudes that vary with pp, explaining why the policy remains insensitive to the confidence target.

Figure 2. Fractions of rollout groups.

To preserve the signal, we remove the group standard deviation while retaining mean centering:

A~i=Ri−R‾.(15)\widetilde{A}_i = R_i - \overline{R}. \tag*{(15)}

Without standard deviation normalization, a reward gap that varies with pp produces advantages that vary accordingly, preserving the direct reward signal across confidence targets. This also exposes the large penalty in Equation (9), which we address next.

Stable Rewards

Removing the group standard deviation preserves the effect of reward magnitudes on the advantages, but also makes policy updates sensitive to those magnitudes. In Equation (9), the rewards for a correct answer and abstention remain fixed at 1 and 0, while the reward for an incorrect answer is −p/(1−p)-p/(1-p), which tends to −∞-\infty as pp approaches one. Groups containing both incorrect answers and other outcomes can therefore produce large advantages. This creates two related problems.

Gradient dominance. Because the policy shares parameters across confidence targets, such groups at large pp can dominate the overall gradient, reducing the influence of lower targets.

Training instability. The number and composition of such groups vary across batches. At large pp, their centered advantages can cause sharp fluctuations in gradient norms and destabilize training.

Multiplying all rewards by a single constant scales the centered advantages equally across confidence targets, leaving their relative influence unchanged. We instead use a positive factor that decreases with pp to reduce the penalty at large pp without changing the theoretical decision to answer or abstain.

A simple way to bound the error penalty is to multiply all rewards by 1−p1-p:

(1−p)Rp={1−p,correct answer,−p,incorrect answer,0,abstention.(16)(1-p)R_p = \begin{cases} 1-p, & \text{correct answer},\\ -p, & \text{incorrect answer},\\ 0, & \text{abstention}. \end{cases} \tag*{(16)}

The magnitude of the error penalty is now bounded by one. This scaling also changes the reward for a correct answer to 1−p1-p, while abstention remains at zero.

To control how strongly rewards at large pp are reduced, we introduce a family of rewards:

Rp(b)=(1−p)bRp={(1−p)b,correct answer,−p(1−p)b−1,incorrect answer,0,abstention,b≥0.(17)R_p^{(b)}=(1-p)^bR_p= \begin{cases} (1-p)^b, & \text{correct answer},\\ -p(1-p)^{b-1}, & \text{incorrect answer},\\ 0, & \text{abstention}, \end{cases} \qquad b\geq0. \tag*{(17)}

Here, bb is a stability coefficient. For a fixed p∈(0,1)p\in(0,1), increasing bb reduces the magnitude of the error penalty and the reward for a correct answer. In particular, b=0b=0 recovers Equation (9), while b=1b=1 recovers Equation (16). Since the same scaling applies to all responses in a rollout group, it also scales their centered advantages:

A~i(b)=(1−p)bA~i(0).(18)\widetilde{A}_i^{(b)}=(1-p)^b\widetilde{A}_i^{(0)}. \tag*{(18)}

Thus, larger values of bb reduce the relative influence of groups with large pp.

Importantly, scaling all rewards at a given pp by the same positive factor does not change the theoretical decision to answer or abstain. Suppose the model answers correctly with probability q(x)q(x). Its expected reward for answering is

E[Rp(b)∣x,ANS]=q(x)(1−p)b−(1−q(x))p(1−p)b−1=(1−p)b−1(q(x)−p).(19)\begin{aligned} \mathbb{E}\left[R_p^{(b)}\mid x,\mathrm{ANS}\right]=q(x)(1-p)^b-(1-q(x))p(1-p)^{b-1} \\ &=(1-p)^{b-1}(q(x)-p). \tag*{(19)} \end{aligned}

Since (1−p)b−1>0(1-p)^{b-1}>0 for p<1p<1, answering has positive expected reward exactly when q(x)>pq(x)>p. Thus, every value of bb preserves the same theoretical answer condition.

Reward Scheduling

The stability coefficient bb balances the strength of the learning signal against training stability. For any fixed p∈(0,1)p\in(0,1), the centered advantages in Equation (18) become smaller in magnitude as bb increases. This reduction is strongest at large pp, where the original error penalty is largest. It can improve stability, but it also weakens the signal for learning when to abstain by reducing the error penalty relative to abstention. Our ablations suggest that keeping bb fixed at a relatively large value, such as b=1b=1, can make it harder for the model to learn when to abstain and to vary its behavior across confidence targets. Details are shown in § 4.4.

By contrast, a smaller bb, such as b=0b=0, preserves stronger centered advantages at large pp, helping the model learn when to abstain and distinguish confidence targets more effectively early in training. However, keeping bb small throughout training can lead to gradient dominance and training instability, as discussed in § 3.2. These effects may harm final performance and general capabilities.

Guided by this trade-off, we linearly increase bb from 0 to 1 over TT training steps. At step tt, we set

bt=tT,0≤t≤T.(20)b_t=\frac{t}{T},\qquad0\leq t\leq T. \tag*{(20)}

This schedule preserves a stronger learning signal early while improving training stability later.

Experiment

In this section, we conduct experiments to answer the following research questions:

  • (RQ1) Can RLEC successfully distinguish different explicit confidence targets?

  • (RQ2) How does RLEC compare with prior methods in calibration and general capabilities?

  • (RQ3) What enables RLEC to distinguish targets while preserving general capabilities?

  • (RQ4) What other properties and practical advantages does RLEC offer?

Experimental Setup

Implementation Details. Our main experiments and ablation studies are based on Qwen3-4B-Base [37], and we further conduct experiments on Qwen3-30B-A3B-Base to evaluate scalability. Our training data combine DAPO-Math [39] and a modified version of HotpotQA [8, 38]. We randomly sample 15,000 examples from each dataset, resulting in 30,000 training examples, and train for five epochs. We set p=0p = 0 with probability 10%. Otherwise, we uniformly sample pp from the interval [0,0.9][0, 0.9] and round it to two decimal places. During evaluation, we test p∈{0,0.1,…,0.9}p \in\{0, 0.1, \ldots, 0.9\} and report results averaged over multiple sampled generations. Additional details are provided in Appendix § 7.§ 1.

Benchmarks. Our evaluation covers eight benchmarks from three domains and different difficulty levels, including (1) mathematical reasoning: GSM8K [5], MATH-500 [12, 21], and AIME 2024; (2) knowledge and hallucination: HotpotQA test set [8, 38], AA-Omniscience [15], and SimpleQA-Verified [11, 34]; and (3) scientific reasoning: MMLU-Pro [33] and GPQA [26]. Details are provided in Appendix § 7.§ 2.

Baselines. We compare RLEC with three types of methods: (1) RLVR: standard RLVR with a binary reward; (2) abstention methods: methods that use a fixed error penalty or fail to learn behavior that varies with pp, including TruthRL [35] and ERT [36], compared in RQ1; and (3) confidence reporting methods: RLCR [8], included in RQ2 as an additional reference for calibration performance under a different formulation.

Adaptation to Confidence Targets

To answer RQ1, we study whether models can change their behavior according to pp. We first compare the training behavior of RLEC and ERT, and then compare the final accuracy and abstention rates of RLEC, RLVR, ERT, and TruthRL across all evaluation benchmarks.

Training Curves. Figure 3 shows the abstention rate on held-out validation sets during training, evaluated with p∈{0.1,0.5,0.9}p \in\{0.1, 0.5, 0.9\}. On both HotpotQA and DAPO-Math, our reproduction of ERT shows the same failure reported in the original study [36]: its curves for different values of pp almost completely overlap. In contrast, RLEC produces a clear dependence on pp, with higher confidence targets generally leading to higher abstention rates. These results show that RLEC successfully learns different behaviors for different confidence targets.

Figure 3. Abstention rates on held-out validation sets during training. Each curve evaluates the same training checkpoints with a fixed confidence target pp. ERT produces nearly overlapping curves for different targets, while RLEC learns higher abstention rates for higher values of pp.

Test Results. Figure 4(a,Figure 4b) reports average abstention rates and answer accuracy across all benchmarks using Qwen3-4B-Base. Among methods that directly decide whether to answer, only RLEC changes clearly with pp: as pp increases from 0.1 to 0.9, its abstention rate rises from approximately 19% to 49%, while its accuracy among answered questions rises from approximately 68% to 81%. RLVR, ERT, and TruthRL remain nearly unchanged. Thus, RLEC successfully learns to adapt its decision to answer, or abstain across confidence targets, answering RQ1.

Figure 4. Adaptation to explicit confidence targets. (a,b) Direct decision methods with Qwen3-4B-Base. (c,d) RLEC, RLCR, and RLVR with Qwen3-30B-A3B-Base.

Comparison with Calibration Methods

As shown in § 4.2, existing methods that directly decide whether to answer or abstain show little sensitivity to the confidence target. To compare calibration performance and answer RQ2, we therefore compare RLEC with RLCR [8], a different type of method that reports a numerical confidence cc together with its answer. Since RLCR cannot directly follow a user provided target pp, we apply an external rule that retains the answer when c≥pc \ge p and replaces it with an abstention otherwise, whereas RLEC conditions its answer or abstention decision directly on pp.

Qualitative Analysis. Figure 4(c,Figure 4d) compares RLEC, RLCR, and RLVR across confidence targets using Qwen3-30B-A3B-Base. RLVR changes little with pp, whereas RLEC and RLCR increase their abstention rates as pp increases. RLEC responds more strongly, especially at high pp, and its accuracy among answered questions follows the target boundary more closely. These results suggest that directly optimizing explicit confidence targets enables clearer adaptation to the requested target.

Quantitative Analysis. We use three metrics to evaluate answer ability, target violations, and utility. Forced Accuracy measures answer accuracy when abstention is disabled. We set p=0p = 0 for RLEC and retain all RLCR answers, testing whether calibration training harms answer ability.

Average Overconfidence (AOC) measures the average magnitude of target violations:

AOC=1∣P∣∑p∈Pmax⁡{0,p−Acc(p)},(21)\mathrm{AOC} = \frac{1}{|\mathcal{P}|} \sum_{p \in\mathcal{P}} \max\{0, p - \mathrm{Acc}(p)\}, \tag*{(21)}

where lower values indicate better compliance with the requested accuracy. Since AOC can be reduced by abstaining, it does not capture utility. We therefore also report

Score=1N∣P∣∑p∈P∑i=1Nzi,p(yi,p−p),(22)\mathrm{Score} = \frac{1}{N|\mathcal{P}|} \sum_{p \in\mathcal{P}} \sum_{i=1}^{N} z_{i,p}(y_{i,p} - p), \tag*{(22)}

where zi,pz_{i,p} indicates whether the model answers and yi,py_{i,p} indicates whether the answer is correct. A correct answer contributes 1−p1-p, an incorrect answer contributes −p-p, and an abstention contributes zero. A higher Score indicates greater utility by accounting for both correctness and coverage.

For RLEC, each Score contribution is exactly its reward with b=1b = 1. For RLCR, twice the corresponding score integrated over all confidence targets equals its reward, as proved in Appendix § 9.

Table 1 reports Micro results pooled over all examples and Macro results averaged equally across benchmarks. Micro reflects aggregate performance and is less affected by difficult benchmarks such as SimpleQA-Verified, whereas Macro prevents larger benchmarks from dominating. RLEC largely preserves the answer ability obtained by RLVR, with comparable Forced Accuracy at both scales. Compared with RLCR, the 4B model achieves lower AOC but also lower Score, indicating a trade-off between target compliance and utility. At 30B, RLEC achieves both lower AOC and higher Score, outperforming RLCR in both respects. These results demonstrate the feasibility of direct training with explicit confidence targets and show that RLEC remains effective at the larger scale.

ModelMetricMethodGSM8KMATHAIMEHotpotOmniSciSimpleQAMMLU-ProGPQAMicroMacro
Qwen3-4B-BaseAcc. ↑\uparrowRLVR93.3369.7020.8369.5013.755.2068.4735.9158.1847.09
Qwen3-4B-BaseRLCR93.1073.9016.2566.3015.584.7566.9434.0057.3546.35
Qwen3-4B-BaseRLEC92.1570.5019.7967.6014.174.7567.3736.2457.4046.57
Qwen3-4B-BaseAOC ↓\downarrowRLVR0.0000.0270.2580.0330.2920.3890.0320.1750.0490.151
Qwen3-4B-BaseRLCR0.0900.0900.0900.0980.2290.3960.0900.1690.0900.156
Qwen3-4B-BaseRLEC0.0000.0000.0900.0100.2590.3810.0150.1570.0130.114
Qwen3-4B-BaseScore ↑\uparrowRLVR0.4800.261-0.1620.237-0.173-0.2130.237-0.0900.1670.072
Qwen3-4B-BaseRLCR0.4820.3420.0530.255-0.004-0.0940.2850.0690.2350.173
Qwen3-4B-BaseRLEC0.4770.3330.0560.264-0.048-0.1070.262-0.0160.2140.153
Qwen3-30B-A3B-BaseAcc. ↑\uparrowRLVR95.0777.1026.8868.9018.179.4076.2546.5363.9752.29
Qwen3-30B-A3B-BaseRLCR94.7377.7026.2566.2019.088.9077.2246.7664.2452.10
Qwen3-30B-A3B-BaseRLEC94.7778.6027.2966.6020.008.3076.7347.0964.1452.42
Qwen3-30B-A3B-BaseAOC ↓\downarrowRLVR0.0000.0140.2210.0380.2650.3720.0200.1080.0450.130
Qwen3-30B-A3B-BaseRLCR0.0000.0030.0000.0380.2700.3670.0000.0570.0250.092
Qwen3-30B-A3B-BaseRLEC0.0000.0030.0000.0090.2090.3680.0020.0720.0010.083
Qwen3-30B-A3B-BaseScore ↑\uparrowRLVR0.5040.329-0.1690.222-0.198-0.3090.3000.0260.1950.088
Qwen3-30B-A3B-BaseRLCR0.5000.3600.0960.218-0.173-0.3110.3550.1230.2360.146
Qwen3-30B-A3B-BaseRLEC0.5040.3530.0910.258-0.015-0.1320.3360.1120.2610.188

Table 1. Forced answer accuracy, average overconfidence (AOC), and average score across model scales. The best Micro and Macro results within each model scale are shown in bold.

Ablation Studies

To answer RQ3, we study two main design choices in RLEC: omitting the reward standard deviation from group advantage estimation and selecting the stability coefficient bb.

Standard Deviation. As shown in Figure 5(b), adding group standard deviation normalization produces nearly overlapping abstention curves across different pp. This supports our analysis that normalization weakens the reward scale signal induced by pp. Also, Appendix § 8.§ 2 shows that removing the group standard deviation does not consistently improve or degrade answer ability.

Figure 5. Ablation training curves. Abstention rates at p∈{0.1,0.5,0.9}p \in\{0.1, 0.5, 0.9\} for (a) RLEC, (b) RLEC with standard deviation normalization, and fixed (c) b=0b = 0, (d) b=0.5b = 0.5, and (e) b=1b = 1.

Choice of Stability Coefficient. Panels (c), (d), and (e) of Figure 5 illustrate the different behaviors induced by fixed choices of bb. With b=0b = 0, the model distinguishes the confidence targets relatively early, but its abstention rate fluctuates sharply. With b=1b = 1, the curves nearly overlap, and the model fails to distinguish the targets. The intermediate choice b=0.5b = 0.5 produces separation only late in training, indicating less efficient learning and potentially less reliable target differentiation.

We also evaluate the different choices on the test benchmarks. Forced Accuracy, AOC, and Score remain broadly consistent, demonstrating that the proposed reward family is robust to the specific choice of bb. The scheduled configuration achieves the highest aggregate Forced Accuracy while retaining competitive calibration performance. Because we consider preserving answer ability a primary requirement for behavioral calibration, we select it as our default configuration. Complete results and further discussion are provided in Appendix § 8.§ 1.

Additional Properties

To answer RQ4, we examine fine-grained target control, the operational implicit confidence profile, and the risk–coverage trade-off.

Fine-Grained Confidence Control

We evaluate RLEC on AIME24 and AIME25 with confidence targets spaced at intervals of 0.01, extending up to p=0.95p = 0.95. We choose these benchmarks because their difficulty provides an informative regime for studying fine-grained control: their relatively low accuracy avoids the saturation observed on easier benchmarks such as GSM8K, while a meaningful subset of problems remains within the model’s reasoning capability. In contrast, errors on knowledge-intensive benchmarks such as SimpleQA more often reflect missing factual knowledge, leaving less room for confidence control to distinguish solvable from unsolvable examples. Each target is evaluated using Avg@1024. Figure 6 reports the abstention rate, attempted accuracy, and the proportions of all responses that are correct and wrong. Because training uses only targets up to p=0.90p = 0.90, the interval p∈(0.90,0.95]p \in(0.90, 0.95] additionally evaluates behavior beyond the training range.

Four line plots showing abstention, attempted accuracy, correct responses, and wrong responses as functions of confidence target

Figure 6. Fine-grained confidence control on AIME24 and AIME25. Targets vary at intervals of 0.01 up to p=0.95p = 0.95, with Avg@1024 for each target. (a) Abstention rate. (b) Accuracy among answered questions. (c,d) Proportions of all responses that are correct and wrong. Training uses targets up to p=0.90p = 0.90.

As shown in Figure 6, the abstention rate rises steadily and attempted accuracy generally improves as pp increases, demonstrating that RLEC responds to fine-grained changes in the target, including values such as p=0.01p = 0.01 and p=0.02p = 0.02. The curves are not strictly monotonic at every adjacent target, which we attribute to finite sample evaluation noise, but their overall trends remain stable. Attempted accuracy also generally meets or exceeds the target.

In contrast, although the RLCR prompt does not restrict the number of decimal places in its confidence output, as shown in Figure 8, all confidence values produced by the evaluated RLCR model lie on a one-decimal grid; we do not observe intermediate values such as 0.12. This observation is consistent with prior findings [4, 7]. Consequently, even if an external system applies arbitrary thresholds to these outputs, it cannot recover additional operating points between adjacent confidence values. The evaluated RLCR model therefore provides substantially coarser control than directly conditioning RLEC on targets at a resolution of 0.01.

Prompt templates used by RLEC and the baselines

Figure 8. RLEC Prompt. Work through this problem step by step, then honestly assess how confident you are that your solution is correct. Only give a final answer if you are at least {p} confident that it is correct; otherwise do not guess and answer with \boxed{I am not sure.} instead. Put your final answer inside \boxed{}. If you are not confident enough, put \boxed{I am not sure.} instead. RLCR Prompt. Work through this problem step by step, then honestly assess how confident you are that your solution is correct. Put your final answer inside \boxed{}, then output your confidence that the answer is correct as a number between 0 and 1 (inclusive) enclosed within <confidence> </confidence> tags. RLVR Prompt. Work through this problem step by step. Put your final answer inside \boxed{}.

Selective Error Filtering. Panels (c) and (d) further show that the number of wrong answers decreases much faster than the number of correct answers as pp increases. This effect is particularly clear at lower confidence targets, where the proportion of correct answers remains nearly unchanged while many wrong answers are filtered out. RLEC therefore initially improves attempted accuracy primarily by abstaining from likely errors rather than discarding correct answers, yielding a favorable trade-off between coverage and answer reliability.

Extrapolation Beyond the Training Range. RLEC is trained only with targets up to p=0.90p = 0.90, whereas Figure 6 extends the evaluation to the unseen range p∈(0.90,0.95]p \in(0.90, 0.95]. Despite small local fluctuations and the reduced coverage at these high targets, all four metrics remain stable and continue their overall trends without an abrupt behavioral change. These results suggest that RLEC remains responsive even to confidence targets beyond the range observed during training.

Implicit Confidence Profile

RLEC directly accepts an explicit confidence target pp and decides whether to answer, rather than outputting a confidence score. Nevertheless, to better understand its aggregate behavior and facilitate comparison with RLCR, we construct an operational approximation of its implicit confidence profile. We measure the abstention rate at p∈{0,0.1,…,0.9}p \in\{0, 0.1, \ldots, 0.9\}, treat it as an estimate of the cumulative mass below pp, and use differences between neighboring targets as interval masses. This construction summarizes the aggregate behavior of the pp-conditioned policies and does not assume a shared sample-level latent confidence variable. For RLCR, we directly use its reported confidence.

Figure 7(a,Figure 7b) compares the resulting profiles on the 30B model. Under this construction, RLEC has a higher standard deviation (0.2335 versus 0.2126 for Micro and 0.2713 versus 0.2646 for Macro), with less mass in the highest interval and more in the intermediate intervals. These results suggest that RLEC supports a broader range of aggregate behavior across confidence targets.

Figure 7. Implicit confidence profiles and risk--coverage trade-offs on the 30B model. (a,b) Micro and Macro profiles, constructed from RLEC’s abstention behavior and RLCR’s reported confidence. (c,d) Micro and Macro risk–coverage curves. RLEC’s operational profile is more dispersed, and its risk–coverage curves extend to lower-coverage operating points.

Risk-Coverage Trade-off

Figure 7(c,Figure 7d) plots risk, the error rate among answered questions, against coverage, the fraction of questions answered, on the 30B model. A lower curve is better, since it means fewer errors at the same coverage. Overall, the two methods perform similarly: RLEC is slightly better in the Micro setting, while RLCR is slightly better in the Macro setting. The main difference is in operating range. In both settings, RLEC extends to a lower coverage than RLCR, so it can abstain more and keep risk lower when the target calls for high reliability, whereas RLCR cannot reach the same conservative operating points. This is a potential practical advantage in applications that demand high reliability.

Related Work

Reinforcement Learning with Verifiable Rewards

Reinforcement learning with verifiable rewards (RLVR) [3, 10, 23, 27, 39] has become a common approach for improving LLM reasoning. A widely used algorithm is Group Relative Policy Optimization (GRPO) [27], which estimates advantages from multiple responses to the same prompt by subtracting the group mean and dividing by the group standard deviation. Prior work has examined the benefits and drawbacks of this normalization. Dr. GRPO [23] removes it to avoid weighting questions according to their difficulty. Turtel et al. [31] remove standard deviation normalization to preserve the effect of large prediction errors, while Bereket and Leskovec [2] show that removing it reduces biased advantages and overconfidence. In contrast, Ge et al. [9] view normalization as adaptive gradient scaling that can improve convergence and stability, while CAL-GRPO [14] also reports that retaining normalization improves sample efficiency across its GRPO variants.

These studies mainly examine how standard deviation normalization affects optimization and model performance. Unlike all of them, RLEC uses the reward scale to encode the confidence target pp. We show that for groups with two distinct reward values and fixed proportions, normalization removes the effect of the reward scale induced by pp on the advantages. Such groups are common during training, helping explain the failure to distinguish confidence targets in prior work [36]. RLEC retains this signal through mean centering without standard deviation normalization.

Hallucination Mitigation and Behavioral Calibration

Hallucination [13, 17] remains a major source of errors in LLMs. In Why Language Models Hallucinate, Kalai et al. [19] argue that common training objectives encourage models to guess rather than admit uncertainty. This has motivated RL methods that reward correct answers, penalize errors, and allow abstention [16, 24, 25, 35]. However, these methods encode a fixed risk preference and cannot adapt their decision to answer or abstain to different user requirements at inference time. Wu et al. [36] propose Explicit Risk Thresholding (ERT), which extends this setting to confidence targets specified in the prompt, but report that it fails to learn distinct behaviors across targets.

Another line of work trains or prompts models to express confidence [22, 30]. For example, Damani et al. [8] propose Reinforcement Learning with Calibration Rewards (RLCR), which trains models to produce an answer together with a calibrated confidence score. However, these confidence estimates typically need to be converted into decisions by an external rule or system, and some approaches further require a separate calibrator or calibration set [20, 32]. In contrast, RLEC enables models to learn how to directly adapt their answer or abstention decisions to explicit confidence targets, without an external rule for converting confidence estimates into decisions.

Conclusion

We study explicit confidence targets as a more flexible and direct way to control LLM abstention. We show that the failure of prior work results from the loss of signals and the unstable scale of reward. Based on this analysis, we propose RLEC, which preserves the confidence target signal, introduces a stable reward family, and uses reward scheduling to balance effective learning and training stability. Experiments across diverse benchmarks and model scales show that explicit confidence target training is feasible. RLEC largely preserves general capabilities, achieves strong calibration, and supports fine-grained control without an external decision system. More broadly, RLEC opens a new and direct path toward behavioral calibration with explicit, user-specified confidence targets.

Detailed Experimental Setup

Implementation Details

Training and Evaluation. We use verl for RL training [28], with the hyperparameters for all RL methods summarized in Table 2. We do not use Clip-Higher, entropy regularization, or a KL penalty. For RLCR, we determine abstention by applying the requested confidence target as an external threshold to its reported confidence. For all other methods, abstentions are identified by matching the designated abstention string. For p=0p = 0, we use a binary reward and evaluation criterion: only a correct answer is treated as correct, whereas incorrect answers and abstentions are both treated as incorrect. For AOC, accuracy is set to zero when no answers are attempted, balancing calibration against utility. Due to limited computational resources, we do not conduct multiple training runs for each setting. Instead, we report Avg@16 for AIME and Avg@2 for all other benchmarks to reduce evaluation variance.

HyperparameterValue
Training
max_response_length4096
train_batch_size256
ppo_mini_batch_size128
optim.lr1×10−61 \times10^{-6}
rollout.temperature1.0
rollout.n8
Evaluation
max_tokens4096
temperature1.0
top_p0.95

Table 2. Hyperparameters.

Prompts. The prompt templates used by RLEC and the baselines are shown in Figure 8. For other abstention methods, including ERT and TruthRL, we use the RLEC prompt during both training and evaluation. For RLVR, we use its standard prompt during training and at p=0p = 0 to represent the conventional RL setting and compare answer ability. At p>0p > 0, we use the RLEC prompt to demonstrate that standard RLVR training alone does not enable target-conditioned abstention. For SimpleQA-Verified and AA-Omniscience, we additionally use the grading prompt shown in Figure 9 and use Qwen3.6-27B as the judge. Since abstention is identified separately, this grader only determines whether the predicted answer is semantically equivalent to the reference answer.

System prompt used to grade SimpleQA-Verified and AA-Omniscience

Figure 9. Correctness-Grading System Prompt. You are a grading assistant. Decide whether the predicted answer is semantically equivalent to the reference gold answer for the given question. Ignore formatting differences, units notation, LaTeX wrappers, and trivial rephrasing. Reply with a single token: YES or NO.

Benchmarks

We use the standard versions of the evaluation benchmarks except for the following two settings.

HotpotQA. We use the modified version [8] of HotpotQA [38]. Each example contains eight paragraphs, while zero, one, or both of the two relevant paragraphs are removed to test whether the model abstains when the available evidence is incomplete.

MMLU-Pro. We evaluate the four science subsets in MMLU-Pro [33]: biology, chemistry, physics, and health. Since the full benchmark contains substantially more examples than the other benchmarks, this selection prevents MMLU-Pro from dominating the aggregate results.

Baselines

RLVR. RLVR uses a binary correctness reward and does not explicitly reward abstention:

RRLVR={1,correct,0,otherwise.(23)R_{\mathrm{RLVR}} = \begin{cases} 1, & \text{correct},\\ 0, & \text{otherwise}. \end{cases} \tag*{(23)}

In the main experiments, we remove its group standard deviation normalization to align its advantage with RLEC and enable a closer comparison. Appendix § 8.§ 2 examines the effect of this choice.

TruthRL. TruthRL [35] uses a fixed ternary reward that penalizes incorrect answers while assigning zero reward to abstentions:

RTruthRL={1,correct,−1,incorrect,0,abstention.(24)R_{\mathrm{TruthRL}} = \begin{cases} 1, & \text{correct},\\ -1, & \text{incorrect},\\ 0, & \text{abstention}. \end{cases} \tag*{(24)}

Our framework is compatible with TruthRL by setting p=0.5p = 0.5, b=0b = 0, and retaining group standard-deviation normalization.

ERT. ERT [36] samples a confidence target pp and applies its corresponding error cost during standard GRPO training:

RERT(p)={1,correct,p1−p,incorrect,0,abstention.(25)R_{\mathrm{ERT}}(p) = \begin{cases} 1, & \text{correct},\\ \frac{p}{1-p}, & \text{incorrect},\\ 0, & \text{abstention}. \end{cases} \tag*{(25)}

Our framework is compatible with ERT by setting b=0b = 0 and retaining group standard-deviation normalization.

RLCR. RLCR [8] trains the model to produce an answer together with a confidence score c∈[0,1]c \in[0, 1]. Let y=1y = 1 if the answer is correct and y=0y = 0 otherwise; its reward combines correctness with the Brier score:

RRLCR(y,c)=y−(c−y)2.(26)R_{\mathrm{RLCR}}(y,c) = y - (c-y)^2. \tag*{(26)}

For RLCR, we remove group standard deviation normalization to obtain better calibration [2, 31]. At evaluation time, we use an external decision system that retains the answer when c≥pc \ge p and replaces it with an abstention otherwise.

Additional Experimental Results

Choice of Stability Coefficient

We examine the choice of bb within our proposed reward family, considering fixed values as well as linear and sinusoidal schedules. As shown in Equation (19), these choices preserve the same theoretical decision rule but induce different balances among signal strength, optimization stability, and answer ability. Table 3 reports their Forced Accuracy, AOC, and Score, reflecting answer ability, target compliance, and utility, respectively. We additionally include LR Schedule as a possible variant, which applies a global cosine learning-rate decay.

MetricMethodGSM8KMATHAIMEHotpotOmniSciSimpleQAMMLU-ProGPQAMicroMacro
Acc. ↑RLVR93.3369.7020.8369.5013.755.2068.4735.9158.1847.09
Ours92.1570.5019.7967.6014.174.7567.3736.2457.4046.57
b=0b = 091.0969.3016.2566.0014.254.6567.3236.3556.9545.65
b=0.5b = 0.592.4670.9015.6267.7014.754.2567.4033.0057.1845.76
b=1b = 191.8170.4019.3867.8014.085.3069.4934.5658.2946.60
LR Schedule91.7769.8016.4664.0014.084.1067.0734.6856.6945.25
Sine Schedule93.0369.6020.0067.8013.754.4565.7735.1256.6346.19
AOC ↓RLVR0.0000.0270.2580.0330.2920.3890.0320.1750.0490.151
Ours0.0000.0000.0900.0100.2590.3810.0150.1570.0130.114
b=0b = 00.0000.0020.0370.0080.1780.3880.0160.1440.0100.096
b=0.5b = 0.50.0000.0010.0180.0000.2070.3700.0150.1410.0080.094
b=1b = 10.0000.0100.0200.0310.2510.3840.0250.1540.0290.109
LR Schedule0.0000.0040.0130.0170.2270.3820.0130.1440.0110.100
Sine Schedule0.0000.0030.0090.0160.2930.3990.0150.1480.0220.111
Score ↑RLVR0.4800.261−0.1620.237−0.173−0.2130.237−0.0900.1670.072
Ours0.4770.3330.0560.264−0.048−0.1070.262−0.0160.2140.153
b=0b = 00.4740.3040.0540.267−0.010−0.0630.264−0.0010.2220.161
b=0.5b = 0.50.4770.3340.0480.276−0.025−0.0610.2580.0150.2220.165
b=1b = 10.4740.3050.0730.241−0.087−0.1290.247−0.0340.1980.136
LR Schedule0.4730.3160.0630.241−0.011−0.0660.2610.0010.2190.160
Sine Schedule0.4780.3300.0740.257−0.101−0.1800.264−0.0070.2040.139

Table 3. Forced answer accuracy, average overconfidence (AOC), and average score for RLVR and different stability coefficient and scheduling choices. The best Micro and Macro results among variants that distinguish confidence targets are shown in bold; RLVR and b=1b = 1 are excluded from this comparison because they cannot distinguish among targets.

Forced Accuracy. Among the configurations that distinguish confidence targets, Forced Accuracy remains broadly stable, with the linear schedule obtaining the highest aggregate Forced Accuracy. The clearest differences appear on AIME, where fixed b=0b = 0, fixed b=0.5b = 0.5, and LR Schedule are all more than three points below the linear schedule. Because AIME is particularly difficult and benefits substantially from reinforcement learning, this result is consistent with training stability being especially important for preserving abilities acquired during this stage. Differences on most other benchmarks are smaller or less consistent, particularly on knowledge benchmarks such as SimpleQA-Verified and AA-Omniscience, where performance largely depends on knowledge acquired during pretraining. LR Schedule performs worse in aggregate Forced Accuracy, indicating that scheduling the relative reward scale across confidence targets is not equivalent to applying a global learning-rate decay.

Calibration Performance. All choices of bb substantially improve calibration over RLVR, while the differences among these choices remain modest. In particular, b=0.5b = 0.5 obtains the lowest aggregate AOC and the highest Macro Score, while retaining relatively strong Forced Accuracy.

Training Dynamics. The training dynamics in Figure 10, however, reveal an additional distinction that is not captured by the final aggregate results. Under the linear schedule, the behavior for p=0.9p = 0.9 separates from the lower-confidence targets around the middle of training and remains distinctly ordered thereafter. Fixed b=0b = 0 also distinguishes the targets relatively early, but exhibits larger fluctuations. With fixed b=0.5b = 0.5, clear separation emerges only after approximately 400 steps despite its favorable final calibration and relatively strong Forced Accuracy, suggesting that it may not establish target conditioned behavior as reliably throughout optimization and may be more sensitive to the available training horizon. With b=1b = 1, the curves remain nearly indistinguishable, consistent with the absence of a target-dependent reward gap. Overall, smaller values of bb strengthen target-dependent learning but reduce stability, whereas larger values stabilize updates but weaken differentiation. The linear schedule balances these effects by establishing the distinctions early and stabilizing them later.

Figure 10. Training dynamics of the abstention rate on HotpotQA under different choices of the stability coefficient and scheduling strategy. RLEC uses the linear schedule of bb, while RLEC Sin. uses a sinusoidal schedule. Curves correspond to target confidence levels p∈{0.1,0.5,0.9}p \in\{0.1, 0.5, 0.9\}.

Final Choice. We treat preserving answer ability as a primary requirement for behavioral calibration, since calibration improvements are less desirable when accompanied by degradation in the model’s underlying answer accuracy. Although the sinusoidal schedule achieves comparable Macro results across all three metrics, it provides no consistent advantage over the linear schedule. We therefore adopt the linear schedule as our default configuration and use it to scale RLEC to the 30B model, because it achieves stronger aggregate Forced Accuracy, establishes earlier and more persistent target separation, and retains a simpler form than the sinusoidal alternative.

Standard Deviation

RLEC removes the group standard deviation from advantage estimation to preserve the reward signal. To enable a closer comparison, we also remove this normalization from RLVR in our main experiments. To further isolate its effect on RLVR’s answer ability, we compare RLVR with centered advantages against standard RLVR under the same binary reward. As shown in Table 4, their aggregate Forced Accuracy is nearly identical, and neither variant consistently outperforms the other across benchmarks. Thus, removing the group standard deviation provides no clear advantage or disadvantage in answer ability in our experiment setting.

MethodGSM8KMATHAIMEHotpotOmniSciSimpleQAMMLU-ProGPQAMicroMacro
RLVR w/o Std.93.3369.7020.8369.5013.755.2068.4735.9158.1847.09
RLVR w/ Std.92.5772.3019.1768.3014.835.1568.9736.8058.4547.26

Table 4. Forced answer accuracy for RLVR with and without group standard deviation normalization. The best result in each column is shown in bold.

Increasing the Rollout Group Size

We further examine whether increasing the rollout group size can help ERT preserve the confidence target signal. We retain the original reward and standard deviation normalization, so this setting differs from ERT only by increasing the number of responses per prompt from 8 to 16. This change approximately doubles the rollout generation cost.

As shown in Figure 11, increasing the rollout group size does not enable ERT to distinguish confidence targets. On HotpotQA, groups containing all three reward values rapidly disappear, and groups containing a single reward dominate later training. DAPO-Math retains more groups containing all three rewards, but its abstention curves still largely overlap across different values of pp. Thus, although a larger rollout group can increase the chance of observing all three outcomes, it does not resolve the loss of target information in groups containing only one or two reward values. These results indicate that increasing rollout size alone cannot replace preserving the target signal during advantage estimation.

Figure 11. Scaling ERT to a rollout group size of 16. (a,b): group compositions during training. (c,d): abstention rates under different confidence targets.

Scaling RLEC to Longer Contexts

Due to computational constraints, our main experiments primarily use a 4K context window for both training and inference. However, difficult reasoning tasks such as AIME may benefit from a larger token budget. To examine whether RLEC can be extended to longer contexts, we increase both the training and inference context windows to 16K while keeping all other hyperparameters identical to those in Table 2.

Training Dynamics. As shown in Figure 12, increasing the context window from 4K to 16K leads to substantially longer responses from the 30B model. This additional budget produces a clear improvement in validation reward on DAPO-Math, whereas the HotpotQA trajectories remain largely unchanged. These results suggest that the longer context primarily benefits reasoning-intensive mathematical problems rather than uniformly improving all tasks. The stronger reasoning performance also changes the model’s abstention behavior on DAPO-Math. As shown in Figure 13, the 16K model abstains less than its 4K counterpart at higher confidence targets, particularly at p=0.9p = 0.9, while preserving the ordering across targets. This behavior is consistent with the stronger model answering more questions even under stringent confidence requirements.

Figure 12. Training dynamics under different model scales and context lengths. We report validation accuracy on HotpotQA and DAPO-Math and the average response length during training.

Figure 13. Abstention dynamics on DAPO-Math for different model scales and context lengths.

Evaluation Results. Table 5 reports the complete evaluation results. As expected, the largest improvement appears on AIME, where Forced Accuracy increases from 27.29% with the 4K context to 38.96% with the 16K context. Results on the remaining benchmarks are mixed and generally show smaller differences. Consequently, aggregate Forced Accuracy improves slightly, largely driven by the gain on AIME, while AOC and Score remain broadly comparable. Overall, the longer context primarily benefits difficult mathematical reasoning without materially affecting behavior calibration.

MetricContextGSM8KMATHAIMEHotpotOmniSciSimpleQAMMLU-ProGPQAMicroMacro
Acc. ↑\uparrow4K94.7778.6027.2966.6020.008.3076.7347.0964.1452.42
16K94.2875.0038.9665.6020.087.5577.9951.6864.8653.89
AOC ↓\downarrow4K0.0000.0030.0000.0090.2090.3680.0020.0720.0010.083
16K0.0000.0050.0000.0130.1820.3580.0040.0710.0040.079
Score ↑\uparrow4K0.5040.3530.0910.258-0.015-0.1320.3360.1120.2610.188
16K0.5020.3290.1570.2510.003-0.0840.3480.1110.2720.202

Table 5. Forced answer accuracy, average overconfidence (AOC), and average score for RLEC-30B with 4K and 16K contexts. The best Micro and Macro results for each metric are shown in bold.

Relation Between Score and the RLCR Reward

We show that the score used in our evaluation directly corresponds to the RLCR reward. Let y∈{0,1}y \in\{0,1\} indicate whether an answer is correct, and let c∈[0,1]c \in[0,1] be the confidence reported by RLCR. Given a confidence target pp, the external decision system answers when c≥pc \ge p and abstains otherwise. Its decision can therefore be written as

zp=1{c≥p}.(27)z_p = \mathbb{1}\{c \ge p\}. \tag*{(27)}

The score at target pp is

sp=zp(y−p)=1{c≥p}(y−p).(28)s_p = z_p(y-p) = \mathbb{1}\{c \ge p\}(y-p). \tag*{(28)}

Integrating this score over all confidence targets gives

∫01sp dp=∫011{c≥p}(y−p) dp(29)\int_0^1 s_p\,\mathrm{d}p = \int_0^1 \mathbb{1}\{c \ge p\}(y-p)\,\mathrm{d}p \tag*{(29)}
=∫0c(y−p) dp(30)= \int_0^c (y-p)\,\mathrm{d}p \tag*{(30)}
=cy−c22.(31)= cy - \frac{c^2}{2}. \tag*{(31)}

RLCR uses the following correctness and calibration reward:

RRLCR(c,y)=y−(c−y)2.(32)R_{\mathrm{RLCR}}(c,y) = y - (c-y)^2. \tag*{(32)}

Because y∈{0,1}y \in\{0,1\}, we have y2=yy^2=y. The reward can therefore be rewritten as

RRLCR(c,y)=y−c2+2cy−y2(33)R_{\mathrm{RLCR}}(c,y) = y-c^2+2cy-y^2 \tag*{(33)}
=2cy−c2(34)= 2cy-c^2 \tag*{(34)}
=2(cy−c22)(35)= 2\left(cy-\frac{c^2}{2}\right) \tag*{(35)}
=2∫01sp dp.(36)= 2\int_0^1 s_p\,\mathrm{d}p. \tag*{(36)}

Thus, when the confidence reported by RLCR is converted into an answer or abstention through thresholding, twice the score integrated over all confidence targets is exactly the RLCR reward.

In our experiments, we evaluate a finite set of confidence targets P\mathcal{P} and compute

Score^=1∣P∣∑p∈Psp.(37)\widehat{\mathrm{Score}} = \frac{1}{|\mathcal{P}|}\sum_{p\in\mathcal{P}}s_p. \tag*{(37)}

For uniformly spaced targets over [0,1)[0,1), this is a Riemann approximation to the integral in Equation (31). The approximation approaches the exact relation in Equation (36) as the interval between evaluated targets decreases.

The same score also directly corresponds to the RLEC reward. For a given pp,

zp(y−p)={1−p,correct answer,−p,incorrect answer,0,abstention.(38)z_p(y-p) = \begin{cases} 1-p, & \text{correct answer},\\ -p, & \text{incorrect answer},\\ 0, & \text{abstention}. \end{cases} \tag*{(38)}

which is exactly Rp(1)R_p^{(1)}, the RLEC reward with b=1b=1. Therefore, Score provides a common measure for comparing RLEC and RLCR under explicit confidence targets.

Limitations and Future Work

Under idealized assumptions, our reward objective induces the decision rule q(x)>pq(x)>p, guaranteeing Acc⁡(p)≥p\operatorname{Acc}(p)\ge p among answered questions. In practice, limited model capacity, imperfect estimates of answer correctness, finite rollouts, and stochastic optimization can prevent the learned policy from fully realizing this rule, while distribution shifts can further weaken target compliance. Our experiments show clear adaptation to explicit confidence targets and improved aggregate target compliance relative to standard RLVR. However, the requested accuracy is not consistently attained on every benchmark: the remaining gaps are most visible on challenging knowledge-intensive tasks such as SimpleQA-Verified and AA-Omniscience, particularly with the 4B model. Thus, the theoretical objective does not guarantee that the trained model will satisfy every requested target in practice. Future work should explore improved training objectives and data to further improve calibration and reduce target violations, especially on difficult tasks and under distribution shifts, while preserving answer ability and fine-grained target control.

References

  1. [1]Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.arxiv.org/abs/2303.08774
  2. [2]Michael Bereket and Jure Leskovec. Uncalibrated reasoning: Grpo induces overconfidence for stochastic outcomes. arXiv preprint arXiv:2508.11800, 2025.arxiv.org/abs/2508.11800
  3. [3]Aili Chen, Nonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, et al. Minimax-ml: Scaling test-time compute efficiently with lightning attention. arXiv preprint arXiv:2506.13585, 2025.
  4. [4]Hui Chen, Antoine Didisheim, and Luciano A Somoza. Out of the black box: Uncertainty quantification for llms via conditional probabilities. Technical report, National Bureau of Economic Research, 2026.
  5. [5]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
  6. [6]Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025.arxiv.org/abs/2507.06261
  7. [7]Yuyang Dai and Yuxia Wang. Rescaling confidence: What scale design reveals about llm metacognition. arXiv preprint arXiv:2603.09309, 2026.
  8. [8]Mehul Damani, Isha Puri, Stewart Slocum, Idan Shenfeld, Leshem Choshen, Yoon Kim, and Jacob Andreas. Beyond binary rewards: Training lms to reason about their uncertainty. In International conference on learning representations, volume 2026, pages 59456–59493, 2026.arxiv.org/abs/2507.16806
  9. [9]Cheng Ge, Caitlyn Heqi Yin, Hao Liang, and Jiawei Zhang. Why grpo needs normalization: A local-curvature perspective on adaptive gradients. arXiv preprint arXiv:2601.23135, 2026.arxiv.org/abs/2601.23135
  10. [10]Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.DOI
  11. [11]Lukas Haas, Gal Yona, Giovanni D’Antonio, Sasha Goldshtein, and Dipanjan Das. Simpleqa verified: A reliable factuality benchmark to measure parametric knowledge. arXiv preprint arXiv:2509.07968, 2025.arxiv.org/abs/2509.07968
  12. [12]Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021.
  13. [13]Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM transactions on information systems, 43(2):1–55, 2025.
  14. [14]Muhammed Emrullah Ildiz, Halil Alperen Gozeten, Ege Onur Taga, and Samet Oymak. Learning to correct: Calibrated reinforcement learning for multi-attempt chain-of-thought. arXiv preprint arXiv:2604.17912, 2026.
  15. [15]Declan Jackson, William Keating, George Cameron, and Micah Hill-Smith. Aa-omniscience: Evaluating cross-domain knowledge reliability in large language models. arXiv preprint arXiv:2511.13029, 2025.arxiv.org/abs/2511.13029
  16. [16]Abha Jha, Akanksha Mahajan, Ashwath Vaithinathan Aravindan, Praveen Saravanan, Sai Sailaja Policharla, and Sonal Chaturbhuj Gehlot. Rewarding intellectual humility learning when not to answer in large language models. arXiv preprint arXiv:2601.20126, 2026.arxiv.org/abs/2601.20126
  17. [17]Ziwei Ji, Nayeon Lee, Rita Frieske, T iezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM computing surveys, 55(12):1–38, 2023.
  18. [18]Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221, 2022.
  19. [19]Adam Tauman Kalai, Ofir Nachum, Santosh S Vempala, and Edwin Zhang. Why language models hallucinate. arXiv preprint arXiv:2509.04664, 2025.arxiv.org/abs/2509.04664
  20. [20]Amita Kamath, Robin Jia, and Percy Liang. Selective question answering under domain shift. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 5684–5696, 2020.arxiv.org/abs/2006.09462
  21. [21]Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In International Conference on Learning Representations, volume 2024, pages 39578–39601, 2024.arxiv.org/abs/2305.20050
  22. [22]Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. arXiv preprint arXiv:2205.14334, 2022.arxiv.org/abs/2205.14334
  23. [23]Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025.arxiv.org/abs/2503.20783
  24. [24]Mohamad Amin Mohamadi, Tianhao Wang, and Zhiyuan Li. Honesty over accuracy: Trustworthy language models through reinforced hesitation. arXiv preprint arXiv:2511.11500, 2025.arxiv.org/abs/2511.11500
  25. [25]Muyu Pan, Shu Zhao, Nan Zhang, Philip Shin, Varun Parekh, Vijaykrishnan Narayanan, and Rui Zhang. Tiar: Trajectory-informed advantage reweighting for llm abstention learning. arXiv preprint arXiv:2605.25850, 2026.arxiv.org/abs/2605.25850
  26. [26]David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanze Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022, 2023.arxiv.org/abs/2311.12022
  27. [27]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.arxiv.org/abs/2402.03300
  28. [28]Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, pages 1279–1297, 2025.DOI
  29. [29]Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, Jianfeng Cai, Xinyuan Cai, Peizhou Cao, Yuxuan Cao, Ziwei Chai, Y Charles, et al. Kimi k3: Open frontier intelligence. arXiv preprint arXiv:2607.24653, 2026.
  30. [30]Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5433–5442, 2023.arxiv.org/abs/2305.14975
  31. [31]Benjamin Turtel, Danny Franklin, Kris Skotheim, Luke Hewitt, and Philipp Schoenegger. Outcome-based reinforcement learning to predict the future. arXiv preprint arXiv:2505.17989, 2025.arxiv.org/abs/2505.17989
  32. [32]Qingni Wang, Yue Fan, and Xin Wang. Safer: Risk-constrained sample-then-filter in large language models. In International Conference on Learning Representations, volume 2026, pages 53723–53747, 2026.arxiv.org/abs/2510.10193
  33. [33]Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. Advances in Neural Information Processing Systems, 37:95266–95290, 2024.arxiv.org/abs/2406.01574
  34. [34]Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality in large language models. arXiv preprint arXiv:2411.04368, 2024.arxiv.org/abs/2411.04368
  35. [35]Zhepei Wei, Xiao Yang, Kai Sun, Jiaqi Wang, Rulin Shao, Jingxiang Chen, Mohammad Kachuee, Teja Gollapudi, Yiwei Liao, Nicolas Scheffer, et al. Truthful: Incentivizing truthful llms via reinforcement learning. arXiv preprint arXiv:2509.25760, 2025.arxiv.org/abs/2509.25760
  36. [36]Jiayun Wu, Jiashuo Liu, Zhiyuan Zeng, Tianyang Zhan, Tianle Cai, and Wenhao Huang. Mitigating llm hallucination via behaviorally calibrated reinforcement learning. arXiv preprint arXiv:2512.19920, 2025.arxiv.org/abs/2512.19920
  37. [37]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengeng Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.arxiv.org/abs/2505.09388
  38. [38]Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 2369–2380, 2018.arxiv.org/abs/1809.09600
  39. [39]Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. Advances in Neural Information Processing Systems, 38:113222–113244, 2026.arxiv.org/abs/2503.14476

Paper details

vlnc:W_b4k22zj5
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents