betaFeedback

WorksComputer Science

Each Insufficient in Its Own Way: Calibrating Language Models Across Varying Evidence States

Abstract

How can a language model know whether it has enough evidence to answer? Language models tend to answer even when the evidence they have is insufficient. Deciding whether to answer or gather more evidence requires a calibrated estimate of how likely an answer is to be correct with the available evidence. However, most calibration work treats the input as fixed, without asking how this estimate should change with the evidence. We introduce evidence-state calibration, which requires a model's confidence before answering to match its probability of answering correctly and to change accordingly whenever the evidence changes this probability. We propose SuffiCE (Sufficiency Confidence Estimation), which fine-tunes the confidence already present in a model's next-token distribution to match how often the model answers correctly from each evidence state. Across clinical diagnosis and multi-hop question answering with four models from two families, SuffiCE reduces the Brier score by up to 60% and increases AUROC by up to 0.16. Its confidence matches the empirical success rate and responds not only to the amount of evidence but also to which evidence is present, showing that models can learn to judge whether their current evidence is enough to answer.

Introduction

Before answering a question, one must judge whether the information at hand is enough to answer it. In high-stakes settings, a physician who diagnoses before gathering enough information may miss the real disease. For example, a patient who has been tired for two months may seem iron deficient, but a careful clinician would order more blood tests, which may reveal celiac disease, an autoimmune cause of poor iron absorption.

Language models facing insufficient information, however, often reach confident but incorrect conclusions [18] (Yin et al. 2023).

A common remedy for such overconfidence is to abstain below a confidence threshold (Wen et al., 2025). This works only if the model is calibrated, so that answers given x% confidence are correct about x% of the time [8, 6]. To achieve this, existing methods train language models to provide calibrated confidence estimates for generated answers [16, 3].

However, most calibration work holds the evidence fixed for each question, so it can check whether confidence is calibrated but not whether it varies correctly as a function of the evidence. In interactive settings, the available information evolves as models ask clarifying questions (Zhang and Choi, 2025) or seek additional information (Li et al., 2024). At each turn, the model must decide whether to answer or keep gathering evidence.

To make this decision, a model needs a confidence estimated from the available evidence alone, before it produces any answer. We call the evidence available to a model at a given moment its evidence state. We then define evidence-state calibration as requiring the confidence before answering to match the probability of answering correctly, so that whenever the evidence changes this probability, the confidence changes by the same amount (Figure 1).

Calibrated and miscalibrated confidence across varying evidence states

Figure 1. Given a question with two different sets of evidence, the model’s probability of answering correctly (gray) differs, and its confidence before answering (yellow) should match it for each. (a) Calibrated confidence matches both. (b) Miscalibrated confidence mismatches the dynamic of the true probability.

Testing this requires observing the same problem under different evidence states. Rather than contrasting the extremes of complete and missing information [18] or following a single fixed sequence of accumulating evidence (Zhang et al., 2026), we construct granular states for each problem, spanning minimal to complete evidence and including different combinations of information. Granularity matters because sufficient evidence states are alike, but each insufficient one is insufficient in its own way.1 Some insufficient states can still be answered from what the model already knows [14], while others lack the one piece of evidence the answer depends on. We therefore measure each state’s sufficiency by how often the model answers correctly from it, not by whether the required facts are present, and teach the model to anticipate this success rate before answering.

With these success rates as targets, we propose SuffiCE (Sufficiency Confidence Estimation), to our knowledge the first method that trains a model’s confidence before answering to be evidence-state calibrated. It reads the confidence directly from the model’s own next-token distribution. The confidence is the model’s probability of choosing to answer instead of asking for more information. No additional classifier or generated number is needed. This avoids an ambiguity in verbalized confidence. When a model states that it is 80% confident, it is unclear whether to trust the stated number or the probability the model assigned to generating it. We instead train the model’s own probability directly to match the success rate across our diverse evidence states. The result is a model that can tell, before answering, whether what it currently knows is enough to answer correctly.

We evaluate on two clinical diagnosis datasets, MIMIC [9] and MedQA [13], and the multi-hop question answering dataset MuSiQue (Trivedi et al., 2022). Across three Qwen models from 4B to 27B parameters and Kimi K2.6, SuffiCE improves calibration for every model and dataset, reducing the Brier score

of the base logit by up to 60% and increasing its AUROC by up to 0.16. Its confidence also follows the evidence state. An estimator trained on complete evidence alone falls short on every metric, so these gains come from training across evidence states. In practice, answering only when SuffiCE is confident gives fewer wrong answers than any baseline.

Related Work

Deciding whether to answer or gather more evidence requires a calibrated judgment of whether the current evidence suffices. This connects three lines of work: interactive information seeking explores how models acquire evidence, information sufficiency and unanswerability examine whether an input supports an answer, and confidence calibration studies whether confidence reflects actual accuracy.

Interactive information seeking. In an interactive setting, a model can gather more information before answering by asking clarifying questions (Zhang and Choi, 2025; Li et al., 2024), retrieving documents [12, 1], or ordering diagnostic tests and searching for medical evidence [5] (Schmidgall et al. 2026). Deciding when enough has been gathered typically relies on a confidence or uncertainty signal [12] (Zhang and Choi 2025; Li et al. 2024), and MediQ shows that improving confidence estimation can enhance interactive diagnostic accuracy (Li et al., 2024). Our approach aims to provide such a signal, estimating before answering how likely the current evidence is to yield a correct answer. Closest to our work, LA-CDM trains multi-turn diagnosis with a verbalized confidence that guides whether to request another test, but this confidence evaluates a hypothesis the model has already generated [2]. We instead estimate success before generating an answer and train this estimate across controlled evidence states of the same problem, capturing how changes in evidence affect the likelihood of success.

Information sufficiency and unanswerability. Whether an input contains enough information to answer a question has been widely studied through unanswerability (Rajpurkar et al., 2018; Yin et al., 2023). Benchmarks test whether models abstain when information is insufficient [18], including settings where key evidence is removed from otherwise answerable questions (Trivedi et al., 2022; Machcha et al., 2026). Detection methods assess sufficiency using model hidden states (Slobodkin et al., 2023; Lavi et al., 2026) or by identifying missing information [11], and training methods teach models to abstain when information is insufficient (Zhai et al., 2026). A shared limitation is that sufficiency is a binary label attached to the input, so a partially informative input counts as either sufficient or not, regardless of which model answers it. Joren et al. [14] take a first step, showing that models often answer correctly from insufficient context. Whether evidence is enough depends on the model answering, so we define sufficiency by the model’s success, how often it answers correctly from each controlled evidence state.

Confidence calibration. A model is calibrated when its confidence matches its accuracy [8]. Training-free estimates based on verbalized judgments (Tian et al., 2023; Xiong et al., 2024), sampled-answer consistency (Kuhn et al., 2023), or token probabilities [15] can be informative, but calibration varies across tasks (Vashurin et al., 2025). Trained methods estimate answer correctness [16], verbalize calibrated confidence [3, 4] (Lin et al. 2022; Li et al. 2025), or predict success before answering [15] (Lugoloobi et al. 2026). However, they do not explicitly train confidence to track success across changing evidence states. Zhang et al. (2026) evaluate confidence as hints accumulate in a fixed sequence but do not train on these changing states. We instead train and evaluate confidence across controlled evidence states, varying both the amount and type of evidence to track changes in empirical success (Appendix § 12).

Method

Our method, SuffiCE, has three stages. We first construct multiple evidence states for each problem (§ 3.2), then estimate a fixed answering policy’s success rate on each state from repeated answers (§ 3.3), and finally train an estimator to predict this rate from the state alone, before answering (§ 3.4).

Problem Formulation

Problem ii consists of a question qiq_i, a pool of evidence EiE_i, and a gold answer yi∗y_i^*. An evidence state x=(qi,I)x = (q_i, I) pairs the question with the evidence I⊆EiI \subseteq E_i available to the model.

A fixed answering policy π0\pi_0 samples an answer y^∼π0(⋅∣x)\hat{y} \sim\pi_0(\cdot\mid x), which a grader scores as g(y^,yi∗)∈{0,1}g(\hat{y}, y_i^*) \in\{0,1\}. The success probability of an evidence state is

p∗(x)=Pr⁡y^∼π0(⋅∣x)[g(y^,yi∗)=1].(1)p^*(x) = \Pr_{\hat{y}\sim\pi_0(\cdot\mid x)}\left[g(\hat{y}, y_i^*) = 1\right]. \tag*{(1)}

We measure how sufficient a state is by p∗(x)p^*(x), because the decision to answer or gather more information hinges on how likely an answer is to be correct. Sufficiency therefore depends on the model as well as the information. A state is sufficient when its information, together with what the model already knows, lets the model reach the correct answer, whether or not the information itself is complete. The same state can thus be sufficient for one model and insufficient for another.

Evidence States

For each problem ii, we construct mim_i evidence states xi,s=(qi,Ii,s)x_{i,s} = (q_i, I_{i,s}), s=1,…,mis = 1,\ldots,m_i, each pairing the question with a subset Ii,s⊆EiI_{i,s} \subseteq E_i of its evidence. We write xx for a generic state when the indices are not needed. All states of a problem share the question and gold answer and differ only in the evidence shown, so differences in success between them are attributable to the evidence. The model is never told which state it is in and sees only the question and the evidence, so it must judge from their content whether the evidence is enough. Training on the same problem under different evidence teaches it which evidence makes a correct answer reachable.

Evidence states vary in both the amount and the composition of evidence. Since success depends on which evidence is present, not on the order it was gathered, states can combine evidence freely, including states that hold the amount fixed and vary the composition (§ 5.3). The states are specific to each dataset (§ 4).

Evidence-state calibration. We seek a confidence cθ(x)∈[0,1]c_\theta(x) \in[0,1] that a model with parameters θ\theta computes from the state xx alone, before any answer is generated. Evidence-state calibration requires it to satisfy two criteria, where expectations are taken over the constructed evidence states.

1. Calibration. Across all states assigned the same confidence cc, the average success probability is cc [8]:

E[p∗(x)∣cθ(x)=c]=c∀c∈range⁡(cθ).(2)\mathbb{E}\left[p^*(x) \mid c_\theta(x)=c\right] = c \quad\forall c \in\operatorname{range}(c_\theta). \tag*{(2)}

2. Evidence sensitivity. A change in evidence drives confidence as much as it moves success:

cθ(xi,t)−cθ(xi,s)≈p∗(xi,t)−p∗(xi,s).(3)c_\theta(x_{i,t})-c_\theta(x_{i,s}) \approx p^*(x_{i,t})-p^*(x_{i,s}). \tag*{(3)}

Here xi,sx_{i,s} and xi,tx_{i,t} denote any two evidence states ss and tt of the same problem ii, so the difference in confidence should match the difference in success.

To see why calibration alone is not enough, we split the squared error of a confidence by problem. For problem ii with mim_i states, let cˉθ,i\bar{c}_{\theta,i} and pˉi∗\bar{p}_i^* denote the averages of cθc_\theta and p∗p^* over its states, and for any two of its states s,t∈{1,…,mi}s,t\in\{1,\ldots,m_i\}, let

δst=(cθ(xi,t)−cθ(xi,s))−(p∗(xi,t)−p∗(xi,s))\delta_{st}=\left(c_\theta(x_{i,t})-c_\theta(x_{i,s})\right)-\left(p^*(x_{i,t})-p^*(x_{i,s})\right)

be the gap between the change in confidence and the change in success from ss to tt, which is zero when Eq. (3) holds exactly.

Error Decomposition

Proposition 1. For any confidence cθc_\theta,

E[(cθ−p∗)2]=Ei[(cˉθ,i−pˉi∗)2]+Ei[12mi2∑s,tδst2],(4)\mathbb{E}\left[(c_\theta-p^*)^2\right]=\mathbb{E}_i\left[(\bar{c}_{\theta,i}-\bar{p}_i^*)^2\right]+\mathbb{E}_i\left[\frac{1}{2m_i^2}\sum_{s,t}\delta_{st}^2\right], \tag*{(4)}

where the sum runs over all ordered pairs of states of problem ii, and each problem is weighted by its number of states.

The first term is the error in each problem’s average success, and the second is the error within each problem, which is zero exactly when criterion 2 holds with equality (proof in Appendix § 11.§ 1). Calibration does not constrain the second term. Giving every state its problem’s average success satisfies criterion 1 exactly but leaves this term at the variance of success within each problem, which only following the evidence can reduce. Since both terms are nonnegative, we seek a confidence that drives the left side to zero and thus satisfies both criteria.

Monte Carlo Estimation of Success Probability

Because computing the true success probability p∗(x)p^*(x) exactly would require grading every answer the policy could generate, we estimate it by sampling from the fixed answering policy (Figure 2, top). A single answer is only correct or incorrect, so we draw KK independent answers per state and average their grades. For each evidence state xx of problem ii, we sample y^(1),…,y^(K)∼π0(⋅∣x)\hat{y}^{(1)},\ldots,\hat{y}^{(K)}\sim\pi_0(\cdot\mid x), grade each as r(k)=g(y^(k),yi∗)r^{(k)}=g(\hat{y}^{(k)},y_i^*), and define the empirical success rate as

Overview of SuffiCE

Figure 2. Overview of SuffiCE. Each evidence state x=(q,I)x=(q,I) pairs an input question qq with the available evidence II. Top: A frozen answering policy samples KK answers per evidence state, and the fraction graded correct is the empirical success rate p^(x)\hat{p}(x). Bottom: The estimator reads its confidence from the softmax over the logits of A (Answer) and S (Seek), and is trained with soft-label binary cross-entropy to match p^(x)\hat{p}(x).

p^(x)=1K∑k=1Kr(k).(5)\hat{p}(x)=\frac{1}{K}\sum_{k=1}^{K}r^{(k)}. \tag*{(5)}

Since r(k)∼Bernoulli⁡(p∗(x))r^{(k)}\sim\operatorname{Bernoulli}(p^*(x)), E[p^(x)∣x]=p∗(x)\mathbb{E}[\hat{p}(x)\mid x]=p^*(x), so p^(x)\hat{p}(x) is an unbiased Monte Carlo estimate of the frozen policy’s success probability.

The estimate p^(x)\hat{p}(x) is specific to the answering policy π0\pi_0, so the sufficiency of a state xx always refers to whether π0\pi_0 can answer correctly given the information in xx, and the confidence we train is calibrated to this policy. Since computing p^(x)\hat{p}(x) requires the gold answer, which is unavailable at decision time, we use it only as a training target.

Sufficiency Confidence Estimator

At decision time, the model must therefore predict p^(x)\hat{p}(x) from the evidence state alone. We read this prediction from the model’s own next-token distribution rather than adding a separate classifier head (Figure 2, bottom). Each evidence state is placed in a prompt that asks the model, without answering the task, whether the currently available information is sufficient for it to answer correctly, and defines two actions, A to answer now because the information is sufficient, and S to seek more information because it is not. Rather than letting the model generate a response, we read the probabilities it assigns to A and S as the first token of its reply (prompt in Appendix § 9.§ 1).

Let ℓA(x)\ell_A(x) and ℓS(x)\ell_S(x) denote the logits assigned to A and S at the first response position from a single forward pass. We take the softmax over these two logits, giving cθ(x)=Pθ(A∣x)=exp⁡(ℓA)/(exp⁡(ℓA)+exp⁡(ℓS))c_\theta(x) = P_\theta(A \mid x) = \exp(\ell_A) / (\exp(\ell_A) + \exp(\ell_S)). We call cθ(x)c_\theta(x) the sufficiency confidence of state xx, and the model that produces it the sufficiency confidence estimator.

We train cθ(x)c_\theta(x) to match the empirical success rate using soft-label binary cross-entropy:

L(x)=−p^(x)log⁡cθ(x)−(1−p^(x))log⁡(1−cθ(x)).(6)\mathcal{L}(x) = -\hat{p}(x)\log c_\theta(x) - (1-\hat{p}(x))\log(1-c_\theta(x)). \tag*{(6)}

Binary cross-entropy is a strictly proper scoring rule ([7]) and is linear in the target, so with the unbiased labels of § 3.3 the expected loss at each state is minimized by cθ(x)=p∗(x)c_\theta(x) = p^*(x) despite sampling noise in p^(x)\hat{p}(x). Training on this loss also controls both terms of Proposition 1.

Corollary 1. For any confidence cθc_\theta, let E[L]\mathbb{E}[\mathcal{L}] be the loss of Eq. (6) averaged over all evidence states and over the KK sampled answers that determine p^(x)\hat{p}(x), and L∗\mathcal{L}^* the minimum expected loss over all confidences, reached exactly at cθ=p∗c_\theta= p^*. Then

E[(cθ−p∗)2]≤12(E[L]−L∗),(7)\mathbb{E}\left[(c_\theta-p^*)^2\right] \le\frac{1}{2}\left(\mathbb{E}[\mathcal{L}]-\mathcal{L}^*\right), \tag*{(7)}

so each term of Eq. (4) is at most 12(E[L]−L∗)\frac{1}{2}\left(\mathbb{E}[\mathcal{L}]-\mathcal{L}^*\right). Lowering the loss therefore lowers an upper bound on both terms, and the bound reaches zero exactly at cθ=p∗c_\theta= p^* (proof in Appendix § 11.§ 2).

Experimental Setup

Datasets. We evaluate on two clinical diagnosis datasets and one multi-hop question answering dataset, which range from mostly answered correctly to mostly not (details in Appendix § 8.§ 1). MIMIC ([9]) consists of real emergency-department admissions for four abdominal diseases, and the model chooses among the four. MedQA ([13]) contains questions from the United States Medical Licensing Examination, and the model names the diagnosis in free text. Both are written as structured clinical cases following AgentClinic ((Schmidgall et al., 2026)). MuSiQue ((Trivedi et al., 2022)) composes multi-hop questions over Wikipedia from single-hop questions, each answered by one supporting paragraph, and the model gives a short answer. All states of a problem are assigned to the same split to avoid leakage between training and test.

Evidence states. Clinical cases are divided into history, physical examination, imaging, and other tests such as laboratory and pathology. Each state adds a different subset of the remaining sections to the history.

In MuSiQue, we use 3-hop questions, whose evidence units are the three supporting paragraphs, one per sub-question. In chain questions, each answer is needed to ask the next sub-question (e1→e2→e3e_1 \to e_2 \to e_3), while in branching questions, e1e_1 and e2e_2 answer independent sub-questions that both feed the final one, e3e_3. States range from the question alone and distractors only (related paragraphs that MuSiQue retrieves but that do not help answer the question) to one, two, or all three supporting paragraphs (Figure 3, examples in Appendix § 13.§ 2). Every state but the question-only one shows three paragraphs, padded with distractors in one shuffled order per question, so neither the number nor the position of paragraphs reveals the state.

Figure 3. Evidence states of MuSiQue for chain and branching questions. Hatched cells give the number of distractors padding each state to three paragraphs.

Baselines. We compare against three baselines on the same untrained model. Base logit (Base) is our sufficiency confidence estimator before training, reading P(A∣x)P(A \mid x) under the prompt of § 3.4. Verbalized asks the model to state the probability that it could answer correctly from the available information ((Tian et al., 2023)). Platt scaling (Platt, 1999) rescales the base logit to fit the success rates in the training set, so it can recalibrate the confidence but not reorder states (prompts in Appendix § 9).

Implementation details. We evaluate four answering models, three Qwen models of increasing size (Qwen3.5-4B, Qwen3.5-9B, and Qwen3.8-27B) and Kimi K2.6 from a different model family. The answering policy π0\pi_0 can be any frozen policy. Our main experiments use the estimator’s own base model, to study how well a model can judge its own evidence state. Success rates are estimated from K=10K = 10 sampled answers per state. Further details are in Appendix § 8.§ 3.

Metrics. Let ci,s=cθ(xi,s)c_{i,s} = c_{\theta}(x_{i,s}) be the sufficiency confidence of state ss of problem ii and p^i,s\hat{p}_{i,s} its empirical success rate. The Brier score, the mean of (ci,s−p^i,s)2(c_{i,s} - \hat{p}_{i,s})^2 over the test set, measures overall error. AUROC, averaged over the KK sampled answers, measures how well confidence ranks correct answers above incorrect ones. Both pool the whole test set, so they reward knowing which problems are hard. The change correlation ρΔ=Corr⁡(ci,t−ci,s,p^i,t−p^i,s)\rho_{\Delta} = \operatorname{Corr}(c_{i,t} - c_{i,s}, \hat{p}_{i,t} - \hat{p}_{i,s}), computed over all pairs of states of the same problem from a state ss to a state tt with more evidence (Appendix § 10.§ 4), instead tests whether confidence follows success as the evidence state changes, as criterion 2 requires.

Results

Overall Performance

SuffiCE achieves the best result on every metric in every setting (Table 1). Each model is calibrated to its own success rates, so methods are compared only for the same model (Appendix § 8.§ 2). SuffiCE lowers the Brier score by 20–60% relative to the base logit and by 5–28% relative to Platt scaling, the strongest baseline. SuffiCE not only calibrates better but also ranks states better. Platt scaling fits the same labels but only rescales the base confidence, leaving the order of states, and thus AUROC and selective accuracy, unchanged, whereas SuffiCE raises AUROC by up to 0.16. SuffiCE also raises the change correlation ρΔ\rho_{\Delta} above every baseline, so its confidence moves with success as the evidence changes, which §5.3 examines in detail.

MethodMIMIC Brier ↓MIMIC AUROC ↑MIMIC ρΔ\rho_{\Delta} ↑MedQA Brier ↓MedQA AUROC ↑MedQA ρΔ\rho_{\Delta} ↑MuSiQue Brier ↓MuSiQue AUROC ↑MuSiQue ρΔ\rho_{\Delta} ↑
Kimi K2.6
Base logit0.219 ± 0.0000.686 ± 0.0000.320 ± 0.0000.223 ± 0.0000.707 ± 0.0000.413 ± 0.0000.184 ± 0.0000.750 ± 0.0000.430 ± 0.000
Verbalized0.131 ± 0.0010.686 ± 0.0030.247 ± 0.0130.162 ± 0.0010.650 ± 0.0010.374 ± 0.0100.174 ± 0.0030.724 ± 0.0030.412 ± 0.011
Platt0.124 ± 0.0000.686 ± 0.0000.296 ± 0.0000.150 ± 0.0000.707 ± 0.0000.438 ± 0.0000.142 ± 0.0000.750 ± 0.0000.431 ± 0.000
SuffiCE0.105 ± 0.0020.844 ± 0.0050.474 ± 0.0450.141 ± 0.0050.795 ± 0.0330.467 ± 0.0640.134 ± 0.0040.787 ± 0.0050.632 ± 0.009
Qwen3.8-27B
Base logit0.127 ± 0.0000.788 ± 0.0000.353 ± 0.0000.143 ± 0.0000.744 ± 0.0000.413 ± 0.0000.182 ± 0.0000.738 ± 0.0000.274 ± 0.000
Verbalized0.112 ± 0.0000.770 ± 0.0070.222 ± 0.0030.174 ± 0.0010.772 ± 0.0010.355 ± 0.0000.203 ± 0.0010.682 ± 0.0020.345 ± 0.008
Platt0.099 ± 0.0000.788 ± 0.0000.286 ± 0.0000.126 ± 0.0000.744 ± 0.0000.438 ± 0.0000.123 ± 0.0000.738 ± 0.0000.423 ± 0.000
SuffiCE0.080 ± 0.0040.857 ± 0.0120.447 ± 0.0400.091 ± 0.0050.826 ± 0.0080.489 ± 0.0190.092 ± 0.0080.830 ± 0.0140.717 ± 0.024
Qwen3.5-9B
Base logit0.124 ± 0.0000.689 ± 0.0000.247 ± 0.0000.268 ± 0.0000.746 ± 0.0000.367 ± 0.0010.197 ± 0.0000.752 ± 0.0000.356 ± 0.000
Verbalized0.143 ± 0.0030.727 ± 0.0020.177 ± 0.0110.290 ± 0.0010.706 ± 0.0020.356 ± 0.0040.145 ± 0.0030.688 ± 0.0040.310 ± 0.012
Platt0.110 ± 0.0000.689 ± 0.0000.254 ± 0.0000.134 ± 0.0000.746 ± 0.0000.514 ± 0.0010.099 ± 0.0000.752 ± 0.0000.389 ± 0.000
SuffiCE0.098 ± 0.0100.819 ± 0.0280.393 ± 0.0340.107 ± 0.0050.815 ± 0.0110.630 ± 0.0120.078 ± 0.0020.828 ± 0.0060.666 ± 0.015
Qwen3.5-4B
Base logit0.143 ± 0.0000.761 ± 0.0000.311 ± 0.0000.132 ± 0.0000.763 ± 0.0000.452 ± 0.0000.122 ± 0.0000.779 ± 0.0000.335 ± 0.000
Verbalized0.190 ± 0.0010.763 ± 0.0010.095 ± 0.0090.250 ± 0.0050.739 ± 0.0060.286 ± 0.0270.164 ± 0.0000.601 ± 0.0010.224 ± 0.003
Platt0.097 ± 0.0000.761 ± 0.0000.274 ± 0.0000.108 ± 0.0000.763 ± 0.0000.488 ± 0.0000.097 ± 0.0000.779 ± 0.0000.400 ± 0.000
SuffiCE0.085 ± 0.0030.830 ± 0.0140.458 ± 0.0970.103 ± 0.0030.773 ± 0.0110.524 ± 0.0720.086 ± 0.0040.818 ± 0.0070.606 ± 0.020

Table 1. Results on the three datasets for Kimi K2.6, Qwen3.8-27B, Qwen3.5-9B, and Qwen3.5-4B, comparing our method with baselines. Results are mean ± SD over three seeds. The best result for each model is in bold.

Calibration across datasets. The base logit is miscalibrated by different amounts across datasets, and SuffiCE improves calibration on all three, moving each curve toward the ideal diagonal (Figure 4, Qwen3.5-9B). The other models follow the same pattern (Appendix § 10.§ 1).

Figure 4. SuffiCE brings confidence closer to perfect calibration on every dataset. Calibration curves on the test set (a) before and (b) after SuffiCE.

Selective answering. If the model answers only the questions it is most confident about and abstains on the rest, SuffiCE is the most accurate at nearly every percentage of questions answered (Table 2 and Appendix § 10.§ 2). On MIMIC, the questions it is most confident about are almost always answered correctly.

AnsweredBaseVerb.SuffiCE
MIMIC10%97.295.3100.0
30%93.495.398.5
50%89.992.495.5
MedQA10%93.774.399.1
30%77.873.484.2
50%65.463.374.0
MuSiQue10%55.252.568.9
30%45.347.756.2
50%39.434.742.6

Table 2. Accuracy (%) when the model answers the given percentage of questions it is most confident about.

Ablation

To test whether the estimator needs diverse evidence states, we train Qwen3.5-9B only on the full state of each training problem and evaluate it on the whole test set (Figure 5). Success on full states still varies across problems, so this estimator learns which problems are hard but never sees missing evidence. SuffiCE outperforms it on every metric and dataset, both with the same number of training epochs and with the same number of steps (Appendix § 10.§ 3).

Figure 5. Training across evidence states improves both calibration and evidence sensitivity. Estimators trained for the same number of epochs on full states only versus on all evidence states (SuffiCE).

Evidence-State Analysis

Confidence rises with the decisive evidence, not with more evidence. SuffiCE closely tracks the empirical success rate across states, rising only when new evidence makes the question answerable, whereas the base logit rises with every paragraph added (Figure 6, more measures in Appendix § 10.§ 4). For both question types, success stays near zero until the final hop e3e_3 is shown. On chain questions, the base logit treats each earlier hop as progress and is confident enough to answer given {e1,e2}\{e_1, e_2\} (0.71), although the model almost never succeeds (0.06), while SuffiCE stays low (0.16) until e3e_3 arrives. On branching questions, the base logit rises with each branch, while SuffiCE stays low for either branch alone and for both, since neither helps without e3e_3. SuffiCE is never told the question type or which paragraph fills which hop, so it judges from the content alone whether the reasoning chain is complete. It thus separates evidence that makes a question answerable from evidence that only adds information (Appendix § 13.§ 1).

Figure 6. SuffiCE tracks the empirical success rate as supporting paragraphs are added, while the base logit rises with every paragraph. MuSiQue test set, for (a) chain and (b) branching questions.

Holding the amount fixed, content decides success. Even when every state has the same amount of evidence, SuffiCE distinguishes which states can be answered (Figure 7). Each state shows two of the three supporting paragraphs, and success is high only when the final hop e3e_3 is among them. The base logit is similarly confident on all three states, responding to how much evidence there is, whereas SuffiCE stays low on {e1,e2}\{e_1, e_2\} and rises for both states with e3e_3.

Figure 7. With the same amount of evidence, SuffiCE tells which states can be answered. Each MuSiQue state shows two of the three supporting paragraphs, for (a) chain and (b) branching questions. Only SuffiCE stays low until the final hop e3e_3 is shown.

Confidence separates decisive from suggestive evidence. In clinical diagnosis, the decisive evidence differs from patient to patient. Table 3 shows a MedQA case of Hirschsprung disease. The Xray is the pivot of the workup. It reveals a distal obstruction and prompts the rectal biopsy, but it cannot tell Hirschsprung disease from other causes of obstruction. Only the biopsy, which finds no ganglion cells, confirms the diagnosis. The base logit is nearly certain in every state, whereas SuffiCE stays uncertain with the X-ray in state {1,4}\{1, 4\} and becomes confident only once the biopsy is added in state {1,3,4}\{1, 3, 4\}. With a suitable confidence threshold, SuffiCE would therefore seek more information before answering. Unlike the base logit, SuffiCE thus recognizes which evidence settles the diagnosis and which only signals that more information is needed.

State{1}\{1\}{1,2}\{1,2\}{1,4}\{1,4\}{1,3,4}\{1,3,4\}{1,2,3,4}\{1,2,3,4\}
Success0.200.400.401.001.00
Base0.960.990.991.001.00
SuffiCE0.500.470.510.970.97

Table 3. A MedQA case of Hirschsprung disease (Qwen3.5-9B). Each state is the set of evidence shown. Success is the fraction of ten sampled answers that are correct. Base and SuffiCE give the sufficiency confidence.

A 2-day-old boy has bilious vomiting, has not passed meconium since birth, and has a distended abdomen. What is the diagnosis? Answer: Hirschsprung disease

1 HISTORYHome birth without complications, bilious vomiting, and no meconium since birth.
2 PEFirm, distended abdomen with decreased bowel sounds and palpable bowel loops.
3 BIOPSYRectal biopsy: no ganglion cells in the submucosa or myenteric plexus.
4 IMAGINGAbdominal X-ray: dilated bowel loops and no rectal gas, suggesting a distal obstruction.

Cross-Policy Confidence Estimation

To test whether the estimator must be the answering model, we fix the answering policy to Kimi K2.6 and train Qwen estimators of three sizes on its success rates (Figure 8). A large enough estimator predicts another model’s success as well as that model does itself. The 27B estimator matches or exceeds Kimi K2.6’s own estimate on every dataset in Brier score, and in ρΔ\rho_\Delta on MedQA and MuSiQue. Trained only on Kimi K2.6’s success rates and never shown its answers, the estimator thus learns to anticipate from the evidence state alone how well another model can answer. Smaller estimators are less accurate, and the 4B estimator falls well short of Kimi K2.6’s own estimate.

Bar charts comparing Qwen estimators with Kimi K2.6 on Brier score and rho delta

Figure 8. Qwen estimators of three sizes are trained to predict the success rates of Kimi K2.6, compared with Kimi K2.6 predicting its own (self).

Conclusion

To our knowledge, this is the first work to train a model’s confidence before answering to be calibrated across varying evidence states. The decision to answer or gather more evidence is made before any answer exists, so a model is only as reliable as this judgment. We formalized it as evidence-state calibration, requiring confidence to be calibrated and to change as much as the chance of success whenever the evidence changes, which calibration alone does not guarantee. SuffiCE outperforms every baseline, and its confidence rises with the decisive evidence, not with every piece added. Such confidence is a step toward language models that can judge, in high-stakes settings, when the evidence at hand is enough to answer.

Limitations

SuffiCE shows strong calibration and evidence sensitivity across four answering models and three datasets, but it has three limitations. First, we pre-construct evidence states rather than gather evidence in live multi-turn interactions, a controlled setting that lets us attribute differences in success and confidence to the evidence itself. Second, to limit cost, we estimate each target from ten sampled answers per state, and more samples would give more precise targets. Third, SuffiCE has not been tested in real clinical scenarios, and its confidence should not replace clinical judgment.

References

Experimental Details

Datasets

The three datasets range widely in difficulty (Table 4). For Qwen3.5-9B, the mean success rate on the test set is 0.81 on MIMIC, 0.50 on MedQA, and 0.25 on MuSiQue, so we test the confidence both where most states are answered correctly and where most are not.

MIMICMedQAMuSiQue
Answer4-way choicefree textshort answer
Problems140 / 60146 / 63200 / 80
States840 / 360728 / 3021,500 / 600
Mean success rate p^\hat{p}
Qwen3.5-4B0.82 / 0.810.45 / 0.440.27 / 0.25
Qwen3.5-9B0.84 / 0.810.50 / 0.500.29 / 0.25
Qwen3.8-27B0.87 / 0.830.64 / 0.620.34 / 0.31
Kimi K2.60.87 / 0.820.76 / 0.710.44 / 0.43

Table 4. Dataset statistics, given as train / test. The mean success rate p^\hat{p} of each answering model is taken over all evidence states.

MIMIC. We use emergency admissions from MIMIC-IV-Ext Clinical Decision Making (Hager et al., [9]). We sample 50 admissions for each of four abdominal diseases, appendicitis, cholecystitis, diverticulitis, and pancreatitis, and the task is to choose one of the four. We convert each admission to the OSCE layout of AgentClinic (Schmidgall et al., 2026). The history, laboratory results, and radiology reports are copied verbatim, and an LLM (GPT-5.1) writes only a one-sentence objective naming the presenting complaints and splits the free-text physical examination into sections without changing its values. Microbiology results are never shown, since cultures return after the diagnostic decision. We split admissions into training and test sets stratified by disease.

MedQA. We use the OSCE-style cases of the extended AgentClinic-MedQA set (Schmidgall et al., 2026), which are derived from USMLE questions (Jin et al., [13]), and the task is to name the diagnosis in free text. We exclude 5 of its 214 cases, four with no test results and one with a nonstandard format, which leaves 209. States that show only imaging or only the other tests exist for the 97 cases that have both kinds of test. We split cases by diagnosis, which keeps near-duplicate vignettes on the same side and makes every test diagnosis unseen during training.

MuSiQue. We use 3-hop questions from MuSiQue-Ans (Trivedi et al., 2022), and the task is to give a short answer. For each question, we sample three distractors from its non-supporting paragraphs. Its three supporting paragraphs and three distractors are shuffled once with a seed derived from the question, and every state shows its paragraphs in that order. Training questions come from the MuSiQue training set and test questions from its development set, since its test set has no released answers, with no single-hop question shared between them. MuSiQue’s own decompositions distinguish two question types (Table 5). In a chain question each hop needs the answer of the previous one, whereas in a branching question the first two hops are independent and only the final hop combines them. A branching question therefore has a state with e2e_2 alone, while in a chain e2e_2 is only useful after e1e_1. The full inputs of the branching example appear in Table 16.

Chain (3hop1) e1→e2→e3e_1 \rightarrow e_2 \rightarrow e_3
Who was appointed arbitrator by the first convert to the faith that is the most common religion in Southeast Asia?
Answer: Abu Musa Ashaarı
HopSub-questionAnswer
e1e_1What is the most common religion in Southeast Asia?Islam
e2e_2Who was the first convert to the faith called #1?Ali
e3e_3Who was appointed arbitrator by #2?Abu Musa Ashaarı
Branching (3hop2) e1+e2→e3e_1 + e_2 \rightarrow e_3
When did the party Alaska generally supports take control of the determiner of rules of the US House and US Senate?
Answer: January 2015
HopSub-questionAnswer
e1e_1Which political party does Alaska generally support?Republicans
e2e_2Who determines the rules of the US House and US Senate?The Senate
e3e_3When did #1 take control of #2?January 2015

Table 5. The two MuSiQue question types, with sub-questions from MuSiQue’s decompositions (capitalization normalized). #k stands for the answer of hop kk, supported by paragraph eke_k. In the chain, each hop depends on the previous one, so each stage has its own color. In the branching question, e1e_1 and e2e_2 are independent and share a color, and e3e_3 combines their answers.

StateParagraphs shown, in order
Question only—
Distractors onlyd1d_1 d2d_2 d3d_3
{e1}\{e_1\}d1d_1 e1e_1 d3d_3
{e2}\{e_2\}e2e_2 d1d_1 d3d_3
{e1,e2}\{e_1,e_2\}e2e_2 d1d_1 e1e_1
{e1,e3}\{e_1,e_3\}d1d_1 e1e_1 e3e_3
{e2,e3}\{e_2,e_3\}e2e_2 d1d_1 e3e_3
{e1,e2,e3}\{e_1,e_2,e_3\}e2e_2 e1e_1 e3e_3

Table 16. Full input and all eight evidence states of the 3hop2 MuSiQue question of Table 13. Supporting paragraphs e1e_1–e3e_3 and distractors d1d_1–d3d_3 are shown verbatim. The six paragraphs of each question are shuffled once into a fixed order, here e2e_2, d1d_1, d2d_2, e1e_1, e3e_3, d3d_3, and every state shows its paragraphs in that order, so a paragraph’s position does not reveal the state. Every state except the question-only one shows three paragraphs, padded with distractors.

Answering Policies

The main experiments study self-calibration. Each model in Table 1 serves as both the answering policy and the estimator, so it estimates its own chance of success, and every method for that model is compared against the exact same targets.

Targets for self-calibration. For each model, the frozen model with thinking disabled samples K=10K = 10 answers per evidence state, and the fraction graded correct is the success rate of that state. These success rates are the only targets used for that model. The base logit reads P(A∣x)P(A \mid x) from this model before training, and verbalized confidence prompts this model for the probability that it could answer correctly. Platt scaling fits its two parameters to this model’s success rates on the training set, and SuffiCE trains a LoRA adapter on a copy of this model to predict the same success rates. All four are evaluated against the success rates of this model on the test set. Methods are therefore compared only for the same model, since each model has its own targets. The cross-policy experiment of § 5.4 is the only exception, where Qwen estimators are trained on and evaluated against the success rates of Kimi K2.6.

Difficulty. Success rates differ across answering models, little on MIMIC but widely on MedQA and MuSiQue (Table 4), so the same evidence state has a different target for each answering policy.

Precision of the targets. Each target is estimated from K=10K = 10 answers, so even a perfect estimator that outputs the true success rate p∗p^* has a nonzero Brier score against it. This noise floor is the expected binomial sampling variance p∗(1−p∗)/Kp^*(1-p^*)/K, which we estimate without bias by p^(1−p^)/(K−1)\hat{p}(1-\hat{p})/(K-1) averaged over the test set. On every dataset, the Brier score of SuffiCE is over 13 times this floor (Table 6). Sampling noise therefore accounts for only a small part of any method’s error, and ten answers per state are enough to compare estimators.

DatasetNoise floorSuffiCE Brier
MIMIC0.0030.092
MedQA0.0080.111
MuSiQue0.0070.097

Table 6. Noise floor, the expected Brier score of a perfect estimator against ten-answer success rates, compared with the Brier score of SuffiCE (Table 1). Both are averaged over the four answering models, on the test set.

Training Details

The settings shared by all models and datasets are listed in Table 7. All models are run with thinking disabled. Answers are sampled at temperature 1.0 without top-pp or top-kk truncation. The estimator is a rank-32 LoRA adapter ([10]) on a copy of the estimator’s base model, which is the answering model itself except in the cross-policy experiment (§ 5.4), initialized to have no effect, so before training it gives the base logit. All training and sampling run on Tinker.

SettingValue
Success-rate estimation
Answers per state KK10
Temperature1.0
Top-pp / top-kknone
LLM judge (if used)GPT-5.6 Luna
Estimator training
LoRA rank32
Losssoft-label BCE
OptimizerAdam ([17])
Learning rate1×10−41 \times10^{-4}
Adam β1,β2,ϵ\beta_1,\beta_2,\epsilon0.9, 0.95, 10−810^{-8}
Learning-rate schedulelinear decay to 0
Batch size8
Seeds3 (batch order, adapter init.)
Verbalized baseline
Runs3

Table 7. Settings shared by all models and datasets.

Grading. Each sampled answer is graded as correct or incorrect against the reference. On MIMIC, an answer is correct if, after lowercasing and removing everything but letters, it is exactly the name of the correct disease among the four options, and any other answer is incorrect. On MedQA, GPT-5.6 Luna judges whether a physician would accept the answer as the same diagnosis as the reference. On MuSiQue, an answer is correct if it matches the reference or one of its aliases after SQuAD normalization (lowercasing and removing punctuation and articles), and otherwise GPT-5.6 Luna judges it given the question and the reference. Both judges decode greedily, and their prompts are in Appendix § 9.§ 4.

Prompts

We list every prompt verbatim, grouped by role. Placeholders are in braces. {question} is the MuSiQue question or the clinical case’s objective, {context} and {case} are the evidence shown in the current state, and {task line} is the dataset-specific line of Table 8. All prompts are given as a single user message with thinking disabled, except the judges, which also receive a system message.

Dataset{task line}
MIMICWhich of the following is the most likely diagnosis: appendicitis, cholecystitis, diverticulitis, or pancreatitis?
MedQAWhat is the most likely diagnosis?
MuSiQue(none, the question is the whole task)

Table 8. The dataset-specific task line of the sufficiency and verbalized prompts.

Sufficiency Confidence

The sufficiency confidence estimator and the base logit read the probabilities of A and S as the first token of the reply (§ 3.4).

Sufficiency prompt (all datasets)

Decide whether the currently available information is
sufficient for you to answer the task correctly. Do not
answer the task itself.

A = the information is sufficient; answer now.
S = the information is insufficient; seek more information.

Task:
{question}
{task line}

Available information:
{context}

Is the available information sufficient? Respond with
exactly one letter: A or S.

Answering

The frozen answering policy samples ten answers per state with these prompts, from which the empirical success rate is computed (§ 3.3).

Answering prompt: MIMIC

{case}

This is all the information available about the patient.
Based on this information, which of the following is the
most likely diagnosis?
appendicitis
cholecystitis
diverticulitis
pancreatitis
Respond with only one of these four words, nothing else.

Answering prompt: MedQA

{case}

This is all the information available about the patient.
Based on this information, what is the most likely
diagnosis?
Respond with only the diagnosis, nothing else.

Answering prompt: MuSiQue

Answer the question using the provided information and your
own knowledge.
If you are unsure, give your best guess.
Give only the final answer, with no explanation, in at
most 100 characters.

{case}
Answer:

Verbalized Baseline

The first number in the reply is taken as the confidence, with “85%” read as 0.85.

Verbalized confidence prompt (all datasets)

Based on the currently available information, estimate the
probability that you could answer the task correctly
without seeking any additional information.
Do not answer the task itself.
Respond with a single number between 0 and 1, where 0
means the current information is certainly insufficient for
a correct answer and 1 means it is certainly sufficient.
Task:
{question}
{task line}
Available information:
{context}
Probability:

LLM Judges

MedQA answers are graded by the judge below. MuSiQue answers that are not a normalized exact match with the reference or one of its aliases are graded by the MuSiQue judge. MIMIC answers are graded by exact match. All methods for a model are evaluated against the same grades, so differences between methods do not come from differences in grading.

MedQA judge: system

You are an expert physician grading a diagnostic answer
against a reference diagnosis.
First explain your reasoning briefly, then give a grade of
1 (correct) or 0 (incorrect).

MedQA judge: user

Correct diagnosis: {correct_diagnosis}
Doctor's diagnosis: {predicted_diagnosis}

Grade 1 if a physician would accept the doctor's diagnosis
as the same diagnosis as the correct one. Accept synonyms,
abbreviations, alternate spellings, historical or modern
terminology, and diagnoses at an equivalent level of
specificity. A more specific form of the correct diagnosis
is also acceptable.

Grade 0 if the doctor's diagnosis:
- is a different condition, even a closely related one,
- is only a broader disease category that contains the
correct diagnosis but does not identify the more specific
diagnosis the correct one requires,
- lists several possibilities without committing to the
correct one, or
- is missing.

Respond in exactly this format:
<reasoning>At most three sentences comparing the two
diagnoses.</reasoning>
<grade>1 or 0</grade>

MuSiQue judge: system

You are grading short answers to multi-hop questions
against a reference answer.
First explain your reasoning briefly, then give a grade of
1 (correct) or 0 (incorrect).

MuSiQue judge: user

Question: {question}
Reference answer: {answer}
Model answer: {predicted}

Grade 1 only if the response clearly commits to an answer
that is semantically correct according to the reference.
Grade 0 if it gives multiple alternative answers without
clearly committing to the correct one, or if it gives a
wrong answer to the question alongside the correct one.
Ignore extra details that do not answer the question, even
if they are inaccurate. Differences in wording or
formatting are acceptable when they preserve the same
underlying answer. A broader place, category, or entity, or
a related-but-different event, is not equivalent to the
reference. Dates are the only exception: differences in
date precision are acceptable when they preserve the
correct time period, so an exact date, month/year, year, or
decade may be treated as equivalent when consistent with
the reference. Extremely broad periods such as centuries
are not sufficient substitutes for a specific year or date.

Respond in exactly this format:
<reasoning>At most two sentences.</reasoning>
<grade>1 or 0</grade>

Additional Results

Calibration Curves for All Models

We extend the calibration curves of Figure 4 to all four answering models (Figure 9).

Figure 9. Calibration curves of Qwen3.5-4B, Qwen3.5-9B, Qwen3.8-27B, and Kimi K2.6 on the test set, (a) before and (b) after SuffiCE.

The base confidence errs in no consistent direction. Under the same prompt, Qwen3.5-9B is overconfident on every dataset, Qwen3.8-27B and Kimi K2.6 are underconfident on every dataset, and Qwen3.5-4B is overconfident on MedQA but underconfident on MIMIC and MuSiQue. No single correction therefore repairs the base confidence across models and datasets. Many base curves are also not monotone, so a state with higher confidence is not reliably more answerable. Platt scaling can shift such a curve toward the diagonal but cannot straighten it, since it keeps the order of states.

SuffiCE moves every curve toward the diagonal. After training, every Qwen curve lies closer to the diagonal. The confidence also adapts to the range of each dataset, spanning low values on MuSiQue, where most states cannot be answered, and concentrating near the top on MIMIC, where most are answered correctly. The same prompt and objective produce both, so the estimator learns the scale of each dataset from its targets rather than imposing a fixed one.

SuffiCE carries over to another model family. Before training, Kimi K2.6 is strongly underconfident on every dataset. SuffiCE moves its curve toward the diagonal on each.

Selective Answering for All Models

Across all four answering models and three datasets, SuffiCE has the highest accuracy at most percentages of questions answered, with the smallest margin for Qwen3.5-4B on the clinical datasets (Figure 10, extending Table 2).

Figure 10. Selective answering on the test set of every dataset (rows) and model (columns). Each curve plots the accuracy of the answered questions against the percentage answered, answering the questions with the highest confidence first. Higher is better. Platt scaling preserves the ranking of the base logit, so their curves coincide.

Step-Matched Ablation

The ablation of § 5.2 trains the full-only estimator for the same number of epochs as SuffiCE. Since it has one state per problem instead of four to eight, it then takes far fewer gradient steps. Here we instead train it for the same number of steps as SuffiCE, with all other hyperparameters unchanged (Table 9). SuffiCE remains better on every metric and dataset, so the gap comes from training across evidence states, not from the number of updates.

MethodBrier ↓\downarrowAUROC ↑\uparrowρΔ↑\rho_{\Delta} \uparrow
MIMIC
Full only0.130 ±0.005\pm0.0050.745 ±0.014\pm0.0140.251 ±0.024\pm0.024
SuffiCE0.098 ±0.010\pm0.0100.819 ±0.028\pm0.0280.393 ±0.034\pm0.034
MedQA
Full only0.131 ±0.004\pm0.0040.782 ±0.007\pm0.0070.554 ±0.008\pm0.008
SuffiCE0.107 ±0.005\pm0.0050.815 ±0.011\pm0.0110.630 ±0.012\pm0.012
MuSiQue
Full only0.101 ±0.006\pm0.0060.794 ±0.015\pm0.0150.532 ±0.049\pm0.049
SuffiCE0.078 ±0.002\pm0.0020.828 ±0.006\pm0.0060.666 ±0.015\pm0.015

Table 9. Step-matched ablation for Qwen3.5-9B. The full-only estimator is trained for the same number of steps as SuffiCE.

Additional Evidence-State Analysis

Ordering of pairs in ρΔ\rho_{\Delta}. Each pair of states enters once, ordered from less to more evidence. The correlation subtracts the average change, so confidence that rises equally whenever evidence is added gets no credit, and ρΔ\rho_{\Delta} rewards confidence that rises more for the evidence that raises success more.

We report two measures of evidence sensitivity beyond ρΔ\rho_{\Delta} for every answering model (Table 10).

MethodMIMIC ΔMAE↓\Delta\mathrm{MAE}\downarrowMIMIC ρwithin↑\rho_{\mathrm{within}}\uparrowMedQA ΔMAE↓\Delta\mathrm{MAE}\downarrowMedQA ρwithin↑\rho_{\mathrm{within}}\uparrowMuSiQue ΔMAE↓\Delta\mathrm{MAE}\downarrowMuSiQue ρwithin↑\rho_{\mathrm{within}}\uparrow
Kimi K2.6
Base logit0.333 ±\pm0.0000.217 ±\pm0.0000.282 ±\pm0.0000.290 ±\pm0.0000.355 ±\pm0.0000.164 ±\pm0.000
Verbalized0.182 ±\pm0.0010.222 ±\pm0.0070.193 ±\pm0.0020.247 ±\pm0.0040.367 ±\pm0.0030.123 ±\pm0.029
Platt0.190 ±\pm0.0000.210 ±\pm0.0000.202 ±\pm0.0000.290 ±\pm0.0000.351 ±\pm0.0000.173 ±\pm0.000
SuffiCE0.188 ±\pm0.0080.496 ±\pm0.0220.188 ±\pm0.0130.430 ±\pm0.0590.292 ±\pm0.0030.216 ±\pm0.021
Qwen3.8-27B
Base logit0.260 ±\pm0.0000.299 ±\pm0.0000.281 ±\pm0.0000.440 ±\pm0.0000.353 ±\pm0.0000.026 ±\pm0.000
Verbalized0.193 ±\pm0.0000.248 ±\pm0.0030.238 ±\pm0.0010.480 ±\pm0.0030.400 ±\pm0.0030.138 ±\pm0.007
Platt0.198 ±\pm0.0000.253 ±\pm0.0000.219 ±\pm0.0000.450 ±\pm0.0000.337 ±\pm0.0000.110 ±\pm0.000
SuffiCE0.180 ±\pm0.0060.497 ±\pm0.0070.218 ±\pm0.0090.634 ±\pm0.0230.240 ±\pm0.0100.257 ±\pm0.025
Qwen3.5-9B
Base logit0.217 ±\pm0.0000.137 ±\pm0.0000.259 ±\pm0.0000.398 ±\pm0.0000.383 ±\pm0.0000.190 ±\pm0.000
Verbalized0.251 ±\pm0.0030.235 ±\pm0.0160.256 ±\pm0.0010.381 ±\pm0.0040.343 ±\pm0.0020.193 ±\pm0.004
Platt0.202 ±\pm0.0000.204 ±\pm0.0000.234 ±\pm0.0000.433 ±\pm0.0000.307 ±\pm0.0000.195 ±\pm0.000
SuffiCE0.182 ±\pm0.0040.454 ±\pm0.1070.205 ±\pm0.0050.592 ±\pm0.0240.243 ±\pm0.0060.229 ±\pm0.015
Qwen3.5-4B
Base logit0.269 ±\pm0.0000.386 ±\pm0.0000.264 ±\pm0.0000.514 ±\pm0.0000.309 ±\pm0.0000.168 ±\pm0.000
Verbalized0.326 ±\pm0.0030.383 ±\pm0.0040.304 ±\pm0.0070.409 ±\pm0.0190.336 ±\pm0.0010.123 ±\pm0.003
Platt0.201 ±\pm0.0000.368 ±\pm0.0000.236 ±\pm0.0000.512 ±\pm0.0000.304 ±\pm0.0000.207 ±\pm0.000
SuffiCE0.178 ±\pm0.0140.487 ±\pm0.0340.222 ±\pm0.0100.530 ±\pm0.0360.258 ±\pm0.0040.230 ±\pm0.011

Table 10. Evidence-state analysis for every answering model. Δ\DeltaMAE compares changes in confidence with changes in empirical success between states of the same problem, and ρwithin\rho_{\mathrm{within}} correlates confidence with success among problems that share a state type. The best result for each model is in bold.

  • Δ\DeltaMAE (lower is better) tests whether confidence changes by the right amount as the evidence changes. For every pair of distinct states s,ts,t of the same problem ii, the same pairs as ρΔ\rho_{\Delta}, it takes the absolute gap between the change in confidence and the change in empirical success, ∣(ci,t−ci,s)−(p^i,t−p^i,s)∣\lvert(c_{i,t}-c_{i,s})-(\hat{p}_{i,t}-\hat{p}_{i,s})\rvert, and averages over all pairs in the test set. It is zero when confidence rises and falls by exactly as much as success. Where ρΔ\rho_{\Delta} tests the direction of each change, Δ\DeltaMAE tests its size.

  • ρwithin\rho_{\mathrm{within}} (higher is better) tests whether confidence separates more and less answerable problems that have the same kind of evidence. It groups the test set by state type, such as all HISTORY states (on MuSiQue, by question type and state), computes the Pearson correlation between confidence and empirical success within each group, and averages over groups. Every state in a group holds the same kind of evidence, so recognizing the state type alone does not raise it.

SuffiCE has the lowest Δ\DeltaMAE in 11 of the 12 combinations of model and dataset, so its confidence changes by about as much as success does, not only in the same direction. The one exception is Kimi K2.6 on MIMIC, where verbalized confidence is slightly lower. For Kimi K2.6, success on MIMIC changes by at most 0.1 for about three quarters of the pairs of states, so a confidence that barely moves has a low Δ\DeltaMAE, and verbalized confidence moves about half as much as SuffiCE. On ρΔ\rho_{\Delta}, which gives no credit for staying still, SuffiCE is far better (Table 1). SuffiCE also has the highest ρwithin\rho_{\mathrm{within}} for every model and dataset, with the largest margin on MIMIC, where whether the evidence suffices depends on the particular patient, so it does not merely learn an average confidence for each state type.

Cross-Domain Transfer

To test whether what SuffiCE learns is specific to one dataset, we train it on one clinical dataset and evaluate it on the other without further training (Table 11). MIMIC asks for one of four diseases and MedQA for a free-text diagnosis, and success is far more common on MIMIC.

MetricBase logitCross-domainIn-domain
Train MIMIC →\to test MedQA
Brier ↓\downarrow0.268 ±\pm0.0000.189 ±\pm0.0270.107 ±\pm0.005
AUROC ↑\uparrow0.746 ±\pm0.0000.779 ±\pm0.0090.815 ±\pm0.011
ρΔ↑\rho_{\Delta}\uparrow0.367 ±\pm0.0010.401 ±\pm0.0270.630 ±\pm0.012
ρwithin↑\rho_{\mathrm{within}}\uparrow0.398 ±\pm0.0000.491 ±\pm0.0290.592 ±\pm0.024
Train MedQA →\to test MIMIC
Brier ↓\downarrow0.124 ±\pm0.0000.167 ±\pm0.0040.098 ±\pm0.010
AUROC ↑\uparrow0.689 ±\pm0.0000.746 ±\pm0.0160.819 ±\pm0.028
ρΔ↑\rho_{\Delta}\uparrow0.247 ±\pm0.0000.316 ±\pm0.0050.393 ±\pm0.034
ρwithin↑\rho_{\mathrm{within}}\uparrow0.137 ±\pm0.0000.321 ±\pm0.0380.454 ±\pm0.107

Table 11. Cross-domain transfer between the clinical datasets (Qwen3.5-9B). Cross-domain and in-domain are SuffiCE trained on the other dataset and on the test dataset itself. The best result in each row is in bold and the second best in italics.

Which evidence matters transfers. In both directions, the transferred estimator ranks states better than the base logit in AUROC and ρΔ\rho_{\Delta}. It also better separates more and less answerable cases that share a state type, recovering about half of the in-domain gain in ρwithin\rho_{\mathrm{within}} (§C.4) over the base logit in both directions. What SuffiCE learns about which evidence settles a case thus carries over across datasets.

The overall success level does not transfer. The transferred confidence stays close to the success rate of the training dataset. Trained on MIMIC, where most states succeed, it remains too confident on MedQA, though less so than the base logit. Trained on MedQA, it is too cautious on MIMIC and is worse calibrated than the base logit. This error shifts the level of confidence, not the order of states, so ranking transfers while calibration does not.

Proofs

Proposition 1 says that the error of a confidence splits into two parts: how far off it is on average for each problem, and how far its changes between states of the same problem are from the changes in success. Calibration constrains only the first part. Corollary 1 shows that the training loss bounds both parts. We first define the notation shared by both results, then restate and prove Proposition 1 (§D.1) and Corollary 1 (§D.2).

Setup. Let there be nn problems, indexed by i=1,…,ni=1,\ldots,n. Problem ii consists of a question qiq_i, a pool of evidence EiE_i, and a gold answer yi∗y_i^* (§ 3.1). It has mi≥1m_i \ge1 evidence states, indexed by s=1,…,mis=1,\ldots,m_i. State ss pairs the question with a subset Ii,s⊆EiI_{i,s} \subseteq E_i of the evidence, xi,s=(qi,Ii,s)x_{i,s}=(q_i,I_{i,s}). Let N=∑i=1nmiN=\sum_{i=1}^{n}m_i be the total number of states. For a state xx, let p∗(x)∈[0,1]p^*(x)\in[0,1] be the success probability of the answering policy π0\pi_0 (Eq. (1)) and cθ(x)∈[0,1]c_\theta(x)\in[0,1] the confidence of the estimator.

Within problem ii, we abbreviate cs=cθ(xi,s)c_s=c_\theta(x_{i,s}) and ps=p∗(xi,s)p_s=p^*(x_{i,s}), and define the error at state ss as ϵs=cs−ps\epsilon_s=c_s-p_s. The problem averages of the confidence, the success probability, and the error are

cˉθ,i=1mi∑s=1mics,pˉi∗=1mi∑s=1mips,\bar{c}_{\theta,i}=\frac{1}{m_i}\sum_{s=1}^{m_i}c_s,\qquad\bar{p}_i^*=\frac{1}{m_i}\sum_{s=1}^{m_i}p_s,
ϵˉ=cˉθ,i−pˉi∗.\bar{\epsilon}=\bar{c}_{\theta,i}-\bar{p}_i^*.

For any s,t∈{1,…,mi}s,t\in\{1,\ldots,m_i\}, the change gap

δst=(ct−cs)−(pt−ps)\delta_{st}=(c_t-c_s)-(p_t-p_s)

is the difference between the change in confidence and the change in success probability from state ss to state tt. Finally, E\mathbb{E} denotes the uniform average over all NN states, and Ei\mathbb{E}_i the average of a per-problem quantity over problems, with problem ii weighted by mi/Nm_i/N, its share of all states. With this weight, averaging any function ff of a state over all states equals first averaging it within each problem and then applying Ei\mathbb{E}_i,

E[f]=1N∑i=1n∑s=1mif(xi,s)=∑i=1nmiN⋅1mi∑s=1mif(xi,s)=∑i=1nmiNfˉi=Ei[fˉi],(8)\begin{aligned} \mathbb{E}[f]=\frac{1}{N}\sum_{i=1}^{n}\sum_{s=1}^{m_i}f(x_{i,s}) =\sum_{i=1}^{n}\frac{m_i}{N}\cdot\frac{1}{m_i}\sum_{s=1}^{m_i}f(x_{i,s}) \\ &=\sum_{i=1}^{n}\frac{m_i}{N}\bar{f}_i=\mathbb{E}_i[\bar{f}_i], \tag*{(8)} \end{aligned}

where fˉi=1mi∑sf(xi,s)\bar{f}_i=\frac{1}{m_i}\sum_s f(x_{i,s}) is the average of ff over the states of problem ii. A plain average over problems, 1n∑ifˉi\frac{1}{n}\sum_i\bar{f}_i, agrees with E[f]\mathbb{E}[f] only when all problems have the same number of states.

Proof of Proposition 1

Proposition 1 (restated). For any confidence cθc_\theta,

E[(cθ−p∗)2]=Ei[(cˉθ,i−pˉi∗)2]+Ei[12mi2∑s=1mi∑t=1miδst2].\mathbb{E}\left[(c_\theta-p^*)^2\right]=\mathbb{E}_i\left[(\bar{c}_{\theta,i}-\bar{p}_i^*)^2\right]+\mathbb{E}_i\left[\frac{1}{2m_i^2}\sum_{s=1}^{m_i}\sum_{t=1}^{m_i}\delta_{st}^2\right].

The left side is the mean squared error over all states. The first term on the right is the error across problems, in each problem’s average success. The second is the error within each problem, half the average squared change gap over all ordered pairs of its states.

Proof. We prove the identity for a single problem ii and then average over problems. All sums over ss and tt run from 11 to mim_i.

Step 1: split the error into its average and its spread. Writing ϵs=ϵˉ+(ϵs−ϵˉ)\epsilon_s=\bar{\epsilon}+(\epsilon_s-\bar{\epsilon}) and expanding the square,

1mi∑sϵs2=ϵˉ2+2ϵˉmi∑s(ϵs−ϵˉ)+1mi∑s(ϵs−ϵˉ)2.\frac{1}{m_i}\sum_s\epsilon_s^2=\bar{\epsilon}^2+\frac{2\bar{\epsilon}}{m_i}\sum_s(\epsilon_s-\bar{\epsilon}) +\frac{1}{m_i}\sum_s(\epsilon_s-\bar{\epsilon})^2.

The middle term is zero. Since ϵˉ=cˉθ,i−pˉi∗=1mi∑sϵs\bar{\epsilon}=\bar{c}_{\theta,i}-\bar{p}_i^*=\frac{1}{m_i}\sum_s\epsilon_s is the average of the errors, the errors above the average exactly cancel those below it,

∑s(ϵs−ϵˉ)=∑sϵs−miϵˉ=miϵˉ−miϵˉ=0.\sum_s(\epsilon_s-\bar{\epsilon})=\sum_s\epsilon_s-m_i\bar{\epsilon}=m_i\bar{\epsilon}-m_i\bar{\epsilon}=0.

Step 2: rewrite the spread with pairs of states. The spread is the last term of Step 1, 1mi∑s(ϵs−ϵˉ)2\frac{1}{m_i}\sum_s(\epsilon_s-\bar{\epsilon})^2, the average squared deviation of the errors from their mean, that is, their variance over the states of problem ii. It is zero exactly when every state of the problem has the same error. We now show that it equals half the average squared difference between the errors of two states. Let ds=ϵs−ϵˉd_s=\epsilon_s-\bar{\epsilon}, so ∑sds=0\sum_s d_s=0 by Step 1. Expanding (dt−ds)2=dt2+ds2−2dsdt(d_t-d_s)^2=d_t^2+d_s^2-2d_sd_t and summing over all ss and tt,

∑s,t(ϵt−ϵs)2=∑s,t(dt−ds)2=2mi∑sds2−2(∑sds)2=2mi∑sds2=2mi∑s(ϵs−ϵˉ)2.\sum_{s,t}(\epsilon_t-\epsilon_s)^2=\sum_{s,t}(d_t-d_s)^2 =2m_i\sum_s d_s^2-2\left(\sum_s d_s\right)^2 =2m_i\sum_s d_s^2 =2m_i\sum_s(\epsilon_s-\bar{\epsilon})^2.

Dividing by 2mi22m_i^2 gives

1mi∑s(ϵs−ϵˉ)2=12mi2∑s,t(ϵt−ϵs)2.\frac{1}{m_i}\sum_s(\epsilon_s-\bar{\epsilon})^2 =\frac{1}{2m_i^2}\sum_{s,t}(\epsilon_t-\epsilon_s)^2.

Step 3: identify the change gap. Rearranging, ϵt−ϵs=(ct−pt)−(cs−ps)=(ct−cs)−(pt−ps)=δst\epsilon_t-\epsilon_s=(c_t-p_t)-(c_s-p_s)=(c_t-c_s)-(p_t-p_s)=\delta_{st}. Combining the three steps, for every problem ii,

1mi∑sϵs2=Step 1ϵˉ2+1mi∑s(ϵs−ϵˉ)2=Step 2ϵˉ2+12mi2∑s,t(ϵt−ϵs)2=Step 3(cˉθ,i−pˉi∗)2+12mi2∑s,tδst2,\frac{1}{m_i}\sum_s\epsilon_s^2 \overset{\text{Step 1}}{=}\bar{\epsilon}^2+\frac{1}{m_i}\sum_s(\epsilon_s-\bar{\epsilon})^2 \overset{\text{Step 2}}{=}\bar{\epsilon}^2+\frac{1}{2m_i^2}\sum_{s,t}(\epsilon_t-\epsilon_s)^2 \overset{\text{Step 3}}{=}(\bar{c}_{\theta,i}-\bar{p}_i^*)^2+\frac{1}{2m_i^2}\sum_{s,t}\delta_{st}^2,

where the last line substitutes ϵˉ=cˉθ,i−pˉi∗\bar{\epsilon}=\bar{c}_{\theta,i}-\bar{p}_i^* and ϵt−ϵs=δst\epsilon_t-\epsilon_s=\delta_{st}. The left side is the mean squared error of problem ii. On the right, the first term is the squared error of the problem’s average confidence, and the second is half the average squared change gap over its ordered pairs of states.

Step 4: average over problems. Steps 1–3 give the identity for one problem. The proposition averages over all states, so we group the states by problem and weight each problem by its share of states, using Eq. (8) with f=(cθ−p∗)2f=(c_\theta-p^*)^2. Substituting the per-problem identity for each problem’s mean squared error and splitting the sum,

E[(cθ−p∗)2]=∑imiN[1mi∑sϵs2]=∑imiN[(cˉθ,i−pˉi∗)2+12mi2∑s,tδst2]=∑imiN(cˉθ,i−pˉi∗)2+∑imiN⋅12mi2∑s,tδst2=Ei[(cˉθ,i−pˉi∗)2]+Ei[12mi2∑s,tδst2],\mathbb{E}\left[(c_\theta-p^*)^2\right] =\sum_i\frac{m_i}{N}\left[\frac{1}{m_i}\sum_s\epsilon_s^2\right] =\sum_i\frac{m_i}{N}\left[(\bar{c}_{\theta,i}-\bar{p}_i^*)^2+\frac{1}{2m_i^2}\sum_{s,t}\delta_{st}^2\right] =\sum_i\frac{m_i}{N}(\bar{c}_{\theta,i}-\bar{p}_i^*)^2+\sum_i\frac{m_i}{N}\cdot\frac{1}{2m_i^2}\sum_{s,t}\delta_{st}^2 =\mathbb{E}_i\left[(\bar{c}_{\theta,i}-\bar{p}_i^*)^2\right]+\mathbb{E}_i\left[\frac{1}{2m_i^2}\sum_{s,t}\delta_{st}^2\right],

where the last line uses the definition of EiE_i as the average over problems with weight mi/Nm_i/N. This is Eq. (4). □

Remarks. Calibration alone does not remove the second term. Suppose cθc_\theta gives every state of a problem the same value, for example the problem’s average success pˉi∗\bar{p}_i^*. Then ct−cs=0c_t-c_s=0, so δst=−(pt−ps)\delta_{st}=-(p_t-p_s), and the second term becomes 12mi2∑s,t(pt−ps)2\frac{1}{2m_i^2}\sum_{s,t}(p_t-p_s)^2, which by the identity of Step 2, applied to psp_s in place of ϵs\epsilon_s, equals the variance of p∗p^* over the states of problem ii. This value does not depend on which constant is chosen, so a confidence that ignores the evidence cannot reduce it, even if it is perfectly calibrated.

The Brier score measures the same error. The Brier score (§ 4) compares the confidence with the empirical success rate p^(x)\hat{p}(x) rather than with p∗(x)p^*(x). Since p^(x)\hat{p}(x) averages KK independent grades, each equal to 1 with probability p∗(x)p^*(x), it satisfies E[p^(x)∣x]=p∗(x)\mathbb{E}[\hat{p}(x)\mid x]=p^*(x) and Var⁡(p^(x)∣x)=p∗(x)(1−p∗(x))/K\operatorname{Var}(\hat{p}(x)\mid x)=p^*(x)(1-p^*(x))/K. For a confidence computed from xx alone, the cross term vanishes, and

E[(cθ−p^)2]=E[(cθ−p∗)2]+1KE[p∗(1−p∗)],\mathbb{E}\left[(c_\theta-\hat{p})^2\right] =\mathbb{E}\left[(c_\theta-p^*)^2\right]+\frac{1}{K}\mathbb{E}\left[p^*(1-p^*)\right],

where the expectation on the left also averages over the sampled grades. The last term does not depend on the confidence, so differences in Brier score between methods reflect differences in the error that Proposition 1 decomposes.

Proof of Corollary 1

Corollary 1 (restated). For any confidence cθc_\theta, let E[L]\mathbb{E}[\mathcal{L}] be the loss of Eq. (6) averaged over all evidence states and over the KK sampled answers that determine p^(x)\hat{p}(x), and L∗\mathcal{L}^* the minimum expected loss over all confidences, reached exactly at cθ=p∗c_\theta=p^*. Then

E[(cθ−p∗)2]≤12(E[L]−L∗),\mathbb{E}\left[(c_\theta-p^*)^2\right]\leq\frac{1}{2}\left(\mathbb{E}[\mathcal{L}]-\mathcal{L}^*\right),

so each term of Proposition 1 is at most 12(E[L]−L∗)\frac{1}{2}\left(\mathbb{E}[\mathcal{L}]-\mathcal{L}^*\right).

Proof. Recall the soft-label binary cross-entropy of Eq. (6), at each evidence state xx,

L(x)=−p^(x)log⁡cθ(x)−(1−p^(x))log⁡(1−cθ(x)).\mathcal{L}(x)=-\hat{p}(x)\log c_\theta(x) -(1-\hat{p}(x))\log(1-c_\theta(x)).

Let H(p)=−plog⁡p−(1−p)log⁡(1−p)H(p)=-p\log p-(1-p)\log(1-p) be the binary entropy and KL⁡(p∥c)=plog⁡pc+(1−p)log⁡1−p1−c\operatorname{KL}(p\parallel c)=p\log\frac{p}{c}+(1-p)\log\frac{1-p}{1-c} the divergence between Bernoulli distributions with means pp and cc, with the convention 0log⁡0=00\log0=0. The confidence of SuffiCE is a soft-max and so lies strictly between 0 and 1, which keeps every logarithm finite. The loss is linear in p^(x)\hat{p}(x), so taking the expectation over the sampled grades replaces p^(x)\hat{p}(x) by E[p^(x)∣x]=p∗(x)\mathbb{E}[\hat{p}(x)\mid x]=p^*(x). At each state, writing p∗p^* and cθc_\theta for p∗(x)p^*(x) and cθ(x)c_\theta(x), the expected loss is therefore

E[L(x)∣x]=−p∗log⁡cθ−(1−p∗)log⁡(1−cθ).\mathbb{E}[\mathcal{L}(x)\mid x] = -p^*\log c_\theta-(1-p^*)\log(1-c_\theta).

To separate the part that depends on cθc_\theta from the part that does not, we use the entropy of the target, H(p∗)=−p∗log⁡p∗−(1−p∗)log⁡(1−p∗)H(p^*)=-p^*\log p^*-(1-p^*)\log(1-p^*). By this definition, H(p∗)+p∗log⁡p∗+(1−p∗)log⁡(1−p∗)=0H(p^*)+p^*\log p^*+(1-p^*)\log(1-p^*)=0, so adding these terms to the expected loss leaves it unchanged:

E[L(x)∣x]=H(p∗)+p∗log⁡p∗+(1−p∗)log⁡(1−p∗)−p∗log⁡cθ−(1−p∗)log⁡(1−cθ).\begin{aligned} \mathbb{E}[\mathcal{L}(x)\mid x] &= H(p^*)+p^*\log p^* \\ &\quad+(1-p^*)\log(1-p^*) \\ &\quad-p^*\log c_\theta\\ &\quad-(1-p^*)\log(1-c_\theta). \end{aligned}

Grouping the two terms with the factor p∗p^* and the two with the factor 1−p∗1-p^*,

E[L(x)∣x]=H(p∗)+p∗(log⁡p∗−log⁡cθ)+(1−p∗)(log⁡(1−p∗)−log⁡(1−cθ)),\begin{aligned} \mathbb{E}[\mathcal{L}(x)\mid x] &= H(p^*)+p^*(\log p^*-\log c_\theta) \\ &\quad+(1-p^*)(\log(1-p^*) \\ &\qquad-\log(1-c_\theta)), \end{aligned}

and writing each difference of logarithms as the logarithm of a ratio,

E[L(x)∣x]=H(p∗)+p∗log⁡p∗cθ+(1−p∗)log⁡1−p∗1−cθ=H(p∗)+KL⁡(p∗∥cθ).\begin{aligned} \mathbb{E}[\mathcal{L}(x)\mid x] &= H(p^*)+p^*\log\frac{p^*}{c_\theta} \\ &\quad+(1-p^*)\log\frac{1-p^*}{1-c_\theta} \\ &= H(p^*)+\operatorname{KL}(p^*\mathbin{\|}c_\theta). \end{aligned}

Averaging this identity over all evidence states gives

E[L]=E[H(p∗)]+E[KL⁡(p∗∥cθ)].\mathbb{E}[\mathcal{L}]=\mathbb{E}[H(p^*)]+\mathbb{E}[\operatorname{KL}(p^*\mathbin{\|}c_\theta)].

The first term depends only on the answering policy, not on the confidence, so no choice of cθc_\theta can change it. The second term is an average of divergences, each nonnegative and zero only when cθ(x)=p∗(x)c_\theta(x)=p^*(x). Every confidence therefore has E[L]≥E[H(p∗)]\mathbb{E}[\mathcal{L}]\ge\mathbb{E}[H(p^*)], with equality exactly when cθ(x)=p∗(x)c_\theta(x)=p^*(x) at every state. Hence the minimum expected loss is L∗=E[H(p∗)]\mathcal{L}^*=\mathbb{E}[H(p^*)], reached exactly at cθ=p∗c_\theta=p^*, and

E[L]−L∗=E[KL⁡(p∗∥cθ)].\mathbb{E}[\mathcal{L}]-\mathcal{L}^*=\mathbb{E}[\operatorname{KL}(p^*\mathbin{\|}c_\theta)].

Pinsker’s inequality for Bernoulli distributions states KL⁡(p∥c)≥2(p−c)2\operatorname{KL}(p\mathbin{\|}c)\ge2(p-c)^2. Applying it at every state and averaging,

E[L]−L∗=E[KL⁡(p∗∥cθ)]≥2E[(cθ−p∗)2],\begin{aligned} \mathbb{E}[\mathcal{L}]-\mathcal{L}^* &= \mathbb{E}[\operatorname{KL}(p^*\mathbin{\|}c_\theta)] \\ &\ge2\mathbb{E}[(c_\theta-p^*)^2], \end{aligned}

which is Eq. (7). By Proposition 1, the two terms are nonnegative and sum to E[(cθ−p∗)2]\mathbb{E}[(c_\theta-p^*)^2], so each is at most 12(E[L]−L∗)\frac{1}{2}(\mathbb{E}[\mathcal{L}]-\mathcal{L}^*). □

For SuffiCE, this means that lowering the training loss lowers an upper bound on both terms of Proposition 1, the error in each problem’s average success and the error in how confidence changes between states of the same problem. The bound holds only over the states the loss is averaged over, which is why SuffiCE trains on many evidence states of each problem rather than on full states alone (§ 5.2).

Comparison with Related Work

We compare our setting with the closest prior work on confidence estimation along five properties (Table 12).

MethodPre-answerTrainedControlled evidence subsetsWithin-problem evaluationOwn token probability
P(IK)P(\mathrm{IK}) [15]✓✓–––
Pre-generation probes (Lugoloobi et al., 2026)✓✓–––
Calibration tuning [16]–✓––✓
Rewarding Doubt [3]–✓–––
LA-CDM [2]–✓–––
P(Sufficient)P(\mathrm{Sufficient}) (Zhang et al., 2026)––––✓
SuffiCE✓✓✓✓✓

Table 12. Comparison with the closest prior work along the five properties defined below. ✓ yes, – no.

  • Pre-answer: the confidence is produced before any answer is generated, so it can decide whether to answer or to gather more evidence.

  • Trained: the confidence estimator is trained, rather than read from an untrained model or elicited by prompting.

  • Controlled evidence subsets: the same problem is observed with different combinations of evidence, not only with evidence that accumulates in a fixed order.

  • Within-problem evaluation: the evaluation tests whether changes in confidence follow changes in success across states of the same problem.

  • Own token probability: the confidence is the model’s next-token probability, with no added classifier head or verbalized number.

P(IK)P(\mathrm{IK}) (Kadavath et al., [15]) trains an added value head to predict, from the question alone, whether the model will answer correctly. It adds source documents and hints only at evaluation and does not relate the change in P(IK)P(\mathrm{IK}) to the change in accuracy. Pre-generation probes (Lugoloobi et al., 2026) train linear probes on the model’s activations before generation to predict success, with each problem shown in a single fixed prompt. Calibration tuning (Kapoor et al., [16]) fine-tunes the model to judge whether a proposed answer is correct, read from its token probability. Rewarding Doubt (Bani-Harouni et al., [3]) uses reinforcement learning to train the model to state a confidence from 0 to 10 in an answer it has already generated, with each

question asked without context. LA-CDM (Bani-Harouni et al., [2]) trains a stated confidence in a diagnosis the agent has already proposed, and the evidence is limited to the tests its agent chose to request. P(Sufficient)P(\mathrm{Sufficient}) (Zhang et al., 2026) prompts the model for the probability that the information so far entails its answer. Its clues arrive in a fixed order, and its evaluation measures per-turn calibration and whether confidence rises over turns.

Each prior method shares some of these properties, and ours is the only one that has all five. No prior method trains the model’s own token probability before an answer exists. P(IK)P(\mathrm{IK}) (Kadavath et al., [15]) and pre-generation probes (Lugoloobi et al., 2026) estimate success before answering but read it from an added head or probe, while calibration tuning (Kapoor et al., [16]) and P(Sufficient)P(\mathrm{Sufficient}) (Zhang et al., 2026) use the model’s token probability but score an answer it has already given. SuffiCE instead trains the model’s own probability of choosing to answer, so the same next-token probability that would decide whether to answer or gather more evidence is calibrated, with no extra head and no answer to generate first. We choose this formulation because the model’s next-token distribution already assigns a probability to answering versus seeking more evidence, so no classifier head is added and the confidence is the model’s own preference between the two actions. A fully-only estimator with the same formulation falls short of SuffiCE (§ 5.2), so the gains come from training across evidence states. Two further properties set SuffiCE apart, and no prior method has either. Training on controlled evidence subsets of the same problem lets the estimator learn which evidence decides success, rather than only which problems are hard (§ 5.2). Evaluating within each problem tests this directly, showing that confidence rises with the decisive evidence and not with every piece added (§ 5.3).

Case Studies

MuSiQue Case Studies

We examine two branching (3hop2) questions from the MuSiQue test set in detail (Tables 13 and 14). In both, the branches e1e_1 and e2e_2 answer two independent sub-questions, and only the final hop e3e_3 combines their answers into the requested date.

Case 1: When the Republicans took control of the Senate. The branches establish that Alaska supports the Republicans (e1e_1) and that the Senate sets its own rules (e2e_2), but only e3e_3 gives the date on which the Republicans regained the Senate. With both branches shown, no sampled answer is correct. The base logit nonetheless jumps from at most 0.02 to 0.50, treating the two branches as progress despite the lack of sampled success. SuffiCE stays at or below 0.02, matching this low success rate. Once e3e_3 is shown, 20% of sampled answers are correct. SuffiCE rises to 0.19, closely matching this rate, while the base logit falls to 0.11 and becomes underconfident.

Case 2: When the Dutch reached New Zealand. The branches establish that The Lord of the Rings was filmed in New Zealand (e1e_1) and that the Dutch first brought enslaved people to New York (e2e_2), but only e3e_3 gives the date on which the Dutch navigator Abel Tasman reached New Zealand. Again no sampled answer is correct with both branches, yet the base logit climbs with each branch to 0.98. SuffiCE reaches only 0.16. With full evidence, 30% of sampled answers are correct. SuffiCE gives 0.34, again close to this rate, while the base logit stays at 0.96 and is far overconfident.

Comparing the two full states. The two questions share the same structure and both show all three supporting paragraphs, yet they differ in how often they are answered correctly. SuffiCE follows this difference, matching each success rate within 0.04, whereas the base logit misses in opposite directions. The confidence of SuffiCE is not fixed by the evidence being complete, so it judges not only whether the reasoning chain is complete but also how likely the answering model is to succeed once it is.

Full Evidence States of the Case Studies

We give the complete inputs behind the clinical case study of § 5.3 and the first MuSiQue case study of §F.1 word for word, with the evidence states

When did the party Alaska generally supports take control of the determiner of rules of the US House and US Senate? Answer: January 2015

e1e_1 Alaska has voted Republican in nearly every presidential election since statehood. → Republicans

e2e_2 Each chamber of Congress sets its own rules (Article I), and the Senate adopts its Standing Rules. → the Senate

e3e_3 Republicans regained the Senate majority in the Congress that began in January 2015, having last held it in January 2007. → January 2015

Evidence∅\varnothinge1e_1e2e_2e1+e2e_1+e_2e1+e2+e3e_1+e_2+e_3
Success0.100.000.000.000.20
Base0.020.020.000.500.11
SuffiCE0.010.010.010.020.19

tbl:13: Case study on a 3hop2 MuSiQue question (Qwen3.5-9B), where the branches e1e_1 and e2e_2 together lead to the final hop e3e_3. Columns are evidence states, marked by which supporting facts are shown (∅\varnothing for none).

When did the people who first imported slaves to New York come to the country where The Lord of the Rings was filmed? Answer: 13 December 1642

e1e_1 The Lord of the Rings trilogy was filmed entirely in New Zealand. → New Zealand

e2e_2 Enslaved Africans were first brought into New York by the Dutch, and later the English. → the Dutch

e3e_3 Dutch navigator Abel Tasman was the first European to sight New Zealand, on 13 December 1642. → 13 December 1642

Evidence∅\varnothinge1e_1e2e_2e1+e2e_1+e_2e1+e2+e3e_1+e_2+e_3
Success0.000.000.000.000.30
Base0.120.150.650.980.96
SuffiCE0.020.030.060.160.34

tbl:14: A second case study on a 3hop2 MuSiQue question (Qwen3.5-9B), in the format of tbl:13. The shaded state has both branches but not the final hop.

of each example and the evidence they contain (Table 15 and Table 16). The state names and evidence labels, such as {1,4}\{1,4\} or e1e_1, are defined only for analysis and are never shown to the model. The model sees only the question and the evidence as it appears in the input, with the section headings of the clinical case but no indication of which state it is in.

ObjectiveAssess and diagnose the infant brought in by a concerned mother due to bilious vomiting and inability to pass meconium, with observed abdominal distension. Answer: Hirschsprung’s disease
1 HISTORYDemographics: 2-day-old male infant. History: The infant was born via a home birth without complications. The mother reports that the infant has had bilious vomiting and has not been able to pass meconium since birth. She also notes that the infant’s abdomen appears distended. Primary symptom: Bilious vomiting. Secondary symptoms: Inability to pass meconium, Abdominal distension. Past medical history: Newborn, no prior medical history. Birth was uneventful. Social history: The infant is the first child of the family. No known family history of genetic or congenital diseases. Review of systems: The infant shows signs of distress, possibly due to abdominal discomfort.
2 PEVital signs: Temperature 37.5°C (99.5°F), Blood pressure not applicable for age, Heart rate 145 bpm, Respiratory rate 40 breaths/min. Abdominal examination: Inspection: Noticeable distension. Auscultation: Decreased bowel sounds. Percussion: Tympanic sound noted, indicating gas. Palpation: The abdomen is firm and distended, with palpable bowel loops; the infant shows signs of discomfort during palpation.
3 BIOPSYRectal biopsy, histopathology: Absence of ganglion cells in the submucosa and myenteric plexus, consistent with Hirschsprung’s disease.
4 IMAGINGAbdominal X-ray, findings: Dilated loops of bowel with no gas in the rectum, suggestive of a distal obstruction.
StateSections shown
{1}HISTORY
{1, 2}HISTORY, PE
{1, 4}HISTORY, IMAGING
{1, 3, 4}HISTORY, BIOPSY, IMAGING
{1, 2, 3, 4}HISTORY, PE, BIOPSY, IMAGING

Table 15. Full input of the MedQA case of Table 3 and the evidence states shown there. Each state shows the objective followed by the listed sections, verbatim.

Question When did the party Alaska generally supports take control of the determiner of rules of the US House and US Senate?

Answer: January 2015

Alaska. Alaska regularly supports Republicans in presidential elections and has done so since statehood. Republicans have won the state’s electoral college votes in all but one election that it has participated in (1964). No state has voted for a Democratic presidential candidate fewer times. Alaska was carried by Democratic nominee Lyndon B. Johnson during his landslide election in 1964, while the 1960 and 1968 elections were close. Since 1972, however, Republicans have carried the state by large margins. In 2008, Republican John McCain defeated Democrat Barack Obama in Alaska, 59.49% to 37.83%. McCain’s running mate was Sarah Palin, the state’s governor and the first Alaskan on a major party ticket. Obama lost Alaska again in 2012, but he captured 40% of the state’s vote in that election, making him the first Democrat to do so since 1968.

Standing Rules of the United States Senate. The Standing Rules of the Senate are the parliamentary procedures adopted by the United States Senate that govern its procedure. The Senate’s power to establish rules derives from Article One, Section 5 of the United States Constitution: “Each House may determine the rules of its proceedings. . . ”

2014 United States Senate elections. The Republicans regained the majority of the Senate in the 114th Congress, which started in January 2015; the Republicans had not controlled the Senate since January 2007. They had needed a net gain of at least six seats to obtain a majority. They held all of their seats, and gained nine Democratic - held seats. Republicans defeated five Democratic incumbents:

Democratic Party of Oregon. The Democratic Party of Oregon, based in Portland, is the official Oregon affiliate of the United States Democratic Party. It is recognized by the state of Oregon as a major political party, along with the Oregon Republican Party. The State Central Committee, made up of two delegates elected from each of Oregon’s 36 counties and one additional delegate for every 15,000 registered Democrats, is the main authoritative body of the party. After Oregon was admitted into the Union in 1859, the Democratic party controlled the state. Oregon elected twice as many Democrats as Republicans between 1859 and 1879 in statewide elections for governor, secretary of state, state treasurer, and congressmen. The party holds 38 members in the State House that has 60 representatives total, and 18 members in the State Senate, out of 30 delegates total. The party also holds the Governor’s office (Kate Brown), Attorney General (Ellen Rosenblum), Labor Commissioner (Val Hoyle), and State Treasurer (Tobias Read). The Democrats also have both U.S senate positions in their state and send four of the five U.S House representatives from Oregon to D.C.

Tammany Hall. Tammany Hall, also known as the Society of St. Tammany, the Sons of St. Tammany, or the Columbian Order, was a New York City political organization founded in 1786 and incorporated on May 12, 1789, as the Tammany Society. It was the Democratic Party political machine that played a major role in controlling New York City and New York State politics and helping immigrants, most notably the Irish, rise up in American politics from the 1790s to the 1960s. It typically controlled Democratic Party nominations and political patronage in Manhattan from the mayoral victory of Fernando Wood in 1854 and used its patronage resources to build a loyal, well - rewarded core of district and precinct leaders; after 1850 the great majority were Irish Catholics.

Political party. At the same time, the political party reached its modern form, with a membership disciplined through the use of a party whip and the implementation of efficient structures of control. The Home Rule League Party, campaigning for Home Rule for Ireland in the British Parliament was fundamentally changed by the great Irish political leader Charles Stewart Parnell in the 1880s. In 1882, he changed his party’s name to the Irish Parliamentary Party and created a well-organized grass roots structure, introducing membership to replace “ad hoc” informal groupings. He created a new selection procedure to ensure the professional selection of party candidates committed to taking their seats, and in 1884 he imposed a firm ‘party pledge’ which obliged MPs to vote as a bloc in parliament on all occasions. The creation of a strict party whip and a formal party structure was unique at the time. His party’s efficient structure and control contrasted with the loose rules and flexible informality found in the main British parties; – they soon came to model themselves on the Parnellite model.

Ethical Considerations

Our clinical experiments use MIMIC-IV-Ext Clinical Decision Making [9], a deidentified dataset accessed through PhysioNet under its data use agreement, and we do not release its data. SuffiCE is a research method and is not intended for clinical use. We used Claude and ChatGPT as AI assistants in preparing this manuscript, in accordance with the ACL policy on AI writing assistance.

References

  1. [1]Akari Asai, Zeqiu Wu, Yizhong Wang, Avr​​up Sil, and H​​​​annah Hajishirzi. 2024. Self-r​​ef: Learning to retrieve, generate, and critique through self-reflection. In ICLR.arxiv.org/abs/2310.11511
  2. [2]David Bani-Harouni, Chantal Pellegrini, Ege Özsoy, Nassir Navab, and Matthias Keicher. 2026a. Language agents for hypothesis-driven clinical decision making with reinforcement learning. In ICLR. proceedings.iclr.cc.
  3. [3]David Bani-Harouni, Chantal Pellegrini, Paul Stangel, Ege Özsoy, Kamilia Zaripova, Nassir Navab, and Matthias Keicher. 2026b. Rewarding doubt: A reinforcement learning approach to calibrated confidence expression of large language models. In ICLR. proceedings.iclr.cc.
  4. [4]Mehul Damani, Isha Puri, Stewart Slocum, Idan Shenfeld, Leshem Choshen, Yoon Kim, and Jacob Andreas. 2026. Beyond binary rewards: Training lms to reason about their uncertainty. In ICLR. proceedings.iclr.cc.arxiv.org/abs/2507.16806
  5. [5]Xilin Dang, Kexin Chen, Xiaorui Su, Ayush Noori, Iñaki Arango, Lucas Vittor, Xinyi Long, Yuyang Du, Marinka Zitnik, and Pheng-Ann Heng. 2026. Knowguard: Knowledge-driven abstention for multi-round clinical reasoning. In ICLR. proceedings.iclr.cc.arxiv.org/abs/2509.24816
  6. [6]Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. 2024. A survey of confidence estimation and calibration in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6577–6595, Mexico City, Mexico. Association for Computational Linguistics.arxiv.org/abs/2311.08298
  7. [7]Tilmann Gneiting and Adrian E. Raftery. 2007. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359–378.
  8. [8]Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On calibration of modern neural networks. In ICML, volume 70 of Proceedings of Machine Learning Research, pages 1321–1330. PMLR.arxiv.org/abs/1706.04599
  9. [9]Paul Hager, Friederike Jungmann, Robbie Holland, Kunal Bhagat, Inga Hubrecht, Manuel Knauer, Jakob Vielhauer, Marcus Makowski, Rickmer Braren, Georgios Kaissis, and Daniel Rueckert. 2024. Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nature Medicine, 30(9):2613–2622.DOI
  10. [10]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuhuai Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.arxiv.org/abs/2106.09685
  11. [11]Akriti Jain and Aparna Garimella. 2026. Knowing what’s missing: Assessing information sufficiency in question answering. In Findings of the Association for Computational Linguistics: EACL 2026, pages 4163–4174, Rabat, Morocco. Association for Computational Linguistics.
  12. [12]Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Active retrieval augmented generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7969–7992, Singapore. Association for Computational Linguistics.
  13. [13]Di Jin, Eileen Pan, Nassim Oufatt​​le, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421.arxiv.org/abs/2009.13081
  14. [14]Hailey Joren, Jianyi Zhang, Chun-Sung Ferng, Da-Cheng Juan, Ankur Taly, and Cyrus Rashtchian. 2025. Sufficient context: A new lens on retrieval augmented generation systems. In ICLR.arxiv.org/abs/2411.06037
  15. [15]Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, and 17 others. 2022. Language models (mostly) know what they know. Preprint, arXiv:2207.05221.arxiv.org/abs/2207.05221
  16. [16]Sanyam Kapoor, Nate Gruver, Manley Roberts, Katherine Collins, Arka Pal, Umang Bhatt, Adrian Weller, Samuel Dooley, Micah Goldblum, and Andrew Gordon Wilson. 2024. Large language models must be taught to know what they don’t know. In NeurIPS.arxiv.org/abs/2406.08391
  17. [17]Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations.arxiv.org/abs/1412.6980
  18. [18]Polina Kirichenko, Mark Ibrahim, Kamalika Chaudhuri, and Samuel J. Bell. 2025. Abstentionbench: Reasoning llms fail on unanswerable questions. In NeurIPS.arxiv.org/abs/2506.09038

Paper details

vlnc:W_vcrqvjz8
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 4

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents