WorksComputer Science
Each Insufficient in Its Own Way: Calibrating Language Models Across Varying Evidence States
Abstract
How can a language model know whether it has enough evidence to answer? Language models tend to answer even when the evidence they have is insufficient. Deciding whether to answer or gather more evidence requires a calibrated estimate of how likely an answer is to be correct with the available evidence. However, most calibration work treats the input as fixed, without asking how this estimate should change with the evidence. We introduce evidence-state calibration, which requires a model's confidence before answering to match its probability of answering correctly and to change accordingly whenever the evidence changes this probability. We propose SuffiCE (Sufficiency Confidence Estimation), which fine-tunes the confidence already present in a model's next-token distribution to match how often the model answers correctly from each evidence state. Across clinical diagnosis and multi-hop question answering with four models from two families, SuffiCE reduces the Brier score by up to 60% and increases AUROC by up to 0.16. Its confidence matches the empirical success rate and responds not only to the amount of evidence but also to which evidence is present, showing that models can learn to judge whether their current evidence is enough to answer.
Introduction
Before answering a question, one must judge whether the information at hand is enough to answer it. In high-stakes settings, a physician who diagnoses before gathering enough information may miss the real disease. For example, a patient who has been tired for two months may seem iron deficient, but a careful clinician would order more blood tests, which may reveal celiac disease, an autoimmune cause of poor iron absorption.
Language models facing insufficient information, however, often reach confident but incorrect conclusions [18] (Yin et al. 2023).
A common remedy for such overconfidence is to abstain below a confidence threshold (Wen et al., 2025). This works only if the model is calibrated, so that answers given x% confidence are correct about x% of the time [8, 6]. To achieve this, existing methods train language models to provide calibrated confidence estimates for generated answers [16, 3].
However, most calibration work holds the evidence fixed for each question, so it can check whether confidence is calibrated but not whether it varies correctly as a function of the evidence. In interactive settings, the available information evolves as models ask clarifying questions (Zhang and Choi, 2025) or seek additional information (Li et al., 2024). At each turn, the model must decide whether to answer or keep gathering evidence.
To make this decision, a model needs a confidence estimated from the available evidence alone, before it produces any answer. We call the evidence available to a model at a given moment its evidence state. We then define evidence-state calibration as requiring the confidence before answering to match the probability of answering correctly, so that whenever the evidence changes this probability, the confidence changes by the same amount (Figure 1).

Figure 1. Given a question with two different sets of evidence, the model’s probability of answering correctly (gray) differs, and its confidence before answering (yellow) should match it for each. (a) Calibrated confidence matches both. (b) Miscalibrated confidence mismatches the dynamic of the true probability.
Testing this requires observing the same problem under different evidence states. Rather than contrasting the extremes of complete and missing information [18] or following a single fixed sequence of accumulating evidence (Zhang et al., 2026), we construct granular states for each problem, spanning minimal to complete evidence and including different combinations of information. Granularity matters because sufficient evidence states are alike, but each insufficient one is insufficient in its own way.1 Some insufficient states can still be answered from what the model already knows [14], while others lack the one piece of evidence the answer depends on. We therefore measure each state’s sufficiency by how often the model answers correctly from it, not by whether the required facts are present, and teach the model to anticipate this success rate before answering.
With these success rates as targets, we propose SuffiCE (Sufficiency Confidence Estimation), to our knowledge the first method that trains a model’s confidence before answering to be evidence-state calibrated. It reads the confidence directly from the model’s own next-token distribution. The confidence is the model’s probability of choosing to answer instead of asking for more information. No additional classifier or generated number is needed. This avoids an ambiguity in verbalized confidence. When a model states that it is 80% confident, it is unclear whether to trust the stated number or the probability the model assigned to generating it. We instead train the model’s own probability directly to match the success rate across our diverse evidence states. The result is a model that can tell, before answering, whether what it currently knows is enough to answer correctly.
We evaluate on two clinical diagnosis datasets, MIMIC [9] and MedQA [13], and the multi-hop question answering dataset MuSiQue (Trivedi et al., 2022). Across three Qwen models from 4B to 27B parameters and Kimi K2.6, SuffiCE improves calibration for every model and dataset, reducing the Brier score
of the base logit by up to 60% and increasing its AUROC by up to 0.16. Its confidence also follows the evidence state. An estimator trained on complete evidence alone falls short on every metric, so these gains come from training across evidence states. In practice, answering only when SuffiCE is confident gives fewer wrong answers than any baseline.
Related Work
Deciding whether to answer or gather more evidence requires a calibrated judgment of whether the current evidence suffices. This connects three lines of work: interactive information seeking explores how models acquire evidence, information sufficiency and unanswerability examine whether an input supports an answer, and confidence calibration studies whether confidence reflects actual accuracy.
Interactive information seeking. In an interactive setting, a model can gather more information before answering by asking clarifying questions (Zhang and Choi, 2025; Li et al., 2024), retrieving documents [12, 1], or ordering diagnostic tests and searching for medical evidence [5] (Schmidgall et al. 2026). Deciding when enough has been gathered typically relies on a confidence or uncertainty signal [12] (Zhang and Choi 2025; Li et al. 2024), and MediQ shows that improving confidence estimation can enhance interactive diagnostic accuracy (Li et al., 2024). Our approach aims to provide such a signal, estimating before answering how likely the current evidence is to yield a correct answer. Closest to our work, LA-CDM trains multi-turn diagnosis with a verbalized confidence that guides whether to request another test, but this confidence evaluates a hypothesis the model has already generated [2]. We instead estimate success before generating an answer and train this estimate across controlled evidence states of the same problem, capturing how changes in evidence affect the likelihood of success.
Information sufficiency and unanswerability. Whether an input contains enough information to answer a question has been widely studied through unanswerability (Rajpurkar et al., 2018; Yin et al., 2023). Benchmarks test whether models abstain when information is insufficient [18], including settings where key evidence is removed from otherwise answerable questions (Trivedi et al., 2022; Machcha et al., 2026). Detection methods assess sufficiency using model hidden states (Slobodkin et al., 2023; Lavi et al., 2026) or by identifying missing information [11], and training methods teach models to abstain when information is insufficient (Zhai et al., 2026). A shared limitation is that sufficiency is a binary label attached to the input, so a partially informative input counts as either sufficient or not, regardless of which model answers it. Joren et al. [14] take a first step, showing that models often answer correctly from insufficient context. Whether evidence is enough depends on the model answering, so we define sufficiency by the model’s success, how often it answers correctly from each controlled evidence state.
Confidence calibration. A model is calibrated when its confidence matches its accuracy [8]. Training-free estimates based on verbalized judgments (Tian et al., 2023; Xiong et al., 2024), sampled-answer consistency (Kuhn et al., 2023), or token probabilities [15] can be informative, but calibration varies across tasks (Vashurin et al., 2025). Trained methods estimate answer correctness [16], verbalize calibrated confidence [3, 4] (Lin et al. 2022; Li et al. 2025), or predict success before answering [15] (Lugoloobi et al. 2026). However, they do not explicitly train confidence to track success across changing evidence states. Zhang et al. (2026) evaluate confidence as hints accumulate in a fixed sequence but do not train on these changing states. We instead train and evaluate confidence across controlled evidence states, varying both the amount and type of evidence to track changes in empirical success (Appendix § 12).
Method
Our method, SuffiCE, has three stages. We first construct multiple evidence states for each problem (§ 3.2), then estimate a fixed answering policy’s success rate on each state from repeated answers (§ 3.3), and finally train an estimator to predict this rate from the state alone, before answering (§ 3.4).
Problem Formulation
Problem consists of a question , a pool of evidence , and a gold answer . An evidence state pairs the question with the evidence available to the model.
A fixed answering policy samples an answer , which a grader scores as . The success probability of an evidence state is
We measure how sufficient a state is by , because the decision to answer or gather more information hinges on how likely an answer is to be correct. Sufficiency therefore depends on the model as well as the information. A state is sufficient when its information, together with what the model already knows, lets the model reach the correct answer, whether or not the information itself is complete. The same state can thus be sufficient for one model and insufficient for another.
Evidence States
For each problem , we construct evidence states , , each pairing the question with a subset of its evidence. We write for a generic state when the indices are not needed. All states of a problem share the question and gold answer and differ only in the evidence shown, so differences in success between them are attributable to the evidence. The model is never told which state it is in and sees only the question and the evidence, so it must judge from their content whether the evidence is enough. Training on the same problem under different evidence teaches it which evidence makes a correct answer reachable.
Evidence states vary in both the amount and the composition of evidence. Since success depends on which evidence is present, not on the order it was gathered, states can combine evidence freely, including states that hold the amount fixed and vary the composition (§ 5.3). The states are specific to each dataset (§ 4).
Evidence-state calibration. We seek a confidence that a model with parameters computes from the state alone, before any answer is generated. Evidence-state calibration requires it to satisfy two criteria, where expectations are taken over the constructed evidence states.
1. Calibration. Across all states assigned the same confidence , the average success probability is [8]:
2. Evidence sensitivity. A change in evidence drives confidence as much as it moves success:
Here and denote any two evidence states and of the same problem , so the difference in confidence should match the difference in success.
To see why calibration alone is not enough, we split the squared error of a confidence by problem. For problem with states, let and denote the averages of and over its states, and for any two of its states , let
be the gap between the change in confidence and the change in success from to , which is zero when Eq. (3) holds exactly.
Error Decomposition
Proposition 1. For any confidence ,
where the sum runs over all ordered pairs of states of problem , and each problem is weighted by its number of states.
The first term is the error in each problem’s average success, and the second is the error within each problem, which is zero exactly when criterion 2 holds with equality (proof in Appendix § 11.§ 1). Calibration does not constrain the second term. Giving every state its problem’s average success satisfies criterion 1 exactly but leaves this term at the variance of success within each problem, which only following the evidence can reduce. Since both terms are nonnegative, we seek a confidence that drives the left side to zero and thus satisfies both criteria.
Monte Carlo Estimation of Success Probability
Because computing the true success probability exactly would require grading every answer the policy could generate, we estimate it by sampling from the fixed answering policy (Figure 2, top). A single answer is only correct or incorrect, so we draw independent answers per state and average their grades. For each evidence state of problem , we sample , grade each as , and define the empirical success rate as

Figure 2. Overview of SuffiCE. Each evidence state pairs an input question with the available evidence . Top: A frozen answering policy samples answers per evidence state, and the fraction graded correct is the empirical success rate . Bottom: The estimator reads its confidence from the softmax over the logits of A (Answer) and S (Seek), and is trained with soft-label binary cross-entropy to match .
Since , , so is an unbiased Monte Carlo estimate of the frozen policy’s success probability.
The estimate is specific to the answering policy , so the sufficiency of a state always refers to whether can answer correctly given the information in , and the confidence we train is calibrated to this policy. Since computing requires the gold answer, which is unavailable at decision time, we use it only as a training target.
Sufficiency Confidence Estimator
At decision time, the model must therefore predict from the evidence state alone. We read this prediction from the model’s own next-token distribution rather than adding a separate classifier head (Figure 2, bottom). Each evidence state is placed in a prompt that asks the model, without answering the task, whether the currently available information is sufficient for it to answer correctly, and defines two actions, A to answer now because the information is sufficient, and S to seek more information because it is not. Rather than letting the model generate a response, we read the probabilities it assigns to A and S as the first token of its reply (prompt in Appendix § 9.§ 1).
Let and denote the logits assigned to A and S at the first response position from a single forward pass. We take the softmax over these two logits, giving . We call the sufficiency confidence of state , and the model that produces it the sufficiency confidence estimator.
We train to match the empirical success rate using soft-label binary cross-entropy:
Binary cross-entropy is a strictly proper scoring rule ([7]) and is linear in the target, so with the unbiased labels of § 3.3 the expected loss at each state is minimized by despite sampling noise in . Training on this loss also controls both terms of Proposition 1.
Corollary 1. For any confidence , let be the loss of Eq. (6) averaged over all evidence states and over the sampled answers that determine , and the minimum expected loss over all confidences, reached exactly at . Then
so each term of Eq. (4) is at most . Lowering the loss therefore lowers an upper bound on both terms, and the bound reaches zero exactly at (proof in Appendix § 11.§ 2).
Experimental Setup
Datasets. We evaluate on two clinical diagnosis datasets and one multi-hop question answering dataset, which range from mostly answered correctly to mostly not (details in Appendix § 8.§ 1). MIMIC ([9]) consists of real emergency-department admissions for four abdominal diseases, and the model chooses among the four. MedQA ([13]) contains questions from the United States Medical Licensing Examination, and the model names the diagnosis in free text. Both are written as structured clinical cases following AgentClinic ((Schmidgall et al., 2026)). MuSiQue ((Trivedi et al., 2022)) composes multi-hop questions over Wikipedia from single-hop questions, each answered by one supporting paragraph, and the model gives a short answer. All states of a problem are assigned to the same split to avoid leakage between training and test.
Evidence states. Clinical cases are divided into history, physical examination, imaging, and other tests such as laboratory and pathology. Each state adds a different subset of the remaining sections to the history.
In MuSiQue, we use 3-hop questions, whose evidence units are the three supporting paragraphs, one per sub-question. In chain questions, each answer is needed to ask the next sub-question (), while in branching questions, and answer independent sub-questions that both feed the final one, . States range from the question alone and distractors only (related paragraphs that MuSiQue retrieves but that do not help answer the question) to one, two, or all three supporting paragraphs (Figure 3, examples in Appendix § 13.§ 2). Every state but the question-only one shows three paragraphs, padded with distractors in one shuffled order per question, so neither the number nor the position of paragraphs reveals the state.
Figure 3. Evidence states of MuSiQue for chain and branching questions. Hatched cells give the number of distractors padding each state to three paragraphs.
Baselines. We compare against three baselines on the same untrained model. Base logit (Base) is our sufficiency confidence estimator before training, reading under the prompt of § 3.4. Verbalized asks the model to state the probability that it could answer correctly from the available information ((Tian et al., 2023)). Platt scaling (Platt, 1999) rescales the base logit to fit the success rates in the training set, so it can recalibrate the confidence but not reorder states (prompts in Appendix § 9).
Implementation details. We evaluate four answering models, three Qwen models of increasing size (Qwen3.5-4B, Qwen3.5-9B, and Qwen3.8-27B) and Kimi K2.6 from a different model family. The answering policy can be any frozen policy. Our main experiments use the estimator’s own base model, to study how well a model can judge its own evidence state. Success rates are estimated from sampled answers per state. Further details are in Appendix § 8.§ 3.
Metrics. Let be the sufficiency confidence of state of problem and its empirical success rate. The Brier score, the mean of over the test set, measures overall error. AUROC, averaged over the sampled answers, measures how well confidence ranks correct answers above incorrect ones. Both pool the whole test set, so they reward knowing which problems are hard. The change correlation , computed over all pairs of states of the same problem from a state to a state with more evidence (Appendix § 10.§ 4), instead tests whether confidence follows success as the evidence state changes, as criterion 2 requires.
Results
Overall Performance
SuffiCE achieves the best result on every metric in every setting (Table 1). Each model is calibrated to its own success rates, so methods are compared only for the same model (Appendix § 8.§ 2). SuffiCE lowers the Brier score by 20–60% relative to the base logit and by 5–28% relative to Platt scaling, the strongest baseline. SuffiCE not only calibrates better but also ranks states better. Platt scaling fits the same labels but only rescales the base confidence, leaving the order of states, and thus AUROC and selective accuracy, unchanged, whereas SuffiCE raises AUROC by up to 0.16. SuffiCE also raises the change correlation above every baseline, so its confidence moves with success as the evidence changes, which §5.3 examines in detail.
| Method | MIMIC Brier ↓ | MIMIC AUROC ↑ | MIMIC ↑ | MedQA Brier ↓ | MedQA AUROC ↑ | MedQA ↑ | MuSiQue Brier ↓ | MuSiQue AUROC ↑ | MuSiQue ↑ |
| Kimi K2.6 | |||||||||
| Base logit | 0.219 ± 0.000 | 0.686 ± 0.000 | 0.320 ± 0.000 | 0.223 ± 0.000 | 0.707 ± 0.000 | 0.413 ± 0.000 | 0.184 ± 0.000 | 0.750 ± 0.000 | 0.430 ± 0.000 |
| Verbalized | 0.131 ± 0.001 | 0.686 ± 0.003 | 0.247 ± 0.013 | 0.162 ± 0.001 | 0.650 ± 0.001 | 0.374 ± 0.010 | 0.174 ± 0.003 | 0.724 ± 0.003 | 0.412 ± 0.011 |
| Platt | 0.124 ± 0.000 | 0.686 ± 0.000 | 0.296 ± 0.000 | 0.150 ± 0.000 | 0.707 ± 0.000 | 0.438 ± 0.000 | 0.142 ± 0.000 | 0.750 ± 0.000 | 0.431 ± 0.000 |
| SuffiCE | 0.105 ± 0.002 | 0.844 ± 0.005 | 0.474 ± 0.045 | 0.141 ± 0.005 | 0.795 ± 0.033 | 0.467 ± 0.064 | 0.134 ± 0.004 | 0.787 ± 0.005 | 0.632 ± 0.009 |
| Qwen3.8-27B | |||||||||
| Base logit | 0.127 ± 0.000 | 0.788 ± 0.000 | 0.353 ± 0.000 | 0.143 ± 0.000 | 0.744 ± 0.000 | 0.413 ± 0.000 | 0.182 ± 0.000 | 0.738 ± 0.000 | 0.274 ± 0.000 |
| Verbalized | 0.112 ± 0.000 | 0.770 ± 0.007 | 0.222 ± 0.003 | 0.174 ± 0.001 | 0.772 ± 0.001 | 0.355 ± 0.000 | 0.203 ± 0.001 | 0.682 ± 0.002 | 0.345 ± 0.008 |
| Platt | 0.099 ± 0.000 | 0.788 ± 0.000 | 0.286 ± 0.000 | 0.126 ± 0.000 | 0.744 ± 0.000 | 0.438 ± 0.000 | 0.123 ± 0.000 | 0.738 ± 0.000 | 0.423 ± 0.000 |
| SuffiCE | 0.080 ± 0.004 | 0.857 ± 0.012 | 0.447 ± 0.040 | 0.091 ± 0.005 | 0.826 ± 0.008 | 0.489 ± 0.019 | 0.092 ± 0.008 | 0.830 ± 0.014 | 0.717 ± 0.024 |
| Qwen3.5-9B | |||||||||
| Base logit | 0.124 ± 0.000 | 0.689 ± 0.000 | 0.247 ± 0.000 | 0.268 ± 0.000 | 0.746 ± 0.000 | 0.367 ± 0.001 | 0.197 ± 0.000 | 0.752 ± 0.000 | 0.356 ± 0.000 |
| Verbalized | 0.143 ± 0.003 | 0.727 ± 0.002 | 0.177 ± 0.011 | 0.290 ± 0.001 | 0.706 ± 0.002 | 0.356 ± 0.004 | 0.145 ± 0.003 | 0.688 ± 0.004 | 0.310 ± 0.012 |
| Platt | 0.110 ± 0.000 | 0.689 ± 0.000 | 0.254 ± 0.000 | 0.134 ± 0.000 | 0.746 ± 0.000 | 0.514 ± 0.001 | 0.099 ± 0.000 | 0.752 ± 0.000 | 0.389 ± 0.000 |
| SuffiCE | 0.098 ± 0.010 | 0.819 ± 0.028 | 0.393 ± 0.034 | 0.107 ± 0.005 | 0.815 ± 0.011 | 0.630 ± 0.012 | 0.078 ± 0.002 | 0.828 ± 0.006 | 0.666 ± 0.015 |
| Qwen3.5-4B | |||||||||
| Base logit | 0.143 ± 0.000 | 0.761 ± 0.000 | 0.311 ± 0.000 | 0.132 ± 0.000 | 0.763 ± 0.000 | 0.452 ± 0.000 | 0.122 ± 0.000 | 0.779 ± 0.000 | 0.335 ± 0.000 |
| Verbalized | 0.190 ± 0.001 | 0.763 ± 0.001 | 0.095 ± 0.009 | 0.250 ± 0.005 | 0.739 ± 0.006 | 0.286 ± 0.027 | 0.164 ± 0.000 | 0.601 ± 0.001 | 0.224 ± 0.003 |
| Platt | 0.097 ± 0.000 | 0.761 ± 0.000 | 0.274 ± 0.000 | 0.108 ± 0.000 | 0.763 ± 0.000 | 0.488 ± 0.000 | 0.097 ± 0.000 | 0.779 ± 0.000 | 0.400 ± 0.000 |
| SuffiCE | 0.085 ± 0.003 | 0.830 ± 0.014 | 0.458 ± 0.097 | 0.103 ± 0.003 | 0.773 ± 0.011 | 0.524 ± 0.072 | 0.086 ± 0.004 | 0.818 ± 0.007 | 0.606 ± 0.020 |
Table 1. Results on the three datasets for Kimi K2.6, Qwen3.8-27B, Qwen3.5-9B, and Qwen3.5-4B, comparing our method with baselines. Results are mean ± SD over three seeds. The best result for each model is in bold.
Calibration across datasets. The base logit is miscalibrated by different amounts across datasets, and SuffiCE improves calibration on all three, moving each curve toward the ideal diagonal (Figure 4, Qwen3.5-9B). The other models follow the same pattern (Appendix § 10.§ 1).
Figure 4. SuffiCE brings confidence closer to perfect calibration on every dataset. Calibration curves on the test set (a) before and (b) after SuffiCE.
Selective answering. If the model answers only the questions it is most confident about and abstains on the rest, SuffiCE is the most accurate at nearly every percentage of questions answered (Table 2 and Appendix § 10.§ 2). On MIMIC, the questions it is most confident about are almost always answered correctly.
| Answered | Base | Verb. | SuffiCE | |
| MIMIC | 10% | 97.2 | 95.3 | 100.0 |
| 30% | 93.4 | 95.3 | 98.5 | |
| 50% | 89.9 | 92.4 | 95.5 | |
| MedQA | 10% | 93.7 | 74.3 | 99.1 |
| 30% | 77.8 | 73.4 | 84.2 | |
| 50% | 65.4 | 63.3 | 74.0 | |
| MuSiQue | 10% | 55.2 | 52.5 | 68.9 |
| 30% | 45.3 | 47.7 | 56.2 | |
| 50% | 39.4 | 34.7 | 42.6 |
Table 2. Accuracy (%) when the model answers the given percentage of questions it is most confident about.
Ablation
To test whether the estimator needs diverse evidence states, we train Qwen3.5-9B only on the full state of each training problem and evaluate it on the whole test set (Figure 5). Success on full states still varies across problems, so this estimator learns which problems are hard but never sees missing evidence. SuffiCE outperforms it on every metric and dataset, both with the same number of training epochs and with the same number of steps (Appendix § 10.§ 3).
Figure 5. Training across evidence states improves both calibration and evidence sensitivity. Estimators trained for the same number of epochs on full states only versus on all evidence states (SuffiCE).
Evidence-State Analysis
Confidence rises with the decisive evidence, not with more evidence. SuffiCE closely tracks the empirical success rate across states, rising only when new evidence makes the question answerable, whereas the base logit rises with every paragraph added (Figure 6, more measures in Appendix § 10.§ 4). For both question types, success stays near zero until the final hop is shown. On chain questions, the base logit treats each earlier hop as progress and is confident enough to answer given (0.71), although the model almost never succeeds (0.06), while SuffiCE stays low (0.16) until arrives. On branching questions, the base logit rises with each branch, while SuffiCE stays low for either branch alone and for both, since neither helps without . SuffiCE is never told the question type or which paragraph fills which hop, so it judges from the content alone whether the reasoning chain is complete. It thus separates evidence that makes a question answerable from evidence that only adds information (Appendix § 13.§ 1).
Figure 6. SuffiCE tracks the empirical success rate as supporting paragraphs are added, while the base logit rises with every paragraph. MuSiQue test set, for (a) chain and (b) branching questions.
Holding the amount fixed, content decides success. Even when every state has the same amount of evidence, SuffiCE distinguishes which states can be answered (Figure 7). Each state shows two of the three supporting paragraphs, and success is high only when the final hop is among them. The base logit is similarly confident on all three states, responding to how much evidence there is, whereas SuffiCE stays low on and rises for both states with .
Figure 7. With the same amount of evidence, SuffiCE tells which states can be answered. Each MuSiQue state shows two of the three supporting paragraphs, for (a) chain and (b) branching questions. Only SuffiCE stays low until the final hop is shown.
Confidence separates decisive from suggestive evidence. In clinical diagnosis, the decisive evidence differs from patient to patient. Table 3 shows a MedQA case of Hirschsprung disease. The Xray is the pivot of the workup. It reveals a distal obstruction and prompts the rectal biopsy, but it cannot tell Hirschsprung disease from other causes of obstruction. Only the biopsy, which finds no ganglion cells, confirms the diagnosis. The base logit is nearly certain in every state, whereas SuffiCE stays uncertain with the X-ray in state and becomes confident only once the biopsy is added in state . With a suitable confidence threshold, SuffiCE would therefore seek more information before answering. Unlike the base logit, SuffiCE thus recognizes which evidence settles the diagnosis and which only signals that more information is needed.
| State | |||||
| Success | 0.20 | 0.40 | 0.40 | 1.00 | 1.00 |
| Base | 0.96 | 0.99 | 0.99 | 1.00 | 1.00 |
| SuffiCE | 0.50 | 0.47 | 0.51 | 0.97 | 0.97 |
Table 3. A MedQA case of Hirschsprung disease (Qwen3.5-9B). Each state is the set of evidence shown. Success is the fraction of ten sampled answers that are correct. Base and SuffiCE give the sufficiency confidence.
A 2-day-old boy has bilious vomiting, has not passed meconium since birth, and has a distended abdomen. What is the diagnosis? Answer: Hirschsprung disease
| 1 HISTORY | Home birth without complications, bilious vomiting, and no meconium since birth. | |
| 2 PE | Firm, distended abdomen with decreased bowel sounds and palpable bowel loops. | |
| 3 BIOPSY | Rectal biopsy: no ganglion cells in the submucosa or myenteric plexus. | |
| 4 IMAGING | Abdominal X-ray: dilated bowel loops and no rectal gas, suggesting a distal obstruction. |
Cross-Policy Confidence Estimation
To test whether the estimator must be the answering model, we fix the answering policy to Kimi K2.6 and train Qwen estimators of three sizes on its success rates (Figure 8). A large enough estimator predicts another model’s success as well as that model does itself. The 27B estimator matches or exceeds Kimi K2.6’s own estimate on every dataset in Brier score, and in on MedQA and MuSiQue. Trained only on Kimi K2.6’s success rates and never shown its answers, the estimator thus learns to anticipate from the evidence state alone how well another model can answer. Smaller estimators are less accurate, and the 4B estimator falls well short of Kimi K2.6’s own estimate.

Figure 8. Qwen estimators of three sizes are trained to predict the success rates of Kimi K2.6, compared with Kimi K2.6 predicting its own (self).
Conclusion
To our knowledge, this is the first work to train a model’s confidence before answering to be calibrated across varying evidence states. The decision to answer or gather more evidence is made before any answer exists, so a model is only as reliable as this judgment. We formalized it as evidence-state calibration, requiring confidence to be calibrated and to change as much as the chance of success whenever the evidence changes, which calibration alone does not guarantee. SuffiCE outperforms every baseline, and its confidence rises with the decisive evidence, not with every piece added. Such confidence is a step toward language models that can judge, in high-stakes settings, when the evidence at hand is enough to answer.
Limitations
SuffiCE shows strong calibration and evidence sensitivity across four answering models and three datasets, but it has three limitations. First, we pre-construct evidence states rather than gather evidence in live multi-turn interactions, a controlled setting that lets us attribute differences in success and confidence to the evidence itself. Second, to limit cost, we estimate each target from ten sampled answers per state, and more samples would give more precise targets. Third, SuffiCE has not been tested in real clinical scenarios, and its confidence should not replace clinical judgment.
References
Experimental Details
Datasets
The three datasets range widely in difficulty (Table 4). For Qwen3.5-9B, the mean success rate on the test set is 0.81 on MIMIC, 0.50 on MedQA, and 0.25 on MuSiQue, so we test the confidence both where most states are answered correctly and where most are not.
| MIMIC | MedQA | MuSiQue | |
| Answer | 4-way choice | free text | short answer |
| Problems | 140 / 60 | 146 / 63 | 200 / 80 |
| States | 840 / 360 | 728 / 302 | 1,500 / 600 |
| Mean success rate | |||
| Qwen3.5-4B | 0.82 / 0.81 | 0.45 / 0.44 | 0.27 / 0.25 |
| Qwen3.5-9B | 0.84 / 0.81 | 0.50 / 0.50 | 0.29 / 0.25 |
| Qwen3.8-27B | 0.87 / 0.83 | 0.64 / 0.62 | 0.34 / 0.31 |
| Kimi K2.6 | 0.87 / 0.82 | 0.76 / 0.71 | 0.44 / 0.43 |
Table 4. Dataset statistics, given as train / test. The mean success rate of each answering model is taken over all evidence states.
MIMIC. We use emergency admissions from MIMIC-IV-Ext Clinical Decision Making (Hager et al., [9]). We sample 50 admissions for each of four abdominal diseases, appendicitis, cholecystitis, diverticulitis, and pancreatitis, and the task is to choose one of the four. We convert each admission to the OSCE layout of AgentClinic (Schmidgall et al., 2026). The history, laboratory results, and radiology reports are copied verbatim, and an LLM (GPT-5.1) writes only a one-sentence objective naming the presenting complaints and splits the free-text physical examination into sections without changing its values. Microbiology results are never shown, since cultures return after the diagnostic decision. We split admissions into training and test sets stratified by disease.
MedQA. We use the OSCE-style cases of the extended AgentClinic-MedQA set (Schmidgall et al., 2026), which are derived from USMLE questions (Jin et al., [13]), and the task is to name the diagnosis in free text. We exclude 5 of its 214 cases, four with no test results and one with a nonstandard format, which leaves 209. States that show only imaging or only the other tests exist for the 97 cases that have both kinds of test. We split cases by diagnosis, which keeps near-duplicate vignettes on the same side and makes every test diagnosis unseen during training.
MuSiQue. We use 3-hop questions from MuSiQue-Ans (Trivedi et al., 2022), and the task is to give a short answer. For each question, we sample three distractors from its non-supporting paragraphs. Its three supporting paragraphs and three distractors are shuffled once with a seed derived from the question, and every state shows its paragraphs in that order. Training questions come from the MuSiQue training set and test questions from its development set, since its test set has no released answers, with no single-hop question shared between them. MuSiQue’s own decompositions distinguish two question types (Table 5). In a chain question each hop needs the answer of the previous one, whereas in a branching question the first two hops are independent and only the final hop combines them. A branching question therefore has a state with alone, while in a chain is only useful after . The full inputs of the branching example appear in Table 16.
| Chain (3hop1) | ||
| Who was appointed arbitrator by the first convert to the faith that is the most common religion in Southeast Asia? | ||
| Answer: Abu Musa Ashaarı | ||
| Hop | Sub-question | Answer |
| What is the most common religion in Southeast Asia? | Islam | |
| Who was the first convert to the faith called #1? | Ali | |
| Who was appointed arbitrator by #2? | Abu Musa Ashaarı | |
| Branching (3hop2) | ||
| When did the party Alaska generally supports take control of the determiner of rules of the US House and US Senate? | ||
| Answer: January 2015 | ||
| Hop | Sub-question | Answer |
| Which political party does Alaska generally support? | Republicans | |
| Who determines the rules of the US House and US Senate? | The Senate | |
| When did #1 take control of #2? | January 2015 |
Table 5. The two MuSiQue question types, with sub-questions from MuSiQue’s decompositions (capitalization normalized). #k stands for the answer of hop , supported by paragraph . In the chain, each hop depends on the previous one, so each stage has its own color. In the branching question, and are independent and share a color, and combines their answers.
| State | Paragraphs shown, in order |
| Question only | — |
| Distractors only | |
Table 16. Full input and all eight evidence states of the 3hop2 MuSiQue question of Table 13. Supporting paragraphs – and distractors – are shown verbatim. The six paragraphs of each question are shuffled once into a fixed order, here , , , , , , and every state shows its paragraphs in that order, so a paragraph’s position does not reveal the state. Every state except the question-only one shows three paragraphs, padded with distractors.
Answering Policies
The main experiments study self-calibration. Each model in Table 1 serves as both the answering policy and the estimator, so it estimates its own chance of success, and every method for that model is compared against the exact same targets.
Targets for self-calibration. For each model, the frozen model with thinking disabled samples answers per evidence state, and the fraction graded correct is the success rate of that state. These success rates are the only targets used for that model. The base logit reads from this model before training, and verbalized confidence prompts this model for the probability that it could answer correctly. Platt scaling fits its two parameters to this model’s success rates on the training set, and SuffiCE trains a LoRA adapter on a copy of this model to predict the same success rates. All four are evaluated against the success rates of this model on the test set. Methods are therefore compared only for the same model, since each model has its own targets. The cross-policy experiment of § 5.4 is the only exception, where Qwen estimators are trained on and evaluated against the success rates of Kimi K2.6.
Difficulty. Success rates differ across answering models, little on MIMIC but widely on MedQA and MuSiQue (Table 4), so the same evidence state has a different target for each answering policy.
Precision of the targets. Each target is estimated from answers, so even a perfect estimator that outputs the true success rate has a nonzero Brier score against it. This noise floor is the expected binomial sampling variance , which we estimate without bias by averaged over the test set. On every dataset, the Brier score of SuffiCE is over 13 times this floor (Table 6). Sampling noise therefore accounts for only a small part of any method’s error, and ten answers per state are enough to compare estimators.
| Dataset | Noise floor | SuffiCE Brier |
| MIMIC | 0.003 | 0.092 |
| MedQA | 0.008 | 0.111 |
| MuSiQue | 0.007 | 0.097 |
Table 6. Noise floor, the expected Brier score of a perfect estimator against ten-answer success rates, compared with the Brier score of SuffiCE (Table 1). Both are averaged over the four answering models, on the test set.
Training Details
The settings shared by all models and datasets are listed in Table 7. All models are run with thinking disabled. Answers are sampled at temperature 1.0 without top- or top- truncation. The estimator is a rank-32 LoRA adapter ([10]) on a copy of the estimator’s base model, which is the answering model itself except in the cross-policy experiment (§ 5.4), initialized to have no effect, so before training it gives the base logit. All training and sampling run on Tinker.
| Setting | Value |
| Success-rate estimation | |
| Answers per state | 10 |
| Temperature | 1.0 |
| Top- / top- | none |
| LLM judge (if used) | GPT-5.6 Luna |
| Estimator training | |
| LoRA rank | 32 |
| Loss | soft-label BCE |
| Optimizer | Adam ([17]) |
| Learning rate | |
| Adam | 0.9, 0.95, |
| Learning-rate schedule | linear decay to 0 |
| Batch size | 8 |
| Seeds | 3 (batch order, adapter init.) |
| Verbalized baseline | |
| Runs | 3 |
Table 7. Settings shared by all models and datasets.
Grading. Each sampled answer is graded as correct or incorrect against the reference. On MIMIC, an answer is correct if, after lowercasing and removing everything but letters, it is exactly the name of the correct disease among the four options, and any other answer is incorrect. On MedQA, GPT-5.6 Luna judges whether a physician would accept the answer as the same diagnosis as the reference. On MuSiQue, an answer is correct if it matches the reference or one of its aliases after SQuAD normalization (lowercasing and removing punctuation and articles), and otherwise GPT-5.6 Luna judges it given the question and the reference. Both judges decode greedily, and their prompts are in Appendix § 9.§ 4.
Prompts
We list every prompt verbatim, grouped by role. Placeholders are in braces. {question} is the MuSiQue question or the clinical case’s objective, {context} and {case} are the evidence shown in the current state, and {task line} is the dataset-specific line of Table 8. All prompts are given as a single user message with thinking disabled, except the judges, which also receive a system message.
| Dataset | {task line} |
| MIMIC | Which of the following is the most likely diagnosis: appendicitis, cholecystitis, diverticulitis, or pancreatitis? |
| MedQA | What is the most likely diagnosis? |
| MuSiQue | (none, the question is the whole task) |
Table 8. The dataset-specific task line of the sufficiency and verbalized prompts.
Sufficiency Confidence
The sufficiency confidence estimator and the base logit read the probabilities of A and S as the first token of the reply (§ 3.4).
Sufficiency prompt (all datasets)
Decide whether the currently available information is
sufficient for you to answer the task correctly. Do not
answer the task itself.
A = the information is sufficient; answer now.
S = the information is insufficient; seek more information.
Task:
{question}
{task line}
Available information:
{context}
Is the available information sufficient? Respond with
exactly one letter: A or S.
Answering
The frozen answering policy samples ten answers per state with these prompts, from which the empirical success rate is computed (§ 3.3).
Answering prompt: MIMIC
{case}
This is all the information available about the patient.
Based on this information, which of the following is the
most likely diagnosis?
appendicitis
cholecystitis
diverticulitis
pancreatitis
Respond with only one of these four words, nothing else.
Answering prompt: MedQA
{case}
This is all the information available about the patient.
Based on this information, what is the most likely
diagnosis?
Respond with only the diagnosis, nothing else.
Answering prompt: MuSiQue
Answer the question using the provided information and your
own knowledge.
If you are unsure, give your best guess.
Give only the final answer, with no explanation, in at
most 100 characters.
{case}
Answer:
Verbalized Baseline
The first number in the reply is taken as the confidence, with “85%” read as 0.85.
Verbalized confidence prompt (all datasets)
Based on the currently available information, estimate the
probability that you could answer the task correctly
without seeking any additional information.
Do not answer the task itself.
Respond with a single number between 0 and 1, where 0
means the current information is certainly insufficient for
a correct answer and 1 means it is certainly sufficient.
Task:
{question}
{task line}
Available information:
{context}
Probability:
LLM Judges
MedQA answers are graded by the judge below. MuSiQue answers that are not a normalized exact match with the reference or one of its aliases are graded by the MuSiQue judge. MIMIC answers are graded by exact match. All methods for a model are evaluated against the same grades, so differences between methods do not come from differences in grading.
MedQA judge: system
You are an expert physician grading a diagnostic answer
against a reference diagnosis.
First explain your reasoning briefly, then give a grade of
1 (correct) or 0 (incorrect).
MedQA judge: user
Correct diagnosis: {correct_diagnosis}
Doctor's diagnosis: {predicted_diagnosis}
Grade 1 if a physician would accept the doctor's diagnosis
as the same diagnosis as the correct one. Accept synonyms,
abbreviations, alternate spellings, historical or modern
terminology, and diagnoses at an equivalent level of
specificity. A more specific form of the correct diagnosis
is also acceptable.
Grade 0 if the doctor's diagnosis:
- is a different condition, even a closely related one,
- is only a broader disease category that contains the
correct diagnosis but does not identify the more specific
diagnosis the correct one requires,
- lists several possibilities without committing to the
correct one, or
- is missing.
Respond in exactly this format:
<reasoning>At most three sentences comparing the two
diagnoses.</reasoning>
<grade>1 or 0</grade>
MuSiQue judge: system
You are grading short answers to multi-hop questions
against a reference answer.
First explain your reasoning briefly, then give a grade of
1 (correct) or 0 (incorrect).
MuSiQue judge: user
Question: {question}
Reference answer: {answer}
Model answer: {predicted}
Grade 1 only if the response clearly commits to an answer
that is semantically correct according to the reference.
Grade 0 if it gives multiple alternative answers without
clearly committing to the correct one, or if it gives a
wrong answer to the question alongside the correct one.
Ignore extra details that do not answer the question, even
if they are inaccurate. Differences in wording or
formatting are acceptable when they preserve the same
underlying answer. A broader place, category, or entity, or
a related-but-different event, is not equivalent to the
reference. Dates are the only exception: differences in
date precision are acceptable when they preserve the
correct time period, so an exact date, month/year, year, or
decade may be treated as equivalent when consistent with
the reference. Extremely broad periods such as centuries
are not sufficient substitutes for a specific year or date.
Respond in exactly this format:
<reasoning>At most two sentences.</reasoning>
<grade>1 or 0</grade>
Additional Results
Calibration Curves for All Models
We extend the calibration curves of Figure 4 to all four answering models (Figure 9).
Figure 9. Calibration curves of Qwen3.5-4B, Qwen3.5-9B, Qwen3.8-27B, and Kimi K2.6 on the test set, (a) before and (b) after SuffiCE.
The base confidence errs in no consistent direction. Under the same prompt, Qwen3.5-9B is overconfident on every dataset, Qwen3.8-27B and Kimi K2.6 are underconfident on every dataset, and Qwen3.5-4B is overconfident on MedQA but underconfident on MIMIC and MuSiQue. No single correction therefore repairs the base confidence across models and datasets. Many base curves are also not monotone, so a state with higher confidence is not reliably more answerable. Platt scaling can shift such a curve toward the diagonal but cannot straighten it, since it keeps the order of states.
SuffiCE moves every curve toward the diagonal. After training, every Qwen curve lies closer to the diagonal. The confidence also adapts to the range of each dataset, spanning low values on MuSiQue, where most states cannot be answered, and concentrating near the top on MIMIC, where most are answered correctly. The same prompt and objective produce both, so the estimator learns the scale of each dataset from its targets rather than imposing a fixed one.
SuffiCE carries over to another model family. Before training, Kimi K2.6 is strongly underconfident on every dataset. SuffiCE moves its curve toward the diagonal on each.
Selective Answering for All Models
Across all four answering models and three datasets, SuffiCE has the highest accuracy at most percentages of questions answered, with the smallest margin for Qwen3.5-4B on the clinical datasets (Figure 10, extending Table 2).
Figure 10. Selective answering on the test set of every dataset (rows) and model (columns). Each curve plots the accuracy of the answered questions against the percentage answered, answering the questions with the highest confidence first. Higher is better. Platt scaling preserves the ranking of the base logit, so their curves coincide.
Step-Matched Ablation
The ablation of § 5.2 trains the full-only estimator for the same number of epochs as SuffiCE. Since it has one state per problem instead of four to eight, it then takes far fewer gradient steps. Here we instead train it for the same number of steps as SuffiCE, with all other hyperparameters unchanged (Table 9). SuffiCE remains better on every metric and dataset, so the gap comes from training across evidence states, not from the number of updates.
| Method | Brier | AUROC | |
| MIMIC | |||
| Full only | 0.130 | 0.745 | 0.251 |
| SuffiCE | 0.098 | 0.819 | 0.393 |
| MedQA | |||
| Full only | 0.131 | 0.782 | 0.554 |
| SuffiCE | 0.107 | 0.815 | 0.630 |
| MuSiQue | |||
| Full only | 0.101 | 0.794 | 0.532 |
| SuffiCE | 0.078 | 0.828 | 0.666 |
Table 9. Step-matched ablation for Qwen3.5-9B. The full-only estimator is trained for the same number of steps as SuffiCE.
Additional Evidence-State Analysis
Ordering of pairs in . Each pair of states enters once, ordered from less to more evidence. The correlation subtracts the average change, so confidence that rises equally whenever evidence is added gets no credit, and rewards confidence that rises more for the evidence that raises success more.
We report two measures of evidence sensitivity beyond for every answering model (Table 10).
| Method | MIMIC | MIMIC | MedQA | MedQA | MuSiQue | MuSiQue |
| Kimi K2.6 | ||||||
| Base logit | 0.333 0.000 | 0.217 0.000 | 0.282 0.000 | 0.290 0.000 | 0.355 0.000 | 0.164 0.000 |
| Verbalized | 0.182 0.001 | 0.222 0.007 | 0.193 0.002 | 0.247 0.004 | 0.367 0.003 | 0.123 0.029 |
| Platt | 0.190 0.000 | 0.210 0.000 | 0.202 0.000 | 0.290 0.000 | 0.351 0.000 | 0.173 0.000 |
| SuffiCE | 0.188 0.008 | 0.496 0.022 | 0.188 0.013 | 0.430 0.059 | 0.292 0.003 | 0.216 0.021 |
| Qwen3.8-27B | ||||||
| Base logit | 0.260 0.000 | 0.299 0.000 | 0.281 0.000 | 0.440 0.000 | 0.353 0.000 | 0.026 0.000 |
| Verbalized | 0.193 0.000 | 0.248 0.003 | 0.238 0.001 | 0.480 0.003 | 0.400 0.003 | 0.138 0.007 |
| Platt | 0.198 0.000 | 0.253 0.000 | 0.219 0.000 | 0.450 0.000 | 0.337 0.000 | 0.110 0.000 |
| SuffiCE | 0.180 0.006 | 0.497 0.007 | 0.218 0.009 | 0.634 0.023 | 0.240 0.010 | 0.257 0.025 |
| Qwen3.5-9B | ||||||
| Base logit | 0.217 0.000 | 0.137 0.000 | 0.259 0.000 | 0.398 0.000 | 0.383 0.000 | 0.190 0.000 |
| Verbalized | 0.251 0.003 | 0.235 0.016 | 0.256 0.001 | 0.381 0.004 | 0.343 0.002 | 0.193 0.004 |
| Platt | 0.202 0.000 | 0.204 0.000 | 0.234 0.000 | 0.433 0.000 | 0.307 0.000 | 0.195 0.000 |
| SuffiCE | 0.182 0.004 | 0.454 0.107 | 0.205 0.005 | 0.592 0.024 | 0.243 0.006 | 0.229 0.015 |
| Qwen3.5-4B | ||||||
| Base logit | 0.269 0.000 | 0.386 0.000 | 0.264 0.000 | 0.514 0.000 | 0.309 0.000 | 0.168 0.000 |
| Verbalized | 0.326 0.003 | 0.383 0.004 | 0.304 0.007 | 0.409 0.019 | 0.336 0.001 | 0.123 0.003 |
| Platt | 0.201 0.000 | 0.368 0.000 | 0.236 0.000 | 0.512 0.000 | 0.304 0.000 | 0.207 0.000 |
| SuffiCE | 0.178 0.014 | 0.487 0.034 | 0.222 0.010 | 0.530 0.036 | 0.258 0.004 | 0.230 0.011 |
Table 10. Evidence-state analysis for every answering model. MAE compares changes in confidence with changes in empirical success between states of the same problem, and correlates confidence with success among problems that share a state type. The best result for each model is in bold.
MAE (lower is better) tests whether confidence changes by the right amount as the evidence changes. For every pair of distinct states of the same problem , the same pairs as , it takes the absolute gap between the change in confidence and the change in empirical success, , and averages over all pairs in the test set. It is zero when confidence rises and falls by exactly as much as success. Where tests the direction of each change, MAE tests its size.
(higher is better) tests whether confidence separates more and less answerable problems that have the same kind of evidence. It groups the test set by state type, such as all HISTORY states (on MuSiQue, by question type and state), computes the Pearson correlation between confidence and empirical success within each group, and averages over groups. Every state in a group holds the same kind of evidence, so recognizing the state type alone does not raise it.
SuffiCE has the lowest MAE in 11 of the 12 combinations of model and dataset, so its confidence changes by about as much as success does, not only in the same direction. The one exception is Kimi K2.6 on MIMIC, where verbalized confidence is slightly lower. For Kimi K2.6, success on MIMIC changes by at most 0.1 for about three quarters of the pairs of states, so a confidence that barely moves has a low MAE, and verbalized confidence moves about half as much as SuffiCE. On , which gives no credit for staying still, SuffiCE is far better (Table 1). SuffiCE also has the highest for every model and dataset, with the largest margin on MIMIC, where whether the evidence suffices depends on the particular patient, so it does not merely learn an average confidence for each state type.
Cross-Domain Transfer
To test whether what SuffiCE learns is specific to one dataset, we train it on one clinical dataset and evaluate it on the other without further training (Table 11). MIMIC asks for one of four diseases and MedQA for a free-text diagnosis, and success is far more common on MIMIC.
| Metric | Base logit | Cross-domain | In-domain |
| Train MIMIC test MedQA | |||
| Brier | 0.268 0.000 | 0.189 0.027 | 0.107 0.005 |
| AUROC | 0.746 0.000 | 0.779 0.009 | 0.815 0.011 |
| 0.367 0.001 | 0.401 0.027 | 0.630 0.012 | |
| 0.398 0.000 | 0.491 0.029 | 0.592 0.024 | |
| Train MedQA test MIMIC | |||
| Brier | 0.124 0.000 | 0.167 0.004 | 0.098 0.010 |
| AUROC | 0.689 0.000 | 0.746 0.016 | 0.819 0.028 |
| 0.247 0.000 | 0.316 0.005 | 0.393 0.034 | |
| 0.137 0.000 | 0.321 0.038 | 0.454 0.107 |
Table 11. Cross-domain transfer between the clinical datasets (Qwen3.5-9B). Cross-domain and in-domain are SuffiCE trained on the other dataset and on the test dataset itself. The best result in each row is in bold and the second best in italics.
Which evidence matters transfers. In both directions, the transferred estimator ranks states better than the base logit in AUROC and . It also better separates more and less answerable cases that share a state type, recovering about half of the in-domain gain in (§C.4) over the base logit in both directions. What SuffiCE learns about which evidence settles a case thus carries over across datasets.
The overall success level does not transfer. The transferred confidence stays close to the success rate of the training dataset. Trained on MIMIC, where most states succeed, it remains too confident on MedQA, though less so than the base logit. Trained on MedQA, it is too cautious on MIMIC and is worse calibrated than the base logit. This error shifts the level of confidence, not the order of states, so ranking transfers while calibration does not.
Proofs
Proposition 1 says that the error of a confidence splits into two parts: how far off it is on average for each problem, and how far its changes between states of the same problem are from the changes in success. Calibration constrains only the first part. Corollary 1 shows that the training loss bounds both parts. We first define the notation shared by both results, then restate and prove Proposition 1 (§D.1) and Corollary 1 (§D.2).
Setup. Let there be problems, indexed by . Problem consists of a question , a pool of evidence , and a gold answer (§ 3.1). It has evidence states, indexed by . State pairs the question with a subset of the evidence, . Let be the total number of states. For a state , let be the success probability of the answering policy (Eq. (1)) and the confidence of the estimator.
Within problem , we abbreviate and , and define the error at state as . The problem averages of the confidence, the success probability, and the error are
For any , the change gap
is the difference between the change in confidence and the change in success probability from state to state . Finally, denotes the uniform average over all states, and the average of a per-problem quantity over problems, with problem weighted by , its share of all states. With this weight, averaging any function of a state over all states equals first averaging it within each problem and then applying ,
where is the average of over the states of problem . A plain average over problems, , agrees with only when all problems have the same number of states.
Proof of Proposition 1
Proposition 1 (restated). For any confidence ,
The left side is the mean squared error over all states. The first term on the right is the error across problems, in each problem’s average success. The second is the error within each problem, half the average squared change gap over all ordered pairs of its states.
Proof. We prove the identity for a single problem and then average over problems. All sums over and run from to .
Step 1: split the error into its average and its spread. Writing and expanding the square,
The middle term is zero. Since is the average of the errors, the errors above the average exactly cancel those below it,
Step 2: rewrite the spread with pairs of states. The spread is the last term of Step 1, , the average squared deviation of the errors from their mean, that is, their variance over the states of problem . It is zero exactly when every state of the problem has the same error. We now show that it equals half the average squared difference between the errors of two states. Let , so by Step 1. Expanding and summing over all and ,
Dividing by gives
Step 3: identify the change gap. Rearranging, . Combining the three steps, for every problem ,
where the last line substitutes and . The left side is the mean squared error of problem . On the right, the first term is the squared error of the problem’s average confidence, and the second is half the average squared change gap over its ordered pairs of states.
Step 4: average over problems. Steps 1–3 give the identity for one problem. The proposition averages over all states, so we group the states by problem and weight each problem by its share of states, using Eq. (8) with . Substituting the per-problem identity for each problem’s mean squared error and splitting the sum,
where the last line uses the definition of as the average over problems with weight . This is Eq. (4). □
Remarks. Calibration alone does not remove the second term. Suppose gives every state of a problem the same value, for example the problem’s average success . Then , so , and the second term becomes , which by the identity of Step 2, applied to in place of , equals the variance of over the states of problem . This value does not depend on which constant is chosen, so a confidence that ignores the evidence cannot reduce it, even if it is perfectly calibrated.
The Brier score measures the same error. The Brier score (§ 4) compares the confidence with the empirical success rate rather than with . Since averages independent grades, each equal to 1 with probability , it satisfies and . For a confidence computed from alone, the cross term vanishes, and
where the expectation on the left also averages over the sampled grades. The last term does not depend on the confidence, so differences in Brier score between methods reflect differences in the error that Proposition 1 decomposes.
Proof of Corollary 1
Corollary 1 (restated). For any confidence , let be the loss of Eq. (6) averaged over all evidence states and over the sampled answers that determine , and the minimum expected loss over all confidences, reached exactly at . Then
so each term of Proposition 1 is at most .
Proof. Recall the soft-label binary cross-entropy of Eq. (6), at each evidence state ,
Let be the binary entropy and the divergence between Bernoulli distributions with means and , with the convention . The confidence of SuffiCE is a soft-max and so lies strictly between 0 and 1, which keeps every logarithm finite. The loss is linear in , so taking the expectation over the sampled grades replaces by . At each state, writing and for and , the expected loss is therefore
To separate the part that depends on from the part that does not, we use the entropy of the target, . By this definition, , so adding these terms to the expected loss leaves it unchanged:
Grouping the two terms with the factor and the two with the factor ,
and writing each difference of logarithms as the logarithm of a ratio,
Averaging this identity over all evidence states gives
The first term depends only on the answering policy, not on the confidence, so no choice of can change it. The second term is an average of divergences, each nonnegative and zero only when . Every confidence therefore has , with equality exactly when at every state. Hence the minimum expected loss is , reached exactly at , and
Pinsker’s inequality for Bernoulli distributions states . Applying it at every state and averaging,
which is Eq. (7). By Proposition 1, the two terms are nonnegative and sum to , so each is at most . □
For SuffiCE, this means that lowering the training loss lowers an upper bound on both terms of Proposition 1, the error in each problem’s average success and the error in how confidence changes between states of the same problem. The bound holds only over the states the loss is averaged over, which is why SuffiCE trains on many evidence states of each problem rather than on full states alone (§ 5.2).
Comparison with Related Work
We compare our setting with the closest prior work on confidence estimation along five properties (Table 12).
Table 12. Comparison with the closest prior work along the five properties defined below. ✓ yes, – no.
Pre-answer: the confidence is produced before any answer is generated, so it can decide whether to answer or to gather more evidence.
Trained: the confidence estimator is trained, rather than read from an untrained model or elicited by prompting.
Controlled evidence subsets: the same problem is observed with different combinations of evidence, not only with evidence that accumulates in a fixed order.
Within-problem evaluation: the evaluation tests whether changes in confidence follow changes in success across states of the same problem.
Own token probability: the confidence is the model’s next-token probability, with no added classifier head or verbalized number.
(Kadavath et al., [15]) trains an added value head to predict, from the question alone, whether the model will answer correctly. It adds source documents and hints only at evaluation and does not relate the change in to the change in accuracy. Pre-generation probes (Lugoloobi et al., 2026) train linear probes on the model’s activations before generation to predict success, with each problem shown in a single fixed prompt. Calibration tuning (Kapoor et al., [16]) fine-tunes the model to judge whether a proposed answer is correct, read from its token probability. Rewarding Doubt (Bani-Harouni et al., [3]) uses reinforcement learning to train the model to state a confidence from 0 to 10 in an answer it has already generated, with each
question asked without context. LA-CDM (Bani-Harouni et al., [2]) trains a stated confidence in a diagnosis the agent has already proposed, and the evidence is limited to the tests its agent chose to request. (Zhang et al., 2026) prompts the model for the probability that the information so far entails its answer. Its clues arrive in a fixed order, and its evaluation measures per-turn calibration and whether confidence rises over turns.
Each prior method shares some of these properties, and ours is the only one that has all five. No prior method trains the model’s own token probability before an answer exists. (Kadavath et al., [15]) and pre-generation probes (Lugoloobi et al., 2026) estimate success before answering but read it from an added head or probe, while calibration tuning (Kapoor et al., [16]) and (Zhang et al., 2026) use the model’s token probability but score an answer it has already given. SuffiCE instead trains the model’s own probability of choosing to answer, so the same next-token probability that would decide whether to answer or gather more evidence is calibrated, with no extra head and no answer to generate first. We choose this formulation because the model’s next-token distribution already assigns a probability to answering versus seeking more evidence, so no classifier head is added and the confidence is the model’s own preference between the two actions. A fully-only estimator with the same formulation falls short of SuffiCE (§ 5.2), so the gains come from training across evidence states. Two further properties set SuffiCE apart, and no prior method has either. Training on controlled evidence subsets of the same problem lets the estimator learn which evidence decides success, rather than only which problems are hard (§ 5.2). Evaluating within each problem tests this directly, showing that confidence rises with the decisive evidence and not with every piece added (§ 5.3).
Case Studies
MuSiQue Case Studies
We examine two branching (3hop2) questions from the MuSiQue test set in detail (Tables 13 and 14). In both, the branches and answer two independent sub-questions, and only the final hop combines their answers into the requested date.
Case 1: When the Republicans took control of the Senate. The branches establish that Alaska supports the Republicans () and that the Senate sets its own rules (), but only gives the date on which the Republicans regained the Senate. With both branches shown, no sampled answer is correct. The base logit nonetheless jumps from at most 0.02 to 0.50, treating the two branches as progress despite the lack of sampled success. SuffiCE stays at or below 0.02, matching this low success rate. Once is shown, 20% of sampled answers are correct. SuffiCE rises to 0.19, closely matching this rate, while the base logit falls to 0.11 and becomes underconfident.
Case 2: When the Dutch reached New Zealand. The branches establish that The Lord of the Rings was filmed in New Zealand () and that the Dutch first brought enslaved people to New York (), but only gives the date on which the Dutch navigator Abel Tasman reached New Zealand. Again no sampled answer is correct with both branches, yet the base logit climbs with each branch to 0.98. SuffiCE reaches only 0.16. With full evidence, 30% of sampled answers are correct. SuffiCE gives 0.34, again close to this rate, while the base logit stays at 0.96 and is far overconfident.
Comparing the two full states. The two questions share the same structure and both show all three supporting paragraphs, yet they differ in how often they are answered correctly. SuffiCE follows this difference, matching each success rate within 0.04, whereas the base logit misses in opposite directions. The confidence of SuffiCE is not fixed by the evidence being complete, so it judges not only whether the reasoning chain is complete but also how likely the answering model is to succeed once it is.
Full Evidence States of the Case Studies
We give the complete inputs behind the clinical case study of § 5.3 and the first MuSiQue case study of §F.1 word for word, with the evidence states
When did the party Alaska generally supports take control of the determiner of rules of the US House and US Senate? Answer: January 2015
Alaska has voted Republican in nearly every presidential election since statehood. → Republicans
Each chamber of Congress sets its own rules (Article I), and the Senate adopts its Standing Rules. → the Senate
Republicans regained the Senate majority in the Congress that began in January 2015, having last held it in January 2007. → January 2015
| Evidence | |||||
| Success | 0.10 | 0.00 | 0.00 | 0.00 | 0.20 |
| Base | 0.02 | 0.02 | 0.00 | 0.50 | 0.11 |
| SuffiCE | 0.01 | 0.01 | 0.01 | 0.02 | 0.19 |
tbl:13: Case study on a 3hop2 MuSiQue question (Qwen3.5-9B), where the branches and together lead to the final hop . Columns are evidence states, marked by which supporting facts are shown ( for none).
When did the people who first imported slaves to New York come to the country where The Lord of the Rings was filmed? Answer: 13 December 1642
The Lord of the Rings trilogy was filmed entirely in New Zealand. → New Zealand
Enslaved Africans were first brought into New York by the Dutch, and later the English. → the Dutch
Dutch navigator Abel Tasman was the first European to sight New Zealand, on 13 December 1642. → 13 December 1642
| Evidence | |||||
| Success | 0.00 | 0.00 | 0.00 | 0.00 | 0.30 |
| Base | 0.12 | 0.15 | 0.65 | 0.98 | 0.96 |
| SuffiCE | 0.02 | 0.03 | 0.06 | 0.16 | 0.34 |
tbl:14: A second case study on a 3hop2 MuSiQue question (Qwen3.5-9B), in the format of tbl:13. The shaded state has both branches but not the final hop.
of each example and the evidence they contain (Table 15 and Table 16). The state names and evidence labels, such as or , are defined only for analysis and are never shown to the model. The model sees only the question and the evidence as it appears in the input, with the section headings of the clinical case but no indication of which state it is in.
| Objective | Assess and diagnose the infant brought in by a concerned mother due to bilious vomiting and inability to pass meconium, with observed abdominal distension. Answer: Hirschsprung’s disease |
| 1 HISTORY | Demographics: 2-day-old male infant. History: The infant was born via a home birth without complications. The mother reports that the infant has had bilious vomiting and has not been able to pass meconium since birth. She also notes that the infant’s abdomen appears distended. Primary symptom: Bilious vomiting. Secondary symptoms: Inability to pass meconium, Abdominal distension. Past medical history: Newborn, no prior medical history. Birth was uneventful. Social history: The infant is the first child of the family. No known family history of genetic or congenital diseases. Review of systems: The infant shows signs of distress, possibly due to abdominal discomfort. |
| 2 PE | Vital signs: Temperature 37.5°C (99.5°F), Blood pressure not applicable for age, Heart rate 145 bpm, Respiratory rate 40 breaths/min. Abdominal examination: Inspection: Noticeable distension. Auscultation: Decreased bowel sounds. Percussion: Tympanic sound noted, indicating gas. Palpation: The abdomen is firm and distended, with palpable bowel loops; the infant shows signs of discomfort during palpation. |
| 3 BIOPSY | Rectal biopsy, histopathology: Absence of ganglion cells in the submucosa and myenteric plexus, consistent with Hirschsprung’s disease. |
| 4 IMAGING | Abdominal X-ray, findings: Dilated loops of bowel with no gas in the rectum, suggestive of a distal obstruction. |
| State | Sections shown |
| {1} | HISTORY |
| {1, 2} | HISTORY, PE |
| {1, 4} | HISTORY, IMAGING |
| {1, 3, 4} | HISTORY, BIOPSY, IMAGING |
| {1, 2, 3, 4} | HISTORY, PE, BIOPSY, IMAGING |
Table 15. Full input of the MedQA case of Table 3 and the evidence states shown there. Each state shows the objective followed by the listed sections, verbatim.
Question When did the party Alaska generally supports take control of the determiner of rules of the US House and US Senate?
Answer: January 2015
Alaska. Alaska regularly supports Republicans in presidential elections and has done so since statehood. Republicans have won the state’s electoral college votes in all but one election that it has participated in (1964). No state has voted for a Democratic presidential candidate fewer times. Alaska was carried by Democratic nominee Lyndon B. Johnson during his landslide election in 1964, while the 1960 and 1968 elections were close. Since 1972, however, Republicans have carried the state by large margins. In 2008, Republican John McCain defeated Democrat Barack Obama in Alaska, 59.49% to 37.83%. McCain’s running mate was Sarah Palin, the state’s governor and the first Alaskan on a major party ticket. Obama lost Alaska again in 2012, but he captured 40% of the state’s vote in that election, making him the first Democrat to do so since 1968.
Standing Rules of the United States Senate. The Standing Rules of the Senate are the parliamentary procedures adopted by the United States Senate that govern its procedure. The Senate’s power to establish rules derives from Article One, Section 5 of the United States Constitution: “Each House may determine the rules of its proceedings. . . ”
2014 United States Senate elections. The Republicans regained the majority of the Senate in the 114th Congress, which started in January 2015; the Republicans had not controlled the Senate since January 2007. They had needed a net gain of at least six seats to obtain a majority. They held all of their seats, and gained nine Democratic - held seats. Republicans defeated five Democratic incumbents:
Democratic Party of Oregon. The Democratic Party of Oregon, based in Portland, is the official Oregon affiliate of the United States Democratic Party. It is recognized by the state of Oregon as a major political party, along with the Oregon Republican Party. The State Central Committee, made up of two delegates elected from each of Oregon’s 36 counties and one additional delegate for every 15,000 registered Democrats, is the main authoritative body of the party. After Oregon was admitted into the Union in 1859, the Democratic party controlled the state. Oregon elected twice as many Democrats as Republicans between 1859 and 1879 in statewide elections for governor, secretary of state, state treasurer, and congressmen. The party holds 38 members in the State House that has 60 representatives total, and 18 members in the State Senate, out of 30 delegates total. The party also holds the Governor’s office (Kate Brown), Attorney General (Ellen Rosenblum), Labor Commissioner (Val Hoyle), and State Treasurer (Tobias Read). The Democrats also have both U.S senate positions in their state and send four of the five U.S House representatives from Oregon to D.C.
Tammany Hall. Tammany Hall, also known as the Society of St. Tammany, the Sons of St. Tammany, or the Columbian Order, was a New York City political organization founded in 1786 and incorporated on May 12, 1789, as the Tammany Society. It was the Democratic Party political machine that played a major role in controlling New York City and New York State politics and helping immigrants, most notably the Irish, rise up in American politics from the 1790s to the 1960s. It typically controlled Democratic Party nominations and political patronage in Manhattan from the mayoral victory of Fernando Wood in 1854 and used its patronage resources to build a loyal, well - rewarded core of district and precinct leaders; after 1850 the great majority were Irish Catholics.
Political party. At the same time, the political party reached its modern form, with a membership disciplined through the use of a party whip and the implementation of efficient structures of control. The Home Rule League Party, campaigning for Home Rule for Ireland in the British Parliament was fundamentally changed by the great Irish political leader Charles Stewart Parnell in the 1880s. In 1882, he changed his party’s name to the Irish Parliamentary Party and created a well-organized grass roots structure, introducing membership to replace “ad hoc” informal groupings. He created a new selection procedure to ensure the professional selection of party candidates committed to taking their seats, and in 1884 he imposed a firm ‘party pledge’ which obliged MPs to vote as a bloc in parliament on all occasions. The creation of a strict party whip and a formal party structure was unique at the time. His party’s efficient structure and control contrasted with the loose rules and flexible informality found in the main British parties; – they soon came to model themselves on the Parnellite model.
Ethical Considerations
Our clinical experiments use MIMIC-IV-Ext Clinical Decision Making [9], a deidentified dataset accessed through PhysioNet under its data use agreement, and we do not release its data. SuffiCE is a research method and is not intended for clinical use. We used Claude and ChatGPT as AI assistants in preparing this manuscript, in accordance with the ACL policy on AI writing assistance.
References
- [1]Akari Asai, Zeqiu Wu, Yizhong Wang, Avrup Sil, and Hannah Hajishirzi. 2024. Self-ref: Learning to retrieve, generate, and critique through self-reflection. In ICLR.arxiv.org/abs/2310.11511
- [2]David Bani-Harouni, Chantal Pellegrini, Ege Özsoy, Nassir Navab, and Matthias Keicher. 2026a. Language agents for hypothesis-driven clinical decision making with reinforcement learning. In ICLR. proceedings.iclr.cc.
- [3]David Bani-Harouni, Chantal Pellegrini, Paul Stangel, Ege Özsoy, Kamilia Zaripova, Nassir Navab, and Matthias Keicher. 2026b. Rewarding doubt: A reinforcement learning approach to calibrated confidence expression of large language models. In ICLR. proceedings.iclr.cc.
- [4]Mehul Damani, Isha Puri, Stewart Slocum, Idan Shenfeld, Leshem Choshen, Yoon Kim, and Jacob Andreas. 2026. Beyond binary rewards: Training lms to reason about their uncertainty. In ICLR. proceedings.iclr.cc.arxiv.org/abs/2507.16806
- [5]Xilin Dang, Kexin Chen, Xiaorui Su, Ayush Noori, Iñaki Arango, Lucas Vittor, Xinyi Long, Yuyang Du, Marinka Zitnik, and Pheng-Ann Heng. 2026. Knowguard: Knowledge-driven abstention for multi-round clinical reasoning. In ICLR. proceedings.iclr.cc.arxiv.org/abs/2509.24816
- [6]Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. 2024. A survey of confidence estimation and calibration in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6577–6595, Mexico City, Mexico. Association for Computational Linguistics.arxiv.org/abs/2311.08298
- [7]Tilmann Gneiting and Adrian E. Raftery. 2007. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359–378.
- [8]Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On calibration of modern neural networks. In ICML, volume 70 of Proceedings of Machine Learning Research, pages 1321–1330. PMLR.arxiv.org/abs/1706.04599
- [9]Paul Hager, Friederike Jungmann, Robbie Holland, Kunal Bhagat, Inga Hubrecht, Manuel Knauer, Jakob Vielhauer, Marcus Makowski, Rickmer Braren, Georgios Kaissis, and Daniel Rueckert. 2024. Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nature Medicine, 30(9):2613–2622.DOI
- [10]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuhuai Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.arxiv.org/abs/2106.09685
- [11]Akriti Jain and Aparna Garimella. 2026. Knowing what’s missing: Assessing information sufficiency in question answering. In Findings of the Association for Computational Linguistics: EACL 2026, pages 4163–4174, Rabat, Morocco. Association for Computational Linguistics.
- [12]Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Active retrieval augmented generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7969–7992, Singapore. Association for Computational Linguistics.
- [13]Di Jin, Eileen Pan, Nassim Oufattle, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421.arxiv.org/abs/2009.13081
- [14]Hailey Joren, Jianyi Zhang, Chun-Sung Ferng, Da-Cheng Juan, Ankur Taly, and Cyrus Rashtchian. 2025. Sufficient context: A new lens on retrieval augmented generation systems. In ICLR.arxiv.org/abs/2411.06037
- [15]Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, and 17 others. 2022. Language models (mostly) know what they know. Preprint, arXiv:2207.05221.arxiv.org/abs/2207.05221
- [16]Sanyam Kapoor, Nate Gruver, Manley Roberts, Katherine Collins, Arka Pal, Umang Bhatt, Adrian Weller, Samuel Dooley, Micah Goldblum, and Andrew Gordon Wilson. 2024. Large language models must be taught to know what they don’t know. In NeurIPS.arxiv.org/abs/2406.08391
- [17]Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations.arxiv.org/abs/1412.6980
- [18]Polina Kirichenko, Mark Ibrahim, Kamalika Chaudhuri, and Samuel J. Bell. 2025. Abstentionbench: Reasoning llms fail on unanswerable questions. In NeurIPS.arxiv.org/abs/2506.09038






