WorksComputer Science
Tell Me How Sure You Are: Communicating Uncertainty in Auxiliary Evidence to Foundation Models
Abstract
In many queries, the context information provided to generative foundation models, such as LLMs and VLMs, is inherently uncertain, including its representations of the physical world. While conventional prompting discards the associated uncertainty, this paper, for the first time in the literature, systematically demonstrates that communicating such uncertainty in context provides useful additional cues for downstream decisions and yields improvements across a wide range of settings. We formulate the decision-making problem as a zero-shot foundation model-assisted image classification task supported by auxiliary text-based characteristics. We choose this due to the availability of well-defined auxiliary properties, associated uncertainty, and unambiguous task correctness. Using five downstream foundation models across three datasets, we extensively evaluate uncertainty arising from both humans and learned models when observing or predicting auxiliary evidence. Across all settings, we find that communicating uncertainty substantially improves downstream LLM classification performance: macro-F1 improves up to percentage points compared to when the uncertainty information is not communicated. We further observe how the foundation models process such uncertainty information and find that auxiliary uncertainty contains predictive information about the correctness of downstream decisions. Together, these results establish uncertainty as useful contextual information and motivate more systematic consideration of uncertainty communication in future foundation-model inference pipelines.
Introduction
Generative foundation models are useful as general-purpose decision layers to combine heterogeneous information at inference time, especially in zero-shot settings. Examples include a medical AI assistant aggregating information from clinical records, medical images, and outputs from specialised predictive models processing clinical test results [22, 26]; or an in-car video agent integrating outputs from specialised object detection models [40]. In such settings, the advantage of a foundation model is its ability to compose different forms of evidence dynamically, without requiring a new task-specific model for every combination of available information. Nevertheless, these pieces of auxiliary evidence are not necessarily equally reliable. They may originate from human observations, retrieval systems, sensors, classifiers, or other predictive models, and may therefore be incomplete, ambiguous, or incorrect. Moreover, many such sources naturally expose confidence scores or other native estimates of uncertainty, raising a basic question: should a downstream foundation model receive only the evidence, or also how certain its source was about that evidence?
Initial studies suggest that communicating or exploiting uncertainty of the auxiliary evidence can benefit foundation-model inference. For example, prior work has examined how LLMs respond to uncertain external knowledge [9, 12, 27], while SRICE [39] and VideoAgent2 [40] incorporate tool confidence into multimodal agentic pipelines. These results provide promising evidence that uncertainty can matter. However, they do not systematically isolate the effect of communicating uncertainty itself: the underlying evidence is not held fixed across conditions, uncertainty may influence subsequent retrieval or routing, and the effect is not examined across different sources of uncalibrated uncertainty. Here we fill all these gaps. We not only require the auxiliary evidence to be accompanied with explicit uncertainty or confidence signal supplied by its source, but also the following: Downstream task correctness can be measured objectively, and the same auxiliary evidence can be communicated to the foundation model either with or without its additional reliability information. This enables a controlled paired comparison in which the evidence itself is fixed and only the communicated uncertainty changes.
We therefore study three benchmarks that satisfy these requirements while covering distinct evidence-generation processes. In CUB [31], human annotators provide visual characteristics of bird images together with ordinal observational certainty labels; in SUN [29], a general-purpose vision model provides image–attribute matching scores; and in Derm7pt [15], specially trained task-specific classifiers provide dermatological findings together with class probability based confidence. We intentionally retain these signals in the form provided by each source rather than forcing them into a common universal numerical representation, since our aim is not to compare their scales directly but to ask whether communicating the reliability information available at the source helps the downstream model. Using both LLMs and VLMs, we examine their effects on downstream classification performance, how those effects change with the amount and composition of auxiliary evidence, and whether upstream uncertainty is informative about downstream zero-shot decision correctness and the language model’s own uncertainty.
Our results reveal four main patterns. First, explicitly communicating uncertainty substantially improves downstream correctness. Across the LLMs, the largest gains reach 7.84 percentage points (pp) Macro F1 on CUB, 26.18 pp on SUN, and 9.10 pp on Derm7pt, with smaller but measurable gains also observed for VLMs. Second, we show that uncertainty helps to sanitize a mixed group of certain and uncertain evidence. Third,
we observe that the foundation model doesn’t merely consider the ordering of evidence in the prompt but inspects the uncertainty values to weight the evidence correctly. And finally, upstream uncertainty can predict downstream correctness, but does not correlate with the downstream model’s own predictive uncertainty: communicating uncertainty can correct a decision without reducing downstream entropy.
Together, these results show that uncertainty is useful contextual information, but its effect depends on the evidence it accompanies and how the downstream model processes that evidence. Our contributions are:
We introduce a paired zero-shot evaluation framework that isolates the effect of communicating uncertainty while holding the downstream model, primary context, and auxiliary evidence fixed. Across settings the gain in Macro F1 improvement is substantial.
We demonstrate this effect across three distinct evidence-generation processes – human judgments, a general-purpose vision model, and specialized classifiers
We show that the benefit depends on if the in-context uncertainty is associated with the evidence: on SUN, correctly aligned uncertainty improves macro-F1 by 11.71–26.60 pp relative to omitting uncertainty and by 14.02–19.53 pp relative to misaligned uncertainty. We further show that upstream uncertainty is informative about downstream correctness, yet not directly reflected in the downstream model’s own predictive uncertainty.
Literature Survey
We briefly review prior work on uncertain auxiliary evidence, intermediate representations, and upstream–downstream uncertainty; an expanded discussion appears in Appendix A.
Prior work has studied uncertain external information through probabilistic reasoning [27], knowledge revision [9], calibrated routing [39], and agentic retrieval [40, 12] (Wu et al. 2025). Foundation models have been used previously to build interpretable image classifiers through concept-based methods using attributes, probabilistic concepts, uncertainty-guided interventions, and descriptor-based zero-shot recognition [17, 16, 28, 25, 8]. Foundation-model uncertainty is commonly estimated through self-evaluation, entropy, verbalized confidence, calibration, or tool-level uncertainty [13, 18, 35, 4, 37, 23 b].
Gaps and novelty. Table 1 shows that prior work addresses complementary parts of this problem. In summary, no prior work isolates the marginal effect of uncertainty communication by holding the auxiliary evidence fixed and examining its impact on final predictions and correctness. None considered downstream predictive uncertainty across heterogeneous sources of uncertainty. To the best of our knowledge, this is the first study to provide this joint evaluation across evidence sources, domains, and foundation models, as summarized in Figure 1.

Figure 1. Overview of our evaluation framework. Upstream sources provide auxiliary evidence and associated uncertainty. We compare paired zero-shot contexts while holding the auxiliary evidence fixed and varying whether its uncertainty is communicated, and study the resulting effects on downstream prediction, evidence use, and decision correctness.
| Method | Communicates Uncertainty | Holds Evidence Fixed | Measures Effect of Uncertainty Communication | Analyzes Downstream Correctness | Analyzes Downstream Uncertainty | Studies Distinct Uncertainty Source Types |
| SRICE (2025a) | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ |
| VideoAgent2 (2025b) | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ |
| Tools in the Loop (2025a; 2025b) | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ |
| ProbCBM (2023) | ✗ | ✗ | ✗ | ✓ | ✓ | ✗ |
| Our study | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
Table 1. Comparison with relevant prior works. A check denotes that the corresponding criterion is explicitly evaluated. “Evidence held fixed” means that auxiliary evidence is unchanged and only the communicated uncertainty varies. “Distinct uncertainty source types” refers to qualitatively different signals, such as human-reported certainty, model matching scores, and classifier confidence.
Experimental strategy
Problem formulation. Let a classification dataset, contain samples of the form where, is an image and is the classification category. Let us call this classification task the primary task, . Auxiliary variables, capture a set of categorical characteristics, of the image . An auxiliary attribute , denotes its observed value for sample , where is the set of possible values for that attribute. The observed value , therefore, constitutes auxiliary evidence that can provide a cue for predicting the primary-task label . For a bird classification task, the color of the wings is an auxiliary variable, and its value for a particular sample bird will be the evidence. Auxiliary tasks identify the evidence of the auxiliary variables.
We assume that both the image, , and the auxiliary variables independently have enough information to build a reasonably accurate classifier for . This follows from the literature that used such evidence to build interpretable classifiers [17].
While one can train an end-to-end image classifier or an auxiliary classifier that predicts from the auxiliary evidence , our goal in this study is not to build the most accurate end-to-end classifier. Instead, we use a generative foundation model, , to predict the label from context information that includes the auxiliary evidence, in a zero-shot setting.
Recording auxiliary evidence. Depending on the applications, there can be two types of auxiliary evidence: ) annotation by a human, and ) predicted by a learned model. Now, for the latter type, one can predict the auxiliary evidence either by training and inferring from a task-specific model; or inferring from a pre-trained general-purpose vision language model. The choice often depends on whether there is enough data to train an auxiliary evidence predictor or whether the auxiliary task is sufficiently general and suitable for a general-purpose pre-trained model. In this paper, we select datasets that cover all these cases.
Constructing the context. Irrespective of the mechanism for recording auxiliary evidence, we construct the prompt with three sections. The first section specifies the context of the primary task (), while the second section defines the auxiliary variables () and their relations to . The final section specifies the auxiliary evidence. Thus, let us represent the context, as a function, . Examples and more details on context construction are in Appendix D.
While it is well known that optimizing prompts can improve performance, we do not do so in this paper for two reasons. First, when comparing performance across multiple foundation models, optimizing prompts is an open problem itself [41]. Second, results on a variety of prompt configurations (Appendix D.2) indicate that the observed benefit is generally robust to prompt wording, although its magnitude varies across prompt formulations.
Uncertainty of auxiliary variables. In the context of the applications considered, auxiliary variables are observed with uncertainty , i.e., each evidence, , is associated with an uncertainty value, . Throughout, we use uncertainty signal as an umbrella term for source-provided reliability information; depending on the source, this may be expressed as uncertainty, certainty, or confidence.
The type of uncertainty directly depends on how the auxiliary evidence is recorded in an application. When a human annotates an auxiliary variable , the uncertainty originates from either a lack of precision in defining the variable or visual noise in the sample. Note that the uncertainty signals obtained from a task-specific model and a general-purpose foundation model can have different semantics and scales, and we do not assume that they are calibrated or directly comparable. There are various ways to measure uncertainty in task-specific models, such as the entropy of the class probabilities or the probability of the most likely class [14]. While Table 16 evaluates several uncertainty representations, including entropy, maximum softmax confidence, and their combination, across multiple prompt configurations for the task-specific model, in the rest of the experiments, we use probability of the most likely auxiliary class and BLIP image–text matching scores as the uncertainty signals for the task-specific model and general-purpose foundation model, respectively.
We assume that all uncertainty values within a context share the same representation and interpretation; mixing heterogeneous uncertainty representations within a single context is outside the scope of this work. To communicate uncertainty to the foundation model, we keep the same prompt structure as above and append the uncertainty information in the final section along with the auxiliary evidence. Thus the context, becomes .
Datasets
We use three datasets exhaustively covering the scenarios discussed above.
CUB. The primary task in CUB [31] is to classify bird species from images. While the dataset contains 200 species and 312 human-annotated visual attributes, to fit the context length we consider candidate-species configurations containing 7, 8, and 9 classes; the class-selection procedure and additional details are provided in Appendix B.2.1. For each sample , contains only the attributes annotated as present. Each retained evidence is accompanied by the ordinal human-certainty signal . These labels represent the annotator’s certainty.
SUN. The SUN dataset [29] asks for scene classification and provides a list of visual and contextual scene attributes. We consider four scene categories encompassing common places in a city life – park, office cubicles, supermarket, and coffee shop – with 20 images per class, and construct a pool of 20 auxiliary variables (Appendix B.2.2 has the details). For each sample , we use the image–text matching head of the pre-trained general-purpose vision–language model BLIP (Li et al., 2022) to compute an image–text matching score for each attribute. We rank the 20 attributes by their BLIP scores and use the top attribute names as the image-specific auxiliary evidence . Each retained evidence is associated with the corresponding BLIP image–text matching score , which we use as its uncertainty.
Derm7pt. The Derm7pt dataset [15] asks for skin-lesion classification (primary task, ) and provides seven dermatological findings (auxiliary variables) from the seven-point checklist. We formulate as a binary classification problem, treating melanoma and basal cell carcinoma as malignant and the remaining diagnoses as non-malignant. For each auxiliary task, we independently train a task-specific ResNet-18 classifier (He et al., 2016) and map its prediction to a binary state, not suspicious or suspicious (Appendix B.2.3). For each sample , these predicted findings constitute the auxiliary evidence , where each is accompanied by the maximum-softmax confidence from the upstream classifier.
Generative Foundation Models
We evaluate five general-purpose generative foundation models spanning text-only and multimodal settings. The LLMs are Llama-3.1-8B-Instruct [10] (8B), Phi-4-14B [1] (14B), and Qwen3-32B [38] (32B), covering distinct model families and parameter scales. For VLMs, we use Qwen2-VL-7B-Instruct [32] and InternVL3-8B [5, 33, 6 c], two similarly sized
(7–8B) models from different families. All models are general-purpose and are evaluated zero-shot without fine-tuning. The LLMs receive only the textual context, while the VLMs additionally receive the image, . Further details are provided in Appendix § 11.§ 3.
Evaluation protocol.
Our primary metric is macro-F1 for the primary task, and we quantify the effect of uncertainty as the paired change from Aux (using context of the form ) to Aux+Unc (using context of the form ) in percentage points. All paired comparisons use identical samples and auxiliary evidence, differing only in whether is communicated. Details of repeated runs and complete results for the remaining evaluated configurations are provided in Appendix § 11.§ 1 and Appendix § 12; the main text reports a balanced view across all three benchmarks.
Communicating Uncertainty Improves Downstream Prediction
Here, we test our core hypothesis: whether communicating the uncertainty associated with auxiliary evidence improves the downstream generative foundation model’s final decision. This is important because auxiliary evidence is rarely equally reliable: a human annotator may be unsure about an observed attribute, a vision-language model may weakly match a scene attribute, or a task-specific classifier may assign low confidence to a predicted finding.
Communicating uncertainty consistently improves LLM performance (Table 2). Across all three benchmarks, communicating improves downstream LLM performance while keeping the auxiliary evidence fixed. On CUB, every reported model–configuration pair improves, with gains of up to 7.84 percentage points. For SUN, all three LLMs improve, with gains ranging from 5.93 to 26.18 points; on Derm7pt, the corresponding gains range from 0.25 to 9.10 points. Thus, the core hypothesis holds across human-reported certainty, model scores, and task-specific classifier confidence.
| Dataset | Model | Aux | Aux + Uncertainty | (pp) |
| CUB (7-class) | Qwen3-32B | 0.4682 0.0165 | 0.5424 0.0131 | +7.42 0.76 |
| CUB (7-class) | Llama-3.1-8B | 0.5770 0.0095 | 0.6368 0.0083 | +5.98 0.94 |
| CUB (7-class) | Phi-4-14B | 0.6924 0.0174 | 0.7385 0.0194 | +4.61 0.84 |
| CUB (8-class) | Qwen3-32B | 0.3602 0.0039 | 0.4387 0.0156 | +7.84 1.72 |
| CUB (8-class) | Llama-3.1-8B | 0.5260 0.0041 | 0.5753 0.0109 | +4.93 1.10 |
| CUB (8-class) | Phi-4-14B | 0.5827 0.0154 | 0.6083 0.0051 | +2.56 1.06 |
| CUB (9-class) | Qwen3-32B | 0.3351 0.0076 | 0.3805 0.0130 | +4.53 0.61 |
| CUB (9-class) | Llama-3.1-8B | 0.4920 0.0070 | 0.5115 0.0067 | +1.95 0.22 |
| CUB (9-class) | Phi-4-14B | 0.5638 0.0102 | 0.6073 0.0249 | +4.36 1.50 |
| SUN () | Qwen3-32B | 0.6066 | 0.8684 | +26.18 |
| SUN () | Llama-3.1-8B | 0.8648 | 0.9241 | +5.93 |
| SUN () | Phi-4-14B | 0.8175 | 0.8843 | +6.69 |
| Derm7pt () | Qwen3-32B | 0.505 | 0.596 | +9.10 |
| Derm7pt () | Llama-3.1-8B | 0.601 | 0.630 | +2.90 |
| Derm7pt () | Phi-4-14B | 0.544 | 0.547 | +0.25 |
Table 2. Macro-F1 across CUB, SUN, and Derm7pt. denotes the Aux+Unc gain over Aux in percentage points. All experiments use three seeds (42, 96, 1001); CUB reports mean standard deviation across seed-specific sampled evaluation sets, while SUN and Derm7pt use fixed evaluation sets and omit 0.0 when deterministic inference yields identical results.
The effect extends to VLMs, although the gains are weaker (Table 3). On CUB, communicating human-reported certainty improves macro-F1 in five of the six VLM settings, with gains reaching 2.07 percentage points; the only decrease is negligible (0.06 points). Thus, uncertainty remains useful even when the downstream model can directly inspect the image, although the effect is smaller than for the text-only LLMs. On SUN, both VLMs already achieve macro-F1 of 1.0, leaving no headroom to measure an additional benefit. The Derm7pt VLM results show similar modest effects and are reported in Appendix § 12.1.3.
| Model | Classes | Image-only | Image+Aux | Image+Aux+Unc | |
| InternVL3-8B | 7 | 0.7873 0.0025 | 0.7963 0.0027 | 0.7993 0.0036 | +0.30 |
| Qwen2-VL-7B-Instruct | 7 | 0.8022 0.0036 | 0.8189 0.0041 | 0.8183 0.0042 | −0.06 |
| InternVL3-8B | 8 | 0.7154 0.0019 | 0.7288 0.0017 | 0.7313 0.0055 | +0.25 |
| Qwen2-VL-7B-Instruct | 8 | 0.7782 0.0046 | 0.7735 0.0056 | 0.7942 0.0067 | +2.07 |
| InternVL3-8B | 9 | 0.7370 0.0069 | 0.7632 0.0056 | 0.7682 0.0062 | +0.50 |
| Qwen2-VL-7B-Instruct | 9 | 0.6862 0.0011 | 0.7048 0.0060 | 0.7124 0.0093 | +0.76 |
Table 3. CUB VLM macro-F1 across 7-, 8-, and 9-class settings. denotes the Image+Aux+Unc gain over Image+Aux in percentage points.
Selecting Auxiliary variables
Here, we consider the case where a subset of auxiliary variables () is given in the context and show that the LLM’s accuracy depends on the selection of . This is important for two reasons. Firstly, where the auxiliary evidence collection is costly, for example, in the Derm7pt dataset, auxiliary evidence corresponds to a clinical test. Secondly, even when auxiliary evidence is available, the context window length may be restrictive. For example, the SUN dataset contains 102 auxiliary variables that extend the context beyond the models’ context length.
Uncertainty helps when includes uncertain evidences (Table 4). For each sample, in SUN dataset we rank the auxiliary variables, according to their BLIP scores. Then we select top ranked variables in to include in the context ( and ).
| Model | Aux | Aux+Unc | Aux | Aux+Unc | Aux | Aux+Unc |
| Qwen3-32B | 6 | 2 | 7 | 2 | 7 | 2 |
| Llama-3.1-8B | 5 | 1 | 5 | 0 | 5 | 2 |
| Phi-4-14B | 2 | 0 | 4 | 2 | 6 | 3 |
| Total | 13 | 3 | 16 | 6 | 18 | 7 |
Table 4. Number of predictions that are correct at the smaller top- setting but become incorrect after lower-ranked auxiliary attributes are added on SUN. Communicating the corresponding BLIP scores substantially reduces these harmful transitions.
Table 4 reports the number of samples (out of 80 test samples in this dataset) whose prediction flips from correct to incorrect when context includes more auxiliary evidence. We observe that such flips increase with increasing , making it obvious that unrelated auxiliary variables (with higher ) hurt. However, the more interesting fact here is that the number of flips in all cases reduces when we include the uncertainty of the auxiliary evidence, . Further analysis of how evidence behaves depending on its correlation across classes is presented in Appendix § 12.§ 2.
Context should contain important auxiliary information (Figure 2). Here, we ask whether the gain from communicating uncertainty changes as more auxiliary variables are included in the context. For Derm7pt, the gain generally increases with the number of auxiliary variables for the LLMs. This is consistent with only a subset of auxiliary variables being strongly informative for the primary task , while additional variables may be less relevant or less reliable. Communicating their uncertainty can therefore help prevent such evidence from disproportionately affecting the prediction for . The VLM results are less uniform, suggesting that this effect also depends on the downstream model.

Figure 2. Effect of uncertainty communication across Derm7pt evidence compositions for LLMs. Boxes show the distribution of macro-F1 (pp) across subsets of size ; black lines and diamonds denote the median and mean, respectively.
For completeness, Appendix § 12.§ 1 reports the extended SUN and Derm7pt results across all evaluated auxiliary-variable subsets.
How Uncertainties are Processed?
Here, we examine how the downstream model uses uncertainty to weight auxiliary variables in a zero-shot way, without task-specific training. We show this in two ways. First, we observe that processes the uncertainty values, not merely looking at the order of the variables in the prompt. Second, performance is affected substantially by whether important auxiliary variables are present in the context. This is important because not all auxiliary variables contribute equally to the primary task.
Table 5: SUN uncertainty-alignment control at . The same ten BLIP-selected attributes are presented in a score-independent random order under all conditions. No-U omits BLIP scores; True-U attaches each attribute’s correct score; and Shuffled-U preserves the same score multiset but reassigns scores to different attributes. Values are macro-F1 (mean standard deviation across three random-order seeds); is reported in percentage points.
| Model | No-Uncertainty | True-Uncertainty | Shuffled-Uncertainty | (pp) | (pp) |
| Qwen3-32B | 0.3902 0.0638 | 0.6562 0.0414 | 0.4609 0.0105 | +26.60 3.12 | +19.53 3.12 |
| Llama-3.1-8B-Instruct | 0.5630 0.0270 | 0.6800 0.0399 | 0.5398 0.0386 | +11.71 5.13 | +14.02 6.08 |
| Phi-4-14B | 0.3846 0.0304 | 0.6439 0.0760 | 0.4694 0.0274 | +25.93 8.36 | +17.44 5.03 |
Table 5. SUN uncertainty-alignment control at .

Figure 3. Supervised feature relevance versus the best-performing Aux+Unc evidence subsets on Derm7pt. Bars show Random-Forest permutation importance [3]; filled cells indicate findings selected in each LLM’s best subset.
Table 6: Downstream correctness prediction across CUB (from human-reported attribute certainty) and SUN (from four BLIP-derived reliability features, , using a stratified five-fold logistic-regression probe). Values report pooled out-of-fold AUROC.
| Dataset | Model | AUROC |
| CUB | Phi-4-14B | 0.558 0.056 |
| CUB | Qwen2-VL-7B-Instruct | 0.472 0.084 |
| SUN | Qwen3-32B | 0.8002 |
| SUN | Phi-4-14B | 0.8150 |
Table 6. Downstream correctness prediction across CUB and SUN.
Uncertainty is useful only when correctly associated with the evidence (Table 5). In Table 5, we show that the gain from communicating uncertainty depends on its correct alignment with the auxiliary evidence. Across all three LLMs, correctly aligned uncertainty performs better than both omitting uncertainty and reassigning the same uncertainty values across attributes. To test this, we first randomly order the same selected auxiliary attributes and then compare contexts with no uncertainty, with the correct uncertainty associated with each evidence item , and with a perturbed association in which is paired with .1 We use SUN for this experiment because its auxiliary attributes can be ranked by their BLIP-derived matching scores, with an uncertainty value available for each attribute. This allows us to perturb the attribute ordering and the uncertainty–attribute association while keeping the selected evidence fixed.
Useful evidence compositions partially align with supervised feature relevance (Figure 3). For Derm7pt, we compare the best-performing evidence subset for each LLM with an independent Random-Forest feature ranking computed from the training data using permutation importance [3]. The highest-ranked supervised feature, dots and globules, appears in the best subset for all three LLMs, while other highly ranked findings also recur across models. However, the agreement is only partial: some best-performing subsets also contain findings with low supervised importance. Thus, the downstream models do not simply recover the supervised feature ranking; their preferred evidence compositions show signs of implicit zero-shot feature prioritization, in which useful auxiliary variables can be identified without task-specific training of the downstream model.
Uncertainty Correlates with Downstream Decisions
Here, we consider whether the uncertainty associated with the auxiliary evidence is informative about the downstream decision itself. This is important because two identical downstream decisions may be supported by evidence of very different reliability. For example, in Derm7pt, consider a lesion image that receives the same final prediction when the supporting dermatological findings are predicted with either high or low confidence. If the downstream generative foundation model meaningfully processes upstream uncertainty, it should therefore be related to whether the final decision is correct and, potentially, to the model’s own predictive uncertainty, which we denote by .2 We examine both relationships below.
Upstream uncertainty is informative of downstream correctness (Table 6 & Table 7). We examine whether the uncertainty associated with auxiliary evidence can distinguish correct from incorrect downstream decisions; the uncertainty features used for each benchmark are detailed in Appendix § 11.§ 5. In CUB, human-reported certainty is only weakly informative, with AUROC remaining close to chance for both tested models. In contrast, the BLIP-derived reliability profile in SUN is strongly predictive of LLM correctness at . In Derm7pt, classifier-derived uncertainty provides a weaker but consistent signal for both the LLM and VLM across the evaluated uncertainty representations. Overall, upstream uncertainty is informative of downstream correctness, although the strength of this relationship depends on the source of uncertainty and the downstream model.
| Model | Upstream Features | AUROC |
| Llama-3.1-8B-Instruct | Confidence | 0.6259 |
| Llama-3.1-8B-Instruct | Entropy | 0.6257 |
| Llama-3.1-8B-Instruct | Full | 0.6480 |
| Qwen2-VL-7B-Instruct | Confidence | 0.6388 |
| Qwen2-VL-7B-Instruct | Entropy | 0.6377 |
| Qwen2-VL-7B-Instruct | Full | 0.6358 |
Table 7. Predicting Derm7pt downstream decision correctness from upstream classifier uncertainty at . For all models, AUROC is obtained using a standardized logistic-regression probe with the indicated uncertainty features under stratified five-fold cross-validation. We report pooled out-of-fold AUROC.
Improved decisions do not necessarily imply lower downstream uncertainty (Table 8). On Derm7pt, we ask whether communicating upstream classifier confidence is reflected in the downstream model’s own predictive uncertainty . The two are not directly coupled: even when uncertainty communication changes an incorrect decision to a correct one, downstream entropy often increases rather than
| Modality | Model | Correctness transition | (%) | |||
| LLM | Llama-3.1-8B-Instruct | Incorrect Correct | 0.6736 | 0.6861 | +0.0125 | 48.3 |
| Correct Incorrect | 0.6836 | 0.6850 | +0.0015 | 52.4 | ||
| Correct Correct | 0.4498 | 0.4737 | +0.0239 | 34.6 | ||
| Incorrect Incorrect | 0.5883 | 0.6065 | +0.0182 | 39.3 | ||
| VLM | Qwen2-VL-7B-Instruct | Incorrect Correct | 0.6315 | 0.6773 | +0.0457 | 58.3 |
| Correct Incorrect | 0.6885 | 0.6796 | -0.0089 | 40.0 | ||
| Correct Correct | 0.2406 | 0.2687 | +0.0280 | 39.9 | ||
| Incorrect Incorrect | 0.2412 | 0.3230 | +0.0819 | 18.9 |
Table 8. Changes in downstream predictive entropy on Derm7pt at after communicating upstream classifier confidence. ; negative values indicate reduced predictive uncertainty. The final column reports the percentage of samples with .
decreases, for both the LLM and VLM. Thus, upstream uncertainty can affect the downstream decision without being directly mirrored by the model’s own predictive uncertainty. We examine this upstream-to-downstream uncertainty relationship more directly through a Bernoulli view in Appendix § 14.
Conclusions
Limitations and future scope. We focus on zero-shot classification, which enables objective measurement of downstream correctness but does not capture open-ended generation or question-answering settings. We do not consider the case where the uncertainty information itself comes from a heterogeneous sources in a single context. Extending the analysis to open-ended tasks, cross-source calibration, and uncertainty-aware prompt optimization, therefore, remains a useful direction for future work.
To conclude, we studied whether communicating uncertainty associated with auxiliary evidence improves zero-shot generative foundation-model inference. Across three benchmarks, three LLMs, and two VLMs, communicating uncertainty generally improves downstream classification performance. In the main reported settings, LLM macro-F1 increases by 1.95–7.84 points on CUB, 5.93–26.18 on SUN, and 0.25–9.10 on Derm7pt; we observe similar patterns for VLMs, although the gains are generally smaller. The benefit depends on the ordering, alignment, and composition of auxiliary evidence, while upstream uncertainty informs downstream correctness without being directly reflected in the model’s own predictive uncertainty. These results support treating uncertainty as part of the context supplied to generative foundation models rather than discarding it before inference.
AI use statement
We used LLM-based tools to assist with writing and editing the manuscript, and with writing, debugging, and refining research code. All LLM-assisted text was reviewed and edited by the authors, as was all LLM-generated or modified code. LLMs were not relied upon to make any final scientific or experimental decisions. The authors take full responsibility for the final content of this work, including all text, claims, results, code, and other artifacts produced with the aid of LLMs.
Detailed Literature Survey
Reasoning with Uncertain Auxiliary Evidence
Prior work on uncertain auxiliary evidence spans formal probabilistic reasoning, belief revision under external information, and uncertainty-aware agentic systems. [27] introduce BLInD, a benchmark in which LLMs answer queries from textual premises containing explicitly quantified probabilities. Direct and chain-of-thought prompting deteriorate as the underlying Bayesian networks become more complex, whereas translating the premises into executable symbolic representations substantially improves accuracy. BLInD therefore evaluates formal probability computation and the value of symbolic execution, rather than whether uncertainty associated with naturally produced evidence should be communicated to a downstream model.
From a complementary behavioral perspective, [9] stratify multiple-choice predictions by the LLM’s initial confidence and inject supportive, opposing, or irrelevant knowledge-graph statements. Low- and moderate-confidence predictions are more responsive to external information, particularly when it supports alternatives that the model already considers plausible. In contrast, high-confidence predictions are comparatively resistant to revision. Their analysis uses the downstream model’s confidence to explain knowledge acceptance rather than studying the uncertainty reported by the evidence source.
Uncertainty has also been incorporated into multimodal agentic pipelines. SRICE [39] is a training-free VQA framework that applies conformal procedures to segmentation and object-detection outputs and uses output uncertainty to select among tool-conditioned pathways. Its ablations additionally attach raw confidence scores to textualized tool evidence; this produces only marginal gains over providing tool outputs alone and underperforms the conformally calibrated pipeline. VideoAgent2 [40] more directly communicates confidence by attaching uncertainty scores to observations returned by captioning, detection, and tracking tools. The LLM is instructed to use these scores when revising its retrieval plan and synthesizing an answer, and removing them reduces average accuracy across the evaluated long-video QA benchmarks. However, because confidence influences subsequent tool selection and retrieval, it can also change which evidence the model eventually observes. Consequently, the contribution of uncertainty communication cannot be separated from changes to the evidence-acquisition trajectory.
Related research examines whether external contexts should be trusted and whether all retrieved evidence is equally useful. [12] formulate situated faithfulness to study how LLMs arbitrate between internal knowledge and external contexts that may be correct or misleading. They show that models frequently over-rely on external information and propose self-guided and rule-based confidence-reasoning procedures for estimating contextual reliability before answer selection. These methods infer whether the external context as a whole should be trusted, rather than communicating the graded uncertainty supplied by its source.
PANDORA (Wu et al., 2025) instead varies the composition of retrieved evidence across seven forms of RAG noise. Its results demonstrate that evidence items differ markedly in utility: counterfactual and other harmful contexts reduce accuracy, while some semantically irrelevant or malformed contexts can unexpectedly improve accuracy and confidence. This heterogeneity motivates examining evidence subsets rather than treating auxiliary information as uniformly useful.
Key Differences. In summary, these studies formalize probabilistic premises, explain knowledge acceptance using model confidence, use uncertainty to calibrate or route agentic pipelines, estimate contextual trustworthiness, or manipulate the quality of retrieved evidence. Unlike these works, our paired design holds task-native auxiliary evidence fixed and changes only whether its associated uncertainty is communicated to the foundation model; we then evaluate its effects across evidence sources, domains, models, and evidence subsets.
Reasoning through Attributes and Intermediate Representations
Concept bottleneck models provide an interpretable alternative to end-to-end prediction by requiring decisions to pass through human-understandable intermediate concepts [17]. This structure permits concept-level inspection and intervention, but its performance depends critically on whether the selected concepts are informative and whether they can be reliably inferred from the input. Moving beyond deterministic concept predictions, [16] introduce Probabilistic Concept Bottleneck Models, which represent each predicted concept through a probabilistic embedding and propagate concept uncertainty to the final class prediction. On CUB and AwA2, concept uncertainty increases when relevant visual information is obscured, correlates with lower concept and class accuracy, and supports more effective uncertainty-guided interventions. ProbCBM therefore incorporates uncertainty directly into a trained, task-specific bottleneck architecture.
The reliability of the concept layer also depends on its supervision. Park et al. [28] shows that annotation noise degrades task accuracy, concept alignment, and the effectiveness of concept-level interventions. Importantly, much of this degradation is concentrated in a relatively small subset of noise-susceptible concepts rather than being distributed uniformly across the bottleneck. They use sharpness-aware minimization to improve robustness during training and predictive entropy to identify concepts to prioritize for correction at inference time. Their results demonstrate that uncertainty can expose particularly damaging intermediate representations, but the uncertainty is inferred within a trained concept model and used to repair its bottleneck under noisy supervision.
Concept-based prediction has also been extended to pretrained vision-language models. Menon and Vondrick [25] introduces classification by description, in which an LLM generates class-specific visual descriptors and their image–text similarities are aggregated to classify an image. The resulting linguistic attributes provide an interpretable and editable intermediate representation while improving zero-shot recognition across several visual domains. However, the descriptor similarities function internally as components of the classifier rather than as uncertainty information explicitly presented to a downstream foundation model. Taking a Bayesian approach to concept construction, Feng et al. [8] introduces BC-LLM, which iteratively uses an LLM to propose and annotate concepts within a Bayesian variable-selection procedure. Observed data can consequently correct unreliable LLM proposals, while posterior inference quantifies uncertainty over the resulting concept sets.
Key Differences. Collectively, these approaches use uncertainty to represent, train, select, evaluate, or repair an intermediate concept layer. In contrast, we begin with task-native attributes produced by external sources, hold both their content and the foundation model fixed, and vary whether their associated uncertainty is explicitly communicated. We additionally vary the supplied attribute subsets and evaluate the resulting final predictions across evidence sources, domains, and foundation models.
Upstream and Downstream Uncertainty in Foundation Models
Foundation-model uncertainty has primarily been studied as a property of the model’s knowledge or of its generated responses. Kadavath et al. [13] show that LLMs can evaluate whether their answers are correct and whether they possess the knowledge required to answer a question. Semantic entropy instead groups generations by meaning, preventing lexical variation among semantically equivalent answers from being interpreted as uncertainty [18]. In black-box settings, Xiong et al. [35] find that directly verbalized confidence is frequently overconfident, while consistency across repeated responses provides a more informative signal of failure. Rather than evaluating uncertainty after generation, Chen et al. [4] estimates whether a query is answerable beforehand, enabling uncertainty-aware retrieval and model cascading. These approaches characterize uncertainty originating within the downstream model.
Subsequent work further distinguishes uncertainty about knowledge from legitimate variability in model responses. Yadkori et al. [37] iteratively prompt an LLM with its previous responses and measure their dependence through mutual information. This separates epistemic uncertainty from high response entropy due to multiple valid answers, thereby improving hallucination detection and calibrated abstention. Kumaran et al. [19] provide causal evidence that internal confidence influences whether an LLM answers or abstains. Through activation steering and explicit manipulation of decision thresholds, they show that log-probability-based and verbal confidence both affect abstention. However, each provides only a partial read-out of a richer internal confidence representation. Notably, verbalized confidence can influence model behavior even when it is comparatively weak at predicting correctness.
External context can itself alter the reliability of downstream uncertainty estimates. Soudani et al. [30] show that conventional estimators in retrieval-augmented generation often report lower uncertainty merely because contextual evidence is present, including when that evidence is irrelevant or contradicts the generated response. They formulate axioms describing how response uncertainty should behave under supportive, contradictory, and irrelevant contexts and introduce evidence-aware calibration that strengthens its association with correctness. Their uncertainty nevertheless remains a property of the downstream response after contextual augmentation. Multimodal models introduce an additional source of variation: Xuan et al. [36] find that verbalized confidence in VLMs is frequently miscalibrated and sensitive to input modality, with semantically aligned visual inputs generally producing poorer calibration than corresponding textual inputs. Their confidence-aware prompting improves the calibration of the VLM’s self-reported confidence but does not communicate uncertainty originating from an upstream evidence source.
A smaller body of work jointly considers uncertainty from multiple components of an augmented system. Lymperopoulos and Sarathy (Lymperopoulos and Sarathy [23]) formulate system-level uncertainty for tool-augmented question answering by combining uncertainty in the external tool with predictive or semantic uncertainty in the final LLM response. Under their strong-tool approximation, the entropy of an external classifier or retriever is added to the entropy of the final answer. The resulting measure is generally more predictive of correctness on synthetic classifier-based QA tasks, although improvements are smaller and model-dependent in their RAG proof of concept. Their objective is to assess the reliability of the completed composite system rather than expose tool uncertainty to the LLM that produces the answer. Within tool-using agents, ProbeCal learns embedding-based estimates of whether alternative prompts and execution traces will succeed, and uses these calibrated scores to select trajectories (Liu et al. [21]). It therefore calibrates confidence in an agent’s tool-use trajectory rather than representing uncertainty associated with the resulting external evidence. Collectively, this literature estimates downstream self-knowledge, semantic variability, query answerability, abstention confidence, context-sensitive response uncertainty, multimodal calibration, composite-system reliability, or tool-use success.
Key Differences. Unlike these approaches, our intervention communicates uncertainty emitted by the original auxiliary-evidence source to an otherwise unchanged foundation model while holding the evidence itself fixed. We then evaluate its effect on final predictions, whether upstream uncertainty predicts downstream correctness, and whether upstream uncertainty is reflected in the foundation model’s own predictive uncertainty across evidence sources, domains, and models.
Additional Implementation Details
Evaluation Protocol
Repeated runs and reproducibility. All experiments are repeated three times using seeds 42, 96, and 1001. For CUB, the seed determines the sampled evaluation set, with 40 images drawn per class for each 7-, 8-, and 9-class configuration; paired conditions use the same sampled images within each run, and results are reported as mean standard deviation across seeds. In contrast, SUN and Derm7pt use fixed evaluation sets of 80 and 395 images, respectively. Because inference is deterministic and the input samples remain unchanged, repeated runs can produce identical predictions, resulting in zero variance. In such cases, we report the common performance value without appending 0.0000 for clarity. All experiments were conducted on Google Colab using NVIDIA A100-SXM4-80GB GPUs.
Classification performance. Our primary evaluation metric is macro-F1, which weights each target class equally. For every paired configuration, the effect of communicating uncertainty is reported as
in percentage points. For Derm7pt, all nonempty subsets of the seven findings are evaluated in the text-only experiments, and results are additionally grouped by subset cardinality.
Decision changes and correctness. To complement aggregate performance, we examine paired sample-level transitions between Aux and Aux+Unc. We distinguish cases in which communicating uncertainty changes an incorrect prediction to a correct one from those in which it changes a correct prediction to an incorrect one. We additionally evaluate whether upstream uncertainty is informative of downstream prediction correctness using AUROC. Because the available uncertainty has different semantics across CUB, SUN, and Derm7pt, the corresponding sample-level score construction is defined separately for each evidence source in Appendix § 11.§ 5.
Additional Dataset Details
CUB Candidate-Class Construction
Rather than selecting candidate species independently for each class count, we construct a nested sequence of increasingly ambiguous classification problems. We represent each of the 200 CUB species by its continuous 312-dimensional class-level attribute profile and use cosine similarity in this attribute space to guide candidate selection.
We begin with a deliberately well-separated three-class configuration. Among the 200 species, we select Chuck-will’s-widow (class 22), Green Jay (75), and White Pelican (101) by maximizing the minimum pairwise cosine distance between their class-level attribute profiles. We then progressively enlarge this fixed set by introducing species whose attribute profiles are more similar to existing candidates. The five-class configuration adds Whip-poor-will (105) and Hooded Warbler (167). The seven-class configuration further adds Ivory Gull (63) and Painted Bunting (16). These additions introduce interpretable competing groups: Whip-poor-will is highly similar to Chuck-will’s-widow, Hooded Warbler and Painted Bunting provide closer attribute-space alternatives to Green Jay, and Ivory Gull is close to White Pelican.
Finally, we extend the seven-class set with Nighthawk (92) and Herring Gull (62). Nighthawk creates a closely related attribute-space group with Whip-poor-will and Chuck-will’s-widow. Herring Gull similarly introduces ambiguity around the light-coloured water-bird candidates, with similarities to Ivory Gull and White Pelican. The resulting eight- and nine-class configurations, therefore, increase candidate ambiguity while preserving all classes from the smaller configurations. The list of classes used in different configurations is summarized in Table 9.
| Configuration | Candidate species added |
| 3-class | Chuck-will’s-widow (22), Green Jay (75), White Pelican (101) |
| 5-class | + Whip-poor-will (105), Hooded Warbler (167) |
| 7-class | + Ivory Gull (63), Painted Bunting (16) |
| 8-class | + Nighthawk (92) |
| 9-class | + Herring Gull (62) |
Table 9. Nested candidate-species configurations used for CUB. Each larger configuration retains all species from the preceding setting.
For each image , we retain only visual attributes annotated as present and exclude those marked as not visible. These retained observations constitute the auxiliary evidence . Each evidence is accompanied by the annotator-reported ordinal certainty signal . For each retained attribute, we also provide its continuous class-level score for each candidate species.
For a fixed image and candidate configuration, the auxiliary-only and uncertainty-aware conditions share the same retained attributes and class-level attribute profiles. The only difference is that the uncertainty-aware condition additionally communicates the corresponding human certainty label for each retained evidence . This construction, therefore, isolates the effect of communicating the available human certainty while holding the underlying auxiliary evidence fixed.
SUN Scene and Attribute Construction
SUN attribute representation. The SUN Attribute Database provides 102 scene attributes together with Amazon Mechanical Turk annotation vote fractions. We retain these vote fractions as continuous values rather than thresholding them into binary labels. In the original annotations, the per-image vote fractions take values in , corresponding to agreement among three annotators. For each candidate scene category, we construct a class-level attribute profile by averaging these continuous vote fractions over images belonging to that class.
Attribute selection. Starting from the 102 SUN attributes, the selection procedure first removes attributes that are rare within the selected scene classes, defined as being present in fewer than 5% of images. For the remaining attributes, discriminativeness is estimated using two complementary statistics: mutual information between the attribute values and the scene category, and the maximum absolute coefficient obtained from an -regularised multinomial logistic-regression classifier. Each attribute is ranked independently according to these two criteria, and the two ranks are averaged to obtain a combined ranking. The highest-ranked candidates are then retained subject to an attribute budget of approximately 20.
The implementation additionally permits selected correlated attribute groups to be retained deliberately rather than automatically removing redundant features. In particular, it defines the pairs {open area, enclosed area},
, and ; a group is protected only when all of its members are non-rare for the selected scene classes. This produces the 20-attribute pool used for subsequent BLIP scoring.
Image-specific BLIP scores. We use the pretrained BLIP image–text-matching model Salesforce/blip-itm-base-coco. For each image and each of the 20 selected attributes, we construct a textual description of the form “a photo with [attribute]”. Attribute names are lightly normalised by removing parenthetical qualifiers and replacing “/” with “or” before forming the caption. All 20 captions are evaluated against the same image using BLIP’s ITM head, and we record
where is the score assigned to the ITM match class. The scores are stored exactly as produced by BLIP, without thresholding or subsequent calibration.
Top- auxiliary evidence. For every image, the 20 attributes are sorted in descending order of . We retain the first attributes as the image-specific auxiliary evidence . For each retained evidence , its corresponding BLIP match score is used as the uncertainty signal . Consequently, increasing produces nested evidence sets for a fixed image: the top-3 attributes are contained in the top-5 set, which is contained in the top-8 and top-10 sets. Ties are resolved deterministically by attribute name. The same selected attributes and ordering are used in both experimental conditions. In Aux, the downstream model observes each attribute together with its candidate-class profile; in Aux+Unc, the identical evidence is shown together with its BLIP match score. Hence, the paired comparison does not alter which attributes are supplied, and isolates the effect of making the underlying score magnitude explicit.
Scene categories. We evaluate four SUN scene categories: park, office cubicles, supermarket, and coffee shop. These categories were specified explicitly as the candidate set prior to attribute selection; the automated preprocessing pipeline does not select the scene categories. Given this fixed class set, the subsequent procedure selects the visual and contextual attributes used for auxiliary evidence.
Derm7pt Auxiliary-Evidence Construction
Seven-point checklist findings. For Derm7pt, the auxiliary evidence is derived from the seven dermatological findings in the seven-point checklist: pigment network, blue-whitish veil, vascular structures, pigmentation, streaks, dots and globules, and regression structures. The original categorical annotation for each finding is mapped to a binary state indicating whether the finding is considered not suspicious or suspicious. The mappings used throughout the experiments are listed in Table 10.
| Finding | Not suspicious | Suspicious |
| Pigment network | absent; typical | atypical |
| Blue-whitish veil | absent | present |
| Vascular structures | absent; arborizing; comma; hairpin; within regression; wreath | dotted; linear irregular |
| Pigmentation | absent; diffuse regular; localized regular | diffuse irregular; localized irregular |
| Streaks | absent; regular | irregular |
| Dots and globules | absent; regular | irregular |
| Regression structures | absent | blue areas; white areas; combinations |
Table 10. Binarisation of the original Derm7pt checklist annotations used to train the seven auxiliary finding predictors.
Upstream finding predictors. We train one independent ResNet-18 classifier for each of the seven binary checklist findings using the Derm7pt training split. All classifiers operate on the clinical-image modality and are initialized from ImageNet-pretrained weights. The original ResNet classification head is replaced with a two-class linear classifier corresponding to not suspicious and suspicious. Images are resized to 224 × 224 pixels and normalized using ImageNet statistics. During training, we additionally apply random cropping, horizontal and vertical flipping, rotation, color jitter, and random erasing. To mitigate the substantial class imbalance across several findings, training uses inverse-frequency-weighted sampling.
Each classifier is trained for 20 epochs using Adam with a learning rate of , weight decay of , a batch size of 32, cross-entropy loss, and a cosine learning-rate schedule. For each finding, the checkpoint attaining the highest validation macro-F1 is retained. Held-out performance of the resulting seven classifiers on the common 395-image test split is reported in Table 11. The models differ substantially in their ability to recover the individual findings, with test macro-F1 ranging from 0.5612 to 0.7078. This variation is useful for our downstream study because the auxiliary evidence is therefore neither uniformly reliable nor artificially noise-free.
| Auxiliary Finding | Macro-F1 | F1 Not-Susp. | F1 Suspicious | Accuracy |
| Blue-whitish veil | 0.7078 | 0.8528 | 0.5628 | 0.7797 |
| Regression structures | 0.6720 | 0.8148 | 0.5291 | 0.7342 |
| Dots and globules | 0.6063 | 0.5674 | 0.6452 | 0.6101 |
| Pigmentation | 0.5949 | 0.7231 | 0.4667 | 0.6354 |
| Vascular structures | 0.5852 | 0.9370 | 0.2333 | 0.8835 |
| Pigment network | 0.5786 | 0.8218 | 0.3353 | 0.7190 |
| Streaks | 0.5612 | 0.6734 | 0.4490 | 0.5899 |
Table 11. Held-out test performance of the seven Derm7pt auxiliary ResNet classifiers. Each model predicts whether the corresponding dermoscopic finding is not-susp or suspicious. All results are computed on the same 395 clinic images from the fixed test split. Macro-F1 is the primary metric; class-wise F1 and accuracy are also reported.
For an input image , predictor produces a probability vector
The predicted binary finding is
and its corresponding maximum-softmax confidence is
We retain this confidence exactly as produced by the upstream classifier and do not apply additional calibration. Consequently, Aux communicates the predicted binary findings, while Aux+Unc communicates the identical findings together with their associated confidence scores.
Downstream target and reference evidence. The downstream task collapses the Derm7pt diagnosis into a binary target, with melanoma and basal cell carcinoma treated as malignant and all remaining diagnoses treated as non-malignant. We evaluate the fixed test split of 395 clinical images, containing 117 malignant and 278 non-malignant cases. The same images are used to obtain the predicted findings and confidence scores from the seven upstream classifiers.
In addition to these predicted-evidence conditions, we construct two reference conditions directly from the ground-truth checklist annotations. Gold Binary applies the same binary mappings in Table 10, whereas Gold Original retains the original categorical annotations. These conditions remove errors introduced by the seven upstream predictors and therefore help distinguish limitations associated with upstream evidence quality from those of the downstream foundation model.
Downstream Generative Foundation Models and Inference
Large language models. We evaluate Qwen3-32B, Llama-3.1-8B-Instruct, and Phi-4-14B using the checkpoints Qwen/Qwen3-32B, meta-llama/Llama-3.1-8B-Instruct, and microsoft/phi-4, respectively. All models are evaluated without fine-tuning using their native chat templates and zero-shot task prompts. For Qwen3-32B, we disable its optional thinking mode using enable_thinking=False; none of the models is explicitly prompted to produce chain-of-thought reasoning.
The models are loaded in bfloat16 precision on CUDA with automatic device placement, and evaluated in inference mode. Decoding is deterministic (do_sample=False), without temperature, top-p, or top-k sampling. Generation is limited to 32 tokens for Derm7pt and 64 tokens for CUB and SUN. Each prompt requests a single class identifier, and raw model responses and sample-level predictions are retained for subsequent analysis.
Vision–language models. For the multimodal evaluation, we use Qwen2-VL-7B-Instruct and InternVL3-8B. Both models receive the original image together with the textual task context. We evaluate the Image-only, Image+Aux, and Image+Aux+Unc conditions; for Derm7pt, we additionally evaluate the Gold Binary and Gold Original reference conditions.
Both VLMs are evaluated without fine-tuning using model-specific image preprocessing and deterministic decoding. For Qwen2-VL-7B-Instruct, the processor uses a pixel-count budget ranging from to , with generation limited to 64 tokens. InternVL3-8B uses dynamic image tiles with at most 12 tiles. Since the classification tasks require a single candidate identifier, the final InternVL3-8B evaluations use max_new_tokens=1. Image preprocessing and all inference settings are held fixed across paired evidence conditions for each model.
Correctness Probe
To evaluate whether upstream uncertainty is informative about downstream decision correctness, we train a lightweight logistic regression probe on uncertainty-derived features and predict whether the downstream decision is correct. We use the same probing protocol across all benchmarks and model configurations.
Features are standardized using StandardScaler and passed to an -regularized logistic-regression model with , the liblinear solver, balanced class weights, and a maximum of 2000 iterations. Evaluation uses five-fold stratified cross-validation with shuffling and random seed 42. Standardization is fitted independently within each training fold through a pipeline to avoid leakage from the held-out fold.
For each sample, the probe outputs a probability that the downstream decision is correct. We concatenate the out-of-fold probabilities across the five folds and report the pooled AUROC as the primary metric. Separate probes are trained for each benchmark, model, inference condition, and uncertainty representation; samples are never pooled across these settings.
Uncertainty Features
To assess whether upstream uncertainty provides information about downstream decision correctness, we construct dataset-specific features directly from the auxiliary evidence and its associated uncertainty signals. For CUB, these features capture the distribution and concentration of human-reported certainty across retained attributes, along with evidence quantity. For SUN, they summarize the strength and dispersion of BLIP image-text matching scores, while Derm7pt uses confidence from the specialized classifiers associated with the selected findings. All features are independent of the downstream LLM’s predictions, token probabilities, and output confidence. Table 12 provides their complete definitions.
| Feature / representation | Definition / interpretation |
| CUB: Human-reported certainty | |
frac_guessing | Fraction of retained attributes annotated as guessing. |
frac_probably | Fraction of retained attributes annotated as probably. |
frac_definitely | Fraction of retained attributes annotated as definitely. |
mean_certainty | Mean ordinal certainty using the original certainty IDs 2, 3, and 4. |
certainty_entropy | Normalized entropy of the three certainty-level fractions: . |
certainty_concentration | Largest fraction among the three certainty levels, . |
num_attributes | Number of retained attributes, capturing the amount of available auxiliary evidence. |
| SUN: BLIP image–text matching scores | |
mean_blip | Mean BLIP matching score across the selected top- attributes. |
min_blip | Lowest BLIP matching score among the selected attributes. |
top1_topk_gap | Difference between the highest and lowest selected matching scores. |
std_blip | Standard deviation of the selected BLIP matching scores. |
| Derm7pt: Specialised classifier predictive distributions | |
| Confidence | Seven-dimensional vector containing the maximum-softmax confidence for each checklist finding . |
| Entropy | Seven-dimensional vector containing the binary predictive entropy of each finding classifier. |
| Full | Twenty-one-dimensional representation containing for each of the seven findings, where and denote the probabilities of not suspicious and suspicious, respectively. |
Table 12. Upstream features used by the downstream correctness probes. Features are constructed solely from the auxiliary evidence and its associated certainty or confidence signals, independently of the downstream model’s predictions.
Obtaining Predictive Uncertainty of the Downstream Generative Foundation Model
For Derm7pt, we measure the downstream model’s predictive uncertainty from its probability distribution over the two valid classes [18], \mathcal{Y} = \{malignant, non\mathchar`-malignant\}. Let denote the probability assigned to class after normalizing the probability mass over these two alternatives. For the LLM, we obtain from the token probabilities assigned to the two valid class labels. For the VLM, we use the logits for the first generated token corresponding to the two candidate classes and normalize them over the same set of alternatives. We then compute predictive entropy as
where a larger indicates greater downstream uncertainty, the same procedure is applied under both Aux and Aux+Unc; only the context supplied to the downstream model changes.
Additional Experiments
Experiments across Different Subset Sizes
SUN LLM Results
The extended SUN results in Table 13 show that the benefit of communicating uncertainty is generally preserved across different evidence budgets. The effect is strongest for Qwen3-32B, particularly as more attributes are included, while Llama-3.1-8B-Instruct and Phi-4-14B show smaller but mostly positive gains. The near-zero change for Llama-3.1-8B-Instruct at Top-5 is the only setting where uncertainty does not help. Overall, the results indicate that the value of uncertainty depends on both the downstream model and the amount of auxiliary evidence available.
| Model | Top- | Aux | Aux + Uncertainty | (pp) |
| Qwen3-32B | 3 | 0.6320 | 0.7028 | +7.09 |
| Qwen3-32B | 5 | 0.7181 | 0.7398 | +2.17 |
| Qwen3-32B | 8 | 0.6728 | 0.8690 | +19.62 |
| Qwen3-32B | 10 | 0.6066 | 0.8684 | +26.18 |
| Llama-3.1-8B | 3 | 0.8881 | 0.9244 | +3.63 |
| Llama-3.1-8B | 5 | 0.8892 | 0.8874 | -0.18 |
| Llama-3.1-8B | 8 | 0.8651 | 0.9120 | +4.69 |
| Llama-3.1-8B | 10 | 0.8648 | 0.9241 | +5.93 |
| Phi-4-14B | 3 | 0.8542 | 0.8843 | +3.02 |
| Phi-4-14B | 5 | 0.8438 | 0.8466 | +0.28 |
| Phi-4-14B | 8 | 0.8589 | 0.8849 | +2.61 |
| Phi-4-14B | 10 | 0.8175 | 0.8843 | +6.69 |
Table 13. SUN macro-F1 across top- auxiliary-attribute settings. denotes the Aux+Unc gain over Aux in percentage points.
Derm7pt LLM Results
The extended Derm7pt results in Table 14 show that communicating uncertainty is generally more beneficial as richer evidence subsets are provided, although the magnitude depends on the model and composition. The gains are small for single findings and become larger for intermediate or full evidence sets, particularly for Qwen3-32B and Llama-3.1-8B-Instruct. The gold-reference conditions remain substantially stronger than the predicted-evidence conditions, indicating that upstream finding quality is still an important bottleneck even when its confidence is communicated.
| Model | # subsets | Aux | Aux+Unc | Gold Binary | Gold Original | (pp) | |
| Qwen3-32B | 1 | 7 | 0.567 0.040 | 0.567 0.049 | 0.655 0.063 | 0.636 0.106 | +0.00 |
| Qwen3-32B | 3 | 35 | 0.573 0.043 | 0.595 0.042 | 0.681 0.062 | 0.715 0.029 | +2.20 |
| Qwen3-32B | 5 | 21 | 0.581 0.029 | 0.612 0.012 | 0.731 0.023 | 0.758 0.028 | +3.14 |
| Qwen3-32B | 7 | 1 | 0.505 | 0.596 | 0.669 | 0.750 | +9.10 |
| Llama-3.1-8B | 1 | 7 | 0.496 0.086 | 0.498 0.096 | 0.550 0.138 | 0.542 0.141 | +0.12 |
| Llama-3.1-8B | 3 | 35 | 0.557 0.047 | 0.577 0.050 | 0.641 0.066 | 0.659 0.094 | +2.05 |
| Llama-3.1-8B | 5 | 21 | 0.584 0.032 | 0.619 0.012 | 0.714 0.024 | 0.739 0.034 | +3.56 |
| Llama-3.1-8B | 7 | 1 | 0.601 | 0.630 | 0.740 | 0.805 | +2.90 |
| Phi-4-14B | 1 | 7 | 0.567 0.040 | 0.579 0.040 | 0.655 0.063 | 0.658 0.057 | +1.19 |
| Phi-4-14B | 3 | 35 | 0.536 0.048 | 0.571 0.042 | 0.671 0.035 | 0.705 0.021 | +3.48 |
| Phi-4-14B | 5 | 21 | 0.528 0.039 | 0.568 0.022 | 0.638 0.042 | 0.683 0.024 | +4.00 |
| Phi-4-14B | 7 | 1 | 0.544 | 0.547 | 0.684 | 0.653 | +0.25 |
Table 14. Derm7pt classification performance across evidence subset sizes. Values are the mean standard deviation of macro-F1 across distinct subsets; for , the single-subset result is reported without deviation. denotes Aux+Unc minus Aux in percentage points. ‘Gold Original’ refers to the original ground-truth auxiliary findings. ‘Gold Binary’ refers to the binarized version of the gold auxiliary findings.
Derm7pt VLM Results
The extended Derm7pt VLM results in Table 15 show a more model-dependent effect of uncertainty communication than for the LLMs. InternVL3-8B benefits across all evidence sizes, whereas Qwen2-VL-7B-Instruct improves only for the larger subsets. In both models, the gold-reference conditions are substantially stronger than the predicted-evidence conditions, again indicating that upstream finding quality remains a major limitation even when confidence is communicated. The remaining gap to the gold-reference conditions also motivates future evaluation with domain-specialized medical VLMs, which may exploit dermatological evidence differently from the general-purpose models considered here.
| Model | Image-only | Image+Aux | Image+Aux+Unc | Gold Binary | Gold Original | (pp) | |
| Qwen2-VL-7B-Instruct | 1 | 0.2649 | 0.5904 | 0.5673 | 0.6655 | 0.6535 | −2.31 |
| Qwen2-VL-7B-Instruct | 3 | 0.5505 | 0.5496 | 0.6710 | 0.7017 | −0.09 | |
| Qwen2-VL-7B-Instruct | 5 | 0.5422 | 0.5483 | 0.6662 | 0.7273 | +0.61 | |
| Qwen2-VL-7B-Instruct | 7 | 0.5264 | 0.5442 | 0.6271 | 0.7306 | +1.78 | |
| InternVL3-8B | 1 | 0.4604 | 0.4808 | 0.5301 | 0.5213 | 0.5946 | +4.93 |
| InternVL3-8B | 3 | 0.5210 | 0.5674 | 0.6315 | 0.6994 | +4.64 | |
| InternVL3-8B | 5 | 0.5021 | 0.5296 | 0.6355 | 0.7014 | +2.75 | |
| InternVL3-8B | 7 | 0.4739 | 0.4913 | 0.6377 | 0.7220 | +1.74 |
Table 15. Derm7pt VLM performance across auxiliary-evidence subset sizes. All conditions include the original clinical image. Image+Aux provides predicted checklist findings, while Image+Aux+Unc provides the same findings together with their corresponding upstream classifier confidence. Gold Binary and Gold Original replace the predicted findings with their ground-truth binarized and original categorical annotations, respectively. Results are macro-F1 on the fixed 395-image test set. denotes the paired change from Image+Aux to Image+Aux+Unc in percentage points.
Analysis of Samples That Change with Additional Evidence for SUN Scene Classification
The preceding analysis shows that communicating BLIP scores can prevent a correct prediction from being overturned when lower-ranked attributes are added. We next ask what characterizes these cases. We focus on the transition from to . For each image, we call its three highest-ranked attributes the original evidence and attributes ranked 4–10 the added evidence. We then measure the strength of the relationship between the two sets of attributes using the continuous SUN attribute annotation matrix.
Importantly, the correlation is not computed within a single image. Each SUN attribute defines a vector of annotation strengths across images, and we compute the Pearson correlation between these vectors. For example, suppose that for one test image the original top-3 attributes include trees, grass, and sky, while the added evidence contains attributes such as road, bench, and building. We ask whether, across SUN images, scenes containing trees also tend to contain road, whether grass co-occurs with bench, and so forth. With three original and seven added attributes, this gives pairwise correlations for each test image.
We summarize these relationships in two complementary ways. The first is the mean absolute correlation,
A value approaching 1 would indicate that the newly added attributes are strongly associated with the original evidence, regardless of whether the association is positive or negative; a value near 0 indicates that the two sets are largely unrelated. We additionally compute the signed mean
Here, a large positive value means that the added attributes generally co-occur with the original evidence, a strongly negative value indicates an opposing association, and a value near zero indicates little net alignment (or a mixture of positive and negative relationships). Finally, we compute the mean BLIP match score of the seven newly introduced attributes to measure how strongly BLIP itself associates them with the image.
We group images by how the downstream prediction changes as increases from to : correctcorrect, correctincorrect, incorrectcorrect, or incorrectincorrect. Under Aux, images that remain correct show the strongest association between the original and added evidence, with mean values of 0.236, 0.219, and 0.218 for Qwen3-32B, Llama-3.1-8B-Instruct, and Phi-4-14B, respectively. Thus, predictions are more likely to remain stable when the newly introduced attributes are structurally related to the evidence already available.
We then examine the particularly informative cases that are correctincorrect under Aux but remain correctcorrect under Aux+Unc—that is, cases in which adding the seven attributes destabilizes the prediction when their scores are omitted, but not when their scores are communicated. The added attributes in these cases receive very low mean BLIP scores (0.041, 0.041, and 0.051 for Qwen3-32B, Llama-3.1-8B-Instruct, and Phi-4-14B) and have weak signed association with the original top-3 evidence (, 0.046, and 0.037, respectively). Although these groups are small, they provide a consistent explanation for the robustness pattern in Table 4: uncertainty communication is particularly useful when the newly introduced evidence is weakly scored and only loosely aligned with the stronger evidence already present.

Figure 4. BLIP match score as a function of attribute rank across the 80 selected SUN images. For each image, the 20 candidate attributes are ranked by BLIP match score; points show the mean score at each rank and error bars denote one standard deviation. Scores decrease with rank, confirming that increasing top- progressively introduces attributes that BLIP itself scores as less strongly matched to the image.
Prompt Templates
LLM and VLM Prompts
For SUN, the LLM prompt closely follows the VLM prompt shown in Figure 7, with the sentence “The original scene image is also provided for direct inspection.” removed. For CUB, the prompt is likewise largely shared across modalities, following Figure 6; for the VLM setting, the opening is changed to “You are given the original bird image.”. In contrast, Derm7pt requires modality-specific prompt designs, shown in Figure 5 and Figure 8 for LLMs and VLMs, respectively. We also observe greater prompt sensitivity on Derm7pt, which we investigate further in Appendix § 13.§ 2.

Figure 5. Derm7pt LLM Prompt. This is an illustrative example with only one auxiliary evidence and its associated classifier confidence. The prompt contains additional background information obtained from [2].

Figure 6. CUB LLM Prompt. This is an illustrative example with 7-classes bird species classification task.

Figure 7. SUN scene classification VLM prompt. This is an illustrative example with three auxiliary evidences and their associated uncalibrated BLIP image–text matching scores.

Figure 8. Derm7pt VLM Prompt. This is an illustrative example with all the seven auxiliary evidence with their associated classifier confidence.
Prompt Sensitivity: Preliminary Investigation
We investigated the sensitivity of Derm7pt classification to the final output instruction. All experiments used the same 395 clinic test cases, all seven ResNet-predicted auxiliary findings, and deterministic inference. The downstream task remained binary classification (non-malignant versus malignant). The dermatologist role, seven-point checklist background knowledge, diagnostic class definitions, and auxiliary evidence were held constant (same as shown in Figure 5). We evaluated three variants of the final instruction.
P1: Original direct-answer instruction.
Instructions:
Based on the dermoscopic findings, classify the lesion as NON-MALIGNANT (0) or MALIGNANT (1).
Output Format:
Output exactly one numeric class label: 0 or 1.
Do not provide reasoning, explanations, or any additional text.
P2: Strict single-character instruction.
Instructions:
Use the dermoscopic findings and the background knowledge to determine whether the lesion is NON-MALIGNANT (0) or MALIGNANT (1).
Perform the classification internally. Do not write out the seven-point checklist, calculate scores in your response, or provide any reasoning.
Output Format:
Your entire response must consist of exactly one character: 0 or 1.
Do not output any other text, punctuation, explanation, or formatting.
P3: Intermediate instruction.
Instructions:
Based on the dermoscopic findings, classify the lesion as NON-MALIGNANT (0) or MALIGNANT (1).
Output Format:
Your entire response must consist of exactly one character: 0 or 1.
Do not provide reasoning, explanations, or any additional text.
Table 16 examines three output-instruction variants while keeping the evidence and domain knowledge fixed. Although the absolute macro-F1 and the magnitude of the uncertainty gain vary across prompts and models, the central trend remains stable, adding uncertainty (in some form of representation) outperforms Aux in 8 of the 9 model–prompt combinations. The size of the gain is nevertheless model- and prompt-dependent—for example, Qwen3-32B gains +10.40, +3.33, and +10.14 pp under P1–P3, respectively. This variation motivates treating uncertainty representation together with the downstream model and prompting protocol, rather than assuming that its effect is model-independent.
| Model | Prompt | Aux | Full | Conf. | Ent. |
| Qwen3-32B | P1 | 0.5053 | 0.6093 | 0.6132 | 0.5807 |
| Qwen3-32B | P2 | 0.5442 | 0.5775 | 0.5775 | 0.5213 |
| Qwen3-32B | P3 | 0.5053 | 0.6067 | 0.5963 | 0.5496 |
| Llama-3.1-8B | P1 | 0.6051 | 0.5915 | 0.5772 | 0.5859 |
| Llama-3.1-8B | P2 | 0.5083 | 0.5106 | 0.5083 | 0.5111 |
| Llama-3.1-8B | P3 | 0.6011 | 0.6194 | 0.6301 | 0.5982 |
| Phi-4-14B | P1 | 0.5442 | 0.5491 | 0.54173 | 0.5417 |
| Phi-4-14B | P2 | 0.5513 | 0.5637 | 0.6030 | 0.5564 |
| Phi-4-14B | P3 | 0.5442 | 0.5467 | 0.5466 | 0.5466 |
Table 16. Sensitivity of Derm7pt classification to the output instruction formulation. All experiments use the fixed 395-case test set and all seven ResNet-predicted findings (). Results are reported in macro-F1. Aux provides predicted finding labels only; Full additionally provides both softmax probabilities and entropy; Conf. provides maximum softmax confidence; and Ent. provides predictive entropy.
Table 17 compares our original VLM prompt with a P3-style formulation adapted from the LLM experiments. Under the original prompt, Aux+Unc improves over Aux for both Qwen2-VL-7B-Instruct and InternVL3-8B (+1.8 and +1.7 pp, respectively). Under the alternative P3 formulation, the effect becomes more model dependent: Qwen2-VL-7B-Instruct shows a substantially larger gain of +9.3 pp, whereas InternVL3-8B decreases by 1.9 pp. Thus, as with the LLMs, prompt formulation changes the magnitude of the uncertainty effect, but the benefit is preserved in three of the four model–prompt combinations. Notably, Gold Original consistently outperforms Gold Binary under both prompts and both VLMs, indicating that richer auxiliary evidence remains useful across prompt formulations. These results further support evaluating uncertainty communication jointly with the downstream model and prompt protocol, rather than assuming a model-independent effect.
| Condition | Qwen2-VL-7B Original | Qwen2-VL-7B P3 | InternVL3-8B Original | InternVL3-8B P3 |
| Image-only | 0.2649 | 0.2270 | 0.4604 | 0.3671 |
| Aux | 0.5264 | 0.4251 | 0.4739 | 0.5104 |
| Aux+Unc | 0.5442 | 0.5183 | 0.4913 | 0.4916 |
| Gold Binary | 0.6271 | 0.5595 | 0.6377 | 0.6479 |
| Gold Original | 0.7306 | 0.6982 | 0.7220 | 0.7612 |
| (Aux+Unc, Aux) [pp] | +1.8 | +9.3 | +1.7 | -1.9 |
Table 17. Prompt sensitivity on the 395 held-out Derm7pt clinic images. Macro-F1 is reported for the original VLM prompt and the P3-style prompt adapted from Figure 5. All evidence based conditions use all seven findings (); Image-only uses no auxiliary findings (). The final row reports the macro-F1 change from Aux to Aux+Unc in percentage points.
A Bernoulli View of Upstream-to-Downstream Uncertainty
For Derm7pt, each auxiliary predictor solves a binary task and therefore defines a Bernoulli predictive distribution. For sample and auxiliary finding , let
The corresponding upstream predictive uncertainty is
The communicated classifier confidence is ; for a binary predictor, is therefore a deterministic function of .
The downstream diagnosis is also binary. Let
where is the uncertainty-aware downstream context. The downstream predictive uncertainty is then
This gives the following view of the upstream-to-downstream pathway:
where .
Importantly, there is no reason to expect a simple mapping . The downstream model observes not only the uncertainty of each auxiliary predictor but also which findings are predicted, how the findings interact, and the remaining task context. Two samples with identical upstream uncertainty values may therefore induce very different downstream predictive distributions.
CUB LLM Prompt
A person briefly observed a bird, but it flew away before they could take a photograph. The person reports the visual characteristics they remember seeing. Each reported characteristic is accompanied by the observer's certainty: 'guessing', 'probably', or 'definitely'.
You must identify the bird from the following candidate species.
Candidate species:
Chuck will Widow
Green Jay
White Pelican
Whip poor Will
Hooded Warbler
Ivory Gull
Painted Bunting
For each characteristic reported by the observer, you are also given a class-level prevalence score for that same characteristic in each candidate species. A larger percentage means that the characteristic is more commonly associated with that species.
The class-level prevalence score and the observer's certainty represent different information. The prevalence score describes how commonly the characteristic occurs in a candidate species, whereas the observer's certainty describes how certain the person was that they actually observed that characteristic.
Observed characteristics and candidate statistics:
Bill shape: all purpose [observer certainty: definitely]
Candidate 1 (Chuck will Widow): 28.4%
Candidate 2 (Green Jay): 48.8%
Candidate 3 (White Pelican): 0.0%
Candidate 4 (Whip poor Will): 38.4%
Candidate 5 (Hooded Warbler): 77.9%
Candidate 6 (Ivory Gull): 43.0%
Candidate 7 (Painted Bunting): 26.9%
Wing color: brown [observer certainty: definitely]
Candidate 1 (Chuck will Widow): 85.3%
Candidate 2 (Green Jay): 0.0%
Candidate 3 (White Pelican): 4.7%
Candidate 4 (Whip poor Will): 56.9%
Candidate 5 (Hooded Warbler): 7.0%
Candidate 6 (Ivory Gull): 0.0%
Candidate 7 (Painted Bunting): 21.6%
Upperparts color: pink [observer certainty: definitely]
Candidate 1 (Chuck will Widow): 0.0%
Candidate 2 (Green Jay): 0.0%
Candidate 3 (White Pelican): 0.0%
Candidate 4 (Whip poor Will): 0.0%
Candidate 5 (Hooded Warbler): 0.0%
Candidate 6 (Ivory Gull): 0.0%
Candidate 7 (Painted Bunting): 3.6%
Underparts color: yellow [observer certainty: definitely]
Candidate 1 (Chuck will Widow): 5.7%
Candidate 2 (Green Jay): 61.3%
Candidate 3 (White Pelican): 2.6%
Candidate 4 (Whip poor Will): 1.4%
Candidate 5 (Hooded Warbler): 93.0%
Candidate 6 (Ivory Gull): 0.0%
Candidate 7 (Painted Bunting): 12.8%
Breast pattern: striped [observer certainty: definitely]
Candidate 1 (Chuck will Widow): 38.3%
Candidate 2 (Green Jay): 0.0%
Candidate 3 (White Pelican): 0.0%
Candidate 4 (Whip poor Will): 29.5%
Candidate 5 (Hooded Warbler): 5.5%
Candidate 6 (Ivory Gull): 0.0%
Candidate 7 (Painted Bunting): 12.4%
Back color: grey [observer certainty: definitely]
Candidate 1 (Chuck will Widow): 21.7%
Candidate 2 (Green Jay): 1.9%
Candidate 3 (White Pelican): 4.1%
Candidate 4 (Whip poor Will): 30.1%
Candidate 5 (Hooded Warbler): 35.5%
Candidate 6 (Ivory Gull): 6.8%
Candidate 7 (Painted Bunting): 12.7%
Tail shape: fan shaped tail [observer certainty: definitely]
Candidate 1 (Chuck will Widow): 34.7%
Candidate 2 (Green Jay): 0.0%
Candidate 3 (White Pelican): 0.0%
Candidate 4 (Whip poor Will): 16.4%
Candidate 5 (Hooded Warbler): 1.0%
Candidate 6 (Ivory Gull): 21.7%
Candidate 7 (Painted Bunting): 8.9%
Upper tail color: orange [observer certainty: definitely]
Candidate 1 (Chuck will Widow): 7.8%
Candidate 2 (Green Jay): 0.0%
Candidate 3 (White Pelican): 0.0%
Candidate 4 (Whip poor Will): 0.0%
Candidate 5 (Hooded Warbler): 0.0%
Candidate 6 (Ivory Gull): 0.0%
Candidate 7 (Painted Bunting): 25.2%
Head pattern: masked [observer certainty: definitely]
Candidate 1 (Chuck will Widow): 0.0%
Candidate 2 (Green Jay): 8.6%
Candidate 3 (White Pelican): 21.1%
Candidate 4 (Whip poor Will): 3.3%
Candidate 5 (Hooded Warbler): 28.6%
Candidate 6 (Ivory Gull): 1.2%
Candidate 7 (Painted Bunting): 6.0%
Breast color: yellow [observer certainty: definitely]
Candidate 1 (Chuck will Widow): 0.0%
Candidate 2 (Green Jay): 44.7%
Candidate 3 (White Pelican): 2.4%
Candidate 4 (Whip poor Will): 0.0%
Candidate 5 (Hooded Warbler): 79.0%
Candidate 6 (Ivory Gull): 0.0%
Candidate 7 (Painted Bunting): 8.9%
Throat color: buff [observer certainty: definitely]
Candidate 1 (Chuck will Widow): 24.0%
Candidate 2 (Green Jay): 0.0%
Candidate 3 (White Pelican): 3.8%
Candidate 4 (Whip poor Will): 30.7%
Candidate 5 (Hooded Warbler): 0.0%
Candidate 6 (Ivory Gull): 5.1%
Candidate 7 (Painted Bunting): 2.0%
Eye color: orange [observer certainty: definitely]
Candidate 1 (Chuck will Widow): 2.8%
Candidate 2 (Green Jay): 0.0%
Candidate 3 (White Pelican): 0.0%
Candidate 4 (Whip poor Will): 0.0%
Candidate 5 (Hooded Warbler): 0.0%
Candidate 6 (Ivory Gull): 0.0%
Candidate 7 (Painted Bunting): 6.3%
Bill length: longer than head [observer certainty: definitely]
Candidate 1 (Chuck will Widow): 2.3%
Candidate 2 (Green Jay): 0.0%
Candidate 3 (White Pelican): 97.8%
Candidate 4 (Whip poor Will): 3.2%
Candidate 5 (Hooded Warbler): 5.9%
Candidate 6 (Ivory Gull): 2.0%
Candidate 7 (Painted Bunting): 5.8%
Forehead color: grey [observer certainty: definitely]
Candidate 1 (Chuck will Widow): 3.0%
Candidate 2 (Green Jay): 5.7%
Candidate 3 (White Pelican): 2.3%
Candidate 4 (Whip poor Will): 36.5%
Candidate 5 (Hooded Warbler): 5.5%
Candidate 6 (Ivory Gull): 5.5%
Candidate 7 (Painted Bunting): 5.8%
Under tail color: brown [observer certainty: definitely]
Candidate 1 (Chuck will Widow): 82.4%
Candidate 2 (Green Jay): 0.0%
Candidate 3 (White Pelican): 1.1%
Candidate 4 (Whip poor Will): 47.6%
Candidate 5 (Hooded Warbler): 9.6%
Candidate 6 (Ivory Gull): 1.3%
Candidate 7 (Painted Bunting): 23.3%
Nape color: brown [observer certainty: definitely]
Candidate 1 (Chuck will Widow): 76.9%
Candidate 2 (Green Jay): 0.0%
Candidate 3 (White Pelican): 0.0%
Candidate 4 (Whip poor Will): 41.1%
Candidate 5 (Hooded Warbler): 5.2%
Candidate 6 (Ivory Gull): 0.0%
Candidate 7 (Painted Bunting): 2.0%
Belly color: brown [observer certainty: definitely]
Candidate 1 (Chuck will Widow): 81.1%
Candidate 2 (Green Jay): 0.0%
Candidate 3 (White Pelican): 0.0%
Candidate 4 (Whip poor Will): 43.6%
Candidate 5 (Hooded Warbler): 1.8%
Candidate 6 (Ivory Gull): 0.0%
Candidate 7 (Painted Bunting): 8.0%
Wing shape: tapered wings [observer certainty: definitely]
Candidate 1 (Chuck will Widow): 4.9%
Candidate 2 (Green Jay): 2.7%
Candidate 3 (White Pelican): 29.6%
Candidate 4 (Whip poor Will): 26.6%
Candidate 5 (Hooded Warbler): 6.2%
Candidate 6 (Ivory Gull): 21.6%
Candidate 7 (Painted Bunting): 14.3%
References
- [1]Abdin, M., Aneja, J., Behl, H., Bubeck, S., Eldan, R., Gunasekar, S., Harrison, M., Hewett, R. J., Javaheripi, M., Kauffmann, P., et al. (2024). Phi-4 technical report. arXiv preprint arXiv:2412.08905.arxiv.org/abs/2412.08905
- [2]Argenziano, G., Fabbrocini, G., Carli, P., De Giorgi, V., Sammarco, E., and Delfino, M. (1998). Epiluminescence microscopy for the diagnosis of doubtful melanocytic skin lesions: comparison of the abcd rule of dermatoscopy and a new 7-point checklist based on pattern analysis. Archives of dermatology, 134(12):1563–1570.pubmed.ncbi.nlm.nih.gov/9875194
- [3]Breiman, L. (2001). Random forests. Machine learning, 45(1):5–32.
- [4]Chen, L., de Melo, G., Suchanek, F., and Varoquaux, G. (2026). Query-level uncertainty in large language models. In International Conference on Learning Representations, volume 2026, pages 34353–34372.arxiv.org/abs/2506.09669
- [5]Chen, Z., Wang, W., Cao, Y., Liu, Y., Gao, Z., Cui, E., Zhu, J., Ye, S., Tian, H., Liu, Z., et al. (2024a). Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271.arxiv.org/abs/2412.05271
- [6]Chen, Z., Wang, W., Tian, H., Ye, S., Gao, Z., Cui, E., Tong, W., Hu, K., Luo, J., Ma, Z., et al. (2024b). How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821.arxiv.org/abs/2404.16821
- [8]Feng, J., Kothari, A., Zier, L., Singh, C., and Tan, Y. S. (2026). Bayesian concept bottleneck models with llm priors. Advances in Neural Information Processing Systems, 38:93889–93920.arxiv.org/abs/2410.15555
- [9]Ghanbari, M. and Dividino, R. (2026). When do llms listen? confidence-guided knowledge acceptance in llms. In Bouzar-Benlabiod, L. and Leung, C., editors, Proceedings of the The 39th Canadian Conference on Artificial Intelligence, volume 318 of Proceedings of Machine Learning Research, pages 1108–1115.
- [10]Grattafiori, A., Dubey, A., Jauhari, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024). The llama 3 herd of models. arXiv preprint arXiv:2407.21783.arxiv.org/abs/2407.21783
- [12]Huang, Y., Chen, S., Cai, H., and Dhingra, B. (2025). To trust or not to trust? enhancing large language models’ situated faithfulness to external contexts. In International Conference on Learning Representations, volume 2025, pages 8625–8654.arxiv.org/abs/2410.14675
- [13]Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., DasSarma, N., Tran-Johnson, E., et al. (2022). Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221.arxiv.org/abs/2207.05221
- [14]Kang, S., Bakman, Y. F., Yaldiz, D. N., Buyukates, B., and Avestimehr, S. (2025). Uncertainty quantification for hallucination detection in large language models: Foundations, methodology, and future directions. IEEE BITS the Information Theory Magazine, 5(1):3–19.arxiv.org/abs/2510.12040
- [15]Kawahara, J., Daneshvar, S., Argenziano, G., and Hamarneh, G. (2019). Seven-point checklist and skin lesion classification using multitask multimodal neural nets. IEEE Journal of Biomedical and Health Informatics, 23(2):538–546.DOI
- [16]Kim, E., Jung, D., Park, S., Kim, S., and Yoon, S. (2023). Probabilistic concept bottleneck models. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J., editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 16521–16540. PMLR.arxiv.org/abs/2306.01574
- [17]Koh, P. W., Nguyen, T., Tang, Y. S., Mussmann, S., Pierson, E., Kim, B., and Liang, P. (2020). Concept bottleneck models. In III, H. D. and Singh, A., editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 5338–5348. PMLR.arxiv.org/abs/2007.04612
- [18]Kuhn, L., Gal, Y., and Farquhar, S. (2023). Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302.09664.
- [19]Kumaran, D., Daw, N., Osindero, S., Velickovic, P., and Patraucеan, V. (2026). Causal evidence that language models use confidence to drive behaviour. Nature Machine Intelligence, pages 1–15.DOI
- [21]Liu, H., Dou, Z.-Y., Wang, Y., Peng, N., and Yue, Y. (2024). Uncertainty calibration for tool-using language agents. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N., editors, Findings of the Association for Computational Linguistics: EMNLP 2024, pages 16781–16805, Miami, Florida, USA. Association for Computational Linguistics.DOI
- [22]Liu, X., McDuff, D., Kovacs, G., Galatzer-Levy, I., Sunshine, J., Zhan, J., Poh, M.-Z., Liao, S., Di Achille, P., and Patel, S. (2023). Large language models are few-shot health learners. arXiv preprint arXiv:2305.15525.arxiv.org/abs/2305.15525
- [23]Lymperopoulos, P. and Sarathy, V. (2025a). Tools in the loop: Quantifying uncertainty of llm question answering systems that use tools. In Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems, AAMAS ’25, page 2645–2647, Richland, SC. International Foundation for Autonomous Agents and Multiagent Systems.arxiv.org/abs/2505.16113
- [25]Menon, S. and Vondrick, C. (2023). Visual classification via description from large language models. In The Eleventh International Conference on Learning Representations.arxiv.org/abs/2210.07183
- [26]Moor, M., Banerjee, O., Abad, Z. S. H., Krumholz, H. M., Leskovec, J., Topol, E. J., and Rajpurkar, P. (2023). Foundation models for generalist medical artificial intelligence. Nature, 616(7956):259–265.DOI
- [27]Nafar, A., Venable, K. B., and Kordjamshidi, P. (2025). Reasoning over uncertain text by generative large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 24911–24920.
- [28]Park, S., Mun, J., Oh, D., and Lee, N. (2026). An analysis of concept bottleneck models: Measuring, understanding, and mitigating the impact of noisy annotations. Advances in Neural Information Processing Systems, 38:5297–5331.arxiv.org/abs/2505.16705
- [29]Patterson, G. and Hays, J. (2012). Sun attribute database: Discovering, annotating, and recognizing scene attributes. In 2012 IEEE conference on computer vision and pattern recognition, pages 2751–2758. IEEE.DOI
- [30]Soudani, H., Kanoulas, E., and Hasibi, F. (2025). Why uncertainty estimation methods fall short in RAG: An axiomatic analysis. In Che, W., Nabende, J., Shutova, E., and Piliehvаr, M. T., editors, Findings of the Association for Computational Linguistics: ACL 2025, Vienna, Austria. Association for Computational Linguistics.arxiv.org/abs/2505.07459
- [31]Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. (2011). The caltech-ucsd birds-200-2011 dataset.
- [32]Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., Fan, Y., Dang, K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., and Lin, J. (2024a). Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191.arxiv.org/abs/2409.12191
- [33]Wang, W., Chen, Z., Wang, W., Cao, Y., Liu, Y., Gao, Z., Zhu, J., Zhu, X., Lu, L., Qiao, Y., and Dai, J. (2024b). Enhancing the reasoning ability of multimodal large language models via mixed preference optimization. arXiv preprint arXiv:2411.10442.arxiv.org/abs/2411.10442
- [35]Xiong, M., Hu, Z., Lu, X., Li, Y., Fu, J., He, J., and Hooi, B. (2024). Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. In International Conference on Learning Representations, volume 2024, pages 23650–23678.arxiv.org/abs/2306.13063
- [36]Xuan, W., Zeng, Q., Qi, H., Wang, J., and Yokoya, N. (2025). Seeing is believing, but how much? a comprehensive analysis of verbalized calibration in vision-language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 1408–1450.arxiv.org/abs/2505.20236
- [37]Yadkori, Y. A., Kuzborskij, I., György, A., and Szepesvari, C. (2024). To believe or not to believe your llm: Iterative prompting for estimating epistemic uncertainty. Advances in Neural Information Processing Systems, 37:58077–58117.
- [38]Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. (2025). Qwen3 technical report. arXiv preprint arXiv:2505.09388.
- [39]Zhi, Z., Feng, C., Daneshmand, A., Orlu, M., Demosthenous, A., Yin, L., Li, D., Liu, Z., and Rodrigues, M. R. (2025a). Seeing and reasoning with confidence: Supercharging multimodal llms with an uncertainty-aware agentic framework. arXiv preprint arXiv:2503.08308.arxiv.org/abs/2503.08308
- [40]Zhi, Z., Wu, Q., Shen, M., Li, W., Li, Y., Shao, K., and Zhou, K. (2025b). Videoagent2: Enhancing the LLM-based agent system for long-form video understanding by uncertainty-aware cot. In Workshop on Scaling Environments for Agents.arxiv.org/abs/2504.04471
- [41]Zhuo, J., Zhang, S., Fang, X., Duan, H., Lin, D., and Chen, K. (2024). Prosa: Assessing and understanding the prompt sensitivity of llms. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1950–1976.