betaFeedback

WorksComputer Science

In the Eye of the Beholder: Human Label Variation in Textual and Visual Entailment

Abstract

Human Label Variation (HLV) is increasingly recognised as an informative signal that reflects meaningful differences in how people interpret the same input. While HLV has been studied extensively in Natural Language Inference (NLI), much less is known about HLV in Visual Entailment (VE; Xie et al., 2019), where the textual premise is replaced by an image, keeping the hypothesis and label space fixed. We introduce VariNLI-VE, a dataset for studying how HLV changes between paired NLI and VE instances. We re-annotate a subset of originally neutral SNLI (Bowman et al., 2015) instances and their visual counterparts, with the same annotators evaluating both the textual and visual versions. Each annotation includes a label, confidence score, and explanation, yielding 2,250 annotations. Labels and explanations are collected jointly, providing ecologically valid explanations. Because the image may reveal information not stated in the text, one might expect disagreement to decrease in VE. Instead, we find no clear overall reduction: disagreement decreases for some instances while it increases for others. Explanations reveal further variation, as annotators can retain the same label while changing their rationale. When evaluating vision-language models (VLMs) on the paired data, we find lower accuracy, weaker alignment with human label distributions, and generally higher uncertainty in VE than in NLI.

Introduction

Many Natural Language Processing tasks assume that each input is associated with a single correct label. In practice, however, annotators often disagree, giving rise to Human Label Variation (HLV; Plank [32]). Rather than being noise, such variation can reflect ambiguity, different interpretations, and multiple valid readings (Aroyo and Welty [3]; Uma et al. [37]). While substantial prior work has studied variation and disagreement in the Natural Language Inference (NLI) task (Pavlick and Kwiatkowski [30]; Nie et al. [29]; Jiang et al. [22], inter alia), human inference is not restricted to textual evidence. In the Visual Entailment (VE) task (Xie et al. [42]), the premise is presented as an image rather than text, while the hypothesis and label space remain unchanged. However, HLV in VE remains underexplored. We therefore use VE as a test bed to study how HLV changes from NLI to VE when the same annotators judge the same hypothesis under textual and visual premises.

This comparison is particularly interesting because an image may provide evidence that is left unstated in a textual premise. Such additional evidence could resolve potential underspecification in the text, narrow the range of plausible interpretations, and reduce disagreement. Figure 1 illustrates such a case: the textual premise elicits disagreement among annotators, while evidence available in the corresponding image leads all annotators to unanimously assign the contradiction label.

Figure 1. Setup and analyses. For each VariNLI-VE instance, we collect five annotations, including a label (E, N, C), confidence score (1-6), and explanation from the same annotators across NLI and VE tasks, and compare with model predictions and uncertainty estimates.

To study this evidence shift in a paired setting, we introduce VARINLI-VE, a dataset of 2,250 new annotations across NLI and VE, constructed from originally neutral SNLI instances (Bowman et al., [7]). Since SNLI premises originate from image captions, each textual premise can be replaced with its corresponding source image. Each version of an NLI–VE pair receives five annotations consisting of a label, a natural-language explanation, and a self-reported confidence score. Crucially, the same five annotators judge both the textual and visual versions, enabling direct comparison across the two settings. Since labels and explanations are collected jointly, the dataset provides ecologically valid explanations (Jiang et al., [22]) that are tied directly to the judgments they explain. To the best of our knowledge, VARINLI-VE is the first dataset to provide multiple annotations, including explanations, from the same individuals for paired NLI and VE instances.

We further ask whether vision-language models (VLMs) reflect the human disagreement and uncertainty observed across NLI and VE. We evaluate a range of VLMs on the same paired instances, comparing their accuracy and predictive distributions with human judgments, and their uncertainty with human disagreement and reported confidence.

We make the following contributions:

  • We release VARINLI-VE, a dataset of 2,250 paired NLI–VE annotations for originally neutral SNLI instances, including labels, explanations, and confidence scores from the same annotators across textual and visual premises.1

  • Using this paired setup, we analyze how human label distributions, explanations, and confidence change between NLI and VE.

  • We evaluate how VLM predictions and uncertainty change across the paired tasks, and to what extent they align with human judgments.

Contrary to the expectation that visual evidence would reduce disagreement, we find no clear overall decrease from NLI to VE. Instead, disagreement decreases for some instances and increases for others. Human confidence remains generally high in both settings, suggesting that disagreement can reflect confidently held competing interpretations.

Explanations add another layer to the HLV analysis: even when an annotator assigns the same label in NLI and VE, the rationale for that judgment can change, revealing variation that label distributions alone do not capture. For models, we find consistently lower accuracy and weaker alignment with human judgment distributions in VE compared to NLI, while model uncertainty is generally higher and only weakly tracks human uncertainty.

Related Work

Human Label Variation & NLI

HLV (Plank, [32]) arises when multiple annotators assign different labels to the same instance. Rather than treating such disagreement as noise, prior work argues that it can reflect meaningful variation in human judgments due to ambiguity, plausible differences in interpretation, or annotator-specific factors ([3, 30, 21]). This has motivated work that preserves disagreement, as models trained on such data better reflect the range of human interpretations ([37, 14]). The ChaosNLI dataset (Nie et al., [29]) shows that HLV is widespread in NLI by collecting annotations from a large number of annotators. Explanations can further reveal reasoning behind disagreement: the e-SNLI dataset (Camburu et al., [8]) adds natural language explanations to SNLI (Bowman et al., [7]), but these were collected post hoc and from annotators who differ from those providing the labels. LiveNLI (Jiang et al., [22]) and VariErr NLI (Weber-Genzel et al., [40]) instead collect ecologically valid explanations, where annotators provide labels and explanations jointly, enabling variation to be studied beyond labels alone.

VQA & Answer Diversity

Annotator variation and diversity also arise in Visual Question Answering (VQA), where question-image pairs may produce different responses due to ambiguity, subjectivity, or difficulty ([16, 23, 44]). Several VQA datasets collect multiple human answers per instance ([2, 15, 17, 28]). Existing approaches have investigated this variation through eye-tracking, linking disagreement to different gaze patterns (Hindennach et al., [18]) or through correlations between VLMs and human answer distributions [26], finding that even the best-performing models struggle to approximate human distributions. While these prior VQA works establish disagreement and diversity as important phenomena in multimodal tasks, we study HLV across paired textual and visual evidence, using the same annotators in both settings.

From NLI to Visual Entailment

VE extends NLI to a multimodal setting by replacing the textual premise with an image while keeping the hypothesis and label space fixed [42].2 This is possible for the SNLI dataset [7], since its premises are derived from Flickr30k image captions [46] and can therefore be replaced with the corresponding images. The e-SNLI-VE dataset [24] further adds natural language explanations from e-SNLI [8]. At the same time, replacing captions with images can change the information available to annotators, and Kayser et al. [24] address resulting label mismatches by re-annotating neutral examples of the e-SNLI-VE development and test sets. These resources make VE a natural setting for studying HLV under a caption-to-image shift. Yet HLV is not a primary focus of existing VE resources, and explanations are only partially ecologically valid. While re-annotated neutral instances receive new explanations and labels, most explanations are inherited from post-hoc explanations from e-SNLI. Thus, existing resources do not provide multiple labels and explanations for paired NLI-VE instances annotated by the same individuals. Our work is a first step towards addressing this gap.

VariNLI-VE

We construct VariNLI-VE from 225 SNLI instances [7] and their source images. Each instance yields two annotation items that share the same hypothesis: an NLI item with the original textual premise and a VE item with the corresponding image. Each version receives five annotations consisting of a label, a natural-language explanation, and a self-reported confidence score. Crucially, the same five annotators judge both the textual and visual versions of each instance, yielding 2,250 annotations in total (225 instances ×\times 2 versions ×\times 5 annotators). Confidence is expressed on a six-point scale, from 1 (lowest) to 6 (highest). Labels, explanations, and confidence scores are collected jointly, making the explanations ecologically valid [22]. Item order is randomized during annotation, while ensuring that each NLI item precedes its corresponding VE item to reduce potential priming from the visual evidence.

We focus on SNLI instances that were originally labeled neutral. This choice is motivated by Kayser et al. [24], who report that neutral instances are particularly sensitive to replacing textual premises with images. To capture different outcomes of this caption-to-image shift, we use the e-SNLI-VE labels [24] to select our 225 instances, balancing them across entailment, neutral, and contradiction in VE. Thus, all instances share a neutral starting point in SNLI, but differ in how they are labeled when the premise is visual.

Because studying HLV requires multiple judgments, and because our paired comparison requires the same annotators across textual and visual premises, we newly annotate all NLI and VE items. Consequently, the majority labels in VariNLI-VE may differ from the original SNLI and e-SNLI-VE labels. Figure 2 summarizes the resulting label structure, tracing each instance from its original SNLI label to the majority labels from our new NLI and VE annotations. Most instances remain neutral after NLI re-annotation, while 31 shift to entailment. From NLI to VE, 51.1% of instances change majority label and 48.9% retain it. Ties denote cases in which no label receives a majority among the five annotators.

Figure 2. Label transitions from the original SNLI label to our newly derived NLI and VE majority labels in VariNLI-VE. Ties indicate cases with no majority label among five annotators.

Further details on the annotation procedure, annotator pool, and annotation guidelines are provided in Appendix § 11, while comparisons with the original e-SNLI-VE labels are reported in Appendix § 15.

Experimental Setup

Models

We evaluate 13 VLMs from five model families on both NLI and VE, including Gemma 3 Team et al., [35], InternVL3.5 Wang et al., [39], LLaVA-OneVision-1.5 An et al., [1], Molmo 2 Clark et al., [11], and Qwen3-VL Bai et al., [6], with model sizes ranging from 2B to 27B parameters. To contextualize VLM performance on the text-only NLI task, we additionally evaluate Qwen3 Yang et al., [43] as a text-only LLM on the NLI task and report results in Appendix § 18.§ 1.

Prompting and Prediction Extraction

We evaluate models using a zero-shot multiple-choice prompting setup. To reduce position and option-letter bias [31, 47], we randomize the mapping between answer letters and labels for each instance. Example prompts are provided in Appendix § 13. For prediction extraction, let Y={Ent,Neu,Con}Y = \{\mathrm{Ent}, \mathrm{Neu}, \mathrm{Con}\} denote the label set. For each instance ii, we extract the raw next-token logits for the three answer option letters and map them back to their corresponding labels. This yields label-level scores zi=[ziEnt,ziNeu,ziCon]\mathbf{z}_i = [z_i^{\mathrm{Ent}}, z_i^{\mathrm{Neu}}, z_i^{\mathrm{Con}}], whose softmax-normalized values define the model judgment distribution pip_i over YY. The predicted label is the label with the highest probability.

Evaluation Metrics

We summarize the metrics used for the human and model analyses below. Formal definitions and further details are provided in Appendix § 14.

Human Analysis. We quantify disagreement using entropy over human label distributions. To measure how human judgments change between NLI and VE, we compare the corresponding label distributions using total variation distance (TVD) and Jensen-Shannon distance (JSD), following prior work [4, 29, 10, 9]. We further analyze human uncertainty using annotator self-reported confidence.

Model Analysis. We report both accuracy against the human majority label, and distributional alignment metrics (TVD and JSD). We further analyze model uncertainty using normalized predictive entropy and conformal prediction with the Least Ambiguous set-valued Classifier (LAC), following Ye et al. [45]. This method returns a calibrated set of plausible labels, with larger sets indicating higher uncertainty.

In our entailment task, the set size ranges from 1 to 3. Finally, we compute Spearman correlations between model and human uncertainty.

Human Label Variation across Textual and Visual Entailment

Disagreement Restructuring

To examine how disagreement changes when textual premises are replaced with visual premises, we compare annotation patterns between NLI and VE. We group instances by their five-annotator label distribution: unanimous (5-0-0), near-unanimous (4-1-0), split between two labels (3-2-0), split between three labels (3-1-1), and tied (2-2-1). The transition matrix in Figure 3 shows that many instances remain in similar agreement regions across modalities, especially in the high-agreement area. However, the off-diagonal cells show movement in both directions: items to the left of the diagonal become more agreed upon in VE, while items to the right become less agreed upon.

Figure 3. Agreement-pattern transitions NLI ↔ VE.

Table 1 summarizes the changes using human label entropy and cross-modal distributional distances. At the aggregate level, we find no clear evidence that visual premises systematically increase or decrease disagreement, as the difference in mean human label entropy between NLI and VE is small and not statistically distinguishable from zero. Importantly, at the instance level, TVD and JSD scores indicate substantial changes in how judgments are distributed across labels.3 Not every distributional change, however, constitutes a change in disagreement. An instance may, for example, switch unanimously from one label to another while remaining fully agreed upon in both modalities. We therefore define disagreement as restructured, see Table 2, when either (i) its amount changes across modalities, measured by a change in item-level entropy, or (ii) disagreement is present in both modalities and its label composition changes despite an unchanged entropy level. Under this operationalization, we observe that 170 of 225 instances (75.6%) exhibit disagreement restructuring. For 12%, the label distribution remains unchanged across modalities, while for the other 12% labels flips unanimously.

MetricMean ± Std.95% CI
Entropy (NLI)0.47 ± 0.47[0.41, 0.53]
Entropy (VE)0.54 ± 0.49[0.48, 0.60]
Δ entropy (VE−NLI)0.07 ± 0.64[−0.01, 0.15]
TVD (VE−NLI)0.48 ± 0.34[0.44, 0.53]
JSD (VE−NLI)0.52 ± 0.32[0.48, 0.57]

Table 1. Human disagreement and NLI–VE distributional shifts with 95% paired bootstrap confidence intervals. Values are mean ± standard deviation.

What changes between NLI and VE?Operational criterionnnDisagreement restructuring?
D↓D \downarrow: Disagreement decreases in VEHVE<HNLIH_{\mathrm{VE}} < H_{\mathrm{NLI}}63Yes
D↑D \uparrow: Disagreement increases in VEHVE>HNLIH_{\mathrm{VE}} > H_{\mathrm{NLI}}85Yes
D↔D \leftrightarrow: Disagreement composition changesHNLI=HVE>0H_{\mathrm{NLI}} = H_{\mathrm{VE}} > 0 and 0<TVD<10 < \mathrm{TVD} < 122Yes
Unanimous label changesHNLI=HVE=0H_{\mathrm{NLI}} = H_{\mathrm{VE}} = 0 and TVD=1\mathrm{TVD} = 127No
Label distribution unchangedHNLI=HVEH_{\mathrm{NLI}} = H_{\mathrm{VE}} and TVD=0\mathrm{TVD} = 028No

Table 2. Operationalization of disagreement restructuring across paired NLI–VE instances (n=225n = 225). We consider disagreement restructured when its amount changes, or when the amount remains the same but the label distributions differ. HH denotes human label entropy, while nn indicates item count. TVD is total variation distance.

HLV through the Lens of Explanations

Label distributions capture changes in assigned labels, but not the reason behind them. Because in VARINLI-VE the same annotators provide explanations, our dataset allows us to examine whether the same label is supported by a similar reason across the textual (NLI) and visual premises (VE). For the 170 instances exhibiting disagreement restructuring, we manually classify all 850 paired NLI–VE explanations (170 instances ×\times 5 annotators) by whether the label and rationale remain the same or change. One author classified all 850 pairs, while a second independently classified 101 of these pairs to assess reliability (89.1% agreement; Cohen’s κ=0.74\kappa= 0.74). Pairs were marked unclear when explanations were too brief or generic to determine a rationale change.

Figure 4 summarizes the resulting patterns. As expected, label changes are usually accompanied by changes in rationale, accounting for 414 pairs (in red), with only a few exceptions. Another 234 keep both the same label and a similar rationale. More notably, 157 pairs (light green) retain the same label but change rationale. These label-preserving rationale shifts show that changes in the available evidence can alter an annotator’s rationale without changing their label. This echoes within-label variation observed in text-based NLI ([22, 19]), but here it occurs within the same annotator across modalities. This shows the importance of explanation, as they reveal variation that remains invisible at the label level. Examples of all patterns are shown in Appendix § 17.

Figure 4. Explanation analysis for instances exhibiting disagreement restructuring. Numbers indicate explanation-pair counts. D↓D \downarrow: Disagreement decreases in VE; D↑D \uparrow: Disagreement increases in VE; D↔D \leftrightarrow: Disagreement composition changes.

Confidence and Disagreement

While entropy captures between-annotator disagreement, self-reported confidence captures annotators’ subjective certainty in their own judgments. This raises the question whether disagreement is accompanied by lower confidence, and how confidence is affected by visual grounding. Table 3 shows that human confidence (ranging from 1-6) is generally high in both NLI and VE and similar across the two settings within the same agreement pattern. Replacing the textual premise with an image therefore does not appear to make substantially more confident. Instead, confidence varies more clearly with the degree of agreement. Unanimously labeled instances receive the highest confidence scores in both settings, while divisive patterns obtain lower scores. At the same time, confidence remains relatively high even for split cases, suggesting that HLV can reflect confidently held competing interpretations. We therefore treat confidence and disagreement as related but complementary signals of human uncertainty.

PatternNLIVEnn (NLI/VE)
5-0-05.39 ± 0.495.41 ± 0.54105/92
4-1-04.94 ± 0.605.00 ± 0.5268/68
3-2-04.67 ± 0.584.76 ± 0.4439/46
3-1-14.82 ± 0.414.69 ± 0.409/11
2-2-14.75 ± 0.774.62 ± 0.684/8

Table 3. Mean confidence by annotation pattern in NLI and VE. Values are mean ± standard deviation on a six-point confidence scale, while nn indicates item count.

Model Behavior across Textual and Visual Entailment

Performance and Prediction Patterns

Accuracy consistently drops from NLI to VE. We first evaluate each model against the majority label from our new NLI and VE annotations. Figure 5 (left panel) shows consistently higher accuracy for NLI than VE across model families. The strongest models reach high accuracy on NLI, with InternVL3.5-4B and Qwen3VL-8B both achieving above 90% accuracy against the human majority label. In contrast, Molmo2-8B only reaches 61.8%

Figure 5. Model performance and uncertainty across NLI and VE. The left half shows accuracy with respect to human majority labels across all instances and on the neutral-only paired instances. The right half shows model uncertainty, measured by normalized predictive entropy and LAC conformal prediction set size, with larger LAC sets indicating higher uncertainty. Accuracy decreases from NLI to VE, while uncertainty generally increases.

accuracy as the best performing VE model. To better understand this large gap, we further break accuracy down by VE majority label. Table 4 shows that models perform well on entailment, with several models scoring close or above 90%. However, performance is lower on neutral instances and drops severely for contradiction. This suggests that current VLMs are effective at identifying visual evidence that supports a hypothesis, but less reliable at detecting visual evidence that contradicts it. This pattern is further supported by the aggregate model prediction distribution (App. Figure 16), showing that models underpredict contradiction and instead assign a larger share to entailment or neutral.

ModelNLIVEVE (by label)
OverallEntNeuCon
Gemma-3-4B71.545.287.755.63.8
Gemma-3-12B79.654.887.751.934.2
Gemma-3-27B82.855.387.753.134.2
InternVL3.5-2B81.059.493.053.141.8
InternVL3.5-4B91.454.889.569.115.2
InternVL3.5-8B89.658.594.765.425.3
LLaVA-OneVision-4B85.151.6100.049.419.0
LLaVA-OneVision-8B87.355.898.253.127.8
Molmo2-4B90.055.384.276.517.2
Molmo2-8B88.761.887.782.721.5
Molmo2-O-7B84.255.393.060.522.8
Qwen3VL-2B70.153.096.542.032.9
Qwen3VL-8B91.460.480.784.021.5

Table 4. Model accuracies across modalities and by VE majority label. Cells are shaded according to decrease in accuracy wrt NLI; darker cells indicate larger drops.

The modality gap remains for neutral cases. Since our NLI subset is neutral-focused while the corresponding VE annotations have a more balanced distribution, the NLI-VE accuracy drop may partly reflect this label shift. To assess this, we examine the subset of paired instances whose VE majority label is neutral, and compare model accuracy with the corresponding NLI instances. As shown in the second plot in Figure 5, even on this controlled subset, all VLMs still perform worse in VE than in NLI, with drops ranging from 12.2% for Molmo2-4B to 41.9% for LLaVA-OneVision-4B. While this indicates that the modality gap cannot be explained by label shift alone, other factors, such as potential textual shortcuts in NLI or uneven reliance on textual versus visual evidence, may also contribute. Disentangling these factors is beyond the scope of the present analysis but remains an interesting direction for future work. Full per-model results are reported in Appendix § 18.§ 2, while Appendix I reports the 95% confidence intervals.

Accuracy decreases with disagreement. Figure 6 shows model accuracy by human annotator agreement pattern. Across modalities, accuracy is highest on unanimous or near-unanimous instances and lower on instances with greater disagreement, especially in VE. Additionally, VE accuracy remains lower than NLI accuracy across agreement patterns, indicating that the modality gap is not limited to highly ambiguous instances.

Figure 6. Per-model accuracy by annotator agreement.

Alignment with Human Judgments

So far, we have considered model behavior through aggregate prediction patterns, but these do not show whether model predictive distributions capture instance-level variation in human judgments. We therefore compare each model’s predicted label distributions to human judgment distributions to inspect whether models reflect HLV rather than only the majority vote. Following prior work ([4, 29, 9, 10]), we use TVD and JSD and report results in Table 5. Across all model families, model predictive distributions align more closely with human label distributions in NLI than in VE, with higher TVD and JSD in VE for every model. The best-aligned models are InternVL3.5-4B in NLI and InternVL3.5-8B in VE for both metrics. However, even the best VE alignment scores remain considerably lower compared to the best NLI scores. As with accuracy, this gap persists on the neutral-only subset. Even when we restrict the comparison to paired instances whose VE majority label is neutral, all VLMs have higher TVD and JSD in VE than on the corresponding NLI examples. A label-wise VE breakdown in Table 14 in the Appendix shows that divergence is highest for contradiction cases, consistent with the low accuracy and the underprediction of this label observed above.

ModelAll instancesNeutral-only instances
TVD NLI ↓\downarrowTVD VE ↓\downarrowJSD NLI ↓\downarrowJSD VE ↓\downarrowTVD NLI ↓\downarrowTVD VE ↓\downarrowJSD NLI ↓\downarrowJSD VE ↓\downarrow
Gemma-3-4B0.3250.5610.3670.5930.2780.4970.3190.541
Gemma-3-12B0.2680.4600.3090.5000.2450.4810.2860.528
Gemma-3-27B0.2450.4770.2890.5080.2040.4920.2480.531
InternVL3.5-2B0.2530.3920.3220.4340.2550.4540.3310.500
InternVL3.5-4B0.1850.4070.2440.4430.1480.3290.2110.382
InternVL3.5-8B0.1960.3880.2490.4260.1770.3410.2330.392
LLaVA-OneVision-4B0.2240.4300.3010.4700.2260.4320.3130.474
LLaVA-OneVision-8B0.2040.3990.2780.4400.2030.3980.2820.439
Molmo2-4B0.1910.4070.2530.4440.1880.2900.2570.346
Molmo2-8B0.2030.4010.2610.4460.1460.2870.2130.349
Molmo2-O-7B0.2240.4100.2870.4530.2190.3880.2780.441
Qwen3VL-2B0.3370.4470.3770.4810.2960.5170.3430.551
Qwen3VL-8B0.1970.4180.2480.4540.1440.2910.1970.352

Table 5. Alignment between model predictive distributions and human judgment distributions. Lower TVD and JSD indicate closer alignment. VE columns are shaded according to the increase over the corresponding NLI value within each block, with darker cells indicating a larger gap.

ModelTVD ↓ EntTVD ↓ NeuTVD ↓ ConJSD ↓ EntJSD ↓ NeuJSD ↓ Con
Gemma-3-4B0.2280.4970.8600.2900.5410.859
Gemma-3-12B0.1980.4810.6130.2600.5280.632
Gemma-3-27B0.2190.4920.6300.2650.5310.642
InternVL3.5-2B0.1390.4540.5000.1950.5000.536
InternVL3.5-4B0.1610.3290.6720.2220.3820.672
InternVL3.5-8B0.1420.3410.6080.2060.3920.619
LLaVA-OneVision-4B0.1210.4320.6560.2090.4740.662
LLaVA-OneVision-8B0.1350.3980.5970.2210.4390.607
Molmo2-4B0.1780.2900.6900.2500.3460.687
Molmo2-8B0.1790.2870.6710.2570.3490.680
Molmo2-O-7B0.1410.3880.6270.2260.4410.636
Qwen3VL-2B0.1420.5170.5900.2030.5510.607
Qwen3VL-8B0.2240.2910.6770.2650.3520.686

Table 14. VE label-wise alignment between model predictive distributions and human judgment distributions. Results are grouped by VE majority label. Lower TVD and JSD indicate closer alignment. Cells are shaded according to the distance value, with darker cells indicating weaker alignment.

Model Uncertainty

We first analyze model uncertainty across NLI and VE using normalized predictive entropy and LAC set size. As shown in Figure 5 (right panel), uncertainty is generally higher in VE: predictive entropy is higher for 11/13 models, with 95% CIs excluding zero for 10 models. For LAC set size, all 13 show higher uncertainty, with CIs excluding zero for 12 models. Full CIs are reported in Appendix I, and detailed model uncertainty scores in Appendix § 18.§ 4.

To assess whether models tend to be more uncertain on instances that humans also find more uncertain, we compute Spearman correlations with human label entropy and inverse confidence. Figure 7 shows that predictive entropy correlates only weakly with human uncertainty, with median ρ=0.235\rho= 0.235 in NLI and 0.1470.147 in VE. The associations are statistically significant for 88% of NLI correlations, but only 54% of VE correlations. LAC set size correlations, reported in Figure 17 in the Appendix, are similarly weak (71% significant in NLI; 50% in VE). Across both measures, model uncertainty only weakly tracks human uncertainty, with significant associations occurring less consistently in VE.

Figure 7. Spearman’s ρ\rho between model predictive entropy and human uncertainty measures (human label entropy and inverse mean confidence). Asterisks indicate statistically significant correlations (p < 0.05).

Discussion

We asked how HLV changes when a premise in NLI is replaced by an image, and whether VLMs reflect the resulting human judgment patterns. Our findings suggest that visual evidence does not simply resolve ambiguity. It can clarify information left underspecified in the text, but also introduce additional aspects of the scene that annotators interpret differently. Confidence provides a complementary view, as annotators can remain confident even in contested cases. Explanations reveal another layer of variation, showing that an annotator’s rationale can change even when the label remains the same.

These findings matter for HLV research, as a majority label cannot distinguish broad agreement from contested cases, while labels alone cannot capture changes in the rationale behind a judgment. We consider label distributions, confidence, and explanations across paired NLI and VE instances, allowing HLV to be studied at several levels rather than reducing it to disagreement over the final label. This is also important for model evaluation. VLMs align less closely with human judgment distributions in VE, while model uncertainty only weakly tracks human uncertainty. An evaluation based only on majority labels can hide whether a model captures the range of human judgments or merely predicts the dominant label.

More broadly, these findings motivate datasets that preserve meaningful variation, disagreement-aware evaluation, and systematic analyses of why judgments change from NLI to VE. Such work could also help identify cases where VLM predictions reflect or amplify social biases, contributing to more transparent and reliable VLMs.

Conclusion

We study HLV in a paired NLI–VE setting, where the same hypothesis is judged against either a textual premise or an image. We introduce VarNLI-VE, with 2,250 labels, explanations, and confidence scores from the same annotators for originally neutral SNLI instances. Contrary to the expectation that visual evidence would reduce disagreement, we find no clear overall decrease: disagreement decreases for some instances while increasing for others, and explanations reveal rationale changes even when labels remain unchanged. VLMs show lower accuracy, weaker alignment with human judgments, and generally higher uncertainty in VE. We hope our work encourages further research into why HLV changes across textual and visual evidence, and how these changes relate to model behavior.

Limitations

One limitation of our work is the scale of our dataset due to resource constraints and the preference to collect high-quality annotations without having to resort to crowdsourcing. While we collect 2,250 annotations, these cover 225 underlying instances with five judgments each for NLI and VE. A larger dataset with more annotators per instance would indeed provide a more stable estimate of HLV. Still, we believe our paired same-annotator design, with labels, confidence, and explanations for both NLI and VE, provides a useful resource for studying HLV across textual and visual evidence.

Another limitation is that we focus on originally neutral NLI instances, since these are the most sensitive to the introduction of visual evidence. This allows us to target cases where a caption-to-image shift in the premise is likely to matter, but it remains open whether similar HLV patterns occur for originally entailment or contradiction instances.

Furthermore, our analysis focuses mainly on aggregate judgment patterns and does not analyze annotator-level variation. Annotators may differ systematically in how they interpret labels or how they use confidence scores. While the released dataset provides the opportunity to investigate whether some annotators are more sensitive to visual evidence than others, we leave this question to future work.

Among our pool of 21 annotators, three were authors of this paper, although they never constituted a majority for any instance. We assess this potential source of bias through an author-exclusion analysis and find the main findings persisting even after excluding author-annotators (cf. Appendix § 16). Further, as described in Appendix § 11.§ 2, most annotators were non-native English speakers, although they generally acquired English early, reported high proficiency, and were allowed to look up unfamiliar terms. Future work could assess whether similar patterns hold with an annotator pool of native speakers.

Our explanation analysis provides an initial account of whether annotator rationales change across textual and visual premises, but does not aim to provide an exhaustive taxonomy of the sources and reasons of disagreement. A more in-depth analysis, as well as a comparison of human and model explanations, remain important future directions enabled by VARINLI-VE.

Ethical Considerations

We study HLV in a paired NLI and VE setup, with the goal of understanding how human judgments and model predictions change when a textual premise is replaced by an image. While we do not foresee direct harmful implications specific to our dataset or analyses, broader ethical considerations remain. Vision-language models may reproduce or amplify social biases present in their training data, which may affect both their predictions and uncertainty estimates. Despite this risk, we believe that explicitly studying, preserving, and modeling human label variation can have a positive downstream impact by informing the development of more transparent and less biased models.

We acknowledge the use of AI tools to support parts of the code development and writing refinement. All AI-generated outputs were carefully reviewed, and we confirm that we take full responsibility for the final content and results.

Acknowledgments

We thank the members of the MaiNLP lab for their insightful feedback and discussions, particularly Diego Frassinelli, Michael Hedderich, Mahdi Dhaini, Shijia Zhou and Andreas Säuberli. We are also grateful to the participants in the NLP course at the Free University of Bozen-Bolzano for their valuable contribution to the annotation of our dataset. Furthermore, we thank the anonymous reviewers for their constructive feedback, which helped improve this paper. We acknowledge funding through the ERC Consolidator Grant DIALECT 101043235, and the German Research Foundation (DFG) project VariVLM (586887264).

Annotation Details

Annotation Procedure

Before starting their annotation session, annotators were given a short oral briefing on the task, the corresponding labels entailment, contradiction, and neutral, as well as the annotation process in general. To ensure their consent and to be able to collect their metadata, they were asked to fill out an online form, using previously assigned annotator codes as identifiers. These codes facilitated individual annotator removal in case of faulty annotations or annotators not following instructions, ensuring the quality of annotations.

For annotators previously unfamiliar with annotation or entailment tasks, we offered the opportunity to explore the tool and task via a dedicated playground interface. This interface was designed to closely match the interface of the real annotation task. The instances presented in it were carefully selected to not overlap with the actual to-be-annotated instances, yet show a broad overview of different types of instances and give annotators an impression of what would be expected in the task.

In the playground interface, as well as in the real task that followed shortly after, written annotation guidelines were shown as soon as annotators opened the interface (for more details see A.3) and

were available to them throughout the whole duration of the task. Moreover, annotators were encouraged to raise any questions they may have and were explicitly allowed to look up any unknown terms they might encounter.

Each annotator was allocated a subset of our data, ensuring that each instance was labeled by exactly five annotators. Moreover, the subsets ensured that one annotator worked on both variants of a given NLI-VE instance, such that potential label shifts from the textual to the visual modality would remain as comparable as possible. To this end, the order of instances in each subset was pseudo-randomized. While the appearance of items was generally random, some restrictions were put in place: 1) NLI instances were always shown before their corresponding VE versions, as we assumed seeing the multi-modal VE variants prior to the textual ones might prime or bias annotators to favor interpretations more closely aligned with the VE ones. 2) Paired versions were not shown consecutively, again to reduce any potential priming or biasing effects of interpretations trickling from one variant to the other.

The annotation task consisted of three parts: 1) label, 2) explanation, and 3) confidence scores (see also Figure 10 and Figure 11). For 1) label, annotators were asked to choose a label out of entailment, contradiction, and neutral, which they then justified in 2) by providing a short rationale on why they chose said label. As we expected some genuinely ambiguous or hard-to-judge instances, especially in NLI, we additionally included 3) confidence scores. Here, annotators indicated on a scale from 1-6 how certain or uncertain they were about having chosen their provided label.

Annotation interface presenting a textual (NLI) instance

Figure 10. Annotation interface presenting a textual (NLI) instance.

Annotation interface presenting a visual (VE) instance

Figure 11. Annotation interface presenting a visual (VE) instance.

Annotator Metadata

Annotator metadata was collected via the online forms between the oral introduction and starting the annotation. In the form, annotators first provided written consent for participating in the study and their annotations to be used within the scope of research. In total, 22 annotators were recruited in this paper, out of which 14 were 1st year M.Sc. computer science students, who participated as part of a university course, and the rest were in-house annotators consisting of three authors of this paper and 5 hired PhD students and post-docs. After the removal of one annotator for poor performance, the remaining set consisted of 21 annotators in total, resulting in 2,250 annotations overall.

Figure 8 shows the distribution of native languages in our annotator sample. 7 annotators were native Italians, 5 native Germans and one was German-Italian. The remaining 8 annotators were from other non-English speaking backgrounds such as Arabic, Bengali, or Slovenian, or reported a mixed upbringing, such as English-Asante Twi.

Figure 8. Distribution of the first languages of the annotators who participated in this study.

Although most of the annotators employed in this study are not English natives, most reported learning English very early (mean age of acquisition (AoA): 7.71, std: 2.65; see Table 6). Additionally, their self-reported English proficiency scores, assessed via a 6-point scale across reading and writing skills, indicate a generally high proficiency among all annotators. As shown in Table 6, mean reading proficiency sits at 5.71 (std: 0.46, range: 5 − 6), while writing proficiency presents only marginally lower at 5.24 (std: 0.62, range: 4 − 6). In addition to self-reported proficiency measures generally being an appropriate approx-

MetricMeanStd.Range
AoA7.712.653 - 14
Reading Proficiency5.710.465 - 6
Writing Proficiency5.240.624 - 6

Table 6. Annotator metadata: Age of Acquisition (AoA) of English, as well as self-assessed English reading and writing proficiency of the annotators used in this study. We report mean, standard deviation (std) and range. Proficiency was measured on a 6-point scale with 1 indicating minimal and 6 high proficiency.

imation of actual language skills [5, 27, 41], annotators were also explicitly allowed to look up any terms they might not know, altogether suggesting sufficient language proficiency and resources to provide adequate annotations.

Annotation Guidelines

Figure 9 depicts the annotation guidelines given to annotators prior to starting their task. These guidelines explain the three labels entailment, contradiction, and neutral and give instructions on providing explanations and confidence scores. Further, annotators were provided with information on how to save their annotations, as well as how to get back to the guidelines if they ever needed to.

Annotation guidelines that were presented to annotators before starting their task and available to them throughout the session

Figure 9. Annotation guidelines that were presented to annotators before starting their task and available to them throughout the session.

The task

For each sample, you are asked to provide:

  • label (entailment, contradiction, or neutral)

  • short explanation for the label (1 short sentence)

  • confidence score (1-6)

Label

You will be asked to select a label based on the relationship between a premise and a hypothesis. The possible relationships between a premise and a hypothesis are:

  • Entailment: The hypothesis is definitely true given the premise.

  • Contradiction: The hypothesis is definitely false given the premise.

  • Neutral: The premise does not provide enough information to determine whether the hypothesis is definitely true or false.

Explanation

After choosing your label, we will ask you to give a short explanation for …

Confidence score

Then, please assign a score indicating how confident you feel about the selected label:

  • 1 = not confident at all

  • 6 = very confident

If you are done with an example, submit your annotations by clicking on <submit>. Please do not skip examples as all examples should be annotated.

Note: You can always come back to these instructions by clicking on "instructions" in the top right corner of your screen.

Annotation Interface

We used LabelStudio (Tkachenko et al., [36] (2025)) for its simple integration of both textual and visual content, while providing remote accessibility to annotators via server connectivity. As can be seen in Figure 10 and Figure 11, for both NLI and VE, the interface included the premise (either textual or visual), followed by the textual hypothesis. Annotators were then asked to provide a label, a rationale for picking the label, as well as a confidence score.

Qualitative Dataset Examples

We additionally conduct a small-scale exploratory qualitative analysis on selected samples, three of which can be found in Figure 18. Each bounding box represents one sample, where its left side shows the textual variant and the right side shows the visual one. Below each instance are the five annotations collected for it. To facilitate by-annotator comparisons across NLI and VE, annotations on the same line signify annotations given by the same annotator. Each annotation includes the label, confidence score, and explanation.

Figure 18. Three selected samples in both NLI and VE, complete with all five annotations. Each bounding box represents one sample. The left side shows the textual variant and the right side shows the visual one. Below each instance we show the five annotations collected for each sample. Annotations on the same line signify annotations given by the same annotator. Each annotation includes the label, confidence score, and explanation.

In the first example, both the textual and visual premise present a dog playing with a football, while the hypothesis to be judged states that the dog is happy. In both NLI and VE, the sample received both neutral (35\frac{3}{5}) and entailment labels (25\frac{2}{5}). Interestingly, individual annotators retained their labels across the two setups: annotators voting neutral in the NLI version also voted neutral in the VE version, and the same applies to the entailment label.

Looking at their explanations, annotators selecting entailment placed their vote on the assumption that a dog playing is most likely happy, while the ones that voted neutral rather focused on the fact that It is hard to say whether a dog is happy or not. As such, this instance seems to represent a particular subset of instances that we encountered in both NLI and VE samples: in these instances, the concept of interest is genuinely up for interpretation; e.g., how can one tell the emotional state of an animal?

The second example shows how an originally unanimously annotated sample in the textual variant (55\frac{5}{5} neutral) can turn into a more contentious one in the visual version (25\frac{2}{5} neutral, 25\frac{2}{5} contradiction, and 15\frac{1}{5} entailment). The hypothesis states that A hunter is posing with his dog, which, in combination with the NLI premise A man, a gun, and a dog results in a neutral label, with annotators pointing out not knowing whether the man is a hunter, whether he is posing, or whether the dog is truly his dog. This changes when the textual premise is replaced by the image, representing another kind of subset we observed: here, rather than taking in the whole image all at once, annotators seem to ”zoom in” and focus only on very specific aspects of it, and place their judgments solely based on this restricted view. In this instance, the annotator choosing entailment states that The man is hunting something and the dog is doing what a hunting dog does, focusing on the aspect of the man being a hunter, while passing over the posing aspect (i.e., while the man could very well be a hunter, that does not necessarily mean that he is posing). Annotators selecting contradiction, on the other hand, focus solely on the posing aspect, stating that, to them, The man is clearly not posing, but shooting. While the remainder of annotators chose a neutral label, raising similar questions and uncertainties as in the NLI version, it seems that some people may employ a cognitive simplification strategy here: when the interpretive space of the image feels too open, annotators may simplify it by focusing on specific aspects rather than ruminating over everything that could or could not be happening within or beyond the frame.

The third example is the only instance that did not converge to a single majority label in both modalities (NLI: 25\frac{2}{5} neutral, 25\frac{2}{5} entailment, and 15\frac{1}{5} contradiction, VE: 25\frac{2}{5} entailment, 25\frac{2}{5} contradiction, and 15\frac{1}{5} neutral). The textual and visual premises present Two large black dogs are snarling at each other and the hypothesis states that The pitbulls are snarling at one another. Interestingly, while the image and text seem quite aligned in terms of posited information, the different modes of presenting it seem to influence annotator judgments. Notably, in the visual variant, interpretation is particularly contingent on annotators’ knowledge about dog breeds and their appearance, with some annotators accepting the dogs present in the image as pitbulls, others stating that they [...] are not pitbulls, whereas others state that they [...] cannot confidently say if they are pitbulls.

Prompt Templates

The prompt templates used for the NLI and VE task are shown in Figure 12 and Figure 13, respectively. To avoid option and position bias [47, 31], we independently randomize, for each instance, the mapping between answer option letters {A,B,C}\{A, B, C\} and labels entailment, neutral, and contradiction, together with their corresponding definitions.

Prompt template used for natural language inference

Figure 12. Prompt template used for natural language inference.

Prompt template used for visual entailment

Figure 13. Prompt template used for visual entailment.

Details on Evaluation Metrics

Human Label Entropy. We quantify annotator disagreement within NLI and VE using Shannon entropy (Shannon, [33]) over the empirical human label distribution. For each example ii and modality m∈{NLI,VE}m \in\{\mathrm{NLI}, \mathrm{VE}\}, we obtain a distribution pimp_i^m by normalizing the counts of the five annotations over the label set Y={ENT,NEU,CON}\mathcal{Y} = \{\mathrm{ENT}, \mathrm{NEU}, \mathrm{CON}\}. We compute human label entropy as:

H(pim)=−∑y∈Ypim(y)log⁡2pim(y).H(p_i^m) = - \sum_{y \in\mathcal{Y}} p_i^m(y) \log_2 p_i^m(y).

Entropy is 0 when all annotators select the same label and increases as votes are distributed across labels. With five annotators and three labels, the maximum attainable entropy occurs for a 2–2–1 vote split and is approximately 1.522. We do not further normalize entropy values.

Jensen-Shannon Distance. We use Jensen-Shannon distance (Endres and Schindelin, [13]) to compare label distributions. Unlike KL divergence (Kullback and Leibler, [25]), it is symmetric and ranges from 0 to 1, with larger values indicating greater distributional differences. In the human analysis, we compute it between NLI and VE label distributions for each paired example. For model analyses, we compute it between the model prediction distribution and the corresponding human label distribution. Given two distributions pp and qq over Y\mathcal{Y} and their average m=12(p+q)m = \frac{1}{2}(p+q), we compute:

JSD⁡(p,q)=12DKL(p∥m)+12DKL(q∥m).\operatorname{JSD}(p,q) = \sqrt{\frac{1}{2}D_{\mathrm{KL}}(p\mathbin{\|}m) + \frac{1}{2}D_{\mathrm{KL}}(q\mathbin{\|}m)}.

where the KL divergence is defined as:

DKL(a∥b)=∑y∈Ya(y)log⁡2a(y)b(y).D_{\mathrm{KL}}(a\mathbin{\|}b) = \sum_{y \in\mathcal{Y}} a(y) \log_2 \frac{a(y)}{b(y)}.

Total Variation Distance. In addition to JSD, we use TVD (Devroye and Lugosi, [12]) to compare label distributions. Unlike JSD, TVD is based on absolute differences between probability masses and can be interpreted as the amount of probability mass that has to be shifted to transform one distribution into the other. As with JSD, we compute TVD to both compare human label distributions across NLI and VE, and to compare human label distributions with model prediction distributions. Given two distributions pp and qq over Y\mathcal{Y}, we compute:

TVD⁡(p,q)=12∑y∈Y∣p(y)−q(y)∣.\operatorname{TVD}(p,q) = \frac{1}{2}\sum_{y \in\mathcal{Y}} \lvert p(y)-q(y)\rvert.

Like JSD, TVD also ranges from 0 to 1, with higher values indicating greater differences in distributions.

Normalized Predictive Entropy. We use normalized predictive entropy as a measure of model uncertainty. For each instance ii and modality m∈{NLI,VE}m \in\{\mathrm{NLI}, \mathrm{VE}\}, let qimq_i^m denote the model distribution over the label set Y={ENT,NEU,CON}\mathcal{Y} = \{\mathrm{ENT}, \mathrm{NEU}, \mathrm{CON}\}, obtained from the answer-option logits. We compute:

Hnorm(qim)=−1log⁡∣Y∣∑y∈Yqim(y)log⁡qim(y).H_{\mathrm{norm}}(q_i^m) = -\frac{1}{\log\lvert\mathcal{Y}\rvert}\sum_{y \in\mathcal{Y}}q_i^m(y)\log q_i^m(y).

Values close to 0 indicate that the model assigns most probability mass to a single label, while values close to 1 indicate near uniform label distribution and therefore greater uncertainty.

Conformal Prediction with Least Ambiguous set-valued Classifier. In addition to normalized predictive entropy, we use conformal prediction with the Least Ambiguous set-valued Classifier (LAC), following Ye et al. [45]. Conformal prediction turns model probabilities into calibrated prediction sets with a target coverage level, providing a set-valued uncertainty estimate. The procedure follows three steps: compute LAC scores on a held-out calibration set, estimate a conformal threshold on these scores, and use this threshold to construct prediction sets for test instances. For a calibration instance ii, let qiq_i denote the model distribution over Y\mathcal{Y} and let yiy_i be the true label. The LAC score is defined as:

si=1−qi(yi).s_i = 1-q_i(y_i).

Given calibration scores s1,…,sns_1,\ldots,s_n, we compute the threshold:

q^=Quantile⁡⌈(n+1)(1−α)⌉/n(s1,…,sn).\hat{q} = \operatorname{Quantile}_{\lceil(n+1)(1-\alpha)\rceil/n}(s_1,\ldots,s_n).

At test time, the prediction set for instance ii is given by

Ci={y′∈Y:1−qi(y′)≤q^}.C_i = \{y' \in\mathcal{Y}: 1-q_i(y') \le\hat{q}\}.

We use the resulting prediction set size as the uncertainty score, with larger sets indicating greater model uncertainty. We set α=0.1\alpha= 0.1 (90% target coverage rate). For calibration, we use a set of paired SNLI/e-SNLI-VE instances with their original single gold labels (n=988n = 988) that are disjoint from the 225 pairs used for human annotation. Ye et al. [45] consider both LAC and Adaptive Prediction Sets (APS). In our three-label setting, APS saturates for VE: when the gold label is ranked last, its cumulative APS score is necessarily 1, which occurs frequently enough in the VE calibration data for the 90% coverage threshold to reach 1. Consequently, all three labels are included in every VE prediction set, leaving no variation for instance-level analyses. We therefore use LAC, which retains variation in prediction set size in our setting.

Spearman’s Rank Correlation. To inspect whether model uncertainty is aligned with human uncertainty, we compute Spearman’s rank correlation ρ\rho (Spearman [34]) between human and model uncertainty scores. For human uncertainty, we use human label entropy and inverse confidence, defined as 7−ci7 - c_i, where cic_i is the mean human confidence score on the original 1–6 scale. For model uncertainty, we use normalized predictive entropy and LAC set size. Given paired uncertainty scores (xi,zi)(x_i, z_i), where xix_i denotes a human uncertainty score, and ziz_i denotes a model uncertainty score, let ri=R(xi)r_i = R(x_i) and si=R(zi)s_i = R(z_i) denote their ranks among the examples. We compute:

ρ=∑i(ri−rˉ)(si−sˉ)∑i(ri−rˉ)2∑i(si−sˉ)2\rho= \frac{\sum_i (r_i - \bar{r})(s_i - \bar{s})}{\sqrt{\sum_i (r_i - \bar{r})^2 \sum_i (s_i - \bar{s})^2}}

We compute correlations separately for each model in NLI and VE. Positive correlations indicate that instances with higher human uncertainty also tend to receive higher model uncertainty.

Original vs. Human-Majority VE Labels.

Figure Figure 14 compares the original single e-SNLI-VE labels with the VE majority obtained from our new annotations (5 per instance). Most contradiction, entailment, and neutral cases retain their original label (82.67%, 69.33%, and 70.67%, respectively). When labels change, they often move through neutral: 22.67% of originally entailment and 14.67% of originally contradiction cases receive a neutral majority, while direct shifts between entailment and contradiction are rare, suggesting that neutral is the main boundary area for HLV.

Figure 14. Label shifts from original single e-SNLI-VE labels to VE majority labels from our new annotations.

Stability Checks for Human Analysis

We assess whether our main human disagreement results are sensitive to the annotations used to estimate the label distributions. We consider two checks: removing one paired NLI–VE judgment per instance and excluding author-annotator judgments.

Paired leave-one-vote-out analysis. Because the 21 annotators labeled different subsets of instances, leaving out one annotator globally would affect only part of the dataset and produce unequal vote counts. We therefore perform a paired leave-one-vote-out analysis at the instance level. For each instance, we remove one annotator’s NLI and VE judgments and recompute the metrics from the remaining four votes. We repeat this for all five possible paired removals and average the resulting values. Table 7 shows that results remain stable under one-vote removal.

Metric5 votes4 votesSensitivity
Entropy (NLI)0.4680.430±0.026
Entropy (VE)0.5380.495±0.026
Δ\Delta entropy (VE–NLI)0.0700.065±0.036
TVD (VE–NLI)0.4840.487±0.013
JSD (VE–NLI)0.5240.526±0.013

Table 7. Paired leave-one-vote-out stability analysis. The 4-vote values average over all five possible paired removals per instance. Sensitivity reflects variation across these removals.

Author-exclusion analysis. Three of the 21 annotators were authors of this paper, which could introduce bias into the human disagreement results. We therefore recompute the main metrics after excluding all author-annotator judgments. Of the 225 instances, 37 (16.4%) were annotated entirely by non-authors, 162 (72.0%) included one authorannotator, and 26 (11.6%) included two. Author-annotators thus never constituted a majority for any instance, and all 225 instances remain represented by three to five non-author annotators after exclusion. We report results in Table 8. The main findings persist after excluding author-annotators: the aggregate entropy difference between NLI and VE remains indistinguishable from zero and moves closer to zero, while TVD and JSD remain nearly unchanged. Because author exclusion leaves between three and five judgments per instance, direct numerical differences between the two columns should be interpreted cautiously.

MetricAll annotatorsw/o authors
Entropy (NLI)0.468 [0.41, 0.53]0.467 [0.40, 0.53]
Entropy (VE)0.538 [0.48, 0.60]0.469 [0.40, 0.54]
Δ\Delta entropy (VE–NLI)0.070 [-0.01, 0.15]0.002 [-0.08, 0.09]
TVD (VE–NLI)0.484 [0.44, 0.53]0.488 [0.44, 0.53]
JSD (VE–NLI)0.524 [0.48, 0.57]0.526 [0.48, 0.57]

Table 8. Human disagreement metrics with all annotators and after excluding author-annotators. Values are means with 95% percentile bootstrap confidence intervals based on 10,000 resamples of the 225 instances.

Examples of Label–Rationale Patterns

Figure 15 provides one illustrative example for each of the four label-rationale patterns used in our explanation analysis. We consider two rationales the same when they rely on the same underlying consideration, even if their wording differs.

Same label, same rationale. In this example, the annotator assigns neutral in both NLI and VE. In both explanations, the judgment rests on the fact that the girl’s age cannot be inferred from the available evidence.

Same label, different rationale. In this case, the annotator assigns contradiction in both NLI and VE, but for different reasons. The NLI explanation focuses on the girl playing rather than learning a new program, whereas the VE explanation instead points to the absence of her parents from the image.

Different label, same rationale. Here, the label changes from neutral to contradiction, but both explanations hinge on whether the scene can be established as being in Colorado.

This pattern is present in 14 out of 850 NLI-VE explanation pairs. While it might seem counterintuitive at first, it does not necessarily signify an annotation error; it may also be indicative of individual differences in label interpretation. For instance, in the present example, the annotator seems to interpret neutral as allowing for the possibility that the scene is in Colorado (They could be in Colorado), while treating the absence of evidence that it is in Colorado as sufficient for contradiction (There is no evidence it is Colorado). While the annotator thus deviates from the original label interpretation, where such uncertainty would generally fall under the neutral label, Hong et al. [20] report comparable cases in which annotators provide similar reasoning but ultimately choose different labels, suggesting a degree of subjectivity in label boundaries.

Different label, different rationale. In this example, the judgment changes from neutral to contradiction, and the rationale changes as well. The NLI explanation leaves open whether the balloon ride is a city tour, while the VE explanation rejects that possibility.

Additional Model Results

Text-only LLM Results

Table 9 reports results for text-only Qwen3 models on NLI. These models serve as a reference point for interpreting VLM performance on the text-only task. The smallest LLM, Qwen3-1.7B, underperforms all VLMs in terms of accuracy, TVD and JSD, while Qwen3-4B and and Qwen3-8B fall within the range of the VLMs on all three metrics, although they remain below the strongest VLMs. This suggests that strong text-only LLMs and VLMs perform comparably on the NLI task. The uncertainty metrics (normalized predictive entropy and LAC set size) reveal a similar pattern, with values that are broadly comparable to the VLM uncertainty scores on NLI.

ModelAcc. ↑\uparrowTVD ↓\downarrowJSD ↓\downarrowEnt.LAC
Qwen3-1.7B42.10.5560.5830.1672.77
Qwen3-4B81.40.2590.3040.0211.74
Qwen3-8B83.70.2490.2970.0562.04

Table 9. Text-only Qwen3 results on NLI. Ent. denotes normalized predictive entropy, and LAC denotes conformal prediction set size using LAC score.

Figure 15. Illustrative examples of the four label-rationale patterns used in the explanation analysis. Each panel shows the textual premise, hypothesis, and paired NLI and VE judgments from the same annotator.

Accuracy on Neutral Instances Across Modalities

Table 10 reports accuracy on the 81 paired instances whose VE majority label is neutral. In the full paired evaluation set, the NLI examples are mostly neutral, while the corresponding VE examples are more evenly distributed across entailment, neutral and contradiction. To check whether the VE drop persists when this label-distribution difference is removed, we restrict the comparison the instances that have a neutral majority label in both modalities. Even under this restriction, all models obtain lower accuracy in VE than NLI.

ModelNLI acc.VE acc.
Gemma-3-4B76.255.6
Gemma-3-12B80.051.9
Gemma-3-27B87.553.1
InternVL3.5-2B82.553.1
InternVL3.5-4B92.569.1
InternVL3.5-8B88.865.4
LLaVA-OneVision-4B91.249.4
LLaVA-OneVision-8B87.553.1
Molmo2-4B88.876.5
Molmo2-8B96.282.7
Molmo2-O-7B83.860.5
Qwen3VL-2B76.242.0
Qwen3VL-8B96.284.0

Table 10. Accuracy on paired instances whose VE majority label is neutral (n=81n = 81). VE cells are shaded according to the decrease in accuracy compared to the corresponding NLI instances, with darker cells indicating larger drops.

Figure 16. Predicted vs human label distribution for VE across our 225 instances.

Model Predictions vs Human Labels

Figure 16 compares the distribution of labels predicted by the models for the VE task with the human-majority VE label distribution. Compared to humans, models tend to underpredict the contradiction label and instead assign more predictions to entailment and neutral.

Table 14 shows a label-wise breakdown of human-model distributional alignment in VE. For each label, we compare the model judgment distribution with the corresponding human label distribution using TVD and JSD. Alignment is strongest for entailment and weaker for neutral, and espe-

@page 19/22 starts_mid=0 ends_mid=1

Figure 15 (PDF p. 19). Illustrative examples of the four label-rationale patterns used in the explanation analysis. Each panel shows the textual premise, hypothesis, and paired NLI and VE judgments from the same annotator.

Accuracy on Neutral Instances Across Modalities

Table 10 reports accuracy on the 81 paired instances whose VE majority label is neutral. In the full paired evaluation set, the NLI examples are mostly neutral, while the corresponding VE examples are more evenly distributed across entailment, neutral and contradiction. To check whether the VE drop persists when this label-distribution difference is removed, we restrict the comparison the instances that have a neutral majority label in both modalities. Even under this restriction, all models obtain lower accuracy in VE than NLI.

ModelNLI acc.VE acc.
Gemma-3-4B76.255.6
Gemma-3-12B80.051.9
Gemma-3-27B87.553.1
InternVL3.5-2B82.553.1
InternVL3.5-4B92.569.1
InternVL3.5-8B88.865.4
LLaVA-OneVision-4B91.249.4
LLaVA-OneVision-8B87.553.1
Molmo2-4B88.876.5
Molmo2-8B96.282.7
Molmo2-O-7B83.860.5
Qwen3VL-2B76.242.0
Qwen3VL-8B96.284.0

Table 10 (PDF p. 19). Accuracy on paired instances whose VE majority label is neutral (n=81n = 81). VE cells are shaded according to the decrease in accuracy compared to the corresponding NLI instances, with darker cells indicating larger drops.

Figure 16 (PDF p. 19). Predicted vs human label distribution for VE across our 225 instances.

Model Predictions vs Human Labels

Figure 16 compares the distribution of labels predicted by the models for the VE task with the human-majority VE label distribution. Compared to humans, models tend to underpredict the contradiction label and instead assign more predictions to entailment and neutral.

Table 14 shows a label-wise breakdown of human-model distributional alignment in VE. For each label, we compare the model judgment distribution with the corresponding human label distribution using TVD and JSD. Alignment is strongest for entailment and weaker for neutral, and especially low for contradiction cases. This mirrors the aggregate label-distribution pattern where models tend to underpredict the contradiction label.

Model Uncertainty Analyses

Detailed Model Uncertainty Results. Table 11 reports detailed model uncertainty scores for NLI and VE. We include normalized predictive entropy and conformal prediction with LAC set size, where larger values indicate greater model uncertainty. Normalized predictive entropy varies considerably across models, but models are more uncertain in VE than NLI in most cases. LAC set size shows a similar trend, with larger prediction sets for VE across models.

ModelEntropyLAC
NLIVENLIVE
Gemma-3-4B0.0450.0411.4762.627
Gemma-3-12B0.0250.1051.7162.200
Gemma-3-27B0.0100.0271.4092.293
InternVL3.5-2B0.3870.3462.0802.129
InternVL3.5-4B0.1140.3441.0981.969
InternVL3.5-8B0.1160.3611.1242.004
LLaVA-OneVision-4B0.5130.5291.3072.196
LLaVA-OneVision-8B0.3860.4681.4002.004
Molmo2-4B0.1810.4071.1112.004
Molmo2-8B0.0890.3381.0002.151
Molmo2-O-7B0.2290.3771.4312.289
Qwen3VL-2B0.0960.2451.9072.258
Qwen3VL-8B0.0270.1541.0802.044

Table 11. Detailed model uncertainty scores measured with normalized predictive entropy and LAC set size. VE cells are shaded according to their increase over the corresponding NLI value.

Spearman Correlations with LAC Set Size

Figure 17 shows Spearman’s rank correlation between LAC set size and the two human uncertainty measures, computed separately for NLI and VE. Because our task has only three labels, LAC set size is a relatively coarse uncertainty measure, taking only values from 1 to 3. For Molmo2-8B in NLI, the set size does not vary across instances, so Spearman’s ρ\rho cannot be computed for this case. Among the remaining correlations, 17 of 24 (71%) are statistically significant in NLI, compared with 13 of 26 (50%) in VE. The correlations are generally small, indicating limited correspondence between LAC set size and human uncertainty.

Paired NLI-VE Confidence Intervals

We report 95% paired bootstrap confidence intervals for NLI–VE differences in model performance

Figure 17. Spearman’s ρ\rho between LAC set size and human uncertainty measures (human label entropy and inverse mean confidence). Asterisks indicate p < 0.05. n/a indicates cases where Spearman’s ρ\rho is undefined because LAC set size does not vary across instances.

and uncertainty (Table 12) and human–model alignment (Table 13). Intervals are based on 10,000 bootstrap resamples of paired instances. For accuracy, we define Δ=NLI−VE\Delta= \mathrm{NLI} - \mathrm{VE}, such that positive values indicate higher accuracy in NLI. For TVD, JSD, normalized predictive entropy, and LAC set size, we define Δ=VE−NLI\Delta= \mathrm{VE} - \mathrm{NLI}, such that positive values indicate poorer alignment or greater uncertainty in VE. An interval containing zero indicates that the corresponding NLI–VE difference is not clearly distinguishable from zero under this analysis.

ModelAccuracy (Δ\Delta = NLI − VE)Uncertainty (Δ\Delta = VE − NLI)
ModelOverallNeutral-onlyNorm. entropyLAC size
Gemma-3-4B26.3 [17.6, 35.0]20.7 [9.9, 31.7]−0.004 [−0.027, 0.017] n.s.1.151 [1.036, 1.267]
Gemma-3-12B24.8 [16.9, 32.5]28.1 [17.3, 39.5]0.080 [0.051, 0.109]0.484 [0.356, 0.613]
Gemma-3-27B27.5 [19.5, 35.5]34.4 [23.5, 45.6]0.016 [0.001, 0.033]0.884 [0.769, 1.000]
InternVL3.5-2B21.5 [13.4, 29.5]29.4 [18.4, 40.7]−0.041 [−0.091, 0.010] n.s.0.049 [−0.098, 0.196] n.s.
InternVL3.5-4B36.6 [29.2, 43.9]23.4 [13.5, 33.3]0.231 [0.184, 0.278]0.871 [0.756, 0.987]
InternVL3.5-8B31.1 [23.4, 38.6]23.3 [12.1, 34.6]0.245 [0.198, 0.293]0.880 [0.764, 0.996]
LLaVA-OneVision-4B33.5 [25.1, 41.7]41.9 [30.9, 52.9]0.016 [−0.028, 0.059] n.s.0.889 [0.760, 1.013]
LLaVA-OneVision-8B31.6 [23.9, 39.2]34.4 [23.2, 45.7]0.082 [0.036, 0.127]0.604 [0.476, 0.733]
Molmo2-4B34.7 [27.0, 42.4]12.2 [2.5, 22.2]0.227 [0.185, 0.268]0.893 [0.791, 0.996]
Molmo2-8B26.9 [19.1, 34.8]13.5 [4.9, 22.2]0.249 [0.212, 0.287]1.151 [1.044, 1.258]
Molmo2-O-7B28.9 [20.6, 36.9]23.3 [12.3, 34.4]0.148 [0.096, 0.201]0.858 [0.724, 0.991]
Qwen3VL-2B17.1 [8.1, 26.1]34.3 [22.2, 46.6]0.149 [0.104, 0.194]0.351 [0.182, 0.520]
Qwen3VL-8B31.0 [23.5, 38.7]12.3 [3.7, 21.0]0.127 [0.093, 0.162]0.964 [0.849, 1.080]

Table 12. Paired NLI–VE differences in model accuracy and uncertainty with 95% bootstrap confidence intervals based on 10,000 paired resamples. Cells report Δ\Delta [95% CI]. For accuracy, Δ\Delta = NLI − VE and is reported in percentage points. For normalized predictive entropy and LAC set size, Δ\Delta = VE − NLI. Non-significant differences (95% CI includes zero) are marked with n.s. and shaded in gray.

ModelAll instancesNeutral-only (nn = 81)
ModelTVDJSDTVDJSD
Gemma-3-4B0.236 [0.170, 0.303]0.226 [0.163, 0.290]0.218 [0.136, 0.301]0.222 [0.139, 0.304]
Gemma-3-12B0.192 [0.134, 0.249]0.191 [0.135, 0.246]0.235 [0.155, 0.318]0.242 [0.162, 0.323]
Gemma-3-27B0.231 [0.173, 0.290]0.219 [0.160, 0.277]0.288 [0.208, 0.369]0.283 [0.201, 0.363]
InternVL3.5-2B0.139 [0.088, 0.190]0.112 [0.067, 0.158]0.198 [0.132, 0.266]0.169 [0.112, 0.227]
InternVL3.5-4B0.222 [0.171, 0.272]0.200 [0.151, 0.248]0.180 [0.113, 0.246]0.171 [0.104, 0.236]
InternVL3.5-8B0.192 [0.143, 0.241]0.178 [0.133, 0.223]0.164 [0.101, 0.225]0.159 [0.098, 0.217]
LLaVA-OneVision-4B0.207 [0.163, 0.250]0.169 [0.133, 0.205]0.207 [0.156, 0.258]0.161 [0.119, 0.205]
LLaVA-OneVision-8B0.195 [0.151, 0.240]0.162 [0.125, 0.200]0.195 [0.137, 0.251]0.158 [0.109, 0.205]
Molmo2-4B0.216 [0.169, 0.264]0.191 [0.149, 0.234]0.102 [0.046, 0.156]0.089 [0.036, 0.139]
Molmo2-8B0.198 [0.147, 0.251]0.185 [0.138, 0.233]0.141 [0.086, 0.195]0.136 [0.080, 0.192]
Molmo2-O-7B0.186 [0.134, 0.236]0.166 [0.121, 0.212]0.170 [0.105, 0.234]0.163 [0.104, 0.222]
Qwen3VL-2B0.110 [0.048, 0.171]0.104 [0.046, 0.162]0.220 [0.133, 0.308]0.208 [0.125, 0.289]
Qwen3VL-8B0.221 [0.167, 0.277]0.206 [0.152, 0.260]0.148 [0.086, 0.211]0.155 [0.087, 0.222]

Table 13. Paired NLI–VE differences in human–model distributional alignment with 95% bootstrap confidence intervals based on 10,000 paired resamples. Cells report Δ\Delta [95% CI], where Δ\Delta = VE − NLI. The neutral-only comparison is restricted to the 81 instances whose VE majority label is neutral.

Implementation Details

We use the Hugging Face implementations4 of all models and their default model-specific parameters. For prediction extraction, we use a single forward pass per model and instance to obtain the next-token logits for the answer option letters. We apply a softmax over these option-letter logits and take the argmax as the predicted label. Predictions are therefore derived directly from answer-option logits rather than from decoded generated responses. Accordingly, we do not average over multiple sampled runs. All models were run locally on NVIDIA A100 GPUs.

We report 95% paired bootstrap confidence intervals for NLI–VE differences in model performance

@page 20/22 starts_mid=1 ends_mid=1 cially low for contradiction cases. This mirrors the aggregate label-distribution pattern where models tend to underpredict the contradiction label.

Model Uncertainty Analyses

Detailed Model Uncertainty Results. Table 11 reports detailed model uncertainty scores for NLI and VE. We include normalized predictive entropy and conformal prediction with LAC set size, where larger values indicate greater model uncertainty. Normalized predictive entropy varies considerably across models, but models are more uncertain in VE than NLI in most cases. LAC set size shows a similar trend, with larger prediction sets for VE across models.

ModelEntropyLAC
NLIVENLIVE
Gemma-3-4B0.0450.0411.4762.627
Gemma-3-12B0.0250.1051.7162.200
Gemma-3-27B0.0100.0271.4092.293
InternVL3.5-2B0.3870.3462.0802.129
InternVL3.5-4B0.1140.3441.0981.969
InternVL3.5-8B0.1160.3611.1242.004
LLaVA-OneVision-4B0.5130.5291.3072.196
LLaVA-OneVision-8B0.3860.4681.4002.004
Molmo2-4B0.1810.4071.1112.004
Molmo2-8B0.0890.3381.0002.151
Molmo2-O-7B0.2290.3771.4312.289
Qwen3VL-2B0.0960.2451.9072.258
Qwen3VL-8B0.0270.1541.0802.044

Table 11 (PDF p. 20). Detailed model uncertainty scores measured with normalized predictive entropy and LAC set size. VE cells are shaded according to their increase over the corresponding NLI value.

Spearman Correlations with LAC Set Size

Figure 17 shows Spearman’s rank correlation between LAC set size and the two human uncertainty measures, computed separately for NLI and VE. Because our task has only three labels, LAC set size is a relatively coarse uncertainty measure, taking only values from 1 to 3. For Molmo2-8B in NLI, the set size does not vary across instances, so Spearman’s ρ\rho cannot be computed for this case. Among the remaining correlations, 17 of 24 (71%) are statistically significant in NLI, compared with 13 of 26 (50%) in VE. The correlations are generally small, indicating limited correspondence between LAC set size and human uncertainty.

Paired NLI-VE Confidence Intervals

We report 95% paired bootstrap confidence intervals for NLI–VE differences in model performance

Figure 17 (PDF p. 20). Spearman’s ρ\rho between LAC set size and human uncertainty measures (human label entropy and inverse mean confidence). Asterisks indicate p < 0.05. n/a indicates cases where Spearman’s ρ\rho is undefined because LAC set size does not vary across instances.

and uncertainty (Table 12) and human–model alignment (Table 13). Intervals are based on 10,000 bootstrap resamples of paired instances. For accuracy, we define Δ=NLI−VE\Delta= \mathrm{NLI} - \mathrm{VE}, such that positive values indicate higher accuracy in NLI. For TVD, JSD, normalized predictive entropy, and LAC set size, we define Δ=VE−NLI\Delta= \mathrm{VE} - \mathrm{NLI}, such that positive values indicate poorer alignment or greater uncertainty in VE. An interval containing zero indicates that the corresponding NLI–VE difference is not clearly distinguishable from zero under this analysis.

Implementation Details

We use the Hugging Face implementations4 of all models and their default model-specific parameters. For prediction extraction, we use a single forward pass per model and instance to obtain the next-token logits for the answer option letters. We apply a softmax over these option-letter logits and take the argmax as the predicted label. Predictions are therefore derived directly from answer-option logits rather than from decoded generated responses. Accordingly, we do not average over multiple sampled runs. All models were run locally on NVIDIA A100 GPUs.

References

  1. [1]Xiang An, Yin Xie, Kaicheng Yang, Wenkang Zhang, Xiuwei Zhao, Zheng Cheng, Yirui Wang, Songcen Xu, Changrui Chen, Didi Zhu, Chunsheng Wu, Hua-jie Tan, Chunyu-an Li, Jing Yang, Jie Yu, Xiaoyao Wang, Bin Qin, Yumeng Wang, Zizhen Yan, and 4 others. 2025. Llava-onevision-1.5: Fully open framework for democratized multimodal training. Preprint, arXiv:2509.23661.arxiv.org/abs/2509.23661
  2. [2]Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425–2433.
  3. [3]Lora Aroyo and Chris Welty. 2015. Truth is a lie: Crowd truth and the seven myths of human annotation. AI Magazine, 36(1):15–24.
  4. [4]Joris Baan, Wilker Aziz, Barbara Plank, and Raquel Fernandez. 2022. Stop measuring calibration when humans disagree. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1892–1915, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  5. [5]Lyle F Bachman and Adrian S Palmer. 1989. The construct validation of self-ratings of communicative language ability. Language testing, 6(1):14–29.DOI
  6. [6]Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shuhong Jiang, Zhaohai Li, Mingsheng Li, and 45 others. 2025. Qwen3-vl technical report. Preprint, arXiv:2511.21631.arxiv.org/abs/2511.21631
  7. [7]Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computational Linguistics.arxiv.org/abs/1508.05326
  8. [8]Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. e-SNLI: Natural language inference with natural language explanations. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc.arxiv.org/abs/1812.01193
  9. [9]Beiduo Chen, Siyao Peng, Anna Korhonen, and Barbara Plank. 2025. A rose by any other name: LLM-generated explanations are good proxies for human explanations to collect label distributions on NLI. In Findings of the Association for Computational Linguistics: ACL 2025, pages 10777–10802, Vienna, Austria. Association for Computational Linguistics.arxiv.org/abs/2412.13942
  10. [10]Beiduo Chen, Xinpeng Wang, Siyao Peng, Robert Litschko, Anna Korhonen, and Barbara Plank. 2024. “seeing the big through the small”: Can LLMs approximate human judgment distributions on NLI from a few explanations? In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 14396–14419, Miami, Florida, USA. Association for Computational Linguistics.arxiv.org/abs/2406.17600
  11. [11]Christopher Clark, Jieyu Zhang, Zixian Ma, Jae Sung Park, Mohammedza Salehi, Rohun Tripathi, Sangho Lee, Zhongzheng Ren, Chris Dongjoo Kim, Yinuo Yang, Vincent Shao, Yue Yang, Weikai Huang, Ziqi Gao, Taira Anderson, Jianrui Zhang, Jitesh Jain, George Stoica, Winson Han, and 2 others. 2026. Molmo2: Open weights and data for vision-language models with video understanding and grounding. Preprint, arXiv:2601.10611.arxiv.org/abs/2601.10611
  12. [12]Luc Devroye and Gábor Lugosi. 2001. Combinatorial Methods in Density Estimation. Springer Series in Statistics. Springer, New York, NY.DOI
  13. [13]D.M. Endres and J.E. Schindelin. 2003. A new metric for probability distributions. IEEE Transactions on Information Theory, 49(7):1858–1860.DOI
  14. [14]Tommaso Fornaciari, Alexandra Uma, Silvio Paun, Barbara Plank, Dirk Hovy, and Massimo Poesio. 2021. Beyond black & white: Leveraging annotator disagreement via soft-label multi-task learning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2591–2597, Online. Association for Computational Linguistics.
  15. [15]Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913.
  16. [16]Danna Gurari and Kristen Grauman. 2017. Crowdverge: Predicting if people will agree on the answer to a visual question. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pages 3511–3522.DOI
  17. [17]Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. 2018. Vizwiz grand challenge: Answering visual questions from blind people. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3608–3617. IEEE.arxiv.org/abs/1802.08218
  18. [18]Susanne Hindennach, Lei Shi, and Andreas Bulling. 2024. Explaining disagreement in visual question answering using eye tracking. In Proceedings of the 2024 Symposium on Eye Tracking Research and Applications, pages 1–7.DOI
  19. [19]Pingjun Hong, Beiduo Chen, Siyao Peng, Marie-Catherine de Marneffe, and Barbara Plank. 2025. LiTEx: A linguistic taxonomy of explanations for understanding within-label variation in natural language inference. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 34065–34085, Suzhou, China. Association for Computational Linguistics.arxiv.org/abs/2505.22848
  20. [20]Pingjun Hong, Beiduo Chen, Siyao Peng, Marie-Catherine de Marneffe, Benjamin Roth, and Barbara Plank. 2026. Agree, disagree, explain: Decomposing human label variation in NLI through the lens of explanations. In Findings of the Association for Computational Linguistics: ACL 2026, pages 26922–26934, San Diego, California, United States. Association for Computational Linguistics.
  21. [21]Nan-Jiang Jiang and Marie-Catherine de Marneffe. 2022. Investigating reasons for disagreement in natural language inference. Transactions of the Association for Computational Linguistics, 10:1357–1374.arxiv.org/abs/2209.03392
  22. [22]Nan-Jiang Jiang, Chenhao Tan, and Marie-Catherine de Marneffe. 2023. Ecologically valid explanations for label variation in NLI. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10622–10633, Singapore. Association for Computational Linguistics.
  23. [23]Kushal Kafle and Christopher Kanan. 2017. Visual question answering: Datasets, algorithms, and future challenges. Computer Vision and Image Understanding, 163:3–20.DOI
  24. [24]Maxime Kayser, Oana-Maria Camburu, Leonard Salewski, Cornelius Emde, Virginie Do, Zeynep Akata, and Thomas Lukasiewicz. 2021. E-vil: A dataset and benchmark for natural language explanations in vision-language tasks. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1244–1254.arxiv.org/abs/2105.03761
  25. [25]S. Kullback and R. A. Leibler. 1951. On Information and Sufficiency. The Annals of Mathematical Statistics, 22(1):79 – 86.DOI
  26. [26]Jian Lan, Diego Frassinelli, and Barbara Plank. 2025. Mind the uncertainty in human disagreement: Evaluating discrepancies between model predictions and human responses in vqa. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 4446–4454.arxiv.org/abs/2410.02773
  27. [27]Minzi Li and Xian Zhang. 2021. A meta-analysis of self-assessment and language performance in language testing and assessment. Language Testing, 38(2):189–218.
  28. [28]Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. Ok-vqa: A visual question answering benchmark requiring external knowledge. In 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 3190–3199. IEEE.arxiv.org/abs/1906.00067
  29. [29]Yixin Nie, Xiang Zhou, and Mohit Bansal. 2020. What can we learn from collective human opinions on natural language inference data? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9131–9143, Online. Association for Computational Linguistics.arxiv.org/abs/2010.03532
  30. [30]Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics, 7:677–694.
  31. [31]Pouya Pezeshkpour and Estevam Hruschka. 2024. Large language models sensitivity to the order of options in multiple-choice questions. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2006–2017, Mexico City, Mexico. Association for Computational Linguistics.
  32. [32]Barbara Plank. 2022. The “problem” of human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10671–10682, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.arxiv.org/abs/2211.02570
  33. [33]C. E. Shannon. 1948. A mathematical theory of communication. The Bell System Technical Journal, 27(3):379–423.
  34. [34]C. Spearman. 1904. The proof and measurement of association between two things. The American Journal of Psychology, 15(1):72–101.
  35. [35]Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Riviére, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, and 197 others. 2025. Gemma 3 technical report. Preprint, arXiv:2503.19786.
  36. [36]Maxim Tkachenko, Mikhail Malyuk, Andrey Holmanyuk, and Nikolai Liubimov. 2020–2025. Label Studio: Data labeling software. Open source software available from https://github.com/HumanSignal/label-studio.
  37. [37]Alexandra N. Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2022. Learning from disagreement: A survey. J. Artif. Int. Res., 72:1385–1470.DOI
  38. [38]Hoa Trong Vu, Claudio Greco, Alijia Erofeeva, Somayeh Jafaritazehjan, Guido Linders, Marc Tanti, Alberto Testoni, Raffaella Bernardi, and Albert Gatt. 2018. Grounded textual entailment. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2354–2368, Santa Fe, New Mexico, USA. Association for Computational Linguistics.arxiv.org/abs/1806.05645
  39. [39]Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, Zhaokai Wang, Zhe Chen, Hongjie Zhang, Ganlin Yang, Haomin Wang, Qi Wei, Jinhui Yin, Wenhao Li, Erfei Cui, and 56 others. 2025. Intervl3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. Preprint, arXiv:2508.18265.arxiv.org/abs/2508.18265
  40. [40]Leon Weber-Genzel, Siyao Peng, Marie-Catherine De Marneffe, and Barbara Plank. 2024. VariErr NLI: Separating annotation error from human label variation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2256–2269, Bangkok, Thailand. Association for Computational Linguistics.
  41. [41]Kenneth M Wilson. 1999. Validity of global self-ratings of esl speaking proficiency based on an fsi/ilr-referenced scale. ETS Research Report Series, 1999(2):i–64.DOI
  42. [42]Ning Xie, Farley Lai, Derek Doran, and Asim Kaday. 2019. Visual entailment: A novel task for fine-grained image understanding. Preprint, arXiv:1901.06706.arxiv.org/abs/1901.06706
  43. [43]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.arxiv.org/abs/2505.09388
  44. [44]Chun-Ju Yang, Kristen Grauman, and Danna Gurari. 2018. Visual question answer diversity. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume 6, pages 184–192.DOI
  45. [45]Fanghua Ye, Mingming Yang, Jianhui Pang, Longyue Wang, Derek Wong, Emine Yilmaz, Shuming Shi, and Zhaopeng Tu. 2024. Benchmarking llms via uncertainty quantification. In Advances in Neural Information Processing Systems, volume 37, pages 15356–15385. Curran Associates, Inc.
  46. [46]Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67–78.
  47. [47]Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2024. Large language models are not robust multiple choice selectors. In International Conference on Learning Representations, volume 2024, pages 19426–19454.arxiv.org/abs/2309.03882

Paper details

vlnc:W_hambqk8h
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY-SA 4.0
Usage

views · 3

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents