betaFeedback

WorksComputer Science

What Does Hard-Negative Fine-Tuning Teach Multimodal Large Language Models? Controlled Evidence from a Compositional Benchmark

Abstract

Fine-tuning on hard negatives is a popular way to improve the compositional reasoning of vision-language models, but when accuracy goes up it is rarely clear what the model actually learned. We ask this question for multimodal large language models. We use CompHall, 2,860 forced-choice questions (spatial, attribute, counting and part-whole) with two graded hard negatives each, three model families (LLaVA-1.5, LLaVA-NeXT and Idefics2), and controls that such studies often lack: gray-image evaluation, three seeds, option-order tests, control arms trained on different question types, and independent benchmarks. We find four things. First, CompHall is not a pure test of vision: part-whole questions can be answered from the text alone (66-95% without the image), and per-image accuracy is unrelated to an object-hallucination probe (r = -0.05), so we see no evidence that compositional failures drive hallucination. Second, the original fine-tuning method scored mostly image-placeholder tokens and hurt both LLaVA models. With a corrected loss, fine-tuning raises accuracy on the spatial-relation questions of CV-Bench in all three families, by +3.2, +23.1 and +6.0 points (averaged over both option orders). Control arms show that the gain needs spatial training questions, and for LLaVA-NeXT about two-thirds of it appears even on relation words that were only wrong options in training. Third, the gain is not general: on CompHall itself it is significant only for LLaVA-NeXT (+6.9 points) and not detectable for Idefics2, it reaches about 3 points at most on true/false statements, and LLaVA-1.5 fine-tuned on other question types learns to answer "true" too often. Fourth, both losses raise the yes-bias of LLaVA-1.5 on POPE.

Introduction

Multimodal large language models (MLLMs), including LLaVA [13], InstructBLIP [1], and GPT-4V [17], can sometimes produce responses that do not accurately reflect the content of the input image. This issue is commonly referred to as hallucination [10, 21, 27]. One approach to addressing related limitations in vision-language models is fine-tuning with hard negatives. These are questions or captions with incorrect answers that differ from the correct ones in only one compositional detail, such as a change in spatial relationships [3, 26]. Although this approach can improve performance on compositional benchmarks, it is often unclear what the model actually learns from the process: improved visual understanding, familiarity with the expected answer format, or patterns in how questions are phrased.

The question. We ask what hard-negative fine-tuning teaches an MLLM. To answer it, we built CompHall, a benchmark of 2,860 forced-choice questions in four compositional types (spatial, attribute, counting, part-whole), each with two graded hard negatives (§ 4), and we ran it on three model families with controls that such studies often lack: gray-image (blind) evaluation, matched prompts and parsers, three training seeds, exact paired tests, option-order tests, control arms that train on different question types, and independent benchmarks.

Motivation. Hard-negative fine-tuning is often motivated by the idea that compositional failures contribute to hallucination: the inability to bind object identity, position, attributes, and part membership into one scene representation, as in the binding problem of perception research [23, 24]. We tested this link first. We found no evidence for it, so the rest of the paper asks the more modest question of what the fine-tuning does teach.

Findings.

  1. The benchmark is not a pure vision test, and compositional accuracy is not linked to hallucination (§ 4–§ 5). Six MLLMs, three of them recent, reach only 28–51% on spatial and 31–45% on counting questions. On 500 further part–whole questions all six models score 66–95% without the image, and under a second prompt every model is more accurate blind than with it. Per-image CompHall accuracy is uncorrelated with an object-hallucination probe (Pearson r≈−0.05r \approx-0.05, 200 images).

  1. The original fine-tuning method had a flaw (§ 6). In CCFT as first implemented, 97% of the scored positions were image-placeholder tokens. That version degrades both LLaVA-1.5-7B and LLaVA-NeXT-7B on independent benchmarks (for example NeXT counting: 63.1% to 12.6%); a corrected loss scores answer tokens only.

  1. Spatial supervision transfers, but by very different amounts (§ 8–§ 8.6). With the corrected loss, accuracy on the spatial-relation questions of the independent CV-Bench, averaged over both option orders, rises from 57.6% to 60.8% for LLaVA-1.5-7B, from 62.8% to 85.9% for LLaVA-NeXT-7B and from 63.8% to 69.8% for Idefics2-8B. Most of this needs spatial training questions: training on spatial questions alone gives 64.5%, 87.1% and 68.7%, and training on the other types alone 57.6%, 66.1% and 66.1%. For the LLaVA models the gain holds on images from another dataset, and for LLaVA-NeXT about two-thirds of it appears even on relation words the model saw only as wrong options. The benchmark’s own option order is misleading here: it makes LLaVA-1.5’s gain look twice as large and Idefics2’s look absent, because the models differ in which answer they prefer when unsure.

  1. The gain is not general. It reaches counting only after spatial-only training (+4.2 pp for LLaVA-NeXT-7B, significant in two of three seeds, and +3.9 pp for Idefics2-8B, in all three; none for LLaVA-1.5-7B). On VSR’s true/false statements it is about 3 points at most (LLaVA-NeXT-7B trained on spatial questions only; none for the other two models), and LLaVA-1.5-7B fine-tuned on other question types becomes biased toward answering “true” (−3 pp). On CompHall itself the gain differs across families: LLaVA-NeXT-7B improves by 6.9 pp, LLaVA-1.5-7B does not improve significantly (51.1 ± 0.5% vs. 48.5%) and loses the image’s contribution to its spatial accuracy, and Idefics2-8B does not change (52.2% to 52.6 ± 0.4%). Both losses also raise the yes-bias of LLaVA-1.5-7B on POPE.

Related Work

Multimodal Large Language Models

MLLMs combine a pre-trained vision encoder (typically a CLIP ViT) with a language model backbone via a lightweight connector module. LLaVA [13] uses a linear projection and LLaVA-1.5 [14] a two-layer MLP; InstructBLIP [1] and BLIP-2 [9] use a Q-Former cross-attention bottleneck; Idefics2 [8] uses a perceiver resampler. LLaVA-NeXT (also known as LLaVA-1.6) [15] introduces dynamic high-resolution visual encoding. Despite architectural diversity, all of these models share the same fundamental limitation: the vision encoder produces a set of independent patch tokens, and cross-modal binding is left to the LLM attention heads without explicit structural inductive bias.

Hallucination in MLLMs

POPE [10] probes binary object existence and measures yes-bias across three difficulty splits. MME [2] evaluates a broader set of visual capabilities. Rohrbach et al. [21] measure caption hallucination in earlier vision-language models. AMBER [25] combines discriminative and generative hallucination evaluation. Ramakrishnan et al. [20] address language-prior shortcuts in VQA. The hallucination benchmarks above measure object-existence or task-level accuracy and are not designed around compositional binding.

Compositional Visual Reasoning

Visual compositional reasoning has been studied via CLEVR [6], GQA [5], and CREPE [16]. CREPE and SugarCREPE [3] evaluate the compositionality of image–text matching models with hard-negative captions, and VALSE [18] probes linguistic phenomena in vision-language models. Fine-tuning on hard negatives of this kind was popularised for contrastive models by NegCLIP [26]; we ask what it teaches an MLLM. CompHall instead asks multiple-choice questions about scene-graph-derived compositional facts, gives each question two graded hard negatives so that the option chosen on an error reveals the kind of mistake, and relates question type to error rate.

Background

The MLLM pipeline and the binding bottleneck. A vision encoder, typically a CLIP or SigLIP ViT [19], maps the image to patch tokens; a connector projects them into the language model’s embedding space (a linear or MLP layer in LLaVA [13, 14], a Q-Former in BLIP-2 and InstructBLIP [1, 9], a perceiver resampler in Idefics2 [8]); and an LLM backbone generates text from the fused sequence. LLaVA-NeXT [15] keeps the MLP connector and adds multi-resolution tiling of the input image. In none of these architectures is a component designed to bind features to objects: attributing a colour or a position to the right object is left to self-attention layers trained on text. Perceptual psychology calls this the binding problem [23, 24]: features processed separately must be assigned to the right object, and failures appear as illusory conjunctions. We therefore expect compositional errors to grow with scene complexity, although we do not test this directly (§ 5.3 measures only existence-question errors against object count). A schematic and further detail are in Appendix § 26.

Hallucination types. We distinguish object-existence errors (T0; the target of POPE, which do not require binding), attribute binding (T2), spatial relations (T1), part–whole and role binding (T4; e.g., “the pants are wearing the man”), and counting (T3, where the number of required bindings grows with the number of instances). CompHall targets T1–T4 and excludes T0 by design: every question concerns objects that are present in the image. A model can do well on a T0 benchmark (POPE accuracy 0.82–0.87 for LLaVA-1.5-7B) while performing near chance on T1 and T3, as our results show.

The CompHall Benchmark

Design Principles

CompHall is built around three design choices. (i) Compositional focus. Every question requires relating at least two pieces of visual information (an object and a position, colour, count, or second object); naming the queried object alone is not sufficient. (ii) Forced-choice format. Each question has three options: the correct answer and two hard negatives. Three-way forced choice sets chance at 33.3% and avoids the yes-bias of binary formats. (iii) Graded negatives. The two negatives differ in how far they depart from the correct answer. A near negative changes one element (the opposite direction, the colour of another annotated object, a count off by one, or a swapped subject and object); a far negative departs further (a different spatial axis, an unrelated colour, a count off by two, or an unrelated object in the scene). The option a model picks when it makes an error therefore shows what kind of mistake it made.

Question Types

CompHall covers four question types corresponding to the types in § 3:

  • T1: Spatial relations. e.g., Where is the person relative to the car? Requires binding two objects to their positions; the options are spatial relations.

  • T2: Attribute binding. e.g., What color is the handle in the image? Requires binding a colour to the queried object; the options are colours.

  • T3: Object counting. e.g., How many car can you see? Requires identifying every instance of a category; the options are counts.

  • T4: Part–whole relations. e.g., Which statement correctly describes the relationship between the man and the pants? Requires assigning the correct roles in a relation; the options are statements.

The four types differ in how many bindings a correct answer involves: T2 binds one attribute to one object, T1 and T4 bind two objects to the roles of a relation, and T3 requires identifying every instance of a category, so the number of bindings grows with scene population. This grouping is a post-hoc reading of Table 1 (the heatmap is in Appendix § 27), where T2 and T4 are answered far more accurately than T1 and T3. It does not separate T1 from T4, which involve similar numbers of bindings but differ widely in accuracy, so binding count alone does not explain the results; part of the gap is that T4 can be answered from the text alone (§ 4.4).

ModelOverallT1T2T3T4T4-L
Chance33.333.333.333.333.333.3
LLaVA-1.5-7B38.328.4{sup}†`56.230.958.167.0
blind33.631.834.328.861.365.6
LLaVA-NeXT-7B52.542.679.038.174.279.8
blind35.233.840.024.574.278.2
Idefics2-8B52.041.977.137.480.687.8
blind41.639.954.325.977.490.0
LLaVA-1.5-7B{sup}‡`48.541.273.332.471.083.2
blind{sup}‡`39.232.450.530.274.290.2
LLaVA-NeXT-7B{sup}‡`51.540.582.933.877.481.2
blind{sup}‡`39.039.249.521.680.682.6
Idefics2-8B{sup}‡`54.643.974.343.290.389.8
blind{sup}‡`45.643.956.230.983.995.2

Table 1. Zero-shot CompHall accuracy (%) with the real image and with a uniform gray image (blind). Chance = 33.3%. †: numerically below the 33.3% chance level (not significant). ‡: second prompt wording (that of the CCFT evaluation). T4-L: 500 additional part–whole questions on new images (95% half-width at most 4.2 pp).

Data Construction

Image sources. Images are drawn from COCO val2017 [11] (those with at least two annotated objects) and Visual Genome [7] (the 2,000 images with the most annotated spatial and part–whole relationships, so they are denser than typical scenes). The two sources map onto question types: all T3 questions and most T1 questions (959 of 1,000 in the full file) use COCO images, and all T2 and T4 questions use Visual Genome images, so question type and image source are confounded (Table 11). COCO val2017 is held out from the standard COCO training split, but Visual Genome has no held-out split, and we did not check overlap between the benchmark images and the training data of the evaluated models.

TypeImage sourceTrainValTest
T1 SpatialCOCO (959 of 1,000)*682164148
T2 AttributeVisual Genome47192105
T3 CountingCOCO701160139
T4 Part–wholeVisual Genome1422531
Total questions1,996441423
Images920199200

Table 11. Questions per split and type, with image source.

Question generation. Questions are generated automatically from COCO and Visual Genome annotations using templates. Each question receives two hard negatives derived from the same annotations (Table 12), and the position of the correct option is randomised. An automatic filter removes questions shorter than 15 characters, questions with duplicated options, and questions containing giveaway words (e.g., “obviously”); each type is capped at 1,000 questions. No human verification was performed.

TypeCorrect answerNear negativeFar negative
T1 SpatialAnnotated relation between two objectsThe opposite relation (e.g. left of → right of)A random relation on another axis (e.g. above)
T2 AttributeAnnotated colour of the queried objectColour of another colour-annotated object in the image; a random colour if there is noneAnother random colour
T3 CountingNumber of annotated instances of a category (at least two)Count ±1\pm1Count ±2\pm2 (or +3+3 if it collides with the near negative)
T4 Part–whole“The S is P the O” from an annotated part–whole or possession relationSubject and object swappedObject replaced by another object from the scene

Table 12. Construction of the correct answer and the two hard negatives.

Scale and splits. The benchmark contains 2,860 questions in four types: 1,996 train, 441 validation, and 423 test (920, 199, and 200 images). Another 93 questions (70 train, 10 validation, 13 test, all with Visual Genome images) were removed because their image was not found at the recorded path: our loader assigned some images to the wrong one of Visual Genome’s two image folders, although the files exist. No test image appears in the training split. The training split is used for CCFT, the validation split for checkpoint selection, and all reported accuracies are on the test split.

Baseline Evaluation

We evaluate three open-source MLLMs under zero-shot prompting (Table 1). Responses are decoded greedily and matched to the closest candidate by normalised edit distance.

LLaVA-1.5-7B scores 28.4% on T1 Spatial and 30.9% on T3 Counting, numerically below but statistically indistinguishable from the 33.3% chance level (two-sided binomial test, p=0.22p=0.22 and 0.59; n=148n=148 and 139). LLaVA-NeXT-7B and Idefics2-8B score 42.6% and 41.9% on T1, modestly above chance (p<0.03p<0.03), and 38.1% and 37.4% on T3 (p>0.2p>0.2), whereas their T2 and T4 accuracy is 74–81%.

Blind baseline. To see how much the questions depend on the image, we repeat the evaluation with a uniform gray image (Table 1). Overall accuracy falls from 38.3% to 33.6% for LLaVA-1.5-7B, from 52.5% to 35.2% for LLaVA-NeXT-7B, and from 52.0% to 41.6% for Idefics2-8B. The image adds 22–39 pp on T2 and 2–14 pp on T3, but between −3 and +9 pp on T1 and between −3 and +3 pp on T4. Under a second prompt wording (rows marked ‡), overall accuracy changes by +10.2, about −1, and +2.6 pp for LLaVA-1.5-7B, LLaVA-NeXT-7B and Idefics2-8B, and the picture is the same: the image adds 18–33 pp on T2, 2–12 pp on T3, 0–9 pp on T1, and −3 to +6 pp on T4. Across both prompts, part–whole questions are answered at 61–84% without the image, and spatial questions gain little from it. The differences between types in Table 1 therefore partly reflect language priors rather than visual binding.

Recent models. Three recent models (Qwen2.5-VL-7B, LLaVA-OneVision-7B, InternVL3-8B) score 55–59% overall with the image under both prompts, but still only 40–51% on spatial and 39–45% on counting questions (Table 14). Without the image they score 37–47% overall and 64–90% on part–whole questions, and the image adds 5–10 pp on spatial questions.

ModelPromptInputOverallT1T2T3T4T4-L
Qwen2.5-VL-7BAimage58.650.784.841.087.190.6
Qwen2.5-VL-7BAblind46.843.957.133.883.986.8
Qwen2.5-VL-7BBimage58.950.089.538.887.193.2
Qwen2.5-VL-7BBblind46.144.659.030.280.694.4
LLaVA-OneVision-7BAimage55.647.382.939.674.285.4
LLaVA-OneVision-7BAblind37.137.241.028.164.575.6
LLaVA-OneVision-7BBimage56.046.681.941.777.488.2
LLaVA-OneVision-7BBblind42.837.855.230.977.489.8
InternVL3-8BAimage56.540.583.845.390.394.8
InternVL3-8BAblind39.233.852.423.790.392.8
InternVL3-8BBimage55.341.285.739.690.393.6
InternVL3-8BBblind40.735.858.121.690.394.6

Table 14. CompHall accuracy (%) of three recent models. Chance = 33.3%. T4-L: 500 additional part–whole questions.

A larger part–whole test set. The part–whole test set has only 31 questions, so we built 500 further part–whole questions (T4-L) from Visual Genome images that appear in no other split, with the same generator and filters and one question per image (Appendix § 17). All six models score 67–95% with the image and 66–95% without it (Table 1 and Table 14). The image adds at most 9.8 pp (LLaVA-OneVision, first prompt), and under the second prompt every model is more accurate blind than with the image, significantly so for LLaVA-1.5-7B (90.2% vs. 83.2%) and Idefics2-8B (95.2% vs. 89.8%). These questions therefore measure language priors, not visual binding (Figure 1).

Figure 1. Accuracy on the 500 part–whole questions of T4-L with the image and with a gray image (blind), for six models under both prompts (bars: 95% Wilson intervals). Under the second prompt every model is at least as accurate blind as with the image.

Compositional Errors and Hallucination

The benchmark results show a correlation between question type and error rate. Question type is confounded with image source (T1 and T3 use COCO images, T2 and T4 use Visual Genome images), so the type-level comparison below cannot separate compositional difficulty from the effect of the image source. The synthetic-scene study in § 5.3 varies the number of objects within a single synthetic image source, but it probes object existence rather than binding.

Type-Level Error Gap

We partition CompHall questions into hard types (T1 Spatial, T3 Counting, requiring more simultaneous bindings) and easy types (T2 Attribute, T4 Part) and compute the error-rate gap per model (Figure 2).

Figure 2. Error-rate gap between hard (T1, T3) and easy (T2, T4) question types per model; a group’s error rate is the mean of its two types’ error rates. The gap is 27.5 pp for LLaVA-1.5-7B, 36.3 pp for LLaVA-NeXT-7B, and 39.2 pp for Idefics2-8B.

The gap widens as overall model accuracy increases (27.5 pp, 36.3 pp, 39.2 pp for LLaVA-1.5-7B, LLaVA-NeXT-7B, and Idefics2-8B respectively), indicating that stronger models improve on T2 and T4 while making smaller gains on T1 and T3 — consistent with compositional complexity having a structural effect not reduced by general capability improvements. With only three models the trend is descriptive rather than inferential, and the gap may partly reflect image source rather than compositional difficulty.

Per-Image Correlation with Object Hallucination

We test the link more directly on 200 test images. For each image we compute each model’s accuracy on the CompHall questions about it, and a hallucination rate from up to six yes/no existence questions asked of LLaVA-1.5-7B (“Is there a object in the image?”): three about objects named in the image’s annotations and three about objects that are not. The correlation is negligible for all three models (Pearson r=−0.05r=-0.05, p≥0.44p \ge0.44; Spearman ∣ρ∣≤0.07|\rho| \le0.07; Table 13), with a 95% interval for rr of about [−0.19,0.09][-0.19, 0.09]. The measurement is noisy: images have few questions, an object missing from the annotations may still be in the image, and one probe model serves all three rows, so only the first row compares a model with itself. The result excludes moderate correlations but gives no support for a link between per-image compositional accuracy and object hallucination.

ModelPearson rr (pp)Spearman ρ\rho (pp)ANOVA ppLowMidHigh
LLaVA-1.5-7B−0.053 (0.46)−0.069 (0.33)0.400.1120.0800.085
LLaVA-NeXT-7B−0.055 (0.44)−0.056 (0.43)0.750.1060.0940.085
Idefics2-8B−0.052 (0.46)−0.053 (0.45)0.0530.1160.0610.116

Table 13. Correlation between per-image CompHall accuracy and hallucination rate (200 images). ANOVA compares the three terciles.

Synthetic-Scene Study

We draw synthetic scenes of 1–5 flat coloured shapes in the style of CLEVR [6] (we do not use its renderer or images) and ask LLaVA-1.5-7B two yes/no object-existence questions per scene, one about a present and one about an absent colour (50 scenes per level; Appendix § 19). The error rate rises from 6% with one object to 42% with five (Figure 7), and 200 independently generated 2-object and 3-object scenes differ by 8.5 pp (t=3.39t=3.39, p=0.0008p=0.0008). These questions do not require binding, so the study measures scene clutter rather than binding; it uses one model and flat shapes.

Figure 7. Error rate of LLaVA-1.5-7B on object-existence questions in synthetic scenes with 1–5 objects (50 scenes per level, two questions each). Error bars are 95% Wilson intervals over scenes.

Compositional Contrastive Fine-Tuning

The benchmark results show that spatial and counting questions are hard and that part of the errors are binding-consistent. We design a training objective intended to reduce such errors. Compositional Contrastive Fine-Tuning (CCFT) fine-tunes LLaVA-1.5-7B (§ 8 adds two more models) on the CompHall training questions, using for each question the correct answer and its two hard negatives (§ 4). It combines three terms.

P1: Hard-negative contrast. Standard supervised fine-tuning (SFT) trains only on the correct answer and does not require the model to separate it from plausible wrong ones. CCFT also requires the model to score the correct answer higher than each hard negative by a margin.

P2: Softmax ranking. A cross-entropy over the scores of the correct answer and the two hard negatives, with the correct answer as the target, moves probability mass towards the correct answer and away from both negatives jointly.

P3: Language-modelling term. The standard language-modelling loss on the correct-answer sequences, which is the SFT objective used as a baseline in § 7.2.

Training Objective

For a question qq about an image xx with correct answer a+a^{+} and hard negatives a1−a_{1}^{-} and a2−a_{2}^{-}, we build the prompt “<image> Question: q Answer: a” for each candidate answer aa. The score ss of an answer is the negative mean token-level cross-entropy under the model. In our original implementation (v1) the mean runs over the whole sequence and is computed for a mini-batch, so the terms below compare mini-batch averages. In the corrected implementation (v2) it runs over the answer tokens only and is computed per example. Let s+s^{+} and sk−s^{-}_{k} denote the scores of the correct answers and of the kk-th hard negatives. The loss is

LCCFT=LLM+λcLcontrast+λrLrank(1)\mathcal{L}_{\mathrm{CCFT}}=\mathcal{L}_{\mathrm{LM}}+\lambda_{c}\mathcal{L}_{\mathrm{contrast}}+\lambda_{r}\mathcal{L}_{\mathrm{rank}} \tag*{(1)}

with

LLM=−s+(2)\mathcal{L}_{\mathrm{LM}}=-s^{+} \tag*{(2)}
Lcontrast=12∑k=12max⁡(0,m−s++sk−)(3)\mathcal{L}_{\mathrm{contrast}}=\frac{1}{2}\sum_{k=1}^{2}\max(0,m-s^{+}+s^{-}_{k}) \tag*{(3)}
Lrank=−log⁡es+es++es1−+es2−(4)\mathcal{L}_{\mathrm{rank}}=-\log\frac{e^{s^{+}}}{e^{s^{+}}+e^{s^{-}_{1}}+e^{s^{-}_{2}}} \tag*{(4)}

where m=0.5m=0.5, λc=0.5\lambda_{c}=0.5, and λr=0.3\lambda_{r}=0.3.

Implementation

We apply CCFT to LLaVA-1.5-7B [14] using LoRA [4] on the attention projections (rank 16, α=32\alpha=32, dropout 0.05). Training runs for 3 epochs on the CompHall training split (1,996 questions) with AdamW (learning rate 2×10−52 \times10^{-5}, weight decay 0.01, cosine schedule with 10% linear warmup), batch size 2, fp16 autocast, and gradient clipping at 1.0. We keep the epoch with the lowest validation loss, computed on 200 validation questions. All hyperparameters are listed in Appendix § 14. We apply the corrected loss to LLaVA-NeXT-7B and Idefics2-8B with the same LoRA rank, learning rate, schedule, epochs and loss weights and one question per optimizer step (for Idefics2-8B, LoRA is on the language model only; Appendix § 28). In v1 the score averages the token loss over all positions, 96.7% of which are the processor’s image-placeholder tokens (40 training questions, plain model) and which contribute 99.3% of the loss; the correct-versus-negative score gap is +0.004 on average and never reaches the margin of 0.5, so the contrastive and ranking terms carry almost no information about the answers. v2 scores only the answer tokens, per example (gap +0.83; margin already met for 47.5% of pairs); everything else is unchanged.

What CCFT Teaches on CompHall

CCFT vs. Baseline

Table 2 compares LLaVA-1.5-7B with and without CCFT, using the same prompt and parser for every run. The prompt matters: with the wording used here, the plain model scores 48.5% rather than the 38.3% of Table 1, so about 10 of the 12.3 pp between our first CCFT model and the Table 1 baseline come from the prompt alone. We report two implementations, each trained with three seeds (42, 1, 2). The original (v1) averages the token loss over all positions, 96.7% of which are image placeholders (§ 6); the corrected one (v2) scores only the answer tokens, per example, with the same weights and margin.

ModelOverallT1T2T3T4Blind
Plain model48.541.273.332.471.039.2
CCFT v149.2±1.941.965.736.783.944.8±1.5
CCFT v251.1±0.547.372.432.679.644.6±0.5

Table 2. CompHall accuracy (%) of LLaVA-1.5-7B, with the image (overall and by type) and without it (Blind). CCFT rows: mean over three seeds (± standard deviation); same prompt and parser throughout. Chance = 33.3%.

With the image, v1 scores 49.2 ± 1.9% and v2 51.1 ± 0.5%, against 48.5% for the plain model. No seed of either differs significantly from the plain model (exact McNemar p≥0.47p\geq0.47 for v1; p=0.09p=0.09, 0.25, 0.15 for v2, which fixes 29–32 questions and breaks 19–20 per seed). Without the image, v1 scores 44.8 ± 1.5% and v2 44.6 ± 0.5%, against 39.2% (p≤0.02p\leq0.02 for every v2 seed). The image therefore adds 6.5 pp after v2 fine-tuning (51.1% vs. 44.6%) against 9.3 pp for the plain model (48.5% vs. 39.2%): the fine-tuning improves accuracy without the image more than with it. On CompHall we find no evidence that CCFT improves the visual binding of LLaVA-1.5-7B. By type, the plain model uses the image on spatial questions (32.4% blind vs. 41.2% with the image), but after either fine-tuning the image adds nothing (v2: 48.0% vs. 47.3%; Table 7 and Figure 5, so the spatial gain of CCFT is a text effect. On the larger part–whole set (T4-L), v2 raises LLaVA-1.5-7B from 83.2% to 87.7 ± 2.0% with the image (each seed p≤0.0014p \le0.0014), but the plain model is already more accurate blind (90.2%) and v2 blind reaches 91.1 ± 0.8%, so here fine-tuning mostly removes the image’s harm; v1 is unstable (87.8 ± 5.1%, seeds 82.8–93.0%; Table 15).

Figure 5. Image contribution (accuracy with the image minus accuracy with a gray image) by question type. In LLaVA-1.5-7B, fine-tuning removes the image’s contribution to spatial (T1) questions; in LLaVA-NeXT-7B, the corrected loss raises it for spatial and counting (T3) questions. Bars are means over three seeds (LLaVA-NeXT original loss: one run); dots are the individual LLaVA-NeXT v2 seeds.

InputConditionT1T2T3T4Overall
LLaVA-1.5-7B (CCFT and controls: mean of three seeds)
ImageBaseline41.273.332.471.048.5
ImageCCFT v141.965.736.783.949.2
ImageCCFT v247.372.432.679.651.1
ImageNon-spatial only45.073.736.281.751.9
ImageSpatial only52.369.532.168.851.1
BlindBaseline32.450.530.274.239.2
BlindCCFT v143.554.030.284.944.8
BlindCCFT v248.054.325.780.644.6
BlindNon-spatial only41.752.129.078.542.8
BlindSpatial only46.446.727.868.842.0
LLaVA-NeXT-7B (CCFT v2 and controls: mean of three seeds; baseline and v1: one run)
ImageBaseline44.679.036.777.453.0
ImageCCFT v141.278.134.5100.052.5
ImageCCFT v259.279.042.077.459.8
ImageNon-spatial only44.679.742.789.255.9
ImageSpatial only60.181.033.369.957.2
BlindBaseline35.151.424.580.639.0
BlindCCFT v142.659.036.0100.048.7
BlindCCFT v243.756.223.776.342.6
BlindNon-spatial only39.958.123.581.742.1
BlindSpatial only38.751.122.161.338.0

Table 7. Per-type accuracy (%) with the image and blind (gray image).

ModelConditionImageppBlindpp
LLaVA-1.5-7BPlain83.2–90.2–
LLaVA-1.5-7Bv1, seed 4287.60.00490.21.00
LLaVA-1.5-7Bv1, seed 193.0<0.000191.60.43
LLaVA-1.5-7Bv1, seed 282.80.9184.20.001
LLaVA-1.5-7Bv2, seed 4290.0<0.000192.00.09
LLaVA-1.5-7Bv2, seed 186.40.000991.00.39
LLaVA-1.5-7Bv2, seed 286.80.001490.41.00
LLaVA-NeXT-7BPlain80.6–83.8–
LLaVA-NeXT-7Bv198.4<0.000199.2<0.0001
LLaVA-NeXT-7Bv2, seed 4283.00.1385.80.17
LLaVA-NeXT-7Bv2, seed 184.20.00686.80.02
LLaVA-NeXT-7Bv2, seed 279.00.3783.20.78

Table 15. Accuracy (%) on T4-L (n=500n=500) and McNemar pp against the plain model.

Figure 3. Accuracy with the image for LLaVA-1.5-7B: plain model vs. CCFT v1 and v2 (means over three seeds; bars show the minimum–maximum range over seeds). The dashed line marks chance (33.3%).

Ablation Study

We ablate the loss terms with the corrected loss (seed 42; same budget, prompt and parser as Table 2). SFT only (answer-token language modelling) scores 51.3%, the contrastive term alone 51.8%, the ranking term alone 52.7%, and the full objective 51.5% (Table 3). All four lie within 1.4 pp of each other, and the seed-to-seed spread of the full objective is 0.5 pp. Without the image they score 45.2–45.6%. The ranking-only model is the only one that differs significantly from the plain model with the image (p=0.020p=0.020), which would not survive correction for the seven comparisons we ran. The contrastive and ranking terms therefore add nothing measurable to fine-tuning on the answers alone.

ConditionImageppBlind
Plain model48.5–39.2
SFT only51.30.11145.2
SFT + Contrastive51.80.07645.4
SFT + Ranking52.70.02045.6
CCFT Full51.50.09245.2

Table 3. Ablation of the loss terms (corrected loss, seed 42): overall accuracy (%) with and without the image, and the exact McNemar pp against the plain model with the image.

What Transfers: Independent Benchmarks and Controls

CompHall Across Three Model Families

LLaVA-NeXT-7B. We apply CCFT to it (the same prompt and parser as the baseline of this run, which scores 53.0% with the image and 39.0% without it). The original loss, one run, gives 52.5% (fixes 69 questions, breaks 71; McNemar p=0.93p=0.93) and 48.7% without the image: the image adds only 3.8 pp instead of 14.0, and the model scores 100% on T4 and 98.4% on T4-L with the image (99.2% blind). The corrected loss, over three seeds, gives 59.8±0.4%59.8 \pm0.4\% (+6.9 pp; p=0.0056p=0.0056, 0.0078, 0.0015 per seed) and 42.6±1.4%42.6 \pm1.4\% without the image (+3.6 pp; p=0.023p=0.023, 0.25, 0.23). The gain is concentrated in spatial (44.6% to 59.2%) and counting (36.7% to 42.0%) questions, where the image’s contribution grows in every seed: the spatial image gain is +9.5 pp for the baseline against +14.2, +14.9 and +17.6 pp, and the counting gain +12.2 pp against +18.2 pp on average (Table 7 in the appendix; Figure 5). This is unlike LLaVA-1.5-7B. On T4-L the corrected loss does not help consistently (82.1±2.7%82.1 \pm2.7\% vs. 80.6%; p=0.13p=0.13, 0.006, 0.37).

Idefics2-8B. To test a model with a different vision encoder, connector and language model (SigLIP, perceiver resampler, Mistral-7B), we train Idefics2-8B with the corrected loss on all question types, on the non-spatial types only and on spatial questions only, three seeds each (Appendix § 28). Without fine-tuning it scores 52.2% with the image and 45.9% without it. Fine-tuning does not change its CompHall accuracy: 52.6±0.4%52.6 \pm0.4\% (all types), 53.6±0.8%53.6 \pm0.8\% (non-spatial) and 51.2±0.5%51.2 \pm0.5\% (spatial only), with McNemar p≥0.40p \ge0.40 in every seed, and even spatial-only training leaves its spatial accuracy at 42.1±1.0%42.1 \pm1.0\% (plain: 42.6%). Its blind accuracy rises slightly (to 48.8±1.0%48.8 \pm1.0\% and 49.9±0.6%49.9 \pm0.6\% for the first two arms), mainly on counting (30.9% to 39.3% and 38.6%; p≤0.04p \le0.04 in every seed) and attribute questions (52.4% to 59.7% and 61.9%), which is a gain from text priors, not from the image (Table 25). The in-distribution improvement is therefore clear in only one of the three families: it is significant for LLaVA-NeXT-7B, small and not significant for LLaVA-1.5-7B, and not detectable for Idefics2-8B.

InputArmT1T2T3T4Overall
With imagePlain42.673.338.887.152.2
All types39.4±2.239.4 \pm2.279.0±1.979.0 \pm1.939.8±1.139.8 \pm1.183.9±3.283.9 \pm3.252.6±0.452.6 \pm0.4
Non-spatial40.1±1.440.1 \pm1.479.4±2.979.4 \pm2.941.5±1.141.5 \pm1.184.9±1.984.9 \pm1.953.6±0.853.6 \pm0.8
Spatial only42.1±1.042.1 \pm1.075.2±1.675.2 \pm1.635.3±1.935.3 \pm1.984.9±6.784.9 \pm6.751.2±0.551.2 \pm0.5
BlindPlain46.652.430.987.145.9
All types41.2±1.441.2 \pm1.459.7±3.859.7 \pm3.839.3±1.139.3 \pm1.190.3±0.090.3 \pm0.048.8±1.048.8 \pm1.0
Non-spatial43.0±2.643.0 \pm2.661.9±1.961.9 \pm1.938.6±0.838.6 \pm0.892.5±1.992.5 \pm1.949.9±0.649.9 \pm0.6
Spatial only41.2±2.941.2 \pm2.953.3±0.053.3 \pm0.030.7±1.730.7 \pm1.787.1±0.087.1 \pm0.044.1±1.644.1 \pm1.6

Table 25. Idefics2-8B on CompHall: accuracy (%) with the image and with a gray image (mean ±\pm sd over three seeds). With the image, no arm differs from the plain model overall (McNemar p≥0.40p \ge0.40 in every seed) or on T1 (p≥0.19p \ge0.19).

POPE Evaluation

We evaluate on the POPE benchmark [10] (n=444n=444 questions per split) to measure how CCFT affects the general object-existence accuracy of LLaVA-1.5-7B (Table 4; the figure is in Appendix § 29); we did not run POPE for the other models. POPE asks whether an object is present in the image. The probe is balanced, so a model with no bias answers “yes” about half of the time, and a yes-bias above 0.5 means that it claims absent objects too readily.

SplitAccuracyYes-bias
Plainv1v2Plainv1v2
Random0.8670.7250.8930.3810.7480.494
Popular0.8420.6490.8390.4050.8240.549
Adversarial0.8180.6100.7700.4300.8630.617

Table 4. POPE accuracy and yes-bias (fraction of “yes” answers) for the plain model, CCFT v1 (original loss) and CCFT v2 (corrected loss), each the mean of three seeds.

CCFT v1 reduces POPE accuracy on all three splits (adversarial: 0.818 to 0.610±0.0920.610 \pm0.092) and raises the yes-bias from 0.430 to 0.863±0.1160.863 \pm0.116; it is also unstable across seeds (accuracy standard deviation 0.09–0.16). The corrected loss is much less harmful and stable (standard deviation at most 0.009): accuracy changes by +2.6 pp (random), −0.3 pp (popular) and −4.8 pp (adversarial), but it still shifts the model towards “yes” on every split (adversarial: 0.430 to 0.617±0.0210.617 \pm0.021; recall +13.9 pp, precision −15.0 pp). The shift does not come from the contrastive or ranking terms: fine-tuning on the answers alone gives an adversarial yes-bias of 0.588 (accuracy 0.773), against 0.586–0.606 with either added term and 0.617 with both (Table 9).

SplitModelPrecisionRecallF1
RandomPlain0.9820.7480.849
v10.6720.9730.789
v20.8980.8870.893
SFT only0.9050.8600.882
Contrastive0.9060.8650.885
Ranking0.8980.8740.886
PopularPlain0.9220.7480.826
v10.6010.9730.740
v20.8090.8870.846
SFT only0.8160.8600.838
Contrastive0.8210.8650.842
Ranking0.8150.8740.843
AdversarialPlain0.8690.7480.804
v10.5690.9730.716
v20.7190.8870.794
SFT only0.7320.8600.791
Contrastive0.7380.8650.797
Ranking0.7210.8740.790

Table 9. Full POPE results: precision, recall and F1.

Figure 4. Overall accuracy with the image (left) and with a gray image (right) for the plain model and the two CCFT implementations. Means over three seeds (bars: standard deviation), except the original loss on LLaVA-NeXT (one run).

An Independent Benchmark: CV-Bench

CompHall’s questions share a generator with the fine-tuning data, so gains on it may reflect its templates. We therefore evaluate the same models on the 2D tasks of CV-Bench [22]: counting and spatial-relation questions with 4–6 and 2 options (chance 23% and 50%), on different images and with different question writers. We drop the 160 questions whose COCO image appears in CompHall, leaving 1,278 questions (723 counting and 555 spatial-relation questions), and use the prompts of the CompHall evaluations, with and without the image (Appendices § 22 and § 23). A model can gain or lose accuracy on a two-option question simply by changing which letter or word it prefers, so we ask every spatial-relation question twice, with the options in both orders, and report order-averaged accuracy (a model that always picks the same position scores exactly 50%) and consistency, the share of questions answered with the same option content in both orders. Significance is a paired bootstrap over questions for each seed. Counting has 4–6 options and is reported in the benchmark’s own order.

Spatial relations. The transfer is positive in all three families, but its size differs a lot (Figure 6; Table 21). Averaged over both option orders, accuracy with the image goes from 57.6% to 60.8±0.1%60.8 \pm0.1\% for LLaVA-1.5-7B (+3.2 points), from 62.8% to 85.9±1.9%85.9 \pm1.9\% for LLaVA-NeXT-7B (+23.1) and from 63.8% to 69.8±1.8%69.8 \pm1.8\% for Idefics2-8B (+6.0), and every seed is significant (paired bootstrap, p≤0.002p \le0.002). LLaVA-NeXT’s answers are also stable when the options are reordered (89% consistent), and its gain is nearly the same in the benchmark’s own order (+24.6).

Figure 6. Accuracy averaged over both option orders on CV-Bench spatial relations (left) and VSR (right), for the plain models, both CCFT implementations (the original loss was run on the LLaVA models only) and the two control arms. Bars are means over three seeds (LLaVA-NeXT original loss: one run), dots the individual seeds.

ModelArmOriginalReversedAveragedConsistentGain (per seed), pp
LLaVA-1.5-7BPlain60.055.157.675.7–
CCFT v157.349.753.5±1.653.5\pm1.651.9−2.9 to −5.9; p≤.049p\le.049
CCFT v266.754.960.8±0.160.8\pm0.174.4+3.2 to +3.3; p<.0001p<.0001
Non-spatial61.953.357.6±0.957.6\pm0.976.7−0.9 to +0.9; p≥.16p\ge.16
Spatial only72.956.264.5±1.364.5\pm1.368.8+5.9 to +8.5; p<.0001p<.0001
LLaVA-NeXT-7BPlain64.161.462.873.2–
CCFT v1 (one run)50.155.552.885.9−10.0; p<.0001p<.0001
CCFT v288.783.185.9±1.985.9\pm1.988.6+21.2 to +24.9; p<.0001p<.0001
Non-spatial69.562.766.1±2.666.1\pm2.671.8+0.6 to +5.9; p=.66,.002,<.0001p=.66, .002, <.0001
Spatial only89.085.187.1±0.687.1\pm0.690.9+23.6 to +24.8; p<.0001p<.0001
Idefics2-8BPlain68.559.163.865.8–
CCFT v269.370.469.8±1.869.8\pm1.864.8+4.1 to +7.7; p≤.002p\le.002
Non-spatial66.565.766.1±1.366.1\pm1.370.7+1.5 to +3.9; p=.20,.0016,.18p=.20, .0016, .18
Spatial only70.666.968.7±1.968.7\pm1.974.0+3.9 to +7.1; p≤.002p\le.002

Table 21. CV-Bench spatial relations: accuracy (%) by option order (mean over seeds), averaged accuracy (±\pm sd over seeds), consistency, and per-seed gain over the plain model.

For the other two models the original order is misleading, in opposite directions. LLaVA-1.5-7B gains +6.7 points there (60.0% to 66.7±0.8%66.7 \pm0.8\%), about twice its order-averaged gain. Idefics2-8B gains only +0.8 (68.5% to 69.3±2.4%69.3 \pm2.4\%, not significant), because its plain model is nine points better in the original order than in the reversed one (68.5% vs. 59.1%) and fine-tuning removes that asymmetry. The gain needs the image: blind accuracy stays between 48.8% and 53.5% for the LLaVA models in every arm, and between 53.2% and 58.7% for Idefics2-8B (55.9% for its plain model).

The original loss degrades both LLaVA models. Order-averaged, it lowers spatial-relation accuracy by 4.1 pp (LLaVA-1.5-7B; p≤0.049p \le0.049 in every seed) and 10.0 pp (LLaVA-NeXT-7B), and in the original order it lowers counting from 53.3% to 39.1±2.7%39.1 \pm2.7\% for LLaVA-1.5-7B (p<0.0001p < 0.0001 in each seed) and from 63.1% to 12.6% for LLaVA-NeXT-7B (chance 23%; p<0.0001p < 0.0001).

Counting. With the corrected loss on all types, counting does not improve in any family: 53.3% to 48.2±1.0%48.2 \pm1.0\% for LLaVA-1.5-7B (p≤0.005p \le0.005 in each seed), 63.1% to 64.2±2.1%64.2 \pm2.1\% for LLaVA-NeXT-7B (not significant) and 61.5% to 59.0±0.8%59.0 \pm0.8\% for Idefics2-8B (p=0.048,0.29,0.10p = 0.048, 0.29, 0.10). Spatial-only training does raise it for two of the three models (Section 8.4).

The blind control is informative only for spatial relations. On a blank image the plain models answer “0” to 32–99% of the counting questions, and “0” is the correct answer for only 0.1% of them, so their blind counting accuracy (1–22%) lies below chance and says nothing about text priors. Fine-tuning changes this, which by itself raises blind counting accuracy (LLaVA-1.5-7B: “0” in 16–39% of the answers of the original-loss models against 70% for the plain model; Idefics2-8B: blind counting accuracy rises from 7.7% to 25–31% in the arms trained on counting questions).

The six plain models. Accuracy with the image ranges from 49% to 90% on spatial relations and from 44% to 73% on counting (Table 17), and blind spatial accuracy stays at 49–60%, so, unlike CompHall’s part–whole questions, this benchmark cannot be answered without the image. The ranking of the six models agrees with CompHall’s for counting (Spearman ρ=0.94\rho= 0.94 and 0.60 for the two prompts) and less for spatial questions (ρ=0.37\rho= 0.37 and 0.46; n=6n=6, so descriptive only).

ModelPromptImage CountImage RelationBlind CountBlind Relation“0”
LLaVA-1.5-7BA44.349.021.949.031.5
B53.360.016.049.769.8
LLaVA-NeXT-7BA62.557.710.250.380.6
B60.262.24.850.590.6
Idefics2-8BA62.464.74.458.791.6
B62.569.44.756.292.8
Qwen2.5-VL-7BA64.078.79.053.285.3
B66.586.33.055.793.4
LLaVA-OneVision-7BA70.578.418.454.859.3
B70.481.37.159.687.3
InternVL3-8BA73.089.90.755.599.4
B71.889.40.756.098.9

Table 17. The six plain models on CV-Bench (%), under both prompts. “0”: share of blank-image counting questions answered “0”.

Where Does the Gain Come From?

The corrected loss is trained on all four question types. To see which supervision the CV-Bench gain needs, we train two control arms with the same loss, hyper-parameters and seeds: one on the non-spatial types only (T2–T4; 1,314 questions) and one on spatial questions only (T1; 682 questions), against 1,996 for the full set. All three models have three seeds per arm (Appendices § 23 and § 28).

Spatial supervision carries most of the gain. Averaged over both orders, training on spatial questions alone raises accuracy by 6.9, 24.3 and 4.9 points (LLaVA-1.5-7B, LLaVA-NeXT-7B, Idefics2-8B; significant in every seed), against 3.2, 23.1 and 6.0 points for all question types. The non-spatial types alone give 0.0, 3.3 and 2.3 points, that is 0%, 14% and 47% of the spatial-only gain, and only three of the nine individual runs (three seeds for each of the three models) are significant (Table 21). Mixing the types dilutes the transfer for LLaVA-1.5-7B but not for the other two models. In the benchmark’s own order the same arms give +1.9, +5.4 and −2.0 points (non-spatial) and +12.9, +24.9 and +2.1 points (spatial only; Table 18).

ModelArmOverallCountRelationpp (Relation, per seed)Blind rel.
LLaVA-1.5-7BPlain56.253.360.0–49.7
CCFT v147.0±2.439.1±2.757.3±3.3.71 / .89 / .0353.5
CCFT v256.2±0.948.2±1.066.7±0.8<.0001 (all)49.7
Non-spatial55.9±0.951.4±0.861.9±1.2.04 / .14 / .6048.9
Spatial only62.3±0.554.1±0.672.9±1.5<.0001 (all)53.0
LLaVA-NeXT-7BPlain63.563.164.1–49.7
CCFT v1 (one run)28.912.650.1<.000148.8
CCFT v274.9±0.964.2±2.188.7±0.6<.0001 (all)51.5
Non-spatial66.2±2.063.6±1.269.5±3.6.52 / <.0001 / <.000150.2
Spatial only76.7±0.967.3±1.389.0±0.5<.0001 (all)52.3

Table 18. CV-Bench accuracy (%) with the image in the original option order, by arm (mean ± sd over seeds), exact McNemar pp of the spatial-relation accuracy against the plain model for each seed, and blind spatial-relation accuracy. LLaVA-NeXT CCFT v1 is a single run (no spread).

It holds on images from another dataset. Split by image source, spatial-only training raises LLaVA-1.5-7B from 61.9% to 71.4% on ADE20K images and from 58.0% to 74.5% on COCO images (p≤0.001p \le0.001 in every seed), and LLaVA-NeXT-7B from 61.2% to 86.6% and from 67.4% to 91.7% (original order; Table 20); we did not split Idefics2-8B’s results by source.

ModelArmADE20KCOCO
LLaVA-1.5-7BPlain61.958.0
CCFT v157.257.4
CCFT v264.968.6
Non-spatial61.262.8
Spatial only71.474.5
LLaVA-NeXT-7BPlain61.267.4
CCFT v150.250.0
CCFT v285.592.3
Non-spatial67.871.3
Spatial only86.691.7

Table 20. CV-Bench spatial-relation accuracy (%) with the image in the original option order, by image source (ADE20K: 291 questions; COCO: 264 questions after excluding images shared with CompHall).

It reaches counting after spatial-only training in two of three families. Spatial-only training raises counting for LLaVA-NeXT-7B (63.1% to 67.3 ± 1.3%; p=0.12p = 0.12, 0.0002, 0.005) and for Idefics2-8B (61.5% to 65.4 ± 0.4%; p≤0.001p \le0.001 in every seed), but for LLaVA-1.5-7B no arm does (p>0.05p > 0.05 for the spatial-only arm in every seed). The other arms do not help counting in any family (LLaVA-NeXT-7B: all types 64.2 ± 2.1%, non-spatial 63.6 ± 1.2%; Idefics2-8B: 59.0 ± 0.8% and 58.6 ± 0.7%).

On CompHall the arms separate image-dependent from text-only gains for the LLaVA models. Spatial-only training is the only arm with a consistent image-conditioned spatial gain for LLaVA-1.5-7B (+11.1 pp; p≤0.03p \le0.03 in each seed). For LLaVA-NeXT-7B it raises spatial accuracy by +15.5 pp with the image but only +3.6 pp blind (three seeds), so the image’s contribution grows from +9.5 to +21.4 pp, while the non-spatial arm leaves spatial accuracy unchanged (44.6%). All three LLaVA-1.5-7B arms raise its blind spatial accuracy (p≤0.03p \le0.03 in every seed; Table 7). For Idefics2-8B no arm changes CompHall accuracy (§ 8.1).

Caveats. The non-spatial arm’s gain on LLaVA-NeXT-7B varies from +1.3 to +8.2 pp across seeds in the original order, which we cannot explain; the spatial-only arm of LLaVA-NeXT-7B also raises blind spatial-relation accuracy slightly (52.3% vs. 49.7%; p≤0.02p \le0.02 in two of three seeds); and these controls cannot tell whether the gain reflects spatial reasoning or the specific spatial vocabulary of CompHall’s options (§ 8.6).

True/False Statements and Answer Biases

On VSR [12], true/false statements about spatial relations (1,218 questions; always answering “true” scores 51.4%), the transfer is small at most. Averaged over both option orders, training on spatial questions alone gives LLaVA-NeXT-7B a small but consistent gain (61.7% to 64.8 ±\pm 0.3%, significant in every seed) and leaves LLaVA-1.5-7B (56.1% to 56.3%) and Idefics2-8B (77.1% to 77.0%) unchanged. The other arms leave LLaVA-NeXT-7B unchanged, lower Idefics2-8B slightly (by 1.0 and 1.5 points) and lower LLaVA-1.5-7B by about 3 points (p<0.0001p < 0.0001 in every seed); the original loss lowers both LLaVA models (−4.7 and −10.0 points). In the benchmark’s own order the LLaVA drops look much larger (6–14 points) because of the answer biases described next (Table 19 and Table 22).

ModelArmImageBlind“A” answers
LLaVA-1.5-7BPlain60.249.874
CCFT v151.4±0.251.698–100
CCFT v254.5±0.451.092–93
Non-spatial54.3±0.552.092–94
Spatial only60.7±0.749.175–80
LLaVA-NeXT-7BPlain65.048.679
CCFT v151.451.4100
CCFT v258.3±2.548.684–92
Non-spatial58.4±2.548.685–92
Spatial only65.5±0.848.572–76

Table 19. VSR accuracy (%, original option order, nn=1,218; always answering “true” scores 51.4%) with and without the image, and the share of “A” (true) answers with the image (range over seeds). v1 on LLaVA-NeXT is a single run.

ModelArmOriginalReversedAveragedConsistentGain (per seed), pp
LLaVA-1.5-7BPlain60.252.156.174.8–
CCFT v151.451.451.4±0.051.4\pm0.098.5−4.7; p<.0001p<.0001
CCFT v254.551.853.1±0.253.1\pm0.293.8−2.8 to −3.2; p<.0001p<.0001
Non-spatial54.351.853.0±0.253.0\pm0.293.8−2.8 to −3.2; p<.0001p<.0001
Spatial only60.751.956.3±0.456.3\pm0.479.4−0.1 to +0.6; p≥.12p\ge.12
LLaVA-NeXT-7BPlain65.058.561.735.8–
CCFT v1 (one run)51.452.151.899.1−10.0; p<.0001p<.0001
CCFT v258.365.461.9±1.361.9\pm1.378.8−1.3 to +1.3; p≥.32p\ge.32
Non-spatial58.466.662.5±2.362.5\pm2.358.3−1.6 to +3.1; p=.07,.002,.50p=.07, .002, .50
Spatial only65.564.164.8±0.364.8\pm0.361.6+2.8 to +3.3; p≤.002p\le.002
Idefics2-8BPlain76.977.377.193.3–
CCFT v274.977.476.1±1.176.1\pm1.187.0−2.2 to −0.1; p=.48,.0012,.88p=.48, .0012, .88
Non-spatial74.576.675.6±0.475.6\pm0.488.3−1.9 to −1.1; p=.04p=.04
Spatial only77.276.777.0±0.277.0\pm0.292.6−0.4 to +0.0; p≥.60p\ge.60

Table 22. VSR: accuracy (%) by option order (mean over seeds), averaged accuracy (±\pm sd over seeds), consistency, and per-seed gain over the plain model.

The biases differ by model. In the original order option A is “true” and B is “false”, and in the reversed order they swap; consistent answers are those with the same meaning in both orders. LLaVA-1.5-7B follows the word: its plain model answers “true” for 74% of the statements in the original order and 98.5% in the reversed order, and the fine-tuned models other than the spatial-only ones for 92–100% in both, with 94–99% consistent answers, so fine-tuning on other question types has taught it an affirmation bias. LLaVA-NeXT-7B’s plain model answers “A” for 79% and 85% of the statements in the two orders (36% consistent answers), a position bias; only its original-loss model answers “true” regardless of order (consistency 99%). Idefics2-8B’s plain model is consistent (93% of its answers stay the same when the order flips) with a mild preference for “true” (62% and 57%).

Held-Out Relation Words

We ran this test for the LLaVA models only. CompHall’s spatial questions use a few direction words, and CV-Bench asks about the same words, so the gain could reflect a learned mapping from those words to image layout rather than spatial reasoning. We test this with two further arms (same loss, hyper-parameters and seeds; three seeds each): one trained only on spatial questions whose correct answer is left or right (about 500 questions), the other only on those whose correct answer is above or below (158 questions). In each arm the words of the other axis occur only as wrong options. We test on CV-Bench spatial relations split by the axis of the correct answer (331 horizontal and 224 vertical questions, averaged over both option orders), where both options of a question belong to the same axis, and on CompHall’s own spatial test questions (103 and 41; Appendix § 25).

A large part of the gain transfers to words never seen as correct answers. For LLaVA-NeXT-7B the horizontal-only arm raises order-averaged accuracy on the held-out vertical questions from 65.4% to 78.8 ±\pm 0.8% (+13.4 pp; p<0.01p < 0.01 in every seed), 66% of the spatial-only arm’s gain on that axis (85.8%), and the vertical-only arm raises the held-out horizontal questions from 61.0% to 78.2 ±\pm 2.7% (+17.2 pp; p<0.01p < 0.01 in every seed), 64% of the spatial-only gain (87.9%); with only 158 training questions it reaches almost the same accuracy on the held-out axis as on its own (79.4%; Table 23). The picture for LLaVA-1.5-7B is smaller and less consistent: the held-out gains are +2.6 pp (vertical, 66.5% to 69.1%; significant in one of three seeds) and +3.9 pp (horizontal, 51.5% to 55.4%; significant in every seed), against +6.3 and +7.4 pp for the spatial-only arm.

Table 23. CV-Bench spatial relations, accuracy (%) averaged over both option orders, by the axis of the correct answer (331 horizontal, 224 vertical questions); mean ± sd over three seeds and per-seed pp.

ModelArmHorizontal-correctVertical-correct
LLaVA-1.5-7BPlain51.566.5
All types54.2±0.554.2\pm0.5; p<.01p < .01 (all)70.5±0.570.5\pm0.5; p<.01,<.01,.07p < .01, < .01, .07
Spatial only58.9±1.258.9\pm1.2; p<.01p < .01 (all)72.8±1.972.8\pm1.9; p<.01,<.01,.02p < .01, < .01, .02
Horizontal only57.3±3.657.3\pm3.6; p<.01p < .01 (all)69.1±1.869.1\pm1.8*; p<.01,.12,.74p < .01, .12, .74
Vertical only55.4±0.755.4\pm0.7*; p<.01,<.01,.01p < .01, < .01, .0168.4±0.368.4\pm0.3; p=.28,.37,.44p = .28, .37, .44
LLaVA-NeXT-7BPlain61.065.4
All types88.4±0.488.4\pm0.4; p<.01p < .01 (all)82.1±4.482.1\pm4.4; p<.01p < .01 (all)
Spatial only87.9±0.887.9\pm0.8; p<.01p < .01 (all)85.8±0.685.8\pm0.6; p<.01p < .01 (all)
Horizontal only85.9±0.585.9\pm0.5; p<.01p < .01 (all)78.8±0.878.8\pm0.8*; p<.01p < .01 (all)
Vertical only78.2±2.778.2\pm2.7*; p<.01p < .01 (all)79.4±2.679.4\pm2.6; p<.01p < .01 (all)

Table 23.

But not all of it. The trained axis does better than the held-out one for the horizontal-only arm (85.9% against 78.8% for LLaVA-NeXT-7B), and in both LLaVA models the held-out axis stays below the spatial-only arm. The gain is therefore partly general and partly specific to the words seen as correct answers.

CompHall’s own test shows no transfer. On CompHall no arm improves on the plain model on its held-out axis (for example, LLaVA-NeXT-7B trained on left/right only scores 41.5% against 46.3% for the plain model on the 41 vertical questions; Table 24). This is what we would expect if the arms learn to reject words they only ever saw as wrong options: in CompHall the held-out words appear in training only as wrong options, whereas on CV-Bench both options belong to the held-out axis, so that shortcut cannot help. The vertical test set is small and we did not test this explanation directly.

Table 24. CompHall spatial test questions, accuracy (%) with the image in the original option order, by the axis of the correct answer (103 horizontal, 41 vertical questions); mean ± sd over three seeds and per-seed pp.

ModelArmHorizontal-correctVertical-correct
LLaVA-1.5-7BPlain41.739.0
All types52.4±1.052.4\pm1.0; p<.01,.02,<.01p < .01, .02, < .0135.0±3.735.0\pm3.7; p=.62,1.0,.38p = .62, 1.0, .38
Spatial only55.7±1.155.7\pm1.1; p<.01,.04,.01p < .01, .04, .0145.5±3.745.5\pm3.7; p=1.0,.51,.22p = 1.0, .51, .22
Horizontal only50.8±2.050.8\pm2.0; p<.01,.09,.19p < .01, .09, .1940.7±3.740.7\pm3.7*; p=.62,1.0,1.0p = .62, 1.0, 1.0
Vertical only38.2±1.138.2\pm1.1*; p=.12,.61,.55p = .12, .61, .5545.5±1.445.5\pm1.4; p=.69,.51,.51p = .69, .51, .51
LLaVA-NeXT-7BPlain42.746.3
All types63.1±1.963.1\pm1.9; p<.01p < .01 (all)49.6±5.149.6\pm5.1; p=.80,1.0,.58p = .80, 1.0, .58
Spatial only61.8±1.161.8\pm1.1; p<.01p < .01 (all)56.1±2.456.1\pm2.4; p=.61,.30,.45p = .61, .30, .45
Horizontal only63.4±1.563.4\pm1.5; p<.01p < .01 (all)41.5±4.941.5\pm4.9*; p=.42,.80,1.0p = .42, .80, 1.0
Vertical only36.6±2.236.6\pm2.2*; p=.30,.08,.33p = .30, .08, .3368.3±2.468.3\pm2.4; p=.02,.06,.01p = .02, .06, .01

Table 24.

Discussion

What hard-negative fine-tuning teaches. On CompHall itself, fine-tuning does not teach a general compositional skill. LLaVA-1.5-7B changes by +2.6 points with the image (not significant) and by +5.4 without it (p≤0.02p \le0.02 in every seed), and the image’s contribution to its spatial questions disappears (from +8.8 to −0.7 points), so what it gains comes more from the question text than from the image. LLaVA-NeXT-7B is the exception: its spatial and counting accuracy rise and the image contributes more in all three seeds (+6.9 points overall). Idefics2-8B does not change, apart from a rise in blind accuracy on counting and attribute questions that comes from text priors. We therefore find no evidence that this kind of fine-tuning improves visual binding on CompHall for two of the three models.

The independent benchmark tells a more positive but still limited story. Averaged over both option orders, the corrected loss raises accuracy on CV-Bench’s spatial-relation questions in all three families: strongly for LLaVA-NeXT-7B (62.8% to 85.9%), modestly for LLaVA-1.5-7B (57.6% to 60.8%) and Idefics2-8B (63.8% to 69.8%). The control arms show where the gain comes from. Training on spatial questions alone gives +6.9, +24.3 and +4.9 points, the other question types alone give 0.0, +3.3 and +2.3 points, and all types together give +3.2, +23.1 and +6.0 points. For the LLaVA models the spatial-only gain holds on images from ADE20K as well as COCO, and for LLaVA-NeXT-7B and Idefics2-8B it extends to counting (+4.2 and +3.9 points), but not for LLaVA-1.5-7B. Training on left/right questions only (or above/below only) gives LLaVA-NeXT-7B about two-thirds of the spatial-only gain on the held-out axis (+13.4 and +17.2 points) and LLaVA-1.5-7B a smaller, less consistent part (+2.6 and +3.9), but nothing on CompHall’s own held-out questions, so the gain is partly general and partly tied to the words seen as correct answers.

Two cautions apply. The gain is largely tied to the multiple-choice format: on VSR’s true/false statements it is about 3 points at most and only for LLaVA-NeXT-7B, and models fine-tuned on other question types can lose up to 3 points (LLaVA-1.5-7B becomes biased toward “true”). And the benchmark’s own option order can hide or exaggerate a gain, which is why we report order-averaged accuracy.

The in-distribution and external results are compatible. Fine-tuning on CompHall’s spatial questions can teach the question format and its cues, which raise blind accuracy there, and, for LLaVA-NeXT-7B, also improves image-dependent spatial judgements. The original loss is the opposite case: it teaches NeXT to answer part–whole questions from the text (100% on T4, 99.2% blind on T4-L) and degrades both LLaVA models on the independent benchmark (NeXT counting: 63.1% to 12.6%).

Which wrong answer do models choose? Each CompHall question has a near and a far negative (§ 4), so the option chosen on an error shows what kind of mistake was made; if errors were uniform over the two negatives, each would be chosen 50% of the time (counts in Table 10, Appendix § 16). On spatial questions (T1), LLaVA-NeXT-7B and Idefics2-8B choose the opposite direction in 71% (60/85) and 64% (55/86) of their errors under the first prompt (p<0.001p<0.001 and p=0.013p=0.013; two-sided binomial test) and in 61% and 64% under the second (p=0.042p=0.042 and 0.015); LLaVA-1.5-7B’s errors are split evenly under the first prompt (51/106) and lean towards the near negative under the second (53/87, p=0.053p=0.053). On attribute questions (T2), 62–77% of the first-prompt errors choose the colour of another annotated object, which is significant only for LLaVA-NeXT-7B (p=0.017p=0.017). Counting errors split evenly between off-by-one and off-by-two (45–52% off-by-one), and T4 has too few errors (5–13 per model) to say anything. These preferences are only partly image-dependent. Under the first prompt they vanish with a gray image (45–56% near; all p>0.28p>0.28), but under the second prompt Idefics2-8B’s preference persists without the image (T1: 66% near vs. 64% with it, p=0.004p=0.004; counting: 62.5%, p=0.018p=0.018), while it weakens for LLaVA-NeXT-7B (52% vs. 61%) and LLaVA-1.5-7B (55% vs. 61%). A preference for the near negative is therefore not specific evidence of visual binding errors; part of it is a language prior. Models answer the same questions, so the tests are not independent across models.

RunT1T2T3T4
LLaVA-1.5-7B51 / 5529 / 1749 / 476 / 7
LLaVA-NeXT-7B60 / 2517 / 539 / 473 / 5
Idefics2-8B55 / 3115 / 945 / 421 / 5
LLaVA-1.5-7B, blind47 / 5435 / 3451 / 487 / 5
LLaVA-NeXT-7B, blind44 / 5429 / 3450 / 553 / 5
Idefics2-8B, blind50 / 3927 / 2160 / 432 / 5
Second prompt, LLaVA-1.5-7B:
plain, image53 / 3421 / 751 / 434 / 5
CCFT, image50 / 3425 / 738 / 501 / 4
plain, blind55 / 4530 / 2252 / 455 / 3
CCFT, blind52 / 3123 / 2456 / 402 / 2
Second prompt, other baselines:
LLaVA-NeXT-7B, image54 / 3412 / 642 / 503 / 4
LLaVA-NeXT-7B, blind47 / 4324 / 2951 / 582 / 4
Idefics2-8B, image53 / 3018 / 940 / 390 / 3
Idefics2-8B, blind55 / 2827 / 1960 / 361 / 4

Table 10. Incorrect answers choosing the near / far negative.

Scope of the hallucination claim. CompHall measures multiple-choice accuracy and the synthetic-scene study measures yes/no existence errors; neither measures free-form hallucination, and the latter does not involve binding. The one direct test of a link, the per-image correlation (§ 5.2), is null though noisy, and CCFT raises the POPE yes-bias. Our results show that compositional questions are hard and that some errors have the pattern binding failures would predict, although part of that pattern is a language prior (previous paragraph); they do not support the claim that compositional failures cause hallucination.

Implications for evaluating hard-negative fine-tuning. Four practices would have changed our conclusions, and we suggest them for similar studies. First, report accuracy without the image: part–whole questions look like a vision test and are not. Second, for two-option tasks, ask every question in both option orders: in the benchmark’s own order the same fine-tuning looked twice as effective for one model and ineffective for another. Third, train control arms on different question types, because a gain on a spatial benchmark may or may not need spatial supervision. Fourth, check what the training loss actually scores: in our first implementation 97% of the scored positions were image-placeholder tokens, and the resulting models left CompHall’s overall accuracy unchanged while degrading on independent benchmarks.

Limitations. CompHall does not cover relational composition beyond the four types (transitive relations, event structures) or temporal composition. Questions are template-generated and not human-verified. It is multiple-choice only, and question type is confounded with image source (COCO for T1 and T3, Visual Genome for T2 and T4). Visual Genome images are the 2,000 most densely annotated, not a random sample. Several construction details limit what the types measure: T3 answers are COCO annotation counts, which may miss small or occluded instances; the T2 near negative is a random colour when an image has no other coloured object, and a colour that appears elsewhere in the image may be chosen for that reason rather than because of mis-binding; and the swapped statement in T4 is often implausible as text, and a blind baseline confirms that language priors alone solve T4 (§ 4.4). The full objective is trained with three seeds per implementation and each ablation variant once, the original T4 test set has only 31 questions (T4-L adds 500), the original loss was run once on LLaVA-NeXT-7B and not on Idefics2-8B, and all models have 7–8B parameters and were fine-tuned with LoRA, so we cannot say how the findings scale. Each seed is tested separately and without correction for multiple comparisons, and the paired tests treat questions as independent, so the pp-values are descriptive; three seeds also limit what we can say about seed-to-seed variance. POPE was run for LLaVA-1.5-7B only, and CV-Bench’s relation questions use only four direction words and two options. The external evaluation uses two benchmarks; blind counting on CV-Bench is uninformative because models answer “0” for a blank image. The controls leave a minority of the gain unexplained (0–47% of the spatial-only gain once option order is controlled, depending on the model), the option-order control covers only the two-option tasks, the held-out-vocabulary arms (LLaVA models only) are small (about 500 and 158 training questions) and CompHall’s held-out vertical test set has only 41 questions, and the gain extends to a true/false format only weakly. Results are prompt-sensitive: under a second prompt wording, overall accuracy moves from 38.3% to 48.5% for LLaVA-1.5-7B, from 52.5% to 51.5% for LLaVA-NeXT-7B, and from 52.0% to 54.6% for Idefics2-8B.

Conclusion

We asked what hard-negative fine-tuning teaches a multimodal large language model, and approached it with CompHall, blind controls, matched prompts, three seeds, option-order tests, two independent benchmarks and three model families.

Spatial supervision transfers, but by very different amounts. With a corrected loss that scores only the answer tokens, accuracy on CV-Bench’s spatial-relation questions, averaged over both option orders, rises from 57.6% to 60.8% for LLaVA-1.5-7B, from 62.8% to 85.9% for LLaVA-NeXT-7B and from 63.8% to 69.8% for Idefics2-8B. Most of the gain needs spatial training questions (spatial only: 64.5%, 87.1% and 68.7%; other types only: 57.6%, 66.1% and 66.1%), for the LLaVA models it holds on images from another dataset, and for LLaVA-NeXT-7B about two-thirds of it appears on relation words that were only wrong options in training.

The gain is not general. On CompHall itself it is significant only for LLaVA-NeXT-7B (+6.9 pp; LLaVA-1.5-7B: 51.1% vs. 48.5%, not significant; Idefics2-8B: no change). It reaches counting only after spatial-only training, it is about 3 points at most on true/false statements, and fine-tuning on other question types can make LLaVA-1.5-7B answer “true” too often.

The original method was flawed: its score was dominated by image-placeholder tokens, and it degraded both LLaVA models on independent benchmarks. The contrastive and ranking terms add nothing over fine-tuning on the answers alone, and both losses raise the POPE yes-bias, the original one unstably. The benchmark itself needs care: part–whole questions can be answered from the text alone (66–95% without the image on 500 further questions), and per-image compositional accuracy is unrelated to an object-hallucination probe (r≈−0.05r \approx-0.05, noisy probe), so we find no evidence that compositional failures drive hallucination. We did not test larger models, other training regimes or free-form hallucination. CompHall and the CCFT code will be released.

Use of AI

We used Claude (Anthropic), through its chat interface, throughout this project. It wrote first drafts and some revisions of the paper, including the abstract, background, related work, and discussion, as well as the LaTeX source for those sections. It debugged and wrote some parts of the code, including the corrected-loss training scripts, the evaluation and statistical-analysis scripts, and we used it to check the draft’s numbers and logic against our result files. The author designed the study, wrote much of the code, ran all experiments on local hardware, and every reported number comes from the author’s own result files. The author reviewed the full paper and takes responsibility for its content.

CompHall Question Examples

Table 5 shows one test question per type with the correct answer and the two hard negatives (near and far; § 4.1).

TypeQuestionCorrectNear negativeFar negative
T1 SpatialWhere is the person relative to the car?right ofleft ofabove
T2 AttributeWhat color is the handle in the image?bluegreenorange
T3 CountingHow many car can you see?111013
T4 Part–wholeWhich statement correctly describes the relationship between the man and the pants?The man is wearing the pantsThe pants is wearing the manThe man is wearing the wii remote

Table 5. CompHall test questions with the correct answer and the two hard negatives. Text is quoted from the released data.

Per-Type Ablation Results

Table 6 extends the ablation of Table 3 of the main paper with a per-question-type breakdown (corrected loss, seed 42, with the image). The baseline row is the plain model under the same prompt, so the Δ\Delta rows are matched comparisons. All four trained conditions differ from each other by at most 1.4 pp overall, and the seed-to-seed spread of the full objective is 0.5 pp (Table 2 of the main paper). Gains over the plain model are concentrated in T1 and T4; T4 has only 31 test questions, so +9.6 pp is three questions.

ConditionT1T2T3T4Overall
Plain model41.273.332.471.048.5
SFT only46.673.333.180.651.3
Δ\Delta+5.40.0+0.7+9.6+2.8
SFT + Contrastive50.072.431.780.651.8
Δ\Delta+8.8−0.9−0.7+9.6+3.3
SFT + Ranking50.773.333.180.652.7
Δ\Delta+9.50.0+0.7+9.6+4.2
CCFT Full48.073.332.480.651.5
Δ\Delta+6.80.00.0+9.6+3.0

Table 6. Per-type accuracy (%) with the image for the loss-term ablation (corrected loss, seed 42), with delta relative to the plain model under the same prompt.

An earlier ablation of the original loss, trained by a separate script (its settings were not recorded; its defaults are 2 epochs on 500 questions) and evaluated with a third prompt, scored 28.4–46.3% where the plain model scored 48.9%. Because that loss is dominated by image-placeholder tokens and its budget and prompt differ from those of Table 2, we replaced it with the ablation above.

With and without the image, both models. Table 7 shows per-type accuracy with the image and with a gray image for the baseline, both CCFT implementations (v1 = original loss, v2 = corrected loss) and the two control arms of Section 8.4 (trained on the non-spatial types only or on spatial questions only). LLaVA-1.5-7B rows are means over three seeds and use the prompt of Table 2 of the main paper; LLaVA-NeXT-7B rows use its own prompt, with v2 and both control arms means over three seeds and the baseline and v1 single runs.

Hyperparameter Details

Table 8 lists the CCFT hyperparameters used in all experiments, as implemented in our training script.

ParameterValue
Base modelLLaVA-1.5-7B (llava-hf)
Fine-tuning methodLoRA on q, k, v, o projections
LoRA rank / α\alpha / dropout16 / 32 / 0.05
OptimiserAdamW, weight decay 0.01
Learning rate2×10−52 \times10^{-5}
LR scheduleCosine, 10% linear warmup
Epochs3
Batch size2 (no gradient accumulation)
Optimisation steps2,994 (998 per epoch)
Gradient clipping1.0
Precisionfp16 autocast, fp32 LoRA weights
Margin mm0.5
λc/λr\lambda_c / \lambda_r0.5 / 0.3
Score ssv1: whole sequence, mini-batch; v2: answer tokens, per example
Checkpointlowest validation loss (200 val. questions)
Random seed42
GPUNVIDIA H100 NVL (95 GB)

Table 8. CCFT hyperparameters.

Extended POPE Analysis

Table 9 gives precision, recall and F1 for the POPE evaluation of § 8.2 (nn=444 per split). v1 and v2 are means over three seeds. Both CCFT implementations trade precision for recall; the corrected loss does so far less than the original. The last three rows per split are single-seed variants of the corrected loss (SFT only, contrastive term only, ranking term only); their accuracy and yes-bias are 0.768–0.779 and 0.586–0.606 (adversarial), 0.833–0.838 and 0.527–0.536 (popular), and 0.885–0.887 and 0.475–0.486 (random), against 0.770 and 0.617, 0.839 and 0.549, and 0.893 and 0.494 for the full objective. Recall is identical across splits for the plain model because the positive questions are shared.

Broader Impact. CompHall is a diagnostic benchmark for compositional errors in MLLMs, and CCFT is a fine-tuning study whose effect on object hallucination is mixed (§ 8.2). We do not anticipate direct harmful applications. Model checkpoints and benchmark data will be released.

Error-Direction Counts

Table 10 gives, for each run, the number of incorrect answers that choose the near or the far negative. Near/far: T1 opposite direction / different axis; T2 colour of another object / unrelated colour; T3 off-by-one / off-by-two; T4 subject–object swap / unrelated scene object. The middle block repeats the evaluation with a gray image instead of the real one; the last two blocks use the second prompt (the prompt of Table 2 in the main paper): LLaVA-1.5-7B with and without CCFT, and the other two baselines.

Benchmark Statistics

Table 11 gives the composition of the CompHall splits, which contain the 2,860 questions used in all experiments. Questions are split by image: no test image appears in the training split.

Table 12 describes how the correct answer and the two hard negatives are built for each type. Counts up to eight are written as words and larger counts as digits, so the options of a counting question can mix the two forms. Option positions are randomised.

The larger part–whole set (T4-L). We built 500 further part–whole questions with the same generator, relationship tagging and quality filter as T4, with two differences: they come from Visual Genome images that appear in no other split (and are not COCO images used elsewhere), with at least three tagged relations and three distinct objects, and there is one question per image. A question was also rejected if its far negative was not an object of the scene. Of the 1,413 candidate images, the first 500 in a seeded random order that produced a question were used (18 images produced none). The predicates are wearing (229), has (177), holding (56), sitting on (16), attached to (8), standing on (6), mounted on (4), carrying (3) and part of (1); 437 of the 500 subject–predicate–object triples are distinct, and the correct letter is A, B and C for 176, 148 and 176 questions. T4-L is used for evaluation only.

Per-Image Correlation

Table 13 gives the per-image correlation between CompHall accuracy and the object-hallucination rate (§ 5.2). Images are split into terciles of compositional accuracy; the last three columns give the mean hallucination rate in the lowest, middle and highest tercile. The hallucination probe is run with LLaVA-1.5-7B for all three rows.

Synthetic-Scene Study

Figure 7 shows the error rate of LLaVA-1.5-7B on the object-existence questions of § 5.3. Each level uses 50 scenes of flat coloured shapes and two questions per scene (one about a colour that is present, one about a colour that is absent), so the 100 questions per level come from 50 scenes; the intervals treat the scene as the unit. The 95% Wilson intervals are [2.1, 16.2] at one object and [29.4, 55.8] at five, and those of two to five objects overlap. In a separate comparison of 200 pairs of independently generated 2-object and 3-object scenes, the mean difference in error rate is 8.5 pp (one-sample tt-test on the 200 differences, t=3.39t=3.39, p=0.0008p=0.0008).

Recent Models

Table 14 gives CompHall accuracy for three recent open models, with the real image and with a gray image (blind), under the two prompt wordings of the main paper (Prompt A: the wording of Table 1; Prompt B: the wording of Table 2). Second-prompt responses are scored with a parser that also accepts answers of the form “(B) right of”.

CCFT on the Larger Part–Whole Set

Table 15 gives accuracy on the 500 questions of T4-L with the image and with a gray image for the plain models and each CCFT seed, with the exact McNemar pp-value against the plain model on the same questions. LLaVA-1.5-7B uses the prompt of Table 2 of the main paper; LLaVA-NeXT-7B uses its own prompt.

The External Benchmark (CV-Bench)

We use the 2D tasks of CV-Bench (788 counting and 650 spatial-relation questions; the counting questions have 4–6 options and always include “0”), without the 160 questions whose COCO image appears in CompHall (n=1,278n = 1{,}278; chance 23% for counting and 50% for spatial relations). Prompts follow the CompHall evaluations, with only the list of letters adapted to each question; responses are scored with a parser restricted to the valid letters. Table 16 gives the fine-tuning results, with the exact McNemar tests of the main text computed per task and seed against the plain model; Table 17 gives the six plain models. A blank image leads the models to answer “0” (the correct answer for 0.1% of the questions), so blind counting accuracy is not a measure of text priors.

ModelConditionOverall imageImage CountImage RelationBlind CountBlind Relation
LLaVA-1.5-7BPlain56.253.360.016.049.7
CCFT v147.039.157.336.153.5
CCFT v256.248.266.723.549.7
LLaVA-NeXT-7BPlain63.563.164.14.449.7
CCFT v128.912.650.119.448.8
CCFT v274.964.288.75.251.5

Table 16. CV-Bench accuracy (%): overall with the image, and by task with and without the image. v1 and v2 are means over three seeds, except LLaVA-NeXT v1 (one run).

Control Arms and the VSR Benchmark

Arms. The two control arms use CCFT v2 (the corrected loss) with unchanged hyper-parameters, prompt and seeds, and differ only in the training and validation questions, which are filtered by type: the non-spatial arm keeps T2–T4 (1,314 training questions with an existing image) and the spatial-only arm keeps T1 (682), against 1,996 for the full set; the validation file used for checkpoint selection is filtered in the same way. Both models have seeds 42, 1 and 2 for both arms. Evaluation is as in Appendix § 22.

Option-Order Control

Every two-option question (CV-Bench spatial relations after excluding images shared with CompHall: 555 questions; VSR: 1,218) is asked in the original option order and with the two options reversed, with the same prompts and the image. “Averaged” is the mean of the two accuracies; “consistent” is the share of questions answered with the same option content in both orders. The last column gives the gain of the averaged accuracy over the plain model for each seed, with the pp-value of a paired bootstrap over questions (5,000 resamples).

Held-Out Relation Words

The horizontal-only arm is trained on the CompHall spatial training questions whose correct option contains “left” or “right” (511 in the training file) and the vertical-only arm on those whose correct option is above, below, over, under, top or beneath (159; 158 remained after the training scripts skipped one question whose image file is unavailable); 17 questions with other correct answers (for example “behind”) are in neither arm. Validation questions used for checkpoint selection are filtered in the same way, and all other settings are those of the full spatial-only arm. Table 23 reports CV-Bench spatial relations split by the axis of the correct answer, averaged over both option orders; the pp-values are paired bootstraps over questions against the plain model, per seed. Table 24 reports CompHall’s spatial test questions in the original option order (exact McNemar tests). An asterisk marks the held-out axis of an arm.

Extended Background

MLLM Architecture and Visual Encoding

Modern MLLMs follow a three-stage pipeline (Figure 8): a vision encoder maps an input image to a sequence of visual tokens; a connector module projects or cross-attends those tokens into the language model’s embedding space; and an LLM backbone generates text conditioned on the fused multimodal context.

Figure 8. Standard MLLM pipeline. Patch tokens from the vision encoder are projected into the LLM’s embedding space, and no component is designed for binding; object-feature attribution is handled by LLM self-attention layers.

Vision encoders. The standard choice is a CLIP ViT [19] trained on image–text contrastive objectives. A ViT divides the image into a P×PP \times P grid of patches, processes each through self-attention layers, and outputs a sequence of patch tokens {v1,…,vN}\{v_1,\ldots,v_N\}. Patch tokens carry local appearance information; spatial relationships between patches are encoded only through learned position embeddings, which do not explicitly represent object-level spatial relations.

Connector modules. LLaVA [13] projects each patch token with a linear layer (LLaVA-1.5 [14] uses a two-layer MLP). InstructBLIP and BLIP-2 [1, 9] use a Q-Former: KK learned query vectors that cross-attend over all NN patch tokens, producing a fixed-length visual sequence. Idefics2 [8] uses a perceiver resampler. LLaVA-NeXT [15] tiles the image at multiple resolutions before encoding, increasing the number of patch tokens per image.

The binding bottleneck. In all of these architectures, the attribution of spatial and attribute information to specific objects is handled by LLM self-attention layers. These layers were trained on text and have no structural inductive bias toward enforcing feature-to-object consistency. There is no explicit operation that associates a colour token with its object rather than a neighbouring one. This bottleneck becomes more acute as scene complexity increases. For a single-object scene, feature attribution is unambiguous. For a scene with KK objects of varying colours, sizes, and positions, the model must resolve KK-way attribution across every feature dimension using only the implicit structure of LLM self-attention weights—a task those weights were not pretrained to perform explicitly. The expected difficulty therefore scales with object count and attribute diversity; § 5.3 measures how existence-question errors change with object count, although those questions do not themselves require binding.

The Binding Problem in Perception

The binding problem [23, 24] refers to the question of how the brain combines independently processed features (colour, shape, position) into correctly attributed object representations. Treisman’s Feature Integration Theory proposes that a serial focal attention operation selects one location at a time and assembles all features at that location into a coherent percept. Without this step, features are susceptible to illusory conjunctions: incorrect attributions in which a feature from one object is assigned to another (e.g., perceiving a red circle and a blue square as a blue circle and a red square).

MLLMs process all patch tokens in parallel without an explicit location-selection step, and may produce the same class of attribution errors: correctly representing that a scene contains “red,” “sphere,” and “left,” but binding them incorrectly.

A Taxonomy of MLLM Hallucination

We organise MLLM hallucination into four types based on which binding step fails:

  • Object-existence (T0). The model reports an object not present in the image. This is the target of POPE and most prior benchmarks. It does not require a compositional binding error; a model can fail here even with correct attribute-to-object binding.

  • Attribute-binding (T2). The model correctly identifies objects but assigns an attribute (colour, size, texture) to the wrong one. e.g., “a large red sphere” when the sphere is blue and the cube is red.

  • Spatial-relation (T1). The model mis-reports the relative or absolute position of an object.

  • Part–whole and role binding (T4). The model assigns the wrong roles in a part–whole or possession relation, e.g., “the pants are wearing the man” for a man wearing pants.

  • Counting (T3). The model reports an incorrect count of a target category. Correct counting requires identifying every instance of the category, so the number of required bindings scales with scene population rather than being fixed.

CompHall targets T1, T2, T3 (counting, which requires per-instance identification), and T4. T0 is excluded by design: every CompHall question involves only objects that are present in the image. This separation matters because a model can do well on a T0 benchmark (POPE accuracy 0.82–0.87 for LLaVA-1.5-7B) while performing near chance on T1 and T3, as our results show.

Accuracy Heatmap

Figure 9. CompHall accuracy by model and question type (first prompt wording; the three recent models are in Table 14). Cells numerically below the 33.3% chance level are marked (†); none is significantly below chance (two-sided binomial test). LLaVA-NeXT-7B and Idefics2-8B score 42.6% and 41.9% on T1 but 74–81% on T2 and T4.

Idefics2-8B

Idefics2-8B (SigLIP vision encoder, perceiver resampler, Mistral-7B language model) is fine-tuned with the corrected loss of § 6 (answer-token score; contrastive and ranking terms with the weights of Table 8), with LoRA of rank 16 (α=32\alpha=32, dropout 0.05) on the q, k, v and o projections of the language model only (13.6M trainable parameters), learning rate 2×10−52 \times10^{-5} (AdamW, weight decay 0.01, cosine schedule with 10% warm-up), 3 epochs, gradient clipping at 1.0, fp16 autocast with fp32 LoRA weights, one question per optimizer step (its correct answer and the two negatives) and no image splitting (64 visual tokens per image); the last checkpoint is used. The three arms are trained on all question types (1,996 questions), on the non-spatial types (1,314) and on spatial questions (682), with seeds 42, 1 and 2. Evaluation uses the letter prompt of the other models (§ 4.4); the plain model scores 52.2% with the image under this implementation (54.6% with the parser of § 4.4). Table 25 gives CompHall by type and Table 26 CV-Bench and VSR in the original option order; the order-averaged results are in Table 21 and Table 22.

With imageBlind
ArmRelationCountingVSRRelationCountingVSR
Plain68.561.576.955.97.748.6
All types69.3±2.469.3 \pm2.4 (.17, .68, .55)59.0±0.859.0 \pm0.8 (.048, .29, .10)74.9±1.374.9 \pm1.3 (.044, .0001, .30)55.628.349.7
Non-spatial66.5±1.866.5 \pm1.8 (.16, 1.0, .23)58.6±0.758.6 \pm0.7 (.016, .12, .14)74.5±0.574.5 \pm0.5 (.005, .009, .0003)57.327.349.2
Spatial only70.6±1.470.6 \pm1.4 (.35, .08, .78)65.4±0.465.4 \pm0.4 (.001, .0002, .0009)77.2±0.577.2 \pm0.5 (.90, .38, .92)53.85.848.7

Table 26. Idefics2-8B on CV-Bench (relation and counting) and VSR in the original option order: accuracy (%) with the image (mean ±\pm sd over three seeds; per-seed exact McNemar pp against the plain model in brackets) and with a gray image (means).

POPE Figure

POPE accuracy and yes-bias for the plain model, CCFT v1 and CCFT v2

Figure 10. POPE accuracy and yes-bias for the plain model, CCFT v1 and CCFT v2 (mean over three seeds; bars show the standard deviation). The dashed line marks a balanced yes/no rate.

References

  1. [1]Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. In Advances in Neural Information Processing Systems, 2023.
  2. [2]Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. MME: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023.arxiv.org/abs/2306.13394
  3. [3]Cheng-Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. SugarCrepe: Fixing hackable benchmarks for vision-language compositionality. In Advances in Neural Information Processing Systems, pages 31096–31116, 2023.arxiv.org/abs/2306.14610
  4. [4]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.arxiv.org/abs/2106.09685
  5. [5]Drew A. Hudson and Christopher D. Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6693–6702, 2019.arxiv.org/abs/1902.09506
  6. [6]Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2901–2910, 2017.arxiv.org/abs/1612.06890
  7. [7]Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123 (1):32–73, 2017.arxiv.org/abs/1602.07332
  8. [8]Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? In Advances in Neural Information Processing Systems, pages 87874–87907, 2024.arxiv.org/abs/2405.02246
  9. [9]Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning, pages 19730–19742, 2023.arxiv.org/abs/2301.12597
  10. [10]Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 292–305, 2023.arxiv.org/abs/2305.10355
  11. [11]Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In European Conference on Computer Vision, 2014.arxiv.org/abs/1405.0312
  12. [12]Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 11:635–651, 2023.
  13. [13]Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems, 2023.
  14. [14]Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26286–26296, 2024.arxiv.org/abs/2310.03744
  15. [15]Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. LLaVA-NeXT: Improved reasoning, OCR, and world knowledge. https://llava-vl.github.io/blog/2024-01-30-llava-next/, 2024.
  16. [16]Zixian Ma, Jerry Hong, Mustafa Omer Gul, Mona Gandhi, Irena Gao, and Ranjay Krishna. CREPE: Can vision-language foundation models reason compositionally? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10910–10921, 2023.arxiv.org/abs/2212.07796
  17. [17]OpenAI. GPT-4V(vision) system card. Technical report, OpenAI, 2023.
  18. [18]Letitia Parcalabescu, Michele Cafagna, Lilitta Muradjan, Anette Frank, Jacer Calixto, and Albert Gatt. VALSE: A task-independent benchmark for vision and language models centered on linguistic phenomena. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8253–8280, 2022.
  19. [19]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748–8763, 2021.arxiv.org/abs/2103.00020
  20. [20]Sainandan Ramakrishnan, Aishwarya Agrawal, and Stefan Lee. Overcoming language priors in visual question answering with adversarial regularization. In Advances in Neural Information Processing Systems, 2018.arxiv.org/abs/1810.03649
  21. [21]Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4035–4045, 2018.arxiv.org/abs/1809.02156
  22. [22]Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai C. Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal LLMs. Advances in Neural Information Processing Systems, 37:87310–87356, 2024.arxiv.org/abs/2406.16860
  23. [23]Anne Treisman. The binding problem. Current Opinion in Neurobiology, 6(2):171–178, 1996.pubmed.ncbi.nlm.nih.gov/8725958
  24. [24]Anne M. Treisman and Garry Gelade. A feature-integration theory of attention. Cognitive Psychology, 12(1):97–136, 1980.
  25. [25]Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Jiaqi Wang, Haiyang Xu, Ming Yan, Ji Zhang, et al. AMBER: An LLM-free multi-dimensional benchmark for MLLMs hallucination evaluation. arXiv preprint arXiv:2311.07397, 2023.arxiv.org/abs/2311.07397
  26. [26]Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? arXiv preprint arXiv:2210.01936, 2022.arxiv.org/abs/2210.01936
  27. [27]Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision-language models. arXiv preprint arXiv:2310.00754, 2023.arxiv.org/abs/2310.00754

Paper details

vlnc:W_dhtzwvnf
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 1

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents