betaFeedback

WorksComputer Science

Can Vision-Language Models Truly Understand Images? A Benchmark Beyond Commonsense Bias

Abstract

Multimodal large language models (MLLMs) have achieved impressive performance on a wide range of vision-language tasks. However, it remains unclear whether their predictions are grounded in the visual content itself or are largely driven by statistical co-occurrence patterns and commonsense correlations learned from large-scale image-text corpora. Existing vision-language benchmarks predominantly consist of samples that conform to human commonsense, allowing models to achieve strong performance by exploiting dataset priors rather than genuinely understanding image content. To address this limitation, we present ContraSense, a systematically constructed benchmark for evaluating faithful vision-language understanding. ContraSense consists of carefully curated counter-intuitive image-text pairs that deliberately violate common expectations, substantially reducing the effectiveness of statistical biases and requiring models to rely on visual evidence for reasoning. We conduct comprehensive evaluations on a diverse set of representative open- and closed-source MLLMs. The results reveal a substantial performance gap between current MLLMs and humans: even the strongest closed-source model achieves only about 60% accuracy, compared with 96.04% for human participants. These findings demonstrate that current MLLMs still struggle with faithful visual understanding and highlight the need for more robust multimodal reasoning grounded in image content. The ContraSense benchmark is publicly available at https://huggingface.co/datasets/LingLin00/ContraSense.

Introduction

WITH the rapid development of Vision-Language Models (VLMs), related models have made significant progress in a variety of tasks such as image-text alignment, Visual Question Answering (VQA), and Image Captioning [1, 2, 3, 4, 5, 6, 7, 8]. They have demonstrated excellent cross-modal migration and generalization capabilities and are able to describe image content more accurately or provide seemingly reasonable answers to questions posed by images [5], [6], [9]. Yet this prompts a fundamental question: Do these models actually see images? In other words, are their outputs truly grounded in the visual content, or do they instead lean on linguistic/visual a priori knowledge—“common sense” built from frequent statistical co-occurrences1 in the multimodal training data? Such co-occurrences can yield superficially correct responses based on high-frequency patterns, allowing models to score artificially high without deep image analysis [10]. This phenomenon can trigger serious consequences in real-world applications, especially when models are deployed in scenarios where accurate perception and reasoning are highly required. For example, in autonomous driving, models may ignore the actual state of lights due to statistically common patterns such as “red-stop” or “green-forward” and thus make incorrect judgments in abnormal lighting or occlusion situations.

An intuitive idea for evaluation is to directly test the multimodal model’s ability to understand image content using existing datasets. However, existing datasets generally suffer from statistical co-occurrence during the construction process. For example, images of camels obtained through natural collection methods often appear in a desert background. Therefore, when the model is asked “where do camels live”, even without understanding the content of the image, it can give a seemingly correct answer of “desert” based on the statistical co-occurrence pattern learned from training. This coupling effect makes it difficult to judge whether the correct answer of the model is based on the real understanding of the semantics of the image or only on the co-occurrence patterns learned from the training corpus. Therefore, although existing datasets play an important role in multimodal model evaluation, their applicability may be limited in specific contexts, especially when decoupled statistical co-occurrence is required to evaluate the model’s real image comprehension. This strongly motivates us to construct a new evaluation benchmark that should deliberately break the statistical co-occurrence relationship of the training data so that the model cannot obtain the correct answer through a priori knowledge or frequent co-occurrence but has to rely on a deep understanding of the image content itself. By minimizing the impact of statistical co-occurrences on the answers, the benchmark can more accurately measure model’s ability to understand the image semantics truly.

In terms of dataset construction, the existing approach of collecting natural image data often inevitably introduces statistical co-occurrence bias, which makes it difficult to assess the realistic comprehension of image content by multimodal models in the inference process. To alleviate the problem, we propose a generation-based approach to construct a multimodal benchmark, ContraSense Benchmark, for counter-intuitive reasoning. Specifically, we crafted a set of prompts that cover five types of common knowledge violations, including contextual violation, attribute violation, size/proportion anomalies, physical impossibility, and function/state perversity. During the data generation process, we randomly select at least one counterintuitive type and construct a conflict situation between the two categories to guide the Large Language Model (LLM) in the form of a prompt to generate the corresponding counterintuitive image captions as well as six question-answer pairs (QA pairs) in a regularized format. Subsequently, we transformed the generated captions into high-resolution (1024×10241024 \times1024) images using a text-to-image model. To ensure the consistency of the images and the quality of the questions and answers, we invited five human experts to manually proofread the image descriptions and the corresponding question-answer pairs and correct the formatting, as well as eliminate samples that are not consistent with the images and text, or that can be answered by logical reasoning. In the end, we constructed the ContraSense Benchmark with 4,698 high-quality counterintuitive QA samples covering a wide range of scenes and styles, including indoor and outdoor environments, painting styles, 3D rendering styles, etc. Figure 1 shows some counterintuitive images from the ContraSense dataset.

Counterintuitive images from the ContraSense dataset

Figure 1. We propose a new benchmark called ContraSense, which is both creative and imaginative. The benchmark consists of 4,698 carefully crafted and manually calibrated high-resolution (1024×10241024 \times1024 pixels) text QA pairs. Each image contains at least one counterintuitive scenario designed to assess whether visual-language models can provide accurate responses based on the image content itself without relying on common-sense prior knowledge from the training data. This benchmark provides a unique and powerful tool for studying the robustness and real-world visual understanding capabilities of VLMs.

In order to comprehensively assess the ability of visual language models (VLMs) to understand image content realistically, we further propose the Contra-Acc metric, which measures the performance of the models in processing counterintuitive images from multiple dimensions. The experimental results show an average performance degradation of about 37% for these models on the ContraSense Benchmark compared to regular image tasks, reflecting the significant performance gap of current VLMs in counterintuitive scenarios.

The main contributions are summarized as follows:

  • We have carefully constructed ContraSense, a image-text multimodal dataset dedicated to counterintuitive comprehension, which contains a total of 4,698 well-designed counterintuitive QA examples, aiming to systematically examine whether a multimodal large model can faithfully comprehend and align image content in the face of common-sense biases.

  • We propose a new evaluation metric, called Contra-Acc metric, to more comprehensively measure a model’s ability to understand an image’s semantics truly. The metric is measured in six counterintuitive dimensions, aiming to exclude common sense from interfering with the model’s behavior and, thus, more accurately measure the model’s fidelity to the content of the image itself.

  • We conducted experimental research on several mainstream multimodal large models. The results show that existing models generally suffer from significant performance bottlenecks in counterintuitive tasks, and they still suffer from significant robustness deficiencies in handling scenarios that violate common sense.

Related Work

Comprehensive Benchmarks for Large VLMs. A series of large-scale, general-purpose benchmarks [11, 12, 13, 14, 15] have emerged to evaluate the capabilities of multimodal large language models (MLLMs). SEED-Bench and its hierarchical successor SEED-Bench-2 provide 19K and 24K multiple-choice questions respectively, spanning up to 27 evaluation dimensions across image, video, and generation tasks [16, 17]. MMBench introduces a 3K bilingual test set with automatic grading, enabling direct comparisons between English and Chinese models [18]. MME categorizes 14 subtasks into ”perception” and ”cognition” branches for finer-grained diagnostic insights [19], while MM-Vet emphasizes integrative skills such as optical character recognition (OCR), spatial reasoning, and mathematical understanding [20]. Beyond static leaderboards, LVLM-eHub aggregates over 40 public datasets into an online evaluation hub, providing continuously updated model rankings [21]. Traditional vision-reasoning corpora, such as GQA, remain popular for compositional question answering and are commonly included in benchmark suites [22]. Additionally, specialized diagnostic benchmarks such as POPE, which probes models for object hallucinations, are now frequently reported alongside holistic metrics to highlight specific failure modes [23]. Zhou et al. [24] introduce a novel probing dataset named ROME to evaluate whether the SOTA pre-trained vision-language models have the reasoning capability to correctly interpret counter-intuitive content. Bitton-Guetta et al. [25] released the WHOOPS! dataset and benchmark to evaluate models’ performance in visual common-sense understanding. This dataset consists of images deliberately constructed by designers using publicly available image generation tools (such as Midjourney) to violate common sense.

However, despite their comprehensive coverage, these benchmark suites inherently reflect commonsense regularities prevalent in natural images (e.g., camel↔desertcamel \leftrightarrow desert, plane↔skyplane \leftrightarrow sky), or it does not primarily emphasize the impact of disrupting statistical co-occurrence on model conclusions to achieve the goal of verifying whether the model remains faithful to the specific content of the image, allowing shortcut-seeking models to answer correctly by exploiting frequent co-occurrence patterns rather than genuine visual understanding. Furthermore, multiple-choice or short-answer formats predominant in these benchmarks favor superficial keyword matching, rarely confronting models with multifaceted contradictions involving context, physics, and function. Unlike the aforementioned work, ContraSense’s core objective is to construct a class of problem samples for which answers cannot be directly inferred from statistical co-occurrence patterns. This tests whether models genuinely possess the ability to understand and reason about image semantics. To achieve this, we systematically designed counterintuitive scenarios, ensuring both task design and problem construction revolved strictly around this central principle. Consequently, existing benchmarks inadequately assess how VLMs behave when commonsense priors conflict with visual evidence—a critical evaluation gap explicitly addressed by our proposed ContraSense benchmark.

The ContraSense

To achieve the goal of constructing counterintuitive multimodal data, we face a core challenge: data directly collected from natural scenes typically follow statistical co-occurrence patterns, making it difficult to obtain representative counterintuitive samples. This collected approach makes it challenging to clearly distinguish whether the performance of visual language models (VLMs) in downstream tasks stems from a true understanding of image content or merely relies on statistical co-occurrence information for inference. To reduce the interference of common-sense bias, we employ a text-to-image generation approach to construct multimodal counterintuitive data, thereby controlling image content while breaking the statistical co-occurrence structure. Figure 2 demonstrates the counterintuitive data collection process. In this section, we will provide a detailed overview of the ContraSense Benchmark construction.

Figure 2. Pipeline of counterintuitive data collection.

Prompt guides LLM to generate counterintuitive text. To generate text with counterintuitive characteristics for the next stage of text-to-image generation tasks, we carefully designed a set of prompts to guide large language models in generating text that meets the requirements of the instructions.

In constructing the prompts, we systematically constrained and designed the object categories, caption fields, structured labels, and six types of question-answering tasks for describing images in the captions.

In the process of generating image captions, we guide large language models to generate image captions with counterintuitive characteristics by embedding instructions in the prompt. Specifically, as shown in Table I, we designed five categories of counterintuitive scenarios, including context violation (e.g., gorillas living underwater), attribute anomalies (e.g., naturally occurring purple lions), size ratio anomalies (e.g., a cat larger than a house), physically impossible scenarios (e.g., a floating piano), and functional state violations (e.g., ice burning). In the generation task, we randomly select one or more combinations of these counterintuitive categories to guide the large language model in fabricating a non-existent image and generating a caption that aligns with the selected counterintuitive settings.

No.Type NameExampleDescriptionCommon Sense Detection Capability
1Context Violation”A small wooden boat inside a modern bathroom, photorealistic”Objects appearing in unlikely backgrounds or scenesScene Common Knowledge
2Attribute Violation”A black banana”Unreasonable attributes such as colour, material, orientation, etc.Attribute recognition
3Size/Proportion Anomalies”A giant cat towering over city buildings”Proportion Extreme AnomaliesScale Comparison
4Physics-Impossible”A full glass of water floating in mid-air”Violation of the laws of physicsPhysics common sense
5Function/State Perversity”A burning ice cube on a metal tray”State does not correspond to the function of the objectState Identification

Table I. ContraSense defines five counterintuitive categories, and each generated sample contains at least one of these categories, or a combination of multiple counterintuitive categories.

In terms of object category settings, we require that each image contain at least two categories to support subsequent category relationship construction tasks. However, directly instructing a large language model to generate two common categories randomly results in a large number of duplicate categories in the generated output, exhibiting pseudo-random characteristics, thereby limiting category diversity. To enhance category generation diversity, we predefined 240 candidate categories and set the generation strategy in the prompt: one category is randomly generated by the LLM, while the other is randomly sampled from the predefined 240 categories with equal probability.

In terms of structured labels, we require the LLM to generate structured labels of different granularity for the two objects mentioned in the caption, including the object category, color, quantity, and a binary label indicating whether each object has counterintuitive characteristics. In addition, we describe the spatial relationship between the two objects in the overall scene to express the relative spatial information between objects more comprehensively.

In terms of designing counterintuitive questions, we embedded explicit instructions in the prompt. We set up six types of single-choice questions for each image, including object category, color attribute, quantity perception, whether the object is counterintuitive, and the spatial relationship between objects. In addition, we specially designed a higher-order question, namely the counterintuitive image-text matching question, aimed at directly assessing whether the model can accurately identify the most prominent counterintuitive features in the image. Each question provides five possible answers, with only one being the correct option. The purpose of this question design is to comprehensively evaluate whether the model truly answers based on the image content itself, rather than simply relying on superficial statistical co-occurrence relationships, across multiple dimensions—including category recognition, fine-grained attribute perception, quantity inference, nuanced understanding of multi-object relationships, added spatial reasoning, and the core counter-intuitive image–text matching.

Text-to-image. After using the above prompt to guide the large language model in generating text, we select the caption field and further employ a text-to-image generation method to generate high-definition images consistent with the semantic description of each given caption. Specifically, we input the caption into the generation model and set the image resolution to 1024×10241024 \times1024 pixels to ensure the output images have high clarity and rich detail. The model can comprehensively process multi-dimensional information such as objects, scenes, attributes, and spatial relationships mentioned in the caption, thereby generating images consistent with the caption’s semantic meaning, ensuring the accuracy and reasonableness of the generated content. Using the above method, we ultimately generated 18,000 image-caption candidate pairs, which were then subjected to subsequent manual screening.

Human calibration. Existing large language models and text-to-image models have demonstrated remarkable text-generated expression capabilities and text-image matching capabilities. However, directly utilizing text generated by LLMs and images generated by text-to-image models as a counterintuitive dataset is not feasible, primarily because these generated results still contain significant biases. Specifically, during the text generation process of LLMs, the generated multiple-choice questions often exhibit issues such as text answer leakage, multiple answers in single-choice questions, and ambiguous referents. For example, the question What color is the green tiger in the image? can be directly answered without even viewing the image. In text-to-image models, the most common issue is that the generated images do not fully match the given captions. To effectively address these issues, we have introduced a human-calibrated process. Specifically, we invited five human experts to independently review whether the generated images and text matched and conducted cross-validation to ensure consistency and accuracy between images and text. During this process, we removed samples where generation failed or where text and images did not match. Additionally, we corrected the text

to ensure the accuracy of labels and the uniqueness of answers in multiple-choice questions. At the same time, to eliminate other potential confounding factors—such as the small scale of objects related to the question answers or their occupancy of only a small portion of the visual field—we excluded images where the answer-related content did not belong to the primary semantic object or was unrelated to it. Through human calibration, we ensured that all questions are directly related to the image content, especially those counterintuitive points that should be clear and explicit. Ultimately, we cleaned up 4,698 high-quality QA samples from the generated 18,000 image-caption pairs, forming our ContraSense Benchmark.

Experiments

This section provides a detailed description of our specific experimental setup and results for performance evaluation using multiple multimodal large models on the ContraSense benchmark.

Experimental Settings and Ethical Statement

Evaluation Metrics

Standard Classification Metrics. To comprehensively evaluate model performance under multi-class and class-imbalanced settings, we report a diverse set of metrics, including Accuracy, F1 Macro, F1 Weighted [26], Precision, Recall, and Matthews Correlation Coefficient (MCC) [27], [28]. These metrics capture complementary aspects of classification behavior, ranging from overall correctness to per-class balance and correlation-aware robustness.

ROUGE [29] & METEOR [30]. To quantitatively evaluate the performance of VLM-generated text, we adopted standardized metrics commonly used in the field of natural language generation. Specifically, we used ROUGE-1, ROUGE-2, and ROUGE-L to measure the overlap between the generated text and the reference text at the unigram, bigram, and longest common subsequence levels. These metrics comprehensively reflect the similarity and fluency of the generated content. Additionally, we introduced the METEOR metric, which comprehensively evaluates semantic similarity through synonym matching and stemming, thereby providing a more detailed assessment of the content fidelity, information richness, and language quality of the generated text.

Contra-Acc. In image understanding tasks, models exhibit significant variations in performance across different task dimensions, reflecting their distinct capabilities when addressing diverse problem types. In multi-task scenarios, directly calculating average performance may conceal these shortcomings. To comprehensively evaluate a model’s multi-dimensional image understanding capabilities and address dimensions: image caption, the categories of two core objects, color, quantity, whether it violates common-sense attributes, and the semantic relationship between the two. Additionally, to thoroughly assess the model’s understanding of image semantics, we designed six detailed multiple-choice questions for each image, comprehensively testing model performance across six dimensions: category, color, quantity, counterintuitive perception, object relationship understanding, and overall counterintuitive recognition.

To get a better understanding of how counterintuitive phenomena are spread out in the dataset, we manually labeled each image with its counterintuitive type. We counted how often each type showed up. As shown in Figure 3, since a single image can have more than one counterintuitive type, the total number of types is more than the total number of images (4,698), which shows that multiple types of counterintuitive phenomena can exist at the same time.

Figure 3. Distribution of different types of common sense violations in ContraSense. Phys-Imposs: Physics-Impossible. Attr-Viol: Attribute Violation. Ctx-Viol: Context Violation. Sz-Prop-Anom: Size/Proportion Anomalies. Func-State-Perv: Function/State Perversity.

the shortcoming of conventional evaluation metrics, which do not sufficiently account for the lower bounds of a model’s capabilities, we propose the Contra-Acc metric. We define the six-dimensional correctness vector for each sample as pi=(pi1,pi2,…,pi6)∈[0,1]6\mathbf{p}_i = (p_{i1}, p_{i2}, \ldots, p_{i6}) \in[0,1]^6, and define the error vector as ei=1−pi\mathbf{e}_i = \mathbf{1} - \mathbf{p}_i. Based on this vector, we formulate the metric as follows:

Si∞=1−∥ei∥∞=1−max⁡1≤j≤6eij=min⁡1≤j≤6pij;1N∑i=1NSi∞.S_i^{\infty} = 1 - \|\mathbf{e}_i\|_{\infty} = 1 - \max_{1 \leq j \leq6} e_{ij} = \min_{1 \leq j \leq6} p_{ij}; \qquad\frac{1}{N}\sum_{i=1}^{N} S_i^{\infty}.

We introduce a Chebyshev consistency metric based on the L∞L^{\infty} norm, leveraging the 1-Lipschitz continuous kernel to prioritize the weakest dimension in modeling multidimensional consistency. Maximum deviation is employed to characterize consistency disparities across these dimensions. Contra-Acc offers a straightforward yet effective approach to pinpointing a model’s weakest links, thereby enhancing our understanding and improving its multidimensional capabilities.

Implemental details. In terms of implementation details, we selected GPT-4.1 [31] as the text generation large language model during the text generation stage to generate counterintuitive text. During the text-to-image generation stage, we employed FLUX.1-dev [32] as the text-to-image model to generate high-resolution images that match the given caption. In the answer option extraction component of the text generation model, we detect the keywords A/B/C/D/E in the responses and ensure that no English letters appear before or after these keywords, using this as the criterion for option extraction. When no answer options are detected, option A is returned by default as the model’s predicted result. This method has been previously applied in the literature [33]. We used 1 NVIDIA L40s graphics card in the testing phase.

Contra-intuitive Type Data Distribution

Through our carefully designed counterintuitive data collection process, we have built the ContraSense dataset, which contains 4,698 high-quality QA pairs. Unlike large-scale datasets that focus on the sheer volume of images, ContraSense does not prioritize the size of the dataset but rather the richness, accuracy, and structured semantic annotations provided for each image. Specifically, each image undergoes multiple rounds of rigorous manual screening, modification, and detailed annotation. The annotation content covers six dimensions: image caption, the categories of two core objects, color, quantity, whether it violates common-sense attributes, and the semantic relationship between the two.

Main Results

To comprehensively assess the impact of statistical co-occurrence on model performance, we created normal data that conformed to statistical co-occurrence characteristics and conducted systematic experiments on this data to compare differences with ContraSense data. Specifically, we followed the same prompt format as ContraSense and replaced the instructions for generating counterintuitive text with instructions for generating normal samples that conform to common sense. Since normal samples do not contain counterintuitive elements, we replaced counterintuitive questions with function questions. Other parts remained unchanged to minimize differences other than counterintuitive points. We use the same structured prompt template as ContraSense, while replacing the counter-intuitive constraints with commonsense-consistent instructions. The complete prompts are publicly available in the released benchmark repository. Subsequently, we conducted contrast experiments on the current state-of-the-art 8 VLM models using ContraSense and normal data. The experimental results are shown in Table II: On the Contra-Acc metric, open-source model performance on ContraSense data decreased by 37%-45% compared to normal data. Among these, LLaVA-NeXT-8B, DeepSeek-VL-7B-Chat, and Llama-3.2-11B-Vision achieved Contra-Acc scores of only 35%-37%, far below the average performance of approximately 80% on normal data, failing to meet expectations. In contrast, closed-source models showed overall performance improvements, but Contra-Acc still decreased by approximately 30% on ContraSense data. These results indicate that when data exceeds the scope of statistical co-occurrence, VLMs exhibit significant deficiencies in understanding and reasoning, often failing to remain faithful to the image content itself and being influenced by statistical co-occurrence bias. Comparison with Human Performance. They ultimately achieved an average Contra-Acc score of 96.04% on the ContraSense benchmark. This result highlights the significant gap between the current VLM and humans in accurately reflecting image content. Such gaps could have catastrophic consequences in critical applications such as autonomous driving and medical image analysis, further emphasizing the necessity of enhancing the model’s true visual understanding capabilities.

ModelImage EncoderLanguage ModelData TypeContraSense Bench Performance
Contra-Acc(%) ↑\uparrowROUGE-1 ↑\uparrowROUGE-2 ↑\uparrowROUGE-L ↑\uparrowMETEOR ↑\uparrowTimes
Human Performance--ContraSense96.04-----
Open-source Models
LLaVA-NeXT-8BCLIP-L-14Llama-3-8B-InstructContraSense36.9114.504.1711.2021.0001:28:16
Normal79.1817.487.5213.7525.9701:22:05
Δ\Delta-42.27-2.98-3.35-2.55-4.97-
DeepSeek-VL-7B-ChatSigLIP-L & SAM-BDeepSeek-LLM-7BContraSense35.8921.806.1616.4024.9901:13:09
Normal80.9724.049.2218.2930.0301:10:11
Δ\Delta-45.08-2.24-3.06-1.89-5.04-
Llama-3.2-11B-VisionViT-H-14Llama3.1-8BContraSense37.5529.577.5922.2120.3600:52:07
Normal81.7433.9811.6826.7125.1100:51:35
Δ\Delta-44.19-4.41-4.09-4.5-4.75-
Qwen2.5-VL-7B-InstructViT(SwiGLU, RMSNorm)Qwen2.5 LLMContraSense54.0215.474.5211.6520.1501:57:11
Normal91.9517.116.7413.0823.1101:56:30
Δ\Delta-37.93-1.46-2.22-1.43-2.96-
Close-source Models
Gemini-1.5-pro--ContraSense51.9813.724.5010.4719.9704:55:35
Normal85.4416.516.9112.8124.2004:40:23
Δ\Delta-33.46-2.79-2.41-2.34-4.23-
Gemini-2.0-flash--ContraSense47.1313.963.9210.5518.8203:24:17
Normal80.7216.716.2512.7822.4503:14:00
Δ\Delta-33.59-2.75-2.33-2.23-3.63-
GPT-4o-GPT-4ContraSense52.1112.353.639.4818.3307:46:48
Normal85.4414.175.4110.9921.5506:38:13
Δ\Delta-33.33-1.82-1.78-1.51-3.22-
GPT-4.1-GPT-4ContraSense60.0312.953.899.7819.0506:44:14
Normal86.0815.345.7111.6022.9706:43:13
Δ\Delta-26.05-2.39-1.82-1.82-3.92-

Table II. Model performance evaluation on the ContraSense Benchmark. We systematically evaluated the overall performance of eight state-of-the-art VLM models on the ContraSense dataset and normal data. The Δ\Delta row indicates the difference between the two datasets across each metric. The experimental results show that all models perform significantly lower on the ContraSense dataset compared to the normal data, indicating that these models still have limited capability in accurately reflecting image content.

Model Performance Across Violation Types

As for the caption generation evaluation, the performance of the four text matching metrics, ROUGE-1, ROUGE-2, ROUGE-L, and METEOR, on the ContraSense dataset is generally significantly degraded compared to the normal data. Specifically, the open-source models of ROUGE-1, ROUGE-2, ROUGE-L, and METEOR show an average performance degradation ranging from 1.5% to 5%. The closed-source models also show a similar downward trend of about 1.5% to 4.2%. This overall decline suggests that the models face challenges not only in choosing answers but also in the quality of the generated text when faced with the counterintuitive graphic matching task. Especially in the ContraSense, literal overlap and semantic alignment between the model-generated answers and the reference answers are disturbed by statistical co-occurrence bias, which makes it challenging to reflect the real information of the images faithfully. This result further reveals the general limitations of current VLMs in dealing with counterintuitive scenarios.

To more clearly analyze the specific impact of the five types of contra-intuitive defined in the ContraSense dataset on the model inference process, we conducted performance evaluations for each type of anomaly separately. As shown in Table III, compared to normal samples, data from all contra-intuitive types resulted in varying degrees of performance degradation in the model. This finding indicates that current multimodal models still exhibit significant limitations in reasoning capabilities when faced with semantically contra-intuitive or counter-intuitive scenarios, particularly in terms of semantic robustness and common-sense consistency, where there is considerable room for improvement.

ModelImage EncoderLanguage ModelData TypeContra-Acc(%) ↑ROUGE-1 ↑ROUGE-2 ↑ROUGE-L ↑METEOR ↑
Open-source Models
LLaVA-NeXT-8BCLIP-L-14Llama-3-8B-InstructNormal79.1817.487.5213.7525.97
Context Violation42.4915.084.3411.4021.42
Attribute Violation38.1015.004.3611.5021.51
Size/Proportion Anomalies34.8314.354.2311.2220.89
Physics-Impossible35.6114.223.9811.0620.66
Function/State Perversity32.7313.723.7910.6320.65
DeepSeek-VL-7B-ChatSigLIP-L & SAM-BDeepSeek-LLM-7BNormal80.9724.049.2218.2930.03
Context Violation41.8522.326.4516.4525.27
Attribute Violation33.5822.616.6816.8325.71
Size/Proportion Anomalies32.2121.726.0216.3524.58
Physics-Impossible39.1521.185.6815.9924.26
Function/State Perversity32.7320.465.1915.6424.28
Llama-3.2-11B-VisionViT-H-14Llama3.1-8BNormal81.7433.9811.6826.7125.11
Context Violation40.5829.947.6822.3520.15
Attribute Violation34.5929.837.8422.2320.29
Size/Proportion Anomalies33.3328.997.5821.9819.71
Physics-Impossible39.1529.177.5521.9820.48
Function/State Perversity38.1829.177.1522.3321.18
Qwen2.5-VL-7B-InstructViT(SwiGLU, RMSNorm)Qwen2.5 LLMNormal91.9517.116.7413.0823.11
Context Violation57.5115.954.6711.8220.65
Attribute Violation52.1315.944.6911.8920.48
Size/Proportion Anomalies50.5615.184.4011.5419.76
Physics-Impossible54.7215.174.3811.3819.89
Function/State Perversity50.0015.184.2811.4020.30
Close-source Models
Gemini-1.5-pro--Normal85.4416.516.9112.8124.20
Context Violation53.3513.454.5610.1319.78
Attribute Violation51.3813.624.6110.3319.94
Size/Proportion Anomalies52.0613.764.4510.5919.88
Physics-Impossible51.1813.944.4110.6420.26
Function/State Perversity46.3613.284.1310.1619.24
Gemini-2.0-flash--Normal80.7216.716.2512.7822.45
Context Violation49.2013.964.1710.4519.04
Attribute Violation43.8614.264.0010.7119.32
Size/Proportion Anomalies47.1913.603.9510.4718.42
Physics-Impossible47.4113.973.8010.5718.66
Function/State Perversity40.0013.303.4610.0218.07
GPT-4o-GPT-4Normal85.4414.175.4110.9921.55
Context Violation55.2712.333.649.3418.51
Attribute Violation52.1312.253.649.4118.43
Size/Proportion Anomalies52.0612.333.579.4618.13
Physics-Impossible50.7112.623.679.7218.39
Function/State Perversity40.0012.183.269.2718.05
GPT-4.1-GPT-4Normal86.0815.345.7111.6022.97
Context Violation62.3012.773.869.6018.91
Attribute Violation58.9012.823.899.7018.98
Size/Proportion Anomalies62.1713.003.929.9019.12
Physics-Impossible61.5613.233.909.9619.25
Function/State Perversity45.4512.373.489.3418.24

Table III. Model performance was evaluated on the ContraSense dataset for different types of counterintuitive samples.

Context violation represents a relatively easier type of counterintuitive case to identify. Most models—including open-source ones such as LLaVA-NeXT and Qwen2.5-VL, as well as closed-source models like GPT-4.1 and Gemini—achieve relatively high accuracy on this category. We speculate that this is because semantic conflicts between objects and scenes are typically more explicit, making it easier for models to localize these counterintuitive regions and perform semantic reasoning. Even in the absence of explicit training for counterintuitive reasoning, models may still detect semantic violations by leveraging statistical co-occurrence patterns learned during pretraining. Attribute violations and size/proportion anomalies consistently yield lower performance, particularly in open-source models. These two types of anomalies often involve fine-grained perceptual cues (e.g., abnormal colors), requiring detailed visual discrimination and precise cross-modal alignment. Their subtle nature further increases the difficulty of detection.

Recognition performance on the Physics-Impossible category is moderate—slightly better than on attribute-related anomalies, yet lower than on context violations. These anomalies, such as objects floating in mid-air or defying gravity, are often visually salient; however, the model’s ability to interpret their implausibility depends on whether it encodes physical common sense, which is typically not explicitly represented during pre-training.

Function/State Perversity poses the greatest challenge, yielding the lowest accuracy across all models. These anomalies involve atypical combinations of object functionality and state logic (e.g., using a banana as a phone), often requiring reasoning that goes beyond superficial visual cues and depends on action understanding and commonsense knowledge. Most models exhibit near-random performance on such samples, reflecting fundamental limitations in semantic reasoning capabilities.

Overall, while closed-source models (e.g., GPT-4.1 and the Gemini series) consistently outperform open-source counterparts across all anomaly types, they still face significant bottlenecks in handling Function/State Perversity and attribute-based anomalies. This suggests that although larger model scales and pre-training data improve robustness to some extent, current models continue to struggle with generalization in semantically complex, fine-grained, and commonsense-reliant scenarios.

Vision-free Testing: Exposing Statistical Co-occurrence Reliance

Image Ablation. In order to deeply understand the impact of statistical co-occurrence on model performance, we designed and implemented image ablation experiments. In terms of the experimental setup, for image input, we replaced all images in the ContraSense and normal data with noise images, completely dissociating them from the text content to ensure that no information relevant to answering the question could be obtained from the images. In terms of question selection, the five categories of questions—Object Class, Object Colour, Object Num, Is Counterintuitive, and Relation — focus on examining image content from different dimensions. However, their answers do not necessarily involve counterintuitive points. Therefore, it is difficult to directly compare the differences in image content between normal questions and counterintuitive questions, making it insufficient to comprehensively validate the impact of statistical co-occurrence under conditions where image content is missing. For this purpose, we selected the Anomaly Recognition Question (corresponding to the Commonsense Reasoning Question in the normal dataset) as the comparison question. This question necessarily involves counterintuitive queries in the ContraSense group, which helps minimize the interference of common sense and thus more accurately assess the impact of statistical co-occurrence. In the experiment, we excluded the explicit influence of statistical co-occurrence in the ContraSense group from both image and text perspectives, using noise images that contain no image information for comparison testing. The experimental results are shown in Figure 4. On the ContraSense dataset, when image content is unavailable, the model’s performance is significantly lower than that on normal data, indicating that the model cannot infer the answers to the Anomaly Recognition Question in ContraSense solely based on statistical co-occurrence. Remarkably, on the normal data, even when image content is unavailable, the model can still achieve an average performance exceeding 95% by relying on statistical co-occurrence. **This result indicates that statistical cooccurrence can significantly enhance model performance under conditions where image information is unavailable, enabling the model to make correct judgments in specific tasks without relying on real visual information. However, in addition to the aforementioned visual-text or object-object level semantic statistical co-occurrence, there exists another easily overlooked yet equally critical form of statistical co-occurrence: statistical co-occurrence at the level of answer choice positions. This co-occurrence does not originate from visual or semantic content itself, but manifests as systematic biases toward specific answer choice positions during the model’s multi-choice decision-making process.

Figure 4. We replaced the input images with noisy images and conducted blind tests on ANOMALY RECOGNITION QUESTIONS and COMMONSENSE REASONING QUESTION. The results showed that the model’s blind test performance on the normal data was significantly higher than its performance on the ContraSense dataset, averaging a high level of 95%, indicating that even in the absence of image content, the model can still obtain the correct answer based on statistical co-occurrence. The red dashed line represents the randomly selected probability baseline.

Analysis of Performance Failures

In Section IV-C, we analyzed the phenomenon of widespread systematic performance degradation when MLLMs encounter problems that significantly violate the statistical co-occurrence-based knowledge prior. Furthermore, the marked differences in performance range across different problem types indicate a highly negative correlation between the degree of deviation from statistical co-occurrence structures and model performance on each problem type: the more severe the deviation, the more pronounced the decline in model performance. The root cause lies in the model’s inability to establish stable, generalizable mappings when semantic statistical co-occurrence is disrupted—whether between objects within an image or between images and text—leading to performance degradation.

In addition to semantic co-occurrence between visual objects and textual concepts, multiple-choice models may also exploit statistical regularities associated with answer positions. We therefore analyze these two effects separately. A complementary Macro-F1 analysis, which reduces the influence of uneven predictions across answer slots, confirms that the performance ranking across question types remains broadly consistent with that obtained using Accuracy; complete per-model results and detailed discussion are provided in Section B and Table VI of the Supplementary Material. In the following, we focus on the second effect and examine systematic option-position preferences through row-normalized confusion matrices.

Option-level statistical co-occurrence. To analyze the impact of option-level statistical co-occurrence on model behavior at the decision-making level, we constructed confusion matrices for the outputs of 8 MLLMs. We visualized the results, as shown in Figure 5. Where each row represents the option position corresponding to the true label, and each column represents the option position predicted by the model (predicted label). Since options A–E correspond to different candidate answers across questions and do not represent fixed semantic categories, we applied row normalization to the confusion matrix. This allows us to characterize the conditional dispersion and misalignment patterns of model predictions across remaining slot positions, given the known true label slot. From this conditional probability perspective, the confusion matrix is no longer used to analyze confusion between semantic categories. Instead, it diagnoses non-semantic systematic behavioral patterns in the model during multi-choice decision scenarios, such as position bias and slot sensitivity. Furthermore, Table IV reports statistical results obtained by column-wise aggregation of row-normalized confusion matrices. Since A–E do not correspond to fixed semantic classes, these column sums reflect the model’s overall selection tendencies toward each option slot under different ground-truth conditions. They can be interpreted as relative preference intensity. Theoretically, if the model exhibits no option slot bias, each column sum should approach 1.

Row-normalized confusion matrices for eight MLLMs

Figure 5. Row-normalized confusion matrices for eight MLLMs on the ContraSense benchmark. Each row corresponds to the true answer slot (A–E), and each column denotes the predicted answer slot.

ModelABCDE
Open-source
LLaVA-NeXT-8B1.3811.2311.1270.7460.515
DeepSeek-VL-7B-Chat1.4781.2071.0000.7130.603
Llama-3.2-11B-Vision-Instruct1.5230.9191.0920.7810.685
Qwen2.5-VL-7B-Instruct1.0161.2001.1510.8590.774
Closed-source
Gemini1.7161.1431.3170.5700.254
Gemini-2.0-Pro1.1290.9641.2290.9300.748
GPT-4o1.2641.0391.3160.7390.643
GPT-4.11.2841.0621.3100.7160.629

Table IV. Column-wise sums of row-normalized confusion matrices, reflecting models’ positional bias over options.

Experimental results show that most models exhibit a significant preference for options A and C, while consistently avoiding the last option. Column E ranks lowest or second-lowest across nearly all models. Empirically, this phenomenon may relate to the distribution of question formats in training data: existing large-scale training corpora predominantly feature four-choice questions (A–D). In contrast, multiple-choice questions with an E option are relatively scarce. This induces statistical co-occurrence of specific option positions at the decision layer during training. This indicates that current models remain influenced by non-semantic positional priors during multiple-choice decision-making, with this bias originating from the structure of the options themselves rather than failures in understanding visual or semantic content. Further comparison across model types reveals that closed-source models generally exhibit a concentrated preference for A/C options, with their consistent distribution across rows indicating higher stability of this bias. In contrast, open-source models exhibit mixed preferences for A/B/C options, with more dispersed bias distributions and more random error transfer. This comparison indicates that stronger models do not eliminate option position bias but instead solidify it in a more stable and consistent form.

In summary, through column-wise aggregation analysis of the confusion matrix, we observe a systematic bias toward certain option positions that is consistent across models: models generally exhibit significant attraction to the first and middle options while systematically avoiding the last option. When statistical co-occurrence at the semantic or visual level is disrupted, models often degrade into relying on the statistical frequency or linguistic prior of the answer slot position itself, and frequently fall back to fixed options. This creates a non-causal statistical co-occurrence structure within the decision space, ultimately leading to systematic positional bias.

Conclusion

In this paper, we introduce a brand-new benchmark dataset: the ContraSense Benchmark. By constructing counterintuitive scenarios under multi-object conditions, we have maximized the elimination of the influence of statistical co-occurrence on question answering, thereby enabling a more accurate assessment of the model’s ability to understand image content faithfully. Additionally, we propose the Contra-Acc metric, designed to comprehensively quantify the model’s understanding of image content across multiple dimensions. Experimental results demonstrate that even current state-of-the-art models significantly lag behind human performance in terms of image content understanding.

Comparison With Existing Multimodal Benchmarks

ContraSense has significant advantages over existing benchmarks in three key areas. First, our benchmark supports multiple answer types, covering both multiple-choice and open-ended task formats. In contrast, most current multimodal evaluation benchmarks only provide open-ended annotations, making it challenging to explore differences in model behavior across different answer settings effectively. Diverse answer types enable more granular capability assessment. Second, ContraSense focuses on contra-intuitive reasoning tasks, a capability that is currently underrepresented in mainstream evaluation frameworks. Existing benchmarks primarily focus on conventional text-image question-answering or description generation tasks, lacking structured evaluation for content that violates common-sense semantics. Therefore, ContraSense fills an important gap in this field, enabling the evaluation of a model’s understanding and reasoning capabilities when faced with semantic anomalies and counter-intuitive scenarios. This minimizes the influence of statistical co-occurrence on model performance. Finally, our dataset employs a fully manual annotation process, covering the construction of image-text pairs, the design of questions and answers, and the manual classification of anomaly types. The annotation process involves human annotators, ensuring semantic consistency and annotation quality, providing a solid foundation for reliable evaluation. Tab. V compares ContraSense with multiple mainstream MLLM benchmarks, highlighting our systematic advantages in semantic counter-intuitive coverage, multi-task support, and annotation accuracy.

Per-Question Evaluation and Analysis

Contra-Acc is used to assess the lower bound of a model’s capability across various task dimensions, thereby highlighting its weakest links. To further analyze the model’s specific behavior across different problem types, we conducted independent performance tests for each of the six problem categories in the ContraSense benchmark: Object Class, Object Colour, Object Num, Counterintuitive, Relation, and Anomaly Recognition. The final row of the table shows the overall performance, representing the average results across all problem types. Since not all problems involve counterintuitive factors, we additionally manually selected a batch of image-problem pairs where the questions explicitly violate statistical co-occurrence patterns. These were used to evaluate the model’s performance in such scenarios. The results are shown in tbl:6, from the accuracy metric perspective, which measures the overall correct selection rate of the model, significant performance differences among various models can be observed across different question types. Even the best-performing model overall, GPT-4.1, achieved only 45.01% accuracy; other models scored between 31% and 43%. This generally low performance indicates that current models exhibit significant degradation when confronted with inputs that deviate from statistical co-occurrence patterns. This suggests their predictive capabilities remain heavily reliant on semantic co-occurrence rather than genuine visual-semantic understanding.

Analysis based on the Accuracy metric indicates that, when comprehensively evaluating the performance of open-source and closed-source models across six problem types, their overall capabilities exhibit a descending order of performance: Anomaly Recognition > Relation ≈ Object Num > Counterintuitive > Object Colour > Object Class.

Across different problem types, nearly all models exhibit the lowest accuracy on Object Class tasks. Specifically, the accuracy scores of open-source models predominantly cluster within the 20%–35% range. While closed-source models generally outperform their open-source counterparts, their performance remains significantly inferior to that of other

ModelsData ScaleAnswer TypeContra-intuitiveHuman Annotation
HumanEval [34]164Open❌❌
MM-Vet [35]218Open❌❌
INTERLEAVEDBENCH [36]815Open❌✅
MMIE [37]20KMulti-Choice & Open❌✅
ContraSense (Ours)4,698Multi-Choice & Open✅✅
ModelImage EncoderLanguage ModelData TypeAccuracy(%) ↑F1 Macro(%) ↑F1 Weighted(%) ↑Precision(%) ↑Recall(%) ↑MCC(%) ↑
Open-source Models
LLaVA-NeXT-8BCLIP-L-14Llama-3-8B-InstructObject Class19.0512.6221.0214.4711.89-7.72
Object Colour30.0025.1632.1727.4728.7211.01
Object Num37.5033.0141.3438.7030.0318.36
Counterintuitive36.0527.5336.4727.3827.919.06
Relation47.6234.4254.5536.8442.6129.41
Anomaly Recognition24.0719.6224.8822.1619.99-2.00
Overall Performance32.7729.1034.3829.5031.5810.54
DeepSeek-VL-7B-ChatSigLIP-L & SAM-BDeepSeek-LLM-7BObject Class31.7521.7632.3122.5122.738.89
Object Colour29.0025.1432.2128.2429.0510.14
Object Num52.0844.3954.4746.2757.4338.43
Relation34.2922.9341.5626.9221.969.95
Counterintuitive37.7734.2237.7233.0238.5611.01
Anomaly Recognition20.9917.4321.6820.3019.87-2.68
Overall Performance32.6329.3133.9930.2632.6611.27
Llama-3.2-11B-VisionViT-H-14Llama3.1-8BObject Class34.9223.4537.0426.3623.8613.29
Object Colour27.0020.0429.3522.6620.295.15
Object Num27.0822.2430.4630.9820.639.08
Counterintuitive30.0427.6828.9226.1635.922.47
Relation39.0526.8846.3331.1233.4117.38
Anomaly Recognition32.7230.2534.4931.7437.8213.36
Overall Performance31.7927.6433.0228.1431.7510.03
Qwen2.5-VL-7B-InstructViT(SwiGLU, RMSNorm)Qwen2.5 LLMObject Class47.6235.9250.4237.3837.1730.08
Object Colour46.0036.3948.9138.6639.1729.29
Object Num56.2546.5657.8551.5848.7942.73
Counterintuitive35.6230.8136.3129.2246.9411.16
Relation60.0042.0066.2742.7646.8840.76
Anomaly Recognition20.3716.8423.4420.3018.25-3.14
Overall Performance39.6634.4841.6634.6039.4520.40
Close-source Models
Gemini-1.5-pro--Object Class42.8629.9945.6631.5629.0021.57
Object Colour31.0024.6433.6027.3924.359.66
Object Num39.5831.5140.9236.5645.7121.89
Counterintuitive46.3534.1346.9234.2934.6524.52
Relation30.4821.9934.3324.3232.727.53
Anomaly Recognition55.5646.8756.1848.5049.8240.49
Overall Performance43.1834.1244.2034.7634.6122.93
Gemini-2.0-flash--Object Class34.9225.6536.9427.6325.7414.16
Object Colour33.0030.3635.0632.1335.3816.11
Object Num45.8339.1151.3044.1737.2132.25
Counterintuitive24.8920.7525.2920.1329.15-2.72
Relation39.0527.9644.9730.6638.0819.43
Anomaly Recognition65.4355.1167.5854.6761.0553.83
Overall Performance39.6635.0941.0834.5840.3420.88
GPT-4o-GPT-4Object Class41.2728.5344.0130.6627.4820.73
Object Colour32.0025.8734.6827.8525.6812.25
Object Num35.4229.6339.3133.7828.3517.71
Counterintuitive28.3322.3228.5022.1131.432.68
Relation39.0527.1543.9931.0034.4222.25
Anomaly Recognition66.0556.6766.9657.9059.8853.98
Overall Performance40.6534.8142.1634.5438.1820.50
GPT-4.1-GPT-4Object Class46.0338.6648.9035.4233.9826.98
Object Colour35.0027.5538.1129.9126.6115.22
Object Num41.6734.8847.8345.1032.0327.34
Counterintuitive30.9023.7131.8423.9524.874.70
Relation45.7132.8150.6334.2342.8326.57
Anomaly Recognition71.6061.5972.3261.6064.1061.77
Overall Performance45.0138.2646.4038.0941.3426.15

Table V. The comparison between ContraSense with other LVLM benchmarks.

question types. This phenomenon suggests that when the task object category significantly deviates from statistical co-occurrence-based prior knowledge, current MLLMs exhibit systemic deficiencies in classifying counterintuitive objects.

Regarding the Object Colour task, despite color typically being considered a relatively “simple” perceptual attribute, overall accuracy remains low, with most models achieving around 30%. This result suggests that such errors do not originate from failures in low-level perception, but are more likely related to erroneous binding between color and object, or prior interference introduced by anomalous color settings. Even for perceptual attribute tasks, model prediction accuracy drops significantly when encountering counterintuitive semantics or weakened object-attribute bindings.

For Counterintuitive problems, performance differences between models are relatively minor, with accuracy typically clustering between 30% and 40%. This suggests that models can perceive the presence of “counterintuitiveness” to some extent but struggle to make stable, generalizable, and correct decisions in such judgments.

For Object Num problems, nearly all models achieved significantly higher accuracy than Object Class, Object Colour, and Counterintuitive tasks. However, overall performance remains suboptimal, with only a few models achieving an accuracy of over 50%.

This indicates that current MLLMs still suffer widespread performance degradation when counting tasks involve counterintuitive objects.

For Relation problems, performance disparities among models widened further. Although these tasks do not yet involve complex spatial reasoning or multi-step logical inference, their difficulty extends beyond purely perceptual capabilities. The increased complexity amplified the differences in models’ abilities.

In Anomaly Recognition tasks, nearly all models achieved the highest accuracy. This is closely tied to the nature of the task: such problems lean more toward holistic semantic judgment, typically presenting candidate descriptions in multiple-choice format and requiring models to match images with counterintuitive text given the image context.

Notably, the performance ranking across different problem types aligns closely with their degree of disruption to statistical co-occurrence structures. When problems directly introduce object category errors or counterintuitive color settings (e.g., Object Class and Object Colour), statistical co-occurrence is severely disrupted, leading to the lowest model performance. In contrast, Counterintuitive and Object Num problems, which involve anomalies only at the object attribute or behavioral level, yield somewhat improved performance. In contrast, Relation and Anomaly Recognition problems exhibit weaker disruption to statistical co-occurrence patterns, focusing more on semantic matching between images and descriptions, thus yielding relatively better performance. Overall, this trend indicates that current model predictions remain heavily reliant on statistical co-occurrence structures and have yet to develop a sufficient understanding of genuine visual semantics.

Disentangles option position statistics from co-occurrence interference. In tbl:6, we report performance results based on F1 Macro. F1 Macro is defined as F1Macro=1N∑i=1NF1iF1_{\mathrm{Macro}} = \frac{1}{N}\sum_{i=1}^{N} F1_i, where NN denotes the number of categories. This metric employs an equal-weight average (also known as a non-weighted average) of the F1 scores across all categories (options). It is commonly used in traditional classification tasks to mitigate evaluation bias caused by class imbalance. It should be noted that in ContraSense’s task design, options A–E do not correspond to fixed semantic categories (e.g., one option always representing cats or dogs). Instead, they denote discrete answer slots for the same question. Therefore, these options do not inherently represent high-frequency or long-tail categories in a semantic sense. Under this premise, F1 Macro no longer measures semantic category imbalance; instead, it imposes an equal-weight constraint on prediction behavior across different answer slots. Essentially, F1 Macro penalizes answer-position bias—the statistical dependency a model develops toward specific answer slots during multi-choice decision-making. Thus, F1 Macro can be viewed as a metric designed to suppress statistical co-occurrence at the answer slot level. By eliminating interference from statistical co-occurrence across slots, F1 Macro effectively removes decision-level statistical noise. This disentangles the statistical interference introduced by slot positions, making the impact of semantic co-occurrence deviation across different question types clearer and more distinguishable in terms of performance. Under this framework, across all models (including open-source and closed-source models) and six question types, the performance ranking based on F1 Macro exhibits high consistency with that of Accuracy. Compared to Accuracy, this ranking demonstrates greater stability while significantly amplifying performance differences between models. This phenomenon reinforces the conclusion drawn from Accuracy: current MLLMs’ predictions remain heavily reliant on statistical co-occurrence structures, lacking sufficient understanding of genuine visual-semantic relationships. When question content deviates strongly from these statistical patterns, models inevitably experience systematic performance degradation.

Impact of Resolution Variation

To systematically assess the impact of image resolution on the performance of multimodal models, we conducted resolution ablation experiments on three types of models: Qwen2.5-VL-7B-Instruct, which uses a dynamic resolution strategy; LLaMA-3.2-11B-Vision, which uses a fixed resolution strategy; and the closed-source model GPT-4.1. We progressively reduced the input images to different resolutions and measured model performance on the ContraSense benchmark, with results shown in Figure 6.

Figure 6. The figure shows the performance trends of mainstream multimodal models at different resolutions.

From an overall trend perspective, all models exhibit varying degrees of performance degradation when the input resolution is reduced. This indicates that the models’ perceptual capabilities are sensitive to the distribution of image details. Specifically, Qwen2.5-VL-7B-Instruct is the most sensitive to resolution changes, showing a significant decline in performance. When the input image resolution is reduced to 64×6464 \times64, its Contra-Acc accuracy rate drops to just 8.68%. This phenomenon highlights the limitations of dynamic resolution strategies when dealing with extremely low-resolution inputs: since their core objective is to balance perceptual capability with computational overhead, when images are forced to be compressed, local structures and micro-semantics are significantly lost, leading to overly sparse visual token representations. As a result, the model struggles to obtain sufficient semantic signals, ultimately causing a sharp decline in performance.

In contrast, LLaMA-3.2-11B-Vision employs a fixed-resolution input strategy, resulting in a more gradual decline in performance. This is primarily due to the fact that during training, the model consistently receives images of uniform dimensions (e.g., 224×224224 \times224 or 448×448448 \times448), enabling it to learn to extract stable and robust features at a specific scale. Even when the original resolution of the input images is reduced, after interpolation or scaling to a fixed size, it can still retain some key semantic structures, thereby maintaining relatively stable inference performance.

GPT-4.1’s performance is least affected by resolution changes and remains almost stable. Empirical analysis suggests that this robustness may stem from its training process covering a wide distribution of image sizes, combined with diverse data augmentation strategies, enabling the model to achieve good scale generalization capabilities.

It is worth noting that ContraSense adopts 1024×10241024 \times1024 images, preserving micro-semantic differences (small objects, background cues, edge details) and producing denser patch-token representations. This design improves image-text alignment and enables high-resolution models such as GPT-4.1 to realize their potential on fine-grained image-text reasoning.

Limitations

The semi-automated nature of the ContraSense dataset construction process introduces considerations of scalability. While the introduction of manual screening and correction ensures high consistency between the pairing image and text and the clarity of counterintuitive content in the image, this approach demands significant human resource costs, hence the decrease in efficiency compared to a fully automated process. Moving forward, exploring more efficient construction workflows to reduce human resource consumption represents a valuable direction for future research.

Ethical Statement

This research dataset is entirely generated by text-to-image models and contains no real individuals, geographic locations, or sensitive events. All content has undergone manual review and security filtering. All samples are used for academic research purposes to evaluate the model’s semantic understanding and robustness, posing no public safety or value-oriented risks. Therefore, there are no potential ethical concerns.

References

  1. [1]X. Chen, X. Wang, S. Changpinyo et al., “Pali: A jointly-scaled multilingual language–image model,” arXiv:2206.07974, 2022.
  2. [2]J.-B. Alayrac, J. Donahue, P. Luc et al., “Flamingo: A visual language model for few-shot learning,” arXiv:2204.14198, 2022.arxiv.org/abs/2204.14198
  3. [3]H. Bao, Q. Zhang et al., “Kosmos-2: Grounding multimodal large language models to the world,” arXiv:2306.14824, 2023.
  4. [4]H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” arXiv:2304.08485, 2023.arxiv.org/abs/2304.08485
  5. [5]OpenAI, “Hello gpt-4o,” 2024, https://openai.com/index/hello-gpt-4o/.
  6. [6]G. Team, P. Georgiev, V. I. Le, R. Burnell, L. Bai, A. Gulati, G. Tanzer, D. Vincent, Z. Pan, S. Wang et al., “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of tokens of context,” arXiv preprint arXiv:2403.05530, 2024.
  7. [7]D. Driess, F. Xia, M. Sajjadi et al., “Palm-e: An embodied multimodal language model,” arXiv:2303.03378, 2023.arxiv.org/abs/2303.03378
  8. [8]J. Shi, X. Yin, Y. Chen, Y. Zhang, Z. Zhang, Y. Xie, and Y. Qu, “Multi-schema proximity network for composed image retrieval,” in ICCV, 2025, pp. 19 999–20 008.DOI
  9. [9]Y. Wang, H. Liu, Y. Feng, Z. Li, X. Wu, and C. Zhu, “Headdiff: Exploring rotation uncertainty with diffusion models for head pose estimation,” IEEE TIP, vol. 33, pp. 1868–1882, 2024.DOI
  10. [10]C. Raj, A. Mukherjee, A. Anastasopoulos, and Z. Zhu, “Biasdora: Exploring hidden biased associations in vision-language models,” arXiv preprint arXiv:2407.02066, 2024.arxiv.org/abs/2407.02066
  11. [11]L. Wang, C. Zhang, A.-A. Liu, B. Yang, M. Hu, X. Qiao, L. Wang, J. He, and Q. Liu, “Toward chinese food understanding: A cross-modal ingredient-level benchmark,” IEEE TMM, 2024.DOI
  12. [12]Z. Li, J. Liao, C. Tang, H. Zhang, Y. Li, Y. Bian, X. Sheng, X. Feng, Y. Li, C. Gao et al., “Ustc-td: A test dataset and benchmark for image and video coding in 2020s,” IEEE TMM, 2025.arxiv.org/abs/2409.08481
  13. [13]S. Li, Z. Wang, J. Fuefei-Xu, Q. Guo, X. Li, and L. Ma, “Common corruption robustness of point cloud detectors: Benchmark and enhancement,” IEEE TMM, vol. 27, pp. 848–859, 2023.arxiv.org/abs/2210.05896
  14. [14]K. Wang, D. Lin, C. Li, Z. Tu, and B. Luo, “Alignment-free rgbt salient object detection: Semantics-guided asymmetric correlation network and a unified benchmark,” IEEE TMM, vol. 26, pp. 10 692–10 707, 2024.arxiv.org/abs/2406.00917
  15. [15]F. Hou, J. Yuan, Y. Yang, Y. Zhang, Y. Liu, Y. Zhang, C. Zhong, Z. Shi, J. Fan, Z. He et al., “Domainverse: A benchmark towards real-world distribution shifts for training-free adaptive domain generalization,” IEEE TMM, 2025.DOI
  16. [16]B. Li, R. Wang, G. Wang et al., “Seed-bench: Benchmarking multimodal large language models with generative comprehension,” arXiv:2307.16125, 2023.arxiv.org/abs/2307.16125
  17. [17]B. Li, Y. Ge, Y. Ge et al., “Seed-bench-2: Benchmarking multimodal large language models,” arXiv:2311.17092, 2023.arxiv.org/abs/2311.17092
  18. [18]S. Ma, F. Liu, L. Li et al., “Mmbench: Is your multi-modal model an all-round player?” arXiv:2307.06281, 2023.
  19. [19]C. Fu, P. Chen, Y. Shen et al., “Mme: A comprehensive evaluation benchmark for multimodal large language models,” arXiv:2306.13394, 2023.arxiv.org/abs/2306.13394
  20. [20]W. Yu, C. Fang, Y. Sun et al., “Mm-vet: Evaluating large multimodal models for integrated vision–language abilities,” arXiv:2308.02490, 2023.arxiv.org/abs/2308.02490
  21. [21]Y. Zhang, J. Zhang, Z. Gan et al., “Lvlm-ehub: A comprehensive evaluation hub for large vision–language models,” arXiv:2306.09265, 2023.arxiv.org/abs/2306.09265
  22. [22]D. A. Hudson and C. D. Manning, “Gqa: A new dataset for real-world visual reasoning and compositional question answering,” in CVPR, 2019.arxiv.org/abs/1902.09506
  23. [23]Y. Li, Y. Du, K. Zhou et al., “Evaluating object hallucination in large vision–language models,” in EMNLP, 2023.arxiv.org/abs/2305.10355
  24. [24]K. Zhou, E. Lai, W. B. A. Yeong, K. Mouratidis, and J. Jiang, “Rome: Evaluating pre-trained vision-language models on reasoning beyond visual common sense,” in EMNLP, 2023.
  25. [25]N. Bitton-Guetta, Y. Bitton, J. Hessel, L. Schmidt, Y. Elovici, G. Stanovsky, and R. Schwartz, “Breaking common sense: Whoops! a vision-and-language benchmark of synthetic and compositional images,” in ICCV, 2023, pp. 2616–2627.arxiv.org/abs/2303.07274
  26. [26]M. Sokolova and G. Lapalme, “A systematic analysis of performance measures for classification tasks,” Information processing & management, vol. 45, no. 4, pp. 427–437, 2009.DOI
  27. [27]B. W. Matthews, “Comparison of the predicted and observed secondary structure of t4 phage lysozyme,” Biochimica et Biophysica Acta (BBA)-Protein Structure, vol. 405, no. 2, pp. 442–451, 1975.pubmed.ncbi.nlm.nih.gov/1180967
  28. [28]J. Gorodkin, “Comparing two k-category assignments by a k-category correlation coefficient,” Computational biology and chemistry, vol. 28, no. 5-6, pp. 367–374, 2004.pubmed.ncbi.nlm.nih.gov/15556477
  29. [29]C. Lin, “Recall-oriented understudy for gisting evaluation (rouge),” Retrieved August, vol. 20, p. 2005, 2005.
  30. [30]A. Lavie and M. J. Denkowski, “The meteor metric for automatic evaluation of machine translation,” Machine translation, vol. 23, pp. 105–115, 2009.DOI
  31. [31]OpenAI, “gpt4one,” in Proceedings of GPT-4 Development, 2025, https: //openai.com/index/gpt-4-1/.
  32. [32]B. F. Labs, “Flux,” https://github.com/black-forest-labs/flux, 2024.
  33. [33]Y. Zhang, A. Unell, X. Wang, D. Ghosh, Y. Su, L. Schmidt, and S. Yeung-Levy, “Why are visually-grounded language models bad at image classification?” in NeurIPS, 2024.arxiv.org/abs/2405.18415
  34. [34]M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374, 2021.arxiv.org/abs/2107.03374
  35. [35]W. Yu, Z. Yang, L. Li, J. Wang, K. Lin, Z. Liu, X. Wang, and L. Wang, “Mm-vet: Evaluating large multimodal models for integrated capabilities,” arXiv preprint arXiv:2308.02490, 2023.arxiv.org/abs/2308.02490
  36. [36]M. Liu, Z. Xu, Z. Lin, T. Ashby, J. Rimchala, J. Zhang, and L. Huang, “Holistic evaluation for interleaved text-and-image generation,” arXiv preprint arXiv:2406.14643, 2024.arxiv.org/abs/2406.14643
  37. [37]P. Xia, S. Han, S. Qiu, Y. Zhou, Z. Wang, W. Zheng, Z. Chen, C. Cui, M. Ding, L. Li et al., “Mme: Massive multimodal interleaved comprehension benchmark for large vision-language models,” arXiv preprint arXiv:2410.10139, 2024.arxiv.org/abs/2410.10139

Paper details

vlnc:W_xvcdk37e
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 4

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents