betaFeedback

WorksComputer Science

On the Limits of Scale: Inverse Scaling in Vision-Language Models

Abstract

Scaling laws suggest that increasing model size, training data, and compute generally yields monotonic performance improvements. While this often holds at the aggregate level, we present systematic evidence that vision-language models (VLMs) exhibit inverse scaling at the subtask level: specific question categories on which larger models perform worse than smaller counterparts within the same model family. We evaluate 19 models from four distinct model families across eight multimodal benchmarks (nine evaluation splits), answering 29,386 questions per model, 558,334 model-question evaluations in total. Using per-question pairwise analysis across all model pairs in each family, we identify 375 subtask-model-pair combinations exhibiting inverse scaling, spanning 4,342 unique queries where a smaller model answers correctly but a larger model fails, and we find that these patterns replicate across independently developed architectures. We identify 13 subtask-dataset pairs that show inverse scaling across multiple families, consistent with the phenomenon being task-driven rather than model-specific. The most affected domains in our study include multi-image spatial reasoning, visual perception, and university-level STEM knowledge, with accuracy gaps exceeding 20 percentage points in several cases, while diagram understanding appears comparatively resistant. These findings challenge the assumption that increasing model size uniformly improves multimodal capabilities and suggest that model selection should be task-aware.

Introduction

Neural scaling laws suggest that model performance generally improves with increasing model size, data, and compute ([13, 11]). This empirical trend has guided the development of foundation models and reinforced a simple working assumption: larger models should perform better. However, recent work shows that this assumption can fail. In language models, McKenzie et al. [19] identify inverse scaling tasks on which performance worsens as model scale increases, while Wei et al. [24] show that some of these trends are better described as non-monotonic or U-shaped behaviour rather than uniformly decreasing. More recently, Gema et al. [10] demonstrate an analogous phenomenon for reasoning models, where increasing test-time compute can also reduce accuracy. In multimodal reasoning, concurrent work by Bae et al. [3] shows that adding visual distractors can reduce VLMs’ accuracy and induce inverse-scaling-like behaviour, though in a constructed distractor setting rather than in standard benchmark evaluation.

Whether inverse scaling also arises in VLMs under standard benchmark evaluation, without artificially introducing distractors, remains an open question. VLMs inherit the same scaling paradigm as language models, with model families typically released in sizes ranging from small to very large. At the same time, VLMs must jointly process visual and textual information, creating additional opportunities for failure ([2, 20]). A larger model may achieve higher overall benchmark accuracy while becoming less accurate on particular forms of visual reasoning. If so, standard aggregate evaluation may obscure regressions that are scientifically important and practically relevant for deployment. Such regressions are already visible, though unremarked, in providers’ own reports: in the Qwen3-VL technical report [4], the 8B instruct model scores above both the 32B and the 30B-A3B on BLINK (69.1 vs. 67.3 and 67.7; Tables 3–4), the 32B scores above the 235B-A22B on VLMs-Are-Blind (87.0 vs. 80.4; Tables 2–3), and CountBench accuracy declines monotonically from 2B to 4B to 8B (88.4, 84.9, 80.5; Table 4). Aggregate reporting records these numbers without identifying which capabilities regress or on which questions.

Figure 1. Scaling curves for four cross-family subtasks. Filled circles: dense models; open diamonds: MoE variants. All show non-monotonic behaviour across families.

In this work, we conduct a systematic study of subtask-level inverse scaling across model sizes in vision-language models on standard multimodal benchmarks, where subtasks are the category labels provided by each benchmark (e.g., “Jigsaw” in BLINK [9] or “Organic Chemistry” in MMMU [27]; see Table 1 for the full list of benchmarks). Our analysis is structured around within-family comparisons: we compare models released under the same family name, which generally share a high-level architecture and training recipe but may differ across sizes in language backbone, vision encoder, active parameter count, training data, alignment procedure, or inference-time visual processing. This design reduces, but does not eliminate, confounds relative to comparisons across independently developed model families. We evaluate 19 models from four distinct model families (Qwen3-VL [4], InternVL3.5 [23], Gemma 3 [22], and Molmo 2 [6]) spanning roughly 1B to 235B total parameters and including both dense and Mixture-of-Experts (MoE) variants.

Across eight established multimodal benchmarks used for evaluating frontier VLMs (nine evaluation splits), this yields 29,386 questions per model. Rather than relying solely on aggregate accuracy, we introduce a per-question flip analysis that compares the behaviour of paired models on the same evaluation items, measuring how often a smaller model answers correctly when a larger model fails, and vice versa. This paired analysis quantifies inverse scaling at the subtask level, revealing patterns that are often hidden in benchmark-level summaries.

Our results show that inverse scaling is widespread at the subtask level (Figure 1). Across all pairwise comparisons within each family, we identify 375 subtask–model-pair cases exhibiting inverse scaling, spanning all four model families and appearing in every benchmark we evaluate. These patterns are not confined to a single architecture or training recipe: 13 subtask–dataset pairs exhibit inverse scaling across multiple families (Figure 2), consistent with a task-driven rather than purely model-specific phenomenon. The strongest effects arise in multi-image spatial reasoning (BLINK), multi-task visual understanding (MMT-Bench), and university-level STEM problem solving (MMMU), whereas diagram understanding (AI2D) appears comparatively resistant. Crucially, these findings persist when restricting to dense-only model pairs (193 cases, 8 cross-family subtasks), confirming that the phenomenon is not driven solely by MoE models.

Figure 2. Cross-family inverse scaling across subtasks. Each bar shows the number of pairwise inverse-scaling cases for a given model family on that subtask; the annotation shows the count and the strongest case’s Net flip. Right column: total cases and number of families. Only the 12 subtasks with the most cases are shown; all 13 appear in Table 36.

Appendix M presents qualitative examples illustrating the range of failure modes, from misidentifying clearly visible objects to selecting plausible but incorrect options. These findings challenge the assumption that increasing scale uniformly improves multimodal capability. More broadly, they suggest that model choice should depend not only on aggregate benchmark performance, but also on the specific capabilities required by the target application domain [15].

Related Work

Scaling laws. Scaling-law work shows that language-model loss and many downstream capabilities improve predictably as parameters, data, and compute increase [13, 11]. Similar empirical regularities have also been reported in vision-language pretraining: Cherti et al. [5] find power-law behaviour for CLIP-style contrastive models across zero-shot classification, retrieval, linear probing, and fine-tuning. Similarly, Aghajanyan et al. [1] reports relevant scaling laws for multimodal models. While these works focus on aggregate task performance, we analyse whether monotonic improvements persist at the finer-grained level of benchmark subtasks within a VLM family, which we assume correlates more directly with specific multimodal skills.

Inverse scaling in LLMs and VLMs. The Inverse Scaling Prize [19] introduced curated text-only tasks on which larger language models performed worse and identified causes such as strong priors, unwanted imitation, and distractor tasks. Wei et al. [24] expanded the model range and found that many of these curves become U-shaped rather than monotonically decreasing, and that one-shot examples or chain-of-thought prompting can mitigate some failures. Along a different axis, Gema et al. [10] construct tasks on which extending the reasoning length of large reasoning models degrades accuracy, yielding inverse scaling in test-time compute rather than model size. Concurrently, Bae et al. [3] study how visual and textual distractors affect multimodal reasoning, finding that visual distractors reduce VLM accuracy without lengthening reasoning chains, a mechanism distinct from text-only distractor effects.

Our work complements these efforts by examining open VLM families on standard multimodal benchmarks rather than on specially constructed tasks and by detecting inverse scaling via per-question pairwise flips within existing subtasks. In line with McKenzie et al. [19], our findings are observational evidence from publicly available VLM checkpoints.

Methodology: Investigating Inverse Scaling in VLMs

Problem Formulation. Let F={f1,f2,…,fK}\mathcal{F} = \{f_1, f_2, \ldots, f_K\} denote a family of VLMs ordered by size, measured in terms of the number of parameters, where f1f_1 is the smallest and fKf_K the largest.2 Let DD be a question-answering (QA) benchmark partitioned into subtasks {S1,S2,…,ST}\{S_1, S_2, \ldots, S_T\}, where each subtask is a set of evaluation questions of the same type (e.g., a subject area or reasoning category), and let acc⁡(fk,St)\operatorname{acc}(f_k, S_t) denote the accuracy of model fkf_k on subtask StS_t. Under a monotonic scaling assumption, one would expect acc⁡(fi,St)\operatorname{acc}(f_i, S_t) to be non-decreasing in kk. We say subtask StS_t exhibits inverse scaling for the pair (fi,fj)(f_i, f_j) with i<ji < j iff acc⁡(fi,St)>acc⁡(fj,St)\operatorname{acc}(f_i, S_t) > \operatorname{acc}(f_j, S_t).

Per-Question Flip Analysis. To quantify inverse scaling at a finer granularity than subtask-level accuracy, we compare individual question outcomes across model pairs. For a pair of models (fi,fj)(f_i, f_j) with i<ji < j and a subtask StS_t, let Q={q1,…,qn}⊆StQ = \{q_1, \ldots, q_n\} \subseteq S_t denote the questions in that subtask. For each question q∈Qq \in Q, let ci(q),cj(q)∈{0,1}c_i(q), c_j(q) \in\{0, 1\} be correctness indicators, where 1 means correct, and 0 means incorrect. We then define:

FlipInv⁡=∑q∈Q1[ci(q)=1∧cj(q)=0](smaller correct, larger wrong)(1)\operatorname{FlipInv} = \sum_{q \in Q} \mathbf{1}[c_i(q) = 1 \land c_j(q) = 0] \qquad\text{(smaller correct, larger wrong)} \tag*{(1)}
FlipNorm⁡=∑q∈Q1[ci(q)=0∧cj(q)=1](larger correct, smaller wrong)(2)\operatorname{FlipNorm} = \sum_{q \in Q} \mathbf{1}[c_i(q) = 0 \land c_j(q) = 1] \qquad\text{(larger correct, smaller wrong)} \tag*{(2)}
Net⁡=FlipInv⁡−FlipNorm⁡(net advantage for the smaller model)(3)\operatorname{Net} = \operatorname{FlipInv} - \operatorname{FlipNorm} \qquad\text{(net advantage for the smaller model)} \tag*{(3)}

Here, FlipInv counts disagreements that favor the smaller model fif_i, FlipNorm counts disagreements that favor the larger model fjf_j, and Net is the difference between the two. A positive Net means that, among discordant questions, the smaller model is correct more often than the larger one.

We apply this analysis to all within-family model pairs. A family with KK models therefore contributes (K2)\binom{K}{2} pairwise comparisons for each subtask.

FamilyModels
Qwen3-VL2B, 4B, 8B, 30B-A3B*(3B), 32B, 235B-A22B*(22B)
InternVL3.51B, 2B, 4B, 8B, 14B, 30B-A3B*(3B), 38B, 241B-A28B*(28B)
Gemma 34B, 12B, 27B
Molmo 24B, 8B

Table 1. Left: Model families evaluated; *MoE models with active parameters in parentheses. Right: Evaluation benchmarks; subtask counts reflect the category granularity used in our analysis. †ScienceQA is filtered to the 2,017 questions that include an image, excluding text-only questions.

DatasetQwen3-VLInternVLGemma 3Molmo 2Total
MMT-Bench10206137
MMMU (dev_val+test)13408667
ScienceQA4010014
MME3140219
AI2D01012
MathVista01001
BLINK13232846
MMBench05027
Total431042620193

Table 4. Dense-only pairwise inverse scaling cases by family and dataset (MoE models excluded). Gemma 3 and Molmo 2 contain no MoE models, so their counts equal those of the full analysis.

DatasetSplitNSub.Type
MMT-BenchVAL3,12732MCQ
MMMUdev_val1,050132MCQ+open
MMMUtest10,500177MCQ+open
ScienceQA†test2,01762MCQ
MMEtest2,37414Yes/No
AI2Dtest3,08815MCQ
MathVistatestmini1,0005MCQ+free
BLINKval1,90114MCQ
MMBenchDEV_EN4,32920MCQ

Table 1 (PDF p. 4). Left: Model families evaluated; *MoE models with active parameters in parentheses. Right: Evaluation benchmarks; subtask counts reflect the category granularity used in our analysis. †ScienceQA is filtered to the 2,017 questions that include an image, excluding text-only questions.

Experimental Setup

Models. We evaluate 19 models across 4 open-weight families3: Qwen3-VL Bai et al., [4], InternVL3.5 Wang et al., [23], Gemma 3 Team et al., [22], and Molmo 2 Clark et al., [6], summarized in Table 1 (left). Each family provides a range of model sizes, including both dense architectures and Mixture-of-Experts (MoE) variants. The families were selected to maximize architectural diversity with at least two sizes per family.

Datasets. We evaluate on 9 benchmark splits covering diverse multimodal capabilities, summarized in Table 1 (right). The suite covers perception and cognition (MME Fu et al., [8]), diagram understanding (AI2D Kembhavi et al., [14]), K–12 science (ScienceQA Lu et al., [17]), multi-image visual perception (BLINK Fu et al., [9]), general multimodal reasoning (MMT-Bench Ying et al., [26] and MMBench Liu et al., [16]), university-level knowledge (MMMU Yue et al., [27]), and mathematical reasoning (MathVista Lu et al., [18]). Because these benchmarks are widely used to compare released VLMs, subtask-level regressions are practically relevant to model selection. In total, we perform 171 inference runs (19 models ×\times 9 evaluation splits), evaluating 29,386 unique questions per model (558,334 model–question evaluations). Full model identifiers, compute details, and API configurations are provided in Appendix § 11.

Evaluation Protocol

Inference. All models are evaluated zero-shot with direct prompting and greedy decoding, with thinking modes disabled to separate model scale from test-time compute Gema et al., [10]; MCQ prompts ask for the option letter and open-ended prompts for a concise answer (settings in Appendix § 11, prompts in Appendix § 17). Answers are verified as in VLMEvalKit Duan et al., [7]: deterministic letter extraction and exact or numeric matching, with GPT-5.4 as a multimodal fallback adjudicator when these rules are inconclusive (0.4% of responses).

Case selection. We report a pairwise inverse-scaling case when the subtask contains at least n≥20n \ge20 shared questions and the net flip advantage is Net≥5\mathrm{Net} \ge5. The first threshold excludes subtasks too small for meaningful comparison; the second excludes cases where the asymmetry is trivially small. The findings are robust to stricter thresholds: 132 cases across 44 subtask–dataset pairs remain at Net≥10\mathrm{Net} \ge10, with 4 cross-family subtasks still replicated (Appendix § 16).

Results

Aggregate Scaling Behavior

At the aggregate level, the largest model in each family achieves the highest or near-highest accuracy on most benchmarks (full results in Appendix § 12). However, several clear exceptions stand out. On BLINK, Qwen 30B-A3B (62.4%) substantially underperforms smaller dense models such as 4B (66.4%), and Molmo 4B (63.5%) outperforms 8B (59.3%). Molmo 4B also outperforms 8B on MMBench (88.3% vs. 87.9%).

Subtask-Level Inverse Scaling

While aggregate performance scales mostly monotonically, the subtask level tells a different story. Inverse scaling appears across all four families and every benchmark we evaluate.

Breadth across families and benchmarks. Table 2 summarizes prevalence, not model quality (aggregate accuracies: Appendix § 12; strongest cases: Table 3; all cases: Appendix § 13). InternVL contributes the most cases (207), reflecting its eight models and 6,216 eligible comparisons; normalized, Qwen3-VL (3.7%), InternVL (3.3%), and Gemma 3 (3.9%) have similar rates, and Molmo 2 a higher one (9.0%, from a single pair). Raw counts are highest for MMMU (115), BLINK (95), and MMT-Bench (87) and lowest for AI2D (9) and MathVista (3); normalized, BLINK leads (14.4%), followed by MMT-Bench (5.8%), MME (5.3%), and MMBench (5.0%), with MMMU at 2.3% and ScienceQA at 1.2%. Because Net = 5 corresponds to an accuracy gap of 0.45pp at n=1,109n = 1{,}109 but 25pp at n=20n = 20, we read counts and rates descriptively, not as intrinsic benchmark susceptibility. The 375 cases span 115 subtask–dataset pairs and 171 family–subtask combinations. Adjacent-size comparisons alone (e.g., 4B→8B, 8B→14B) yield 197 cases across 107 subtask–dataset pairs, including 12 cross-family subtasks, so the patterns include local regressions, not only distant sizes. Of the 171 family–subtask pairs, 87% have non-monotonic accuracy curves rather than consistent degradation, mirroring U-shaped trends in language-model inverse scaling (Wei et al., [24]). The findings are qualitatively stable across Net thresholds (Appendix § 16).

DatasetQwen3-VLInternVLGemma 3Molmo 2Total
MMT-Bench31/48049/8966/961/3287/1504
MMMU (dev–val+test)34/162067/30248/3246/108115/5076
ScienceQA7/4500/84010/900/3017/1410
MME10/21023/3920/422/1435/658
AI2D0/1958/3640/391/139/611
MathVista1/752/1400/150/53/235
BLINK37/21048/3922/428/1495/658
MMBench2/9010/1680/182/614/282
Total122/3330207/621626/66620/222375/10434

Table 2. Pairwise inverse scaling cases by family and dataset. Each cell reports the number of detected cases over all eligible model-pair–subtask comparisons (n≥20n \ge20 shared questions); a case is a unique (subtask, smaller model, larger model) tuple meeting the thresholds in Section 4.

#FamilyDatasetSubtaskSmallLargennFIFNNetAcc Δ\Delta
1Qwen3-VLBLINKJigsaw4B30B-A3B1504593624.0
2Qwen3-VLBLINKJigsaw4B32B15044113322.0
3Molmo 2BLINKJigsaw4B8B15043123120.7
4InternVLBLINKForensic Detection2B30B-A3B1323003022.7
5Qwen3-VLBLINKMulti-view Reasoning8B32B1333883022.6
6InternVLBLINKForensic Detection4B30B-A3B1322902922.0
7InternVLBLINKForensic Detection14B30B-A3B1322902922.0
8Qwen3-VLBLINKMulti-view Reasoning2B32B1333782921.8
9InternVLMMELandmark1B2B4005526297.2
10Gemma 3ScienceQABasic Economic Principles4B27B713132839.4

Table 3. Top 10 pairwise cases ranked by Net (FlipInv − FlipNorm); Acc Δ\Delta in pp.

Statistical significance. As a descriptive measure of paired asymmetry, we report one-sided McNemar exact pp-values for each candidate case, i.e., for each subtask–model-pair comparison, computed over the subtask’s shared questions. Because these tests are performed on many overlapping model-pair/subtask combinations and because cases are selected using the same observed data, these pp-values should be interpreted as exploratory rather than confirmatory. With this caveat, 133 of 375 cases (35.5%) have uncorrected p<0.05p < 0.05, rising from 10.4% of cases with Net 5–7 to 100% with Net ≥21\ge21; Appendix § 14 reports the Benjamini–Hochberg correction over the reported cases and over the full space of eligible comparisons from which they were selected. Cross-family replication provides complementary robustness evidence that the patterns are not confined to a single family (§ 5.4).

Strongest cases. Table 3 presents the strongest individual cases. BLINK dominates: Jigsaw appears three times in the top 10 across the Qwen and Molmo families, led by Qwen 4B outperforming 30B-A3B by 24.0pp (Net = 36), and Forensic Detection three times for InternVL, where cases #4, #6, and #7 have FlipNorm = 0: no single question where the MoE model succeeds and the smaller model fails. The only non-BLINK entries are MME Landmark (InternVL 1B vs. 2B) and ScienceQA Basic Economic Principles, where Gemma 3 4B outperforms 27B by 39.4pp. Accuracy gaps range from 7.2pp to 39.4pp among the top 10. The six BLINK cases among them keep a positive Net under every deterministic reordering of their answer options (Appendix § 20).

Per-Family Analysis

Qwen3-VL (122 cases). The most striking inverse scaling appears on BLINK, where the MoE model 30B-A3B consistently underperforms dense models. On Jigsaw, the 4B model achieves 84.0% accuracy compared to 30B-A3B’s 60.0% (+24.0pp, Net = 36). On Multi-view Reasoning, multiple smaller models outperform the 32B model by over 20pp.

InternVL3.5 (207 cases). InternVL has the largest raw count (207), partly reflecting its larger family, while its normalized rate (3.3%) is comparable to Qwen (3.7%) and Gemma (3.9%). Key transitions include 8B → 14B (where 14B degrades on MMMU subtasks, MME categories, and BLINK), and 38B → 241B-A28B (the MoE model fails to improve on many specialized subtasks). On BLINK Forensic Detection, four dense models (2B–14B) outperform 30B-A3B with FlipNorm = 0.

Gemma 3 (26 cases). Despite having only 3 models (and thus (32)\binom{3}{2} = 3 pairs), Gemma 3 shows 26 cases. ScienceQA contributes 10 cases, including the striking “Particle motion and energy” where 12B achieves 95.1% vs. 27B’s 51.2% (+43.9pp, Net = 18).

Molmo 2 (20 cases). With only 2 models (4B and 8B), Molmo 2 shows 20 cases. BLINK contributes 8 cases, with Jigsaw showing Net = 31 (4B: 79.3% vs. 8B: 58.7%). The 4B model outperforms 8B at the aggregate level on both BLINK (63.5% vs. 59.3%) and MMBench (88.3% vs. 87.9%).

Cross-Family Patterns

Cross-family replication provides additional evidence that inverse scaling is systematic. We first identify pairwise cases using the question-level criteria in § 4. Separately, for cross-family summarization, we highlight subtask–dataset pairs that recur in at least three of the four families. This criterion is a reporting choice for emphasizing the strongest replications, not part of the underlying pairwise case definition. Table 3 presents the 13 subtask–dataset pairs meeting this criterion, organized by dataset. An additional 29 subtasks appear in exactly two families (Appendix § 15).

BLINK dominates cross-family inverse scaling. Five of 14 BLINK subtasks show cross-family inverse scaling, with Multi-view Reasoning appearing in all four families (Qwen, InternVL, Gemma 3, Molmo 2). These subtasks involve spatial reasoning across multiple images. BLINK contributes high Max Net values (up to 36), with multi-image spatial reasoning showing the strongest inverse scaling effects. Figure 1 shows representative scaling curves for four cross-family subtasks, illustrating the non-monotonic accuracy patterns across families.

MMBench coarse perception. Coarse perception shows inverse scaling in Qwen, InternVL, and Molmo 2 (Max Net = 21).

MMMU STEM subjects. Organic Chemistry (Gemma 3, InternVL, Molmo 2), Investment (Gemma 3, InternVL, Qwen), and Thermodynamics (InternVL, Molmo 2, Qwen) show cross-family inverse scaling on university-level problems. These require specialized domain knowledge combined with visual understanding of diagrams and figures.

MMT-Bench and MME. Autonomous Driving on MMT-Bench (3 families, Max Net = 23) and Scene on MME (3 families, Max Net = 16) also show cross-family replication.

Other Findings

MoE models frequently appear in inverse scaling pairs. Mixture-of-Experts models frequently appear as the larger model in inverse scaling pairs. Qwen 30B-A3B (3B active parameters) underperforms the dense 32B model on several BLINK subtasks. InternVL 241B-A28B (28B active) fails to improve over the dense 38B on AI2D (both 87.1%) and degrades on several BLINK subtasks. However, MoE and dense models differ in active parameter count, training recipe, and alignment procedure, so we cannot isolate the contribution of expert routing from these confounds.

The findings above show that inverse scaling is widespread and cross-family, but is it simply the largest model underperforming, or a more complex relationship between scale and capability? Three observations suggest the latter.

Scaling trajectories are non-monotonic. Rather than a single large-model drop, many subtasks exhibit multiple peaks and valleys across the size range. For example, InternVL’s performance on “Mechanics of Materials” on MMMU shows: 1B = 27.6%, 2B = 48.3%, 4B = 44.8%, 8B = 31.0%, 14B = 51.7%, 30B-A3B = 58.6%, 38B = 62.1%, 241B-A28B = 27.6%. This suggests that different model sizes have qualitatively different strengths rather than uniformly improving capabilities.

In extreme cases, the smallest model wins. This non-monotonicity can be so pronounced that the smallest model in a family achieves the highest subtask accuracy. Qwen 2B is the best on MMMU test AI Content Detection (65.0% vs. 235B’s 45.0%). Molmo 4B outperforms 8B on all BLINK subtasks showing inverse scaling. These cases show that inverse scaling is not limited to marginal regressions at the top of the size range.

Inverse scaling survives a twofold increase in model size. Restricting to pairs in which the larger model has at least twice the total parameters of the smaller leaves 326 of the 375 cases (3.3% of 9,768 comparisons). Using active parameters, which matters for the MoE models, leaves 239 cases (2.9% of 8,214 comparisons). Both restrictions keep 11 of the 13 cross-family subtasks (Appendix § 19). At the largest size gap in each family (smallest vs. largest model, on every eligible subtask), the smaller model is ahead by Net ≥5\ge5 in 9 of 666 comparisons (Qwen3-VL 2B vs. 235B-A22B: 4; InternVL 1B vs. 241B-A28B: 0; Gemma 3 4B vs. 27B: 5); Molmo 2 has a single pair. Cases therefore persist at twofold size gaps but are limited across the full size range.

The pattern persists without MoE models. MoE architectures, with fewer active parameters and distinct routing, could drive the phenomenon. We therefore repeat the pairwise analysis on dense pairs only, excluding all MoE variants (Qwen 30B-A3B and 235B-A22B; InternVL 30B-A3B and 241B-A28B). This still yields 193 pairwise cases (Table 4 and Table 9), with 8 subtask–dataset pairs replicating across at least three families, including BLINK Multi-view Reasoning, which appears in all four. Inverse scaling is therefore not solely an MoE artifact.

DatasetSubtaskFamiliesMax Net
BLINKMulti-view Reasoning430
BLINKJigsaw333
MMMU testOrganic Chemistry315
MMEScene314
MMMU testInvestment313
BLINKObject Localization311
MMT-BenchRelation Reasoning39
MMT-BenchAutonomous Driving38

Table 9. Dense-only cross-family replication: subtask–dataset pairs with inverse scaling in at least three of the four families using only dense model pairs.

Error Taxonomy

To understand why larger models fail on inverse-scaling questions, we use GPT-5.4 as a multimodal annotator to assign failure categories to inverse-scaling flip examples. We sample up to 10 flip questions per pairwise case with stratified dataset quotas and a per-family cap of 600, yielding 1,300 questions spanning eight datasets and all 4 families after filtering degenerate responses (empty or single non-letter characters). Each question, along with its associated image(s), is independently classified (temperature = 0) into one of three failure categories. The annotator prompt is blinded to model identities (models are labeled “Model A” and “Model B”) to reduce bias. As a secondary check, we also run Claude Opus 4.6 with the same prompt; the two judges agree on 90.8% of questions (Cohen’s κ=0.30\kappa= 0.30). Full prompts and API model identifiers are provided in Appendix § 17.

To validate the LLM annotations, three authors independently labeled a stratified subsample of 127 questions using the same three categories. Human inter-annotator agreement was substantial (Fleiss κ=0.65\kappa= 0.65, pairwise agreement 87–92%). The human majority vote classified 84.3% as genuine errors and 15.7% as bad annotation or ambiguous, confirming that humans identify more annotation issues than either LLM judge (GPT: 2.6%, Claude: 10.5%). The majority vote agreed with GPT on 88.2% and with Claude on 84.3% of questions (full breakdown in Appendix § 18).

Of the 1,300 sampled flips, GPT-5.4 classified 97.4% (1,266) as genuine errors, 2.6% (34) as bad annotation or ambiguous, and none as format mismatch. All five raters (three human, two LLM) classify the majority as genuine errors, though with different thresholds: GPT-5.4 labels 97.4% as genuine, while the human majority vote applies a stricter standard, classifying 84.3% as genuine and flagging 15.7% as bad annotation or ambiguous. We use the human rate as our conservative estimate: at least 84% of inverse scaling flips reflect real model failures. The human-flagged bad-annotation cases are concentrated in MME (known annotation issues) and knowledge-heavy benchmarks where ground-truth answers are debatable; on visual reasoning benchmarks (BLINK, ScienceQA), both humans and LLMs agree that nearly all flips are genuine errors.

Discussion

Why does inverse scaling occur in VLMs? While our study does not establish causal explanations, several hypotheses fit the observed patterns. First, localized capability regression: on some perception-centric subtasks (BLINK, MME), larger released variants are less accurate than smaller ones, even though the larger model is ahead in 74.8% of eligible comparisons. These regressions affect particular skills, not whole benchmarks, and our data cannot attribute them to parameter count alone. Second, sensitivity to distractors: on knowledge-heavy benchmarks (MMMU, MMT-Bench), larger models may be more susceptible to plausible but incorrect options, consistent with the distractor memorization hypothesis of McKenzie et al. [19]. Third, MoE-specific factors: MoE variants frequently appear in inverse scaling pairs, though routing, lower active parameter counts, and training differences cannot be separated in our observational data. Fourth, training data composition: larger models typically see more training data, whose composition may introduce biases that hurt specific subtasks.

Can chain-of-thought prompting recover the errors? To test whether inverse scaling errors reflect shallow reasoning deficits, we re-evaluated both models of every case under chain-of-thought (CoT) prompting on all shared questions (66,698 matched question instances across 375 cases). The CoT prompt adds a step-by-step instruction to the direct prompt (Appendix § 22); decoding is unchanged. CoT attenuates inverse scaling but does not remove it: the median Net falls from +8 to 0, yet 96 of 375 cases (25.6%) retain Net ≥5\ge5, 101 reverse, and the larger model recovers 43.5% of the questions it had lost under direct prompting. Persistence varies by benchmark, from 48.6% of MME and 42.1% of BLINK cases to 7.6% on MMMU test (full results in Appendix § 22). Because the smaller model is also re-evaluated, the persisting cases cannot be attributed to the smaller model reasoning where the larger one does not. CoT thus corrects fewer than half of the individual errors and leaves a quarter of the cases intact, and part of the reduction in Net comes from the smaller model losing accuracy under CoT (in 251 of 375 cases) rather than from the larger model gaining. It also remains an open question whether CoT recovery reflects genuine improvements in reasoning, as recent work suggests that intermediate steps can sometimes mask underlying vulnerabilities (Xu et al., [25]; Kambhampati et al., [12]; Shojaee et al., [21]).

Implications for model selection. Our results have practical implications. The common practice of selecting the largest available model is suboptimal for some task categories. On a minority of evaluated subtask–model-pair comparisons, concentrated in multi-image spatial reasoning, visual perception, and specific STEM subtasks, smaller models outperform their larger counterparts. Practitioners should evaluate candidate models on their downstream tasks, not on aggregate scores.

Limitations. Our study has several limitations. First, we evaluate only open-weight models, since the unknown architectures and parameter counts of proprietary models make scaling analysis unverifiable. Second, the flip analysis weights all questions equally, regardless of difficulty. Third, cross-family replication reports empirical evidence and does not establish any causal mechanism. Fourth, models released under one family name may differ in vision encoder (Appendix § 21), training data, alignment, and active parameters, so our results describe subtask-level regressions among released variants rather than causal effects of parameter count alone; a controlled study training VLMs of several sizes under one fully released recipe would isolate parameter count and make such analyses reproducible, which we leave to future work. Fifth, larger checkpoints may have seen more benchmark data, and we cannot rule out size-dependent contamination. Sixth, results depend on option order; the strongest BLINK cases survive reorderings (Appendix § 20), but paraphrases are untested.

Conclusion

To our knowledge, this is the first systematic study of inverse scaling in vision-language models. We evaluate 19 models across 4 families and eight benchmarks. Through per-question pairwise analysis, we identify 375 cases in which smaller models outperform larger ones, with 13 subtask types showing this pattern across multiple families. Our findings demonstrate that VLM performance does not scale monotonically at the subtask level, with multi-image spatial reasoning, visual perception, and specialized STEM knowledge most affected. These results motivate work on the mechanisms of inverse scaling and suggest task-aware model deployment. Finally, this work complements recent evidence of deficiencies in current VLMs [e.g. 2], highlighting the need for robust multimodal models that leverage their full capacity across modalities.

AI USE STATEMENT

In this work, we used generative AI tools as a writing aid to improve fluency and check grammar, and for coding assistance. We have reviewed all AI-assisted work; AI-assisted code was verified by the authors. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI.

ACKNOWLEDGMENTS

Rohit Saxena was supported by the Engineering and Physical Sciences Research Council (EPSRC) through the AI Hub in Generative Models (grant number EP/Y028805/1). Pasquale Minervini was partially funded by ELIAI (The Edinburgh Laboratory for Integrated Artificial Intelligence), EPSRC (grant no. EP/W002876/1), and a donation from Accenture LLP. This work was supported by the Edinburgh International Data Facility (EIDF) and the Data-Driven Innovation Programme at the University of Edinburgh.

REPRODUCIBILITY STATEMENT

All 19 evaluated models are open-weight; their exact HuggingFace identifiers are listed in Appendix § 11 (Table 5), which also states the compute used and the API models that support evaluation. Generation settings, including those of the chain-of-thought control, are given in Appendix § 11; the answer-verification pipeline and the case-selection thresholds are given in § 4.1. The exact prompts for every benchmark and the annotation prompts for the error-taxonomy judges are reproduced in Appendix § 17. Benchmark splits and sizes are listed in Table 1.

FamilySizeHuggingFace Identifier
Qwen3-VL2BQwen/Qwen3-VL-2B-Instruct
Qwen3-VL4BQwen/Qwen3-VL-4B-Instruct
Qwen3-VL8BQwen/Qwen3-VL-8B-Instruct
Qwen3-VL30B-A3BQwen/Qwen3-VL-30B-A3B-Instruct
Qwen3-VL32BQwen/Qwen3-VL-32B-Instruct
Qwen3-VL235B-A22BQwen/Qwen3-VL-235B-A22B-Instruct
InternVL3.51BOpenGVLab/InternVL3.5-1B-Instruct
InternVL3.52BOpenGVLab/InternVL3.5-2B-Instruct
InternVL3.54BOpenGVLab/InternVL3.5-4B-Instruct
InternVL3.58BOpenGVLab/InternVL3.5-8B-Instruct
InternVL3.514BOpenGVLab/InternVL3.5-14B-Instruct
InternVL3.530B-A3BOpenGVLab/InternVL3.5-30B-A3B-Instruct
InternVL3.538BOpenGVLab/InternVL3.5-38B-Instruct
InternVL3.5241B-A28BOpenGVLab/InternVL3.5-241B-A28B-Instruct
Gemma 34Bgoogle/gemma-3-4b-it
Gemma 312Bgoogle/gemma-3-12b-it
Gemma 327Bgoogle/gemma-3-27b-it
Molmo 24Ballenai/Molmo2-4B
Molmo 28Ballenai/Molmo2-8B

Table 5. HuggingFace model identifiers for all 19 evaluated models.

Additional Experiment Details

Model identifiers. All models are open-weight and publicly available on HuggingFace. Table 5 lists the exact HuggingFace model identifiers used for inference.

Compute. All inference was performed on NVIDIA H100 GPUs with no quantization. The total evaluation comprises 171 inference runs (19 models ×\times 9 evaluation splits), i.e., 558,334 model–question evaluations.

API models. Two proprietary models are used for evaluation support, not as evaluation targets:

  • GPT-5.4 (gpt-5.4 via OpenAI API): used as the multimodal fallback adjudicator for answer verification (0.4% of responses) and as the primary error taxonomy annotator (§ 5.6).

  • Claude Opus 4.6 (claude-opus-4-6 via Anthropic API): used as the secondary error taxonomy annotator for inter-judge agreement analysis.

Generation settings. All models are evaluated zero-shot with direct prompting and greedy decoding (do_sample=False, max_new_tokens=2048), with thinking modes disabled. For multiple-choice questions the prompt instructs the model to respond with only the letter of the correct option; for open-ended questions (MMMU, MathVista) it asks for a concise answer. The chain-of-thought control (Appendix § 22) also uses greedy decoding, with max_new_tokens=4096. Prompts are listed in Appendix § 17.

Answer verification. We first apply deterministic answer extraction and matching: answer-letter extraction for multiple-choice questions and exact or numeric matching for open-ended responses. When these rules are inconclusive, GPT-5.4 (OpenAI API, model identifier gpt-5.4) adjudicates as a multimodal fallback, receiving the image, question, options, and model response (prompt in Appendix § 17). Across all models and datasets, 0.4% of responses require adjudication: 4.1% on BLINK, 2.9% on MathVista, under 0.05% on AI2D, MMT-Bench, ScienceQA, and MMMU, and none on MME and MMBench.

Aggregate Accuracy Tables

Dataset2B4B8B30B-A3B32B235B-A22B
MMT-Bench60.267.368.569.771.773.3
MMMU dev_val44.051.053.159.661.767.3
MMMU test41.648.751.056.057.064.2
ScienceQA87.392.694.495.796.298.0
MME82.587.289.690.490.991.6
AI2D76.683.485.187.387.989.5
MathVista57.270.973.674.579.481.5
BLINK54.766.465.962.466.768.8
MMBench84.188.689.891.791.792.5

Table 6. Overall accuracy (%) for Qwen3-VL. Bold = best in family.

Dataset1B2B4B8B14B30B-A3B38B241B-A28B
MMT-Bench53.158.965.267.668.467.371.972.6
MMMU dev_val40.447.457.157.059.361.663.465.2
MMMU test37.644.050.352.754.055.457.663.2
ScienceQA84.191.797.397.598.399.298.899.5
MME77.779.284.088.385.987.688.790.0
AI2D70.578.082.483.384.586.487.187.1
MathVista48.461.767.269.870.772.974.976.7
BLINK39.944.546.747.449.548.355.357.9
MMBench77.983.886.987.888.789.591.291.7

Table 7. Overall accuracy (%) for InternVL3.5. Bold = best in family.

DatasetGemma 3 4BGemma 3 12BGemma 3 27BMolmo 2 4BMolmo 2 8B
MMT-Bench54.360.161.564.066.5
MMMU dev_val40.747.954.850.552.9
MMMU test39.347.450.646.348.0
ScienceQA78.582.282.797.898.1
MME79.183.485.376.879.3
AI2D73.581.583.685.785.9
MathVista48.057.163.057.964.2
BLINK43.653.757.463.5<u>59.3</u>
MMBench80.084.186.788.3<u>87.9</u>

Table 8. Overall accuracy (%) for Gemma 3 and Molmo 2. Bold = best in family. Underline = largest model is not best.

Full Pairwise Inverse Scaling Results

Tables below list all 375 pairwise cases identified by the flip analysis (§ 3): Net ≥5\ge5 on n≥20n \ge20 matched questions. Cases are sorted by Net within each dataset. Columns: Subtask: the dataset category; Small/Large: the model pair; nn: matched questions; Acc Δ\Delta: how many percentage points the smaller model’s accuracy exceeds the larger model’s; FI: FlipInv (smaller correct, larger wrong); FN: FlipNorm (reverse); Net: FI − FN.

Qwen3-VL (122 cases)

MMT-Bench (31).

SubtaskSmallLargennAcc Δ\DeltaFIFNNet
autonomous_driving30B-A3B235B-A22B10023.023023
autonomous_driving2B235B-A22B10019.023419
autonomous_driving8B235B-A22B10017.018117
autonomous_driving32B235B-A22B10016.017116
autonomous_driving4B235B-A22B10014.016214
cross_image_matching8B30B-A3B6023.321714
medical_understanding4B8B10013.014113
medical_understanding4B32B10013.016313
medical_understanding4B235B-A22B10013.017413
medical_understanding4B30B-A3B10012.016412
visual_illusion30B-A3B235B-A22B7915.217512
anomaly_detection4B30B-A3B10011.0211011
anomaly_detection8B30B-A3B10010.018810
temporal_understanding8B30B-A3B9910.1231310
temporal_understanding4B30B-A3B999.121129
cross_image_matching4B30B-A3B6013.31468
SubtaskSmallLargennAcc Δ\DeltaFIFNNet
visual_illusion32B235B-A22B7910.11138
autonomous_driving30B-A3B32B1007.0817
visual_illusion2B235B-A22B797.622166
action_recognition4B8B1006.01046
temporal_understanding8B32B996.11486
autonomous_driving2B4B1005.01385
multiple_image_analysis2B8B4012.5945
visual_illusion2B8B796.323185
emotion4B8B1204.21055
relation_reasoning4B8B806.21055
action_recognition4B30B-A3B1005.01055
temporal_understanding4B32B995.116115
action_recognition4B235B-A22B1005.01495
pixel_level_perception8B30B-A3B1144.420155
gui_navigation30B-A3B32B806.21495

Table C.1.

MMMU test (34).

SubtaskSmallLargennAcc Δ\DeltaFIFNNet
Surveying and Mapping8B30B-A3B4124.1634617
Electrical Circuit30B-A3B32B12813.324717
Cost Management30B-A3B32B5824.117314
Data Structure and Algorithm4B30B-A3B9212.018711
Cost Management30B-A3B235B-A22B5819.015411
Data Structure and Algorithm8B30B-A3B929.81789
Engineering Graphics8B30B-A3B8910.119109
Physiology2B8B7311.01688
Calculus4B8B1615.023158
Financial Accounting30B-A3B32B1077.520128
Physiology2B4B739.61477
Control System4B30B-A3B4515.61037
Control System8B30B-A3B4515.61037
AI Content Detection2B4B2030.0936
Cost Management2B8B5810.3826
Artificial Intelligence4B8B4314.01156
Microbiology4B8B6010.01156
Operation Research4B8B4912.21046
Principles of Management4B8B4114.6936
Financial Management30B-A3B32B3616.7936
Operating System30B-A3B32B5411.11266
Structural Engineering30B-A3B32B946.41376
Graph Theory2B4B786.415105
Investment4B8B1194.219145
Surveying and Mapping4B30B-A3B4121.254495
Thermodynamics8B30B-A3B1633.122175
Engineering Graphics8B32B895.620155
Surveying and Mapping8B32B4121.246415
Civil Engineering8B235B-A22B4112.21165
Calculus30B-A3B32B1613.117125
Design History30B-A3B32B846.0725
Graph Theory30B-A3B32B786.41385
Human Geography30B-A3B32B4710.6725
Electrical Circuit30B-A3B235B-A22B1283.915105

Table 10.

ScienceQA (7).

SubtaskSmallLargennAcc Δ\DeltaFIFNNet
Velocity, acceleration, and forces30B-A3B32B6214.51239
The Americas: geography2B4B4517.81468
Velocity, acceleration, and forces8B32B6212.91138
Velocity, acceleration, and forces4B32B629.7826
The Americas: geography30B-A3B32B4513.3716
Ecosystems4B8B736.8615
Astronomy30B-A3B32B3613.9615

MME (10).

SubtaskSmallLargennAcc Δ\DeltaFIFNNet
scene2B235B-A22B4002.820911
scene32B235B-A22B4002.818711
scene30B-A3B235B-A22B4002.514410
scene4B235B-A22B4001.51486
celebrity8B32B3401.820146
scene8B235B-A22B4001.51376
position30B-A3B235B-A22B6010.0716
scene2B4B4001.21495
scene2B8B4001.218135
posters32B235B-A22B2941.7945

MathVista (1).

SubtaskSmallLargennAcc Δ\DeltaFIFNNet
math word problem8B30B-A3B1863.81587

BLINK (37).

SubtaskSmallLargennAcc Δ\DeltaFIFNNet
Jigsaw4B30B-A3B15024.045936
Jigsaw4B32B15022.0441133
Multi-view Reasoning8B32B13322.638830
Multi-view Reasoning2B32B13321.837829
Object Localization4B30B-A3B12218.0351322
Multi-view Reasoning4B32B13316.5371522
Forensic Detection4B32B13215.225520
Forensic Detection4B235B-A22B13215.227720
Jigsaw4B235B-A22B15013.3301020

continued MMBench (2).

SubtaskSmallLargennAcc Δ\DeltaFIFNNet
Multi-view Reasoning30B-A3B32B13315.0391920
Jigsaw4B8B15012.727819
IQ Test8B30B-A3B15011.325817
Jigsaw8B30B-A3B15011.3341717
Relative Reflectance32B235B-A22B13412.723617
Forensic Detection4B30B-A3B13212.123716
Forensic Detection8B32B13210.6251114
Jigsaw8B32B1509.3342014
Forensic Detection8B235B-A22B13210.6261214
Multi-view Reasoning8B235B-A22B13310.5321814
Multi-view Reasoning2B235B-A22B1339.8352213
Object Localization4B8B1229.0281711
Object Localization8B30B-A3B1229.0281711
Object Localization4B32B1228.2302010
Forensic Detection8B30B-A3B1327.6241410
Multi-view Reasoning8B30B-A3B1337.5221210
IQ Test8B32B1506.7201010
Multi-view Reasoning2B30B-A3B1336.823149
Relative Reflectance4B30B-A3B1346.024168
Functional Correspondence32B235B-A22B1306.225178
Multi-view Reasoning2B4B1335.319127
Jigsaw2B30B-A3B1504.739327
Relative Depth8B30B-A3B1245.61037
Forensic Detection4B8B1324.516106
Relative Depth4B30B-A3B1244.81376
Multi-view Reasoning4B235B-A22B1334.530246
Object Localization4B235B-A22B1224.922166
Relative Reflectance8B30B-A3B1344.529236

Table 11.

SubtaskSmallLargennAcc Δ\DeltaFIFNNet
coarse_perception30B-A3B32B11091.1382612
attribute_reasoning32B235B-A22B6990.919136

Table 12.

InternVL3.5 (207 cases)

MMT-Bench (49).

SubtaskSmallLargennAcc Δ\DeltaFIFNNet
cross_image_matching14B30B-A3B6031.721219
localization8B30B-A3B1699.5291316
localization14B30B-A3B1699.5311516
image_retrieval14B30B-A3B14010.022814
image_retrieval38B241B-A28B14010.019514
temporal_understanding38B241B-A28B9913.120713
visual_illusion1B2B7915.221912
cross_image_matching1B30B-A3B6020.021912
counting8B14B1408.6291712
image-to-image_translation8B14B4030.013112
cross_image_matching38B241B-A28B6020.016412
counting2B14B1407.9291811
cross_image_matching14B241B-A28B6018.316511
multiple_image_analysis1B2B4022.51239
relation_reasoning1B2B8011.21789
visual_illusion1B8B7911.423149
visual_illusion4B8B7911.41459
SubtaskSmallLargennAcc Δ\DeltaFIFNNet
image-to-image_translation8B30B-A3B4022.51239
threed38B241B-A28B4022.51129
cross_image_matching2B30B-A3B6013.318108
cross_image_matching4B30B-A3B6013.31688
image_retrieval8B30B-A3B1405.724168
visual_recognition14B30B-A3B6581.248408
doc_understanding8B30B-A3B1245.61367
ocr8B30B-A3B6011.7927
image-to-image_translation8B38B4017.51147
image-to-image_translation8B241B-A28B4017.51037
doc_understanding14B30B-A3B1245.61367
cross_image_matching1B8B6010.01596
visual_illusion1B30B-A3B797.619136
visual_illusion1B38B797.622166
counting2B4B1404.325196
visual_illusion4B30B-A3B797.61486
visual_illusion4B38B797.616106
cross_image_matching8B30B-A3B6010.01266
multiple_image_analysis1B8B4012.51165
autonomous_driving2B4B1005.0945
hallucination4B8B806.2835
counting4B14B1403.622175
ocr4B30B-A3B608.3835
ocr8B14B608.3725
counting8B30B-A3B1403.625205
counting8B38B1403.629245
visual_captioning8B38B1593.11165
image_evaluation_judgement14B30B-A3B4012.5725
image_retrieval14B241B-A28B1403.61495
temporal_understanding14B241B-A28B995.118135
emotion30B-A3B241B-A28B1204.21165
localization38B241B-A28B1693.025205

Table 13.

MMMU dev_val (10).

SubtaskSmallLargennAcc Δ\DeltaFIFNNet
Mechanics of Materials38B241B-A28B2934.510010
Mechanics of Materials30B-A3B241B-A28B2931.0909
Music8B14B3522.91138
Market Research4B14B3520.0927
Mechanics of Materials14B241B-A28B2924.1817
Mechanics of Materials2B241B-A28B2920.71046
Music4B14B3517.1936
Mechanics of Materials2B8B2917.2945
Mechanics of Materials4B241B-A28B2917.2835
Market Research8B14B3514.3725

Table 14.

MMMU test (57).

SubtaskSmallLargennAcc Δ\DeltaFIFNNet
Organic Chemistry8B14B3224.7523715
Engineering Graphics4B14B8915.722814
Music38B241B-A28B3343.9635013
Organic Chemistry4B14B3223.7513912
Engineering Graphics4B38B8913.5281612
Genetics14B30B-A3B5721.115312
Investment14B30B-A3B11910.1241212

Table 15.

continued MME (23).

SubtaskSmallLargennAcc Δ\DeltaFIFNNet
Music8B14B3343.3554411
Probability and Statistics8B14B5320.813211
Neuropathology4B14B9410.614410
Neuropathology4B30B-A3B9410.6201010
Electrical Circuit8B14B1287.818810
Geometry30B-A3B38B1486.8261610
Engineering Graphics2B14B8910.119109
Artificial Intelligence4B8B4320.91019
Investment4B8B1197.625169
Engineering Dynamics8B14B6513.81569
Structural Engineering8B14B949.61899
Mechanical Design1B8B928.721138
Financial Accounting4B8B1077.519118
Genetics4B30B-A3B5714.01578
Thermodynamics8B14B1634.936288
Managerial Accounting14B30B-A3B1545.226188
Engineering Graphics30B-A3B38B899.022148
Fluid Dynamics1B2B4615.21587
Mechanical Design1B4B927.619127
Engineering Graphics2B38B897.922157
Engineering Graphics4B8B897.91697
Artificial Intelligence4B14B4316.3817
Structural Engineering4B14B947.417107
Artificial Intelligence4B30B-A3B4316.3927
Engineering Graphics8B14B897.918117
Graph Theory8B14B789.01697
Probability and Statistics8B30B-A3B5313.21367
Plant Pathology14B30B-A3B2662.632257
Engineering Dynamics2B14B659.21486
Neuropathology4B8B946.416106
Investment4B30B-A3B1195.018126
Calculus8B14B1613.718126
Materials Science8B14B1394.317116
Compiler Principle8B30B-A3B4513.31046
Computer Network8B30B-A3B5012.01046
Engineering Dynamics8B30B-A3B659.217116
Organic Chemistry8B30B-A3B3221.941356
Structural Engineering8B30B-A3B946.417116
Genetics14B38B5710.51156
Analog Electronics1B2B3912.8505
Industrial Design2B8B4411.4615
Mechanical Design2B8B925.417125
Fluid Dynamics8B14B4610.91055
Genetics8B30B-A3B578.81055
Music8B30B-A3B3341.563585
Project Management8B30B-A3B2520.0615
Engineering Graphics8B38B895.622175
Probability and Statistics8B38B539.4945
Epidemiology14B30B-A3B4231.261565
Principals of Economics30B-A3B38B578.81385

Table 16.

SubtaskSmallLargennAcc Δ\DeltaFIFNNet
landmark1B2B4007.2552629
celebrity8B14B3407.431625
landmark8B14B4005.0341420
scene8B241B-A28B4004.020416
landmark8B30B-A3B4003.8352015
SubtaskSmallLargennAcc Δ\DeltaFIFNNet
celebrity8B30B-A3B3404.1311714
scene8B38B4003.521714
artwork8B14B4003.2241113
scene2B241B-A28B4003.021912
scene2B38B4002.5211110
scene8B14B4002.21789
scene30B-A3B241B-A28B4002.21789
scene8B30B-A3B4001.817107
scene14B241B-A28B4001.818117
scene30B-A3B38B4001.817107
color1B2B6010.0716
scene2B4B4001.51596
scene4B241B-A28B4001.516106
celebrity8B38B3401.817116
artwork1B2B4001.254495
scene2B14B4001.217125
artwork4B14B4001.225205
scene14B38B4001.216115

Table 17. continued

AI2D (8).

SubtaskSmallLargennAcc Δ\DeltaFIFNNet
moonPhaseEquinox30B-A3B38B2775.8291316
moonPhaseEquinox30B-A3B241B-A28B2773.220119
photosynthesisRespiration14B38B797.6826
rockCycle38B241B-A28B679.0936
partsOfA14B30B-A3B4871.033285
photosynthesisRespiration14B30B-A3B796.3505
foodChainsWebs38B241B-A28B11620.438335
partsOfA38B241B-A28B4871.031265

Table 18.

MathVista (2).

SubtaskSmallLargennAcc Δ\DeltaFIFNNet
geometry problem solving8B14B2084.333249
visual question answering38B241B-A28B1793.420146

Table 19.

SubtaskSmallLargennAcc Δ\DeltaFIFNNet
Forensic Detection2B30B-A3B13222.730030
Forensic Detection4B30B-A3B13222.029029
Forensic Detection14B30B-A3B13222.029029
Forensic Detection8B30B-A3B13221.228028
Visual Correspondence14B30B-A3B1729.9432617
Multi-view Reasoning1B4B13312.0523616
Forensic Detection1B30B-A3B13212.118216
Jigsaw2B30B-A3B15010.0382315
Relative Reflectance14B38B13411.2352015
Art Style38B241B-A28B11712.824915
Relative Reflectance14B30B-A3B13410.4352114
Art Style14B30B-A3B11711.1362313
Jigsaw2B8B1508.0271512
Relative Reflectance4B38B1349.0382612
SubtaskSmallLargennAcc Δ\DeltaFIFNNet
Multi-view Reasoning30B-A3B38B1339.0352312
Multi-view Reasoning1B14B1338.3291811
Relative Reflectance4B30B-A3B1348.2382711
Jigsaw2B14B1506.7413110
Multi-view Reasoning8B14B1337.5352510
Jigsaw4B30B-A3B1506.034259
Art Style8B30B-A3B1177.732239
Multi-view Reasoning1B2B1336.062548
Multi-view Reasoning1B38B1336.024168
Multi-view Reasoning2B4B1336.025178
Spatial Relation2B4B1435.61688
Art Style2B30B-A3B1176.835278
Semantic Correspondence8B14B1395.826188
Relative Reflectance8B38B1346.033258
Object Localization8B241B-A28B1226.623158
Spatial Relation2B8B1434.91477
Relative Depth4B8B1245.61587
Relative Reflectance8B30B-A3B1345.235287
Multi-view Reasoning8B38B1335.338317
Multi-view Reasoning30B-A3B241B-A28B1335.335287
Object Localization30B-A3B241B-A28B1225.723167
Relative Depth30B-A3B241B-A28B1245.622157
Art Style2B4B1175.130246
Jigsaw2B4B1504.021156
Jigsaw2B38B1504.031256
Jigsaw4B8B1504.022166
Object Localization8B14B1224.919136
Relative Reflectance14B241B-A28B1344.528226
Relative Depth38B241B-A28B1244.817116
Visual Similarity38B241B-A28B1354.423176
IQ Test1B2B1503.331265
Visual Similarity4B14B1353.724195
Jigsaw14B30B-A3B1503.344395
Object Localization38B241B-A28B1224.118135

Table 20.

caption:

:::

MMBench (10).

SubtaskSmallLargennAcc Δ\DeltaFIFNNet
coarse_perception2B8B11091.9492821
attribute_reasoning8B14B6992.6351718
coarse_perception2B4B11091.5422517
attribute_reasoning4B14B6992.3291316
attribute_reasoning2B14B6992.1291415
coarse_perception30B-A3B241B-A28B11091.1271512
finegrained_perception (cross-instance)14B30B-A3B5331.736279
coarse_perception2B241B-A28B11090.739318
coarse_perception38B241B-A28B11090.732248
coarse_perception14B241B-A28B11090.534286

Table 21.

Gemma 3 (26 cases)

MMT-Bench (6).

SubtaskSmallLargennAcc Δ\DeltaFIFNNet
visual_recognition12b27b6582.3372215
autonomous_driving12b27b1008.01468
medical_understanding4b27b1007.020137
SubtaskSmallLargennAcc Δ\DeltaFIFNNet
action recognition4b12b1006.01266
relation reasoning4b12b806.21385
threed12b27b4012.5835

Table 22.

MMMU test (8).

SubtaskSmallLargennAcc Δ\DeltaFIFNNet
Investment4b12b11910.9311813
Fluid Mechanics12b27b2254.4493910
Corporate Finance12b27b1316.925169
Pathology12b27b3321.2707
Physiology4b12b738.216106
Management Models12b27b728.31266
Mechanics of Materials4b12b3641.473685
Organic Chemistry12b27b3221.643385

ScienceQA (10).

SubtaskSmallLargennAcc Δ\DeltaFIFNNet
Basic economic principles4b27b7139.431328
Particle motion and energy12b27b4143.918018
Basic economic principles12b27b7123.921417
Particle motion and energy4b27b4134.119514
Magnets12b27b11011.8311813
Basic economic principles4b12b7115.514311
Magnets4b27b11010.0362511
Physical Geography4b12b5111.8606
Solutions12b27b5411.1716
Oceania: geography4b12b519.81495

BLINK (2).

SubtaskSmallLargennAcc Δ\DeltaFIFNNet
Multi-view Reasoning4b27b1335.3927
Multi-view Reasoning12b27b1335.31477

Molmo 2 (20 cases)

MMT-Bench (1).

SubtaskSmallLargennAcc Δ\DeltaFIFNNet
emotion4B8B1205.01156

MMMU test (6).

SubtaskSmallLargennAcc Δ\DeltaFIFNNet
Materials Science4B8B13911.524816
Music4B8B3344.2544014
Organic Chemistry4B8B3223.7423012

continued MME (2).

SubtaskSmallLargennAcc Δ\DeltaFIFNNet
Thermodynamics4B8B1635.51569
Mechanics of Materials4B8B3641.932257
Microbiology4B8B608.3835

Table 23 (PDF p. 24).

SubtaskSmallLargennAcc Δ\DeltaFIFNNet
artwork4B8B4004.3452817
scene4B8B4003.221813

Table 24 (PDF p. 24).

AI2D (1).

SubtaskSmallLargennAcc Δ\DeltaFIFNNet
foodChainsWebs4B8B11620.636297

Table 25 (PDF p. 24).

BLINK (8).

SubtaskSmallLargennAcc Δ\DeltaFIFNNet
Jigsaw4B8B15020.7431231
Visual Correspondence4B8B1729.3291316
Multi-view Reasoning4B8B13311.3735815
Art Style4B8B11710.321912
Relative Reflectance4B8B1346.721129
Semantic Correspondence4B8B1396.524159
Forensic Detection4B8B1326.127198
Object Localization4B8B1224.122175

Table 26 (PDF p. 24).

MMBench (2).

SubtaskSmallLargennAcc Δ\DeltaFIFNNet
coarse_perception4B8B11091.6331518
logic_reasoning4B8B4121.228235

Table 27 (PDF p. 24).

Statistical Significance Analysis

We report one-sided McNemar exact pp-values as a descriptive measure of paired asymmetry for each of the 375 candidate cases; a case is one subtask–model-pair comparison (a subtask, a smaller model, and a larger model of the same family), and each test is computed over that subtask’s shared questions. For each case, we construct a 2×22 \times2 contingency table of per-question outcomes between the smaller and larger model. Under the null hypothesis that both models are equally likely to answer correctly, the number of discordant-inverse outcomes (FlipInv) follows a Binomial(FlipInv + FlipNorm, 0.5) distribution.

These pp-values are exploratory, not confirmatory, for two reasons: (1) cases were pre-selected using the same observed data (Net ≥5\ge5, n≥20n \ge20), which biases the tested set toward positive effects; and (2) the tests involve many overlapping model-pair/subtask combinations. They should be read as a descriptive characterization of effect strength, not as family-wise error-controlled evidence. To address (2), we also report Benjamini–Hochberg (BH) adjusted pp-values (FDR 0.05), corrected either over the 375 cases or, more conservatively, over the full space of 10,434 eligible within-family comparisons (n≥20n \ge20) from which the cases were selected.

133 of 375 cases (35.5%) have uncorrected p<0.05p < 0.05, rising monotonically with Net from 10.4% (Net 5–7) to 100% (Net ≥21\ge21); near the threshold, a net advantage of 5–7 reaches significance only when it arises from few discordant questions (median 9 among the significant cases vs. 26 among the rest). After BH correction, 54 cases remain significant over the reported cases and 16 over all eligible comparisons; the latter 16 all have Net ≥16\ge16 (Table 35). For scale, the 375 cases are 3.6% of the 10,434 eligible comparisons. The larger model is ahead in 74.8% of comparisons, and by Net ≤−5\le-5 in 5,000 (47.9%); under the mirrored one-sided test (the larger model winning the discordant questions), corrected over the same 10,434 comparisons, 2,478 of these are significant. Scaling therefore helps in most comparisons and detectably reverses in a minority. Under CoT (Appendix § 22), 18 of the 375 tests remain BH-significant, all among the 96 persisting cases. Cross-family replication (§ 5.4) provides complementary evidence that these patterns are not confined to a single family.

Net RangeCasesSignificant (p<0.05p < 0.05)%BH (cases)BH (all)
5–71831910.400
8–10761925.030
11–15684972.1160
16–20323093.8204
≥21\ge211616100.01512
Total37513335.55416

Table 35. McNemar’s exact test significance by Net range, uncorrected and after Benjamini–Hochberg (BH) correction at FDR 0.05. “BH (cases)” corrects over the 375 reported cases; “BH (all)” corrects over all 10,434 eligible comparisons.

Cross-Family Replication

Table 36 lists the 13 subtask–dataset pairs with inverse scaling in at least three families. Table 37 lists 29 additional subtask–dataset pairs where inverse scaling appears in exactly two of the four families.

DatasetSubtaskFamiliesMax NetCases
BLINKMulti-view Reasoning43022
BLINKJigsaw33616
BLINKForensic Detection33013
BLINKObject Localization32210
BLINKRelative Reflectance31711
MMBenchCoarse Perception3218
MMMU testOrganic Chemistry3155
MMMU testInvestment3135
MMMU testThermodynamics393
MMT-BenchAutonomous Driving3239
MMEScene31621
MMT-BenchRelation Reasoning393
MMT-BenchEmotion363

Table 36. Subtask–dataset pairs with pairwise inverse scaling in at least three of the four families. “Max Net” is the highest Net value across all model pairs and families for that subtask. “Cases” is the total number of pairwise cases.

Table 37: Subtask–dataset pairs with pairwise inverse scaling in exactly two families. Max Net is the strongest case across all model pairs in those families.

DatasetSubtaskFamiliesMax Net
MMEcelebrityInternVL, Qwen25
MMT-Benchcross_image_matchingInternVL, Qwen19
MMBenchattribute_reasoningInternVL, Qwen18
MMMU testElectrical CircuitInternVL, Qwen17
BLINKIQ TestInternVL, Qwen17
MMEartworkInternVL, Molmo 217
BLINKVisual CorrespondenceInternVL, Molmo 217
MMMU testMaterials ScienceInternVL, Molmo 216
MMT-Benchvisual_recognitionGemma 3, InternVL15
BLINKArt StyleInternVL, Molmo 215
MMMU testEngineering GraphicsInternVL, Qwen14
MMMU testMusicInternVL, Molmo 214
MMT-Benchmedical_understandingGemma 3, Qwen13
MMT-Benchtemporal_understandingInternVL, Qwen13
MMT-Benchvisual_illusionInternVL, Qwen12
MMT-Benchmultiple_image_analysisInternVL, Qwen9
MMMU testArtificial IntelligenceInternVL, Qwen9
MMMU testStructural EngineeringInternVL, Qwen9
MMT-BenchthreedGemma 3, InternVL9
BLINKSemantic CorrespondenceInternVL, Molmo 29
MMMU testPhysiologyGemma 3, Qwen8
MMMU testCalculusInternVL, Qwen8
MMMU testFinancial AccountingInternVL, Qwen8
MMMU testGraph TheoryInternVL, Qwen7
BLINKRelative DepthInternVL, Qwen7
AI2DfoodChainsWebsInternVL, Molmo 27
MMMU testMechanics of MaterialsGemma 3, Molmo 27
MMT-Benchaction_recognitionGemma 3, Qwen6
MMMU testMicrobiologyMolmo 2, Qwen6

Table 37.

Net Flip Distribution

Table 38 shows how the number of pairwise cases, unique subtask–dataset pairs, and cross-family replicated subtasks vary with the Net threshold (n≥20n \ge20 throughout). The qualitative patterns are stable across thresholds: the same benchmarks and subtasks dominate at all cutoff levels.

Table 38: Robustness to threshold choice: pairwise inverse scaling cases (n≥20n \ge20) at different Net thresholds.

Net threshold≥5\ge5≥10\ge10≥15\ge15≥20\ge20≥25\ge25≥30\ge30
Pairwise cases3751325621125
Unique subtask–dataset pairs1154425963
Cross-family (≥3\ge3 families)1342000

Table 38.

Evaluation Prompts

All models receive identical prompts per dataset. We use direct (zero-shot, no chain-of-thought) prompting throughout to isolate model capacity from reasoning scaffolding.

Multiple-choice datasets. For MCQ datasets (MMT-Bench, MMMU, ScienceQA, MMBench), the prompt is constructed as:

[Question]
A. [Option A]
B. [Option B]
C. [Option C]
...
Please select the correct answer from the options above.
Respond with only the letter of the correct option. Do not
explain. Answer:

MMBench additionally prepends a hint field (image context description) before the question when available. MMMU questions may contain embedded <image N> placeholders referencing multiple sub-images within the question text.

Open-ended questions. MMMU contains a subset of fill-in-the-blank questions without multiple-choice options. For these, the suffix is replaced with:

Answer concisely with just the final value or term. Answer:

Yes/No dataset (MME). MME questions are presented as-is without any additional suffix. Each question expects a “Yes” or “No” response.

Pre-formatted datasets. AI2D, MathVista, and BLINK use pre-formatted prompts from their respective source distributions. AI2D uses the VLMEvalKit-processed version with masked diagrams. MathVista uses the query field from AI4Math/MathVista, which includes task-specific hints and answer format instructions. BLINK prompts are used verbatim from the original benchmark.

Error Taxonomy Judge

For the error analysis (§ 5.6), we classify each inverse scaling flip question using GPT-5.4 and Claude Opus 4.6 as multimodal annotators. GPT-5.4 is accessed via the OpenAI API (model identifier: gpt-5.4). Claude Opus 4.6 is accessed via the Anthropic API (model identifier: claude-opus-4-6). Each question is sent with its associated image(s) and the following text prompt (temperature = 0, max tokens = 200). The prompt is blinded to model identities; models are labeled “Model A” (correct) and “Model B” (incorrect) without revealing their sizes:

You are analyzing VLM (Vision-Language Model)
benchmark results. Two models answered this
question. Model A answered correctly and Model B
answered incorrectly. Your job is to figure out
why Model B failed.

Question: {question}
Options: {options}
Correct Answer: {answer}

Model A response: {response_small}
    -> CORRECT

Model B response: {response_large}
    -> INCORRECT

Categorize Model B’s failure into exactly
ONE of these categories:

1. GENUINE_ERROR - Model B made a clear mistake --
   whether through visual misidentification, wrong
   reasoning, incorrect knowledge, falling for a
   plausible distractor, or simply picking the wrong
   answer. This covers any case where the model
   genuinely got it wrong.
2. FORMAT_MISMATCH - Model B effectively gives the
   correct answer content, but in a form that would
   not match the benchmark answer format (e.g.,
   numeric value instead of option letter, or full
   option text instead of just the letter). If the
   response is self-contradictory or clearly commits
   to the wrong answer, use GENUINE_ERROR.
3. BAD_ANNOTATION_OR_AMBIGUOUS - Use this only when
   there is strong evidence that the ground-truth
   answer is wrong, the question is genuinely
   ambiguous, or multiple answers would reasonably
   be acceptable. Do NOT use this category just

because Model B’s answer seems plausible; if the benchmark answer is still clearly better supported, use GENUINE_ERROR.

Reply with ONLY a JSON object (no markdown, no extra text): {"category": "CATEGORY_NAME", "confidence": "high/medium/low", "explanation": "One concise sentence explaining why."}

Each annotator assigns exactly one category per question. We report GPT-5.4 results in the main text and use Claude Opus 4.6 as a secondary check. The two judges agree on 90.8% of questions (Cohen’s κ=0.30\kappa= 0.30; § 5.6). We treat these annotations as exploratory qualitative labels, not as validated ground truth.

Human Annotation Validation

Three authors independently annotated a stratified subsample of 127 taxonomy questions using the same three categories (Genuine Error, Bad Annotation / Ambiguous, Format Mismatch). The sample overrepresents questions flagged as Bad Annotation by either LLM judge to ensure sufficient coverage of the minority class.

Genuine ErrorBad Annot.Format Mis.
Annotator 1105 (82.7%)22 (17.3%)0
Annotator 2110 (86.6%)17 (13.4%)0
Annotator 3101 (79.5%)26 (20.5%)0
Majority vote107 (84.3%)20 (15.7%)0

Table 39. Human annotation results on 127 sampled taxonomy questions.

Inter-annotator agreement. Pairwise agreement ranged from 86.6% to 92.1% (κ=0.53\kappa= 0.53–0.740.74). Fleiss κ\kappa across all three raters was 0.65 (substantial agreement).

Human vs. LLM judges. The human majority vote agreed with GPT-5.4 on 88.2% and with Claude Opus 4.6 on 84.3% of questions. Humans labeled 15.7% of questions as Bad Annotation or Ambiguous, compared to 2.6% (GPT) and 10.5% (Claude), indicating that human annotators apply a more liberal standard for annotation quality issues than either LLM judge. Nonetheless, all five raters (three human, two LLM) agree that the large majority of sampled flips reflect genuine model errors.

Restricting to Comparisons of at Least Twofold Size

The main analysis compares every pair of sizes within a family. Some of these pairs are close in size: consecutive dense checkpoints such as InternVL 8B and 14B differ by less than a factor of two, and when models are ordered by total parameter count, a Mixture-of-Experts checkpoint can rank above a dense one while activating fewer parameters per token (Qwen 30B-A3B activates 3B, the dense 8B activates 8B). Here we repeat the case count under two restrictions that remove such comparisons: (i) the larger model has at least twice the total parameters of the smaller, which removes 3 pairs (InternVL 30B-A3B→38B, InternVL 8B→14B, Qwen3-VL 30B-A3B→32B); and (ii) the larger model has at least twice the active parameters, which additionally removes the 9 pairs in which the larger model is an MoE with fewer than twice the active parameters of the smaller (InternVL 14B→30B-A3B, InternVL 22B→30B-A3B, InternVL 38B→241B-A28B, InternVL 4B→30B-A3B, InternVL 8B→30B-A3B, Qwen3-VL 2B→30B-A3B, Qwen3-VL 32B→235B-A22B, Qwen3-VL 4B→30B-A3B, Qwen3-VL 8B→30B-A3B). Table 40 and 41 give the resulting counts in the format of Table 2; Table 42 summarizes them next to the unrestricted analysis.

DatasetQwen3-VLInternVLGemma 3Molmo 2Total
MMT-Bench29/44846/8326/961/3282/1408
MMMU (dev_val+test)24/151250/28088/3246/10888/4752
ScienceQA4/4200/78010/900/3014/1320
MME10/19618/3640/422/1430/616
AI2D0/1827/3380/391/138/572
MathVista1/701/1300/150/52/220
BLINK36/19644/3642/428/1490/616
MMBench1/849/1560/182/612/264
Total105/3108175/577226/66620/222326/9768

Table 40. Pairwise inverse scaling cases over eligible comparisons, restricted to pairs in which the larger model has at least twice the total parameters of the smaller. Gemma 3 and Molmo 2 have no pair below this ratio, so their counts are unaffected.

Pairs keptComparisonsCasesRateLarger aheadReplicated
All pairs104343753.6%5000 (47.9%)13
Total parameters ≥2×\ge2\times97683263.3%4856 (49.7%)11
Active parameters ≥2×\ge2\times82142392.9%4190 (51.0%)11

Table 42. Summary of the two restrictions, with the unrestricted analysis for reference. “Larger ahead” counts comparisons with Net ≤−5\le-5; “Replicated” is the number of subtask–dataset pairs with a case in at least three families.

The total-parameter restriction changes little: 326 of the 375 cases remain, at a rate of 3.3%, and 11 of the 13 replicated subtasks survive. The active-parameter restriction removes the pairs in which the case rate is highest: the 2,220 excluded comparisons contain 136 cases (6.1%) and no replicated subtask. Under this restriction 239 cases remain from 8,214 comparisons (2.9%), the larger model is ahead by Net ≤−5\leq-5 in 51.0% of comparisons, and 11 of the 13 replicated subtasks survive; BLINK falls from 95 to 58 cases and MathVista to none. Cross-family replication is therefore insensitive to how the pairs are restricted, whereas the case count depends on whether a checkpoint that activates fewer parameters is counted as the larger model.

DatasetQwen3-VLInternVLGemma 3Molmo 2Total
MMT-Bench20/35224/7046/961/3251/1184
MMMU (dev_val+test)26/118833/23768/3246/10873/3996
ScienceQA7/3300/66010/900/3017/1110
MME8/15416/3080/422/1426/518
AI2D0/1433/2860/391/134/481
MathVista0/550/1100/150/50/185
BLINK21/15427/3082/428/1458/518
MMBench1/667/1320/182/610/222
Total83/2442110/488426/66620/222239/8214

Table 41. As Table 40, restricted to pairs in which the larger model has at least twice the active parameters of the smaller. Gemma 3 and Molmo 2 are dense and have no pair below this ratio, so their counts are unaffected.

Option-Order Sensitivity

All main-text results keep each benchmark’s provided option order, the standard protocol, which preserves comparability with published numbers. Within a pair, both checkpoints receive the same prompt, option order, decoding, and answer-extraction pipeline, so a presentation effect shared by both models cannot by itself produce Net >0> 0; a differential sensitivity of the two checkpoints to option position could. To check this, we re-evaluate the six BLINK cases among the ten strongest cases of Table 3 under deterministic reorderings of the answer options: for two-option subtasks the single reversal, and for four-option subtasks the reversal and the three cyclic shifts. The reordering is applied identically to every question of the subtask and to both models of the pair, the gold letter is remapped accordingly, and every other setting is unchanged. The identity ordering is re-run as well, so that the effect of reordering can be read against the run-to-run variation of greedy decoding on different hardware (the re-runs use a different cluster from the main evaluation).

The smaller model’s net advantage stays positive in all 17 runs, 11 reorderings and 6 identity replays. Reordering changes the magnitude, most visibly for Qwen 4B vs. 32B on Jigsaw, where the reversal reduces Net from +30 to +9 because the 32B improves by 9pp under the reversed order, and for InternVL on Forensic Detection, where the 30B-A3B gains a few correct answers under every reordering but never more than 8pp. The identity replays also differ from the main-evaluation values in Table 3 by up to 11 Net points in either direction, which bounds the run-to-run variation of greedy decoding across hardware; the reordering effects are of the same order. Option position therefore modulates the size of these cases but does not reverse them.

#FamilySubtaskPairOrderingAcc SAcc LFIFNNet
1Qwen3-VLJigsaw4B vs. 30B-A3Bidentity80.764.03813+25
reversal76.059.33813+25
2Qwen3-VLJigsaw4B vs. 32Bidentity80.760.74010+30
reversal76.070.02920+9
3Molmo 2Jigsaw4B vs. 8Bidentity80.058.74412+32
reversal80.754.0499+40
4InternVL3.5Forensic Detection2B vs. 30B-A3Bidentity23.52.3280+28
reversal20.56.8224+18
shift +122.03.8262+24
shift +222.77.6288+20
5Qwen3-VLMulti-view Reasoning8B vs. 32Bidentity51.924.1447+37
reversal49.628.64012+28
8Qwen3-VLMulti-view Reasoning2B vs. 32Bidentity51.924.1436+37
reversal46.628.63814+24

Table 43. Option-order sensitivity for the six BLINK cases of Table 3. Each row is one deterministic reordering of the answer options; “identity” replays the original order. Acc is accuracy (%) of the smaller (S) and larger (L) model; FI, FN and Net as in the main text.

Vision Encoders and Image Preprocessing

Within-family comparisons hold the training recipe roughly fixed but not necessarily the vision encoder. Table 44 lists, for each model, the vision encoder as stated in the family’s technical report or model card and confirmed against the released configuration files. Two families change encoder across sizes: Qwen3-VL uses SigLIP2-Large (300M) for the 2B and 4B models and SigLIP2-SO-400M for the larger ones [4], and InternVL3.5 uses InternViT-300M up to the 14B model and 30B-A3B and InternViT-6B for the 38B and 241B-A28B [23]; in both, the encoder is trained jointly, so same-tier weights may still differ across sizes. Gemma 3 shares one frozen 417M SigLIP encoder across all three sizes [22], and Molmo 2 uses the same SigLIP 2 SO-400M configuration for both sizes [6]. All models use their native dynamic-resolution or tiling preprocessing at the defaults of their released processors, with no image resizing on our side.

Table 44: Vision encoders by model. “Same weights” indicates whether the encoder weights are shared across the sizes in the row according to the family’s report.

FamilyModelsEncoderParamsSame weights
Qwen3-VL2B, 4BSigLIP2-Large0.3Bno (jointly trained)
Qwen3-VL8B, 30B-A3B, 32B, 235B-A22BSigLIP2-SO-400M0.4Bno (jointly trained)
InternVL3.51B, 2B, 4B, 8B, 14B, 30B-A3BInternViT-300M0.3Bno (jointly trained)
InternVL3.538B, 241B-A28BInternViT-6B5.5Bno (jointly trained)
Gemma 34B, 12B, 27BSigLIP0.4Byes (frozen)
Molmo 24B, 8BSigLIP 2 SO-400M0.4Bsame configuration

Table 44.

Of the 375 cases, 271 (72.3%) compare two models with the same encoder tier (Qwen3-VL 68, InternVL3.5 157, Gemma 3 26, Molmo 2 20); the 46 Gemma 3 and Molmo 2 cases compare models whose encoders are shared or configuration-identical, so only the language model differs in size. The remaining 104 cases cross an encoder tier (Qwen3-VL 54, InternVL3.5 50), and in every InternVL3.5 instance the larger model is the one with the roughly twenty times larger encoder. Restricting the cross-family analysis to same-tier cases leaves 10 of the 13 replicated subtasks, including BLINK Multi-view Reasoning in all four families.

Chain-of-Thought Control Experiment

Design. We re-evaluate both models of every case under CoT prompting on all shared questions of the subtask: 375 cases, 66,698 matched question instances, and 92,694 model–question evaluations over 522 (model, dataset, subtask) runs. The CoT prompt appends a step-by-step instruction to the dataset prompt; for multiple-choice questions it reads: “Answer the preceding multiple choice question. The last line of your response should be of the following format: ‘Answer: $LETTER’ (without quotes) where LETTER is one of options. Think step by step before answering.” Decoding is greedy with a 4,096-token budget, and answers are scored with the same pipeline as the direct condition. Because both models are re-run, Net is recomputed from the CoT outputs rather than adjusted.

Result. Table 45 reports the outcome per family and Table 46 per dataset. CoT attenuates inverse scaling but does not remove it. The median Net falls from +8 to 0; 96 of 375 cases (25.6%) retain Net ≥5\ge5, 178 fall below the threshold, and 101 reverse sign (Net ≤−5\le-5). The larger model recovers 3,640 of the 8,361 questions (43.5%) it had lost under direct prompting. At the same time, the smaller model’s accuracy decreases under CoT in 251 of 375 cases (mean −3.9-3.9pp, vs. +5.2+5.2pp for the larger model), so part of the reduction in Net reflects the smaller model losing its advantage rather than the larger model gaining.

Table 45: Both-model chain-of-thought control, by family. Persist counts cases still meeting Net ≥5\ge5 under CoT; reverse counts cases where the larger model gains a net advantage of 5 or more. Recovery is the share of the questions the larger model lost in direct mode that it answers correctly under CoT.

FamilyCasesMedian NetPersistDissolveReverseRecovery
directCoT
Qwen3-VL122+8035 (28.7%)592840.1%
InternVL3.5207+7054 (26.1%)995445.4%
Gemma 326+7−22 (7.7%)141046.9%
Molmo 220+10−45 (25.0%)6939.7%
All375+8096 (25.6%)17810143.5%

Table 45.

DatasetCasesMedian NetPersistReverseRecovery
directCoT
MME35+7+417 (48.6%)746.2%
BLINK95+10+340 (42.1%)1746.4%
AI2D*9+6+13 (33.3%)114.8%
MMT-Bench87+7022 (25.3%)1637.3%
ScienceQA17+8−22 (11.8%)659.0%
MMMU dev_val10+6−11 (10.0%)248.9%
MMMU test105+7−38 (7.6%)4142.6%
MMBench14+12−141 (7.1%)1049.4%
MathVista3+7+52 (66.7%)147.1%

Table 46. Both-model chain-of-thought control, by dataset. Inverse scaling survives CoT on perception-centric benchmarks and collapses on knowledge-centric ones.

Where it persists. Persistence differs markedly across benchmarks. On MME and BLINK, close to half of the cases survive CoT (48.6% and 42.1%), whereas on MMMU the effect largely disappears: 7.6% of test-split cases persist and 41 of 105 reverse. One reading is that CoT helps more where the difficulty lies in reasoning over recognized content than where it lies in extracting the answer from the image, but we have not tested this directly.

Significance. Applying the test of Appendix § 14 to the CoT outcomes, 44 of the 96 persisting cases have uncorrected p<0.05p < 0.05, and 18 remain significant after BH correction over the 375 CoT tests (14 with Net ≥21\ge21), vs. 54 of the same cases under direct prompting.

Qualitative Examples of Inverse Scaling

We present representative examples where a smaller model answers correctly but a larger model from the same family fails. All models were prompted with direct answering (“Respond with only the letter”), yet several produce chain-of-thought reasoning unprompted (Examples 1, 2, 5, 6, and 9) and

still arrive at the wrong answer, suggesting the errors are not due to insufficient reasoning. Long responses are shortened, with omissions marked “. . . ”; Examples 9 and 12 also show answers from the chain-of-thought control (Appendix § 22), and full responses are provided in the supplementary material.

Example 1: BLINK / Jigsaw (Molmo 4B vs 8B, Net = 31).

Three images from the BLINK Jigsaw task

Figure 1 (PDF p. 32).

Question: Which image is the missing part in the first image?

Answer: B

Molmo 4B ✓: “Let’s carefully analyze this question step by step: 1. The first image has a missing lower right corner, creating a black square. 2. We need to compare this with the second and third images to determine which would fit best. . . ” → B

Molmo 8B ✗: “Before answering, let’s consider the details of each image. . . Image 1 shows two men in an outdoor setting, with the lower right corner missing. . . Image 3 shows a group of Asian individuals. . . ” → C

Analysis: Both models reason step by step, but the larger model selects the wrong piece.

Example 2: BLINK / Multi-view Reasoning (Qwen 2B vs 32B, Net = 29). Question: The images are frames from a video. Is the camera moving left or right?

Two frames showing a light blue thermos

Figure 3 (PDF p. 33).

Answer: B (right)

Qwen 2B ✓: “To determine the direction of the camera movement, we need to analyze the change in the object’s position. . . The light blue thermos is the central subject. In the first image, it is positioned more towards the center. In the second image, it has been shifted slightly to the right. . .” → B

Qwen 32B ✗: “To determine the direction of the camera movement, we compare the two images. In the first image, the object (a light blue thermos) is positioned slightly to the right. . . The vertical blind on the left is now more [visible]. . .” → A

Analysis: Both models analyze the same visual cues but the larger model infers the opposite direction.

Example 3: MME / Celebrity (InternVL 8B vs 14B, Net = 25).

Arnold Schwarzenegger inside a red bounding box

Figure 4 (PDF p. 33).

Question: Is the actor inside the red bounding box called Arnold Schwarzenegger?

Answer: Yes

InternVL 8B ✓: Yes

InternVL 14B ✗: No

Analysis: A straightforward visual identification failure: the person is clearly Arnold Schwarzenegger.

Example 4: MMT-Bench / Autonomous Driving (Qwen 2B vs 235B-A22B, Net = 19). Question: Which traffic lights can you see from the image?

Photograph of traffic lights at an intersection

Figure 5 (PDF p. 34).

Options:

  • A. Wait on

  • B. Red ✓

  • C. Yellow

  • D. Green

Qwen 2B ✓: B

Qwen 235B-A22B ✗: C (Yellow)

Analysis: The smallest Qwen model correctly identifies red traffic lights. The largest (235B MoE) says yellow, a safety-critical error in an autonomous driving context.

Example 5: BLINK / Visual Correspondence (Molmo 4B vs 8B, Net = 16).

Two colorful grid images showing visual correspondence points

Figure 6 (PDF p. 34).

Question: Which point is corresponding to the reference point?

Answer: A

Molmo 4B ✓: A

Molmo 8B ✗: “The point in the first image that corresponds to the point in the second image is Point B. In the first image, there’s a red circle with a black dot in the center, labeled ‘REF’…” → B

Analysis: The larger model reasons about the spatial correspondence but arrives at the wrong point.

Example 6: MathVista / Geometry (InternVL 8B vs 14B, Net = 9).

Geometric diagram of a circle with points A, B, C, D, and O

Figure 8.

Question: AB is the diameter of ⊙O\odot O, point D is on the extended line of AB, passing point D is the tangent of ⊙O\odot O, and the tangent point is C. If ∠A=25∘\angle A = 25^\circ, then ∠D=\angle D =?

Options:

  • A. 25∘25^\circ

  • B. 40∘40^\circ ✓

  • C. 50∘50^\circ

  • D. 65∘65^\circ

InternVL 8B ✓: “Connect OC. Since CD is tangent to the circle, OC⊥CDOC \perp CD, so ∠OCD=90∘\angle OCD = 90^\circ. Since OA=OCOA = OC, ∠A=∠OCA=25∘\angle A = \angle OCA = 25^\circ. Thus ∠DOC=50∘\angle DOC = 50^\circ, and ∠D=90∘−50∘=40∘\angle D = 90^\circ- 50^\circ= 40^\circ.” → B

InternVL 14B ✗: “Since AB is the diameter, ∠ACB=90∘\angle ACB = 90^\circ. Given ∠A=25∘\angle A = 25^\circ, ∠CBA=90∘−25∘=65∘\angle CBA = 90^\circ- 25^\circ= 65^\circ. Since CD is a tangent to the circle. . . ” → wrong answer

Analysis: Both models attempt multi-step geometric reasoning. The smaller model correctly uses tangent-radius perpendicularity and isosceles triangle properties, while the larger model misapplies the tangent-chord relationship.

Example 7: MMT-Bench / Action Recognition (Gemma 3 4B vs 12B, Net = 6).

A man sitting on a railing

Figure 7.

Question: What is the action type shown in this image? (A) lie (B) squat (C) run (D) sit

Answer: D (sit)

Gemma 3 4B ✓: D

Gemma 3 12B ✗: B

Analysis: The man is plainly seated on the railing with his legs hanging down; the larger model labels the pose as squatting.

Example 8: ScienceQA / Astronomy (Qwen 30B-A3B vs 32B, Net = 5).

PlanetVolume (km3\mathrm{km}^{3})Primary composition
Mercury6.08×10106.08 \times10^{10}rock
Venus9.28×10119.28 \times10^{11}rock
Earth1.08×10121.08 \times10^{12}rock
Mars1.63×10111.63 \times10^{11}rock
Jupiter1.43×10151.43 \times10^{15}gas
Saturn8.27×10148.27 \times10^{14}gas
Uranus6.83×10136.83 \times10^{13}ice
Neptune6.25×10136.25 \times10^{13}ice

Table 8 (PDF p. 35).

Question: Is the following statement about our solar system true or false? Earth’s volume is more than ten times as great as Mars’s volume. (A) false (B) true Answer: A (false)

Qwen 30B-A3B ✓: A

Qwen 32B ✗: B

Analysis: Ten times the volume of Mars is 1.63×10121.63 \times10^{12} km3^{3}, more than the volume of Earth (1.08×10121.08 \times10^{12} km3^{3}), so the statement is false. The answer requires reading two values from the table and one comparison.

Example 9: BLINK / Forensic Detection (Qwen 4B and 8B vs 235B-A22B, Net = 20 and 14; under CoT, 16 and 17).

Four images of elephants

Figure 1 (PDF p. 36).

Question: Which image is most likely to be a real photograph? (A) the first image (B) the second image (C) the third image (D) the fourth image

Answer: A (the first image)

Qwen 4B ✓: “. . . The other images, especially B, C, and D, have elements that suggest digital manipulation or AI generation — the overly clean water sprays, the unnatural lighting, and the overly perfect textures. Therefore, the most likely to be a real photograph is: (A) the first image.” (CoT: A)

Qwen 8B ✓: “. . . The other three images exhibit telltale signs of AI generation or heavy digital manipulation, such as unrealistic water effects, inconsistent lighting, and unnatural textures. Therefore, the correct answer is (A) the first image.” (CoT: A)

Qwen 235B-A22B, CoT ✗: “Let’s analyze each image step by step for signs of realism . . . Image 1: . . . The pillars have very uniform, repetitive texture — almost too perfect . . . Suspicious for AI generation due to texture uniformity and lack of natural interaction with environment. . . . Image 3: . . . Water spray is dynamic and realistic . . . Conclusion: Image 3 stands out as the most realistic . . . Final Answer: (C) the third image.” (direct: C)

Analysis: Both smaller models identify the real photograph with and without chain-of-thought. The largest Qwen model reasons through every image and rejects the real one as “suspicious for AI generation”, under both direct and CoT prompting.

Example 10: MMMU test / Investment (Gemma 3 4B vs 12B, Net = 13).

CharacteristicsConvertible BondCommon Equity
Par value$1,000 | —
Coupon (annual payment)4%—
Current market price980∣980 |35 per share
Straight bond value$925 | —
Conversion ratio25—
Conversion optionAt any time—
Dividend—$0
Expected market price in 1 year1,125∣1,125 |45 per share

Table 1 (PDF p. 36).

Question: Rich McDonald, CFA, is evaluating his investment alternatives in Ytel Incorporated by analyzing a Ytel convertible bond and Ytel common equity. Characteristics of the two securities are given in the exhibit. Calculate, based on the exhibit, the expected 1-year rate of return for the Ytel common equity. (A) 27.39% (B) 26.44% (C) 28.57%

Answer: C (28.57%)

Gemma 3 4B ✓: C

Gemma 3 12B ✗: A

Analysis: With no dividend, the expected return is the price change, ($45−$35)/$35=28.57%(\$45 - \$35)/\$35 = 28.57\%. The smaller model selects this value; the larger model selects a distractor.

Example 11: MME / Landmark (InternVL 1B vs 2B, Net = 29).

Church of Our Lady of Sorrows in Riga

Figure 9 (PDF p. 37).

Question: Is this a picture of Church of Our Lady of Sorrows in Riga? Please answer yes or no.

Answer: Yes

InternVL 1B ✓: Yes.

InternVL 2B ✗: “No, this is not the Church of Our Lady of Sorrows in Riga. The building in the image appears to be a church with a distinctive architectural style, but it does not match the specific characteristics of the Church of Our Lady of Sorrows in Riga. . . ”

Analysis: The larger model rejects the correct landmark and justifies the rejection at length. This pair is among the ten strongest cases (Table 3).

Example 12: BLINK / Relative Reflectance (Qwen 32B vs 235B-A22B, Net = 17; under CoT, 8).

Interior with two marked points on a sofa and wall

Figure 10 (PDF p. 37).

Question: Consider the surface color (color without shading) of the two points in the image. Which one is darker, or the color is about the same? (A) A is darker (B) B is darker (C) About the same

Answer: B (B is darker)

Qwen 32B, CoT ✓: “Point A is located on the wall, just above the sofa, between the two framed pictures. The wall is painted a light beige or cream color . . . Point B is located on the dark gray or charcoal-colored sofa . . . Therefore, Point B has a darker surface color than Point A. Final Answer: (B) B is darker.” (direct: B)

Qwen 235B-A22B, CoT ✗: “Point A is located on the backrest of the dark gray couch. Point B is located on the seat cushion of the same couch. Both points are on the same piece of furniture . . . the surface color (albedo) should be the same. Therefore, the correct answer is: (C) About the same.” (direct: C)

Analysis: The largest model misplaces point A on the sofa rather than the wall and reasons correctly from the wrong premise, under both direct and CoT prompting. The 32B locates both points correctly.

References

  1. [1]Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer. Scaling laws for generative mixed-modal language models. In International Conference on Machine Learning, pp. 265–279. PMLR, 2023.arxiv.org/abs/2301.03728
  2. [2]Mohammad Asadi, Jack W O’Sullivan, Fang Cao, Tahoura Nedaee, Kamyar Rajabalifardi, Fei-Fei Li, Ehsan Adeli, and Euan Ashley. Mirage the illusion of visual understanding. arXiv preprint arXiv:2603.21687, 2026.arxiv.org/abs/2603.21687
  3. [3]Jiyun Bae, Hyunjong Ok, Sangwoo Mo, and Jaeho Lee. Do reasoning vision-language models inversely scale in test-time compute? a distractor-centric empirical analysis, 2025. URL https://arxiv.org/abs/2511.21397.
  4. [4]Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chুনjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan Liu, Dunjie Lu, Ruilin Luo, Chenxu Lv, Rui Men, Lingchen Meng, Xuanchen Ren, Xingzhan Ren, Sibo Song, Yuchong Sun, Jun Tang, Jianhong Tu, Jianqiang Wan, Peng Wang, Pengfei Wang, Qiuyue Wang, Yuxuan Wang, Tianbao Xie, Yiheng Xu, Haiyang Xu, Jin Xu, Zhibo Yang, Mingkun Yang, Jianxin Yang, An Yang, Bowen Yu, Fei Zhang, Hang Zhang, Xi Zhang, Bo Zheng, Humen Zhong, Jingren Zhou, Fan Zhou, Jing Zhou, Yuanzhi Zhu, and Ke Zhu. Qwen3-vl technical report, 2025. URL https://arxiv.org/abs/2511.21631.
  5. [5]Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. 2818–2829. IEEE, 2023. doi: 10.1109/CVPR52729.2023.00276. URL https://doi.org/10.1109/CVPR52729.2023.00276.
  6. [6]Christopher Clark, Jieyu Zhang, Zixian Ma, Jae Sung Park, Mohammadreza Salehi, Rohun Tripathi, Sangho Lee, Zhongzheng Ren, Chris Dongjoo Kim, Yinuo Yang, Vincent Shao, Yue Yang, Weikai Huang, Ziqi Gao, Taira Anderson, Jianrui Zhang, Jitesh Jain, George Stoica, Winson Han, Ali Farhadi, and Ranjay Krishna. Molmo2: Open weights and data for vision-language models with video understanding and grounding, 2026. URL https://arxiv.org/abs/2601.10611.
  7. [7]Haodong Duan, Xinyu Fang, Junming Yang, Xiangyu Zhao, Yuxuan Qiao, Mo Li, Amit Agarwal, Zhe Chen, Lin Chen, Yuan Liu, Yubo Ma, Hailong Sun, Yifan Zhang, Shiyin Lu, Tack Hwa Wong, Weiyun Wang, Peiheng Zhou, Xiaozhe Li, Chaoyou Fu, Junbo Cui, Jixuan Chen, Enxin Song, Song Mao, Shengyuan Ding, Tianhao Liang, Zicheng Zhang, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, Dahua Lin, and Kai Chen. Vmevalkit: An open-source toolkit for evaluating large multi-modality models, 2025. URL https://arxiv.org/abs/2407.11691.
  8. [8]Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. MME: A comprehensive evaluation benchmark for multimodal large language models. CoRR, abs/2306.13394, 2023. doi: 10.48550/ARXIV.2306.13394. URL https://doi.org/10.48550/arXiv.2306.13394.
  9. [9]Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, and Ranjay Krishna. BLINK: multimodal large language models can see but not perceive. CoRR, abs/2404.12390, 2024. doi: 10.48550/ARXIV.2404.12390. URL https://doi.org/10.48550/arXiv.2404.12390.
  10. [10]Aryo Pradipta Gema, Alexander Hägele, Runjin Chen, Andy Arditi, Jacob Goldman-Wetzler, Kit Fraser-Taliente, Henry Sleight, Linda Petrini, Julian Michael, Beatrice Alex, Pasquale Minervini, Yanda Chen, Joe Benton, and Ethan Perez. Inverse scaling in test-time compute. CoRR, abs/2507.14417, 2025. doi: 10.48550/ARXIV.2507.14417. URL https://doi.org/10.48550/arXiv.2507.14417.
  11. [11]Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Henighan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae, and Laurent Sifre. Training compute-optimal large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 97817113871088.arxiv.org/abs/2203.15556
  12. [12]Subbarao Kambhampati, Kaya Stechly, Karthik Valmeekam, Lucas Paul Saldyt, Siddhant Bhambri, Vardhan Palod, Atharva Gundawar, Soumya Rani Samineni, Durgesh Kalwar, and Upasana Biswas. Stop anthropomorphizing intermediate tokens as reasoning/thinking traces! In NeurIPS 2025 Workshop on Bridging Language, Agent, and World Models for Reasoning and Planning, 2025.arxiv.org/abs/2504.09762
  13. [13]Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. URL https://arxiv.org/abs/2001.08361.
  14. [14]Aniruddha Kembhavi, Mike Salvato, Eric Kolove, Min Joon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV, volume 9908 of Lecture Notes in Computer Science, pp. 235–251. Springer, 2016. doi: 10.1007/978-3-319-46493-0\_15. URL https://doi.org/10.1007/978-3-319-46493-0_15.
  15. [15]Xianyuan Liu, Jiayang Zhang, Shuo Zhou, Thijs L van der Plas, Avish Vijayaraghavan, Anastasiia Grishina, Mengdie Zhuang, Daniel Schofield, Christopher Tomlinson, Yuhan Wang, et al. Towards deployment-centric multimodal ai beyond vision and language. Nature Machine Intelligence, pp. 1–13, 2025.arxiv.org/abs/2504.03603
  16. [16]Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player? CoRR, abs/2307.06281, 2023. doi: 10.48550/ARXIV.2307.06281. URL https://doi.org/10.48550/arXiv.2307.06281.
  17. [17]Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tåfjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. CoRR, abs/2209.09513, 2022. doi: 10.48550/ARXIV.2209.09513. URL https://doi.org/10.48550/arXiv.2209.09513.
  18. [18]Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. MathVista: Evaluating math reasoning in visual contexts with gpt-4V, bard, and other large multimodal models. CoRR, abs/2310.02255, 2023. doi: 10.48550/ARXIV.2310.02255. URL https://doi.org/10.48550/arXiv.2310.02255.
  19. [19]Ian R. McKenzie, Alexander Lyzhov, Michael Pieler, Alicia Parrish, Aaron Mueller, Ameya Prabhu, Euan McLean, Aaron Kirtland, Alexis Ross, Alisa Liu, Andrew Gritsevskyi, Daniel Wurg aft, Derik Kauffman, Gabriel Recchia, Jiacheng Liu, Joe Cavanagh, Max Weiss, Sicing Huang, The Floating Droid, Tom Tseng, Tomasz Korbak, Xudong Shen, Yuhui Zhang, Zhengping Zhou, Najoung Kim, Samuel R. Bowman, and Ethan Perez. Inverse scaling: When bigger isn’t better. Trans. Mach. Learn. Res., 2023, 2023. URL https://openreview.net/forum?id=DwgRm72GQF.arxiv.org/abs/2306.09479
  20. [20]Rohit Saxena, Alessandro Suglia, and Pasquale Minervini. Vlm-robustbench: A comprehensive benchmark for robustness of vision-language models. arXiv preprint arXiv:2603.06148, 2026.arxiv.org/abs/2603.06148
  21. [21]Parshin Shojaee, Iman Mirzadeh, Keivan Alizadeh, Maxwell Horton, Samy Bengio, and Mehrdad Farajtabar. The illusion of thinking: Understanding the strengths and limitations of reasoning models via the lens of problem complexity. arXiv preprint arXiv:2506.06941, 2025.arxiv.org/abs/2506.06941
  22. [22]Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Riviére, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gaël Liu, Francesco Visin, Kathleen Kenealy, Lucas Beyer, Xiaohai Zhai, Anton Tsitsulin, Robert Busa-Fekete, Alex Feng, Noveen Sachdeva, Benjamin Coleman, Yi Gao, Basil Mustafa, Iain Barr, Emilio Parisotto, David Tian, Matan Eyal, Colin Cherry, Jan-Thorsten Peter, Danila Sinopalnikov, Surya Bhupatiraju, Rishabh Agarwal, Mehran Kazemi, Dan Malkin, Ravin Kumar, David Vilar, Idan Br usilovsky, Jiaming Luo, Andreas Steiner, Abe Friesen, Abhanshu Sharma, Abheesht Sharma, Adi Mayrav Gilady, Adrian Goedeckemeyer, Alaa Saade, Alex Feng, Alexander Kolesnikov, Alexei Bendebury, Alvin Abdagic, Amit Vadi, András György, André Susano Pinto, Anil Das, Ankur Bapna, Antoine Miech, Antoine Yang, Antonia Paterson, Ashish Shenoy, Ayan Chakrabarti, Bilal Piot, Bo Wu, Bobak Shahriari, Bryce Petrini, Charlie Chen, Charline Le Lan, Christopher A. Choquette-Choo, CJ Carey, Cormac Brick, Daniel Deutsch, Danielle Eisenbud, Dee Cattle, Derek Cheng, Dimitris Paparas, Divyashree Shivakumar Sreepathihalli, Doug Reid, Dustin Tran, Dustin Zelle, Eric Noland, Erwin Huizenga, Eugene Kharitonov, Frederick Liu, Gagik Amirkhanyan, Glenn Cameron, Hadi Hashemi, Hanna Klimczak-Plucińska, Harman Singh, Harsh Mehta, Harshal Tushar Lehri, Hussein Hazimeh, Ian Ballantyne, Idan Spektor, Ivan Nardini, Jean Pouget-Abadie, Jetha Chan, Joe Stanton, John Wieting, Jonathan Lai, Jordi Orbay, Joseph Fernandez, Josh Newlan, Ju yeong Ji, Jyotinder Singh, Kat Black, Kathy Yu, Kevin Hui, Kiran Vodrahalli, Klaus Greff, Linhai Qiu, Marcella Valentine, Marina Coelho, Marvin Ritter, Matt Hoffman, Matthew Watson, Mayank Chaturvedi, Michael Moynihan, Min Ma, Nabila Babar, Natasha Noy, Nathan Byrd, Nick Roy, Nikola Momchev, Nilay Chauhan, Noveen Sachdeva, Oskar Bunyan, Pankil Botarda, Paul Caron, Paul Kishan Rubenstein, Phil Culliton, Philipp Schmid, Pier Giuseppe Sessa, Pingmei Xu, Piotr Stanczyk, Pouya Tafti, Rakesh Shivanna, Renjie Wu, Renke Pan, Reza Rokni, Rob Willoughby, Rohit Vallu, Ryan Mullins, Sammy Jerome, Sara Smoot, Sertan Girgin, Shariq Iqbal, Shashir Reddy, Shruti Sheth, Siim Pöder, Sijal Bhatnagar, Sindhu Raghuram Panyam, Sivan Eiger, Susan Zhang, Tianqi Liu, Trevor Yacovone, Tyler Liechty, Uday Kalra, Utku Evci, Vedant Misra, Vincent Roseberry, Vlad Feinberg, Vlad Kolesnikov, Woohyun Han, Woosuk Kwon, Xi Chen, Yinlam Chow, Yuvein Zhu, Zichuan Wei, Zoltan Egyed, Victor Cotruta, Minh Giang, Phoebe Kirk, Anand Rao, Kat Black, Nabila Babar, Jessica Lo, Erica Moreira, Luiz Gustavo Martins, Omar Sanseviero, Lucas Gonzalez, Zach Gleicher, Tris Warkentin, Vahab Mirrokni, Evan Senter, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, Yossi Matias, D. Sculley, Slav Petrov, Noah Fiedel, Noam Shazeer, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Jean-Baptiste Alayrac, Rohan Anil, Dmitry, Lepikhin, Sebastian Borgeaud, Olivier Bachem, Armand Joulin, Alek Andreev, Cassidy Hardin, Robert Dadashi, and Léonard Hussenot. Gemma 3 technical report, 2025. URL https://arxiv.org/abs/2503.19786.
  23. [23]Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, Zhaokai Wang, Zhe Chen, Hongjie Zhang, Ganlin Yang, Haomin Wang, Qi Wei, Jinhui Yin, Wenhao Li, Erfei Cui, Guanzhou Chen, Zichen Ding, Changyao Tian, Zhenyu Wu, Jingjing Xie, Zehao Li, Bowen Yang, Yuchen Duan, Xuehui Wang, Zhi Hou, Haoran Hao, Tianyi Zhang, Songze Li, Xiangyu Zhao, Haodong Duan, Nianchen Deng, Bin Fu, Yinan He, Yi Wang, Conghui He, Botian Shi, Junjun He, Yingtong Xiong, Han Lv, Lijun Wu, Wenqi Shao, Kaipeng Zhang, Huipeng Deng, Biqing Qi, Jiaye Ge, Qipeng Guo, Wenwei Zhang, Songyang Zhang, Maosong Cao, Junyao Lin, Kexian Tang, Jianfei Gao, Haian Huang, Yuzhe Gu, Chengqi Lyu, Huanze Tang, Rui Wang, Haijun Lv, Wanli Ouyang, Limin Wang, Min Dou, Xizhou Zhu, Tong Lu, Dahua Lin, Jifeng Dai, Weijie Su, Bowen Zhou, Kai Chen, Yu Qiao, Wenhai Wang, and Gen Luo. InternVL3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency, 2025. URL https://arxiv.org/abs/2508.18265.
  24. [24]Jason Wei, Najoung Kim, Yi Tay, and Quoc V. Le. Inverse scaling can become u-shaped. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 15580–15591. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.EMNLP-MAIN.963. URL https://doi.org/10.18653/v1/2023.emnlp-main.963.
  25. [25]Weive Xu, Jiahao Wang, Weiyun Wang, Zhe Chen, Wengang Zhou, Aijun Yang, Lewei Lu, Houqiang Li, Xiaohua Wang, Xizhou Zhu, et al. VisuLogic: A benchmark for evaluating visual reasoning in multi-modal large language models. arXiv preprint arXiv:2504.15279, 2025.arxiv.org/abs/2504.15279
  26. [26]Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, Jiayi Lei, Quanfeng Lu, Runjian Chen, Peng Xu, Renrui Zhang, Haozhe Zhang, Peng Gao, Yali Wang, Yu Qiao, Ping Luo, Kaipeng Zhang, and Wenqi Shao. Mmt-bench: A comprehensive multimodal benchmark for evaluating large vision-language models towards multitask AGI. CoRR, abs/2404.16006, 2024. doi: 10.48550/ARXIV.2404.16006. URL https://doi.org/10.48550/arXiv.2404.16006.
  27. [27]Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI. CoRR, abs/2311.16502, 2023. doi: 10.48550/ARXIV.2311.16502. URL https://doi.org/10.48550/arXiv.2311.16502.

Paper details

vlnc:W_h8ahxncv
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 13

downloads · 1

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

generative AI tools

View statement
Looking for related papers…
Contents