betaFeedback

WorksComputer Science

Seen but Unsaid: Locating and Repairing Cross-Image Comparison in Vision-Language Models

Abstract

Vision-language models (VLMs) often fail to match a point or an object across two images, even when the answer can be decoded from their own hidden states. We trace this failure from the visual tokens to the answer letter, in Qwen3-VL (2B-32B), InternVL3.5-8B, and Gemma-3-12B, on synthetic shape matching, same-instance point correspondence (HPatches), and semantic correspondence (SPair-71k), and find two distinct failure points, a small set of attention heads that carries the comparison, and a repair that touches 0.02% of the parameters. Two failure points: exact copies are compared but not read out. An answer-position probe exceeds the calibrated output by 15 to 25 points in every family, for nameable and unnameable shapes alike, and steering along the probe's direction moves no more answers than random directions. Semantic correspondences are present in the visual tokens but never reach the answer. One pathway: redistributing the cross-image attention of 16-32 heads, with its mass preserved, removes 17-22 points of synthetic and 15-32 points of HPatches accuracy, against at most 7 for random or attention-matched heads. A lever that training finds on its own: training only these heads' query and key projections raises SPair-71k accuracy by 28.3 points (Qwen3-VL-8B) and 25.1 points (InternVL3.5-8B), beats both layer-matched and attention-matched heads, and transfers to HPatches and BLINK. An unconstrained all-layer adapter routes 21-51% of its gain through the attention of the same heads, against at most 7% for random heads, and does so by changing what they read (83-86% of the effect) rather than their weights. Every decision rule was frozen before its run, and every reported number is recomputed by independent verification scripts.

Introduction

Vision–language models (VLMs) answer open-ended questions about images, yet remain unreliable when a task requires matching a point, a shape, or an object across two images [6]. Many such failures do not originate in the vision encoder. VLMs underperform direct readouts of their own encoders on vision-centric tasks [5]; image value tokens inside the language model support semantic correspondence zero-shot even when the model’s answer is wrong [16]; counting fails even when numerical information remains linearly separable [24]; and visual evidence can be encoded about as strongly when the answer is wrong as when it is right [23]. VLMs often see more than they say.

For cross-image correspondence, Shahgir et al. [28] argue that this gap is widest for entities without names: on synthetic “squiggle” correspondence, Qwen3-VL-2B answers 29.0% correctly while a representation probe over its hidden states reaches 74.2%, and teaching arbitrary names narrows the gap. That readout compares hidden states at the annotated locations of the reference and the options, which the answering model must find itself. Such numbers are useful upper references, but they do not measure an input-matched gap, and they do not say where inside the language model the information is lost, nor what would recover it.

Open VLMs encode each image independently, so cross-image comparison must be computed inside the language model [34]. This makes the comparison a concrete, localizable computation: tokens of the second image must retrieve evidence from the first through attention. We follow the signal along this path (Fig. 1). We measure what is available at each stage with probes that receive only what the model receives, and compare every gap against a calibrated output. We then localize the components that carry the comparison, characterize what they attend to, and test the localization by repairing it: we train only these components, compare them against matched alternatives, and ask, through attention transplantation, whether unconstrained training discovers the same components on its own. Decision rules for every run were frozen in advance; the main claims are confirmed on fresh items.

Locating the loss matters because different loci call for different remedies. If the answer is computed but not emitted, a remedy must act on how the output reads it; if the information never reaches the answer position, no change to the output side can help, and the computation that brings it there must change instead. We find both cases, in different tasks, inside the same models.

The answer in brief. The comparison between two images runs through a small, identifiable pathway: 16–32 attention heads through which tokens of the second image retrieve evidence from the first. We establish this pathway with four kinds of evidence that do not depend on each other. Causal role: redistributing the heads’ crossimage attention with its mass preserved removes 15–32 pppp of synthetic and same-instance accuracy, also on the items whose residual states stay in distribution, while random and attention-matched heads remove at most 7 pppp. Specificity: the heads’ attention is selective for the reference, gated by its salience, and predicts on fresh items whether the comparison reaches the answer position. Sufficiency as a lever: training only their query and key projections recovers most of the semantic-correspondence deficit (+25 to +28 pppp) and beats heads matched in layer or in attention mass. Convergence: an adapter that may change every layer routes a third to a half of its gain through the same heads in Qwen3-VL-8B, by changing what they read; training other heads matched in attention mass raises the selectivity of these heads as well, and selectivity tracks accuracy across all fourteen trained models. The exact-copy failure, by contrast, lies downstream of this pathway: the comparison is computed and decodable at the answer position, yet the output does not read it, and the model’s own letter directions at that position, unlike the probe’s, move the answer.

Where correspondence information is lost

Figure 1. Where correspondence information is lost. Top: the task. Each image is encoded separately; under causal attention, tokens of each option in image 2 attend back to salient candidate locations in image 1, and the true option (here B) attends more strongly to the reference. Bottom: the four measurements along this path for Qwen3-VL-8B on test items (ρREF\rho_{\mathrm{REF}} on fresh items). Exact copies are compared strongly and reach the answer position but are not emitted; semantic correspondences are present in the visual tokens but only weakly compared. Shading is a qualitative summary of the values shown. The cross-image heads are causal: redistributing their attention removes 17–32 pp of SYN and HP accuracy (§ 5.1). Training only their query–key projections raises SP accuracy from .43 to .71 (§ 6).

Our contributions are:

  • A measurement that separates seeing, comparing, and saying. Input-matched probes and a calibrated output baseline show that annotated-region readouts on synthetic matching measure pixel identity, and that the gap left after calibration is the same for unnameable and nameable shapes (§ 4.1).

  • Two distinct failure points in the same models, consistent across three families: exact copies are compared but not read out; semantic correspondences are present in visual tokens but do not reach the answer position (§ 4.2–§ 4.3).

  • The cross-image comparison pathway. A small set of heads whose attention is causal for exact-copy and same-instance correspondence under mass-preserving knockouts, also on the items whose residual states stay in the natural range, beyond random and attention-matched heads, and which the base model does not yet use for semantic correspondence—the gap that repair closes. Their salience-gated, content-weighted selectivity predicts on fresh items whether the comparison reaches the answer position (§ 5).

  • A repair at 0.02% of the parameters, on a pathway that fine-tuning finds on its own. Training only these heads’ query–key projections recovers most of the semantic-correspondence deficit and transfers to benchmarks never used for training. Attention transplantation and its decomposition show that an unconstrained adapter routes part of its gain through the same heads by a different mechanism—changing what they read rather than how they compare—so two independent routes to repair converge on one set of heads, a sufficient but redundant pathway (§ 6).

  • A verifiable protocol. Every run has frozen decision rules, every reported number is recomputed by independent verification scripts, and all outcomes, supported

or not, are listed (Appendix § 9, Appendix § 15); Table 7 maps each claim to its evidence and controls.

ClaimEvidenceControlsWhere
Exact copies are compared but not read outR3−R4aff=+21.3 ppR_{3}-R_{4}^{\mathrm{aff}}=+21.3\,pp (Qwen3-VL-8B, Syn-squiggle); +15+15 to +25 pp+25\,pp in every family; answer-position edits move R4R_{4} by at most 0.2 pp0.2\,pp; probe-direction steering at the level of random directionsOutput calibration; identical probe protocol on HP/SP (gap ≤2.6 pp\le2.6\,pp); 20 magnitude-matched random subspaces; letter-direction positive control (switches 2.82.8–18.6%18.6\%)§ 4.1–§ 4.2
The gap is not about nameabilityGap difference squiggle vs. known: +1.8 pp+1.8\,pp (Qwen3-VL-8B), +2.8 pp+2.8\,pp (Qwen3-VL-2B)Size- and stroke-matched nameable shapes§ 4.1
Semantic correspondence is present in visual tokensP−R3P-R_{3} on SP: +16.3+16.3 / +25.9 pp+25.9\,pp, +14.3 pp+14.3\,pp for Gemma-3-12BMarker-free images; layers chosen on development data§ 4.3
A small set of cross-image heads is causalMass-preserving knockout of CkC_k: R4−17R_{4}-17 to −22 pp-22\,pp on Syn, −15-15 to −32 pp-32\,pp on HP10 random and 3 attention-matched head sets (≤6.8 pp\le6.8\,pp); in-band items; within-image-2 knockout§ 5.1
Their attention is salience-gated and content-weightedΔREF\Delta_{\mathrm{REF}}: HP .051.051 / .046.046, SP .011.011 / .009.009; decoy markers raise ΔREF\Delta_{\mathrm{REF}} to .109.109; moved marker makes the new target selective (.052.052)Fresh items; head sets and rules frozen in advance§ 5.2
Training these heads repairs semantic correspondenceSP +28.3+28.3 / +25.1 pp+25.1\,pp; transfer to HP +15.2+15.2 / +30.6 pp+30.6\,pp and BLINKLayer-matched random heads (A−B +3.6+3.6 / +2.1 pp+2.1\,pp); attention-matched heads (A−M +3.3+3.3 / +2.5 pp+2.5\,pp); query-only arms (AQ_Q−BQ_Q +3.5+3.5 / +4.5 pp+4.5\,pp, AQ_Q−MQ_Q +2.1+2.1 / +4.3 pp+4.3\,pp)§ 6.1
Broad adapters route gain through the same headsTransplanting CkC_k rows of D recovers .345.345–.514.514 / .213.213–.295.295 of its gain; 8383 / 86%86\% of it with base query–key weightsFive random head sets (−0.04-0.04 to +0.07+0.07); residual norms in distribution§6.3
The pathway is redundantRestoring CkC_k rows in D removes .133.133–.166.166 / .022.022–.049.049Five random head sets§6.3

Table 7. Claims, evidence, and controls at a glance. Numbers are for Qwen3-VL-8B / InternVL3.5-8B unless stated; pp are percentage points.

Related Work

Representation versus utilization. Benchmarks expose VLM failures on perception tasks that are easy for humans, including visual and semantic correspondence [6]. Fu et al. [5] show that encoder readouts outperform the VLM on vision-centric tasks and attribute the bottleneck to the language model’s use of visual information. Liu et al. [16] find that image value tokens inside the language model support segmentation and semantic correspondence zero-shot and that correct perceptual information is often present but not surfaced. Pang et al. [24] place the counting bottleneck in the mapping to number tokens, and Veikherman et al. [30] bound how often errors are encoded but unused. Shahgir et al. [28] link the gap to nameability, and SOCO reports a similar asymmetry between visual and textual queries on real images [29]; Qiu et al. [26] find that VLM judgments of visual sameness rely on semantic labels. We measure the gap with input-matched probes, find that it is not specific to unnameable entities, and identify the computation step at which it arises.

Readouts, probes, and calibration. Probe accuracy must be read against control tasks [10], and decodability differences across layers do not add up to information lost [37]. In text-only models, hidden-state probes can recover answers when sequence scoring collapses because of output biases [38], so a gap that output calibration closes is not evidence of a deeper failure. In VLMs, probe decodability and steering recovery can overstate grounding unless the image is ablated [14]. We report every gap against a calibrated output and include image-ablation controls.

Attention heads in VLMs and multiple-choice answering. Activation patching and mediation analysis [32, 20] have located where visual evidence gains or loses influence in single-image settings: arbitration between percept and prior [23], heads through which priors override vision [15], object localization [27], and visual entities resolved too late for factual recall [31, 3]. Specialized heads ground text to image regions [25], retrieve evidence in long multimodal contexts (Li et al., 2026), and, in Qwen3-VL-8B, track the region being described, so that redirecting their attention steers the description [7]. Multiple-choice answering involves symbol-producing heads [35] and a decide-then-bind sequence [36]. These studies concern text-to-image or within-image computation; we study region-to-region comparison between two images. Interventions have known artifacts: conclusions depend on the counterfactual and metric [39], subspace patching can activate dormant pathways [19], patched states can leave the natural distribution [9], and mid-layer interventions in VLMs can yield null results [40]. We pair each intervention with random-component controls and a residual-norm check.

Mechanism-guided repair. Training-free repairs gate causal routes [4], reweight attention [17], edit sensitivity-selected heads [12], or attenuate a language-prior subspace [41]. Parameter-efficient fine-tuning [11] has been restricted to visual tokens and selected attention paths [18] and to heads selected by their mechanistic role [22]. We restrict adaptation to heads selected by a causal comparison criterion, compare the result against head-matched and parameter-matched alternatives, and test whether unconstrained adaptation converges on the same heads.

Setup

Tasks. Every item is a four-way multiple-choice question over two images: image 1 contains a location labelled REF; image 2 contains four candidates labelled A–D, one of which corresponds to REF. In SYN-squiggle, shapes are closed cubic splines through n=30n=30 anchors; SYN-known uses ten nameable outline classes matched in size and stroke. In both, image 2 contains the same four shapes as image 1 at new positions, identical up to translation, so every option has a copy in image 1 and only REF-specific comparison solves the task; within-item shape pairs are separated by a minimum Chamfer distance. HP is same-instance point correspondence generated from HPatches homographies [2]; SP is semantic key-point correspondence from SPair-71k [21]. Distractors are at least 8% of the image diagonal from the target and from each other. Splits are by seed range (SYN), sequence (HP), or official split (SP), with 1,000 development and 1,000 test items per task (500 each for SYN-known) unless stated. For transfer we also evaluate the BLINK semantic- and visual-correspondence validation splits [6].

Models. Qwen3-VL-2B / 4B / 8B / 32B [1], InternVL3.5-8B [33], and Gemma-3-12B [8]. Qwen3-VL uses SigLIP2-Large for 2B/4B and SigLIP2-SO-400M for 8B/32B, so scale comparisons are made only within encoder-matched pairs. For Qwen3-VL-8B we verified that perturbing image 1 leaves image 2’s encoder output unchanged, so cross-image comparison can occur only in the language model. Mechanistic analyses and repairs use Qwen3-VL-8B and InternVL3.5-8B.

Scoring. One forward pass returns the log-probabilities of the four letter tokens at the scoring position, with the final normalization and unembedding computed in fp32, which removes exact ties present in bf16. The answer format was selected on development data by a fixed rule (median letter mass ≥0.9\ge0.9 and generation–argmax agreement ≥95%\ge95\%). Each item is scored under four cyclic letter orders that keep shape positions fixed.

Readouts. R4R_4 is behavioural accuracy. R4affR_4^{\mathrm{aff}} is accuracy after a multinomial logistic calibration of the four letter log-probabilities, fitted on development data. R3R_3 is an input-matched probe on the answer-position residual stream (PCA to 256 dimensions, multinomial logistic regression; layer and ℓ2\ell_2 strength by five-fold cross-validation on development data). PP is a privileged readout over annotated regions: the maximum of a training-free MaxSim score R1R_1 and a paired probe R2R_2 (the latter counted only with selectivity ≥5 pp\ge5\ pp over shuffled labels), with layers chosen on development data. Confidence intervals are 2,000-sample item bootstraps. Appendix § 10.1 collects all symbols.

Evaluation sets. Three disjoint groups of test items are used: the measurement test sets of § 4 (letter order 0), the fresh confirmation items of § 5.2, and 1,000 further SP test pairs for the repair experiments of § 6 (mean over the four letter orders, or order 0 where stated). Base accuracies therefore differ slightly between sections (Qwen3-VL-8B on SP: .438, .430, and .436); every comparison we report is paired within a single item set.

Protocol. Decision rules and thresholds were frozen before each run; amendments were made only before any result existed. Every reported number was recomputed from raw outputs, or, for refitted probes, asserted to reproduce stored per-item values, by verification scripts that do not import the experiment code (Appendix § 15). Region-to-token mappings were validated per family by a patch-level perturbation test (precision 1.0). Total compute was about 90 GPU-hours, mainly on a single RTX PRO 6000. Code, item generators, split identifiers, head indices, and trained adapters will be released.

Where the Signal Is Lost

Privileged Readouts and Nameability

Privileged readouts are saturated. On Syn-squiggle (test), Qwen3-VL-8B answers 41% correctly while R3R_3 reaches 64% and R1/R2R_1/R_2 reach 1.00; Qwen3-VL-2B answers 31% with R3=36%R_3 = 36\% and R1/R2=0.93/1.00R_1/R_2 = 0.93/1.00. The privileged readouts are near ceiling already at the vision-encoder output, because the reference and the target are pixel-identical. On this task they measure pixel identity rather than anything the language model computes.

Nameability does not explain the gap. The gap R3−R4R_3 - R_4 is essentially the same for squiggles and nameable shapes: the difference is +1.8 pp+1.8\ pp for Qwen3-VL-8B and +2.8 pp+2.8\ pp for Qwen3-VL-2B. Nameability raises both R3R_3 and R4R_4; it does not change the distance between them.

The Qwen3-VL-8B gap survives calibration. Calibrating the letter log-probabilities leaves the Qwen3-VL-8B gap intact on both conditions, while it closes the small Qwen3-VL-2B gap on known shapes (Table 1). For known shapes, Qwen3-VL-8B names the ref shape and finds the letter of a named shape (letter binding 100%), yet answers the combined question correctly on 63% of items, below the product of the two step accuracies (≈82%\approx82\%).

ModelConditionR3−R4affR_3 - R_4^{\mathrm{aff}} (pp)
Qwen3-VL-2Bsquiggle+6.0+6.0 [+2.2+2.2, +9.5+9.5]
Qwen3-VL-2Bknown−0.2-0.2 [−5.0-5.0, +4.4+4.4]
Qwen3-VL-8Bsquiggle+21.3+21.3 [+17.9+17.9, +24.6+24.6]
Qwen3-VL-8Bknown+19.0+19.0 [+14.8+14.8, +23.0+23.0]

Table 1. Calibration does not close the Qwen3-VL-8B gap. R3−R4affR_3 - R_4^{\mathrm{aff}} on SYN test, order 0, with paired 95% CIs.

Breadth: The Readout Gap Is Specific to Exact Copies

Across families the calibration-robust gap on Syn-squiggle persists (+15+15 to +25 pp+25\ pp in all models except Qwen3-VL-2B). On real correspondence it does not (Table 2): calibration changes R4R_4 by at most 2.6 pp across all six models, and R3R_3 is within a few points of R4affR_4^{\mathrm{aff}} in every model. HP is solved well above chance (answer-position probe 0.54–0.86), whereas SP is near chance both at the answer position and in the output. With both images replaced by grey, the transferred probe stays at or below 0.315. Within encoder-matched pairs, the SYN gap grows from 2B to 4B and shrinks from 8B to 32B (where known shapes reach ceiling); it does not change with scale on HP or SP.

ModelTaskR4R_4R4raffR_4^{\mathrm{raff}}R3R_3R3−R4raffR_3-R_4^{\mathrm{raff}}[CI]
Qwen-8BHP.770.778.804+2.6[+0.6, +4.8]
SP.438.460.433−2.7[−6.1, +0.7]
InternVL-8BHP.543.536.542+0.6[−2.3, +3.2]
SP.355.369.310−5.9[−9.7, −2.1]
Gemma-12BHP.590.591.601+1.0[−1.7, +3.7]
SP.407.404.351−5.3[−8.8, −1.8]

Table 2. Real correspondence shows no hidden answer-position gap. Test, order 0, n=1,000n = 1{,}000.

The gap is not a property of the probe. Every task and model uses the same probe architecture, dimensionality, and model-selection protocol; the probe is fitted per task, with its layer and regularization chosen on that task’s development data. If the extra capacity of this probe family relative to a calibrated four-way output were enough to create a gap, the gap would appear everywhere. It does not: on HP and SP the probe exceeds R4affR_4^{\mathrm{aff}} by at most 2.6 pp, and falls below it in three of six model–task cells, and on Syn-known for Qwen3-VL-2B calibration closes the gap entirely, while for Qwen3-VL-8B the same protocol exceeds R4affR_4^{\mathrm{aff}} by about 20 pp on both SYN conditions. The SYN gap is therefore specific to exact copies in the models that compare them strongly (§ 5.2), not a by-product of the readout.

The decoded answer is not on the output path. To test whether the output uses the answer-position signal, we edit the residual stream at the answer position, at the probe’s layer, along the probe’s own three-dimensional subspace SS: we remove it or scale it by γ∈{2,4}\gamma\in\{2,4\}, and compare against 20 random subspaces matched in edit magnitude. No label information enters the edit, and the rules were frozen in advance. Neither edit moves the answer. On SYN-squiggle and on HP, in both Qwen3-VL-8B and InternVL3.5-8B, R4R_4 changes by at most 0.2 pp, within the range of the random subspaces, and residual norms stay in the natural band (Appendix § 13).

The layer itself is not inert. Adding, at the same position and layer, the output direction of a letter the model did not choose switches the answer to that letter on 2.8–3.8% of SYN and 9.6–18.6% of HP items at eight times the probe subspace’s natural scale, against at most 0.4% and 1.8% for ten random directions of equal norm. Steering along the probe’s own class direction, toward a class it did not predict, moves at most 1.6% of the items, within the range of the random directions (Appendix § 14). The answer that the probe reads at the answer position is thus not the one the model reports. On HP, where the output is mostly correct, the model computes it through other channels; on SYN, the decodable comparison signal sits beside the output path and is not read out, in line with reports that decodable signals in VLMs need not be causal for the answer ([40, 30]). Amplifying it at a single layer does not recover it, so the loss is not a weak linear readout that more gain would fix.

Semantic Correspondence Is Present in the Visual Tokens

We compute the privileged readout PP over 30×3030 \times30-pixel regions centred on each point, on images without drawn markers. Table 3 reports P−R3P-R_3. On SP, the region readout exceeds the answer-position probe in all three models: by +16.3 pp for Qwen3-VL-8B and +25.9 pp for InternVL3.5-8B, both above the pre-registered 15 pp threshold, and by +14.3 pp for Gemma-3-12B. In no cell is PP attained only at the vision-encoder output, so the language model’s own visual tokens carry the correspondence, in line with Liu et al. [16]. Selecting the readout layer on the test set instead of development data changes R1R_1 by 0–1.8 pp, with every interval including zero.

ModelSYN-squiggleHPSP
Qwen3-VL-8B+36.0+5.8+16.3
InternVL3.5-8B+25.8+37.0+25.9
Gemma-3-12B+35.1+18.1+14.3

Table 3. Region readout minus answer-position probe, P−R3P-R_3 in pp (images without markers, test).

Marker control. When everything except the drawn markers is replaced by grey, the region readout falls 5.8–16.1 pp below its value on marked images in five of six model-task cells; only InternVL3.5-8B on HP is largely marker-driven (0.780 versus 0.804). Table 3 therefore uses images without markers.

How the Two Images Are Compared

Cross-Image Heads

Image-1 tokens precede image-2 tokens, so under causal attention a cross-image comparison must run from image-2 (option) queries to image-1 keys, or through later positions. We scored each head by how strongly option tokens attend to their corresponding region in image 1 on SYN development items and blocked the image-2→\toimage-1 attention of the top-kk heads, with kk chosen on development data. This gives a set C32C_{32} of 32 heads in layers 1–24 of Qwen3-VL-8B and a set C16C_{16} of 16 heads in layers 3–23 of InternVL3.5-8B; we write CkC_k for either.

Causal effect. On the 1,000 SYN-squiggle test items, blocking this attention lowers R4R_4 by 15.2 pp for Qwen3-VL-8B and 24.2 pp for InternVL3.5-8B, more than the 95th percentile of 20 layer-matched random head sets, while blocking the same heads’ attention within image 2 has a much smaller effect.

Mass-preserving knockouts. Blocking removes attention mass, and for Qwen3-VL-8B 53% of the blocked items leave the natural residual-norm range (1st–99th percentile of the base model; InternVL3.5-8B: 1.9%). We therefore repeated the knockout with two operations that keep the total attention of each image-2 query on image 1 and change only where it goes: Uniform spreads it evenly over the image-1 tokens, and Shuffle permutes it among them. Each operation was applied to CkC_k, to ten layer-matched random head sets, and to the three attention-matched head sets of § 6.1 (Table 4). On SYN and HP, both operations lower R4R_4 by 15–32 pp and R3R_3 by 20–48 pp in both models; for every operation and both measures, CkC_k ranks first against the ten random sets, and its R4R_4 drop exceeds that of all three attention-matched sets, none of which lowers R4R_4 by more than 6.8 pp. Both operations keep 94–98% of the SYN items in the norm band, and restricted to these items the drops are unchanged (18.0 and 19.3 pp for Qwen3-VL-8B). Under Block, by contrast, the Qwen3-VL-8B drop on in-band items is only 3.1 pp: the earlier Qwen3-VL-8B effect was carried by off-distribution items, and the mass-preserving knockouts replace it as the causal evidence. On HP, 6–21% of items leave the band, above the frozen 5% limit, but the in-band drops are within 2 pp of the overall drops. On SP, the knockout has little effect (2.2–3.6 pp for Qwen3-VL-8B, none for InternVL3.5-8B): the base models’ above-chance semantic answers hardly run through these heads. That is where the repair of § 6 acts.

TaskBaseUniformShuffleBlockCtrl
Qwen3-VL-8B
Syn41.217.2 (18.0)18.2 (19.3)17.4 (3.1)3.6
HP76.232.2 (31.8)31.8 (31.4)21.2 (20.9)2.4
SP46.62.2 (2.4)3.2 (3.7)3.6 (3.8)2.0
InternVL3.5-8B
Syn48.620.2 (19.9)22.0 (22.0)22.4 (22.2)6.8
HP53.616.8 (16.2)14.6 (12.6)16.2 (14.9)4.2
SP34.8−0.8 (−0.8)−0.6 (−0.6)−0.6 (−0.6)1.0

Table 4. Knockouts of the image-2→image-1 attention of CkC_k. Drop in R4R_4 (pp) on 500 test items per task; in parentheses, the drop on items whose residual norm stays in the base band. Uniform and Shuffle preserve attention mass; Block removes it (on these 500 items; the 15.2 and 24.2 pp above are on all 1,000 Syn test items). Ctrl: largest drop of any of ten random and three attention-matched head sets under any operation.

Where the heads look. On HP and SP, the true option’s attention peaks on the REF region in 94–100% of items, but distractors also attend mostly to REF rather than to their own correspondence, so per-option retrieval accuracy is 0.28–0.33. Because REF is the only marked location in image 1, we tested selectivity and salience directly.

Content-Selective, Salience-Gated Attention

On fresh items never used before (600 Syn, 474 HP, 400 SP), with head sets frozen from the knockout experiment and rules frozen before any data, we measure, averaged over the heads in CkC_k, the attention mass mref(o)m_{\mathrm{ref}}(o) from option oo to the ref region, the selectivity Δref=mref(target)−mref(distractor)\Delta_{\mathrm{ref}} = m_{\mathrm{ref}}(\mathrm{target}) - m_{\mathrm{ref}}(\mathrm{distractor}), and the ratio of means ρref\rho_{\mathrm{ref}}.

Selectivity, its drop, and prediction. In both models the true option attends more to the reference than distractors do (Δref>0\Delta_{\mathrm{ref}} > 0 on HP and SP); selectivity is lower on SP than on HP, in both the difference and the ratio (Figure 2); and Δref\Delta_{\mathrm{ref}} predicts whether the answer-position probe is correct, with AUC 0.62–0.70 on HP (Table 5). Removing items whose correspondence regions overlap leaves these values essentially unchanged.

ModelTaskΔref\Delta_{\mathrm{ref}} [95% CI]ρref\rho_{\mathrm{ref}}AUC
Qwen-8BSyn+.252 [.244, .260]10.36.48
HP+.051 [.046, .056]1.88.70
SP+.011 [.008, .013]1.30.66
InternVL-8BSyn+.155 [.148, .163]2.88.69
HP+.046 [.035, .057]1.32.62
SP+.009 [.003, .016]1.13.56

Table 5. Selectivity of cross-image heads on fresh test items. AUC uses Δref\Delta_{\mathrm{ref}} as a score for answer-position probe correctness.

Figure 2. Selectivity falls from copies to semantic correspondence. Ratio of the true option’s attention on the reference to that of distractors, averaged over CkC_k, on fresh test items with 95% intervals. The SP−HP difference in Δref\Delta_{\mathrm{ref}} is −0.040 [−0.046, −0.034] for Qwen3-VL-8B and −0.036 [−0.049, −0.024] for InternVL3.5-8B.

Salience gating. When identical unlabelled markers are drawn at the distractors’ correspondence locations in image 1, distractors shift their attention to their own, now marked, locations (ratio of attention on REF to attention on the own location: 0.19–0.77), and selectivity remains positive and becomes larger (Δref=+0.109\Delta_{\mathrm{ref}} = +0.109 on HP for Qwen3-VL-8B). Without such markers, distractors place almost no attention on their own correspondence (0.011–0.017). Attention thus goes to salient candidate locations in image 1 and, among them, is weighted by content similarity.

Selectivity follows content. Moving the REF marker to a distractor’s correspondence location, with image 2 unchanged, makes the new target selective (Δref=+0.052\Delta_{\mathrm{ref}} = +0.052 on HP for Qwen3-VL-8B), while the old target keeps a smaller residual (+0.009), consistent with content-driven attention to its now unmarked correspondence. The pre-registered switch criterion required this residual to be non-positive and was therefore not met.

Sharpening is not a repair. Multiplying the image-2→image-1 attention logits of CkC_k by β∈{1.5,2}\beta\in\{1.5, 2\} does not raise calibrated accuracy on HP or SP in either model, and on Syn it lowers the answer-position probe. If the comparison stage limits semantic correspondence, it is the similarity that these heads compute, not their temperature, that must change. This is what we test next.

Repairing the Comparison

If the cross-image heads limit semantic correspondence, changing what they compare should help, and should help more than changing other components of similar size. We test this with adapters restricted to CkC_k and with matched alternatives, and then ask whether adapters that are free to change any layer end up changing the same heads.

Targeted Adapters

Arms. All arms share one recipe and differ only in their trainable parameters, with base weights frozen. A trains rank-16 LoRA updates [11] on the query and key projections of the heads in CkC_k (for grouped-query attention, the key update acts on the key–value group of each selected head): 1.68M parameters for Qwen3-VL-8B and 1.50M for InternVL3.5-8B. AQA_Q trains the query updates only (0.85M / 0.75M); because query projections are separate per head, AQA_Q changes the computation of the selected heads and of no other head. B applies arm A to three layer-matched random head sets of the same size. C trains the value projections of CkC_k with nearly the same parameter count as A (1.65M). D is a standard LoRA on all attention projections of all layers at rank 2 (1.92M parameters, the closest match to A). E is a rank-16 LoRA on all attention projections of all layers (15.3M), a common default that is not parameter-matched. C and E were run for Qwen3-VL-8B only.

Training and evaluation. Training uses 2,000 items from the SPair-71k training split in the same four-option format, cross-entropy on the correct letter at the scoring position, AdamW for two epochs with batch size 8 and a cosine schedule; the learning rate is chosen per arm from {10−4,3⋅10−4,10−3}\{10^{-4}, 3 \cdot10^{-4}, 10^{-3}\} on 400 validation items. A, C, and D use three seeds. The primary test set is 1,000 SP test pairs never used in earlier runs, scored as the mean over the four letter orders. Transfer is measured on the fresh HP items of § 5.2 and on BLINK semantic and visual correspondence, none of which is used for training or selection; the two SYN conditions serve as collateral checks.

Training the comparison heads repairs most of the deficit. Arm A raises SP accuracy by +28.3 pppp [25.3, 31.1] for Qwen3-VL-8B (.430→.712) and by +25.1 pppp [22.4, 27.8] for InternVL3.5-8B (.354→.605), with about 0.02% of the model’s parameters (Table 6). With only 500 training items the gains are still +23.9 and +19.0 pppp. The query update alone (AQ_Q), which touches no head outside CkC_k, obtains 86–90% of the effect (+24.4 and +22.7 pppp), so the repair does not depend on keys shared with other heads. The gains transfer without any training on the target benchmarks: on HP, A improves by +15.2 pppp (Qwen3-VL-8B) and +30.6 pppp (InternVL3.5-8B); on BLINK semantic correspondence by +18.2 and +5.8 pppp (the latter with an interval including zero); and on BLINK visual correspondence by +13.4 and +12.6 pppp. For Qwen3-VL-8B, A also improves both SYN conditions.

Arm (trainable parameters)ParamsSPHPBLINK-SCBLINK-VCSYN-squiggleSYN-known
QWEN3-VL-8B
base (none)–.430.782.367.669.414.686
A: QK of C32C_{32}1.68M.712.934.549.802.551.833
AQ_{\mathrm{Q}}: Q of C32C_{32}0.85M.674.922.504.791.480.842
B: QK of random heads1.69M.677.899.499.808.451.701
C: V of C32C_{32}1.65M.747.925.561.876.488.785
D: all layers, rank 21.92M.792.953.655.899.402.806
E: all layers, rank 1615.3M.790.958.655.907.451.911
INTERNVL3.5-8B
base (none)–.354.531.309.436.526.926
A: QK of C16C_{16}1.50M.605.837.367.562.366.776
AQ_{\mathrm{Q}}: Q of C16C_{16}0.75M.581.836.410.558.315.784
B: QK of random heads1.51M.584.829.369.622.474.906
D: all layers, rank 21.92M.728.941.614.845.492.993

Table 6. Repair arms. Accuracy (mean over four letter orders; seeds averaged) after training on 2,000 SP items. Only SP is in-domain; HP and BLINK measure transfer, SYN measures collateral effects. Best value per column and model in bold.

Location matters, but is not the whole story. Adapting the comparison heads beats adapting the same number of random heads in the same layers: A−B is +3.6 pppp [2.0, 5.2] for Qwen3-VL-8B and +2.1 pppp [0.8, 3.4] for InternVL3.5-8B, and the intervals stay above zero when the random head sets are resampled together with the items ([1.4, 6.0] and [0.2, 4.1]). The advantage holds, and grows for InternVL3.5-8B, when only query projections are trained, so that no update can reach a head outside the chosen set: AQ_Q beats query-only training of the same random sets (BQ_Q) by +3.5 pppp [1.6, 5.5] and +4.5 pppp [2.1, 7.0] (hierarchical intervals). Query updates of random heads leave the selectivity of CkC_k almost unchanged (ΔREF=.013\Delta_{\mathrm{REF}}=.013–.016 and .010–.011, against .034 and .029 for AQ_Q).

Attention mass does not explain the advantage. Random heads in the same layers send only 32–47% as much attention across images as CkC_k, so A−B could reflect access to the other image rather than the role of the heads. We therefore built three further head sets per model, drawn from any layer and matched to CkC_k head by head in cross-image attention mass on SYN development items (80–99% of the mass of CkC_k; 1.3–1.7 times as much on SP), while excluding the heads with the highest correspondence scores. We trained them with the recipes of A (arm M) and AQ_Q (arm MQ_Q), with every matched arm having more trainable parameters than its reference. Matched heads do about as well as random ones (SP .679 vs. .677 for Qwen3-VL-8B and .580 vs. .584 for InternVL3.5-8B), and CkC_k beats them: A−M is +3.3 pppp [1.6, 5.1] and +2.5 pppp [0.8, 4.2], and AQ_Q−MQ_Q is +2.1 pppp [0.3, 4.0] and +4.3 pppp [2.4, 6.4] (hierarchical intervals; Appendix § 14). Neither layer (B) nor cross-image attention mass (M) accounts for the advantage of CkC_k; what sets these heads apart is what they attend to. The matched arms also locate the synthetic-task cost of the targeted InternVL3.5-8B repair: training M leaves SYN-squiggle and SYN-known at .473 and .912, whereas training CkC_k lowers them to .366 and .776 (Table 6). Most of the cost comes from retraining these particular heads for semantic correspondence, as expected if the same heads also carry exact-copy comparison.

The parameter-matched all-layer adapter D is stronger than A, by 8.0 pppp (Qwen3-VL-8B) and 12.3 pppp (InternVL3.5-8B), and adapting the value projections of the same heads (C) is 3.5 pppp stronger than adapting their query–key projections in Qwen3-VL-8B. The comparison heads are thus an effective but not an optimal place to intervene: a distributed adapter can also change the features being compared, which a query–key update cannot.

The Repair Increases Selectivity

Arm A raises the selectivity of CkC_k on SP from .0107 to .0426 in Qwen3-VL-8B (rise +.032 [.029, .035]) and from .0084 to .0298 in InternVL3.5-8B (rise +.021 [.016, .027]), more than twice the rise under the random-head arm B (+.014 and +.007), although A and B reach similar accuracy. Across the base model and all trained arms, selectivity measured over the fixed heads CkC_k orders SP accuracy almost perfectly (Figure 3a; Spearman 0.96 over seven Qwen3-VL-8B models and 0.90 over five InternVL3.5-8B models). The relation holds for the attention-matched arms M and MQ_Q of § 6.1, which were trained after it was observed: with them, Spearman is 0.93 over nine and 0.96 over seven models. This includes D and E, which can change any layer: whatever else they do, they also make the comparison heads more selective. Training the attention-matched heads M, which leaves the weights of CkC_k untouched, also raises the selectivity of CkC_k (to .019–.029 in Qwen3-VL-8B and .012–.017 in InternVL3.5-8B, rises of +.008 to +.019 and +.004 to +.009), though less than A does. Because selectivity is read from the attention of a fixed head set, it needs neither a probe nor further training, which makes it a cheap check of whether a candidate repair acts on the comparison step. Zeroing the A adapters on half of CkC_k lowers SP accuracy by 5.8–9.4 pppp in Qwen3-VL-8B and 4.2–6.1 pppp in InternVL3.5-8B over three random halves, and by 6.0 and 10.7 pppp for the half with the highest retrieval score, so the repair is spread across the set rather than carried by a few heads.

Selectivity tracks the repair across arms and attention rows of $C_k$: transplanted vs. restored

Figure 3. The repair runs through the comparison heads. (a) Each point is one trained model (seed-averaged); selectivity is measured over the fixed heads CkC_k as in § 5.2. Open markers: AQ_{\mathrm{Q}} and E. Diamonds: attention-matched arms M and MQ_{\mathrm{Q}} (mean over three head sets), trained after this relation was first observed. Accuracy rises with selectivity in both families, including for the all-layer adapters D and E, which were never told about CkC_k. (b) Order-0 SP test, seed 0. Filled bars: fraction of each arm’s gain obtained by inserting only the image-2 attention rows of CkC_k from the trained model into the base model. Open bars: fraction lost by restoring those rows to the base model inside the trained model. Grey band: range of both fractions over five layer-matched random head sets, across arms. Arms whose training changed other heads (B) or only values (C) are hardly recovered by transplanting CkC_k attention.

Does Unconstrained Training Converge on the Same Heads?

The arm-level pattern in Figure 3a was observed after the fact. We tested it with rules frozen before any data. For a trained arm with gain gg over the base model, we manipulate only the post-softmax attention rows of the heads in CkC_k for image-2 query positions, leaving values, other heads, and all other positions untouched. Transplant runs the base model with these rows taken from the trained model on the same input; the recovered fraction is its accuracy gain over base divided by gg. Restore runs the trained model with these rows taken from the base model; the removed fraction is its accuracy loss divided by gg. Each operation is repeated on five layer-matched random head sets. The frozen criteria call a head set sufficient (or necessary) if the recovered (or removed) fraction is at least 0.25, its numerator is positive under the bootstrap, and it exceeds all five random sets. The decisive case is arm D, which never targeted CkC_k.

The comparison heads are sufficient. Transplanting the CkC_k attention rows of the all-layer adapter D into the base model recovers .345 [.273, .413] of its +34.8 pp gain in QWEN3-VL-8B, and .466 and .514 for two further training seeds; for INTERNVL3.5-8B it recovers .284, .295, and .213 over the three seeds (seed-0 gain +37.0 pp). The rank-16 adapter E gives .404 in QWEN3-VL-8B. Random head sets recover between −0.04 and +0.07 (Figure 3b). In accuracy terms, for the arms that meet the sufficiency criterion and all their seeds, the base model with transplanted rows reaches .549–.619 on SP (Qwen3-VL-8B, base .436) and .425–.463 (InternVL3.5-8B, base .354), against .42–.46 and .35–.37 for random transplants. Transplanted states stay in distribution: 1–5% of items fall outside the base residual-norm band, against 2% for the base model itself. The effect extends to HP, where transplanting the rows of D recovers .612 (Qwen3-VL-8B) and .481 (InternVL3.5-8B) of D’s gain, and those of E recover .789. For seed 0, the sufficiency criterion holds for A, D, and E in Qwen3-VL-8B and for A and D in InternVL3.5-8B, and fails, as expected, for B and C, whose training changed other heads or only values.

What changes is what the heads read. An attention row depends on the head’s query–key weights and on the residual stream it reads. We split Transplant accordingly. Upstream computes the rows of CkC_k with the base model’s query–key weights applied to the trained model’s inputs. Local applies the trained weights to the base model’s inputs. Each set of rows is then transplanted into the base model as before (Table 18). For D on SP, Upstream recovers .282 [.207, .348] of the gain in Qwen3-VL-8B and .233 [.166, .298] in InternVL3.5-8B, 83% and 86% of the full transplant measured in the same run, whereas Local recovers .043 and .035 with intervals reaching zero, and the interaction is near zero. E and the HP items show the same ordering, and no random head set recovers more than 7.4% of the gain on SP under any operation. The all-layer adapter thus routes its gain through the comparison heads not by rewriting them but by changing what they read: the heads’ own weights, applied to better inputs, already produce most of the transplanted effect.

Table 18. Recovered fraction of each arm’s gain under transplanted CkC_k rows (95% intervals). Interaction = FULL − UPSTREAM − LOCAL. Rank: position of CkC_k among CkC_k and five random sets (FULL / UPSTREAM / LOCAL).

ModelArmSplitGain (pp)FULLUPSTREAMLOCALInteractionRank
QWEN3-VL-8BDSP34.8.339 [.268, .404].282 [.207, .348].043 [−.013, .095]+.014 [−.059, .088]1 / 1 / 1
DHP16.2.585 [.417, .759].692 [.550, .857].077 [−.062, .207]−.185 [−.339, −.034]1 / 1 / 1
ESP35.3.411 [.340, .484].360 [.291, .435].028 [−.012, .066]+.023 [−.031, .078]1 / 1 / 2
EHP16.8.806 [.672, .953].791 [.646, .962].090 [.000, .190]−.075 [−.213, .061]1 / 1 / 2
INTERNVL3.5-8BDSP37.4.270 [.201, .339].233 [.166, .298].035 [.000, .068]+.003 [−.041, .054]1 / 1 / 1
DHP39.0.462 [.366, .558].442 [.344, .539].026 [−.029, .077]−.006 [−.078, .069]1 / 1 / 1

Table 18.

But they are not necessary. Restoring the base attention rows inside D removes only .161 [.101, .220] of its gain in Qwen3-VL-8B (.133 and .166 for the other seeds) and .022–.049 in InternVL3.5-8B. The pre-registered necessity criterion for D is therefore not met in either model, and the frozen summary is not supported. For arm A, which changes only these heads’ queries and the keys of their groups, restoring the image-2 rows removes .347–.415 in Qwen3-VL-8B over three seeds but only .053 in InternVL3.5-8B. Restoring the heads’ attention at all query positions removes .746 [.671, .823] and .370 [.248, .479] of A’s gain (seed 0), more than any of five random head sets (at most .139 and .004), so arm A meets the necessity criterion in both models; restoring every query head that shares their keys removes A’s gain exactly, as it must. For D, restoring all positions removes .198 [.137, .260] and .080 [.029, .135]—more than any random set (at most .063 and .037), but far below the criterion. The residual-norm check is uninformative for Restore, because the trained models themselves lie outside the base band.

Interpretation. The improvement that an unconstrained adapter learns contains, as one of its parts, a change in how the comparison heads attend across images; this part alone carries a third to a half of the gain in Qwen3-VL-8B, and it is produced upstream of the heads: the change in their own query–key weights contributes little on its own. The trained models also carry the improvement through other routes, which compensate when the comparison heads are reset. The targeted repair is carried by the comparison heads; the unconstrained repair uses them but does not depend on them. The comparison heads are thus a sufficient but redundant pathway: a point through which gradient descent repairs semantic correspondence on its own, but not a bottleneck through which every repair must pass.

Discussion

Two failure points and one lever. For exact copies, the comparison succeeds—selectivity is high and the answer position carries the result—but the output does not use it: neither calibration nor amplifying the decoded signal recovers it. For semantic correspondence, the visual tokens carry the correspondence, but the cross-image heads compare reference and candidates only weakly, the base model’s semantic answers barely depend on them, and the result does not reach the answer position. Same-instance correspondence lies in between and is largely solved. The repair experiments close the loop for the semantic case: changing only the similarity computed by the comparison heads recovers most of the deficit, raises their selectivity, and transfers to unseen correspondence benchmarks; and training that is free to change anything still changes these heads’ attention, by changing what they read, in a way that is sufficient to reproduce a large share of its gain. Training turns heads that the base model does not use for semantic correspondence into a route for it.

Relation to prior accounts. Large gaps measured with annotated-region readouts are reproduced by our privileged readouts, but on real correspondence they reflect information that the language model does not bring to the answer, rather than an answer that is computed and withheld. This picture does not depend on whether the entities can be named. It agrees with the observation that perceptual information inside the language model often goes unused ([5, 16]), and it adds the step at which this happens for cross-image tasks and a component that can be trained to change it. Like gaze heads for single-image description ([7]), the comparison heads are a small set whose attention controls what reaches the answer; unlike them, they compare across images, and they are a sufficient but redundant route rather than a unique one.

Why the all-layer adapter wins. A query–key update can only change which existing features the heads match. Sharpening the same heads does not help, while learning their similarity does, so the comparison function is a limiting factor. The parameter-matched all-layer adapter also changes the features themselves and the path from the comparison to the answer; that it outperforms the targeted arm, and that restoring the comparison heads inside it removes little of its gain, both indicate that the comparison is one of several limiting steps. The decomposition of §6.3 makes the first point concrete: the part of D’s gain that passes through CkC_k comes from changed inputs to the heads, which no update of the heads’ own projections can produce. A practical consequence is that the comparison heads are a good diagnostic for repairs—their selectivity tracks accuracy across every arm we trained—even where they are not the most efficient target.

Implications. Three practical points follow. First, the locus decides where a remedy must act: for exact copies the comparison is already computed, so the target is how the answer is read out, where neither calibration nor single-layer amplification suffices; for semantic correspondence the target is the comparison itself, and that is where training succeeds. Second, the comparison heads are an inexpensive training target: about 0.02% of the parameters, +19+19 to +24 pp+24\,pp on SP from only 500 training items, and transfer to benchmarks never used for training or selection. Third, because broad fine-tuning routes a third to a half of its SP gain through the same heads in Qwen3-VL-8B, their selectivity offers a way to inspect what a fine-tuned model has changed, without retraining or probing.

Conclusion

Input-matched, calibration-controlled measurement shows that the representation–output gap in cross-image correspondence is not a property of unnameable entities. Exact-copy matching is computed and lost at readout; semantic correspondence is encoded in visual tokens but only weakly compared by a small set of salience-gated, content-selective cross-image heads. Training only these heads recovers most of the semantic-correspondence deficit and transfers to other benchmarks, and broad adapters route part of their gain through them by changing what they read: the comparison heads are a sufficient but redundant pathway for repair. Locating the loss in this way separates what a model sees from what it compares and from what it says. It turns that separation into a small, verifiable target for training and gives a cheap readout, the selectivity of a fixed head set, for checking what fine-tuning has changed.

Limitations

Mechanistic results and repairs cover two families at the 8B scale; Gemma-3-12B enters the measurement results only. Blocking the attention of CkC_k leaves the natural residual-norm range for about half of the Qwen3-VL-8B synthetic items; the mass-preserving knockouts keep 94–98% of Syn items in range, but on HP 6–16% of items leave it (21% under Block), although restricting to in-range items leaves the drops nearly unchanged. In the base models, knocking out CkC_k has little effect on SP, so causal evidence for semantic correspondence comes from the trained models. The attention-matched control heads are not layer-matched, and three of their four learning-rate sweeps selected the largest rate of the grid, as did A in InternVL3.5-8B (AQ reuses A’s rate in both models). Attention mass describes where heads look; causal evidence comes from knockouts and transplants. Markers make the reference salient by design; the decoy-marker experiment addresses this, but natural queries specify the reference in other ways. The targeted repair is below the parameter-matched all-layer adapter, and in InternVL3.5-8B it lowers accuracy on both synthetic conditions, which the all-layer adapter does not. Answer-position edits show that the probe’s direction is not the one the output reads, while the output direction of each letter at the same layer is; they do not identify the channel through which the comparison is lost. SOCO could not be evaluated; the BLINK splits contain fewer than 200 items each, so their intervals are wide. Synthetic results use one stimulus family and one answer format per model, and the answer format differs from the prefill used by Shahgir et al. [28], so absolute accuracies are not directly comparable to theirs. Natural extensions include head sets matched to CkC_k in layer and attention mass at once, and general-capability evaluations after targeted repair.

Ethical Considerations

This work analyses publicly released models on public or synthetic data. The adapters improve a perceptual capability on a narrow task family and introduce no new capability class. Head-targeted training changes model behaviour in ways that general evaluations may not reveal, as the synthetic-task drop under the targeted InternVL3.5-8B adapter shows; any such repair should be evaluated for unintended effects on unrelated behaviour before deployment.

Experimental Chronology and Frozen Outcomes

Each run fixed its decision rules, thresholds, item sets, and statistical tests before execution; the only amendments were made before any result of that run existed. Table 8 lists the runs in order with their frozen outcomes. Outcomes that did not support a hypothesis are reported alongside those that did.

RunQuestionFrozen outcome
Readout ladderWhere does the SYN gap fall (readout vs. retention/computation)?QWEN3-VL-8B: readout-failure candidate; QWEN3-VL-2B: retention/computation candidate (within 0.2 pp of the rule boundary).
Gap diagnosticsIs the gap specific to unnameable shapes? Letter or position coding? Probe saturation? Composition?Gap not nameability-specific in either model; mixed letter/position coding; QWEN3-VL-8B probe not saturated; QWEN3-VL-8B composition gap.
CalibrationDoes output calibration close the gap?QWEN3-VL-8B: gap persists on both conditions; QWEN3-VL-2B: partial (squiggle), closed (known).
Breadth and scaleIs the gap general across tasks and families?Not general across tasks: present on SYN-squiggle in every family, absent on HP and SP.
Privileged readout, real imagesIs correspondence information present in visual tokens?SP: present in two of three models (third 0.7 pp below threshold); no model without information.
Cross-image headsDo causal heads exist; does per-option retrieval dissociate across tasks?Heads causal in both models; per-option retrieval criteria not met (mixed).
Attention analysisWhere do option tokens attend?Exploratory: attention concentrated on REF, positive selectivity, lower on SP.
Fresh-item confirmationAre selectivity, its drop, and its predictive value confirmed?Selectivity, drop, and prediction confirmed in both models; switch criterion not met; sharpening does not help.
Targeted repairDoes training only the query–key projections of CkC_k repair SP, and through these heads?Helps (both models); beats random heads (QWEN3-VL-8B; INTERNVL3.5-8B +2.1 pp against a 3 pp threshold, interval excluding zero); selectivity rise exceeds random heads (both); transfers (both); does not match the parameter-matched all-layer adapter (both); value arm stronger than query–key arm (QWEN3-VL-8B); not data-efficient relative to the all-layer adapter at 500 items; synthetic collateral drop in INTERNVL3.5-8B.
ConvergenceIs each arm’s gain carried by the attention of CkC_k?Sufficient for A, D, E (QWEN3-VL-8B) and A, D (INTERNVL3.5-8B); necessary for A in QWEN3-VL-8B only; summary for D: not supported (necessity).
Additional controlsDoes the output use the answer-position signal? Is the repair query-localized? Is A’s gain carried by CkC_k at all positions?Answer-position edits change no output on either task (underweighted-readout and used-on-HP criteria not met; unused-on-SYN met); query-localized (both); full-position restore necessary for A (both), not for D (both).
Matched controls and decompositionIs the advantage of CkC_k explained by cross-image attention mass? Is the answer-position layer causal for the output? Do transplanted rows owe their effect to inputs or to weights? Do mass-preserving knockouts reproduce the causal effect?Localized beyond attention mass (both); strong-localization 3 pp threshold not met (AQ_Q−MQ_Q +2.1 pp in QWEN3-VL-8B, A−M +2.5 pp in INTERNVL3.5-8B); layer causal (both); probe direction does not steer, no HP/SYN dissociation (both); upstream-dominant (both); knockout causal and beyond attention-matched on SYN and HP for every operation (both), on SP for SHUFFLE and BLOCK in QWEN3-VL-8B only; in-distribution criterion not met (HP 5.6–20.6% outside the band).

Table 8. Chronology of pre-registered runs and their frozen outcomes.

Notation {#sec:B}

Table 8.

Notation

SymbolMeaning
R4R_4Behavioural accuracy: argmax over the four letter log-probabilities at the scoring position.
R4affR^{\mathrm{aff}}_4Accuracy after multinomial logistic calibration of the four letter log-probabilities, fitted on development data.
R3R_3Input-matched probe on the answer-position residual stream (PCA to 256 dimensions, multinomial logistic regression).
R1R_1, R2R_2Privileged readouts over annotated regions: training-free MaxSim score (R1R_1) and paired probe (R2R_2).
PPmax⁡(R1,R2)\max(R_1,R_2), with R2R_2 counted only if its selectivity over shuffled labels is at least 5 pp.
CkC_kCross-image head set: C32C_{32} (32 heads, layers 1–24) for Qwen3-VL-8B, C16C_{16} (16 heads, layers 3–23) for InternVL3.5-8B.
mREF(o)m_{\mathrm{REF}}(o)Attention mass from the tokens of option oo to the REF region, averaged over the heads in CkC_k.
ΔREF\Delta_{\mathrm{REF}}, ρREF\rho_{\mathrm{REF}}Selectivity: mREFm_{\mathrm{REF}} of the target minus the mean over distractors, and the ratio of the two means.
A, AQA_QLoRA on the query–key (A) or query-only (AQA_Q) projections of CkC_k.
B, BQB_QAs A and AQA_Q, on three layer-matched random head sets of the same size.
M, MQM_QAs A and AQA_Q, on three head sets matched to CkC_k in cross-image attention mass.
CLoRA on the value projections of CkC_k.
D, ELoRA on all attention projections of all layers, rank 2 (parameter-matched to A) and rank 16.
Transplant, RestoreInsert the trained model’s image-2 attention rows of a head set into the base model, or the base model’s rows into the trained model (§6.3).
Upstream, LocalTransplanted rows computed with base query–key weights on trained inputs, or with trained weights on base inputs.
Uniform, Shuffle, BlockKnockouts of image-2→image-1 attention: spread evenly or permuted (mass preserved), or removed.
Syn, HP, SPSynthetic shape matching, HPatches point correspondence, SPair-71k semantic correspondence.

Table 9. Symbols used in the paper.

Repair Details

Adapters. In arm A the query adapter of each layer acts only on the output dimensions of the selected heads in that layer; the key adapter acts on the key–value group of each selected head, so with grouped-query attention it also changes the keys seen by the other query heads of that group. Arm C applies the same restriction to the value projection. Arms D and E adapt the query, key, value, and output projections of every decoder layer. A smoke test verified that training changes only the intended adapter parameters and leaves the base weights unchanged by checksum.

Selection and data. Training pairs come from the SPair-71k training split and validation pairs from its validation split; test pairs are from the test split and were not used in any earlier run. The learning rate was chosen per arm on the validation items. For Qwen3-VL-8B arm E, the largest learning rate diverged and was not selected. Accuracies in Table 6 average three seeds for A, C, and D and three random head sets for B; differences use a paired item bootstrap with seeds averaged first; hierarchical intervals resample the three random head sets and the items jointly. BQ uses the AQA_Q recipe on the three B head sets, with the learning rate chosen on set 0 (3⋅10−43 \cdot10^{-4} for Qwen3-VL-8B, 10−310^{-3} for InternVL3.5-8B).

Comparisons. Table 10 lists the pre-registered comparisons on the SP test set.

ComparisonQwen3-VL-8BInternVL3.5-8B
A − base (N=2,000N=2{,}000)+28.3 [25.3, 31.1]+25.1 [22.4, 27.8]
A − mean(B)+3.6 [2.0, 5.2]+2.1 [0.8, 3.4]
A − mean(B), hierarchical+3.6 [1.4, 6.0]+2.1 [0.2, 4.1]
AQ_Q − mean(BQ_Q), hierarchical+3.5 [1.6, 5.5]+4.5 [2.1, 7.0]
A − D−8.0−12.3
A − C−3.5–
A − E−7.8–
A − base (N=500N=500)+23.9+19.0
A − D (N=500N=500)−5.0−12.7
Selectivity rise, A+.032 [.029, .035]+.021 [.016, .027]
Selectivity rise, mean(B)+.014+.007

Table 10. Pre-registered comparisons on SP test (pp, paired 95% intervals where available).

Attention Transplantation: Full Results

Table 11 gives the restore and transplant fractions for every analysed arm and seed on the 1,000 SP test items (order 0). Random head sets give fractions between −0.05 and +0.08 for every arm.

ModelArmSeedGain (pp)Removed by RestoreRecovered by Transplant
Qwen3-VL-8BA0+28.9.415 [.333, .504].391
Qwen3-VL-8BA1.347.434
Qwen3-VL-8BA2.356.434
Qwen3-VL-8BC0+31.1.109.151
Qwen3-VL-8BD0+34.8.161 [.101, .220].345 [.273, .413]
Qwen3-VL-8BD1.133.466
Qwen3-VL-8BD2.166.514
Qwen3-VL-8BE0+35.6.171.404
Qwen3-VL-8BBset 0+24.3.086.041
InternVL3.5-8BA0+22.7.053.313
InternVL3.5-8BD0+37.0.030.284
InternVL3.5-8BD1.022.295
InternVL3.5-8BD2.049.213
InternVL3.5-8BBset 0–−.023.066

Table 11. Removed and recovered fractions for CkC_k, SP test, order 0.

Answer-Position Edits and Full-Position Restore

Answer-position edits. For each model and task, the answer-position probe of § 4.1 was refitted from the stored development features and reproduced the earlier cross-validation score, per-item test correctness, and per-item calibrated credit exactly. SS is the span of its centred class directions mapped back to the residual space (rank 3). Edits act on the answer-position residual at the probe’s layer (Qwen3-VL-8B: 31 for Syn, 28 for HP; InternVL3.5-8B: 32 and 33): Ablate sets h←h−PS(h−μ)h \leftarrow h - P_S(h-\mu) and Amplify sets h←h+(γ−1)PS(h−μ)h \leftarrow h + (\gamma- 1)P_S(h-\mu), with μ\mu the development mean. Each of 20 random rank-3 subspaces from the top-256 principal components is scaled so that its edit moves the residual by the same mean squared distance as SS on development data. Table 12 reports the change in R4R_4; calibrated accuracy changes by at most 0.2 pp in every cell. Between 1.7% and 2.4% of items edited along SS leave the base residual-norm band (1.2% to 4.2% for the random subspaces), against 2.0% by construction for unedited items.

ModelTaskBase R4R_4AblateAmplify γ=2\gamma=2Amplify γ=4\gamma=4Random range
Qwen3-VL-8BSyn-squiggle.413+0.0+0.0−0.1−0.2 to +0.1
Qwen3-VL-8BHP.767−0.2+0.1+0.1−0.3 to +0.3
InternVL3.5-8BSyn-squiggle.514+0.0+0.0+0.1−0.2 to +0.3
InternVL3.5-8BHP.543−0.1+0.0+0.0−0.5 to +0.4

Table 12. Answer-position edits along the probe subspace (test, n=1,000n=1{,}000, order 0). ΔR4\Delta R_4 in pp; random range over 20 subspaces and the three operations.

Full-position restore. Restore-all replaces the attention of the heads in CkC_k at every query position with that of the base model; Restore-group does so for every query head in the key–value groups that CkC_k touches. Table 13 gives the removed fractions on the 1,000 SP test items (order 0, seed 0) against the five random head sets of §6.3. For arm A, Restore-group reproduces the base model’s log-probabilities on every item in both models. Random sets share some key–value groups with CkC_k (four to nine of 20 in Qwen3-VL-8B, at most two of 14 in InternVL3.5-8B), so random Restore-group values are not a clean control (in Qwen3-VL-8B they range from .125 to .606 for arm A, without a monotone relation to the overlap) and enter no criterion. The gains in Table 13 differ from the seed-0 gains in Table 11 by at most 0.4 pp: these runs scored 14 prompts per batch instead of 4, which changes bf16 numerics on a few items.

ModelArmGain (pp)Restore-all(CkC_k)Random Restore-allRestore-group(CkC_k)
Qwen3-VL-8BA+28.7.746 [.671, .823].024 to .1391.000
Qwen3-VL-8BD+34.9.198 [.137, .260]−.006 to .063.183 [.122, .245]
InternVL3.5-8BA+22.7.370 [.248, .479]−.013 to .0041.000
InternVL3.5-8BD+37.4.080 [.029, .135]−.013 to .037.120 [.064, .172]

Table 13. Fraction of the gain removed by full-position restore (SP test, order 0, seed 0; 95% intervals).

Matched Controls, Positive Controls, Decomposition, and Mass-Preserving Knockouts

These four experiments were run together, with rules, item sets, and thresholds frozen before execution, a budget of 24 GPU-hours (18.9 used), and no cut. Two amendments were made before any training result existed: the attention-matched head sets were drawn from any layer rather than from the layers of CkC_k, and the trainable-parameter count of the matched arms was reported rather than asserted equal to that of the reference arm.

Attention-matched heads. For each head hh, X(h)X(h) is the attention mass from image-2 query positions to all image-1 key positions, averaged over Syn development items; R(h)R(h) is the correspondence score used to select CkC_k. The three sets M0M_0–M2M_2 are disjoint and built greedily: for each head e∈Cke\in C_k in turn, the head with XX closest to X(e)X(e) among those not in CkC_k, not among the 3∣Ck∣3|C_k| heads with the highest RR, and not yet used. Table 14 compares them with the layer-matched random sets of arm B. Because the matched heads are spread over more key–value groups, every matched arm has more trainable parameters than its reference (Qwen3-VL-8B: M 1.96–2.75M vs. 1.68M, MQM_Q 0.98–1.38M vs. 0.85M; InternVL3.5-8B: 1.64–1.77M vs. 1.50M, 0.82–0.88M vs. 0.75M). Training follows the recipe of §6.1 with one seed per set; the learning rate was chosen on set 0 from the same grid (3⋅10−43\cdot10^{-4} for Qwen3-VL-8B M, 10−310^{-3} for the other three arms). The grid’s largest rate was thus selected in three of four sweeps, as for A in InternVL3.5-8B; AQA_Q reuses A’s rate in both models, without a sweep of its own. Table 15 gives the comparisons. In Qwen3-VL-8B, MQM_Q closes about 40% of the gap between AQA_Q and BQB_Q; in the other three comparisons matched heads do about as well as random ones. On the synthetic conditions, which no arm is trained on, InternVL3.5-8B M and MQM_Q are 10.7–16.3 pp above A and AQA_Q, extending the collateral drop of the targeted InternVL3.5-8B arms noted in § 6.1; on HP and blink-sc they are below.

ModelSetXX ratioRR ratioXSPX_{\mathrm{SP}} ratio
Qwen3-VL-8BM0M_0 / M1M_1 / M2M_2.985 / .950 / .803.107 / .043 / .0811.670 / 1.575 / 1.319
B0B_0 / B1B_1 / B2B_2.350 / .430 / .368.131 / .152 / .127.448 / .554 / .487
InternVL3.5-8BM0M_0 / M1M_1 / M2M_2.969 / .875 / .827.101 / .125 / .1051.709 / 1.390 / 1.401
B0B_0 / B1B_1 / B2B_2.316 / .467 / .464.017 / .115 / .112.522 / .701 / .717

Table 14. Head-set statistics relative to CkC_k. Ratios of set means; XSPX_{\mathrm{SP}} is measured on 200 SP development items.

ModelComparisonCkC_k armMatched (per set)B meanDifferencePaired CIHierarchical CI
Qwen3-VL-8BA − M71.2467.90 (68.28, 67.40, 68.03)67.66+3.34[1.73, 4.94][1.62, 5.07]
AQ−MQA_Q - M_Q67.3565.23 (65.03, 66.03, 64.65)63.82+2.12[0.50, 3.78][0.30, 4.02]
InternVL3.5-8BA − M60.5158.03 (57.38, 59.18, 57.53)58.38+2.48[1.19, 3.84][0.75, 4.24]
AQ−MQA_Q - M_Q58.0553.73 (52.78, 54.87, 53.52)53.52+4.32[2.77, 5.98][2.42, 6.39]

Table 15. CkC_k against attention-matched heads (SP test, 1,000 items, mean over four letter orders, %). Paired intervals average the three sets first; hierarchical intervals resample sets and items.

ModelArmsHPBLINK-SCBLINK-VCSyn-squiggleSyn-known
Qwen3-VL-8BA / M93.4 / 91.754.9 / 48.980.2 / 79.855.1 / 51.183.3 / 79.1
AQA_Q / MQM_Q92.2 / 89.750.4 / 45.679.1 / 74.648.0 / 49.584.2 / 79.0
InternVL3.5-8BA / M83.7 / 82.536.7 / 33.656.2 / 57.236.6 / 47.377.6 / 91.2
AQA_Q / MQM_Q83.7 / 79.041.0 / 34.855.8 / 52.931.5 / 47.878.4 / 92.9

Table 16. Transfer and collateral accuracy of the matched arms (%, mean over the three sets; A and AQA_Q from Table 6).

Answer-position positive controls. On the first 500 test items of Syn-squiggle and HP (order 0), at the position and layer of Appendix § 13, edits add αsu\alpha s u to the residual, with ss the standard deviation of the residual within the probe subspace SS, α∈{1,2,4,8}\alpha\in\{1, 2, 4, 8\}, and uu a unit direction. For the letter direction, u∝WU[c]−WU[c′]c′≠cu \propto W_U[c] - W_U[c']_{c' \ne c}, where cc is the first letter that is not the model’s base answer; the switch rate is the fraction of items whose answer becomes cc. For the probe direction, uu is the class direction of the probe for the first class it does not predict, mapped back to the residual space; the frozen steer rate is the fraction of items whose answer becomes that class’s letter. Because this rate also counts items already answered with that letter (4.4–10.0%, identical in every row), Table 17 also reports the rate on the remaining items. Controls are ten random unit directions from the top-256 principal components at the same αs\alpha s. At most 4.8% of letter- or probe-edited items leave the base norm band (10.2% for random directions at α=8\alpha= 8). The frozen steering criterion (an excess of at least 5 pp over the random mean) is not reached at any α\alpha in either model, and the HP−Syn difference in steer rate has an interval including zero (−1.2 pp [−4.0, 1.4] and +2.2 pp [−1.4, 5.8]).

ModelTaskLetter switch, α=1/2/4/8\alpha= 1/2/4/8Random maxProbe movedRandom moved
Qwen3-VL-8BSyn-squiggle0.4 / 0.4 / 0.8 / 2.80.40.40.2–0.4
HP0.8 / 1.2 / 3.0 / 9.60.80.40.0–0.6
InternVL3.5-8BSyn-squiggle0.6 / 0.6 / 1.8 / 3.80.40.90.0–1.5
HP3.2 / 4.6 / 9.4 / 18.61.81.60.9–1.6

Table 17. Answer-position positive controls (%, 500 test items per task). Last three columns at α=8\alpha= 8. Moved: steer rate on the items not already answered with the target letter.

Decomposition of Transplant. Upstream runs the trained model and computes the image-2 rows of CkC_k with the query–key adapter updates switched off for these rows only; Local runs the base model and computes them with the updates switched on. Rows are transplanted into the base model as in §6.3, on the 1,000 SP and 400 HP items of that section (order 0, seed 0), and the same three operations are applied to the five random head sets. Fractions use numerator and gain from the same bootstrap resamples. A reproduction gate with the original batch layout matched §6.3 exactly on the first 50 items; the runs reported here score 32 prompts per batch, which changes bf16 numerics on 1–19 items per arm, so FULL differs slightly from Table 11. For D on HP in Qwen3-VL-8B, Upstream exceeds FULL and the interaction interval excludes zero.

Mass-preserving knockouts. Operations act on the post-softmax image-2 query rows of a head set, image-1 key block only. UNIFORM replaces the distribution over image-1 keys by a uniform one with the same total mass; SHUFFLE permutes the values over image-1 keys with one fixed permutation per item and layer; BLOCK is the knockout of § 5.1. Items are the first 500 test items per task (order 0); controls are the first ten layer-matched random sets of § 5.1 and the three attention-matched sets. Calibrated accuracy follows R4R_4 within 4 pp in every cell. Under every operation, random sets leave the norm band on at most 5.6% of items and matched sets on at most 8.0%.

Table 19. Knockouts of CkC_k, full results (pp; paired 95% intervals). Random, Matched: largest R4R_4 drop among ten random and three attention-matched sets. Out: % of items outside the base residual-norm band.

TaskOperationR4R_4 dropR3R_3 dropRandomMatchedOut
QWEN3-VL-8B (base R4R_4 / R3R_3: SYN 41.2 / 62.2, HP 76.2 / 81.2, SP 46.6 / 42.0)
SYNUNIFORM17.2 [12.6, 22.0]38.2 [32.6, 43.8]2.80.64.6
SHUFFLE18.2 [13.8, 22.8]37.6 [31.6, 43.2]3.00.25.6
BLOCK17.4 [12.2, 22.8]34.8 [28.8, 40.4]3.6−0.849.0
HPUNIFORM32.2 [27.6, 37.0]35.0 [30.0, 39.8]1.21.25.6
SHUFFLE31.8 [27.2, 36.4]32.4 [27.8, 37.6]1.01.85.8
BLOCK21.2 [17.4, 25.4]26.2 [21.6, 31.0]2.40.215.8
SPUNIFORM2.2 [−0.4, 4.8]2.2 [−1.4, 6.0]1.00.01.8
SHUFFLE3.2 [0.4, 6.0]2.2 [−1.2, 6.0]1.2−0.61.6
BLOCK3.6 [0.8, 6.6]4.2 [0.2, 8.4]1.42.01.0
INTERNVL3.5-8B (base R4R_4 / R3R_3: SYN 48.6 / 73.4, HP 53.6 / 54.0, SP 34.8 / 29.4)
SYNUNIFORM20.2 [15.4, 25.0]48.4 [43.6, 53.4]3.46.63.6
SHUFFLE22.0 [17.2, 26.8]46.0 [41.0, 51.0]1.86.22.0
BLOCK22.4 [17.4, 27.2]46.4 [41.4, 51.4]2.66.82.0
HPUNIFORM16.8 [13.0, 20.8]20.6 [16.2, 25.2]2.81.815.8
SHUFFLE14.6 [10.8, 18.4]19.6 [15.0, 24.0]4.21.416.2
BLOCK16.2 [12.2, 20.6]18.2 [13.4, 23.0]2.41.820.6
SPUNIFORM−0.8 [−3.0, 1.6]−1.2 [−3.8, 1.4]0.00.01.8
SHUFFLE−0.6 [−3.0, 1.8]−1.6 [−4.2, 1.0]0.41.01.4
BLOCK−0.6 [−3.0, 2.0]−0.8 [−3.4, 1.8]0.2−0.21.4

Table 19.

Verification and Provenance

Every number in the main text was recomputed from raw per-item outputs or, for probe and calibration refits, checked against stored per-item values, by scripts that do not import the experiment code, with no mismatches (between 238 and 3.6×1053.6 \times10^{5} checks per run). Per-item answer-position calibration and probe refits used in the fresh-item confirmation were asserted to reproduce the earlier per-item values before any forward pass. Region-to-token mappings were validated with a perturbation test at the patch-embedding output; for InternVL3.5, the pixel footprint of each region was defined from the support of the resize kernel. Item sets for the fresh-item confirmation use seeds, sequences, and image pairs disjoint from all earlier runs, except that most HP image pairs recur with new reference points and distractors. Repair test pairs are disjoint from all earlier runs, and overlap between training, validation, and test splits was checked by pair identifier.

References

  1. [1]Shuai Bai, Yuxuan Cai, Ruizhe Chen, Kevin Chen, Xionghui Chen, et al. Qwen3-VL Technical Report. arXiv:2511.21631, 2025.arxiv.org/abs/2511.21631
  2. [2]Vasileios Balntas, Karel Lenc, Andrea Vedaldi, and Krystian Mikolajczyk. HPatches: A Benchmark and Evaluation of Handcrafted and Learned Local Descriptors. In CVPR, 2017.arxiv.org/abs/1704.05939
  3. [3]Eden Biran, Daniela Gottesman, Sohee Yang, Mor Geva, and Amir Globerson. Hopping Too Late: Exploring the Limitations of Large Language Models on Multi-Hop Queries. In EMNLP, 2024.
  4. [4]Zhe Cheng, Wen­yu Chen, Fode Zhang, and Dehuan Shen. Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gating. In ICML, pages 18828–18860, 2026.arxiv.org/abs/2605.24024
  5. [5]Stephanie Fu, Tyler Bonnen, Devin Guillory, and Trevor Darrell. Hidden in Plain Sight: VLMs Overlook Their Visual Representations. In COLM, 2025.arxiv.org/abs/2506.08008
  6. [6]Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, and Ranjay Krishna. BLINK: Multimodal Large Language Models Can See but Not Perceive. In ECCV, 2024.arxiv.org/abs/2404.12390
  7. [7]Rohit Gandikota and David Bau. Gaze Heads: How VLMs Look at What They Describe. arXiv:2606.14703, 2026.arxiv.org/abs/2606.14703
  8. [8]Gemma Team. Gemma 3 Technical Report. arXiv:2503.19786, 2025.arxiv.org/abs/2503.19786
  9. [9]Satchel Grant, Simon Jerome Han, Alexa R. Tartaglini, and Christopher Potts. Addressing Divergent Representations from Causal Interventions on Neural Networks. In ICLR, 2026.arxiv.org/abs/2511.04638
  10. [10]John Hewitt and Percy Liang. Designing and Interpreting Probes with Control Tasks. In EMNLP-IJCNLP, pages 2733–2743, 2019.arxiv.org/abs/1909.03368
  11. [11]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. In ICLR, 2022.arxiv.org/abs/2106.09685
  12. [12]Junyang Ji, Qifan Liu, Wenming Yang, and Zhihai He. Causal-Lens: Sensitivity-Guided Multi-Head Causal Intervention for Hallucination Mitigation in Large Vision-Language Models. In CVPR, pages 4199–4209, 2026.
  13. [14]Chih-Ting Liao, Fei Shen, Xin Cao, and Tat-Seng Chua. Decodable Is Not Grounded: A Vision-Ablation Arbiter for VLM Spatial Reasoning. arXiv:2606.31257, 2026.arxiv.org/abs/2606.31257
  14. [15]Niclas Lietzow, Danielle Bitterman, Carsten Eickhoff, William Rudman, and Michal Golanovensky. Vision-Default, Prior-Override: Causal Mechanisms of Perception-Knowledge Conflict in Vision-Language Models. arXiv:2606.28273, 2026.arxiv.org/abs/2606.28273
  15. [16]Benlin Liu, Amita Kamath, Madeleine Grunde-McLaughlin, Winson Han, and Ranjay Krishna. Visual Representations inside the Language Model. In COLM, 2025.arxiv.org/abs/2510.04819
  16. [17]Zhining Liu, Ziyi Chen, Hui Liu, Chen Luo, Xianfeng Tang, Suhang Wang, Jingying Zeng, Zhenwei Dai, Zhan Shi, Tianxin Wei, Benoit Dumoulin, and Hanghang Tong. Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMs. In ICLR, 2026.arxiv.org/abs/2510.17771
  17. [18]Tiange Luo, Lajanu­gen Logeswaran, Jaekyeom Kim, Justin Johnson, and Honglak Lee. Towards Minimal Fine-Tuning of VLMs. arXiv:2512.19219, 2025.
  18. [19]Aleksandar Makelov, George Lange, Atticus Geiger, and Neel Nanda. Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching. In ICLR, 2024.arxiv.org/abs/2311.17030
  19. [20]Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and Editing Factual Associations in GPT. In NeurIPS, 2022.arxiv.org/abs/2202.05262
  20. [21]Juhong Min, Jongmin Lee, Jean Ponce, and Minsu Cho. SPair-71k: A Large-scale Benchmark for Semantic Correspondence. arXiv:1908.10543, 2019.arxiv.org/abs/1908.10543
  21. [22]Chancharik Mitra, Yusen Luo, Raj Saravanan, Dantong Niu, Anirudh Pai, Jesse Thomason, Trevor Darrell, Abrar Anwar, Deva Ramanan, and Roei Herzig. Mechanistic Finetuning of Vision-Language-Action Models via Few-Shot Demonstrations. arXiv:2511.22697, 2025.arxiv.org/abs/2511.22697
  22. [23]Farhad Nooralahzadeh, Omid Rohaniann, Yi Zhang, Jonathan Fürst, and Kurt Stockinger. Arbitration Failure, Not Perceptual Blindness: How Vision-Language Models Resolve Visual-Linguistic Conflicts. arXiv:2604.09364, 2026.arxiv.org/abs/2604.09364
  23. [24]Xingzhou Pang, Yifan Hou, Junling Wang, and Mrinmaya Sachan. Unveiling the Visual Counting Bottleneck in Vision-Language Models. In ICML, 2026.arxiv.org/abs/2605.30170
  24. [25]Chanho Park, Daeheyeon Choi, Jihyun Lee, and Minhhyuk Sung. Retrieval Heads Meet Vision: Uncovering How VLMs Locate and Extract Visual Information. arXiv:2608.27417, 2026.arxiv.org/abs/2608.27417
  25. [26]Jason Qiu, Zachary Meurer, Xavier Thomas, and Deepti Ghadiyaram. Semantic Richness or Geometric Reasoning? The Fragility of VLM’s Visual Invariance. In COLM, 2026.arxiv.org/abs/2604.01848
  26. [27]Timothy Schaumlöffel, Martina G. Vilas, and Gemma Roig. Mechanisms of Object Localization in Vision-Language Models. In CVPR, pages 31356–31365, 2026.arxiv.org/abs/2605.19792
  27. [28]Haza Shahgir, Xiaofuo Chen, Yu Fu, Erfan Shavegani, Nael Abu-Ghazaleh, Yova Kementchedjhieva, and Yue Dong. VLMs Need Words: Vision Language Models Ignore Visual Detail In Favor of Semantic Anchors. In COLM, 2026.arxiv.org/abs/2604.02486
  28. [29]Basavaraj Sunagad, Haoran Wang, David T. Hoffmann, Christian Theobalt, and Adam Kortylewski. SOCO: Benchmarking Semantic Object Correspondence in Vision Foundation Models. arXiv:2605.31597, 2026.arxiv.org/abs/2605.31597
  29. [30]Anat Veikherman, Mor Ventura, Nitay Calderon, Lihi Zelnik-Manor, and Roi Reichart. Encoded but Unused: Bounding the Sources of VLM Errors. arXiv:2610.29241, 2026.arxiv.org/abs/2610.29241
  30. [31]Constantin Venhoff, Ashkan Khakzar, Sonia Joseph, Philip Torr, and Neel Nanda. Too Late to Recall: Explaining the Two-Hop Problem in Multimodal Knowledge Retrieval. In NeurIPS, 2025.
  31. [32]Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating Gender Bias in Language Models Using Causal Mediation Analysis. In NeurIPS, 2020.
  32. [33]Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, et al. InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency. arXiv:2508.18265, 2025.arxiv.org/abs/2508.18265
  33. [34]Zirui Wang, Junwei Yu, Adam Yala, David M. Chan, Joseph E. Gonzalez, and Trevor Darrell. Stateful Visual Encoders for Vision-Language Models. arXiv:2606.04433, 2026.arxiv.org/abs/2606.04433
  34. [35]Sarah Wiegreffe, Oyvind Tafjord, Yonatan Belinkov, Hannah Hajishirzi, and Ashish Sabharwal. Answer, Assemble, Ace: Understanding How LMs Answer Multiple Choice Questions. In ICLR, 2025.arxiv.org/abs/2407.15018
  35. [36]Hugh Mee Wong, Rick Nouwen, and Albert Gatt. When Models Decide and When They Bind: A Two-Stage Computation for Multiple-Choice Question-Answering. arXiv:2601.03914, 2026.arxiv.org/abs/2601.03914
  36. [37]Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon. A Theory of Usable Information under Computational Constraints. In ICLR, 2020.arxiv.org/abs/2002.10689
  37. [38]Qiyao Yan, Chenpeng Wang, and Liangming Pan. Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scores. arXiv:2608.31068, 2026.arxiv.org/abs/2608.31068
  38. [39]Fred Zhang and Neel Nanda. Towards Best Practices of Activation Patching in Language Models: Metrics and Methods. In ICLR, 2024.arxiv.org/abs/2309.16042
  39. [40]Genpei Zhang. Encoded but Disconnected: Decomposing Vision-Language Model Failures under a Patching Null. arXiv:2610.00024, 2026.arxiv.org/abs/2610.00024
  40. [41]Yingxuan Zhuang, Jingxiao Yang, Miao Pan, Cheng Tan, Yuxian Cai, Siwei Tan, Chen Zhi, Xuhong Zhang, Jianwei Yin, and Jintao Chen. Mitigating Manifold Departure: Uncertainty-Aware Subspace Rectification for Trustworthy MLLM Decoding. In ICML, 2026.arxiv.org/abs/2606.09859

Paper details

vlnc:W_xkkundae
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents