WorksComputer Science
Seen but Unsaid: Locating and Repairing Cross-Image Comparison in Vision-Language Models
Abstract
Vision-language models (VLMs) often fail to match a point or an object across two images, even when the answer can be decoded from their own hidden states. We trace this failure from the visual tokens to the answer letter, in Qwen3-VL (2B-32B), InternVL3.5-8B, and Gemma-3-12B, on synthetic shape matching, same-instance point correspondence (HPatches), and semantic correspondence (SPair-71k), and find two distinct failure points, a small set of attention heads that carries the comparison, and a repair that touches 0.02% of the parameters. Two failure points: exact copies are compared but not read out. An answer-position probe exceeds the calibrated output by 15 to 25 points in every family, for nameable and unnameable shapes alike, and steering along the probe's direction moves no more answers than random directions. Semantic correspondences are present in the visual tokens but never reach the answer. One pathway: redistributing the cross-image attention of 16-32 heads, with its mass preserved, removes 17-22 points of synthetic and 15-32 points of HPatches accuracy, against at most 7 for random or attention-matched heads. A lever that training finds on its own: training only these heads' query and key projections raises SPair-71k accuracy by 28.3 points (Qwen3-VL-8B) and 25.1 points (InternVL3.5-8B), beats both layer-matched and attention-matched heads, and transfers to HPatches and BLINK. An unconstrained all-layer adapter routes 21-51% of its gain through the attention of the same heads, against at most 7% for random heads, and does so by changing what they read (83-86% of the effect) rather than their weights. Every decision rule was frozen before its run, and every reported number is recomputed by independent verification scripts.
Introduction
Vision–language models (VLMs) answer open-ended questions about images, yet remain unreliable when a task requires matching a point, a shape, or an object across two images [6]. Many such failures do not originate in the vision encoder. VLMs underperform direct readouts of their own encoders on vision-centric tasks [5]; image value tokens inside the language model support semantic correspondence zero-shot even when the model’s answer is wrong [16]; counting fails even when numerical information remains linearly separable [24]; and visual evidence can be encoded about as strongly when the answer is wrong as when it is right [23]. VLMs often see more than they say.
For cross-image correspondence, Shahgir et al. [28] argue that this gap is widest for entities without names: on synthetic “squiggle” correspondence, Qwen3-VL-2B answers 29.0% correctly while a representation probe over its hidden states reaches 74.2%, and teaching arbitrary names narrows the gap. That readout compares hidden states at the annotated locations of the reference and the options, which the answering model must find itself. Such numbers are useful upper references, but they do not measure an input-matched gap, and they do not say where inside the language model the information is lost, nor what would recover it.
Open VLMs encode each image independently, so cross-image comparison must be computed inside the language model [34]. This makes the comparison a concrete, localizable computation: tokens of the second image must retrieve evidence from the first through attention. We follow the signal along this path (Fig. 1). We measure what is available at each stage with probes that receive only what the model receives, and compare every gap against a calibrated output. We then localize the components that carry the comparison, characterize what they attend to, and test the localization by repairing it: we train only these components, compare them against matched alternatives, and ask, through attention transplantation, whether unconstrained training discovers the same components on its own. Decision rules for every run were frozen in advance; the main claims are confirmed on fresh items.
Locating the loss matters because different loci call for different remedies. If the answer is computed but not emitted, a remedy must act on how the output reads it; if the information never reaches the answer position, no change to the output side can help, and the computation that brings it there must change instead. We find both cases, in different tasks, inside the same models.
The answer in brief. The comparison between two images runs through a small, identifiable pathway: 16–32 attention heads through which tokens of the second image retrieve evidence from the first. We establish this pathway with four kinds of evidence that do not depend on each other. Causal role: redistributing the heads’ crossimage attention with its mass preserved removes 15–32 of synthetic and same-instance accuracy, also on the items whose residual states stay in distribution, while random and attention-matched heads remove at most 7 . Specificity: the heads’ attention is selective for the reference, gated by its salience, and predicts on fresh items whether the comparison reaches the answer position. Sufficiency as a lever: training only their query and key projections recovers most of the semantic-correspondence deficit (+25 to +28 ) and beats heads matched in layer or in attention mass. Convergence: an adapter that may change every layer routes a third to a half of its gain through the same heads in Qwen3-VL-8B, by changing what they read; training other heads matched in attention mass raises the selectivity of these heads as well, and selectivity tracks accuracy across all fourteen trained models. The exact-copy failure, by contrast, lies downstream of this pathway: the comparison is computed and decodable at the answer position, yet the output does not read it, and the model’s own letter directions at that position, unlike the probe’s, move the answer.

Figure 1. Where correspondence information is lost. Top: the task. Each image is encoded separately; under causal attention, tokens of each option in image 2 attend back to salient candidate locations in image 1, and the true option (here B) attends more strongly to the reference. Bottom: the four measurements along this path for Qwen3-VL-8B on test items ( on fresh items). Exact copies are compared strongly and reach the answer position but are not emitted; semantic correspondences are present in the visual tokens but only weakly compared. Shading is a qualitative summary of the values shown. The cross-image heads are causal: redistributing their attention removes 17–32 pp of SYN and HP accuracy (§ 5.1). Training only their query–key projections raises SP accuracy from .43 to .71 (§ 6).
Our contributions are:
A measurement that separates seeing, comparing, and saying. Input-matched probes and a calibrated output baseline show that annotated-region readouts on synthetic matching measure pixel identity, and that the gap left after calibration is the same for unnameable and nameable shapes (§ 4.1).
Two distinct failure points in the same models, consistent across three families: exact copies are compared but not read out; semantic correspondences are present in visual tokens but do not reach the answer position (§ 4.2–§ 4.3).
The cross-image comparison pathway. A small set of heads whose attention is causal for exact-copy and same-instance correspondence under mass-preserving knockouts, also on the items whose residual states stay in the natural range, beyond random and attention-matched heads, and which the base model does not yet use for semantic correspondence—the gap that repair closes. Their salience-gated, content-weighted selectivity predicts on fresh items whether the comparison reaches the answer position (§ 5).
A repair at 0.02% of the parameters, on a pathway that fine-tuning finds on its own. Training only these heads’ query–key projections recovers most of the semantic-correspondence deficit and transfers to benchmarks never used for training. Attention transplantation and its decomposition show that an unconstrained adapter routes part of its gain through the same heads by a different mechanism—changing what they read rather than how they compare—so two independent routes to repair converge on one set of heads, a sufficient but redundant pathway (§ 6).
A verifiable protocol. Every run has frozen decision rules, every reported number is recomputed by independent verification scripts, and all outcomes, supported
or not, are listed (Appendix § 9, Appendix § 15); Table 7 maps each claim to its evidence and controls.
| Claim | Evidence | Controls | Where |
| Exact copies are compared but not read out | (Qwen3-VL-8B, Syn-squiggle); to in every family; answer-position edits move by at most ; probe-direction steering at the level of random directions | Output calibration; identical probe protocol on HP/SP (gap ); 20 magnitude-matched random subspaces; letter-direction positive control (switches –) | § 4.1–§ 4.2 |
| The gap is not about nameability | Gap difference squiggle vs. known: (Qwen3-VL-8B), (Qwen3-VL-2B) | Size- and stroke-matched nameable shapes | § 4.1 |
| Semantic correspondence is present in visual tokens | on SP: / , for Gemma-3-12B | Marker-free images; layers chosen on development data | § 4.3 |
| A small set of cross-image heads is causal | Mass-preserving knockout of : to on Syn, to on HP | 10 random and 3 attention-matched head sets (); in-band items; within-image-2 knockout | § 5.1 |
| Their attention is salience-gated and content-weighted | : HP / , SP / ; decoy markers raise to ; moved marker makes the new target selective () | Fresh items; head sets and rules frozen in advance | § 5.2 |
| Training these heads repairs semantic correspondence | SP / ; transfer to HP / and BLINK | Layer-matched random heads (A−B / ); attention-matched heads (A−M / ); query-only arms (A−B / , A−M / ) | § 6.1 |
| Broad adapters route gain through the same heads | Transplanting rows of D recovers – / – of its gain; / of it with base query–key weights | Five random head sets ( to ); residual norms in distribution | §6.3 |
| The pathway is redundant | Restoring rows in D removes – / – | Five random head sets | §6.3 |
Table 7. Claims, evidence, and controls at a glance. Numbers are for Qwen3-VL-8B / InternVL3.5-8B unless stated; pp are percentage points.
Related Work
Representation versus utilization. Benchmarks expose VLM failures on perception tasks that are easy for humans, including visual and semantic correspondence [6]. Fu et al. [5] show that encoder readouts outperform the VLM on vision-centric tasks and attribute the bottleneck to the language model’s use of visual information. Liu et al. [16] find that image value tokens inside the language model support segmentation and semantic correspondence zero-shot and that correct perceptual information is often present but not surfaced. Pang et al. [24] place the counting bottleneck in the mapping to number tokens, and Veikherman et al. [30] bound how often errors are encoded but unused. Shahgir et al. [28] link the gap to nameability, and SOCO reports a similar asymmetry between visual and textual queries on real images [29]; Qiu et al. [26] find that VLM judgments of visual sameness rely on semantic labels. We measure the gap with input-matched probes, find that it is not specific to unnameable entities, and identify the computation step at which it arises.
Readouts, probes, and calibration. Probe accuracy must be read against control tasks [10], and decodability differences across layers do not add up to information lost [37]. In text-only models, hidden-state probes can recover answers when sequence scoring collapses because of output biases [38], so a gap that output calibration closes is not evidence of a deeper failure. In VLMs, probe decodability and steering recovery can overstate grounding unless the image is ablated [14]. We report every gap against a calibrated output and include image-ablation controls.
Attention heads in VLMs and multiple-choice answering. Activation patching and mediation analysis [32, 20] have located where visual evidence gains or loses influence in single-image settings: arbitration between percept and prior [23], heads through which priors override vision [15], object localization [27], and visual entities resolved too late for factual recall [31, 3]. Specialized heads ground text to image regions [25], retrieve evidence in long multimodal contexts (Li et al., 2026), and, in Qwen3-VL-8B, track the region being described, so that redirecting their attention steers the description [7]. Multiple-choice answering involves symbol-producing heads [35] and a decide-then-bind sequence [36]. These studies concern text-to-image or within-image computation; we study region-to-region comparison between two images. Interventions have known artifacts: conclusions depend on the counterfactual and metric [39], subspace patching can activate dormant pathways [19], patched states can leave the natural distribution [9], and mid-layer interventions in VLMs can yield null results [40]. We pair each intervention with random-component controls and a residual-norm check.
Mechanism-guided repair. Training-free repairs gate causal routes [4], reweight attention [17], edit sensitivity-selected heads [12], or attenuate a language-prior subspace [41]. Parameter-efficient fine-tuning [11] has been restricted to visual tokens and selected attention paths [18] and to heads selected by their mechanistic role [22]. We restrict adaptation to heads selected by a causal comparison criterion, compare the result against head-matched and parameter-matched alternatives, and test whether unconstrained adaptation converges on the same heads.
Setup
Tasks. Every item is a four-way multiple-choice question over two images: image 1 contains a location labelled REF; image 2 contains four candidates labelled A–D, one of which corresponds to REF. In SYN-squiggle, shapes are closed cubic splines through anchors; SYN-known uses ten nameable outline classes matched in size and stroke. In both, image 2 contains the same four shapes as image 1 at new positions, identical up to translation, so every option has a copy in image 1 and only REF-specific comparison solves the task; within-item shape pairs are separated by a minimum Chamfer distance. HP is same-instance point correspondence generated from HPatches homographies [2]; SP is semantic key-point correspondence from SPair-71k [21]. Distractors are at least 8% of the image diagonal from the target and from each other. Splits are by seed range (SYN), sequence (HP), or official split (SP), with 1,000 development and 1,000 test items per task (500 each for SYN-known) unless stated. For transfer we also evaluate the BLINK semantic- and visual-correspondence validation splits [6].
Models. Qwen3-VL-2B / 4B / 8B / 32B [1], InternVL3.5-8B [33], and Gemma-3-12B [8]. Qwen3-VL uses SigLIP2-Large for 2B/4B and SigLIP2-SO-400M for 8B/32B, so scale comparisons are made only within encoder-matched pairs. For Qwen3-VL-8B we verified that perturbing image 1 leaves image 2’s encoder output unchanged, so cross-image comparison can occur only in the language model. Mechanistic analyses and repairs use Qwen3-VL-8B and InternVL3.5-8B.
Scoring. One forward pass returns the log-probabilities of the four letter tokens at the scoring position, with the final normalization and unembedding computed in fp32, which removes exact ties present in bf16. The answer format was selected on development data by a fixed rule (median letter mass and generation–argmax agreement ). Each item is scored under four cyclic letter orders that keep shape positions fixed.
Readouts. is behavioural accuracy. is accuracy after a multinomial logistic calibration of the four letter log-probabilities, fitted on development data. is an input-matched probe on the answer-position residual stream (PCA to 256 dimensions, multinomial logistic regression; layer and strength by five-fold cross-validation on development data). is a privileged readout over annotated regions: the maximum of a training-free MaxSim score and a paired probe (the latter counted only with selectivity over shuffled labels), with layers chosen on development data. Confidence intervals are 2,000-sample item bootstraps. Appendix § 10.1 collects all symbols.
Evaluation sets. Three disjoint groups of test items are used: the measurement test sets of § 4 (letter order 0), the fresh confirmation items of § 5.2, and 1,000 further SP test pairs for the repair experiments of § 6 (mean over the four letter orders, or order 0 where stated). Base accuracies therefore differ slightly between sections (Qwen3-VL-8B on SP: .438, .430, and .436); every comparison we report is paired within a single item set.
Protocol. Decision rules and thresholds were frozen before each run; amendments were made only before any result existed. Every reported number was recomputed from raw outputs, or, for refitted probes, asserted to reproduce stored per-item values, by verification scripts that do not import the experiment code (Appendix § 15). Region-to-token mappings were validated per family by a patch-level perturbation test (precision 1.0). Total compute was about 90 GPU-hours, mainly on a single RTX PRO 6000. Code, item generators, split identifiers, head indices, and trained adapters will be released.
Where the Signal Is Lost
Privileged Readouts and Nameability
Privileged readouts are saturated. On Syn-squiggle (test), Qwen3-VL-8B answers 41% correctly while reaches 64% and reach 1.00; Qwen3-VL-2B answers 31% with and . The privileged readouts are near ceiling already at the vision-encoder output, because the reference and the target are pixel-identical. On this task they measure pixel identity rather than anything the language model computes.
Nameability does not explain the gap. The gap is essentially the same for squiggles and nameable shapes: the difference is for Qwen3-VL-8B and for Qwen3-VL-2B. Nameability raises both and ; it does not change the distance between them.
The Qwen3-VL-8B gap survives calibration. Calibrating the letter log-probabilities leaves the Qwen3-VL-8B gap intact on both conditions, while it closes the small Qwen3-VL-2B gap on known shapes (Table 1). For known shapes, Qwen3-VL-8B names the ref shape and finds the letter of a named shape (letter binding 100%), yet answers the combined question correctly on 63% of items, below the product of the two step accuracies ().
| Model | Condition | (pp) |
| Qwen3-VL-2B | squiggle | [, ] |
| Qwen3-VL-2B | known | [, ] |
| Qwen3-VL-8B | squiggle | [, ] |
| Qwen3-VL-8B | known | [, ] |
Table 1. Calibration does not close the Qwen3-VL-8B gap. on SYN test, order 0, with paired 95% CIs.
Breadth: The Readout Gap Is Specific to Exact Copies
Across families the calibration-robust gap on Syn-squiggle persists ( to in all models except Qwen3-VL-2B). On real correspondence it does not (Table 2): calibration changes by at most 2.6 pp across all six models, and is within a few points of in every model. HP is solved well above chance (answer-position probe 0.54–0.86), whereas SP is near chance both at the answer position and in the output. With both images replaced by grey, the transferred probe stays at or below 0.315. Within encoder-matched pairs, the SYN gap grows from 2B to 4B and shrinks from 8B to 32B (where known shapes reach ceiling); it does not change with scale on HP or SP.
| Model | Task | [CI] | ||||
| Qwen-8B | HP | .770 | .778 | .804 | +2.6 | [+0.6, +4.8] |
| SP | .438 | .460 | .433 | −2.7 | [−6.1, +0.7] | |
| InternVL-8B | HP | .543 | .536 | .542 | +0.6 | [−2.3, +3.2] |
| SP | .355 | .369 | .310 | −5.9 | [−9.7, −2.1] | |
| Gemma-12B | HP | .590 | .591 | .601 | +1.0 | [−1.7, +3.7] |
| SP | .407 | .404 | .351 | −5.3 | [−8.8, −1.8] |
Table 2. Real correspondence shows no hidden answer-position gap. Test, order 0, .
The gap is not a property of the probe. Every task and model uses the same probe architecture, dimensionality, and model-selection protocol; the probe is fitted per task, with its layer and regularization chosen on that task’s development data. If the extra capacity of this probe family relative to a calibrated four-way output were enough to create a gap, the gap would appear everywhere. It does not: on HP and SP the probe exceeds by at most 2.6 pp, and falls below it in three of six model–task cells, and on Syn-known for Qwen3-VL-2B calibration closes the gap entirely, while for Qwen3-VL-8B the same protocol exceeds by about 20 pp on both SYN conditions. The SYN gap is therefore specific to exact copies in the models that compare them strongly (§ 5.2), not a by-product of the readout.
The decoded answer is not on the output path. To test whether the output uses the answer-position signal, we edit the residual stream at the answer position, at the probe’s layer, along the probe’s own three-dimensional subspace : we remove it or scale it by , and compare against 20 random subspaces matched in edit magnitude. No label information enters the edit, and the rules were frozen in advance. Neither edit moves the answer. On SYN-squiggle and on HP, in both Qwen3-VL-8B and InternVL3.5-8B, changes by at most 0.2 pp, within the range of the random subspaces, and residual norms stay in the natural band (Appendix § 13).
The layer itself is not inert. Adding, at the same position and layer, the output direction of a letter the model did not choose switches the answer to that letter on 2.8–3.8% of SYN and 9.6–18.6% of HP items at eight times the probe subspace’s natural scale, against at most 0.4% and 1.8% for ten random directions of equal norm. Steering along the probe’s own class direction, toward a class it did not predict, moves at most 1.6% of the items, within the range of the random directions (Appendix § 14). The answer that the probe reads at the answer position is thus not the one the model reports. On HP, where the output is mostly correct, the model computes it through other channels; on SYN, the decodable comparison signal sits beside the output path and is not read out, in line with reports that decodable signals in VLMs need not be causal for the answer ([40, 30]). Amplifying it at a single layer does not recover it, so the loss is not a weak linear readout that more gain would fix.
Semantic Correspondence Is Present in the Visual Tokens
We compute the privileged readout over -pixel regions centred on each point, on images without drawn markers. Table 3 reports . On SP, the region readout exceeds the answer-position probe in all three models: by +16.3 pp for Qwen3-VL-8B and +25.9 pp for InternVL3.5-8B, both above the pre-registered 15 pp threshold, and by +14.3 pp for Gemma-3-12B. In no cell is attained only at the vision-encoder output, so the language model’s own visual tokens carry the correspondence, in line with Liu et al. [16]. Selecting the readout layer on the test set instead of development data changes by 0–1.8 pp, with every interval including zero.
| Model | SYN-squiggle | HP | SP |
| Qwen3-VL-8B | +36.0 | +5.8 | +16.3 |
| InternVL3.5-8B | +25.8 | +37.0 | +25.9 |
| Gemma-3-12B | +35.1 | +18.1 | +14.3 |
Table 3. Region readout minus answer-position probe, in pp (images without markers, test).
Marker control. When everything except the drawn markers is replaced by grey, the region readout falls 5.8–16.1 pp below its value on marked images in five of six model-task cells; only InternVL3.5-8B on HP is largely marker-driven (0.780 versus 0.804). Table 3 therefore uses images without markers.
How the Two Images Are Compared
Cross-Image Heads
Image-1 tokens precede image-2 tokens, so under causal attention a cross-image comparison must run from image-2 (option) queries to image-1 keys, or through later positions. We scored each head by how strongly option tokens attend to their corresponding region in image 1 on SYN development items and blocked the image-2image-1 attention of the top- heads, with chosen on development data. This gives a set of 32 heads in layers 1–24 of Qwen3-VL-8B and a set of 16 heads in layers 3–23 of InternVL3.5-8B; we write for either.
Causal effect. On the 1,000 SYN-squiggle test items, blocking this attention lowers by 15.2 pp for Qwen3-VL-8B and 24.2 pp for InternVL3.5-8B, more than the 95th percentile of 20 layer-matched random head sets, while blocking the same heads’ attention within image 2 has a much smaller effect.
Mass-preserving knockouts. Blocking removes attention mass, and for Qwen3-VL-8B 53% of the blocked items leave the natural residual-norm range (1st–99th percentile of the base model; InternVL3.5-8B: 1.9%). We therefore repeated the knockout with two operations that keep the total attention of each image-2 query on image 1 and change only where it goes: Uniform spreads it evenly over the image-1 tokens, and Shuffle permutes it among them. Each operation was applied to , to ten layer-matched random head sets, and to the three attention-matched head sets of § 6.1 (Table 4). On SYN and HP, both operations lower by 15–32 pp and by 20–48 pp in both models; for every operation and both measures, ranks first against the ten random sets, and its drop exceeds that of all three attention-matched sets, none of which lowers by more than 6.8 pp. Both operations keep 94–98% of the SYN items in the norm band, and restricted to these items the drops are unchanged (18.0 and 19.3 pp for Qwen3-VL-8B). Under Block, by contrast, the Qwen3-VL-8B drop on in-band items is only 3.1 pp: the earlier Qwen3-VL-8B effect was carried by off-distribution items, and the mass-preserving knockouts replace it as the causal evidence. On HP, 6–21% of items leave the band, above the frozen 5% limit, but the in-band drops are within 2 pp of the overall drops. On SP, the knockout has little effect (2.2–3.6 pp for Qwen3-VL-8B, none for InternVL3.5-8B): the base models’ above-chance semantic answers hardly run through these heads. That is where the repair of § 6 acts.
| Task | Base | Uniform | Shuffle | Block | Ctrl |
| Qwen3-VL-8B | |||||
| Syn | 41.2 | 17.2 (18.0) | 18.2 (19.3) | 17.4 (3.1) | 3.6 |
| HP | 76.2 | 32.2 (31.8) | 31.8 (31.4) | 21.2 (20.9) | 2.4 |
| SP | 46.6 | 2.2 (2.4) | 3.2 (3.7) | 3.6 (3.8) | 2.0 |
| InternVL3.5-8B | |||||
| Syn | 48.6 | 20.2 (19.9) | 22.0 (22.0) | 22.4 (22.2) | 6.8 |
| HP | 53.6 | 16.8 (16.2) | 14.6 (12.6) | 16.2 (14.9) | 4.2 |
| SP | 34.8 | −0.8 (−0.8) | −0.6 (−0.6) | −0.6 (−0.6) | 1.0 |
Table 4. Knockouts of the image-2→image-1 attention of . Drop in (pp) on 500 test items per task; in parentheses, the drop on items whose residual norm stays in the base band. Uniform and Shuffle preserve attention mass; Block removes it (on these 500 items; the 15.2 and 24.2 pp above are on all 1,000 Syn test items). Ctrl: largest drop of any of ten random and three attention-matched head sets under any operation.
Where the heads look. On HP and SP, the true option’s attention peaks on the REF region in 94–100% of items, but distractors also attend mostly to REF rather than to their own correspondence, so per-option retrieval accuracy is 0.28–0.33. Because REF is the only marked location in image 1, we tested selectivity and salience directly.
Content-Selective, Salience-Gated Attention
On fresh items never used before (600 Syn, 474 HP, 400 SP), with head sets frozen from the knockout experiment and rules frozen before any data, we measure, averaged over the heads in , the attention mass from option to the ref region, the selectivity , and the ratio of means .
Selectivity, its drop, and prediction. In both models the true option attends more to the reference than distractors do ( on HP and SP); selectivity is lower on SP than on HP, in both the difference and the ratio (Figure 2); and predicts whether the answer-position probe is correct, with AUC 0.62–0.70 on HP (Table 5). Removing items whose correspondence regions overlap leaves these values essentially unchanged.
| Model | Task | [95% CI] | AUC | |
| Qwen-8B | Syn | +.252 [.244, .260] | 10.36 | .48 |
| HP | +.051 [.046, .056] | 1.88 | .70 | |
| SP | +.011 [.008, .013] | 1.30 | .66 | |
| InternVL-8B | Syn | +.155 [.148, .163] | 2.88 | .69 |
| HP | +.046 [.035, .057] | 1.32 | .62 | |
| SP | +.009 [.003, .016] | 1.13 | .56 |
Table 5. Selectivity of cross-image heads on fresh test items. AUC uses as a score for answer-position probe correctness.
Figure 2. Selectivity falls from copies to semantic correspondence. Ratio of the true option’s attention on the reference to that of distractors, averaged over , on fresh test items with 95% intervals. The SP−HP difference in is −0.040 [−0.046, −0.034] for Qwen3-VL-8B and −0.036 [−0.049, −0.024] for InternVL3.5-8B.
Salience gating. When identical unlabelled markers are drawn at the distractors’ correspondence locations in image 1, distractors shift their attention to their own, now marked, locations (ratio of attention on REF to attention on the own location: 0.19–0.77), and selectivity remains positive and becomes larger ( on HP for Qwen3-VL-8B). Without such markers, distractors place almost no attention on their own correspondence (0.011–0.017). Attention thus goes to salient candidate locations in image 1 and, among them, is weighted by content similarity.
Selectivity follows content. Moving the REF marker to a distractor’s correspondence location, with image 2 unchanged, makes the new target selective ( on HP for Qwen3-VL-8B), while the old target keeps a smaller residual (+0.009), consistent with content-driven attention to its now unmarked correspondence. The pre-registered switch criterion required this residual to be non-positive and was therefore not met.
Sharpening is not a repair. Multiplying the image-2→image-1 attention logits of by does not raise calibrated accuracy on HP or SP in either model, and on Syn it lowers the answer-position probe. If the comparison stage limits semantic correspondence, it is the similarity that these heads compute, not their temperature, that must change. This is what we test next.
Repairing the Comparison
If the cross-image heads limit semantic correspondence, changing what they compare should help, and should help more than changing other components of similar size. We test this with adapters restricted to and with matched alternatives, and then ask whether adapters that are free to change any layer end up changing the same heads.
Targeted Adapters
Arms. All arms share one recipe and differ only in their trainable parameters, with base weights frozen. A trains rank-16 LoRA updates [11] on the query and key projections of the heads in (for grouped-query attention, the key update acts on the key–value group of each selected head): 1.68M parameters for Qwen3-VL-8B and 1.50M for InternVL3.5-8B. trains the query updates only (0.85M / 0.75M); because query projections are separate per head, changes the computation of the selected heads and of no other head. B applies arm A to three layer-matched random head sets of the same size. C trains the value projections of with nearly the same parameter count as A (1.65M). D is a standard LoRA on all attention projections of all layers at rank 2 (1.92M parameters, the closest match to A). E is a rank-16 LoRA on all attention projections of all layers (15.3M), a common default that is not parameter-matched. C and E were run for Qwen3-VL-8B only.
Training and evaluation. Training uses 2,000 items from the SPair-71k training split in the same four-option format, cross-entropy on the correct letter at the scoring position, AdamW for two epochs with batch size 8 and a cosine schedule; the learning rate is chosen per arm from on 400 validation items. A, C, and D use three seeds. The primary test set is 1,000 SP test pairs never used in earlier runs, scored as the mean over the four letter orders. Transfer is measured on the fresh HP items of § 5.2 and on BLINK semantic and visual correspondence, none of which is used for training or selection; the two SYN conditions serve as collateral checks.
Training the comparison heads repairs most of the deficit. Arm A raises SP accuracy by +28.3 [25.3, 31.1] for Qwen3-VL-8B (.430→.712) and by +25.1 [22.4, 27.8] for InternVL3.5-8B (.354→.605), with about 0.02% of the model’s parameters (Table 6). With only 500 training items the gains are still +23.9 and +19.0 . The query update alone (A), which touches no head outside , obtains 86–90% of the effect (+24.4 and +22.7 ), so the repair does not depend on keys shared with other heads. The gains transfer without any training on the target benchmarks: on HP, A improves by +15.2 (Qwen3-VL-8B) and +30.6 (InternVL3.5-8B); on BLINK semantic correspondence by +18.2 and +5.8 (the latter with an interval including zero); and on BLINK visual correspondence by +13.4 and +12.6 . For Qwen3-VL-8B, A also improves both SYN conditions.
| Arm (trainable parameters) | Params | SP | HP | BLINK-SC | BLINK-VC | SYN-squiggle | SYN-known |
| QWEN3-VL-8B | |||||||
| base (none) | – | .430 | .782 | .367 | .669 | .414 | .686 |
| A: QK of | 1.68M | .712 | .934 | .549 | .802 | .551 | .833 |
| A: Q of | 0.85M | .674 | .922 | .504 | .791 | .480 | .842 |
| B: QK of random heads | 1.69M | .677 | .899 | .499 | .808 | .451 | .701 |
| C: V of | 1.65M | .747 | .925 | .561 | .876 | .488 | .785 |
| D: all layers, rank 2 | 1.92M | .792 | .953 | .655 | .899 | .402 | .806 |
| E: all layers, rank 16 | 15.3M | .790 | .958 | .655 | .907 | .451 | .911 |
| INTERNVL3.5-8B | |||||||
| base (none) | – | .354 | .531 | .309 | .436 | .526 | .926 |
| A: QK of | 1.50M | .605 | .837 | .367 | .562 | .366 | .776 |
| A: Q of | 0.75M | .581 | .836 | .410 | .558 | .315 | .784 |
| B: QK of random heads | 1.51M | .584 | .829 | .369 | .622 | .474 | .906 |
| D: all layers, rank 2 | 1.92M | .728 | .941 | .614 | .845 | .492 | .993 |
Table 6. Repair arms. Accuracy (mean over four letter orders; seeds averaged) after training on 2,000 SP items. Only SP is in-domain; HP and BLINK measure transfer, SYN measures collateral effects. Best value per column and model in bold.
Location matters, but is not the whole story. Adapting the comparison heads beats adapting the same number of random heads in the same layers: A−B is +3.6 [2.0, 5.2] for Qwen3-VL-8B and +2.1 [0.8, 3.4] for InternVL3.5-8B, and the intervals stay above zero when the random head sets are resampled together with the items ([1.4, 6.0] and [0.2, 4.1]). The advantage holds, and grows for InternVL3.5-8B, when only query projections are trained, so that no update can reach a head outside the chosen set: A beats query-only training of the same random sets (B) by +3.5 [1.6, 5.5] and +4.5 [2.1, 7.0] (hierarchical intervals). Query updates of random heads leave the selectivity of almost unchanged (–.016 and .010–.011, against .034 and .029 for A).
Attention mass does not explain the advantage. Random heads in the same layers send only 32–47% as much attention across images as , so A−B could reflect access to the other image rather than the role of the heads. We therefore built three further head sets per model, drawn from any layer and matched to head by head in cross-image attention mass on SYN development items (80–99% of the mass of ; 1.3–1.7 times as much on SP), while excluding the heads with the highest correspondence scores. We trained them with the recipes of A (arm M) and A (arm M), with every matched arm having more trainable parameters than its reference. Matched heads do about as well as random ones (SP .679 vs. .677 for Qwen3-VL-8B and .580 vs. .584 for InternVL3.5-8B), and beats them: A−M is +3.3 [1.6, 5.1] and +2.5 [0.8, 4.2], and A−M is +2.1 [0.3, 4.0] and +4.3 [2.4, 6.4] (hierarchical intervals; Appendix § 14). Neither layer (B) nor cross-image attention mass (M) accounts for the advantage of ; what sets these heads apart is what they attend to. The matched arms also locate the synthetic-task cost of the targeted InternVL3.5-8B repair: training M leaves SYN-squiggle and SYN-known at .473 and .912, whereas training lowers them to .366 and .776 (Table 6). Most of the cost comes from retraining these particular heads for semantic correspondence, as expected if the same heads also carry exact-copy comparison.
The parameter-matched all-layer adapter D is stronger than A, by 8.0 (Qwen3-VL-8B) and 12.3 (InternVL3.5-8B), and adapting the value projections of the same heads (C) is 3.5 stronger than adapting their query–key projections in Qwen3-VL-8B. The comparison heads are thus an effective but not an optimal place to intervene: a distributed adapter can also change the features being compared, which a query–key update cannot.
The Repair Increases Selectivity
Arm A raises the selectivity of on SP from .0107 to .0426 in Qwen3-VL-8B (rise +.032 [.029, .035]) and from .0084 to .0298 in InternVL3.5-8B (rise +.021 [.016, .027]), more than twice the rise under the random-head arm B (+.014 and +.007), although A and B reach similar accuracy. Across the base model and all trained arms, selectivity measured over the fixed heads orders SP accuracy almost perfectly (Figure 3a; Spearman 0.96 over seven Qwen3-VL-8B models and 0.90 over five InternVL3.5-8B models). The relation holds for the attention-matched arms M and M of § 6.1, which were trained after it was observed: with them, Spearman is 0.93 over nine and 0.96 over seven models. This includes D and E, which can change any layer: whatever else they do, they also make the comparison heads more selective. Training the attention-matched heads M, which leaves the weights of untouched, also raises the selectivity of (to .019–.029 in Qwen3-VL-8B and .012–.017 in InternVL3.5-8B, rises of +.008 to +.019 and +.004 to +.009), though less than A does. Because selectivity is read from the attention of a fixed head set, it needs neither a probe nor further training, which makes it a cheap check of whether a candidate repair acts on the comparison step. Zeroing the A adapters on half of lowers SP accuracy by 5.8–9.4 in Qwen3-VL-8B and 4.2–6.1 in InternVL3.5-8B over three random halves, and by 6.0 and 10.7 for the half with the highest retrieval score, so the repair is spread across the set rather than carried by a few heads.

Figure 3. The repair runs through the comparison heads. (a) Each point is one trained model (seed-averaged); selectivity is measured over the fixed heads as in § 5.2. Open markers: A and E. Diamonds: attention-matched arms M and M (mean over three head sets), trained after this relation was first observed. Accuracy rises with selectivity in both families, including for the all-layer adapters D and E, which were never told about . (b) Order-0 SP test, seed 0. Filled bars: fraction of each arm’s gain obtained by inserting only the image-2 attention rows of from the trained model into the base model. Open bars: fraction lost by restoring those rows to the base model inside the trained model. Grey band: range of both fractions over five layer-matched random head sets, across arms. Arms whose training changed other heads (B) or only values (C) are hardly recovered by transplanting attention.
Does Unconstrained Training Converge on the Same Heads?
The arm-level pattern in Figure 3a was observed after the fact. We tested it with rules frozen before any data. For a trained arm with gain over the base model, we manipulate only the post-softmax attention rows of the heads in for image-2 query positions, leaving values, other heads, and all other positions untouched. Transplant runs the base model with these rows taken from the trained model on the same input; the recovered fraction is its accuracy gain over base divided by . Restore runs the trained model with these rows taken from the base model; the removed fraction is its accuracy loss divided by . Each operation is repeated on five layer-matched random head sets. The frozen criteria call a head set sufficient (or necessary) if the recovered (or removed) fraction is at least 0.25, its numerator is positive under the bootstrap, and it exceeds all five random sets. The decisive case is arm D, which never targeted .
The comparison heads are sufficient. Transplanting the attention rows of the all-layer adapter D into the base model recovers .345 [.273, .413] of its +34.8 pp gain in QWEN3-VL-8B, and .466 and .514 for two further training seeds; for INTERNVL3.5-8B it recovers .284, .295, and .213 over the three seeds (seed-0 gain +37.0 pp). The rank-16 adapter E gives .404 in QWEN3-VL-8B. Random head sets recover between −0.04 and +0.07 (Figure 3b). In accuracy terms, for the arms that meet the sufficiency criterion and all their seeds, the base model with transplanted rows reaches .549–.619 on SP (Qwen3-VL-8B, base .436) and .425–.463 (InternVL3.5-8B, base .354), against .42–.46 and .35–.37 for random transplants. Transplanted states stay in distribution: 1–5% of items fall outside the base residual-norm band, against 2% for the base model itself. The effect extends to HP, where transplanting the rows of D recovers .612 (Qwen3-VL-8B) and .481 (InternVL3.5-8B) of D’s gain, and those of E recover .789. For seed 0, the sufficiency criterion holds for A, D, and E in Qwen3-VL-8B and for A and D in InternVL3.5-8B, and fails, as expected, for B and C, whose training changed other heads or only values.
What changes is what the heads read. An attention row depends on the head’s query–key weights and on the residual stream it reads. We split Transplant accordingly. Upstream computes the rows of with the base model’s query–key weights applied to the trained model’s inputs. Local applies the trained weights to the base model’s inputs. Each set of rows is then transplanted into the base model as before (Table 18). For D on SP, Upstream recovers .282 [.207, .348] of the gain in Qwen3-VL-8B and .233 [.166, .298] in InternVL3.5-8B, 83% and 86% of the full transplant measured in the same run, whereas Local recovers .043 and .035 with intervals reaching zero, and the interaction is near zero. E and the HP items show the same ordering, and no random head set recovers more than 7.4% of the gain on SP under any operation. The all-layer adapter thus routes its gain through the comparison heads not by rewriting them but by changing what they read: the heads’ own weights, applied to better inputs, already produce most of the transplanted effect.
Table 18. Recovered fraction of each arm’s gain under transplanted rows (95% intervals). Interaction = FULL − UPSTREAM − LOCAL. Rank: position of among and five random sets (FULL / UPSTREAM / LOCAL).
| Model | Arm | Split | Gain (pp) | FULL | UPSTREAM | LOCAL | Interaction | Rank |
| QWEN3-VL-8B | D | SP | 34.8 | .339 [.268, .404] | .282 [.207, .348] | .043 [−.013, .095] | +.014 [−.059, .088] | 1 / 1 / 1 |
| D | HP | 16.2 | .585 [.417, .759] | .692 [.550, .857] | .077 [−.062, .207] | −.185 [−.339, −.034] | 1 / 1 / 1 | |
| E | SP | 35.3 | .411 [.340, .484] | .360 [.291, .435] | .028 [−.012, .066] | +.023 [−.031, .078] | 1 / 1 / 2 | |
| E | HP | 16.8 | .806 [.672, .953] | .791 [.646, .962] | .090 [.000, .190] | −.075 [−.213, .061] | 1 / 1 / 2 | |
| INTERNVL3.5-8B | D | SP | 37.4 | .270 [.201, .339] | .233 [.166, .298] | .035 [.000, .068] | +.003 [−.041, .054] | 1 / 1 / 1 |
| D | HP | 39.0 | .462 [.366, .558] | .442 [.344, .539] | .026 [−.029, .077] | −.006 [−.078, .069] | 1 / 1 / 1 |
Table 18.
But they are not necessary. Restoring the base attention rows inside D removes only .161 [.101, .220] of its gain in Qwen3-VL-8B (.133 and .166 for the other seeds) and .022–.049 in InternVL3.5-8B. The pre-registered necessity criterion for D is therefore not met in either model, and the frozen summary is not supported. For arm A, which changes only these heads’ queries and the keys of their groups, restoring the image-2 rows removes .347–.415 in Qwen3-VL-8B over three seeds but only .053 in InternVL3.5-8B. Restoring the heads’ attention at all query positions removes .746 [.671, .823] and .370 [.248, .479] of A’s gain (seed 0), more than any of five random head sets (at most .139 and .004), so arm A meets the necessity criterion in both models; restoring every query head that shares their keys removes A’s gain exactly, as it must. For D, restoring all positions removes .198 [.137, .260] and .080 [.029, .135]—more than any random set (at most .063 and .037), but far below the criterion. The residual-norm check is uninformative for Restore, because the trained models themselves lie outside the base band.
Interpretation. The improvement that an unconstrained adapter learns contains, as one of its parts, a change in how the comparison heads attend across images; this part alone carries a third to a half of the gain in Qwen3-VL-8B, and it is produced upstream of the heads: the change in their own query–key weights contributes little on its own. The trained models also carry the improvement through other routes, which compensate when the comparison heads are reset. The targeted repair is carried by the comparison heads; the unconstrained repair uses them but does not depend on them. The comparison heads are thus a sufficient but redundant pathway: a point through which gradient descent repairs semantic correspondence on its own, but not a bottleneck through which every repair must pass.
Discussion
Two failure points and one lever. For exact copies, the comparison succeeds—selectivity is high and the answer position carries the result—but the output does not use it: neither calibration nor amplifying the decoded signal recovers it. For semantic correspondence, the visual tokens carry the correspondence, but the cross-image heads compare reference and candidates only weakly, the base model’s semantic answers barely depend on them, and the result does not reach the answer position. Same-instance correspondence lies in between and is largely solved. The repair experiments close the loop for the semantic case: changing only the similarity computed by the comparison heads recovers most of the deficit, raises their selectivity, and transfers to unseen correspondence benchmarks; and training that is free to change anything still changes these heads’ attention, by changing what they read, in a way that is sufficient to reproduce a large share of its gain. Training turns heads that the base model does not use for semantic correspondence into a route for it.
Relation to prior accounts. Large gaps measured with annotated-region readouts are reproduced by our privileged readouts, but on real correspondence they reflect information that the language model does not bring to the answer, rather than an answer that is computed and withheld. This picture does not depend on whether the entities can be named. It agrees with the observation that perceptual information inside the language model often goes unused ([5, 16]), and it adds the step at which this happens for cross-image tasks and a component that can be trained to change it. Like gaze heads for single-image description ([7]), the comparison heads are a small set whose attention controls what reaches the answer; unlike them, they compare across images, and they are a sufficient but redundant route rather than a unique one.
Why the all-layer adapter wins. A query–key update can only change which existing features the heads match. Sharpening the same heads does not help, while learning their similarity does, so the comparison function is a limiting factor. The parameter-matched all-layer adapter also changes the features themselves and the path from the comparison to the answer; that it outperforms the targeted arm, and that restoring the comparison heads inside it removes little of its gain, both indicate that the comparison is one of several limiting steps. The decomposition of §6.3 makes the first point concrete: the part of D’s gain that passes through comes from changed inputs to the heads, which no update of the heads’ own projections can produce. A practical consequence is that the comparison heads are a good diagnostic for repairs—their selectivity tracks accuracy across every arm we trained—even where they are not the most efficient target.
Implications. Three practical points follow. First, the locus decides where a remedy must act: for exact copies the comparison is already computed, so the target is how the answer is read out, where neither calibration nor single-layer amplification suffices; for semantic correspondence the target is the comparison itself, and that is where training succeeds. Second, the comparison heads are an inexpensive training target: about 0.02% of the parameters, to on SP from only 500 training items, and transfer to benchmarks never used for training or selection. Third, because broad fine-tuning routes a third to a half of its SP gain through the same heads in Qwen3-VL-8B, their selectivity offers a way to inspect what a fine-tuned model has changed, without retraining or probing.
Conclusion
Input-matched, calibration-controlled measurement shows that the representation–output gap in cross-image correspondence is not a property of unnameable entities. Exact-copy matching is computed and lost at readout; semantic correspondence is encoded in visual tokens but only weakly compared by a small set of salience-gated, content-selective cross-image heads. Training only these heads recovers most of the semantic-correspondence deficit and transfers to other benchmarks, and broad adapters route part of their gain through them by changing what they read: the comparison heads are a sufficient but redundant pathway for repair. Locating the loss in this way separates what a model sees from what it compares and from what it says. It turns that separation into a small, verifiable target for training and gives a cheap readout, the selectivity of a fixed head set, for checking what fine-tuning has changed.
Limitations
Mechanistic results and repairs cover two families at the 8B scale; Gemma-3-12B enters the measurement results only. Blocking the attention of leaves the natural residual-norm range for about half of the Qwen3-VL-8B synthetic items; the mass-preserving knockouts keep 94–98% of Syn items in range, but on HP 6–16% of items leave it (21% under Block), although restricting to in-range items leaves the drops nearly unchanged. In the base models, knocking out has little effect on SP, so causal evidence for semantic correspondence comes from the trained models. The attention-matched control heads are not layer-matched, and three of their four learning-rate sweeps selected the largest rate of the grid, as did A in InternVL3.5-8B (AQ reuses A’s rate in both models). Attention mass describes where heads look; causal evidence comes from knockouts and transplants. Markers make the reference salient by design; the decoy-marker experiment addresses this, but natural queries specify the reference in other ways. The targeted repair is below the parameter-matched all-layer adapter, and in InternVL3.5-8B it lowers accuracy on both synthetic conditions, which the all-layer adapter does not. Answer-position edits show that the probe’s direction is not the one the output reads, while the output direction of each letter at the same layer is; they do not identify the channel through which the comparison is lost. SOCO could not be evaluated; the BLINK splits contain fewer than 200 items each, so their intervals are wide. Synthetic results use one stimulus family and one answer format per model, and the answer format differs from the prefill used by Shahgir et al. [28], so absolute accuracies are not directly comparable to theirs. Natural extensions include head sets matched to in layer and attention mass at once, and general-capability evaluations after targeted repair.
Ethical Considerations
This work analyses publicly released models on public or synthetic data. The adapters improve a perceptual capability on a narrow task family and introduce no new capability class. Head-targeted training changes model behaviour in ways that general evaluations may not reveal, as the synthetic-task drop under the targeted InternVL3.5-8B adapter shows; any such repair should be evaluated for unintended effects on unrelated behaviour before deployment.
Experimental Chronology and Frozen Outcomes
Each run fixed its decision rules, thresholds, item sets, and statistical tests before execution; the only amendments were made before any result of that run existed. Table 8 lists the runs in order with their frozen outcomes. Outcomes that did not support a hypothesis are reported alongside those that did.
| Run | Question | Frozen outcome |
| Readout ladder | Where does the SYN gap fall (readout vs. retention/computation)? | QWEN3-VL-8B: readout-failure candidate; QWEN3-VL-2B: retention/computation candidate (within 0.2 pp of the rule boundary). |
| Gap diagnostics | Is the gap specific to unnameable shapes? Letter or position coding? Probe saturation? Composition? | Gap not nameability-specific in either model; mixed letter/position coding; QWEN3-VL-8B probe not saturated; QWEN3-VL-8B composition gap. |
| Calibration | Does output calibration close the gap? | QWEN3-VL-8B: gap persists on both conditions; QWEN3-VL-2B: partial (squiggle), closed (known). |
| Breadth and scale | Is the gap general across tasks and families? | Not general across tasks: present on SYN-squiggle in every family, absent on HP and SP. |
| Privileged readout, real images | Is correspondence information present in visual tokens? | SP: present in two of three models (third 0.7 pp below threshold); no model without information. |
| Cross-image heads | Do causal heads exist; does per-option retrieval dissociate across tasks? | Heads causal in both models; per-option retrieval criteria not met (mixed). |
| Attention analysis | Where do option tokens attend? | Exploratory: attention concentrated on REF, positive selectivity, lower on SP. |
| Fresh-item confirmation | Are selectivity, its drop, and its predictive value confirmed? | Selectivity, drop, and prediction confirmed in both models; switch criterion not met; sharpening does not help. |
| Targeted repair | Does training only the query–key projections of repair SP, and through these heads? | Helps (both models); beats random heads (QWEN3-VL-8B; INTERNVL3.5-8B +2.1 pp against a 3 pp threshold, interval excluding zero); selectivity rise exceeds random heads (both); transfers (both); does not match the parameter-matched all-layer adapter (both); value arm stronger than query–key arm (QWEN3-VL-8B); not data-efficient relative to the all-layer adapter at 500 items; synthetic collateral drop in INTERNVL3.5-8B. |
| Convergence | Is each arm’s gain carried by the attention of ? | Sufficient for A, D, E (QWEN3-VL-8B) and A, D (INTERNVL3.5-8B); necessary for A in QWEN3-VL-8B only; summary for D: not supported (necessity). |
| Additional controls | Does the output use the answer-position signal? Is the repair query-localized? Is A’s gain carried by at all positions? | Answer-position edits change no output on either task (underweighted-readout and used-on-HP criteria not met; unused-on-SYN met); query-localized (both); full-position restore necessary for A (both), not for D (both). |
| Matched controls and decomposition | Is the advantage of explained by cross-image attention mass? Is the answer-position layer causal for the output? Do transplanted rows owe their effect to inputs or to weights? Do mass-preserving knockouts reproduce the causal effect? | Localized beyond attention mass (both); strong-localization 3 pp threshold not met (A−M +2.1 pp in QWEN3-VL-8B, A−M +2.5 pp in INTERNVL3.5-8B); layer causal (both); probe direction does not steer, no HP/SYN dissociation (both); upstream-dominant (both); knockout causal and beyond attention-matched on SYN and HP for every operation (both), on SP for SHUFFLE and BLOCK in QWEN3-VL-8B only; in-distribution criterion not met (HP 5.6–20.6% outside the band). |
Table 8. Chronology of pre-registered runs and their frozen outcomes.
Notation {#sec:B}
Table 8.
Notation
| Symbol | Meaning |
| Behavioural accuracy: argmax over the four letter log-probabilities at the scoring position. | |
| Accuracy after multinomial logistic calibration of the four letter log-probabilities, fitted on development data. | |
| Input-matched probe on the answer-position residual stream (PCA to 256 dimensions, multinomial logistic regression). | |
| , | Privileged readouts over annotated regions: training-free MaxSim score () and paired probe (). |
| , with counted only if its selectivity over shuffled labels is at least 5 pp. | |
| Cross-image head set: (32 heads, layers 1–24) for Qwen3-VL-8B, (16 heads, layers 3–23) for InternVL3.5-8B. | |
| Attention mass from the tokens of option to the REF region, averaged over the heads in . | |
| , | Selectivity: of the target minus the mean over distractors, and the ratio of the two means. |
| A, | LoRA on the query–key (A) or query-only () projections of . |
| B, | As A and , on three layer-matched random head sets of the same size. |
| M, | As A and , on three head sets matched to in cross-image attention mass. |
| C | LoRA on the value projections of . |
| D, E | LoRA on all attention projections of all layers, rank 2 (parameter-matched to A) and rank 16. |
| Transplant, Restore | Insert the trained model’s image-2 attention rows of a head set into the base model, or the base model’s rows into the trained model (§6.3). |
| Upstream, Local | Transplanted rows computed with base query–key weights on trained inputs, or with trained weights on base inputs. |
| Uniform, Shuffle, Block | Knockouts of image-2→image-1 attention: spread evenly or permuted (mass preserved), or removed. |
| Syn, HP, SP | Synthetic shape matching, HPatches point correspondence, SPair-71k semantic correspondence. |
Table 9. Symbols used in the paper.
Repair Details
Adapters. In arm A the query adapter of each layer acts only on the output dimensions of the selected heads in that layer; the key adapter acts on the key–value group of each selected head, so with grouped-query attention it also changes the keys seen by the other query heads of that group. Arm C applies the same restriction to the value projection. Arms D and E adapt the query, key, value, and output projections of every decoder layer. A smoke test verified that training changes only the intended adapter parameters and leaves the base weights unchanged by checksum.
Selection and data. Training pairs come from the SPair-71k training split and validation pairs from its validation split; test pairs are from the test split and were not used in any earlier run. The learning rate was chosen per arm on the validation items. For Qwen3-VL-8B arm E, the largest learning rate diverged and was not selected. Accuracies in Table 6 average three seeds for A, C, and D and three random head sets for B; differences use a paired item bootstrap with seeds averaged first; hierarchical intervals resample the three random head sets and the items jointly. BQ uses the recipe on the three B head sets, with the learning rate chosen on set 0 ( for Qwen3-VL-8B, for InternVL3.5-8B).
Comparisons. Table 10 lists the pre-registered comparisons on the SP test set.
| Comparison | Qwen3-VL-8B | InternVL3.5-8B |
| A − base () | +28.3 [25.3, 31.1] | +25.1 [22.4, 27.8] |
| A − mean(B) | +3.6 [2.0, 5.2] | +2.1 [0.8, 3.4] |
| A − mean(B), hierarchical | +3.6 [1.4, 6.0] | +2.1 [0.2, 4.1] |
| A − mean(B), hierarchical | +3.5 [1.6, 5.5] | +4.5 [2.1, 7.0] |
| A − D | −8.0 | −12.3 |
| A − C | −3.5 | – |
| A − E | −7.8 | – |
| A − base () | +23.9 | +19.0 |
| A − D () | −5.0 | −12.7 |
| Selectivity rise, A | +.032 [.029, .035] | +.021 [.016, .027] |
| Selectivity rise, mean(B) | +.014 | +.007 |
Table 10. Pre-registered comparisons on SP test (pp, paired 95% intervals where available).
Attention Transplantation: Full Results
Table 11 gives the restore and transplant fractions for every analysed arm and seed on the 1,000 SP test items (order 0). Random head sets give fractions between −0.05 and +0.08 for every arm.
| Model | Arm | Seed | Gain (pp) | Removed by Restore | Recovered by Transplant |
| Qwen3-VL-8B | A | 0 | +28.9 | .415 [.333, .504] | .391 |
| Qwen3-VL-8B | A | 1 | .347 | .434 | |
| Qwen3-VL-8B | A | 2 | .356 | .434 | |
| Qwen3-VL-8B | C | 0 | +31.1 | .109 | .151 |
| Qwen3-VL-8B | D | 0 | +34.8 | .161 [.101, .220] | .345 [.273, .413] |
| Qwen3-VL-8B | D | 1 | .133 | .466 | |
| Qwen3-VL-8B | D | 2 | .166 | .514 | |
| Qwen3-VL-8B | E | 0 | +35.6 | .171 | .404 |
| Qwen3-VL-8B | B | set 0 | +24.3 | .086 | .041 |
| InternVL3.5-8B | A | 0 | +22.7 | .053 | .313 |
| InternVL3.5-8B | D | 0 | +37.0 | .030 | .284 |
| InternVL3.5-8B | D | 1 | .022 | .295 | |
| InternVL3.5-8B | D | 2 | .049 | .213 | |
| InternVL3.5-8B | B | set 0 | – | −.023 | .066 |
Table 11. Removed and recovered fractions for , SP test, order 0.
Answer-Position Edits and Full-Position Restore
Answer-position edits. For each model and task, the answer-position probe of § 4.1 was refitted from the stored development features and reproduced the earlier cross-validation score, per-item test correctness, and per-item calibrated credit exactly. is the span of its centred class directions mapped back to the residual space (rank 3). Edits act on the answer-position residual at the probe’s layer (Qwen3-VL-8B: 31 for Syn, 28 for HP; InternVL3.5-8B: 32 and 33): Ablate sets and Amplify sets , with the development mean. Each of 20 random rank-3 subspaces from the top-256 principal components is scaled so that its edit moves the residual by the same mean squared distance as on development data. Table 12 reports the change in ; calibrated accuracy changes by at most 0.2 pp in every cell. Between 1.7% and 2.4% of items edited along leave the base residual-norm band (1.2% to 4.2% for the random subspaces), against 2.0% by construction for unedited items.
| Model | Task | Base | Ablate | Amplify | Amplify | Random range |
| Qwen3-VL-8B | Syn-squiggle | .413 | +0.0 | +0.0 | −0.1 | −0.2 to +0.1 |
| Qwen3-VL-8B | HP | .767 | −0.2 | +0.1 | +0.1 | −0.3 to +0.3 |
| InternVL3.5-8B | Syn-squiggle | .514 | +0.0 | +0.0 | +0.1 | −0.2 to +0.3 |
| InternVL3.5-8B | HP | .543 | −0.1 | +0.0 | +0.0 | −0.5 to +0.4 |
Table 12. Answer-position edits along the probe subspace (test, , order 0). in pp; random range over 20 subspaces and the three operations.
Full-position restore. Restore-all replaces the attention of the heads in at every query position with that of the base model; Restore-group does so for every query head in the key–value groups that touches. Table 13 gives the removed fractions on the 1,000 SP test items (order 0, seed 0) against the five random head sets of §6.3. For arm A, Restore-group reproduces the base model’s log-probabilities on every item in both models. Random sets share some key–value groups with (four to nine of 20 in Qwen3-VL-8B, at most two of 14 in InternVL3.5-8B), so random Restore-group values are not a clean control (in Qwen3-VL-8B they range from .125 to .606 for arm A, without a monotone relation to the overlap) and enter no criterion. The gains in Table 13 differ from the seed-0 gains in Table 11 by at most 0.4 pp: these runs scored 14 prompts per batch instead of 4, which changes bf16 numerics on a few items.
| Model | Arm | Gain (pp) | Restore-all() | Random Restore-all | Restore-group() |
| Qwen3-VL-8B | A | +28.7 | .746 [.671, .823] | .024 to .139 | 1.000 |
| Qwen3-VL-8B | D | +34.9 | .198 [.137, .260] | −.006 to .063 | .183 [.122, .245] |
| InternVL3.5-8B | A | +22.7 | .370 [.248, .479] | −.013 to .004 | 1.000 |
| InternVL3.5-8B | D | +37.4 | .080 [.029, .135] | −.013 to .037 | .120 [.064, .172] |
Table 13. Fraction of the gain removed by full-position restore (SP test, order 0, seed 0; 95% intervals).
Matched Controls, Positive Controls, Decomposition, and Mass-Preserving Knockouts
These four experiments were run together, with rules, item sets, and thresholds frozen before execution, a budget of 24 GPU-hours (18.9 used), and no cut. Two amendments were made before any training result existed: the attention-matched head sets were drawn from any layer rather than from the layers of , and the trainable-parameter count of the matched arms was reported rather than asserted equal to that of the reference arm.
Attention-matched heads. For each head , is the attention mass from image-2 query positions to all image-1 key positions, averaged over Syn development items; is the correspondence score used to select . The three sets – are disjoint and built greedily: for each head in turn, the head with closest to among those not in , not among the heads with the highest , and not yet used. Table 14 compares them with the layer-matched random sets of arm B. Because the matched heads are spread over more key–value groups, every matched arm has more trainable parameters than its reference (Qwen3-VL-8B: M 1.96–2.75M vs. 1.68M, 0.98–1.38M vs. 0.85M; InternVL3.5-8B: 1.64–1.77M vs. 1.50M, 0.82–0.88M vs. 0.75M). Training follows the recipe of §6.1 with one seed per set; the learning rate was chosen on set 0 from the same grid ( for Qwen3-VL-8B M, for the other three arms). The grid’s largest rate was thus selected in three of four sweeps, as for A in InternVL3.5-8B; reuses A’s rate in both models, without a sweep of its own. Table 15 gives the comparisons. In Qwen3-VL-8B, closes about 40% of the gap between and ; in the other three comparisons matched heads do about as well as random ones. On the synthetic conditions, which no arm is trained on, InternVL3.5-8B M and are 10.7–16.3 pp above A and , extending the collateral drop of the targeted InternVL3.5-8B arms noted in § 6.1; on HP and blink-sc they are below.
| Model | Set | ratio | ratio | ratio |
| Qwen3-VL-8B | / / | .985 / .950 / .803 | .107 / .043 / .081 | 1.670 / 1.575 / 1.319 |
| / / | .350 / .430 / .368 | .131 / .152 / .127 | .448 / .554 / .487 | |
| InternVL3.5-8B | / / | .969 / .875 / .827 | .101 / .125 / .105 | 1.709 / 1.390 / 1.401 |
| / / | .316 / .467 / .464 | .017 / .115 / .112 | .522 / .701 / .717 |
Table 14. Head-set statistics relative to . Ratios of set means; is measured on 200 SP development items.
| Model | Comparison | arm | Matched (per set) | B mean | Difference | Paired CI | Hierarchical CI |
| Qwen3-VL-8B | A − M | 71.24 | 67.90 (68.28, 67.40, 68.03) | 67.66 | +3.34 | [1.73, 4.94] | [1.62, 5.07] |
| 67.35 | 65.23 (65.03, 66.03, 64.65) | 63.82 | +2.12 | [0.50, 3.78] | [0.30, 4.02] | ||
| InternVL3.5-8B | A − M | 60.51 | 58.03 (57.38, 59.18, 57.53) | 58.38 | +2.48 | [1.19, 3.84] | [0.75, 4.24] |
| 58.05 | 53.73 (52.78, 54.87, 53.52) | 53.52 | +4.32 | [2.77, 5.98] | [2.42, 6.39] |
Table 15. against attention-matched heads (SP test, 1,000 items, mean over four letter orders, %). Paired intervals average the three sets first; hierarchical intervals resample sets and items.
| Model | Arms | HP | BLINK-SC | BLINK-VC | Syn-squiggle | Syn-known |
| Qwen3-VL-8B | A / M | 93.4 / 91.7 | 54.9 / 48.9 | 80.2 / 79.8 | 55.1 / 51.1 | 83.3 / 79.1 |
| / | 92.2 / 89.7 | 50.4 / 45.6 | 79.1 / 74.6 | 48.0 / 49.5 | 84.2 / 79.0 | |
| InternVL3.5-8B | A / M | 83.7 / 82.5 | 36.7 / 33.6 | 56.2 / 57.2 | 36.6 / 47.3 | 77.6 / 91.2 |
| / | 83.7 / 79.0 | 41.0 / 34.8 | 55.8 / 52.9 | 31.5 / 47.8 | 78.4 / 92.9 |
Table 16. Transfer and collateral accuracy of the matched arms (%, mean over the three sets; A and from Table 6).
Answer-position positive controls. On the first 500 test items of Syn-squiggle and HP (order 0), at the position and layer of Appendix § 13, edits add to the residual, with the standard deviation of the residual within the probe subspace , , and a unit direction. For the letter direction, , where is the first letter that is not the model’s base answer; the switch rate is the fraction of items whose answer becomes . For the probe direction, is the class direction of the probe for the first class it does not predict, mapped back to the residual space; the frozen steer rate is the fraction of items whose answer becomes that class’s letter. Because this rate also counts items already answered with that letter (4.4–10.0%, identical in every row), Table 17 also reports the rate on the remaining items. Controls are ten random unit directions from the top-256 principal components at the same . At most 4.8% of letter- or probe-edited items leave the base norm band (10.2% for random directions at ). The frozen steering criterion (an excess of at least 5 pp over the random mean) is not reached at any in either model, and the HP−Syn difference in steer rate has an interval including zero (−1.2 pp [−4.0, 1.4] and +2.2 pp [−1.4, 5.8]).
| Model | Task | Letter switch, | Random max | Probe moved | Random moved |
| Qwen3-VL-8B | Syn-squiggle | 0.4 / 0.4 / 0.8 / 2.8 | 0.4 | 0.4 | 0.2–0.4 |
| HP | 0.8 / 1.2 / 3.0 / 9.6 | 0.8 | 0.4 | 0.0–0.6 | |
| InternVL3.5-8B | Syn-squiggle | 0.6 / 0.6 / 1.8 / 3.8 | 0.4 | 0.9 | 0.0–1.5 |
| HP | 3.2 / 4.6 / 9.4 / 18.6 | 1.8 | 1.6 | 0.9–1.6 |
Table 17. Answer-position positive controls (%, 500 test items per task). Last three columns at . Moved: steer rate on the items not already answered with the target letter.
Decomposition of Transplant. Upstream runs the trained model and computes the image-2 rows of with the query–key adapter updates switched off for these rows only; Local runs the base model and computes them with the updates switched on. Rows are transplanted into the base model as in §6.3, on the 1,000 SP and 400 HP items of that section (order 0, seed 0), and the same three operations are applied to the five random head sets. Fractions use numerator and gain from the same bootstrap resamples. A reproduction gate with the original batch layout matched §6.3 exactly on the first 50 items; the runs reported here score 32 prompts per batch, which changes bf16 numerics on 1–19 items per arm, so FULL differs slightly from Table 11. For D on HP in Qwen3-VL-8B, Upstream exceeds FULL and the interaction interval excludes zero.
Mass-preserving knockouts. Operations act on the post-softmax image-2 query rows of a head set, image-1 key block only. UNIFORM replaces the distribution over image-1 keys by a uniform one with the same total mass; SHUFFLE permutes the values over image-1 keys with one fixed permutation per item and layer; BLOCK is the knockout of § 5.1. Items are the first 500 test items per task (order 0); controls are the first ten layer-matched random sets of § 5.1 and the three attention-matched sets. Calibrated accuracy follows within 4 pp in every cell. Under every operation, random sets leave the norm band on at most 5.6% of items and matched sets on at most 8.0%.
Table 19. Knockouts of , full results (pp; paired 95% intervals). Random, Matched: largest drop among ten random and three attention-matched sets. Out: % of items outside the base residual-norm band.
| Task | Operation | drop | drop | Random | Matched | Out |
| QWEN3-VL-8B (base / : SYN 41.2 / 62.2, HP 76.2 / 81.2, SP 46.6 / 42.0) | ||||||
| SYN | UNIFORM | 17.2 [12.6, 22.0] | 38.2 [32.6, 43.8] | 2.8 | 0.6 | 4.6 |
| SHUFFLE | 18.2 [13.8, 22.8] | 37.6 [31.6, 43.2] | 3.0 | 0.2 | 5.6 | |
| BLOCK | 17.4 [12.2, 22.8] | 34.8 [28.8, 40.4] | 3.6 | −0.8 | 49.0 | |
| HP | UNIFORM | 32.2 [27.6, 37.0] | 35.0 [30.0, 39.8] | 1.2 | 1.2 | 5.6 |
| SHUFFLE | 31.8 [27.2, 36.4] | 32.4 [27.8, 37.6] | 1.0 | 1.8 | 5.8 | |
| BLOCK | 21.2 [17.4, 25.4] | 26.2 [21.6, 31.0] | 2.4 | 0.2 | 15.8 | |
| SP | UNIFORM | 2.2 [−0.4, 4.8] | 2.2 [−1.4, 6.0] | 1.0 | 0.0 | 1.8 |
| SHUFFLE | 3.2 [0.4, 6.0] | 2.2 [−1.2, 6.0] | 1.2 | −0.6 | 1.6 | |
| BLOCK | 3.6 [0.8, 6.6] | 4.2 [0.2, 8.4] | 1.4 | 2.0 | 1.0 | |
| INTERNVL3.5-8B (base / : SYN 48.6 / 73.4, HP 53.6 / 54.0, SP 34.8 / 29.4) | ||||||
| SYN | UNIFORM | 20.2 [15.4, 25.0] | 48.4 [43.6, 53.4] | 3.4 | 6.6 | 3.6 |
| SHUFFLE | 22.0 [17.2, 26.8] | 46.0 [41.0, 51.0] | 1.8 | 6.2 | 2.0 | |
| BLOCK | 22.4 [17.4, 27.2] | 46.4 [41.4, 51.4] | 2.6 | 6.8 | 2.0 | |
| HP | UNIFORM | 16.8 [13.0, 20.8] | 20.6 [16.2, 25.2] | 2.8 | 1.8 | 15.8 |
| SHUFFLE | 14.6 [10.8, 18.4] | 19.6 [15.0, 24.0] | 4.2 | 1.4 | 16.2 | |
| BLOCK | 16.2 [12.2, 20.6] | 18.2 [13.4, 23.0] | 2.4 | 1.8 | 20.6 | |
| SP | UNIFORM | −0.8 [−3.0, 1.6] | −1.2 [−3.8, 1.4] | 0.0 | 0.0 | 1.8 |
| SHUFFLE | −0.6 [−3.0, 1.8] | −1.6 [−4.2, 1.0] | 0.4 | 1.0 | 1.4 | |
| BLOCK | −0.6 [−3.0, 2.0] | −0.8 [−3.4, 1.8] | 0.2 | −0.2 | 1.4 |
Table 19.
Verification and Provenance
Every number in the main text was recomputed from raw per-item outputs or, for probe and calibration refits, checked against stored per-item values, by scripts that do not import the experiment code, with no mismatches (between 238 and checks per run). Per-item answer-position calibration and probe refits used in the fresh-item confirmation were asserted to reproduce the earlier per-item values before any forward pass. Region-to-token mappings were validated with a perturbation test at the patch-embedding output; for InternVL3.5, the pixel footprint of each region was defined from the support of the resize kernel. Item sets for the fresh-item confirmation use seeds, sequences, and image pairs disjoint from all earlier runs, except that most HP image pairs recur with new reference points and distractors. Repair test pairs are disjoint from all earlier runs, and overlap between training, validation, and test splits was checked by pair identifier.
References
- [1]Shuai Bai, Yuxuan Cai, Ruizhe Chen, Kevin Chen, Xionghui Chen, et al. Qwen3-VL Technical Report. arXiv:2511.21631, 2025.arxiv.org/abs/2511.21631
- [2]Vasileios Balntas, Karel Lenc, Andrea Vedaldi, and Krystian Mikolajczyk. HPatches: A Benchmark and Evaluation of Handcrafted and Learned Local Descriptors. In CVPR, 2017.arxiv.org/abs/1704.05939
- [3]Eden Biran, Daniela Gottesman, Sohee Yang, Mor Geva, and Amir Globerson. Hopping Too Late: Exploring the Limitations of Large Language Models on Multi-Hop Queries. In EMNLP, 2024.
- [4]Zhe Cheng, Wenyu Chen, Fode Zhang, and Dehuan Shen. Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gating. In ICML, pages 18828–18860, 2026.arxiv.org/abs/2605.24024
- [6]Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, and Ranjay Krishna. BLINK: Multimodal Large Language Models Can See but Not Perceive. In ECCV, 2024.arxiv.org/abs/2404.12390
- [7]Rohit Gandikota and David Bau. Gaze Heads: How VLMs Look at What They Describe. arXiv:2606.14703, 2026.arxiv.org/abs/2606.14703
- [8]Gemma Team. Gemma 3 Technical Report. arXiv:2503.19786, 2025.arxiv.org/abs/2503.19786
- [9]Satchel Grant, Simon Jerome Han, Alexa R. Tartaglini, and Christopher Potts. Addressing Divergent Representations from Causal Interventions on Neural Networks. In ICLR, 2026.arxiv.org/abs/2511.04638
- [10]John Hewitt and Percy Liang. Designing and Interpreting Probes with Control Tasks. In EMNLP-IJCNLP, pages 2733–2743, 2019.arxiv.org/abs/1909.03368
- [11]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. In ICLR, 2022.arxiv.org/abs/2106.09685
- [12]Junyang Ji, Qifan Liu, Wenming Yang, and Zhihai He. Causal-Lens: Sensitivity-Guided Multi-Head Causal Intervention for Hallucination Mitigation in Large Vision-Language Models. In CVPR, pages 4199–4209, 2026.
- [14]Chih-Ting Liao, Fei Shen, Xin Cao, and Tat-Seng Chua. Decodable Is Not Grounded: A Vision-Ablation Arbiter for VLM Spatial Reasoning. arXiv:2606.31257, 2026.arxiv.org/abs/2606.31257
- [15]Niclas Lietzow, Danielle Bitterman, Carsten Eickhoff, William Rudman, and Michal Golanovensky. Vision-Default, Prior-Override: Causal Mechanisms of Perception-Knowledge Conflict in Vision-Language Models. arXiv:2606.28273, 2026.arxiv.org/abs/2606.28273
- [16]Benlin Liu, Amita Kamath, Madeleine Grunde-McLaughlin, Winson Han, and Ranjay Krishna. Visual Representations inside the Language Model. In COLM, 2025.arxiv.org/abs/2510.04819
- [17]Zhining Liu, Ziyi Chen, Hui Liu, Chen Luo, Xianfeng Tang, Suhang Wang, Jingying Zeng, Zhenwei Dai, Zhan Shi, Tianxin Wei, Benoit Dumoulin, and Hanghang Tong. Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMs. In ICLR, 2026.arxiv.org/abs/2510.17771
- [18]Tiange Luo, Lajanugen Logeswaran, Jaekyeom Kim, Justin Johnson, and Honglak Lee. Towards Minimal Fine-Tuning of VLMs. arXiv:2512.19219, 2025.
- [19]Aleksandar Makelov, George Lange, Atticus Geiger, and Neel Nanda. Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching. In ICLR, 2024.arxiv.org/abs/2311.17030
- [20]Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and Editing Factual Associations in GPT. In NeurIPS, 2022.arxiv.org/abs/2202.05262
- [21]Juhong Min, Jongmin Lee, Jean Ponce, and Minsu Cho. SPair-71k: A Large-scale Benchmark for Semantic Correspondence. arXiv:1908.10543, 2019.arxiv.org/abs/1908.10543
- [22]Chancharik Mitra, Yusen Luo, Raj Saravanan, Dantong Niu, Anirudh Pai, Jesse Thomason, Trevor Darrell, Abrar Anwar, Deva Ramanan, and Roei Herzig. Mechanistic Finetuning of Vision-Language-Action Models via Few-Shot Demonstrations. arXiv:2511.22697, 2025.arxiv.org/abs/2511.22697
- [23]Farhad Nooralahzadeh, Omid Rohaniann, Yi Zhang, Jonathan Fürst, and Kurt Stockinger. Arbitration Failure, Not Perceptual Blindness: How Vision-Language Models Resolve Visual-Linguistic Conflicts. arXiv:2604.09364, 2026.arxiv.org/abs/2604.09364
- [24]Xingzhou Pang, Yifan Hou, Junling Wang, and Mrinmaya Sachan. Unveiling the Visual Counting Bottleneck in Vision-Language Models. In ICML, 2026.arxiv.org/abs/2605.30170
- [25]Chanho Park, Daeheyeon Choi, Jihyun Lee, and Minhhyuk Sung. Retrieval Heads Meet Vision: Uncovering How VLMs Locate and Extract Visual Information. arXiv:2608.27417, 2026.arxiv.org/abs/2608.27417
- [26]Jason Qiu, Zachary Meurer, Xavier Thomas, and Deepti Ghadiyaram. Semantic Richness or Geometric Reasoning? The Fragility of VLM’s Visual Invariance. In COLM, 2026.arxiv.org/abs/2604.01848
- [27]Timothy Schaumlöffel, Martina G. Vilas, and Gemma Roig. Mechanisms of Object Localization in Vision-Language Models. In CVPR, pages 31356–31365, 2026.arxiv.org/abs/2605.19792
- [28]Haza Shahgir, Xiaofuo Chen, Yu Fu, Erfan Shavegani, Nael Abu-Ghazaleh, Yova Kementchedjhieva, and Yue Dong. VLMs Need Words: Vision Language Models Ignore Visual Detail In Favor of Semantic Anchors. In COLM, 2026.arxiv.org/abs/2604.02486
- [29]Basavaraj Sunagad, Haoran Wang, David T. Hoffmann, Christian Theobalt, and Adam Kortylewski. SOCO: Benchmarking Semantic Object Correspondence in Vision Foundation Models. arXiv:2605.31597, 2026.arxiv.org/abs/2605.31597
- [30]Anat Veikherman, Mor Ventura, Nitay Calderon, Lihi Zelnik-Manor, and Roi Reichart. Encoded but Unused: Bounding the Sources of VLM Errors. arXiv:2610.29241, 2026.arxiv.org/abs/2610.29241
- [31]Constantin Venhoff, Ashkan Khakzar, Sonia Joseph, Philip Torr, and Neel Nanda. Too Late to Recall: Explaining the Two-Hop Problem in Multimodal Knowledge Retrieval. In NeurIPS, 2025.
- [32]Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating Gender Bias in Language Models Using Causal Mediation Analysis. In NeurIPS, 2020.
- [33]Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, et al. InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency. arXiv:2508.18265, 2025.arxiv.org/abs/2508.18265
- [34]Zirui Wang, Junwei Yu, Adam Yala, David M. Chan, Joseph E. Gonzalez, and Trevor Darrell. Stateful Visual Encoders for Vision-Language Models. arXiv:2606.04433, 2026.arxiv.org/abs/2606.04433
- [35]Sarah Wiegreffe, Oyvind Tafjord, Yonatan Belinkov, Hannah Hajishirzi, and Ashish Sabharwal. Answer, Assemble, Ace: Understanding How LMs Answer Multiple Choice Questions. In ICLR, 2025.arxiv.org/abs/2407.15018
- [36]Hugh Mee Wong, Rick Nouwen, and Albert Gatt. When Models Decide and When They Bind: A Two-Stage Computation for Multiple-Choice Question-Answering. arXiv:2601.03914, 2026.arxiv.org/abs/2601.03914
- [37]Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon. A Theory of Usable Information under Computational Constraints. In ICLR, 2020.arxiv.org/abs/2002.10689
- [38]Qiyao Yan, Chenpeng Wang, and Liangming Pan. Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scores. arXiv:2608.31068, 2026.arxiv.org/abs/2608.31068
- [39]Fred Zhang and Neel Nanda. Towards Best Practices of Activation Patching in Language Models: Metrics and Methods. In ICLR, 2024.arxiv.org/abs/2309.16042
- [40]Genpei Zhang. Encoded but Disconnected: Decomposing Vision-Language Model Failures under a Patching Null. arXiv:2610.00024, 2026.arxiv.org/abs/2610.00024
- [41]Yingxuan Zhuang, Jingxiao Yang, Miao Pan, Cheng Tan, Yuxian Cai, Siwei Tan, Chen Zhi, Xuhong Zhang, Jianwei Yin, and Jintao Chen. Mitigating Manifold Departure: Uncertainty-Aware Subspace Rectification for Trustworthy MLLM Decoding. In ICML, 2026.arxiv.org/abs/2606.09859
