betaFeedback

WorksComputer Science

Beyond the Sink Token: Sink-Guided Key Editing in Vision-Language Models

Abstract

In vision-language models (VLMs), a few sink tokens attract a disproportionate share of attention. While previous studies have examined why these tokens attract attention, we investigate how the sink-key direction affects attention to image and question tokens. Across four probed VLMs, boosting the magnitude of positive sink-aligned components increases attention on average, whereas boosting negative components decreases it; these responses persist through later layers. This identifies the native sink key as a reference for controlling attention beyond its own position. Motivated by this finding, we propose SHRINK, a training-free method that partially reduces sink-aligned components across visible keys during generation. On nine benchmarks under sampled reasoning, it improves mean accuracy by 5.1 and 2.7 percentage points on LLaVA-OneVision-7B and Qwen2.5-VL-3B, with smaller gains on the other two main models, Qwen2.5-VL-7B and Gemma-3-4B. Beyond SHRINK, rescaling sink-aligned key components can replace the attention operations of two existing attention-steering methods, GIFT and ACG, with comparable accuracy on hallucination benchmarks.

Introduction

Vision-language models (VLMs) rely on attention to process images and questions when generating answers [3, 78]. Yet some heads assign substantial attention to a few sink tokens, known as attention sinks, regardless of the input [83]. Previous studies have investigated why these tokens attract attention [22, 69]. Attention redistribution can improve VLM predictions [36, 48], and structural attention biases can extend beyond individual sinks [30]. Editing only the sink’s score changes its attention share but preserves attention ratios among the other tokens [25]. We ask:

Key Question

Does the sink’s influence extend to how image and question tokens compete with one another for attention?

We investigate this broader influence through the sink’s

key vector. Self-attention scores tokens by comparing their keys with the current token’s query [78]. Prior work links sink attraction to its key [22], while analyses of query-key geometry reveal shared directional structure [19]. We study one sink per model, the most-attended template token before the image and usually the first token; its key is identical across inputs. Other keys can contain a part pointing along or against it: a positive or negative sink-aligned component.

We find that the sink-key direction can control how much attention image and question tokens receive. As illustrated in Figure 1, boosting the magnitude of a positive sink-aligned component increases attention on average, whereas boosting a negative one decreases it. These responses persist through later layers, extending the sink-key direction’s role beyond attraction to the sink itself.

Figure 1. Boosting the sink-aligned component. For a query positively aligned with the sink-key direction, illustrative bars show attention increasing when a positive component is amplified and decreasing when a negative component is amplified.

We then ask whether reducing the sink-key direction’s contribution improves answer accuracy. Shrinking sink-aligned components makes attention scores depend more on the remaining key components. We introduce SHRINK, a training-free decoding method that applies this shrinkage across all visible keys in heads attending to the image or question. Because the components differ across tokens, the edit changes their competition with one another as well as with the sink.

Under sampled reasoning, SHRINK improves mean accuracy on the four main VLMs, with gains of 5.1 and 2.7 percentage points on LLaVA-OneVision-7B and Qwen2.5-VL-3B. Rescaling sink-aligned key components can also replace the attention operations of GIFT [57] and ACG [32] with comparable accuracy on hallucination benchmarks.

Our contributions are threefold:

  • We show that existing signed components along the native sink-key direction control attention to image and question tokens. Boosting positive components increases average attention; boosting negative components decreases it, with both effects persisting through later layers.

  • We introduce SHRINK, a training-free intervention that changes relative attention among content tokens by shrinking their sink-aligned components, extending the scope of sink-only editing.

  • We show that rescaling sink-aligned key components can replace the attention operations of GIFT and ACG, with comparable accuracy on hallucination benchmarks.

Related Work

Origins and function of sinks. Studies connect sink attraction to massive activations [69], representation geometry [7, 22, 42, 63], and circuits [60, 70]. Related analyses examine concentrated query-key activations [31] and visual re-sinking [34]. Shared geometry connects tokens beyond the primary sink: queries and keys can drift along a common axis [19], and middle-layer MLPs align secondary-sink hidden states with the primary sink [81]. SinkRouter places BOS keys near the key mean in PCA [47]; ValueDiff reports anti-alignment with the content-key mean [56], and KeyDiff uses key similarity to the mean for attention prediction and cache eviction [55]. These measurements motivate our mean-key comparison. Sink representations also serve computational roles: they support dormant heads and stable biases [12, 24], limit over-mixing [4], and enable streaming [83], compression [59], and visual aggregation [14, 29, 53]. Catch, Tag, Release studies shared tags carried through sink values [92]; SinkTrack and OutRo exploit sink information [50, 86], with OutRo finding that zeroing sink-key coordinates can hurt performance. We therefore preserve token values at the edited layer when studying attention attraction.

Steering attention during inference. Inference-time methods reweight visual or textual attention [10, 48, 85, 91] and counter distracting attention [21, 26]. Sink-aware methods calibrate attention [88], edit initial-token logits [25], redistribute visual-sink attention [36], or use sink-triggered [67] and position-aware decoding [93]. AttnReal redirects attention from output sinks to image tokens [77], while FarSight uses mask-based registers to absorb attention diverted to outliers [72]. SPAR broadens the focus beyond isolated sinks, correcting textual bias over visual features with an attention template [30]. We study this broader competition through the sink-key direction. Native attention also guides where to intervene [11, 35, 64, 73]; we select heads reading the image, question, or options. We also test key edits within GIFT’s saliency reweighting and ACG’s contrastive guidance [32, 57].

Directional editing of keys and queries. Directional editing is established in steering and concept removal [1, 6, 41, 49, 54, 58, 61, 62, 84]. Related interventions include learned key scaling [13], TAME’s query-key eigenspectrum regulation [71], spectral edits of the query-key weight product [74], and cached-state steering [5]. Single-channel scaling targets positional bias [87]. OrthoRank instead uses sink-relative hidden-state geometry to select tokens [66]. Q-Filters projects keys onto a calibrated, context-agnostic dominant query axis to predict attention and rank cache entries, without editing those components [20]. SEKA edits highlighted keys using contrastively estimated relevance subspaces; AdaSEKA combines expert projections [43]. We build on projection and key-query equivalence as established tools. Concurrent work [90] also studies sink-key alignment and copying, but moves visual-sink components to an image-key median for crop selection. We use the native sink key as the reference, without estimating a query subspace, and test how editing existing signed components changes attention and accuracy.

Finding: Signed Components in Input Keys

We test whether sink attraction transfers to another token and whether the sink-key direction can control attention to image and question tokens. Key copying replaces only a target token’s key with the sink’s key; value copying replaces only its value with the sink’s value. We then isolate the directional contribution by editing the component already present in a target key (Fig. 2).

Attention notation. To isolate the sink direction’s contribution to attention, we separate it from the rest of each key. Within an attention head, qq denotes the current token’s query, with its token index omitted for simplicity; kjk_j and vjv_j denote the key and value at position jj. These vectors are taken after positional encoding. Scores are zj=clqTkjz_j = c_l q^\mathsf{T} k_j with scale cl>0c_l > 0, and aj=exp⁡zj/∑i∈Vtexp⁡zia_j = \exp z_j / \sum_{i \in\mathcal{V}_t} \exp z_i over visible positions Vt\mathcal{V}_t. Layer and head indices are suppressed unless needed. The nonzero key at sink position ss defines the sink-key direction u=ks/∥ks∥2\mathbf{u} = k_s / \lVert k_s \rVert_2.

An input key’s sink-aligned component is (kjTu)u(k_j^\mathsf{T}\mathbf{u})\mathbf{u}. Writing its remaining part as kj⊥k_j^\perp separates the score:

zj=clqTkj⊥+(kjTu)clqTu⏟sink-aligned contribution.(1)z_j = c_l q^\mathsf{T} k_j^\perp+ \underbrace{(k_j^\mathsf{T}\mathbf{u})c_l q^\mathsf{T}\mathbf{u}}_{\text{sink-aligned contribution}} . \tag*{(1)}

The component can point with or against u\mathbf{u}; we call kjTu\mathbf{k}_j^{\mathsf{T}}\mathbf{u} its sink-aligned coefficient, which is positive or negative accordingly.

Attraction transfers through the key. To test whether sink attraction follows the key or value, we copy either vector from the sink token to another prompt token positioned before the image and question. We then measure attention to this receiving token from the final prompt query, before answer generation. The original sink is blocked as an attention target, and a matched no-copy control preserves the receiving token’s original key and value. Copying the sink key substantially increases attention to the receiving token across all four models, whereas copying its value leaves attention near the control (Figure 2a). On Qwen2.5-VL-3B, key copying raises it from about 0.02% to 3.0% and 5.1% (Appendix § 12). We next examine the sink-aligned component already present in ordinary input keys.

Figure 2. Attraction follows the key, and its direction gives signed control. (a) Attention to a receiving token after copying the sink’s key or value, for one model per family; the original sink is blocked. (b) Boosting positive and negative components for a positively aligned query. (c) Attention relative to Base for all four models; multipliers above 1 boost the component’s magnitude and 0 removes it. Whiskers and colored bands: 95% confidence intervals; hatched grey band: range of four size-matched random-direction means.

Boosting sink-aligned components. To test directional control without replacing the whole key, we boost the magnitude of a token’s existing sink-aligned component while preserving its sign. We edit one image or question token during prompt processing and measure attention to it from later prompt positions. We vary the component’s strength using

kj(α)=kj+(α−1)(kjTu)u.(2)\mathbf{k}_j(\alpha)=\mathbf{k}_j+(\alpha-1)(\mathbf{k}_j^{\mathsf{T}}\mathbf{u})\mathbf{u}. \tag*{(2)}

A boost uses α>1\alpha> 1: a positive coefficient becomes more positive, and a negative coefficient becomes more negative (Figure 2b). We also include the unchanged key (α=1\alpha= 1) and component removal (α=0\alpha= 0). The remaining key component and value are unchanged at the edited layer.

To test whether the component’s sign predicts the attention response, we group measurements by its sign in the unedited key. Figure 2c reports the geometric mean of edited-to-Base attention ratios, with 1 indicating unchanged attention. Across all four models, boosting positive components raises the curves above 1, while boosting negative components lowers them below 1. Removing the component reverses this pattern. Equal-size random edits orthogonal to u\mathbf{u} separate the two groups much less, indicating that the signed response is tied to the sink-key direction.

Why boosting can raise or lower attention. To explain the two responses in Figure 2b, we hold the query and all other keys fixed. Boosting changes the target’s score by

Δzj=(α−1)cl(kjTu)(qTu).\Delta z_j = (\alpha- 1)c_l\left(\mathbf{k}_j^{\mathsf{T}}\mathbf{u}\right)\left(\mathbf{q}^{\mathsf{T}}\mathbf{u}\right).

For a boost (α>1\alpha> 1) and a positively aligned query, the score change has the same sign as the sink-aligned component. A positive component therefore raises the target’s score and attention; a negative component lowers both. Negative query alignment reverses these effects.

The calculation predicts the immediate response at the edited layer. The full-pass measurements in Figure 2c show that the average increase for positive components and decrease for negative components persist through later layers. Most final prompt queries are positively aligned with the sink-key direction. Appendix § 14 gives the alignment statistics, measurement protocol, and full results.

These interventions identify a shared sink-related contribution to attention among image and question tokens. Reducing it across keys can adjust their competition while preserving the remaining key components, which motivates SHRINK.

Method

SHRINK reduces the shared sink-related contribution identified in § 3 to improve accuracy. It partially shrinks sink-aligned components across visible keys, reducing their contribution to attention scores while preserving the remaining key components at the edited layer. Applying the edit across keys changes competition among content tokens. We focus on heads currently reading the image or question.

Preliminaries

We use the attention notation of § 3 and make head indices explicit. At decoding step tt, Vt\mathcal{V}_t contains the visible prompt and generated positions. Visibility follows the model’s original attention mask; keys outside this set receive zero attention. Query head hh uses query qh\mathbf{q}^{h} and the keys kjg(h)\mathbf{k}^{g(h)}_j and values vjg(h)\mathbf{v}^{g(h)}_j from its key/value (KV) head g(h)g(h). Under grouped-query attention, several query heads can share a KV head. Before editing, attention weights and the head output are

ah=softmax⁡j∈Vt(cl(qh)Tkjg(h)),\mathbf{a}_h = \operatorname{softmax}_{j\in\mathcal{V}_t}\left(c_l\left(\mathbf{q}^{h}\right)^{\mathsf{T}}\mathbf{k}^{g(h)}_j\right),
oh=∑j∈Vtah,jvjg(h),\mathbf{o}_h = \sum_{j\in\mathcal{V}_t}a_{h,j}\mathbf{v}^{g(h)}_j,
o=Concat⁡h(oh)WOl.(3)\mathbf{o} = \operatorname{Concat}_h(\mathbf{o}_h)\mathbf{W}^{l}_{O}. \tag*{(3)}

Here WOl\mathbf{W}^{l}_{O} is the layer’s output projection. The weights determine how the values are combined. SHRINK changes the query-key interaction while keeping these values unchanged at the edited layer.

The sink occupies a fixed prefix position ss before the image and question. Causality makes its key independent of those subsequent inputs. Its unit key direction ug\mathbf{u}_g, defined as in § 3 for each layer and KV head, therefore supplies an input-independent reference. No labels are needed to estimate this direction.

Algorithm 1 SHRINK Attention for One Layer and Query

Input: Pre-edit qh\mathbf{q}^{h}, kjg(h)\mathbf{k}^{g(h)}_j, vjg(h)\mathbf{v}^{g(h)}_j; map g(h)g(h); directions ug\mathbf{u}_g.

Input: Visible positions Vt\mathcal{V}_t; image/question sets I,T\mathcal{I}, \mathcal{T}; sink ss.

Input: Layer ll, eligible layers Ledit\mathcal{L}_{\mathrm{edit}}, answer-query flag bb; strength β∈(0,1]\beta\in(0,1], scale clc_l, output map WOl\mathbf{W}^{l}_{O}.

Output: Attention-layer output o\mathbf{o}.

  1. for each Query head hh do

  2.     g←g(h)g \gets g(h)

  3.     ah←softmax⁡j∈Vt{cl(qh)Tkjg}\mathbf{a}_h \gets\operatorname{softmax}_{j\in\mathcal{V}_t}\{c_l(\mathbf{q}^{h})^{\mathsf{T}}\mathbf{k}^{g}_j\}     ▷ pre-edit attention

  4.     if bb and l∈Leditl\in\mathcal{L}_{\mathrm{edit}} and s∈Vts\in\mathcal{V}_t then

  5.         mI←∑j∈I∩Vtah,jm_{\mathcal{I}}\gets\sum_{j\in\mathcal{I}\cap\mathcal{V}_t}a_{h,j}; mT←∑j∈T∩Vtah,jm_{\mathcal{T}}\gets\sum_{j\in\mathcal{T}\cap\mathcal{V}_t}a_{h,j}

  6.         if mI≥0.2m_{\mathcal{I}}\geq0.2 or mT≥0.2m_{\mathcal{T}}\geq0.2 then

  7.             kj′←kjg−β((kjg)Tug)ug\mathbf{k}'_j\gets\mathbf{k}^{g}_j-\beta((\mathbf{k}^{g}_j)^{\mathsf{T}}\mathbf{u}_g)\mathbf{u}_g for all j∈Vtj\in\mathcal{V}_t

  8.             ah←softmax⁡j∈Vt{cl(qh)Tkj′}\mathbf{a}_h\gets\operatorname{softmax}_{j\in\mathcal{V}_t}\{c_l(\mathbf{q}^{h})^{\mathsf{T}}\mathbf{k}'_j\}

  9.         end if

  10.     end if

  11.     oh←∑j∈Vtah,jvjg(h)\mathbf{o}_h\gets\sum_{j\in\mathcal{V}_t}a_{h,j}\mathbf{v}^{g(h)}_j

  12. end for

  13. return o=Concat⁡h(oh)WOl\mathbf{o}=\operatorname{Concat}_h(\mathbf{o}_h)\mathbf{W}^{l}_{O}

All-Key Shrinkage

For each eligible query head (§ 4.3), SHRINK applies the following edit, with head indices omitted:

kj′=kj−β(kjTu)u,j∈Vt,0<β≤1.(4)\mathbf{k}'_j=\mathbf{k}_j-\beta(\mathbf{k}_j^{\mathsf{T}}\mathbf{u})\mathbf{u},\quad j\in\mathcal{V}_t,\quad0<\beta\leq1. \tag*{(4)}

The fraction removed is β=1−α\beta=1-\alpha in Eq. (2); β=1\beta=1 removes it entirely. Validation selects the strength. The remaining key component and value are unchanged at the edited layer.

Applying the edit to every visible key, including the sink, input, and generated keys, has an equivalent query-side interpretation:

q′h=qh−β((qh)Tug(h))ug(h).(5)\mathbf{q}'^{h}=\mathbf{q}^{h}-\beta((\mathbf{q}^{h})^{\mathsf{T}}\mathbf{u}_{g(h)})\mathbf{u}_{g(h)}. \tag*{(5)}

It reduces the query’s interaction with the sink-key direction at every visible position. The resulting change in relative attention is

Δlog⁡ajai=−βcl(qTu)((kj−ki)Tu).(6)\Delta\log\frac{a_j}{a_i}=-\beta\mathbf{c}_l(\mathbf{q}^{\mathsf{T}}\mathbf{u})((\mathbf{k}_j-\mathbf{k}_i)^{\mathsf{T}}\mathbf{u}). \tag*{(6)}

The softmax denominator cancels when comparing two input tokens, so a sink-only edit preserves their attention ratio. Equation (6) shows why all-key shrinkage changes it: keys with different sink-aligned coefficients receive different score changes. For a positively aligned query, a token with a smaller sink-aligned coefficient gains attention relative to one with a larger coefficient; for a negatively aligned query the order reverses. This allows SHRINK to adjust competition among image and question tokens alongside their competition with the sink. Appendix § 11 gives the derivation. By the query-side form, the edit itself needs only one projection per eligible query head and leaves the KV cache unchanged.

Head Selection and Generation

We focus the edit on heads already reading the task inputs. Let I\mathcal{I} contain image positions and T\mathcal{T} question and answer-option positions. A head qualifies when at least 20% of its pre-edit attention goes to visible image positions or at least 20% goes to visible question or option positions. The image threshold follows VAR’s initial image-attention filter [36], with related attention-based head selection in SPIN [64].

A label-free calibration pass over a small set of unedited prompts fixes the sink position and eligible layers once per model. During generation, only head eligibility is recomputed at each step.

Editing begins with the query predicting the first generated token; earlier prompt positions are left unedited. At each step, eligible heads recompute attention with projected keys and the original values, skipping the edit if the sink is not visible. Query heads sharing a KV head use the same direction but are assessed independently. Algorithm 1 summarizes the procedure; Appendix § 11 specifies calibration, layer schedules, and implementation.

Experiments

We evaluate SHRINK’s accuracy, test directional key edits within GIFT [57] and ACG [32], identify what SHRINK’s gains depend on, and examine the resulting attention allocation.

Experimental Setup

Models and benchmarks. We evaluate LLaVA-OneVision-7B [40], Qwen2.5-VL-3B/7B (Qwen-3B/7B) [3], and Gemma-3-4B [18]; InternVL3-8B [94] results appear in Appendix § 15. Table 1 reports the four models on Core-5 (MMBench-EN [51], POPE [44], CV-Bench [75], EmbSpatial-Bench [15], MMStar [8]) and Spatial-4 (VSR [45], WhatsUp, COCO-QA, VG-QA [10, 33]). Additional benchmark results are reported in the appendix.

BenchmarkLLaVA-OneVision 7B BaseLLaVA-OneVision 7B SHRINKLLaVA-OneVision 7B Δ\DeltaQwen2.5-VL 3B BaseQwen2.5-VL 3B SHRINKQwen2.5-VL 3B Δ\DeltaQwen2.5-VL 7B BaseQwen2.5-VL 7B SHRINKQwen2.5-VL 7B Δ\DeltaGemma-3 4B BaseGemma-3 4B SHRINKGemma-3 4B Δ\Delta
Core benchmarks
MMBench64.270.9+6.7*73.874.5+0.787.387.1−0.282.682.4−0.2
POPE61.263.1+1.9*80.981.9+1.0*86.386.5+0.284.384.2−0.1
CV-Bench51.254.9+3.6*58.261.0+2.8*73.774.2+0.459.460.8+1.5*
EmbSpatial51.153.0+1.9*47.349.4+2.1*66.766.9+0.253.955.0+1.2*
MMStar38.143.3+5.2*44.745.4+0.759.460.7+1.352.452.5+0.1
Spatial relations
VSR42.653.4+10.8*63.869.1+5.3*84.085.2+1.2*67.667.4−0.2
WhatsUp68.971.5+2.6*67.274.5+7.4*88.788.2−0.558.864.7+5.9*
COCO-QA37.542.1+4.6*56.257.2+1.064.864.6−0.250.651.6+0.9*
VG-QA34.743.0+8.3*51.354.8+3.5*63.365.7+2.4*48.046.5−1.5†
Core-5 avg.53.257.1+3.9*61.062.4+1.5*74.775.1+0.466.567.0+0.5*
Spatial-4 avg.45.952.5+6.6*59.663.9+4.3*75.275.9+0.7*56.357.5+1.3*
All-nine avg.49.955.0+5.1*60.463.1+2.7*74.975.5+0.5*62.062.8+0.8*

Table 1. Main four-model comparison. Test accuracy (%) on nine benchmarks under sampled reasoning with strict scoring and validation-selected strengths. Base: unedited decoding; Δ\Delta: SHRINK minus Base in percentage points, computed before rounding. Bold: higher accuracy in each pair. */†: gains/losses whose pointwise paired-bootstrap 95% CI excludes zero. Benchmark averages are equally weighted.

Decoding and scoring. Base denotes unedited decoding. Base and SHRINK use identical reasoning-then-answer prompts, three paired sampling seeds, temperature 0.7, and top-pp = 1. We refer to this protocol as sampled reasoning. For each model and benchmark, β\beta is chosen by mean validation accuracy and evaluated on a disjoint test split. Accuracy counts only naturally completed responses with a correct, unambiguous final answer; all others count as wrong. Benchmark means are equally weighted. Appendix § 9 specifies preprocessing, including LLaVA-OneVision’s 384 × 384 resizing.

Uncertainty. Table markers denote pointwise 95% paired-bootstrap intervals over test questions, conditioned on the selected strength.

Main Results

SHRINK improves nine-benchmark mean accuracy on all four main models (Table 1). LLaVA-OneVision and Qwen-3B gain 5.1 and 2.7 percentage points and improve on every benchmark. Qwen-7B and Gemma gain 0.5 and 0.8 points, with more variation across tasks. On ten additional benchmarks, LLaVA-OneVision and Qwen-3B again improve on every benchmark, significantly on four and six (Appendix § 18). All four mean gains survive Holm correction over the five evaluated models (Appendix § 23).

The improvement is strongest on spatial relations: Spatial-4 gains exceed Core-5 gains for every model. LLaVA-OneVision gains 6.6 versus 3.9 points across the two groups; Qwen-3B gains 4.3 versus 1.5. Spatial relations respond most to the edit, while Core-5 means remain positive for every model.

The gains also persist with deterministic reasoning: greedy Core-5 accuracy rises by 3.7 and 1.5 points on LLaVA-OneVision and Qwen-3B (Appendix § 16).

Directional Edits within Existing Methods

We next test directional key edits within existing methods, retaining their selection or guidance. Each variant rescales sink-aligned key components (Eq. (2)) with token-specific strengths. GIFT (Key) replaces GIFT’s saliency reweighting with saliency-weighted key edits; ACG (Key) replaces ACG’s image-masked contrast branch with a key-edited branch. Both use greedy yes/no decoding. For matched tuning, GIFT’s own exponent is also retuned on the same validation pools. ACG keeps validation-chosen guidance (γ=2.4\gamma= 2.4 on 7B, 4 on 13B/BakLLaVA), tuning only the new branch’s strength.

In Table 2, five of six key variants have higher benchmark means, three with positive pointwise intervals; GIFT on LLaVA-1.5-7B favors the original by 1.22 points. Appendix L gives results on further benchmarks. Directional key edits can thus replace the attention operations of both methods with comparable accuracy on hallucination benchmarks.

ModelMethodPOPE-A-OKVQAPOPE-GQAAMBERAvg.
LLaVA-1.5 7BBase78.3576.0872.4175.62
GIFT (tuned)82.2681.9979.7081.32
GIFT (Key)81.76†81.27†77.24†80.09†
ACG (tuned)80.7379.7376.5579.00
ACG (Key)81.51*80.77*77.53*79.93*
LLaVA-1.5 13BBase76.8574.6672.0474.52
GIFT (tuned)79.1278.6277.1578.30
GIFT (Key)79.4679.1878.51*79.05*
ACG (tuned)80.3779.7876.4778.87
ACG (Key)80.9280.49*76.2179.20
BakLLaVA 7BBase76.2974.9368.6273.28
GIFT (tuned)81.7681.0076.8879.88
GIFT (Key)81.9480.44†77.57*79.98
ACG (tuned)79.5278.9175.6678.03
ACG (Key)80.42*79.71*76.46*78.87*

Table 2. Key edits inside GIFT and ACG. Accuracy (%) with greedy yes/no decoding and matched tuning. Bold: higher in each pair; */†: Key minus tuned original has a 95% CI above/below zero.

What the Gains Depend On

Editing beyond the sink. SHRINK changes competition among content tokens. We first compare it with sink-only editing, which preserves their attention ratios. With head and layer selection fixed and strengths tuned separately for each scope, all-key editing gains 3.9 Core-5 points on LLaVA-OneVision versus 2.4 for sink-only editing. It is also favored on Qwen-3B (1.5 vs. 0.6) and Gemma (0.5 vs. −0.3), with the 95% CI of the difference above zero on all three models; the two scopes perform similarly on Qwen-7B (Appendix § 19).

The reference direction. We next test whether the gains depend on the reference direction, fixing scope, head and layer selection, and one model-level strength, in a separate run (Table 3; Appendix § 22). Mean query tests whether stronger attraction predicts greater benefit from shrinkage. Mean key and the leading key axis test shared key geometry; a fixed template key tests another input-independent reference, and random directions test generic projection.

ReferenceLLaVA-OV 7BQwen2.5-VL 3B
Random (range of 3)−0.24 to +0.40−0.76 to +0.41
Leading key axis−11.08†−5.03†
Fixed template key+0.14+0.15
Mean key−0.39+2.26∗
Mean query+2.49∗+2.33∗
Sink key+4.57∗+3.55∗

Table 3. Reference direction. Nine-benchmark accuracy change (pp) from a separate run at fixed model-level strengths. ∗/†: 95% CIs above/below zero for individual directions.

The sink reference gives the largest gains at the tested strengths (+4.57 and +3.55). Mean query is more aligned with the queries than the sink key (mean absolute alignment 0.76 and 0.75 versus 0.47 and 0.43) and attracts more strongly when added to a key at equal edit size (Appendix § 14), yet gains less when shrunk (+2.49 and +2.33). Stronger attraction therefore does not imply a larger benefit from shrinkage. Template and random edits change little; the leading key axis reduces accuracy (−11.08 and −5.03). The mean key helps Qwen-3B (+2.26) but not LLaVA-OneVision (−0.39), so shared key geometry alone does not account for the gains. In paired comparisons on both models, every alternative direction falls below the sink key, with each 95% CI below zero.

One global strength. We finally test whether the gains require adapting shrinkage strength to each dataset. Table 4 compares dataset-specific, model-specific, and global validation selection. Model-specific selection stays within 0.33 points of dataset-specific selection on every model, so per-dataset tuning adds little. One global β=0.3\beta= 0.3 retains positive means on all four main models, with pointwise intervals above zero. Model-specific selection can still help: LLaVA-OneVision gains 3.88 points globally versus 5.14 with its model-specific choice. For Qwen-3B and Qwen-7B, the global strength coincides with the model-specific one. For Qwen-3B, the global choice is above dataset-specific selection (3.05 vs. 2.72); for Qwen-7B and Gemma, it stays within 0.13 points. A single strength therefore transfers across these models and benchmarks.

ModelPer datasetPer modelGlobal
LLaVA-OV-7B+5.10∗+5.14∗+3.88∗
Qwen2.5-VL-3B+2.72∗+3.05∗+3.05∗
Qwen2.5-VL-7B+0.53∗+0.48∗+0.48∗
Gemma-3-4B+0.84∗+0.72∗+0.71∗

Table 4. Strength sharing. Nine-benchmark mean accuracy change (pp). Bold: row maximum. Global uses β=0.3\beta= 0.3; ∗: 95% CI above zero.

Attention Allocation

The experiments above measure accuracy. We now examine how the edits move attention: group-level redistribution under SHRINK (Figure 3) and token-level examples from SHRINK, ACG (Key), and GIFT (Key) (Figure 4).

Figure 3. Attention changes under SHRINK. Attention leaving the sink goes to image, template, and generated positions. Bars show SHRINK minus Base in percentage points on a common scale; the five groups partition visible attention. Both runs follow the same answer continuation, with heads selected separately in each. Means weight the nine benchmarks equally.

Figure 4. Selected token-level attention changes. Blue/red: increases/decreases, saturated at ±0.9\pm0.9 pp. Numbers give summed image-attention changes. SHRINK panels compare runs on the same response continuation; ACG and GIFT are measured before the first answer token. ACG guidance is matched; GIFT uses released settings.

Image attention gains exceed question/option gains in all four models. Together, these two groups receive 46% to 58% of the attention leaving the sink; template and generated positions receive the rest. Image tokens gain most in LLaVA-OneVision and Gemma. Qwen2.5-VL-7B shows both the smallest sink reduction (5.9 pp) and the smallest image gain (2.7 pp). Redistribution magnitude does not track accuracy: Gemma’s image increase exceeds Qwen-3B’s despite its smaller gain, and its sink reduction resembles LLaVA-OneVision’s. Head eligibility is recomputed in each pass, so the averages cover the heads that each run selects rather than one fixed set.

Changes within an input. To examine attention changes within an input, we look beyond the group totals to individual tokens. In Figure 4a, attention rises on “perpendicular” and falls on “bed” while image attention increases. This example shows how a small net text-attention change can contain opposing token-level shifts. Appendix § 21 specifies the trace protocols.

Discussion

Implications

The finding extends sink analysis to competition among input tokens. The sink token’s key direction contributes to their scores (Eq. (1)), and all-key editing can change their relative attention (Eq. (6)). Attraction strength alone, however, does not determine which direction is useful to shrink. Mean query attracts more strongly when added to a key yet gives smaller shrinkage gains at the tested strengths (Table 3). Reference selection must therefore account for the effect of reducing existing components.

Limitations and Future Directions

Our evaluation covers closed-form tasks, in which the model selects among given options (including yes/no), and models of 3B to 13B parameters. Extending it to open-ended answering, captioning, long-form generation, and larger models remains future work.

More broadly, our results suggest that attention sinks matter beyond the attention they absorb: the sink token’s key direction also shapes how attention is shared among the remaining input tokens. Sink-aligned components may thus offer an inexpensive indicator of token salience, complementing fixed-direction scoring for cache compression [20], and could be regulated during training rather than edited at inference. Extending this analysis to video, multi-image, and text-only models is a natural next step.

Conclusion

The sink-key direction controls attention beyond the sink position: boosting positive sink-aligned components increases average attention to image and question tokens, while boosting negative components decreases it. SHRINK uses this finding to reduce the direction’s contribution across visible keys during generation. It improves mean accuracy on the four main VLMs under sampled reasoning. Together, these results show that the key direction of the sink token offers a handle for controlling attention among task-input tokens.

LLM Usage Statement

OpenAI Codex was used during manuscript preparation to assist with drafting and revising text, reviewing related literature, preparing figures and tables, analyzing experimental results, and discussing research design and interpretation. Fable 5.2 was also used to help identify and explore relevant literature, and Claude was used to help write and revise the manuscript text. All AI-assisted content was reviewed by the authors, who remain responsible for the research methodology, scientific claims, experimental results, and final manuscript.

Supplementary Material

Experimental Details

Model checkpoints and inputs. Table 5 lists the checkpoints. Each uses its own chat template with one user turn: image, then text. Qwen2.5-VL receives the system prompt “You are a helpful assistant.” LLaVA-OneVision images are resized to 384×384384 \times384 with bicubic interpolation. LLaVA-1.5 images are padded to a square with the CLIP mean color.

Reasoning instruction. Every arm receives the following instruction after the question and lettered options:

Reason step by step before choosing an option. Explain the evidence and check the candidate options. Do not give only an option letter. Write your reasoning inside <think> and </think> tags (ordinary text formatting). Finish with a separate final line exactly like “<answer>A</answer>”, outside those tags, replacing A with your chosen option letter.

MMStar retains its original question text, including options. Other benchmarks use a question followed by an explicit options list. Strict scoring accepts an answer tag or an explicit final-answer line; a sentence such as “Therefore, option D is the correct choice” alone is rejected. Conflicting explicit decisions are also rejected.

Benchmark splits. Table 6 lists validation and test sizes. The first four core benchmarks use fixed 20/80 hash splits; POPE is grouped by image. MMStar assigns all 18 calibration questions to validation, then hashes the remaining questions. VSR uses its official zero-shot splits, and WhatsUp, COCO-QA, and VG-QA use AdaptVis’s validation/test splits. We retain those splits when converting each subset’s open-ended answers into lettered options over fixed option words.

M3CoT uses an image-grouped 20/80 hash split of its released validation set. The nine further benchmarks also use grouped hash splits: questions sharing an image stay together, as do MMVP’s question pairs and HallusionBench’s figure sets. The validation and test groups are disjoint. All reported scores use only test questions, with β\beta selected on validation.

Dataset composition and metric. MMBench-EN uses its development questions in a single pass without circular evaluation. POPE contains 3,000 questions in each of its adversarial, popular, and random subsets over 500 COCO val2014 images. CV-Bench contains 788 count, 650 relation, 600 depth, and 600 distance questions. EmbSpatial uses its released test set; MMStar contains 1,500 questions before splitting. WhatsUp includes Controlled Images A+B; COCO-QA and VG-QA include the one- and two-object subsets. POPE, MME, and HallusionBench present Yes/No as lettered choices. We report strict per-question accuracy on all benchmarks, including MMVP, MME, and HallusionBench; their scores here therefore use this common metric rather than benchmark-specific pair or aggregate scores.

Inference and stopping. The main sampled evaluation uses vLLM 0.3.1 [38], bf16, and H100 GPUs, with at most 2,048 new tokens. A loop guard stops generation if the last 96 tokens repeat with a period of at most 24 tokens. Sampling uses temperature 0.7 and top-p=1top\text{-}p = 1, without top-ktop\text{-}k truncation or repetition penalties. Seeds 1–3 are shared between arms for each question and replicate. Greedy evaluation uses one run per arm and selects strengths independently (Appendix § 16).

Strict scoring. A response counts as correct only if it ends naturally and gives an unambiguous explicit decision matching the label. We accept an answer tag or a final line such as “Final answer: A” or “The answer is A.” Cap or loop-guard stops, conflicting decisions, and prose without an accepted decision count as wrong. The instruction and accepted answer forms above apply to every arm of the main evaluation.

Uncertainty and averaging. Accuracy is averaged over the three sampling seeds. Confidence intervals use 10,000 paired bootstrap resamples over test questions, seed 0, applied to each question’s seed-mean edited-minus-unedited change. For Core-5, Spatial-4, and All-nine averages, questions are resampled independently within each benchmark and the resulting changes are averaged with equal benchmark weights. Pointwise 95% intervals condition on the validation-selected strength. A gain (*) or loss (†) is marked only when its interval excludes zero; boldface in the main tables indicates a numerical comparison, not significance. All-19 averages also weight benchmarks equally but are descriptive, without aggregate significance tests. Changes are computed before rounding.

Calibration and Native Key Geometry

Sink-position and emergence profiling uses 16 label-free questions (eight each from MMStar and M3CoT) for LLaVA-OneVision-7B, Qwen2.5-VL-7B, InternVL3-8B, InternVL3.5-8B, and both Gemma models. It uses 20 questions (ten each) for Qwen2.5-VL-3B, Qwen3-VL-4B, and LLaVA-1.5-7B. The extreme-logit diagnostic uses 20 questions per model. All MMStar calibration questions are assigned to its validation split, so none is reported. M3CoT calibration uses test questions, disjoint from the validation-split evaluation used here. Labels are never read. Table 7 gives the native projection and sign statistics.

Modelzs<0z_s < 0rj<0r_j < 0: image / text / generatedMean image cosine: early / middle / late
LLaVA-OneVision-7B1.641.3/36.1/35.9+0.46/ +0.17/ +0.01
Qwen2.5-VL-3B2.116.3/10.0/13.3+0.30/ +0.32/ +0.36
Qwen2.5-VL-7B1.97.4/3.9/4.9+0.56/ +0.35/ +0.26
Gemma-3-4B0.112.7/4.0/2.9+0.26/ +0.38/ +0.37
InternVL3-8B1.146.5/45.1/47.1+0.41/ +0.13/ −0.04
LLaVA-1.5-7B15.596.2/95.9/95.9−0.52/ −0.60/ −0.62
Qwen3-VL-4B0.416.1/11.2/16.4+0.39/ +0.24/ +0.24
InternVL3.5-8B0.324.9/12.8/19.5+0.28/ +0.16/ +0.16
Gemma-3-12B0.220.8/8.8/6.3+0.17/ +0.37/ +0.22

Table 7. Native geometry on the 20-question diagnostic. Shares are percentages. Negative sink-score shares are over query head-steps selected at τ=0.2\tau= 0.2; negative projection ratios exclude the sink. Image-key cosines are means within early/middle/late layer thirds.

ModelHugging Face checkpoint
LLaVA-OneVision-7Bllava-hf/llava-onevision-qwen2-7b-ov-hf
Qwen2.5-VL-3BQwen/Qwen2.5-VL-3B-Instruct
Qwen2.5-VL-7BQwen/Qwen2.5-VL-7B-Instruct
Gemma-3-4Bgoogle/gemma-3-4b-it
InternVL3-8BOpenGVLab/InternVL3-8B-hf
LLaVA-1.5-7Bllava-hf/llava-1.5-7b-hf
Qwen3-VL-4BQwen/Qwen3-VL-4B-Instruct
InternVL3.5-8BOpenGVLab/InternVL3.5-8B-HF
Gemma-3-12Bgoogle/gemma-3-12b-it

Table 5. Model checkpoints. The first five models form the main evaluation; the remaining four extend the architecture sweep.

BenchmarkValidationTest
MMBench-EN dev8663,463
POPE1,8007,200
CV-Bench5282,110
EmbSpatial-Bench7282,912
MMStar2781,222
VSR3401,222
WhatsUp163657
COCO-QA5372,150
VG-QA2901,161
M3CoT196912
AI2D7442,344
ScienceQA-IMG3741,643
A-OKVQA226919
MMVP66234
RealWorldQA (MC)85343
MME4541,920
HallusionBench189762
BLINK (single-image)153640
MMMU (single-image MC)161640

Table 6. Validation and test questions. Additional benchmarks use the same strength-selection and strict-scoring protocol as the main table.

In the 20-question label-free diagnostic, we teacher-force unedited chain-of-thought answers through the fp32 Hugging Face reference. Among decoding head-steps selected by Eq. (14), the sink score is positive in 97.9–99.9% for eight of the nine VLMs, and in 84.5% for LLaVA-1.5-7B. Mean image-key cosines with u\mathbf{u} reach 0.6 in magnitude. Negative projection ratios occur in 41.3% of LLaVA-OneVision image keys, 7.4% of Qwen2.5-VL-7B image keys, and 96.2% of LLaVA-1.5 image keys. These decoding diagnostics differ from the all-head prompt-pass measurements in Appendix F.

SHRINK Formulation and Implementation

This appendix gives the full key-set formulation, derivations, calibration rules, and implementation details of § 4. The main method applies the projection to all visible keys in selected query heads; the sink-only variant is a comparison.

Sink Identification and Native Direction

Attention notation and the key decomposition are given in Appendices E and F. SHRINK shrinks the sink-aligned component of every visible key in selected query heads. The sink-only variant serves as a comparison that changes competition with the sink without changing relative non-sink attention.

The sink position ss is a template token before the image. We identify it once per model using 16–20 calibration questions from MMStar and M3CoT. Calibration reads only the model’s own unedited chain-of-thought answers and attention, never labels. We run an fp32 teacher-forced pass and choose the pre-image template position with maximum attention averaged over decoding rows, heads, questions, and the last third of layers. We assign all MMStar calibration questions to its validation split, so none of them is reported (§ 5.1). The M3CoT calibration items come from its test split and are disjoint from our validation-split evaluation. Appendix B gives the per-model allocation.

Because the prefix through ss and its position are fixed, causality makes ks\mathbf{k}_s identical for every input. It defines a unit vector for each layer and KV head,

ul,g=ksl,g∥ksl,g∥2.(7)\mathbf{u}^{l,g} = \frac{\mathbf{k}_s^{l,g}}{\lVert\mathbf{k}_s^{l,g} \rVert_2}. \tag*{(7)}

All query heads in a GQA group share this direction. To measure another key’s component along it, we define the projection ratio

rj=kjTu∥ks∥2.(8)r_j = \frac{\mathbf{k}_j^{\mathsf{T}}\mathbf{u}}{\lVert\mathbf{k}_s\rVert_2}. \tag*{(8)}

This is the coefficient in Eq. (22): rs=1r_s = 1, whereas rj=0r_j = 0 for a key orthogonal to u\mathbf{u}. Content keys need not be orthogonal to u\mathbf{u}. Their components along this direction make directional shrinkage distinct from changing only the sink logit.

Sink-Direction Key Projection

For strength β∈(0,1]\beta\in(0,1] and key set St⊆VtS_t \subseteq\mathcal{V}_t, SHRINK applies

kj′={kj−β(kjTu)u,j∈St,kj,j∉St.(9)\mathbf{k}'_j = \begin{cases} \mathbf{k}_j - \beta(\mathbf{k}_j^{\mathsf{T}}\mathbf{u})\mathbf{u}, & j \in S_t,\\ \mathbf{k}_j, & j \notin S_t. \end{cases} \tag*{(9)}

The corresponding logit change is

Δzt,j=−βcl(qtTu)(kjTu),j∈St.(10)\Delta z_{t,j} = -\beta c_l(\mathbf{q}_t^{\mathsf{T}}\mathbf{u})(\mathbf{k}_j^{\mathsf{T}}\mathbf{u}), \qquad j \in S_t. \tag*{(10)}

Equivalently, SHRINK replaces rjr_j by (1−β)rj(1-\beta)r_j in Eq. (23), leaving kj′\mathbf{k}_j' unchanged. At β=1\beta= 1 the component is removed completely; smaller strengths retain part of it.

SINK: project the sink key. For the sink-only comparison, setting St={s}S_t = \{s\} gives ks′=(1−β)ks\mathbf{k}'_s = (1-\beta)\mathbf{k}_s and zt,s′=(1−β)zt,sz'_{t,s} = (1-\beta)z_{t,s}; every other logit is unchanged. Hence every non-sink key’s margin over the sink changes by the same βzt,s\beta z_{t,s}. The conditional attention distribution among non-sink keys remains unchanged; when zt,s>0z_{t,s} > 0, their total mass increases by a shared renormalization.

ALL: project every visible key. Our main method sets St=VtS_t = \mathcal{V}_t, covering the sink, input, and generated keys visible to the current query. This yields an exact query-side interpretation. Since uuT\mathbf{u}\mathbf{u}^{\mathsf{T}} is symmetric,

qtT(I−βuuT)kj=[(I−βuuT)qt]Tkj.(11)\mathbf{q}_t^{\mathsf{T}}(I-\beta\mathbf{u}\mathbf{u}^{\mathsf{T}})\mathbf{k}_j = [(I-\beta\mathbf{u}\mathbf{u}^{\mathsf{T}})\mathbf{q}_t]^{\mathsf{T}}\mathbf{k}_j. \tag*{(11)}

Thus projecting all keys is equivalent to using qt′=(I−βuuT)qt\mathbf{q}'_t = (I-\beta\mathbf{u}\mathbf{u}^{\mathsf{T}})\mathbf{q}_t. Combining Eqs. (8) and (10), the change in key jj’s margin over the sink is

Δ(zt,j−zt,s)=βcl(qtTu)∥ks∥2(1−rj).(12)\Delta(z_{t,j}-z_{t,s}) = \beta c_l(\mathbf{q}_t^{\mathsf{T}}\mathbf{u})\lVert\mathbf{k}_s\rVert_2(1-r_j). \tag*{(12)}

When qtTu>0\mathbf{q}_t^{\mathsf{T}}\mathbf{u} > 0, keys with rj<1r_j < 1 gain on the sink, and keys with smaller projection ratios gain more. More generally, the change in the log attention ratio of any two visible positions is

Δlog⁡at,jat,i=−βzt,s(rj−ri).(13)\Delta\log\frac{a_{t,j}}{a_{t,i}} = -\beta z_{t,s}(r_j-r_i). \tag*{(13)}

Thus the all-key operation changes competition among content tokens whenever their projection ratios differ. This is not equivalent to subtracting a common bias from every logit, which would cancel under softmax. It does not guarantee that every content token gains attention, and no positive-sink-score gate is used in the SHRINK implementation.

Selecting Heads at Each Token

Let I\mathcal{I} be the image positions and T\mathcal{T} the question and answer-option positions. Selection uses the native attention from Eq. (18), before any edit. At every generated token, we select

Htl={h:∑j∈Iat,jnat,l,h≥τ ∨ ∑j∈Tat,jnat,l,h≥τ}.(14)\mathcal{H}_t^l = \left\{h : \sum_{j\in\mathcal{I}} a_{t,j}^{\mathrm{nat},l,h} \ge\tau\ \lor\ \sum_{j\in\mathcal{T}} a_{t,j}^{\mathrm{nat},l,h} \ge\tau\right\}. \tag*{(14)}

We use τ=0.2\tau= 0.2. The image condition follows VAR’s initial image-attention filter [36]; VAR additionally selects by the fraction of visual attention assigned to non-sink tokens. SPIN likewise uses native image attention to select heads [64]. Our text condition extends selection to heads reading the question or choices. Because we implement Eq. (10) per query head, an edited query head does not alter unselected siblings that share its KV head under GQA. The head-selection rule is recomputed from the native attention of the current row, before that row’s edit.

On unedited calibration decoding, this rule selects an average of 8–19% of query heads per step in the edited layers.

Scheduling Layers

Let aˉll\bar{a}_l^l denote mean calibration-set sink attention in layer ll. Within the first half of an LL-layer decoder, we define the sink emergence layer as the largest one-layer rise,

Table 8. Frozen sink and layer configuration. A dash denotes no excluded layer.

Modelssℓstart\ell_{\mathrm{start}}Lexcl\mathcal{L}_{\mathrm{excl}}
LLaVA-OneVision-7B000, 27
LLaVA-1.5-7B02–
Qwen2.5-VL-3B030
Qwen2.5-VL-7B240, 1
Qwen3-VL-4B17–
InternVL3-8B240, 1
InternVL3.5-8B07–
Gemma-3-4B05–
Gemma-3-12B014

Table 8.

ℓemg=arg⁡max⁡1≤l<L/2(aˉsl−aˉsl−1).(15)\ell_{\mathrm{emg}} = \underset{1 \leq l < L/2}{\arg\max}\left(\bar{a}_{s}^{l} - \bar{a}_{s}^{l-1}\right). \tag*{(15)}

We also exclude any layer in which more than 1% of selected calibration head-steps have an extreme native sink logit,

Lexcl={l:Pr⁡(∣zt,sl,h∣>50∣h∈Htl)>0.01}.(16)\mathcal{L}_{\mathrm{excl}} = \left\{l : \Pr\left(\left|z_{t,s}^{l,h}\right| > 50 \mid h \in\mathcal{H}_{t}^{l}\right) > 0.01\right\}. \tag*{(16)}

Let ℓstart=ℓemg\ell_{\mathrm{start}} = \ell_{\mathrm{emg}} by default. LLaVA-OneVision, the first model we configured, keeps its initial schedule: edits start at layer 0, and layers 0 and 27 are excluded. Layer 0 meets the extreme-logit rule, and layer 27 has 0.8% extreme head-steps; its measured emergence layer is 4. The edited layer set is

Ledit={l:l≥ℓstart}∖Lexcl.(17)\mathcal{L}_{\mathrm{edit}} = \{l : l \geq\ell_{\mathrm{start}}\} \setminus\mathcal{L}_{\mathrm{excl}}. \tag*{(17)}

The sink and emergence profiles use 16–20 questions; the extreme-logit diagnostic uses 20 questions. No labels tune these choices.

The sink token is <bos> for both Gemma 3 models, <s> for LLaVA-1.5, <|im_start|> for LLaVA-OneVision, Qwen2.5-VL-3B, and InternVL3.5-8B, user for Qwen3-VL-4B, and the newline after the role tag for Qwen2.5-VL-7B and InternVL3-8B.

Implementation Details

We edit the query that predicts the first answer token and every generated-token query thereafter; earlier prompt rows remain native. The implementation reads the native post-RoPE sink key for each layer and KV head and computes the projection in vLLM [38]. The stored keys and values are never modified. An edit is skipped whenever the sink is outside the query’s attention window. All reported results use the direct path: transient projected keys rounded to the key dtype. An optimized query-side path also exists, based on Eq. (11), but produces none of the reported results. The equivalence is exact in real arithmetic.

Against an independent fp32 Hugging Face implementation, next-token log-probabilities differ by at most 2.5×10−42.5 \times10^{-4} over 18 key-projection test cases per model, and by at most 4×10−54 \times10^{-5} for six of the nine VLMs. Every job must pass this check before running. The intervention requires no training or additional model forward pass. We report shrinkage strengths and evaluation choices in § 5.

Head selection and attention recomputation add inference work, although no additional full model forward pass is required.

Controlled Key and Value Transfer

Figure 2a shows LLaVA-OneVision-7B, Qwen2.5-VL-7B, and Gemma-3-4B, one model per family. Table 9 retains Qwen2.5-VL-3B and all six conditions. Each model uses the same 240 MMStar and 240 CV-Bench 2D questions, with 24 disjoint calibration questions per dataset to identify the sink and recipient before evaluation. Recipient attention is measured from the last prompt query, averaged over all heads and the final third of decoder layers, then over questions. The 95% interval uses 10,000 question-bootstrap resamples within category strata, seed 20261009, pairing conditions within each resample. Each question has a distinct image.

ModelDatasetSTDBLOCKRRSRRSSSSR−RR [95% CI]
LLaVA-OneVision-7BMMStar0.00649.8580.06242.3350.00849.754+42.27 [+41.19, +43.36]
CV-Bench 2D0.00551.5560.05846.6020.00751.430+46.54 [+46.02, +47.04]
Qwen2.5-VL-7BMMStar0.07242.1140.09976.4330.08943.546+76.33 [+76.18, +76.49]
CV-Bench 2D0.06141.9750.08777.2900.06943.863+77.20 [+77.11, +77.29]
Gemma-3-4BMMStar0.2790.0030.51115.0430.62746.365+14.53 [+14.36, +14.71]
CV-Bench 2D0.2970.0020.50416.1060.63245.025+15.60 [+15.45, +15.76]
Qwen2.5-VL-3BMMStar0.02426.0930.0282.9970.07426.289+2.97 [+2.81, +3.14]
CV-Bench 2D0.01926.0250.0265.0550.06426.207+5.03 [+4.86, +5.19]

Table 9. Recipient attention (%) under all six key/value-transfer conditions, with 240 questions per model and dataset. The last column gives the paired key-copy increase over the matched no-copy control, in percentage points, with its 95% interval. Figure 2a shows the first three models.

Qwen2.5-VL-7B and Gemma use the original reference runs; LLaVA-OneVision and Qwen2.5-VL-3B use the same protocol in the fp32 reference. A 24-question check per dataset reproduces Qwen2.5-VL-7B’s sink, recipient, and prompt lengths, with mean attention differences at most 0.0151 percentage points and maximum individual difference 0.250 points. Gemma’s backend supports bidirectional image attention. The probe questions overlap the main evaluation splits (107 validation and 373 test questions), but this probe does not select SHRINK strengths. The final-third means include propagation through earlier edited layers.

The recipient is the first natively eligible pre-content template position after the sink. All transplanted vectors come from the same prompt’s unedited source pass. The original sink is blocked as a key for other queries, with its native self-row retained; all other native permissions remain. RR fixes the ordinary recipient key and value, SR transfers the sink key only, RS the sink value only, and SS both. BLOCK allows natural relocation without clamping; the matched key-copy effect is SR minus RR. Joint key/value transfer can differ from key-only transfer after propagation, particularly on Gemma, so key-mediated attraction does not imply that values are functionally interchangeable. The VLM CV-Bench sample contains count and relation questions. This fp32 direct-answer probe is separate from the bf16 reasoning-generation evaluation.

For Qwen2.5-VL-7B, accuracy is measured with direct-answer option probabilities on the same 240 questions per dataset. Key-only transfer reduces accuracy by 35.0 points on MMStar (95% CI [−43.3, −26.7]) and 40.8 on CV-Bench (2D) ([−47.9, −33.3]). Joint transfer changes accuracy by +1.2 and +0.8 points from STD accuracies of 64.6% and 78.3%. Its recipient attention is also different (43.5–43.9%), so this is not a fixed-attention comparison of values. Intervals use 10,000 paired category-stratified question-bootstrap resamples, seed 20261009; the original planned contrast family uses Holm correction. These propagated outcomes show why transferable attraction need not imply functional equivalence.

Attention Response to Additive Key Transfer

The transplant in § 3 tests full key replacement. Here we derive the response to a fixed additive transfer. For a query qtq_t, post-positional-encoding keys kik_i, and visible positions Vt\mathcal{V}_t, scaled dot-product attention [78] gives

zt,i=clqtTki,at,i=exp⁡zt,i∑m∈Vtexp⁡zt,m,(18)z_{t,i}=c_l q_t^{\mathsf{T}} k_i,\qquad a_{t,i}=\frac{\exp z_{t,i}}{\sum_{m\in\mathcal{V}_t}\exp z_{t,m}}, \tag*{(18)}

where cl>0c_l > 0 is the layer’s attention scale. Hold the query and all other keys fixed, omit the query index, and write zs=clqtTksz_s=c_l q_t^{\mathsf{T}} k_s. The analytical additive response below is distinct from the empirical native-component sweep in Appendix § 14.

Key addition transfers part of the sink’s score. For a recipient j≠sj\ne s, add a fraction δ≥0\delta\ge0 of the sink key while retaining the recipient’s original value:

kj(δ)=kj+δks,k_j(\delta)=k_j+\delta k_s,
zj(δ)=clqtT(kj+δks)=zj+δzs.(19)z_j(\delta)=c_l q_t^{\mathsf{T}}(k_j+\delta k_s)=z_j+\delta z_s. \tag*{(19)}

The recipient therefore gains δ\delta times the sink’s native score. Its value remains unchanged: the intervention changes how strongly the token is retrieved, while preserving the content it supplies.

The score change produces a graded attention response. Only the recipient’s score changes. Substituting Eq. (19) into the softmax gives

aj(δ)=exp⁡(zj+δzs)exp⁡(zj+δzs)+∑i≠jexp⁡zi=ajexp⁡(δzs)1+aj[exp⁡(δzs)−1],(20)a_j(\delta)=\frac{\exp(z_j+\delta z_s)}{\exp(z_j+\delta z_s)+\sum_{i\ne j}\exp z_i} =\frac{a_j\exp(\delta z_s)}{1+a_j[\exp(\delta z_s)-1]}, \tag*{(20)}

where the sum is over the other visible positions and aj=aj(0)a_j=a_j(0) is the native attention. Differentiating yields

∂aj(δ)∂δ=zsaj(δ)[1−aj(δ)].(21)\frac{\partial a_j(\delta)}{\partial\delta}=z_s a_j(\delta)[1-a_j(\delta)]. \tag*{(21)}

When zs>0z_s>0, the derivative is positive: increasing the transfer strength continuously increases recipient attention, approaching one as δ\delta grows. The recipient’s attention ratio against any unedited token is multiplied by exp⁡(δzs)\exp(\delta z_s). The sink key therefore serves as an attention control signal: δ\delta sets the transfer strength, and the query’s alignment with the sink key determines the signal’s effect. If zs=0z_s=0 the edit has no effect; if zs<0z_s<0 its effect reverses. High sink attention alone does not establish this sign condition.

This is an exact fixed-query response, not an additional empirical sweep. Appendix § 14 instead tests scaling an existing native component through the full network.

Native Sink-Component Probe

This probe tests whether changing a token’s existing sink-direction component moves its attention as predicted by Eq. (26). It is distinct from the full key/value replacement in Appendix § 12 and the fixed additive edit in Eq. (19). Only prompt attention is measured; the probe does not evaluate accuracy or attention during generation.

Exact local response. For a nonzero sink key, decompose an ordinary key into an orthogonal component kj⊥k_j^{\perp} and a parallel component:

kj=kj⊥+rjks,rj=kjTks∥ks∥22.(22)k_j=k_j^{\perp}+r_jk_s,\qquad r_j=\frac{k_j^{\mathsf{T}}k_s}{\lVert k_s\rVert_2^2}. \tag*{(22)}

Its native score separates as

zj=clqlTkj⊥+rjzs.(23)z_j = c_l q_l^{\mathsf{T}} k_j^{\perp} + r_j z_s. \tag*{(23)}

The probe scales the parallel component while retaining the orthogonal component and value locally. With u=ks/∥ks∥2\mathbf{u} = \mathbf{k}_s/\lVert\mathbf{k}_s\rVert_2,

kj(η)=kj+η(kjTu)u=kj⊥+(1+η)rjks.(24)\begin{aligned} \mathbf{k}_j(\eta) = \mathbf{k}_j + \eta(\mathbf{k}_j^{\mathsf{T}}\mathbf{u})\mathbf{u} \\ &= \mathbf{k}_j^{\perp} + (1+\eta)r_j\mathbf{k}_s. \tag*{(24)} \end{aligned}

With the query and all other keys fixed, Eqs. (23) and (24) give

Δzj=ηrjzs,(25)\Delta z_j = \eta r_j z_s, \tag*{(25)}

Substituting this score change into the softmax,

logit⁡aj(η)=logit⁡aj+ηrjzs,(26)\operatorname{logit} a_j(\eta) = \operatorname{logit} a_j + \eta r_j z_s, \tag*{(26)}

where logit⁡p=log⁡[p/(1−p)]\operatorname{logit} p = \log[p/(1-p)]. Equivalently, the probability follows Eq. (20) with score increment ηrjzs\eta r_j z_s. This increment can have either sign even for η>0\eta> 0, whereas fixed additive transfer increases the coefficient by α\alpha irrespective of its original sign.

Protocol and Controls

Models and questions. We probe the four main models: LLaVA-OneVision-7B, Qwen2.5-VL-3B/7B, and Gemma-3-4B. Each receives the same 100 validation questions: 50 MMStar and 50 CV-Bench (2D), sampled separately with seed 0. No test question or answer label is used. Prompts contain the image, the model’s chat template, and the same reasoning instruction as the main evaluation. Each condition is a single full prompt forward pass through the Hugging Face reference, with fp32 computation on bf16-rounded weights. This differs from the bf16 vLLM decoding evaluation.

Recipients and edit schedule. The primary run draws one image token uniformly per question and, separately, one question-text token, seeded by question identifier. Question text excludes answer options and instructions. Each recipient is edited alone, after positional encoding, in every KV head and every layer from sink emergence onward (Table 10). This schedule deliberately differs from SHRINK’s selected heads and excluded layers. At each layer the operation leaves the orthogonal key component and value tensor unchanged locally. Edits to earlier layers can nevertheless change the hidden states, queries, other keys, and values arriving at later layers. Thus this probe does not hold the entire forward pass fixed or test the simultaneous editing of many tokens.

ModelEdited layersQuery / KV headsSink positionMean prompt lengthUnits
LLaVA-OneVision-7B4–2728 / 401,61267,200
Qwen2.5-VL-3B3–3516 / 2040052,800
Qwen2.5-VL-7B4–2728 / 4240067,200
Gemma-3-4B5–338 / 4039023,200

Table 10. Native-component probe configuration. Layers and positions are zero-indexed. Every KV head is edited from sink emergence through the final layer; no head selection or layer exclusion is applied. The last column counts question-layer-query-head units per recipient kind, over 100 questions.

Conditions. The unedited pass provides native attention and projection coefficients. We apply Eq. (24) at η∈{0.5,1,2,4}\eta\in\{0.5, 1, 2, 4\}, and remove the component at η=−1\eta= -1. For each positive strength, additional conditions use perturbation magnitude

d=∣η∣∣kjTu∣.(27)d = |\eta| \lvert\mathbf{k}_j^{\mathsf{T}}\mathbf{u}\rvert. \tag*{(27)}

RAND adds dwd\mathbf{w}, where w\mathbf{w} is a random unit vector orthogonal to u\mathbf{u}, fixed by question, recipient kind, layer, and KV head across strengths. NORM adds dkj/∥kj∥2d\mathbf{k}_j/\lVert\mathbf{k}_j\rVert_2, lengthening the key in its own direction. Because this also scales its sink coefficient, NORM is not a clean null for the component hypothesis. MQ adds dqˉ/∥qˉ∥2d\bar{\mathbf{q}}/\lVert\bar{\mathbf{q}}\rVert_2, where qˉ\bar{\mathbf{q}} averages post-positional-encoding queries over later prompt rows and query heads in the recipient’s GQA group. This uses prompt queries as a reference direction, rather than as a proposed decoding intervention. Each arm computes its perturbation from the tensors arriving at that layer; those tensors can differ between arms after earlier edits propagate.

The sweep parameter satisfies α=1+η\alpha= 1+\eta in the main text’s notation. SHRINK corresponds to η=−β\eta=-\beta and edits all visible keys simultaneously; this single-recipient sweep tests controllability, not its chosen decoding strengths. Positional components can reflect position as well as content; the probe does not separate those sources.

Measurement and intervals. For each question, edited layer, and query head, we first average attention to the recipient over all prompt rows after it. We then form edited-to-unedited ratios and report their geometric mean over these units. Sign strata use rjr_j from the unedited pass at each layer and KV head, repeated across its query heads. A token can belong to opposite strata in different layers or heads. The final prompt row, which predicts the first answer token, is also measured separately. To handle zeros in the saved fp16 attention summaries, ratios use (aedit+ϵ)/(anative+ϵ)(a^{\mathrm{edit}}+\epsilon)/(a^{\mathrm{native}}+\epsilon) with ϵ=10−12\epsilon=10^{-12}. The 95% intervals use 2,000 bootstrap resamples of questions, seed 0, moving all units of a question together because layer-head units within a question are correlated. These original tables retain ϵ=10−12\epsilon=10^{-12}. Figure 2(c) is recomputed with ϵ=6×10−8\epsilon=6\times10^{-8}, near the smallest positive fp16 value, using the same question bootstrap. The signed-stratum responses persist, but the magnitudes of extreme decreases depend on storage precision. For example, the LLaVA-OneVision negative image stratum at η=4\eta=4 changes from 1.8×10−61.8\times10^{-6} to 1.7×10−31.7\times10^{-3}; restricting to nonzero summaries gives 1.3×10−21.3\times10^{-2} and excludes a substantial fraction of units. None resolves probabilities rounded to zero. These large amplification settings lie outside SHRINK’s partial-removal range.

Signed Response and Alternative Directions

The grey range in Figure 2c spans four RAND means at each matched edit size: image and question recipients, each split by the native coefficient’s sign. It is a descriptive minimum-to-maximum range, not a confidence interval. Figure 5 shows the four curves separately. A random direction orthogonal to the sink direction need not be orthogonal to a query, so its score change need not vanish.

Figure 5. The four random-control means summarized by the grey range in Figure 2c. Filled and open markers distinguish positive and negative native components; circles and triangles distinguish image and question recipients. Perturbations match the local key-change size of the sink-direction edit. A ratio of 1 means unchanged attention.

Both signs occur in the unedited keys. Pooling layers and heads, negative components account for 48.2%, 19.4%, 9.8%, and 7.9% of image-key measurements on LLaVA-OneVision, Qwen2.5-VL-3B, Qwen2.5-VL-7B, and Gemma-3-4B, respectively. We abbreviate the last three as Qwen-3B, Qwen-7B, and Gemma.

Table 11 and Table 12 give the primary results with stratum sizes and intervals. At η=1\eta= 1, attention moves in the coefficient-predicted direction in 67–92% of positive-coefficient units and 85–99% of negative-coefficient units. Removal reverses these directions in 86–100% and 89–97%, respectively. At the same strength, RAND agrees with the coefficient-based direction in 46–62% of units, with a modest increase for positive coefficients.

ModelRecipientNNShare (%)BoostRemovalRAND
LLaVA-OneVision-7Bimage34,79751.85.18 [4.82, 5.58]0.0135 [0.0128, 0.0142]1.8 [1.74, 1.86]
question39,52258.85.59 [5.31, 5.89]0.0112 [0.0106, 0.0119]1.46 [1.42, 1.49]
Qwen2.5-VL-3Bimage42,56080.68.6 [7.72, 9.62]0.00705 [0.0067, 0.00742]1.17 [1.12, 1.21]
question49,36093.58.37 [7.97, 8.81]0.00671 [0.00634, 0.00712]1.19 [1.16, 1.22]
Qwen2.5-VL-7Bimage60,59990.23.14 [2.78, 3.57]0.0333 [0.0316, 0.0352]1.17 [1.14, 1.2]
question64,96096.75.57 [5.2, 5.98]0.022 [0.0208, 0.0231]1.28 [1.26, 1.3]
Gemma-3-4Bimage21,36092.19.63 [8.5, 11]0.0111 [0.00981, 0.0125]1.32 [1.27, 1.38]
question22,81498.315.4 [14.3, 16.5]0.00216 [0.00195, 0.00241]1.35 [1.32, 1.37]

Table 11. Random recipients, units with rj>0r_j > 0: geometric-mean recipient-attention ratios and 95% question-bootstrap intervals. Boost doubles the native coefficient (η=1\eta= 1); removal sets it to zero (η=−1\eta= -1). RAND uses a random orthogonal direction at η=1\eta= 1. NN counts question-layer-query-head units from 100 questions per model. Share is the percentage of all units in the stratum.

ModelRecipientNNShare (%)BoostRemovalRAND
LLaVA-OneVision-7Bimage32,40348.20.0245 [0.0204, 0.0291]9.92 [9.23, 10.7]1.01 [0.994, 1.03]
question27,67841.20.105 [0.0981, 0.112]4.67 [4.45, 4.91]0.988 [0.977, 1]
Qwen2.5-VL-3Bimage10,24019.40.129 [0.118, 0.142]3.37 [3.18, 3.58]0.988 [0.973, 1]
question3,4406.50.146 [0.126, 0.166]3.03 [2.77, 3.33]0.989 [0.962, 1.01]
Qwen2.5-VL-7Bimage6,6019.80.127 [0.101, 0.162]2.71 [2.5, 2.93]0.964 [0.94, 0.987]
question2,2403.30.326 [0.285, 0.372]1.89 [1.78, 2]0.989 [0.959, 1.02]
Gemma-3-4Bimage1,8407.90.128 [0.102, 0.16]4.6 [4.31, 4.93]1.05 [1.01, 1.1]
question3861.70.268 [0.23, 0.317]2.26 [2.08, 2.45]0.996 [0.969, 1.02]

Table 12. Random recipients, units with rj<0r_j < 0: geometric-mean recipient-attention ratios and 95% question-bootstrap intervals. Boost doubles the native coefficient (η=1\eta= 1); removal sets it to zero (η=−1\eta= -1). RAND uses a random orthogonal direction at η=1\eta= 1. NN counts question-layer-query-head units from 100 questions per model. Share is the percentage of all units in the stratum.

Figure 6 compares the native-component edit with RAND, NORM, and MQ. MQ raises attention by 14.9–64.3 across all units at η=1\eta= 1, and by 3.7–31.8 in negative-coefficient units where native-component scaling lowers it. Across strengths up to four, the all-unit ratio reaches 1,288. Query alignment can therefore attract more attention than the sink direction at the same local perturbation norm. The property studied here is that ordinary keys already carry a signed component along the native sink direction, which can be isolated and scaled.

Figure 6. The sink direction is a signed control direction, not a maximum-attraction direction. Random-recipient results at η=1\eta= 1, split by the native coefficient. Points are geometric-mean attention ratios over later prompt rows; bars are 95% question-bootstrap intervals. All conditions use the local perturbation norm in Eq. (27). NORM also scales the sink component. The mean-query reference (MQ) attracts more strongly and raises attention even in the negative-coefficient stratum.

The all-unit averages in Table 13 show how the sign mix shapes whole-token responses: LLaVA-OneVision image-recipient attention falls to 0.39 at η=1\eta= 1, because 48.2% of its units have negative coefficients and their attention falls sharply. Its question-recipient ratio is 1.09. In Qwen and Gemma, all-unit ratios rise by 2.3–14.4, while negative-coefficient units account for only 1.7–19.4% of the primary-run strata. The final prompt row shows the same positive/negative response pattern. No confidence intervals were computed for those last-row ratios.

ModelRecipientBoostMQrj>0r_j > 0rj<0r_j < 0
LLaVA-OneVision-7Bimage0.392 [0.346, 0.442]45.7 [40.5, 52.1]6.1/0.01090.002/12.8
question1.09 [1.02, 1.15]19.6 [17.7, 21.5]4.95/0.00540.0588/5.39
Qwen2.5-VL-3Bimage3.81 [3.49, 4.19]21.5 [18.9, 24.4]22.2/0.002720.0553/3.8
question6.43 [6.11, 6.75]22.8 [21 24.9]9.44/0.00370.115/3.33
Qwen2.5-VL-7Bimage2.29 [2.05, 2.58]14.9 [12.8, 17.6]4.54/0.01460.0262/3.72
question5.07 [4.7, 5.46]35.3 [31.1, 40.2]7.68/0.01090.203/1.98
Gemma-3-4Bimage6.83 [6.04, 7.81]30.9 [27.1, 35.2]18.8/4.66 × 10−410^{-4}0.0539/8.05
question14.4 [13.4, 15.4]64.3 [58.5, 70.2]24.4/2.41 × 10−410^{-4}0.141/2.23

Table 13. Random recipients: whole-population and first-answer-query results. The first two numeric columns aggregate all units using attention averaged over later prompt rows (95% intervals). MQ is the matched-norm mean-query reference. The last two columns use only the final prompt row, split by native coefficient sign, and give boost / removal ratios; these last-row ratios are shown without intervals. Boost and MQ use η=1\eta= 1; removal uses η=−1\eta= -1.

Selecting Recipients with More Negative Coefficients

An additional unedited pass measures rjr_j for every image or question-text candidate across edited layers and KV heads. We choose the candidate with the largest fraction of negative coefficients, breaking ties by the lowest mean coefficient, and repeat the same conditions except MQ. This selection increases the fraction of negative units but does not make a recipient negative in every layer and head: selected image/question recipients average 61/51% negative units in LLaVA-OneVision, 34/14% in Qwen-3B, 31/11% in Qwen-7B, and 21/4% in Gemma. The exact query-head-weighted shares appear in Table 14.

ModelRecipientShare (%)BoostRemovalRANDFall (%)
LLaVA-OneVision-7Bimage61.30.0123 [0.0104, 0.0145]14.3 [13.3, 15.3]1.02 [0.999, 1.05]96
question50.60.105 [0.0989, 0.112]6.47 [6.16, 6.8]1.01 [0.998, 1.02]93
Qwen2.5-VL-3Bimage34.10.118 [0.106, 0.131]4.81 [4.57, 5.08]1.01 [0.99, 1.02]95
question13.60.169 [0.152, 0.187]3.97 [3.63, 4.3]1.03 [1.01, 1.06]95
Qwen2.5-VL-7Bimage30.70.19 [0.16, 0.228]3.83 [3.63, 4.06]0.978 [0.948, 1.01]82
question11.40.28 [0.261, 0.302]2.19 [2.1, 2.28]1 [0.976, 1.03]84
Gemma-3-4Bimage21.50.366 [0.292, 0.466]4.18 [3.99, 4.37]1.05 [1.03, 1.08]72
question3.90.205 [0.171, 0.253]2.81 [2.7, 2.92]1.01 [0.986, 1.04]90

Table 14. Recipients selected for the largest fraction of negative native coefficients: results within their rj<0r_j < 0 units. Ratios and 95% intervals use mean attention over later prompt rows. All conditions use η=1\eta= 1 except removal (η=−1\eta= -1). Share gives the fraction of the selected recipient's units with rj<0r_j < 0; Fall is the percentage of those units whose attention decreases under the boost.

Within negative units, η=1\eta= 1 gives ratios 0.012–0.37, with decreases in 72–96% of units; removal raises attention by 2.2–14.3 (Table 14 and Figure 7). RAND gives 0.98–1.05. In LLaVA-OneVision, the selected image recipient also loses attention over all units: 0.143 (95% CI [0.127, 0.161]) at η=1\eta= 1 and 0.016 ([0.014, 0.019]) at η=2\eta= 2. In Qwen and Gemma, the selected recipients remain positive in most units, and their all-unit ratios rise by 2.4–13.6. For these models, the decrease appears in the negative-coefficient units.

Figure 7. Recipients selected for negative coefficients reproduce the signed response. Curves show only units with native rj<0r_j < 0, using the recipient-selection procedure in the text. Increasing the component lowers attention; removing it raises attention. The statistic, intervals, and grey RAND curves follow Figure 2c. A selected recipient can still have positive coefficients in other units, which are excluded from these curves.

Exact Local Identity and Propagation through Depth

At the first edited layer, queries and other keys are unchanged from the native pass, so Eq. (26) is an identity. Maximum absolute attention errors are 3.9×10−63.9 \times10^{-6} for random recipients and 3.5×10−63.5 \times10^{-6} for negative-selected recipients; the run rejects any question with error above 10−310^{-3}. Deeper layers no longer satisfy the fixed-query assumption. Across positive-strength arms and recipient kinds in the primary run, correlations between predicted and measured log attention ratios range from 0.91–0.97 for LLaVA-OneVision, 0.65–0.93 for Qwen-3B, 0.70–0.93 for Qwen-7B, and 0.56–0.87 for Gemma. Median measured-to-predicted changes range from 0.97–1.00, 0.53–0.99, 0.45–0.96, and 0.11–0.96, respectively. The local identity explains the edit, while the full-pass measurements establish the response after propagation.

The last-row sink score is positive in 93–100% of question-layer-query-head units across these four models. els. This all-head prompt statistic is distinct from the selected-head decoding diagnostic in Appendix § 10. The sink’s mean share falls by 1.7–6.8 percentage points in positive-coefficient strata at η=1\eta= 1, exceeding the recipient’s absolute gain, so the released mass also reaches other keys. These probes measure attention movement under a controlled key edit; § 5 evaluates task accuracy.

Full Sampled Accuracy Results

Table 15 and Table 16 give the core and spatial results for all five sampled models, using the same validation-selected strengths and three-seed test evaluation as Table 1. Together with the additional ten benchmarks in Table 20, they retain all 19 benchmark results for every model. Table 17 gives the five models’ All-19 averages. On the nine benchmarks of Table 1, InternVL3-8B’s mean is essentially unchanged (−0.03 points).

Table 15. Full core-five evaluation: test accuracy (%). Core avg. weights the five benchmarks equally. All strengths use the validation-selected settings of the main evaluation. Superscripts on SHRINK accuracies mark changes from unedited decoding: ∗/† indicate a pointwise paired-bootstrap 95% CI above/below zero.

ModelMethodMMBenchPOPECV-BenchEmbSpatialMMStarCore avg.
LLaVA-OneVision-7BUnedited64.261.251.251.138.153.2
+SHRINK70.9∗63.1∗54.9∗53.0∗43.3∗57.1∗
Qwen2.5-VL-3BUnedited73.880.958.247.344.761.0
+SHRINK74.581.9∗61.0∗49.4∗45.462.4∗
Qwen2.5-VL-7BUnedited87.386.373.766.759.474.7
+SHRINK87.186.574.266.960.775.1
Gemma-3-4BUnedited82.684.359.453.952.466.5
+SHRINK82.484.260.8∗55.0∗52.567.0∗
InternVL3-8BUnedited89.088.279.570.364.078.2
+SHRINK88.987.6†78.769.866.6∗78.3

Table 15.

Table 16. Full spatial evaluation: test accuracy (%). Spatial avg. weights these four benchmarks equally; All avg. weights all nine core and spatial benchmarks equally. Superscripts on SHRINK accuracies mark changes from unedited decoding: ∗/† indicate a pointwise paired-bootstrap 95% CI above/below zero.

ModelMethodVSRWhatsUpCOCO-QAVG-QASpatial avg.All avg.
LLaVA-OneVision-7BUnedited42.668.937.534.745.949.9
+SHRINK53.4∗71.5∗42.1∗43.0∗52.5∗55.0∗
Qwen2.5-VL-3BUnedited63.867.256.251.359.660.4
+SHRINK69.1∗74.5∗57.254.8∗63.9∗63.1∗
Qwen2.5-VL-7BUnedited84.088.764.863.375.274.9
+SHRINK85.2∗88.264.665.7∗75.9∗75.5∗
Gemma-3-4BUnedited67.658.850.648.056.362.0
+SHRINK67.464.7∗51.6∗46.5†57.5∗62.8∗
InternVL3-8BUnedited81.690.369.270.677.978.1
+SHRINK82.486.9†69.172.6∗77.778.0

Table 16.

Table 17. Full five-model evaluation: test accuracy (%) averaged equally over all 19 benchmarks. Δ\Delta is SHRINK minus Base in points, computed before rounding. Bold marks the higher mean. No significance test is applied to these averages.

ModelBaseSHRINKΔ\Delta
LLaVA-OneVision-7B52.455.9+3.5
Qwen2.5-VL-3B57.560.1+2.7
Qwen2.5-VL-7B72.773.1+0.4
Gemma-3-4B63.163.6+0.6
InternVL3-8B75.274.8−0.4

Table 17.

BenchmarkLLaVA-OV 7BQwen2.5-VL 3BQwen2.5-VL 7BGemma-3 4BInternVL3 8B
M3CoT41.8±0.3→43.6±1.641.8 \pm0.3 \to43.6 \pm1.6<br>(+1.8)(+1.8) [0.3]42.5±0.7→43.6±0.942.5 \pm0.7 \to43.6 \pm0.9<br>(+1.0)(+1.0) [0.2]61.4±0.5→61.7±1.061.4 \pm0.5 \to61.7 \pm1.0<br>(+0.3)(+0.3) [0.3]50.7±1.6→50.9±1.650.7 \pm1.6 \to50.9 \pm1.6<br>(+0.2)(+0.2) [0.5]56.9±1.6→57.7±0.856.9 \pm1.6 \to57.7 \pm0.8<br>(+0.8)(+0.8) [0.3]
AI2D59.7±1.0→61.8±1.259.7 \pm1.0 \to61.8 \pm1.2<br>(+2.1)∗(+2.1)^{*} [0.3]52.3±1.2→56.4±1.152.3 \pm1.2 \to56.4 \pm1.1<br>(+4.0)∗(+4.0)^{*} [0.2]75.2±0.6→75.4±1.075.2 \pm0.6 \to75.4 \pm1.0<br>(+0.2)(+0.2) [0.2]71.7±0.2→71.5±0.071.7 \pm0.2 \to71.5 \pm0.0<br>(−0.2)(-0.2) [0.2]78.6±0.4→79.3±0.378.6 \pm0.4 \to79.3 \pm0.3<br>(+0.7)(+0.7) [0.2]
ScienceQA-IMG74.1±0.5→78.3±0.874.1 \pm0.5 \to78.3 \pm0.8<br>(+4.2)∗(+4.2)^{*} [0.5]66.1±1.1→69.0±1.566.1 \pm1.1 \to69.0 \pm1.5<br>(+2.9)∗(+2.9)^{*} [0.3]87.8±0.7→88.1±0.787.8 \pm0.7 \to88.1 \pm0.7<br>(+0.3)(+0.3) [0.2]80.8±0.4→81.3±0.080.8 \pm0.4 \to81.3 \pm0.0<br>(+0.4)(+0.4) [0.2]94.7±0.2→94.4±0.594.7 \pm0.2 \to94.4 \pm0.5<br>(−0.3)(-0.3) [0.3]
A-OKVQA68.8±2.0→72.9±1.468.8 \pm2.0 \to72.9 \pm1.4<br>(+4.1)∗(+4.1)^{*} [0.5]66.0±0.3→70.9±2.266.0 \pm0.3 \to70.9 \pm2.2<br>(+4.9)∗(+4.9)^{*} [0.5]84.4±0.5→84.8±0.484.4 \pm0.5 \to84.8 \pm0.4<br>(+0.4)(+0.4) [0.3]79.8±1.0→79.3±0.979.8 \pm1.0 \to79.3 \pm0.9<br>(−0.4)(-0.4) [0.3]85.2±0.6→85.0±1.185.2 \pm0.6 \to85.0 \pm1.1<br>(−0.2)(-0.2) [0.2]
MMVP60.3±3.0→60.8±1.760.3 \pm3.0 \to60.8 \pm1.7<br>(+0.6)(+0.6) [0.3]58.0±0.5→58.5±2.258.0 \pm0.5 \to58.5 \pm2.2<br>(+0.6)(+0.6) [0.3]73.8±1.3→74.6±2.073.8 \pm1.3 \to74.6 \pm2.0<br>(+0.8)(+0.8) [0.5]68.4±1.1→69.4±1.968.4 \pm1.1 \to69.4 \pm1.9<br>(+1.0)(+1.0) [0.2]76.9±1.3→74.4±3.776.9 \pm1.3 \to74.4 \pm3.7<br>(−2.6)(-2.6) [0.5]
RealWorldQA46.1±2.5→48.3±1.946.1 \pm2.5 \to48.3 \pm1.9<br>(+2.2)(+2.2) [0.3]47.2±1.1→49.7±0.847.2 \pm1.1 \to49.7 \pm0.8<br>(+2.4)(+2.4) [0.2]60.5±1.9→61.8±3.160.5 \pm1.9 \to61.8 \pm3.1<br>(+1.3)(+1.3) [0.2]53.4±1.0→55.5±1.853.4 \pm1.0 \to55.5 \pm1.8<br>(+2.1)(+2.1) [0.5]65.5±1.2→61.3±1.565.5 \pm1.2 \to61.3 \pm1.5<br>(−4.2)†(-4.2)^{\dagger} [0.7]
MME66.9±0.8→68.3±1.166.9 \pm0.8 \to68.3 \pm1.1<br>(+1.4)(+1.4) [0.3]74.4±0.3→76.4±0.774.4 \pm0.3 \to76.4 \pm0.7<br>(+2.0)∗(+2.0)^{*} [0.3]84.5±1.0→85.1±0.484.5 \pm1.0 \to85.1 \pm0.4<br>(+0.6)(+0.6) [0.5]79.4±0.1→79.9±0.779.4 \pm0.1 \to79.9 \pm0.7<br>(+0.6)(+0.6) [0.2]87.2±0.2→87.0±0.487.2 \pm0.2 \to87.0 \pm0.4<br>(−0.2)(-0.2) [0.2]
HallusionBench51.9±0.5→52.8±0.651.9 \pm0.5 \to52.8 \pm0.6<br>(+0.8)(+0.8) [0.5]56.7±1.4→60.2±0.556.7 \pm1.4 \to60.2 \pm0.5<br>(+3.5)∗(+3.5)^{*} [0.5]68.9±0.1→68.9±0.668.9 \pm0.1 \to68.9 \pm0.6<br>(0.0)(0.0) [0.2]60.4±1.0→62.5±0.360.4 \pm1.0 \to62.5 \pm0.3<br>(+2.1)∗(+2.1)^{*} [0.2]66.2±0.9→67.2±1.066.2 \pm0.9 \to67.2 \pm1.0<br>(+1.0)(+1.0) [0.3]
BLINK40.9±1.5→40.9±0.840.9 \pm1.5 \to40.9 \pm0.8<br>(+0.1)(+0.1) [0.7]45.4±1.9→47.0±2.145.4 \pm1.9 \to47.0 \pm2.1<br>(+1.6)(+1.6) [0.5]55.9±1.7→55.9±2.655.9 \pm1.7 \to55.9 \pm2.6<br>(−0.1)(-0.1) [0.5]48.3±0.5→47.3±1.648.3 \pm0.5 \to47.3 \pm1.6<br>(−1.0)(-1.0) [0.7]55.0±3.5→54.4±1.455.0 \pm3.5 \to54.4 \pm1.4<br>(−0.6)(-0.6) [0.2]
MMMU34.9±1.0→39.0±1.534.9 \pm1.0 \to39.0 \pm1.5<br>(+4.1)∗(+4.1)^{*} [0.3]40.1±1.2→43.1±0.540.1 \pm1.2 \to43.1 \pm0.5<br>(+3.0)∗(+3.0)^{*} [0.3]55.1±0.5→53.9±0.855.1 \pm0.5 \to53.9 \pm0.8<br>(−1.2)(-1.2) [0.5]47.6±1.1→46.3±0.947.6 \pm1.1 \to46.3 \pm0.9<br>(−1.3)(-1.3) [0.5]60.4±1.2→58.8±1.660.4 \pm1.2 \to58.8 \pm1.6<br>(−1.6)(-1.6) [0.5]

Table 20. Additional test-split strict accuracy (%): unedited → SHRINK, mean ± standard deviation. Each second line gives the change in points and validation-selected β\beta. The protocol and significance symbols match Table 1. MME and HallusionBench use per-question lettered Yes/No accuracy; MMVP uses per-question accuracy. BLINK and MMMU include only single-image multiple-choice items.

Greedy Decoding

Greedy evaluation uses one run, selecting strengths independently on each validation split. Batched bf16 inference is not bitwise deterministic, and repeated unedited greedy runs with different batch compositions differ by at most 1.0 point in per-benchmark accuracy. We therefore use greedy decoding as a complement to the three-seed sampled evaluation. The Core-5 averages improve significantly for LLaVA-OneVision and Qwen2.5-VL-3B, change within their intervals for Qwen2.5-VL-7B and Gemma-3-4B, and decrease for InternVL3-8B.

The reported greedy comparison is restricted to Core-5. The spatial results in the main evaluation use the sampled protocol.

Strength Selection and Sensitivity

The strength grid is {0.3,0.5,0.7}\{0.3, 0.5, 0.7\} for LLaVA-OneVision, {0.2,0.3,0.5}\{0.2, 0.3, 0.5\} for Qwen2.5-VL-3B/7B, and {0.2,0.3,0.5,0.7}\{0.2, 0.3, 0.5, 0.7\} for Gemma-3-4B and InternVL3-8B.

Each model-benchmark pair uses the validation strength with the largest mean edited-minus-unedited accuracy change, breaking ties toward smaller strengths. Sampled selection averages three seeds; greedy selection uses one run. The test questions play no part in strength selection. Selected strengths for the greedy and additional-benchmark evaluations appear in Table 18 and Table 20.

BenchmarkLLaVA-OV 7BQwen2.5-VL 3BQwen2.5-VL 7BGemma-3 4BInternVL3 8B
MMBench74.1→81.8 (+7.7)* [0.5]79.4→81.4 (+2.0)* [0.5]87.7→87.8 (+0.1) [0.2]82.8→81.0 (−1.8)† [0.7]89.3→88.2 (−1.0)† [0.7]
POPE69.5→72.9 (+3.4)* [0.3]85.7→86.4 (+0.7)* [0.2]86.8→87.6 (+0.8)* [0.2]84.2→84.0 (−0.2) [0.5]88.6→87.2 (−1.3)† [0.7]
CV-Bench62.7→61.9 (−0.8) [0.7]66.6→67.1 (+0.4) [0.3]74.4→75.8 (+1.4) [0.3]60.3→62.5 (+2.2)* [0.2]79.6→79.5 (−0.1) [0.3]
EmbSpatial62.9→64.4 (+1.5)* [0.5]53.9→55.9 (+2.0)* [0.2]69.0→68.6 (−0.3) [0.2]53.3→53.8 (+0.5) [0.2]71.3→70.1 (−1.2) [0.2]
MMStar43.5→50.2 (+6.6)* [0.5]47.5→49.8 (+2.2) [0.2]61.4→62.2 (+0.8) [0.2]51.5→50.6 (−0.9) [0.5]65.5→63.3 (−2.3)† [0.5]
Core-five avg.62.6→66.2 (+3.7)*66.6→68.1 (+1.5)*75.9→76.4 (+0.6)66.4→66.4 (0.0)78.8→77.7 (−1.2)†

Table 18. Greedy test-split strict accuracy (%): unedited → SHRINK, with change and independently validation-selected β\beta beneath each score. */† denote paired-bootstrap 95% CIs above/below zero. The comparison covers Core-5, with benchmarks weighted equally.

Table 19 reports the original-grid validation sweeps underlying strength selection. Stronger projection can reduce accuracy: on Gemma/WhatsUp, the validation change moves from +6.1 points at β=0.2\beta= 0.2 to −2.5 at β=0.7\beta= 0.7. These sweeps support selecting partial removal on validation. Appendix § 22 evaluates fixed strengths across the main nine benchmarks.

Additional Benchmarks

We evaluate M3CoT [9], AI2D [37], ScienceQA-IMG [52], A-OKVQA [65], MMVP [76], RealWorldQA [82], MME [16], HallusionBench [23], BLINK [17], and MMMU [89]. RealWorldQA uses its multiple-choice questions; BLINK and MMMU include only single-image multiple-choice items. Every cell follows the validation-to-test selection protocol of the main evaluation.

No M3CoT change is significant. Across the other nine benchmarks, LLaVA-OneVision and Qwen2.5-VL-3B improve in every cell, significantly on four and six benchmarks.

Qwen2.5-VL-7B has no significant change on these nine benchmarks. Gemma-3-4B gains 2.10 points significantly on HallusionBench, with no significant loss. InternVL3-8B loses 4.18 points significantly on RealWorldQA and has no significant gain. The small validation splits of MMVP and RealWorldQA make strength selection less precise; we retain their validation-selected strengths.

Sink-Only Comparison

This accuracy comparison covers the five main models on Core-5. The additional checkpoints (LLaVA-1.5-7B [46], Qwen3-VL-4B [2], InternVL3.5-8B [80], and Gemma-3-12B [18]) are included in the native-geometry diagnostics of Appendix § 10; no extended greedy accuracy comparison is reported here.

Sink-only comparison. The sink-only control applies the same projection to the sink key alone, retaining SHRINK’s eligible layers, native-attention head selector, and decoding onset. It can alter the sink’s competition with content positions but preserves non-sink attention ratios within each edited row. Both interventions use the original strength grids, with strengths selected independently by validation accuracy before testing on the same core-five questions.

Table 21 reports all five models and benchmarks at the selected strengths. Each intervention’s gain is measured against its own unedited baseline run; the contrast is the difference of gains. Pointwise 95% CIs use 10,000 paired test-question bootstrap resamples (seed 0) of per-question mean gain differences. Core-five intervals resample each benchmark separately and weight their means equally.

Benchmarkβ\beta: SHRINK/controlSHRINK Δ\DeltaControl Δ\DeltaDifference of gains [95% CI]
LLaVA-OneVision-7B
MMBench-EN0.7/0.5+6.73+4.04+2.69 [+1.36, +3.98]
POPE0.7/0.3+1.94+0.94+1.01 [+0.12, +1.88]
CV-Bench0.3/0.5+3.63+1.56+2.07 [+0.39, +3.73]
EmbSpatial0.5/0.7+1.95+3.15−1.20 [−2.64, +0.23]
MMStar0.5/0.7+5.24+2.26+2.97 [+0.68, +5.24]
Core-5 avg.–+3.90+2.39+1.51 [+0.81, +2.22]
Qwen2.5-VL-3B
MMBench-EN0.2/0.5+0.70−0.46+1.16 [+0.00, +2.28]
POPE0.3/0.5+1.01+1.43−0.42 [−1.06, +0.23]
CV-Bench0.3/0.5+2.80+1.04+1.75 [+0.11, +3.38]
EmbSpatial0.2/0.5+2.07+1.01+1.06 [−0.35, +2.47]
MMStar0.2/0.5+0.74+0.14+0.60 [−1.45, +2.73]
Core-5 avg.–+1.46+0.63+0.83 [+0.17, +1.50]
Qwen2.5-VL-7B
MMBench-EN0.5/0.2−0.20+0.02−0.22 [−0.90, +0.46]
POPE0.3/0.3+0.19+0.20−0.01 [−0.36, +0.33]
CV-Bench0.3/0.3+0.41+0.19+0.22 [−0.96, +1.42]
EmbSpatial0.2/0.5+0.21−0.56+0.77 [−0.37, +1.92]
MMStar0.3/0.3+1.34+1.55−0.22 [−1.96, +1.55]
Core-5 avg.–+0.39+0.28+0.11 [−0.39, +0.62]
Gemma-3-4B
MMBench-EN0.2/0.7−0.20−0.02−0.18 [−0.82, +0.46]
POPE0.5/0.2−0.09−0.15+0.06 [−0.30, +0.43]
CV-Bench0.3/0.7+1.45−0.73+2.18 [+0.71, +3.65]
EmbSpatial0.2/0.5+1.18−0.50+1.68 [+0.58, +2.78]
MMStar0.3/0.5+0.08+0.05+0.03 [−1.69, +1.75]
Core-5 avg.–+0.48−0.27+0.75 [+0.20, +1.29]
InternVL3-8B
MMBench-EN0.2/0.5−0.12−0.10−0.02 [−0.58, +0.56]
POPE0.7/0.7−0.66+0.00−0.66 [−1.06, −0.25]
CV-Bench0.2/0.3−0.84−0.27−0.57 [−1.77, +0.60]
EmbSpatial0.3/0.2−0.53+0.08−0.61 [−1.74, +0.53]
MMStar0.3/0.5+2.56+0.65+1.91 [+0.16, +3.66]
Core-5 avg.–+0.09+0.07+0.01 [−0.49, +0.52]

Table 21. Sink-only control under the main sampled protocol. Values are test-accuracy changes in percentage points against each run's own unedited baseline. Strengths are selected independently on validation. The final column is SHRINK’s gain minus the control's gain, with pointwise paired-question bootstrap 95% CIs. Core-5 means weight benchmarks equally.

The mean gain difference favors SHRINK on LLaVA-OneVision, Qwen2.5-VL-3B, and Gemma, with intervals above zero. Qwen2.5-VL-7B and InternVL3 intervals include zero. Individual tasks also differ: the control has a supported advantage on InternVL3/POPE. These results support a benefit of all-key editing beyond editing only the sink on these three models.

Sink-Direction Editing in Existing Methods

This study applies the directional control from Sec. 3 within GIFT [57] and ACG [32]. Their intervention rules differ from SHRINK’s all-visible-key projection and 20% head selector. The results in Table 2 therefore evaluate integrations of the finding, rather than the standalone ALL method.

Evaluation Protocol

Models and inputs. We use LLaVA-1.5-7B/13B [46] and BakLLaVA-7B [68], whose language backbone is Mistral-7B [28]. All three use the LLaVA-1.5 chat template with its system line and one user image/question turn, matching GIFT’s prompt. BakLLaVA uses this template rather than its hub template. Evaluation uses bf16, greedy decoding, at most 10 new tokens, and one question at a time, with one generation per arm and question. Intervals resample test questions, not generation seeds. Unlike the letter-decision scoring in Sec. 5.1, we use the LLaVA POPE yes/no rule: take the output’s first sentence, remove commas, and predict “no” if its space-separated words contain no, No, or not; otherwise predict “yes.” Accuracy is the fraction matching the gold yes/no label.

Benchmarks and splits. POPE-A-OKVQA and POPE-GQA [27, 44, 65] pool random, popular, and adversarial questions from the VCD release [39]. We split by image, assigning a group to validation when the hash of its dataset and image identifier modulo five is zero. The test sets contain 7,452 and 7,128 questions. The AMBER discriminative set [79] contributes all 14,216 questions over 1,004 images and is never used for selection. We also evaluate POPE-COCO (7,344 test questions) and MME (1,948 test questions, per-question yes/no accuracy). Their results are retained in Table 25. The same questions are used for every method and model.

ModelBenchmarkBaseOriginalKey editΔ\DeltaOriginalKey branchΔ\Delta
LLaVA-1.5-7BPOPE-COCO84.8785.7085.99+0.29+0.2986.1985.74−0.45-0.45
LLaVA-1.5-7BMME77.8777.0576.59−0.46-0.4676.3973.36−3.03-3.03†
LLaVA-1.5-13BPOPE-COCO84.1284.3385.28+0.95+0.95*84.4287.12+2.70+2.70*
LLaVA-1.5-13BMME80.2479.2675.67−3.59-3.59†80.3479.67−0.67-0.67
BakLLaVA-7BPOPE-COCO83.4285.3985.69+0.30+0.3083.8585.70+1.85+1.85*
BakLLaVA-7BMME77.8278.0877.52−0.56-0.5676.0877.93+1.85+1.85*

Table 25. Additional benchmarks for the comparison in Table 2. Values are accuracy (%); Δ\Delta compares each key-edited variant with the corresponding original method. * /† indicate an image-bootstrap 95% interval entirely above/below zero. These retain the exploratory settings in Table 22; MME uses per-question accuracy.

Selection and uncertainty. Settings maximize mean validation accuracy over POPE-COCO, POPE-A-OKVQA, and POPE-GQA, with ties favoring smaller edits. AMBER and all test questions are excluded. We first reproduce the released GIFT configuration and independently select its key-edit strength. For the matched comparison, GIFT’s saliency exponent is also tuned on the same validation pools. ACG’s guidance strength is selected on validation; the matched key variant retains that strength while tuning its branch edit. The exploratory image-bootstrap comparisons use 10,000 resamples, seed 20261102; matched-comparison CIs resample paired questions within each benchmark, with 10,000 resamples and seed 0. Means are computed before rounding. All intervals are pointwise and conditional on the selected settings.

Directional Operation and Integrations

We identify the sink once per model on 24 POPE-COCO validation questions, taking the pre-image template position with the largest mean last-row attention over the final third of layers. All three models select position zero, ⟨s⟩\langle s\rangle. For the relevant layer and head, its native post-RoPE key defines u=ks/∥ks∥2\mathbf{u}=\mathbf{k}_s/\lVert\mathbf{k}_s\rVert_2. With a token-specific strength xjx_j, the local operation is

kj′=kj+xj(kj⊤u)u.(28)\mathbf{k}'_j=\mathbf{k}_j+x_j(\mathbf{k}_j^\top\mathbf{u})\mathbf{u}. \tag*{(28)}

It rescales the existing parallel component by 1+xj1+x_j, without directly editing the query, value, or orthogonal key component. Negative strengths between −1-1 and zero shrink the component; xj=−1x_j=-1 removes it, and xj<−1x_j<-1 reverses its sign. The latter settings exceed SHRINK’s bounded shrinkage range. Attention changes depend on query-key alignment, as in § 3.

GIFT. We retain the released saliency calculation, enhancement layers, last-query-row scope, and image/question head and token selections. The saliency map averages positive visual-attention shifts over content words and the top half of heads, clips at three standard deviations, and normalizes to hj∈[0,hmax⁡]h_j\in[0,h_{\max}] with hmax⁡=5h_{\max}=5. Its original image reweighting uses Sj=exp⁡(hj)S_j=\exp(h_j). Our replacement uses Eq. (28) with xj=λhj/hmax⁡x_j=\lambda h_j/h_{\max} on image keys in the selected image heads. On question keys in the selected question heads, it uses xj=λlog⁡(qs)/hmax⁡x_j=\lambda\log(q_s)/h_{\max}, where qsq_s is GIFT’s ratio of enhanced to native selected image mass. We preserve these weighting rules, replacing the attention operation rather than adding an extra saliency selector. GIFT uses saliency layer 11 and enhancement layers 12–22 for the 7B-depth models; for 13B these are layer 10 and layers 14–20. BakLLaVA reuses the 7B settings. Token positions for its question are located directly to account for tokenizer differences.

ACG. We retain its conditional attention output, last-query-row scope in every layer/head, and orthogonalized contrastive guidance. Original ACG defines its contrast branch by masking all image logits. Our contrast branch instead applies Eq. (28) to every image key with a shared strength xx. Both branches use the original values. Guidance uses the difference between the conditional and modified outputs, removes its component along the modified output, and adds the residual to the conditional output with scale γ\gamma, as in ACG. The exploratory xx values are negative; the matched 13B configuration instead selects positive xx. A key-edited contrast branch is therefore not generally a softer approximation to image masking.

The GIFT edit grid for LLaVA-1.5 includes both signs of {0.1,0.25,0.5,0.75,1,1.25,1.5,2,4,8}\{0.1, 0.25, 0.5, 0.75, 1, 1.25, 1.5, 2, 4, 8\}. BakLLaVA uses {0.5,1,−0.25,−0.5,−0.75,−1,−1.25,−1.5,−2}\{0.5, 1, -0.25, -0.5, -0.75, -1, -1.25, -1.5, -2\}. ACG’s original guidance grid is {1,1.5,2,2.4,3,4,5,6,8}\{1, 1.5, 2, 2.4, 3, 4, 5, 6, 8\} for LLaVA-1.5 and {1,1.5,2,2.4,3,4,5}\{1, 1.5, 2, 2.4, 3, 4, 5\} for BakLLaVA. Key-edited ACG explores positive and negative strengths, including {0.5,1,2,4,8,16,−0.5,−0.75,−1,−1.5,−2,−3,−4}\{0.5, 1, 2, 4, 8, 16, -0.5, -0.75, -1, -1.5, -2, -3, -4\} for LLaVA-1.5, alongside auxiliary strength/guidance pairs. Tables 22 and 23 specify the reported configurations. The exploratory 13B/BakLLaVA ACG comparisons include a guidance-scale difference; matched comparisons use the same guidance on both sides.

Matched tuning. GIFT’s saliency exponent is selected from 21 validation candidates spanning 0.5–25, with selected values 10, 25, and 9 for 7B, 13B, and BakLLaVA. The 13B pick lies at the grid boundary, with a flat validation response over 15–25. The key-edited GIFT configurations are unchanged. For matched ACG, both sides use γ=2.4\gamma=2.4 on 7B and 4 on 13B/BakLLaVA. The latter key branches select x=2x=2 and x=−0.25x=-0.25 from 15 candidates spanning −4-4–16; 7B retains x=−0.75x=-0.75. All test arms for each matched comparison share a run; repeated released-GIFT, GIFT-Key, and ACG-4 accuracies agree with the exploratory runs.

ModelGIFT exponentKey λ\lambdaACG γ\gammaKey xx
LLaVA-1.5-7B10−1-12.4−0.75-0.75
LLaVA-1.5-13B25−1.25-1.2542
BakLLaVA-7B9−2-24−0.25-0.25

Table 23. Settings for the matched host comparisons. The same ACG guidance scale is used on both sides.

Reproduction details. Our GIFT implementation follows the released code at commit e20b8a9; ACG follows dee49c5, whose masking and orthogonalized-guidance construction is also described in the paper’s third version. All methods use bf16 here, whereas the original LLaVA implementations use fp16. ACG’s branch outputs and guidance are computed in fp32 before casting back. GIFT’s prompt saliency pass is retained. Because precision, input files, and splits differ, reproduced accuracies differ from those reported in the original papers.

Results and Selection Scope

The models and benchmarks in Table 2 were chosen based on the exploratory reproductions, and matched tuning was then applied to them. Table 26 gives the six paired mean contrasts. Three CIs lie above zero, two include zero, and one lies below zero. At matched guidance, the 13B ACG gain is 0.33 points, compared with 4.05 points in the exploratory comparison at different guidance scales.

ModelHostTuned hostKeyChange95% CI
LLaVA-1.5-7BGIFT81.3280.09−1.22-1.22[−1.49-1.49, −0.95-0.95]
LLaVA-1.5-13BGIFT78.3079.05+0.75+0.75[+0.45+0.45, +1.05+1.05]
BakLLaVA-7BGIFT79.8879.98+0.10+0.10[−0.16-0.16, +0.36+0.36]
LLaVA-1.5-7BACG79.0079.93+0.93+0.93[+0.61+0.61, +1.24+1.24]
LLaVA-1.5-13BACG78.8779.20+0.33+0.33[−0.03-0.03, +0.67+0.67]
BakLLaVA-7BACG78.0378.87+0.83+0.83[+0.50+0.50, +1.17+1.17]

Table 26. Matched host comparisons on the selected three test benchmarks: accuracy (%) and Key-minus-host change (points). GIFT is validation-tuned; ACG uses the same γ\gamma on both sides. CIs are pointwise 95% intervals.

The exploratory released-host comparisons appear in Table 24 and Table 25 and include 17 positive key-versus-host intervals. On the three-benchmark mean, GIFT-Key exceeds released GIFT by 0.86, 3.07, and 1.37 points; tuning GIFT changes those contrasts to −1.22, +0.75, and +0.10.

ModelMethodBenchmarkMethod − baseKey edit − baseKey edit − method
LLaVA-1.5 7BGIFTPOPE-A-OKVQA+2.48+2.48 [+1.89+1.89, +3.07+3.07]+3.41+3.41 [+2.74+2.74, +4.08+4.08]+0.93+0.93 [+0.50+0.50, +1.37+1.37]
LLaVA-1.5 7BGIFTPOPE-GQA+3.73+3.73 [+3.06+3.06, +4.43+4.43]+5.19+5.19 [+4.43+4.43, +5.96+5.96]+1.46+1.46 [+0.95+0.95, +1.96+1.96]
LLaVA-1.5 7BGIFTAMBER+4.65+4.65 [+4.17+4.17, +5.13+5.13]+4.83+4.83 [+4.28+4.28, +5.38+5.38]+0.18+0.18 [−0.19-0.19, +0.56+0.56]
LLaVA-1.5 7BACGPOPE-A-OKVQA+2.38+2.38 [+1.74+1.74, +2.99+2.99]+3.15+3.15 [+2.44+2.44, +3.86+3.86]+0.78+0.78 [+0.15+0.15, +1.41+1.41]
LLaVA-1.5 7BACGPOPE-GQA+3.65+3.65 [+2.95+2.95, +4.35+4.35]+4.69+4.69 [+3.89+3.89, +5.50+5.50]+1.04+1.04 [+0.35+0.35, +1.74+1.74]
LLaVA-1.5 7BACGAMBER+4.14+4.14 [+3.72+3.72, +4.56+4.56]+5.11+5.11 [+4.51+4.51, +5.72+5.72]+0.98+0.98 [+0.46+0.46, +1.52+1.52]
LLaVA-1.5 13BGIFTPOPE-A-OKVQA+0.47+0.47 [+0.09+0.09, +0.86+0.86]+2.60+2.60 [+1.95+1.95, +3.29+3.29]+2.13+2.13 [+1.50+1.50, +2.76+2.76]
LLaVA-1.5 13BGIFTPOPE-GQA+1.16+1.16 [+0.74+0.74, +1.60+1.60]+4.52+4.52 [+3.82+3.82, +5.23+5.23]+3.35+3.35 [+2.71+2.71, +3.98+3.98]
LLaVA-1.5 13BGIFTAMBER+2.75+2.75 [+2.39+2.39, +3.11+3.11]+6.47+6.47 [+5.92+5.92, +7.02+7.02]+3.72+3.72 [+3.26+3.26, +4.18+4.18]
LLaVA-1.5 13BACGPOPE-A-OKVQA+3.52+3.52 [+2.59+2.59, +4.46+4.46]+5.93+5.93 [+5.01+5.01, +6.87+6.87]+2.42+2.42 [+1.72+1.72, +3.13+3.13]
LLaVA-1.5 13BACGPOPE-GQA+5.12+5.12 [+4.25+4.25, +6.00+6.00]+8.89+8.89 [+7.91+7.91, +9.85+9.85]+3.77+3.77 [+2.97+2.97, +4.57+4.57]
LLaVA-1.5 13BACGAMBER+4.43+4.43 [+3.90+3.90, +4.96+4.96]+10.40+10.40 [+9.78+9.78, +11.04+11.04]+5.97+5.97 [+5.44+5.44, +6.50+6.50]
BakLLaVA 7BGIFTPOPE-A-OKVQA+4.64+4.64 [+3.85+3.85, +5.43+5.43]+5.65+5.65 [+4.72+4.72, +6.59+6.59]+1.01+1.01 [+0.39+0.39, +1.64+1.64]
BakLLaVA 7BGIFTPOPE-GQA+4.66+4.66 [+3.79+3.79, +5.51+5.51]+5.51+5.51 [+4.49+4.49, +6.51+6.51]+0.86+0.86 [+0.13+0.13, +1.57+1.57]
BakLLaVA 7BGIFTAMBER+6.71+6.71 [+6.24+6.24, +7.18+7.18]+8.95+8.95 [+8.36+8.36, +9.52+9.52]+2.24+2.24 [+1.82+1.82, +2.65+2.65]
BakLLaVA 7BACGPOPE-A-OKVQA+3.23+3.23 [+2.28+2.28, +4.21+4.21]+5.11+5.11 [+4.19+4.19, +6.05+6.05]+1.88+1.88 [+1.01+1.01, +2.78+2.78]
BakLLaVA 7BACGPOPE-GQA+3.98+3.98 [+2.89+2.89, +5.05+5.05]+5.82+5.82 [+4.81+4.81, +6.80+6.80]+1.84+1.84 [+0.91+0.91, +2.72+2.72]
BakLLaVA 7BACGAMBER+7.04+7.04 [+6.34+6.34, +7.76+7.76]+8.12+8.12 [+7.52+7.52, +8.73+8.73]+1.08+1.08 [+0.41+0.41, +1.72+1.72]

Table 24. Accuracy differences (percentage points) for the exploratory configurations in Table 22. Brackets show paired-image bootstrap 95% intervals (10,000 resamples). The three columns compare each original method with unedited decoding, its key-edited variant with unedited decoding, and the variant with the original method.

Results on POPE-COCO and MME are mixed. With matched γ=4\gamma= 4, 13B ACG-Key raises POPE-COCO accuracy from 84.42% to 85.62% and lowers MME from 80.34% to 72.79%, giving a five-set mean contrast of −1.07 points (95% CI [−1.52, −0.64]). BakLLaVA’s matched five-set ACG contrast is +1.26 points ([+0.90, +1.62]). For GIFT, the matched five-set contrasts are −0.97, +0.14, and −0.05 points for 7B, 13B, and BakLLaVA.

The validation-selected 13B branch strength depends on the guidance scale, changing from x=−1x = -1 at γ=2.4\gamma= 2.4 to x=+2x = +2 at γ=4\gamma= 4.

We also ran both integrations on LLaVA-NeXT-7B/13B, with single-view inputs, and on Qwen2-VL-7B. The released methods themselves give little or no gain on LLaVA-NeXT-7B and Qwen2-VL-7B, and the key variants do not consistently improve on them across these models.

Exploratory integrations with several other attention-intervention methods did not consistently improve on those methods. Our claims concern the GIFT and ACG integrations and settings reported in Table 2.

Absolute Attention Allocation

Figure 8 uses the exploratory host settings in Table 22, on POPE-A-OKVQA (7,452 questions), POPE-GQA (7,128), and AMBER (14,216). These configurations differ from the matched-tuning settings in Table 2. They characterize attention allocation under the exploratory settings. Table 27 reports the full absolute allocation for Base, Original, and Key edit, both per benchmark and as an equally weighted benchmark mean. The plotted changes are Key edit minus Original, computed from unrounded shares before rounding the labels.

ModelGIFT λ\lambdaACG γ\gammaKey branch xxγ\gamma
LLaVA-1.5-7B−1-12.42.4−0.75-0.752.42.4
LLaVA-1.5-13B−1.25-1.2544−1-12.42.4
BakLLaVA-7B−2-244−0.75-0.752.42.4

Table 22. Exploratory settings for the released-host comparisons and attention traces, distinct from the matched settings below.

BenchmarkConditionGIFT scopeACG scope
SinkImageQ./opt.OtherSinkImageQ./opt.Other
LLaVA-1.5-7B
POPE-A-OKVQABase35.786.5212.3845.3337.228.837.9745.99
Original30.1414.2021.4834.1839.626.544.6449.20
Key edit27.2920.4422.7029.5833.6213.366.3046.72
POPE-GQABase35.766.4812.4445.3237.218.788.0146.01
Original30.1014.2421.5434.1139.616.524.5949.28
Key edit27.1820.6322.7329.4633.6913.206.3246.79
AMBERBase35.566.0013.3645.0936.998.588.5545.88
Original28.5515.4223.6532.3839.376.515.7048.42
Key edit26.7418.3326.0428.8933.9312.996.2346.85
MeanBase35.706.3312.7245.2537.148.738.1845.96
Original29.6014.6222.2233.5639.536.534.9748.97
Key edit27.0719.8023.8329.3133.7513.186.2846.79
LLaVA-1.5-13B
POPE-A-OKVQABase69.396.3815.079.1672.617.326.5213.55
Original54.4415.8826.023.6775.115.715.4513.73
Key edit35.2537.9525.970.8271.499.254.6114.66
POPE-GQABase69.376.3315.199.1172.617.296.5613.54
Original54.4315.7826.113.6875.145.725.4313.71
Key edit35.2338.1425.800.8371.509.204.6614.64
AMBERBase67.966.1815.939.9372.387.126.8413.66
Original53.4315.3627.243.9874.925.505.8113.77
Key edit34.9837.5226.560.9471.398.954.8114.85
MeanBase68.916.3015.409.4072.537.246.6413.59
Original54.1015.6726.453.7775.055.645.5613.74
Key edit35.1537.8726.110.8671.469.134.6914.72
BakLLaVA-7B
POPE-A-OKVQABase35.7111.6224.2128.4638.6412.3615.6533.36
Original24.3425.1135.1315.4246.347.9611.0934.60
Key edit24.8134.5726.7413.8833.2420.1812.1234.46
POPE-GQABase35.6511.5424.3928.4338.6412.2215.7533.39
Original24.3024.9535.3815.3646.347.8511.1434.67
Key edit24.7534.3627.0313.8633.2620.0312.1734.55
AMBERBase35.789.7025.1129.4038.9210.7016.5233.87
Original24.2523.8136.9514.9945.807.4511.5035.25
Key edit25.3731.5729.2013.8633.5918.3512.4235.64
MeanBase35.7110.9624.5728.7638.7311.7615.9733.54
Original24.3024.6235.8215.2646.167.7511.2534.84
Key edit24.9833.5027.6613.8633.3619.5212.2434.88

Table 27. Absolute attention shares (%) underlying Figure 8. Q./opt. denotes question and option tokens. Mean weights the three benchmarks equally; GIFT and ACG retain their own measurement scopes. Rounding can affect displayed totals.

Figure 8. Exploratory host attention changes. Bars show attention-share changes (pp), key edit minus original at answer onset. Within each group: sink, image, question/options, other template; generated share is zero. Colors follow Figure 3. Traces use released GIFT settings and the exploratory ACG guidance, not the matched settings of Table 2.

Measurement position and scope. We measure the attention row that predicts the first answer token, averaging all query heads and the specified layers, then questions within each benchmark. GIFT measurements cover its enhancement layers: indices 12–22 for LLaVA-1.5-7B and BakLLaVA-7B, and 14–20 for LLaVA-1.5-13B (zero-based, inclusive). ACG measurements cover all 32 layers for the 7B models and all 40 for the 13B model. For ACG, the measured distribution is the normalized attention in the conditional main branch before contrastive guidance, not a signed

Separate Base averages. The same unedited forward pass provides Base for both methods, but the layer averages differ: GIFT uses its enhancement layers and ACG uses all layers. Each Original and Key edit condition is compared with Base under its own method’s scope. Combining these Base averages would change the comparison. The replacement conditions retain the host methods’ selection and guidance structures as described in Appendix § 20. Original and key edit ACG both use γ=2.4\gamma= 2.4 for LLaVA-1.5-7B; for LLaVA-1.5-13B and BakLLaVA, Original uses γ=4\gamma= 4 and Key edit uses γ=2.4\gamma= 2.4. Their differences therefore include the guidance-strength change as well as branch replacement.

Token groups and conservation. The sink is the first template token. Image and question/option positions are disjoint from it; the question group combines content words, function words, and any options. Remaining template and generated positions complete the visible keys, with no generated positions at answer onset. The host figure omits generated positions, which have zero mass at answer onset; the tables below retain their absolute shares with template/generated combined as Other. The partition sums to 100% before rounding and describes attention allocation, not a causal accuracy mechanism. Standalone SHRINK measurements. Table 28 gives the corresponding absolute shares for Figure 3, on the four improved models and nine benchmarks in Table 1. The two passes follow the same unedited sampled continuation, capped at 256 answer tokens. Within each question, attention is averaged over the selected layer/head/query cells across traced answer steps in eligible layers; question means are then averaged within each benchmark, and benchmark means are weighted equally. Head selection is recomputed independently in each pass, so the selected populations can differ. These are full attention distributions within each selected population. The disjoint group partition is the same as for the host methods, but Other now includes generated tokens as well as remaining template tokens. We verified the raw group sums and checked Other against both the remaining groups and the complement of sink, image, and question/option attention. No renormalization of the sink, image, and question/option groups is applied.

BenchmarkBaseSHRINK
SinkImageQ./opt.OtherSinkImageQ./opt.Other
LLaVA-OneVision-7B
MMBench-EN20.1123.0224.8731.994.0633.5624.7137.67
POPE20.8825.6821.6331.803.9534.4420.9240.69
CV-Bench19.2618.6130.4231.708.8822.3132.0636.76
EmbSpatial19.0216.9232.1331.935.7222.6233.5638.10
MMStar19.7222.8625.4631.965.6030.7726.4937.14
VSR19.9622.1026.7531.194.2531.7925.2938.68
WhatsUp18.6925.8222.8332.668.1931.3923.6536.77
COCO-QA19.7128.6519.5032.138.3034.2320.6236.85
VG-QA19.1526.2521.9632.643.5135.6621.1039.73
Mean19.6123.3225.0632.005.8330.7525.3838.04
Qwen2.5-VL-3B
MMBench-EN17.3319.3127.2036.168.9821.6829.3739.97
POPE19.3129.4916.1235.087.8833.7417.1841.20
CV-Bench16.6919.2128.5835.526.7721.7430.4041.09
EmbSpatial16.2914.6132.7836.328.5216.0735.0840.33
MMStar17.4620.5226.4235.609.1523.2028.2939.36
VSR17.7220.3926.7735.124.8224.5127.0043.66
WhatsUp16.2923.3824.4735.876.3427.0825.4341.15
COCO-QA17.1327.0319.4136.436.7930.3720.7542.09
VG-QA16.4925.5321.0036.988.7127.5722.6541.07
Mean17.1922.1624.7535.907.5525.1126.2441.10
Qwen2.5-VL-7B
MMBench-EN22.7520.5925.7530.9114.5624.9326.4934.02
POPE24.6332.1214.2229.0218.2335.6014.8331.34
CV-Bench21.4419.8927.6231.0515.8322.0228.4033.75
EmbSpatial21.0115.3631.2632.3716.6016.6032.3534.45
MMStar21.8519.9326.6431.5715.7722.7227.3134.20
VSR23.2423.3824.4228.9717.3526.0425.0731.53
WhatsUp21.6124.2922.5931.5017.2026.5422.8533.41
COCO-QA21.9428.8717.5131.6916.0231.8018.1034.08
VG-QA21.3926.6119.4232.5815.6329.2920.1734.90
Mean22.2123.4523.2731.0716.3626.1723.9533.52
Gemma-3-4B
MMBench-EN20.5123.7822.6733.059.5128.5024.3637.64
POPE22.7836.1311.6729.422.4549.8210.6137.13
CV-Bench19.7824.2424.2731.715.8830.4925.3238.31
EmbSpatial18.7720.6527.6232.968.4024.5629.3937.65
MMStar20.5325.6021.2832.586.1032.3222.5739.01
VSR20.6628.1621.9529.230.7242.6017.3639.31
WhatsUp19.9328.3720.4331.269.3334.0221.2635.40
COCO-QA20.8231.6516.0331.519.6538.0816.6035.68
VG-QA20.5730.4617.2431.739.5136.6518.0335.81
Mean20.4827.6720.3531.506.8435.2320.6137.33

Table 28. Absolute attention shares (%) for standalone SHRINK in Figure 3. The scope is traced answer steps in eligible layers and independently selected heads in each pass. Q./opt. denotes question and option tokens; Other includes remaining template and generated tokens. Mean weights the nine benchmarks equally. Displayed totals can differ from 100% through rounding.

Direction Controls and Strength Transfer

Reference Directions

The scalar projection rule, layer schedule, native head selector, and decoding protocol are fixed; only the removed direction changes. Each direction is a unit vector per layer and KV head in post-positional key space. From each of the nine validation benchmarks, 10 prompts estimate the directions and 10 disjoint prompts evaluate their geometry. No test labels enter direction construction or strength selection.

The mean-query reference averages unit queries from prompt rows after the sink across the associated query heads. The mean-key reference averages all non-sink prompt keys. The top key PC is the first axis of their centered covariance. A template reference uses the first fixed template token after the sink, before task content. Three seeded random unit vectors are drawn independently, without orthogonalization. The native sink direction is unchanged. Each alternative uses the same model-level strength selected for SHRINK: 0.7, 0.3, 0.3, 0.2, 0.3 for LLaVA-OneVision, Qwen-3B, Qwen-7B, Gemma, and InternVL3.

The accuracy comparison uses all nine test benchmarks, with every arm and its baseline evaluated in a single shared run at SHRINK’s model-level strengths, rather than the per-benchmark selection in Table 1. Table 29 and Table 30 report all five models and paired comparisons. CIs resample test questions independently within benchmarks, weighting their means equally (10,000 draws, seed 0).

DirectionLLaVA-OV 7BQwen2.5-VL 3BQwen2.5-VL 7BGemma-3 4BInternVL3 8B
Sink (SHRINK)+4.57{sup}*` [+3.58,+5.54]+3.55{sup}*` [+2.69,+4.42]+0.16 [-0.44,+0.76]+0.33 [-0.31,+0.97]−0.12 [−0.73,+0.48]
Mean query+2.49{sup}*` [+1.50,+3.45]+2.33{sup}*` [+1.46,+3.20]+0.24 [−0.37,+0.88]+0.39 [−0.24,+1.04]−0.98{sup}†` [−1.59,−0.37]
Mean key−0.39 [−1.35,+0.56]+2.26{sup}*` [+1.43,+3.11]+0.47 [−0.13,+1.09]+0.43 [−0.15,+1.01]−0.50 [−1.09,+0.10]
Top key PC−11.08{sup}†` [−12.05,−10.11]−5.03{sup}†` [−5.89,−4.14]−0.20 [−0.78,+0.40]+0.01 [−0.54,+0.57]+0.39 [−0.15,+0.96]
Template token+0.14 [−0.79,+1.07]+0.15 [−0.63,+0.94]+0.61{sup}*` [+0.06,+1.18]+0.19 [−0.44,+0.81]−0.09 [−0.64,+0.48]
Random 1+0.40 [−0.30,+1.10]+0.41 [−0.28,+1.10]−0.17 [−0.65,+0.30]−0.09 [−0.51,+0.33]−0.40 [−0.87,+0.06]
Random 2+0.18 [−0.52,+0.90]−0.11 [−0.77,+0.56]+0.07 [−0.41,+0.55]−0.00 [−0.42,+0.41]−0.03 [−0.51,+0.45]
Random 3−0.24 [−0.95,+0.48]−0.76{sup}†` [−1.45,−0.07]−0.17 [−0.65,+0.31]−0.05 [−0.48,+0.37]−0.23 [−0.70,+0.25]

Table 29. Accuracy changes from unedited decoding (percentage points), equal-weight nine-benchmark means. Brackets: pointwise paired-question 95% CIs; */†: intervals above/below zero. All directions use SHRINK-selected model strengths.

DirectionLLaVA-OV 7BQwen2.5-VL 3BQwen2.5-VL 7BGemma-3 4BInternVL3 8B
Mean query−2.08{sup}†` [−3.02,−1.13]−1.22{sup}†` [−2.01,−0.42]+0.08 [−0.53,+0.69]+0.06 [−0.52,+0.64]−0.86{sup}†` [−1.44,−0.27]
Mean key−4.97{sup}†` [−5.93,−4.02]−1.29{sup}†` [−2.11,−0.45]+0.31 [−0.28,+0.93]+0.10 [−0.51,+0.70]−0.38 [−0.98,+0.22]
Top key PC−15.66{sup}†` [−16.64,−14.68]−8.58{sup}†` [−9.47,−7.67]−0.36 [−0.99,+0.28]−0.33 [−0.98,+0.32]+0.51 [−0.10,+1.13]
Template token−4.43{sup}†` [−5.40,−3.47]−3.40{sup}†` [−4.26,−2.53]+0.45 [−0.15,+1.06]−0.14 [−0.70,+0.39]+0.03 [−0.57,+0.64]
Random 1−4.17{sup}†` [−5.15,−3.17]−3.14{sup}†` [−4.00,−2.29]−0.33 [−0.93,+0.25]−0.42 [−1.06,+0.23]−0.28 [−0.90,+0.33]
Random 2−4.40{sup}†` [−5.37,−3.43]−3.66{sup}†` [−4.52,−2.80]−0.09 [−0.69,+0.50]−0.34 [−0.98,+0.31]+0.09 [−0.53,+0.69]
Random 3−4.81{sup}†` [−5.79,−3.86]−4.31{sup}†` [−5.17,−3.44]−0.33 [−0.93,+0.26]−0.39 [−1.03,+0.27]−0.10 [−0.72,+0.50]

Table 30. Paired accuracy contrasts against SHRINK (percentage points), equal-weight nine-benchmark means. Brackets: pointwise paired-question 95% CIs; */†: intervals above/below zero. All directions use SHRINK-selected model strengths.

On LLaVA-OneVision and Qwen-3B, every alternative has a negative paired contrast against the sink direction, with its pointwise interval below zero. The mean-query direction has greater mean absolute query alignment (0.76 and 0.75, versus 0.47 and 0.43 for the sink), yet produces smaller accuracy gains. The mean-key direction is also distinct geometrically: absolute cosine with the sink ranges from 0.33 to 0.58 across models. These results support the sink reference at the tested strengths. On the other three models, where SHRINK’s gains are smaller, most direction contrasts have intervals that include zero; InternVL3’s mean-query contrast lies below zero.

Strength Transfer

We reuse the main sampled evaluation, with no additional generation. Benchmark-specific selection maximizes each validation mean; per-model selection maximizes the equal-weight mean over nine validation benchmarks. Global selection maximizes the mean over models using the shared grid {0.3, 0.5} and chooses 0.3. Ties favor smaller strengths throughout. Test questions are excluded from every choice. Table 31 retains all three selection rules and their paired-question CIs. Per-model selection changes the nine-benchmark gain by at most 0.33 points from benchmark-specific selection. Global selection preserves positive intervals for the four main models; LLaVA-OneVision gains a further 1.26 points with its per-model setting.

ModelPer benchmarkPer modelGlobal
LLaVA-OV-7B+5.10{sup}*` [+4.55, +5.64]+5.14{sup}*` [+4.56, +5.71]+3.88{sup}*` [+3.35, +4.41]
Qwen2.5-VL-3B+2.72{sup}*` [+2.21, +3.23]+3.05{sup}*` [+2.54, +3.57]+3.05{sup}*` [+2.54, +3.57]
Qwen2.5-VL-7B+0.53{sup}*` [+0.19, +0.87]+0.48{sup}*` [+0.12, +0.83]+0.48{sup}*` [+0.12, +0.83]
Gemma-3-4B+0.84{sup}*` [+0.42, +1.27]+0.72{sup}*` [+0.31, +1.12]+0.71{sup}*` [+0.28, +1.14]
InternVL3-8B−0.03 [−0.38, +0.33]−0.07 [−0.43, +0.28]−0.07 [−0.43, +0.28]

Table 31. Validation-selected strength transfer: nine-benchmark mean accuracy changes (points). Brackets are pointwise paired-question 95% CIs. The per-model strengths are 0.7, 0.3, 0.3, 0.2, 0.3 in row order; the global strength is 0.3. *: CI above zero.

Benchmarkβ=0.2\beta= 0.2β=0.3\beta= 0.3β=0.5\beta= 0.5β=0.7\beta= 0.7
LLaVA-OneVision-7B
MMBench-EN–+7.5+6.2+8.2
POPE–+1.6+1.2+2.1
CV-Bench–+1.1+0.4+1.1
EmbSpatial–+2.6+3.2+2.3
MMStar–+2.2+4.7+0.8
VSR–+4.0+8.4+11.3
WhatsUp–+3.9+3.1+1.6
COCO-QA–+4.7+3.2+4.3
VG-QA–+3.7+5.1+7.1
Qwen2.5-VL-3B
MMBench-EN+2.9+2.4+1.0–
POPE+0.9+1.1-1.1–
CV-Bench+4.8+5.4+5.1–
EmbSpatial+3.1+1.2+2.0–
MMStar+0.7+0.7-1.1–
VSR+6.9+6.4+7.9–
WhatsUp+0.4+4.5+2.9–
COCO-QA+2.2+3.8+2.9–
VG-QA+5.1+2.4+4.7–
Qwen2.5-VL-7B
MMBench-EN-0.9-0.8-0.5–
POPE+0.1+0.9+0.9–
CV-Bench+0.1+0.4-0.5–
EmbSpatial+1.5+0.6+0.1–
MMStar-0.5+0.6+0.1–
VSR-0.4+0.5+0.3–
WhatsUp+1.4-0.8-0.8–
COCO-QA-0.2+0.3+0.1–
VG-QA-1.1+0.5+0.5–
Gemma-3-4B
MMBench-EN+0.3-0.6-0.2-1.1
POPE+0.2+0.7+0.7-0.1
CV-Bench+1.3+1.9+1.6-0.3
EmbSpatial+1.2+1.0-0.4-1.3
MMStar-0.6+0.6+0.4-1.2
VSR+1.0+0.9+1.0+1.2
WhatsUp+6.1+3.9-0.4-2.5
COCO-QA+1.1-0.2-0.6-4.3
VG-QA-1.0-3.0-4.0-6.0
InternVL3-8B
MMBench-EN+0.2-0.5-0.2-1.0
POPE+0.5+0.2+0.8+0.8
CV-Bench-0.9-1.5-1.0-2.9
EmbSpatial-0.3+0.5-2.1-1.6
MMStar-1.0-0.2-3.4-2.0
VSR-0.2+0.8+0.0-0.7
WhatsUp-1.8-0.4-3.1-4.7
COCO-QA-0.1-0.4-0.8-2.2
VG-QA-3.1-3.4-1.8-3.4

Table 19. Original-grid validation sensitivity: mean accuracy change (percentage points) from unedited decoding. Bold: validation-selected strength. Dashes indicate strengths outside the model's original grid.

Multiplicity-Adjusted Accuracy Comparisons

We distinguish two test families defined before inspecting their adjusted pp-values: the five model-level nine-benchmark means, and the 45 model-benchmark cells. Each uses the original validation-selected strengths and three-seed per-question mean changes. Two-sided bootstrap pp-values are twice the smaller tail probability at zero, floored at 1/10,0001/10{,}000. Model means resample questions separately within each benchmark before equal weighting. These corrections address the stated accuracy families.

Holm correction retains all four positive model-level means. Among the 45 cells, 26 have raw p<0.05p < 0.05; 18 survive Holm correction (16 gains, two losses), and 23 survive Benjamini–Hochberg correction at q<0.05q < 0.05. Both Holm-retained losses occur on InternVL3. Table 32 and Table 33 report the corrected comparisons. The stars in Table 1 continue to denote pointwise intervals, not these adjusted tests.

ModelChange (points)Raw ppHolm pp
LLaVA-OV-7B+5.100.00010.0005
Qwen2.5-VL-3B+2.720.00010.0005
Qwen2.5-VL-7B+0.530.00200.0040
Gemma-3-4B+0.840.00010.0005
InternVL3-8B-0.030.85560.8556

Table 32. Model-level nine-benchmark accuracy changes and two-sided bootstrap pp-values, with Holm correction over five models. Bootstrap resolution is 10−410^{-4}.

ModelBenchmarkChangeRaw ppHolm ppBH qq
LLaVA-OV-7BMMBench+6.730.00010.00450.0003
LLaVA-OV-7BPOPE+1.940.00010.00450.0003
LLaVA-OV-7BCV-Bench+3.630.00010.00450.0003
LLaVA-OV-7BEmbSpatial+1.950.00220.05940.0052
LLaVA-OV-7BMMStar+5.240.00010.00450.0003
LLaVA-OV-7BVSR+10.830.00010.00450.0003
LLaVA-OV-7BWhatsUp+2.640.03020.63420.0544
LLaVA-OV-7BCOCO-QA+4.640.00010.00450.0003
LLaVA-OV-7BVG-QA+8.300.00010.00450.0003
Qwen2.5-VL-3BPOPE+1.010.00020.00640.0006
Qwen2.5-VL-3BCV-Bench+2.800.00010.00450.0003
Qwen2.5-VL-3BEmbSpatial+2.070.00060.01800.0015
Qwen2.5-VL-3BVSR+5.320.00010.00450.0003
Qwen2.5-VL-3BWhatsUp+7.360.00010.00450.0003
Qwen2.5-VL-3BVG-QA+3.470.00010.00450.0003
Qwen2.5-VL-7BVSR+1.170.01660.41500.0356
Qwen2.5-VL-7BVG-QA+2.410.00060.01800.0015
Gemma-3-4BCV-Bench+1.450.02680.58960.0502
Gemma-3-4BEmbSpatial+1.180.01980.45540.0387
Gemma-3-4BWhatsUp+5.940.00010.00450.0003
Gemma-3-4BCOCO-QA+0.930.04040.80800.0699
Gemma-3-4BVG-QA-1.490.01740.41760.0356
InternVL3-8BPOPE-0.660.00060.01800.0015
InternVL3-8BMMStar+2.560.00040.01240.0012
InternVL3-8BWhatsUp-3.450.00010.00450.0003
InternVL3-8BVG-QA+2.010.00540.14040.0122

Table 33. Cells with raw p<0.05p < 0.05 from the family of 45 model-benchmark accuracy contrasts. Holm and Benjamini–Hochberg (BH) corrections use all 45 cells, including those not displayed. Changes are percentage points.

Direct-Answer Evaluation

We also repeat the nine-benchmark comparison with greedy decoding and the instruction “Answer with the option’s letter from the given choices directly.” Scoring retains the strict option decision rule. Strengths are selected on validation separately for each model, benchmark, and image setting, using the original grids, heads, and layers. Each model/setting is evaluated in one shared run.

The native setting restores processor defaults: AnyRes for LLaVA-OneVision, dynamic resolution for Qwen2.5-VL, and tiles for InternVL3. Gemma already uses its native setting. The fixed setting retains the main experiment’s preprocessing. Qwen native-resolution runs cover fewer test questions than their fixed counterparts: test counts in benchmark order are 2,425, 5,040, 1,478, 2,040, 851, 854, 448, 1,510, and 818. Other runs retain the main test counts. Each contrast pairs the arms on their common questions; native-versus-fixed differences are not controlled estimates of resolution alone.

Table 34 gives the means; Table 35 and Table 36 retain every benchmark. Mean changes at validation-selected strengths range from −0.23-0.23 to +0.10+0.10 points, with some small negative intervals excluding zero. Under this protocol, unedited accuracy is higher than under sampled reasoning for the two most responsive models, and unparsed responses average at most 0.3%, so accuracy changes are small.

ModelResolutionBaseSHRINKChange95% CI
LLaVA-OV-7BNative76.076.0+0.02[−0.14, +0.18]
LLaVA-OV-7BFixed75.675.4−0.23†[−0.40, −0.06]
Qwen2.5-VL-3BNative74.674.6−0.05[−0.28, +0.18]
Qwen2.5-VL-3BFixed74.273.9−0.22†[−0.43, −0.02]
Qwen2.5-VL-7BNative79.379.1−0.17†[−0.32, −0.02]
Qwen2.5-VL-7BFixed79.179.1+0.04[−0.11, +0.18]
Gemma-3-4BNative57.957.7−0.22[−0.59, +0.15]
InternVL3-8BNative82.382.3+0.04[−0.13, +0.22]
InternVL3-8BFixed82.282.3+0.10[−0.07, +0.27]

Table 34. Direct-answer greedy evaluation: nine-benchmark mean accuracy (%), with per-benchmark validation-selected strengths. Fixed denotes the main experiment’s image preprocessing. Gemma is already native. Changes and pointwise CIs are in percentage points.

BenchmarkLLaVA-OV 7BQwen2.5-VL 3BQwen2.5-VL 7BGemma-3 4BInternVL3 8B
MMBench86.6 / +0.2385.0 / −0.3789.2 / −0.0880.0 / −0.52†89.8 / −0.17
POPE89.7 / +0.0890.2 / −0.2289.0 / −0.0684.6 / −0.46†90.8 / −0.06
CV-Bench78.7 / −0.1974.1 / +0.1480.4 / +0.2055.7 / −1.18†83.2 / +0.09
EmbSpatial72.4 / −0.52†63.9 / −0.2970.7 / +0.0549.2 / +0.9675.4 / +0.14
MMStar62.5 / −0.3355.2 / −0.2461.7 / −0.8244.8 / +0.0867.8 / −0.33
VSR76.5 / −0.2580.0 / +0.4788.1 / +0.0067.4 / −0.0886.0 / −0.16
WhatsUp94.8 / +0.76*92.0 / −0.2295.5 / −0.4550.2 / −0.9194.1 / −0.15
COCO-QA60.6 / −0.2367.5 / −0.3369.7 / +0.0048.0 / +2.00*72.9 / +0.00
VG-QA61.8 / +0.60*63.8 / +0.6169.2 / −0.3741.3 / −1.89†80.4 / +1.03*
Mean76.0 / +0.0274.6 / −0.0579.3 / −0.17†57.9 / −0.2282.3 / +0.04

Table 35. Direct-answer accuracy with native image preprocessing. Each cell gives Base accuracy (%) / SHRINK change (points); */† denote pointwise 95% CIs above/below zero. Strengths are selected on validation, separately for each benchmark and resolution.

BenchmarkLLaVA-OV 7BQwen2.5-VL 3BQwen2.5-VL 7BInternVL3 8B
MMBench86.4 / +0.0385.3 / −0.40†89.3 / −0.26†89.6 / −0.32†
POPE88.1 / +0.0689.4 / +0.0088.9 / −0.1090.1 / −0.19†
CV-Bench77.4 / −0.0572.7 / +0.0079.5 / +0.0082.4 / +0.09
EmbSpatial72.7 / −0.52†62.8 / −0.69†73.3 / −0.62†75.5 / −0.07
MMStar60.0 / −0.4953.7 / −0.7459.8 / +0.4967.3 / +0.16
VSR76.6 / −0.4981.2 / −0.2587.8 / +0.0886.7 / +0.08
WhatsUp96.2 / +0.0090.9 / −0.3095.7 / +0.1595.3 / +0.61*
COCO-QA62.1 / −0.42†66.9 / −0.0568.8 / +0.42*73.4 / +0.09
VG-QA61.0 / −0.1764.6 / +0.4368.6 / +0.1779.8 / +0.43
Mean75.6 / −0.23†74.2 / −0.22†79.1 / +0.0482.2 / +0.10

Table 36. Direct-answer accuracy with fixed main-protocol image preprocessing. Each cell gives Base accuracy (%) / SHRINK change (points); */† denote pointwise 95% CIs above/below zero. Strengths are selected on validation, separately for each benchmark and resolution.

Interpreting the sampled gains. The existing paired decomposition separates pairs with at least one unparsed response from pairs where both responses parse, using all questions as denominator. These parts sum to the accuracy change before rounding. Table 37 and Table 38 report the main results. In the direction study, the sink’s LLaVA-OneVision gain is +4.57+4.57 points: +5.55+5.55 from pairs involving unparsed responses and −0.98-0.98 from pairs where both parse; Qwen-3B has +3.57+3.57 and −0.03-0.03 contributions to its +3.55+3.55-point gain (rounded separately). Much of the sampled-reasoning gain therefore comes from responses that reach a clear final answer under SHRINK.

BenchmarkLLaVA-OV 7BQwen2.5-VL 3BQwen2.5-VL 7BGemma-3 4BInternVL3 8B
MMBench23.5→13.523.5 \to13.510.3→8.510.3 \to8.50.5→0.40.5 \to0.40.0→0.00.0 \to0.00.2→0.20.2 \to0.2
POPE29.2→26.829.2 \to26.85.7→3.95.7 \to3.90.3→0.20.3 \to0.20.0→0.00.0 \to0.00.1→0.60.1 \to0.6
CV-Bench25.7→20.225.7 \to20.210.7→5.110.7 \to5.10.9→0.60.9 \to0.60.0→0.10.0 \to0.10.2→0.20.2 \to0.2
EmbSpatial18.5→13.018.5 \to13.012.3→9.112.3 \to9.11.4→1.11.4 \to1.10.1→0.00.1 \to0.00.2→0.10.2 \to0.1
MMStar27.3→17.827.3 \to17.814.9→12.514.9 \to12.52.0→1.22.0 \to1.22.4→2.12.4 \to2.11.9→1.51.9 \to1.5
VSR44.1→27.944.1 \to27.911.9→2.511.9 \to2.50.3→0.10.3 \to0.10.0→0.00.0 \to0.00.0→0.00.0 \to0.0
WhatsUp18.4→14.818.4 \to14.814.5→4.114.5 \to4.10.7→0.40.7 \to0.40.1→0.20.1 \to0.20.4→0.10.4 \to0.1
COCO-QA34.9→27.534.9 \to27.59.3→5.89.3 \to5.80.4→0.50.4 \to0.50.0→0.10.0 \to0.10.3→0.20.3 \to0.2
VG-QA33.9→24.933.9 \to24.913.7→9.013.7 \to9.00.3→0.10.3 \to0.10.1→0.10.1 \to0.10.2→0.20.2 \to0.2

Table 37. Unparsed test responses (%), unedited →\to SHRINK, at the validation-selected strengths in Table 1.

BenchmarkLLaVA-OV 7BQwen2.5-VL 3BQwen2.5-VL 7BGemma-3 4BInternVL3 8B
MMBench+8.27/−1.54+8.27 / -1.54+1.21/−0.51+1.21 / -0.51+0.03/−0.23+0.03 / -0.23+0.00/−0.20+0.00 / -0.20−0.04/−0.08-0.04 / -0.08
POPE+1.99/−0.05+1.99 / -0.05+1.50/−0.49+1.50 / -0.49+0.03/+0.15+0.03 / +0.15+0.00/−0.09+0.00 / -0.09−0.48/−0.18-0.48 / -0.18
CV-Bench+3.82/−0.19+3.82 / -0.19+3.18/−0.38+3.18 / -0.38+0.16/+0.25+0.16 / +0.25−0.03/+1.48-0.03 / +1.48−0.02/−0.82-0.02 / -0.82
EmbSpatial+3.37/−1.42+3.37 / -1.42+2.14/−0.07+2.14 / -0.07+0.22/−0.01+0.22 / -0.01+0.05/+1.13+0.05 / +1.13+0.00/−0.53+0.00 / -0.53
MMStar+5.54/−0.30+5.54 / -0.30+1.04/−0.30+1.04 / -0.30+0.52/+0.82+0.52 / +0.82+0.14/−0.05+0.14 / -0.05+0.22/+2.35+0.22 / +2.35
VSR+12.22/−1.39+12.22 / -1.39+6.68/−1.36+6.68 / -1.36+0.19/+0.98+0.19 / +0.98+0.00/−0.22+0.00 / -0.22+0.00/+0.76+0.00 / +0.76
WhatsUp+3.50/−0.86+3.50 / -0.86+8.02/−0.66+8.02 / -0.66+0.25/−0.76+0.25 / -0.76−0.10/+6.04-0.10 / +6.04+0.25/−3.70+0.25 / -3.70
COCO-QA+4.76/−0.12+4.76 / -0.12+2.05/−1.04+2.05 / -1.04−0.12/−0.11-0.12 / -0.11+0.02/+0.91+0.02 / +0.91+0.09/−0.12+0.09 / -0.12
VG-QA+6.06/+2.24+6.06 / +2.24+2.99/+0.49+2.99 / +0.49+0.09/+2.33+0.09 / +2.33+0.00/−1.49+0.00 / -1.49+0.00/+2.01+0.00 / +2.01

Table 38. Paired format / answer contributions (percentage points) to sampled test accuracy. Both use all test questions as denominator and sum to the total change before rounding. Strengths are those in Table 1.

References

  1. [1]Andy Arditi, Oscar Obeso, Aa quib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction. In Advances in Neural Information Processing Systems, 2024. 2arxiv.org/abs/2406.11717
  2. [2]Shuai Bai, Yuxuan Cai, Ruizhe Chen, et al. Qwen3-VL technical report, 2025. S11arxiv.org/abs/2511.21631
  3. [3]Shuai Bai, Keqin Chen, Xuejing Liu, et al. Qwen2.5-VL technical report, 2025. 1, 5arxiv.org/abs/2502.13923
  4. [4]Federico Barbero, Alvaro Arroyo, Xiangming Gu, Christos Perivolaropoulos, Michael Bronstein, Petar Veličković, and Razvan Pascanu. Why do llms attend to the first token? arXiv preprint arXiv:2504.02732, 2025. 2arxiv.org/abs/2504.02732
  5. [5]Max Belitsky, Dawid J Kopiczko, Michael Dorkenwald, M Jehanzez Mirza, James R Glass, Cees GM Snoek, and Yuki M Asano. Kv cache steering for controlling frozen llms. arXiv preprint arXiv:2507.08799, 2025. 2arxiv.org/abs/2507.08799
  6. [6]Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. Leace: Perfect linear concept erasure in closed form. Advances in Neural Information Processing Systems, 36:66044–66063, 2023. 2arxiv.org/abs/2306.03819
  7. [7]Nicola Cancedda. Spectral filters, dark signals, and attention sinks. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4792–4808, Bangkok, Thailand, 2024. Association for Computational Linguistics. 2
  8. [8]Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are we on the right way for evaluating large vision-language models? In Advances in Neural Information Processing Systems, 2024. 5arxiv.org/abs/2403.20330
  9. [9]Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che. M³CoT: A novel benchmark for multimodal multi-step multi-modal chain-of-thought. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pages 8199–8221, 2024. S10DOI
  10. [10]Shiqi Chen, Tongyao Zhu, Ruocheng Zhou, Jinghan Zhang, Siyang Gao, Juan Carlos Niebles, Mor Geva, Junxian He, Jiajun Wu, and Manling Li. Why is spatial reasoning hard for VLMs? an attention mechanism perspective on focus areas. In Proceedings of the 42nd International Conference on Machine Learning, pages 9910–9932. PMLR, 2025. 2, 5arxiv.org/abs/2503.01773
  11. [11]Yangneng Chen, Junlin Li, Weijun Yao, Xilai Ma, Guodong Du, Wenya Wang, and Jing Li. Vocabulary hijacking in LVLMs: Unveiling critical attention heads by excluding invert tokens to mitigate hallucination. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 38458–38485. Association for Computational Linguistics, 2026. 2
  12. [12]Yung-Chin Chen, Chung Peng Lee, Ze-Wei Liou, and Naveen Verma. Massive spikes in LLMs are bias vectors: Mechanistic uncovering and spike-free quantization, 2026. 2arxiv.org/abs/2606.02288
  13. [13]Jiho Choi, Jaemin Kim, Sanghwan Kim, Seunghoon Hong, and Jin-Hwi Park. When sinks help or hurt: Unified framework for attention sink in large vision-language models, 2026. 2arxiv.org/abs/2604.03316
  14. [14]Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In International Conference on Learning Representations, 2024. 2arxiv.org/abs/2309.16588
  15. [15]Mengfei Du, Binhao Wu, Zejun Li, Xuanjing Huang, and Zhongyu Wei. EmbSpatial-Bench: Benchmarking spatial understanding for embodied tasks with large vision-language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 346–355. Association for Computational Linguistics, 2024. 5
  16. [16]Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. MME: A comprehensive evaluation benchmark for multimodal large language models, 2023. S10
  17. [17]Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, and Ranjay Krishna. BLINK: Multimodal large language models can see but not perceive. In European Conference on Computer Vision, 2024. S10arxiv.org/abs/2404.12390
  18. [18]Gemma Team, Aishwarya Kamath, Johan Ferret, et al. Gemma 3 technical report, 2025. 5, S11
  19. [19]Nathan Godey, Éric Clergerie, and Benoît Sagot. Anisotropy is inherent to self-attention in transformers. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 35–48. Association for Computational Linguistics, 2024. 1, 2
  20. [20]Nathan Godey, Alessio Devoto, Yu Zhao, Simone Scardapane, Pasquale Minervini, Éric de la Clergerie, and Benoît Sagot. Q-Filters: Leveraging query-key geometry for efficient key-value cache compression. In ICLR Workshop on Sparsity in LLMs, 2025. 2, 8
  21. [21]Xuan Gong, Tianshi Ming, Xinpeng Wang, and Zhihua Wei. DAMRO: Dive into the attention mechanism of LVLM to reduce object hallucination. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7696–7712, Miami, Florida, USA, 2024. Association for Computational Linguistics. 2arxiv.org/abs/2410.04514
  22. [22]Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhu Zhang, Cuxiao Du, Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view. In In- ternational Conference on Learning Representations, 2025. 1, 2arxiv.org/abs/2410.10781
  23. [23]Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. HallusionBench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14375–14385, 2024. S10
  24. [24]Tianyu Guo, Druv Pai, Yu Bai, Jiantao Jiao, Michael I Jordan, and Song Mei. Active-dormant attention heads: Mechanistically demystifying extreme-token phenomena in llms. arXiv preprint arXiv:2410.13835, 2024. 2arxiv.org/abs/2410.13835
  25. [25]Feijiang Han, Xiaodong Yu, Jianheng Tang, Delip Rao, Weihua Du, and Lyle Ungar. ZeroTuning: Unlocking the initial token’s power to enhance large language models without training. In International Conference on Learning Representations, 2026. 1, 2arxiv.org/abs/2505.11739
  26. [26]Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaq i Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. OPERA: Alleviating hallucination in multimodal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13418–13427, 2024. 2DOI
  27. [27]Drew A. Hudson and Christopher D. Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019. S12arxiv.org/abs/1902.09506
  28. [28]Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. S12arxiv.org/abs/2310.06825
  29. [29]Nicholas Jiang, Amil David, Alexei Efros, and Yossi Gandelsman. Vision transformers don’t need trained registers. In Advances in Neural Information Processing Systems, pages 56557–56595, 2025. 2arxiv.org/abs/2506.08010
  30. [30]Pengkun Jiao, Bin Zhu, Jingjing Chen, and Yu-Gang Jiang. Semantic-structural decoupling: Disentangling semantic attention from structural bias in the attention manifold. In Proceedings of the 34th ACM International Conference on Multimedia, pages 1–21. ACM, 2026. 1, 2
  31. [31]Mingyu Jin, Kai Mei, Wujiang Xu, Mingjie Sun, Ruxiang Tang, Mengnan Du, Zirui Liu, and Yongfeng Zhang. Massive values in self-attention modules are the key to contextual knowledge understanding. In Proceedings of the 42nd International Conference on Machine Learning, pages 28063–28096. PMLR, 2025. 2arxiv.org/abs/2502.01563
  32. [32]Yujin Jo, Sangyoon Bae, and Taesup Kim. Attention-space contrastive guidance for efficient hallucination mitigation in LVLMs. In Findings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2026. 2, 5, S12arxiv.org/abs/2601.13707
  33. [33]Amita Kamath, Jack Hessel, and Kai-Wei Chang. What’s “up” with vision-language models? investigating their struggle with spatial reasoning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9161–9175. Association for Computational Linguistics, 2023. 5
  34. [34]Zhehan Kan, Xin Li, Yanli Liu, Xiaoch en Yang, Xinghua Jiang, Yinsong Liu, Deqiang Jiang, Xing Sun, Qingmin Liao, and Wenming Yang. RAR: Reversing visual attention sinking for unlocking potential in multimodal large language models. In The Fourteenth International Conference on Learning Representations, 2026. 2
  35. [35]Seil Kang, Jinyeong Kim, Junhyeok Kim, and Seong Jae Hwang. Your large vision-language model only needs a few attention heads for visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9339–9350, 2025. 2arxiv.org/abs/2503.06287
  36. [36]Seil Kang, Jinyeong Kim, Junhyeok Kim, and Seong Jae Hwang. See what you are told: Visual attention sink in large multimodal models. In International Conference on Learning Representations, 2025. 1, 2, 5, S3arxiv.org/abs/2503.03321
  37. [37]Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In European Conference on Computer Vision, pages 235–251, 2016. S10arxiv.org/abs/1603.07396
  38. [38]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th ACM Symposium on Operating Systems Principles, pages 611–626, 2023. S1, S4arxiv.org/abs/2309.06180
  39. [39]Sicong Leng, Hang Zhang, Guangzhou Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13872–13882, 2024. S12arxiv.org/abs/2311.16922
  40. [40]Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. LLaVA-OneVision: Easy visual task transfer, 2024. 5arxiv.org/abs/2408.03326
  41. [41]Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from a language model. In Advances in Neural Information Processing Systems, 2023. 2arxiv.org/abs/2306.03341
  42. [42]Siqian Li, Kaiqi Jiang, Jiachen Sun, and Tianyang Hu. The structural origin of attention sink: Variance discrepancy, super neurons, and dimension disparity, 2026. 2arxiv.org/abs/2605.06611
  43. [43]Weixian Waylon Li, Yuchen Niu, Yongxin Yang, Keshuang Li, Tiejun Ma, and Shao B. Cohen. Spectral attention steering for prompt highlighting. In The Fourteenth International Conference on Learning Representations, 2026. 2arxiv.org/abs/2603.01281
  44. [44]Yifan Li, Yifan Du, Kun Zhou, Junpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 292–305. Association for Computational Linguistics, 2023. 5, S12arxiv.org/abs/2305.10355
  45. [45]Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 11:635–651, 2023. 5
  46. [46]Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26286–26296. IEEE, 2024. S11, S12arxiv.org/abs/2310.03744
  47. [47]Junman Liu, Xinyan Liu, Peifeng Gao, Zhaobo Qi, Beichen Zhang, Weigang Zhang, and Antoni Bert Chen. SinkRouter: Sink-aware routing for efficient long-context decoding in large language and multimodal models. arXiv preprint arXiv:2604.16883, 2026. 2arxiv.org/abs/2604.16883
  48. [48]Shi Liu, Kechen Zheng, and Wei Chen. Paying more attention to image: A training-free method for alleviating hallucination in LVMs. In European Conference on Computer Vision, 2024. 1, 2arxiv.org/abs/2407.21771
  49. [49]Sheng Liu, Haotian Ye, and James Y Zou. Reducing hallucinations in large vision-language models via latent space steering. In International Conference on Learning Representations, pages 72402–72419, 2025. 2arxiv.org/abs/2410.15778
  50. [50]Xu Liu, Guikun Chen, and Wenguan Wang. SinkTrack: Attention sink based context anchoring for large language models. In International Conference on Learning Representations, 2026. 2arxiv.org/abs/2604.10027
  51. [51]Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. MMBench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, 2024. 5arxiv.org/abs/2307.06281
  52. [52]Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In Advances in Neural Information Processing Systems, 2022. S10arxiv.org/abs/2209.09513
  53. [53]Jiayun Luo, Wan-Cyuan Fan, Lyuyang Wang, Xiangteng He, Tanzila Rahman, Purang Abolmaesumi, and Leonid Sigal. To sink or not to sink: Visual information pathways in large vision-language models. In International Conference on Learning Representations, 2026. 2arxiv.org/abs/2510.08510
  54. [54]Jiaqi Mu, Suma Bhat, and Pramod Viswanath. All-but-the-top: Simple and effective postprocessing for word representations, 2018. 2
  55. [55]Junyong Park, Dalton Jones, Matthew Morse, Raghavv Goel, Mingu Lee, and Christopher Lott. KeyDiff: Key similarity-based KV cache eviction for long-context LLM inference in resource-constrained environments. In Advances in Neural Information Processing Systems, 2025. 2arxiv.org/abs/2504.15364
  56. [56]Junyong Park, Jungwook Choi, and Mingu Lee. ValueDiff: Value-geometric KV cache eviction for sink-suppressed LLMs. arXiv preprint arXiv:2609.23314, 2026. 2arxiv.org/abs/2609.23314
  57. [57]Zheng Qi, Chao Shang, Evangelia Spiliopoulou, and Nikolaos Pappas. Capturing gaze shifts for guidance: Cross-modal fusion enhancement for VLM hallucination mitigation. In International Conference on Machine Learning, 2026. 2, 5, S12arxiv.org/abs/2510.22067
  58. [58]Yifu Qiu, Zheng Zhao, Yftah Ziser, Anna Korhonen, Edoardo M. Ponti, and Shay B. Cohen. Spectral editing of activations for large language model alignment. In Advances in Neural Information Processing Systems, 2024. 2arxiv.org/abs/2405.09719
  59. [59]Enrique Queipo-de Llano, Álvaro Arroyo, Federico Barbero, Xiaowen Dong, Michael Bronstein, Yann LeCun, and Ravid Shwartz-Ziv. Attention sinks and compression valleys in LLMs are two sides of the same coin. In International Conference on Learning Representations, 2026. 2
  60. [60]Yuval Ran-Milo, Hila Ofek, and Shahar Mendel. A mechanistic account of attention sinks in GPT-2: One circuit, broader implications for mitigation. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics, 2026. 2
  61. [61]Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null it out: Guarding protected attributes by iterative nullspace projection. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 7237–7256, 2020. 2arxiv.org/abs/2004.07667
  62. [62]Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. Steering llama 2 via contrastive activation addition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15504–15522, 2024. 2
  63. [63]Valeria Ruscio, Umberto Nanni, and Fabrizio Silvestri. What are you sinking? a geometric approach on attention sink, 2025. 2
  64. [64]Sreetama Sarkar, Yue Che, Alex Gavin, Peter Anthony Beereel, and Souvik Kundu. Mitigating hallucinations in vision-language models through image-guided head suppression. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 12481–12500, Suzhou, China, 2025. Association for Computational Linguistics. 2, 5, S3
  65. [65]Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-OKVQA: A benchmark for visual question answering using world knowledge. In European Conference on Computer Vision, 2022. S10, S12arxiv.org/abs/2206.01718
  66. [66]Seungjun Shin, Jaehoon Oh, and Dokwan Oh. Ortho-Rank: Token selection via sink token orthogonality for efficient LLM inference. In Proceedings of the 42nd International Conference on Machine Learning, pages 55172–55194. PMLR, 2025. 2
  67. [67]Tripti Shukla and Zsolt Kira. SAGE: Sink-aware grounded decoding for multimodal hallucination mitigation. arXiv preprint arXiv:2603.27898, 2026. 2arxiv.org/abs/2603.27898
  68. [68]SkunkworksAI. BakLLaVA-1.1 model card. Hugging Face, 2023. S12
  69. [69]Mingjie Sun, Xinlei Chen, J. Zico Kolter, and Zhuang Liu. Massive activations in large language models. In First Conference on Language Modeling, 2024. 1, 2arxiv.org/abs/2402.17762
  70. [70]Shangwen Sun, Alfredo Canziani, Yann LeCun, and Jiachen Zhu. The spike, the sparse and the sink: Anatomy of massive activations and attention sinks, 2026. 2arxiv.org/abs/2603.05498
  71. [71]Feilong Tang, Zile Huang, Chengzhi Liu, Qiang Sun, Harry Yang, and Ser-Nam Lim. Intervening anchor token: Decoding strategy in alleviating hallucinations for mllms. In International Conference on Learning Representations, 2025. 2
  72. [72]Feilong Tang, Chengzhi Liu, Zhongxing Xu, Ming Hu, Zile Huang, Haochen Xue, Ziyang Chen, Zelin Peng, Zhiwei Yang, Sijin Zhou, Wenxue Li, Yulong Li, Wenxuan Song, Shiyan Su, Wei Feng, Jionglong Su, Mingquan Lin, Yifan Peng, Xuelian Cheng, Imran Razzak, and Zongyuan Ge. Seeing far and clearly: Mitigating hallucinations in MLLMs with attention causal decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26147–26159, 2025. 2arxiv.org/abs/2505.16652
  73. [73]Lexiang Tang, Xianwei Zhuang, Bang Yang, Zhiyuan Hu, Hongxiang Li, Lu Ma, Jinghan Ru, and Yuexian Zou. Not all tokens and heads are equally important: Dual-level attention intervention for hallucination mitigation, 2025. 2arxiv.org/abs/2506.12609
  74. [74]Karn Tiwari, Varnith Chordia, and A. P. Prathosh. Spectral query-key product weight steering for training-free VLM hallucination mitigation, 2026. 2arxiv.org/abs/2606.20419
  75. [75]Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middpuegu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, Austin Wang, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-1: A fully open, vision-centric exploration of multimodal LLMs. In Advances in Neural Information Processing Systems, 2024. 5arxiv.org/abs/2406.16860
  76. [76]Shengbang Tong, Zhuang Liu, Yuexian Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal LLMs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9568–9578, 2024. S10arxiv.org/abs/2401.06209
  77. [77]Chongjun Tu, Peng Ye, Dongzhan Zhou, Lei Bai, Gang Yu, Tao Chen, and Wanli Ouyang. Attention reallocation: Towards zero-cost and controllable hallucination mitigation of MLLMs. arXiv preprint arXiv:2503.08342, 2025. 2
  78. [78]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. 1, S5
  79. [79]Junyang Wang, Yuhang Wang, Quan Xiu, Jing Zhang, Yukai Gu, Haitao Jia, Jiaqi Wang, Haiyang Xu, Ming Yan, Ji Zhang, and Jitao Sang. AMBER: An LLM-free multi-dimensional benchmark for MLLMs hallucination evaluation, 2023. S12arxiv.org/abs/2311.07397
  80. [80]Weiyun Wang, Zhangwei Gao, Lixin Gu, et al. InternVL3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency, 2025. S11arxiv.org/abs/2508.18265
  81. [81]Jeffrey T. H. Wong, Cheng Zhang, Louis Mahon, Wayne Luk, Anton Ispoussu, and Yiren Zhao. On the existence and behavior of secondary attention sinks. In ICLR Workshop on Unifying Concept Representation Learning, 2026. 2arxiv.org/abs/2512.22213
  82. [82]xAI. Grok-1.5 vision preview. https://x.ai/news/grok-1.5v, 2024. S10
  83. [83]Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In International Conference on Learning Representations, 2024. 1, 2arxiv.org/abs/2309.17453
  84. [84]Le Yang, Ziwei Zheng, Boxu Chen, Zhengyu Zhao, Chenhao Lin, and Chao Shen. Nullu: Mitigating object hallucinations in large vision-language models via hallucspace projection. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14635–14645. IEEE, 2025. 2arxiv.org/abs/2412.13817
  85. [85]Hao Yin, Guangzhou Si, and Zilei Wang. Clearsight: Visual signal enhancement for object hallucination mitigation in multimodal large language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14625–14634, 2025. 2arxiv.org/abs/2503.13107
  86. [86]Suho Yoo, Youngjoon Jang, and Joon Son Chung. On the nature of attention sink that shapes decoding strategy in Omni-LLMs, 2026. Version 3. 2arxiv.org/abs/2603.14337
  87. [87]Yijiong Yu, Huiqiang Jiang, Xufang Luo, Qianhui Wu, Chin-Yew Lin, Dongsheng Li, Yuqing Yang, Yongfeng Huang, and Lili Qiu. Mitigate position bias in LLMs via scaling a single hidden states channel. In Findings of the Association for Computational Linguistics: ACL 2025, pages 6092–6111. Association for Computational Linguistics, 2025. 2DOI
  88. [88]Zhongzhi Yu, Zheng Wang, Yonggan Fu, Huihong Shi, Khalid Shaikh, and Yingyan Celine Lin. Unveiling and harnessing hidden attention sinks: Enhancing large language models without training through attention calibration. In Proceedings of the 41st International Conference on Machine Learning, pages 57659–57677, 2024. 2arxiv.org/abs/2406.15765
  89. [89]Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9556–9567, 2024. S10arxiv.org/abs/2311.16502
  90. [90]Dengjia Zhang, Alexander Martin, Pengyu Yan, Xihan Sun, Jiaqi Li, Benjamin Van Durme, Andrew Yates, Reno Kriz, and William Walden. Attention sinks in VLMs are borrowed, not seen. arXiv preprint arXiv:2610.15855, 2026. 2arxiv.org/abs/2610.15855
  91. [91]Qingru Zhang, Chandan Singh, Liyuan Liu, Xiaodong Liu, Bin Yu, Jianfeng Gao, and Tuo Zhao. Tell your model where to attend: Post-hoc attention steering for LLMs. In International Conference on Learning Representations, 2024. 2arxiv.org/abs/2311.02262
  92. [92]Stephen Zhang, Mustafa Khan, and Vardan Papyan. Attention sinks: A ‘Catch, Tag, Release’ mechanism for embeddings. In Advances in Neural Information Processing Systems, 2025. 2arxiv.org/abs/2502.00919
  93. [93]Xiaofeng Zhang, Yuanchao Zhu, Qiyan Zhao, Xiaosong Yuan, Jiawei Cao, and Yuhang Chen. TokenPenalty: Alleviating attention sinks and positional decay in LVLMs. In Findings of the Association for Computational Linguistics: ACL 2026, pages 12894–12906. Association for Computational Linguistics, 2026. 2DOI
  94. [94]Jinguo Zhu, Weiyun Wang, Zhe Chen, et al. InternVL3: Exploring advanced training and test-time recipes for open-source multimodal models, 2025. 5arxiv.org/abs/2504.10479

Paper details

vlnc:W_eva5ta48
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Codex, Fable 5.2, Claude

View statement
Looking for related papers…
Contents