betaFeedback

WorksComputer Science

BOS Summary: The First Token Matters

Abstract

Large vision-language models (LVLMs) and language models (LLMs) often exhibit attention sinks, whereby particular positions, such as the beginning-of-sequence (BOS) token, receive a disproportionate share of attention. Under standard causal masking, however, the BOS query attends only to itself, so the representation that later tokens repeatedly access carries no information about the input. We introduce BOS Summary, a training-free inference-time method that repurposes this persistent pathway to carry input context. At a single transformer layer during prefill, BOS Summary relaxes the causal constraint for the BOS query row alone, letting BOS aggregate all visual and textual input tokens into a self-derived context representation while every other token retains standard causal attention. The enriched representation then remains accessible to the model throughout decoding, with no additional parameters, auxiliary modules, or decoding-time overhead. Across multiple VLM and LLM families and eight multimodal and text-only benchmarks, BOS Summary improves over direct and Chain-of-Thought prompting and outperforms recent sink-intervention methods on multi-step reasoning tasks. A mechanistic analysis shows that making BOS input-specific also weakens its sink dominance and redistributes attention toward content-bearing prompt tokens, indicating that changing what a sink carries reshapes how later layers allocate attention.

Introduction

Recent advances in vision-language models (VLMs) have enabled strong performance across multimodal understanding, visual reasoning, and instruction-following tasks [4, 28]. Despite this progress, VLMs remain prone to hallucination and context loss: during long-form generation, they may produce statements unsupported by the image or gradually deviate from the original instruction. These limitations are well documented by hallucination benchmarks [26, 35, 38] and have motivated both training-free decoding methods [13, 22, 25] and instruction-tuning approaches [27]. One mechanism associated with these failures is attention drift [22, 32]: as generation proceeds, attention can shift toward previously generated tokens and away from the grounding image and input instruction. This problem becomes particularly important in extended multimodal generation [9], where visual evidence must remain accessible across many decoding steps even as the generated context continues to grow.

This motivates us to look for a representation that remains available throughout decoding and is already repeatedly accessed by later tokens. Transformers naturally provide such a candidate through the attention-sink phenomenon, in which the first token receives a disproportionate share of attention from subsequent tokens, often largely independent of its content [5, 19, 37, 40]. Related sink behavior has also been observed in vision and vision-language models [14, 23], and recent work suggests that sink representations can influence subsequent computation rather than merely absorbing attention [43]. In our setting, this first position is the beginning-of-sequence (BOS) token. Because later tokens repeatedly access BOS throughout decoding, conditioning its representation on the visual input and instruction could turn this existing pathway into a persistent carrier of prompt context.

Key Question

Can we turn the first token from an uninformative attention sink into a context-dependent anchor that gathers information from the visual input and instruction, allowing the model to access this information throughout generation?

Recent inference-time methods demonstrate that attention sinks can serve as useful control points for shaping model behavior, either by rescaling the attention they receive or by injecting additional information into their representations [21, 32]. Rather than externally modifying the sink or its incoming attention, we explore whether BOS can derive a prompt representation directly through the model’s own attention. This offers a complementary way to exploit the sink: changing what BOS carries using information already present in the prompt.

As illustrated in Figure 1, BOS Summary turns the existing BOS token into a training-free, one-token summary of the visual input and instruction. Under standard causal attention, BOS attends only to itself, while subsequent tokens can attend back to BOS throughout generation. We relax the causal constraint only for the BOS query row at a selected layer during prefill, allowing BOS to aggregate the visual and textual prompt while leaving the rest of the causal attention mask unchanged. This simple intervention repurposes an existing attention pathway into a persistent carrier of prompt context without changing the model architecture or decoding procedure.

Figure 1. BOS Summary. Standard causal attention leaves BOS attending only to itself, making it an uninformative sink despite receiving heavy attention from all later tokens (top). We unmask the BOS row at one selected layer so BOS attends to the visual and instruction tokens, turning it into a context aggregator that later tokens read from throughout generation (bottom). All other attention patterns, the architecture are unchanged.

Building on this design, our contributions are threefold:

  • We introduce BOS Summary, a training-free, plug-and-play intervention that turns BOS into a one-token, prompt-conditioned representation via a single prefill-time mask edit, with no retraining, auxiliary modules, or decoding-time cost.

  • We show that BOS Summary improves reasoning and grounding across diverse VLMs, LLMs, and benchmarks targeting context forgetting and object hallucination, where faithful access to the input matters most.

  • We provide a mechanistic analysis showing that an input-specific BOS weakens the sink’s attention dominance and shifts attention back toward content-bearing input tokens, revealing that changing what a sink carries reshapes how later layers allocate attention.

Related Work

Attention Sink. Transformer language models often concentrate disproportionate, content-independent attention on the first token, forming an attention sink [40]; related sinks also appear in VLMs on structural and weakly informative visual tokens [23, 47]. Prior work studies their mechanisms [6, 20, 34, 37], emergence [19, 36], and functional role [5, 46], with analogous phenomena motivating register tokens in vision transformers [14] and finer VLM sink analyses [12]. Recent work further suggests that sink representations can carry global information and shape decoding [43], motivating inference-time interventions [21, 30, 32, 44]. We build on this view by asking what the sink can be made to carry and how a minimal intervention changes its downstream role.

Repurposing the sink at inference time. A growing body of training-free methods intervenes on the attention sink to shape model behavior without modifying model weights, differing primarily in what aspect of the sink they modify. The first group changes how much attention the sink receives: ZeroTuning [21] calibrates per-head attention to the initial token, and ACT [44] rescales attention for activation steering. The second group changes what the sink carries: SinkTrack [32] injects externally derived context features, and learned gating can regulate the information stored in the sink [12]. Closest to our work, OutRo [43] also relaxes the causal mask on sink positions, but couples it with sink detection and head-output rotation applied throughout decoding, and reports that the mask relaxation alone yields only marginal gains on video and audio-visual QA. A third group treats sink dominance as an efficiency signal for skipping saturated attention heads [30]. BOS Summary isolates mask relaxation as a prefill-only edit applied solely to the BOS query row, while leaving the causal attention mask for all other query rows and the decoding process unchanged. Without injection, gating, detection, rotation, or added parameters, it shows that this minimal intervention is effective for both image-grounded and text-only reasoning across several models and benchmarks.

Attention control for grounding and hallucination. Beyond sink-specific interventions, training-free methods mitigate VLM hallucination through attention and decoding. Attention-based approaches reweight visual tokens or suppress non-informative attention [2, 17, 31, 41, 45], while decoding methods contrast logits across visual inputs, depth, or image-biased distributions [10, 13, 25, 39], or use attention-aware beam-search penalties [22]. Other methods revise outputs post hoc [42] or reduce hallucination through training [27]. In contrast, BOS Summary modifies one sink position and uses its downstream influence rather than reweighting content tokens, altering logits, or correcting outputs. Further discussion appears in Appendix § 8.

Methodology

Motivation. During autoregressive generation, the visual input and instruction are represented by a fixed set of prompt tokens, whereas the decoding history grows at every step. Information from the original prompt must therefore compete with an increasingly long sequence of generated tokens. A natural way to preserve access to this information is to encode it into a fixed representation that remains available throughout decoding. The BOS token is particularly well suited for this role: it is present from the beginning of the sequence, remains in the cached context, and often receives substantial attention from later tokens due to the attention-sink phenomenon [5, 40].

Under standard causal masking, however, the BOS query can attend only to itself and therefore cannot incorporate information from the visual and textual tokens that follow it. This creates an underused pathway: later tokens repeatedly attend to BOS, yet the representation they access is not conditioned on the prompt. BOS Summary exploits this asymmetry by relaxing the causal constraint only for the BOS query at a selected layer during prefill, allowing BOS to aggregate information from the multimodal prompt while leaving the causal attention pattern of every other token unchanged.

The resulting prompt-conditioned BOS state propagates through subsequent transformer layers and remains accessible to later generated tokens, providing a persistent pathway to information from the original prompt through a sequence position that the model already attends to.

Formal Definition. We consider a decoder-only transformer (VLM or LLM) with LL layers. Let X(ℓ)∈Rn×dX^{(\ell)} \in\mathbb{R}^{n \times d} denote the input hidden states to layer ℓ\ell, where nn is the sequence length, dd is the hidden dimension, and X0(ℓ)X^{(\ell)}_0 corresponds to the BOS token. For clarity, we present a single-head formulation and omit architecture-specific operations (e.g., normalization and positional encoding), since BOS Summary modifies only the attention mask, and the same mask edit is applied uniformly across all heads. Queries, keys, and values are computed as

Q=X(ℓ)WQ,K=X(ℓ)WK,V=X(ℓ)WV,(1)\begin{aligned} Q &= X^{(\ell)}W_Q,\\ K &= X^{(\ell)}W_K,\\ V &= X^{(\ell)}W_V, \tag*{(1)} \end{aligned}

where WQW_Q, WKW_K, WV∈Rd×dkW_V \in\mathbb{R}^{d \times d_k} are the model’s frozen projection matrices. The attention weights and head output are

A=softmax⁡(QKTdk+M),O=AV,(2)A = \operatorname{softmax}\left(\frac{QK^{\mathsf{T}}}{\sqrt{d_k}} + M\right), \qquad O = AV, \tag*{(2)}

where M∈{0,−∞}n×nM \in\{0,-\infty\}^{n \times n} is the additive attention mask applied before the softmax. Under standard causal masking, Mijcausal=0M^{\mathrm{causal}}_{ij}=0 for j≤ij \le i and −∞-\infty otherwise, so the BOS query attends only to itself, yielding A0,⋅A_{0,\cdot} one-hot at position 0 and O0=V0O_0=V_0. BOS Summary applies its intervention at a single layer ℓ⋆∈{1,…,L}\ell^\star\in\{1,\ldots,L\} during the prefill pass, i.e., while the entire multimodal prompt is encoded before autoregressive decoding. The mask MM equals McausalM^{\mathrm{causal}} at every layer except ℓ⋆\ell^\star; at this layer we replace only its first row,

M0,j=0,j=0,…,n−1,(3)M_{0,j}=0,\qquad j=0,\ldots,n-1, \tag*{(3)}

while leaving every other row unchanged. Since M0,j=0M_{0,j}=0 for all jj, the softmax term in Equation (2) reduces at position 0 to an unmasked attention distribution, and the BOS attention output becomes a content-weighted aggregation of the entire prompt,

O0=∑j=0n−1A0jVj,A0j=softmax⁡j(Q0KjTdk).(4)O_0=\sum_{j=0}^{n-1}A_{0j}V_j,\qquad A_{0j}=\operatorname{softmax}_j\left(\frac{Q_0K_j^{\mathsf{T}}}{\sqrt{d_k}}\right). \tag*{(4)}

The resulting BOS attention output is incorporated into the BOS hidden state through the transformer’s standard residual update. At every layer above ℓ⋆\ell^\star, this self-derived summary shapes the BOS key and value representations, allowing later tokens to access information from the original prompt through the BOS position; the summary’s influence thus depends on the choice of ℓ⋆\ell^\star (see Appendix § 12). During autoregressive decoding, BOS Summary does not modify the attention mask; the model therefore follows its standard causal attention pattern. Algorithm 1 provides pseudocode for our method, while Appendix § 9 presents a visual overview of the procedure.

Algorithm 1 BOS Summary
Input: Hidden states X(ℓ)∈Rn×dX^{(\ell)} \in\mathbb{R}^{n \times d}, current layer ℓ\ell, intervention layer ℓ⋆\ell^\star, and prefill indicator pp
Output: Attention output OO
1: function BOSSummaryAttention(X(ℓ),ℓ,ℓ⋆,p)(X^{(\ell)},\ell,\ell^\star,p)
2: (Q,K,V)←ProjectQKV⁡(X(ℓ))(Q,K,V) \leftarrow\operatorname{ProjectQKV}(X^{(\ell)})
3: M←NativeCausalMask⁡(n)M \leftarrow\operatorname{NativeCausalMask}(n)
4: if p∧ℓ=ℓ⋆p \land\ell=\ell^\star then
5: for j←0j \leftarrow0 to n−1n-1 do
6: M0,j←0M_{0,j}\leftarrow0
7: end for
8: end if
9: A←Softmax⁡(QKTdk+M)A\leftarrow\operatorname{Softmax}\left(\frac{QK^{\mathsf{T}}}{\sqrt{d_k}}+M\right)
10: O←AVO\leftarrow AV
11: return OO
12: end function

Table 1.

Properties. BOS Summary is training-free by construction: it introduces no additional tokens, parameters, or auxiliary modules, and the summary is derived entirely from the model’s own hidden states and frozen attention projections rather than injected from an external encoder. The intervention is also confined to prefill, it modifies one attention row at one layer, an O(n)O(n) change to a prefill pass that already costs O(n2)O(n^2) per layer, and once the BOS representation is formed during prompt encoding, decoding proceeds under standard causal attention with no further mask modification and no decoding-time overhead.

MethodMMStar Acc.MMStar Macro-F1M3CoT Acc.M3CoT Macro-F1POPE Acc.POPE Macro-F1MMBench Acc.MMBench Macro-F1Average Acc.Average Macro-F1
Qwen2.5-VL-7B-Instruct
Direct52.00±6.4052.00\pm6.4068.16±5.8768.16\pm5.8747.24±2.8247.24\pm2.8262.22±3.9762.22\pm3.9783.53±1.4083.53\pm1.4083.19±1.4883.19\pm1.4885.03±1.4285.03\pm1.4291.93±0.8791.93\pm0.8766.95±1.7866.95\pm1.7876.38±1.9276.38\pm1.92
CoT58.60±4.6158.60\pm4.6173.77±3.4373.77\pm3.4362.99±2.3362.99\pm2.3375.87±4.6175.87\pm4.6184.91±1.7084.91\pm1.7085.07±1.7585.07\pm1.7583.99±1.5783.99\pm1.5791.27±0.9191.27\pm0.9172.62±1.5472.62\pm1.5481.50±1.8181.50\pm1.81
ZeroTuning59.87±4.3659.87\pm4.3674.68±3.4474.68\pm3.4463.72±1.9863.72\pm1.9875.21±2.6775.21\pm2.6785.66±1.0085.66\pm1.0085.59±1.0385.59\pm1.0386.69±1.0086.69\pm1.0092.90±0.5892.90\pm0.5873.98±1.0973.98\pm1.0982.09±1.5182.09\pm1.51
SinkTrack59.87±2.3859.87\pm2.3874.56±2.1474.56\pm2.1462.68±2.1462.68\pm2.1474.89±3.4474.89\pm3.4486.47±0.7486.47\pm0.7486.37±0.7886.37\pm0.7886.44±0.8186.44\pm0.8192.75±0.4892.75\pm0.4873.86±0.9773.86\pm0.9782.14±1.1582.14\pm1.15
BOS Summary61.47±2.7761.47\pm2.7775.55±2.4475.55\pm2.4465.14±2.4465.14\pm2.4474.94±8.7774.94\pm8.7785.01±0.3485.01\pm0.3485.15±0.3985.15\pm0.3985.47±1.4385.47\pm1.4392.17±0.8792.17\pm0.8774.27±1.0374.27\pm1.0381.95±1.8481.95\pm1.84
Qwen3-VL-8B-Instruct
Direct48.07±5.7848.07\pm5.7864.60±5.8164.60\pm5.8131.79±3.1031.79\pm3.1046.62±2.3046.62\pm2.3089.23±1.1389.23\pm1.1389.20±1.1589.20\pm1.1584.08±1.9984.08\pm1.9991.32±1.2091.32\pm1.2063.29±1.6763.29\pm1.6772.94±1.5272.94\pm1.52
CoT66.67±3.6066.67\pm3.6079.80±2.5479.80\pm2.5465.92±1.4865.92\pm1.4877.36±3.6077.36\pm3.6088.08±0.7788.08\pm0.7788.09±0.8188.09\pm0.8188.38±1.5788.38\pm1.5793.84±0.8593.84\pm0.8577.26±1.1877.26\pm1.1884.77±1.3184.77\pm1.31
ZeroTuning64.80±3.1564.80\pm3.1578.55±2.5178.55\pm2.5159.15±3.2459.15\pm3.2471.09±5.3071.09\pm5.3088.29±0.4588.29\pm0.4588.32±0.4188.32\pm0.4187.04±0.8187.04\pm0.8193.01±0.5593.01\pm0.5574.82±1.1974.82\pm1.1982.74±0.6682.74\pm0.66
SinkTrack63.40±2.9763.40\pm2.9777.54±2.4377.54\pm2.4359.53±2.1659.53\pm2.1671.99±3.6371.99\pm3.6388.48±0.4388.48\pm0.4388.49±0.4488.49\pm0.4487.09±0.8887.09\pm0.8893.05±0.4893.05\pm0.4874.62±0.9174.62\pm0.9182.77±0.9882.77\pm0.98
BOS Summary68.07±2.2868.07\pm2.2881.05±1.5981.05\pm1.5968.08±0.7368.08\pm0.7380.02±1.1280.02\pm1.1289.29±0.5389.29\pm0.5389.21±1.1489.21\pm1.1489.05±0.8589.05\pm0.8594.19±0.4794.19\pm0.4778.62±0.7478.62\pm0.7486.12±0.8686.12\pm0.86
Gemma-3-4B-Instruct
Direct36.80±2.1436.80\pm2.1453.25±2.1053.25\pm2.1032.14±1.8732.14\pm1.8744.64±2.4344.64\pm2.4383.94±0.8483.94\pm0.8483.93±0.8583.93\pm0.8585.43±0.6985.43\pm0.6991.35±1.2091.35\pm1.2059.58±0.9459.58\pm0.9468.29±1.0868.29\pm1.08
CoT50.67±4.1750.67\pm4.1767.01±3.5567.01\pm3.5549.44±2.7549.44\pm2.7558.76±5.7958.76\pm5.7982.18±0.9882.18\pm0.9882.12±0.9782.12\pm0.9785.54±2.2385.54\pm2.2391.28±2.5491.28\pm2.5466.96±1.4266.96\pm1.4274.79±1.6774.79\pm1.67
ZeroTuning49.20±2.9949.20\pm2.9965.46±2.6465.46\pm2.6449.66±1.6749.66\pm1.6757.94±5.3957.94\pm5.3982.42±0.9882.42\pm0.9882.39±0.9982.39\pm0.9980.94±1.0780.94\pm1.0789.41±0.5989.41\pm0.5965.56±1.5765.56\pm1.5773.80±1.3373.80\pm1.33
SinkTrack48.53±5.2548.53\pm5.2564.90±4.8564.90\pm4.8550.99±2.8250.99\pm2.8261.02±4.0261.02\pm4.0282.63±0.7882.63\pm0.7882.60±0.7882.60\pm0.7881.17±1.2581.17\pm1.2589.53±0.6789.53\pm0.6765.83±1.2365.83\pm1.2374.51±1.3174.51\pm1.31
BOS Summary52.49±1.9752.49\pm1.9768.39±1.8568.39\pm1.8551.73±2.5951.73\pm2.5958.48±6.7658.48\pm6.7682.76±0.6982.76\pm0.6982.76±0.7182.76\pm0.7186.01±1.4286.01\pm1.4291.41±0.9591.41\pm0.9568.25±1.0868.25\pm1.0875.26±1.5475.26\pm1.54
Gemma-3-12B-Instruct
Direct45.60±4.0645.60\pm4.0661.69±4.0661.69\pm4.0645.38±2.9245.38\pm2.9256.17±6.4856.17\pm6.4884.57±1.1584.57\pm1.1584.55±1.1684.55\pm1.1685.44±1.4785.44\pm1.4791.37±1.8991.37\pm1.8965.25±1.4965.25\pm1.4973.44±1.7373.44\pm1.73
CoT60.00±4.7060.00\pm4.7074.68±3.9374.68\pm3.9359.62±2.9059.62\pm2.9069.56±4.2069.56\pm4.2085.14±0.5985.14\pm0.5985.13±0.6185.13\pm0.6186.00±1.5586.00\pm1.5591.74±1.7891.74\pm1.7872.69±1.3672.69\pm1.3680.28±1.5180.28\pm1.51
ZeroTuning58.40±2.0558.40\pm2.0573.08±3.0273.08\pm3.0258.37±3.9658.37\pm3.9669.70±4.9669.70\pm4.9684.69±0.2084.69\pm0.2084.76±0.2384.76\pm0.2384.50±1.3484.50\pm1.3491.58±0.7391.58\pm0.7371.49±1.1671.49\pm1.1679.78±0.2179.78\pm0.21
SinkTrack57.80±1.8557.80\pm1.8572.48±1.8972.48\pm1.8958.07±3.1058.07\pm3.1068.16±4.4468.16\pm4.4484.70±0.3684.70\pm0.3684.80±0.3684.80\pm0.3684.34±1.2384.34\pm1.2391.48±0.7091.48\pm0.7071.23±1.0271.23\pm1.0279.23±1.1779.23\pm1.17
BOS Summary62.58±1.3562.58\pm1.3576.43±0.8876.43\pm0.8861.06±1.1061.06\pm1.1072.25±4.0872.25\pm4.0884.73±1.0184.73\pm1.0184.73±1.0384.73\pm1.0387.13±0.8687.13\pm0.8691.32±0.5291.32\pm0.5273.88±0.8973.88\pm0.8981.18±1.0881.18\pm1.08

Table 1 (PDF p. 4). Performance of different inference methods across four multimodal benchmarks. Per-benchmark results are reported as mean Accuracy (%) and Macro-F1 with standard deviation. The Average columns report the arithmetic mean of the benchmark scores and the arithmetic mean of their benchmark-wise standard deviations. The best mean within each model block is shown in bold, while the second-best mean is underlined.

Experiments

Experimental Setup

Models. We evaluate BOS Summary on four instruction-tuned VLMs: Qwen2.5-VL-7B-Instruct [4], Qwen3-VL-8B-Instruct [3], Gemma-3-4B-Instruct, and Gemma-3-12B-Instruct [16]; and on two instruction-tuned LLMs: Llama-3.1-8B-Instruct [18] and Gemma-3-12B-Instruct [16]. Together, these models span multiple families, sizes, and modalities.

Benchmarks. For vision-language evaluation, we use MMBench [33] and MMStar [7] for general multimodal understanding, M3CoT [9] for multi-step multimodal reasoning, and POPE [26] for object hallucination. For text-only evaluation, we use RACE [24] for reading comprehension, LogiQA [29] for logical reasoning, MathQA [1] for mathematical reasoning, and QuAC [11] for conversational question answering.

Baselines. We compare BOS Summary against Direct prompting, Chain-of-Thought (CoT) [48], Sink-Track [32], and ZeroTuning [21]. To ensure a controlled comparison, all intervention-based methods are applied on top of the same CoT prompting setup. For SinkTrack, we follow the reported configuration and apply the intervention every five layers starting from layer 5, without additional task-specific tuning. For ZeroTuning, we sweep the attention-scaling coefficient over α∈{0.5,0.9,1.0,1.1,1.5}\alpha\in\{0.5, 0.9, 1.0, 1.1, 1.5\}. We omit Outro [43] from quantitative comparisons because its released pipeline targets audio-visual Omni-LLMs; applying it to our image-text and text-only backbones would require implementation adaptation and model-specific tuning of its rotation strength. Baseline details and the scope of this exclusion are discussed in Appendix C.

Metrics. For the multiple-choice benchmarks, we report Accuracy and Macro-F1. Accuracy is the percentage of questions answered correctly, while Macro-F1 computes F1 for each answer class and then averages across classes, reducing the effect of class imbalance. Since QuAC is an extractive question-answering benchmark, we follow its standard evaluation and report token-level F1 and exact match.

Thus, the two metric columns correspond to Accuracy and Macro-F1 for multiple-choice benchmarks, and token-level F1 and exact match for QuAC. All results are averaged over three independent runs with different random seeds and reported as mean with standard deviation. Further details are provided in Appendix § 11.

Inference protocol. Direct uses the standard task-specific prompt, whereas CoT, SinkTrack, ZeroTuning, and BOS Summary prepend the zero-shot chain-of-thought instruction “Let’s think step by step.” and enforce a fixed Answer: X output format. For the main evaluation, all methods decode with temperature T=1.0T = 1.0, and results are averaged over three random seeds. Hyperparameters are selected separately for each (model, benchmark) pair using a 10% held-out calibration split. We select the setting with the highest held-out accuracy: the intervention layer ℓ∗\ell^* for BOS Summary and the attention-scaling coefficient α\alpha for ZeroTuning. Final performance is measured on the remaining 90% of the data, which is not used during hyperparameter selection. Full layer-sweep results and the selected hyperparameters for all model–benchmark pairs are reported in Appendix § 12.

Main Results

Vision-language model results. Table 1 shows that BOS Summary achieves the highest Accuracy on all eight model–benchmark pairs on MMStar and M3CoT, exceeding the strongest baseline by 0.74–2.58 percentage points. It also leads in average Accuracy for all four backbones, with gains of 0.29–1.36 points, and in average Macro-F1 for three. The improvement extends beyond CoT prompting: BOS Summary also outperforms SinkTrack and ZeroTuning in Accuracy on both benchmarks for every model, indicating that the choice of sink intervention matters even under the same prompting protocol.

The benefits vary across tasks and metrics. Qwen3-VL leads both metrics on all four benchmarks, whereas other models show mixed results on POPE and MMBench. On M3CoT, the Accuracy gains for Qwen2.5-VL and Gemma-3-4B also do not yield the highest Macro-F1. These differences indicate that improvements in overall correctness need not extend uniformly across metrics. The consistent Accuracy gains on MMStar and M3CoT are compatible with the proposed benefit of maintaining access to prompt evidence during reasoning, while the mixed results elsewhere suggest that the benefit depends on the model and task.

Language model results. Table 2 shows that the benefit extends to text-only inputs. BOS Summary achieves the highest values in both Average columns for both models, exceeding the next-best method by 3.60 and 1.20 points on Llama-3.1-8B and by 1.44 and 1.65 points on Gemma-3-12B. It attains the highest Accuracy on RACE, LogiQA, and MathQA for both backbones. On LogiQA, Llama-3.1-8B reaches 46.34%, improving over the strongest baseline by 3.63 points. On MathQA, the Accuracy gains over the strongest sink-intervention baseline are 0.87 and 0.94 points for Llama and Gemma, respectively; the corresponding Macro-F1 gains are 1.14 and 1.83 points.

MethodRACE Acc.RACE F1LogiQA Acc.LogiQA F1MathQA Acc.MathQA F1QuAC Acc.QuAC F1Average Acc.Average F1
Llama-3.1-8B-Instruct
Direct83.31±0.8583.13±0.9542.71±5.4040.59±5.8231.76±1.0230.42±1.2051.60±1.1717.30±1.4352.35±1.5542.86±1.90
CoT78.62±1.1178.43±1.0442.24±2.5142.13±2.4661.24±1.2361.32±1.1942.69±1.9019.20±1.5756.20±1.5150.27±1.56
ZeroTuning79.53±1.4679.40±1.3541.47±4.9641.30±5.0462.58±1.8362.63±1.8143.63±1.4621.32±1.3856.80±2.0151.16±2.21
SinkTrack78.96±1.3978.78±1.3642.39±4.0642.02±3.8663.48±2.4863.57±2.3644.21±1.4221.72±1.3857.26±1.7251.52±1.78
BOS Summary84.01±0.6783.51±1.0346.34±3.1542.33±2.1764.35±1.0164.71±1.2148.73±1.9120.33±1.6660.86±1.6052.72±1.55
Gemma-3-12B-Instruct
Direct86.34±0.5986.17±0.6358.52±4.7858.02±4.6132.63±1.0931.11±0.9058.80±1.4034.92±1.6059.07±1.7652.56±1.49
CoT86.25±0.9486.03±1.1058.22±2.5157.44±2.4570.05±1.9770.35±1.9357.61±1.0032.73±1.2568.03±1.6161.64±1.68
ZeroTuning86.42±0.9186.22±1.0459.90±2.5159.34±2.9976.35±1.8176.31±1.8054.71±0.8130.08±1.0369.34±1.3462.99±1.29
SinkTrack86.34±0.9286.11±1.1059.60±1.7159.07±1.3476.15±1.4576.15±1.4154.66±0.9730.11±1.4569.19±1.1662.86±1.30
BOS Summary86.78±0.9486.22±1.0160.34±1.1159.30±3.4577.29±1.4178.14±0.9358.69±1.2934.91±1.3570.78±1.1464.64±1.44

Table 2. Performance across four language understanding and reasoning benchmarks. Per-benchmark results are reported as mean Accuracy (%) and F1 with standard deviation. The Average columns report the arithmetic mean of the benchmark scores and the arithmetic mean of their benchmark-wise standard deviations. The best mean within each model block is shown in bold, while the second-best mean is underlined.

The gains remain task-dependent. RACE improvements over the strongest baseline are smaller, and on QuAC, Direct retains the highest token-level F1 for both models. BOS Summary is close to Direct on Gemma’s QuAC scores but remains below it on Llama’s token-level F1; SinkTrack leads Llama’s exact match. Thus, the average improvements coexist with individual metrics that favor other methods. Together with the VLM results, this pattern supports the usefulness of an input-conditioned BOS across modalities, with the clearest gains on reasoning benchmarks rather than a uniform advantage across all tasks.

Analysis

We analyze the mechanism of BOS Summary through three questions:

  1. Does BOS become input-specific?

  2. Does the resulting representation alter its downstream role as an attention sink?

  3. Where is the displaced attention redistributed?

These questions trace the effect of the intervention from the BOS hidden state to the attention patterns of subsequent layers.

❶ BOS becomes input-dependent and input-specific. We ask whether BOS Summary makes BOS sensitive to the input. Let Xi,0(ℓ)X_{i,0}^{(\ell)} denote BOS’s hidden state for sample ii at layer ℓ\ell, and Xˉ0(ℓ)=1N∑i=1NXi,0(ℓ)\bar{X}_{0}^{(\ell)} = \frac{1}{N}\sum_{i=1}^{N}X_{i,0}^{(\ell)} its mean over NN samples. We measure dispersion as

D(ℓ)=1N∑i=1N∥Xi,0(ℓ)−Xˉ0(ℓ)∥2∥Xˉ0(ℓ)∥2.(5)D^{(\ell)} = \frac{\frac{1}{N}\sum_{i=1}^{N}\left\lVert X_{i,0}^{(\ell)}-\bar{X}_{0}^{(\ell)}\right\rVert_{2}}{\left\lVert\bar{X}_{0}^{(\ell)}\right\rVert_{2}}. \tag*{(5)}

Figure Figure 2(a) shows the Standard BOS is nearly input-invariant in both models. With BOS Summary, Qwen2.5-VL-7B reaches dispersion comparable to an ordinary prompt token, and Qwen3-VL-8B exceeds it, indicating strong input dependence.

Figure 2. BOS Summary makes the BOS representation input-dependent and input-specific. Results are shown for Qwen2.5-VL-7B-Instruct and Qwen3-VL-8B-Instruct on MMStar. (a) Dispersion measures how much the BOS representation varies across inputs. BOS Summary increases dispersion toward the level of ordinary prompt tokens. (b) Retrieval accuracy measures whether the BOS representation can identify its source input. BOS Summary performs well above chance, peaking at about 68% for Qwen2.5-VL-7B and 97% for Qwen3-VL-8B.

Dispersion alone does not show BOS preserves which input produced it. For sample ii with nin_i tokens, define its content signature Si(ℓ)=1ni−1∑t=1ni−1Xi,t(ℓ)S_i^{(\ell)}=\frac{1}{n_i-1}\sum_{t=1}^{n_i-1}X_{i,t}^{(\ell)}, the mean over non-BOS positions from the unmodified model. With (⋅~)(\widetilde{\cdot}) denoting mean-centering across samples, retrieval accuracy is

R(ℓ)=1N∑i=1N1[i=arg⁡max⁡i′cos⁡(X~i′,0(ℓ),S~i′(ℓ))].(6)R^{(\ell)} = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\left[i=\arg\max_{i'}\cos\left(\widetilde{X}_{i',0}^{(\ell)},\widetilde{S}_{i'}^{(\ell)}\right)\right]. \tag*{(6)}

Chance is 1/N1/N with N=500N=500. Figure 2(b) shows retrieval far above chance, peaking at 68% for Qwen2.5-VL-7B and 97% for Qwen3-VL-8B before declining at late layers, where the summary is progressively transformed for downstream use: BOS Summary makes BOS both input-dependent and input-specific.

❷ The input-specific BOS weakens sink dominance. We next examine whether the modified BOS representation changes how later layers attend to it. Figure 3(a) shows that BOS Summary reduces the attention received by BOS across all evaluated model–modality settings, although the magnitude varies by architecture. This effect is indirect: the intervention modifies only the BOS query row at one layer, thereby changing the resulting BOS hidden state. In subsequent layers, this modified state changes the key and value representations associated with BOS, which in turn affects how later tokens attend to it. Thus, a local prefill-time modification is sufficient to weaken the downstream role of BOS as an attention sink. This links a change in BOS content to a change in downstream attention allocation, even though all non-BOS query rows retain their original causal visibility.

Figure 3. BOS Summary reduces attention to the BOS sink and redistributes it toward the prompt. Results are shown for five model–modality settings using 500 samples each. (a) Attention assigned to BOS across model depth for the unmodified model (gray) and BOS Summary (blue); each panel reports the mean change. (b) Change in attention share for BOS, Context (image or passage), and Question (question and instruction tokens), measured in percentage points. All measurements use a deterministic prefill pass with eager attention.

❸ Displaced attention shifts back to the input. Finally, we examine where the attention released from BOS is redistributed. Figure 3(b) shows the destination is architecture-dependent: the Gemma VLMs route most released attention to image tokens, whereas Qwen3-VL shifts it almost entirely to the question and instruction. BOS Summary does not prescribe this destination; the differences reflect how each pretrained model reallocates attention once sink dominance is reduced. The shared effect is that attention moves to content-bearing prompt tokens rather than remaining on BOS.

Taken together, the analyses suggest two complementary ways in which BOS Summary can support access to the input. The representation probes show that BOS carries information identifying the current prompt, while the attention measurements show that subsequent layers allocate more attention to content-bearing tokens. The destination of this additional attention varies across architectures: visual context receives most of the released attention in the Gemma VLMs, whereas the question and instruction receive it in Qwen3-VL. A common mask modification can therefore interact differently with each model’s existing attention patterns while improving performance across model families.

The magnitude of the attention shift alone does not explain the gains. Llama-3.1-8B exhibits the smallest reduction in BOS attention (2 percentage points), yet improves its average score by 3.60 points over the strongest baseline. Qwen3-VL exhibits the largest reduction (52 points), with a smaller average score gain of 1.36 points. This contrast suggests that the information carried by BOS and the allocation of attention should be considered jointly when assessing a sink intervention. Enriching BOS may be useful even when

its attention share changes little, while a larger redistribution may help through renewed access to the prompt. These measurements are consistent with both pathways, but do not isolate their individual contributions to task performance.

Fine-Tuning with BOS Summary

We evaluate whether BOS Summary also benefits models during fine-tuning. We compare standard LoRA fine-tuning with LoRA fine-tuning that applies BOS Summary during both training and evaluation, using Llama-3.1-8B, Gemma-3-12B, and Gemma-3-4B on LogiQA and RACE. Within each model–dataset pair, both variants use the same training and evaluation settings, differing only in the BOS Summary intervention. All results use direct prompting and a single random seed.

Table 3 shows that BOS Summary fine-tuning outperforms standard fine-tuning across all six model–dataset pairs. The largest improvement is on Llama-3.1-8B on LogiQA, where accuracy increases from 50.54% to 53.61%, a gain of 3.07 percentage points. Both Gemma models also improve on both datasets. These results indicate that the benefit of conditioning BOS on the input extends beyond training-free inference to fine-tuning. Under matched LoRA training, the additional gains suggest that an input-conditioned BOS pathway can complement task-specific parameter adaptation. The effect appears across both model families and datasets, motivating the use of sink positions as explicit context carriers during training. This opens a direction for learning how to aggregate and reuse prompt information through existing attention pathways, while retaining the backbone architecture and standard causal decoding.

MethodLogiQALogiQARACERACE
Acc.Δ (pp)Acc.Δ (pp)
Llama-3.1-8B
Before FT42.24−83.30−
Standard FT50.54+8.3087.31+4.00
BOS Summary FT53.61+11.3788.34+5.03
Gemma-3-12B
Before FT58.53−86.34−
Standard FT60.83+2.3089.19+2.86
BOS Summary FT63.44+4.9289.88+3.54
Gemma-3-4B
Before FT44.85−73.90−
Standard FT51.77+6.9184.19+10.29
BOS Summary FT52.84+7.9985.48+11.58

Table 3. Fine-tuning accuracy (%) on LogiQA (n=651n = 651) and RACE (n=3498n = 3498). Δ\Delta denotes the change in percentage points relative to the before-FT baseline.

Qualitative Results

As shown in Figure 4, BOS Summary improves grounding at both the scene and fine-grained levels. These behaviors follow from conditioning the existing BOS attention channel on the input. Because attention to BOS remains stable during decoding, the input-specific BOS helps preserve broader scene context as the generation history grows. At the same time, aggregating information across all prompt positions allows weak or peripheral evidence to remain available, helping the model recover small or partially obscured details such as the distant person in the background.

Figure 4. Qualitative comparison on two representative examples. The CoT baseline overlooks relevant visual evidence, while BOS-Summary preserves broader scene context and attends to small, distant details, leading to correct answers.

Limitations

Although BOS Summary improves standard fine-tuning, we have not analysed how optimisation changes the BOS pathway or why this interaction yields additional gains. Understanding this interaction could guide its use during training and inform more transferable layer-selection rules, beyond the current model–task-specific calibration. Establishing the scope of these benefits also requires evaluation beyond the multiple-choice and extractive QA tasks studied here, including open-ended visual questions and unrestricted long-form generation.

Conclusion

We introduced BOS Summary, a training-free prefll-time mask edit that turns BOS into a prompt-conditioned context carrier. Across VLMs and LLMs, it improves reasoning and grounding while reshaping downstream attention, and its fine-tuning gains motivate studying the same pathway during learning. More broadly, our findings motivate research on how attention sinks can acquire, preserve, and reuse task-relevant information. Connecting their representational content to their downstream influence could guide both inference-time interventions and training objectives that use existing attention pathways to maintain access to the input during extended generation and adaptation.

LLM Usage Statement

OpenAI Codex was used during manuscript preparation to assist with drafting and revising text, reviewing related literature, checking tables against experimental results, and improving document formatting and presentation. All AI-assisted content was reviewed by the authors, who remain responsible for the accuracy of the scientific claims, references, experimental results, and final manuscript.

Detailed Related Work

Attention Sinks in Transformers and VLMs

The tendency of transformers to route a disproportionate share of attention to the first token of a sequence was formalized as the attention sink in the context of streaming language models [40]. Subsequent analysis has traced the phenomenon to massive activations in a small subset of hidden dimensions [6, 37], characterized the conditions under which it emerges during pretraining [19], linked it to outlier-driven rescaling and specialized sink heads [34], and argued that it functions as an implicit no-op that regulates information mixing across depth [5]. Analogous high-norm tokens observed in vision transformers have motivated explicit register tokens [14]. In VLMs, sinks additionally arise on weakly informative visual patches: Kang et al. [23] identify visual attention sinks and analyze which heads are image-centric, while Choi et al. [12] distinguish sinks that propagate from the vision encoder from those that emerge within the language-model layers and quantify a trade-off between global scene-level encoding and precise local evidence. Recent analysis further argues that the sink is not inert but encodes structured, global information that shapes decoding [43]. Together these findings frame the sink as a plausible intervention target for multimodal grounding.

Inference-Time Interventions on the Attention Sink

A growing line of training-free methods intervenes on the sink at inference time. One group rescales the attention received by the initial token: ZeroTuning [21] applies head-specific scaling to the attention that downstream tokens direct at the sink, and ACT [44] identifies and calibrates hidden non-beneficial sinks to reshape attention distributions. A second group changes what the sink encodes. SinkTrack [32] runs a parallel cross-attention track that injects externally derived features (e.g., a global image embedding) into the <BOS> representation, requiring auxiliary embeddings, image-position tracking, and per-layer refresh logic. LSG [12] adds a lightweight gating module that dynamically scales V-sink contributions, learned via next-token prediction with the backbone frozen. OutRo [43] lifts the causal constraint on sink positions so that the sink query attends to the full sequence, but combines this with per-layer sink detection and a gated head-output rotation applied throughout decoding, and reports on video and audiovisual question answering that the mask-relaxation component alone yields only marginal gains. Orthogonally, SinkRouter [30] treats the sink as an efficiency signal, skipping sink-dominated attention head computations to accelerate long-context decoding. Our BOS Summary sits within this family as a minimal variant: it isolates the causal-mask relaxation on <BOS> at prefill only, with no auxiliary embedding, no learned parameters, no sink detection, and no head-output rotation.

Attention and Decoding Interventions for VLM Hallucination

A broader landscape of training-free methods targets VLM hallucination by manipulating attention across content tokens or by intervening on the output distribution. Within the attention-manipulation family, PAI [31] uniformly amplifies attention on visual tokens, VAF [41] enhances visual attention within mid-layer fusion sites, PAINT [2] selectively boosts image-informed local and summary tokens, and VAR [23] redistributes attention away from visual sinks toward content-bearing patches within image-centric heads. In the LLM setting, PASTA [45] similarly reweights attention toward user-specified spans on a small subset of heads. FastV [8] approaches visual redundancy from the opposite direction by pruning visual tokens after the earliest layers. Beyond attention, OPERA [22] diagnoses over-trust on summary tokens and applies a logit penalty with beam-search rollback, VCD [25] contrasts logits against a degraded-image counterpart, and DoLa [13] contrasts logits across depth. Training-based approaches such as LRV [27] instead mitigate hallucination through robust instruction tuning, and Woodpecker [42] takes a post-hoc correction approach that revises generated outputs against evidence extracted from the image. Standard evaluation is carried out on hallucination-focused benchmarks such as POPE [26], AMBER [38], and CHAIR [35], as well as general capability benchmarks such as MMBench [33] and MME [15], with multimodal chain-of-thought reasoning as a further evaluation setting [9, 48]. In contrast to methods operating on the distribution of attention across content tokens or on the output logits, our method modifies the content of the sink itself, so that its role as a broadcast channel carries a self-derived summary of the visual context throughout generation.

Visualization of the BOS Summary Mechanism

Figure 5 illustrates the mechanism at the injection layer. In a standard decoder-only transformer, the additive causal mask restricts each query to its own position and those preceding it. For the first position this leaves a degenerate case: the BOS query has no preceding context, so its attention distribution is one-hot on itself and its output reduces to its own value vector, independent of the input. BOS Summary removes the mask entries on this single row, so that qBOSq_{\mathrm{BOS}} attends across the full set of prompt keys. Because attention is computed in parallel over positions, this requires no additional forward pass: the same prefill computation now produces a BOS output that aggregates the visual and textual prompt in proportion to the model’s own attention scores. Every other row of the mask is untouched, so all non-BOS tokens retain exactly the causal attention pattern they had before the intervention. Answer tokens are shown in the figure for completeness but are not present during prefill; they are produced during decoding, at which point no mask modification is applied and BOS is read through the frozen key and value entries already written to the cache.

Figure 5. Attention pattern under BOS Summary at the injection layer ℓ∗\ell^{*} during prefill. Each input token is projected to a query, key, and value. Under standard causal masking (gray dashed arrows), every token attends only to itself and to preceding positions, so the BOS query attends solely to its own key (red arrow). BOS Summary removes the mask on the BOS query row alone, allowing qBOSq_{\mathrm{BOS}} to additionally attend to the keys of all visual tokens v1,…,vNv_{1},\ldots,v_{N} and query tokens q1,…,qMq_{1},\ldots,q_{M} (green arrows). The resulting attention output writes a content-weighted aggregate of the full prompt into the BOS hidden state. All other query rows are unchanged, and answer tokens a1,…,aTa_{1},\ldots,a_{T} are not yet present during prefill; at decoding time no mask modification is applied.

Baselines

Direct. The model is prompted with the task input and instructed to produce the final answer immediately, without intermediate reasoning. This is the standard zero-shot protocol and establishes the backbone’s baseline capability under unmodified inference. It also serves as a reference point for benchmarks where extended reasoning is unnecessary or counterproductive, such as short yes/no probes, on which chain-of-thought prompting can degrade performance.

Chain-of-Thought (CoT). The model is prompted to produce intermediate reasoning steps before its final answer using the zero-shot instruction “Let’s think step by step.” [48] CoT is the most direct comparison for our setting: it improves multi-step reasoning purely through prompting, without modifying the model’s computation. Because BOS Summary and the remaining baselines are all applied on top of the same CoT prompt, CoT also isolates the contribution of each intervention from the effect of the prompt itself.

ZeroTuning. ZeroTuning [21] is a training-free intervention that rescales the attention that downstream tokens direct at the initial token, applying head-specific adjustments to the sink’s attention logits. It represents the family of methods that modify how much attention the sink receives while leaving its representation unchanged, making it the natural contrast to our approach of changing what the sink encodes. We sweep the scaling coefficient over α∈{0.5,0.9,1.0,1.1,1.5}\alpha\in\{0.5, 0.9, 1.0, 1.1, 1.5\} and select α\alpha on the held-out calibration split.

SinkTrack. SinkTrack [32] injects externally derived contextual features into the BOS representation through an auxiliary cross-attention track, refreshing the anchor at multiple layers during prefill. It is the closest baseline to BOS Summary in objective, both aim to make the sink carry input context, but differs in mechanism: SinkTrack supplies that context from an externally maintained embedding, whereas BOS Summary derives it within the model’s own attention. We follow the reported configuration and apply the intervention every five layers starting from layer 5, without additional task-specific tuning.

OutRo (not included in quantitative comparisons). OutRo [43] combines sink-query mask relaxation with gated rotation of non-sink attention-head outputs toward the sink value direction. Its main evaluation concerns video and audio-visual QA with Omni-LLMs. The released implementation1 integrates Qwen2.5-Omni and provides AVH-Bench and DailyOmni inference pipelines. The paper also reports visual-only VLM experiments and image–text and text-only generalisation, including Llama-3.1-8B. Its rotation strength γ\gamma is selected separately for each model; Appendices B.5 and B.7 of that work describe the sensitivity analysis and validation-based selection. This is inference-hyperparameter tuning, with no model-weight updates. A comparison on our backbones and benchmarks would require porting the released pipeline and calibrating γ\gamma under our evaluation protocol. We leave this adaptation outside the present quantitative comparison.

Metrics

This section provides further details on the metrics described in the main text. All methods use the same prompts, decoding settings, answer-extraction procedure, and scoring code. Reported differences therefore arise from the inference methods rather than from changes in evaluation.

Multiple-Choice Metrics

The multiple-choice benchmarks are M3CoT, MMStar, POPE, and MMBench for the vision-language evaluation, and RACE, LogiQA, and MathQA for the text-only evaluation. We report Accuracy and Macro-F1 for these benchmarks.

Let {(y^i,yi)}i=1N\{(\hat{y}_i, y_i)\}_{i=1}^{N} denote the predicted and gold labels for NN questions. Accuracy is the fraction of predictions that match the corresponding gold labels:

Acc=1N∑i=1N1[y^i=yi].(7)\mathrm{Acc} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}[\hat{y}_i = y_i]. \tag*{(7)}

For Macro-F1, each answer option is treated as a separate class. For class c∈Cc \in\mathcal{C}, let TPc\mathrm{TP}_c, FPc\mathrm{FP}_c, and FNc\mathrm{FN}_c denote the numbers of true positives, false positives, and false negatives. Per-class precision, recall, and F1 are defined as

Pc=TPcTPc+FPc,Rc=TPcTPc+FNc,Fc=2PcRcPc+Rc.(8)\begin{aligned} P_c &= \frac{\mathrm{TP}_c}{\mathrm{TP}_c + \mathrm{FP}_c}, \\ R_c &= \frac{\mathrm{TP}_c}{\mathrm{TP}_c + \mathrm{FN}_c}, \\ F_c &= \frac{2P_cR_c}{P_c + R_c}. \tag*{(8)} \end{aligned}

We set Fc=0F_c = 0 when Pc+Rc=0P_c + R_c = 0. Macro-F1 is the unweighted mean of the per-class F1 scores:

Macro-F1=1∣C∣∑c∈CFc.(9)\mathrm{Macro\text{-}F1} = \frac{1}{|\mathcal{C}|} \sum_{c \in\mathcal{C}} F_c. \tag*{(9)}

Because all classes contribute equally, Macro-F1 is less affected by dominant classes than Accuracy. This distinction is especially relevant for POPE, which is evaluated as a binary yes/no classification task.

QuAC Metrics

QuAC is the only extractive question-answering benchmark in our evaluation. We follow its standard QuAC/SQuAD-style evaluation and report token-level F1 and exact match. Predictions and references are normalized by converting text to lowercase, removing punctuation and the articles a, an, and the, and collapsing repeated whitespace. Token-level F1 is then computed from the overlap between the normalized prediction and reference tokens. Let num_same\mathrm{num\_same} denote the number of shared tokens. Precision, recall, and F1 are

P=num_same∣prediction tokens∣,R=num_same∣reference tokens∣,F1=2PRP+R.(10)\begin{aligned} P &= \frac{\mathrm{num\_same}}{|\mathrm{prediction\ tokens}|}, \\ R &= \frac{\mathrm{num\_same}}{|\mathrm{reference\ tokens}|}, \\ F_1 &= \frac{2PR}{P + R}. \tag*{(10)} \end{aligned}

We set F1=0F_1 = 0 when the prediction and reference have no token overlap. Exact match is 1 when their normalized strings are identical and 0 otherwise. QuAC provides multiple reference answers for each question. We therefore take the maximum score over all valid references. For unanswerable questions, we follow QuAC’s majority rule. When at least half of the references are CANNOTANSWER, the question is treated as unanswerable and the model must also abstain to receive credit. Otherwise, the CANNOTANSWER references are removed before scoring.

For CoT outputs, the predicted span is taken from the text following Answer:. Any generated continuation into the next dialogue turn is removed, as are stray literal spanmarkup tags.

Aggregation Across Seeds

Each reported result is averaged over three independent runs with different random seeds. We report the mean and standard deviation across these runs. All methods use the same test splits and evaluation code, so variation across seeds reflects run-level differences rather than changes in scoring.

Complete Results and Hyperparameters

We report the complete layer-wise results for every model–benchmark pair.

Hyperparameters

Models and benchmarks. The vision-language experiments use Qwen2.5-VL-7B, Qwen3-VL-8B, Gemma-3-4B, and Gemma-3-12B. The text experiments use Llama-3.1-8B and Gemma-3-12B. All models are used off the shelf, and all inference-time interventions are training-free. The vision-language benchmarks are M3CoT, MMStar, POPE, and MMBench. The text benchmarks are RACE, LogiQA, MathQA, and QuAC. Accuracy and Macro-F1 are reported for the multiple-choice tasks, including the binary yes/no POPE benchmark. QuAC is evaluated using token-level F1 and exact match, as defined in Appendix § 11. Every result is reported as the mean and standard deviation over three random seeds.

Compared methods. All methods use the same task prompts, decoding settings, answer-extraction procedure, and scoring code within each model–benchmark setting. Direct answers without producing an explicit reasoning trace. Chain-of-Thought (CoT) instructs the model to reason before returning its prediction in the fixed format Answer: X.

SinkTrack intervenes at every fifth layer, namely layers 5, 10, 15, 20, and 25 when those layers are present. At each selected layer, the first token replaces its standard self-attention update with cross-attention to a global representation of the input. This representation consists of the image-token states for vision-language models and the prompt hidden states for text models. The input representation is read again at each intervention layer without positional encoding. The intervention is applied only during prefill and always targets the first token.

ZeroTuning scales the first token’s post-softmax attention weight by a single factor α\alpha across all layers, heads, and decoding steps, followed by renormalization. The identity setting is α=1\alpha= 1. We sweep

α∈{0.5,0.9,1.0,1.1,1.5}\alpha\in\{0.5, 0.9, 1.0, 1.1, 1.5\}

and report the best value separately for each model–dataset pair.

BOS-Summary opens the first token’s attention row over all real input tokens at the selected injection layer during prefill. After this update, decoding follows the model’s ordinary causal attention pattern. We report two variants. The main variant uses one injection layer selected separately for each model–dataset pair by sweeping all model layers. The inject-all variant applies the same intervention at every layer during prefill. No model weights are changed in either setting.

Decoding settings. Both evaluation tracks use sampling with do_sample=True and temperature T=1.0T = 1.0. For vision-language models, generation is limited to 1024 new tokens for CoT prompting and 64 for Direct prompting. For text models, CoT generation is limited to 1024 new tokens. Direct evaluation on the multiple-choice text benchmarks uses a single constrained forward pass rather than free-form generation: the next-token distribution is restricted to the candidate labels, and the highest-scoring label is selected, following the standard MMLU/lm-eval-harness protocol. Direct generation on QuAC is limited to 48 new tokens.

Fine-tuning study. The supplementary fine-tuning experiment trains Llama-3.1-8B on the LogiQA training split using LoRA. The LoRA rank is r=32r = 32, with scaling parameter α=2r=64\alpha= 2r = 64 and dropout 0.05. LoRA adapters are applied to the query, key, value, and output projections, as well as the gate, up, and down projections. Training runs for four epochs with a learning rate of 10−410^{-4}, a cosine schedule, and a warmup ratio of 0.03. The per-device batch size is 4, with gradient accumulation used to obtain the effective batch size, and training uses bfloаt16. All examples use the direct-answer prompt format.

Two training arms use identical settings, except that BOS-Summary is active during training in one arm. The study reports three evaluation conditions: standard fine-tuning, BOS-Summary fine-tuning evaluated with the intervention enabled, and the same BOS-Summary-trained weights evaluated with the intervention disabled. Each condition is run with three seeds and reported as mean performance with standard deviation.

Figure 6. VLM layer sweep results

Figure 7. LLM layer sweep results

Complete Results

Figures 6 and 7 report the BOS-Summary layer sweeps for the vision-language and text models, respectively. Each curve shows Accuracy as a function of the injection layer for a fixed model–dataset pair; QuAC uses token-level F1 in place of Accuracy.

Across models and datasets, performance remains nearly flat over much of the network depth. BOS-Summary is therefore largely insensitive to the exact injection layer, even though the best layer is selected separately for the main results. This stability is consistent with the first token acting as a stable input-summary position throughout the network: once its attention row is opened to the prompt, similar information can be collected across a broad range of layers. The inject-all results provide an additional reference for whether repeated intervention offers benefits beyond a single well-placed update. Together, the complete tables and layer sweeps show that the reported gains do not depend on a narrow or fragile layer choice.

Dataset Prompt Templates

This section reports the prompts used for Direct and chain-of-thought (CoT) inference. Placeholders enclosed in angle brackets are replaced with fields from each example. Vision–language inputs are passed through each model’s chat template as a single user message containing the image and prompt text. Text-only inputs are provided as raw completion prompts.

Vision–Language Datasets

All vision–language benchmarks use the following shared templates.

Vision–Language — Direct

Question: <task input>

Output Format:
Give the final answer directly using the
    exact format:
**Answer: <final answer>**

Vision–Language — CoT

Question: <task input>

Let’s think step by step!

Output Format:
After the reasoning, give the final answer
    using the exact format:
**Answer: <final answer>**

The contents of <task input> depend on the benchmark. For M3CoT, the question is followed by its letter-labeled answer choices. MMBench uses the same format, with an optional hint placed before the question. MMStar stores its answer choices directly in the question field and therefore requires no separate option block. POPE contains a binary question without an explicit choice list, and the final answer is either Yes or No.

For all four datasets, the image and text prompt are provided in one user message of the form [{type:image}, {type:text, text:<prompt>}]. Model-specific special tokens are inserted by the corresponding processor’s chat template.

Text-Only Datasets

RACE, LogiQA, and MathQA use a common multiple-choice structure:

Text Multiple Choice — Direct

<task instruction>
<context field>
Question: <question>
<letter-labeled choices>
Answer:

Text Multiple Choice — CoT

<task instruction>
<context field>
Question: <question>
<letter-labeled choices>
Let’s think step by step. Briefly explain
    your reasoning, then on a new line
give the final answer in the exact format
    ’Answer: X’.

For RACE, the context field is introduced as Article:; for LogiQA, it is introduced as Passage:; and for MathQA, the input begins with Problem: and does not contain a separate passage. RACE and LogiQA use options A–D, while MathQA uses options A–E.

QuAC uses a separate extractive question-answering template:

QuAC — Direct

Answer the question using a span from the
    passage. If the passage does not
contain an answer, reply exactly
    CANNOTANSWER.

Passage: <passage>
Conversation so far:
<previous question--answer pairs>
Question: <current question>
Answer:

QuAC — CoT

Answer the question using a span from the
    passage. If the passage does not
contain an answer, reply exactly
    CANNOTANSWER.

Passage: <passage>
Conversation so far:
<previous question--answer pairs>
Question: <current question>
Let’s think step by step. Briefly explain
    your reasoning, then write
’Answer:’ followed by a span copied from
    the passage, or
’Answer: CANNOTANSWER’.

The conversation-history block is omitted for the first turn of each QuAC dialogue. For later turns, it contains all preceding question–answer pairs in chronological order.

References

  1. [1]Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannan Hejazi. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. In NAACL-HLT, pages 2357–2367, 2019.arxiv.org/abs/1905.13319
  2. [2]Kazi Hasan Ibn Arif, Sajib Acharjee Dip, Khizar Hussain, Lang Zhang, and Chris Thomas. Paint: Paying attention to informed tokens to mitigate hallucination in large vision-language model. arXiv preprint arXiv:2501.12206, 2025.arxiv.org/abs/2501.12206
  3. [3]Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631, 2025.arxiv.org/abs/2511.21631
  4. [4]Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-VL technical report. arXiv preprint arXiv:2502.13923, 2025.arxiv.org/abs/2502.13923
  5. [5]Federico Barbero, Álvaro Arroyo, Xiangming Gu, Christos Perivolaropoulos, Michael Bronstein, Petar Velicković, and Razvan Pascanu. Why do LLMs attend to the first token? In Conference on Language Modeling (COLM), 2025.arxiv.org/abs/2504.02732
  6. [6]Nicola Cancedda. Spectral filters, dark signals, and attention sinks. In ACL (Long Papers), pages 4792–4808, 2024.
  7. [7]Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaq i Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? Advances in Neural Information Processing Systems, 37:27056–27087, 2024.arxiv.org/abs/2403.20330
  8. [8]Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In European Conference on Computer Vision, pages 19–35. Springer, 2024.arxiv.org/abs/2403.06764
  9. [9]Qigong Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che. M3Cot: A novel benchmark for multi-domain multi-step multi-modal chain-of-thought. In ACL, pages 8199–8221, 2024.DOI
  10. [10]Zhaor uen Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. Halc: Object hallucination reduction via adaptive focal-contrast decoding. arXiv preprint arXiv:2403.00425, 2024.arxiv.org/abs/2403.00425
  11. [11]Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. QuAC: Question answering in context. In EMNLP, pages 2174–2184, 2018. 4
  12. [12]Jiho Choi, Jaemin Kim, Sanghwan Kim, Seunghoon Hong, and Jin-Hwi Park. When sinks help or hurt: Unified framework for attention sink in large vision-language models. arXiv preprint arXiv:2604.03316, 2026. 2, 11arxiv.org/abs/2604.03316
  13. [13]Yung-Sung Chung, Yujia Xie, Hongyin Luo, Yoon Kim, James R Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models. In International Conference on Learning Representations, pages 54158–54183, 2024. 1, 2, 11
  14. [14]Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In International Conference on Learning Representations (ICLR), 2024. 1, 2, 11arxiv.org/abs/2309.16588
  15. [15]Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. Advances in Neural Information Processing Systems, 38, 2026. 11
  16. [16]Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, et al. Gemma 3 technical report, 2025. 4
  17. [17]Xuan Gong, Tianshi Ming, Xinpeng Wang, and Zhihua Wei. Damro: Dive into the attention mechanism of lvlm to reduce object hallucination. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7696–7712, 2024. 2arxiv.org/abs/2410.04514
  18. [18]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. 4arxiv.org/abs/2407.21783
  19. [19]Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view. In International Conference on Learning Representations, pages 97114–97144, 2025. 1, 2, 11arxiv.org/abs/2410.10781
  20. [20]Tianyu Guo, Druv Pai, Yu Bai, Jiantao Jiao, Michael I. Jordan, and Song Mei. Active-dormant attention heads: Mechanistically demystifying extreme-token phenomena in LLMs. arXiv preprint arXiv:2410.13835, 2024. 2arxiv.org/abs/2410.13835
  21. [21]Feijiang Han, Xiaodong Yu, Jianheng Tang, Delip Rao, Weihua Du, and Lyle Ungar. Zerotuning: Unlocking the initial token’s power to enhance large language models without training. arXiv preprint arXiv:2505.11739, 2025. 1, 2, 4, 11, 12arxiv.org/abs/2505.11739
  22. [22]Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multimodal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13418–13427, 2024. 1, 2, 11DOI
  23. [23]Seil Kang, Jinyeong Kim, Junhyeok Kim, and Seong Jae Hwang. See what you are told: Visual attention sink in large Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. In EMNLP, pages 785–794, 2017.
  24. [25]Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13872–13882, 2024.arxiv.org/abs/2311.16922
  25. [26]Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 conference on empirical methods in natural language processing, pages 292–305, 2023.arxiv.org/abs/2305.10355
  26. [27]Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning. In International Conference on Learning Representations, pages 57689–57733, 2024.arxiv.org/abs/2306.14565
  27. [28]Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
  28. [29]Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. In IJCAI, pages 3622–3628, 2021.arxiv.org/abs/2007.08124
  29. [30]Junnan Liu, Xinyan Liu, Peifeng Gao, Zhaobo Qi, Beichen Zhang, Weigang Zhang, and Antoni Bert Chen. Sinkrouter: Sink-aware routing for efficient long-context decoding in large language and multimodal models. arXiv preprint arXiv:2604.16883, 2026.arxiv.org/abs/2604.16883
  30. [31]Shi Liu, Kecheng Zheng, and Wei Chen. Paying more attention to image: A training-free method for alleviating hallucination in lvmls. In European Conference on Computer Vision, pages 125–140. Springer, 2024.arxiv.org/abs/2407.21771
  31. [32]Xu Liu, Guikun Chen, and Wenguan Wang. Sinktrack: Attention sink based context anchoring for large language models. arXiv preprint arXiv:2604.10027, 2026.
  32. [33]Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216–233. Springer, 2024.arxiv.org/abs/2307.06281
  33. [34]Zihan Qiu, Zeyu Huang, Kaiyue Wen, Peng Jin, Bo Zheng, Yuxin Zhou, Haofeng Huang, Zekun Wang, Xiao Li, Huaqing Zhang, et al. A unified view of attention and residual sinks: Outlier-driven rescaling is essential for transformer training. arXiv preprint arXiv:2601.22966, 2026.arxiv.org/abs/2601.22966
  34. [35]Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. In EMNLP, pages 4035–4045, 2018.arxiv.org/abs/1809.02156
  35. [36]Valeria Ruscio, Umberto Nanni, and Fabrizio Silvestri. What are you sinking? a geometric approach on attention sink multimodal models. arXiv preprint arXiv:2503.03321, 2025.
  36. [37]Mingjie Sun, Xinlei Chen, J. Zico Kolter, and Zhuang Liu. Massive activations in large language models. In First Conference on Language Modeling (COLM), 2024.arxiv.org/abs/2402.17762
  37. [38]Junyang Wang, Yuhan Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Jiaqi Wang, Haiyang Xu, Ming Yan, Ji Zhang, et al. Amber: An llm-free multi-dimensional benchmark for llms hallucination evaluation. arXiv preprint arXiv:2311.07397, 2023.arxiv.org/abs/2311.07397
  38. [39]Xintong Wang, Jingheng Pan, Liang Ding, and Chris Biemann. Mitigating hallucinations in large vision-language models with instruction contrastive decoding. In Findings of the Association for Computational Linguistics: ACL 2024, pages 15840–15853, 2024.arxiv.org/abs/2403.18715
  39. [40]Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2309.17453
  40. [41]Hao Yin, Guangzong Si, and Zilei Wang. Clearsight: Visual signal enhancement for object hallucination mitigation in multimodal large language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 14625–14634, 2025.arxiv.org/abs/2503.13107
  41. [42]Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Shi, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large language models. Science China Information Sciences, 67(12):220105, 2024.DOI
  42. [43]Suho Yoo, Youngjoon Jang, and Joon Son Chung. On the nature of attention sink that shapes decoding strategy in omnillms, 2026.arxiv.org/abs/2603.14337
  43. [44]Zhongzhi Yu, Zheng Wang, Yonggan Fu, Huihong Shi, Khalid Shaikh, and Yingyan Celine Lin. Unveiling and harnessing hidden attention sinks: Enhancing large language models without training through attention calibration. arXiv preprint arXiv:2406.15765, 2024.
  44. [45]Qingru Zhang, Chandan Singh, Liyuan Liu, Xiaodong Liu, Bin Yu, Jianfeng Gao, and Tuo Zhao. Tell your model where to attend: Post-hoc attention steering for llms. arXiv preprint arXiv:2311.02262, 2023.arxiv.org/abs/2311.02262
  45. [46]Stephen Zhang, Mustafa Khan, and Vardan Papyan. Attention sinks: A ‘catch, tag, release’ mechanism for embeddings. Advances in Neural Information Processing Systems, 38:83140–83181, 2026.arxiv.org/abs/2502.00919
  46. [47]Xiaofeng Zhang, Yuanchao Zhu, Chaochen Gu, Jiawei Cao, Hao Cheng, and Kaijie Wu. What drives attention sinks? a study of massive activations and rotational positional encoding in large vision–language models. Information Processing & Management, 63(2):104431, 2026.
  47. [48]Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.00923, 2023.arxiv.org/abs/2302.00923

Paper details

vlnc:W_zf8uuyg5
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Codex

View statement
Looking for related papers…
Contents