betaFeedback

WorksComputer Science

Entropogram-guided Visual Realignment Decoding: Capturing the Rhythm of Multimodal Reasoning

Abstract

Multimodal large language models (MLLMs) have demonstrated remarkable progress on cross-modal reasoning tasks, yet they continue to suffer from hallucinations in long-chain reasoning processes. In this work, we revisit hallucination in MLLMs from the perspective of reasoning generation dynamics. We argue that entropy is not merely a static measure of uncertainty but a structured signal that reflects how modality information is integrated throughout decoding. Through systematic analysis, we identify two characteristic entropy patterns associated with hallucination behaviors: (1) a generally high-confidence regime corresponds to premature convergence to inappropriate multimodal interactions; and (2) local abrupt entropy transitions are associated with reasoning instability and hallucination onset. Building on these observations, we propose an entropy-aware self-introspective framework for MLLM reasoning. Our method continuously tracks entropy dynamics during decoding and performs targeted visual re-alignment when suspicious hallucination states are detected, encouraging the model to re-examine salient visual evidence at critical reasoning stages. Extensive experiments across six multimodal reasoning benchmarks spanning nine distinct evaluation settings demonstrate that our approach delivers robust improvements in reasoning accuracy and hallucination robustness, while providing new insights into the generation dynamics underlying MLLM failures. The code is available at: https://github.com/jden4524/SelfRetrospectiveDecoding.

Introduction

Multimodal large language models (MLLMs) have achieved significant progress on cross-modal reasoning tasks ([31, 3] (Zhu et al. 2025; Bai et al. 2025)), yet they continue to suffer from hallucinated outputs that are unfaithful to the provided visual context ([18, 16, 13, 21]). The issue becomes particularly pronounced in long-horizon reasoning processes such as Chain-of-Thought (CoT), where errors at intermediate steps can propagate and compound.

Existing efforts largely attribute such failures to insufficient reasoning or weak representations, and therefore attempt to mitigate them through stronger supervision signals such as visual-reward reinforcement learning ([20, 27, 23, 4]). While effective, these approaches necessitate expensive training. Consequently, recent studies also explore training-free interventions, including contrastive decoding and reasoning-aware decoding strategies ([16] (Xu et al. 2025; Xu et al. 2026)), which improve generation by identifying hallucination-associated tokens and applying interventions to suppress their impact. Despite their effectiveness, these methods predominantly address what tokens are hallucinated, but do not investigate how multimodal interactions shape and steer decoding trajectories, which can ultimately lead to hallucinations.

In this work, we revisit MLLM hallucination from the perspective of entropy-driven generation dynamics. While recent studies have begun leveraging entropy for decoding in both unimodal (He et al., 2026; Sharma & Lossfunk, 2025; Wang et al., 2025; Cheng et al., 2026; Zhu et al., 2026) and multimodal (Xu et al., 2026; Zou et al., 2025) settings, they primarily treat entropy as a token-level confidence signal. We instead argue that entropy encodes richer, trajectory-level information about how modality information is integrated over time. Through systematic analysis, we identify two recurring entropy patterns that statistically distinguish correct from hallucinated generation: (1) prolonged low-entropy (high-confidence) regimes, which counterintuitively correspond to ineffective multimodal consolidation and premature trajectories; and (2) abrupt entropy transitions associated with local instability and hallucination onset (see Figure 1–Figure 2 and Table 1). These findings indicate that hallucination is closely associated with dynamic failures in cross-modal interaction over time, beyond static reasoning difficulty alone.

Comparison of generation dynamics between correct and incorrect multimodal reasoning trajectories

Figure 1. Comparison of generation dynamics between correct and incorrect multimodal reasoning trajectories. (a) Average reasoning entropy across the generation process over samples, where correct predictions consistently maintain higher entropy than incorrect ones. (b) Evolution of visual attention during CoT generation, showing that correct reasoning trajectories preserve stronger visual grounding at the early stage of reasoning, while incorrect trajectories exhibit progressively weakened but still higher attention to visual evidence. Shaded regions denote standard deviations.

Entropy transitions and visual attention patterns during hallucination emergence

Figure 2. A case study illustrating hallucination emergence during MLLM reasoning. The left panel shows entropy over generated tokens, while the right panel visualizes the corresponding visual attention patterns at selected reasoning steps (P5–P8). The gray region denotes the span of incorrect generation. Hallucinated reasoning coincides with sharp entropy spikes (P8) and progressively strong but diffused visual attention. More examples can be found in Appendix § 9.

ΔH\Delta H@onsetLow-entropy durationSpike frequencyMax ΔH\Delta H
Correct (n=465n=465)0.01217.60.0851.762
Hallucinated (n=763n=763)0.24521.40.1011.974
pp-value4×10−44 \times10^{-4}<10−35< 10^{-35}<10−9< 10^{-9}<10−16< 10^{-16}

Table 1. Population-level entropy statistics on 1,228 VQA trajectories (Qwen3-VL-8B). ΔH\Delta H@onset is the mean entropy change at the aligned hallucination onset (p=0p=0); the remaining columns are per-trajectory statistics. pp-values are from two-sided Mann–Whitney U tests.

Building on this insight, we propose SEED (Self-introspective Entropy-aware visual rE-alignment Decoding), an entropy-aware framework for MLLM reasoning. SEED continuously monitors the entropy dynamics of the reasoning trajectory during decoding; when the entropy signal indicates that generation is entering a hallucination-prone regime, SEED performs targeted re-alignment of visual attention, effectively correcting over-reliance on language priors at critical moments. Unlike prior approaches that intervene uniformly or rely on fixed heuristics, SEED adapts its intervention to the evolving state of cross-modal grounding. In summary, our contributions are threefold:

  • We provide a systematic empirical analysis showing that hallucinations in MLLMs are consistently associated, at the population level, with two characteristic entropy patterns: sustained low-entropy convergence and abrupt entropy transitions, indicating an association with disrupted modality interaction dynamics beyond token-level uncertainty alone.

  • We propose SEED, a training-free, entropy-aware decoding framework that monitors generation dynamics in real time and performs targeted visual re-alignment when hallucination-prone states are detected, directly addressing the identified failure modes.

  • We conduct extensive experiments across multiple base MLLMs and diverse benchmarks. Results demonstrate that SEED delivers broad and substantial improvements in reasoning accuracy and hallucination robustness over strong baselines, introducing less than 5% computational overhead.

The Pattern of Entropy Dynamics in MLLM Reasoning

To understand the intrinsic mechanism underlying MLLM reasoning, we use information entropy to characterize the decoding process. While entropy is commonly interpreted as a measure of prediction uncertainty or LLMs’ confidence, we extend this perspective by asking whether entropy trajectories can reveal how modalities interact during generation. Specifically, we employ Qwen3-VL-8B-Instruct (Bai et al., 2025) to conduct CoT reasoning on the M3CoT (Chen et al., 2024) benchmark. During reasoning, the entropy at each decoding step tt is tracked and calculated as Ht=−∑w∈Vp(w∣x<t)log⁡p(w∣x<t)H_t = -\sum_{w\in\mathcal{V}} p(w \mid x_{<t}) \log p(w \mid x_{<t}), where x<tx_{<t} denotes the preceding token sequence and V\mathcal{V} the vocabulary.

Higher controlled uncertainty supports more effective MLLM reasoning. The first observation arises from 2,000 randomly selected Visual Question Answering (VQA) instances, stratified into correctly and incorrectly solved ones. As shown in Figure 1(a), we observe a consistent entropy-level separation: correctly answered samples maintain persistently higher entropy throughout the reasoning trajectory compared to incorrect ones. Interestingly, this trend remains stable across all decoding stages, suggesting that successful reasoning is not associated with maximal confidence, but rather with sustained uncertainty during generation.

We further analyze the evolution of visual attention during reasoning. Figure 1(b) shows that correctly answered samples initially allocate more attention to visual tokens, which then quickly decays and stabilizes at a relatively low level. In contrast, incorrect cases exhibit a slower decay of visual attention, maintaining higher visual attention in later stages of generation. These two findings are temporally coupled and suggest a strong connection between entropy dynamics and modality interaction: higher-entropy trajectories correspond to faster convergence of visual attention, while lower-entropy trajectories correspond to prolonged but ineffective visual dependency. This alignment suggests that entropy not only reflects uncertainty in reasoning generation, but also implicitly captures how rapidly and effectively visual information is integrated into the reasoning process.

Localized entropy spikes mark hallucination onset. To contextualize how hallucinations emerge during generation, we first conduct an instance-level trajectory analysis. One example is illustrated in Figure 2, where the “tidy room” has been falsely interpreted as “the room is a mess”. The hallucination emerges progressively over the course of the reasoning trajectory. Prior to the exact hallucinated content, the entropy curve exhibits multiple sharp spikes (P2, P4, P6, and P8), which correspond to the largest-magnitude transitions observed across the entire decoding process. Meanwhile, the model gradually loses concentration on relevant visual evidence and begins generating unsupported claims. Notably, the hallucinated reasoning span coincides with the unstable entropy transitions and the attention diffusion, suggesting a localized breakdown of multimodal coordination during reasoning.

Population-level evidence. To verify that this pattern extends beyond individual cases, we analyze 1,228 randomly sampled VQA trajectories, including 763 hallucinated responses and 465 correct controls. For each hallucinated response, we align the first hallucinated token to position p=0p=0 and compute the token-to-token entropy change ΔHt=Ht−Ht−1\Delta H_t = H_t - H_{t-1} within a ±20\pm20-token window; for each correct response, p=0p=0 is sampled uniformly at random. At hallucination onset, the mean entropy jump is more than 20× larger than in correct trajectories, while differences before the onset are substantially smaller (Table 1; full profile in Appendix § 9.§ 3). Across the complete trajectories, we further measure the longest low-entropy run, the fraction of steps with ΔHt>τrise\Delta H_t > \tau_{\mathrm{rise}}, and the maximum entropy change. Hallucinated responses show higher values on all three measures, with significantly longer low-entropy runs, more frequent spikes, and larger maximum changes (two-sided Mann–Whitney U tests, all p<10−3p < 10^{-3}). These consistent associations motivate SEED’s use of entropy dynamics as a proxy signal for hallucination-prone states.

✎ To summarize, the two findings suggest that entropy trajectories encode substantially richer information than prediction uncertainty alone. Beyond serving as a confidence signal, entropy appears to implicitly reflect the evolving state of multimodal information consolidation throughout reasoning. Existing entropy-based decoding approaches largely inherit assumptions from unimodal LLMs and therefore interpret entropy primarily as a token-level uncertainty indicator. Our observations instead suggest that, in MLLMs, entropy also captures the stability of cross-modal grounding dynamics. We therefore hypothesize that effective multimodal reasoning requires maintaining a controlled entropy regime that enables stable multimodal information consolidation throughout generation.

Literature Review

Multimodal Large Language Models

Recent MLLMs have made substantial progress in multimodal reasoning, which requires both integrating visual perception with language understanding and reasoning over multimodal clues (Wu et al., 2024; Deitke et al., 2024; Zhu et al., 2025; Bai et al., 2025). A typical MLLM architecture consists of a modality alignment module (e.g., Q-Former [17]) bridging visual encoders (e.g., ViT-L/14 [6]) and LLMs (e.g., Flan-T5 [24], GPT-3 (Brown et al., 2020)).

To further enhance reasoning capabilities, Chain-of-Thought (CoT) prompting (Zheng et al., 2023; Gao et al., 2025; Zhang et al., 2024) and supervised fine-tuning (SFT) (Dai et al., 2023; Liu et al., 2023; Bai et al., 2025) have been widely adopted to refine step-by-step reasoning behaviors. More recently, reinforcement learning, such as Group Relative Policy Optimization (GRPO), and process-level supervision have been introduced to improve reasoning faithfulness and long-horizon planning abilities (Zhou et al., 2025a; DeepSeek-AI et al., 2026; Deng et al., 2026b; Song et al., 2025; Xing et al., 2026).

Mitigating Hallucinations in MLLM Reasoning

Hallucinations in MLLMs refer to outputs that are either inconsistent with the multimodal context or entirely ungrounded, including incorrect object attribution, fabricated attributes, and irrelevant reasoning ([18, 21]). Unlike hallucinations in language models, multimodal hallucinations arise from the intricate coupling between visual perception and language generation, making their causal factors harder to disentangle and mitigate.

In the context of large language models, a substantial body of work has studied hallucination mitigation through inference-time decoding strategies ([12, 25, 29, 1] (Zhu et al. 2026)). These approaches leverage internal confidence signals, such as entropy or token probabilities, to detect and correct unreliable generations. For example, Wang et al. [29] find that only a small subset of generated tokens exhibit high entropy and that these tokens correlate with critical reasoning transitions, motivating entropy-aware interventions that selectively regulate uncertain decoding steps.

Motivated by their efficiency, recent studies have extended similar ideas to MLLM hallucinations, developing training-free decoding interventions that adjust generation ([16, 14] (Xu et al. 2025; Xu et al. 2026)). Representative approaches include contrastive decoding and reasoning-aware decoding, which aim to reduce hallucinations by suppressing hallucination-prone tokens or re-weighting generation based on uncertainty estimates ([16] (Xu et al. 2025; Xu et al. 2026)). However, these methods largely inherit assumptions from unimodal settings and thus treat hallucination as a token-level or uncertainty-driven phenomenon.

Methodology

This section presents SEED (Self-introspective Entropy-aware Visual Re-alignment Decoding), illustrated in Figure 3. SEED is a training-free framework for improving MLLM reasoning that leverages entropy as a lens for monitoring the reasoning trajectory. We introduce a self-introspective mechanism to dynamically re-align visual attention throughout the reasoning trajectory.

Figure 3. Framework overview of the proposed SEED. The figure presents three parts: (a) A standard MLLM architecture processes system instructions and visual tokens through stacked decoder layers to perform multimodal reasoning. (b) A self-introspective mechanism monitors token-level entropy dynamics throughout the decoding process, capturing locally important spikes that signal shifts in reasoning dynamics. (c) When both detection conditions are met, our method performs an attention intervention during decoding to rectify the multimodal interaction. We refer to this process as Visual Attention Re-alignment Decoding.

Problem Formulation

Given a multimodal large language model (MLLM) Mθ\mathcal{M}_\theta parameterized by θ\theta, we consider it as a conditional sequence generator that produces an output token sequence based on both textual and visual inputs. For an image-grounded reasoning task, the input consists of a textual query xx and an image vv. These inputs are jointly encoded into a unified token sequence, which conditions the autoregressive decoding process. The model then generates an output sequence y=(y1,y2,…,yT)y = (y_1, y_2, \ldots, y_T) token by token, following the distribution

yt∼pθ(⋅∣x,v,y<t)∝softmax⁡(fθ(⋅∣x,v,y<t)),(1)y_t \sim p_\theta(\cdot\mid x,v,y_{<t}) \propto\operatorname{softmax}(f_\theta(\cdot\mid x,v,y_{<t})), \tag*{(1)}

where y<ty_{<t} denotes the previously generated tokens, and fθf_\theta represents the unnormalized logits over the vocabulary. Under this formulation, multimodal reasoning can be viewed as a sequential decision process over tokens. Hallucination occurs when the generated sequence yy deviates from the visual context in vv. Rather than treating hallucination as an isolated prediction error, we study it as a dynamic phenomenon emerging along the generation trajectory. To this end, the token-level entropy is recorded to characterize the model’s decoding behavior at each step:

Ht=−∑w∈Vpθ(w∣x,v,y<t)log⁡pθ(w∣x,v,y<t).(2)H_t=-\sum_{w\in\mathcal{V}}p_\theta(w\mid x,v,y_{<t})\log p_\theta(w\mid x,v,y_{<t}). \tag*{(2)}

Self-introspective Entropy Dynamics Awareness

Given the entropy sequence {Ht}\{H_t\} obtained from Eq. (2), we maintain an online estimate of the entropy trajectory during decoding. Motivated by the association between hallucination onset and characteristic entropy transitions, we identify two criteria that jointly characterize unstable reasoning states. A multimodal reasoning trajectory is intervened upon when it satisfies both:

(1) Relative Low-entropy Regime. At a decoding step within the monitoring window, we identify relative overconfidence when the current entropy falls below the mean of the preceding trajectory:

Ht<τlow,t∈[t0,t1],(3)H_t<\tau_{\mathrm{low}},\qquad t\in[t_0,t_1], \tag*{(3)}

where t0<t1t_0<t_1 exclude the initial decoding steps and reserve sufficient steps for completing the CoT process. The threshold is adapted to the current trajectory as τlow=1t−1∑i<tHi\tau_{\mathrm{low}}=\frac{1}{t-1}\sum_{i<t}H_i.

(2) Local Entropy Spike. When a sharp transient increase in entropy exceeds a high threshold, the MLLM is considered to be experiencing instability arising from cross-modal conflicts. We detect this by:

ΔHt=Ht−Ht−1>τrise,(4)\Delta H_t=H_t-H_{t-1}>\tau_{\mathrm{rise}}, \tag*{(4)}

where τrise\tau_{\mathrm{rise}} is a hyperparameter. Jointly, we characterize hallucination-prone states as those in which the model operates under an overall overconfident convergence regime while simultaneously exhibiting localized high uncertainty. Such a global-local mismatch indicates a breakdown in multimodal coordination and substantially increases the risk of erroneous generation.

Visual Attention Re-Alignment Decoding

Once a hallucination-prone state is detected, instead of performing token-level interventions at the final generation layer, we carry out an attention-level intervention designed to restore visual grounding and better coordinate multimodal interactions without modifying the MLLM’s architecture. The implementation consists of two phases.

Prefill phase. Given the input vision tokens v={v1,v2,…,vn}v=\{v_1,v_2,\ldots,v_n\}, to decouple visual saliency from textual reasoning, we hook the self-attention map A\mathcal{A} in the image tokens from the final layer of the vision encoder and set it as the reference visual importance map:

Aref=MEAN⁡heads(∑ihAi)∈Rn×n,where {Ai}∈Rh×n×n.(5)\mathcal{A}_{\mathrm{ref}}=\operatorname{MEAN}_{\mathrm{heads}}\left(\sum_i^h\mathcal{A}_i\right)\in\mathbb{R}^{n\times n},\qquad\text{where }\{\mathcal{A}_i\}\in\mathbb{R}^{h\times n\times n}. \tag*{(5)}

We derive a visual guidance representation evise_{\mathrm{vis}} by using the reference attention weights to aggregate the visual embeddings VV:

evis=∑j=1n(Aref,j∑Aref)⋅Vj.(6)e_{\mathrm{vis}}=\sum_{j=1}^{n}\left(\frac{\mathcal{A}_{\mathrm{ref},j}}{\sum\mathcal{A}_{\mathrm{ref}}}\right)\cdot V_j. \tag*{(6)}

Therefore, evise_{\mathrm{vis}} compresses the high-dimensional visual information into a single vector that prioritizes salient tokens over “background” tokens, while effectively maintaining the original semantic structure of the input.

Injection phase. We inject the re-aligned representation at the first layer of the language model to influence the hidden states at the earliest possible stage. The hidden state ht\mathbf{h}_t is modified via a residual addition: ht′=ht+α⋅evis\mathbf{h}'_t = \mathbf{h}_t + \alpha\cdot\mathbf{e}_{\mathrm{vis}}, where α\alpha is the intervention strength hyperparameter that controls the degree of visual re-alignment. For Qwen3-VL, V\mathbf{V} denotes the layer-0 hidden states at the visual-token positions, which already lie in the residual-stream space. For Llama-3.2-Vision, whose visual features enter through cross-attention, evis\mathbf{e}_{\mathrm{vis}} is taken from the first cross-attention layer and rescaled to the root-mean-square magnitude of ht\mathbf{h}_t before injection, so that α\alpha has a consistent meaning across architectures.

Experiments

Experimental Setup

Benchmarks. Our method is evaluated across 6 VLM benchmarks covering 9 evaluation settings, grouped into two categories: vision-language reasoning and visual hallucination tasks. 1) Vision-language reasoning tasks (VStar, MMVP, VisOnly, and M3^{3}CoT) mainly assess a model’s ability to perform semantically grounded reasoning by jointly integrating visual evidence, world knowledge and linguistic understanding, including compositional reasoning, contextual interpretation, and cross-modal inference. 2) Visual hallucination tasks include POPE ([18]) and HallusionBench ([11]), two specialized benchmarks designed to assess whether a model’s responses remain faithful to the provided context (e.g., images), thereby directly evaluating models’ reliability. More details about the benchmarks can be found in Appendix § 10.§ 1.

Baseline models. We adopt three popular MLLMs (Qwen3-VL-8B-Instruct, Qwen3-VL-4B-Instruct ((Bai et al., 2025)), and Llama-3.2-11B-Vision ((Grattafiori et al., 2024))) as backbones to verify that our proposed method generalizes effectively across models of different scales and architectures. For performance and efficiency comparison, we also evaluate against existing training-free hallucination mitigation and reasoning enhancement methods, including VCD ([16]), MemVR ((Zou et al., 2025)) and OPERA ([13]). We keep generation hyperparameters, including max_tokens and temperature, consistent for fair comparison. More details about how these methods are designed to mitigate hallucinations can be found in Appendix § 10.§ 1.

Implementation details and hyperparameter protocol. For all experiments, we monitor the entropy trajectory from decoding step 5 (t0t_0) to step 60 (t1t_1). We set the entropy-rise threshold τrise=0.25\tau_{\mathrm{rise}}=0.25 and inject visual features with strength α=0.5\alpha=0.5 for two decoding steps after each trigger. The monitoring window is motivated by an analysis of 270 VQA trajectories, which shows that hallucinated spans concentrate in the first 20% of generation (Appendix § 9.§ 2). Following prior training-free methods ((Zou et al., 2025)), we select τrise\tau_{\mathrm{rise}} and α\alpha on a small held-out subset of M3^{3}CoT. We then fix all intervention hyperparameters across backbones and benchmarks. This tune-once configuration avoids per-setting optimization and simplifies the application of SEED. § 5.3 reports the corresponding sensitivity analysis. The maximum generation length is 512 tokens, and all inference is conducted on four NVIDIA RTX 5090 GPUs with 32GB VRAM.

Main Experimental Results

Tables 2 and 3 present a comprehensive comparison of zero-shot performance across multiple multimodal reasoning benchmarks and specialized hallucination benchmarks under varying model scales and decoding strategies. The core insights from the main experiments are summarized below.

Consistent accuracy uplift on comprehensive benchmarks. Compared to other baseline models, SEED delivers consistently larger and more stable performance gains across multiple benchmarks and baseline models, achieving average improvements of 3.2%–3.7% depending on the backbone and improving over the base model in 13 of 15 settings (vs. 9/15 for VCD and 3/14 for MemVR). In particular, MemVR and VCD often gain improvements on MMVP and VisOnly but fail consistently on M3^{3}CoT, which requires longer and more complex reasoning. In contrast, SEED maintains robust improvements across different models and evaluation cases.

Reduced hallucination. SEED demonstrates a specialized capability in mitigating hallucinations in MLLMs. On two hallucination-focused benchmarks, POPE and HallusionBench, SEED improves the Qwen3-VL-8B and Qwen3-VL-4B backbones by 2.5% and 3.6% on average, respectively. These results support our hypothesis that temporal entropy dynamics can serve as effective indicators of emerging hallucination-prone states, enabling timely intervention during reasoning.

MethodMultimodal Reasoning & UnderstandingSummary
VStar↑MMVP↑VisOnlys_s↑VisOnlyr_r↑M3^3CoT↑Δ\Delta (%) / improved
Qwen3-VL-8B73.8276.2342.2054.6078.21–
+ VCD54.45(-19.4)80.00(+3.8)42.57(+0.4)55.71(+1.1)77.68(-0.5)−2.93 / 3/5
+ MemVR74.87(+1.0)80.67(+4.4)41.86(-0.3)54.44(-0.2)77.68(-0.5)+0.89 / 2/5
+ SEED (Ours)83.25(+9.4)76.33(+0.1)47.00(+4.8)54.37(-0.2)79.97(+1.8)+3.17 / 4/5
Qwen3-VL-4B68.0676.3342.1452.2276.79–
+ VCD54.97(-13.1)78.67(+2.3)43.25(+1.1)52.73(+0.5)75.28(-1.5)−2.13 / 3/5
+ MemVR66.49(-1.6)76.33(-0.0)42.14(-0.0)54.22(+2.0)76.08(-0.7)−0.06 / 1/5
+ SEED (Ours)76.76(+8.7)77.33(+1.0)45.57(+3.4)56.42(+4.2)78.03(+1.2)+3.71 / 5/5
Llama-3.2-11B50.2662.3325.0033.4064.66–
+ VCD42.41(-7.8)63.67(+1.3)27.14(+2.1)33.91(+0.5)63.13(-1.5)−1.08 / 3/5
+ MemVR40.31(-9.9)60.30(-2.0)21.57(-3.4)27.78(-5.6)-−5.26 / 0/4
+ SEED (Ours)57.61(+7.3)61.33(-1.0)28.00(+3.0)36.90(+3.5)67.58(+2.9)+3.15 / 4/5

Table 2. Comparisons across general reasoning benchmarks. The summary column reports the macro-average change in percentage points and the number of strictly positive changes over available settings. Base MLLMs use the default CoT setting.

MethodMultimodal HallucinationSummary
HalluBench↑POPE-R↑POPE-P↑POPE-A↑Δ\Delta (%) / improved
Qwen3-VL-8B58.5090.2587.5386.42–
+ VCD56.24(-2.3)91.67(+1.4)87.03(-0.5)86.30(-0.1)−0.37 / 1/4
+ MemVR55.87(-2.6)90.63(+0.4)88.12(+0.6)86.85(+0.4)−0.31 / 3/4
+ SEED (Ours)59.28(+0.8)97.39(+7.1)90.94(+3.4)85.07(-1.3)+2.49 / 3/4
Qwen3-VL-4B54.8090.9288.6487.96–
+ VCD53.84(-1.0)90.60(-0.3)87.87(-0.8)87.37(-0.6)−0.66 / 0/4
+ MemVR55.27(+0.5)90.77(-0.2)87.69(-1.0)88.39(+0.4)−0.05 / 2/4
+ SEED (Ours)54.84(+0.0)98.45(+7.5)92.84(+4.2)90.43(+2.5)+3.56 / 4/4
Llama-3.2-11B42.3891.3388.0285.04–
+ VCD43.66(+1.3)86.23(-5.1)80.80(-7.2)76.43(-8.6)−4.91 / 1/4
+ MemVR43.16(+0.8)91.02(-0.3)87.86(-0.2)83.83(-1.2)−0.23 / 1/4
+ SEED (Ours)46.42(+4.0)91.10(-0.2)87.10(-0.9)83.27(-1.8)+0.28 / 1/4

Table 3. Comparisons across hallucination benchmarks. The summary column reports the macro-average change in percentage points and the number of positive changes over available settings. Base MLLMs use the default CoT setting.

Failure analysis. SEED remains net positive on Llama-3.2-11B overall, improving average performance by 3.15 points on reasoning benchmarks and 0.28 points on hallucination benchmarks. However, it decreases accuracy on MMVP and POPE by 0.2–1.8 points. On Llama POPE, VCD degrades substantially across all three splits by 5.1–8.6 points, whereas MemVR and SEED exhibit smaller and broadly comparable declines; their absolute scores differ by at most 0.8 points. This cross-method pattern suggests that the localized weakness is not specific to SEED. Two factors may contribute. First, Llama-3.2-Vision uses an earlier cross-attention-based design that injects visual information through a separate pathway, so ϵvis\epsilon_{\mathrm{vis}} requires scale alignment before being added to the textual residual stream. Second, POPE’s short binary answers amplify small token-level perturbations. A paired transition analysis on Llama POPE-Adversarial (Appendix B.5) supports the latter explanation: the number of false-positive answers is nearly unchanged (462→467), whereas invalid-format outputs increase (610→646). Thus, the net drop stems mainly from failures to produce responses in the required yes/no format rather than increased object hallucination.

Ablation Studies

Variants of attention intervention strategies. As shown in Table 4, we further analyze multiple variants of visual intervention strategies. The results indicate that performance is highly sensitive to the design of attention intervention, highlighting the importance of properly coordinating multimodal attention during reasoning. Specifically, “Equal attention” resets the text-to-image attention map to equal weights; “Truncated SVD Project” performs SVD on the attention key matrix and retains only the top-kk principal components to form a low-rank orthogonal subspace, so that the MLLM receives compressed, important visual information for re-alignment; “ViT attention intervene” randomly perturbs the attention distribution in the ViT layers to isolate the effect of visual information. More details of the variants can be found in Appendix § 10.§ 6.

Attention Intervention StrategyQwen3-VL-8BQwen3-VL-4B
Accuracy ↑Efficiency ↑Accuracy ↑Efficiency ↑
None (baseline)78.210.2076.790.22
+ Equal attention80.24(+2.0)0.05(×0.25)78.02(+1.2)0.07(×0.32)
+ Truncated SVD Project54.28(-23.9)0.17(×0.85)50.23(-26.6)0.18(×0.82)
+ ViT attention intervene60.32(-17.9)0.15(×0.75)55.36(-21.4)0.16(×0.73)
+ SEED (Ours)79.97(+1.8)0.19(×0.95)78.03(+1.2)0.21(×0.95)

Table 4. Comparison of attention intervention strategies across backbones on M3^{3}CoT (Chen et al., 2024). Each backbone is evaluated in terms of accuracy and efficiency (instance/s). All hyperparameters are kept consistent. The best and second-best results are highlighted in green and blue, respectively.

Although some variants (e.g., “Equal attention”) effectively improve models’ accuracy by 2.0%, they often sacrifice inference efficiency with 4× longer inference time, while efficient variants suffer severe performance drops. In contrast, SEED achieves a favorable balance between effectiveness and computational efficiency across various backbones. End-to-end, SEED increases latency by less than 5% (3.30 to 3.45 ms/token), compared with 54% for VCD, 157% for MemVR, and 333% for OPERA; full results appear in Appendix § 6.

Hyperparameter sensitivity. Since SEED uses a single hyperparameter configuration across all models and benchmarks, hyperparameter sensitivity is examined in this section. On M3^{3}CoT with Qwen3-VL-8B, no tested setting across α∈[0.4,0.7]\alpha\in[0.4, 0.7], τrise∈[0.225,0.300]\tau_{\mathrm{rise}} \in[0.225, 0.300], and t1∈[40,70]t_1 \in[40, 70] falls below the base model, and all but α=0.7\alpha=0.7 improve on it (Appendix § 10.7). Together, the fixed cross-setting protocol and broad stable range indicate that SEED does not depend on narrowly tuned hyperparameter values.

Conclusion

In this work, we rethink hallucination in MLLMs as a dynamical phenomenon emerging from the decoding trajectory. By interpreting entropy as a structured signal of multimodal interaction, we show that hallucinations are closely tied to two coupled behaviors: a globally overconfident regime that reflects premature reliance on ineffective reasoning, and locally unstable entropy fluctuations that expose conflicts in cross-modal reasoning. Motivated by these findings, we propose an entropy-aware self-introspective framework that uses entropy as a real-time diagnostic and intervention signal for targeted visual re-alignment during critical reasoning steps. This shifts hallucination mitigation from passive uncertainty modeling to active control over multimodal interaction dynamics. Empirically, SEED improves 21 of 27 model–benchmark settings, validating the proposed approach, as well as the broader premise that structured decoding signals can help identify and partially correct unreliable reasoning states. Looking forward, promising directions include query-dependent visual representations, and adaptive intervention strengths, which may further improve SEED’s robustness across model families. We believe that such entropy-centric analysis provides a foundation for more controllable and interpretable multimodal reasoning systems.

AI Use Statement

Generative AI tools were used to refine the writing of this manuscript and to assist with code development and debugging. All research design, methodological decisions, experimental procedures, and interpretation of results were executed by the authors. All AI-assisted text and code were reviewed, verified, and, where applicable, tested by the authors. The authors take full responsibility for the final content of this work.

References

More Preliminary Experiment Results

Localized entropy spikes mark hallucination onset

As illustrated in Figure 4, we provide more cases to understand how hallucinations emerge. These cases show that hallucination is closely coupled with sharp transitions in the model’s local entropy. As shown in the figure, hallucinations typically manifest during entropy spike transitions (e.g., the P3→P4P3 \to P4 transition in the top case and P4→P6P4 \to P6 in the bottom case), where the model shifts from a stable reasoning state into a regime of high uncertainty. During these high-entropy phases, the MLLM begins to generate inconsistent content. For the top example, the MLLM selectively attributes fragility to only a subset of objects, despite clear visual evidence to the contrary.

More examples illustrating hallucination emergence during MLLM reasoning

Figure 4. More examples illustrating hallucination emergence during MLLM reasoning. The left panel shows entropy over generated tokens, while the right panel visualizes the corresponding visual attention patterns at selected reasoning steps. The gray region denotes the span of incorrect generation. Hallucinated reasoning coincides with regions containing sharp entropy spikes and increasingly strong yet diffuse visual attention.

Furthermore, we observe a distinct temporal coupling between these entropy spikes and the diffuseness of visual attention. While successful reasoning is characterized by a rapid convergence and stabilization of visual attention, hallucinatory trajectories exhibit a prolonged yet ineffective visual dependency. The spikes therefore provide a useful marker of a breakdown in modality integration: the model fails to effectively map visual tokens to its linguistic output, leading to the generation of “confident” but factually ungrounded falsehoods. This alignment suggests that entropy does not merely reflect token-level static output uncertainty, but also tracks changes in multimodal information interaction.

Hallucinations mainly occur at the early stage

To characterize when erroneous reasoning emerges, we analyze 270 VQA reasoning trajectories and record the start and end tokens of each erroneous span (Figure 5). These spans are concentrated within the first 20% of generation and typically extend for approximately 50–70 tokens within the maximum length of 512 tokens. Their early occurrence, together with their association with local entropy spikes in Figure 4, motivates SEED’s focus on early decoding. When an entropy trigger is detected, SEED injects the attention-weighted visual feature evise_{\mathrm{vis}} at Layer 0 to reinforce visual grounding from the start of the language-model computation.

Entropy Dynamics and Breaking Claim Spans during CoT Generation

Figure 5. Statistical analysis of wrong reasoning span sizes. The reasoning processes of 270 VQA samples are recorded. Results show that wrong reasoning claims concentrate in the first 20% of the generation process.

Population-level entropy analysis

To complement the case studies above, we analyze 1,228 randomly sampled VQA trajectories generated by Qwen3-VL-8B, of which 763 contain hallucinations and 465 are correct and serve as controls (following the original accuracy distribution). For each hallucinated trajectory, we identify the first token at which hallucinated content appears and align it to relative position p=0p=0. We then compute the mean token-to-token entropy change ΔH=Ht−Ht−1\Delta H = H_t - H_{t-1} in a ±20\pm20-token window around the onset. Since correct trajectories have no hallucination onset, p=0p=0 is sampled uniformly at random for them. Table 5 reports the resulting profile. At the onset, the hallucination group exhibits a mean entropy jump of 0.245, more than 20× larger than that of the control group (0.012), followed by a sharp drop at p=+1p=+1. A two-sided Mann–Whitney U test on the per-response ΔH\Delta H at p=0p=0 gives p=4×10−4p=4 \times10^{-4}.

pp−20 ∼ −11−5−4−3−2−10+1+2 ∼ +5+6 ∼ +20
Hallucinated0.00200.05600.00110.0188−0.04230.00650.2447−0.1728−0.0227 ∼ 0.0081−0.0025
Correct0.00070.01190.0140−0.0010−0.01090.00470.0117−0.0125−0.0404 ∼ 0.02650.0007

Table 5. Mean token-to-token entropy change ΔH\Delta H aligned to the hallucination onset (p=0p=0). For ranges, the mean over the range (or its min–max) is reported.

We further compare three trajectory-level statistics (Table 1 in the main text): the length of the longest low-entropy run (low-entropy duration), the fraction of steps whose ΔH\Delta H exceeds τrise\tau_{\mathrm{rise}} (spike frequency), and the maximum ΔH\Delta H within a trajectory. Hallucinated trajectories show significantly longer low-entropy durations (p<10−35p < 10^{-35}), more frequent spikes (p<10−9p < 10^{-9}), and larger maximum entropy changes (p<10−35p < 10^{-35}), consistent with the two patterns identified in § 2. These results establish a robust statistical association; they do not by themselves establish that entropy spikes cause hallucinations, and SEED only relies on them as a detection signal.

Experiment Details

Benchmarks

Our method is evaluated across multiple VLM benchmarks, grouped into two main categories: general vision-language understanding and visual hallucination task. 1) General multimodal reasoning task assesses how MLLM can effectively incorporate information and use it as the reference for complex reasoning tasks. There are four benchmarks in this category:

  • M3M^{3}CoT (Chen et al., 2024) specifically addresses three critical limitations found in existing MCoT evaluations: the absence of visual modal reasoning, the prevalence of single-step visual reasoning, and a lack of domain diversity. By facilitating a multi-domain, multi-step scenario, M3M^{3}CoT provides a more rigorous environment for testing how MLLMs leverage knowledge from both textual and visual modalities for complex (long-horizon) and step-by-step reasoning. In our experiments, we use its validation set for testing, which contains 1,110 VQA instances.

  • MMVP (Tong et al., [28]) is a compact benchmark that contains 300 image-question pairs. It is designed to evaluate vision-language models on “CLIP-blind” visual features. Higher performance on MMVP indicates the MLLMs can genuinely utilize visual information rather than relying on language priors.

  • VisOnly (Synthetic and Real splits) (Kamoi et al., [15]) evaluates MLLMs on 12 geometric perception tasks, and reveals that base MLLMs often cannot accurately perceive basic geometric information in images. The dataset contains a synthetic split (700 questions) and a real split (900 questions).

  • VStar (Wu & Xie, [30]) proposes a visual search mechanism together with a benchmark to address a critical limitation in current MLLMs: the inability to selectively process visual details in high-resolution or crowded images.

  1. Visual hallucination task. In addition to assessing general MLLM capabilities, we conduct evaluations on two specialized benchmarks, where our method demonstrates strong effectiveness in mitigating hallucination:

  • POPE (Li et al., [18]) is a diagnostic benchmark for probing object hallucination in MLLMs via binary (yes/no) questions about object presence. It comprises 3,000 questions

constructed under different sampling strategies (e.g., random, popular, adversarial) to systematically test whether models can correctly assert the existence of certain objects.

  • HallusionBench [11] comprises 346 images paired with 1,129 questions, all meticulously crafted by human experts. It is specifically designed to evaluate multimodal hallucination across diverse visual reasoning scenarios.

Baseline models

  • Base Models. We select Qwen3-VL-4B-Instruct, Qwen3-VL-8B-Instruct (Bai et al., 2025), and Llama-3.2-11B-Vision (Grattafiori et al., 2024) as our primary backbones to ensure diversity in both model scale and architectural design. Specifically, Qwen3-VL models represent strong decoder-only self-attention architectures at different parameter scales (4B vs 8B), enabling us to analyze scaling effects on our proposed method. In contrast, Llama-3.2-11B-Vision adopts a cross-attention-based design, providing a complementary paradigm for multimodal fusion. This combination allows us to comprehensively evaluate the generality of our method across distinct model architectures and capacities.

  • Other Baseline Models. We also consider alternative training-free decoding strategies that aim at alleviating MLLMs’ hallucinations, including VCD [16] and MemVR (Zou et al., 2025). Specifically, VCD mitigates hallucination by contrasting model predictions under visual-conditioned and text-only contexts during decoding, reweighting candidate tokens to favor those consistent with visual evidence. MemVR improves visual reasoning by maintaining a visual memory bank of vision representations, enabling the model to retrieve and reinforce previously grounded information to ensure consistency across multi-step reasoning.

Reproducibility

Besides M3^{3}CoT (Chen et al., 2024), we employ OpenCompass VLMEvalKit [7] to conduct evaluations for fair comparison. We follow the official evaluation document of M3^{3}CoT for its own evaluation. During the inference of baseline models (MemVR and VCD), we follow their prompt instructions if they are available.

Computation costs

As summarized in Table 6, SEED introduces negligible inference overhead (∼\sim5% increase in time cost per 100 tokens) while improving performance over the greedy baseline.

MethodLatency ↓ (ms/token)Throughput ↑ (instance/s)Time Cost ↓ (min)
Greedy3.30 (×1.00)0.941 (×1.00)20.0 (×1.00)
OPERA14.28 (×4.33)0.224 (×0.24)85.0 (×4.25)
VCD5.08 (×1.54)0.598 (×0.63)31.5 (×1.58)
MemVR8.47 (×2.57)0.359 (×0.38)52.5 (×2.63)
SEED3.45 (×1.05)0.897 (×0.95)21.0 (×1.05)

Table 6. Performance comparison of SOTA methods and SEED in latency, throughput, and time cost. All methods are built on the Qwen3-VL-8B-Instruct backbone. The experiment is conducted on M3^{3}CoT and all models are prompted to generate CoT. Note that all results are evaluated under an inference acceleration framework.

Failure analysis

Paired transition analysis. To understand when SEED’s interventions are harmful, we compare the per-sample predictions of the base model and SEED on the representative failure case, Llama-3.2-11B-Vision on POPE-Adversarial (Table 7). SEED corrects 408 of the 1,137 samples the base model answers incorrectly, while flipping 454 correct answers, a net change of 46 samples. Breaking down this net degradation, the number of false-positive answers (i.e., hallucinated objects) barely changes (462→467), whereas invalid-format outputs increase from 610 to 646. Hence the drop is mainly associated with instruction-following failures under constrained binary evaluation—specifically, outputs that do not follow the required yes/no format—rather than a systematic increase in object hallucination. Interestingly, persistently wrong samples receive the strongest intervention signals (highest trigger ratio, most trigger events, and longest intervention span) yet remain hard to correct, suggesting that entropy dynamics identify difficult reasoning states even though not all of them are amenable to visual re-alignment. POPE’s short trajectories and binary outputs further make it sensitive to local token-level perturbations introduced during intervention.

TransitionInterpretationCountRateTrigger ratioAvg. triggersAvg. span
Correct → WrongHarmful intervention45415.13%46.26%2.047.95
Wrong → CorrectSuccessful correction40813.60%43.38%2.208.69
Wrong → WrongUnresolved error72924.30%60.22%2.9311.84

Table 7. Paired transition analysis between the base model and SEED on POPE-Adversarial with Llama-3.2-11B-Vision.

Ablation Variants

Here are the implementation details of attention intervention strategies.

  • Equal attention: Earlier work [8] shows that even (i.e., uniform) visual attention can help MLLMs alleviate hallucinations. Following this idea, we set up equal attention, which naively re-guides the model to attend to important visual evidence.

  • Truncated SVD: Following Lin et al. [19], we construct a low-rank visual projection matrix using truncated singular value decomposition (SVD) to constrain attention intervention within a compact visual subspace. Given the visual key matrix Kv∈RNv×dK_v \in\mathbb{R}^{N_v \times d}, we first perform singular value decomposition:

Kv=UΣVTK_v = U \Sigma V^{\mathsf{T}}

We then retain the top-kk principal components from VV to obtain a low-rank orthogonal basis Vk∈Rd×kV_k \in\mathbb{R}^{d \times k}. The visual projection matrix is constructed as

Pvis=VkVkTP_{\mathrm{vis}} = V_k V_k^{\mathsf{T}}

which projects attention intervention into the dominant visual subspace while avoiding explicit matrix inversion. The method is designed to keep the salient visual information and re-inject it into the reasoning process.

  • ViT attention intervene: Similar to multimodal contrastive methods (Liu et al., 2024a; Zhou et al., 2025b) that remove images to isolate the impact of language priors from the reasoning, we instead randomly perturb the attention map during the ViT self-attention process so the generated visual embedding will be ineffective. By doing so, we conduct contrastive decoding by actively comparing the results between intervened visual information and normal ones. The experimental results in Table 4 show that such contrastive decoding still requires careful design of interactions.

Hyperparameter analysis

All hyperparameters are selected once and shared across all backbones and benchmarks. Table 8 reports the sensitivity of SEED (Qwen3-VL-8B, M3M^{3}CoT) to the injection coefficient α\alpha, the trigger threshold τrise\tau_{\mathrm{rise}}, and the end of the monitoring window t1t_1, varying one hyperparameter at a time while fixing the others at their default values (α=0.5\alpha=0.5, τrise=0.25\tau_{\mathrm{rise}}=0.25, t0=5t_0=5, t1=60t_1=60). No setting falls below the base model, and all but one improve on it. Figure 6 visualizes the sensitivity to α\alpha and τrise\tau_{\mathrm{rise}}, and further results, including the start of the monitoring window, are shown in Figure 7.

Hyperparameter analysis of injection ratios and entropy threshold

Figure 6. Hyperparameter analysis of injection ratios (left) and entropy threshold (right).

0.400.450.500.70
α\alpha78.30 (+0.09)78.79 (+0.58)79.89 (+1.68)78.21 (+0.00)
0.2250.2500.2750.300
τrise\tau_{\mathrm{rise}}78.42 (+0.21)79.89 (+1.68)79.20 (+0.99)78.53 (+0.32)
40506070
t1t_179.12 (+0.91)79.48 (+1.27)79.89 (+1.68)78.91 (+0.70)

Table 8. Hyperparameter sensitivity on M3^{3}CoT with Qwen3-VL-8B (base model: 78.21). Gains over the base model are shown in parentheses.

Additional hyperparameter analysis plots

Figure 7. Additional hyperparameter analysis of the entropy threshold, injection ratio, and the start and end steps of the monitoring window.

Case studies

We further present a case study to investigate the effectiveness of our method and its impact on the entropy trajectory during decoding. As shown in Figure 8, although both the baseline Qwen3-VL-8B and SEED yield the same correct final answer, SEED exhibits improved visual grounding and produces more faithful intermediate descriptions. Specifically, Qwen3-VL-8B incorrectly rejects the premise by claiming that the umbrella handle is not broken, which is not supported by the visual evidence, where the umbrella handle is obviously missing. In contrast, SEED better aligns its reasoning with the image and incorporates this detail into a coherent explanation. This suggests that SEED not only preserves answer correctness but also enhances the reliability of the reasoning process, which is more important in long-horizon reasoning applications.

Case study illustrating SEED's improved answer accuracy and intermediate reasoning

Figure 8. A case study illustrating that SEED not only improves final answer accuracy but also produces more reliable intermediate reasoning steps. Hallucinated content is highlighted in purple, while correct reasoning is highlighted in yellow.

Figure 9 provides a comprehensive case study of how SEED improves both reasoning quality and underlying modality interaction dynamics. From the top qualitative analysis, the baseline model produces partially unreliable reasoning, ignoring the fact that the umbrella has a broken handle, leading to hallucination-prone intermediate steps. In contrast, SEED generates more coherent and visually grounded reasoning, where key inference steps are supported by image evidence, resulting in the correct answer.

Figure 9. Full analysis on one example to demonstrate the effectiveness of SEED. The hallucination related text is highlighted in yellow and the statement related to correct reasoning is highlighted in blue.

This behavioral difference is further reflected in the attention dynamics (bottom). As shown in the vision attention plot, the baseline exhibits relatively unstable attention with a slow decay, maintaining non-trivial attention weights even in later stages of generation. This suggests prolonged but ineffective reliance on visual information, which correlates with its inconsistent reasoning. In contrast, after the intervention trigger, SEED significantly boosts visual attention in the early-to-mid stages, followed by a sharper decay and stabilization, indicating more efficient integration of visual evidence into the reasoning process. Importantly, this pattern closely matches the attention dynamics identified in Figure 1(b), suggesting that the intervention effectively steers the model toward the desirable modality interaction regime.

Limitations

Although SEED effectively mitigates hallucinations during multimodal reasoning, it has several limitations. First, the relationship between entropy dynamics and hallucination established in this work is correlational; SEED uses it as a detection signal without claiming causality. Second, the injected visual representation is query-independent, and aligning it with the cross-attention-based Llama-3.2-Vision backbone introduces an additional compatibility constraint associated with localized degradations (Appendix § 10.§ 5); architecture-specific feature alignment may further improve robustness. Third, while the fixed trigger policy simplifies deployment across settings, an adaptive policy may better capture model-specific reasoning dynamics. In particular, the model could learn to self-evaluate the trigger conditions and the attention reweighting strategy across different reasoning dynamics.

Future work could investigate adaptive intervention mechanisms that learn to modulate visual realignment based on uncertainty evolution, reasoning trajectories, or model-specific behaviors.

Social Impacts

The social impacts of this work are primarily centered on improving the reliability and trustworthiness of multimodal AI systems deployed in real-world settings. By mitigating hallucinations in multimodal large language models, the proposed approach can enhance decision-support systems in high-stakes domains such as education, healthcare, and accessibility technologies, where incorrect or overconfident model outputs may lead to misleading conclusions or suboptimal decisions. Improved grounding interactions between visual and textual modalities also contribute to more transparent and interpretable AI behavior, which is critical for user trust and regulatory compliance. Furthermore, reducing hallucination-induced errors can lower the risk of misinformation propagation in AI-assisted content generation systems, thereby supporting healthier information ecosystems.

References

  1. [1]Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with Exploration: An Entropy Perspective. In Proceedings of the AAAI Conference on Artificial Intelligence, 2026.arxiv.org/abs/2506.14758
  2. [3]Matt Deitke, Christopher Clark, Sangho Lee, Rohn Tripathi, Yue Yang, Jae Sung Park, Mohamadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, Yensung Chen, Ajay Patel, Mark Yatskar, Chris Callison-Burch, Andrew Head, Rose Hendrix, Favyen Bastani, Eli Vanderbelt, Nathan Lambert, Yvonne Chou, Arnavi Chheda, Jenna Sparks, Sam Skjonsberg, Michael Schmitz, Aaron Sarnat, Byron Bischoff, Pete Walsh, Chris Newell, Piper Wolters, Tanmay Gupta, Kuo-Hao Zeng, Jon Borchardt, Dirk Groeneveld, Crystal Nam, Sophie Lebrecht, Caitlin Wittlif, Carissa Schoenick, Oscar Michel, Ranjay Krishna, Luca Weihs, Noah A Smith, Hannah Hajiishirzi, Ross Girshick, Ali Farhadi, Aniruddha Kembhavi, and † Allen. Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models. arXiv preprint arXiv:2409.17146, 2024. URL https://molmo.allenai.org/blog.arxiv.org/abs/2409.17146
  3. [4]Jiaqi Deng, Zonghan Wu, Zhan Heng, Xiaoshui Huang, Huan Huo, and Guangdong Xu. Mind What Matters for Reasoning: Aligning Cross-Modal Attention via Selective Probability Mass Concentration. arXiv preprint arXiv:2609.29940, September 2026a. doi: 10.48550/arXiv.2609.29940. URL https://arxiv.org/abs/2609.29940.
  4. [6]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In 9th International Conference on Learning Representations, ICLR 2021, 10 2020.
  5. [7]Haodong Duan, Xinyu Fang, Junming Yang, Xiangyu Zhao, Yuxuan Qiao, Mo Li, Amit Agarwal, Zhe Chen, Lin Chen, Yuan Liu, Yubo Ma, Hailong Sun, Yifan Zhang, Shiyin Lu, Tack Hwa Wong, Weiyun Wang, Peiheng Zhou, Xiaozhe Li, Chaoyou Fu, Junbo Cui, Jixuan Chen, Enxin Song, Song Mao, Shengyuan Ding, Tianhao Liang, Zicheng Zhang, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, Dahua Lin, and Kai Chen. VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models. MM 2024 - Proceedings of the 32nd ACM International Conference on Multimedia, pp. 11198–11201, 7 2024. doi: 10.1145/3664647.3685520. URL https://arxiv.org/pdf/2407.11691.
  6. [8]Mehrdad Fazli, Bowen Wei, Ahmet Sari, and Ziwei Zhu. Mitigating Hallucination in Large Vision-Language Models via Adaptive Attention Calibration. arXiv preprint arXiv:2505.21472, 5 2025. URL https://arxiv.org/pdf/2505.21472.
  7. [11]Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 14375–14385, 3 2024. URL http://arxiv.org/abs/2310.14566.
  8. [12]Jiashu He, Meizhu Liu, Olaitan P Olaleye, Amit Agarwal, M Avendi, Yassi Abbasi, Matthew Rowe, Hitesh Laxmichand Patel, Paul Li, Tao Sheng, Sujith Ravi, and Dan Roth. Think Twice Before You Write – an Entropy-based Decoding Strategy to Enhance LLM Reasoning. arXiv preprint arXiv:2604.00018, 2026.arxiv.org/abs/2604.00018
  9. [13]Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaq​​i Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.arxiv.org/abs/2311.17911
  10. [14]Fushuo Huo, Wenchao Xu, Zhong Zhang, Haozhao Wang, Zhicheng Chen, and Peilin Zhao. Self-introspective Decoding: Alleviating Hallucinations For Large Vision-Language Models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://github.com/huofushuo/SID.arxiv.org/abs/2408.02032
  11. [15]Ryo Kamoi, Yusen Zhang, Sarkar Snigdha Sarathi Das, Ranran Haoran Zhang, and Rui Zhang. VisOnlyQA: Large Vision Language Models Still Struggle with Visual Perception of Geometric Information. arXiv preprint arXiv:2412.00947, 7 2025. URL http://arxiv.org/abs/2412.00947.
  12. [16]Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. URL https://github.com/DAMO-NLP-SG/VCD.arxiv.org/abs/2311.16922
  13. [17]Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. In Proceedings of the 40th International Conference on Machine Learning, 1 2023a.arxiv.org/abs/2301.12597
  14. [18]Yifan Li, Yifan Du, Kun Zhou, Jipeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating Object Hallucination in Large Vision-Language Models. EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings, pp. 292–305, 10 2023b. URL http://arxiv.org/abs/2305.10355.
  15. [19]Yangguang Lin, Quan Fang, Yufei Li, Jiachen Sun, Junyu Gao, and Jitao Sang. HulluEdit: Single-Pass Evidence-Consistent Subspace Editing for Mitigating Hallucinations in Large Vision-Language Models. arXiv preprint arXiv:2602.22727, 2 2026. URL https://arxiv.org/pdf/2602.22727v1.
  16. [20]Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yaoob, and Lijuan Wang. Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning. In 12th International Conference on Learning Representations, ICLR 2024. International Conference on Learning Representations, ICLR, 6 2024a. URL https://arxiv.org/pdf/2306.14565.
  17. [21]Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. A Survey on Hallucination in Large Vision-Language Models. arXiv preprint arXiv:2402.00253, 2024b. URL https://github.com/lhanchao777/.arxiv.org/abs/2402.00253
  18. [23]Miao Pan, Wangjie Gan, Jintao Chen, Wenqi Zhang, Sun Bing, Jianwei Yin, and Xuhong Zhang. Ground What You See: Hallucination-Resistant MLLMs via Caption Feedback, Diversity-Aware Sampling, and Conflict Regularization. Proceedings of the AAAI Conference on Artificial Intelligence, 40(10):8242–8250, 3 2026. ISSN 2374-3468. doi: 10.1609/AAAI.V40I10.37772. URL https://ojs.aaai.org/index.php/AAAI/article/view/37772.DOI
  19. [24]Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research, 21:5485 – 5551, 10 2019.arxiv.org/abs/1910.10683
  20. [25]Aman Sharma and Chopra Lossfunk. Think Just Enough: Sequence-Level Entropy as a Confidence Signal for LLM Reasoning. arXiv preprint arXiv:2510.08146, 10 2025. URL https://arxiv.org/pdf/2510.08146.
  21. [27]Guohao Sun, Hang Hua, Jian Wang, Jiebo Luo, Sohail Dianat, Majid Rabbani, Raghuveer Rao, and Zhiqiang Tao. Latent Chain-of-Thought for Visual Reasoning. arXiv preprint arXiv:2510.23925, 10 2025. URL http://arxiv.org/abs/2510.23925.
  22. [28]Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saïng Xie. Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 9568–9578, 4 2024. URL http://arxiv.org/abs/2401.06209.
  23. [29]Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xiong-Hui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin. Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning. In The Thirty-Ninth Annual Conference on Neural Information Processing Systems, 2025.arxiv.org/abs/2506.01939
  24. [30]Penghao Wu and Saining Xie. V∗: Guided Visual Search as a Core Mechanism in Multimodal LLMs. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 13084–13094, 12 2023. URL http://arxiv.org/abs/2312.14135.
  25. [31]Tong Wu, Guandao Yang, Zhibing Li, Kai Zhang, Ziwei Liu, Leonidas Guibas, Dahua Lin, and Gordon Wetzstein. GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generation. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 22227–22238, 1 2024. ISSN 10636919. doi: 10.1109/CVPR52733.2024.02098. URL https://arxiv.org/pdf/2401.04092.

Paper details

vlnc:W_hzzpb37p
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Generative AI tools

View statement
Looking for related papers…
Contents