betaFeedback

WorksComputer Science

SARGO: Agreement-Gated Reinforcement and Suppression via On-Policy Self-Guidance for Visual Hallucination Mitigation

Abstract

Vision-language models remain susceptible to visual hallucinations, often mixing grounded information with unsupported claims in the same response. Response-level reinforcement learning assigns a shared advantage to all tokens in a response, causing negative updates to suppress correct information alongside unsupported claims. To reduce hallucination while preserving useful visual information, we introduce SARGO (Self-guided Advantage-Routed Grounding Optimization), a post-training framework for selective reinforcement and suppression through on-policy self-guidance. Direct and reciprocal weighting gives better-supported tokens stronger reinforcement on positive updates and weaker penalties on negative updates. Agreement between overall and hallucination-specific advantages selects when to apply this guidance. At the response level, a reference-calibrated reward couples factuality with the retention and expansion of grounded information, discouraging lower hallucination scores obtained through omission. Experiments on Qwen3-VL-4B, Qwen3-VL-8B, and Gemma3-4B demonstrate that SARGO outperforms the evaluated mitigation baselines across Obj-Hal, AMBER, THRONE, and SHR evaluations. SARGO achieves these gains while preserving useful visual information and general multimodal capabilities.

Introduction

Instruction-tuned vision-language models (VLMs) support a broad range of multimodal understanding and reasoning tasks [2, 7, 15, 29]. However, they remain prone to visual hallucinations, producing content unsupported by the input image [26, 34]. Hallucinated claims often coexist with accurate observations within the same response. Effective mitigation should suppress unsupported content while preserving grounded visual information.

Existing methods [30, 39, 44] can suppress hallucinated content at the expense of useful visual information, reducing object coverage. The objective should reward correction together with information retention. Response-level reinforcement learning (RL) [37], however, assigns every token the same advantage, suppressing correct visual details alongside hallucinated claims on negative updates.

On-policy self-distillation (OPSD) provides token-level guidance by conditioning the model on privileged context [49]. A natural combination adds its teacher–student KL objective to the RL loss. Such distribution matching follows teacher preferences independently of the response advantage, potentially reinforcing tokens that RL seeks to suppress. We instead use teacher support to redistribute reward-driven updates. For hallucination mitigation, this guidance remains valuable even on responses with negative advantages, where grounded details coexist with unsupported claims.

This motivates SARGO (Self-guided Advantage-Routed Grounding Optimization), a post-training framework that uses on-policy self-guidance for selective reinforcement and suppression. The current policy scores its own sampled responses with and without privileged visual context to obtain token-level grounding support. Direct teacher-support weights emphasize well-supported tokens, but applying them to negative advantages would also amplify their penalties. Our central principle is that better-supported tokens should receive stronger reinforcement on positive updates and weaker penalties on negative updates. SARGO therefore uses direct weights for reinforcement and reciprocal weights for suppression.

Teacher guidance should align with the objective driving each update. The overall reward combines factuality with information retention and response quality, potentially yielding an advantage that differs in sign from the hallucinationspecific advantage. Our hallucination-aware router applies teacher weighting when the two advantages agree in sign and uses uniform RL weights for all other responses.

Lower hallucination scores can result from omitting useful visual information. We design a reference-calibrated response objective that couples factuality with coverage retention and expansion. Coverage losses are measured against each image’s reference coverage, and gains against the remaining headroom. A coverage gate reduces factuality credit when coverage falls, rewarding error reduction together with preserved visual information.

As illustrated in Figure 1, SARGO achieves the lowest Obj-Hal hallucination rates among evaluated methods on Qwen3-VL-4B, Qwen3-VL-8B, and Gemma3-4B. It consistently outperforms mitigation baselines across hallucination benchmarks and improves coverage over Vanilla while preserving general multimodal capabilities. Ablations establish the complementary contributions of selective reinforcement and suppression, hallucination-aware routing, and reward design. Our contributions are as follows:

Figure 1. Object hallucination and information retention on Obj-Hal. Comparison of SARGO and baselines on (a) Qwen3-VL-4B, (b) Qwen3-VL-8B, and (c) Gemma3-4B. Lower CHAIRi_i and higher object recall (ObjRec) indicate better performance. SARGO achieves the lowest CHAIRi_i across all three backbones while retaining high object recall.

  • We propose SARGO for selective reinforcement and suppression, using direct and reciprocal weighting to modulate token-level update strength and hallucination-aware routing to select when on-policy self-guidance applies.

  • We introduce a reference-calibrated response objective rewarding factuality alongside retained and newly recovered visual information. Coverage penalties and gains encourage retention and expansion, while a coverage gate ties factuality credit to retained information.

  • We demonstrate joint hallucination–coverage gains and lower SHR across Qwen3-VL and Gemma3 backbones, with ablations establishing the roles of selective attribution, agreement routing, and reward design.

Related Work

Visual reliability and hallucination mitigation. Maintaining consistency between visual evidence and model outputs is central to reliable visual understanding. Adversarial attacks can disrupt image–text alignment [10] and manipulate visual predictions [9]. On natural images, VLMs can still generate content unsupported by or inconsistent with the visual evidence, a phenomenon known as visual hallucination [26, 34] [17]. Existing solutions can be broadly grouped into training-free and training-based approaches. Training-free methods improve grounding through contrastive decoding in VCD [24], attention-guided decoding in OPERA [18], and adaptive focal grounding in HALC [5]. CMI-VLD calibrates decoding through conditional mutual information between visual and textual tokens [11]. Woodpecker and LURE instead correct unsupported claims in descriptions, improving their consistency with the visual evidence [45, 51].

Training-based approaches strengthen factual supervision through instruction refinement [28, 46], human corrections [47], and model-generated feedback [48]. Building on DPO [33], HA-DPO and HDPO construct hallucination-aware preferences [14, 50], while OPA-DPO and SymMPO improve preference supervision through reference-policy alignment and symmetric margin regulation, respectively [30, 44]. Visual dependence is strengthened through contrastive learning in HACL [19], joint image and response preferences in mDPO [39], and entity-centric alignment in EMPO [41]. Recent studies address hallucination in multimodal reasoning models [8, 12, 22], multi-image settings [43], and model merging [25].

Reinforcement learning and on-policy distillation. RL post-training optimizes language models using response rewards. RLHF trains a reward model on human preferences and optimizes the policy with PPO [32, 35]. GRPO estimates advantages from rewards within a response group without a separate critic [36]. To refine supervision beyond complete responses, Fine-Grained RLHF learns segment-level rewards for factuality, relevance, and completeness [42]. Process-supervision methods evaluate intermediate reasoning steps using automated annotations [31] or implicit rewards learned from outcome feedback [6].

Knowledge distillation provides teacher guidance through output or distribution matching. Sequence-level distillation trains on teacher-generated responses [21], whereas on-policy distillation queries the teacher on student-generated trajectories. MiniLLM and generalized knowledge distillation use such trajectories for token-level distribution matching [1, 16]. OPSD obtains this guidance from the same model under privileged conditioning [49]. Distilled RL trains reasoning models on mathematical tasks with binary correctness rewards and incorporates teacher likelihood ratios into policy-gradient weights, retaining uniform weights for nonpositive advantages [38]. SARGO targets visual hallucination, where unfavorable responses can still contain grounded information. Reciprocal weighting enables selective suppression, while advantage agreement determines when to apply on-policy self-guidance. A reference-calibrated reward complements these token-level updates by jointly evaluating factuality, information retention, and coverage growth.

Method

SARGO combines reference-calibrated rewards with selective reinforcement and suppression, as illustrated in Figure 2. Its core principle is to give better-supported tokens stronger reinforcement on positive updates and weaker penalties on negative updates. Given an image and instruction xx, the rollout policy πold\pi_{\mathrm{old}} samples GG responses {yi}i=1G\{y_i\}_{i=1}^{G}.

Overview of SARGO

Figure 2. Overview of SARGO. The frozen initial policy supplies reference responses for reward calibration, while the current policy scores its own rollouts under ordinary and privileged contexts to obtain token-level grounding support. Agreement between overall and hallucination-specific advantages selects when self-guidance applies. Direct and reciprocal weighting modulate the relative strength of token-level reinforcement and suppression, illustrated by bar magnitude. Neutral routes retain uniform token weights.

Reference-Calibrated Response Reward

Hallucination reduction should preserve and extend correctly grounded content. We use the frozen initial policy π0\pi_0 to obtain a reference response y0y^0 for each input xx. Its grounded content provides an image-specific baseline for evaluating retention and coverage gains. Let O(x)\mathcal{O}(x) denote annotated object categories and M(y)M(y) those extracted from yy.

Dual-granularity hallucination. A small fraction of unsupported categories can still leave false object claims. We therefore penalize both the fraction of unsupported categories and the presence of any unsupported category:

hiobj=∣M(yi)∖O(x)∣∣M(yi)∣,hiresp=I[∣M(yi)∖O(x)∣>0],h_i^{\mathrm{obj}} = \frac{\lvert M(y_i) \setminus\mathcal{O}(x)\rvert}{\lvert M(y_i)\rvert}, \qquad h_i^{\mathrm{resp}} = \mathbb{I}[\lvert M(y_i) \setminus\mathcal{O}(x)\rvert> 0],
rihall=1−αhiobj−(1−α)hiresp.(1)r_i^{\mathrm{hall}} = 1 - \alpha h_i^{\mathrm{obj}} - (1-\alpha)h_i^{\mathrm{resp}}. \tag*{(1)}

The coefficient α\alpha balances category-level hallucination severity and response-level hallucination occurrence.

Coverage retention and expansion. A response can reduce hallucination by avoiding uncertain object mentions while also omitting correct visual details. We therefore reward the retention and expansion of object coverage relative to the initial policy’s reference response. Let Ci=∣M(yi)∩O(x)∣/∣O(x)∣C_i = \lvert M(y_i) \cap\mathcal{O}(x)\rvert/\lvert\mathcal{O}(x)\rvert and C0=∣M(y0)∩O(x)∣/∣O(x)∣C^0 = \lvert M(y^0) \cap\mathcal{O}(x)\rvert/\lvert\mathcal{O}(x)\rvert for inputs with nonempty annotations. Relative loss and gain are

Δi,−=[C0−Ci]+C0,Δi,+=[Ci−C0]+1−C0.(2)\Delta_{i,-} = \frac{[C^0-C_i]_+}{C^0}, \qquad\Delta_{i,+} = \frac{[C_i-C^0]_+}{1-C^0}. \tag*{(2)}

Loss is normalized by reference coverage and gain by the remaining headroom, accounting for differences in initial coverage across images. We define a coverage violation and a retention gate:

dicov=12+12Δi,−−12Δi,+,gicov=1−Δi,−.(3)d_i^{\mathrm{cov}} = \frac{1}{2} + \frac{1}{2}\sqrt{\Delta_{i,-}} - \frac{1}{2}\Delta_{i,+}, \qquad g_i^{\mathrm{cov}} = 1-\Delta_{i,-}. \tag*{(3)}

For equal normalized changes, the square-root penalty gives greater weight to coverage loss than the linear reward gives to coverage expansion, prioritizing the retention of grounded information already captured in the reference response.

Response quality and reward composition. To discourage overly brief or repetitive responses during hallucination mitigation, we introduce a response-quality gate. It combines the word counts LiL_i and L0L^0 of the sampled and reference responses with reference-relative word nn-gram diversity retention δidiv\delta_i^{\mathrm{div}}:

qiresp=(1−[1−LiβL0]+)(1−1−δidiv).(4)q_i^{\mathrm{resp}} = \left(1-\sqrt{\left[1-\frac{L_i}{\beta L^0}\right]_+}\right)\left(1-\sqrt{1-\delta_i^{\mathrm{div}}}\right). \tag*{(4)}

The length factor saturates at βL0\beta L^0, while the square roots amplify shortfalls in length and diversity. Appendix § 7 defines the reference-relative diversity retention.

The retention gate couples factuality to information preservation: d~ihall=1−gicovrihall\tilde{d}_i^{\mathrm{hall}} = 1-g_i^{\mathrm{cov}}r_i^{\mathrm{hall}} discounts factuality when coverage falls below the reference. We combine this violation with coverage and response quality:

riRL=qiresp(1−(d~ihall)2+(dicov)22).(5)r_i^{\mathrm{RL}} = q_i^{\mathrm{resp}}\left(1-\sqrt{\frac{(\tilde{d}_i^{\mathrm{hall}})^2+(d_i^{\mathrm{cov}})^2}{2}}\right). \tag*{(5)}

Squaring emphasizes the larger content violation, while the outer square root preserves the scale of the individual penalties. This encourages the policy to improve factuality while preserving existing grounded information and recovering additional correct visual details.

Hallucination-Aware Trajectory Routing

We obtain teacher guidance from the rollout-policy snapshot conditioned on grounding context zz, comprising object annotations and reference captions already available for the COCO-Stuff images: πpriv(⋅∣x,z,y<t)=πold(⋅∣x,z,y<t)\pi_{\mathrm{priv}}(\cdot\mid x,z,y_{<t}) = \pi_{\mathrm{old}}(\cdot\mid x,z,y_{<t}). Comparing token probabilities with and without zz at the same sampled prefix measures how additional grounding evidence changes support for each token in the student’s response. The composite reward evaluates coverage and response quality alongside factuality, and its advantage can differ in sign from the hallucination-specific advantage. We apply teacher guidance when these advantages agree in sign, aligning its use with the purpose of the response-level update. To determine this agreement, we separately normalize riRLr_i^{\mathrm{RL}} and rihallr_i^{\mathrm{hall}} within each rollout group:

Aik=rik−μkσk,k∈{RL,hall},(6)A_i^k = \frac{r_i^k-\mu_k}{\sigma_k}, \qquad k \in\{\mathrm{RL},\mathrm{hall}\}, \tag*{(6)}

where μk\mu_k and σk\sigma_k are the group mean and standard deviation. The route sis_i selects teacher-guided reinforcement, teacher-guided suppression, or uniform attribution:

si={+1,AiRL>0∧Aihall>0,−1,AiRL<0∧Aihall<0,0,otherwise,(7)s_i = \begin{cases} +1, & A_i^{\mathrm{RL}} > 0 \land A_i^{\mathrm{hall}} > 0,\\ -1, & A_i^{\mathrm{RL}} < 0 \land A_i^{\mathrm{hall}} < 0,\\ 0, & \text{otherwise}, \end{cases} \tag*{(7)}

A response with above-average factuality can still receive a negative composite advantage because of insufficient coverage or poor response quality. Grounding-specific suppression would then emphasize a different property from the one driving the penalty. Conversely, gains in coverage or response quality can yield a positive composite advantage despite below-average factuality. The composite reward determines whether to reinforce or suppress a response; agreement determines whether grounding-specific teacher support is appropriate for allocating that update across tokens. Neutral routing preserves the response-level update with uniform token weights when the two assessments disagree.

Advantage-Aware Token Credit Assignment

Teacher support should have complementary effects across update directions: stronger support should amplify reinforcement but attenuate suppression. Direct weighting achieves the former, yet imposes larger penalties on better-supported tokens under a negative advantage. Reciprocal weighting reverses this allocation for suppression.

On-policy teacher support. For each token in the student’s sampled response, we compare the two distributions at the same prefix to assess its grounding support:

ρi,t=πpriv(yi,t∣x,z,yi,<t)πold(yi,t∣x,yi,<t).(8)\rho_{i,t} = \frac{\pi_{\mathrm{priv}}(y_{i,t} \mid x,z,y_{i,<t})}{\pi_{\mathrm{old}}(y_{i,t} \mid x,y_{i,<t})}. \tag*{(8)}

The ratio measures how grounding evidence changes support for a token relative to the policy’s ordinary preference for it. This contrast supplies a token-level grounding signal even for responses whose overall advantage is negative. Scoring the student’s current rollouts aligns teacher guidance with its evolving generation behavior, enabling selective reinforcement and suppression within its own responses.

Let TiT_i be the number of tokens in response yiy_i. We stabilize the support ratios by clipping ρi,t\rho_{i,t} to [1/c,c][1/c,c] with bound c>1c > 1 and centering its logarithm within each response [38]:

ai,t=log⁡ρˉi,t−1Ti∑u=1Tilog⁡ρˉi,u,ρˉi,t=clip⁡(ρi,t,1/c,c).(9)a_{i,t} = \log\bar{\rho}_{i,t} - \frac{1}{T_i}\sum_{u=1}^{T_i}\log\bar{\rho}_{i,u}, \qquad\bar{\rho}_{i,t} = \operatorname{clip}(\rho_{i,t},1/c,c). \tag*{(9)}

Log-centering expresses support relative to the response’s geometric mean, providing a shared reference for reinforcement and suppression when assigning credit across tokens.

Direct and reciprocal weighting. Using the route sis_i from Eq. (7) and centered support ai,ta_{i,t}, we define the token weight and effective advantage as

wi,t=exp⁡(siai,t),Ai,teff=AiRLwi,t.(10)w_{i,t} = \exp(s_i a_{i,t}), \qquad A^{\mathrm{eff}}_{i,t} = A^{\mathrm{RL}}_i w_{i,t}. \tag*{(10)}

On positive routes, direct weighting assigns greater credit to tokens with stronger relative teacher support. On negative routes, reciprocal weighting concentrates penalties on less-supported tokens, allowing unfavorable responses to provide corrective feedback while reducing suppression of their better-supported content. Neutral routes preserve the original RL update with uniform token weights.

Policy Optimization

Let ui,t(θ)=πθ(yi,t∣x,yi,<t)/πold(yi,t∣x,yi,<t)u_{i,t}(\theta) = \pi_\theta(y_{i,t} \mid x,y_{i,<t})/\pi_{\mathrm{old}}(y_{i,t} \mid x,y_{i,<t}) denote the importance ratio between the updated policy πθ\pi_\theta and the rollout policy. We incorporate the effective token advantages into the clipped group-relative policy objective [35, 36]:

ℓi,t=min⁡(ui,tAi,teff,clip⁡(ui,t,1−ϵ,1+ϵ)Ai,teff),Lpolicy(θ)=−E[1G∑i=1G1Ti∑t=1Tiℓi,t],(11)\ell_{i,t} = \min\left(u_{i,t}A^{\mathrm{eff}}_{i,t},\operatorname{clip}(u_{i,t},1-\epsilon,1+\epsilon)A^{\mathrm{eff}}_{i,t}\right), \\ \mathcal{L}_{\mathrm{policy}}(\theta) = -\mathbb{E}\left[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{T_i}\sum_{t=1}^{T_i}\ell_{i,t}\right], \tag*{(11)}

Teacher support enters the objective through reward-conditioned token weights, refining the allocation of an RL update without adding a separate distribution-matching objective. The response advantage determines the update direction, while the support weights determine which tokens receive greater emphasis. This aligns teacher guidance with reward-driven reinforcement and suppression.

Experiments

Experimental Setup

Models and datasets. We conduct experiments on Qwen3-VL-4B-Instruct, Qwen3-VL-8B-Instruct [2], and Gemma3-4B-IT [15]. All three models are trained on the same 10,000 COCO-Stuff images with accompanying visual grounding information [3, 27]. The training set is strictly disjoint from the image sets used for evaluation across all benchmarks.

Evaluation benchmarks. We assess object hallucination on Obj-Hal using response-level CHAIRs\mathrm{CHAIR}_s, object-level CHAIRi\mathrm{CHAIR}_i [34], and object recall. On AMBER [40], we report CHAIR, hallucination rate (HalRate), and coverage (Cover). THRONE [20] evaluates object-presence decisions with classwise precision, recall, and F0.5F_{0.5}; @5 and @9 denote majority and unanimous agreement among nine votes. We further use GPT-4-assisted evaluation [50] to assess fine-grained hallucinations in generated image descriptions through the Sentence-level Hallucination Ratio (SHR). General multimodal capabilities on Qwen3-VL are evaluated using the total MME score [13] and MMStar accuracy [4].

Baselines. We compare with the original backbone (Vanilla), OPSD [49], GRPO [36], and OPSD+GRPO. OPSD applies on-policy teacher–student distribution matching. GRPO uses the same response reward in Eq. (5) with uniform token weights. OPSD+GRPO combines the two losses additively. We also compare with strong training-based baselines, including mDPO [39], OPA-DPO [44], and SymMPO [30], trained following their original settings.

Implementation details. We train all three backbones for one epoch using AdamW with a learning rate of 10−610^{-6} and an effective batch of 512 response sequences. Each rollout samples G=8G = 8 responses per image with temperature 1.0 and top-p=0.95p = 0.95. Reference descriptions for reward calibration are generated by the initial backbone and cached before training. We set the hallucination mixture coefficient α=0.7\alpha= 0.7, reference scaling factor β=0.35\beta= 0.35, teacher-ratio clipping bound c=3c = 3, and policy clipping margin ϵ=0.2\epsilon= 0.2. Diversity retention uses word 1–4-grams. To reduce sampling-induced variance and maintain consistent comparisons across benchmarks, we use greedy decoding for all model responses during evaluation. Appendix § 6 provides additional training details and benchmark descriptions.

Hallucination Mitigation and Information Retention

Factuality and coverage improve together. As shown in Table 1, SARGO achieves lower AMBER CHAIR and higher coverage than all evaluated baselines across all three backbones. On Qwen3-VL-4B, for example, CHAIR decreases from OPA-DPO’s 5.9 to 3.9, while coverage increases from 66.6 to 76.2. On Qwen3-VL-4B and Qwen3-VL-8B, SymMPO achieves a similar HalRate to SARGO but has substantially higher CHAIR and lower coverage. Strong training-based baselines such as OPA-DPO and SymMPO reduce hallucination on Qwen3-VL with coverage below Vanilla, suggesting gains partly at the expense of grounded information. SARGO lowers CHAIR while raising coverage above Vanilla across all three backbones. The Obj-Hal results show a similar pattern: SARGO obtains the lowest CHAIRs_s and CHAIRi_i across all three backbones, with object recall ranking first on Qwen3-VL-8B and second on Qwen3-VL-4B and Gemma3-4B. Relative to Vanilla, CHAIRi_i decreases by 40.6%–49.1% across the three backbones. On Gemma3-4B, SARGO lowers OPA-DPO’s CHAIRi_i from 7.8 to 5.7 and increases object recall from 68.8 to 72.2.

ModelMethodObj-Hal CHAIRs_s ↓Obj-Hal CHAIRi_i ↓Obj-Hal Recall↑AMBER CHAIR↓AMBER HalRate↓AMBER Cover↑THRONE F0.5_{0.5} @5↑THRONE F0.5_{0.5} @9↑GPT-4-assisted SHR↓
Qwen3-VL-4BVanilla55.010.375.78.060.874.274.281.234.5
Qwen3-VL-4BOPSD55.710.174.87.660.374.473.881.233.2
Qwen3-VL-4BGRPO45.310.375.37.246.471.576.586.025.2
Qwen3-VL-4BOPSD+GRPO54.311.976.39.660.273.574.983.530.6
Qwen3-VL-4BmDPO48.39.073.06.444.168.876.285.230.8
Qwen3-VL-4BOPA-DPO44.38.771.55.938.566.675.284.031.2
Qwen3-VL-4BSymMPO44.09.072.76.036.566.977.386.128.8
Qwen3-VL-4BSARGO34.75.875.93.936.776.282.388.013.7
Qwen3-VL-8BVanilla54.710.174.17.860.574.773.781.230.3
Qwen3-VL-8BOPSD56.710.573.97.962.575.172.881.329.6
Qwen3-VL-8BGRPO44.78.377.06.343.374.277.984.125.8
Qwen3-VL-8BOPSD+GRPO53.311.575.28.855.772.975.184.128.3
Qwen3-VL-8BmDPO48.38.672.06.543.668.876.085.128.2
Qwen3-VL-8BOPA-DPO45.78.871.36.951.571.873.180.927.3
Qwen3-VL-8BSymMPO43.39.172.76.339.368.276.985.727.8
Qwen3-VL-8BSARGO35.36.077.84.740.176.380.986.415.7
Gemma3-4BVanilla54.311.271.16.854.068.771.285.524.8
Gemma3-4BOPSD51.310.371.06.752.368.571.785.323.6
Gemma3-4BGRPO47.09.272.66.646.769.572.786.919.7
Gemma3-4BOPSD+GRPO52.011.471.79.159.769.571.384.622.4
Gemma3-4BmDPO47.78.969.76.552.869.072.485.722.0
Gemma3-4BOPA-DPO43.77.868.85.543.065.773.487.021.6
Gemma3-4BSymMPO46.09.271.66.653.268.772.586.122.6
Gemma3-4BSARGO30.75.772.25.235.670.277.088.317.8

Table 1. Comparison of SARGO and baseline methods on hallucination benchmarks using Qwen3-VL-4B, Qwen3-VL-8B, and Gemma3-4B. THRONE reports classwise F0.5_{0.5} at majority (@5) and unanimous (@9) agreement. All scores are percentages. Bold and underlined values indicate the best and second-best results within each backbone, respectively.

Integrating on-policy self-guidance. Under the same response reward, adding OPSD’s distribution-matching loss to GRPO increases object hallucination on all three backbones. AMBER CHAIR rises from 7.2 to 9.6 on Qwen3-VL-4B, from 6.3 to 8.8 on Qwen3-VL-8B, and from 6.6 to 9.1 on Gemma3-4B. Obj-Hal CHAIRi_i worsens in all three cases as well. Both hallucination metrics are also higher than Vanilla, showing that additive distribution matching fails to preserve GRPO’s factuality gains in this setting. SARGO improves the joint outcome, achieving lower CHAIR and higher AMBER coverage than OPSD+GRPO across all three backbones. On Qwen3-VL-8B, it reduces OPSD+GRPO’s CHAIRi_i from 11.5 to 6.0 while increasing recall from 75.2 to 77.8. On both 4B backbones, it approximately halves OPSD+GRPO’s CHAIRi_i while maintaining comparable object recall. AMBER coverage increases over OPSD+GRPO by 2.7, 3.4, and 0.7 points on Qwen3-VL-4B, Qwen3-VL-8B, and Gemma3-4B, respectively. These comparisons highlight the importance of how teacher guidance enters RL: SARGO uses the response advantage to determine the update direction and teacher support to distribute reinforcement and suppression across tokens.

Grounding gains extend to THRONE. SARGO achieves the highest reported THRONE F0.5_{0.5} at both voting thresholds across all three backbones. At majority agreement, it exceeds SymMPO, the strongest Qwen3-VL-4B baseline, by 5.0 points. On Gemma3-4B, the gains over OPA-DPO are 3.6 points at @5 and 1.3 points at @9, showing improved object-presence judgments under both agreement criteria.

Improved sentence-level factuality. As shown in Table 1, SARGO achieves the lowest SHR among all evaluated meth-

Effects of token weighting and advantage routingVector (PDF)

caption: Effects of token weighting and advantage routing on Qwen3-VL-4B. All variants share the same response reward. No weighting uses uniform token weights. Positive only and Negative only apply teacher weighting in one branch, retaining uniform weights in the other. RL-only routing uses the overall advantage alone to select the weighting branch. Full denotes SARGO.

ods across the three backbones. Compared with Vanilla, SHR decreases from 34.5 to 13.7 on Qwen3-VL-4B, from 30.3 to 15.7 on Qwen3-VL-8B, and from 24.8 to 17.8 on Gemma3-4B. These results extend the observed gains from object-level grounding to sentence-level factuality.

Routing and Selective Token Attribution

Figure 3 compares five Qwen3-VL-4B settings under the same reference-calibrated reward. No weighting isolates token attribution; the single-direction variants retain ordinary RL in the opposite direction. RL-only routing retains both branches and tests whether the composite advantage alone can select the teacher’s role.

Token attribution improves response-level optimization. With the same response reward, Full lowers GRPO’s Obj-Hal CHAIRi_i from 10.3 to 5.8 and AMBER CHAIR from 7.2 to 3.9, while coverage rises from 71.5 to 76.2. Token

attribution thus improves both factuality and information retention beyond response-level optimization.

Negative attribution strengthens hallucination suppression. Adding the negative branch to positive-only weighting lowers AMBER CHAIR from 5.2 to 3.9 and HalRate from 47.9 to 36.7, with coverage changing from 77.0 to 76.2. Since the positive branch and response reward are retained, this comparison isolates the benefit of allocating negative

Positive attribution preserves grounded information. Relative to negative-only weighting, adding the positive branch lowers Obj-Hal CHAIRi_i from 6.9 to 5.8 while increasing recall from 74.4 to 75.9. AMBER CHAIR and coverage also improve. The single-branch variants expose a trade-off: positive-only weighting retains more objects, whereas negative-only weighting yields lower AMBER CHAIR. Their combination achieves lower CHAIR scores than either variant, with positive attribution recovering grounded information beyond negative-only updates.

THRONE provides a complementary view of the two branches. Positive-only weighting has a slightly higher F0.5F_{0.5} than negative-only weighting at majority agreement, whereas negative-only weighting performs better at unanimous agreement. Their combination achieves the highest score at both thresholds, reaching 82.3 at @5 and 88.0 at @9. The combined benefit extends beyond the CHAIR–recall comparison to object-presence judgments, with neither single branch matching the full model across the two agreement criteria.

Advantage agreement selects when to apply guidance. RL-advantage-only routing retains both direct and reciprocal weights, replacing Eq. (7) with siRL=sign⁡(AiRL)s_i^{\mathrm{RL}} = \operatorname{sign}(A_i^{\mathrm{RL}}). Restoring hallucination-advantage agreement improves every reported Obj-Hal and AMBER metric and both THRONE scores. CHAIRi_i decreases from 7.4 to 5.8 and AMBER CHAIR from 5.1 to 3.9, relative reductions of 21.6% and 23.5%. THRONE F0.5F_{0.5} rises from 79.1 to 82.3 at @5 and from 85.3 to 88.0 at @9. With identical weighting formulas, these gains isolate the value of requiring factuality agreement when allocating teacher-guided updates.

Reward Design and Information Retention

Table 2 isolates reward components on Qwen3-VL-4B. Coverage rewards retention and gains relative to each image’s reference; removing it lowers AMBER coverage from 76.2 to 62.3 and Obj-Hal recall from 75.9 to 68.3. The coverage gate makes factuality rewards depend on retained information: without it, AMBER coverage falls to 69.8. Both removals also raise AMBER CHAIR from 3.9 to 4.1. Their identical CHAIR scores accompany different losses of correct information, illustrating why hallucination and coverage need to be assessed together.

MethodObj-Hal CHAIRs_s ↓Obj-Hal CHAIRi_i ↓Obj-Hal Recall↑AMBER CHAIR↓AMBER HalRate↓AMBER Cover↑THRONE F0.5@5F_{0.5}@5↑THRONE F0.5@9F_{0.5}@9↑
SARGO34.75.875.93.936.776.282.388.0
w/o response quality11.03.072.71.76.858.082.987.8
w/o hallucination51.010.479.95.649.577.279.485.8
w/o coverage30.75.468.34.135.362.381.086.4
w/o coverage gate35.06.070.44.137.569.880.886.5

Table 2. Ablation study of SARGO’s response reward on Qwen3-VL-4B. Each variant removes one component from the full reward.

The two coverage mechanisms are complementary. Once a response matches the reference coverage, the retention gate saturates, while the explicit coverage term continues to reward further correct discoveries through its gain component. Removing this term preserves the gate’s protection against information loss but eliminates the incentive for coverage expansion, consistent with the larger drops in AMBER coverage and Obj-Hal recall.

The hallucination term suppresses unsupported content. Removing it raises AMBER coverage to 77.2 and Obj-Hal recall to 79.9, but AMBER CHAIR rises to 5.6 and Obj-Hal CHAIRi_i to 10.4. SARGO combines AMBER CHAIR 3.9 with coverage 76.2 and achieves the highest THRONE F0.5F_{0.5} of 88.0 at @9, rewarding informative, grounded responses.

Response quality, qirespq_i^{\mathrm{resp}}, favors informative responses. Removing this gate produces extremely short answers with little useful visual content: AMBER CHAIR drops to 1.7 while coverage falls to 58.0. The lower hallucination score accompanies substantial content omission, making the response-quality gate important for retaining informative outputs.

Preserving General Multimodal Capabilities

Figure 4 shows that SARGO achieves the highest MME scores at both Qwen3-VL scales, improving Vanilla by 24 points on 4B and 12 on 8B. MMStar accuracy rises from 55.4 to 56.8 on 4B and from 60.3 to 61.6 on 8B, ranking second at both scales behind OPA-DPO. On 8B, SARGO achieves MME 2398 and MMStar 61.6, exceeding GRPO’s 2392 and 61.1. GRPO falls below Vanilla on 4B MMStar, while OPA-DPO leads on MMStar but falls below Vanilla on 8B MME. SARGO is the only evaluated method to improve over Vanilla on both benchmarks at both Qwen3-VL scales.

Figure 4. General multimodal performance on Qwen3-VL-4B and Qwen3-VL-8B. Comparison on (a) MME total score and (b) MMStar accuracy (%). SARGO improves both metrics over Vanilla at both scales.

These results show that stronger hallucination suppression can coexist with preserved general multimodal capabilities.

Conclusion

This work investigates visual hallucination mitigation through response-level reward design and token-level credit assignment. We propose SARGO, which uses on-policy teacher support to selectively reinforce grounded content and suppress unsupported claims. Direct and reciprocal weighting adapt this guidance to positive and negative updates, while hallucination-aware routing aligns its use with the purpose of each update. A reference-calibrated reward complements these updates by jointly valuing factuality, information retention, and coverage gains. Experiments on Qwen3-VL-4B, Qwen3-VL-8B, and Gemma3-4B demonstrate joint improvements in hallucination mitigation and information retention. Evaluations on Qwen3-VL further show improved general multimodal capabilities. These findings highlight the importance of coordinating selective reinforcement and suppression with reward objectives that preserve and extend grounded visual information.

Experimental Details

Training Configuration

We train Qwen3-VL-4B, Qwen3-VL-8B [2], and Gemma3-4B [15] on the same 10,000 COCO image–instruction examples with accompanying grounding annotations [27]. COCO-Stuff [3] supplies semantic labels for scene and background regions, which complement object annotations and reference captions in the teacher’s grounding context. Training and evaluation use disjoint image sets. All three backbones are trained for one epoch using AdamW with a learning rate of 10−610^{-6} and an effective batch of 512 response sequences. Each rollout samples G=8G = 8 responses per image with temperature 1.0 and top-p=0.95p = 0.95. Reference responses from the initial model are cached before training.

Evaluation Benchmarks

Object HalBench. Object HalBench (Obj-Hal) assesses whether objects mentioned in generated descriptions are supported by image annotations. We report CHAIRs\mathrm{CHAIR}_{s} and CHAIRi\mathrm{CHAIR}_{i} [34], which measure the fraction of responses containing hallucinated objects and the fraction of object mentions that are hallucinated, respectively. Object recall also measures the recovery of annotated objects.

AMBER. AMBER [40] evaluates hallucinations in generative and discriminative settings. We use its generative setting and report CHAIR, HalRate, and Cover. CHAIR measures the proportion of hallucinated objects, while HalRate measures how often a response contains hallucination. Cover measures the recovery of annotated objects, capturing the amount of grounded information retained in a response.

THRONE. Following the original THRONE protocol [20], object-presence predictions are assessed against COCO 2017 validation annotations covering 80 object categories. Object-presence judgments are obtained from FLAN-T5 base, large, and XL with three probes each, yielding nine votes. We report classwise F0.5F_{0.5} at majority (@5) and unanimous (@9) agreement. This score combines precision and recall with greater emphasis on precision than recall. Scores are macro-averaged over object categories.

GPT-4-assisted evaluation. Following Zhao et al. [50], we assess fine-grained hallucinations using Visual Genome annotations [23] and GPT-4o-mini. The evaluator compares generated descriptions with the annotated visual content and identifies hallucinated statements. We report the Sentence-level Hallucination Ratio (SHR) as a percentage. This evaluation complements object-based metrics by assessing factual consistency at the sentence level.

Training Procedure and Prompts

On-Policy Training

Algorithm 1 summarizes the core policy optimization. The frozen initial policy π0\pi_0 supplies cached references for reward calibration. The current policy snapshot πold\pi_{\mathrm{old}} is refreshed for each training batch and serves both as the rollout policy and as the teacher under privileged conditioning.

Grounding Context and Teacher Prompt

The teacher receives the student’s image and instruction together with COCO object labels, human reference captions, and available COCO-Stuff scene cues. Figure 5 shows the grounding prompt, whose placeholders are filled with the evidence for each image. The scene-related portion is omitted when that evidence is unavailable.

Privileged Grounding Prompt

Figure 5. Privileged Grounding Prompt [Privileged Grounding Evidence] Use the following information only as private grounding evidence for answering the original request. Verified COCO Object Annotations {object labels} Verified Scene and Background Regions {selected scene labels, when available} Human Observations of the Same Image 1. {reference caption} … Answer the original request directly. Teacher prompt for on-policy self-guidance. Image-specific object labels, scene cues, and reference captions provide grounding context for scoring the current policy’s sampled responses.

Algorithm 1 Core policy optimization in SARGO

Require: Training set D={(x,O(x),z)}\mathcal{D} = \{(x, \mathcal{O}(x), z)\}; initial policy π0\pi_0; group size GG; parameters α\alpha, β\beta, cc, ϵ\epsilon

Ensure: Trained policy πθ\pi_\theta

  1. Freeze π0\pi_0 and initialize πθ←π0\pi_\theta\leftarrow\pi_0

  2. Cache reference y0(x)y^0(x) and its calibration statistics from π0\pi_0 for every training input xx

  3. for each training batch B⊂DB \subset\mathcal{D} do

  4. πold←πθ\pi_{\mathrm{old}} \leftarrow\pi_\theta

  5. for each (x,O(x),z)∈B(x, \mathcal{O}(x), z) \in B do

  6. Sample GG student responses {yi}i=1G∼πold(⋅∣x)\{y_i\}_{i=1}^{G} \sim\pi_{\mathrm{old}}(\cdot\mid x)

  7. Compute hallucination rewards rihallr_i^{\mathrm{hall}} from yiy_i and O(x)\mathcal{O}(x) for all ii

  8. Calibrate coverage and response quality against cached y0(x)y^0(x); compose rewards riRLr_i^{\mathrm{RL}}

  9. Aik←(rik−μk)/σkA_i^k \leftarrow(r_i^k - \mu_k)/\sigma_k for k∈{RL,hall}k \in\{\mathrm{RL},\mathrm{hall}\}, using this input’s response group

  10. for each response yiy_i do

  11. si←sign⁡(AiRL)I[Aihall>0]s_i \leftarrow\operatorname{sign}(A_i^{\mathrm{RL}})\mathbb{I}[A_i^{\mathrm{hall}} > 0]

  12. pi,tord←πold(yi,t∣x,yi,<t)p_{i,t}^{\mathrm{ord}} \leftarrow\pi_{\mathrm{old}}(y_{i,t} \mid x, y_{i,<t}), pi,tpriv←πold(yi,t∣x,z,yi,<t)p_{i,t}^{\mathrm{priv}} \leftarrow\pi_{\mathrm{old}}(y_{i,t} \mid x, z, y_{i,<t}) for all tokens tt

  13. ρˉi,t←clip⁡(pi,tpriv/pi,tord,1/c,c)\bar{\rho}_{i,t} \leftarrow\operatorname{clip}(p_{i,t}^{\mathrm{priv}}/p_{i,t}^{\mathrm{ord}}, 1/c, c)

  14. ai,t←log⁡ρˉi,t−Ti−1∑u=1Tilog⁡ρˉi,ua_{i,t} \leftarrow\log\bar{\rho}_{i,t} - T_i^{-1}\sum_{u=1}^{T_i}\log\bar{\rho}_{i,u}

  15. wi,t←exp⁡(siai,t)w_{i,t} \leftarrow\exp(s_i a_{i,t}), Ai,teff←AiRLwi,tA_{i,t}^{\mathrm{eff}} \leftarrow A_i^{\mathrm{RL}}w_{i,t}

  16. end for

  17. end for

  18. Compute the clipped policy loss Lpolicy\mathcal{L}_{\mathrm{policy}} with effective token advantages and ratio πθ/πold\pi_\theta/\pi_{\mathrm{old}}

  19. Update θ\theta with AdamW on Lpolicy\mathcal{L}_{\mathrm{policy}}

  20. end for

  21. return πθ\pi_\theta

Object extraction. Object mentions are mapped to distinct COCO categories using a category lexicon and lemmatization. The reward annotation set combines instance labels with recognized categories from human captions.

Response-Quality Gate

The response-quality gate combines sufficient content with diversity retention relative to the initial policy. Let Gn(y)\mathcal{G}_n(y) be the multiset of word nn-grams in response yy. The fraction of distinct nn-grams is

Vi(n)=∣Unique⁡(Gn(yi))∣∣Gn(yi)∣.(12)V_i^{(n)} = \frac{\left|\operatorname{Unique}(\mathcal{G}_n(y_i))\right|}{\left|\mathcal{G}_n(y_i)\right|}. \tag*{(12)}

For each n∈{1,2,3,4}n \in\{1, 2, 3, 4\}, diversity retention relative to the cached reference y0y^0 is

δi(n)={1,∣Gn(y0)∣=0,min⁡(Vi(n)V0(n),1),otherwise.(13)\delta_i^{(n)} = \begin{cases} 1, & \left|\mathcal{G}_n(y^0)\right| = 0,\\ \min\left(\dfrac{V_i^{(n)}}{V_0^{(n)}}, 1\right), & \text{otherwise.} \end{cases} \tag*{(13)}

Orders absent from the reference contribute neutral retention. Combining the four orders gives δidiv=∏n=14δi(n)\delta_i^{\mathrm{div}} = \prod_{n=1}^{4}\delta_i^{(n)}, which enters the response-quality gate as

qiresp=gilen(1−1−δidiv),(14)q_i^{\mathrm{resp}} = g_i^{\mathrm{len}}\left(1-\sqrt{1-\delta_i^{\mathrm{div}}}\right), \tag*{(14)}
gilen=1−[1−LiβL0]+.g_i^{\mathrm{len}} = 1-\sqrt{\left[1-\frac{L_i}{\beta L^0}\right]_+}.

Combining multiple orders accounts for repeated words and longer recurring phrases. This gate reduces the reward of repetitive or insufficient responses, complementing the factuality and coverage terms that evaluate their visual content.

General Multimodal Capabilities

Table 3 gives the numerical results underlying the capability comparison in Figure 4. SARGO ranks first on MME and second on MMStar at both model scales, improving both metrics over Vanilla.

MethodQwen3-VL-4B MME↑Qwen3-VL-4B MMStar↑Qwen3-VL-8B MME↑Qwen3-VL-8B MMStar↑
Vanilla232155.4238660.3
OPSD233755.2236860.2
GRPO234055.0239261.1
OPSD+GRPO222853.9237460.1
mDPO230154.3234656.8
OPA-DPO233857.4238263.5
SymMPO233955.0235961.0
SARGO2345<u>56.8</u>2398<u>61.6</u>

Table 3. General multimodal results on MME and MMStar. MME reports total scores and MMStar reports accuracy (%). Bold and underlined values denote the best and second-best results within each backbone, respectively.

Case Study

We provide qualitative comparisons among OPSD, GRPO, SymMPO, and SARGO on six hallucination examples in Figure 6–Figure 8. These qualitative examples reflect the overall performance gap observed in our quantitative evaluations: compared with SARGO, the baselines produce more hallucinated content, including nonexistent objects and unsupported background details, while SARGO reduces such errors without sacrificing grounded visual information, resulting in descriptions that are both more faithful and informative.

Figure 6. Describe this image.

OPSD: … The image captures a joyful moment at a beach. A white dog is running excitedly across the sand, with a rocky shoreline and a bright blue sky in the background. … There are large, light-colored rocks scattered across the sand and along the shoreline. …

GRPO: … The photo captures a joyful moment at a beach. … The scene is a sandy beach with a gray, slightly overcast sky. There are several large, gray rocks scattered across the sand, adding texture and visual interest. …

SymMPO: … A white dog is in mid-stride, running towards the viewer with a playful expression. … There are large, light-colored rocks scattered across the sand, adding texture and visual interest. … The background shows a glimpse of the ocean with a light blue sky. …

SARGO: … The image shows a white dog running excitedly on a sandy beach. … Several large rocks are scattered around the beach, and the dog is near a body of water (likely the ocean) given the presence of the sand and the beach setting. …

Describe this image in detail.

OPSD: … A narrow, asphalt road curves gently. It’s relatively empty, with only a few parked cars visible. … A few scattered items – a bicycle, a discarded can – are visible on the pavement. …

GRPO: … A line of bare trees dominates the mid-ground, their branches silhouetted against the sky. They suggest it’s winter or early spring. … A few scattered items – a bicycle leaning against the wall, and a single shoe – are visible on the pavement. …

SymMPO: … A paved road curves gently to the left, suggesting a residential street. It appears slightly worn and has a few scattered items – a bicycle leaning against the wall, and what looks like a discarded shoe. …

SARGO: … A road curves into the frame, bordered by a brick wall that runs along the side of the street. … A suitcase, heavily laden with what appears to be trash and various items, is prominently placed on the pavement near the wall, suggesting it has been discarded or left behind. …

Visualization results comparing OPSD, GRPO, SymMPO, and SARGO. Hallucinations are marked in red.

Figure 8. Visualization results comparing OPSD, GRPO, SymMPO, and SARGO. Hallucinations are marked in red.

Impact Statement

This work aims to improve the reliability of vision-language models by reducing unsupported claims while preserving useful visual information. More faithful multimodal responses can support applications such as visual assistance and image-grounded information access, where hallucinated content may mislead users.

Figure 7. OPSD: … Two adult polar bears are the primary focus. They are standing on a large, undulating sheet of ice, which appears to be part of a larger ice floe or glacier. … The background is a pale blue sky, suggesting a cold, arctic environment. … GRPO: … The image captures a serene and beautiful scene in the Arctic. It features two polar bears standing on a vast expanse of icy terrain. … The background is a pale, hazy blue sky, typical of Arctic conditions. It creates a sense of distance and vastness. … SymMPO: … Two adult polar bears are the focal point of the image. They are standing on a sheet of ice that appears to be part of a larger, fragmented ice floe. … The background is a pale blue sky, suggesting a clear, cold day. The horizon is slightly blurred, giving a sense of depth. … SARGO: … The image shows two polar bears standing on a large expanse of snow and ice, likely a frozen sea or a large ice floe. The bears are positioned on the ice, and there is a body of water visible in the background. … OPSD: … The surrounding environment is a bustling urban street lined with lush, leafy trees. Pedestrians are scattered along the sidewalks, some walking, others standing, creating a sense of everyday city life. A few individuals are seen holding umbrellas, suggesting a possible drizzle or the need for shade. … GRPO: … On the right, a crowd of people is gathered, possibly waiting for the bus or simply observing the street. A few individuals are holding umbrellas, suggesting a possible drizzle or the presence of rain. … SymMPO: … On the right side of the image, a group of pedestrians is gathered near the curb, some holding umbrellas, suggesting a possible rain shower or simply a typical urban scene. Their clothing and posture indicate they are waiting or observing the traffic. … SARGO: … A red bus is visible on a road with trees and a sidewalk, with people standing near the pavement and a car visible further down the street. The bus is positioned on the road, with its rear facing the viewer, and appears to be moving along a paved surface that includes visible lane markings. … Figure 7. Visualization results comparing OPSD, GRPO, SymMPO, and SARGO. Hallucinations are marked in red.

References

  1. [1]Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In International Conference on Learning Representations, 2024.arxiv.org/abs/2306.13649
  2. [2]Shuai Bai, Yuxuan Cai, Ruize Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan Liu, Dunjie Lu, Ruilin Luo, Chenxu Lv, Rui Men, Lingchen Meng, Xuancheng Ren, Xingzhang Ren, Sibo Song, Yuchong Sun, Jun Tang, Jianhong Tu, Jianqiang Wan, Peng Wang, Pengfei Wang, Qiuyue Wang, Yuxuan Wang, Tianbao Xia, Yiheng Xu, Haiyang Xu, Jin Xu, Zhibo Yang, Mingkun Yang, Jianxin Yang, An Yang, Bowen Yu, Fei Zhang, Hang Zhang, Xi Zhang, Bo Zheng, Humen Zhong, Jingren Zhou, Fan Zhou, Jing Zhou, Yuanzhi Zhu, and Ke Zhu. Qwen3-VL technical report. arXiv preprint arXiv:2511.21631, 2025.arxiv.org/abs/2511.21631
  3. [3]Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. COCO-Stuff: Thing and stuff classes in context. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1209–1218, 2018.arxiv.org/abs/1612.03716
  4. [4]Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zhang, Zehui Chen, Haodong Duan, Jiaq Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are we on the right way for evaluating large vision-language models? In Advances in Neural Information Processing Systems, 2024.arxiv.org/abs/2403.20330
  5. [5]Zhaoruan Chen, Chuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. HALC: Object hallucination reduction via adaptive focal-contrast decoding. In Proceedings of the 41st International Conference on Machine Learning, pages 7824–7846, 2024.arxiv.org/abs/2403.00425
  6. [6]Ganqiu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Yuchen Zhang, Jiacheng Chen, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qin Xu, Weize Chen, Jiarui Yuan, Huayu Chen, Kaiyan Zhang, XingtaI Lv, Shuo Wang, Yuan Yao, Xu Han, Hao Peng, Yu Cheng, Zhiyuan Liu, Maosong Sun, Bowen Zhou, and Ning Ding. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456, 2025.arxiv.org/abs/2502.01456
  7. [7]Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. In Advances in Neural Information Processing Systems, pages 49250–49267, 2023.
  8. [8]Bowen Dong, Minheng Ni, Zitong Huang, Guanglei Yang, Wangmeng Zuo, and Lei Zhang. MIRAGE: Assessing hallucination in multimodal reasoning chains of MLLM. In Advances in Neural Information Processing Systems, 2025.arxiv.org/abs/2505.24238
  9. [9]Hao Fang, Jiawei Kong, Bin Chen, Tao Dai, Hao Wu, and Shu-Tao Xia. CLIP-guided generative networks for transferable targeted adversarial attacks. In European Conference on Computer Vision, pages 1–19, 2024.arxiv.org/abs/2407.10179
  10. [10]Hao Fang, Jiawei Kong, Wenbo Yu, Bin Chen, Jiawei Li, Hao Wu, Shu-Tao Xia, and Ke Xu. One perturbation is enough: On generating universal adversarial perturbations against vision-language pre-training models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4090–4100, 2025.arxiv.org/abs/2406.05491
  11. [11]Hao Fang, Changle Zhou, Jiawei Kong, Kuofeng Gao, Bin Chen, Tao Liang, Guojun Ma, and Shu-Tao Xia. Grounding language with vision: A conditional mutual information calibrated decoding strategy for reducing hallucinations in LVLMs. In Advances in Neural Information Processing Systems, 2025.arxiv.org/abs/2505.19678
  12. [12]Hao Fang, Jinyu Li, Jiawei Kong, Tianqu Zhuang, Kuofeng Gao, Bin Chen, and Shu-Tao Xia. Seeing through the chain: Mitigate hallucination in multimodal reasoning models via CoT compression and contrastive preference optimization. arXiv preprint arXiv:2602.03380, 2026.
  13. [13]Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, Rongrong Ji, Caifeng Shan, and Ran He. MME: A comprehensive evaluation benchmark for multimodal large language models. In Advances in Neural Information Processing Systems, 2025.
  14. [14]Yuhan Fu, Ruobing Xie, Xingwu Sun, Zhanhui Kang, and Xirong Li. Mitigating hallucination in multimodal large language model via hallucination-targeted direct preference optimization. In Findings of the Association for Computational Linguistics: ACL 2025, pages 16563–16577, 2025.arxiv.org/abs/2411.10436
  15. [15]Gemma Team. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025.
  16. [16]Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. MiniLLM: Knowledge distillation of large language models. In International Conference on Learning Representations, 2024. Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. Hallusion-Bench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14375–14385, 2024.
  17. [18]Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. OPERA: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13418–13427, 2024.arxiv.org/abs/2311.17911
  18. [19]Chaoya Jiang, Haiyang Xu, Mengfan Dong, Jiaxing Chen, Wei Ye, Ming Yan, Qinghao Ye, Ji Zhang, Fei Huang, and Shikun Zhang. Hallucination augmented contrastive learning for multimodal large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27636–27046, 2024.arxiv.org/abs/2312.06968
  19. [20]Prannay Kaul, Zhizhong Li, Hao Yang, Yonatan Dukler, Ashwin Swaminathan, C. J. Taylor, and Stefano Soatto. THRONE: An object-based hallucination benchmark for the free-form generations of large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.arxiv.org/abs/2405.05256
  20. [21]Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317–1327, 2016.arxiv.org/abs/1606.07947
  21. [22]Jiawei Kong, Hao Fang, Shunxiang Liao, Jinyu Li, Bin Chen, Hao Wu, Shu-Tao Xia, and Min Zhang. Reasoning matters: Mitigate hallucination in multimodal large reasoning models via reasoning-conditioned preference optimization. arXiv preprint arXiv:2605.27906, 2026.arxiv.org/abs/2605.27906
  22. [23]Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalanditis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123(1):32–73, 2017.arxiv.org/abs/1602.07332
  23. [24]Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyuang Miao, and Lidong Bing. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.arxiv.org/abs/2311.16922
  24. [25]Jinyu Li, Hao Fang, Zhiming Zhang, Jiawei Kong, Bin Chen, and Shu-Tao Xia. MergeHEIR: Mitigating multimodal hallucinations as the tax of model merging. arXiv preprint arXiv:2609.32422, 2026.
  25. [26]Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In Proceedings of EMNLP, pages 292–305, 2023.
  26. [27]Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft COCO: Common objects in context. arXiv preprint arXiv:1405.0312, 2014.arxiv.org/abs/1405.0312
  27. [28]Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning. In International Conference on Learning Representations, 2024.arxiv.org/abs/2306.14565
  28. [29]Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems, pages 34892–34916, 2023.
  29. [30]Wenqi Liu, Xuemeng Song, Jiaxi Li, Yinwei Wei, Na Zheng, Jianhua Yin, and Liqiang Nie. Mitigating hallucination through theory-consistent symmetric multimodal preference optimization. In Advances in Neural Information Processing Systems, 2025.arxiv.org/abs/2506.11712
  30. [31]Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, and Abhinav Rastogi. Improve mathematical reasoning in language models by automated process supervision. arXiv preprint arXiv:2406.06592, 2024.arxiv.org/abs/2406.06592
  31. [32]Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, pages 27730–27744, 2022.arxiv.org/abs/2203.02155
  32. [33]Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, pages 53728–53741, 2023.arxiv.org/abs/2305.18290
  33. [34]Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. In Proceedings of EMNLP, pages 4035–4045, 2018.arxiv.org/abs/1809.02156
  34. [35]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.arxiv.org/abs/1707.06347
  35. [36]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
  36. [37]Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chun-yuan Li, Yikang Shen, Chuang Gan, Liangyan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, and Trevor Darrell. Aligning large multimodal models with factually augmented RLHF. In Findings of the Association for Computational Linguistics: ACL 2024, pages 13088–13110, 2024.arxiv.org/abs/2309.14525
  37. [38]Chen Wang, Zhaochun Li, Jionghao Bai, Yining Zhang, Hexuan Deng, Ge Lan, and Yue Wang. Distilled reinforcement learning for LLM post-training. arXiv preprint arXiv:2607.17247, 2026.arxiv.org/abs/2607.17247
  38. [39]Fei Wang, Wenxuan Zhou, James Y. Huang, Nan Xu, Sheng Zhang, Hoifung Poon, and Muhao Chen. mDPO: Conditional preference optimization for multimodal large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8078–8088, 2024.
  39. [40]Juny​​ang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Jiaqi Wang, Haiyang Xu, Ming Yan, Ji Zhang, and Jitao Sang. AMBER: An LLM-free multi-dimensional benchmark for MLLMs hallucination evaluation. arXiv preprint arXiv:2311.07397, 2023.arxiv.org/abs/2311.07397
  40. [41]Jiulong Wu, Zhenqiang Shi, Shuaqiang Wang, Jizhou Huang, Dawei Yin, Lingyong Yan, Min Cao, and Min Zhang. Mitigating hallucinations in large vision-language models via entity-centric multimodal preference optimization. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 19445–19461, 2025.arxiv.org/abs/2506.04039
  41. [42]Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. In Advances in Neural Information Processing Systems, pages 59008–59033, 2023.arxiv.org/abs/2306.01693
  42. [43]Xiaochen Yang, Hao Fang, Jiawei Kong, Yaoxin Mao, Bin Chen, and Shu-Tao Xia. Looking back and forth: Cross-image attention calibration and attentive preference learning for multi-image hallucination mitigation. In Computer Vision – ECCV 2026, pages 259–277. Springer, 2026.arxiv.org/abs/2603.07048
  43. [44]Zhihe Yang, Xufang Luo, Dongqi Han, Yunjian Xu, and Dongsheng Li. Mitigating hallucinations in large vision-language models via DPO: On-policy data hold the key. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.arxiv.org/abs/2501.09695
  44. [45]Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large language models. Science China Information Sciences, 67(12):220105, 2024.DOI
  45. [46]Qifan Yu, Juncheng Li, Longhui Wei, Liang Pang, Wentao Ye, Bosheng Qin, Siliang Tang, Qi Tian, and Yueting Zhuang. HalluciDoctor: Mitigating hallucinatory toxicity in visual instruction data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12944–12953, 2024.arxiv.org/abs/2311.13614
  46. [47]Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, and Tat-Seng Chua. RLHF-V: Towards trustworthy MLLMs via behavior alignment from fine-grained correctional human feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.arxiv.org/abs/2312.00849
  47. [48]Tianyu Yu, Haoye Zhang, Qiming Li, Qixin Xu, Yuan Yao, Da Chen, Xiaoman Lu, Ganqu Cui, Yunkai Dang, Taiwen He, Xiaocheng Feng, Jun Song, Bo Zheng, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. RLAIF-V: Open-source AI Feedback leads to super GPT-4V trustworthiness. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.arxiv.org/abs/2405.17220
  48. [49]Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models. arXiv preprint arXiv:2601.18734, 2026.arxiv.org/abs/2601.18734
  49. [50]Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. Beyond hallucinations: Enhancing LVLMs through hallucination-aware direct preference optimization. arXiv preprint arXiv:2311.16839, 2023.
  50. [51]Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision-language models. In International Conference on Learning Representations, 2024.arxiv.org/abs/2310.00754

Paper details

vlnc:W_28yj4ayq
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents