betaFeedback

WorksComputer Science

VEPO: Mitigating Multilingual Hallucinations in LVLMs via Visual Evidence-guided Preference Optimization

Abstract

Large vision-language models (LVLMs) achieve strong multimodal performance, yet they remain prone to object hallucination—generating answers that are linguistically fluent but visually ungrounded. This issue is especially severe in low-resource languages, where weaker cross-modal alignment causes models to rely more on language priors than on image evidence. Existing remedies either require costly multilingual annotations or external translation/reward models, or intervene at inference time in ways that can distort the output language. We propose VEPO(Visual Evidence-guided Preference Optimization), a training-time method that transfers reliable visual grounding from English to other languages without external models. VEPO first localizes key visual evidence using English queries via mid-layer cross-modal attention, then constructs preference pairs that share the same correct answer but differ in evidence sufficiency (full image vs.\ key-region masked image). Direct preference optimization on these pairs encourages low-resource inputs to prefer answers supported by complete visual evidence. Experiments on multilingual POPE and MME with Chinese, French, and Hindi show consistent hallucination reduction and perception gains, with the largest improvements on the lowest-resource language. Analyses further confirm that VEPO sharpens question-relevant attention and that evidence-guided masking outperforms random masking.

Introduction

Large vision-language models (LVLMs) have advanced rapidly on visual question answering, image captioning, and multimodal dialogue ([7, 6, 1, 8]). Despite strong overall capability, they still suffer from object hallucination: producing answers that appear plausible yet lack visual support ([15, 4] (Li et al. 2023)). Critically, this failure is language-dependent. As shown in Figure 1, Qwen-VL-Chat exhibits substantially lower accuracy on low-resource queries than high-resource ones on POPE-COCO, indicating uneven multimodal perception across languages.

Multilingual performance gap of Qwen-VL-Chat on POPE-COCO

Figure 1. Multilingual performance gap of Qwen-VL-Chat on POPE-COCO: non-English, especially low-resource, languages lag behind English.

We term this phenomenon multilingual hallucination: semantically equivalent questions in different languages induce inconsistent evidence retrieval and decision behavior, causing low-resource inputs to hallucinate more frequently. Prior work mitigates the gap mainly through data-centric adaptation. Supervised fine-tuning (SFT) with multilingual visual instructions improves instruction following (Rasheed et al. [11]), but high-quality low-resource annotations are expensive and hard to scale. Preference-based methods such as MHR (Qu et al. 2026) construct cross-lingual preference signals via translation or strong multilingual models, thus still relying on external systems. CLAIM (Ye et al. 2025) instead intervenes in cross-lingual attention at inference time to transfer English visual focus to other languages. While effective for grounding, such test-time intervention can bias generation toward English semantic pathways, yielding visually correct yet linguistically incorrect outputs (e.g., answering a Hindi question in English).

This raises a more fundamental question: can we improve low-resource visual grounding using only mechanisms already present inside the LVLM, without external models or inference-time language distortion? Empirically, for semantically equivalent questions, the image regions that support a correct answer should be language-invariant. In practice, however, attention maps under English and low-resource queries often diverge: English inputs focus on question-relevant objects, whereas low-resource inputs attend more diffusely or to irrelevant regions. This suggests that cross-lingual attention consistency is a useful optimization signal.

We propose VEPO (Visual Evidence-guided Preference Optimization). Rather than preferring a “correct” answer over an “incorrect” one, VEPO prefers the same correct answer under a complete visual input over the same answer under a key-evidence-masked input. Concretely, we (i) use English queries as probes to identify the top-ρ\rho image patches most attended during answer generation (default ρ=0.2\rho=0.2), (ii) mask those patches to form rejected images while keeping the original image as the chosen image, and (iii) apply direct preference optimization (DPO) ([10, 12]) on low-resource queries with fixed target answers. Training thus instills a preference for evidence-sufficient grounding while preserving target-language generation, avoiding the language drift of inference-time attention transfer.

Our contributions are threefold:

  • We formulate multilingual hallucination as insufficient

visual-evidence reliance under low-resource inputs, and propose VEPO, which constructs evidence-sufficiency preference pairs without external translators or reward models.

  • We introduce an English-attention-guided masking pipeline that localizes key visual evidence from mid-layer attention (layers 11–14 in LLaVA-1.5) and transfers grounding preferences to other languages via training-time DPO.

  • Experiments on multilingual POPE and MME show consistent gains across Chinese, French, and Hindi (e.g., Hindi POPE average F1 0.302→0.644); ablations verify the necessity of evidence-guided (vs. random) masking and of mixing English data for extremely low-resource languages.

Related Work

Object hallucination in LVLMs. Object hallucination has been extensively studied for English-centric LVLMs [3, 15] (Li et al. 2023). Mitigation strategies include decoding-time contrast [4], preference/RLHF alignment [12] (Yu et al. 2024), and curated correctional feedback. Most of this literature assumes English evaluation; multilingual degradation remains comparatively underexplored, even though language priors that drive hallucination are often stronger when visual grounding is weak. Closing this gap requires methods that improve evidence use under low-resource prompts, rather than only increasing target-language fluency.

Multilingual multimodal models. Multilingual visual instruction tuning improves coverage across languages [11], but residual hallucination persists, especially when visual grounding is weak. Cross-lingual preference learning (Qu et al. 2026) and attention intervention (Ye et al. 2025) target multilingual hallucination more directly: the former often relies on translation/judging systems, while the latter may distort the output language at inference time. VEPO differs by optimizing evidence sufficiency at training time using internal English attention, without external scorers or inference-time intervention.

Algorithm 1 VEPO Preference Construction and Optimization

Input: Image II, English query xenx^{\mathrm{en}}, low-resource query xlrx^{\mathrm{lr}}, answer yy, mask ratio ρ\rho, layer set L\mathcal{L}

Output: Updated policy πθ\pi_\theta

  1. Run πref\pi_{\mathrm{ref}} on (I,xen)(I, x^{\mathrm{en}}) and collect mid-layer attentions {A(l)}l∈L\{A^{(l)}\}_{l\in\mathcal{L}}

  2. Compute evidence scores sis_i from the last query token to each image token

  3. Select S=TopK⁡({si},ρ)S = \operatorname{TopK}(\{s_i\}, \rho); set Iw=II^w=I, Il=Mask⁡(I,S)I^l=\operatorname{Mask}(I,S)

  4. Form preference pair (Iw,xlr,y)≻(Il,xlr,y)(I^w, x^{\mathrm{lr}}, y) \succ(I^l, x^{\mathrm{lr}}, y)

  5. Update πθ\pi_\theta by minimizing LVEPO\mathcal{L}_{\mathrm{VEPO}} on the constructed pairs

Preference optimization for multimodal models. DPO [10] and multimodal variants such as mDPO [12] improve alignment by contrasting preferred and rejected responses. Standard multimodal DPO typically contrasts different answer texts under a fixed image. VEPO instead contrasts different image conditions under a shared answer, explicitly teaching the model when visual evidence is adequate to support that answer. This shift matters for multilingual hallucination: when language priors are strong and visual grounding is weak, answer-level preferences can be satisfied without consulting the image, whereas evidence-sufficiency preferences force the model to notice whether the supporting regions are present.

Method

VEPO consists of three stages: (i) localizing key visual evidence with an English query, (ii) constructing an evidence-occluded rejected image, and (iii) optimizing a low-resource preference objective that favors the same correct answer under complete evidence over incomplete evidence. Figure 2 illustrates how English mid-layer attention is converted into a key-evidence mask; Figure 3 then contrasts the resulting preference construction with standard multimodal DPO. Algorithm 1 summarizes the full pipeline.

Diagram showing construction of a key-evidence masked image from English attention

Figure 2. Constructing a key-evidence masked image from English attention. Mid-layer attention from the last query token ranks image patches; the top-ρ\rho patches are blacked out to form the rejected image, while the original image remains the chosen image.

Comparison between VEPO and standard multimodal DPO

Figure 3. Comparison between VEPO (left) and standard multimodal DPO (right). VEPO contrasts evidence-sufficient vs. evidence-missing images under the same correct answer; standard DPO contrasts different answers under the same image. The training examples illustrated in the figure are presented in Chinese, and the semantic content of the questions is consistent with that shown in Figure 2.

Key Visual Evidence Localization

Given an image II and an English query xen={t1en,…,tnen}x^{\mathrm{en}} = \{t^{\mathrm{en}}_1,\ldots,t^{\mathrm{en}}_n\}, the LVLM generates an answer autoregressively. We treat English as a high-resource probe language: because English visual grounding is typically stronger after large-scale pre-training and instruction tuning, its cross-modal attention provides a more reliable estimate of which image regions truly support the answer. At layer ll, with hidden states H(l−1)H^{(l-1)},

Q(l)=H(l−1)WQ(l),K(l)=H(l−1)WK(l),V(l)=H(l−1)WV(l).(1)\begin{aligned} Q^{(l)} &= H^{(l-1)}W_Q^{(l)},\\ K^{(l)} &= H^{(l-1)}W_K^{(l)},\\ V^{(l)} &= H^{(l-1)}W_V^{(l)}. \tag*{(1)} \end{aligned}

The attention matrix is A(l)=softmax⁡(Q(l)K(l)⊤/d)A^{(l)} = \operatorname{softmax}(Q^{(l)}K^{(l)\top}/\sqrt{d}). We use the attention from the last query token tnent^{\mathrm{en}}_n to each image token viv_i as an evidence signal. Empirically, this token aggregates the full question semantics after encoding and yields more compact, question-aligned maps than averaging all prompt tokens or using generated tokens (Section 5). Because shallow layers capture local texture patterns and deep layers are biased toward language generation, we average mid-layer attentions over a layer set L\mathcal{L} (layers 11–14 in LLaVA-1.5-7B):

si=1∣L∣∑l∈LA(l)(tnen,vi).(2)s_i = \frac{1}{|\mathcal{L}|} \sum_{l \in\mathcal{L}} A^{(l)}(t_n^{\mathrm{en}}, v_i). \tag*{(2)}

Higher sis_i indicates stronger support for the English answering process. The resulting ranking of image patches defines the occlusion targets used to build rejected images in the next stage.

Key Evidence Occlusion

Let {si}i=1m\{s_i\}_{i=1}^{m} be scores over mm image tokens. We select the top ρm\rho m tokens as key evidence,

S=TopK⁡({si}i=1m,ρm),(3)S = \operatorname{TopK}(\{s_i\}_{i=1}^{m}, \rho m), \tag*{(3)}

and obtain the rejected image by masking the corresponding patches,

Il=Mask⁡(I,S),Iw=I,(4)I^{l} = \operatorname{Mask}(I,S), \qquad I^{w} = I, \tag*{(4)}

where ρ=0.2\rho=0.2 by default. Mask⁡(⋅)\operatorname{Mask}(\cdot) zeros the selected patches rather than randomly destroying the image, so IlI^{l} retains scene structure and background context while removing question-critical evidence. Intuitively, if English attention for a color question concentrates on a bird’s feathers, those patches are blacked out in IlI^{l}, forcing a hard “evidence-missing” negative without inventing a new answer label. This design is crucial: the rejected sample is not an arbitrary corrupted image, but a carefully constructed case where the visual evidence needed for yy is selectively removed.

As shown on the left of Figure 3, VEPO therefore compares two image conditions under a fixed correct answer. In contrast, standard multimodal DPO (right) keeps the image fixed and contrasts a correct answer against an incorrect answer. The former directly supervises visual grounding; the latter mainly supervises answer text preference and can still be satisfied by language priors when the image is under-used. Because the rejected image preserves global scene layout, the preference is not about image realism or captionability, but about whether the patches that English attention marks as decisive remain available to the low-resource query.

Evidence-Sufficiency Preference Optimization

For a low-resource query xlrx^{\mathrm{lr}} with correct answer yy, we form the conditional preference pair

(Iw,xlr,y),(Il,xlr,y).(5)(I^{w}, x^{\mathrm{lr}}, y), \qquad(I^{l}, x^{\mathrm{lr}}, y). \tag*{(5)}

The chosen sample provides complete evidence; the rejected sample shares the same answer text but lacks key visual support. With policy πθ\pi_{\theta} and reference model πref\pi_{\mathrm{ref}}, VEPO minimizes

LVEPO=−E(Iw,Il,xlr,y)∼D[log⁡σ(βlog⁡πθ(y∣Iw,xlr)πref(y∣Iw,xlr)−βlog⁡πθ(y∣Il,xlr)πref(y∣Il,xlr))].(6)\mathcal{L}_{\mathrm{VEPO}} = -\mathbb{E}_{(I^{w},I^{l},x^{\mathrm{lr}},y)\sim\mathcal{D}}\left[\log\sigma\left(\beta\log\frac{\pi_{\theta}(y \mid I^{w},x^{\mathrm{lr}})}{\pi_{\mathrm{ref}}(y \mid I^{w},x^{\mathrm{lr}})} - \beta\log\frac{\pi_{\theta}(y \mid I^{l},x^{\mathrm{lr}})}{\pi_{\mathrm{ref}}(y \mid I^{l},x^{\mathrm{lr}})}\right)\right]. \tag*{(6)}

This objective raises the relative likelihood of yy under complete evidence and lowers it under missing evidence, teaching the model to condition answers on visual sufficiency rather than language priors alone. Importantly, because yy remains in the target language for both sides of the pair, VEPO improves grounding without encouraging English-only decoding—a failure mode that inference-time English-attention transfer can introduce.

In practice, VEPO is applied as a lightweight post-training stage: the reference model is frozen, only LoRA adapters are updated, and the preference pairs reuse existing multilingual instruction answers. Consequently, the method does not require new human labels for rejected answers, nor does it depend on an external reward model to score hallucinations. The only language asymmetry is intentional and confined to evidence localization, where English serves as a probe; the optimization itself remains fully target-language.

Experiments

Setup

Data. Training uses the PALO multilingual visual-instruction corpus [11] (∼\sim2.1M multilingual visual instructions spanning English, Chinese, French, Spanish, Russian, Japanese, Arabic, Hindi, Bengali, and Urdu). For each selected training sample, we pair the target-language question with its English counterpart on the same image: English is used only to localize evidence, while preference optimization is performed on the target-language question with the original supervised answer yy. This design avoids collecting new low-resource annotations and does not require an external translator at training time beyond the already available English parallel questions in PALO. Evaluation uses POPE (Li et al. 2023) for object hallucination (F1 on random/popular/adversarial splits over COCO, A-OKVQA, and GQA sources) and MME (Fu et al. 2026) for perception and cognition (14 subtasks). We report Chinese (Zh), French (Fr), and Hindi (Hi) as representative mid- to low-resource settings.

Models and baselines. All methods use LLaVA-1.5-7B [6] as the backbone:

  • LLaVA: pretrained model without further adaptation.

  • LLaVA+SFT: supervised fine-tuning on target-language PALO instructions.

  • LLaVA+VEPO: VEPO applied directly to the pretrained model.

  • LLaVA+SFT+VEPO: SFT followed by VEPO (full pipeline).

For Zh/Fr, VEPO pairs are built from target-language data only; for Hi, we mix English pairs at a 1:1 ratio for training stability (ablated in § 5).

Implementation. We fine-tune with LoRA (rank 128, α\alpha=256) for one epoch, learning rate 2×10−52 \times10^{-5}, batch size 8, DPO temperature β\beta=0.1, and mask ratio ρ\rho=0.2. Mid-layer attention uses layers 11–14 and the last query token, validated in § 5. The reference model πref\pi_{\mathrm{ref}} is frozen as the initialization of πθ\pi_{\theta}. All multilingual POPE/MME prompts follow the official English templates translated into the target language with the same yes/no answer format. Unless otherwise noted, we report a single run with the above hyperparameters; qualitative attention analyses use the same checkpoints as the quantitative tables.

Main Results on POPE

Table 1 reports F1 scores. On the pretrained backbone, VEPO improves average F1 for Zh (0.731→0.766, +3.5), Fr (0.742→0.794, +5.2), and Hi (0.302→0.644, +34.2). Gains are largest for Hindi, consistent with transferring English visual localization to languages with weaker native grounding. Improvements hold across COCO/A-OKVQA/GQA and across random/popular/adversarial splits. Notably, adversarial F1 also rises (e.g., Zh COCO 0.731→0.766; Hi COCO

Lang.MethodCOCO RandomCOCO PopularCOCO AdversarialAOKVQA RandomAOKVQA PopularAOKVQA AdversarialGQA RandomGQA PopularGQA AdversarialAvg.
ZhLLaVA-1.50.7860.7640.7310.7450.7110.6930.7520.7020.6920.731
+VEPO0.8180.8150.7660.7830.7640.7200.7870.7310.7070.766
FrLLaVA-1.50.7650.7980.7430.7440.7500.7100.7440.7220.7060.742
+VEPO0.8220.8580.7840.7980.8140.7450.7980.7830.7430.794
HiLLaVA-1.50.3450.3320.3370.2970.3020.2980.2670.2730.2700.302
+VEPO0.6320.6690.6610.6310.6390.6370.6350.6560.6330.644
Zh+SFT0.8230.8220.7750.7650.7550.7150.7690.7470.7180.765
+SFT+VEPO0.8450.8390.7820.8090.7900.7370.8110.7580.7260.789
Fr+SFT0.7680.7300.7210.7360.7160.7020.7370.7160.7050.726
+SFT+VEPO0.7970.8340.7640.7730.7830.7260.7700.7460.7190.768

Table 1. POPE F1 of VEPO on original and SFT models across Chinese (Zh), French (Fr), and Hindi (Hi). Best results within each language block are in bold.

Overall MME scores before and after VEPO across languages

Figure 4. Overall MME scores before and after VEPO across languages.

Fine-grained MME results

Figure 5. Fine-grained MME results: gains are concentrated on perception-oriented subtasks.

0.337→\to0.661), indicating that VEPO reduces reliance on co-occurrence priors rather than only correcting easy random negatives. In other words, even when distractor objects frequently co-occur with queried categories, the model becomes less likely to affirm their presence without sufficient visual support.

After SFT, VEPO still helps: Zh 0.765→\to0.789 and Fr 0.726→\to0.768. Thus SFT and VEPO are complementary—SFT improves instruction following and target-language fluency, while VEPO strengthens evidence-based answering and further suppresses object hallucination. This complementarity is practically useful: one can first adapt an LVLM to a target language with standard supervised data, then apply VEPO as a lightweight second stage to specifically harden visual grounding.

We further ablate two design choices on Chinese and Hindi POPE. Table 2 contrasts evidence-guided masking with random masking at the same occlusion ratio, and Table 3 studies whether mixing English preference pairs stabilizes Hindi training.

MethodCOCOAOKVQAGQAAvg.
rand.pop.adv.rand.pop.adv.rand.pop.adv.
LLaVA-1.50.7860.7640.7310.7450.7110.6930.7520.7020.6920.731
+Random Mask0.7930.7890.7430.7530.7310.7060.7670.7190.7040.745
+VEPO0.8180.8150.7660.7830.7640.7200.7870.7310.7070.766

Table 2. Chinese POPE: evidence-guided masking vs. random masking under the same occlusion ratio ρ=0.2\rho=0.2.

Methodrand.pop.adv.Avg.
LLaVA-1.50.3450.3320.3370.338
VEPO (Hi only)0.3090.4990.4090.406
VEPO (Hi+En)0.6320.6690.6610.654
SFT0.7060.7110.7090.709
SFT+VEPO (Hi only)0.6860.7240.6890.700
SFT+VEPO (Hi+En)0.7720.7830.7520.769

Table 3. Hindi COCO-POPE: mixing English preference data stabilizes VEPO.

Results on MME

Figures 4 and 5 summarize MME scores. Overall scores rise for Zh (1233→\to1270), Fr (1313→\to1365), and Hi (745→\to909).

Perception accounts for most of the gain: Zh 992→\to1061, Fr 1059→\to1149, Hi 544→\to664, with clear improvements on existence, color, position, poster, artwork, and OCR—tasks that require precise local visual evidence. Cognition changes are mixed: Zh/Fr show mild drops on some reasoning subtasks (e.g., commonsense and numerical calculation), while Hi improves (200→\to245). Importantly, total scores still increase in all languages. For Hindi, both perception and cognition rise, suggesting that stronger visual evidence can also stabilize downstream reasoning when language ability itself is weak. Overall, MME corroborates POPE: VEPO mainly improves visually grounded perception, with the largest absolute benefit in the lowest-resource setting.

Analysis

Attention redistribution. To verify that VEPO redirects attention toward question-relevant evidence, we visualize patch-level attention changes before vs. after training (Figure 6; red: increase; blue: decrease). VEPO does not uniformly strengthen or weaken visual attention; instead, it exhibits a clear question-conditioned redistribution: gains concentrate on regions needed to answer the query, while attention on irrelevant areas decreases.

Attention change after VEPO

Figure 6. Attention change after VEPO: red/blue indicate increased/decreased attention on image patches.

Concrete cases match this pattern. For “Is there a snowboard?”, attention rises near the person and ski-slope activity regions rather than sky or distant background, indicating a shift from scene-level priors to local evidence supporting the queried object. For “Is there a sink?”, gains localize around the washbasin area in the lower image, while furniture and person regions are relatively suppressed. For “Is there a chair?”, attention increases on audience seating rather than the salient foreground athlete, showing that VEPO can retrieve background objects that are diagnostic but visually less dominant. For “Is there a bed?”, gains cover the bed surface and nearby structure, again aligning with the queried category.

Overall, VEPO is semantically selective: it amplifies regions that support the correct answer, not necessarily the most salient foreground. Regions that previously attracted attention but are weakly related to the question are suppressed. This suggests that VEPO strengthens an evidence-based answering preference and transfers a more English-like visual localization pattern to low-resource inputs, rather than only changing final yes/no logits.

Layer and token choice for masking. When constructing English-guided masks, we black out patches ranked by mid-layer attention from the last query token. To justify this choice, we compare alternative constructions along two axes (Figure 7): (i) layer depth—shallow, mid, and deep attentions (and mid-layer averages); and (ii) token choice—last query token, all prompt tokens, first generated token, and all generated tokens (averaged when multiple tokens are used). The goal is to identify which language-side anchor and which depth best recover the visual evidence needed for the answer.

Effect of layer depth and token selection on English-attention masks used for VEPO

Figure 7. Effect of layer depth and token selection on English-attention masks used for VEPO.

Layer comparisons show a clear progression. Shallow masks are sparse and fragmented: they often hit texture edges

(sky/snow borders, room bottoms, scattered background) and only partially cover foreground objects, reflecting local appearance patterns rather than stable semantic localization. Mid-layer masks are more compact and question-aligned: they cover activity regions on the slope, person–mirror interactions indoors, and athlete/court regions in sports scenes, while reducing random background responses. This indicates that mid layers are where vision–language alignment becomes effective. Deep masks retain some object focus but shrink, shift, or collapse into blocky clusters, consistent with representations being compressed toward answer generation; intervening with deep attention therefore risks decoding bias rather than reliable evidence localization. These observations motivate averaging layers 11–14 in LLaVA-1.5.

Token comparisons likewise favor the last query token. After full question encoding, this token aggregates the main query semantics and yields the most compact, question-driven masks. Averaging all prompt tokens mixes content words with function words and instruction templates, producing larger, more fragmented masks that spill into background. Attentions from generated tokens are biased by decoding: they partly serve answer organization rather than evidence retrieval, and later tokens further depend on already generated text. First-generated-token masks are more focused than full-generation averages but remain less stable than last-query-token masks. Thus, for both interpretability and intervention quality, the last query token is the preferred language-side anchor.

These analyses also clarify why VEPO helps beyond output-level preference shifts. Before training, attention under complex scenes often spreads to weakly related candidates, occluders, or background clutter, encouraging false positives or over-interpretation of local cues. After VEPO, attention concentrates on entity-, attribute-, or relation-relevant regions; when those regions are masked, the model becomes more sensitive to missing evidence and less willing to affirm presence without support. This matches the training signal: contrasting full vs. evidence-occluded images teaches the model to separate evidence-sufficient from evidence-missing conditions, improving intermediate evidence selection rather than only final answers.

Evidence-guided vs. random masking. Table 2 replaces evidence-guided occlusion with random occlusion at the same ratio ρ=0.2\rho=0.2. Random masking yields only a modest gain (avg. 0.731→0.745), whereas evidence-guided VEPO reaches 0.766, with consistent gaps across COCO/A-OKVQA/GQA and across random/popular/adversarial splits. Random masks frequently remove background or unrelated objects and therefore fail to create informative “evidence-missing” negatives: the occluded image remains nearly equally answerable, so the preference pair provides a weak learning signal about when confidence should drop. Evidence-guided masks preferentially remove question-critical patches, producing harder negatives that force the model to link answers to specific visual support. Hence VEPO’s gains are not from occlusion augmentation alone, but from targeted destruction of decisive evidence; both the preference objective and the region-selection strategy are necessary.

Mixing English data for low-resource languages. Table 3 studies whether VEPO should mix English preference pairs for Hindi. Target-only Hindi VEPO is unstable: without SFT, COCO-POPE random F1 drops from 0.345 to 0.309 and the average is only 0.406; after SFT, Hi-only VEPO (0.700) even falls below SFT alone (0.709). Mixing English pairs at 1:1 raises averages to 0.654 (no SFT) and 0.769 (with SFT), with balanced gains on random/popular/adversarial splits. Qualitatively, Hindi-only optimization can collapse to overly conservative “no” answers or unstable yes/no decisions, while Hi+En training yields consistent improvements.

We attribute this to data scale and distribution sensitiv-

ity. The backbone has limited Hindi exposure, so optimizing solely on scarce Hindi preference pairs can over-adapt parameters toward a small target-language distribution and damage previously stable vision–language alignments. English pairs supply denser multimodal gradients and act as cross-lingual regularization that constrains update magnitude. For higher-resource Zh/Fr, target-only VEPO already suffices; English mixing is therefore reserved for extremely low-resource languages in our main experiments. A practical recipe follows: start with target-only pairs, and introduce English pairs when validation hallucination metrics become unstable or degrade relative to the SFT baseline.

Conclusion

We presented VEPO, a visual evidence-guided preference optimization method for mitigating multilingual hallucinations in LVLMs. By localizing key evidence with English mid-layer attention and optimizing preferences between full and evidence-masked images under a shared target-language answer, VEPO strengthens low-resource visual grounding without external models or inference-time language distor-

References

  1. [1]Bai, J.; Bai, S.; Yang, S.; Wang, S.; Tan, S.; Wang, P.; Lin, J.; Zhou, C.; and Zhou, J. 2023. Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond. arXiv:2308.12966.arxiv.org/abs/2308.12966
  2. [3]Gunjal, A.; Yin, J.; and Bas, E. 2024. Detecting and Preventing Hallucinations in Large Vision Language Models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 18135–18143.arxiv.org/abs/2308.06394
  3. [4]Leng, S.; Zhang, H.; Chen, G.; Li, X.; Lu, S.; Liu, C.; and Zhang, H. 2024. Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 13872–13882.arxiv.org/abs/2311.16922
  4. [6]Liu, H.; Li, C.; Li, Y.; and Lee, Y. J. 2024. Improved Baselines with Visual Instruction Tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 26296–26306.arxiv.org/abs/2310.03744
  5. [7]Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023. Visual Instruction Tuning. In Advances in Neural Information Processing Systems, volume 36.
  6. [8]OpenAI. 2023. GPT-4V(vision) System Card. Technical Report.
  7. [10]Rafailov, R.; Sharma, A.; Mitchell, E.; Manning, C. D.; Ermon, S.; and Finn, C. 2024. Direct Preference Optimization: Your Language Model Is Secretly a Reward Model. Advances in Neural Information Processing Systems, 36.arxiv.org/abs/2305.18290
  8. [11]Rasheed, H.; Maaz, M.; Shaker, A.; Khan, S.; Cholakah, H.; Anwer, R. M.; Baldwin, T.; Felsberg, M.; and Khan, F. S. 2025. PALO: A Large Multilingual Multimodal Language Model. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision.
  9. [12]Wang, F.; Zhou, W.; Huang, J. Y.; Xu, N.; Zhang, S.; Poon, H.; and Chen, M. 2024. mDPO: Conditional Preference Optimization for Multimodal Large Language Models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 8078–8088. Association for Computational Linguistics.arxiv.org/abs/2406.11839
  10. [15]Zhou, Y.; Cui, C.; Yoon, J.; Zhang, L.; Deng, Z.; Finn, C.; Bansal, M.; and Yao, H. 2024. Analyzing and Mitigating Object Hallucination in Large Vision-Language Models. In The Twelfth International Conference on Learning Representations.arxiv.org/abs/2310.00754

Paper details

vlnc:W_xnjeh7bu
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents