betaFeedback

WorksComputer Science

Seeing Isn't Resolving: Visual and Linguistic Disambiguation with Conversational Grounding

Abstract

Visual grounding benchmarks usually do not consider the contextual uncertainty of real-world scenarios. In this study, we construct a novel benchmark, the Japanese evaluation for Ambiguous visual Grounding with Dialogue (JAG-D) dataset. It consists of human-robot conversational multimodal interactions covering different types of ambiguity, arising from visual and linguistic categories. We evaluate the capabilities of different state-of-the-art Visual Language Models (VLMs) for ambiguity recognition and resolution. We compare dialogue context grounding performance for visual and linguistic ambiguity resolution. Our experiments show that VLMs can substantially improve the resolution of linguistic ambiguity, especially Japanese zero anaphora, by using dialogue context. We further conduct a comparative analysis to understand whether grounding improvements arise from the semantic content of the dialogue or from its conversational structure.

Introduction

Visual grounding is the task of localizing visual regions described by referring expressions in natural language, and has achieved substantial progress on standard benchmarks [33, 34]. However, these benchmarks evaluate models under idealized conditions where referring expressions are unambiguous, single-target, and context-independent [43], even if such assumptions are not valid for real-world settings. This limitation motivates the need to address ambiguity in referring expressions in realistic scenarios such as human-robot interaction.

Ambiguity resolution in language has been widely studied, particularly at the lexical and sentential levels [9, 24, 36, 16].

Multimodal ambiguity resolution is equally important, and tasks such as visual question answering have attracted particular attention [38, 31, 48]. However, such tasks capture only part of the referential communication required in human-robot interaction. To achieve appropriate interaction grounded to the world, computational systems must be able to properly interpret references that depend more heavily on the situational context. In real-world scenarios, user utterances contain diverse forms of instructions and omissions, and resolving them requires the use of various types of information, such as dialogue history [6], pointing [4], and gaze [11]. In particular, appropriately interpreting context by combining dialogue with visual input is one of the capabilities required of dialogue agents [5].

For example, consider the situation shown in Figure 1, where two mugs are placed side by side on a table in a living room and a human instructs a robot to “take that mug.” To resolve this ambiguity, it is necessary to use the dialogue context that indicates what “that” refers to. In this case, the ambiguity to be resolved through the dialogue context exists in both the visual and linguistic modalities. Visual ambiguity can arise when multiple similar objects are present in the visual scene, or when multiple objects share the same attributes [47, 45]. Linguistic ambiguity [3], on the other hand, prominently involves demonstratives, overt pronouns, and zero pronouns (zero anaphora). Zero anaphora is a linguistic phenomenon in which a referring expression, such as a subject or object, is omitted but recoverable from context. It is particularly prevalent in pro-drop languages such as Japanese [37], Chinese [19], and Korean [32]. In this work, we especially focus on phenomena in Japanese.

Figure 1. Example of visual and lingusitic ambiguity resolution through dialogue history integration, where dialogue turns provide additional context to the ambiguous deictic instruction (take that mug). The dialogue text has been translated from Japanese, the original language of the human-robot interactions.

Existing benchmarks do not jointly control the source of ambiguity and the context used to resolve it, making it impossible to determine whether grounding failures arise from perceptual or discourse-level limitations. To clarify the extent to which multimodal systems can address these problems, and where their limitations lie, we introduce Japanese evaluation for Ambiguous visual Grounding with Dialogue (JAG-D). JAG-D contains 480 carefully curated instances, each combining an image, an instruction, and a dialogue history. We prioritize annotation quality over benchmark scale, limiting the dataset size so that human experts, with more than eight years of experience in discourse structure annotation, can thoroughly verify each instance. The utterances in JAG-D contain diverse ambiguities arising from both visual and linguistic factors, and the dataset consists of situations in which these ambiguities are expected to be resolved by referring to the dialogue history.

Using this benchmark, we discuss the ambiguity recognition and ambiguity resolution capabilities of visual grounding systems, including current VLMs. First, we evaluate the ability to determine whether a given pair of current instruction and image situation contains ambiguity. Second, we discuss ambiguity resolution ability when dialogue context is provided, namely the improvement in grounding accuracy. As evaluation targets, we use representative visual grounding models including GLIP [23] and MDETR [15], as well as Qwen3-VL [2], a general-purpose VLM with visual grounding capabilities.

Our experiments reveal that current models can robustly recognize linguistically ambiguous scenarios; however, fail when the ambiguity arises from the visual scene. We also find that using additional dialogue context substantially resolves linguistic ambiguity, especially ambiguity caused by zero anaphora, whereas its contribution to resolving visual ambiguity is limited. This asymmetry suggests that current multimodal models rely heavily on linguistic surface cues rather than actively reasoning about the structure of the visual scene. Closing this gap will require models that treat perception and discourse as jointly informative sources [12], rather than treating dialogue as a post-hoc filter over linguistically driven predictions.

Our contributions are as follows:

  1. We introduce JAG-D, a Japanese benchmark for studying ambiguity resolution in human-robot visual grounding. JAG-D provides controlled coverage of diverse linguistic and visual ambiguities, as well as dialogue history used as disambiguation context.

  1. We provide a systematic evaluation of ambiguity recognition and ambiguous visual grounding. Our results show that visual ambiguity remains a systematic blind spot for both monolingual Japanese and multilingual VLMs.

  1. We show that dialogue history improves visual grounding by resolving ambiguities that cannot be addressed from a single instruction alone. We further introduce a dialogue intervention analysis to quantify how different dialogue aspects affect grounding accuracy.

Related Work

Visual Grounding. Visual grounding is a fundamental task that links referring expressions to regions in the visual world. The literature includes foundational benchmarks like RefCOCO and its variations [28 Kazemzadeh et al.,] 2014; Yu et al., [44]), Flickr30k Entities (Plummer et al., [33]), and Visual Genome (Krishna et al., 2017), alongside an evolution of model development: from early methods based on alignment (Karpathy and Fei-Fei, [17]; Rohrbach et al., [34]) to transformer architectures like MDETR (Kamath et al., [15]), GLIP (Li et al., [23]), and GroundingDINO (Liu et al., [25]) that achieve strong performance on the grounding task. Additionally, several studies have also explored extensions of visual grounding to more complex settings, such as including pronouns in addition to noun phrases (Lu et al., [26]) and grounding in contexts of human-robot interaction (Mi et al., [29]). Despite these advances, most existing benchmarks still lack a well-defined categorization of ambiguity and real-world conversations for human-robot interaction.

Ambiguity in Vision–Language Tasks. Ambiguity is widely recognized as a fundamental challenge in multimodal understanding. Prior work has primarily studied this phenomenon in tasks such as visual question answering (VQA) ([11, 39, 14, 13]), as well as multimodal machine translation (Ma et al., [27]). In human-robot interaction, systems are required not only to answer questions but also to localize and use intended objects in the visual scene. While these studies partially address such challenges, they do not sufficiently explore ambiguity that can arise from visual similarity or linguistic underspecification in dialogue contexts.

Visual Grounding with Context. Integrating dialogue as additional context for visual grounding has also been explored. Benchmarks like SIMMC 2.0 (Kottur et al., [20]), GuessWhat?! (Tu et al., [40]) and InViG (Zhang et al., [46]) contain dialogues with disambiguation questions, allowing model evaluation under open-ended ambiguous queries. As a challenging setting, the J-CRe3 dataset (Ueda et al., [41]) provides visual grounding in Japanese dialogue as one of its tasks, offering insight into visual grounding under multimodal ambiguity. Our benchmark extends J-CRe3 and investigates more practical ambiguities that depend on the context of human-robot interaction.

Problem Definition

Let II be a human-robot interaction image from robot’s first-person view, and uu a textual human-to-robot instruction, potentially ambiguous due to multiple possible references in the image. Additionally, let H={u1,…,uK}H = \{u_1, \ldots, u_K\} be a KK-turns dialogue history that contains the human-robot exchanges uiu_i leading to the instruction uu. Finally, b^∗∈B(I)\hat{b}^* \in\mathcal{B}(I) is a single ground-truth bounding box that contains the target object for the grounding task, where B(I)\mathcal{B}(I) denotes the set of candidate bounding boxes in II.

Task Formulation

Multimodal Ambiguity Recognition. We evaluate models on ambiguity recognition, defined as predicting whether (I,u)(I,u) is ambiguous, i.e., whether the instruction uu alone is not sufficient to correctly identify a clear target object b^\hat{b} in image II. The models receive as input the pair (I,u)(I,u) and output “ambiguous” or “non-ambiguous” based on their processing.

Let ϕ(u,I,b)∈{0,1}\phi(u,I,b) \in\{0,1\} denote whether region bb in image II satisfies the semantic interpretation of instruction uu. We define

R(u,I)={b∈B(I)∣ϕ(u,I,b)=1}\mathcal{R}(u,I) = \{b \in\mathcal{B}(I) \mid\phi(u,I,b) = 1\}

as the set containing all the bounding boxes that ground a reference in the instruction uu.

Although supervision provides a single intended referent b∗b^*, ambiguity is defined independently of annotation, and for an image-instruction input pair (u,I)(u,I) the possible outcomes arise:

y={1if ∣R(u,I)∣>1(ambiguous)0if ∣R(u,I)∣=1(unambiguous)y = \begin{cases} 1 & \text{if } |\mathcal{R}(u,I)| > 1 \quad\text{(ambiguous)} \\ 0 & \text{if } |\mathcal{R}(u,I)| = 1 \quad\text{(unambiguous)} \end{cases}

We model ambiguity recognition as a binary classification task. Given (I,u)(I,u), a classifier

c(I,u)∈[0,1]c(I,u) \in[0,1]

predicts the probability that the instruction is ambiguous:

y^=I[c(I,u)>τ],\hat{y} = \mathbb{I}[c(I,u) > \tau],

where τ\tau is a decision threshold.

Visual Grounding Evaluation. In addition to the recognition task, we study visual grounding under ambiguous input in real-world conversations. Given an image-instruction pair (I,u)(I,u), the grounding model predicts a bounding box b^\hat{b} as follows:

b^=f(I,u),b^∈B(I)(1)\hat{b} = f(I,u), \quad\hat{b} \in\mathcal{B}(I) \tag*{(1)}

The performance is evaluated using Intersection over Union (IoU) between b^\hat{b} and the ground-truth bounding box b∗b^* ([7]).

For the visual grounding evaluation, we consider two settings:

  1. Instruction-only grounding: The model receives the instruction uu only.

  1. Dialogue-enhanced grounding: The dialogue history HH is integrated with the instruction uu to obtain a disambiguated instruction u^\hat{u} as follows:

u^=g(u,H),\hat{u} = g(u, H),

where gg is the dialogue-integration function that processes the information from HH and uses it to better interpret uu.

Ambiguity Categorization

In our work, we assume two types of ambiguity that can occur in multimodal inputs to conversational systems. These ambiguity categories are used to balance the data during dataset construction and are also used in the subsequent analysis.

Visual Ambiguity. The instruction uu is linguistically explicit, but multiple objects in image II match the description. We distinguish two types of visual ambiguity:

  • Same Object: multiple instances of the same object candidate, for instance, “pass me the mug” (“マグカップを取って”) where there is more than one mug in the image.

  • Same Attribute: multiple objects share the specified attribute(s), such as “pass me the white object” (“白い物体を取って”) where there is more than one white candidate in the visual input.

Linguistic Ambiguity. The instruction uu is underspecified at the discourse level, as it lacks sufficient contextual information to identify the target object in the image II uniquely. Consequently, we categorize it as follows:

  • Pronoun Ambiguity: the use of an explicit pronoun that lacks a clear antecedent within the current utterance, such as “put it on the table” (“テーブルにそれを置いて”) where “it” refers to a previously discussed object.

  • Zero Anaphora: an omitted argument that requires contextual recovery, such as “pass” (“取って”). Zero anaphora is very common in Japanese language and requires the dialogue history HH to be resolved.

JAG-D Overview

We present JAG-D, a Japanese benchmark for evaluating visual grounding under controlled multimodal ambiguity in conversational settings. The dataset isolates ambiguity categories and explicitly measures both ambiguity recognition and dialogue-based resolution. We focus on Japanese because zero anaphora is frequently used in the language, making it natural for linguistic ambiguity to arise in ways that strongly depend on dialogue context ([37] (Ono and Thompson 1997)).

Source Images

The images in the test set are sampled from J-CRe3 ([41]), an egocentric human-robot interaction dataset collected in real-world environments (e.g., kitchens, offices, living rooms). We construct 480 instances starting from J-CRe3 visual content, prioritizing scenes with sufficient object density and diversity to enable controlled ambiguity under different environment combinations.

Design Principles

Controlled Multimodal Ambiguity. Each instance contains a single ground-truth bounding box b∗b^*. Ambiguity is introduced by ensuring that multiple regions satisfy the semantic constraints of instruction uu, while only one corresponds to the intended referent.

Explicit Ambiguity Categorization. Ambiguous instructions are categorized into four types, described in § 3.2: same_object, same_attribute, pronoun_ambiguity, and zero_anaphora. This categorization enables fine-grained evaluation across distinct variants of ambiguity.

Dialogue-Guided Disambiguation. Each instance consists of an instruction uu and dialogue history HH under three modes: (i) resolving, where HH disambiguates uu, (ii) non-resolving, where ambiguity persists due to irrelevant context, (iii) supportive, where HH provides redundant but consistent information for an unambiguous uu. Examples are provided in Appendix A.2.

Data Generation and Validation

For each image, we generate an ambiguous or unambiguous instruction paired with dialogue history consistent with the designated ambiguity type and dialogue mode. The data generation pipeline is described in Appendix A.3.

All instances are manually reviewed and corrected by two native Japanese annotators, with more than eight years of experience, ensuring that (i) the ambiguity matches the assigned category, (ii) the dialogue mode functions as intended, and (iii) the target bounding box aligns with the instruction.

The final benchmark comprises 480 instances with balanced ambiguous and unambiguous conditions and controlled distributions across ambiguity types and dialogue settings (Table 1). JAG-D provides a carefully curated, human-evaluated test set of high-quality real-world disambiguation dialogues.

CategoryCount%
Instruction Ambiguity
Ambiguous23248.3
same_object5812.1
same_attribute5511.5
pronoun6112.7
zero_anaphora5812.1
Unambiguous24851.7
Dialogue Mode
Resolving17436.2
Non-resolving11824.6
Supportive18839.2

Table 1. Distribution of the JAG-D benchmark (N=480N = 480). Instruction ambiguity is divided into unambiguous and ambiguous types, with the latter spanning four categories. Dialogue modes are assigned as follows: resolving for ambiguous instructions, supportive for unambiguous instructions, and non-resolving for both.

Experimental Setup

Multimodal Ambiguity Recognition

We evaluate Japanese and multilingual VLMs on the multimodal ambiguity recognition task, using our benchmark. Specifically, we provide the models with two inputs: an image and an instruction, where the instruction is either ambiguous or unambiguous according to our ambiguity categorization.

The models tested are Japanese VLMs, i.e., LLM-jp VILA2 (Sasagawa et al., [35]), Sarashina23, and EvoVLM-JP3 (Akiba et al., [1]), and two multilingual VLMs, i.e., Qwen3-VL4 (Bai et al., [2]) and Gemma35 (Gemma Team, [8]). They were prompted under different settings (zero-shot and few-shot) and in both Japanese and English for the multilingual models (the prompts are provided in Appendix B).

We evaluate the models on macro F1 score for the overall recognition performance, and we also compute per-category recall following our benchmark’s ambiguity categorization.

Visual Grounding Evaluation

We evaluate models on ambiguous visual grounding settings with dialogue history integration as disambiguation context. Our evaluation includes the Japanese versions of two widely used grounding models, GLIP and MDETR, trained on both English datasets (Krishna et al., 2017; Hudson and Manning, [10]) and Japanese datasets (Nakayama et al., [30]; Ueda et al., [41]). We also evaluate Qwen3-VL, which has demonstrated strong multilingual performance and visual grounding capabilities (Bai et al., [2]).

The evaluation pipeline comprises a two-phase evaluation protocol, designed to isolate grounding performance from raw user instruction (first phase) and the benefit of additional dialogue context to solve ambiguous scenarios (second phase). From the ones mentioned, Qwen is the only model that can handle long input, such as a complete dialogue history, while standard visual grounding models usually process short sentences or a list of phrases only. Consequently, we leverage an established method in the literature, by Willemsen et al. [42], to integrate the dialogue information into a single simple input sentence for the second-phase grounding task for GLIP and MDETR. This integration method consists of using an LLM (GPT-5.27 in our case) to retrieve the important information from a dialogue and condense it into a shorter textual input. Both the Qwen’s prompt and the one used for the dialogue integration can be found in Appendix C.

The metrics used in the visual grounding evaluation are mean Intersection over Union (mIoU) on the top-1 prediction and Acc@τ\tau, measuring if the model’s top-scoring bounding box attains IoU ≥τ\geq\tau, with τ\tau of 0.5 and 0.75 in our experiments.

Dialogue Intervention Analysis

We study which properties of the dialogue history support grounding through controlled input interventions on Qwen3-VL. Keeping the image and final ambiguous instruction fixed, we replace the Phase-2 dialogue with counterfactual variants.

We define twelve interventions across four dimensions. Reference conditions include the original Phase-2 input (“Full dialogue”) and the Phase-1 instruction (human order) without dialogue (“Order only”). Order perturbations preserve turns while disrupting sequence through random shuffling over five seeds (“Shuffle turns”) or full reversal (“Reverse order”). Content perturbations separate informativeness from token count using irrelevant in-domain dialogue (“Irrelevant substitution”), semantically neutral acknowledgements (“Neutral substitution”), or augmented histories in which irrelevant filler is added at the end (“Irrelevant suffix”) or between turns (“Irrelevant interleaving”). Selective exposure retains only subsets of the dialogue: the first or last k=2k=2 turns (“First K turns”, “Last K turns”) or speaker-specific transcripts (“Robot only”, “Human only”).

We report the following metrics: mean Intersection over Union (mIoU), [email protected], and Δ\DeltamIoU relative to “full dialogue” performance.

Results

We organise our analysis around three research questions: RQ1: Can current VLMs recognize multimodal ambiguity, and does the difficulty vary between visual and linguistic categories? RQ2: Does dialogue history improve visual grounding under ambiguity, and to what extent does the benefit differ across ambiguity types? RQ3: Does dialogue content contribute to visual grounding under ambiguity beyond surface properties such as length and format?

Our experiments reveal a consistent asymmetry across both tasks: VLMs handle linguistic ambiguity substantially better than visual ambiguity. Additionally, dialogue provides the largest grounding gains precisely where it can supply the missing linguistic referent for pronoun and zero anaphora, leaving visual ambiguity as a persistent bottleneck.

RQ1: Multimodal Ambiguity Recognition

Overall performance. Table 2 reports macro-F1 and per-category recall for all models. Multilingual VLMs clearly outperform their Japanese-specific counterparts: Qwen3-VL and Gemma3 reach the highest overall scores (83.0 and 78.5 F1-M, respectively), whereas the best Japanese model, LLM-jp VILA, achieves 60.1. The remaining Japanese models, Sarashina2 (40.3) and EvoVLM-JP (32.6), perform below the chance prediction level, underscoring a wide capability gap between model families on multimodal ambiguity recognition.

ModelOverall (F1-M)Non-Ambig (R)Visual Ambiguity (R) Same ObjVisual Ambiguity (R) Same AttrLinguistic Ambiguity (R) PronounLinguistic Ambiguity (R) Zero Ana
Qwen3-VL83.094.432.854.596.7100.0
Gemma378.585.939.752.796.793.1
LLM-jp VILA60.167.346.654.547.563.8
Sarashina240.37.798.398.2100.0100.0
EvoVLM-JP32.60.0100.0100.0100.0100.0

Table 2. Multimodal ambiguity recognition on the full test set (N=480N=480; binary ambiguous vs. non-ambiguous). We report macro-averaged F1 (F1-M) for the overall results and recall (R) for the category-specific performance. Bold represents higher overall performance. The red color represents models that are highly biased toward the ambiguous label. This table reports the few-shot models’ results only, while a complete results table can be found in Appendix § 13.

Visual and linguistic asymmetry. The per-category breakdown exposes a sharp split. Multilingual VLMs (Qwen3-VL and Gemma3) achieve near-ceiling recall on linguistic ambiguity: 96.7–100.0 on pronoun and 93.1–100.0 on zero_anaphora. In contrast, performance collapses on visual categories, with same_object recall, for instance, dropping to 32.8 and 39.7 for these same models. This asymmetry directly answers RQ1: VLMs are strong at detecting underspecification in the text (pronouns, missing arguments) but largely fail to detect that multiple regions in the image satisfy the instruction constraints, where multimodal reasoning is actually required.

Degenerate baselines. Two Japanese-specific VLMs exhibit near-degenerate decision behaviour: EvoVLM-JP predicts ambiguous for every input (0.0 non-ambiguous recall, 100.0 across all positive categories), and Sarashina2 oscillates between the two extremes across prompt variants. High per-category recall values for these models are therefore artefacts of class bias rather than genuine understanding.

RQ2: Visual Grounding under Ambiguity

Dialogue-enhanced grounding improvements. Table 3 shows that adding dialogue context substantially improves all three grounding models. For GLIP and MDETR, [email protected] roughly doubles (26.9→53.3 and 25.4→54.0, respectively) and mean IoU increases by 0.229 and 0.210 points. Qwen3-VL, the strongest model overall, still gains 15.5 [email protected] points (63.5→79.0) and 0.132 IoU points when dialogue is available.

ModelPhase[email protected] (%)[email protected] (%)mIoU#Correct
GLIPInstruction-Only26.924.40.272129
GLIPDialogue-Enhanced53.349.40.501256
MDETRInstruction-Only25.417.90.239122
MDETRDialogue-Enhanced54.032.90.449259
Qwen3-VLInstruction-Only63.556.50.585305
Qwen3-VLDialogue-Enhanced79.068.30.717379

Table 3. Visual grounding on our Japanese multimodal test set (N=480N=480) on two evaluation phases. Instruction-Only: image plus the human instruction only. Dialogue-Enhanced: image plus dialogue-informed text. #Correct: number of instances with a top-1 IoU ≥0.5\ge0.5.

An important methodological distinction underlies these results. For GLIP and MDETR, the “Dialogue-Enhanced” condition passes dialogue through a GPT-5.2 integration step that condensates the conversational turns into a single sentence before grounding. The reported gains, therefore, reflect the performance of a pipeline (grounding model + LLM integrator), not the grounding model’s intrinsic ability to use dialogue. Qwen3-VL, by contrast, receives the full conversation history directly and shows stronger evidence that a VLM can itself exploit conversational context for grounding.

Grounding visual–linguistic asymmetry. Table 4 disaggregates accuracy gains by ambiguity type and reveals a pattern that mirrors the recognition results.

Ambiguity TypeGLIPMDETRQwen3-VL
Non-ambiguous61.7 (↑\uparrow +21.8)63.3 (↑\uparrow +24.2)87.9 (↑\uparrow +0.4)
Ambiguous44.4 (↑\uparrow +31.5)44.0 (↑\uparrow +33.2)69.4 (↑\uparrow +31.5)
same_object32.8 (↑\uparrow +12.1)25.9 (↑\uparrow +12.1)62.1 (↑\uparrow +6.9)
same_attribute38.2 (↑\uparrow +18.2)38.2 (↑\uparrow +14.5)67.3 (↑\uparrow +14.5)
pronoun52.5 (↑\uparrow +44.3)54.1 (↑\uparrow +50.8)63.9 (↑\uparrow +42.6)
zero_anaphora53.4 (↑\uparrow +50.0)56.9 (↑\uparrow +53.4)84.5 (↑\uparrow +60.3)

Table 4. [email protected] (%) under Dialogue-Enhanced grounding, broken down by dataset slice (non-ambiguous vs. ambiguous instructions, and the four ambiguity categories among ambiguous ones). Values in parentheses give the absolute gain over Instruction-Only on the same slice (Δ\Delta[email protected]).

Linguistic categories benefit dramatically: zero-anaphora gains reach +50.0, +53.4, and +60.3 [email protected] for GLIP, MDETR, and Qwen3-VL, respectively, consistent with the hypothesis that dialogue context restores the missing argument needed to uniquely identify the referent. Visual categories remain low even after dialogue integration, proving that the dialogue enhancement does not eliminate the visual ambiguity gap; rather, it reallocates improvements to cases where linguistic clarification can resolve previously underspecified references.

Unambiguous instructions benefits. In the non-ambiguous condition, dialogue still produces modest gains (+21.8 and +24.2 for GLIP/MDETR; +0.4 for Qwen, where the instruction-only baseline is already high at 87.5). This suggests that even grounding-sufficient instructions benefit from the additional relational and spatial context supplied by prior turns.

RQ3: Is Dialogue Content Causally Responsible?

Content, not format, drives the gains. Figure 2 reports results for twelve targeted dialogue interventions on Qwen3-VL. The most informative comparison is between the content-removal conditions, such as neutral substitution (Δ\DeltaIoU = −0.292) and irrelevant substitution (−0.279), all of which reduce ambiguous performance to near the instruction-only baseline (−0.280-0.280) despite preserving dialogue length and structure. Additionally, structural perturbations that preserve content while disrupting ordering (shuffled turns: −0.029-0.029; reversed turns: −0.072-0.072) introduce substantially smaller penalties than content removal. Similarly, retaining only the final KK turns (−0.043-0.043) preserves most of the benefit, while retaining only the first KK (−0.174-0.174) is more costly, thus consistent with an expectation that the most recent referential context is most informative for grounding. These results directly answers RQ3: the grounding improvements are attributable to the semantic content of dialogue, not to any formatting or structural artefact.

Figure 2. Change in grounding performance under targeted interventions on the dialogue history for Qwen. Points show the mean per-sample Δ\DeltaIoU vs. the full-dialogue baseline; horizontal bars are 95% confidence intervals across samples. Ambiguous instances (blue) are substantially more sensitive to interventions than non-ambiguous instances (orange). Extended results appear in Appendix § 15.

Non-ambiguous instances robustness. Across every intervention, Δ\DeltaIoU for non-ambiguous instances remains near zero (range: −0.019-0.019 to +0.007+0.007). This robustness is expected: when the instruction is already sufficient to uniquely identify the referent, dialogue carries at most redundant information, and its removal or perturbation has no systematic effect. The sharp contrast with ambiguous instances (Δ\DeltaIoU up to −0.292-0.292) provides strong evidence that the causal role of dialogue is specifically to resolve multimodal ambiguity rather than to supply generic perceptual context.

Non-resolving dialogue modest benefits. Regarding the dialogue-mode analysis (Figure 3), the

Figure 3. Qwen3-VL grounding performance ([email protected]) comparing Instruction-Only vs Dialogue-Enhanced across dialogue resolution modes: Resolving (dialogue disambiguates the referent), Supportive (order is already clear; dialogue adds context), and Non-resolving (dialogue does not disambiguate).

resolving condition yields the largest absolute gain (39→\to73% accuracy), as expected. Interestingly, the non-resolving condition still improves over instruction-only (62→\to74%), despite dialogue that was designed not to disambiguate. The intervention results suggest a partial explanation: dialogue may supply auxiliary grounding cues (spatial relations, object attributes) that are useful even when they do not narrow the set of valid referents.

Conclusion

We present a systematic analysis of ambiguity in visual grounding through JAG-D, a new Japanese test set that evaluates visual and linguistic ambiguity in human-robot interaction scenarios, providing dialogue as a disambiguation context. Across both ambiguity recognition and grounding, we observe a consistent asymmetry: Japanese and multilingual VLMs resolve discourse-driven (linguistic) ambiguity substantially better than scene-driven (visual) ambiguity. We show that current VLMs can recognize linguistic ambiguity with high confidence (93.1-100.0 recall), while performing poorly on visual ambiguity (32.8-54.5 recall). For visual grounding, dialogue context substantially improves performance for linguistic cases (up to +60.3 [email protected] for zero-anaphora), whereas visual ambiguity remains a persistent bottleneck, with 25.9 accuracy in the worst case even after dialogue enhancement. Overall, our findings identify visual ambiguity as a central unresolved challenge for robust conversational grounding and motivate models capable of reasoning over multiple modalities beyond surface-level linguistic pattern matching.

Limitations

First, our benchmark size is relatively small compared to standard grounding datasets. However, it is explicitly designed as a highly curated diagnostic test set requiring specific real-world objects and precise multi-turn dialogue. We prioritized linguistic fidelity via native Japanese annotator validation to ensure the accurate representation of complex and context-dependent phenomena like zero anaphora. Second, our findings are currently limited to Japanese; future work must explore whether this visual-linguistic asymmetry persists in a range of distinct languages. Finally, our dialogue-enhanced evaluation for GLIP and MDETR relies on an LLM-based (GPT-5.2) text integration pipeline. This was necessitated by the current lack of Japanese-specific visual grounding models capable of natively processing long-context dialogue histories. Consequently, reported gains for these architectures reflect the pipeline’s effectiveness rather than the grounding models’ intrinsic multi-turn capabilities.

Ethical Considerations

Our test set utilizes real-world human-robot interaction images from the J-CRe3 dataset [41]. In building upon this dataset, we ensured that our newly generated multi-turn dialogues and annotations did not introduce any sensitive or personally identifiable information regarding the actors in the visual sources. Furthermore, all manual dataset validation was conducted by native Japanese annotators who were compensated according to the task. Lastly, deploying multimodal models in physical robotic systems carries inherent safety risks if ambiguous commands are misinterpreted. By exposing current VLM failure modes regarding perceptual ambiguity, we aim to encourage the development of safer agents that actively seek clarification rather than executing potentially harmful actions based on underspecified instructions.

Acknowledgments

We would like to thank Hideko Habe and Yoko Matsui for their valuable contributions to the annotation of our benchmark. A part of this work was supported by JST, PRESTO Grant Number JPMJPR24TC, Japan.

JAG-D Benchmark

J-CRe3 Images

Our benchmark’s images are sampled from J-CRe3 ([41]). J-CRe3 contains egocentric human–robot interaction scenarios in real-world environments. These environments represent different elements of rooms where interactions with social robots usually take place, such as the living room, dining room, and kitchen (Figure 4).

Figure 4. J-CRe3 scenario examples

Dialogue Mode Examples

In our benchmark, we divide dialogue history data into three specific modes:

  • Resolving: the dialogue content helps in disambiguating the ambiguous human instruction.

  • Non-resolving: the dialogue doesn’t contain any relevant information for the current instruction (either ambiguous or not).

  • Supportive: the dialogue contains relevant information for the current unambiguous instruction. The last is already clear, but still, the dialogue can help the grounding model with additional context.

Table 5 contains examples for every introduced dialogue mode.

Dialogue ModeFinal OrderDialogue History
resolving“Hand me the rectangular item.”“robot”: “I see a green backpack on the couch. Is there something you need from it?” “master”: “Yes, there’s an item inside the main compartment.” “robot”: “I notice a black electronic device between your hands as you adjust the zipper. Is that the item?” “master”: “Exactly, that’s the one I need.” “robot”: “It seems to be rectangular and could be a tablet or a laptop. Should I retrieve it for you?” “master”: “Yes, please hand me the rectangular item.”
non-resolving“Hand me the mug.”“robot”: “I noticed there are two white mugs on the coffee table.” “master”: “Yes, they both look quite similar, don’t they?” “robot”: “Indeed. One is sitting right next to the yellow tissue box.” “master”: “And the other is right beside it, in front of me.” “robot”: “They’re both conveniently placed on the black coffee table.” “master”: “Could you hand me the mug?”
supportive“Hand me the notebook.”“robot”: “I’m here to assist. Is there something specific you need today?” “master”: “Yes, I need the notebook. It’s on the couch.” “robot”: “I see multiple items. Is it near anything in particular?” “master”: “The notebook is dark blue and should be to the left of the green backpack.” “robot”: “Ah, and is it under a piece of white paper with Japanese text?” “master”: “Yes, that’s the one. Hand me the notebook.”

Table 5. Example of dialogue mode in JAG-D. The Japanese version is omitted due to space constraints. Note that the real data is in Japanese.

Data Generation Pipeline

Starting from J-CRe3, the remaining data for JAG-D has been synthetically generated using a combination of three models: Qwen3-VL8, GPT-5.19, and Grounding DINO10. Figure 5 represents the data generation pipeline for our benchmark. Specifically, Qwen3-VL receive the social scenario image as input and is prompted to generate a detailed image caption. From this caption, GPT-5.1 is prompted to generate complex textual data: the ambiguous/non-ambiguous instruction and the related dialogue history. GPT-5.1 and Grounding DINO are also used to generate the bounding box candidates for the visual grounding task. The synthetic data is then human-curated and collected by Japanese native professional annotators to assess the data quality of our designed benchmark. All

JAG-D data generation pipeline combining Qwen3-VL, GPT-5.1, and Grounding DINO. The pipeline includes Qwen3-VL, GPT-5.1, Grounding DINO, dialogue history, and bounding boxes, with labels Caption, Image Caption, Ambiguity Instruction, Object Extracted, and Target Object.

Figure 5. JAG-D data generation pipeline combining Qwen3-VL, GPT-5.1, and Grounding DINO.

prompts used in the pipeline can be found in our code repository.

Ambiguity Recognition Experiment

We evaluate Japanese and multilingual VLMs on the ambiguity recognition task. Figure 6 and Figure 7 contain the zero-shot and few-shot prompts for these models. We evaluate them under the two prompt settings, in Japanese and in English, for multilingual models only. Table 6 reports the extended results for the VLMs tested. In addition, Table 7 reports fine-grained performance, comparing different categories of ambiguity.

Zero-shot Ambiguity Recognition Prompt. Look at the image and the instructions below. Determine whether the instruction is ambiguous or not. An instruction is considered "ambiguous" if the mentioned object is not clearly identifiable in the image. An instruction is considered "non-ambiguous" if the mentioned object is clearly identifiable in the image. Reply with ONLY ’ambiguous’ or ’non-ambiguous’. Instruction: \{\{instruction\}\}

Figure 6. Zero-shot ambiguity recognition prompt. The Japanese version is the same translated prompt.

Few-shot ambiguity recognition prompt

Figure 7. Few-shot ambiguity recognition prompt. The Japanese version is the same translated prompt.

ModelPromptLangAcc (%)Prec (%)Rec (%)F1 (%)F1-M (%)
Qwen3-VLFew-shotEN83.392.271.680.683.0
Qwen3-VLFew-shotJA82.592.569.479.382.1
Gemma3Few-shotEN78.882.571.176.478.5
Qwen3-VLZero-shotJA76.979.570.374.676.7
Gemma3Few-shotJA76.570.488.478.476.3
Gemma3Zero-shotJA74.277.865.170.973.8
Qwen3-VLZero-shotEN69.892.240.956.766.8
Gemma3Zero-shotEN61.569.436.247.658.6
LLM-jp VILAFew-shotJA60.460.353.056.460.1
LLM-jp VILAZero-shotJA56.757.042.248.555.6
Sarashina2Zero-shotJA52.1100.00.91.735.0
Sarashina2Few-shotJA51.950.199.166.640.3
EvoVLM-JPZero-shotJA48.348.3100.065.232.6
EvoVLM-JPFew-shotJA48.348.3100.065.232.6

Table 6. Multimodal ambiguity recognition with full classification metrics on the complete test set (N=480N=480): accuracy, precision, recall, micro-F1, and macro-F1 (F1-M), spanning prompt protocol (few-shot vs. zero-shot) and language for multilingual VLMs. Rows are sorted by accuracy (descending).

ModelConfigNon-Ambig RecallVisual Ambiguity Same ObjVisual Ambiguity Same AttrLinguistic Ambiguity PronounLinguistic Ambiguity Zero Ana
Qwen3-VLFS-EN94.432.854.596.7100.0
Qwen3-VLFS-JA94.832.852.795.194.8
Qwen3-VLZS-JA83.156.941.886.993.1
Qwen3-VLZS-EN96.86.914.572.167.2
Gemma3FS-EN85.939.752.796.793.1
Gemma3FS-JA65.377.676.4100.098.3
Gemma3ZS-JA82.748.338.290.281.0
Gemma3ZS-EN85.120.727.347.548.3
LLM-jp VILAFS-JA67.346.654.547.563.8
LLM-jp VILAZS-JA70.241.432.741.053.4
Sarashina2FS-JA7.798.398.2100.0100.0
Sarashina2ZS-JA100.01.70.00.01.7
EvoVLM-JPFS-JA0.0100.0100.0100.0100.0
EvoVLM-JPZS-JA0.0100.0100.0100.0100.0

Table 7. Fine-grained recall (%) for multimodal ambiguity recognition: Non-Ambig reports recall on gold-unambiguous instances; visual (same_object, same_attribute) vs. linguistic (pronoun, zero_anaphora) columns condition on gold ambiguity type among ambiguous instances. Each row is one fully specified model configuration (few-shot/zero-shot and English/Japanese).

Visual Grounding Experiment

The visual grounding experiment compares two grounding models (GLIP and MDETR) and Qwen3-VL as a multilingual VLM. The models are tested in two different phases:

  1. Instruction-only: the models receive as input only the human instruction, which can be ambiguous or not.

  2. Dialogue-enhanced: in addition to the previous setting, the models receive a dialogue history as additional context, to potentially disambiguate the human instruction.

GLIP and MDETR can’t handle long context, such as the dialogue history. For this reason, we use a method from Willemsen et al. [42] to integrate the dialogue information into a single instruction (Figure 8).

GPT-5.2 dialogue integration prompt for the visual grounding experiment

Figure 8. GPT-5.2 dialogue integration prompt for the visual grounding experiment.

In contrast, Qwen3-VL handles long context, and we can pass it the dialogue history directly. Figure 9 and Figure 10 contains the instruction-only and the dialogue-enhanced prompts for Qwen, respectively.

Qwen3-VL instruction-only prompt for the visual grounding experiment

Figure 9. Qwen3-VL instruction-only prompt for the visual grounding experiment.

Qwen3-VL dialogue-enhanced prompt for the visual grounding experiment

Figure 10. Qwen3-VL dialogue-enhanced prompt for the visual grounding experiment.

Dialogue Intervention Experiment

To understand how the dialogue history impacts visual grounding in Qwen3-VL, we run controlled counterfactual interventions on the dialogue-enhanced grounding evaluation phase. The interventions are the following:

  • Full dialogue: the original, unaltered dialogue history (Phase 2 baseline).

  • Order only: the human instruction only, with no dialogue (Phase 1 baseline).

  • Shuffled turns: the same dialogue turns in a random order.

  • Reversed turns: the dialogue with its turn order reversed.

  • Irrelevant filler: an unrelated dialogue, length-matched to the original to control for token volume.

  • Last KK turns: only the final KK turns of the dialogue (KK=2).

  • First KK turns: only the initial KK turns of the dialogue (KK=2).

  • Neutral substitution: the dialogue is replaced with generic robot/human turns matched to the original length.

  • Irrelevant suffix: the true dialogue followed by irrelevant turns.

  • Irrelevant interleaving: the original dialogue with one irrelevant line inserted after each original turn.

  • Robot turns only: the dialogue restricted to robot utterances.

  • Human turns only: the dialogue restricted to human utterances.

The results of the intervention experiment are reported in Table 8.

InterventionAmbiguousAmbiguousAmbiguousNon-ambiguousNon-ambiguousNon-ambiguous
mIoUΔ\DeltaIoU[email protected]mIoUΔ\DeltaIoU[email protected]
Full dialogue (baseline)0.6410.0000.6940.7870.0000.879
Neutral substitution0.349-0.2920.3620.7910.0040.887
Order only0.361-0.2800.3790.7940.0070.875
Irrelevant substitution0.362-0.2790.3790.786-0.0010.883
First KK turns0.467-0.1740.4870.786-0.0010.879
Human only0.565-0.0760.6080.785-0.0020.879
Reversed turns0.569-0.0720.6080.768-0.0190.843
Last KK turns0.599-0.0430.6290.7870.0000.879
Shuffled turns0.612-0.0290.6470.774-0.0130.863
Robot only0.617-0.0240.6640.785-0.0020.875
Irrelevant interleaving0.618-0.0240.6640.781-0.0060.879
Irrelevant suffix0.627-0.0140.6720.7920.0050.887

Table 8. Qwen3-VL visual grounding under dialogue counterfactuals. Full dialogue: original unedited dialogue history from our benchmark. mIoU: mean IoU of the top-1 prediction; [email protected]: accuracy at IoU ≥0.5\ge0.5. Δ\DeltaIoU: paired mean IoU gap versus full dialogue on the same instance. Results are split into ambiguous (n=232n=232) vs. non-ambiguous (n=248n=248) instruction subsets.

References

  1. [1]Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. 2025. Evolutionary optimization of model merging recipes. Nature Machine Intelligence, 7(2):195–204.DOI
  2. [2]Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, and 45 others. 2025. Qwen3-VL technical report. Preprint, arXiv:2511.21631.arxiv.org/abs/2511.21631
  3. [3]Yevgeni Berzak, Andrei Barbu, Daniel Harari, Boris Katz, and Shimon Ullman. 2015. Do you see what I mean? visual resolution of linguistic ambiguities. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1477–1487.arxiv.org/abs/1603.08079
  4. [4]Stefan Constantin, Fevziye Irem Eyiokur, Dogucan Yaman, Leonard Bärmann, and Alex Waibel. 2023. Interactive multimodal robot dialog using pointing gesture recognition. In Computer Vision – ECCV 2022 Workshops, pages 640–657.
  5. [5]Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, Jose M. F. Moura, Devi Parikh, and Dhruv Batra. 2017. Visual dialog. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  6. [6]David DeVault and Matthew Stone. 2009. Learning to interpret utterances using dialogue history. In Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009), pages 184–192.DOI
  7. [7]Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman. 2010. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88(2):303–338.DOI
  8. [8]Gemma Team. 2025. Gemma 3 technical report. Preprint, arXiv:2503.19786.arxiv.org/abs/2503.19786
  9. [9]Julio Gonzalo, Anselmo Penas, and Felisa Verdejo. 1999. Lexical ambiguity and information retrieval revisited. In 1999 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora.
  10. [10]Drew A. Hudson and Christopher D. Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).arxiv.org/abs/1902.09506
  11. [11]Shun Inadumi, Seiya Kawano, Akishige Yuguchi, Yasotumo Kawanishi, and Koichiro Yoshino. 2024. A gaze-grounded visual question answering dataset for clarifying ambiguous Japanese questions. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 558–571.arxiv.org/abs/2403.17545
  12. [12]Shun Inadumi, Nobuhiro Ueda, and Koichiro Yoshino. 2025. Disambiguating reference in visually grounded dialogues through joint modeling of textual and multimodal semantic structures. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11183–11198.
  13. [13]Jihyoung Jang and Hyounghun Kim. 2026. AQua: Toward strategic response generation for ambiguous visual questions. In The Fourteenth International Conference on Learning Representations.arxiv.org/abs/2603.07394
  14. [14]Pu Jian, Donglei Yu, Wen Yang, Shuo Ren, and Jiajun Zhang. 2025. Teaching vision-language models to ask: Resolving ambiguity in visual questions. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 361–3638.arxiv.org/abs/2507.13773
  15. [15]Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. 2021. Mdetr - modulated detection for end-to-end multi-modal understanding. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 1760–1770.arxiv.org/abs/2104.12763
  16. [16]Antonia Karamolegkou, Oliver Eberle, Phillip Rust, Carina Kauf, and Anders Søgaard. 2025. Trick or neat: Adversarial ambiguity and language model evaluation. In Findings of the Association for Computational Linguistics: ACL 2025, pages 18542–18561.
  17. [17]Andrej Karpathy and Li Fei-Fei. 2017. Deep visual-semantic alignments for generating image descriptions. IEEE Trans. Pattern Anal. Mach. Intell., 39(4):664–676.
  18. [19]Fang Kong and Guodong Zhou. 2010. A tree kernel-based unified framework for Chinese zero anaphora resolution. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 882–891. Association for Computational Linguistics.
  19. [20]Satwik Kottur, Seungwhan Moon, Alborz Geramifard, and Babak Damavandi. 2021. SIMMC 2.0: A task-oriented dialog dataset for immersive multimodal conversations. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4903–4912.
  20. [23]Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuан Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao. 2022. Grounded language-image pre-training. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10955–10965.arxiv.org/abs/2112.03857
  21. [24]Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta, Noah Smith, and Yejin Choi. 2023. We’re afraid language models aren’t modeling ambiguity. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 790–807.arxiv.org/abs/2304.14399
  22. [25]Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. 2024. Grounding DINO: Marrying dino with grounded pre-training for open-set object detection. In Proceedings of the 18th European Conference on Computer Vision (ECCV), pages 38–55.arxiv.org/abs/2303.05499
  23. [26]Panzhong Lu, Xin Zhang, Meishan Zhang, and Min Zhang. 2022. Extending phrase grounding with pronouns in visual dialogues. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7614–7625.arxiv.org/abs/2210.12658
  24. [27]Xinyu Ma, Xuebo Liu, Derek F. Wong, Jun Rao, Bei Li, Liang Ding, Lidia S. Chao, Dacheng Tao, and Min Zhang. 2024. 3AM: An ambiguity-aware multimodal machine translation dataset. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 1–13.arxiv.org/abs/2404.18413
  25. [28]Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan Yuille, and Kevin Murphy. 2016. Generation and comprehension of unambiguous object descriptions. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 11–20.arxiv.org/abs/1511.02283
  26. [29]Jinpeng Mi, Jianzhi Liu, Song Tang, Qingdu Li, and Jianwei Zhang. 2020. Interactive natural language grounding via referring expression comprehension and scene graph parsing. Frontiers in Neurorobotics, Volume 14 - 2020.DOI
  27. [30]Hideki Nakayama, Akihiro Tamura, and Takashi Ninomiya. 2020. A visually-grounded parallel corpus with phrase-to-region linking. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4204–4210.
  28. [31]Heejeong Nam, Jinwoo Ahn, Keummin Ka, Jiwan Chung, and Youngjae Yu. 2025. Vague: Visual contexts clarify ambiguous expressions. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1537–1547. Tsuyoshi Ono and Sandra A Thompson. 1997. Deconstructing "zero anaphora" in japanese. In Annual Meeting of the Berkeley Linguistics Society, pages 481–491.
  29. [32]Arum Park, Seunghee Lim, and Munpyo Hong. 2015. Zero object resolution in Korean. In Proceedings of the 29th Pacific Asia Conference on Language, Information and Computation (PACLIC), pages 439–448, Shanghai, China.
  30. [33]Bryan A. Plummer, Liwei Wang, Christopher M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2017. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. International Journal of Computer Vision, 123(1):74–93.
  31. [34]Anna Rohrbach, Marcus Rohrbach, Ronghang Hu, Trevor Darrell, and Bert Schiele. 2016. Grounding of Textual Phrases in Images by Reconstruction, page 817–834.DOI
  32. [35]Keito Sasagawa, Koki Maeda, Issa Sugiura, Shuhei Kurita, Naoki Okazaki, and Daisuke Kawahara. 2025. Constructing multimodal datasets from scratch for rapid development of a Japanese visual language model. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (System Demonstrations), pages 470–484.
  33. [36]Stefan Schouten, Peter Bloem, Ilia Markov, and Piek Vossen. 2023. Reasoning about ambiguous definite descriptions. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 4479–4484.arxiv.org/abs/2310.14657
  34. [37]Kazuhiro Seki, Atsushi Fujii, and Tetsuya Ishikawa. 2002. A probabilistic method for analyzing Japanese anaphora integrating zero pronoun detection and resolution. In COLING 2002: The 19th International Conference on Computational Linguistics.
  35. [38]Elias Stengel-Eskin, Jimena Guallar-Blasco, Yi Zhou, and Benjamin Van Durme. 2023. Why did the chicken cross the road? rephrasing and analyzing ambiguous questions in VQA. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10220–10237.
  36. [39]Alberto Testoni, Barbara Plank, and Raquel Fernández. 2025. RacQUET: Unveiling the dangers of overlooked referential ambiguity in visual LLMs. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 23627–23647.
  37. [40]Tao Tu, Qing Ping, Govindarajan Thattai, Gokhan Tur, and Prem Natarajan. 2021. Learning better visual dialog agents with pretrained visual-linguistic representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5622–5631.arxiv.org/abs/2105.11541
  38. [41]Nobuhiro Ueda, Hideko Habe, Akishige Yuguchi, Seiya Kawano, Yasutomo Kawanishi, Sadao Kurohashi, and Koichiro Yoshino. 2024. J-CRe3: A Japanese conversation dataset for real-world reference resolution. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 9489–9502.
  39. [42]Bram Willemsen, Livia Qian, and Gabriel Skantze. 2023. Resolving references in visually-grounded dialogue via text generation. In Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 457–469.arxiv.org/abs/2309.13430
  40. [43]Linhui Xiao, Xiaoshan Yang, Xiangyuan Lan, Yaowei Wang, and Changsheng Xu. 2026. Toward visual grounding: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 48(3):2749–2771.DOI
  41. [44]Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. 2016. Modeling context in referring expressions. In Computer Vision – ECCV 2016, pages 69–85.arxiv.org/abs/1608.00272
  42. [45]Yunan Zeng, Yan Huang, Jinjin Zhang, Zequn Jie, Zhenhua Chai, and Liang Wang. 2024. Investigating compositional challenges in vision-language models for visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14141–14151.DOI
  43. [46]Hanbo Zhang, Jie Xu, Yuchen Mo, and Tao Kong. 2024. InViG: Benchmarking open-ended interactive visual grounding with 500k dialogues. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 5508–5518.DOI
  44. [47]Hanwang Zhang, Yulei Niu, and Shih-Fu Chang. 2018. Grounding referring expressions in images by variational context. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).arxiv.org/abs/1712.01892
  45. [48]Kankan Zhou, Eason Lai, Kyriakos Mouratidis, and Jing Jiang. 2025. FOCUS: Evaluating pre-trained vision-language models on underspecification reasoning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 25765–25784.

Paper details

vlnc:W_nw348e2j
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 1

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents