WorksComputer Science
Ask Precisely, Assume Wisely: Interactive Evaluation of Ambiguity Resolution in Large Language Models
Abstract
Ambiguity is a core challenge in human–AI interaction, where models must decide when to make assumptions and when to seek clarification. Existing approaches to ambiguity evaluation largely focus on coverage over multiple interpretations using static metrics, offering limited insight into how efficiently a model converges toward a single user-intended interpretation through interaction. We introduce a preference-grounded simulated user framework for controlled, multi-turn evaluation of ambiguity resolution, grounding simulator responses strictly in user disambiguation preferences to avoid hallucination and over-disclosure. Alongside this, we define three observable dissatisfaction signals — incorrect assumptions, repeating clarifications, and irrelevant clarifications — that characterize interaction quality beyond task accuracy. Our analysis reveals a clear trade-off: models that frequently seek clarification tend to over-clarify and prolong interactions, while assumption-driven models are more prone to misaligning with user intent. We demonstrate that system prompts optimized on these dissatisfaction signals achieve a Pareto-optimal balance across all three metrics, reducing overall error rates by up to 7.65%, underscoring the value of our framework in designing agents that ask precisely and assume wisely.
Introduction
Ambiguity is a fundamental challenge in human–AI interaction. Real-world user queries are often under-specified, requiring language models (LMs) to infer missing intent or engage in clarification through interaction Keyvan and Huang [ [12]. While recent LLMs have demonstrated strong capabilities across reasoning and conversational tasks, effectively resolving ambiguity remains difficult. Models that aggressively assume a single interpretation risk misunderstanding user intent Berant et al. [ [4], whereas models that over-rely on clarification can prolong interactions through repetitive or irrelevant questions Acikgoz et al. [ [1]; Andukuri et al. [ [2]. Designing agents that can ask precisely and assume wisely therefore remains an open problem in interactive language systems.
Despite growing interest in interactive evaluation, existing approaches remain limited in two important ways. First, evaluation frameworks largely measure final task success, overlooking the quality and efficiency of the conversational process used to arrive at the answer Zhang and Choi [ [35]; Min et al. [ [21]. Second, many interactive evaluations rely on LLM-only user simulators to facilitate multi-turn interaction. While flexible, these proxies are difficult to control and reproduce, often prone to hallucination and over-disclosure of information Dou et al. [ [8]; Seshadri et al. [ [27].
We address both limitations by introducing a preference-grounded user simulator for evaluating ambiguity resolution across multiple turns. We formalize each ambiguous query through a user disambiguation preference set as the minimal set of specifications required to uniquely identify the user’s intended interpretation. This simulator restricts its responses strictly to these preferences, enabling reproducible and controlled evaluation without risk of information leakage. Derived from this are three observable dissatisfaction signals — incorrect assumptions, repeating clarifications, and irrelevant clarifications — that together provide a more nuanced view of how efficiently a model navigates ambiguity during interaction.
Our analysis spanning ambiguous question answering, semantic parsing and text-to-visualization benchmarks reveal a clear trade-off: assumption-driven strategies have higher frequency of wrong attempts while clarification-seeking models prolong conversations while asking ill-scoped questions. By prompt-optimizing on our dissatisfaction signals (DisPO), we achieve pareto optimal balance, leading to cleaner interactions without sacrificing task success. Our overall contributions include:
We propose a preference-grounded user simulator for controlled and reproducible evaluation towards ambiguity resolution in a multi-turn setting.
We introduce three dissatisfaction signals that correlate to inefficient ambiguity handling in conversations.
We demonstrate a clear trade-off between assumption-driven and clarification-seeking strategies, where optimization on our framework yields systems that ask only necessary and precise questions.
Problem Formulation
We formalize ambiguity and define how different disambiguation preferences correspond to different user interpretations. We then introduce our simulated user framework, which grounds interactions on these preferences, and describe how it enables measuring user dissatisfaction signals as indicators of ambiguity resolution effectiveness.
User Disambiguation Preferences
User Disambiguation Preferences are resolution strategies that guide an ambiguous query toward a specific intended interpretation A query is ambiguous if the cardinality of its solution space satisfies , where denotes the set of valid solutions consistent with the query. For example, in Figure 1, seniority could be interpreted in multiple ways: age, organizational role, or joining date, leading to multiple solutions. To resolve ambiguity toward a target solution , we introduce a user disambiguation preference set ,such that admits a unique solution . We require to satisfy: (i) completeness, where removing any reintroduces ambiguity; (ii) non-redundancy, where no specification can be removed or changed without altering solution ; and (iii) disentangled scope, where each resolves a distinct aspect of ambiguity, either independently or conditioned on previously resolved ambiguities. For example, in Figure 1, the user preferences resolve: (1) how performance should be evaluated; (2) what constitutes seniority; (3) the threshold or parameter associated with that definition (e.g., age threshold or role hierarchy). Each of these are disjoint decision points when a solution is computed. Different assignments of therefore model different user interpretation toward a specific task. The cardinality symbolizes the extent of uncertainty characterized by the number of decision points at which the model encounters multiple plausible interpretations. This definition extends to queries whose interpretation space is unbounded. In such cases, contains only the specifications the user cares about, and every aspect left out is one for which any reasonable assumption is acceptable. is therefore finite for a given user, even when the space of possible interpretations is not.
Figure 1. Demonstrates different ambiguity resolution styles and signals of user dissatisfaction during interaction. Users may choose different resolution strategies towards a given ambiguity. Solution-driven models often make incorrect assumptions while clarification-seeking models ask repeating or irrelevant clarifications.
Preference-Grounded User Simulator
We propose a user simulator that generates responses grounded on the disambiguation preference set corresponding a given interpretation. Prior work on multi-turn evaluation commonly relies on a single LLM call to serve as user simulator. While they can produce natural conversations Dou et al. [ [8], the interactions remain difficult to control and reproduce consistently across evaluations, and are prone to hallucinated user behavior and unnecessary information disclosure beyond what is required to resolve the ambiguity.
To address these limitations, we design a simulator where the user vocabulary is restricted to the preference set (see Figure 2). At each interaction turn , the evaluated agent may either provide a candidate solution or seek clarification. If a solution is proposed, the framework checks whether the generated answer satisfies all specifications in the user preference set . Any unresolved or incorrectly assumed ambiguity is rectified by responding with: . If the agent instead asks a clarification question, the framework determines whether the question meaningfully targets a specific ambiguity dimension resolved by some . In such cases, the simulated user returns the associated clarification response ; otherwise, the user responds with “Please proceed with your best judgement” Vijayvargiya et al. [ [29], if the set is empty. The conversation terminates when agent’s solution aligns with . Our goal is not to faithfully emulate real users, but to enable controlled information injection into interaction trajectories for systematically analyzing ambiguity handling by models Vijayvargiya et al. [ [29].
Figure 2. Preference-Grounded User Simulator that rectifies or clarifies based on preference set .
Measuring Ambiguity Resolution
Beyond task success, our framework measures how efficiently the model resolves ambiguity toward the user’s intended interpretation, focusing on three dissatisfaction signals: incorrect assumptions, repeated clarifications, and irrelevant clarifications Sandbank et al. [ [25]; Shea et al. [ [28]. We formalize these dissatisfaction signals through two indicator functions. The *Answer indicator function , takes the value 1 if the user reveals preference at turn and 0 otherwise. Similarly, we define a Clarification indicator function , which takes the value 1 if the agent asks clarification at turn for any preference that may or may not be part of , else 0. denotes complement of , suggesting those questions that resolve a preference , i.e. irrelevant towards resolving ambiguity.
Incorrect Assumption. We count assumptions that do not match the user’s preference throughout the conversation. This is tracked by counting the number of repair utterances from the user Walker et al. [ [30] when clarification is not asked.
Repeating Clarification. We capture the number of times the agent asks a clarification question, whether relevant to or not, that has already been previously asked or resolved by the user in its response earlier, thereby introducing unnecessary conversational repetition and user frustration.
Irrelevant Clarification. We measure how often the agent asks clarification questions unrelated to resolving the intended ambiguity. Formally, this corresponds to questions where .
We scale each irrelevant question detected by , where denotes the number of sensible interpretations (cardinality of assignment space of ) to ambiguous query , and is a monotonic function that stores the count of preferences that were either revealed by the user or the agent got them right at turn : , . The scaling factor is added to allow agents room to ask clarification questions more freely earlier when the user intent is unclear, and be more restrictive when a certain amount of information has been revealed over the turns. During the course of the conversation as the simulated user reveals its preference, the solution space reduces by a multiplicative factor and so does the room for asking irrelevant clarifications. Consequently, the penalty increases exponentially over the course of the interaction.
These metrics are not directly comparable, as they arise from different decision contexts and opportunity spaces. Incorrect assumptions occur when the agent chooses to act, while clarification-based errors occur when the agent chooses to ask. Repeating clarification depends on interaction history, whereas irrelevant clarification depends on the remaining ambiguity space. This creates a turn-wise asymmetry: early interactions allow many irrelevant clarifications but few repetitions, while later turns increase repetition likelihood and space for irrelevant queries shrink. To account for this, we normalize irrelevant clarification with lower penalties in early turns and higher penalties in later turns, aligning it with repeated clarification. For example, in a query with two interpretations and a single ambiguity , two early irrelevant clarifications is equal to one repeated clarification.
Dataset and Metrics
AmbigQA covers ambiguity in open-domain question answering tasks Min et al. [ [21]. Built on top of NaturalQuestions Kwiatkowski et al. [ [14], the dataset provides annotated disambiguation variants for each ambiguous query, which we leverage to construct user disambiguation preference sets. We split 1,142 unique queries into 3,649 variants, with an average of 3.2 interpretations per query and an . Given the open-ended nature of responses, we employ GPT-5 (reasoning) as LLM-judge Zheng et al. [ [39] to evaluate, instead of relying on lexical similarity metrics such as EDIT-F1 or BLEU Papineni et al. [ [23], which are overly strict and have higher false positives.
Ambrosia covers ambiguity in text-to-SQL semantic parsing tasks Saparina and Lapata [ [26]. The dataset captures diverse forms of ambiguity in tabular queries, including scope ambiguity, attachment ambiguity, and vagueness, along with their corresponding unambiguous variants, which are used to construct the preference set. From 1,141 unique ambiguous queries, we build a total interaction set of 2,689 instances, with an average of 2.36 interpretations per query and . For evaluation, we measure execution accuracy by comparing the execution result of the predicted SQL query with that of the gold query.
nvBench2.0 covers ambiguous queries in Text2VIS systems, where models generate visualizations from tabular data Luo et al. [ [19]. It introduces ambiguity on existing table-query pairs from nvBench Luo et al. [ [17] and BIRD Li et al. [ [15], and provides stepwise disambiguation reasoning paths, which we leverage to construct user preference sets. From 747 unique ambiguous queries, we derive 2,251 interaction instances, with an average of 3.01 interpretations per query and . Following Luo et al. [ [18, 17], we evaluate predictions by comparing the generated Vega-Lite specifications with the corresponding gold.
We expand each query into all possible interpretations, enabling ambiguity resolution to be evaluated over the complete interpretation space rather than being biased toward specific user preferences. In addition to task success, we report the Clean Interaction Rate (CIR%), which measures the percentage of benchmark queries whose conversations contain no Incorrect Assumption, Repeating Clarification and Irrelevant Clarification.
Evaluation
RQ1 How different models and strategies effectively resolve ambiguity?
RQ2 Can dissatisfaction signals reliably guide when agents should ask clarifying questions versus make assumptions?
RQ3 How resolution strategies change with increasing ambiguity?
RQ4 Can our user simulator overcome limitations of existing LLM-only baselines?
Ambiguity Resolution in LLMs (RQ1)
We evaluate LLMs and different clarification strategies by measuring their ability to resolve ambiguity through interactions with our user simulator.
Evaluating Models
Framework. We evaluate both open-weight: Phi-4 Microsoft Research [2024], Deepseek-R1-Distill-Qwen-32B (abbrv. as Deepseek-R1-Distill) DeepSeek-AI [2025]; and proprietary models: Claude-Opus-4.6 Anthropic [2026] and GPT-5.4 OpenAI [2026]. Each model interacts (upto 10 turns) with our user simulator. No explicit instruction is given to indicate that the query is ambiguous, and all models operate in a zero-shot setting (see Appendix § 12 for exact prompts). We use GPT-5 (chat) throughout to drive our user simulator.
Results. In the absence of explicit indicators about query ambiguity, models exhibit an inherent solution-driven behavior and attempt to resolve queries in the first turn Zhang et al. [2025]. As a result, incorrect assumptions occur frequently, while repeating and irrelevant clarifications remain negligible (Table 1). Models with shorter conversation lengths generally achieve higher CIR, suggesting a greater ability to infer the intended interpretation in the first turn. Despite user repair utterances following incorrect assumptions, open-weight models consistently underperform, particularly on the open-domain ambiguity benchmark AmbigQA, where their accuracy is nearly half that of closed-weight models, indicating weaker pre-training knowledge. NvBench2.0 is especially challenging for models to infer correctly, as reflected by its comparatively lower CIR, with models make at least one incorrect assumption in most interactions.
| Model | Acc(%) | IncAssmp | RepClrf | IrrelClrf | CIR(%) | |
| AmbigQA | ||||||
| Phi-4 | 34.34 | 2.03 | 1.13 | 0.02 | 0.02 | 21.54 |
| DeepSeek-R1-Distill | 30.79 | 2.05 | 1.07 | 0.06 | 0.04 | 23.36 |
| Claude-Opus-4.6 | 65.11 | 1.85 | 0.88 | 0.05 | 0.02 | 30.50 |
| GPT-5.4 | 63.47 | 1.90 | 0.81 | 0.12 | 0.02 | 31.84 |
| Ambrosia | ||||||
| Phi-4 | 44.55 | 1.92 | 0.98 | 0.01 | 0.00 | 34.81 |
| DeepSeek-R1-Distill | 45.15 | 1.74 | 0.79 | 0.00 | 0.00 | 36.26 |
| Claude-Opus-4.6 | 53.48 | 1.83 | 0.89 | 0.00 | 0.00 | 37.60 |
| GPT-5.4 | 66.20 | 1.84 | 0.89 | 0.03 | 0.00 | 36.04 |
| nvBench 2.0 | ||||||
| Phi-4 | 49.27 | 1.95 | 1.32 | 0.00 | 0.00 | 16.30 |
| DeepSeek-R1-Distill | 51.78 | 2.01 | 1.28 | 0.00 | 0.04 | 15.77 |
| Claude-Opus-4.6 | 69.44 | 1.84 | 1.17 | 0.00 | 0.00 | 19.24 |
| GPT-5.4 | 69.48 | 1.83 | 1.16 | 0.00 | 0.00 | 19.90 |
Table 1. Reports task success, measured by accuracy Acc(%), alongside interaction efficiency through Clean Interaction Rate CIR(%) and average conversation length , after the model aligns with the user’s intended interpretation of an ambiguous query. and denote metrics where higher and lower values are preferable, respectively. Closed-weight models generally achieve stronger overall performance and efficiency.
Evaluating Resolution Strategies
Framework. In this setup, we pick one of the best performers i.e. GPT-5.4, and evaluate different ambiguity resolution strategies. For baselines, we use (i) None: where no explicit guidance for handling ambiguity is provided; (ii) Clrf: model is instructed to ask clarification upon detecting ambiguity; (iii) Self-Ask: implements chain-of-thought to determine if and what clarification to ask — “Is a follow-up question needed here?”, adapted to shift focus from task decomposition Press et al. [ [24]; (iv) Intent-Sim: determines if clarification is necessary by estimating entropy over possible user interpretations Zhang and Choi [ [36]. Clarification is asked when entropy exceeds 0.5 (sample size , Temperature ).
Results. For a given model (GPT-5.4), accuracy remains largely stable across ambiguity resolution strategies after attaining user alignment, reflecting a capability ceiling that each model reaches regardless of how it resolves ambiguity (as empirically observed in Table 2). What differs across strategies is the quality and efficiency of the interaction path used which is captured via CIR(%). Under NONE, models answer directly, resulting in shorter conversations but higher incorrect assumptions. In contrast, CLRF reduces incorrect assumptions by aggressively seeking clarification, but at the cost of longer interactions and increased repeated and irrelevant clarifications from nearly zero to average 0.97 and 1.05 respectively, revealing inefficiencies in unconstrained clarification-seeking. SELF-ASK and INTENT-SIM occupy a middle ground, maintaining shorter conversations while keeping repeated and irrelevant clarifications minimal, thereby achieving a more balanced trade-off between assumption risk and conversational overhead.
| Strategy | Acc(%)↑ | |Conv|↓ | IncAssmp↓ | RepClrf↓ | IrrelClrf↓ | CIR(%)↑ |
| AmbigQA | ||||||
| None | 63.47 | 1.90 | 0.81 | 0.12 | 0.02 | 31.84 |
| Self-Ask | 65.72 | 1.96 | 0.85 | 0.14 | 0.02 | 30.25 |
| Intent-Sim | 62.02 | 1.85 | 0.86 | 0.07 | 0.02 | 29.60 |
| DisPO | 64.68 | 2.10 | 0.68 | 0.15 | 0.11 | 39.49 |
| Clrf | 59.80 | 4.20 | 0.28 | 1.83 | 0.91 | 24.61 |
| Ambrosia | ||||||
| None | 66.20 | 1.84 | 0.89 | 0.03 | 0.00 | 36.04 |
| Self-Ask | 67.39 | 1.90 | 0.84 | 0.07 | 0.02 | 37.45 |
| Intent-Sim | 68.50 | 1.83 | 0.82 | 0.02 | 0.02 | 35.78 |
| DisPO | 68.72 | 2.00 | 0.68 | 0.09 | 0.10 | 44.78 |
| Clrf | 61.66 | 3.33 | 0.34 | 0.99 | 1.64 | 17.36 |
| nvBench 2.0 | ||||||
| None | 69.48 | 1.83 | 1.16 | 0.00 | 0.00 | 19.90 |
| Self-Ask | 69.35 | 1.95 | 1.06 | 0.04 | 0.11 | 26.03 |
| Intent-Sim | 68.50 | 1.91 | 1.13 | 0.03 | 0.04 | 20.61 |
| DisPO | 69.30 | 2.05 | 0.93 | 0.03 | 0.20 | 32.03 |
| Clrf | 70.06 | 2.30 | 0.73 | 0.08 | 0.60 | 29.63 |
Table 2. Interaction efficiency for different ambiguity resolution strategies. DisPO achieves a Pareto-optimal trade-off between incorrect assumptions and repeated or irrelevant clarifications.
Dissatisfaction-Guided Resolution (RQ2)
We use Dissatisfaction-guided Prompt Optimization (DisPO) to define instructions on when agents should ask clarifying questions and how precise those clarifications should be, using dissatisfaction signals collected from conversations Cheng et al. [ [6]; Yang et al. [ [33]. The optimization leverages conversations generated by both extremes of solution-driven (None) and clarification-seeking (Clrf) models. We collected 100 random GPT-5.4 conversations from each benchmark (AmbigQA, Ambrosia, nvBench2.0) under both settings, yielding 600 conversations. Using GitHub Copilot CLI GitHub [ [10] in an automated review loop, we inspected each conversation alongside its dissatisfaction signal to identify recurring mistake patterns Liu et al. [ [16], and iteratively refined the prompt to address them (exact prompt in Appendix § 12). To reduce order sensitivity, we repeated the process with the conversations in reverse order and retained only the instructions common to both resulting prompts. We performed no further optimization on this sample to avoid overfitting, and the final DisPO prompt was fixed. Table 2 shows that DisPO achieves the highest CIR% across all benchmarks, establishing a target for models to strive toward in achieving efficient ambiguity resolution.
Impact of Increasing Ambiguity (RQ3)
Figure 3 shows how dissatisfaction signals get affected across different ambiguity resolution strategies a extent of ambiguity increases. Incorrect assumption across all strategies increase, suggesting that there is more room for getting an interpretation wrong if the agent is making an assumption. Same goes for repeating clarification, as the model now has a wider base where its questions resolved earlier resurface in a later turn. However the trend is inconsistent as repetition occurs minimally in Ambrosia and nvBench 2.0, that have tasks more goal-oriented than AmbigQA, and therefore require fewer exploratory clarification turns. Conversely, irrelevant clarifications decrease slightly as increases, suggesting that clarifying questions are more likely to align with at least one plausible interpretation as the interpretation space expands, leaving fewer questions truly irrelevant. The DisPO mode achieves a Pareto-optimal profile across all three metrics and all depth levels, consistently maintaining lower incorrect assumptions than solution-driven modes while avoiding the clarification degradation exhibited by Clrf. Across datasets, repetitive and misdirected clarifications are substantially more common in AmbigQA and Ambrosia than in nvBench 2.0. This is largely because nvBench 2.0 involves structured chart-generation tasks with ambiguity constrained on tables, whereas AmbigQA contains open-domain questions with broader interpretation spaces and ever-changing world-knowledge dependencies.
Figure 3. Variation of dissatisfaction signals as extent of ambiguity increases going vertically down. Individual bars report average incorrect assumptions (blue), repeating clarifications (red), and irrelevant clarifications (green). NONE incurs incorrect assumptions that worsen monotonically with , while CLRF observes reduction in IrrelClrf with . DISPO achieves a Pareto-optimal profile across all three metrics and ambiguity depth.
Simulated User Benefits (RQ4)
We compare our simulator against the single-LLM design standard in multi-turn evaluation Wu et al. [ [31]. Because prior simulators were built for different tasks and settings (Section 5), reusing their prompts directly would confound the comparison with domain mismatch. Instead, we unify the instructions they share into a single generic baseline, keeping those that apply to ambiguity resolution (Appendix D). This keeps the comparison on methodology, an unconstrained LLM user versus a preference-grounded one, rather than on reproducing prior experimental setups.
Experimental Setup. We use the same sample set employed for DisPO and run simulations with both our simulator and the baseline under the None and Clrf strategies across all benchmarks using GPT-5.4, resulting in a total of 600 simulations per simulator. We
further extend the comparison by evaluating both Phi-4 and GPT-5 as the underlying LLMs driving the respective user simulators. We use LLM-as-judge Wu et al. [ [31] to compute Preference Violation, Hallucination and Over-disclosure rate (all prompts in Appendix § 12).
Results. Table 3 highlights qualitative and statistical benefits offered by our Preference-Grounded User Simulator. Accuracy for the baseline user simulator varies significantly with the choice of underlying LLM, suggesting that the evaluated GPT-5.4 agent degrades when paired with less capable simulator models. In contrast, our simulator remains robust across changes in the underlying model, indicating broader applicability without dependence on access to highly capable proprietary systems. The baseline simulator also exhibits substantially higher standard deviation in accuracy, indicating inconsistent responses across iterations that are difficult to reproduce reliably for scientific evaluation. Additionally, conversations with our simulator tend to be longer due to lower rates of over-disclosure. The baseline, by contrast, often leaks explicit hints or partially solves the task on behalf of the evaluated model, enabling faster convergence to an intent-aligned response with fewer turns. This behavior is further reflected in its higher preference violation rate, where conversations terminate early despite the final solution not aligning with the user’s intended interpretation.
| Metric | GPT-5 Base | GPT-5 Ours | Phi-4 Base | Phi-4 Ours |
| Acc (%) | 63.21 6.89 | 61.71 0.93 | 45.39 14.48 | 62.16 0.44 |
| |Conv| | 1.98 0.02 | 2.23 0.31 | 2.56 0.05 | 3.02 0.02 |
| Pref. Viol. (%) | 5.26 | 1.01 | 7.05 | 4.87 |
| Hallucin. (%) | 0.00 | 0.00 | 11.00 | 0.00 |
| Overdisc. (%) | 3.06 | 1.68 | 2.67 | 0.33 |
Table 3. User simulator comparison.
Related Works
Ambiguity Resolution. Ambiguity has been studied across many domains Ivanova et al. [ [11]. Prior work commonly treats clarification as an effective resolution strategy Yu et al. [2019]; Kuhn et al. [ [13], overlooking the prolonged, frustrating conversations it can cause. Existing evaluations largely measure how well a model enumerates or covers all interpretations, using beam search or metrics such as F1 Zhang and Choi [ [35]; Bhaskar et al. [ [5]; Min et al. [ [21], which limits their applicability when ambiguity is resolved through user interaction Wu et al. [ [31]; Zhang et al. [2025]. We extend this line of work by linking open-ended user dissatisfaction signals Sandbank et al. [ [25] to resolution inefficiency, bridging static evaluation and real-world conversational utility.
User Simulator. Prior simulators typically rely on a single LLM call, prompted to emulate a human’s answering style Wu et al. [ [31], role-play personas or domain characteristics Wu et al. [ [32]; Zhao et al. [ [38], or operate on constrained knowledge Shea et al. [ [28]. They are often further steered toward the final solution with hints Vijayvargiya et al. [ [29] or intermediate reasoning steps Dutta et al. [ [9]. In contrast, our simulator restricts its responses to user preferences critical to disambiguate the query, improving reproducibility and extensibility across models.
Conclusion
We introduce a preference-grounded user simulator for controlled and reproducible evaluation of ambiguity resolution in multi-turn interactions, reducing hallucination and over-disclosure common in unconstrained simulators. We further define dissatisfaction signals: incorrect assumptions, repeating clarifications, and irrelevant clarifications, to capture observable inefficiencies during ambiguity resolution. Across question answering, semantic parsing, and text-to-viz. tasks, we reveal clear trade-offs between solution-driven and clarification-driven strategies. DisPO, optimized on dissatisfaction signals, achieves a Pareto-optimal balance, demonstrating what models should strive for in their attempt to ask precisely and assume wisely.
Limitations
Our simulated user framework is inherently dependent on the quality and completeness of the disambiguation preferences generated. Ambiguity in benchmarks such as AmbigQA, Ambrosia, and NvBench 2.0 is well-bounded through high-quality disambiguated queries, enabling the construction of reliable preference sets. However, extending this process to naturally occurring ambiguous queries remains challenging, as they can admit a large or unbounded number of interpretations, which would trivially reduce IrrelClrf to zero. To address this, we bound the interpretation space by prioritizing critical and commonly intended interpretations over peripheral ones, though this introduces a degree of subjectivity in determining what constitutes a valid or popular interpretation. Nevertheless unbounded ambiguity space can be modeled in this framework and can be run against a specific user preference set of interest.
Additionally, we do not consider single-turn approaches in which the model generates multiple interpretations alongside their corresponding answers simultaneously in the same turn. Our primary objective is to evaluate conversational efficiency, so we treat such multi-solution outputs as equivalent to separate dialogue turns, each resolving a distinct interpretation. In this setting, the inefficiency of multi-answer generation appears as extra token and computation cost within a single turn rather than as extra turns, and each generated answer can be read as an individual model assumption, which reduces the evaluation to the None setting.
Ethical Considerations
We use AmbigQA Min et al. [ [21], Ambrosia Saparina and Lapata [ [26] and nvBench 2.0 Luo et al. [ [19] for evaluation which are open-source under the license CC-BY-4.01. The user simulator introduced in this work operates under a very strong assumption that its vocabulary is restricted to the disambiguation preferences set for a given interpretation. While this enables controlled and reproducible evaluation of ambiguity resolution in multi-turn interactions, it does not fully reflect the variability and unpredictability of real user behavior. Our notion of a user simulator is intended only as an abstraction for studying human-LLM interactions under ambiguity, rather than as a faithful attempt to replicate real users. Moreover, the dissatisfaction signals derived from this setup may not fully capture the degree of dissatisfaction a real user may experience during such interactions. We instead treat them as immediate observable proxies, derived from our simulator design, for measuring the effectiveness of ambiguity resolution in interactive settings. This was further approved by the ethics review board by the anonymous IT company. None of the interpretations in this study had implication of bias towards specific race, gender, ethnicity, or other protected attributes. ChatGPT was used solely for grammar polishing and improving writing clarity during the preparation of this paper. It was not involved in the formulation of hypotheses, experimental design, analysis, or the scientific discoveries and conclusions presented in this work.
Dissatisfaction Signals Formulation
In this section, we discuss in detail how the formulas projected in § 2.3 are justified.
Incorrect Assumption. We compute the sum across all turns in a conversation for scenarios when the user responded with a subset of , when no clarification question was asked by the model. This implies that the user, if not ending conversation, is unhappy with misaligned interpretations and applies penalty for the ambiguity dimensions it replies with.
Repeating Clarification. is measured by counting the occurrences of those questions at a given turn that have either been asked or clarified by the user in their response in prior turns. The search space spans both relevant and irrelevant preferences .
Irrelevant Clarification. is measured by the number of occurrences of those questions that do not resolve any ambiguity resolved by any preference in . In the absence of scaling, a model would be penalized more towards asking questions, thereby creating bias towards making assumptions. It is for this reason we allow some breathing space for models initially to ask questions and increase the penalty as information gets revealed over the conversation. We normalize each question for a given irrelevant preference at turn with , where is a hyperparameter loosely approximating the number of interpretations for a given ambiguity dimension, ranging somewhere between 2 − 3. For example, "How many goals did Ronaldo score?" can either be interpreted as Christiano Ronaldo from Portugal or Ronaldo Nazario from Brazil. We use an exponential scaling since every time an ambiguity is resolved, the number os answers in the solution space contracts by . We further translate constant and represent it as number of interpretations for a given query. Hence the normalization factor . Different queries with different , would have different appetite for accepting irrelevant clarifications.
Qualitative Analysis
Qualitative analysis reveals recurring patterns across all dissatisfaction signals.
Incorrect Assumptions.
W-1 Default entity trap: The model defaults to globally dominant interpretations (e.g., US over UK, film over radio), overriding explicitly intended non-default meanings.
W-2 Ordinal/positional default: Ordinal constraints such as “second win” or “episode 11” are ignored in favor of first, latest, or most common interpretations.
W-3 Granularity mismatch: The model returns answers at an incorrect granularity, such as returning a team instead of a league or a captain instead of a vice-captain.
W-4 Temporal cutoff violations: Temporal constraints such as “as of 2017” are ignored or only partially enforced.
W-5 Axis/channel guessing: In NvBench2.0, models guess chart axis mappings instead of requesting clarification when mappings are ambiguous.
W-6 Sample-as-dataset confusion: Models incorrectly treat example rows as the complete dataset and reject valid chart requests because queried values are absent from the examples.
Repeating Clarifications.
R-1 Over-decomposition cascade: After a user narrows an ambiguity dimension (e.g., interpreting “why” as capability rather than motivation), the model repeatedly sub-divides the same dimension into finer variants, leading to long clarification loops.
R-2 Confirmation-seeking: The model correctly infers the user’s intent but still asks redundant confirmation questions such as “is this what you mean?” instead of proceeding.
R-3 Multi-spec partial resolution: In queries containing multiple specifications, the model resolves only one ambiguity at a time while repeatedly revisiting previously resolved dimensions. R-4 Lexical re-opening: Previously resolved ambiguities, such as chart axis assignments, are reopened under different wording or interaction contexts.
Irrelevant Clarifications.
I-1 Aggregation over-asking: Models ask unnecessary aggregation questions (e.g., “sum, average, or count?”) even when aggregation intent is already implied.
I-2 Unsolicited filter questions: The model introduces additional filtering options unrelated to the ambiguity specified in the query.
I-3 Questioning the chart type: Instead of resolving missing information, the model questions whether the requested chart type is appropriate.
I-4 Format vs. content confusion: The model asks formatting-related questions when the ambiguity concerns factual content.
I-5 Re-opening specified entities: Explicitly provided entities are unnecessarily questioned instead of being treated as fixed constraints.
Statistical Significance Tests
We rerun GPT-5.4 (chat) with DisPO for five independent iterations on AmbigQA, Ambrosia, and nvBench2.0, and report the standard deviation and variance of the resulting metrics, averaged across the three benchmarks, in Table 4. Accuracy and CIR are reported in %.
| Metric | Std Dev | Variance |
| Acc(%) | 0.92 | 0.85 |
| 0.0515 | 0.003375 | |
| IncAssmp | 0.0540 | 0.003579 |
| RepClrf | 0.0208 | 0.000561 |
| IrrelClrf | 0.0478 | 0.003311 |
| CIR(%) | 2.76 | 9.77 |
Table 4. Standard deviation and variance of GPT-5.4 with DisPO over five independent runs, averaged across AmbigQA, Ambrosia, and nvBench2.0.
The low variation across runs indicates that the trends in Table 1 and Table 2 are stable.
Prompts Used
We list down all the system prompts used across all LLM operated systems in this paper.
Within-Query Sensitivity Analysis
To test how sensitive our metrics are to the choice of user preference set , we measure within-query variation across interpretations for GPT-5.4. For each query, we compute the standard deviation of each metric across its interpretations (i.e., across different assignments of ), and report the mean of these per-query standard deviations in Table 5. Higher values indicate greater variation across interpretations. Accuracy and CIR are binary (0/1) per conversation, IncAssmp, RepClrf and IrrelClrf are raw counts, and is the number of turns.
| Data | Strategy | Acc | IncAssmp | RepClrf | IrrelClrf | CIR | |Conv| |
| AmbigQA | None | 0.30 | 0.52 | 0.17 | 0.02 | 0.43 | 0.62 |
| AmbigQA | Self-Ask | 0.29 | 0.53 | 0.19 | 0.02 | 0.39 | 0.62 |
| AmbigQA | Intent-Sim | 0.31 | 0.54 | 0.12 | 0.03 | 0.42 | 0.58 |
| AmbigQA | DisPO | 0.30 | 0.51 | 0.23 | 0.10 | 0.40 | 0.67 |
| AmbigQA | Clrf | 0.32 | 0.28 | 1.78 | 0.65 | 0.28 | 1.68 |
| Ambrosia | None | 0.24 | 0.77 | 0.03 | 0.00 | 0.47 | 0.71 |
| Ambrosia | Self-Ask | 0.22 | 0.78 | 0.07 | 0.02 | 0.48 | 0.77 |
| Ambrosia | Intent-Sim | 0.22 | 0.74 | 0.04 | 0.02 | 0.48 | 0.73 |
| Ambrosia | DisPO | 0.22 | 0.69 | 0.09 | 0.10 | 0.43 | 0.71 |
| Ambrosia | Clrf | 0.23 | 0.40 | 1.15 | 0.76 | 0.14 | 1.16 |
| nvBench2.0 | None | 0.09 | 0.53 | 0.00 | 0.00 | 0.33 | 0.37 |
| nvBench2.0 | Self-Ask | 0.09 | 0.53 | 0.02 | 0.06 | 0.33 | 0.39 |
| nvBench2.0 | Intent-Sim | 0.09 | 0.55 | 0.04 | 0.06 | 0.34 | 0.46 |
| nvBench2.0 | DisPO | 0.10 | 0.50 | 0.02 | 0.15 | 0.30 | 0.37 |
| nvBench2.0 | Clrf | 0.10 | 0.43 | 0.09 | 0.35 | 0.25 | 0.40 |
Table 5. Within-query sensitivity of GPT-5.4 across ambiguity-resolution strategies. Values are the mean per-query standard deviation across interpretations; higher values indicate more variation across user preference sets .
Accuracy is most sensitive to the interpretation on AmbigQA (0.30) and least on nvBench2.0 (0.09). CIR varies substantially across interpretations on all datasets (roughly 0.3–0.5 for non-clarifying strategies), indicating that interaction cleanliness depends on which interpretation the user holds. Clarification increases the variability of repeated and irrelevant clarifications and of conversation length, while reducing the variability of IncAssmp and CIR. IncAssmp is most volatile for strategies that do not ask (None, Self-Ask, Intent-Sim), suggesting that a single assumption may fit one interpretation but not others.
AmbigQA System Prompt
You are a helpful AI assistant. When providing an answer to the user’s question, you think out loud and reason step by step. You clearly mention any assumptions that were made while answering the question.
→ Nature Insert: (C) or (G) — see above
Swapped per evaluation mode
Your final answer, which should be a concise one or two words (not a full sentence) should be represented in the following format:
```json { "answer": "your final answer here" } ```
This should follow after you have stated your reasoning and assumptions (if any) for the final answer provided. For example,
query: Who is the current chairman of african union commission?
response: As of my latest information (through Sept 2025), the fourth and the current Chairperson of the African Union Commission is Moussa Faki Mahamat from Chad.
``` { "answer": "Moussa Faki Mahamat" } ```
If asking for clarification, simply respond in plain text without using the json tags. For example,
query: Who is the current chairman of african union commission?
response: Are you asking about the current chairman as of 2025 or any other year?
You should use json tags only when you are providing a final answer to the user’s query. If you are asking for clarifications, do not use the json tags and just ask the question in plain text.
Ambrosia System Prompt
The task is to write SQL queries based on the provided questions in English. Do not include any explanations, and do not select extra columns beyond those requested in the question.
→ Nature Insert: (C) or (G) — see above
Swapped per evaluation mode
If you are providing a solution, enclose the SQL query inside ```sql``` tags. If you are providing feedback or asking for clarification, respond in plain text without any code formatting. Presence of ```sql``` tags will imply that you are providing a solution.
nvBench2.0 System Prompt
You are an expert data visualization assistant. Given a natural language visualization request and a data table schema with example values, generate a Vega-Lite chart specification. Only use column names that exist in the provided schema. Do not include any explanations or extra fields beyond what is needed to represent the chart.
↪ Nature Insert: (C) or (G) — see below
Swapped per evaluation mode
If you are providing a solution, enclose your answer in ‘‘‘json‘‘‘ tags using the following output format: ‘‘‘json
{chart_spec}
‘‘‘
Each Vega chart must only include the following three keys (mark, encoding, transform):
mark (chart type)
Possible chart types:
bar,line,arc,point,rect,boxplot. Note that arc is identical to pie chart, and rect is identical to heatmap.If the NL query mentions one exact chart type, select that single chart type.
Else if the NL query does not mention chart type but indicates a data analysis task in (trend; distribution; correlation), infer the chart type by (task:chart) = (trend:[bar,line]; distribution:[bar,arc,line,boxplot]; correlation:[point,heatmap]).
Otherwise, when no hints for chart type are mentioned in the NL query, all chart types are possible.
encoding (with only field, aggregate, sort, bin, and timeUnit within the channels)
Obligatory/optional channels per chart type (channels with * are obligatory): bar:[x*,y*,color]; line:[x*,y*,color]; arc:[color*,theta*]; point:[x*,y*,color,size]; rect:[x*,y*,color*]; boxplot:[x*,y*].
aggregate: Three types: sum, mean, count. For count, it is a special computed column that must not include a field.sort: If sort is explicitly requested, use channels like x, -x, y, -y, theta, or -theta.bin: Used for quantitative columns, e.g.,{"bin": {"maxbins": 10}}. For temporal columns, use timeUnit, e.g.,{"timeUnit": "year"}.
transform
Include only filter if explicitly mentioned in the NL query; otherwise omit transform.
If you need clarification or are providing feedback, respond in plain text without any code formatting. The presence of ‘‘‘json‘‘‘ tags indicates you are submitting a solution.
Nature Insert (C): clarification mode
Your Nature: The queries that you will face would be ambiguous. Always clarify all the ambiguities by asking relevant clarification questions to the user before providing an answer. Do not make any assumptions.
Nature Insert (G): DisPO mode
Your Nature: The queries you will face are ambiguous. Ask clarifying questions to resolve genuine ambiguities before answering. Follow these rules strictly:
Ask only about genuine ambiguities. Do not ask about anything already specified or implied by the query, or that has a clear conventional default.
Accept every clarification as final. Once the user answers, act on it immediately — never rephrase the same question or ask “just to confirm.”
Do not sub-divide a resolved dimension. If the user narrows an ambiguity to a category, use your best judgment within that category rather than decomposing it further.
Do not default to the most prominent interpretation. If a query is ambiguous between a well-known reading and a less common one, ask — do not silently assume the most familiar one.
Apply every constraint in the query. Never silently drop a qualifying constraint (temporal, numerical, geographic, categorical, etc.).
Treat provided examples as illustrative, not exhaustive. Do not refuse or flag errors because a query value is absent from example data — trust the query and proceed.
Infer well-established conventions. Apply standard, widely accepted approaches without asking; clarify only when the choice meaningfully changes the outcome.
Batch all ambiguities into one turn. Ask about all unresolved dimensions at once, then produce your response as soon as all are answered.
Baseline User Proxy System Prompt
You are simulating a user in a conversation with an AI assistant.
Context
The user has an ambiguous query and a set of preferences that resolve the ambiguity. The AI assistant is trying to help the user.
Your Task
Given the conversation history, you must produce the next user reply. Follow these rules strictly:
If the assistant’s latest message contains a final answer (e.g. a SQL query, a code block, a direct answer), inspect whether it matches all of the user’s preferences.
If all preferences are satisfied, reply only with: “Thanks! I am satisfied with your solution”
If any preference is violated, point out the violated preference(s) and ask the assistant to try again.
If the assistant’s latest message is asking a clarification question, answer it using the relevant preference(s) that address the ambiguity being asked about.
If none of the preferences address the clarification question, reply: “I have no preferences. Please proceed with your best judgement”
Input
query: The original ambiguous question from the user.preferences: The user’s preferences for resolving the ambiguity, as a list.conversation: The full conversation history up to and including the assistant’s latest message.
Behavioral Guidelines
Stay in character as a human user. You are NOT an AI. Do not respond like an assistant.
Goal-oriented: keep the conversation focused on resolving the original query. Avoid small talk or digressions.
Keep your responses concise and to the point.
Output
Produce only the user’s next reply as a short plain-text message. Do not include any explanation or meta-commentary.
Hallucination Judge System Prompt
You are an expert evaluator assessing whether a simulated USER in a multi-turn conversation has hallucinated.
Definition. A hallucination occurs when the simulated user:
Claims an action was performed (by themselves or the agent) when it was not.
Claims an action was not performed when it actually was.
Makes a request or complaint that has no relationship to any of the stated user preferences.
Input
query: The original ambiguous question.amb_specs: The list of user preferences that resolve the ambiguity.conversation: The full conversation showing both user and assistant turns.
Task. Review only the USER turns and determine whether any of the three hallucination types occurred.
Output — strict JSON:
{ "hallucinated": true | false, "reasoning": "<brief explanation>" }
Over-Disclosure Judge System Prompt
You are an expert evaluator assessing whether a simulated USER in a multi-turn conversation over-disclosed information that should only be discovered by the AI agent.
Definition. Over-disclosure occurs when the simulated user:
Attempts to solve the question themselves (e.g. computes or constructs the answer).
Leaks the correct answer or a critical hint that makes the solution too obvious (e.g. reveals the specific column name, value, or formula the agent should discover).
Gives away the intended resolution in a way that circumvents the agent’s reasoning process.
Note: It is acceptable for the user to answer clarification questions using the provided preferences. That is not over-disclosure.
Input
query: The original ambiguous question.amb_specs: The list of user preferences that resolve the ambiguity.conversation: The full conversation showing both user and assistant turns.
Task. Review only the USER turns and determine whether over-disclosure occurred.
Output — strict JSON:
{ "overdisclosed": true | false, "reasoning": "<brief explanation>" }References
- [1]Emre Can Acikgoz, Jinoh Oh, Joo Hyuk Jeon, Jie Hao, Heng Ji, Dilek Hakkani-Tur, Gokhan Tur, Xiang Li, Chengyuan Ma, and Xing Fan. Mac: A multi-agent framework for interactive user clarification in multi-turn conversations. In Proceedings of the 16th International Workshop on Spoken Dialogue System Technology, pages 1–17, 2026.arxiv.org/abs/2512.13154
- [2]Chinmaya Andukuri, Jan-Philipp Fränken, Tobias Gerstenberg, and Noah D Goodman. Star-gate: Teaching language models to ask clarifying questions. arXiv preprint arXiv:2403.19154, 2024.arxiv.org/abs/2403.19154
- [4]Jonathan Berant, Maximillian Chen, Adam Fisch, Reza Aghajani, Fantine Huot, Mirella Lapata, and Jacob Eisenstein. Learning steerable clarification policies with collaborative self-play. arXiv preprint arXiv:2512.04068, 2025.arxiv.org/abs/2512.04068
- [5]Adithya Bhaskar, Tushar Tomar, Ashutosh Sathe, and Sunita Sarawagi. Benchmarking and improving text-to-SQL generation under ambiguity. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7053–7074, Singapore, December 2023. Association for Computational Linguistics.arxiv.org/abs/2310.13659
- [6]Jiale Cheng, Xiao Liu, Kehan Zheng, Pei Ke, Hongning Wang, Yuxiao Dong, Jie Tang, and Minlie Huang. Black-box prompt optimization: Aligning large language models without model training, 2024.arxiv.org/abs/2311.04155
- [8]Yao Dou, Michel Galley, Baolin Peng, Chris Kedzie, Weixin Cai, Alan Ritter, Chris Quirk, Wei Xu, and Jianfeng Gao. SimulatorArena: Are user simulators reliable proxies for multi-turn evaluation of AI assistants? In Christos Christodoulopoulos, Tammoy Chakraborty, Carolyn Rose, and Violet Peng, editors, Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 35212–35290, Suzhou, China, November 2025. Association for Computational Linguistics.arxiv.org/abs/2510.05444
- [9]Avik Dutta, Priyanshu Gupta, Hosein Hasanbeig, Rahul Pratap Singh, Harshit Nigam, Sumit Gulwani, Arjun Radhakrishna, Gustavo Soares, and Ashish Tiwari. Condabench: Interactive evaluation of language models for data analysis, 2025.
- [10]GitHub. Github copilot. https://github.com/features/copilot, 2026. Accessed: 2026-05-25.
- [11]Anastasia Ivanova, Bakaeva Eva, Zoya Volovikova, Alexey Kovalev, and Aleksandr Panov. AmbiK: Dataset of ambiguous tasks in kitchen environment. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 33216–33241, Vienna, Austria, July 2025. Association for Computational Linguistics.arxiv.org/abs/2506.04089
- [12]Kimiya Keyvan and Jimmy Xiangji Huang. How to approach ambiguous queries in conversational search: A survey of techniques, approaches, tools, and challenges. ACM Computing Surveys, 55(6):1–40, 2022.DOI
- [13]Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Clam: Selective clarification for ambiguous questions with generative language models, 2023.arxiv.org/abs/2212.07769
- [14]Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452–466, 2019.DOI
- [15]Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Rongyu Cao, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin C. Chang, Fei Huang, Reynold Cheng, and Yongbin Li. Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls, 2023.arxiv.org/abs/2305.03111
- [16]Yuhan Liu, Michael JQ Zhang, and Eunsol Choi. User feedback in human-LLM dialogues: A lens to understand users but noisy as a learning signal. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors, Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 2666–2681, Suzhou, China, November 2025. Association for Computational Linguistics.arxiv.org/abs/2507.23158
- [17]Yuyu Luo, Nan Tang, Guoliang Li, Chengliang Chai, Wenbo Li, and Xuedi Qin. Synthesizing natural language to visualization (nl2vis) benchmarks from nl2sql benchmarks. In Proceedings of the 2021 International Conference on Management of Data, SIGMOD ’21, page 1235–1247, New York, NY, USA, 2021. Association for Computing Machinery.DOI
- [18]Yuyu Luo, Nan Tang, Guoliang Li, Jiawei Tang, Chengliang Chai, and Xuedi Qin. Natural language to visualization by neural machine translation. IEEE Transactions on Visualization and Computer Graphics, 28(1):217–226, 2022.DOI
- [19]Tianqi Luo, Chuhan Huang, Leixian Shen, Boyan Li, Shuyu Shen, Wei Zeng, Nan Tang, and Yuyu Luo. nvbench 2.0: Resolving ambiguity in text-to-visualization through stepwise reasoning, 2026.arxiv.org/abs/2503.12880
- [21]Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. Ambigqa: Answering ambiguous open-domain questions, 2020.arxiv.org/abs/2004.10645
- [23]Kishore Papineni, Salim Roukos, Todd Ward, and Wei Jing Zhu. Bleu: a method for automatic evaluation of machine translation. 10 2002.
- [24]Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models, 2023.arxiv.org/abs/2210.03350
- [25]Tommy Sandbank, Michal Shmueli-Scheuer, Jonathan Herzig, David Konopnicki, John Richards, and David Piorkowski. Detecting egregious conversations between customers and virtual agents. pages 1802–1811, 01 2018.arxiv.org/abs/1711.05780
- [26]Irina Saparina and Mirella Lapata. Ambrosia: A benchmark for parsing ambiguous questions into database queries, 2024.arxiv.org/abs/2406.19073
- [27]Preethi Seshadri, Samuel Cahyawijaya, Ayomide Odumakinde, Sameer Singh, and Seraphina Goldfarb-Tarrant. Lost in simulation: Llm-simulated users are unreliable proxies for human users in agentic evaluations, 2026.
- [28]Ryan Shea, Yunan Lu, Liang Qiu, and Zhou Yu. Sage: A top-down bottom-up knowledge-grounded user simulator for multi-turn agent evaluation, 2026.arxiv.org/abs/2510.11997
- [29]Sanidhya Vijayvargiya, Xuhui Zhou, Akhila Yerukola, Maarten Sap, and Graham Neubig. Interactive agents to overcome ambiguity in software engineering, 2025.arxiv.org/abs/2502.13069
- [30]Marilyn A. Walker, Diane J. Litman, Candace A. Kamm, and Alicia Abella. Paradise: A framework for evaluating spoken dialogue agents, 1997.
- [31]Shirley Wu, Michel Galley, Baolin Peng, Hao Cheng, Gavin Li, Yao Dou, Weixin Cai, James Zou, Jure Leskovec, and Jianfeng Gao. Collabllm: From passive responders to active collaborators. In International Conference on Machine Learning (ICML), 2025.arxiv.org/abs/2502.00640
- [32]Shujin Wu, Yi R. Fung, Cheng Qian, Jeonghwan Kim, Dilek Hakkani-Tur, and Heng Ji. Aligning LLMs with individual preferences via interaction. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert, editors, Proceedings of the 31st International Conference on Computational Linguistics, pages 7648–7662, Abu Dhabi, UAE, January 2025. Association for Computational Linguistics.arxiv.org/abs/2410.03642
- [33]Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. In ICLR 2024, 2024.arxiv.org/abs/2309.03409
- [35]Michael J. Q. Zhang and Eunsol Choi. Clarify when necessary: Resolving ambiguity through interaction with lms, 2023.arxiv.org/abs/2311.09469
- [36]Michael JQ Zhang and Eunsol Choi. Clarify when necessary: Resolving ambiguity through interaction with LMs. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Findings of the Association for Computational Linguistics: NAACL 2025, pages 5526–5543, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics.arxiv.org/abs/2311.09469
- [38]Zheng Zhao, Clara Vania, Subhadeep Kayal, Naila Khan, Shay B. Cohen, and Emine Yilmaz. Personalones: A benchmark for personalization evaluation in conversational ai assistants, 2025.arxiv.org/abs/2506.09902
- [39]Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.


