WorksComputer Science
AmbiSteer: Selectively Eliciting Ambiguity-Aware Responses in Vision-Language Models
Abstract
Vision-language models (VLMs) often exhibit overconfidence, providing firm answers even when visual evidence is insufficient. A prominent case is referential ambiguity in Visual Question Answering (VQA), where multiple entities in an image are plausible referents of a question. Addressing such cases requires recognizing when ambiguity is present: ambiguous questions can be handled by asking for clarification or enumerating plausible referents, while unambiguous questions should receive a direct answer. Current VLMs struggle with this distinction, often defaulting to a single interpretation. To investigate whether activation steering can serve as an inference-time mechanism to elicit such behaviors, we introduce AmbiSteer, a lightweight framework to selectively elicit ambiguity-aware behavior. Across three VLMs, we find that steering alone has limited and model-dependent effects, while prompting is generally more effective. Combining prompting and steering substantially increases clarification and enumeration behaviors, but also elicits them on unambiguous cases. AmbiSteer addresses this using a linear classifier to predict input ambiguity and condition the steering direction accordingly, increasing the proportion of inputs handled appropriately from 59.0% to 77.0% on average across models. Our work sets the ground for lightweight approaches to reduce model overconfidence in uncertain multimodal settings.
Introduction
Visual Question Answering (VQA) requires models to interpret natural-language questions in visual context [2]. This often involves resolving which entity in the image a question refers to. However, visual context does not always uniquely identify the intended referent. For example, in Figure 1, the question “What is the man holding?” is referentially ambiguous because multiple people in the image are equally plausible referents. Motivated by Testoni et al. [27], we focus on two ambiguity-aware strategies observed in human responses to referential ambiguity: asking which referent is intended (clarification) and providing answers for multiple plausible referents (enumeration). Yet, their analysis shows that current VLMs often overcommit to a single arbitrary interpretation without acknowledging ambiguity.
Figure 1. Referentially ambiguous questions call for ambiguity-aware responses, such as asking for clarification or enumerating multiple plausible referents, whereas non-ambiguous questions call for a direct, single-interpretation response.
Crucially, clarification and enumeration should not be elicited unconditionally. Figure 1 also illustrates the complementary case: when the referent is clear, a direct answer is most appropriate, while clarification or enumeration would be unnecessary. and violate Grice’s maxim of quantity by providing more information than required [8]. Effective ambiguity handling is therefore inherently selective: ambiguity-aware responses are appropriate when the referent is unresolved, but not when the intended referent is already clear.
Two recent works address referential ambiguity through task-specific model training: ClearVQA trains models to ask clarification questions for ambiguous visual questions [10], while AQuA trains models to adapt their response strategy to different levels of ambiguity [9]. Explicit prompting provides a training-free alternative, but is not consistently effective across models [27, 9]. We investigate activation steering as a lightweight inference-time control mechanism that leaves the VLM’s weights unchanged. By modifying internal representations during generation, activation steering offers a way to selectively promote or suppress target behaviors without fine-tuning.
To this end, we introduce AMBISTEER, a framework for selective ambiguity handling in VQA. Using ClearVQA and AQuA to study clarification and enumeration behaviors, respectively, we derive steering vectors from contrastive pairs consisting of an ambiguity-aware response and a response that commits to a single interpretation. Because effective ambiguity handling requires knowing both how to respond and when to intervene, AMBISTEER combines activation steering with classifier-gated ambiguity prediction. A linear classifier over VLM representations predicts whether each image-question pair is ambiguous, and its prediction determines the steering direction, encouraging ambiguity-aware behavior for ambiguous inputs and suppressing it otherwise.
Experiments across three VLMs show that steering alone has limited effects, while prompting is generally stronger but varies considerably across models. Interestingly, combining prompting and steering consistently increases ambiguity-aware behavior beyond either method alone, in many cases by a large margin. This tendency, though, extends to non-ambiguous inputs, where clarification or enumeration is undesirable. AMBISTEER improves this balance and achieves the best trade-off between eliciting ambiguity-aware responses when needed and avoiding them when the referent is clear, increasing the proportion of appropriately handled inputs from 59.0% for combined prompting and steering to 77.0% on average.
Related Work
Referential Ambiguity in VQA. Ambiguity as a resolvable property of a question has been studied in text-only open-domain QA, where Min et al. [16] introduced AmbigQA to resolve multiple valid answers through rewrites. Early multimodal work followed a similar direction: Stengel-Eskin et al. [25] identified ambiguous VQA questions via annotator disagreement and trained models to rephrase them, while Prasad et al. [19] used Large Language Models (LLMs) to rephrase underspecified questions in a zero-shot setting. Recent benchmarks explicitly target referential ambiguity in visual scenes, highlighting that standard VLMs struggle when visual context leaves the referent underspecified [27, 17]. More recent studies train models to handle ambiguity through clarification questions or referent enumeration [10, 9]. In contrast, we selectively elicit both behaviors at inference time without modifying model weights.
Linear Separability and Activation Steering. Linear probing studies suggest that high-level concepts are often organized along linearly separable directions in the activation space, supporting the linear representation hypothesis [1, 15, 18]. Building on this, activation steering modifies a model’s internal representations at inference time to elicit or suppress specific behaviors without updating model parameters. Early approaches, such as Activation Addition and Inference-Time Intervention [28, 14, 34], derive steering directions from activation differences obtained from a single contrastive pair or from free-form text exhibiting the target behavior. Contrastive Activation Addition (CAA; [22]) instead constructs steering vectors from a dataset of minimal pairs, averaging the resulting activation differences across all pairs to obtain a less noisy direction. A key limitation of unconditional steering is that it intervenes regardless of whether steering is needed. Most closely related to our work, conditional Activation Steering (CAST; [12]) addresses this by learning a condition direction and steering only when a criterion is met. Recent VLM steering studies focus mainly on object hallucination mitigation [32, 26], instruction following [24], and configurable refusal behavior ([31]), extending text-only findings ([3]) to the visual domain. However, existing work has not addressed visual ambiguity, nor principled criteria for deciding when steering for ambiguity should be applied. Our approach follows the premise of conditional steering, but addresses this gap by using an internal ambiguity classifier to dynamically determine whether to promote or suppress ambiguity-aware behavior.
Figure 2. Overview of the AMBISTEER framework. We adapt two datasets to obtain ambiguity-aware and single-interpretation contrastive pairs, extract steering vectors from their activation differences, and apply steering at inference time. Conditional steering uses an ambiguity classifier to select the steering direction.
AmbiSteer Framework
Figure 2 summarizes our proposed AmbiSteer framework, described in the following subsections.
Contrastive Pairs
To study the two central disambiguation behaviors, we build contrastive pairs using two distinct datasets: ClearVQA ([10]), which investigates clarification-seeking behavior, and AQuA ([9]), which studies enumeration behavior. Because these benchmarks are not designed for representation-based intervention, for each dataset we generate paired responses for identical image-question contexts, contrasting an ambiguity-aware response against a single-interpretation response that arbitrarily commits to a specific referent. These contrastive pairs serve as the training signal for deriving steering vectors as well as evaluating selective ambiguity handling. Complete descriptions of the dataset adaptation and examples of resulting contrastive pairs are detailed in Appendices A and B, respectively.
Steering Vector Extraction
Contrastive constructions. We seek a direction in activation space that captures the distinction between ambiguity-aware and single-interpretation behavior, and consider two ways of constructing this contrast. The first follows Arditi et al. [3], where we form two sequences by appending the ambiguity-aware and single-interpretation responses separately to the same image–question context and extract the resulting activations. We refer to this as the free-form (FF) construction because the contrast is formed directly from the open-ended responses. This closely matches the setting in which steering is ultimately applied. However, the two responses may differ substantially in length, wording, and lexical content, so their activation difference may capture incidental linguistic variation in addition to the intended response strategy.
To reduce this source of variation, we additionally consider Contrastive Activation Addition (CAA; [22]), where the two responses are presented as options in a shared multiple-choice formatted prompt and the contrasted sequences differ only in the final answer-letter token . CAA therefore provides a more controlled contrast, but introduces a format gap between extraction and generation, since the direction is derived from multiple-choice responses and later applied to open-ended generation. We therefore test both constructions, since it is unclear whether a cleaner contrast or a closer match to open-ended generation is more important for effective steering. Steering vector computation. For either construction, let and denote the extracted activations for example at decoder layer . We compute the steering vector as the mean activation difference across examples:
We derive vectors independently for each decoder layer, model, and dataset.
Inference-Time Steering
At inference time, we add the steering vector to the decoder residual stream at each generation step. At decoder layer , we modify the hidden state as , where is the steering vector extracted at that layer and controls the strength and sign of the intervention. Positive coefficients are designed to promote the ambiguity-aware behavior, while negative coefficients are designed to suppress it; corresponds to generation without steering.
We apply steering at either a single layer or across a contiguous set of layers (more details are provided in § 4.6 and Appendix § 14). In the multi-layer setting, the corresponding layer-specific vector is added at each selected layer.
Conditional Steering
The intervention described above specifies how to promote or suppress ambiguity-aware behavior, but not when each is appropriate: applying positive steering indiscriminately can trigger unnecessary clarification or hallucinated referents on non-ambiguous inputs. We therefore condition the steering intervention on predicted input ambiguity, using a linear probe trained to classify each image-question pair as ambiguous or non-ambiguous.
Experimental Setup
Data Splits and Models
Data Splits. Table 1 summarizes our data splits. Since training data is more constrained in ClearVQA, we use a single split containing ambiguous and non-ambiguous samples for both extracting steering vectors and training the ambiguity classifier. In contrast, the larger scale of AQuA allows us to use an exclusively ambiguous split to obtain the steering vectors. Since ambiguity classification requires a balanced supervisory signal across both conditions, we allocate a distinct split with an equal number of ambiguous and non-ambiguous examples to train the classifier. Full details on the dataset structure and adaptation are provided in Appendix § 10.2
| Dataset | Split | Ambig. | Non-ambig. | |
| ClearVQA | Train | 728 | 395 | 333 |
| ClearVQA | Dev | 300 | 300 | — |
| ClearVQA | Test | 600 | 300 | 300 |
| AQuA | Train (steering) | 900 | 900 | — |
| AQuA | Train (classifier) | 900 | 450 | 450 |
| AQuA | Dev | 300 | 300 | — |
| AQuA | Test | 600 | 300 | 300 |
Table 1. Data splits used in our experiments.
Models. We evaluate three open-weight VLMs of comparable scale: Qwen3-VL-8B-Instruct ([4]), InternVL3.5-8B ([29]), and Molmo2-8B ([5]). Steering vectors are extracted separately for each model. Generation settings are kept fixed across experimental conditions for a given model. Additional model and decoding details are provided in Appendix § 19.
Generation Settings
We compare AMBISTEER with four generation settings that vary whether activation steering is applied and whether a task-specific prompt instruction is provided. In addition, we report a Random baseline as lower bound and an Oracle as upper bound using gold ambiguity labels.
Baseline. We generate responses using the original model without an additional task-specific prompt or activation steering. This measures the model’s default response behavior.
Steer only. We apply activation steering without providing an additional prompt instruction. This isolates the effect of the steering intervention itself.
Prompt only. We provide an explicit task instruction but do not modify the model’s activations. For ClearVQA, the prompt asks the model to seek clarification if the intended referent is unclear. For AQuA, it asks the model to enumerate plausible referents when more than one interpretation is possible. Exact prompts are provided in Appendix § 13.§ 1.
Prompt + steer. We combine the task-specific prompt with activation steering. This tests whether steering provides additional gains and control over response behavior beyond prompting alone.
Ambiguity Classification
We derive binary ambiguity labels separately for each dataset, leveraging human annotations. Details and examples are provided in Appendix § 12.
Using the obtained labels, for each dataset we train a separate logistic-regression classifier to predict whether an image-question pair is ambiguous. The classifiers take as input the Qwen3-VL-8B-Instruct residual-stream activation at the final prompt token. We train one classifier per decoder layer and select the layer with the highest 5-fold cross-validation accuracy on the training data, yielding layer 22 for ClearVQA and layer 23 for AQuA. The selected classifiers achieve 77.2% and 93.8% test accuracy for ClearVQA and AQuA, respectively, and their predictions determine the steering sign at inference time across all three VLMs used for steering. Additional classifier details and comparisons are provided in Appendix § 15.
Evaluation Metrics
We evaluate two central aspects: whether the interventions elicit the intended ambiguity-aware behavior, measured by clarification and enumeration rate, and whether this behavior is applied selectively, measured by behavioral accuracy. We additionally report a token-level degeneration rate to monitor repetitive generation under steering.
Clarification and enumeration rate. We measure how often generated responses exhibit the target behavior, i.e., asking a relevant clarification question on ClearVQA or considering multiple plausible referents and answering for each on AQuA. We use Meta-Llama-3.1-8B-Instruct (Grattafiori et al., [7]) as a few-shot judge based on the original question and generated response. Exact evaluation prompts and few-shot demonstrations are provided in Appendix § 13.§ 2.
Behavioral accuracy. Clarification and enumeration rates measure how frequently the target behavior occurs, but not whether it occurs in the appropriate cases. We therefore report behavioral accuracy, counting a response as behaviorally correct when the target behavior is present on an ambiguous input and absent on a non-ambiguous input.
Degeneration rate. We measure whether steering causes excessive repetition. A response is considered degenerate if any 4-gram occurs at least eight times. Degenerate generations are counted as behaviorally incorrect.
Human Validation
LLM-judge validation. To validate the LLM-based clarification and enumeration rates, two annotators independently label 100 generations from each dataset using the same criteria as the LLM judge. On ClearVQA, the annotators agree on 97% of examples (), while the LLM judge agrees with the individual annotators on 90–91% of examples. On AQuA, human agreement is 91% (), and LLM-human agreement is 89–90%. Full details of the annotation procedure, annotators, and agreement scores are shown in Appendix § 16.§ 1.
Factual consistency. For AQuA, the enumeration metric captures whether a response considers multiple plausible referents, but not whether the information provided for those referents is supported by the image. We therefore conduct a separate human evaluation following the factual consistency criterion proposed by the authors of the AQuA dataset (Jang and Kim, [9]). A human annotator is shown the image, question, and generated response and judge whether the response is grounded in the visual evidence. We report further details of the annotation procedure in Appendix § 16.§ 2.
Steering Hyperparameter Selection
We select steering layers and coefficients on the development sets using clarification or enumeration rates from open-ended generations. To limit per-condition tuning, we select a set of intervention layers using the multiple-choice CAA vectors, and reuse these layers for the free-form construction. We consider , where denotes the number of consecutive layers being steered. We sweep each candidate layer with coefficients and , compute the difference in target-behavior rate, and select the layer, or contiguous five-layer window, with the largest difference averaged across models and datasets. This selects layer 21 for and layers 20–24 for . After fixing the layers, we tune the steering coefficient separately for each model, dataset, vector construction, , and steered generation setting. Positive and negative coefficients are selected to maximize and minimize the target-behavior rate, respectively. Full sweep ranges, normalization details, and selected coefficients are provided in Appendix § 14.
Results and Analysis
We evaluate both free-form (FF) and multiple-choice (CAA) steering. Since CAA is more consistently effective across models and datasets, we focus on CAA with in the main text for clarity. Appendix H reports the corresponding FF results and results for both constructions.
Probing Ambiguity-Aware Behavior
Prior work has used linear probes to examine whether attributes targeted by activation interventions are linearly represented in model activations [14, 23]. We similarly assess whether the CAA vectors distinguish ambiguity-aware from single-interpretation responses. At each layer, the vector is defined by the difference between the mean training activations of the two response types. We use this direction as a linear probe, with the midpoint between the two means defining the decision boundary. As a control, we also measure separability by A/B answer label, allowing us to distinguish behavioral encoding from merely encoding the option letter.
Figure 3 shows a clear transition across layers. Across all three models and both datasets, response behavior becomes nearly perfectly separable between layers 12 and 16 and remains at 99.2–100% accuracy across the steered layers. In contrast, A/B separability is high in early layers but drops to chance as behavioral separability emerges, reaching at most 50.2% in the steered layers. This mirrors the transition from answer-letter to behavioral clustering observed by Rimsky et al. [22], indicating that the CAA vectors used for intervention capture the intended response distinction rather than the A/B answer position.
Figure 3. Layer-wise probing accuracy of CAA directions for response behavior and A/B label assignment on ClearVQA and AQuA. The A/B line shows the mean across models, with shading indicating the range. Vertical shading marks the steered layers (20–24).
Eliciting Ambiguity-Aware Behavior
We first evaluate whether activation steering and prompting elicit clarification or enumeration on ambiguous inputs. Figure 4 reports results on the 300 ambiguous test instances from each dataset.
Figure 4. Target-behavior rates on ambiguous test instances. Panel (a) reports clarification rates on ClearVQA and panel (b) enumeration rates on AQuA, with 300 ambiguous test examples per dataset.
Interestingly, steering alone provides only limited and uneven control over the response behavior. On ClearVQA, clarification is essentially absent at baseline, and steer-only rates remain below 0.20 for all three models. On AQuA, the models enumerate more often, and steering produces somewhat larger shifts, particularly for Qwen3-VL. Overall, however, steering does not reliably elicit ambiguity-aware behavior across models and datasets.
Prompting is substantially more effective, although the models differ in how strongly they follow the instruction. Most importantly, combining prompting and steering is key: it gives the highest clarification or enumeration rate for every model on both datasets. For Qwen3-VL on ClearVQA, for example, steering alone yields a clarification rate of 0.03 and prompting alone 0.44, while their combination reaches 0.92. The size of this additional gain varies by model: Molmo2 already responds strongly to prompting and gains less, while InternVL3.5 benefits much more from the combined intervention. For our ambiguity-handling tasks, steering therefore appears most effective as a complement to prompting, reinforcing ambiguity-aware behavior elicited by the task instruction, while steering alone remains less reliable. Our experiments with the free-form (FF) construction corroborate these findings (see Appendix H).
Selective Ambiguity Handling
The previous analysis considers only ambiguous inputs, where increasing clarification or enumeration is desirable. A useful intervention, however, should promote these behaviors without inducing them on non-ambiguous ones. We therefore compare Prompt, Prompt+Steer, and AMBISTEER on balanced test sets of ambiguous and non-ambiguous inputs. We omit Baseline and Steer, as they do not reliably elicit ambiguity-aware behavior. Figure 5 shows the resulting scores separately for amambiguous and non-ambiguous inputs. Prompt+Steer strongly increases clarification and enumeration on ambiguous examples (86.4% on average), but also substantially increases them on non-ambiguous inputs (66.3%), where they are inappropriate. AMBISTEER reduces this undesired behavior to 17.0% across all models and datasets, while maintaining a 73.6% target-behavior rate on ambiguous inputs. The Oracle selects the steering direction from the gold ambiguity label, providing an upper bound.
Figure 5. Clarification and enumeration rates on balanced test sets with 300 ambiguous and 300 non-ambiguous inputs per dataset. P = Prompt, PS = Prompt+Steer, AS = AMBISTEER, and O = Oracle.
To evaluate the overall appropriateness of model responses, Table 2 reports behavioral accuracy, which rewards clarification or enumeration on ambiguous inputs and direct responses on non-ambiguous inputs. AMBISTEER achieves the highest non-oracle behavioral accuracy across models and datasets, outperforming Prompt by 8.7 to 25.8 percentage points and Prompt+Steer by 12.5 to 22.5 points, as well as consistently outperforming the Random Gate. Degeneration remains low, with a maximum of 3.8%. AMBISTEER comes close to
| Dataset | Model | Prompt | Prompt+Steer | Random Gate | AmbiSteer | Oracle Gate |
| ClearVQA | InternVL3.5 | 55.2 (0.0) | 60.7 (0.5) | 55.3 (0.3) | 73.2 (0.0) | 87.0 (0.2) |
| Molmo2 | 60.7 (0.0) | 55.5 (0.0) | 52.4 (2.4) | 75.2 (2.7) | 96.2 (2.5) | |
| Qwen3-VL | 63.0 (0.2) | 51.3 (5.8) | 49.0 (6.8) | 73.8 (3.8) | 92.7 (5.0) | |
| AQuA | InternVL3.5 | 60.7 (0.0) | 67.3 (0.3) | 61.1 (0.2) | 86.5 (0.3) | 91.0 (0.3) |
| Molmo2 | 66.0 (0.2) | 55.5 (0.3) | 56.1 (1.9) | 74.7 (1.8) | 77.8 (2.3) | |
| Qwen3-VL | 64.2 (0.3) | 63.5 (2.7) | 63.2 (2.3) | 78.5 (2.3) | 81.2 (2.2) |
Table 2. Behavioral accuracy (%) on the balanced test sets, with degeneration rate (%) shown in parentheses. Random Gate selects the steering direction with probability 0.5 and is averaged over 200 random draws; Oracle Gate uses the gold ambiguity label. Bold indicates the best non-oracle behavioral accuracy.
Oracle performance on AQuA, while a larger gap remains on ClearVQA, consistent with the lower classifier accuracy on that dataset.
Error Analysis
Sources of behavioral errors. To better understand the remaining failures under AMBISTEER, we separate degenerate generations from non-degenerate behavioral errors. We further distinguish errors associated with an incorrect ambiguity-classifier prediction from those that occur despite a correct prediction, which we refer to as classification and generation errors, respectively. As shown in Table 3, degeneration is rare, with at most 23 of 600 generations affected in any setting. Behavioral errors are more frequent, and their source differs by dataset. On ClearVQA, most non-degenerate errors coincide with an incorrect ambiguity-classifier prediction, accounting for 72.0-95.5% of errors across models, identifying ambiguity classification as the main bottleneck. On AQuA, the opposite pattern emerges: 58.2-83.0% of errors occur despite a correct classifier prediction, indicating that the model often fails to produce the desired behavior even when the intended steering condition is applied.
| Dataset | Model | Deg. | Total | Class. | Gen. |
| ClearVQA | InternVL3.5 | 0 | 161 | 116 (72.0) | 45 (28.0) |
| Molmo2 | 16 | 133 | 127 (95.5) | 6 (4.5) | |
| Qwen3-VL | 23 | 134 | 127 (94.8) | 7 (5.2) | |
| AQuA | InternVL3.5 | 2 | 79 | 33 (41.8) | 46 (58.2) |
| Molmo2 | 11 | 141 | 24 (17.0) | 117 (83.0) | |
| Qwen3-VL | 14 | 115 | 28 (24.3) | 87 (75.7) |
Table 3. Error breakdown for AMBISTEER. Degenerate generations (Deg.) are reported separately and excluded from behavioral errors. Class. denotes errors with an incorrect ambiguity-classifier prediction, while Gen. denotes generation errors despite a correct prediction. Percentages are computed over behavioral errors.
Behavioral Accuracy and Factual Consistency. Behavioral correctness measured by enumeration rate alone does not necessarily imply proper visual grounding. We therefore jointly analyze behavioral accuracy and human-rated factual consistency for 100 of the final AQuA outputs (Figure 6). On ambiguous inputs, roughly half of behaviorally correct responses (26-31/50 across models) are also factually consistent, while a further 9-16/50 exhibit the correct enumeration behavior but contain factual errors. Thus, correct enumeration does not guarantee accurate description. Factual errors are less frequent on non-ambiguous inputs, though the overall error profile varies across models. Representative generations illustrating both successful and failure cases are provided in Appendix § 18.
Figure 6. Behavioral and factual outcomes on AQuA.
Conclusion
We study whether activation steering can support selective handling of referential ambiguity in
VLMs. Across three VLMs and two ambiguity-handling tasks, steering alone yields limited, model-dependent control, while prompting is generally more effective. Combining prompting and steering consistently yields the highest ambiguity-aware behavior rates, but also induces these behaviors on non-ambiguous inputs, where they are inappropriate. By conditioning the steering direction on predicted ambiguity, AMBISTEER improves this trade-off across all models and tasks, consistently increasing behavioral accuracy. Our error analysis further shows that appropriate ambiguity handling does not ensure visual grounding, highlighting factual consistency as a key direction for future work.
More broadly, our findings place referential ambiguity within a wider challenge in current VLMs: a systemic overconfidence that leads models to provide firm answers even with insufficient evidence. By demonstrating that a classifier over model activations can dynamically gate steering vectors to impose pragmatic caution, AMBISTEER provides an inference-time solution for referential ambiguity. This lightweight paradigm could extend to broader settings where models must recognize uncertainty.
Limitations
Our study has a number of limitations that also point to interesting directions for future investigation. First, the pool of evaluated models is restricted to three open-weight VLMs at a comparable scale (approximately 8B parameters), due to limited computational resources. It remains an open question whether our results, in particular the benefits of AMBISTEER, extend to different or larger models. Second, while our classifier achieves reasonable accuracy (77.2% on ClearVQA and 93.8% on AQuA), performance on ClearVQA leaves room for improvement, and our error analysis in § 5.4 suggests that classifier errors can meaningfully affect the effectiveness of conditional steering. Third, we rely on a single model (Qwen3-8B) to generate the single-interpretation rephrasings used in our contrastive pairs. Although this provides a consistent procedure for constructing the pairs, the choice of generator may introduce lexical or structural regularities into the negative responses, which could partially influence the resulting steering alongside the ambiguity-related target signal. Finally, despite our manual re-labeling work, the ambiguous category in ClearVQA may still contain some degree of noise. In particular, we found that the original dataset contains a non-negligible number of examples that are not genuinely ambiguous with respect to the target, and some residual inconsistencies may therefore remain after filtering.
Future work could extend this study along several directions. A natural next step would be to broaden the pool of evaluated models, both in number and in scale, to examine whether the observed trade-offs between steering, prompting, and AMBISTEER continue to hold in larger and more diverse VLMs. It would also be interesting to compare our inference-time steering approach against models explicitly fine-tuned for ambiguity-aware behavior in order to better understand how activation steering compares with task-specific training, as a complementary approach to ours.
Ethical Considerations
Activation steering is a general-purpose technique for manipulating a model’s internal representations, and like other similar techniques its use therefore requires care. The same mechanism used here could instead be applied to suppress safety-relevant behaviors, bypass alignment training, or push a model toward generating harmful, deceptive, or otherwise undesired content. Since steering works directly on activations instead of through the model’s training objective, it can be more difficult to constrain than prompting-based interventions, which makes responsible use and careful validation of any steering direction particularly important before usage. We do not derive or apply any steering vectors seeking harmful behaviors in the present work, and we encourage future work to evaluate steering interventions for unintended side effects, before applying them in user-facing systems.
Separately, activation steering intervenes on the model’s residual stream during generation, and such interventions are known to carry a risk of degrading output quality or increasing hallucination, independently of the specific target behavior. Future work applying steering more broadly should be aware that inducing a desired high-level behavior does not guarantee that the resulting generations remain factually grounded, and should evaluate reliability alongside behavior, rather than treating the latter as a sufficient indicator of a successful intervention.
We acknowledge the use of AI tools to support parts of the code development and writing refinement. All AI-generated outputs were carefully reviewed, and we confirm that we take full responsibility for the final content and results.
Acknowledgments
We would like to thank several members of the MaiNLP lab, particularly Philipp Mondorf, Ryan Soh-Eun Shim, and Xinyuan Cheng, along with Arnisa Fazla, for their valuable feedback on earlier drafts of this paper. We also thank Leonardo Bertolazzi for his helpful ideas and stimulating discussions. Special thanks go to Jasmin Orth for her dedicated work on annotating ambiguity labels for the ClearVQA dataset, contributing to the validation of our LLM judge, and evaluating factual consistency. We acknowledge funding through the ERC Consolidator Grant DIALECT 101043235, and the German Research Foundation (DFG) project VariVLM (586887264).
Dataset Adaptation
We use ClearVQA ([10]) and AQuA ([9]), two English-language VQA datasets targeting referential ambiguity in general-domain visual scenes, as the basis for our experiments. Because these two benchmarks are designed for open-ended generation or fine-tuning rather than representation intervention, we adapt instances from ClearVQA and AQuA to construct contrastive pairs. Each adapted instance consists of a shared image and question paired with two alternative responses: a positive, ambiguity-aware response (requesting clarification or enumerating plausible referents) and a negative, single-interpretation response (arbitrarily selecting a single referent). Both adaptations were run using Qwen3-8B ([30]) to rephrase the original datasets. Figure 7 illustrates this adaptation process for both benchmarks, with details for each benchmark provided below.
Figure 7. Adaptation of ClearVQA and AQuA into ambiguity-aware and single-interpretation responses.
To ensure the quality of the generated single-interpretation responses, two authors of the paper qualitatively inspected a random sample of 50 responses per dataset (100 in total). We verified that each response commits to a single referent rather than preserving the ambiguity, while remaining faithful to the original content. Out of the 100 inspected responses, only one failed to commit to a single interpretation, yielding a 99% compliance rate.
ClearVQA. The dataset was constructed from originally unambiguous VQA examples, each consisting of an image, a question, and a short ground-truth answer. The authors then use an LLM-based pipeline to introduce ambiguity by removing or blurring information from the question and to generate a corresponding clarification question. Each ClearVQA instance therefore contains the original unambiguous question and gold answer alongside an ambiguous question and clarification question. We augmented clarification questions with an ambiguity-aware response and unambiguous questions with a single-interpretation response. For our adaptation, we use the clarification question as the ambiguity-aware response and prompt Qwen3-8B ([30]) to rewrite the gold answer as a complete sentence conditioned on the original unambiguous question, yielding the corresponding single-interpretation response (see Figure 8).

Figure 8. Prompt used to construct ClearVQA single-interpretation responses.
AQuA. The dataset organizes image-question pairs into four levels of increasing ambiguity (from 0 to 3). We use Level 2, where multiple plausible referents remain and the gold responses enumerate
them, making it a natural setting for our enumeration task. For each Level 2 instance, the gold response serves as the ambiguity-aware response. For the corresponding single-referent response, we prompt Qwen3-8B ([30]) with the question and gold response and ask it to answer using only one of the referents listed in the gold response, as shown in Figure 9.

Figure 9. Prompt used to construct AQuA single-interpretation responses.
Contrastive Pair Examples
Figure 10 shows representative contrastive pairs for ClearVQA and AQuA under the multiple-choice (CAA) and free-form (FF) constructions. For CAA, positive and negative sequences differ only in the final answer-letter token. The ambiguity-aware re- response is assigned to option A or B in balanced proportions to avoid systematic answer-position bias. For FF, the corresponding responses are appended directly to the shared image-question context, resulting in sequences that may differ in length and wording.
Figure 10. Examples of multiple-choice (CAA) and free-form contrastive pairs for ClearVQA and AQuA.
Ambiguity Labels Derivation
We derive binary ambiguity labels separately for ClearVQA and AQuA.
ClearVQA Ambiguity Labels
For ClearVQA, although the dataset is intended to contain ambiguous questions, manual inspection revealed that a substantial number of instances in the released referential subset are not genuinely ambiguous with respect to the corresponding image. We therefore manually label each example as ambiguous, non-ambiguous, uncertain, or poor quality, retaining only the first two categories.
Following Jian et al. [10], we consider a question referentially ambiguous when its referring expression does not uniquely identify the intended referent. For each example, one annotator was shown the question and corresponding image, together with the original unambiguous question and its gold answer from the ClearVQA dataset as reference. Examples were presented individually in random order.
For each example, the annotator assigned one of four labels: ambiguous if the intended referent could not be uniquely identified from the question and image; non-ambiguous if the intended referent remained identifiable and the question could be answered directly; uncertain if the annotator could not confidently distinguish between these two cases; and poor quality if the example did not support a reliable judgment, e.g. because the question did not fit the image or was nonsensical.
Of the 1,928 unique questions, 995 (51.6%) were labeled ambiguous, 633 (32.8%) non-ambiguous, 114 (5.9%) uncertain, and 186 (9.6%) poor quality. We retain the 1,628 examples labeled ambiguous or non-ambiguous and discard the remaining categories. Figure 11 shows one representative example for each of the four labels.
Figure 11. Examples for the human ambiguity labels.
The annotation was carried out by a master’s student in Computational Linguistics, employed as a student assistant and compensated according to national standards. The annotator was informed of the intended research use of the annotations and agreed to perform the task.
AQuA Ambiguity Labels
The AQuA dataset (Jang and Kim [9]) organizes examples into four levels of increasing ambiguity. Level 0 contains unambiguous questions; at Level 1, the referent is underspecified in the question but can still be recovered from the image; at Level 2, multiple plausible referents remain and the gold response enumerates them; and Level 3 requires clarification because the intended referent cannot be resolved. We map Levels 0 and 1 to non-ambiguous and Level 2 to ambiguous. We do not use Level 3 because it targets clarification behavior, which we cover with ClearVQA. In addition, ClearVQA retains the original unambiguous question and gold answer from which each ambiguous question was derived, allowing us to construct a single-interpretation response for the negative contrastive pair. AQuA does not provide such an unambiguous counterpart for Level 3.
Generation and Evaluation Prompts
Generation Instructions
All generation settings use the image and question as input. In Baseline and Steer, no additional instruction is provided. In Prompt, Prompt+Steer and AMBISTEER, we additionally provide the dataset-specific instruction for clarification and
enumeration shown in Figure 12 and Figure 13, respectively. These generation prompts are the same for CAA and FF; the two constructions differ only in how the steering vectors are obtained.
!Clarification Prompt: [image], ⟨question⟩, Instruction: If the question is ambiguous, ask a clarifying question instead of answering. bbox=0.12,0.61,0.49,0.75
Figure 12. Generation instruction used for ClearVQA.
!Enumeration Prompt: [image], ⟨question⟩, Instruction: If the question is referentially ambiguous, explicitly list each possible referent instead of assuming a single interpretation. bbox=0.52,0.08,0.88,0.22
Figure 13. Generation instruction used for AQuA.
The clarification judge template is as follows:
You are an evaluator. You are given the user’s original question and a short text generated by a model in response to it. Your task is to decide whether that text contains a clarification question for the user or not. Only mark as yes if the question is truly a clarification question, i.e. it asks the user to clarify, specify, or disambiguate what they meant, and not a random question that is not asking for clarification. In addition, only mark as yes if a generated clarification truly refers to and is related to the user’s original question. A question that instead offers to perform a different or additional task, or that asks whether the user would like some other or further information, is not a clarification question. Mark as no if the text already commits to a referent before asking for clarification. Answer with exactly one word: “yes” if the text contains such a clarification question, or “no” if it does not. Do not explain your answer.
[Four few-shot demonstrations are inserted here]
User’s original question: {question}
Text to evaluate: {response}
LLM-Judge Prompts
We use Meta-Llama-3.1-8B-Instruct [7] as a few-shot judge for assessing the presence of clarification and enumeration behavior in model generations. For each task, the conversation consists of the task description, four demonstrations as alternating user and assistant turns, and the item to be evaluated as the final user turn. Figure 14 and 15 show the two judge prompts for the clarification and enumeration judges, while Figure 16 and Figure 17 show their corresponding few-shot demonstrations.

Figure 14. Prompt template used by the clarification judge.

Figure 16. Few-shot demonstrations provided to the clarification judge.

Figure 17. Few-shot demonstrations provided to the enumeration judge.
Enumeration judge template
System prompt
You are an evaluator. You are given a question a user asked about an image and a short text generated by a model in response to it. Your task is to decide whether the response enumerates multiple possible referents of the referring expression in the question.
Mark yes only if both of the following hold: (1) the response presents two or more distinct objects, people, or regions as possible referents of the expression in the question, and (2) it gives a separate answer to the question for each of those referents. The response does not need to explicitly state that the question is ambiguous.
Mark no if the response commits to a single referent, even if it gives a detailed answer, and even if it describes other objects or people as part of the surroundings or as supporting detail.
Mark no if the response lists, counts, or describes several objects or people without giving separate answers to the question for them as possible referents.
Answer with exactly one word: yes or no. Do not explain your answer.
Few-shot demonstrations
[Four few-shot demonstrations are inserted here]
Final user turn
QUESTION: {question}
MODEL GENERATION: {response}
fig:15: Prompt template used by the enumeration judge.
Clarification judge: few-shot demonstrations
Demonstration 1
User’s original question: What color is his jacket?
Text to evaluate: Based on the image provided, there are four men standing in front of a bench. The man on the far right is wearing a jacket. The color of his jacket is green. It appears to be a dark green or forest green color, and he is also wearing a beanie.
Assistant: no
Demonstration 2
User’s original question: What color is his jacket?
Text to evaluate: The question is ambiguous because it refers to “his jacket” without specifying which person is being referred to. There are four men in the image, each wearing a different jacket:
The man on the far left is wearing a blue jacket.
The man second from the left is wearing a gray jacket.
The man third from the left is wearing a dark blue or black jacket.
The man on the far right is wearing a green jacket.
Could you please specify which person you are referring to?
Assistant: yes
Demonstration 3
User’s original question: How many times does it appear here?
Text to evaluate: The Apple logo appears 5 times in the image. Here’s a breakdown of where each one is located:
On the white keyboard box (top right corner, partially visible).
On the white keyboard (center, on the spacebar).
On the white mouse (center right, on the top surface).
On the left user guide (top center, on the cover).
On the right user guide (top center, on the cover).
The logo is also visible on the gray box in the background, but it’s partially obscured and not clearly distinct from the other items. The most clearly visible and distinct instances are the 5 listed above.
Assistant: no
Demonstration 4
User’s original question: How many times does it appear here?
Text to evaluate: The question “How many times does it appear here?” is ambiguous without specifying what “it” refers to. Could you clarify:
What specific object, text, or feature are you asking about?
Are you referring to the Apple logo, the word “Wireless,” the keyboard, the mouse, or something else?
Once clarified, I can give you an accurate count.
Assistant: yes
Enumeration judge: few-shot demonstrations
Demonstration 1
QUESTION: What is sitting on top of it?
MODEL GENERATION: The question is referentially ambiguous because “it” could refer to either:
The wooden bench (which has a cat sitting on it)
The wooden piano (which also has items on its top surface)
Therefore, the answer is:
A cat is sitting on top of the bench.
The piano also has sheet music and bowls on its top surface.
Assistant: yes
Demonstration 2
QUESTION: What is sitting on top of it?
MODEL GENERATION: Based on the image provided, a cat is sitting on top of the piano bench.
Here are the details:
The cat is a long-haired, gray tabby with a striped pattern.
It is lying down comfortably on the wooden bench, which is positioned in front of the upright piano.
So, the animal sitting on top of the bench is a cat.
Assistant: no
Demonstration 3
QUESTION: What color is that house behind him?
MODEL GENERATION: The image contains multiple houses in the background, so the color is not uniquely determined by a single referent. Here are the possible referents:
A large, multi-story house on the left side of the image, which has sections of gray siding and white trim around windows and balconies.
A house to the right of the first one, which is white.
Assistant: yes
Demonstration 4
QUESTION: What color is that house behind him?
MODEL GENERATION: Based on the image provided, the house directly behind the man is a gray color. It appears to be a multi-story residential building with a gray exterior and a dark roof that has skylights. There is also a white house visible to the right of the gray one.
Assistant: no
Layer and Coefficient Selection
Layer selection. We consider , where denotes the number of consecutive decoder layers being steered. Following prior work, we sweep all decoder layers on the development sets using coefficients and [22]. However, instead of selecting layers from multiple-choice answer probabilities, we evaluate their effect directly on open-ended generation, motivated by recent work showing that steering effects measured in multiple-choice settings may not transfer reliably to open-ended behavior [20]. For each layer, we compare the target-behavior rate under positive and negative steering, where the target behavior is clarification on ClearVQA and enumeration on AQuA. We measure this rate using the LLM-based judge described in § 4.4. To keep the layer choice fixed across comparisons and limit per-condition tuning, we pool these results for layer selection. We average the resulting difference across models and datasets and, for each , we select the contiguous -layer window with the largest average difference in target-behavior rate. This gives layer 21 for and layers 20-24 for . Selecting layers separately on ClearVQA and AQuA yields the same choices as the pooled selection. We use these same layers for vectors extracted from both free-form and multiple-choice contrastive pairs.
Coefficient selection. After selecting the intervention layers, we tune the steering coefficient
on each dataset’s development set separately for each model, contrastive construction, number of steered layers, and generation setting. To make steering magnitudes comparable across the two target behaviors, we norm-match the vectors on a per-layer basis, following [22], using ClearVQA as the reference and rescaling the corresponding vectors derived from AQuA accordingly. To make coefficient sweeps comparable across different numbers of steered layers, we scale the coefficient applied at each layer by where is the number of steered layers. Because vectors extracted from the free-form construction have larger norms, we scale the FF coefficient magnitudes by 0.5 relative to CAA. Table 5 reports the coefficient grids used for each construction and . For our Steer generation setup, we tune only positive coefficients, since this condition tests whether steering can elicit the target behavior. For Prompt+Steer, we tune both signs, which are subsequently also reused by AMBISTEER to promote ambiguity-aware behavior on inputs predicted as ambiguous and suppress it on inputs predicted as non-ambiguous. For positive steering, we select the coefficient with the highest target-behavior rate. For negative steering, we select the coefficient with the lowest rate. In both cases we consider only coefficients with at most 5% degenerate generations (as defined in § 4.4). If all coefficients exceed the 5% degeneration threshold, we select the coefficient with the lowest degeneration rate. Table 4 reports the final selected coefficients for all experiments.
| Dataset | Model | Steer | Steer | Prompt+Steer | Prompt+Steer | Prompt+Steer | Prompt+Steer |
| CAA | |||||||
| ClearVQA | InternVL3.5 | 1.5 | 0.6 | 2.5 | 0.6 | −4.0 | −0.7 |
| ClearVQA | Molmo2 | 1.5 | 0.6 | 1.0 | 0.2 | −2.5 | −0.6 |
| ClearVQA | Qwen3-VL | 0.5 | 0.2 | 0.5 | 0.3 | −2.0 | −0.6 |
| AQuA | InternVL3.5 | 2.5 | 0.5 | 2.5 | 0.7 | −3.0 | −0.5 |
| AQuA | Molmo2 | 0.5 | 0.7 | 1.5 | 0.6 | −3.5 | −0.4 |
| AQuA | Qwen3-VL | 1.0 | 0.4 | 1.5 | 0.5 | −0.5† | −0.1 |
| Free-form (FF) | |||||||
| ClearVQA | InternVL3.5 | 0.75 | 0.20 | 1.00 | 0.25 | −1.75 | −0.35 |
| ClearVQA | Molmo2 | 0.50 | 0.15 | 0.75 | 0.20 | −1.25 | −0.35 |
| ClearVQA | Qwen3-VL | 0.25 | 0.10 | 0.25 | 0.10 | −1.00 | −0.40 |
| AQuA | InternVL3.5 | 1.25 | 0.35 | 1.00 | 0.20 | −1.00 | −0.25 |
| AQuA | Molmo2 | 1.00 | 0.30 | 0.75 | 0.15 | −0.75 | −0.15 |
| AQuA | Qwen3-VL | 0.50 | 0.10 | 0.25 | 0.10 | −0.50 | −0.10 |
Table 4. Selected steering coefficients for each condition. Prompt+Steer and Prompt+Steer denote positive and negative steering with the task-specific prompt, respectively. † For Qwen3-VL on AQuA with CAA and , all negative coefficients exceeded the 5% degeneration threshold; we therefore use (6.3% degeneration rate).
| Vectors | Coefficient magnitudes | |
| CAA | 1 | 0.5, 1, 1.5, 2, 2.5, 3, 3.5, 4 |
| CAA | 5 | 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8 |
| FF | 1 | 0.25, 0.5, 0.75, 1, 1.25, 1.5, 1.75, 2 |
| FF | 5 | 0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4 |
Table 5. Coefficient ranges used for the development-set sweeps. The same sweep ranges are used for both clarification and enumeration behaviors.
Additional Ambiguity Classifier Analyses
Our main experiments use a linear probe over Qwen3-VL residual-stream representations to predict input ambiguity and gate steering for all three evaluated VLMs. We further analyze this classifier setup by examining (i) how ambiguity decodability varies across Qwen3-VL layers, (ii) whether performance depends on which of the three evaluated VLMs provides the gating representation, and (iii) how the Qwen3-VL probe compares with classifiers based on models outside the set of steered VLMs.
Layer-wise Ambiguity Decodability
To examine how ambiguity information is represented across Qwen3-VL layers, we fit a separate logistic-regression probe to the final prompt-token residual representation at each decoder layer. For each layer, we measure 5-fold cross-validation accuracy on the training split and select the layer with the highest cross-validation accuracy for held-out evaluation. Figure 18 shows the resulting layer-wise performance. On ClearVQA, accuracy increases from 63.9% at layer 0 to 80.2% at the selected layer 22. In contrast, AQuA ambiguity is already highly decodable at layer 0 (88.8%), with accuracy reaching 95.1% at the selected layer 23. At these selected layers, held-out test accuracy is 77.2% on ClearVQA and 93.8% on AQuA.
Figure 18. Layer-wise ambiguity decodability from Qwen3-VL residual-stream representations. Curves show 5-fold cross-validation accuracy on the train split.
The different layer-wise patterns suggest that ClearVQA benefits more from deeper representations for ambiguity classification, whereas much of the ambiguity signal in AQuA is already accessible in early representations suggesting a possible presence of stronger surface-level linguistic cues in AQuA.
Choice of gating VLM
We next assess whether classification performance depends on the VLM providing the gating representation. We fit logistic-regression probes to Qwen3-VL, Molmo2, and InternVL3.5 residual-stream representations, selecting the layer for each model by 5-fold cross-validation on the training split.
As shown in Table 6, Qwen3-VL achieves the highest cross-validation accuracy on both datasets (80.2% on ClearVQA; 95.1% on AQuA), motivating its use as the shared gating VLM. Held-out accuracy remains similar across all three models, ranging from 77.2-78.3% on ClearVQA and 93.2-93.8% on AQuA, suggesting that performance is not particularly sensitive to the choice of gating VLM.
| ClearVQA | AQuA | |||
| Gating VLM | CV | Test | CV | Test |
| Qwen3-VL | 80.2 | 77.2 | 95.1 | 93.8 |
| Molmo2 | 79.1 | 78.3 | 94.8 | 93.8 |
| InternVL3.5 | 77.2 | 78.0 | 94.0 | 93.2 |
Table 6. Ambiguity-classification accuracy (%) for the three VLMs used in our steering experiments.
External classifier comparison
We finally compare the Qwen3-VL probe with classifiers based on models outside the three VLMs considered for steering. These include a text-only TF-IDF classifier and multimodal classifiers based on CLIP (Radford et al., [21]), SigLIP (Zhai et al., [33]), BLIP (Li et al., [13]), and ViLT (Kim et al., [11]) representations. For each alternative classifier, model selection is performed using 5-fold cross-validation on the training split, followed by evaluation on the held-out test set.
As shown in Figure 19, the Qwen3-VL probe achieves the highest accuracy on both datasets, with 77.2% on ClearVQA and 93.8% on AQuA. External multimodal classifiers reach 59.7–66.3% on ClearVQA and 88.5–91.7% on AQuA, while the text-only classifier reaches 59.2% and 83.7%, respectively.
Figure 19. Held-out ambiguity-classification accuracy of the Qwen3-VL probe and external classifiers on ClearVQA and AQuA.
Additional Human Evaluation Details
Details on LLM-Judge Validation
We validate the clarification and enumeration judgments on 100 generated responses from each dataset. The same responses were independently labeled by two annotators: one internally hired master’s student in computational linguistics and one author of the paper. Annotators were shown the input question and generated response and assigned a binary label using the same criteria as the LLM judge. The LLM-judge labels were not shown during annotation.
We selected the validation samples to ensure that both the presence and absence of the target behavior were sufficiently represented. For ClearVQA, we uniformly sampled 100 ambiguous development examples from Qwen3-VL-8B generations under the Prompt generation setting, a setting
in which clarification occurs with non-trivial frequency. For AQuA, we stratified the sample by ambiguity level, using 50 ambiguous Level-2 examples and 50 non-ambiguous examples, split evenly between Levels 0 and 1. Half of the non-ambiguous examples were generated with CAA steering () so that the sample also included cases in which enumeration was elicited unnecessarily. All validation examples were disjoint from the few-shot demonstrations used by the LLM judge.
Table 7 reports the full pairwise agreement scores. We distinguish the paper author from the internally hired annotator to make the comparison with the LLM judge explicit. LLM–author agreement is one percentage point higher than LLM–student agreement on both datasets, indicating only a minor difference across annotators.
| Dataset | Comparison | Agreement | |
| ClearVQA | Author–Student | 97% | 0.931 |
| ClearVQA | LLM–Author | 91% | 0.804 |
| ClearVQA | LLM–Student | 90% | 0.779 |
| AQuA | Author–Student | 91% | 0.795 |
| AQuA | LLM–Author | 90% | 0.778 |
| AQuA | LLM–Student | 89% | 0.760 |
Table 7. Pairwise agreement for the human validation of the LLM-Judge. Agreement is raw label agreement; is Cohen’s kappa.
Details on Factual Consistency Evaluation
We evaluate factual consistency on 100 AQuA test items for each of the three models, for a total of 300 AmbiSTEER responses. For each model, the sample contains 50 ambiguous and 50 non-ambiguous inputs. The same set of 100 items is used across models. We evaluate the responses produced by AmbiSTEER under positive or negative steering based on the ambiguity classifier.
The samples are annotated by an internally hired master’s student in computational linguistics. For each response, the annotator is shown the image, question, and generated response; the gold answer is not shown. The annotation instructions are adapted from the factual-consistency criterion and evaluation protocol of AQuA [9] and are shown in Figure 20. Responses are presented in shuffled order.

Figure 20. Factual-consistency instructions shown to the human annotator.
Table 8 reports the resulting factual-consistency rates. Overall, 77.3% of AmbiSTEER responses are judged factually consistent, with rates of 71.3% on ambiguous and 83.3% on non-ambiguous inputs.
| Model | Amb. | Non-amb. | Overall |
| InternVL3.5 | 80.0 | 92.0 | 86.0 |
| Molmo2 | 66.0 | 70.0 | 68.0 |
| Qwen3-VL | 68.0 | 88.0 | 78.0 |
| Average | 71.3 | 83.3 | 77.3 |
Table 8. Factual consistency (%) of AMBISTEER responses on AQuA.
Additional Steering Results
The main text focuses on CAA vectors applied across five layers (, L20–24). Here, we report results for the three remaining configurations: CAA with single-layer steering (, L21), free-form (FF) steering with , and FF steering with . For easier comparison, we also include the CAA results reported in the main text. The layers and all coefficients are selected on the development sets following the procedure in Appendix § 14.
CAA with Single-Layer Steering
Figure 21 (second row) reports the results for CAA when steering only layer 21. As in the five-layer setting, steering alone has limited effects in most cases, whereas combining steering with prompting increases the target-behavior rate over prompting alone for every model and dataset. The gains range from 16.3 to 54.7 percentage points, showing that the main pattern does not depend on intervening across multiple layers.
Figure 21. Target-behavior rates on ambiguous test inputs across steering configurations. Rows show CAA with , CAA with , FF with , and FF with ; columns show ClearVQA and AQuA. The CAA row repeats the main-text results for comparison. Single-layer steering uses L21, while five-layer steering uses L20–24 B = Baseline, S = Steer, P = Prompt, and PS = Prompt+Steer.
AMBiSTEER with also reproduces the behavioral-accuracy pattern observed for CAA with in the main text. As shown in Table 9, it outperforms Prompt+Steer for all six model-dataset settings, increasing mean behavioral accuracy from 60.8% to 78.2%. It also outperforms the shared Prompt baseline in every setting. For comparison, the table includes the CAA results reported in the main text.
| Dataset | Model | Prompt | CAA PS | CAA AS | CAA PS | CAA AS | FF PS | FF AS | FF PS | FF AS |
| ClearVQA | InternVL3.5 | 55.2 | 60.7 | 73.2 | 57.7 | 67.3 | 58.7 | 61.2 | 55.0 | 59.0 |
| Molmo2 | 60.7 | 55.5 | 75.2 | 54.7 | 76.2 | 53.0 | 75.5 | 53.0 | 76.5 | |
| Qwen3-VL | 63.0 | 51.3 | 73.8 | 59.7 | 72.0 | 59.3 | 70.2 | 60.0 | 73.7 | |
| AQuA | InternVL3.5 | 60.7 | 67.3 | 86.5 | 66.5 | 87.2 | 61.0 | 65.5 | 62.2 | 64.5 |
| Molmo2 | 66.0 | 55.5 | 74.7 | 59.8 | 90.7 | 56.5 | 86.3 | 53.5 | 87.7 | |
| Qwen3-VL | 64.2 | 63.5 | 78.5 | 66.7 | 76.0 | 65.2 | 66.5 | 66.7 | 68.3 |
Table 9. Behavioral accuracy (%) on the balanced test sets for Prompt, Prompt+Steer (PS), and AMBISTEER (AS) across steering configurations. Prompt is shared across configurations because no activation steering is applied. Shading shows the gain of AS over the corresponding PS setting; darker cells indicate larger gains. Bold and underlined values indicate the best and second-best result in each row, respectively.
Free-form Steering
We next evaluate steering vectors obtained with the free-form (FF) construction. Figure 21 (bottom two rows) reports results for FF with and , respectively. For both settings, Prompt+Steer increases the target-behavior rate over Prompt alone for every model and dataset. The magnitude of this increase is more model-dependent than for CAA: FF is particularly effective for Molmo2, while CAA yields larger increases for InternVL3.5 and Qwen3-VL in most settings. This model-dependent pattern is similar for and .
The benefit of AMBiSTEER extends to FF. It improves behavioral accuracy over Prompt+Steer for every model and dataset under both and (Table 9), and also exceeds the shared Prompt baseline in every setting. Mean behavioral accuracy increases from 58.9% to 70.9% for and from 58.4% to 71.6% for . Thus, the improvement from conditioning is not specific to the CAA construction, although its magnitude depends on the model and steering configuration.
Qualitative Examples
Figure 22 shows example generations under our different generation settings for ClearVQA and AQuA. Figure 23 shows qualitative examples of the different error types analyzed in § 5.4: ambiguity classification errors, generation errors (despite correct ambiguity classification), and factual errors (despite correct enumeration behavior).
Figure 22. Example generations under different generation settings on ClearVQA and AQuA. ClearVQA — Molmo2: “What color is his shirt?” Baseline: The tennis player is wearing a white short-sleeved shirt. Steer: The man in the image is wearing a white shirt. Prompt: The image shows the man wearing a white shirt. Prompt + steer: The question is ambiguous because it doesn't specify which person's shirt color is being asked about. There are multiple people visible in the image, including the tennis player and the spectators in the stands. Could you please clarify which individual's shirt color you're interested in? AQuA — InternVL3.5: “What color is this bottle on the counter?” Baseline: The bottle on the counter is red. The color red may add a pop of vibrancy to the kitchen's overall aesthetic, creating a visually appealing contrast against the other elements in the space. Steer: The bottle on the counter is red. The presence of this bottle suggests that it might contain some liquid, possibly for culinary or decorative purposes. Its color adds a pop of contrast to the overall design of the kitchen. Prompt: The bottle on the counter is red. Prompt + steer: The image contains two bottles on the counter. One bottle is clear, and the other is red.
Figure 23. Examples of the failure modes analyzed in our error analysis: ambiguity classification errors, generation errors with incorrect behavior despite correct ambiguity classification, and factual errors despite correct enumeration behavior.
Implementation Details
We use the Hugging Face implementations3 of all models with their default chat templates and preprocessing, and load all models in bfloat16 with FlashAttention-2 [6]. All generations use greedy decoding with a maximum of 512 new tokens. Steering-specific hyperparameters are set as described in § 4.6 and Appendix § 14. All experiments are run on NVIDIA A100 80GB GPUs.
References
- [1]Guillaume Alain and Yoshua Bengio. 2016. Understanding intermediate layers using linear classifier probes. Preprint, arXiv:1610.01644.arxiv.org/abs/1610.01644
- [2]Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar-garet Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual question answering. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).*
- [3]Andy Arditi, Oscar Obeso, Aaquīb Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. 2024. Refusal in Language Models Is Mediated by a Single Direction. Preprint, arXiv:2406.11717.arxiv.org/abs/2406.11717
- [4]Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, and 45 others. 2025. Qwen3-VL Technical Report. Preprint, arXiv:2511.21631.arxiv.org/abs/2511.21631
- [5]Christopher Clark, Jieyu Zhang, Zixian Ma, Jae Sung Park, Rohun Tripathi, Sangho Lee, Mohammadreza Salehi, Jason Ren, Chris Dongjoo Kim, Yinuo Yang, Vincent Shao, Yue Yang, Weikai Huang, Ziqi Gao, Taira Anderson, Jianrui Zhang, Jitesh Jain, George Stoica, Ali Farhadi, and Ranjay Krishna. 2026. Molmo2: Open weights and data for vision-language models with video understanding and grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 28652–28668.arxiv.org/abs/2601.10611
- [6]Tri Dao. 2024. Flashattention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations.arxiv.org/abs/2307.08691
- [7]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 others. 2024. The Llama 3 herd of models. Preprint, arXiv:2407.21783.arxiv.org/abs/2407.21783
- [8]H. P. Grice. 1975. Logic and conversation. In Peter Cole and Jerry L. Morgan, editors, Speech Acts: Syntax and Semantics Volume 3, pages 41–58. Academic Press, New York.DOI
- [9]Jihyoung Jang and Hyounghun Kim. 2026. AQuA: Toward Strategic Response Generation for Ambiguous Visual Questions. Preprint, arXiv:2603.07394.arxiv.org/abs/2603.07394
- [10]Pu Jian, Donglei Yu, Wen Yang, Shuo Ren, and Jiajun Zhang. 2025. Teaching Vision-Language Models to Ask: Resolving Ambiguity in Visual Questions. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3619–3638, Vienna, Austria. Association for Computational Linguistics.arxiv.org/abs/2507.13773
- [11]Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021. ViLT: Vision-and-language transformer without convolution or region supervision. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 5583–5594. PMLR.arxiv.org/abs/2102.03334
- [12]Bruce W. Lee, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Erik Mehling, Pierre Dognin, Manish Nagireddy, and Amit Dhurandhar. 2025. Programming refusal with conditional activation steering. Preprint, ICLR:2409.05907.arxiv.org/abs/2409.05907
- [13]Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 12888–12900. PMLR.arxiv.org/abs/2201.12086
- [14]Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2024. Inference-time intervention: Eliciting truthful answers from a language model. Preprint, arXiv:2306.03341.arxiv.org/abs/2306.03341
- [15]Samuel Marks and Max Tegmark. 2024. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. In First Conference on Language Modeling.arxiv.org/abs/2310.06824
- [16]Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. AmbigQA: Answering Ambiguous Open-domain Questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5783–5797, Online. Association for Computational Linguistics.arxiv.org/abs/2004.10645
- [17]Heejeong Nam, Jinwoo Ahn, Keummin Ka, Jiwan Chung, and Youngjae Yu. 2025. VAGUE: Visual contexts clarify ambiguous expressions. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1537–1547.arxiv.org/abs/2411.14137
- [18]Kiho Park, Yo Joong Choe, and Victor Veitch. 2023. The Linear Representation Hypothesis and the Geometry of Large Language Models. Preprint, arXiv:2311.03658.arxiv.org/abs/2311.03658
- [19]Archiki Prasad, Elias Stengel-Eskin, and Mohit Bansal. 2023. Rephrase, augment, reason: Visual grounding of questions for vision-language models. Preprint, arXiv:2310.05861.arxiv.org/abs/2310.05861
- [20]Itamar Pres, Laura Ruis, Ekdeep Singh Lubana, and David Krueger. 2024. Towards reliable evaluation of behavior steering interventions in llms. Preprint, arXiv:2410.17245.arxiv.org/abs/2410.17245
- [21]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748–8763. PMLR.arxiv.org/abs/2103.00020
- [22]Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Huebinger, and Alexander Turner. 2024. Steering Llama 2 via Contrastive Activation Addition. In* Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15504–15522, Bangkok, Thailand. Association for Computational Linguistics.
- [23]Ryan Soh-Eun Shim, Kwanghee Choi, Kalvin Chang, Ming-Hao Hsu, Florian Eichin, Zhizheng Wu, Alane Suhr, Michael A. Hedderich, David Harwath, David R. Mortensen, and Barbara Plank. 2026. Linear script representations in speech foundation models enable zero-shot transliteration. In Findings of the Association for Computational Linguistics: ACL 2026, pages 9532–9544, San Diego, California, United States. Association for Computational Linguistics.
- [24]Anushka Sivakumar, Andrew Zhang, Zaber Ibn Abdul Hakim, and Chris Thomas. 2025. Steer VLM: Robust model control through lightweight activation steering for vision language models. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 26340–26365. Association for Computational Linguistics.
- [25]Elias Stengel-Eskin, Jimena Guallar-Blasco, Yi Zhou, and Benjamin Van Durme. 2023. Why did the chicken cross the road? rephrasing and analyzing ambiguous questions in VQA. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10220–10237, Toronto, Canada. Association for Computational Linguistics.
- [26]Jingran Su, Jingfan Chen, Hongxin Li, Yuntao Chen, Li Qing, and Zhaoxiang Zhang. 2025. Activation Steering Decoding: Mitigating Hallucination in Large Vision-Language Models through Bidirectional Hidden State Intervention. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12964–12974, Vienna, Austria. Association for Computational Linguistics.DOI
- [27]Alberto Testoni, Barbara Plank, and Raquel Fernández. 2025. RACQUET: Unveiling the dangers of overlooked referential ambiguity in visual LLMs. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 23627–23647. Association for Computational Linguistics.
- [28]Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. 2023. Steering Language Models With Activation Engineering. Preprint, arXiv:2308.10248.arxiv.org/abs/2308.10248
- [29]Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, Zhaokai Wang, Zhe Chen, Hongjie Zhang, Ganlin Yang, Haomin Wang, Qi Wei, Jinhui Yin, Wenhao Li, Erfei Cui, and 56 others. 2025. InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency. Preprint, arXiv:2508.18265.arxiv.org/abs/2508.18265
- [30]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengang Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.arxiv.org/abs/2505.09388
- [31]Jiaxi Yang, Shicheng Liu, Yuchen Yang, and Dongwon Lee. 2026. Steering to Say No: Configurable Refusal via Activation Steering in Vision Language Models. Preprint, arXiv:2602.07013.arxiv.org/abs/2602.07013
- [32]Jianghao Yin, Qin Chen, Kedi Chen, Jie Zhou, Xingjiao Wu, and Liang He. 2026. Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models. Preprint, arXiv:2602.21704.arxiv.org/abs/2602.21704
- [33]Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 11975–11986.arxiv.org/abs/2303.15343
- [34]Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, and 2 others. 2023. Representation Engineering: A Top-Down Approach to AI Transparency. Preprint, arXiv:2310.01405.arxiv.org/abs/2310.01405













