WorksComputer Science
Understanding Cross-Modal Inconsistency through Representation Subspaces in Omni Models
Abstract
Omni Models process audio, vision, and text within a unified language backbone, yet often produce different answers for semantically equivalent inputs across modalities. Existing representation-level analyses do not distinguish between information that predicts whether disagreement will occur and the representation whose perturbation affects the model's answer. In this paper, we estimate a low-dimensional Answer Subspace from answer-position representations and use it to separate answer-related components from the remaining representation. Across three Omni Models and two multimodal reasoning tasks, information predictive of cross-modal inconsistency lies mainly outside the Answer Subspace, concentrating in a reproducible two-dimensional representation that is not explained by a fixed modality offset. Matched interventions reveal distinct behavioral roles: perturbing the Answer Subspace substantially changes the model's predictions, whereas perturbing the disagreement representation, like a random subspace of the same dimensionality, leaves them almost unchanged. Our key finding is that cross-modal disagreement can be induced by misaligning modality-specific answer representations: perturbing the Answer Subspace for only one modality induces disagreement, whereas applying the same perturbation to both largely preserves agreement. Thus, the representation that predicts disagreement is distinct from the answer-level representation through which disagreement can be induced, showing that linear decodability does not imply behavioral efficacy.
Introduction
Recent Multimodal Large Language Models (MLLMs) have evolved beyond image–text understanding toward Omni Models that jointly process diverse modalities, including audio, and support any-to-any multimodal interaction. Omni Models integrate information extracted by modality-specific encoders within a shared language backbone to generate responses to heterogeneous sensory inputs. This capability has positioned Omni Models as promising foundation models for applications such as embodied agents and physical AI, which must understand and reason about the real world from multiple sensory signals. Such applications, in particular, require models to make consistent decisions even when the same environment or state is observed through different modalities. In practice, however, Omni Models often exhibit cross-modal inconsistency, producing different answers when semantically equivalent information is presented through different modalities, thereby undermining model reliability.
To address this issue, prior work has focused on constructing benchmarks for quantitatively evaluating cross-modal consistency and on mitigating inconsistency through training strategies such as modality alignment and preference optimization. These studies have established an important foundation by systematically measuring modality-dependent behavioral differences and demonstrating that consistency can be improved through additional training. However, because these approaches primarily evaluate models in terms of final-answer accuracy, agreement, or performance gains after training, they provide limited insight into how inputs from different modalities are transformed and computed within the language backbone to produce divergent answers.
Recent studies have therefore begun to diagnose cross-modal inconsistency by examining internal model states, including hidden representations, attention patterns, and activations. Nevertheless, representations derived from different modalities jointly encode modality identity, semantic content, and answer-related information. Consequently, directly comparing such representations makes it difficult to isolate the components involved in answer formation.
In this work, we analyze cross-modal inconsistency at the representation level by focusing on answer-position representations in the shared language backbone of Omni Models. We first estimate a low-dimensional subspace from which the model’s current prediction can be linearly decoded and use it as a reference coordinate system. Based on this representation, we investigate: (i) where information predictive of modality-dependent disagreement resides and how low-dimensional it is; (ii) what this representation reflects, in particular whether it corresponds to a fixed modality difference or an item-specific component; and (iii) how it differs from the answer representation under matched interventions.
Our results show that information predictive of cross-modal disagreement lies mainly outside the Answer Subspace, forms a reproducible two-dimensional representation, and is not explained by a fixed modality offset. Under matched interventions, however, the two subspaces behave differently: perturbing the Answer Subspace changes the model’s prediction, whereas perturbing the disagreement subspace leaves it essentially unchanged. Moreover, perturbing the Answer Subspace for only one modality reliably induces cross-modal disagreement, while the same perturbation applied to both modalities does not.
Our main contributions are as follows:
We estimate a low-dimensional subspace at the answer position and use it to separate representation components carrying the model’s answer from the remaining representation.
Using this separation, we show that information predictive of cross-modal inconsistency lies outside the Answer Subspace and is concentrated in a reproducible two-dimensional representation that is not explained by a fixed modality offset.
We show that, under matched additive interventions, the disagreement-predictive representation has no greater behavioral effect than random directions, whereas asymmetric perturbations of the Answer Subspace are sufficient to induce cross-modal disagreement.
Related Work
Cross-modal Consistency in Multimodal Language Models
Multimodal language models can produce different answers when semantically equivalent information is presented through different modalities. Such cross-modal inconsistency has been observed in vision–language models [27] (Zhang et al. 2025b), across text, image, and mixed-modal inputs [29], in omni-modal settings [7], and in complex multimodal documents (Yan et al., 2025). Related benchmarks diagnose visual conflicts and hallucinations [24, 20, 18, 13], and decoding and training strategies have been proposed to reduce such inconsistencies [17] (Zhang et al. 2025a).
At the representation level, linear probing, representational similarity analysis, and centered kernel alignment have been used to compare what is encoded across modalities [1, 16, 15], alongside work on shared embedding spaces [25, 12] (Zhu et al. 2024). These approaches compare representations as a whole, and therefore do not distinguish the components that carry the model’s answer from those that do not.
Mechanistic Interpretability and Decision Representations
Mechanistic interpretability relates model behavior to internal representations through targeted interventions. Causal tracing and activation patching measure the influence of specific activations by replacing or restoring them [22, 30, 9] (Zhang and Nanda 2023), while iterative nullspace projection and amnesic probing remove linearly decodable information and examine the resulting behavioral change [26, 11]. Decoding studies ask what decision-related information the hidden states already carry: Tuned Lens and Future Lens estimate future output distributions from intermediate states [4, 23], and probing studies examine whether factual knowledge, truthfulness, and confidence are linearly decodable [6, 3, 21, 5, 14]. Prior work has not established the internal mechanism by which semantically equivalent inputs lead to different answers across modalities. We address this gap by locating disagreement-predictive representations relative to answer-related ones and testing, through intervention, their roles in producing cross-modal inconsistency.
Method
Preliminaries
Multimodal Large Language Models. MLLMs extend autoregressive language models to process multiple modalities, including images and audio, in addition to text. Inputs from each modality are transformed by modality-specific encoders into representations that can be processed by the language backbone, after which inference is performed within the shared language backbone [2, 19, 8].
Let denote the set of modalities, and let denote a specific input modality. We denote the input representation of modality to the language backbone by , and the hidden representation of the -th token at the -th layer by .
Omni Models. An Omni Model is an MLLM extended to process diverse input and output modalities, including images, audio, and text, within a single model (Zhan et al., 2024; Xu et al., 2025). Although the encoder architecture and input integration mechanism vary across models, this work focuses on Omni Models in which inputs from different modalities are jointly processed by a shared language backbone.
Each modality may undergo a distinct encoding and preprocessing pipeline, but the hidden representations produced by the language backbone lie in a common representation space. We therefore directly compare hidden representations from different input modalities at both the layer and token levels within this shared space.
Problem Formulation
We present each item to an Omni Model through semantically equivalent inputs in two different modalities and require the model to generate a single answer choice. Rather than focusing on the final output itself, we analyze how modality-dependent behavioral differences are reflected in the answer-position representation at the point where the answer is generated.
At layer , we define the hidden representation at the answer-generation position as
where denotes the position at which the logits for the answer choice are computed, indexes the layer of the language backbone, denotes the input modality, and is the hidden dimension. Because the number of layers differs across models, we report layer depth in normalized form. The representation extraction procedure is described in Appendix § 6.§ 1.
To connect this representation to model behavior, we define the prediction for item as
Here, denotes item presented in modality , and denotes the four answer choices shared across the two modalities. The prediction is the answer-choice label generated by the model for the corresponding input.
We define cross-modal disagreement as the case in which the model produces different predictions across the two modalities:
In Eq. (3), denotes agreement and denotes disagreement.
Subspace Estimation
Answer Subspace Probe. The answer-position representation contains not only information associated with the model’s prediction but also variation arising from input format and modality-specific encoding. To isolate the components associated with the model’s answer, we train a multinomial logistic regression probe at each layer using answer-position representations from both modalities. The prediction target is the answer-choice label generated by the model for each input.
Let denote the coefficient matrix learned at layer . Since softmax predictions are invariant to adding the same constant to all class logits, the effective rank of is at most three. We define the Answer Subspace as
We obtain an orthonormal basis for by QR decomposition and define the component orthogonal to this subspace as
This decomposition allows disagreement-predictive information to be localized relative to the Answer Subspace. The probe-training procedure and computation of are described in Appendix § 6.§ 2.
Disagreement Subspace Probe. Because disagreement is defined between two modality conditions, each probe sample corresponds to one item and combines the residual representations from both conditions:
A binary probe is trained to predict in Eq. (3). Restricting the input to the orthogonal complement ensures that the recovered directions exclude those already used to decode the answer. The learned coefficient vector is grouped according to the three feature blocks in Eq. (6), yielding . We define the disagreement subspace using the leading two right singular vectors of ,
Although the probe is trained on paired features in , the resulting basis lies in the original single-state space and can therefore be applied to individual hidden states. Full estimation details and a comparison with the two-block alternative are provided in Appendices § 6.§ 3 and § 6.§ 4.
Subspace Intervention
Linear decodability alone does not establish that a representation contributes to model behavior. We therefore intervene directly on each subspace and measure the resulting changes in the model’s prediction.
Let denote an orthonormal basis for the target subspace. We apply the shift intervention directly to the answer-position representation:
where denotes a direction in the target-subspace coordinates and controls the intervention magnitude. All comparisons use the same layer, normalization, magnitude, and operation, differing only in the target subspace.
To test whether cross-modal disagreement can be induced, we additionally apply Eq. (7) asymmetrically to items on which the two modalities initially agree. We perturb the Answer Subspace for one modality while leaving the other unchanged, and compare this condition with applying the same perturbation to both modalities.
The construction and normalization of , random-subspace controls, the removal intervention, and additional implementation details are provided in Appendix § 6.§ 5.
Analysis
Experimental Setup
Models. We evaluate three Omni Models with shared language backbones: Qwen2.5-Omni (Xu et al., 2025), Gemma-4-E4B ([28]), and MiniCPM-o 4.5 ([10]).
Data. We use the spatial and external task families from XModBench (Wang et al., 2026). Each item is presented under matched audio and image conditions with identical questions, answer choices, and A–D assignments; only modality-specific prompt wording differs. The same items are evaluated for all three models, yielding six model–task conditions.
Evaluation. Probes are evaluated with item-grouped cross-validation, keeping the two modality conditions of each item in the same fold, and balanced accuracy is reported throughout. Condition sizes and intervention layers are provided in Appendix § 7.§ 1.
Where Is Disagreement-Predictive Information Encoded?
Answer-related representation. Figure 1a shows the layer-wise accuracy of the answer probe introduced in § 3.3. Probe accuracy increases with depth and approaches perfect recovery in the final layer, first reaching the midpoint between chance and its final layer value around the middle of the backbone in every condition. Answer-related information therefore becomes reliably decodable well before the output, and the learned three-dimensional subspace provides a compact representation from which the model’s predicted answer can be linearly decoded.

Figure 1. Layer-wise probe performance over normalized backbone depth. (a) Answer-probe accuracy shows that the model’s prediction becomes linearly decodable from the answer-position representation in the middle layers. (b) Disagreement-probe accuracy shows that disagreement-predictive information lies primarily outside the answer-related representation, with its orthogonal complement outperforming both the answer-related representation and matched random subspaces.
Disagreement-related representation. Figure 1b compares disagreement-probe performance using the answer-related representation, its orthogonal complement, and matched random subspaces. The orthogonal complement consistently outperforms both the answer-related representation and the random controls across all six conditions, with gains of +0.042–0.225 over the answer-related representation. These results indicate that information predictive of cross-modal disagreement resides primarily outside the answer-related representation, with the full per-condition breakdown provided in Appendix § 7.§ 2.
Low-dimensional approximation. To determine how many directions are needed to capture the dominant disagreement-predictive variation, we progressively retain the right singular vectors of the reshaped disagreement-probe coefficient matrix. Because this matrix has rank at most three, we evaluate one, two, and three retained directions. Figure 2 shows that disagreement-probe performance increases substantially from one to two directions, with little or no consistent improvement from the third. We therefore retain the leading two directions as a compact representation for the subsequent analyses and interventions.

Figure 2. Disagreement-probe accuracy using the leading one, two, or three right singular vectors of the reshaped disagreement-probe coefficient matrix, selected within each training fold. Performance increases substantially from one to two directions, with little additional improvement from the third. Dashed and dotted lines denote the full hidden-state and Answer Subspace probes, respectively. Appendix § 7.§ 3 reports per-condition results and bootstrap stability.
What Do the Two Subspaces Represent?
The preceding analyses identify two low-dimensional representation spaces: a three-dimensional space that decodes the model’s prediction and a two-dimensional space that predicts cross-modal disagreement. Decodability alone, however, does not establish what these spaces represent. We therefore test whether the former tracks the model’s current prediction and whether the latter reflects disagreement-specific structure rather than modality or answer identity.
Answer Subspace. The final-layer representation projected onto the answer-related space recovers the model’s prediction with an accuracy of 0.937–0.997, including items for which the two modalities produce different answers. Appendix § 7.§ 4 reports the full per-condition results. Thus, the recovered coordinates preserve the prediction associated with each condition rather than collapsing the two modalities into a shared answer. We further test whether these coordinates track changes in the model’s prediction by permuting the answer contents assigned to labels A–D.
As shown in Figure 3, model behavior varies substantially across conditions, whereas the decoded answer continues to match the model’s new prediction at 0.948–0.989. The decoded answer therefore reflects the model’s current prediction rather than either the original label assignment or the relocated answer content. Appendix § 7.§ 5 provides the detailed results. Together, these results validate the Answer Subspace as a faithful linear representation of the model’s current answer.

Figure 3. Answer decoding under option permutation. Despite substantial variation in model behavior, the decoded answer consistently matches the model’s updated prediction. Squares, diamonds, and circles denote content following, letter retention, and decoded-answer agreement, respectively.
Disagreement-Predictive Subspace. Because the disagreement probe jointly reads both modality conditions, the recovered representation could reflect either a fixed modality difference or the answers produced under the two conditions. Figure 4 argues against the first explanation: although modality is almost perfectly decodable from the answer-position representation, the recovered directions are nearly orthogonal to the modality axis, and removing the mean modality offset has negligible effect on disagreement-probe accuracy. Figure 5 addresses the second by restricting the analysis to items with the same audio-condition prediction.

Figure 4. Alignment of the recovered subspaces with the modality-discriminating axis. Despite near-perfect modality separability, both subspaces are nearly orthogonal to this axis. Removing the mean modality offset has negligible effect on disagreement-probe accuracy. Appendix § 7.6 reports the per-condition values.

Figure 5. Disagreement-probe accuracy after controlling for the audio-condition prediction. Filled and open circles denote answer-controlled and unrestricted accuracy, respectively, and dashed ticks mark the label-permutation null. Appendix § 7.7 reports the strata and sample sizes.
Disagreement remains decodable above the label-permutation baseline in every condition and performs at least as well as in the unrestricted analysis. Together, these controls validate the recovered representation as a Disagreement-Predictive Subspace whose signal is not explained by modality or answer identity.
What Are the Behavioral Roles of the Two Subspaces?
The preceding analyses identify two distinct representations: the Answer Subspace carries the answer produced under each modality condition, whereas the Disagreement-Predictive Subspace contains information associated with whether the two conditions will diverge. We first examine how naturally occurring disagreement is reflected in the Answer Subspace and then test the behavioral roles of the two subspaces.
Figure 6 plots the two modality conditions of each item in the Answer Subspace. For agreement items, the two states lie close together within the same answer region. For disagreement items, they occupy different answer regions, and their median distance is 1.9–5.3 times larger across the six conditions. After accounting for dimensionality, this excess separation is concentrated within the Answer Subspace rather than its orthogonal complement, by a factor of 2.9–8.6. Appendices § 7.8 and § 7.6 report the full three-dimensional pair distances and modality-relative geometry. Thus, information predicting which items will diverge lies outside the Answer Subspace, whereas naturally occurring divergence is reflected as a separation between the two modality-specific answer states.

Figure 6. Paired audio and image representations in the Answer Subspace across six model–task conditions. Gray lines connect the two modality conditions of each item. Agreement pairs remain within the same answer region, whereas disagreement pairs separate across answer regions.
Behavioral effect. Figure 7 compares matched interventions across a range of magnitudes. Perturbations within the Answer Subspace produce a clear dose-dependent behavioral effect, changing 0.83–1.00 of predictions at the largest magnitude across the six conditions. By contrast, perturbations within the Disagreement-Predictive Subspace remain near baseline throughout the sweep and show no consistent advantage over a matched-dimensional random subspace. Appendix § 8.§ 1 reports the full per-condition dose-response results.

Figure 7. Prediction changes under matched interventions across shift magnitudes. Left and right panels show the external and spatial task families, respectively. Lines denote the mean across three models and shaded regions the range across models. Perturbations of the Answer Subspace produce a strong dose-dependent effect, whereas perturbations of the Disagreement-Predictive Subspace remain comparable to a random two-dimensional subspace drawn from the Answer Subspace complement.
The effect is also depth-dependent. Applying the same intervention before the answer becomes decodable reduces the effect of the Answer Subspace from 89–100 to 0–3 changed predictions out of 100. Appendix § 8.§ 2 reports the per-condition results. Behavioral sensitivity therefore emerges only after the Answer Subspace has formed, whereas the Disagreement-Predictive Subspace shows no comparable behavioral effect under matched interventions.
Generating inconsistency. If manipulating the Disagreement-Predictive Subspace does not alter the model’s prediction, what does? We address this question using items for which the two modalities initially produce the same answer. We perturb the Answer Subspace in only one modality while leaving the other unchanged, and compare this asymmetric intervention with applying the same perturbation to both modalities. As shown in Figure 8, perturbing only one modality induces substantially more disagreement than perturbing both. The asymmetric intervention creates disagreement in 0.767–1.000 of the 60 items per condition, whereas the symmetric intervention yields rates of only 0.050–0.250, a gap of at least a factor of 3.1 in every condition. No prediction changes are observed in the non-intervened modality. This comparison requires a magnitude at which the answer representation can be relocated; the asymmetric arm exceeds its control in all six conditions at the magnitude reported here and in five of six one step below, but not at smaller magnitudes (Appendix § 8.6).

Figure 8. Disagreement induced by symmetric and asymmetric Answer Subspace interventions. Applying the perturbation to one modality produces substantially more disagreement than applying it to both under the same layer, magnitude, and operation.
Implications. These results distinguish the representation that predicts disagreement from the representation through which disagreement can be generated. The Disagreement-Predictive Subspace contains information associated with cross-modal divergence but has little behavioral effect under matched interventions. By contrast, the Answer Subspace does not predict whether disagreement will occur, yet perturbing it asymmetrically is sufficient to induce disagreement. Thus, linear decodability does not imply behavioral efficacy, and these two properties are distributed across distinct subspaces. Appendix § 8.§ 3 reports the cohort analysis, Appendix § 8.§ 4 the removal interventions and their limitations, and Appendix § 8.§ 5 verifies that the interventions reached their target states.
Conclusion
We investigated cross-modal inconsistency relative to the representation that encodes an Omni Model’s answer. By estimating a low-dimensional Answer Subspace, we separated answer-related components from the remaining hidden state and localized disagreement-predictive information. This information lies outside the Answer Subspace and is concentrated in a reproducible two-dimensional subspace not explained by modality or answer identity. Matched interventions reveal distinct behavioral roles: perturbing the Disagreement-Predictive Subspace produces no greater effect than a matched random subspace, whereas perturbing the Answer Subspace changes predictions in a dose-dependent manner. Naturally occurring disagreement appears as separation between modality-specific states in the Answer Subspace, and asymmetric perturbation of this subspace is sufficient to induce disagreement. These results indicate that cross-modal inconsistency is expressed through misaligned modality-specific answer representations, while the representation predicting this mismatch does not directly control answer selection under the tested interventions. More broadly, a representation from which a behavioral property is decoded need not be the representation through which it is produced.
Limitations
The disagreement-predictive coordinates are estimated from both modality conditions and therefore cannot flag disagreement before both predictions are available. Our use of predictive refers to statistical covariation with the disagreement label rather than a deployable forecast. We do not establish what the recovered variation represents. Because disagreement items may differ systematically in difficulty or ambiguity, the recovered directions may partly reflect such factors. Holding the audio-condition prediction fixed removes one answer-related confound but not the remaining item-level variation. Our modality control establishes non-alignment rather than the absence of modality information. Modality remains partly decodable within the two-dimensional subspace, and removing its direction leaves too few dimensions to test the stronger claim. Finally, our experiments cover three Omni Models and two task families from a single benchmark. Although the models differ in depth, hidden dimension, and normalization, all place modality encoders before a decoder-only backbone. Generalization to other integration designs, modalities, task families, and training regimes remains untested. We also study only inference-time interventions and do not evaluate training-time mitigation.
Ethical considerations
This work analyzes existing multimodal models and benchmark data and does not involve human participants, personal data, or sensitive content. The findings concern model behavior and should not be interpreted as guarantees of reliability in safety-critical applications. AI assistants were used to assist with code implementation, and all generated code was manually reviewed, validated, and approved by the authors before being used in the experiments.
Acknowledgement
This work was supported by Korea Internet & Security Agency(KISA) grant funded by the Korea government(PIPC) (No.RS-2026-25526342, Development of Technologies for Preventing Sensitive Information Inference and Risk Assessment in Foundation Model Operations). This work was also supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT) (RS-2026-25483747). In addition, this research was supported by Culture, Sports and Tourism R&D Program through the Korea Creative Content Agency grant funded by the Ministry of Culture, Sports and Tourism in 2026 (Project Name: Develop AI agent technology to connect knowledge through public cultural facility-based discussion and communication, Project Number: RS-2026-25520645). Furthermore, this work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP, AI Computing Support Project for R&D) grant funded by the Korea government(MSIT) (High-Performance Research AI Computing Infrastructure Support at the 2 PFLOPS Scale, RS-2026-25505492). Lastly, we gratefully acknowledge VESSL for providing GPU resources.
Implementation Details
Representation Extraction
For each item we render two prompts that differ only in the modality of the evidence and in the modality-specific wording of the question; the question stem, the four answer choices, and their assignment to the letters A–D are identical across the two conditions. The gold label never appears in the prompt. Each prompt is passed through the model once with output_hidden_states enabled and no sampling. We record the hidden state at the position whose logits determine the answer letter, which is the final position of the rendered prompt, at every layer of the language backbone. The per-condition tensors are stored as answer_a and answer_v of shape together with the model’s own predicted letter for each condition. Prediction is read as the arg-max over the logits of the four letter tokens at that position, using the token id of each letter as it is tokenized in the answer slot. Items on which the answer letter cannot be resolved are dropped before any analysis; the retained counts appear in Table 2.
| Model | Family | Layers | Items | Used | Agree | Disagree | Half-dec. layer | Half-dec. depth | Interv. depth |
| Qwen2.5-Omni | Ext. | 29 | 1350 | 1350 | 1038 | 312 | 15 | 53.600 | 75.000 |
| Qwen2.5-Omni | Spa. | 29 | 465 | 465 | 165 | 300 | 19 | 67.900 | 75.000 |
| Gemma-4-E4B | Ext. | 43 | 1350 | 1350 | 528 | 822 | 18 | 42.900 | 76.200 |
| Gemma-4-E4B | Spa. | 43 | 465 | 465 | 213 | 252 | 24 | 57.100 | 76.200 |
| MiniCPM-o 4.5 | Ext. | 37 | 1350 | 1350 | 864 | 486 | 17 | 47.200 | 75.000 |
| MiniCPM-o 4.5 | Spa. | 37 | 465 | 465 | 141 | 324 | 20 | 55.600 | 75.000 |
Table 2. Conditions used throughout. Used is the number of items on which the answer letter could be resolved in both conditions; Agree and Disagree split those items by whether the two modality conditions produced the same letter. Half-dec. layer is the layer at which answer-probe accuracy first reaches the midpoint between chance and its final-layer value, and Half-dec. depth the corresponding normalized depth. Interv. depth is the normalized depth of the layer at which interventions are applied.
Answer Subspace Estimation
At each layer we pool the answer-position states of both modality conditions and fit a multinomial logistic regression on standardized features (StandardScaler followed by LogisticRegression, max_iter = 20000, tol = 10^{-6}, default penalty, fixed seed) with the model’s own predicted letter as the target. Cross-validation is grouped by item, so the two conditions of an item always fall in the same fold, and we report the mean over three schemes: stratified grouped 5-fold, plain grouped 5-fold, and stratified 5-fold. All three give the same half-decodability depth in every condition; the spread across schemes is shown as the band in Figure 1a. The coefficient matrix has effective rank at most three because softmax predictions are invariant to a constant shift of all class logits. We take a thin QR decomposition of and keep the first three columns of as the orthonormal basis . Coordinates are obtained as and the complement by the projector in Eq. (5). Bases are re-estimated inside each fold; all reported accuracies are held-out.
Disagreement Subspace Estimation
The disagreement probe uses the same pipeline — standardization, logistic regression with the same settings, item-grouped stratified 5-fold — with the binary label of Eq. (3) and one sample per item. Its input is the concatenation of Eq. (6), formed from the complement components rather than the raw states, so no direction already used to decode the answer can enter. Reshaping the fitted coefficient vector into and taking the leading two right singular vectors gives ; the basis is re-orthogonalized against and re-QR’d, which leaves a residual leakage into the Answer Subspace below in every condition. Alongside the two feature sets reported in the body we evaluate a rank-matched random readout of the complement, averaged over five draws, and a PCA readout of the same rank; these appear in Table 3.
| Model | Family | Inside | Compl. | Random | PCA | Full |
| Qwen2.5-Omni | Ext. | 0.510 | 0.704 | 0.511 | 0.518 | 0.705 |
| Qwen2.5-Omni | Spa. | 0.516 | 0.692 | 0.510 | 0.517 | 0.706 |
| Gemma-4-E4B | Ext. | 0.586 | 0.692 | 0.534 | 0.623 | 0.699 |
| Gemma-4-E4B | Spa. | 0.645 | 0.687 | 0.572 | 0.631 | 0.692 |
| MiniCPM-o 4.5 | Ext. | 0.500 | 0.725 | 0.679 | 0.685 | 0.718 |
| MiniCPM-o 4.5 | Spa. | 0.495 | 0.659 | 0.516 | 0.544 | 0.651 |
Table 3. Balanced accuracy of the disagreement probe at the final layer under five feature sets: coordinates inside the Answer Subspace, inside its orthogonal complement, a rank-matched random readout of that complement (mean of five draws), a rank-matched PCA readout of the complement, and the full hidden state. Chance is 0.5. The complement matches the full state to within 0.014 in every condition, while the two rank-matched readouts of the same complement fall well short of it, so the result does not follow from the complement’s dimensionality alone.
Feature Blocks
The difference block is a linear function of the other two and therefore does not enlarge the hypothesis class of a linear probe. We keep it for a different reason: with three blocks the reshaped coefficient matrix has rank at most three, so saturation at two directions is an observation about the data. With two blocks the rank ceiling would be exactly two and the same fact would be a construction artifact. Table 1 confirms that the choice does not change the result. Discriminability with two blocks matches the three-block value to within 0.019, and the planes recovered under the two parameterizations overlap at 0.972–0.991.
| Model | Family | 3 blocks | 2 blocks | Overlap |
| Qwen2.5-Omni | Ext. | 0.679 | 0.674 | 0.972 |
| Qwen2.5-Omni | Spa. | 0.704 | 0.687 | 0.985 |
| Gemma-4-E4B | Ext. | 0.718 | 0.712 | 0.988 |
| Gemma-4-E4B | Spa. | 0.681 | 0.662 | 0.989 |
| MiniCPM-o 4.5 | Ext. | 0.723 | 0.723 | 0.978 |
| MiniCPM-o 4.5 | Spa. | 0.639 | 0.638 | 0.991 |
Table 1. Effect of removing the difference block. The first two columns are balanced accuracy of the disagreement probe on the full complement; Overlap is the mean principal-angle cosine between the two-dimensional subspaces recovered under the two parameterizations.
Intervention Details
All interventions add a fixed vector to the answer-position state at a single layer during a forward pass, as in Eq. (7), and leave every other position and layer untouched.
Direction. For the Answer Subspace, points from the centroid of the currently decoded answer region to the centroid of the next letter in cyclic order, so the perturbation targets a specific alternative rather than an arbitrary direction. For the Disagreement-Predictive Subspace, is the difference between the centroids of disagreement and agreement items in the two-dimensional coordinates. For the random control, is an isotropic Gaussian draw with a fixed seed.
Normalization. Each is rescaled to unit length and multiplied by the coordinate standard deviation of its own subspace, so one unit of is one standard deviation of that subspace’s coordinates. This matters because the two subspaces differ in both dimensionality and scale: at the intervention depth of Qwen2.5-Omni on external, the coordinate spread is 7.24 in the Answer Subspace and 0.84 in the Disagreement-Predictive Subspace, so an unnormalized comparison would favor one arm by roughly a factor of nine.
Controls. The random control is a two-dimensional subspace drawn from the orthogonal complement of the Answer Subspace and dimension-matched to the Disagreement-Predictive Subspace. The pre-formation control applies the identical operation at a shallower layer, chosen per condition to fall before the depth at which the answer first becomes decodable.
Depths. Interventions use a layer at approximately 75% of normalized depth; the exact layer and the pre-formation layer for each condition appear in Table 2.
Per-condition Results
Conditions, Depths and Cohort Sizes
Table 2 lists every condition used in the paper, together with the depths at which the answer first becomes decodable and at which interventions are applied.
Localization of Disagreement-Predictive Information
Table 3 gives the per-condition values behind Figure 1b, evaluated at the final layer; interventions instead use the depths in Table 2.
Dimensionality and Stability
Directions are estimated separately within each training fold and evaluated on the corresponding held-out fold. Because the reshaped coefficient matrix has rank at most three, we evaluate the leading one, two, and three right singular vectors. Table 4 reports how disagreement discriminability changes with the number of retained directions and how stable the leading two-dimensional subspace is under item-level resampling.
| Model | Family | Random | Bootstrap align. | Min | Null | |||
| Qwen2.5-Omni | Ext. | 0.593 | 0.692 | 0.694 | 0.515 | 0.733 | 0.683 | 0.033 |
| Qwen2.5-Omni | Spa. | 0.666 | 0.694 | 0.691 | 0.512 | 0.725 | 0.632 | 0.031 |
| Gemma-4-E4B | Ext. | 0.649 | 0.670 | 0.664 | 0.535 | 0.713 | 0.674 | 0.037 |
| Gemma-4-E4B | Spa. | 0.629 | 0.660 | 0.674 | 0.566 | 0.719 | 0.670 | 0.039 |
| MiniCPM-o 4.5 | Ext. | 0.625 | 0.721 | 0.718 | 0.568 | 0.724 | 0.693 | 0.029 |
| MiniCPM-o 4.5 | Spa. | 0.611 | 0.639 | 0.641 | 0.507 | 0.747 | 0.700 | 0.028 |
Table 4. Number of retained directions and stability of the recovered plane. The first four columns are balanced accuracy of the disagreement probe on directions of the complement, and on a random two-dimensional subspace of the same complement. Bootstrap align. is the mean principal-angle cosine between the plane recovered from an item-level bootstrap resample (30 resamples) and the plane recovered from the full sample; Min is the smallest such value and Null the 95th percentile obtained when the resampled directions are replaced by random ones.
Answer Recovery under the Original Prompt
Table 5 gives the values behind the recovery accuracy quoted in § 4.
| Model | Family | Agree Aud. | Agree Img. | Disagree Aud. | Disagree Img. |
| Qwen2.5-Omni | Ext. | 0.992 | 0.990 | 0.978 | 0.955 |
| Qwen2.5-Omni | Spa. | 0.939 | 0.982 | 0.970 | 0.997 |
| Gemma-4-E4B | Ext. | 0.987 | 0.991 | 0.987 | 0.996 |
| Gemma-4-E4B | Spa. | 0.967 | 0.981 | 0.937 | 0.952 |
| MiniCPM-o 4.5 | Ext. | 0.983 | 0.987 | 0.949 | 0.969 |
| MiniCPM-o 4.5 | Spa. | 0.979 | 0.993 | 0.954 | 0.985 |
Table 5. Agreement between the answer decoded from the final-layer Answer Subspace and the model’s actual prediction, reported separately for each modality condition and for items where the two conditions agree or disagree. Agreement exceeds 0.93 throughout, including items where the two conditions produced different letters.
Option Permutation
Table 6 decomposes the permutation experiment by what the model did, showing that the decoded answer follows the model in every behavior group.
| Model | Family | Content | Letter | Other | Model | Model | content | Model | letter | |
| Qwen2.5-Omni | Ext. | 2700 | 0.919 | 0.028 | 0.053 | 0.985 | 0.991 | 0.895 |
| Qwen2.5-Omni | Spa. | 1320 | 0.618 | 0.175 | 0.207 | 0.948 | 0.971 | 0.935 |
| Gemma-4-E4B | Ext. | 2700 | 0.632 | 0.184 | 0.184 | 0.984 | 0.989 | 0.978 |
| Gemma-4-E4B | Spa. | 1320 | 0.366 | 0.305 | 0.329 | 0.967 | 0.969 | 0.970 |
| MiniCPM-o 4.5 | Ext. | 2700 | 0.880 | 0.043 | 0.076 | 0.985 | 0.990 | 0.966 |
| MiniCPM-o 4.5 | Spa. | 1320 | 0.777 | 0.081 | 0.142 | 0.989 | 0.997 | 0.963 |
Table 6. Option permutation, decomposed. is the number of (item, modality) pairs. Model behavior is how the model’s own prediction moved: to the letter now holding the original content, staying on the original letter, or to a third letter. The remaining columns are the fraction of pairs on which the answer decoded from the Answer Subspace agrees with the model’s new prediction, overall and restricted to the two behavior groups. The readout follows the model in every group, including when the model stays on the original letter.
Geometry Relative to Modality
Table 7 gives the raw pair distances summarized in § 4 and the per-condition angles behind Figure 4.
| Model | Family | Agree | Disagree | Ratio | Excess/dim | Separability | cos answer | cos disagree |
| Qwen2.5-Omni | Ext. | 0.170 | 0.225 | 1.324 | 7.310 | 1.000 | 0.054 | 0.023 |
| Qwen2.5-Omni | Spa. | 0.173 | 0.218 | 1.260 | 3.170 | 1.000 | 0.034 | 0.023 |
| Gemma-4-E4B | Ext. | 0.681 | 0.931 | 1.367 | 8.620 | 1.000 | 0.065 | 0.036 |
| Gemma-4-E4B | Spa. | 0.841 | 0.899 | 1.069 | 4.750 | 1.000 | 0.040 | 0.060 |
| MiniCPM-o 4.5 | Ext. | 0.892 | 1.060 | 1.188 | 2.920 | 1.000 | 0.046 | 0.017 |
| MiniCPM-o 4.5 | Spa. | 1.266 | 1.316 | 1.039 | 3.070 | 1.000 | 0.010 | 0.043 |
Table 7. Geometry at the intervention depth. Pair distance is the mean distance between the two modality conditions of the same item inside the Answer Subspace, split by cohort, with their ratio. Excess/dim is how much more the excess separation of disagreeing pairs concentrates inside the Answer Subspace than in its complement, per dimension. Separability is the balanced accuracy of a probe predicting which modality the input arrived in from the answer-position state; the two cos columns are the cosine of the principal angle between the single axis that probe defines and each subspace. Removing the mean modality offset changes disagreement accuracy by at most 0.0031.
Answer-matched Control
Table 8 reports the strata, sample sizes and unrestricted values for the control shown in Figure 5.
| Model | Family | Strata | Unrestr. | Matched | Null | |
| Qwen2.5-Omni | Ext. | 4 | 675 | 0.635 | 0.670 | 0.509 |
| Qwen2.5-Omni | Spa. | 4 | 233 | 0.558 | 0.726 | 0.587 |
| Gemma-4-E4B | Ext. | 4 | 675 | 0.642 | 0.712 | 0.528 |
| Gemma-4-E4B | Spa. | 2 | 157 | 0.648 | 0.703 | 0.586 |
| MiniCPM-o 4.5 | Ext. | 4 | 675 | 0.683 | 0.697 | 0.534 |
| MiniCPM-o 4.5 | Spa. | 2 | 171 | 0.605 | 0.711 | 0.524 |
Table 8. Holding the audio-condition prediction fixed. A fully answer-pair-matched comparison is impossible because a fixed pair of predictions determines the disagreement label, so we stratify on the audio-condition prediction alone and pool the within-stratum results. Strata is the number of audio-condition answers that contain both cohorts, the number of items retained, and Null the 95th percentile of a label-permutation null computed under the same stratification. The matched values exceed that null in every condition and are no lower than the unrestricted ones; the unrestricted value for Qwen2.5-Omni on spatial falls below the null, which is why the body claim rests on the matched series.
Pair Displacement
Table 9 gives the pair distances computed in the full subspace rather than in the projection shown in Figure 6.
| Model | Family | Disagree | Dist. disagree | Dist. agree | Ratio | |
| Qwen2.5-Omni | Ext. | 1350 | 312 | 2.199 | 0.418 | 5.263 |
| Qwen2.5-Omni | Spa. | 465 | 300 | 2.694 | 1.117 | 2.413 |
| Gemma-4-E4B | Ext. | 1350 | 822 | 2.877 | 0.573 | 5.021 |
| Gemma-4-E4B | Spa. | 465 | 252 | 2.247 | 0.913 | 2.460 |
| MiniCPM-o 4.5 | Ext. | 1350 | 486 | 2.113 | 0.896 | 2.358 |
| MiniCPM-o 4.5 | Spa. | 465 | 324 | 2.615 | 1.362 | 1.921 |
Table 9. Median distance between the two modality conditions of the same item, computed in the full three-dimensional Answer Subspace rather than in the two-dimensional projection shown in Figure 6, in units of the coordinate standard deviation. Disagree is the number of disagreement items among ; Dist. disagree and Dist. agree are the median pair distances within each cohort and Ratio their quotient.
Intervention Results
Magnitude Sweep
Table 10 gives the per-condition values behind Figure 7.
| Subspace | Qwen Ext. | Qwen Spa. | Gemma Ext. | Gemma Spa. | MiniCPM Ext. | MiniCPM Spa. | |
| Answer | 2 | 0.040 | 0.065 | 0.020 | 0.110 | 0.105 | 0.085 |
| Answer | 5 | 0.090 | 0.135 | 0.075 | 0.275 | 0.180 | 0.195 |
| Answer | 10 | 0.170 | 0.250 | 0.295 | 0.500 | 0.740 | 0.810 |
| Answer | 20 | 0.640 | 0.485 | 0.860 | 0.760 | 0.900 | 0.995 |
| Answer | 40 | 0.980 | 0.895 | 0.980 | 0.825 | 1.000 | 0.995 |
| Disagreement | 2 | 0.000 | 0.005 | 0.000 | 0.010 | 0.005 | 0.005 |
| Disagreement | 5 | 0.005 | 0.000 | 0.000 | 0.010 | 0.005 | 0.010 |
| Disagreement | 10 | 0.005 | 0.005 | 0.000 | 0.020 | 0.000 | 0.005 |
| Disagreement | 20 | 0.000 | 0.015 | 0.000 | 0.030 | 0.005 | 0.015 |
| Disagreement | 40 | 0.015 | 0.025 | 0.000 | 0.050 | 0.015 | 0.030 |
| Random | 2 | 0.005 | 0.005 | 0.000 | 0.005 | 0.005 | 0.005 |
| Random | 5 | 0.000 | 0.000 | 0.000 | 0.010 | 0.005 | 0.005 |
| Random | 10 | 0.000 | 0.000 | 0.000 | 0.015 | 0.005 | 0.010 |
| Random | 20 | 0.005 | 0.005 | 0.005 | 0.045 | 0.005 | 0.010 |
| Random | 40 | 0.005 | 0.010 | 0.010 | 0.090 | 0.010 | 0.015 |
Table 10. Fraction of predictions changed across intervention magnitudes , measured in target-subspace standard deviations, over 100 items per condition. Answer Subspace interventions increase monotonically and exceed both controls from onward.
Prediction Changes by Subspace and Depth
Table 11 gives the per-condition counts, including the pre-formation control.
| Model | Family | At intervention depth: Answer | At intervention depth: Answer | At intervention depth: Disagree | At intervention depth: Random | Before answer forms: Answer | Before answer forms: Disagree |
| Qwen2.5-Omni | Ext. | 28 | 98 | 1 | 0 | 0 | 0 |
| Qwen2.5-Omni | Spa. | 45 | 96 | 6 | 2 | 1 | 2 |
| Gemma-4-E4B | Ext. | 36 | 100 | 0 | 1 | 1 | 0 |
| Gemma-4-E4B | Spa. | 56 | 89 | 6 | 18 | 2 | 9 |
| MiniCPM-o 4.5 | Ext. | 75 | 100 | 1 | 3 | 0 | 1 |
| MiniCPM-o 4.5 | Spa. | 85 | 99 | 5 | 4 | 3 | 4 |
Table 11. Number of predictions changed out of 100 items. All arms use the same layer, operation and magnitude within a depth group. Perturbing the Answer Subspace at the intervention depth changes 89–100 predictions, and the same perturbation applied before the answer becomes decodable changes 0–3. The Disagreement-Predictive Subspace and the random control are both near zero and show no consistent ordering: the random control exceeds the disagreement subspace in three of the six conditions, most visibly for Gemma-4-E4B on spatial, where a random two-dimensional subspace changes 18 predictions. In that condition the two arms are therefore not distinguishable and the claim rests on the gap against the Answer Subspace rather than on an ordering between the controls.
Cohort Split
Table 12 splits the Answer Subspace intervention by whether the two modality conditions originally agreed.
| Model | Family | Agree | Disagree | Changed at Agree | Changed at Disagree | Changed at | Settling depth (%) Agree | Settling depth (%) Disagree | Settling depth (%) |
| Qwen2.5-Omni | Ext. | 69 | 31 | 0.159 | 0.548 | 0.389 | 49.900 | 57.300 | 7.400 |
| Qwen2.5-Omni | Spa. | 34 | 66 | 0.471 | 0.439 | -0.031 | 59.600 | 59.600 | 0.000 |
| Gemma-4-E4B | Ext. | 59 | 41 | 0.390 | 0.317 | -0.073 | 50.800 | 49.900 | -0.900 |
| Gemma-4-E4B | Spa. | 47 | 53 | 0.426 | 0.679 | 0.254 | 58.200 | 62.900 | 4.700 |
| MiniCPM-o 4.5 | Ext. | 70 | 30 | 0.714 | 0.833 | 0.119 | 48.000 | 52.900 | 4.900 |
| MiniCPM-o 4.5 | Spa. | 36 | 64 | 0.778 | 0.891 | 0.113 | 53.600 | 55.300 | 1.700 |
Table 12. Answer Subspace intervention split by cohort, at the smaller magnitude. Disagreement items change more often in four of the six conditions, and the sign of that difference agrees with the difference in settling depth — the normalized depth from which the decoded answer stops changing — in every condition (, ). At both cohorts are near ceiling and the difference disappears. Cohort sizes are unequal because the split follows the model’s own behavior, so these comparisons are the least powered in the paper and we do not build any claim in the body on them.
Removal Interventions
Instead of shifting the state along a direction, a removal intervention projects the state onto the orthogonal complement of the target subspace, deleting that subspace’s contribution altogether. We report it separately because its outcome is weaker and more ambiguous than the shift intervention, and because the ambiguity is informative. Table 13 gives the change in the margin between the model’s top two answer letters after removing (i) the two-dimensional disagreement subspace, (ii) the three-dimensional Answer Subspace, and (iii) random subspaces of each dimensionality, over 240 (item, modality) pairs per cell with paired bootstrap intervals. Removing the Answer Subspace moves the margin significantly in five of the six conditions, and most strongly for MiniCPM-o 4.5 on external (−0.416). Removing the disagreement subspace is indistinguishable from zero in five conditions and marginally significant in one (−0.014); in that same condition a random two-dimensional subspace is also significant and of comparable magnitude (−0.013), so the effect is not attributable to the disagreement directions. Two limits keep us from resting any claim on this experiment. First, the removed energy differs across arms: a two-dimensional removal deletes 0.5–5.3% of the state norm while removing the three-dimensional Answer Subspace deletes 3.4–8.5%, so the arms are not matched in the way the shift intervention is. Second, the number of outright prediction flips is small in every arm (0–9 of 240), so the margin is the only sensitive readout and does not directly measure discrete prediction changes. The shift intervention reported in the body avoids both problems.
| Model | Family | Disagree () | Random () | Random (, b) | Answer () | Random () |
| Qwen2.5-Omni | Ext. | -0.002 | +0.002 | +0.008 | -0.103 | -0.033 |
| Qwen2.5-Omni | Spa. | +0.002 | -0.000 | -0.000 | -0.041 | -0.009 |
| Gemma-4-E4B | Ext. | +0.008 | +0.043 | -0.027 | -0.054 | +0.006 |
| Gemma-4-E4B | Spa. | +0.003 | -0.004 | -0.007 | +0.039 | -0.002 |
| MiniCPM-o 4.5 | Ext. | -0.015 | +0.006 | -0.013 | -0.416 | -0.019 |
| MiniCPM-o 4.5 | Spa. | -0.005 | -0.001 | -0.011 | -0.027 | -0.018 |
Table 13. Change in the top-two answer margin after removing each subspace, over 240 (item, modality) pairs. Values in bold differ from zero at the 95% level under a paired bootstrap. Two independent random draws at are shown to indicate the spread of the control itself.
Verification that Interventions Reach Their Target
A null behavioral result is only interpretable if the intervention actually moved the state where it was aimed. We verified this on Qwen2.5-Omni, external, over 40 items, by reading the post-intervention state back out at the intervention layer. Injecting produced a displacement of 39.97 coordinate standard deviations along the target direction, so the intended shift is delivered essentially exactly. The disagreement probe called the perturbed state a disagreement item in every case, confirming that the state was moved into the region the probe associates with disagreement. No prediction changed. Two observations qualify this. Measured at the deeper layers where the answer is read, the displacement has shrunk to 3.88 standard deviations, and after the final layer normalization the residual displacement is 0.87 standard deviations. We report these as observations only: whether the shrinkage reflects an active correction or the ordinary contraction of an off-distribution perturbation is not something this measurement distinguishes. The verification establishes that the intervention reached its target at the layer where it was applied, which is what the null result requires; it does not establish that the perturbation survives to the output unchanged.
Asymmetric Intervention across Magnitudes
The asymmetric experiment in the body uses a single magnitude. Table 14 reports an independent rerun of both arms over the full magnitude grid. The result bounds the claim: the asymmetric intervention exceeds its symmetric control in every condition at , in five of six at , and in neither direction consistently below that. At the symmetric control is in fact higher in every condition.
| Model | Family | 2 | 5 | 10 | 20 | 40 | 2 | 5 | 10 | 20 | 40 | |
| Qwen2.5-Omni | Ext. | 60 | 0.000 | 0.000 | 0.017 | 0.500 | 0.983 | 0.033 | 0.083 | 0.183 | 0.300 | 0.050 |
| Qwen2.5-Omni | Spa. | 60 | 0.033 | 0.033 | 0.100 | 0.367 | 0.833 | 0.200 | 0.300 | 0.550 | 0.600 | 0.233 |
| Gemma-4-E4B | Ext. | 60 | 0.000 | 0.000 | 0.200 | 0.817 | 0.950 | 0.033 | 0.133 | 0.383 | 0.183 | 0.083 |
| Gemma-4-E4B | Spa. | 60 | 0.000 | 0.217 | 0.400 | 0.700 | 0.767 | 0.067 | 0.283 | 0.433 | 0.550 | 0.250 |
| MiniCPM-o 4.5 | Ext. | 60 | 0.017 | 0.033 | 0.717 | 0.950 | 1.000 | 0.083 | 0.217 | 0.233 | 0.200 | 0.100 |
| MiniCPM-o 4.5 | Spa. | 60 | 0.050 | 0.100 | 0.867 | 1.000 | 1.000 | 0.133 | 0.333 | 0.333 | 0.067 | 0.067 |
Table 14. Fraction of originally agreeing items on which disagreement appears, by intervention magnitude in units of the coordinate standard deviation. Asymmetric perturbs the Answer Subspace under one modality only; the symmetric control applies the identical shift to both. is the number of originally agreeing items evaluated. The asymmetric arm exceeds the control in all six conditions at (by at least a factor of 3.07), in five of six at , and in none at .
The mechanism is straightforward. Creating disagreement asymmetrically requires the perturbed condition’s own answer to move, and at the Answer Subspace intervention changes only 0.071 of predictions, so most items are untouched in both conditions. The symmetric arm has a different failure mode: applying the same shift to both conditions moves their predictions at different magnitudes, and that incidental desynchronization produces disagreement more often than a deliberate asymmetry does when neither condition is reliably displaced. The experiment therefore presupposes a magnitude at which the answer representation can actually be relocated.
One observation holds at every magnitude: the prediction of the untouched condition never moved, in any of the 1800 item-by-magnitude combinations.
References
- [1]Guillaume Alain and Yoshua Bengio. 2016. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644.arxiv.org/abs/1610.01644
- [2]Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. 2022. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204.14198.arxiv.org/abs/2204.14198
- [3]Amos Azaria and Tom Mitchell. 2023. The internal state of an llm knows when it’s lying. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 967–976.arxiv.org/abs/2304.13734
- [4]Nora Belrose, Igor Ostrovsky, Lev McKinney, Zach Furman, Logan Smith, Danny Halawi, Stella Biderman, and Jacob Steinhardt. 2023. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.08112.arxiv.org/abs/2303.08112
- [5]Lennart Bürger, Fred A Hamprecht, and Boaz Nadler. 2024. Truth is universal: Robust detection of lies in llms. Advances in Neural Information Processing Systems, 37:138393–138431.arxiv.org/abs/2407.12831
- [6]Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. 2024. Discovering latent knowledge in language models without supervision. Preprint, arXiv:2212.03827.arxiv.org/abs/2212.03827
- [7]Lichang Chen, Hexiang Hu, Mingda Zhang, Yiwen Chen, Zifeng Wang, Yandong Li, Pranav Shyam, Tianyi Zhou, Heng Huang, Ming-Hsuan Yang, and Boqing Gong. 2024. Omnixr: Evaluating omni-modality language models on reasoning across modalities. Preprint, arXiv:2410.12219.arxiv.org/abs/2410.12219
- [8]Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al. 2024. Qwen2-audio technical report. arXiv preprint arXiv:2407.10759.arxiv.org/abs/2407.10759
- [9]Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adria Garriga-Alonso. 2023. Towards automated circuit discovery for mechanistic interpretability. Advances in Neural Information Processing Systems, 36:16318–16352.arxiv.org/abs/2304.14997
- [10]Junbo Cui, Bokai Xu, Chongyi Wang, Tianyu Yu, Weiyue Sun, Yingjing Xu, Tianran Wang, Zhihui He, Wenshuo Ma, Tianchi Cai, et al. 2026. MiniCPM-o 4.5: Towards real-time full-duplex omni-modal interaction. arXiv preprint arXiv:2604.27393.arxiv.org/abs/2604.27393
- [11]Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. 2021. Amnesic probing: Behavioral explanation with amnesic counterfactuals. Preprint, arXiv:2006.00995.arxiv.org/abs/2006.00995
- [12]Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. 2023. Imagebind: One embedding space to bind them all. Preprint, arXiv:2305.05665.arxiv.org/abs/2305.05665
- [13]Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. 2024. Hallusionbench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. Preprint, arXiv:2310.14566.arxiv.org/abs/2310.14566
- [14]Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221.arxiv.org/abs/2207.05221
- [15]Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. 2019. Similarity of neural network representations revisited. Preprint, arXiv:1905.00414.arxiv.org/abs/1905.00414
- [16]Nikolaus Kriegeskorte, Marieke Mur, and Peter A Bandettini. 2008. Representational similarity analysis—connecting the branches of systems neuroscience. Frontiers in systems neuroscience, 2:249.DOI
- [17]Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. 2024. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13872–13882.arxiv.org/abs/2311.16922
- [18]Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. 2023. Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 conference on empirical methods in natural language processing, pages 292–305.arxiv.org/abs/2305.10355
- [19]Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. Advances in neural information processing systems, 36:34892–34916.
- [20]Xiaoyuan Liu, Wenxuan Wang, Youliang Yuan, Jen tse Huang, Qiuzhi Liu, Pinjia He, and Zhaopeng Tu. 2025. Insight over sight: Exploring the vision-knowledge conflicts in multimodal llms. Preprint, arXiv:2410.08145.arxiv.org/abs/2410.08145
- [21]Samuel Marks and Max Tegmark. 2023. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv preprint arXiv:2310.06824.arxiv.org/abs/2310.06824
- [22]Kevin Meng, David Bau, Alex J Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt. In Advances in neural information processing systems.arxiv.org/abs/2202.05262
- [23]Kovena Pal, Jiuding Sun, Andrew Yuan, Byron C Wallace, and David Bau. 2023. Future lens: Anticipating subsequent tokens from a single hidden state. In Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL), pages 548–560.DOI
- [24]Pouya Pezeshkpour, Moin Aminnaser, and Estevam Hruschka. 2025. Mixed signals: Decoding vlms’ reasoning and underlying bias in vision-language conflict. Preprint, arXiv:2504.08974.arxiv.org/abs/2504.08974
- [25]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. Preprint, arXiv:2103.00020.arxiv.org/abs/2103.00020
- [26]Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. 2020. Null it out: Guarding protected attributes by iterative nullspace projection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7237–7256.arxiv.org/abs/2004.07667
- [27]Zhenwei Tang, Difan Jiao, Blair Yang, and Ashton Anderson. 2025. Seam: Semantically equivalent across modalities benchmark for vision-language models. Preprint, arXiv:2508.18179.arxiv.org/abs/2508.18179
- [28]Gemma Team, Sherif El Abd, Vaibhav Aggarwal, Robin Algayres, Alek Andreev, Olivier Bachem, Ian Ballantyne, Cormac Brick, Victor Cărbune, Michelle Casbon, et al. 2026. Gemma 4 technical report. arXiv preprint arXiv:2607.02770.
- [29]Angela van Sprang, Laurens Samson, Ana Lucic, Erman Acar, Sennay Ghebreab, and Yuki M. Asano. 2026. Same content, different answers: Cross-modal inconsistency in mlms. Preprint, arXiv:2512.08923.arxiv.org/abs/2512.08923
- [30]Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegers, and Jacob Steinhardt. 2022. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. Preprint, arXiv:2211.00593.arxiv.org/abs/2211.00593