betaFeedback

WorksComputer Science

Layer Roles Matter: Mitigating Modality Imbalance in Multimodal Emotion Reasoning via Adaptive Layer Profiling

Abstract

Multimodal Emotion Reasoning (MER) requires perceiving and integrating emotion cues from multiple modalities. Recent advances increasingly leverage Multimodal Large Language Models (MLLMs) to enhance reasoning capacity. However, due to the inherent heterogeneity in conveying emotions across modalities, these models often suffer from modality imbalance and confront flawed emotion understanding. In this paper, we conduct an in-depth analysis on MER models and uncover two key empirical insights. (i) In MER, modality imbalance manifests two distinct failure modes: unimodal over-reliance and insufficient cross-modal utilization, which call for different intervention strategies. (ii) MLLM layers exhibit implicit functional hierarchies in multimodal emotion processing, playing separable roles in stabilizing unimodal representations and integrating cross-modal information. Motivated by these observations, we propose a Layer-Aware Modality Rebalancing (LAMR) framework by explicitly aligning intervention strategies with layer roles. Specifically, we first introduce two quantized metrics for adaptive layer role profiling. In each layer, Modality Entropy (ME) measures the dispersion of modality contributions, and Cross-modal Intensity (CMI) captures the strength of cross-modal interactions. Then, we respectively enhance modality equilibrium in stabilization layers and cross-modal propagation in integration layers. Extensive experiments on two comprehensive MER benchmarks demonstrate that LAMR effectively mitigates modality imbalance and improves emotion reasoning across diverse MLLM backbones, without additional training.

This work is licensed under a Creative Commons Attribution 4.0 International License. MIM ’26, Rio de Janeiro, Brazil © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2213-4/2026/11 https://doi.org/10.1145/3767308.3835292

CCS Concepts

• Computing methodologies → Activity recognition and understanding.

Keywords

Multimodal emotion reasoning, multimodal large language model, modality imbalance, attention rebalance, training-free

ACM Reference Format:

Pengyu Huang, Peipei Song, Beier Zhu, Zhiyuan Han, Jinpeng Hu, and Xun Yang. 2026. Layer Roles Matter: Mitigating Modality Imbalance in Multimodal Emotion Reasoning via Adaptive Layer Profiling. In Proceedings of the 34th ACM International Conference on Multimedia (MM ’26), November 10–14, 2026, Rio de Janeiro, Brazil. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3767308.3835292

Introduction

Multimodal Emotion Reasoning (MER) [4, 10, 13, 23, 37] is a challenging task that requires models not only to extract emotion-related cues from individual modalities, but also to perform deep reasoning over multimodal interactions to capture complex emotion expressions and contexts. With the emergence of Multimodal Large Language Models (MLLMs) [38], MER has shifted toward a unified generative reasoning paradigm. Recent approaches, such as Emotion-LLaMA [4], AffectGPT [20], and Emotion-Qwen [13], leverage MLLMs to jointly process and integrate multimodal data, achieving notable improvements in emotion recognition and reasoning.

In MER, different modalities are often complementary, providing additional cues for semantic and emotional disambiguation. However, due to inherent and complex differences in how modalities capture and express information, as well as the varying informational richness of each modality, the MER models inevitably face modality balance dilemma [28, 48]. This implies two ensuing issues, as highlighted in the statistical analysis of AffectGPT (Figure 1 (a)): unimodal over-reliance, where models tend to favor a dominant modality; and cross-modal under-utilization, where self-modal and cross-modal interactions are severely imbalanced. These imbalance issues ultimately result in flawed emotion understanding. As shown in Figure 1 (b), audio dominance induces incorrect recognition of urgency, while text-only reasoning causes context confusion (e.g., Monica asking for advice).

Figure 1. Diagnostic analysis of attention imbalance and layer-wise heterogeneity in MLLM-based emotion reasoning. (a) Attention statistics of AffectGPT. (b) An example of erroneous emotion reasoning given by AffectGPT. (c) Layer-wise self-modal and cross-modal attention heatmaps. (d) Layer-wise evolution of modality attention gaps and cross-modal attention.

Recently, some efforts have attempted to address the modality balance dilemma in multimodal emotion understanding. EMOE [7] proposes dynamic modality weighting to prevent over-reliance on dominant modalities during training. MoSEAR [11] employs modality-specific experts with a regularized gating mechanism in fine-tuning heads. MODA [48] further tackles attention deficit disorder through modular duplex attention. However, the additional fine-tuning costs in these methods limit their scalability and practicality in applications. Training-free methods such as visual augmentation [8] and contrastive decoding [16] have been explored to mitigate modality bias. Although promising, existing methods still exhibit crucial limitations. (1) they primarily focus on alleviating unimodal dominance, while overlooking the distinct challenge of the imbalance between self-modal and cross-modal interactions; (2) they typically adopt layer-agnostic strategies, which is suboptimal given the functional heterogeneity of multimodal processing across layers [9, 14, 49].

To address the above limitations, we delve into how MLLM-based MER models process multimodal tokens and how it affects the generation of modality imbalance. As shown in Figure 1 (c), the self- and cross-modal attention heatmaps across modalities reveal two coexisting failure modes of modality imbalance. Figure 1 (d) further shows a clear layer-wise dependency, such as layers with small modality gaps consistently exhibit weak cross-modal interactions. These observations underscore the need for rebalance strategies that explicitly account for the two-sided attention dilemma and layer-wise functional roles.

In this paper, we propose Layer-Aware Modality Rebalancing (LAMR), which specifies intervention strategies aligned with layer roles. First, we introduce an adaptive layer role profiling strategy that characterizes the evolution of multimodal attention across layers. Instead of a heuristic solution, we propose two metrics of Modality Entropy (ME) and Cross-Modal Intensity (CMI), which respectively quantify the balance of modality participation and the strength of cross-modal interactions at each layer. These metrics are combined to identify a functional bifurcation across layers and partition the MLLM into stabilization and integration layers, where the former primarily stabilizes modality-specific representations, and the latter facilitates cross-modal semantic integration. Second, we propose layer-wise attention re-balancing tailored to the identified layer roles. In stabilization layers, we enforce modality equilibrium to suppress abnormal unimodal dominance and stabilize attention flow. In integration layers, we control emotion propagation via a cross-modal affinity graph, facilitating effective diffusion of semantically relevant information across modalities while preserving the original representational structure. LAMR is fully plug-and-play and can be seamlessly integrated into existing MLLM-based MER models, scaling the emotion intelligence during the inference stage. Our contributions are summarized as follows:

  • We identify a two-sided attention dilemma in MLLM-based emotion reasoning—unimodal over-reliance and cross-modal under-utilization—and reveal its intrinsic connection to layer-wise functional heterogeneity.

  • We propose LAMR for MER, a novel inference-time framework that automatically profiles layer roles via two quantitative metrics and performs role-conditioned attention rebalancing.

  • Extensive experiments on different MER benchmarks and MLLM backbones demonstrate that LAMR effectively mitigates modality imbalance and improves emotion recognition and reasoning without any retraining.

Related Work

Multimodal Emotion Reasoning

Multimodal emotion reasoning requires joint reasoning over visual, linguistic, and acoustic cues. The recent emergence of MLLMs has advanced MER from early single-label emotion recognition [12, 36, 41, 47] to open-vocabulary and explainability-oriented frameworks [21, 23]. Building on generalist MLLMs [38], a line of emotion-specialized models has emerged through post-training adaptation, including Emotion-LLaMA [4], R1-Omni [51], AffectGPT [20], and Emotion-Qwen [13]. Beyond affective computing, modal joint learning has also been explored in unified autonomous driving systems such as MOJITO [5]. In parallel, numerous efforts have been made to establish standardized benchmarks for evaluating multimodal emotional intelligence. Recent benchmarks such as EmoBench [32], MER-UniBench [20], and MME-Emotion [46] provide holistic and rigorous analytical evaluation protocols and reveal the great potential in multimodal emotion reasoning.

Modality Imbalance

Modality imbalance has emerged as a critical challenge in multimodal learning. In the context of general MLLMs, several studies show that imbalanced modality utilization often induces hallucinations and unreliable reasoning [16, 40, 50]. For instance, AVCD [16] suppresses modality-induced hallucinations in audio-visual LLMs by contrastive decoding. In MER, this issue becomes more pronounced due to the subjective and context-dependent nature of emotional expressions [7, 8, 28]. Recent efforts such as MoSEAR [11] explicitly benchmark emotion conflicts across modalities and introduce expert-based mechanisms to alleviate modality bias in multimodal emotion reasoning. MODA [48] further proposes modular duplex attention that disentangles self-modal and cross-modal attention to improve multimodal perception and emotion understanding. Despite their effectiveness, existing approaches typically rely on additional fine-tuning and overlook layer-wise functional heterogeneity in MLLMs. In contrast, we aim to improve the emotional reasoning ability of MLLMs solely during inference by performing layer-wise rebalancing.

Layer-wise Interpretability of MLLMs

Layer-wise interpretability of MLLMs has attracted a great deal of attention in the community. DoLa [6] reveals that factual knowledge in LLMs is generally localized to particular transformer layers. Neo et al. [29] study the processing of visual tokens in the vision-language model and observe their increasing interpretability in the vocabulary space across layers. CMIF [49] further elucidates the cross-layer information flow between different modalities. Building on these analytical insights, LISA [9] leverages the functional roles of transformer layers to mitigate object hallucinations. Devils [14] identifies that middle layers disproportionately contribute to hallucination in Large Vision-Language Models (LVLMs) and adjusts visual attention accordingly. Yue et al. [42] argue that hallucination attention heads are concentrated in the middle and deeper layers, displaying a strong attention bias toward text tokens. They propose two methods that are targeted for hallucination heads to reduce their reliance on text tokens. Inspired by these findings, we focus on developing a layer-wise intervention strategy tailored to multimodal emotion reasoning.

Methodology

Preliminary

We formulate MER within the framework of MLLMs [4], where visual, acoustic, and textual inputs are jointly represented. Each modality is first encoded into the MLLM embedding space to form modality-specific token sequences Tm=Enc⁡m(xm)∈RNm×d\mathbf{T}_m = \operatorname{Enc}_m(x_m) \in\mathbb{R}^{N_m \times d}, where m∈{v,a,t}m \in\{v, a, t\}. The initial input is constructed by concatenation: X0=[Tt;Tv;Ta]∈RN×d\mathbf{X}_0 = [\mathbf{T}_t; \mathbf{T}_v; \mathbf{T}_a] \in\mathbb{R}^{N \times d}, where N=Nt+Nv+NaN = N_t + N_v + N_a denotes the total sequence length. The MLLM then processes X0\mathbf{X}_0 through a stack of Transformer layers. At the ℓ\ell-th layer, the input Xℓ−1\mathbf{X}_{\ell-1} is projected into queries, keys, and values: {Qℓ,Kℓ,Vℓ}=Xℓ−1WℓQ,K,V\{\mathbf{Q}_{\ell}, \mathbf{K}_{\ell}, \mathbf{V}_{\ell}\} = \mathbf{X}_{\ell-1}\mathbf{W}_{\ell}^{Q,K,V}. The attention matrix and layer output are defined as:

Aℓ=softmax⁡(QℓKℓTdh),Xℓ=AℓVℓ(1)\mathbf{A}_{\ell} = \operatorname{softmax}\left(\frac{\mathbf{Q}_{\ell}\mathbf{K}_{\ell}^{\mathsf{T}}}{\sqrt{d_h}}\right), \quad\mathbf{X}_{\ell} = \mathbf{A}_{\ell}\mathbf{V}_{\ell} \tag*{(1)}

where Aℓ∈RN×N\mathbf{A}_{\ell} \in\mathbb{R}^{N \times N} encodes both self-modal and cross-modal interactions, which play different roles in emotion reasoning. For elements i,ji, j with modality labels mi,mjm_i, m_j, entries with mi=mjm_i = m_j correspond to self-modal attention, which primarily consolidates intra-modal emotional cues (e.g., prosodic patterns in audio or facial dynamics in vision), contributing to modality-specific representation stabilization. In contrast, entries with mi≠mjm_i \ne m_j form cross-modal attention, which enables the exchange and alignment of complementary evidence across modalities, facilitating multimodal integration. In this work, we focus on analyzing and refining the attention matrix Aℓ\mathbf{A}_{\ell}, as it explicitly governs how multimodal evidence is weighted and propagated across tokens.

Multimodal Attention Diagnostics

Compared with general multimodal tasks, emotion reasoning fundamentally relies on the subtle interplay among modalities. For example, a speaker’s words may convey politeness, while their tone reveals frustration. This characteristic makes MER particularly susceptible to modality imbalance, yet it remains unclear how such imbalance emerges within MLLM-based MER models. To shed light on this issue, we investigate the layer-wise attention patterns in three representative MER models, including AffectGPT [20], PandaGPT [38], and EmotionLLaMA [4], using the CMU-MOSEI and CMU-MOSI datasets [1]. The attention matrix Aℓ\mathbf{A}_{\ell} provides a natural diagnostic lens to examine how multimodal emotional evidence is processed during inference.

Observation 1: Two kinds of balance dilemma. As shown in Figure 1 (c), our analysis reveals two co-existing but conceptually distinct imbalance phenomena. First, attention primarily concentrates on a dominant modality (such as audio), marginalizing complementary cues from other modalities. Second, even when multiple modalities are attended to, the majority of attention remains confined within self-modal interactions, while cross-modal attention remains sporadic. This observation suggests that unimodal over-reliance and cross-modal under-utilization are two orthogonal bottlenecks in multimodal emotion reasoning, each imposing distinct constraints on attention regulation.

Observation 2: Layered functional heterogeneity. Beyond global imbalance, we observe that these attention behaviors are not uniformly distributed across Transformer layers. As illustrated in Figure 1 (d), layers with small modality-wise attention disparities tend to exhibit weak cross-modal interactions, whereas layers with pronounced unimodal dominance often coincide with stronger cross-modal attention. This layer-wise divergence indicates that different Transformer layers are implicitly specialized for distinct functional roles during emotion inference. Specifically, layers with balanced but weak interactions primarily serve to stabilize modality-specific representations, whereas layers with pronounced cross-modal attention focus on integrating complementary emotional cues across modalities. This functional bifurcation suggests that Transformer layers can be naturally partitioned into two categories, each calling for a distinct intervention strategy. We will formalize this principle in the following sections.

Adaptive Layer Role Profiling

Motivated by the aforementioned findings, we propose an adaptive layer role profiling strategy to characterize the functional roles of Transformer layers in multimodal emotion reasoning. Instead of predefining layer roles heuristically, our approach includes two intrinsic metrics derived from attention patterns—Modality Entropy (ME) and Cross-Modal Intensity (CMI)—to identify whether a layer primarily serves to stabilize unimodal evidence or actively integrate cross-modal information.

Modality Entropy

ME is defined based on Shannon entropy [35] to quantify the balance of attention allocation across modalities. At the ℓ\ell-th layer, we obtain:

MEℓ=−∑mpℓ(m)log⁡(pℓ(m)+ϵ),pℓ(m)=1N∗∑i,jI{mj=m}⋅Aℓ[i,j],(2)\begin{aligned} \mathrm{ME}_{\ell} &= -\sum_m p_{\ell}(m)\log\left(p_{\ell}(m)+\epsilon\right), \\ p_{\ell}(m) &= \frac{1}{N^{*}}\sum_{i,j}\mathbb{I}\{m_j=m\}\cdot A_{\ell}[i,j], \tag*{(2)} \end{aligned}

where pℓ(m)p_{\ell}(m) represents the probability mass of attention allocated to modality mm at layer ℓ\ell. I{⋅}\mathbb{I}\{\cdot\} is the indicator function, m∈{v,a,t}m\in\{v,a,t\} denotes the modality of token, and N∗N^{*} is the normalization constant. Given that entropy is maximized when probability mass is evenly distributed and minimized when concentrated on a single component, ME provides a principled measure of unimodal dominance. A higher MEℓ\mathrm{ME}_{\ell} indicates that attention is more evenly distributed across modalities, whereas a lower value reflects strong reliance on a dominant modality.

Cross-modal Intensity

In AℓA_{\ell}, cross-modal attention enables the exchange and integration of complementary emotional cues between modalities, whereas self-modal attention refines relationships within each modality. Focusing on cross-modal flow, CMI quantifies cross-modal attention intensity at the ℓ\ell-th layer:

CMIℓ=1N∗∑i,jI{mi≠mj}⋅Aℓ[i,j].(3)\mathrm{CMI}_{\ell}=\frac{1}{N^{*}}\sum_{i,j}\mathbb{I}\{m_i\ne m_j\}\cdot A_{\ell}[i,j]. \tag*{(3)}

Since Aℓ[i,j]A_{\ell}[i,j] is normalized by applying softmax given both cross-modal and self-modal pairs, CMIℓ\mathrm{CMI}_{\ell} increases with an increase in cross-modal scores relative to the self-modal scores, suggesting that attention is actively propagated across modalities.

ME and CMI capture two complementary aspects of multimodal attention. However, there is a pitfall: a layer may exhibit high CMI by repeatedly propagating attention from a single modality to others; conversely, a layer with high ME may distribute attention evenly across modalities but lack meaningful cross-modal interactions. Effective multimodal reasoning requires both balanced modality engagement and active cross-modal integration. To this end, we compute the unified layer score SℓS_{\ell} as the harmonic mean [31] of ME and CMI. This formulation requires both metrics to be elevated for a high score:

Sℓ=2⋅MEℓ⋅CMIℓMEℓ+CMIℓ.(4)S_{\ell}=\frac{2\cdot\mathrm{ME}_{\ell}\cdot\mathrm{CMI}_{\ell}}{\mathrm{ME}_{\ell}+\mathrm{CMI}_{\ell}}. \tag*{(4)}

To adaptively identify the transition boundary ℓ∗\ell^{*}, we apply the Kneedle algorithm [33] to the set of scores SℓS_{\ell} sorted in descending order. This algorithm mathematically locates the point of maximum curvature, allowing us to pinpoint the precise layer where functional roles diverge without relying on rigid manual heuristics. Intuitively, a higher SℓS_{\ell} indicates that the layer concurrently supports active cross-modal interaction and balanced modality participation, making it suitable for semantic integration, whereas other layers primarily function to stabilize modality-specific representations. Therefore, we designate layers with Sℓ≥Sℓ∗S_\ell\ge S_\ell^* as integration layers, and layers with lower SℓS_\ell as stabilization layers.

Layer-wise Attention Rebalance

In accordance with the distinct functional roles of stabilization and integration layers, we design a layer-specific rebalancing strategy that addresses unimodal over-utilization in stabilization layers and cross-modal under-utilization in integration layers, respectively.

Stabilization Layer: Modality Anchoring Equilibrium.

To establish a reliable multimodal evidence basis, we aim to anchor attention distributions across modalities into a balanced regime. Rather than relying on absolute attention magnitudes, we construct a layer-wise reference to characterize typical attention intensity. Given AℓA_\ell, where each row represents a token-wise attention distribution, we compute a local anchor by average pooling along the row dimension A~ℓ=AvgPool⁡(Aℓ)\widetilde{A}_\ell= \operatorname{AvgPool}(A_\ell). This anchor A~ℓ\widetilde{A}_\ell captures the expected attention scale of layer ℓ\ell and provides a modality-agnostic anchor against token-level fluctuations.

Importantly, unimodal over-reliance manifests not as uniformly elevated attention, but as extreme deviations concentrated on a subset of tokens or modalities. To distinguish pathological dominance from legitimate salient cues, we leverage the Interquartile Range (IQR) method [34, 39], a statistical technique known for its robustness in outlier detection. Specifically, we define an equilibrium interval [L,U][L,U] that delineates the normal range of attention values here:

L=Q1−1.5×IQR,U=Q3+1.5×IQR,(5)L = Q_1 - 1.5 \times IQR,\qquad U = Q_3 + 1.5 \times IQR, \tag*{(5)}

where Q1Q_1 and Q3Q_3 denote the first and third quartiles of AℓA_\ell, and IQR=Q3−Q1IQR = Q_3 - Q_1. Attention values outside [L,U][L,U] are considered indicative of abnormal confidence concentration.

Finally, outlier attentions are shrunk toward the local anchor A~ℓ\widetilde{A}_\ell:

Aℓ′={(1−αs)Aℓ±αsA~ℓ,Aℓ∉[L,U],Aℓ,otherwise.(6)A'_\ell= \begin{cases} (1-\alpha_s)A_\ell\pm\alpha_s\widetilde{A}_\ell, & A_\ell\notin[L,U],\\ A_\ell, & \text{otherwise}. \end{cases} \tag*{(6)}

where αs\alpha_s controls the anchoring strength. Instead of hard clipping, this soft regularization contracts attention variance while preserving relative structure, thereby mitigating unimodal dominance without enforcing uniformity.

Integration Layer: Intermodal Diffusive Propagation.

Emotion reasoning requires that semantically aligned cues from different modalities reinforce each other. This motivates a content-driven substrate to model latent semantic correlations for information diffusion. As shown in Figure 2, we first construct a semantic affinity graph based on the key representations:

Figure 2. Overview of the proposed LAMR framework. Adaptive Layer Role Profiling automatically partitions Transformer layers into stabilization layers and integration layers. Layer-wise Attention Rebalance performs layer-specific attention interventions during inference, mitigating unimodal over-reliance in stabilization layers and enhancing cross-modal propagation in integration layers.

Gℓ=KℓKℓT∈RN×N.(7)G_\ell= K_\ell K_\ell^{\mathsf{T}} \in\mathbb{R}^{N \times N}. \tag*{(7)}

Unlike raw attention scores, the key-based affinity captures symmetric and current query-independent associations. However, GℓG_\ell typically reflects strong self-token similarity. To enable cross-modal information transfer, we perform row-wise normalization on GℓG_\ell, which redistributes the self-affinity toward semantically related cross-modal tokens and yields a transition matrix Gnorm,ℓG_{\mathrm{norm},\ell}.

Then, we define a diffusive propagation operator that corresponds to a lazy random walk on Gnorm,ℓG_{\mathrm{norm},\ell}:

Dℓ=(1−αd)I+αdGnorm,ℓ.(8)D_\ell= (1-\alpha_d)I + \alpha_d G_{\mathrm{norm},\ell}. \tag*{(8)}

where αd\alpha_d controls the diffusion strength, and II is the identity matrix. Since DℓD_\ell is row-stochastic, it preserves the row-wise normalization of attention. The biased attention AℓA_\ell is thus refined by applying the diffusion operator Aℓ′=DℓAℓA'_\ell= D_\ell A_\ell. For each query token ii, it yields:

Aℓ′(i,:)=(1−αd)Aℓ(i,:)+αd∑jGnorm,ℓ(i,j)Aℓ(j,:).(9)A'_\ell(i,:) = (1-\alpha_d)A_\ell(i,:) + \alpha_d \sum_j G_{\mathrm{norm},\ell}(i,j)A_\ell(j,:). \tag*{(9)}

That is, each query keeps a fraction (1−αd)(1-\alpha_d) of its original attention while absorbing an αd\alpha_d-weighted mixture of attention patterns from semantically related cross-modal neighbors, thereby alleviating cross-modal under-utilization in emotion reasoning.

Experiments

Experimental Setting

Datasets and Metrics.

We evaluate our method on two multimodal emotion benchmarks: MER-UniBench [20] and MME-Emotion [46]. MER-UniBench is a comprehensive evaluation benchmark designed to cover typical multimodal emotion recognition tasks, including basic emotion recognition, sentiment analysis, and fine-grained emotion recognition. For the three tasks, we select nine datasets: MER2023 [22], MER2024 [24], MELD [30], EMOCAP [2], CMU-MOSI [45], CMU-MOSEI [1], CH-SIMS [44], CH-SIMS v2 [26], and OV-MERD+ [21]. We adopt the official evaluation metrics provided by MER-UniBench, tailored for the free-form natural language output style of MLLMs. MME-Emotion is the largest emotional intelligence benchmark for MLLMs, which assesses both emotional understanding and reasoning capabilities. It contains 6,500 video clips with task-specific question-answering (QA) pairs, spanning 27 distinct scenario types. In this work, we report results on six emotional tasks, including emotion recognition in the lab (ER-Lab), emotion recognition under noise (Noise-ER), multi-label emotion recognition (ML-ER), sentiment analysis (SA), fine-grained sentiment analysis (FG-SA), and intent recognition (IR). Evaluation metrics include recognition (Rec) score, reasoning (Rea) score, and Chain-of-Thought (CoT) score, which are computed via a multi-agent system framework.

Implementation Details.

For layer profiling, we construct a small validation set Dval\mathcal{D}_{\mathrm{val}} by combining the CMU-MOSEI and CMU-MOSI datasets [1], which are widely used benchmarks for multimodal sentiment analysis. This validation set is used to estimate layer-wise attention statistics and determine the layer threshold Sℓ∗S_{\ell^*}. We apply our method to three representative MER baselines: PandaGPT [38], Emotion-LLaMA [4], and AffectGPT [20]. The proposed attention rebalance strategy is performed only during inference on the test set Dtest\mathcal{D}_{\mathrm{test}}. For the two types of layers identified by our method, the rebalance hyperparameters (αs,αd)(\alpha_s,\alpha_d) are set to (0.1,0.1)(0.1,0.1) for AffectGPT, and (0.2,0.2)(0.2,0.2) for the other baselines, respectively. For LLM-based evaluation on MME-Emotion, we employ gpt-4.1-2025-04-14 and gpt-4o-2024-11-20 as judge models.

Performance Comparison

Emotion Recognition

For a comprehensive comparison, we include representative generalist MLLMs (e.g., Video-LLaVA [25]) and specialized MER models (e.g., AVERE [3]), and apply our method to three baseline architectures. As shown in Table 1, applying LAMR to PandaGPT, Emotion-LLaMA, and AffectGPT results in consistent performance improvements across all nine datasets, with average gains of 1.71%, 2.76%, and 1.68%, respectively. This indicates that our inference-time rebalancing strategy is robust and model-agnostic. Notably, these gains are observed not only on basic emotion recognition and sentiment analysis, but also on fine-grained emotion recognition tasks (OV-MERD+ dataset), which require more delicate cross-modal evidence aggregation. LAMR improves the multimodal reasoning process by alleviating unimodal dominance and strengthening cross-modal integration.

MethodsMER23↑MER24↑MELD↑IEMO↑MOSI↑MOSEI↑SIMS↑SIMS v2↑OV-MERD+↑Mean↑
Video-LLaVA [25]36.9330.2530.7338.9556.3761.6453.2857.4534.0044.40
Video-ChatGPT [27]44.8646.8037.3356.8354.4263.1264.8265.8039.8052.64
VideoChat2 [18]33.6754.5036.6448.7066.8454.3269.4970.6639.2152.67
LLaMA-VID [19]50.7257.6042.7546.0261.7863.8969.3567.4845.0156.07
VideoChat [17]48.7357.3041.1148.3865.1363.6169.5272.1444.5256.72
Chat-UniVi [15]57.6265.6745.6152.3754.5363.1868.1566.3648.0057.94
mPLUG-Owl [43]56.8659.8949.1155.5472.4072.9172.1375.0048.1862.45
AVERE [3]56.8768.7838.4244.9249.8942.1476.3676.1648.3955.77
PandaGPT [38]44.9649.6338.3046.4267.6569.1267.1264.6037.9253.97
+Ours48.3550.6139.5449.3368.7069.8369.3566.3539.0555.68
Emotion-LLaMA [4]64.9166.1463.4647.3063.6071.8367.6772.8846.8062.73
+Ours68.7169.4366.5348.5665.5775.3771.7975.6247.8465.49
AffectGPT [20]73.6073.2852.1655.5083.0176.6986.4181.9159.4471.33
+Ours76.3573.6253.7856.1785.1377.0588.4585.5461.0473.01

Table 1. Comparison on MER-UniBench [20] for multimodal emotion recognition. This benchmark comprises nine datasets covering three tasks: basic emotion recognition, sentiment analysis, and fine-grained emotion recognition. “Mean” represents the average performance across all datasets. We compare various mainstream MLLMs and apply our method to three baselines.

Emotional Intelligence

Beyond recognition accuracy, we evaluate LAMR on the MME-Emotion benchmark for holistic emotional intelligence. As reported in Table 2, LAMR consistently improves performance across six emotional tasks, with clear gains on reasoning-related metrics. For example, for the FG-SA task, AffectGPT’s Rec score increases from 15.6 to 18.6, while CoT score rises from 39.6 to 43.5. LAMR enhances not only the final predictions but also the coherence and logical structure of the reasoning process. These improvements are attributed to LAMR operating at the level of internal multimodal evidence aggregation, rather than merely adjusting output-level predictions.

MethodsER-Lab RecER-Lab ReaER-Lab CoTFG-SA RecFG-SA ReaFG-SA CoTIR RecIR ReaIR CoTML-ER RecML-ER ReaML-ER CoTNoise-ER RecNoise-ER ReaNoise-ER CoTSA RecSA ReaSA CoTMean RecMean ReaMean CoT
PandaGPT [38]15.542.630.634.847.841.313.848.531.22.037.821.421.240.931.335.251.043.120.444.833.2
+Ours19.745.631.137.049.443.215.848.832.35.042.022.022.041.531.441.254.547.923.547.034.7
Emotion-LLaMA [4]11.732.822.326.854.441.011.045.128.08.346.927.644.060.352.436.261.548.923.050.236.7
+Ours14.837.526.229.055.241.713.847.330.68.749.929.345.660.852.936.563.650.124.752.438.5
AffectGPT [20]16.254.235.215.663.539.63.050.426.70.649.925.628.865.347.025.567.646.715.058.536.8
+Ours19.359.639.518.668.443.55.254.529.90.754.627.631.867.449.625.570.748.116.962.539.7

Table 2. Comparison on MME-Emotion [46] for emotional understanding and reasoning. Results cover six emotional tasks and three evaluation metrics, all computed via a multi-agent system framework.

Ablation Studies

Impact of Adaptive Layer Role Profiling

To validate the necessity of adaptive layer role profiling, we compare LAMR with two alternative strategies: Random Assignment, which randomly selects 30%, 50%, or 70% of the layers as integration layers, and Static Assignment, which heuristically partitions layers into shallow, middle, and deep stages and applies integration to one or two fixed stages. As shown in Table 3, both strategies reduce the baseline performance. In particular, random assignment leads to the most severe performance degradation, as it blindly applies inappropriate interventions, disrupting the internal organization of multimodal processing. In contrast, our adaptive profiling strategy yields the best results, demonstrating that accurate identification of layer-specific functions is critical for effective multimodal attention rebalancing.

StrategyMELDIEMOCAPSIMSMean
AffectGPT52.1655.5086.4164.69
Random Assignment
30% Integration53.9951.5380.4461.99
50% Integration49.7147.5180.7959.34
70% Integration47.1546.2880.6858.04
Static Assignment
Shallow51.4551.5182.8861.95
Middle53.5252.4483.1563.04
Deep52.9651.8681.0461.95
Shallow + Middle52.5651.5383.7762.62
Middle + Deep52.7052.2380.6061.84
Shallow + Deep51.2950.9582.1661.47
Ours53.7856.1788.4566.13

Table 3. Ablation study on layer partition strategies. “Shallow”, “Middle”, and “Deep” denote static heuristic partitions that evenly divide the layers into three contiguous stages. We designate one or two of these stages as integration layers.

Impact of CMI and ME

Our layer role profiling relies on a unified layer score SℓS_{\ell} that jointly combines CMI and ME. To assess the contribution of each metric, we compare LAMR with two ablated variants that rely on a single metric. As shown in Table 4, both single-metric strategies lead to suboptimal performance, with the mean score dropping from 66.13 to 63.18 (only CMI) and 63.42 (only ME), respectively. These results validate our design motivation that only layers exhibiting both high cross-modal intensity and high modality entropy should be identified as integration layers for cross-modal semantic aggregation.

CMIMEMELDIEMOCAPSIMSMean
✓✗52.2553.6283.6763.18
✗✓51.6053.1885.4763.42
✓✓53.7856.1788.4566.13

Table 4. Ablation study on the quantitative metrics used for adaptive layer role profiling, i.e., CMI and ME.

Impact of Layer-wise Attention Rebalance

To examine whether the effectiveness of LAMR stems from matching intervention types to layer-specific functions, we test three variants: Uniform Propagation, which applies diffusive propagation to all layers; Uniform Equilibrium, which applies modality equilibrium to all layers; and Reversed Intervention, which swaps the two strategies between stabilization and integration layers. As shown in Table 5, all three variants lead to significant performance degradation. These results demonstrate that effective attention rebalancing requires alignment with the intrinsic functional role of each layer, thereby confirming the necessity of our layer-wise intervention design.

StrategyMELDIEMOCAPSIMSMean
AffectGPT52.1655.5086.4164.69
Uniform Propagation47.7551.2381.5360.17
Uniform Equilibrium53.5351.1183.9262.85
Reversed Intervention53.2851.4484.2662.99
Ours53.7856.1788.4566.13

Table 5. Ablation study on layer-wise attention rebalance. “Reversed Intervention” denotes exchanging intervention strategies for two types of layers.

Further Analysis

Robustness Under Emotion Conflicts

CA-MER [11] is a benchmark designed to examine MLLMs under realistic emotion conflicts. It constitutes a challenging setting for MER, where only one or all modalities reflect the true emotion. As shown in Table 6, AffectGPT performs worse on the Video-aligned subset compared to the Audio-aligned subset, indicating a clear bias toward dominant modalities. By re-balancing attention at inference time, LAMR effectively mitigates modality dominance and improves performance across all subsets, achieving a 1.9% gain in average score.

ModelVideo-aligned AccVideo-aligned RecallVideo-aligned AvgAudio-aligned AccAudio-aligned RecallAudio-aligned AvgConsistent AccConsistent RecallConsistent Avg
AffectGPT44.9250.8047.8652.2655.5453.9051.8656.9054.38
+ Ours45.8952.6649.2852.5660.0056.2852.7259.8456.28

Table 6. Performance comparison on the CA-MER benchmark [11] under realistic emotion conflicts. The three subsets—video-aligned, audio-aligned, and consistent—represent settings in which the video modality, the audio modality, or all modalities reflect the true emotion, respectively.

Comparison with Training-Free Rebalancing Methods

To evaluate the efficacy of existing training-free rebalancing approaches for MER, we re-implemented three representative methods, Devils [14], AVCD [16], and LISA [9], on the same backbone following their original intervention principles for fair comparison. These methods encompass diverse rebalancing paradigms: AVCD performs contrastive decoding, Devils focuses on modality token-level attention enhancement, and LISA implements a layer-wise integration and suppression approach based on predefined layer functions. As shown in Table 7, these methods often yield unstable performance when transferred to MER: while moderate gains are observed on some datasets, performance degrades on others. This suggests that generic interventions, which rely on fixed layer priors or non-adaptive modality adjustments, fail to account for the fine-grained functional roles of layers in multimodal emotion reasoning. In contrast, LAMR achieves consistent improvements by explicitly grounding its observation and intervention in MER.

MethodsMER23MER24MELDIEMOSIMSSIMSv2OV.
AffectGPT73.6073.2852.1655.5086.4181.9159.44
+ Devils [14]75.2770.1352.3449.4079.9381.9559.80
+ AVCD [16]75.3575.0553.5651.8380.9782.4360.69
+ LISA [9]73.1475.7952.8855.5386.4385.4160.33
+ Ours76.3573.6253.7856.1788.4585.5461.04

Table 7. Comparison with other training-free rebalancing methods on MER-UniBench.

Scalability Across Model Sizes

In Table 8, we evaluate LAMR on AffectGPT backbones of increasing capacity (1.5B, 7B, and 14B parameters). LAMR consistently yields performance gains across all model scales, indicating that modality imbalance persists even in large-capacity MLLMs and cannot be resolved solely by scaling up model size. More importantly, LAMR is scalable and benefits both lightweight and large-scale backbones.

Model SetupMER24MELDIEMOCAPAvg. Δ
AffectGPT-1.5B75.3949.3452.32–
+ Ours76.6849.9752.840.81↑
AffectGPT-7B73.2852.1655.50–
+ Ours73.6253.7856.170.88↑
AffectGPT-14B77.3655.0160.80–
+ Ours80.0255.5765.442.62↑

Table 8. Scalability analysis across different model sizes (1.5B, 7B, and 14B).

4.4.4 Results of Attention Rebalance. Figure 3 provides a quantitative view of how LAMR corrects both facets of modality imbalance. As shown in Figure 3 (a), LAMR significantly balances the attention allocated to three modalities. In AffectGPT, attention to the non-dominant text and video modalities increases by +14.9% and +3.9%, respectively, while attention to the dominant audio modality decreases by −3.6%. Figure 3 (b) shows an increase in cross-modal attention relative to self-modal attention. In particular, AffectGPT exhibits a +58.8% gain in audio-to-video attention, and EmotionLLaMA shows a +76.5% increase in video-to-text interaction, indicating that LAMR actively promotes inter-modal information propagation. Overall, LAMR corrects both unimodal over-reliance and cross-modal under-utilization, directly addressing the two-sided attention dilemma identified in our analysis.

Effect of LAMR on modality-level and cross-modal attention rebalancing

Figure 3. Effect of LAMR on modality-level and cross-modal attention rebalancing. (a/b) Modality-wise and cross-modal attention allocation with LAMR. Percentages indicate changes relative to the baseline.

Qualitative Results

As shown in Figure 4, we examine two challenging cases that reflect distinct forms of modality imbalance in multimodal emotion reasoning. In the first case of unimodal over-reliance, the baseline overemphasizes acoustic cues (e.g., high pitch) and erroneously predicts excitement. In the second case of cross-modal under-utilization, the baseline fails to reconcile positive visual/audio signals (i.e., visual smile/enthusiastic tone) with hostile text, resulting in a superficial interpretation of playful. In contrast, LAMR achieves more nuanced emotional understanding by suppressing excessive modality reliance and enhancing cross-modal interaction.

Figure 4. Two challenging examples of modality imbalance. Cyan, green, and red fonts indicate shared correct, correct, and erroneous interpretations, respectively.

Conclusion

In this work, we aim to address the modality imbalance problem in multimodal emotion reasoning, manifested as unimodal over-reliance and cross-modal under-utilization. We propose Layer-Aware Modality Rebalancing (LAMR), which rebalances multimodal attention through adaptive layer role profiling and role-conditioned attention reallocation during inference. Extensive experiments on multiple MER benchmarks demonstrate that LAMR effectively mitigates modality imbalance and consistently improves emotion reasoning performance across diverse MLLM backbones without requiring any additional training.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (62402471, U22A2094), the Yangtze River Delta Science and Technology Innovation Community Joint Research (Basic Research) Project (2025CSJZN01600), and the National Science and Technology Major Project (2025ZD0215301).

References

References

  1. [1]AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018. Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph. In ACL. 2236–2246.DOI
  2. [2]Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. IEMOCAP: Interactive emotional dyadic motion capture database. Language Resources and Evaluation 42, 4 (2008), 335–359.DOI
  3. [3]Ashutosh Chaubey, Jiacheng Pang, Maksim Siniukov, and Mohammad Soleymani. 2026. AVERE: Improving Audiovisual Emotion Reasoning with Preference Optimization. In The Fourteenth International Conference on Learning Representations.arxiv.org/abs/2602.07054
  4. [4]Zegang Cheng, Zhi-Qi Cheng, Jun-Yan He, Jingdong Sun, Kai Wang, Yuxi ang Lin, Zheng Lian, Xiaojing Peng, and Alexander G. Hauptmann. 2024. Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning. In NeurIPS, Vol. 37. Curran Associates, Inc., 110805–110853. doi:10.5220/079017-3518
  5. [5]Zhijing Cheng, Xuancheng Zhang, Donglin Di, Lei Fan, Baorui Ma, Hao Li, and Xun Yang. 2026. MOJITO: Modal Joint Learning for Unified End-to-End Autonomous Driving. arXiv:2607.23511 [cs.CV] https://arxiv.org/abs/2607.23511
  6. [6]Yung-Sung Chuang, YuJia Xie, Hongyin Luo, Yoon Kim, James R. Glass, and Pengcheng He. 2024. DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models. In The Twelfth International Conference on Learning Representations.
  7. [7]Yiyang Fang, Wenke Huang, Guancheng Wan, Kehua Su, and Mang Ye. 2025. EMOE: Modality-Specific Enhanced Dynamic Emotion Experts. In CVPR. 14314–14324.DOI
  8. [8]Yiyang Fang, Jian Liang, Wenke Huang, He Li, Kehua Su, and Mang Ye. 2025. Catch Your Emotion: Sharpening Emotion Perception in Multimodal Large Language Models. In ICML, Vol. 267. PMLR, 16029–16039.
  9. [9]Zhihui Guo, Xin Man, Hui Xu, Jie Shao, Zhiguo Jiang, Xianchao Zhang, and Heng Tao Shen. 2025. LISA: A Layer-wise Integration and Suppression Approach for Hallucination Mitigation in Multimodal Large Language Models. arXiv preprint arXiv:2507.19110 (2025).arxiv.org/abs/2507.19110
  10. [10]Zhiyuan Han, Beier Zhu, Yanlong Xu, Peipei Song, and Xun Yang. 2025. Bench-marking and Bridging Emotion Conflicts for Multimodal Emotion Reasoning. In ACM MM. 5528–5537.arxiv.org/abs/2606.25325
  11. [11]Zhiyuan Han, Beier Zhu, Yanlong Xu, Peipei Song, and Xun Yang. 2025. Bench-marking and Bridging Emotion Conflicts for Multimodal Emotion Reasoning. In ACM MM. 5528–5537.
  12. [12]Jinpeng Hu, Hongchang Shi, Chongyuan Dai, Zhuo Li, Peipei Song, and Meng Wang. 2025. Beyond Emotion Recognition: A Multi-Turn Multimodal Emotion Understanding and Reasoning Benchmark. In ACM MM. Association for Computing Machinery, 5814–5823. doi:10.1145/3746027.3755726
  13. [13]Dawei Huang, Qing Li, Chuan Yan, Zebabg Cheng, Yurong Huang, Xiang Li, Bin Li, Xiaohui Wang, Zheng Lian, and Xiaojian Peng. 2025. Emotion-Qwen: Training Hybrid Experts for Unified Emotion and General Vision-Language Understanding. arXiv preprint arXiv:2505.06685 (2025).
  14. [14]Zhangqi Jiang, Junkai Chen, Beier Zhu, Tingjin Luo, Yankun Shen, and Xu Yang. 2025. Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens. In CVPR. 25004–25014.arxiv.org/abs/2411.16724
  15. [15]Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. 2024. Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding. In CVPR. 13700–13710.arxiv.org/abs/2311.08046
  16. [16]Chaeyoung Jung, Youngjoon Jang, and Joon Son Chung. 2025. AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decoding. In NeurIPS, Vol. 38. Curran Associates, Inc., 63143–63174.arxiv.org/abs/2505.20862
  17. [17]KunChang Li, Yinah He, Yi Wang, Yizhuo Li, Weihai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. 2025. Videochat: Chat-centric video understanding. Science China Information Sciences 68, 10 (2025), 200102.arxiv.org/abs/2305.06355
  18. [18]Kunchang Li, Yali Wang, Yinah He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Lou, Limin Wang, and Yu Qiao. 2024. MVBench: A Comprehensive Multi-modal Video Understanding Benchmark. In CVPR. 22195–22206.arxiv.org/abs/2311.17005
  19. [19]Weiyun Li, Chengyao Wang, and Jiaya Jia. 2025. LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models. In ECCV. 323–340.arxiv.org/abs/2311.17043
  20. [20]Zheng Lian, Haoyou Chen, Chen Han, Haiyang Sun, Licai Sun, Yong Ren, Zegang Cheng, Bin Liu, Rui Liu, Xiaojian Peng, Jiangyan Yi, and Jianhua Tao. 2025. AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Models. In ICML, Vol. 267. PMLR, 36993–37014.
  21. [21]Zheng Lian, Haiyang Sun, Licai Sun, Haoyu Chen, Hao Gu, Zhuofan Wen, Shun Chen, Zhang Siyuan, Hailiang Yao, Bin Liu, Rui Liu, Shan Liang, Ya Li, Jiangyan Yi, and Jianhua Tao. 2025. OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition. In ICML, Vol. 267. PMLR, 37015–37050.
  22. [22]Zheng Lian, Haiyang Sun, Licai Sun, Kaichen Chen, Mingxuan Xu, Kevin Wang, Ke Xu, Yu He, Ying Li, Jinming Zhao, Ye Liu, Bin Liu, Jiangyan Yi, Meng Wang, Erik Cambria, Guoying Zhao, Björn W. Schuller, and Jianhua Tao. 2023. MER 2023: Multi-modal Learning, Modality Robustness, and Semi-Supervised Learning. In ACM MM. 9610–9614.arxiv.org/abs/2304.08981
  23. [23]Cheng Lian, Haiyang Sun, Licai Sun, Hao Gu, Zhuofan Wen, Siyuan Zhang, Shun Chen, Mingyu Xu, Ke Xu, Kang Chen, et al. 2023. Explainable multimodal emotion recognition. arXiv preprint arXiv:2306.15401 (2023).arxiv.org/abs/2306.15401
  24. [24]Zheng Lian, Haiyang Sun, Licai Sun, Zhuofan Wen, Siyuan Zhang, Shun Chen, Hao Gu, Jinming Zhao, Ziyang Ma, Xie Chen, et al. 2024. MER 2024: Semi-Supervised Learning, Noise Robustness, and Open-Vocabulary Multimodal Emotion Recognition. In MRAC. 41–48.arxiv.org/abs/2404.17113
  25. [25]Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. 2024. Video-LLaVA: Learning United Visual Representation by Alignment Before Projection. In EMNLP. 5971–5984.arxiv.org/abs/2311.10122
  26. [26]Yihe Liu, Ziqi Yuan, Huisheng Mao, Zhiyun Liang, Wanquiyue Yang, Yuanzhe Qiu, Tie Cheng, Xiaoteng Li, Hua Xu, and Kai Gao. 2022. Make Acoustic and Visual Cues Matter: CH-SIMS v2.0 Dataset and AV-Mixup Consistent Module. In ICML. ACM, 247–258. doi:10.1145/3536221.3556630DOI
  27. [27]Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. 2024. Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models. In ACL. 12585–12602.
  28. [28]Ankan Mullick, Saransh Sharma, Abhik Jana, and Pawan Goyal. 2025. Text Takes Over: A Study of Modality Bias in Multimodal Intent Detection. In EMNLP. 24208–24058.
  29. [29]Clement Neo, Luke Ong, Philip Torr, Mor Geva, David Krueger, and Fazl Barez. 2025. Towards Interpreting Visual Information Processing in Vision-Language Models. In The Thirteenth International Conference on Learning Representations.arxiv.org/abs/2410.07149
  30. [30]Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2019. MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations. In ACL. 527–536.arxiv.org/abs/1810.02508
  31. [31]Ronald Rousseau, Leo Egghe, and Raf Guns. 2018. Chapter 4 - Statistics. In Becoming Metric-Wise. Chandos Publishing, 67–97.
  32. [32]Sahand Sabour, Siyang Liu, Zheyuan Zhang, June Liu, Jinfeng Zhou, Alvionna Sunaryo, Talia Lee, Rada Mihalcea, and Minlie Huang. 2024. EmoBench: Evaluating the Emotional Intelligence of Large Language Models. In ACL. 5986–6004.
  33. [33]Ville Satopaa, Jeannie Albrecht, David Irwin, and Barath Raghavan. 2011. Finding a "Kneedle" in a Haystack: Detecting Knee Points in System Behavior. In ICDCS. 166–171.
  34. [34]Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. 2025. LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models. In ICCV. 22857–22867.arxiv.org/abs/2403.15388
  35. [35]C. E. Shannon. 1948. A mathematical theory of communication. The Bell System Technical Journal 27, 3 (1948), 379–423.
  36. [36]Peipei Song, Dan Guo, Xun Yang, Shengeng Tang, and Meng Wang. 2024. Emotional Video Captioning With Vision-Based Emotion Interpretation Network. IEEE Transactions on Image Processing 33 (2024), 1122–1135. doi:10.1109/TIP.2024.3359045DOI
  37. [37]Peipei Song, Zhiyan Zhang, Weidong Chen, Jinping Hu, Xun Yang, and Xiaojun Chang. 2026. Bridging Subjectivity in Affective Explanation Captioning via Consensus-Prompted Emotion Reasoning. IEEE Transactions on Image Processing 35 (2026), 6760–6774. doi:10.1109/TIP.2026.3703790DOI
  38. [38]Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. 2023. PandaGPT: One Model To Instruction-Follow Them All. In Proceedings of the 1st Workshop on Taming Large Language Models: Controllability in the era of Interactive Assistants!, Devamanyu Hazarika, Xiangru Robert Tang, and Di Jin (Eds.). Association for Computational Linguistics, Prague, Czech Republic, 11–23.
  39. [39]H. P. Vinutha, B. Poornima, and B. M. Sagar. 2018. Detection of Collusions Using Interquartile Range Technique from Intrusion Dataset. In Information and Decision Sciences. 511–518.DOI
  40. [40]Xinhao Xu, Hui Chen, Mengyao Lyu, Sicheng Zhao, Yizhe Xiong, Zijia Lin, Jungoh Han, and Guiguang Ding. 2025. Mitigating Hallucinations in Multi-modal Large Language Models via Image Token Attention-Guided Decoding. In NAACL. 1571–1590.
  41. [41]Qu Yang, Mang Ye, and Bo Du. 2024. EmoLLM: Multimodal emotional understanding meets large language models. arXiv preprint arXiv:2406.16442 (2024).arxiv.org/abs/2406.16442
  42. [42]Tianyun Yang, Ziniu Li, Juan Cao, and Chang Xu. 2025. Understanding and mitigating hallucination in large vision-language models via modular attribution and intervention. In The Thirteenth International Conference on Learning Representations.
  43. [43]Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. 2024. mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration. In CVPR. 13040–13051.arxiv.org/abs/2311.04257
  44. [44]Wenmeng Yu, Hua Xu, Fanyang Meng, Yilin Zhu, Yixiao Ma, Jiele Wu, Jiyun Zhou, and Kaicheng Yang. 2020. CH-SIMS: A Chinese Multimodal Sentiment Analysis Dataset with Fine-grained Annotation of Modality. In ACL. 3718–3727.DOI
  45. [45]Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. 2016. Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv preprint arXiv:1606.06259 (2016).arxiv.org/abs/1606.06259
  46. [46]Fan Zhang, Zegang Cheng, Chong Deng, Haoxuan Li, Zheng Lian, Qian Chen, Huadai Liu, Wen Wang, Yi-Fan Zhang, Renrui Zhang, Ziyu Guo, Zhihong Zhu, Hao Wu, Haixin Wang, Yefeng Zheng, Xiaojiang Peng, Xian Wu, Kun Wang, Xiangyang Li, Jieping Ye, and Pheng-Ann Heng. 2026. MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Models. In The Fourteenth International Conference on Learning Representations.
  47. [47]Zhiyan Zhang, Peipei Song, Jingpeng Hu, Jingyang Jia, Xun Yang, and Xiaojun Chang. 2026. Benchmarking Dynamic Affective Reasoning: A Viewer-Centric Video Emotion Dataset. arXiv preprint arXiv:2607.10238 (2026).arxiv.org/abs/2607.10238
  48. [48]Zhicheng Zhang, Wuyou Xia, Chenxi Zhao, Zhou Yan, Xiaoqiang Liu, Yongjie Zhu, Wenyu Qin, Pengfei Wan, Di Zhang, and Jufeng Yang. 2025. MODA: Modular Duplex Attention for Multimodal Perception, Cognition, and Emotion Understanding. In ICML, Vol. 267. PMLR, 76385–76396.arxiv.org/abs/2507.04635
  49. [49]Zhi Zhang, Shrishti Yadav, Fengze Han, and Ekaterina Shutova. 2025. Cross-modal Information Flow in Multimodal Large Language Models. In CVPR. 19781–19791.arxiv.org/abs/2411.18620
  50. [50]Zongmeng Zhang, Wengang Zhou, Jie Zhao, and Houqiang Li. 2025. Robust Multimodal Large Language Models Against Modality Conflict. In ICML, Vol. 267. PMLR, 77233–77253.arxiv.org/abs/2507.07151
  51. [51]Jiaxing Zhao, Xihan Wei, and Liefeng Bo. 2025. R1-omni: Explainable omni-multimodal emotion recognition with reinforcement learning. arXiv preprint arXiv:2503.05379 (2025).arxiv.org/abs/2503.05379

Paper details

vlnc:W_venfwjjz
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents