betaFeedback

WorksComputer Science

UEmo-D1: An Efficient and Calibrated Decision Model for Unified Multimodal Emotion Recognition

Abstract

Multimodal Emotion Recognition (MER) seeks to identify emotions conveyed through or elicited by media comprising diverse modality compositions. Research in this field has largely evolved along two paradigms: task-specific models and approaches based on Multimodal Large Language Models (MLLMs). Despite their distinct strengths, neither fully reconciles three capabilities essential to human-like emotion understanding: cross-task generalization, efficient decision-making, and receptiveness to multiple plausible interpretations. Inspired by recent advances in decision models, we explore a unified framework that maps multimodal inputs, guided by natural language instructions, directly to distributions over candidate options, addressing these requirements through direct probabilistic decisions. We instantiate this framework in UEmo-D1, the first decision model for MER to our knowledge. To support joint training and systematic evaluation, we also curate UEmo-Data, comprising 977,270 training samples and 41 evaluation subsets across six MER tasks. Extensive experiments demonstrate UEmo-D1's combination of strong recognition performance, low inference latency, and well-calibrated confidence estimates. Complementary analyses further reveal how training strategies and data composition influence these capabilities, providing practical insights into unified MER modeling. Together, our findings establish decision models as a feasible direction toward human-level emotion recognition. Project page: https://github.com/wdqqdw/UEmo-D1.

Introduction

Emotion plays an integral role in shaping human behavior and social interactions [29]. Understanding how people express and respond to emotions is therefore important for effective communication [60], with broader implications for building meaningful interpersonal relationships [43] and monitoring mental health [15]. Multimodal Emotion Recognition (MER) seeks to equip machines with this ability by drawing on multimodal content to identify emotions directly expressed by people [53], conveyed through media [36], or evoked by media [70]. Given its close connection to human experience and broad application potential, MER has attracted rapidly growing interest from both academia and industry [52].

After decades of research and debate, MER has evolved into two broad paradigms with distinct characteristics: compact task-specific models [77] and approaches based on multimodal large language models (MLLMs) [74]. Task-specific models excel at learning mappings between predefined input and output spaces, typically through supervised training on dedicated datasets. Although computationally efficient, these models often generalize poorly beyond their training settings [59], suffering substantial performance degradation when input modalities are disrupted [75], the target label space changes [39], or even the input domain shifts [18]. By contrast, MLLM-based MER reverses this trade-off, drawing on extensive world knowledge and strong instruction-following capabilities [57] to achieve broader generalization at greater computational cost [66]. More recently, the field is increasingly embracing explicit affective reasoning before prediction to make emotion judgments more explainable and accurate [31, 37].

Despite their respective strengths, the two paradigms face differing limitations in approaching human-level performance along three key dimensions. First, humans draw on partly shared cognitive representations [56] to flexibly apply their emotional understanding across diverse contexts, from perceiving a speaker’s emotions in face-to-face interactions to recognizing emotions evoked by media [11]. Achieving comparable versatility remains challenging even for MLLM-based approaches, whose generalization typically operates under fixed task requirements rather than across different task demands [40]. Second, human emotion perception is often intuitive and rapid, requiring little deliberate thought [58]. This resembles the direct inference of compact task-specific models, albeit without their generalization bottlenecks, and challenges the need for elaborate reasoning and its attendant computational overhead in MLLM-based approaches. Third, human emotion understanding inherently accommodates multiple plausible interpretations, reflecting a distribution of possibilities rather than a single categorical judgment [48]. This flexibility facilitates meaningful interactions with individuals from diverse backgrounds and different expressive styles, a capacity that neither paradigm is sufficiently adept at capturing and reproducing.

The emerging paradigm of decision models, often termed System 1 models [2, 17, 34], offers a promising avenue for bridging these gaps by combining flexible input processing with direct probabilistic decisions in place of explicit reasoning. Inspired by this paradigm, we introduce UEmo-D1, to our knowledge, the first decision model for unified MER. Guided by natural-language instructions and user-defined closed-set options, UEmo-D1 maps heterogeneous inputs, ranging from facial and vocal expressions to social media posts and audiovisual conversations, to task-specific decisions within a single framework, as illustrated in Figure 1(a). Each decision is accompanied by a probability distribution over the candidate options, obtained at a single decision-token position. To support this unified formulation with diverse supervision and comprehensive evaluation, we construct UEmo-Data, bringing together nearly one million training samples and 41 test subsets spanning diverse MER tasks, including facial expression recognition [61], conversational multimodal emotion recognition [24], and image-text sentiment analysis [69]. As shown in Figure 1(b), extensive comparisons against leading emotion-oriented, open-source, and proprietary MLLMs demonstrate that UEmo-D1 achieves strong recognition performance across tasks while maintaining low inference latency and markedly better-calibrated predictions. By bringing these capabilities together, UEmo-D1 offers a unified, efficient, and calibrated alternative to the two prevailing paradigms, advancing decision modeling as a practical route toward more human-like emotion recognition. Collectively, our main contributions are threefold:

Figure 1. Illustration of UEmo-D1’s decision-making framework and comparison of its accuracy, efficiency, and calibration with advanced MLLMs, with performance averaged over 41 subsets spanning six MER tasks. Decision mode compares first-token option probabilities, whereas reasoning mode allows for deliberation. Inference latency is measured on two NVIDIA RTX 4080 GPUs.

  • We introduce UEmo-D1, the first decision model for unified MER, bringing together versatile emotion recognition, low-latency inference, and well-calibrated confidence estimation within a single framework.

  • We present UEmo-Data, a comprehensive resource that consolidates nearly one million heterogeneous samples from diverse MER sources into a unified interface, enabling joint learning and systematic evaluation.

  • Extensive experiments establish UEmo-D1’s strengths in fine-grained recognition and confidence calibration, while elucidating how training strategies and data recipes shape decision-model performance. These findings provide practical guidance for developing decision models as a unified alternative for MER.

Related Work

Multimodal Emotion Recognition. MER represents a broad research area in which studies adopt the same MER designation despite differences in recognition targets, input modalities, and task requirements. Representative tasks include recognizing speakers’ emotions from audiovisual utterances [33], identifying affect conveyed by image–text social media posts [30], and interpreting emotional states alongside human behaviors in multimodal videos [76]. With the emergence of MLLM-based approaches, the scope of MER has expanded further. Their language-centered design makes text an integral input modality, allowing emotion recognition from vision or audio alone to fall within a common multimodal framework [74].

Early MLLM-based efforts, including EmoLA [35], EmoVIT [66], and Emotion-LLaMA [10], leverage natural-language instructions to bring previously separate recognition settings into shared models, enabling generalization across datasets and label taxonomies within their respective task domains. Building on this foundation, recent work such as AffectGPT [38] and EmoCaliber [63] places increasing emphasis on affective reasoning to improve explainability and recognition accuracy. Structured reasoning grounds emotional judgments in perceptual evidence, progressing from salient entities and their attributes to contextual relationships and, ultimately, emotional interpretations.

plified by UniEmo [78]. Despite their effectiveness, these approaches remain constrained in input-modality coverage, often rely on elaborate reasoning, and fail to adequately reflect the distributional nature of emotional interpretations, as quantitatively reflected in Figure 1(b). In contrast, UEmo-D1 adopts an alternative decision-modeling paradigm, delivering clear and simultaneous gains in recognition performance, inference efficiency, and confidence calibration.

Unified Modeling of MER Tasks

Motivated by the interconnected nature of human affective processing, a newly emerging wave of research has focused on generalization across MER tasks. Nano-EmoX [26] integrates facial, vocal, and linguistic cues to jointly recognize emotions and sentiments, infer intentions, and explain emotional causes. OneEmo [25] further incorporates social understanding, including humor and sarcasm, within a unified multitask framework. EmObserver [64] explores a complementary direction, bringing diverse aspects of visual emotional comprehension into a common statement-judgement formulation. Other efforts extend unification beyond understanding to emotion-evoking image generation, as exemplified by UniEmo [78].

Construction of UEmo-Data

Data form the foundation of modern model development. To enable rigorous evaluation and informative supervision, we first construct UEmo-Data based on the task taxonomy established in a recent MER survey [74], extending it to include higher-level social understanding. The resulting resource covers six complementary MER tasks, with 41 evaluation subsets and 977,270 training samples. These tasks are defined as follows: (1) Facial Expression Recognition (FER) [45, 61], which identifies emotions expressed through facial configurations and movements in images or videos; (2) Audio Emotion Recognition (AER) [42], which recognizes emotions conveyed through vocal expressions and acoustic cues; (3) Visual Content Emotion Comprehension (VCEC) [66, 70], which infers the emotional responses that visual content is likely to evoke in viewers; (4) Conversational Multimodal Emotion Recognition (CMER) [24, 53], which identifies a speaker’s emotional state from the visual, acoustic, and linguistic cues accompanying an utterance; (5) Image–Text Sentiment Analysis (ITSA) [36, 62], which determines the sentiment conveyed jointly by an image and its accompanying text; and (6) Multimodal Social Understanding (MSU) [9, 73], which integrates multimodal cues to infer communicative intent and recognize social meanings such as humor and sarcasm.

The construction of UEmo-Data is outlined in Figure 2. We begin by collecting datasets spanning diverse input modalities and task requirements, then process them through a five-stage pipeline summarized below, with further details provided in the appendix. First, with nearly all source datasets containing at most 26 categories, we adopt alphabetic option identifiers and map each dataset’s predefined categories to consecutive letters in ascending order. Second, to account for shared multimodal sources across datasets, we perform content-based deduplication and adjust class distributions within individual datasets. Third, we conduct quality control by excluding samples with inaccessible media and correcting issues such as modality misalignment and annotation inconsistencies. Fourth, we coordinate training, validation, and test partitioning across the entire collection to prevent cross-dataset leakage while preserving original split assignments wherever specified. For datasets without predefined splits, we target 500 samples per test subset, subject to availability, and allocate additional available samples to training. We then reserve a validation set from the training pool, comparable in size to the test set. Finally, we systematically vary instruction phrasing, option ordering, and other aspects of task presentation to mitigate systematic biases and reduce reliance on shortcuts during both training and evaluation. This process yields UEmo-Data with three disjoint splits, whose representative examples and task distributions are demonstrated in Figure 3.

Figure 2. Overview of the UEmo-Data construction pipeline. Heterogeneous datasets from diverse sources spanning six MER tasks are collected and processed through five stages: label mapping, deduplication, quality control, split verification, and instruction formatting. The processed data are then consolidated into UEmo-Data, a unified resource for MER training and evaluation.

Representative examples and task distributions of UEmo-Data

Figure 3. Representative examples and task distributions of UEmo-Data. Reflecting the abundance of high-quality visual resources, VCEC and FER account for the largest shares of the 977,270 training samples, while the validation and test splits, comprising 19,545 and 19,072 samples respectively, maintain more balanced distributions across the six MER tasks.

Acquisition of UEmo-D1

Consistent with recent studies on decision-model training [17, 34], our explorations reveal that dedicated training strategies struggle to accommodate heterogeneous data while maintaining reliable confidence estimates. Having dispensed with explicit reasoning, we also forgo an additional reinforcement learning stage given the risk of over-confidence associated with GRPO [5, 54, 63], opting instead for cross-entropy-based supervised fine-tuning (SFT) with three simple yet effective modifications.

Figure 4 illustrates the overall pipeline. We initialize our MLLM, denoted by πθ\pi_\theta, from Qwen2.5-Omni-3B [67] for its balance between efficiency and accuracy. Its native support for heterogeneous inputs allows missing visual or audio modalities to be omitted, while preserving the original encoding procedures for the available modalities. Within the language backbone, our first modification replaces the causal attention used for next-token pretraining with BERT-style [14] Bidirectional Attention over the input tokens during both training and inference. This enables each input token to attend to all others at every layer, strengthening contextual modeling within and across modalities. We denote the adapted model by πθ∗\pi^*_\theta.

Figure 4. Overview of the training and calibration pipeline for UEmo-D1. Starting from the UEmo-Data train split, samples with different modality combinations undergo unchanged modality-specific encoding before entering the language backbone, where causal attention is replaced with bidirectional attention to strengthen contextual modeling. At the output decision-token position, option-wise regularization then restricts softmax normalization to valid options for cross-entropy supervision. Throughout training, only LoRA parameters [23] in the language backbone are updated, with all original model parameters frozen. After training, softmax temperature scaling is fitted on the validation split to yield the calibrated UEmo-D1.

Given an input xx containing the available multimodal content, an instruction, and candidate options, πθ∗\pi^*_\theta predicts a single decision token dd without intermediate reasoning or subsequent autoregressive generation:

d=arg⁡max⁡y∈C πθ∗(y∣x),(1)d = \underset{y \in\mathcal{C}}{\arg\max}\ \pi^*_\theta(y \mid x), \tag*{(1)}

where C\mathcal{C} contains the valid option-letter tokens. Under standard training, the model produces logits zvz_v over the vocabulary V\mathcal{V} at the decision position, followed by full-vocabulary softmax normalization and cross-entropy supervision. In place of this standard formulation, our second modification introduces Option-wise Regularization to focus learning on the candidate options. It confines probability normalization and supervision to the valid option set C\mathcal{C}, with the resulting probability assigned to token vv defined as:

pv(τ)=Iv∈Cexp⁡(zv/τ)∑u∈Cexp⁡(zu/τ)(2)p_v(\tau) = \frac{\mathbb{I}_{v\in\mathcal{C}}\exp(z_v/\tau)}{\sum_{u\in\mathcal{C}}\exp(z_u/\tau)} \tag*{(2)}

where I\mathbb{I} is an indicator function and τ\tau is the temperature. With y∗y^* denoting the ground-truth option token, the training objective is:

LSFT=−∑v∈CIv=y∗log⁡pv(τ=1).(3)\mathcal{L}_{\mathrm{SFT}} = -\sum_{v\in\mathcal{C}}\mathbb{I}_{v=y^*}\log p_v(\tau=1). \tag*{(3)}

Cross-entropy training provides a principled foundation for calibrated probability estimation [19]. Building on the learned decision probabilities, our third modification refines confidence through post-training Temperature Scaling without updating model parameters. We fit the softmax temperature on a held-out validation set Dval\mathcal{D}_{\mathrm{val}} according to the expected calibration error (ECE) [50]:

τ∗=arg⁡min⁡τ>0∑b=1B∣Bb∣∣Dval∣∣acc⁡(Bb)−conf⁡τ(Bb)∣.(4)\tau^* = \underset{\tau>0}{\arg\min}\sum_{b=1}^{B}\frac{|\mathcal{B}_b|}{|\mathcal{D}_{\mathrm{val}}|}\left|\operatorname{acc}(\mathcal{B}_b)-\operatorname{conf}_{\tau}(\mathcal{B}_b)\right|. \tag*{(4)}

Here, Bb\mathcal{B}_b denotes the bb-th confidence bin based on pv(1)p_v(1), while acc⁡\operatorname{acc} and conf⁡τ\operatorname{conf}_{\tau} denote the empirical accuracy and mean confidence within each bin. At inference, we use pv(τ∗)p_v(\tau^*) as the calibrated option distribution.

Experiments

Preparations

Following the training procedure described above, we train UEmo-D1 for one epoch on globally shuffled data, using a learning rate of 10−410^{-4} with 500 warmup steps followed by cosine decay, a LoRA rank of 16, and a batch size of 64. Training takes approximately 20 hours on four NVIDIA RTX 4090 GPUs. Post-training temperature scaling uses 10 equal-width confidence bins, yielding τ∗=1.079\tau^* = 1.079.

With UEmo-D1 in place, we conduct a comprehensive experimental evaluation against open-source MLLMs [12, 67, 68, proprietary models [1, 21, 46, 49], and emotion-oriented MLLMs [10, 25, 26, 38, 64]. We consider two inference modes where supported: native reasoning mode (R), which allows intermediate reasoning before producing an answer, and decision mode (D), which selects an answer by comparing candidate-option probabilities at the first output-token position. We evaluate both modes on UEmo-Data’s test split, comprising 41 subsets derived from 4 FER datasets [4, 20, 28, 32], 3 AER datasets [6, 8, 42], 3 VCEC datasets [44, 51, 65, 70, 71], 3 CMER datasets [3, 8, 53], 3 ITSA datasets [27, 47, 72], and 4 MSU datasets [7, 9, 22, 73]. Unless otherwise specified, all evaluation prompts

feature diversified instruction phrasing and shuffled option orders, with identical configurations used across models.

We report four metrics: accuracy (Acc), macro-averaged F1 score (F1), expected calibration error (ECE), and Brier score. ECE measures the sample-weighted discrepancy between accuracy and confidence across confidence bins. The Brier score measures the mean squared distance between predicted probability distributions and one-hot ground-truth labels. Higher Acc and F1 and lower ECE and Brier scores indicate better performance.

Main Comparisons

Comparison of Recognition Accuracy. Table 1 compares representative models in native reasoning mode and a decision mode analogous to UEmo-D1. Overall, open-source and proprietary MLLMs exhibit relatively modest performance differences, underscoring the value of affect-specific optimization beyond general-purpose capabilities. Another notable finding is that decision mode outperforms reasoning mode in most comparisons, even for emotion-oriented MLLMs explicitly optimized for reasoning. This provides quantitative evidence that explicit reasoning may contribute more to explainability than to recognition accuracy itself. The results further reveal generalization challenges across both tasks and task formulations. Strengths such as OneEmo on FER and EmObserver on VCEC do not consistently generalize across tasks, and Nano-EmoX’s strong reported performance does not carry over to our evaluation with diversified instructions and shuffled options. Against this backdrop, UEmo-D1 consistently outperforms all competing models across the six tasks, achieving accuracy gains of 4.8–18.4% over the strongest baseline in each task. Its average accuracy reaches 75.6%, exceeding the runner-up, Gemini-3.5-Flash, by 13.3%.

ModelModeFER (%) AccFER (%) F1AER (%) AccAER (%) F1VCEC (%) AccVCEC (%) F1CMER (%) AccCMER (%) F1ITSA (%) AccITSA (%) F1MSU (%) AccMSU (%) F1Avg. AccAvg. F1
Open-Source MLLMs
Qwen2.5-Omni [67] 7BR37.027.428.422.259.557.158.052.662.657.565.162.251.846.5
Qwen2.5-Omni [67] 7BD38.229.154.448.761.058.870.355.664.157.667.163.459.252.2
Qwen3-Omni [68] 30A3BR51.438.342.134.763.461.271.457.063.959.173.872.861.053.9
Qwen3-Omni [68] 30A3BD49.839.146.838.564.262.471.758.263.255.872.671.061.454.2
MiniCPM-o 4.5 [12] 9BR52.739.634.223.163.261.567.654.760.255.469.367.557.950.3
MiniCPM-o 4.5 [12] 9BD54.341.437.927.463.061.369.153.060.455.271.069.559.351.3
Proprietary MLLMs
Qwen3-VL-Plus [1]R56.045.7––63.261.568.554.763.558.467.264.956.050.3
Kimi-K2.6 [46]R52.140.6––63.962.071.556.364.561.367.967.756.350.8
Gemini-3.5-Flash [21]R51.543.153.947.865.263.067.854.663.660.872.071.062.356.7
GPT-5.6-Luna [49]R52.644.0––60.459.267.853.763.457.865.365.454.549.5
Emotion-Oriented MLLMs
AffectGPT [38] 7BR11.08.320.513.129.723.654.843.556.849.755.655.638.132.3
AffectGPT [38] 7BD15.29.423.013.534.328.062.450.658.150.861.958.742.535.2
Emotion-LLaMA [10] 7BR36.024.539.032.950.343.427.424.560.051.739.229.942.034.5
Emotion-LLaMA [10] 7BD39.623.221.416.741.537.221.718.550.042.834.724.934.827.2
Nano-EmoX [26] 1.5BR0.50.716.514.33.75.316.417.337.236.65.29.113.313.9
Nano-EmoX [26] 1.5BD14.46.519.012.432.427.454.635.123.918.838.630.130.521.7
EmObserver [64] 8BR32.027.8––64.264.560.038.655.250.012.011.940.234.9
EmObserver [64] 8BD52.639.4––68.166.366.546.761.053.459.756.054.346.4
OneEmo [25] 4BR39.533.825.38.050.950.161.645.959.654.660.459.549.642.0
OneEmo [25] 4BD56.240.927.57.557.255.367.345.462.652.360.957.755.343.2
UEmo-D1 (Ours) 3BD66.854.172.871.272.970.677.662.781.877.681.580.775.669.5

Table 1. Recognition performance comparison of advanced open-source, proprietary, and emotion-oriented MLLMs on the test split of UEmo-Data. For models without AER support, uniform-random expected AER scores are used to compute the average scores. R and D denote reasoning and decision modes. Bold and <u>underline</u> indicate the best and second-best results in each column.

Figure 5 further compares leading MLLMs across all 41 fine-grained evaluation subsets. Except for Gemini-3.5-Flash, all models use decision mode for their stronger overall performance. UEmo-D1 leads on most subsets, with gains exceeding 15% on 10 subsets. Of the 11 subsets where it trails the strongest competing model, five show gaps below 0.5%. These results further substantiate UEmo-D1’s broad advantages in recognition accuracy.

Figure 5. Accuracy comparison of leading MLLMs on all 41 subsets spanning six MER tasks. Each subset is annotated with its number of candidate options, UEmo-D1’s accuracy (%), and its margin over the best-performing competing model in percentage points.

Comparison of Confidence Estimation. Table 2 compares confidence estimation across models, offering a practical perspective on the quality of their predictive distributions when ground-truth emotion distributions are unavailable. For a consistent comparison, we derive option probabilities from first-token logits in decision mode rather than extracting answer-token probabilities after reasoning as explored in [63], since the latter are not always readily accessible in practice. Under this protocol, UEmo-D1 achieves markedly better calibration and probabilistic accuracy, with an average ECE of 4.7, approximately one-third of the next-best result, and a Brier score of 33.5, only 60% of the next-best score. Together, these findings suggest that UEmo-D1’s strengths extend to a distributional perspective on emotion recognition, assigning confidence more reliably across emotional alternatives rather than merely selecting a single categorical label.

ModelFER ECEFER BrierAER ECEAER BrierVCEC ECEVCEC BrierCMER ECECMER BrierITSA ECEITSA BrierMSU ECEMSU BrierAvg. ECEAvg. Brier
Open-Source MLLMs
Qwen2.5-Omni [67] 7B21.980.117.261.424.360.312.441.426.158.817.248.919.958.5
Qwen3-Omni [68] 30A3B30.377.145.595.928.862.021.347.632.867.818.944.329.665.8
MiniCPM-o 4.5 [12] 9B27.870.547.1102.627.061.822.651.233.870.615.542.529.066.5
Emotion-Oriented MLLMs
AffectGPT [38] 7B52.9127.046.2113.347.1106.623.160.235.875.227.362.738.790.8
Emotion-LLaMA [10] 7B14.165.312.963.88.341.232.765.98.936.029.662.517.855.8
Nano-EmoX [26] 1.5B25.694.728.395.119.178.119.159.026.389.025.873.724.081.6
EmObserver [64] 8B38.082.1––24.855.127.458.635.673.228.265.030.866.8
OneEmo [25] 4B11.457.019.187.214.154.713.046.213.551.38.049.013.257.6
UEmo-D1 (Ours) 3B3.945.04.936.28.637.74.131.62.824.24.126.24.733.5

Table 2. Calibration comparison of advanced MLLMs on the test split of UEmo-Data, all evaluated in decision mode. ECE and Brier scores are multiplied by 100. Unavailable AER results are excluded when computing the averages.

Further Analysis

Influence of Optimization Strategies. To examine the contributions of individual components, Table 4 evaluates the removal of our three adopted modifications alongside alternative optimization strategies, each applied independently to UEmo-D1. Removing bidirectional attention or option-wise regularization slightly weakens recognition performance and probabilistic accuracy, while removing temperature scaling increases ECE without affecting classification accuracy. We attribute these modest effects partly to the large and diverse training collection, which may attenuate the gains from individual optimization refinements, as also suggested by prior observations on data scale [55].

VariantAcc ↑F1 ↑ECE ↓Brier ↓
UEmo-D175.669.54.733.5
− Bidirectional Attn.74.9 (−0.7)69.1 (−0.4)4.8 (+0.1)33.6 (+0.1)
− Option-wise Reg.75.2 (−0.4)69.1 (−0.4)4.7 (0.00)33.7 (+0.2)
− Temperature Scaling75.6 (0.00)69.5 (0.00)5.0 (+0.3)33.5 (0.00)
+ 1 Extra Epoch74.8 (−0.8)69.3 (−0.2)10.5 (+5.8)36.9 (+3.4)
+ 2 Extra Epochs74.6 (−1.0)69.2 (−0.3)14.2 (+9.5)39.3 (+5.8)
+ Brier Loss75.6 (0.00)69.4 (−0.1)5.1 (+0.4)33.6 (+0.1)
+ R-Drop [41]75.4 (−0.2)69.4 (−0.1)5.0 (+0.3)33.6 (+0.1)
+ GRPO [54]75.7 (+0.1)69.5 (0.00)9.3 (+4.6)40.0 (+6.5)
+ RLCR [13]75.3 (−0.3)69.0 (−0.5)7.6 (+2.9)37.0 (+3.5)

Table 4. Ablation study of adopted and explored optimization strategies. Relative changes are shown alongside each metric, with green indicating improvements and red indicating degradations.

We also find no clear joint improvement from additional optimization strategies. Extra training epochs reduce accuracy and substantially worsen calibration, consistent with overfitting through repeated exposure to the same data. Regularization constraints, such as Brier loss and R-Drop, likewise offer no overall benefit. We further explore reinforcement learning, following its use in decision optimization [2, 16]. GRPO improves accuracy by only 0.1% while nearly doubling ECE, echoing its reported tendency toward overconfidence [5]. Explicit calibration rewards through RLCR partially alleviate this deterioration but still fall short of UEmo-D1. Together, these findings suggest that additional optimization complexity offers limited benefits over a simple supervised recipe for MER decision modeling under our training setup.

Correlation among MER Tasks. Beyond optimization strategies, we examine how different training tasks interact. The upper block of Table 3 compares models fine-tuned on individual tasks against the base model (#1), revealing uneven cross-task transfer, with the largest gains generally concentrated on the task used for training. Interestingly, training on most other tasks improves ITSA and MSU, whereas the reverse is less consistent. This asymmetry suggests that these two tasks may require higher-level understanding and integration that benefit from improvements in more fundamental MER capabilities.

IndexTraining Tasks FERTraining Tasks AERTraining Tasks VCECTraining Tasks CMERTraining Tasks ITSATraining Tasks MSUTest Accuracy (%) ↑ FERTest Accuracy (%) ↑ AERTest Accuracy (%) ↑ VCECTest Accuracy (%) ↑ CMERTest Accuracy (%) ↑ ITSATest Accuracy (%) ↑ MSUAvg.
#1––––––34.547.060.964.759.557.954.1
#2✓–––––65.4 (+30.9)45.2 (−1.8)58.4 (−2.5)57.2 (−7.5)61.2 (+1.7)58.9 (+1.0)57.7 (+3.6)
#3–✓––––38.4 (+3.9)73.7 (+26.7)57.0 (−3.9)64.1 (−0.6)59.8 (+0.3)59.4 (+1.5)58.7 (+4.6)
#4––✓–––29.3 (−5.2)50.7 (+3.7)72.7 (+11.8)63.1 (−1.6)59.8 (+0.3)59.3 (+1.4)55.8 (+1.7)
#5–––✓––37.4 (+2.9)37.4 (−9.6)60.0 (−0.9)77.0 (+12.3)62.6 (+3.1)60.7 (+2.8)55.9 (+1.8)
#6––––✓–39.5 (+5.0)41.5 (−5.5)59.8 (−1.1)67.8 (+3.1)82.4 (+22.9)56.6 (−1.3)57.9 (+3.8)
#7–––––✓29.4 (−5.1)41.5 (−5.5)42.6 (−18.3)63.9 (−0.8)29.7 (−29.8)78.4 (+20.5)47.6 (−6.5)
#8✓✓✓✓✓✓66.872.872.977.681.881.575.6
#9–✓✓✓✓✓30.2 (−36.6)74.4 (+1.6)72.8 (−0.1)77.9 (+0.3)82.6 (+0.8)80.7 (−0.8)69.8 (−5.8)
#10✓–✓✓✓✓65.7 (−1.1)45.3 (−27.5)73.1 (+0.2)77.4 (−0.2)82.6 (+0.8)80.3 (−1.2)70.7 (−4.9)
#11✓✓–✓✓✓65.7 (−1.1)75.1 (+2.3)60.8 (−12.1)77.3 (−0.3)82.4 (+0.6)79.4 (−2.1)73.5 (−2.1)
#12✓✓✓–✓✓65.5 (−1.3)75.0 (+2.2)72.4 (−0.5)67.5 (−10.1)83.5 (+1.7)81.1 (−0.4)74.2 (−1.4)
#13✓✓✓✓–✓65.4 (−1.4)75.0 (+2.2)72.9 (0.0)77.8 (+0.2)60.7 (−21.1)80.4 (−1.1)72.0 (−3.6)
#14✓✓✓✓✓–64.8 (−2.0)75.3 (+2.5)72.5 (−0.4)77.7 (+0.1)83.0 (+1.2)60.8 (−20.7)72.4 (−3.2)

Table 3. Contributions of different task families to model training. Row #1 reports the Qwen2.5-Omni-3B baseline, with changes in rows #2–#7 measured relative to it. Row #8 reports UEmo-D1, with changes in rows #9–#14 measured relative to it.

Conversely, the lower block examines the effects of excluding each task from UEmo-D1 (#8). Every exclusion reduces average accuracy, highlighting the contribution of each task to overall performance. More specifically, all exclusions impair FER and MSU, whereas AER and ITSA improve whenever a task other than themselves is removed. These patterns do not simply mirror the single-task results, suggesting that transfer depends on the composition of the training mixture. Overall, these findings highlight both the value of joint training and the interplay between task complementarity and interference. While a deeper investigation could inform more effective data composition, the scope and space constraints of the present study leave this promising direction for future work.

Case Analysis. Figure 6 visualizes representative cases for an intuitive illustration of UEmo-D1’s decision behavior. Even when predicting correctly, UEmo-D1 assigns appreciable probability to plausible alternatives, such as anger alongside surprise in CMER and applicable gratitude in MSU. Although the VCEC and ITSA predictions disagree with the ground-truth labels, their distributions capture plausible ambiguity between joy and love, and between anger and disgust, respectively. These examples illustrate how probabilistic decisions can reflect competing emotional interpretations beyond a single categorical label, even when the top-ranked prediction is counted as incorrect.

Figure 6. Visualization of UEmo-D1’s recognition results on representative cases from each of the six MER tasks. The multimodal content and instructions are shown on the left, where GT denotes the ground-truth label. The predicted probability distributions are shown on the right, with incorrect predictions for VCEC and ITSA and correct predictions for the remaining tasks.

Conclusion

In this work, we introduce UEmo-D1, the first decision model for unified MER, combining efficient inference, superior performance, and enhanced calibration. To support systematic evaluation and joint optimization, we also curate UEmo-Data, a comprehensive data resource encompassing 41 evaluation subtasks and nearly one million training examples spanning six MER tasks. Beyond demonstrating UEmo-D1’s strengths across tasks, our systematic experiments shed light on how training strategies and task composition shape model performance. Together, we hope that UEmo-D1 will provide a practical, reproducible baseline for future research on unified MER and contribute to broader efforts toward human-level emotion recognition.

LLM Usage Statement

During this study, we used MLLM-based agents (OpenAI Codex, GPT-6 Astra) to assist with data-resource integration and downloading, training-code execution, and data organization for quantitative analyses. During manuscript preparation, we used Codex to refine the language of our original writing and assist with preparing figures and tables. All AI-assisted outputs were carefully reviewed and refined by the authors before inclusion in the manuscript.

Details of UEmo-Data

Data Composition

Table 5 summarizes the test split, which comprises 19,072 samples across 41 subtasks, with 500 samples retained per subtask wherever sufficient data are available. To accommodate their differing task requirements, these subtasks use four input configurations: image–text (I, T), video–text (V, T), audio–text (A, T), and video–audio–text (V, A, T). Beyond input diversity, the candidate label spaces range from 2 to 25 classes, allowing evaluation at different levels of classification granularity. Most subtasks involve emotion or sentiment categories, while others assess complementary aspects of affective understanding. Specifically, MVEI evaluates the validity of emotion-related statements, whereas subsets of MSU and MSED Desire concern communicative intentions, social meanings, and desires. Together, these variations provide a comprehensive assessment across input configurations, label spaces, and recognition targets.

Source DatasetSubtask#SamplesInput#ClassesOption TypeSource DatasetSubtask#SamplesInput#ClassesOption Type
FERCMER
CAER-S [32]CAER-S500I,T7EmotionCMU-MOSEI [3]MOSEI Anger500V,A,T2Emotion
DFEW [28]DFEW500V,T7EmotionMOSEI Disgust500V,A,T2Emotion
FERPlus [4]FERPlus500I,T8EmotionMOSEI Fear500V,A,T2Emotion
FER2013 [20]FER2013500I,T7EmotionMOSEI Happiness500V,A,T2Emotion
AERMOSEI Sadness500V,A,T2Emotion
CREMA-D [8]CREMA-D500A,T6EmotionMOSEI Surprise500V,A,T2Emotion
RAVDESS [42]RAVDESS Song176A,T6EmotionMOSEI Sentiment500V,A,T3Sentiment
RAVDESS Speech240A,T8EmotionMELD [53]MELD Emotion500V,A,T7Emotion
IEMOCAP [6]IEMOCAP500A,T4EmotionMELD Sentiment500V,A,T3Sentiment
VCECCREMA-D [8]CREMA-D500V,A,T6Emotion
MVEI [65]MVEI-EI500I,T2JudgmentITSA
MVEI-PS500I,T2JudgmentMSED [27]MSED Desire500I,T7Desire
MVEI-SC500I,T2JudgmentMSED Emotion500I,T6Emotion
MVEI-SP399I,T2JudgmentMSED Sentiment500I,T3Sentiment
Abstract [44]Abstract-8229I,T8EmotionMVSA-Multiple [47]MVSA-Multiple500I,T3Sentiment
ArtPhoto [44]ArtPhoto-8500I,T8EmotionTwitter-2017 [72]Twitter-2017500I,T3Sentiment
EmoSet [70]EmoSet-8500I,T8EmotionMSU
FI [71]FI-2500I,T2SentimentHFM [7]HFM500I,T2Sarcasm
FI-8500I,T8EmotionMIntRec [73]MIntRec Coarse445V,A,T2Intention
WebEmo-2500I,T2SentimentMIntRec Fine445V,A,T20Intention
WebEmo [51]WebEmo-25500I,T25EmotionMUStARD [9]MUStARD138V,A,T2Sarcasm
WebEmo-7500I,T7EmotionUR-FUNNY [22]UR-FUNNY500V,A,T2Humor
UnbiasedEmo [51]Unbiased Emo-6500I,T6EmotionOverall41 subsets19,072I,V,A,T2–25Mixed

Table 5. Composition of the UEmo-Data test split across six MER tasks. I, V, A, and T denote image, video, audio, and text, respectively.

To support this breadth of evaluation, the training split draws on a wider collection of data resources, as detailed in Table 6. It supplements the task coverage of the test split with additional label inventories and diverse instruction formulations, exposing the model to varied expressions of related recognition requirements. The resulting 977,270 training samples provide rich supervision for learning heterogeneous MER tasks within a unified framework.

Source Dataset#SamplesInput#Classes#Instr.Option TypeSource Dataset#SamplesInput#Classes#Instr.Option Type
FERVCEC
CAER-S [32]49,007I,T7200EmotionINSETS-462K [64]232,213I,T2200Judgment
DFEW [28]9,356V,T7200EmotionEmoSet [70]94,474I,T8200Emotion
FERPlus [4]24,723I,T8200EmotionFI [71]19,637I,T2, 8400Emotion, Sentiment
FER2013 [20]28,322I,T7200EmotionWebEmo [51]226,658I,T2, 7, 25400Emotion, Sentiment
ExpW [86]88,600I,T7200EmotionAffection [79]4,910I,T824Emotion
FindingEmo [84]13,093I,T2424Emotion
AERITSA
CREMA-D [8]4,672A,T6200EmotionMSED [27]18,381I,T3, 6, 7600Desire, Emo., Senti.
RAVDESS [42]1,540A,T6, 8400EmotionMVSA-Multiple [47]12,958I,T3200Sentiment
IEMOCAP [6]3,128A,T4200EmotionTwitter-2017 [72]3,545I,T3200Sentiment
TESS [85]2,600A,T7200EmotionMVSA-Single [47]3,604I,T3200Sentiment
SpeechCraft [83]1,000A,T7197EmotionTwitter-2015 [72]3,174I,T3200Sentiment
EmoV-DB [80]6,893A,T5200Emotion, StyleMSU
LISTEN [81]6,626A,T6, 10, 1624EmotionHFM [7]19,695I,T2200Sarcasm
ESD [87]23,650A,T524EmotionMIntRec [73]2,668V,A,T2, 20400Intention
CMERMUStARD [9]499V,A,T2198Sarcasm
CMU-MOSEI [3]40,197V,A,T2, 31,400Emotion, SentimentUR-FUNNY [22]7,612V,A,T2200Humor
MELD [53]19,026V,A,T3, 7400Emotion, SentimentOverall977,270V,A,T2–258,091Mixed
CREMA-D [8]4,809V,A,T6200Emotion

Table 6. Composition of the UEmo-Data training split. #Instr. counts distinct instruction formulations within each data source.

Detailed Construction of UEmo-Data

Building on the overview in § 3, we elaborate here on the procedures used to construct UEmo-Data.

Label Mapping. In the source datasets, each sample consists of multimodal content and a ground-truth label under predefined task requirements and candidate label sets. To support probabilistic decisions at a single output-token position, we map each candidate option to an uppercase English letter. Specifically, we prefix each option with its assigned letter (e.g., “A. Joy”) and incorporate this mapping into both the instruction format and the supervision target to ensure consistency between the presented options and the expected output.

Deduplication. With the label mapping established, we perform content-based deduplication across modalities. Image screening uses decoded RGB content and perceptual similarity to identify identical or near-duplicate images. Audio screening similarly uses waveform identity and source metadata to detect repeated recordings and shared source utterances. Beyond redundancy, weakly labeled sources such as WebEmo and SpeechCraft require additional attention to annotation reliability and data composition. We exclude conflicting labels for identical images within the same WebEmo classification setting and limit the contribution of SpeechCraft’s disproportionately large neutral-labeled collection. Specifically, its 352,261 records are reduced to 1,000 samples through deterministic round-robin selection across speakers and chapters, retaining 1,000 distinct speakers. Together, these adjustments reduce redundancy and source dominance while preserving diversity without enforcing uniform class distributions.

Quality Control. Beyond deduplication, we address quality issues that could compromise evaluation reliability or introduce misleading supervision, focusing on media integrity and alignment as well as annotation consistency. We first check for missing or empty files, decoding failures, and mismatches between source records and their associated media, then correct temporal inconsistencies where necessary. For example, UR-FUNNY audio and video are reconstructed on a common timeline using their original timestamps to preserve synchronization despite differences in starting points or durations. Complementing these media-level checks, annotation screening follows criteria tailored to each source. In Affection, this requires the leading emotion to receive a strict majority of annotation votes and excludes images labeled “something else.” For INSETS-462K, we follow its established annotation-screening criteria by adopting the corresponding filtered collection while preserving the labels of retained records. The documented processing repairs alignment for approximately 7,600 audiovisual samples and filters out more than 160,000 source records, providing cleaner inputs and more reliable supervision for training and evaluation.

Split Verification. The subsequent split assignment is relatively straightforward, but requires coordination across the collection. Many constituent datasets share multimodal sources, creating potential leakage when their independently defined partitions are combined. We retain datasets specifically intended for evaluation as test resources and, for others, select up to 500 samples per subtask from their original test partitions wherever available. To maximize separation between training and test content, we account for relationships at the speaker, dialogue, and source-video levels. RAVDESS uses a common actor-disjoint partition for speech and song, while CREMA-D similarly separates actors across splits. When incorporating LISTEN, we also exclude training records associated with held-out MELD dialogues or MOSEI source videos, preventing shared content from re-entering through another dataset. These measures help safeguard the validity of our evaluation.

Instruction Formatting. Following split assignment, preliminary experiments revealed pronounced sensitivity to option order and limited cross-task transfer. We attribute these observations partly to fixed instruction formulations and option arrangements, consistent with the performance degradation of Nano-EMoX under our diversified evaluation in Table 1. To mitigate these biases, we use LLMs to extensively reformulate task instructions, with cross-agent quality checks and human spot checks to preserve explicit and unambiguous recognition requirements. Complementing this linguistic diversity, candidate options are randomly permuted and their corresponding answer letters updated to reduce systematic associations between label identities and option positions. Together, these adjustments increase task-presentation diversity while reducing potential shortcuts. The resulting multimodal content [MEDIA], instruction [INST], and candidate options [CHOICES] are then assembled into a shared input template, which is applied uniformly across models evaluated in decision mode:

Unified Decision Prompt

[MEDIA]

[INST]

Choose your answer from: A. [CHOICE 1]; B. [CHOICE 2]; C. [CHOICE 3]; ...

Answer with only the single option letter, without explanation.

Other Experiments

Comparison of Decision Efficiency

Due to space constraints, the main paper provides only a brief account of inference efficiency. Table 7 complements Figure 1 with task-level latency results and the average number of output words. These results confirm that UEmo-D1 offers competitive inference efficiency. We emphasize, however, that this efficiency largely follows from the decision-modeling paradigm and the use of a lightweight backbone, without dedicated efficiency optimizations. The same benefit is apparent across the compared models, which consistently run faster in decision mode than in their native mode.

ModelModeFER TimeFER WordsAER TimeAER WordsVCEC TimeVCEC WordsCMER TimeCMER WordsITSA TimeITSA WordsMSU TimeMSU WordsAvg. TimeAvg. Words
Open-Source MLLMs
Qwen2.5-Omni [67] 7BR1.21121.31.31723.61.79132.83.48411.21.70530.62.89912.82.22622.9
Qwen2.5-Omni [67] 7BD0.1661.00.0811.00.1831.02.8981.00.1681.02.1071.01.0661.0
Qwen3-Omni [68] 30A3BR0.9531.00.9111.01.0671.02.5551.01.0761.02.0761.41.5281.0
Qwen3-Omni [68] 30A3BD0.8521.00.8101.00.9311.02.3731.00.9461.01.7741.01.3681.0
MiniCPM-o 4.5 [12] 9BR1.28318.95.4061.01.32929.32.6652.41.11518.52.1519.32.12215.2
MiniCPM-o 4.5 [12] 9BD0.1631.00.0881.00.0601.01.3181.00.0601.01.0561.00.5011.0
Proprietary MLLMs
Qwen3-VL-Plus [1]R–1.0–––1.0–1.0–1.0–1.0–1.0
Kimi-K2.6 [46]R–1.0–––1.0–1.0–1.0–1.0–1.0
Gemini-3.5-Flash [21]R–1.1–1.0–1.2–1.0–1.0–1.0–1.1
GPT-5.6-Luna [49]R–1.0–––1.0–1.0–1.0–1.0–1.0
Emotion-Oriented MLLMs
AffectGPT [38] 7BR0.9251.31.1331.51.2322.31.9401.10.8502.01.7212.61.3781.8
AffectGPT [38] 7BD0.1451.00.1131.00.0941.01.1021.00.0951.00.6431.00.4141.0
Emotion-LLaMA [10] 7BR1.1193.22.1385.41.3241.52.5591.01.4451.32.7381.31.8402.1
Emotion-LLaMA [10] 7BD0.5001.00.6681.00.5111.02.3031.00.5121.02.2521.01.1141.0
Nano-EmoX [26] 1.5BR5.04110.94.24227.33.36414.93.58810.42.55833.64.62421.23.67018.1
Nano-EmoX [26] 1.5BD0.2211.00.1781.00.1551.01.8921.00.1541.01.0871.00.7011.0
EmObserver [64] 8BR21.610548.4––25.881643.220.903453.539.3501006.341.1161007.726.505643.1
EmObserver [64] 8BD0.1501.0––0.1341.03.6671.00.1161.02.4821.01.4051.0
OneEmo [25] 4BR5.55586.15.74083.55.92789.86.88880.45.80288.35.96080.36.13485.4
OneEmo [25] 4BD0.3101.00.2091.00.2891.04.0621.00.2791.02.7061.01.4971.0
UEmo-D1 (Ours) 3BD0.1531.00.0381.00.0991.01.4171.00.1191.01.1701.00.5531.0

Table 7. Inference latency and output length of MLLMs. Time denotes mean recorded latency in seconds, measured on two NVIDIA RTX 4080 GPUs for locally deployed models. Words measure output length, excluding internal reasoning of proprietary models. R and D denote native reasoning and decision modes, respectively. Avg. is computed over available evaluated samples.

It is also worth clarifying that “reasoning mode” refers here to a model’s native response behavior, rather than an explicit requirement to reason. As a result, models trained to return concise predictions, such as AffectGPT and Emotion-LLaMA, can produce very short responses even in this mode. The roughly one-word outputs of proprietary models require a different interpretation: they represent only the visible answers, while any internal reasoning may be returned separately or remain hidden.

Detailed Comparison of Recognition Accuracy

Complementing Figure 5, Table 8–Table 15 provide detailed comparisons across individual subsets. We report decision-mode results for open-source and emotion-oriented models, as this mode yields better recognition performance in most comparisons, while proprietary models are evaluated in native reasoning mode.

ModelFER: Accuracy (%) ↑
CAER-S [32]DFEW [28]FERPlus [4]FER2013 [20]Overall
Open-Source MLLMs
Qwen2.5-Omni [67] 7B29.444.042.037.638.2
Qwen3-Omni [68] 30A3B30.454.259.255.449.8
MiniCPM-o 4.5 [12] 9B28.053.278.058.054.3
Proprietary MLLMs
Qwen3-VL-Plus [1]31.256.474.861.656.0
Kimi-K2.6 [46]29.852.670.455.652.1
Gemini-3.5-Flash [21]31.950.067.656.651.5
GPT-5.6-Luna [49]28.853.073.055.452.6
Emotion-Oriented MLLMs
EmObserver [64] 8B24.653.271.461.252.6
OneEmo [25] 4B27.259.683.055.056.2
UEmo-D1 (Ours) 3B51.669.882.863.066.8

Table 8. Accuracy comparison on the four FER subsets of UEmo-Data. Open-source and emotion-oriented models are evaluated in decision mode, while proprietary models use native inference. Overall denotes the unweighted mean across the four subsets. Bold and underline indicate the best and second-best results in each column, respectively.

ModelHFM [?]MIntRec [?] CoarseMIntRec [?] FineMUStARD [?]UR-FUNNY [?]Overall
Open-Source MLLMs
Qwen2.5-Omni [?] 7B74.474.663.463.060.267.1
Qwen3-Omni [?] 30A3B75.277.165.875.469.472.6
MiniCPM-o 4.5 [?] 9B75.676.963.165.973.671.0
Proprietary MLLMs
Qwen3-VL-Plus [?]74.267.661.165.267.867.2
Kimi-K2.6 [?]78.054.460.773.972.667.9
Gemini-3.5-Flash [?]78.967.360.581.671.672.0
GPT-5.6-Luna [?]78.670.154.463.859.665.3
Emotion-Oriented MLLMs
EmObserver [?] 8B75.463.438.263.857.659.7
OneEmo [?] 4B67.468.347.655.865.460.9
UEmo-D1 (Ours) 3B95.486.578.972.574.081.5

Table 15. Accuracy comparison on the MSU subsets of UEmo-Data.

For FER, Table 8 covers both static images and dynamic facial expressions. UEmo-D1 achieves an overall accuracy of 66.8%, exceeding the runner-up, OneEmo, by 10.6%. This advantage spans both static-image subsets CAER-S and FER2013 and outperforms the strongest baseline on the video-based DFEW by 10.2%. On FERPlus, it remains within 0.2% of the best result, showing consistently competitive performance across different forms of facial input.

For AER, Table 9 reveals substantial variation across models. Emotion-oriented specialization does not consistently outperform general-purpose models, as illustrated by OneEmo’s weak performance on RAVDESS, particularly on speech. UEmo-D1, by contrast, leads on both RAVDESS subsets and IEMOCAP and falls only 0.4% below the best result on CREMA-D. Its overall accuracy of 72.8% surpasses Qwen2.5-Omni by 18.4%, representing the largest task-level improvement among the six MER tasks.

ModelAER: Accuracy (%) ↑
CREMA-D [8]RAVDESS [42]IEMOCAP [6]Overall
SongSpeech
Open-Source MLLMs
Qwen2.5-Omni [67] 7B54.851.746.364.854.4
Qwen3-Omni [68] 30A3B65.623.932.165.846.8
MiniCPM-o 4.5 [12] 9B60.818.215.457.437.9
Proprietary MLLMs
Gemini-3.5-Flash [21]68.235.839.272.553.9
Emotion-Oriented MLLMs
OneEmo [25] 4B54.418.26.730.627.5
UEmo-D1 (Ours) 3B67.870.579.273.672.8

Table 9. Accuracy comparison on the four AER subsets of UEmo-Data.

The VCEC results are divided into two tables. Table 10 presents MVEI and WebEmo, which assess emotion-statement judgment and hierarchical emotion recognition, respectively. UEmo-D1 leads on three MVEI subtasks and trails the best SP result by only 0.5%, demonstrating strong performance in judging emotional context and subjectivity. On WebEmo, its margin over the strongest competi… tor increases from −2.0% to +9.0% and +9.2% across the 2-, 7-, and 25-class settings, highlighting its advantage as recognition becomes more fine-grained. Table 11 reveals a more mixed picture across visual domains. Strong results

ModelVCEC: Accuracy (%) ↑
MVEI [65]WebEmo [51]
EIPSSCSP2-class7-class25-class
Open-Source MLLMs
Qwen2.5-Omni [67] 7B78.667.678.675.778.047.823.8
Qwen3-Omni [68] 30A3B78.873.882.076.473.248.026.8
MiniCPM-o 4.5 [12] 9B78.875.475.675.774.848.024.6
Proprietary MLLMs
Qwen3-VL-Plus [1]82.074.278.086.775.647.424.0
Kimi-K2.6 [46]81.880.680.283.777.650.221.4
Gemini-3.5-Flash [21]75.384.278.184.785.048.522.8
GPT-5.6-Luna [49]75.472.274.877.271.444.422.6
Emotion-Oriented MLLMs
EmObserver [64] 8B78.479.884.285.273.653.026.6
OneEmo [25] 4B77.868.482.677.968.041.020.4
UEmo-D1 (Ours) 3B87.284.692.086.283.062.036.0

Table 10. Accuracy comparison on the first group of VCEC subsets in UEmo-Data.

ModelAbstract [44]ArtPhoto [44]EmoSet [70]FI 2-classFI 8-classUnbiasedEmo [51]Overall
Open-Source MLLMs
Qwen2.5-Omni [67] 7B16.243.655.491.054.282.061.0
Qwen3-Omni [68] 30A3B29.345.260.692.459.288.864.2
MiniCPM-o 4.5 [12] 9B24.947.057.692.460.683.863.0
Proprietary MLLMs
Qwen3-VL-Plus [1]19.246.055.293.256.083.663.2
Kimi-K2.6 [46]31.046.253.490.454.479.663.9
Gemini-3.5-Flash [21]35.249.746.194.559.584.565.2
GPT-5.6-Luna [49]18.343.854.893.254.483.060.4
Emotion-Oriented MLLMs
EmObserver [64] 8B43.246.067.290.672.086.068.1
OneEmo [25] 4B11.438.241.688.447.880.657.2
UEmo-D1 (Ours) 3B34.141.480.896.078.286.072.9

Table 11. Accuracy comparison on the second group of VCEC subsets in UEmo-Data. Overall includes results from Table 10.

on EmoSet and both FI subsets contribute to an overall accuracy of 72.9%, exceeding EmObserver by 4.8%. However, UEmo-D1 does not lead on Abstract, ArtPhoto, or UnbiasedEmo. These exceptions identify distinctive visual styles and evaluation distributions as areas where broader training coverage may prove valuable.

The CMER results are also presented in two tables. On CMU-MOSEI (Table 12), UEmo-D1 ranks first in two emotion categories and second in three, with no category falling more than 0.4% below the best result. Its advantage is clearer on sentiment recognition, where it exceeds the strongest baseline by 4.4%. This contrast becomes more pronounced on MELD (Table 13), with gains of 13.4% and 10.8% in emotion and sentiment recognition, respectively, suggesting more effective use of multimodal cues in conversational settings. Together, these gains yield an overall CMER accuracy of 77.6%, compared with 71.7% for the strongest baseline, Qwen3-Omni.

ModelAngerDisgustFearHappinessSadnessSurpriseSentiment
Open-Source MLLMs
Qwen2.5-Omni [67] 7B80.487.089.266.870.886.865.8
Qwen3-Omni [68] 30A3B81.487.687.469.670.690.468.6
MiniCPM-o 4.5 [12] 9B79.686.488.064.071.290.462.0
Proprietary MLLMs
Qwen3-VL-Plus [1]79.684.287.668.072.470.864.8
Kimi-K2.6 [46]79.285.888.465.470.890.860.8
Gemini-3.5-Flash [21]76.580.682.561.164.685.555.1
GPT-5.6-Luna [49]76.084.086.267.470.889.461.4
Emotion-Oriented MLLMs
EmObserver [64] 8B78.284.689.669.671.290.465.4
OneEmo [25] 4B77.884.489.856.470.491.860.6
UEmo-D1 (Ours) 3B81.288.089.475.472.291.673.0

Table 12. Accuracy comparison on the first group of CMER subsets in UEmo-Data.

ModelMELD [53] EmotionMELD [53] SentimentCREMA-D [8]Overall
Open-Source MLLMs
Qwen2.5-Omni [67] 7B50.251.254.870.3
Qwen3-Omni [68] 30A3B55.449.457.071.7
MiniCPM-o 4.5 [12] 9B54.451.843.269.1
Proprietary MLLMs
Qwen3-VL-Plus [1]44.455.458.268.5
Kimi-K2.6 [46]48.663.861.671.5
Gemini-3.5-Flash [21]50.649.471.667.8
GPT-5.6-Luna [49]35.446.261.267.8
Emotion-Oriented MLLMs
EmObserver [64] 8B34.238.443.466.5
OneEmo [25] 4B45.049.847.067.3
UEmo-D1 (Ours) 3B68.874.662.277.6
ModelMSED [?] DesireMSED [?] EmotionMSED [?] SentimentMVSA [?] MultipleTwitter [?] 2017Overall
Open-Source MLLMs
Qwen2.5-Omni [?] 7B55.672.070.470.452.064.1
Qwen3-Omni [?] 30A3B54.673.469.868.250.263.2
MiniCPM-o 4.5 [?] 9B38.672.071.469.051.060.4
Proprietary MLLMs
Qwen3-VL-Plus [?]51.072.870.269.254.463.5
Kimi-K2.6 [?]45.476.671.669.259.864.5
Gemini-3.5-Flash [?]48.375.672.669.951.763.6
GPT-5.6-Luna [?]51.071.670.870.253.663.4
Emotion-Oriented MLLMs
EmObserver [?] 8B49.669.070.069.646.861.0
OneEmo [?] 4B52.471.072.069.248.262.6
UEmo-D1 (Ours) 3B88.688.090.472.269.881.8

Table 13. Accuracy comparison on the second group of CMER subsets in UEmo-Data. Overall includes results from Table 12.

Finally, Tabs. 14 and Table 15 extend the comparison to ITSA and MSU. UEmo-D1 reaches overall accuracies of 81.8% and 81.5%, improving over the strongest baselines by 17.3% and 8.9%, respectively, and leads on every subset except MUStARD. These tasks require combining diverse multimodal cues to resolve sentiment, communicative intent, and social meaning. The breadth of these gains suggests that UEmo-D1’s strengths extend beyond recognizing individual emotional signals into integrating evidence into higher-level judgments, although the exception on MUSTARD leaves room for improvement in interpreting conversational sarcasm.

Visualization of Confidence Calibration

While the main paper focuses on aggregate quantitative comparisons of confidence estimation, Figure 7–Figure 12 provide complementary visualizations of UEmo-D1 across all subtasks. For FER (Figure 7), empirical accuracy generally follows the ideal calibration range at higher predicted confidence levels (≥0.5\ge0.5), whereas larger deviations appear in lower-confidence bins, a pattern also visible in the pooled

Figure 7. Visualization of UEmo-D1’s confidence estimation on FER subsets. Green squares mark the perfectly calibrated bins. Blue dots indicate empirical accuracy within each bin, with error bars showing 95% Wilson confidence intervals.

Figure 12. Visualization of UEmo-D1’s confidence estimation on MSU subsets.

results. Several AER subsets (Figure 8) exhibit similar variability at low confidence, with sparse samples contributing to wider confidence intervals. These results support the reliability of UEmo-D1’s more confident decisions, while identifying low-confidence predictions as an area for further calibration improvement.

Figure 8. Visualization of UEmo-D1’s confidence estimation on AER subsets.

Figure 11. Visualization of UEmo-D1’s confidence estimation on ITSA subsets.

For VCEC (Figure 9), the pooled results show generally good alignment between confidence and accuracy, although some subsets exhibit larger deviations, including low-confidence bins in FI-8 and Unbiased Emo-6. Abstract and ArtPhoto reveal an additional challenge in visually distinctive domains. Despite a broadly positive relationship between confidence and accuracy, their predictions remain overconfident across much of the confidence range, indicating a need for calibration that better accommodates distribution shifts. The subsequent results for CMER, ITSA, and MSU (Figure 10 to Figure 12) reveal no additional systematic failure pattern beyond these subset- and confidence-dependent deviations. Taken together, these visualizations support the

Figure 9. Visualization of UEmo-D1’s confidence estimation on VCEC subsets.

Figure 10. Visualization of UEmo-D1’s confidence estimation on CMER subsets.

broad reliability of UEmo-D1’s confidence estimates across heterogeneous subtasks, while clarifying where further refinement remains necessary.

Other Diagnostic Experiments

Sensitivity to Option Order. Emotion-oriented models are typically optimized on specialized data, potentially at the expense of some generalization capabilities. While reduced proficiency in unrelated domains such as mathematics may be less consequential for their intended use, this trade-off becomes more concerning when it affects how candidate answers are presented, as users may arrange them in arbitrary orders. To examine this sensitivity, we cyclically shift the options of every test sample in UEmo-Data while keeping its instruction and multimodal content unchanged. As shown in Table 16, AffectGPT, Nano-EmoX, and OneEmo exhibit average accuracy drops of 2.0–5.4%, with the largest task-level decline reaching 29.4% for Nano-EmoX on CMER. Such deterioration under a purely positional change reveals a reliance on option arrangements beyond their semantic content.

ModelModeFER (%) OriginalFER (%) ShiftedAER (%) OriginalAER (%) ShiftedVCEC (%) OriginalVCEC (%) ShiftedCMER (%) OriginalCMER (%) ShiftedITSA (%) OriginalITSA (%) ShiftedMSU (%) OriginalMSU (%) ShiftedAvg. OriginalAvg. Shifted
Emotion-Oriented MLLMs
AffectGPT [38] 7BD15.210.423.028.134.333.462.454.358.159.761.956.842.540.5
Δ\Delta−4.8+5.1−0.9−8.1+1.6−5.1−2.0
Nano-EmoX [26] 1.5BD14.414.919.015.632.432.854.625.223.921.038.642.130.525.3
Δ\Delta+0.5−3.4+0.4−29.4−2.9+3.5−5.2
EmObserver [64] 8BD52.653.7––68.169.366.565.661.062.459.756.854.354.3
Δ\Delta+1.1–+1.2−0.9+1.4−2.90.0
OneEmo [25] 4BD56.250.127.517.357.251.367.365.762.660.060.955.155.349.9
Δ\Delta−6.1−10.2−5.9−1.6−2.6−5.8−5.4
UEmo-D1 (Ours) 3BD66.866.572.873.472.972.677.677.581.881.981.581.575.675.6
Δ\Delta−0.3+0.6−0.3−0.1+0.10.00.0

Table 16. Performance of models under different candidate-option orders on the UEmo-Data test split. Original uses the same evaluation inputs and option orders as Table 1, while Shifted cyclically moves the first option to the last position, keeping instructions and multimodal inputs unchanged. Avg. denotes the unweighted mean across the six tasks, with uniform-random expected AER accuracy used for EmObserver. Each Δ\Delta row reports Shifted − Original, with green indicating improvements and red indicating degradations.

Against this backdrop, UEmo-D1 and EmObserver maintain nearly unchanged average accuracy. UEmo-D1’s more consistent behavior may partly stem from option-order randomization during training, illustrating the value of a simple measure against positional shortcuts. EmObserver’s emphasis on input diversity may offer a related benefit, helping prevent the learned decision space from collapsing into rigid associations between semantic labels and option positions.

Influence of Conflicting Modalities. CA-MER [82] is designed to evaluate emotion recognition under modality conflicts through three conditions: consistent, where both modalities agree with the ground-truth emotion, and audio-aligned and video-aligned, where only the corresponding modality reflects it. As shown in Table 17, UEmo-D1 outperforms the compared models across all three conditions, but its performance also reveals room for improvement. In particular, accuracy reaches 54.0% on audio-aligned samples but falls to 41.2% on video-aligned samples. This asymmetry is consistent with the audio-dominant tendency reported in the original CA-MER study, suggesting that UEmo-D1 may likewise rely more heavily on acoustic evidence when modalities disagree. Thus, despite its overall advantages, resolving conflicting cues remains a challenge, motivating more targeted training on modality conflicts to encourage balanced use of audio and visual evidence.

ModelCA-MER: Accuracy (%) ↑\uparrow ConsistentCA-MER: Accuracy (%) ↑\uparrow Conflicting Audio-alignedCA-MER: Accuracy (%) ↑\uparrow Conflicting Video-alignedCA-MER: Accuracy (%) ↑\uparrow Conflicting Avg.
Emotion-Oriented MLLMs
AffectGPT [38] 7B49.840.630.035.3
Nano-EmoX [26] 1.5B12.617.411.614.5
OneEmo [25] 4B63.446.036.841.4
UEmo-D1 (Ours) 3B66.254.041.247.6

Table 17. Accuracy comparison on CA-MER under consistent and conflicting multimodal conditions, with 500 samples per condition. Bold and underline indicate the best and second-best results in each column, respectively.

References

  1. [79]Panos Achlioptas, Maks Ovsjanikov, Leonidas Guibas, and Sergey Tulyakov. Affection: learning affective explanations for real-world visual data. In CVPR, pages 6641–6651, 2023.arxiv.org/abs/2210.01946
  2. [80]Adaeze Adigwe, Noé Tits, Kevin El Haddad, Sarah Ostad-abbas, and Thierry Dutoit. The emotional voices database: towards controlling the emotion dimension in voice generation systems. arXiv preprint arXiv:1806.09514, 2018.arxiv.org/abs/1806.09514
  3. [81]Jingyi Chen, Zhimeng Guo, Jiyun Chun, Pichao Wang, Andrew Perrault, and Micha Elsner. Do audio LLMs really LISTEN, or just transcribe? measuring lexical vs. acoustic emotion cues reliance. In EACL, pages 5848–5877, 2026.
  4. [82]Zhiyuan Han, Beier Zhu, Yanlong Xu, Peipei Song, and Xun Yang. Benchmarking and bridging emotion conflicts for multimodal emotion reasoning. In ACM MM, pages 5528–5537, 2025.
  5. [83]Zeyu Jin, Jia Jia, Qixin Wang, Kehan Li, Shuoyi Zhou, Songtao Zhou, Xiaoyu Qin, and Zhiyong Wu. SpeechCraft: a fine-grained expressive speech dataset with natural language description. In ACM MM, pages 1255–1264, 2024.DOI
  6. [84]Laurent Mertens, Elahe’ Yargholi, Hans Op de Beeck, Jan Van den Stock, and Joost Vennekens. FindingEmo: an image dataset for emotion recognition in the wild. In NeurIPS, pages 4956–4996, 2024.arxiv.org/abs/2402.01355
  7. [85]M. Kathleen Phoria-Fuller and Kate Dupuis. Toronto emotional speech set (TESS). Borealis, 2020.
  8. [86]Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang. From facial expression recognition to interpersonal relation prediction. IJCV, 126(5):550–569, 2018.arxiv.org/abs/1609.06426
  9. [87]Kun Zhou, Berrak Sisman, Rui Liu, and Haizhou Li. Emotional voice conversion: theory, databases and ESD. Speech Communication, 137:1–18, 2022.arxiv.org/abs/2105.14762

Paper details

vlnc:W_thzx8s4j
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Codex, GPT-6 Astra (bare)

View statement
Looking for related papers…
Contents