betaFeedback

WorksComputer Science

PI-MLLM: Psychophysics-Inspired Multimodal Large Language Model for Oddly Satisfying Video Emotion Analysis

Abstract

Oddly Satisfying Videos (OSVs) can evoke strong viewer responses through repetitive motion, visual tactility, and ASMR-like sounds despite containing few explicit semantic emotion cues. This makes OSVs a challenging setting for Video Emotion Analysis (VEA), since most existing methods focus on semantic video content and under-model the physical sensory dynamics that shape viewer responses. To support Oddly Satisfying Video Emotion Analysis (OSVEA), we construct the Oddly Satisfying Video Dataset (OSVD), comprising 12,000 annotated 5-second clips. Each clip provides viewer-emotion labels under video-only, audio-only, and audio-visual conditions, together with explanations grounded in visual and acoustic sensory evidence. Building on OSVD, we propose PI-MLLM, a psychophysics-inspired multimodal large language model that integrates Modality-Specific Dynamics Alignment (MSDA) and Coalition-Based Audio-Visual Attribution (CAVA). MSDA uses auxiliary prediction objectives to align modality representations with visual motion and acoustic energy-onset dynamics. CAVA evaluates emotion scores across modality coalitions and applies Shapley attribution to identify the dominant modality and determine whether visual and auditory evidence is synergistic or conflicting. Experiments show that PI-MLLM improves both viewer-side emotion prediction and sensory-grounded explanation quality, suggesting the value of physical sensory cues to video emotion analysis.

Introduction

Understanding how multimedia content shapes human emotion is a central goal in affective computing ([24, 39, 30]). Research on video emotion understanding has primarily focused on recognizing emotions expressed by on-screen individuals. A complementary line of research instead predicts emotions elicited in viewers by treating the video itself as an affective stimulus ([11, 3, 20, 10] (Guo et al. 2025)). This viewer-centered task, known as Video Emotion Analysis (VEA), is particularly challenging for Oddly Satisfying Videos (OSVs), whose repetitive movements, visual tactility, and ASMR-like sounds can evoke satisfaction or relaxation despite sparse facial, linguistic, and narrative emotion cues ([22, 25]). Accordingly, Oddly Satisfying Video Emotion Analysis (OSVEA) requires models to move beyond recognizing objects, scenes, and events and capture how physical sensory dynamics shape the viewer’s affective experience ([26, 22]).

However, existing VEA is not well suited to sensory-driven affect modeling in terms of supervision, temporal representation, and multimodal attribution (Figure 1). Datasets such as EmoCause focus on semantic content and causes, leaving modality-specific sensory dynamics under-explored ([19]). Although StimuVAR uses optical-flow peaks for frame selection, it may miss the subtle repetitive patterns central to OSVs ([16]). Likewise, joint-fusion methods such as eMotions obscure modality dominance and whether audio-visual cues reinforce or conflict with each other ((Wu et al. 2025)). Consequently, a model may recognize a cutting action and predict interest, yet fail to explain how predictable motion, rhythmic sounds, and their coordination elicit that response. OSVEA therefore requires modality-specific sensory supervision, dynamics-aware representations, and explicit audio-visual attribution.

Figure 1. Existing VEA methods rely on sensory-ungrounded semantic annotations and implicit multimodal fusion. Our approach addresses these limitations with psychophysics-aware sensory annotations and explicit multimodal modeling that preserves modality-specific dynamics and cross-modal relations for viewer-emotion reasoning.

To address the lack of an OSVEA-specific dataset, we construct the Oddly Satisfying Video Dataset (OSVD), comprising 12,000 annotated 5-second clips segmented and selected from 7,242 online OSV source videos. As illustrated in Figure 2, the raw videos are segmented, semantically deduplicated, and filtered for resolution and audio completeness before annotation. Domain-specific visual and audio expert models then process the isolated streams independently: the visual expert annotates the scene, motion, and state change, while the audio expert annotates the sound character, temporal envelope, and spatial impression. A fusion expert reconciles the two reports to generate audio-visual annotations and explicitly records whether the sensory cues reinforce, conflict with or complement each other. Each clip is annotated under video-only, audio-only and audio-visual conditions using the 27-class Cowen–Keltner taxonomy adopted by VCE ([8, 21]). Human annotations on a sampled subset are compared with the model-generated labels using Cohen’s κ\kappa to assess reliability.

OSVD construction and annotation pipeline

Figure 2. OSVD construction and annotation pipeline. Word-cloud-derived keywords guide data collection. Then segmented clips are semantically deduplicated, filtered for resolution, and checked for audio integrity. Visual annotations are structured as scene, motion, and state change, while audio annotations are structured as sound character, temporal envelope, and spatial impression. Finally, the modality-specific reports are fused into audio-visual annotations and evaluated against human judgments.

Building on OSVD, we propose PI-MLLM, a psychophysics-inspired MLLM framework composed of Modality-Specific Dynamics Alignment (MSDA) and Coalition-Based Audio-Visual Attribution (CAVA). Within MSDA, Visual Motion Dynamics (VMD) and Acoustic Energy-Onset Dynamics (AED) align visual and audio representations with signal-derived optical-flow, RMS-loudness, and onset-strength descriptors, encouraging the encoders to preserve sensory dynamics that semantic supervision may overlook. CAVA evaluates emotion scores under different modality coalitions and derives Shapley-style visual and auditory contributions and their interaction [28], allowing PI-MLLM to distinguish modality dominance, complementarity, and conflict. On OSVD, PI-MLLM achieves an emotion classification accuracy of 43.2% and an explanation semantic similarity of 0.661, outperforming the evaluated video-only and audio-visual MLLMs as well as the matched autoregressive fine-tuning baseline. Ablations further show that VMD and AED provide complementary gains and that coalition-based attribution yields additional improvements.

The main contributions are summarized as follows:

  • We introduce OSVD, to our knowledge the first dedicated dataset for OSVEA with fine-grained viewer-side emotion labels and detailed physical sensory-grounded explanations across modalities.

  • We propose PI-MLLM, a psychophysics-inspired MLLM framework for OSVEA that introduces MSDA to align modality representations with non-semantic visual and acoustic dynamics and CAVA to quantify modality contributions and audio-visual interactions.

  • Extensive experiments on OSVD demonstrate the effectiveness of PI-MLLM, with comparisons against state-of-the-art methods validating its superior performance.

Video Emotion Analysis Dataset

Video emotion analysis dataset targets the emotion triggered in viewers. It is distinguished from traditional multimodal sentiment analysis datasets like MOSI and MO-SEI [36, 3], which focus on the emotion expressed by the character appeared in the video. Originally, movies were an important material to analyze viewers’ emotion. MediaEval16 [9] and LIRIS-ACCEDE [5] were created to annotate audience feeling after watching specific movie clips. With the development of video platforms, videos are widespread in these platforms for people to share their emotion. YF-E6 (Xu et al. 2016) and VE-8 [20] were established like mushrooms after rain. They both based on videos from popular video platforms YouTube and Flickr, but annotated based on different emotion classification policy. Recently, VEA datasets come to more detailed annotation. With VCE [21] annotate the probability distribution of emotion evoked by the video, EmoCause [19] annotate emotions with abundant explanations including video,audio, shot and content analysis. However, they still lack the annotation of some physical sensory features and the modality-specific emotion labels. In this work, we introduce OSVD, a dataset with unimodal and cross-modal sensory annotation and detailed explanation of physical sensory features which also dominant to viewer’s emotion.

Video Emotion Analysis Methods

The goal of VEA is to extract the relevant features to bridge the gap between the video content and the triggered emotion. Early studies on VEA mainly center on designing hand-crafted features to represent emotion. For example, Acar et al. utilized Mel-frequency cepstral coefficients to capture speech features relevant to viewer’s emotion [1]. However, the correlation between the simple feature and triggered emotion is hard to be modeled. With the emergence of various VEA dataset, methods begun to focus on the multi-modal information fusion. Representative works like TAM [23] designed a cross-temporal multimodal fusion module that applies attention-based fusion to different modalities within and across video segments in order to capture the cross-modal feature central to emotion analysis. However, the fusion process may lack intuitive explanation and thus affecting the credibility of the results. Recent work begun to treat external information as additional cues for emotion analysis. For example, LCAffect [10] utilize Danmaku as an auxiliary modality for emotion analysis. EmoDETective [19] utilize detailed annotation of audio visual content and shot description to improve emotion analysis explainability. However, these methods may lack relative information in many situations. Motivated by predictive-coding and processing-fluency accounts, we use optical-flow dynamics as a descriptor of visual motion predictability relevant to affective responses [13, 26].

OSVD Construction

To support OSVEA, we construct the Oddly Satisfying Video Dataset (OSVD), with 12,000 annotated 5-second clips segmented and selected from 7,242 online short-form OSV source videos. As shown in Table 1, existing VEA datasets typically annotate only one viewing condition and, at most, provide semantic explanations. OSVD instead provides viewer-emotion labels under audio, visual, and audio-visual conditions, together with both semantic and physical sensory explanations. Figure 2 summarizes its construction and annotation pipeline.

DatasetScaleAVAVSemanticPhysical sensory
LIRIS-ACCEDE [5]9,800××✓××
VE-8 [20]1,101××✓××
YF-E6 (Xu et al. 2016)1,637××✓××
VAD [31]19,267××✓××
VCE [21]61,046×✓×××
EmoCause [19]14,000××✓✓×
OSVD (ours)12,000✓✓✓✓✓

Table 1. Comparison between OSVD and representative VEA datasets. Emotion-label columns indicate whether annotations are provided under audio-only (A), video-only (V), and audio-visual (AV) viewing conditions. Emotion explanations are divided into semantic descriptions and physical sensory descriptions.

Collection and segmentation. We derive OSV-related search terms from a word cloud of representative descriptions and use them to retrieve raw videos from online video platforms. Each collected video is standardized and segmented into two non-overlapping 5-second clips. OSVs are often organized around short repeated actions such as slicing, crushing, pouring, cleaning, or brushing [22, 14]; a 5-second clip usually captures one complete micro-action sufficient for emotion analysis.

Data cleaning and filtering. We conduct several cleaning and filtering steps to ensure data quality. Firstly, we calculate pairwise semantic similarity between candidate videos and remove one video from each pair whose score exceeds 0.9. Secondly, we discard videos whose orientation-normalized resolution is below 360 × 640 pixels. Thirdly, we inspect media-container and stream metadata, removing data with missing or corrupted audio.

Emotion and physical sensory annotation. The visual and audio experts independently annotate each clip under isolated unimodal conditions. For the visual stream, scene describes the semantic setting and activity, motion captures time-varying movement patterns, and state change records observable transformations caused by the activity. With only video stream provided, Molmo2 [7] uses these information to produce a video-only emotion label and explanation. For the audio stream, sound character describes timbral and spectral qualities, temporal envelope captures onset, evolution, and decay, and spatial impression represents perceived source proximity and sound-field extent. Given only this stream, Music Flamingo [15] uses the three fields to produce an audio-only label and explanation. Both labels use the same fixed 27-class viewer-emotion taxonomy. Claude Sonnet 4.6 [2] integrates the visual and auditory content reports to generate the audio-visual label and explanation, describing whether the two streams reinforce, conflict with, or complement each other.

Annotation reliability validation. To assess whether the expert-model labels reflect human viewer judgments, we randomly sample 100 clips from OSVD and ask five independent annotators to assign emotion labels for each modality view. Table 2 reports the average Cohen’s κ\kappa between the annotators and the corresponding OSVD labels under the expert models’ view. The values show that OSVD annotations are consistently aligned with human judgments across modalities.

MetricVideo-onlyAudio-onlyAudio-visual
Avg. Cohen’s κ\kappa0.700.670.71

Table 2. Average human agreement with OSVD labels on 100 randomly sampled clips under three modality views.

Method

Overview

Given an audio-visual clip x=(V,A)x = (V, A), PI-MLLM predicts y=(c,e)y = (c, e), where cc is the viewer-side emotion label and ee is its sensory-grounded explanation. As illustrated in Figure 3, the visual and audio encoders produce features FVF_V and FAF_A for the MLLM decoder to generate cc and ee. PI-MLLM contains two complementary modules: Modality-Specific Dynamics Alignment (MSDA) and Coalition-Based Audio-Visual Attribution (CAVA). We describe the two modules below.

Figure 3. Overall framework of PI-MLLM. Visual and audio inputs are encoded into FVF_V and FAF_A to generate viewer emotion cc and a sensory-grounded explanation ee. MSDA aligns the visual representation with optical-flow dynamics and the audio representation with RMS loudness and onset strength. CAVA uses modality-conditioned emotion scores to estimate visual and auditory contributions and their interaction. The total objective combines generation, dynamics-alignment, coalition-classification, and attribution losses.

Modality-Specific Dynamics Alignment

Visual Motion Dynamics (VMD). Prediction-error accounts suggest that aesthetic pleasure can arise when prediction errors become progressively reduced as perceptual regularities are learned [13, 26]. Taking a metal-cutting video as an example, after observing repeated cuts, viewers can infer the knife-motion pattern and anticipate subsequent changes. The resulting reduction in prediction error may contribute to the satisfying response. Motivated by this theory, we use mean optical-flow magnitude to represent frame-to-frame visual discrepancy and guide the model to better capture motion predictability. We attach a lightweight motion-regression head hmh_m to the visual encoder to predict mean optical-flow magnitude in order to help capture visual motion dynamics which are otherwise neglected by previous semantic training strategy. Each clip is divided into TT temporal windows. At step tt, we compute the mean Farneback optical-flow magnitude gv,t∈Rg_{v,t} \in\mathbb{R} over consecutive frame pairs in the corresponding window [12]. We supervise the time-aligned visual features with a mean-squared error loss:

Lmotion=1T∑t=1T∥hm(FV)t−gv,t∥22.(1)\mathcal{L}_{\mathrm{motion}} = \frac{1}{T}\sum_{t=1}^{T}\lVert h_m(F_{V})_t-g_{v,t}\rVert_2^2. \tag*{(1)}

This objective encourages the visual representation to retain visual motion dynamics that may otherwise be weakly exposed to the text decoder for emotion prediction.

Acoustic energy-onset dynamics (AED). Poerio et al. report that ASMR experiences combine overall calmness with localized physiological activation [25]. This coexistence suggests that affective audio should be characterized by both its sustained energy profile and its brief local features. Accordingly, we use frame-wise root-mean-square (RMS) loudness to describe the overall acoustic-energy trajectory relevant to perceived calmness, and onset strength to capture localized acoustic transients relevant to brief activation. At the same TT time steps, these two complementary descriptors form the target ga,t∈R2g_{a,t} \in\mathbb{R}^{2}. A single regression head hah_a is attached to the audio encoder to predict the descriptors from the audio hidden features FAF_A, and we also supervise it with a mean-squared error loss:

Laudio=1T∑t=1T∥ha(FA)t−ga,t∥22.(2)\mathcal{L}_{\mathrm{audio}} = \frac{1}{T}\sum_{t=1}^{T}\lVert h_a(F_A)_t-g_{a,t}\rVert_2^2. \tag*{(2)}

By reconstructing both descriptors, AED encourages the audio representation to preserve the non-semantic acoustic dynamics that distinguish sustained calmness from localized activation. These dynamics also provide complementary sensory evidence for predicting and explaining viewer-side emotion.

Coalition-Based Audio-Visual Attribution

Audio and vision do not always contribute to OSV affect consistently. A sharp cutting sound may reduce the relaxation suggested by smooth visual motion, while a soft repetitive sound may strengthen a visually calm ASMR clip. Shapley values provide a coalition-based attribution of model outputs [28]. PaSE incorporates Shapley-based modality equilibrium into multimodal sentiment training (He et al. 2026). PI-MLLM instead requires modality contributions and interactions to represent genuine dominance, conflict, or complementarity rather than forcing the modalities to balance. CAVA therefore considers two sensory modalities, VV and AA, and four coalitions S∈{∅,{V},{A},{V,A}}\mathcal{S} \in\{\varnothing,\{V\},\{A\},\{V,A\}\}, whose emotion scores are used to derive both Shapley contributions and pairwise interaction.

Specifically, we use a shared auxiliary classifier qψq_{\psi} over the 27 emotion classes to predict the emotion labels. For each coalition S\mathcal{S}, the coalition classifier is written as:

qψ(c∣S)=softmax⁡(fψ(S)c).(3)q_{\psi}(c \mid\mathcal{S})=\operatorname{softmax}(f_{\psi}(\mathcal{S})_c). \tag*{(3)}

where ψ\psi denotes the classifier parameters, fψ(S)f_\psi(S) denotes the 27-class logits computed from the available modality features in SS, and qψ(⋅∣S)q_\psi(\cdot\mid S) is the resulting coalition-level emotion distribution. OSVD provides three supervised coalition labels: the video-only emotion label yvy_v, the audio-only emotion label yay_a, and the fused audio-visual emotion label yavy_{av}. We train the head with:

Lcls=CE⁡(qψ(⋅∣{V}),yv)+CE⁡(qψ(⋅∣{A}),ya)+CE⁡(qψ(⋅∣{V,A}),yav).(4)\mathcal{L}_{\mathrm{cls}} = \operatorname{CE}(q_\psi(\cdot\mid\{V\}), y_v) + \operatorname{CE}(q_\psi(\cdot\mid\{A\}), y_a) + \operatorname{CE}(q_\psi(\cdot\mid\{V,A\}), y_{av}) . \tag*{(4)}

The coalition-level score for the final label yavy_{av} is defined in log-odds space:

sy(S)=log⁡q~ψ(yav∣S)1−q~ψ(yav∣S),(5)s_y(S) = \log\frac{\widetilde{q}_\psi(y_{av} \mid S)}{1-\widetilde{q}_\psi(y_{av} \mid S)} , \tag*{(5)}

where q~ψ(yav∣S)=clip⁡(qψ(yav∣S),δ,1−δ)\widetilde{q}_\psi(y_{av} \mid S) = \operatorname{clip}(q_\psi(y_{av} \mid S), \delta, 1-\delta), and δ\delta is a small clipping constant for numerical stability.

Shapley contribution computation. We compute the Shapley contributions of vision and audio:

ϕV=12[sy({V})−sy(∅)]+12[sy({V,A})−sy({A})],ϕA=12[sy({A})−sy(∅)]+12[sy({V,A})−sy({V})].(6)\begin{aligned} \phi_V &= \frac{1}{2}\left[s_y(\{V\})-s_y(\varnothing)\right] + \frac{1}{2}\left[s_y(\{V,A\})-s_y(\{A\})\right],\\ \phi_A &= \frac{1}{2}\left[s_y(\{A\})-s_y(\varnothing)\right] + \frac{1}{2}\left[s_y(\{V,A\})-s_y(\{V\})\right]. \tag*{(6)} \end{aligned}

Here, a positive value means the modality increases the log-odds score of the final viewer-side emotion. To make the attribution signal learnable during training, we further introduce a modality contribution loss. We directly use the signed Shapley contributions ϕV\phi_V and ϕA\phi_A as the visual and auditory contribution scores, and normalize their magnitudes into a two-way contribution target:

ψ=[∣ϕV∣∣ϕV∣+∣ϕA∣+ϵ,∣ϕA∣∣ϕV∣+∣ϕA∣+ϵ],(7)\psi= \left[ \frac{|\phi_V|}{|\phi_V|+|\phi_A|+\epsilon}, \frac{|\phi_A|}{|\phi_V|+|\phi_A|+\epsilon} \right], \tag*{(7)}

where ϵ\epsilon is a small constant. A contribution head predicts a distribution ψpred\psi_{\mathrm{pred}} over the visual and auditory modalities, and we supervise it with:

Lmod-contrib=KL⁡(stopgrad⁡(ψ)∥ψpred).(8)\mathcal{L}_{\mathrm{mod\text{-}contrib}} = \operatorname{KL}\left(\operatorname{stopgrad}(\psi)\mathbin{\|}\psi_{\mathrm{pred}}\right). \tag*{(8)}

This modality contribution loss teaches the model which modality contributes more to the final viewer-side emotion.

Shapley interaction computation. We further define the pairwise audio-visual interaction as:

IVA=sy({V,A})−sy({V})−sy({A})+sy(∅).(9)I_{VA} = s_y(\{V,A\}) - s_y(\{V\}) - s_y(\{A\}) + s_y(\varnothing). \tag*{(9)}

A positive IVAI_{VA} indicates synergy beyond the sum of unimodal evidence, while a negative IVAI_{VA} indicates conflict or redundancy. This formulation provides an interpretable estimate of modality interaction rather than forcing the two modalities to agree. We also normalize the interaction effect as:

I^VA=IVA∣ϕV∣+∣ϕA∣+∣IVA∣+ϵ.(10)\hat{I}_{VA} = \frac{I_{VA}}{|\phi_V|+|\phi_A|+|I_{VA}|+\epsilon}. \tag*{(10)}

and train an interaction head to predict IpredI_{\mathrm{pred}}:

Lmod-int=Huber⁡(Ipred,stopgrad⁡(I^VA)).(11)\mathcal{L}_{\mathrm{mod\text{-}int}} = \operatorname{Huber}\left(I_{\mathrm{pred}}, \operatorname{stopgrad}(\hat{I}_{VA})\right). \tag*{(11)}

This modality interaction loss keeps the interaction target robust when one coalition score is noisy, while the sign of I^VA\hat{I}_V^A still indicates whether audio and vision form a positive or negative interaction. The combined-modality loss is denoted as:

Lcm=Lmod-contrib+Lmod-int.(12)\mathcal{L}_{\mathrm{cm}} = \mathcal{L}_{\mathrm{mod\text{-}contrib}} + \mathcal{L}_{\mathrm{mod\text{-}int}}. \tag*{(12)}

Joint Training Objective

Combining the autoregressive generation loss Lgen\mathcal{L}_{\mathrm{gen}} with the auxiliary losses from sessions above, the total training objective is:

L=Lgen+λ1Lmotion+λ2Laudio+λ3Lcls+λ4Lcm,(13)\mathcal{L} = \mathcal{L}_{\mathrm{gen}} + \lambda_1\mathcal{L}_{\mathrm{motion}} + \lambda_2\mathcal{L}_{\mathrm{audio}} + \lambda_3\mathcal{L}_{\mathrm{cls}} + \lambda_4\mathcal{L}_{\mathrm{cm}}, \tag*{(13)}

where Lgen\mathcal{L}_{\mathrm{gen}}, Lmotion\mathcal{L}_{\mathrm{motion}}, Laudio\mathcal{L}_{\mathrm{audio}}, Lcls\mathcal{L}_{\mathrm{cls}}, and Lcm\mathcal{L}_{\mathrm{cm}} denote the autoregressive generation, motion, audio, classification, and combined-modality losses respectively. The weights λ1,…,λ4\lambda_1,\ldots,\lambda_4 balance the contribution of each component and are set to zero when the corresponding module is ablated.

Experiments

Experimental Settings

Dataset and protocol. The primary evaluation is on OSVD because it provides OSV-specific, modality-aware viewer-side labels and sensory-grounded explanations. We use a class-balanced OSVD test set to prevent majority emotions from dominating the metric. Emotion classification is evaluated using top-1 accuracy (ACC%) over the 27-class Cowen–Keltner taxonomy, while explanation quality is measured using Sentence-BERT-based Reason Similarity (RS) (Reimers and Gurevych [27]). We further report an out-of-distribution protocol on VCE and VE-8 after converting their labels to the OSVD taxonomy.

Backbone and implementation. We instantiate the backbone with Qwen3-Omni (Xu et al. 2025b) and fine-tune it with LoRA (Hu et al. 2022) (r=64r=64).

Baselines. External baselines span four groups: video MLLMs (VideoLLaMA3 (Zhang et al. [37]), LLaVA-Video (Zhang et al. [38]), Qwen2.5-VL (Bai et al. [4])), audio-visual MLLMs (Qwen2.5-Omni (Xu et al. 2025a), video-SALMONN-2+ (Tang et al. [29])), a closed commercial model (Claude Sonnet 4.6 (Anthropic [2])) and video emotion analysis methods (Emotion-LLaMA (Cheng et al. [6]), StimuVAR (Guo et al. [16]), EmoDETective (Huang et al. [19])). We additionally report Qwen3-Omni (Xu et al. 2025b) as the matched baseline for autoregressive(AR) finetuning and PI-MLLM.

Comparison with SOTA Methods

As shown in Table 3, we report the classification accuracy and semantic similarity of different methods on OSVD. Overall, PI-MLLM achieves the best performance, with 43.2% accuracy and 0.661 semantic similarity.

MethodACC (%)RS
Video MLLMs
VideoLLaMA3-7B (Zhang et al. [37])11.80.322
LLaVA-Video-7B (Zhang et al. [38])14.60.487
Qwen2.5-VL-7B (Bai et al. [4])23.80.472
Audio-visual MLLMs
Qwen2.5-Omni-7B (Xu et al. 2025a)17.60.498
video-SALMONN-2+ (Tang et al. [29])18.50.463
Closed commercial model
Claude Sonnet 4.6 (Anthropic [2])32.60.508
Video emotion analysis methods
Emotion-LLaMA-7B (Cheng et al. [6])11.30.301
StimuVAR-7B (Guo et al. [16])9.10.461
Qwen3-Omni baseline and our methods
Qwen3-Omni-30B (base) (Xu et al. 2025b)29.20.535
Qwen3-Omni-30B + AR fine-tuning35.10.636
Qwen3-Omni-30B + PI-MLLM43.20.661

Table 3. Comparison with SOTA methods on OSVD. ACC (%) denotes 27-class viewer-side emotion classification accuracy, and RS denotes semantic similarity between generated and reference explanations. Best per column in bold.

For vision-only models, Qwen2.5-VL and LLaVA-Video achieve the best results in accuracy (23.8%) and semantic similarity (0.487), respectively. PI-MLLM outperforms them by 19.4% and 0.174, showing the benefit of incorporating auditory information.

Audio-visual models remain limited, with video-SALMONN-2+ reaching 18.5% accuracy and Qwen2.5-Omni achieving 0.498 similarity. PI-MLLM still improves over them by 24.7% and 0.163, indicating that simple multi-modal fusion is insufficient for OSVD.

Video emotion analysis methods perform even worse, where Emotion-LLaMA and StimuVAR achieve 11.3% accuracy and 0.461 similarity, respectively. PI-MLLM surpasses them by 31.9% and 0.200, highlighting the difficulty of modeling OSVD with conventional approaches.

Among closed-source baselines, Claude Sonnet 4.6 is the strongest, with 32.6% accuracy and 0.508 similarity. PI-MLLM still leads by 10.6% and 0.153.

Overall, these results demonstrate that PI-MLLM effectively improves viewer emotion understanding in OSVD by leveraging modality-specific sensory dynamics and cross-modal interactions.

Module Ablation

Table 4 presents the contribution of each component in PI-MLLM. Following the Qwen3-Omni autoregressive baseline, we progressively introduce the proposed supervision objectives to evaluate their individual and combined effects.

ConfigurationVMDAEDCLSMCMIACC (%)RSΔ\Delta ACC (%)
Qwen3-Omni + AR ((Xu et al. 2025b))–––––35.10.636–
AR + VMD✓––––38.80.642+3.7
AR + AED–✓–––39.70.646+4.6
AR + VMD + AED✓✓–––42.10.655+7.0
AR + VMD + AED + MC✓✓–✓–42.70.659+7.6
AR + VMD + AED + CLS + MC✓✓✓✓–42.80.659+7.7
PI-MLLM✓✓✓✓✓43.20.661+8.1

Table 4. Ablation study on OSVD. VMD and AED denote the visual motion dynamics and acoustic energy-onset dynamics auxiliary objectives inside MSDA, respectively. CLS, MC, and MI denote the coalition emotion classification loss, modality contribution loss, and modality interaction loss inside CAVA, respectively.

MSDA consistently improves performance. Introducing the visual motion dynamics objective alone increases the classification accuracy from 35.1% to 38.8%, while the acoustic energy-onset dynamics objective achieves a larger improvement of 39.7%. Combining both objectives further raises the classification accuracy to 42.1% and the semantic similarity to 0.655, demonstrating that visual motion and acoustic dynamics provide complementary sensory cues for viewer emotion understanding.

We then evaluate the CAVA components. Based on the model equipped with both MSDA objectives, adding modality contribution supervision improves the classification accuracy from 42.1% to 42.7%. Introducing coalition emotion classification further increases the accuracy to 42.8%. Finally, incorporating modality interaction supervision yields the full PI-MLLM, achieving the best performance with 43.2% classification accuracy and 0.661 semantic similarity.

Overall, every proposed component consistently contributes to the final performance. The MSDA objectives enable the model to capture visual and auditory physical cues, while the CAVA objectives explicitly supervise coalition emotion prediction, modality contribution estimation, and cross-modal interaction modeling. Their complementary effects lead to the strongest overall performance on OSVD.

Out-of-Distribution Generalization

Table 5 evaluates the out-of-distribution generalization ability of PI-MLLM on VCE and VE-8. Since neither dataset originally follows the OSVD annotation protocol, both are re-annotated using the same 27-class viewer-side emotion taxonomy and sensory-grounded explanation format to enable a fair comparison.

MethodVCEVE-8
ACC (%)RSACC (%)RS
Qwen3-Omni ((Xu et al. 2025b))27.60.48629.10.501
Video emotion analysis methods
Emotion-LLaMA ([6])22.40.36824.70.383
StimuVAR ([16])34.30.50535.50.471
EmoDETective ([19])31.90.49731.40.518
Our methods
Qwen3-Omni + AR finetuning30.90.52432.60.536
Qwen3-Omni + PI-MLLM33.80.55236.10.567

Table 5. Out-of-distribution evaluation on VCE and VE-8. Both datasets are evaluated after re-annotation under the OSVD viewer-side 27-class taxonomy and sensory-explanation protocol.

As expected, all methods perform worse than on OSVD because of the domain shift in video content, emotion distributions, and annotation styles. Nevertheless, PI-MLLM demonstrates strong generalization across both datasets. On VCE, StimuVAR achieves the highest classification accuracy (34.3%), while PI-MLLM obtains a comparable classification accuracy of 33.8% and the highest semantic similarity score of 0.552. On VE-8, PI-MLLM achieves the best performance on both metrics, reaching 36.1% classification accuracy and 0.567 semantic similarity. Compared with the strongest competing methods, PI-MLLM improves the semantic similarity by 0.047 on VCE and improves the classification accuracy and semantic similarity on VE-8 by 0.6 percentage points and 0.049, respectively.

Compared with the autoregressive fine-tuning baseline, PI-MLLM consistently improves both classification accuracy and semantic similarity on the two datasets, demonstrating that the proposed supervision generalizes beyond the OSVD training distribution. Overall, these results indicate that explicitly modeling modality-specific sensory dynamics together with modality contributions and interactions produces representations that transfer effectively to unseen video emotion datasets.

Conclusion

This work investigates viewer emotion understanding in Oddly Satisfying Videos and highlights the importance of audio-visual sensory dynamics that are often overlooked by semantic-centered VEA methods. Through OSVD and PI-MLLM, we incorporated modality-aware emotion supervision, modality-specific dynamics alignment, and coalition-based audio-visual attribution into a unified framework for OSVEA. Experimental results demonstrate that modeling these factors improves both viewer-emotion prediction and sensory-grounded explanation, while the ablation studies confirm the complementary roles of MSDA and CAVA. Overall, our findings suggest that viewer-centered video emotion analysis should consider not only what a video depicts, but also how its sensory dynamics unfold and how individual modalities shape the resulting affective response.

OSVD Details

Source. OSVD was collected from YouTube videos posted within the five years preceding collection, retrieved with queries such as “stress relief,” “oddly satisfying,” and related keywords.

Modality-Isolated Annotation Prompts

Molmo2, Music Flamingo, and Claude Sonnet 4.6 produce the video-only, audio-only, and fused annotations, respectively. The prompts used in the annotation pipeline are shown below. In every prompt, emotion must be exactly one of the 27 viewer-emotion labels defined in the main paper.

Video-only prompt.

You are the visual expert for oddly satisfying video emotion analysis. You receive only the silent five-second video. Predict the dominant emotion elicited in a typical viewer, not an emotion expressed by a person or animal. Never infer sound. Ground the analysis in: (1) scene, objects, and materials; (2) motion speed, repetition, smoothness, regularity, and predictability; (3) visible state change, deformation, or fragmentation; and (4) color and lighting. Select exactly one label from the 27-label list. Return JSON only: {emotion, scene, motion, state_change, physical_cues, reason}.

Audio-only prompt.

You are the audio expert for oddly satisfying video emotion analysis. You receive only the audio track. Predict the dominant emotion elicited in a typical listener. Never infer visible objects or actions. Ground the analysis in: (1) timbre, texture, and frequency content; (2) attack, evolution, repetition, rhythm, and decay; (3) loudness stability and onset/transient dynamics; and (4) spatial impression, source proximity, and speech presence. Select exactly one label from the 27-label list. Return JSON only: {emotion, sound_character, temporal_envelope, spatial_impression, physical_cues, reason}.

Audio-visual fusion prompt.

You receive the independently generated video-only and audio-only reports. Predict the dominant viewer emotion after experiencing both modalities. Do not add facts absent from the reports. Cite concrete evidence from both modalities and identify their relation as visual-dominant, audio-dominant, synergy, conflict, or complementary. The final label need not equal either unimodal label. Return JSON only: {emotion, visual_evidence, audio_evidence, relation, bridge, reason}.

Full-Dataset Annotation Distributions

Figure S1 shows that Satisfaction constitutes the largest overall share across the three annotation views, while a broad range of other viewer-emotion categories remains represented. The modality-specific distributions are not identical: Satisfaction dominates the video-only annotations, whereas the audio-only distribution shifts toward Calmness, showing that isolated acoustic cues can induce an affective prior different from that of visual dynamics.

Figure S1. Full OSVD three-view label distributions. Counts are computed over all 12,000 clips for audio-visual, video-only, and audio-only annotation.

Annotation Reliability Validation

The main paper reports only the mean Cohen’s κ\kappa for each modality. Table S1 gives the complete per-annotator results for five human annotators against the corresponding model-generated labels on the same 100 clips. The closely clustered scores and the standard deviation of 0.03 for every view indicate stable agreement patterns across annotators, while the κ\kappa range of 0.63–0.76 demonstrates consistently high agreement between individual annotators and the automatic labels.

AnnotatorVideo-onlyAudio-onlyAudio-visual
Annotator 10.660.630.67
Annotator 20.710.690.73
Annotator 30.680.650.69
Annotator 40.700.670.71
Annotator 50.740.720.76
Mean ± SD0.70±0.030.67±0.030.71±0.03

Table S1. Human–model Cohen’s κ\kappa (NN=100 clips per annotator and view).

Definition of the Empty Coalition in CAVA

Coalition-Based Audio-Visual Attribution (CAVA) represents no video and no audio by a modality-independent zero vector:

z∅=e∅=0,z{V}=PV(F‾V),z_{\varnothing}=e_{\varnothing}=0,\qquad z_{\{V\}}=P_{V}(\overline{F}_{V}),
z{A}=PA(F‾A),z{V,A}=PV(F‾V)+PA(F‾A),(14)z_{\{A\}}=P_{A}(\overline{F}_{A}),\qquad z_{\{V,A\}}=P_{V}(\overline{F}_{V})+P_{A}(\overline{F}_{A}), \tag*{(14)}

where PVP_{V} and PAP_{A} project pooled modality features into the shared space. With the shared classifier, fψ(∅)=Wψ0+bψ=bψf_{\psi}(\varnothing)=W_{\psi}0+b_{\psi}=b_{\psi}. Thus, f(∅)f(\varnothing) is the classifier-bias baseline used by the Shapley equations. It contains no encoded black video, silent waveform, text token, or pseudo emotion label.

Case Study

The same checkpoint is evaluated under video-only, audio-only, and audio-visual input. Figure S2 reports the generated labels and explanations, normalized Shapley contributions, and signed interaction. Case 1 shows that regular visual cutting can dominate an unpleasant sound; Case 2 shows that an ASMR-like sound can dominate while the gentle visual rhythm reinforces it.

Figure S2. Two PI-MLLM cases. Case 1 is visually dominant (V=0.86V=0.86, A=0.14A=0.14) with negative interaction (−0.071-0.071, conflict); Case 2 is audio dominant (V=0.16V=0.16, A=0.84A=0.84) with positive interaction (+0.059+0.059, synergy).

Ablation of Physical-Feature Annotation

The semantic-only target retains object, scene, and event descriptions; the semantic+physical target additionally retains the visual and acoustic fields listed in the prompts above. All other settings are fixed.

Training annotationMethodACC (%)RS
Semantic onlyQwen3-Omni + AR fine-tuning33.20.574
Semantic + physicalQwen3-Omni + AR fine-tuning35.10.636
Semantic onlyQwen3-Omni + PI-MLLM38.90.602
Semantic + physicalQwen3-Omni + PI-MLLM43.20.661

Table S2. Effect of physical-feature annotation. ACC is 27-class top-1 accuracy; RS is explanation cosine similarity.

Table S2 shows that adding physical-feature annotations improves both training approaches over their semantic-only counterparts: AR fine-tuning rises from 33.2/0.574 to 35.1/0.636 in ACC/RS, while PI-MLLM rises from 38.9/0.602 to 43.2/0.661. Under the matched semantic+physical setting, AR fine-tuning still trails PI-MLLM by 8.1 ACC points and 0.025 RS. Together with PI-MLLM’s larger ACC gain from physical annotations (4.3 versus 1.9 points), these results indicate that the proposed method effectively learns and exploits the importance of physical sensory features for viewer-emotion analysis.

Experimental Hyperparameter Design

The experimental hyperparameter settings are shown in Table S3.

HyperparameterValueHyperparameterValue
BackboneQwen3-Omni-30B-A3B-ThinkingPrecisionbf16
Frozen encodersVisual and audioLoRA targetDecoder q/k/v/o
LoRA rank / alpha / dropout64/128/0.05Optimizer / decayAdamW / 0
Learning rate / schedule10−410^{-4} / cosineWarmup / grad. clip10%/1.0
Loader batch / accumulation8/1Epochs / seed5/42
Video rate / pixel range1 fps / 16,384–50,176Temporal steps TT10
λ1\lambda_{1}, λ2\lambda_{2} (VMD, AED)0.1, 0.1λ3\lambda_{3}, λ4\lambda_{4} (CLS, CM)0.1, 0.1
CAVA classes / hidden / dropout27/512/0.1

Table S3. PI-MLLM experimental hyperparameter settings.

References

  1. [1]Acar, E.; Hopfgartner, F.; and Albayrak, S. 2017. A Comprehensive Study on Mid-Level Representation and Ensemble Learning for Emotional Analysis of Video Material. Multimedia Tools and Applications, 76(9): 11809–11837.DOI
  2. [2]Anthropic. 2026. Introducing Claude Sonnet 4.6. Anthropic News.
  3. [3]Bagher Zadeh, A.; Liang, P. P.; Poria, S.; Cambria, E.; and Morency, L.-P. 2018. Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2236–2246. Association for Computational Linguistics.DOI
  4. [4]Bai, S.; Chen, K.; Liu, X.; Wang, J.; Ge, W.; Song, S.; Dang, K.; Wang, P.; Wang, S.; Tang, J.; Zhong, H.; Zhu, Y.; Yang, M.; Li, Z.; Wan, J.; Wang, P.; Ding, W.; Fu, Z.; Xu, Y.; Ye, J.; Zhang, X.; Xie, T.; Cheng, Z.; Zhang, H.; Yang, Z.; Xu, H.; and Lin, J. 2025. Qwen2.5-VL Technical Report. arXiv preprint arXiv:2502.13923.arxiv.org/abs/2502.13923
  5. [5]Baveye, Y.; Dellandréa, E.; Chamaret, C.; and Chen, L. 2015. LIRIS-ACCED: A Video Database for Affective Content Analysis. IEEE Transactions on Affective Computing, 6(1): 43–55.DOI
  6. [6]Cheng, Z.; Cheng, Z.-Q.; He, J.-Y.; Sun, J.; Wang, K.; Lin, Y.; Lian, Z.; Peng, X.; and Hauptmann, A. G. 2024. Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning. In Advances in Neural Information Processing Systems, volume 37, 110805–110853.
  7. [7]Clark, C.; Zhang, J.; Ma, Z.; Park, J. S.; Tripathi, R.; Lee, S.; Salehi, M.; Ren, J.; Kim, C. D.; Yang, Y.; Shao, V.; Yang, Y.; Huang, W.; Gao, Z.; Anderson, T.; Zhang, J.; Jain, J.; Stoica, G.; Farhadi, A.; and Krishna, R. 2026. Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 28652–28668.arxiv.org/abs/2601.10611
  8. [8]Cowen, A. S.; and Keltner, D. 2017. Self-report captures 27 distinct categories of emotion bridged by continuous gradients. Proceedings of the National Academy of Sciences, 114(38): E7900–E7909.DOI
  9. [9]Dellandréa, E.; Chen, L.; Baveye, Y.; Sjöberg, M.; and Chamaret, C. 2016. The MediaEval 2016 Emotional Impact of Movies Task. CEUR Workshop Proceedings, 1739.
  10. [10]Deng, Z.; Ananthram, A.; and McKeown, K. 2025. Enhancing Multimodal Affective Analysis with Learned Live Comment Features. In Proceedings of the Thirty-Ninth AAAI Conference on Artificial Intelligence, volume 39, 16253–16261.
  11. [11]Dhall, A.; Goecke, R.; Lucey, S.; and Gedeon, T. 2012. Collecting Large, Richly Annotated Facial-Expression Databases from Movies. IEEE MultiMedia, 19(3): 34–41.DOI
  12. [12]Farnebäck, G. 2003. Two-Frame Motion Estimation Based on Polynomial Expansion. In Image Analysis: 13th Scandinavian Conference, SCIA 2003, volume 2749 of Lecture Notes in Computer Science, 363–370. Springer.DOI
  13. [13]Friston, K. 2012. Predictive coding, precision and synchrony. Cognitive Neuroscience, 3(3-4): 238–239.DOI
  14. [14]Gallagher, R. 2016. Eliciting Euphoria Online: The Aesthetics of “ASMR” Video Culture. Film Criticism, 40(2).DOI
  15. [15]Ghosh, S.; Goel, A.; Koroshinadze, L.; Lee, S.-g.; Kong, Z.; Santos, J. F.; Duraiswami, R.; Manocha, D.; Ping, W.; Shoeybi, M.; and Catanzaro, B. 2026. Music Flamingo: Scaling Music Understanding in Audio Language Models. In The Fourteenth International Conference on Learning Representations.arxiv.org/abs/2511.10289
  16. [16]Guo, Y.; Siddiqui, F.; Zhao, Y.; Chellappa, R.; and Lo, S.-Y. 2025. StimuVAR: Spatiotemporal Stimuli-Aware Video Affective Reasoning with Multimodal Large Language Models. International Journal of Computer Vision, 133(9): 6456–6472.arxiv.org/abs/2409.00304
  17. [19]Huang, X.; Zhou, Y.; Li, J.; Lu, S.; and Wang, S. 2025. EmoDEtective: Detecting, Exploring, and Thinking Emotional Cause in Videos. In Proceedings of the 33rd ACM International Conference on Multimedia, 5735–5744.DOI
  18. [20]Jiang, Y.-G.; Xu, B.; and Xue, X. 2014. Predicting Emotions in User-Generated Videos. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 28, 73–79.DOI
  19. [21]Mazeika, M.; Tang, E.; Zou, A.; Basart, S.; Chan, J. S.; Song, D.; Forsyth, D. A.; Steinhardt, J.; and Hendrycks, D. 2022. How Would the Viewer Feel? Estimating Wellbeing from Video Scenarios. In Advances in Neural Information Processing Systems, volume 35, 18571–18585.arxiv.org/abs/2210.10039
  20. [22]Nansen, B.; and Balanzategui, J. 2022. Visual Tactility: ‘Oddly Satisfying’ Videos, Sensory Genres and Ambiguities in Children’s YouTube. Convergence: The International Journal of Research into New Media Technologies, 28(6): 1555–1576.DOI
  21. [23]Pan, J.; Wang, S.; and Fang, L. 2022. Representation Learning through Multimodal Attention and Time-Sync Comments for Affective Video Content Analysis. In Proceedings of the 30th ACM International Conference on Multimedia, 42–50.DOI
  22. [24]Picard, R. W. 1997. Affective Computing. MIT Press. ISBN 978026216701.
  23. [25]Poerio, G. L.; Blakey, E.; Hostler, T. J.; and Veltri, T. 2018. More than a feeling: Autonomous sensory meridian response (ASMR) is characterized by reliable changes in affect and physiology. PLOS ONE, 13(6): e0196545.DOI
  24. [26]Reber, R.; Schwarz, N.; and Winkielman, P. 2004. Processing Fluency and Aesthetic Pleasure: Is Beauty in the Perceiver’s Processing Experience? Personality and Social Psychology Review, 8(4): 364–382.
  25. [27]Reimers, N.; and Gurevych, I. 2019. Sentence-BERT: Sentence Embeddings Using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 3982–3992. Hong Kong, China: Association for Computational Linguistics.
  26. [28]Shapley, L. S. 1953. A Value for n-Person Games. In Kuhn, H. W.; and Tucker, A. W., eds., Contributions to the Theory of Games II, 307–317. Princeton University Press.
  27. [29]Tang, C.; Li, Y.; Yang, Y.; Zhuang, J.; Sun, G.; Li, W.; Ma, Z.; and Zhang, C. 2025. video-SALMONN 2: Captioning-Enhanced Audio-Visual Large Language Models. arXiv preprint arXiv:2506.15220.arxiv.org/abs/2506.15220
  28. [30]Wang, D.; and Zhao, X. 2022. Affective Video Recommender Systems: A Survey. Frontiers in Neuroscience, 16: 984404.DOI
  29. [31]Wang, S.; Li, X.; Zheng, F.; Pan, J.; Li, X.; Chang, Y.; Zhu, Z.; Li, Q.; Wang, J.; and Xiao, Y. 2024. VAD: A Video Affective Dataset with Danmu. IEEE Transactions on Affective Computing, 15(4): 1889–1905.DOI
  30. [36]Zadeh, A.; Chen, M.; Poria, S.; Cambria, E.; and Morency, L.-P. 2017. Tensor Fusion Network for Multimodal Sentiment Analysis. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 1103–1114. Association for Computational Linguistics.arxiv.org/abs/1707.07250
  31. [37]Zhang, B.; Li, K.; Cheng, Z.; Hu, Z.; Yuan, Y.; Chen, G.; Leng, S.; Jiang, Y.; Zhang, H.; Li, X.; Jin, P.; Zhang, W.; Wang, F.; Bing, L.; and Zhao, D. 2025. VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding. arXiv preprint arXiv:2501.13106.arxiv.org/abs/2501.13106
  32. [38]Zhang, Y.; Wu, J.; Li, W.; Li, B.; Ma, Z.; Liu, Z.; and Li, C. 2024. Video Instruction Tuning With Synthetic Data. arXiv preprint arXiv:2410.02713.arxiv.org/abs/2410.02713
  33. [39]Zhao, S.; Wang, S.; Soleymani, M.; Joshi, D.; and Ji, Q. 2019. Affective Computing for Large-Scale Heterogeneous Multimedia Data: A Survey. ACM Transactions on Multimedia Computing, Communications, and Applications, 15(3s): 1–32.DOI

Paper details

vlnc:W_tv3xa3nx
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents