betaFeedback

WorksComputer Science

EmoReward: Self-Evolving Reward Calibration for Emotional Video Generation RL

Abstract

Group Relative Policy Optimization (GRPO) has advanced generative model post-training in verifiable domains such as code and mathematics, but evaluating video quality, especially emotional expression, often depends on subjective aesthetics and has no uniquely correct target. Existing approaches typically rely on fixed rewards for visual quality, motion quality, and text alignment that poorly capture emotional intent and can invite reward hacking, while RLHF is difficult to scale and subject to individual aesthetic differences that complicate the calibration of facial, body motion, and chromatic cues across narrative contexts. We introduce EmoReward, a new self-evolving training mechanism that distills evidence reliability from reward traces collected during an actual training run to adaptively guide multi-reward weight allocation in subsequent training by determining which emotional evidence to trust and how much to weight it. EmoReward first plans specific grounding cues from the emotional prompt to guide evaluation, then uses heterogeneous signal primitives to score the generated video during the act stage. Reflective Reward Gating calibrates their contributions through instance-level gates for phase, face confidence, conflict, and subtlety, then renormalizes weights over the active set to produce the sample's fused reward. RewardMemory then estimates each primitive's reliability from rollout scores and outcomes within each emotional regime in the finished training run, then fixes bounded multipliers that calibrate fusion weights in the next training run without updating reward models. Experiments on HunyuanVideo 1.5 show improvements across multiple RL algorithms and compare EmoReward with other agentic reward methods under SAGE-GRPO. Ablations and transfer experiments on SD 3.5 and FLUX further support adaptive reward calibration for visual generation.

*Work done during internship at Tencent Hunyuan

‡Project leader †Corresponding author

Project Page

GitHub Repo

Introduction

Reinforcement learning (RL), especially Group Relative Policy Optimization (GRPO), has advanced model post-training in verifiable domains such as code and mathematics, where correctness provides a direct learning signal [20, 54, 59, 84]. In contrast, evaluating video quality, especially emotional expression, often depends on subjective aesthetics and has no uniquely correct target. Visual post-training adapts diffusion and flow-based generation models through RL and preference objectives [3, 16, 33, 49, 69, 81, 92]. Learned evaluators provide scores for image preference and video quality [21, 26, 34, 42, 77, 78], but a visually clear, temporally stable video can match its prompt without conveying the intended emotion. Facial analysis, dynamic expression recognition, and multimodal emotion models supply complementary evidence [2, 4, 8, 30, 89], whose reliability varies with context: facial cues require visible faces, strong motion can undermine restrained sadness, and saturated colors may reflect style rather than emotional intent. This raises a central question: Which emotional evidence should the reward system trust in the current video, and how can it adaptively determine how much weight to assign to that evidence?

Agentic systems coordinate interaction, refine outputs through feedback, and organize evaluation into rubrics [19, 43, 56, 71, 74]. These capabilities motivate adaptive evaluation, yet existing video GRPO methods typically rely on fixed rewards for visual quality, motion quality, and text alignment that inadequately capture emotional intent. Reinforcement learning from human feedback (RLHF) and preference optimization address subjective goals [9, 45, 51, 57], but human annotation is costly at scale, and aesthetic differences complicate the calibration of emotional cues across contexts. Fixed fusion can mix informative, missing, and saturated signals, allowing policy updates to exploit scoring shortcuts rather than improve emotional fidelity. This motivates reward calibration at two complementary levels: assessing which cues are valid within each video, and learning from a completed training run how much those cues should influence the reward in the next run.

We introduce EmoReward, a self-evolving training mechanism for emotional video-generation RL (Figure 1). It distills evidence reliability from reward–outcome traces collected during an actual, completed training run, then uses this experience to guide multi-reward weight allocation in the next run to decide which emotional evidence to trust and how much to weight it. Plan translates emotional intent into an EmoScript specifying affect targets, facial and body cues, and an expression window; Act scores generated videos with heterogeneous primitives, keeping criterion scores separate from evidence validity. Within each video, Reflective Reward Gating calibrates these cues using the training phase, face confidence, semantic conflict, and subtlety, then renormalizes weights over active providers. After a training run, RewardMemory estimates provider reliability for each emotional regime from traces. Providers with stronger agreement are given larger weight multipliers for the next run, with limits on the size of the adjustment. These multipliers and all reward models stay fixed throughout that run; for each new video, reflection further downweights or excludes missing or unsuitable cues before the active scores are combined into a weighted average. EmoReward integrates with several RL algorithms alongside general video rewards, using training experience to calibrate reward allocation without retraining the evaluators.

Figure 1. From fixed scoring to contextual reward calibration. The reward design progresses from a fixed evaluator to a heterogeneous provider pool and then to EmoReward. Reflective Reward Gating selects relevant evidence for the current video; RewardMemory calibrates provider trust from experience. All three stages ultimately return a scalar reward. The reward providers remain frozen while RL updates the video generation model.

We evaluate EmoReward on HunyuanVideo 1.5 [73] with DanceGRPO [81], FlowGRPO [33], and SAGE-GRPO [92], finding higher held-out emotion accuracy and rank scores across all three RL algorithms and higher VideoAlign Overall with FlowGRPO and SAGE-GRPO. Under a shared architecture and training budget, RewardMemory yields the highest reliability selectivity among the seven reward policies. Ablation studies and mechanism analyses examine the contributions of reflective gating, provider pools, and RewardMemory (§ 4.2). Transfer experiments on Stable Diffusion 3.5 and FLUX show gains in selected configurations alongside objective-dependent trade-offs, while user studies show that EmoReward receives the highest proportion of Good ratings and the lowest proportion of Bad ratings for emotional fidelity, temporal coherence, and visual quality.

Our contributions are threefold.

  • We introduce EmoReward, the first self-evolving training mechanism for emotional video generation RL that calibrates reward providers across training runs without updating reward model parameters.

  • We propose RewardMemory, the first memory designed for multi-reward systems in video generation. It distills evidence reliability into bounded fusion weights, enabling iterative calibration of RL reward signals across training runs.

  • We introduce a cross-run self-evolution paradigm for post-training multimodal foundation models: a completed training run informs reward allocation in the next, making training experience reusable at the run level. Extensive experiments across three video RL algorithms and two image backbones, with ablations and human evaluations, validate EmoReward’s effectiveness.

Related Work

From verifiable RL to agentic feedback for non-verifiable rewards. RLHF learns subjective preferences [9, 45], whereas verifiable RL exploits correctness checks [20, 84]. Agentic evaluators [12, 87] and adaptive rubrics [19, 79, 88] structure open-ended feedback. Related approaches route experts [44], refine responses [43], evolve rewards [39, 85], or organize experience [56, 80]. JitRL derives closed-form policy updates from retrieved action advantages [29]. EmoReward instead transfers reward–outcome evidence across completed training runs to calibrate frozen providers, combining sample-level gating with persistent fusion weights.

Agents for visual content creation. Agentic visual creation organizes generation into planning, tool use, and feedback-driven refinement. Video agents coordinate multi-shot narratives, script evaluation, audiovisual planning, and learned camera control [15, 22, 68, 76, 90, 91]. Image agents combine knowledge retrieval [5, 18], learned creation and editing policies [27, 31], and experience reuse [6, 7]. Other approaches jointly improve generation and evaluation [32, 60]. EmoReward naturally achieves a self-evolving mechanism in video generation RL training, rather than building a workflow or retraining reward models.

RLHF for visual generation and the emotion–reward gap. Visual reward models capture preferences and multidimensional quality [26, 34, 42, 72, 75, 77, 78]. Optimization using rewards and preferences extends from images [3, 16, 49, 69] to video and flow models [33, 35, 81, 92], with benchmarks assessing quality and compositional alignment [24, 36, 58]. Related 3D, audiovisual, and objectives for human animation address complementary aspects of visual generation [23, 28, 64, 65]. Motion editing and transfer [41, 62, 63], shape-aware and consistent image editing [37, 40], and pose- and identity-preserving animation [46, 66, 67] offer finer control over visual content. Distillation [17, 25, 83] and feature reuse [38] improve generation efficiency, complementing reward-based post-training. Affect-grounded augmentation [70] and reflective action-conditioned generation [82] further connect feedback with video synthesis. Temporal video modeling and dynamic facial-expression recognition provide additional perceptual foundations [4, 61]. Emotion understanding supplies multimodal evidence [8, 30, 89] and grounded affect representations [2, 14, 52]; EmoReward addresses how their reliability should govern reward fusion across scenes and training runs.

Method

Preliminaries

Multi-reward optimization with GRPO. Let pp be a prompt and xx a generated video from a policy πθ(x∣p)\pi_\theta(x \mid p); a scalar reward is R(x,p)∈RR(x,p) \in\mathbb{R}. For a prompt pp, GRPO samples a group G(p)={xi}i=1G\mathcal{G}(p) = \{x_i\}_{i=1}^{G} and forms standardized group advantages

Ai=R(xi,p)−μGσG+ϵ,μG=1G∑jR(xj,p),A_i = \frac{R(x_i,p)-\mu_{\mathcal{G}}}{\sigma_{\mathcal{G}}+\epsilon}, \qquad\mu_{\mathcal{G}} = \frac{1}{G}\sum_j R(x_j,p),
σG2=1G∑j(R(xj,p)−μG)2.(1)\sigma_{\mathcal{G}}^2 = \frac{1}{G}\sum_j \left(R(x_j,p)-\mu_{\mathcal{G}}\right)^2. \tag*{(1)}

which enter the clipped, KL-regularized objective in Appendix § 13. The policy update depends on reward differences inside the group. A common additive constant cancels exactly; positive rescaling changes only the relative effect of ϵ\epsilon. A coefficient of variation measures dispersion, not policy-learning signal quality; we also assess within-prompt ranking agreement and downstream quality.

Why static emotion fusion fails. Given reward primitives T={1,…,K}\mathcal{T} = \{1,\ldots,K\} with normalized scores sk(x,p)∈[0,1]s_k(x,p) \in[0,1], a fixed fusion is Remostatic=∑kwk0skR_{\mathrm{emo}}^{\mathrm{static}} = \sum_k w_k^0s_k with wk0>0,∑kwk0=1w_k^0 > 0,\sum_k w_k^0 = 1. This assumes a fixed trust allocation across scenes, although facial validity and the relevance of motion depend on the intended emotion. Writing each primitive as a constant plus a group-varying part sk=ck+δks_k = c_k + \delta_k and splitting into an active set A\mathcal{A} and a dead/saturated set D\mathcal{D},

Remostatic=∑k∈Awk0δk+∑k∈Awk0ck+∑k∈Dwk0ck,(2)R_{\mathrm{emo}}^{\mathrm{static}} = \sum_{k\in\mathcal{A}} w_k^0\delta_k + \sum_{k\in\mathcal{A}} w_k^0c_k + \sum_{k\in\mathcal{D}} w_k^0c_k, \tag*{(2)}

the constant terms cancel from the centered GRPO numerator. Removing them alone need not change the standardized advantage. Calibration matters when provider validity, errors, or relative contributions vary across candidates, or when the emotion branch is mixed with another reward. EmoReward addresses these cases through gates and memory-conditioned fusion.

Overview

EmoReward grounds emotional intent, scores visual evidence, and calibrates provider trust:

p→Plane(p),(x,e)→Act{sk,uk},(s,u,e,x)→ReflectRemo,M→EvolvewRM(z).(3)\begin{aligned} p &\xrightarrow{\mathrm{Plan}} e(p), \qquad(x,e) \xrightarrow{\mathrm{Act}} \{s_k,u_k\},\\ (s,u,e,x) &\xrightarrow{\mathrm{Reflect}} R_{\mathrm{emo}}, \qquad\mathcal{M} \xrightarrow{\mathrm{Evolve}} w^{\mathrm{RM}}(z). \tag*{(3)} \end{aligned}

Here e(p)e(p) is an EmoScript, uku_k denotes evidence validity, and zz is the emotional-context signature. Reward-Memory M\mathcal{M} stores completed reference-run traces; Evolve produces wRM(z)w^{\mathrm{RM}}(z) and exports the corresponding regime multipliers mk(r)m_k(r) (§ 3.6). Reflect combines these fixed multipliers with current-video gates. The resulting emotion reward is mixed with a general video reward:

Rtotal(x,p)=αRvid(x,p)+βRemo(x,p)α+β,(4)R_{\mathrm{total}}(x,p)=\frac{\alpha R_{\mathrm{vid}}(x,p)+\beta R_{\mathrm{emo}}(x,p)}{\alpha+\beta}, \tag*{(4)}

where RvidR_{\mathrm{vid}} is the evaluation score supplied by the original reward system, for example a general video-quality evaluator [34]; α,β≥0\alpha,\beta\ge0 control its mixture with the calibrated emotion reward, with α+β>0\alpha+\beta>0. Figure 2 connects these stages through their shared fusion interface.

EmoReward calibration process

Figure 2. EmoReward calibrates emotional evidence within videos and across training runs. Plan translates intent into evaluation criteria, and Act obtains provider scores and evidence validity. Reflect combines current-video gates with memory-derived multipliers and normalizes over active providers. Evolve estimates contextual reliability from a completed reference run to construct the fixed multipliers used in subsequent training. The calibrated emotion reward complements the original video reward; GRPO updates the video generation model while the providers and memory table remain fixed.

Plan and Act: Emotional Grounding

Plan: prompt-conditioned criteria. The planner converts the prompt into an EmoScript independently of RewardMemory:

e(p)=(v⋆,a⋆,qAU⋆,b⋆,Ω).(5)e(p)=(v^\star,a^\star,q^\star_{\mathrm{AU}},b^\star,\Omega). \tag*{(5)}

The implemented representation contains scalar valence and arousal targets, target facial AU intensities, body cues, and an expression window. The memory regime is inferred later from observed context. Continuous VAD trajectories and dominance targets are outside this implementation. The EmoScript separates intended affect from its observable cues. For example, restrained sadness need not imply large motion or an exaggerated face: target intensities, body cues, and the expression window define what to assess, while reliability is determined from the generated evidence and completed-run experience.

Act: observation and scoring. Act evaluates a video already sampled from πθ(⋅∣p)\pi_\theta(\cdot\mid p); it does not generate or revise the video. Each provider returns

(sk,uk)=Sk(x,e(p)),sk,uk∈[0,1],(6)(s_k,u_k)=S_k(x,e(p)), \qquad s_k,u_k\in[0,1], \tag*{(6)}

where SkS_k is provider kk’s scoring operator, which aggregates frame-level measurements or evaluates the whole clip. The seven fusion providers comprise a VLM judge, facial AUs, body cues, temporal evidence, CLIP-AUTT, R1-Omni, and motion naturalness; chromatic atmosphere and kinetic intensity are auxiliary readouts, giving nine observation channels. Score sks_k measures criterion agreement, while uku_k marks usable evidence. A missing face channel’s neutral default is not evidence and is excluded through its validity mask. Auxiliary readouts do not enter the seven-provider calibration policy. Appendix § 10 specifies scoring interfaces; Figure 6 illustrates the toolkit studied in § 4.2.

Figure 6. Figure 6 Ablation of provider-pool construction. Three frames per clip compare four configurations on the same prompt. See Appendix § 15 for quantitative results.

Reflect: Instance-Level Calibration

Reflect computes provider-specific gates from the current evidence, prompt, and training step. The factorization is

qk=qkphaseqkconflictqkfaceqksubtleuk,qk∈[0,1],(7)q_k=q_k^{\mathrm{phase}}q_k^{\mathrm{conflict}}q_k^{\mathrm{face}}q_k^{\mathrm{subtle}}u_k,\qquad q_k\in[0,1], \tag*{(7)}

with neutral factors for channels to which a condition does not apply. Phase denotes the training-step warmup, face gating controls face-dependent evidence, conflict gating attenuates disagreement between semantic providers, and subtlety gating restrains intensity-sensitive cues. Motion naturalness uses its observation confidence. These controls are computed, not learned (Appendix § 11).

Evolve: RewardMemory

Algorithm 1 EmoReward reward computation with experience-calibrated weights

Require: prompt pp, generated video xx, fixed table TM\mathcal{T}_{\mathcal{M}} from a completed reference run, base weights w0w^0

  1. Build EmoScript e(p)e(p)

  2. Compute primitive scores and validity {sk,uk}k=1K\{s_k,u_k\}_{k=1}^{K} ▷ Eq. (6)

  3. Compute instance gates {gk}k=1K\{g_k\}_{k=1}^{K} ▷ Eq. (7)

  4. Form state signature z←z(p,x,e,s,g)z \gets z(p,x,e,s,g)

  5. Look up regime r(z)r(z) in TM\mathcal{T}_{\mathcal{M}} ▷ Appendix § 12

  6. if a supported regime or global entry is available then

  7. Read fixed multipliers mkm_k, constructed by Eq. (10)

  8. else

  9. mk←1m_k \gets1 ▷ cold start: fall back to base

  10. end if

  11. Form active weights ak←wk0mkgka_k \gets w_k^0m_kg_k

  12. Fuse active primitives with renormalization →Remo\to R_{\mathrm{emo}} ▷ Eq. (8)

  13. Compute total reward Rtotal(x,p)R_{\mathrm{total}}(x,p) ▷ Eq. (4)

  14. Log the reward trace for inspection; keep TM\mathcal{T}_{\mathcal{M}} fixed

  15. return Rtotal(x,p)R_{\mathrm{total}}(x,p)

Algorithm 2 RewardMemory construction from reflective reward traces

Require: Traces M\mathcal{M} from a completed reference run, provider count KK, temperature τ\tau, log-multiplier cap cc, minimum support nmin⁡n_{\min}

  1. for each context and the global fallback do

  2. Select matching traces; use the global set if the context has fewer than nmin⁡n_{\min} traces

  3. for each provider kk do

  4. ρk←Corr⁡(si,k,yi)\rho_k \gets\operatorname{Corr}(s_{i,k},y_i); use zero for constant scores

  5. end for

  6. ρˉ←K−1∑jρj\bar{\rho} \gets K^{-1}\sum_j\rho_j

  7. for each provider kk do

  8. mk←exp⁡(clip⁡((ρk−ρˉ)/τ,−c,c))m_k \gets\exp(\operatorname{clip}((\rho_k-\bar{\rho})/\tau,-c,c))

  9. end for

  10. Store the context multipliers

  11. end for

  12. Export once; freeze the table for a subsequent generator-training run

Evidence-dependent trust. The phase factor schedules reward evidence according to training progress, separately from the within-video expression window. Face gating limits channels requiring facial evidence; conflict and subtlety gating temper disagreeing semantic assessments and intensity-sensitive cues. Their provider-specific application changes relative influence; a shared positive attenuation would largely cancel under normalization.

Composition with experience. For a fixed base weight wk0w_k^0, define the effective contribution ak(x,p)=wk0gk(x,p)mk(r(x,p))a_k(x,p)=w_k^0g_k(x,p)m_k(r(x,p)). Reflective Reward Gating uses mk=1m_k=1; EmoReward uses the fixed table produced by Evolve. With A={k:ak>ϵg}\mathcal{A}=\{k:a_k>\epsilon_g\}, active-provider normalization gives

Remo(x,p)=∑k∈Aak(x,p)sk(x,p)∑k∈Aak(x,p)+ϵ.(8)R_{\mathrm{emo}}(x,p)=\frac{\sum_{k\in\mathcal{A}}a_k(x,p)s_k(x,p)}{\sum_{k\in\mathcal{A}}a_k(x,p)+\epsilon}. \tag*{(8)}

Invalid evidence receives no mass. A common positive factor cancels except through the stabilizer; calibration acts through relative provider contributions and differences between videos. Regime rr selects a table entry; gate gkg_k controls current evidence validity. A provider favored by memory still contributes nothing when its current evidence is invalid. With no memory entry, unit multipliers recover Reflect-only fusion; the scoring interfaces and the policy objective remain the same.

Evolve: RewardMemory

RewardMemory treats reward allocation as a decision over emotional context zz: the providers are actions, their fusion weights form a policy, and a stored outcome yy supplies a non-parametric reliability signal. Experience is collected during a completed reference training run with Reflective Reward Gating. Each trace mi=(zi,{sik},{gik},wi,Ri,yi)m_i = (z_i, \{s_i^k\}, \{g_i^k\}, w_i, R_i, y_i) records context, scores, gates, weights, and outcomes. Evolve groups these traces into a regime neighborhood N(z)\mathcal{N}(z), with a global fallback for insufficient support, and computes

ρk(z)=Cov⁡i∈N(z)(sik,yi)σskσy+ϵ,\rho_k(z) = \frac{\operatorname{Cov}_{i \in\mathcal{N}(z)}(s_i^k, y_i)}{\sigma_{s^k}\sigma_y + \epsilon},
Akrel(z)=ρk(z)−1K∑jρj(z).(9)A_k^{\mathrm{rel}}(z) = \rho_k(z) - \frac{1}{K}\sum_j \rho_j(z). \tag*{(9)}

Reliability uses raw provider scores; constant scores receive zero correlation. It measures agreement with the reference outcome, not independent emotional fidelity. The centered advantage tilts the base allocation through bounded multipliers:

Δk=clip⁡(Akrel/τ,−c,c),\Delta_k = \operatorname{clip}(A_k^{\mathrm{rel}}/\tau, -c, c),
wkRM(z)=wk0eΔk∑jwj0eΔj.(10)w_k^{\mathrm{RM}}(z) = \frac{w_k^0 e^{\Delta_k}}{\sum_j w_j^0 e^{\Delta_j}}. \tag*{(10)}

This is the closed-form optimum of a KL-regularized allocation with clipped utilities (Appendix § 13). The exported mk(r)=eΔkm_k(r) = e^{\Delta_k} compose with current instance gates in Eq. (8). The table is fixed throughout the next training run: gradients update the video generation model alone, and the table is read-only. Algorithms 1 and 2 separate reward computation from between-run memory construction. Appendix § 12 explains context grouping and table construction; Appendix § 16 provides a trace-based worked example.

Experiments

Experimental settings. We use HunyuanVideo 1.5 with DanceGRPO, FlowGRPO, and SAGE-GRPO, fusion α=β=0.5\alpha= \beta= 0.5, batch size 2, and group size G=4G = 4. We use Qwen3-VL to create emotional prompts for training and use Claude to newly create 302 prompts in same style for evaluation (Appendix § 8). GT denotes the pretrained video generation model without RL. The controlled seven-policy comparison retains its separate evaluation protocol. VideoAlign supplies RvidR_{\mathrm{vid}} and the reference outcome yy. Memory calibration uses K=7K = 7, τ=0.5\tau= 0.5, a log-multiplier cap c=1c = 1, and minimum context support nmin⁡=20n_{\min} = 20.

Main Experiments

Quantitative evaluation. We compute Directional Emotion Accuracy (DEA) and Target Emotion Rank Score (TERS) with a frozen CLIP ViT-H/14 encoder [53], which is different from the separate CLIP-AUTT model [86] used as the facial reward during training (details in Appendix § 14.§ 3). Table 2 compares EmoReward with each RL algorithm’s base reward and includes the pretrained video generation model (GT) as a reference. All three RL baselines score below GT on DEA and TERS. EmoReward improves both metrics under every RL algorithm, with TERS gains of 0.016–0.031. FlowGRPO and SAGE-GRPO also surpass GT on both metrics. The gains have the same direction with a second frozen CLIP encoder. FlowGRPO and SAGE-GRPO improve on all four VideoAlign measures. Both visual metrics increase with EmoReward under SAGE-GRPO, with mixed changes across the other algorithms. Figure 4 illustrates the changes in emotional expression across all three algorithms.

MethodDEATERSOverallVQMQTAPickScoreCLIPScore
GT (reference)0.655±0.0290.7904±0.0145-1.1938±0.0520-0.5899±0.0255-0.4289±0.0220-0.1750±0.03630.7059±0.00180.5623±0.0037
DanceGRPO0.605±0.0300.7640±0.0153-1.2812±0.0513-0.6392±0.0230-0.3867±0.0235-0.2554±0.03570.7177±0.00180.5675±0.0036
+ EmoReward0.632±0.0300.7824±0.0149-1.3930±0.0559-0.6279±0.0284-0.3833±0.0243-0.3818±0.03400.6986±0.00170.5599±0.0037
FlowGRPO0.621±0.0300.7862±0.0144-0.7719±0.0569-0.3567±0.0290-0.2753±0.0271-0.1398±0.03610.7061±0.00190.5634±0.0036
+ EmoReward0.670±0.0290.8018±0.0142-0.5376±0.0556-0.3398±0.0291-0.2302±0.02750.0323±0.03560.7115±0.00180.5605±0.0037
SAGE-GRPO0.617±0.0300.7881±0.0143-0.8269±0.0532-0.4106±0.0261-0.3275±0.0263-0.0888±0.03600.7091±0.00190.5624±0.0036
+ EmoReward0.674±0.0290.8193±0.0132-0.2624±0.0600-0.1710±0.0304-0.0926±0.03240.0012±0.03640.7107±0.00180.5663±0.0035

Table 2. EmoReward across video RL algorithms. DEA uses the 261 prompts with a usable opposite emotion. TERS ranks the target emotion on all 302 prompts. All metrics report mean ± SEM across prompts. GT is the pretrained no-RL reference. Bold marks an increase over its corresponding RL baseline.

Qualitative comparison across video RL algorithms

Figure 4. Qualitative comparison across video RL algorithms. Each pair shares a prompt and seed. Red boxes mark the hands and facial cues. (Top) A bowed head and hands held over the face convey grief. (Middle) The reporter’s expression tightens as she reacts to the screen. (Bottom) A broad smile gives way to a fixed stare.

Emotional expression in generated videos

Figure 3. Emotional expression in generated videos. Eight clips are shown at three moments each. (Top) A performer looks up from a letter. (Middle rows) Gaze, gesture, and expression convey emotion in scenes such as a violin performance and a rooftop piano scene. (Bottom) Laughter fades into a quiet gaze.

Qualitative analysis. Figure 4 compares base and EmoReward results for each RL algorithm with matched prompts and seeds. For DanceGRPO, the man stays bowed with his face covered with better visual effects, and for FlowGRPO the reporter’s strained expression is clearer from a more frontal angle, while under SAGE-GRPO the musician’s smile fades into a fixed stare. These examples complement the clip-level emotion scores by showing the order in which the visible cues emerge.

Ablations and Mechanism Analysis

Emotion-category analysis. Figure 5 compares GT with SAGE-GRPO plus EmoReward across eight emotion categories. All four VideoAlign measures increase in every category, while PickScore and CLIPScore vary. This comparison captures the effect of post-training as a whole.

Figure 5. Figure 5 Emotion-category quality. GT (gray) versus SAGE-GRPO with EmoReward (red); mean ±\pm SEM. Axes are truncated; categories overlap within the evaluated set.

Reward-policy ablation. We compare seven reward policies with a fixed backbone and training budget (Table 1). RewardMemory leads in Sel. (0.428), GD, and GR, while Reflect leads in EF, the ImageReward preference proxy. SNR∗\mathrm{SNR}^{*} measures relative dispersion. The predefined core and emo scores are descriptive summaries, since min–max scaling can magnify small differences (Appendix § 14).

Table 1 Reward-policy ablation under SAGE-GRPO. Controlled policies share the backbone and training budget. Citations identify components. SNR∗^{*} is relative dispersion. Bold marks diagnostic and pillar maxima. Core and emo are shown for reference.

Reward methodSNR∗^{*}Sel.↑\uparrowEF↑\uparrowGD↑\uparrowGR↑\uparrowcore↑\uparrowemo↑\uparrow
VideoAlign [34]0.3780.3070.5180.0000.1970.2380.308
Single provider [89]2.4030.3140.4540.3050.1470.3020.340
Random routing [74]1.1050.3400.0000.1090.1460.0850.064
Round-robin [74]1.2170.3760.2910.1260.3810.2660.272
Fixed fusion [78]0.7730.3710.0260.0600.6530.2460.191
EmoReward (w/o RewardMemory)0.4510.2931.0000.0110.0000.3370.503
EmoReward (w/ RewardMemory)7.0220.4280.7671.0001.0000.9220.884

Table 1.

Reward-provider pool. Figure 6 compares VideoAlign, semantic emotion, motion naturalness, and a balanced pool. Weights and curricula co-vary with the pool (Appendix § 15). Seven channels enter memory calibration. Chromatic and kinetic cues supply auxiliary readouts.

Memory-conditioned trust. The largest global multipliers go to macro (1.833) and motion (1.510). Three constant channels receive 0.778. The gates still exclude invalid evidence. Macro shares an evaluator with the reference outcome, so its correlation reflects objective agreement. Appendices § 12 and § 16 describe and illustrate the allocation.

Generalization to image backbones

We test SD3.5 and FLUX across three RL algorithms with EmoReward. EmoReward improves HPSv3 for SD3.5, while FLUX’s baseline still leads on the same metric. We tested on several prompts and find EmoReward enhances the text following ability and reduce deformities. Appendix § 17 details the settings and qualitative comparisons.

Human evaluation

Eighteen participants rated three methods on 30 prompts for emotion fidelity, temporal coherence, and visual quality. They used the same Bad/Normal/Good scale for each criterion. SAGE-GRPO receives more Good and more Bad ratings than GT on all three criteria. EmoReward has the highest Good shares (60.0%, 54.3%, 57.2%), compared with 53.0%, 51.5%, and 45.0% for SAGE-GRPO, and the lowest Bad shares (Table 3). The same participants rated multiple prompts across methods, and Appendix § 9 details the protocol.

Table 3 Human evaluation. Rating percentages from 18 participants on 30 prompts. Each method and criterion block contains 540 judgments. Bold marks the highest Good share per criterion; percentages are rounded to one decimal place.

MethodEmotion FidelityTemporal CoherenceVisual Quality
Bad ↓\downarrowNormalGood ↑\uparrowBad ↓\downarrowNormalGood ↑\uparrowBad ↓\downarrowNormalGood ↑\uparrow
GT21.9%34.8%43.3%23.5%37.8%38.7%25.2%38.5%36.3%
SAGE-GRPO31.5%15.6%53.0%28.9%19.6%51.5%30.9%24.1%45.0%
SAGE-GRPO + EmoReward15.4%24.6%60.0%20.2%25.6%54.3%15.9%26.9%57.2%

Table 3.

Conclusion

In this paper, we proposed EmoReward, a self-evolving reward calibration framework for emotional video generation reinforcement learning. EmoReward combines instance-level Reflective Reward Gating with RewardMemory, which transfers provider-reliability estimates from a completed training run to subsequent optimization without updating reward-model parameters. Experiments across video optimizers, complemented by ablations, image-transfer analyses, and human ratings, support the utility of adaptive calibration while identifying objective-dependent trade-offs. These findings suggest that completed-run experience can inform the allocation of trust among heterogeneous reward providers, providing a foundation for further research on reward-system evolution across training.

LLM Usage Statement

OpenAI ChatGPT and Codex assisted with manuscript editing and organization, checks of the mathematical exposition, and preparation of figure layouts and LaTeX source. We used Qwen3-VL to create emotional prompts for training and Claude to author a separate dataset of prompts for evaluation. The main comparison uses 302 prompts from this dataset (Appendix § 8). The vision–language and generative models used in the reward pipeline and experiments are described in the paper and supplement. Experimental comparison images and video frames are outputs of the evaluated models; no additional media were generated to illustrate experimental outcomes. The authors are responsible for the scientific claims, evidence, and final manuscript.

Limitations and Future Work

Beyond emotional video generation. EmoReward studies a training-adaptive reward system that transfers evidence about provider reliability between generator-training runs without updating reward-model parameters. The present evaluation focuses on emotion, where different scenes call for different combinations of facial, gestural, semantic, and motion evidence. A broader application would require a task-appropriate provider pool and a meaningful outcome for calibration. Action fidelity and audio–visual correspondence are natural visual extensions; language models and tool-use agents could similarly combine semantic judges, execution checks, and trajectory-level assessments. Whether the same calibration rule transfers effectively remains an empirical question, especially when providers share errors or outcomes favor only part of the task.

Iterative evolution across training runs. A completed training run can be treated as a unit of experience for its successor. Repeating this procedure could produce a sequence of generators and reward-allocation tables, with each generation contributing new traces. Our experiments evaluate a reference-to-calibrated transition, not sustained improvement across many generations. Such an extension must address distribution shift, stale evidence, and feedback loops that reinforce a proxy’s errors. Retaining diverse historical traces and evaluating held-out outcomes would be important tests of whether repeated calibration remains useful.

Model-level human feedback. The trace representation also suggests a way to carry human assessments of a completed model into subsequent training. For example, a reviewer who finds movement too weak could request greater emphasis on suitable motion providers in the affected contexts; excessive motion could motivate the opposite adjustment. This would complement sample-level preference feedback with structured assessments of a model’s final behavior, translated into bounded changes to the next run’s reward allocation. Mapping subjective feedback to providers and regimes, reconciling different raters, and preventing compensating failures remain open design questions. The ordinal study reported here evaluates outputs; it does not implement this proposed feedback mechanism.

Training and Evaluation Prompts Preparation

Training corpus. We use Qwen3-VL-30B-A3B [1, 50] to write emotional video prompts. Our 62-category taxonomy draws on Plutchik’s eight primary emotions and their derivatives, Cowen and Keltner’s 27–28-category semantic-space studies, Shaver’s three-level hierarchy as presented in Parrott’s collection, Ekman and Friesen’s Facial Action Coding System (FACS), Hume AI’s facial expression model, and GoEmotions’ 28 labels including neutral [10, 11, 13, 14, 47, 48, 55]. Each category has a Chinese and an English label and specifies valence, arousal, facial action units, body cues, and temporal dynamics. We assign a seed to each row, sample a category from that seed, and use the corresponding label to condition prompt generation. The resulting corpus contains 10,000 distinct prompts to diversify rollouts across training. Each prompt is used at most once in a complete online on-policy training run.

Example. For row_seed=478163327row\_seed = 478163327, we draw Love/Affection, with positive valence and medium arousal. Its required facial actions are AU6 (cheek raiser) and AU12 (lip corner puller). The body cues are intimate distance (0–45 cm), gentle stroking, and a soft, nonthreatening gaze. The target expression develops slowly and sustains a sense of warmth. The system prompt below is shown in English translation.

Write a visual description for text-to-video generation in Simplified Chinese. Output a single paragraph with no title, list, or quotation marks. Center the scene on people, with visible facial expressions, gaze, lip movements, and body language. Show subtle shifts in emotion and microexpressions. Convey love or affection through the scene and the performances and avoid repeating the emotion label. Write roughly 180–350 Chinese characters. Use cinematic language to describe one or two of framing, camera angle, lighting, and camera movement, and establish the atmosphere of the setting. Exclude prohibited content and excessively graphic violence. Output only the Chinese paragraph, with no explanatory preface or closing note.

Random seed for inspiration only (do not reproduce the number) 156744132

Showing the emotion through acting gives the facial and body reward providers visible cues to score. Varying the inspiration seed encourages different scenes when the same category is sampled again. An example rollout prompt is shown below in English. The slanting sunlight of dusk filters through the old window lattice, bathing the room in a warm orange glow. A middle-aged couple sits on worn wicker chairs; she is bent over, mending an old sweater, her fingers deftly guiding the needle and thread. He watches her quietly, his fingers absently tracing the rim of his cup. . . Suddenly, she looks up at him, a faint smile lifting the corners of her lips, fine lines crinkling at the edges of her eyes—gentle ripples of time-worn tenderness. . . he reaches out, gently brushing a strand of her hair, his touch slow and reverent. . . they share a glance, a silent understanding flowing between them. The camera slowly pushes in, focusing on the delicate smile lines at the corner of her eye and the quiet, shimmering light deep within his gaze.

Evaluation corpus. We used Claude to create a separate emotional prompt dataset for evaluation, following the training corpus’s emotion specifications and cinematic style. These prompts were held out from RL training. Table 2 uses the same 302-prompt subset for each RL algorithm and its matched base and EmoReward runs. Appendix § 14.§ 3 describes how the automatic metrics are aggregated, including DEA’s requirement for a usable opposite emotion.

Human Evaluation: Ordinal Ratings

Study and comparison groups. The study includes 18 participants and 30 prompts, with one video per prompt from each of three methods: the pretrained video generation model without RL (GT), SAGE-GRPO, and SAGE-GRPO with EmoReward. GT denotes the pretrained model baseline. Participants consented to take part in the study. Participants assessed each output separately on three criteria using Bad, Normal, or Good. The supplied aggregate results contain 540 judgments for each method–criterion combination, consistent with the full participant–prompt product. Across all three methods and criteria, there are 4,860 categorical judgments. These ordinal counts are separate from the earlier pairwise preference protocol; pairwise choices are not converted into categorical ratings.

Criteria. Emotion fidelity concerns how convincingly the video conveys the prompt’s intended affect, including whether expression and gesture fit its emotional tone. Temporal coherence concerns continuity of expression, identity, and movement through the clip. Visual quality concerns perceptual detail, anatomy, and visible artifacts. The categories distinguish poor, intermediate, and good quality on each axis. They are rated separately rather than collapsed into a single preference or an assumption that stronger emotional intensity is always better.

Aggregation and rounding. For method mm, criterion cc, and rating category bb, the percentage is:

p^m,c,b=100Nm,c,b∑b′∈{Bad,Normal,Good}Nm,c,b′,∑b′Nm,c,b′=540.(11)\hat{p}_{m,c,b}=100\frac{N_{m,c,b}}{\sum_{b'\in\{\mathrm{Bad},\mathrm{Normal},\mathrm{Good}\}}N_{m,c,b'}},\qquad\sum_{b'}N_{m,c,b'}=540. \tag*{(11)}

Table 3 rounds each percentage independently to one decimal place and bolds only the highest Good percentage within each criterion. Rounding may make a three-category total differ from 100.0% by 0.1 percentage points. The raw integer counts are given in Table 4 and were not rounded before normalization. All nine aggregate blocks have the stated denominator; the totals alone do not establish participant-level completeness or the absence of duplicate records.

MethodEmotion fidelityTemporal coherenceVisual quality
BadNormalGoodBadNormalGoodBadNormalGood
GT118188234127204209136208196
SAGE-GRPO17084286156106278167130243
SAGE-GRPO + EmoReward8313332410913829386145309

Table 4. Table 4 Aggregate ordinal counts. Each method–criterion block contains 540 ratings. The counts reproduce the percentages in main-paper Table 3.

Reading the distributions. Relative to SAGE-GRPO, EmoReward’s Good share increases by 7.0, 2.8, and 12.2 percentage points for emotion fidelity, temporal coherence, and visual quality, respectively. These differences are computed from the unrounded counts. Its Bad share falls from 31.5% to 15.4% for emotion fidelity, from 28.9% to 20.2% for temporal coherence, and from 30.9% to 15.9% for visual quality. The complete distribution matters: compared with GT, SAGE-GRPO has more Good ratings but also more Bad ratings on every criterion, so a Good-only summary would omit part of the observed pattern.

Interface for ordinal human evaluation

Figure 7. Interface for ordinal human evaluation. The interface presents the prompt and generated video with independent ratings of emotional fidelity, temporal coherence, and visual quality, each using three ordered categories: Bad, Normal, and Good. This reconstruction illustrates the reported rating protocol rather than the original data-collection interface.

Statistical scope. The present report uses the supplied aggregated counts. It does not treat repeated judgments from the same participant or prompt as independent observations. Participant- and prompt-level records are needed for uncertainty estimates that respect both sources of dependence; no confidence interval or significance claim is inferred from the aggregate margins. The aggregates do not document recruitment, presentation order, blinding, exclusions, or prompt-selection provenance. Those collection details must be established from the study records rather than inferred from the rating totals.

Act: Provider Interface and Reference Scoring Rules

This appendix specifies the scoring operators used by the provider interface in Eq. (6). The six primitive families (facial expression; semantic emotion understanding; body and temporal evidence; chromatic atmosphere; kinetic intensity; motion naturalness) are instantiated by nine observation channels; Table 5 maps each family to its provider(s), the backing estimator, and the defining equation. Throughout, TT is the number of sampled frames, F⊂{1,…,T}\mathcal{F} \subset\{1,\ldots,T\} the face-detected subset, ωt\omega_t the EmoScript time weight (per-window kernels ψℓ(t)=1[t∈ωℓ]/(∑t′1[t′∈ωℓ]+ϵ)\psi_{\ell}(t)=\mathbf{1}[t\in\omega_{\ell}]/(\sum_{t'}\mathbf{1}[t'\in\omega_{\ell}]+\epsilon) over Ω={[tℓs,tℓe]}\Omega=\{[t_{\ell}^{s},t_{\ell}^{e}]\}), and κ(x;σ)=exp⁡(−max⁡(0,x)/σ)\kappa(x;\sigma)=\exp(-\max(0,x)/\sigma) a bounded exponential score. Each provider kk returns a score sk∈[0,1]s_k\in[0,1] and a validity signal uku_k.

FamilyProviderEstimator / signalEq.
FacialAU matchingFacial action-unit head12
FacialCLIP-AUTTTemporal AU–text similarity13
SemanticR1-OmniValence–arousal estimation14
SemanticMacroAlignment and AU compliance15
BodyBody cuesBody-marker satisfaction16
TemporalPeak alignmentAU peak versus target window18
AppearanceChromaticSaturation and warm hue19
AppearanceKineticInter-frame differences20
MotionNaturalnessOptical-flow consistency21

Table 5. Table 5 Primitive family, provider, backing estimator, and defining equation. The two appearance channels (chromatic, kinetic) are always-on lightweight signals; the remaining seven form the memory support whose per-provider reliability is reported in the calibration analysis (Table 8).

ProviderMacroMotionCLIPR1-OmniMicroBodyTemporal
ρk\rho_k0.4290.3320.0730.0470.0000.0000.000
mkm_k1.8331.5100.8990.8540.7780.7780.778

Table 8. Table 8 RewardMemory calibration. Provider reliability and unnormalized multipliers, estimated from 3,232 reflective reward traces (τ=0.5\tau= 0.5, c=1c = 1). Macro includes VideoAlign information, so its association is partly shared with the outcome.

Facial expression: AU matching and expression similarity

AU matching (micro). A face detector followed by a multi-task action-unit head emits eight AU intensities per detected face, q^bj,t∈[0,1]\hat{q}_{bj,t} \in[0,1] for j∈AFACS={1,2,4,6,9,12,25,26}j \in\mathcal{A}_{\mathrm{FACS}} = \{1,2,4,6,9,12,25,26\}. Against the EmoScript target intensities qj⋆q_j^\star we score agreement by a per-AU intersection-over-union overlap (rather than an ℓp\ell_p kernel, so a target and an observation are rewarded for occupying the same activation interval), averaged over face frames:

IoU⁡(qj⋆,q^j,t)=min⁡(qj⋆,q^j,t)max⁡(qj⋆,q^j,t)+ϵ,smicro=1∣A⋆∣∑j∈A⋆1∣F∣∑t∈FIoU⁡(qj⋆,q^j,t),(12)\operatorname{IoU}(q_j^\star,\hat{q}_{j,t}) = \frac{\min(q_j^\star,\hat{q}_{j,t})}{\max(q_j^\star,\hat{q}_{j,t})+\epsilon}, \qquad s_{\mathrm{micro}} = \frac{1}{\lvert A^\star\rvert}\sum_{j\in A^\star}\frac{1}{\lvert F\rvert}\sum_{t\in F}\operatorname{IoU}(q_j^\star,\hat{q}_{j,t}), \tag*{(12)}

with AU compliance and validity cAU=1∣Areq⋆∣∑j∈Areq⋆1[max⁡t∈Fq^j,t>θAU]c_{\mathrm{AU}} = \frac{1}{\lvert A^\star_{\mathrm{req}}\rvert}\sum_{j\in A^\star_{\mathrm{req}}}\mathbf{1}[\max_{t\in F}\hat{q}_{j,t}>\theta_{\mathrm{AU}}] and umicro=∣F∣/Tu_{\mathrm{micro}}=\lvert F\rvert/T. The provider is inactive when no frame yields a face with AU readings. In our implementation, θAU=0.3\theta_{\mathrm{AU}}=0.3.

Expression similarity (clip_autt). A temporal vision–language encoder embeds a TclipT_{\mathrm{clip}}-frame clip into v\mathbf{v} and compares it to NauN_{\mathrm{au}} action-unit text-prompt embeddings {tj}\{ \mathbf{t}_j \} by cosine similarity, mapping the mean similarity from [−1,1][-1,1] to [0,1][0,1]:

sˉau=1Nau∑jcos⁡(v,tj),sclip=12(sˉau+1).(13)\bar{s}_{\mathrm{au}}=\frac{1}{N_{\mathrm{au}}}\sum_j \cos(\mathbf{v},\mathbf{t}_j), \qquad s_{\mathrm{clip}}=\frac{1}{2}(\bar{s}_{\mathrm{au}}+1). \tag*{(13)}

An auxiliary discrete head additionally yields a negative-affect probability pneg=softmax⁡(logits⁡)negp_{\mathrm{neg}}=\operatorname{softmax}(\operatorname{logits})_{\mathrm{neg}}, from which the scorer derives v^=1−pneg\hat{v}=1-p_{\mathrm{neg}} and a^=min⁡(1,λapneg+ba)\hat{a}=\min(1,\lambda_a p_{\mathrm{neg}}+b_a); the linear rescale in Eq. (13) (rather than clip) preserves the ordering on clips where sˉau<0\bar{s}_{\mathrm{au}}<0. In our implementation, Tclip=16T_{\mathrm{clip}}=16, Nau=47N_{\mathrm{au}}=47, λa=1.5\lambda_a=1.5, and ba=0.2b_a=0.2.

Semantic emotion understanding: Valence–arousal reading and the alignment seed

Valence–arousal reading (r1omni). A multimodal emotion-understanding model produces a free-text emotion description that is parsed to an emotion label and continuous valence–arousal estimates (v^,a^)(\hat{v},\hat{a}). Given EmoScript targets (v⋆,a⋆)(v^\star,a^\star), the reward supports three modes (default: valence):

sr1={12[(1−∣v^−v⋆∣)+(1−∣a^−a⋆∣)],ALIGNMENT,v^,VALENCE,12(v^+a^),MEAN.(14)s_{\mathrm{r1}} = \begin{cases} \frac{1}{2}\left[(1-\lvert\hat{v}-v^\star\rvert)+(1-\lvert\hat{a}-a^\star\rvert)\right], & \text{ALIGNMENT},\\ \hat{v}, & \text{VALENCE},\\ \frac{1}{2}(\hat{v}+\hat{a}), & \text{MEAN}. \end{cases} \tag*{(14)}

Text-alignment seed (macro). When a general video–text alignment model is available it blends its text-alignment head TA∈[0,1]\mathrm{TA}\in[0,1] with AU compliance weighted by face rate wface=min⁡(face rate,1)w_{\mathrm{face}}=\min(\text{face rate},1); otherwise a heuristic reads compliance directly:

smacro={λTATA+(1−λTA)cAUwface,VideoAlign reflector,bmacro+(1−bmacro)cAUπconf,heuristic (active iff face rate ≥θmacro).(15)s_{\mathrm{macro}} = \begin{cases} \lambda_{\mathrm{TA}}\mathrm{TA}+(1-\lambda_{\mathrm{TA}})c_{\mathrm{AU}}w_{\mathrm{face}}, & \text{VideoAlign reflector},\\ b_{\mathrm{macro}}+(1-b_{\mathrm{macro}})c_{\mathrm{AU}}\pi_{\mathrm{conf}}, & \text{heuristic (active iff face rate }\geq\theta_{\mathrm{macro}}\text{)}. \end{cases} \tag*{(15)}

In our implementation, λTA=0.6\lambda_{\mathrm{TA}} = 0.6, bmacro=0.3b_{\mathrm{macro}} = 0.3, and θmacro=0.1\theta_{\mathrm{macro}} = 0.1. The confidence πconf\pi_{\mathrm{conf}} belongs to the heuristic reflector. Because the alignment-seeded branch shares an evaluator with the reference outcome, its correlation cannot serve as independent emotion validation.

Body and temporal evidence (body, temporal)

The body provider counts each EmoScript body marker m∈M⋆m \in\mathcal{M}^{\star} satisfied by at least one frame; the temporal provider locates the frame t⋆t^{\star} of peak expected-AU energy and scores its distance to the EmoScript window [ts,te][t_s,t_e]:

sbody=1∣M⋆∣∑m∈M⋆1[∃t:bm,t=1],(16)s_{\mathrm{body}} = \frac{1}{|\mathcal{M}^{\star}|} \sum_{m \in\mathcal{M}^{\star}} \mathbf{1}[\exists t: b_{m,t} = 1], \tag*{(16)}
t⋆=arg max⁡j∈A⋆q^j,t,(17)t^{\star} = \underset{j \in\mathcal{A}^{\star}}{\operatorname{arg\,max}} \widehat{q}_{j,t}, \tag*{(17)}
stemporal={1,ts≤t⋆≤te,max⁡(0,1−min⁡(∣t⋆−ts∣,∣t⋆−te∣)max⁡(te−ts,δΩ)),else.(18)s_{\mathrm{temporal}} = \begin{cases} 1, & t_s \le t^{\star} \le t_e,\\ \max\left(0,1-\dfrac{\min(|t^{\star}-t_s|,|t^{\star}-t_e|)}{\max(t_e-t_s,\delta_{\Omega})}\right), & \text{else.} \end{cases} \tag*{(18)}

The window stabilizer is δΩ=0.1\delta_{\Omega} = 0.1 in our implementation. These scores require their corresponding observations; unavailable evidence receives the neutral fallback sneutrals_{\mathrm{neutral}} and validity zero. We use sneutral=0.5s_{\mathrm{neutral}} = 0.5. In the analyzed traces, the micro, body, and temporal channels are constant at 0.5; this does not imply that every generated video lacks a face or body.

Chromatic atmosphere (chromatic)

A per-frame chromatic signal blends mean saturation S‾t∈[0,1]\overline{S}_t \in[0,1] with the fraction of warm-hue pixels on the standard [0,179][0,179] hue scale:

Wt=1[h≤θwarm−∨h≥θwarm+].W_t = \mathbf{1}[h \le\theta_{\mathrm{warm}}^{-} \lor h \ge\theta_{\mathrm{warm}}^{+}].

The clip-level score aligns the frame mean to target valence v⋆v^{\star}, linearly mapped to [0,1][0,1]:

ct=λsatS‾t+(1−λsat)Wt,c‾=1T∑tct,schromatic=clip⁡[0,1](1−∣c‾−v⋆∣).(19)c_t = \lambda_{\mathrm{sat}}\overline{S}_t + (1-\lambda_{\mathrm{sat}})W_t,\qquad\overline{c} = \frac{1}{T}\sum_t c_t,\qquad s_{\mathrm{chromatic}} = \operatorname{clip}_{[0,1]}(1-|\overline{c}-v^{\star}|). \tag*{(19)}

In our implementation, λsat=0.5\lambda_{\mathrm{sat}} = 0.5 and the warm-hue thresholds are θwarm−=30\theta_{\mathrm{warm}}^{-} = 30, θwarm+=150\theta_{\mathrm{warm}}^{+} = 150 on the stated hue scale.

Kinetic intensity (kinetic)

A per-frame kinetic signal normalizes the mean absolute frame difference using saturation scale σkin\sigma_{\mathrm{kin}}, and the clip-level score aligns the frame mean to the target arousal a⋆a^{\star}:

kt=clip⁡[0,1](1σkinBI∣It−It−1∣),k‾=1T∑tkt,skinetic=clip⁡[0,1](1−∣k‾−a⋆∣).(20)k_t = \operatorname{clip}_{[0,1]}\left(\frac{1}{\sigma_{\mathrm{kin}}B_I}|I_t-I_{t-1}|\right),\qquad\overline{k} = \frac{1}{T}\sum_t k_t,\qquad s_{\mathrm{kinetic}} = \operatorname{clip}_{[0,1]}(1-|\overline{k}-a^{\star}|). \tag*{(20)}

In our implementation, σkin=0.20\sigma_{\mathrm{kin}} = 0.20 and the pixel range is BI=255B_I = 255. Appearance features describe potential cues; they are not part of the seven-provider memory update.

Motion naturalness (motion)

Dense optical flow ft\mathbf{f}_t on grayscale, short-side-LflowL_{\mathrm{flow}} frames yields per-frame magnitude mt=∥ft∥m_t = \|\mathbf{f}_t\| and a global (dominant) motion vector gt=ft\mathbf{g}_t = \mathbf{f}_t. Four bounded sub-scores reward smooth flow, low camera jerk, low warp residual, and low compensated flicker:

rflow=κ(∣Δmt∣;σflow),rcam=κ(∥Δ2gt∥;σcam),r_{\mathrm{flow}} = \kappa(|\Delta m_t|;\sigma_{\mathrm{flow}}),\qquad r_{\mathrm{cam}} = \kappa(\|\Delta^2\mathbf{g}_t\|;\sigma_{\mathrm{cam}}),
rwarp=κ(∣W(It−1;ft)−It∣;σwarp),rflick=κ(∣S(It−1;gt)−It∣;σflick),(21)r_{\mathrm{warp}} = \kappa(|W(I_{t-1};\mathbf{f}_t)-I_t|;\sigma_{\mathrm{warp}}),\qquad r_{\mathrm{flick}} = \kappa(|S(I_{t-1};\mathbf{g}_t)-I_t|;\sigma_{\mathrm{flick}}), \tag*{(21)}

where W\mathcal{W} is flow-warping and S\mathcal{S} a global-shift compensation. With base rbase=∑j∈{flow,cam,warp,flick}λjrjr_{\mathrm{base}} = \sum_{j\in\{\mathrm{flow,cam,warp,flick}\}} \lambda_j r_j and activity E=1−exp⁡(−mˉ/σE)E = 1 - \exp(-\bar{m}/\sigma_E), the arousal-conditioned score and confidence are

smotion=λMrbase+(1−λM)(1−∣E−a∗∣),(22)s_{\mathrm{motion}} = \lambda_M r_{\mathrm{base}} + (1-\lambda_M)(1-|E-a^*|), \tag*{(22)}
umotion=λUmin⁡(1,T−nUnU)+(1−λU)min⁡(1,std⁡(mt)σU),(23)u_{\mathrm{motion}} = \lambda_U \min\left(1,\frac{T-n_U}{n_U}\right) + (1-\lambda_U)\min\left(1,\frac{\operatorname{std}(m_t)}{\sigma_U}\right), \tag*{(23)}

reverting to smotion=rbases_{\mathrm{motion}} = r_{\mathrm{base}} when no target arousal is supplied. The motion gate scales linearly with this observation confidence (Appendix § 11). The mixture coefficients obey ∑jλj=1\sum_j \lambda_j = 1. In our implementation, Lflow=160L_{\mathrm{flow}} = 160, (σflow,σcam,σwarp,σflick)=(0.35,0.65,0.085,0.11)(\sigma_{\mathrm{flow}},\sigma_{\mathrm{cam}},\sigma_{\mathrm{warp}},\sigma_{\mathrm{flick}}) = (0.35,0.65,0.085,0.11), and (λflow,λcam,λwarp,λflick)=(0.34,0.24,0.24,0.18)(\lambda_{\mathrm{flow}},\lambda_{\mathrm{cam}},\lambda_{\mathrm{warp}},\lambda_{\mathrm{flick}}) = (0.34,0.24,0.24,0.18). We use σE=1.5\sigma_E = 1.5, λM=0.82\lambda_M = 0.82, λU=0.6\lambda_U = 0.6, nU=8n_U = 8, and σU=0.035\sigma_U = 0.035.

Score normalization and validity

The evaluated fusion consumes the providers’ bounded scores directly. An optional running-statistic normalization is not an additional transformation applied to those scores. Availability and confidence are retained separately from score magnitude: unavailable evidence receives a neutral fallback but zero effective contribution. Motion confidence is incorporated once through its channel gate, while an activity mask handles unavailable motion observations. This convention avoids counting the same confidence twice. The micro, body, and temporal channels are constant in the analyzed reference traces; their neutral readings do not establish the absence of visible faces or bodies in the generated videos.

Reflect: Instance-Dependent Gate Construction

Reflect separates training progress, semantic consistency, observation availability, and the prompt’s intended intensity. A factor is neutral for providers to which it does not apply. Let tt denote the video generation model update step, s0<s1s_0 < s_1 the phase-transition endpoints, γmin⁡\gamma_{\min} a phase floor, and qfq_f the available face-evidence ratio:

h(t)=clip⁡[0,1](t−s0s1−s0),b=max⁡(γmin⁡,h(t))gconflict,f=min⁡(1,qf/θf).(24)h(t) = \operatorname{clip}_{[0,1]}\left(\frac{t-s_0}{s_1-s_0}\right), \qquad b = \max(\gamma_{\min},h(t))g_{\mathrm{conflict}}, \qquad f = \min(1,q_f/\theta_f). \tag*{(24)}

The semantic-conflict factor attenuates disagreement d=∣sCLIP−AUTT−sR1−Omni∣d = |s_{\mathrm{CLIP-AUTT}}-s_{\mathrm{R1-Omni}}| above a threshold θd\theta_d, with lower bound γd\gamma_d. Subtlety is detected from prompt cues or a∗≤θaa^* \le\theta_a; it selects restraint factors ξf\xi_f, ξt\xi_t, and ξe\xi_e for facial/body, temporal, and semantic channels, respectively. Otherwise these restraint factors equal one. Before applying observation-availability masks, the gates are

gmacro=b,gmicro=gbody=bfξf,gtemporal=bfξt,g_{\mathrm{macro}} = b, \qquad g_{\mathrm{micro}} = g_{\mathrm{body}} = bf\xi_f, \qquad g_{\mathrm{temporal}} = bf\xi_t,
gCLIP−AUTT=gR1−Omni=bξe,gmotion=λg+(1−λg)cmotion.(25)g_{\mathrm{CLIP-AUTT}} = g_{\mathrm{R1-Omni}} = b\xi_e, \qquad g_{\mathrm{motion}} = \lambda_g + (1-\lambda_g)c_{\mathrm{motion}}. \tag*{(25)}

Here cmotionc_{\mathrm{motion}} is the optical-flow observation confidence. Availability masking and continuous confidence together instantiate the validity term in Eq. (7); confidence is not multiplied a second time.

Implementation settings. In our implementation, the default warmup endpoints are s0=40s_0 = 40 and s1=100s_1 = 100, with γmin⁡=0.1\gamma_{\min} = 0.1. Face gating uses θf=0.35\theta_f = 0.35; semantic conflict uses θd=0.20\theta_d = 0.20 and γd=0.35\gamma_d = 0.35. For subtle prompts, detected using θa=0.4\theta_a = 0.4 or prompt cues, (ξf,ξt,ξe)=(0.85,0.75,0.80)(\xi_f,\xi_t,\xi_e) = (0.85,0.75,0.80); the motion floor is λg=0.55\lambda_g = 0.55. These are rule-based controls rather than learned reward-model parameters. Historical exported traces retain their recorded gates; the default specification alone does not reconstruct a past run’s full configuration.

Effect on normalized fusion. A factor shared by every active provider cannot by itself redistribute trust. For gk=γhkg_k = \gamma h_k with γ>0\gamma> 0, active normalization yields

Remo=∑kwkhksk∑kwkhk+ϵ/γ.(26)R_{\mathrm{emo}} = \frac{\sum_k w_k h_k s_k}{\sum_k w_k h_k + \epsilon/\gamma}. \tag*{(26)}

Except for the stabilizer, the common scale cancels. Reflect changes the relative allocation through provider-specific factors, invalidation, and composition with the original reward branch. The face-activity proxy used to group exported traces in Appendix § 12 is distinct from the current video’s face-evidence ratio.

Evolve: RewardMemory Construction and Use

Record the reference run (Figure 8A). A completed run with Reflective Reward Gating supplies the experience used by RewardMemory. Each trace keeps the prompt–video context, raw provider scores, gates, applied weights, and VideoAlign outcome separate. The context describes target affect and available evidence; its face-activity proxy records whether selected scoring branches depart from their neutral defaults, rather than measuring visible-face coverage. The recorded scores and outcomes provide the basis for comparing providers.

Figure 8. Figure 8 From completed-run evidence to calibrated fusion. Recorded observations (A) supply provider scores and outcomes; RewardMemory (B) supplies fixed multipliers estimated from the complete reference bank, rather than the four displayed clips. A controlled replay (C) applies these multipliers to illustrative initial weights for rollout #2. The resulting 0.578→0.563 calculation demonstrates reallocation with fixed scores; it is distinct from the logged reward 0.721 and does not represent a second training run.

Build the calibration table (Figure 8B). The complete reference bank is grouped by target arousal and face-branch activity, with a global fallback for contexts with insufficient support. Within each group, reliability is the correlation between a provider’s raw scores and VideoAlign outcomes, before gating. Algorithm 2 compares each provider with the group’s average reliability and converts the difference into a capped weight multiplier (Eq. (10)). This calibrates reward allocation without fitting a value network or changing any reward model.

Use the memory in the next run (Figure 8C). Algorithm 1 looks up the matching multipliers, combines them with the base weights and current-video gates, and normalizes over active providers. Memory sets the allocation from prior experience; reflection can still suppress unavailable or unsuitable cues. The table remains fixed throughout training of the video generation model. Panel C illustrates this multiply-and-normalize step using recorded scores and illustrative starting weights: it is a controlled replay, not an additional training run. Reflect-only fusion omits the memory adjustment; Appendix § 16 explains the worked example.

Derivation of the Closed-Form KL Update

For positive base weights on the supported simplex, consider

max⁡w∈ΔK−1∑kwkAkrel−τ∑kwklog⁡wkwk0.(27)\max_{w\in\Delta^{K-1}} \sum_k w_k A_k^{\mathrm{rel}}-\tau\sum_k w_k\log\frac{w_k}{w_k^0}. \tag*{(27)}

The Lagrangian has stationary condition Akrel−τ(log⁡(wk/wk0)+1)+λ=0A_k^{\mathrm{rel}}-\tau(\log(w_k/w_k^0)+1)+\lambda=0. Normalizing the resulting exponentials gives wk=wk0eAkrel/τ/∑jwj0eAjrel/τw_k=w_k^0e^{A_k^{\mathrm{rel}}/\tau}/\sum_jw_j^0e^{A_j^{\mathrm{rel}}/\tau}. Strict concavity yields a unique optimum. A shared additive baseline in the utilities cancels from this unclipped solution.

For Δk=clip⁡(Akrel/τ,−c,c)\Delta_k=\operatorname{clip}(A_k^{\mathrm{rel}}/\tau,-c,c), the same derivation applies to clipped utilities A~k=τΔk\widetilde{A}_k=\tau\Delta_k. Since e−c≤∑jwj0eΔj≤ece^{-c}\le\sum_jw_j^0e^{\Delta_j}\le e^c, normalized weights satisfy

e−2c≤wkRMwk0≤e2c.(28)e^{-2c}\le\frac{w_k^{\mathrm{RM}}}{w_k^0}\le e^{2c}. \tag*{(28)}

The cap bounds relative changes; it does not guarantee a one-hot distribution as τ→0\tau\to0. Zero base weights remain outside the support.

Scope of the bound and cold start. The interval [e−c,ec][e^{-c},e^c] bounds the exported, unnormalized multipliers; the normalized weight ratio has the wider bound above. Clipping the utilities is not equivalent to solving the original objective subject to a prescribed hard KL constraint. If all advantages are zero, the update returns w0w^0. A constant provider has ρk=0\rho_k=0, but its centered advantage is negative only when the mean reliability is positive; constant scores therefore do not imply unconditional down-weighting. In the supplied traces, positive-reliability providers make this relative attenuation observable. None of these algebraic properties proves improved generalization or emotional fidelity.

Group-relative invariance. If Ri′=aRi+bR_i'=aR_i+b with a>0a>0, then μ′=aμ+b\mu'=a\mu+b, σ′=aσ\sigma'=a\sigma, and

Ai′=Ri−μσ+ϵ/a.(29)A_i'=\frac{R_i-\mu}{\sigma+\epsilon/a}. \tag*{(29)}

Translation cancels exactly. By contrast, CV⁡(R)=σ/(∣μ∣+ϵ)\operatorname{CV}(R)=\sigma/(|\mu|+\epsilon) is not translation invariant. Consequently, a reward offset can increase relative dispersion without improving the standardized group advantage. This is an algebraic property, not an experimental claim about learning efficiency.

RL Objective for Video Generation. For completeness, we state the clipped GRPO objective underlying the group advantages in Eq. (1) [54]. Let hi,th_{i,t} denote the generation state and ai,ta_{i,t} its sampled transition in candidate trajectory ii. Define ϱi,t(θ)=πθ(ai,t∣hi,t,p)πθold(ai,t∣hi,t,p)\varrho_{i,t}(\theta) = \frac{\pi_\theta(a_{i,t} \mid h_{i,t},p)}{\pi_{\theta_{\mathrm{old}}}(a_{i,t} \mid h_{i,t},p)}. With clipping radius εclip\varepsilon_{\mathrm{clip}} and reference-policy regularization coefficient κ\kappa,

JGRPO(θ)=Ep,{xi}∼πθold[1G∑i=1G1Ti∑t=1Ti(min⁡{ϱi,tAi,clip⁡(ϱi,t,1−εclip,1+εclip)Ai}−κDKL(πθ(⋅∣hi,t,p)∥πref(⋅∣hi,t,p)))].(30)J_{\mathrm{GRPO}}(\theta) = \mathbb{E}_{p,\{x_i\}\sim\pi_{\theta_{\mathrm{old}}}}\left[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{T_i}\sum_{t=1}^{T_i}\left(\min\left\{\varrho_{i,t}A_i,\operatorname{clip}\left(\varrho_{i,t},1-\varepsilon_{\mathrm{clip}},1+\varepsilon_{\mathrm{clip}}\right)A_i\right\}-\kappa D_{\mathrm{KL}}\left(\pi_\theta(\cdot\mid h_{i,t},p)\mathbin\Vert\pi_{\mathrm{ref}}(\cdot\mid h_{i,t},p)\right)\right)\right]. \tag*{(30)}

Here TiT_i is the number of sampled transitions, πθold\pi_{\theta_{\mathrm{old}}} is the rollout policy, and πref\pi_{\mathrm{ref}} is a fixed reference policy. The scalar reward of the completed video determines AiA_i; advantages and reward-provider outputs are treated as constants during the policy update. This expression states the common clipped objective; individual video RL algorithms specify their transition likelihoods, sampled timesteps, and regularization. EmoReward changes the scalar reward supplied to the RL algorithm and preserves those algorithm-specific choices. The policy KL penalty acts on the video generation model, while the reward-allocation KL term acts on the provider weights.

Reward-Gating Metrics and Reproducibility

Table 1 reports two reward diagnostics (SNR* and Sel.), three normalized pillars (EF, GD, GR), and two composites (core, emo). All seven policies use the same backbone and optimization budget. The exported evaluation contains 160 four-candidate groups for VideoAlign, single-provider, and random routing, and 152 groups for each remaining policy. Thus the available sample counts differ, even though the nominal training budget is shared. The references identify scoring models or routing mechanisms, not reproductions of complete agentic systems.

Reward Diagnostics

Let RiR_i denote the fused training reward, yiy_i the VideoAlign reference outcome, and si,ks_{i,k} provider kk’s score. A group gg contains candidates for one prompt.

Reward–outcome agreement (ROA). ROA is an auxiliary input to GR rather than a displayed column in Table 1. It measures within-group pairwise ranking agreement. For outcome-nontied pairs Cg={(i,j):i<j,∣yi−yj∣>εy}C_g = \{(i,j): i < j,\lvert y_i-y_j\rvert> \varepsilon_y\}, the intended statistic is

Agr⁡(g)=1∣Cg∣∑(i,j)∈Cg[1{(Ri−Rj)(yi−yj)>0}+121{Ri=Rj}].(31)\operatorname{Agr}(g)=\frac{1}{\lvert C_g\rvert}\sum_{(i,j)\in C_g}\left[\mathbf{1}\{(R_i-R_j)(y_i-y_j)>0\}+\frac{1}{2}\mathbf{1}\{R_i=R_j\}\right]. \tag*{(31)}

The reported agreement is ROA = Mean⁡gAgr⁡(g)\operatorname{Mean}_{g}\operatorname{Agr}(g). The supplied pairwise helper uses εy=10−9\varepsilon_y = 10^{-9} and returns 1/21/2 when no outcome-nontied pairs exist. However, its treatment of reward ties differs from the symmetric half-credit convention above. The original aggregation script is unavailable; we therefore retain the archived ROA values as inputs to GR, without claiming to have recomputed their tie handling. VideoAlign-only obtains one by construction when its reward and reference ordering coincide. ROA measures agreement with that evaluator, not independent human or emotional quality.

Within-group relative dispersion (SNR).* With population standard deviation within each group,

Dg=Std⁡i∈g(Ri)∣Mean⁡i∈g(Ri)∣+ϵD,SNR∗=1∣G∣∑g∈GDg.(32)D_g=\frac{\operatorname{Std}_{i\in g}(R_i)}{\lvert\operatorname{Mean}_{i\in g}(R_i)\rvert+\epsilon_D},\qquad\mathrm{SNR}^{*}=\frac{1}{\lvert G\rvert}\sum_{g\in G}D_g. \tag*{(32)}

The supplied diagnostic implementation uses ϵD=10−6\epsilon_D = 10^{-6}. This is a coefficient of variation, not a signal-to-noise power ratio. A reward offset can increase it while leaving centered GRPO advantages unchanged; a high value neither establishes nor is necessary for better policy gradients. We report the archived per-policy reductions, separately from the aggregate-level checks below.

Reliability selectivity (Sel.). Let ρk=Corr⁡(s⋅,k,y)\rho_k = \operatorname{Corr}(s_{\cdot,k}, y) over the policy’s evaluated candidates, using the nine-channel diagnostic pool P\mathcal{P}. For Z=∑k∈P∣ρk∣>0Z = \sum_{k \in\mathcal{P}} \lvert\rho_k \rvert> 0,

qk=∣ρk∣Z,Sel.=1−−∑k∈Pqklog⁡qklog⁡∣P∣,0log⁡0=0.(33)q_k = \frac{\lvert\rho_k \rvert}{Z}, \qquad\mathrm{Sel.} = 1 - \frac{-\sum_{k \in\mathcal{P}} q_k \log q_k}{\log\lvert\mathcal{P} \rvert}, \qquad0\log0 = 0. \tag*{(33)}

The pool includes chromatic and kinetic readouts as well as the seven fusion providers. Sel. describes concentration of absolute provider–outcome correlations, not applied fusion weights. It discards correlation signs and does not establish causal provider usefulness. All seven reported policies have Z>0Z > 0; the auxiliary zero-sample row and its all-zero-profile convention are excluded. Recomputing Sel. from the exported correlation profiles agrees with the stored values within 5×10−55 \times10^{-5}.

Normalized Pillars and Composite Scores

For the fixed comparison set A\mathcal{A} of seven policies, define

N(va)=va−min⁡b∈Avbmax⁡b∈Avb−min⁡b∈Avb.(34)N(v_a) = \frac{v_a - \min_{b \in\mathcal{A}} v_b}{\max_{b \in\mathcal{A}} v_b - \min_{b \in\mathcal{A}} v_b}. \tag*{(34)}

The supplied reducer returns zero for a constant input across policies; none of the three reported inputs is constant. Normalization is specific to the compared set, so the resulting scores are not absolute measures transferable across benchmarks.

EF: preference-proxy pillar. EFa=N(Ia)\mathrm{EF}_a = N(I_a), where IaI_a is the archived mean ImageReward score [77]. EF retains the benchmark’s “Emotion Fidelity” abbreviation, but its input is a general visual preference model. We do not interpret it as an independent measurement of emotion fidelity. The raw means and normalization inputs are included in the supplement.

GD: relative-dispersion pillar. GDa=N(SNRa∗)\mathrm{GD}_a = N(\mathrm{SNR}^{*}_a). The benchmark calls this Gating Discriminativeness; mathematically, it is the normalized coefficient of variation above. Its inclusion does not turn reward dispersion into a gradient-quality measurement.

GR: agreement-weighted selectivity pillar. GRa=N(Sel.aROAa)\mathrm{GR}_a = N(\mathrm{Sel.}_a\mathrm{ROA}_a). GR combines correlation-profile concentration with agreement with the reference evaluator. Both inputs are diagnostic summaries; the product does not establish an independent reliability ground truth.

core and emo: two specified aggregations. The equal-weight composite and preference-emphasized composite are

corea=EFa+GDa+GRa3,emoa=λEFEFa+λGDGDa+λGRGRa.(35)\mathrm{core}_a = \frac{\mathrm{EF}_a + \mathrm{GD}_a + \mathrm{GR}_a}{3}, \qquad\mathrm{emo}_a = \lambda_{\mathrm{EF}}\mathrm{EF}_a + \lambda_{\mathrm{GD}}\mathrm{GD}_a + \lambda_{\mathrm{GR}}\mathrm{GR}_a. \tag*{(35)}

Here the nonnegative coefficients sum to one; our implementation uses (λEF,λGD,λGR)=(0.5,0.25,0.25)(\lambda_{\mathrm{EF}}, \lambda_{\mathrm{GD}}, \lambda_{\mathrm{GR}}) = (0.5, 0.25, 0.25). We report the predefined composites as descriptive summaries. Min–max normalization fixes the endpoints for this set of policies, so small raw differences can lead to large changes in a normalized pillar.

Emotion and visual metrics

Facial reward during training. The facial reward provider uses CLIP-AUTT [86] with a CLIP ViT-B/32 backbone and an action-unit (AU) text adapter and temporal classifier trained on BioVid. It produces AU similarity scores and probabilities for the BioVid pain and neutral classes.

Emotion metrics at evaluation. DEA and TERS use the standard frozen CLIP-ViT-H-14-laion2B-s32B-b79K checkpoint, trained on the LAION-2B subset of LAION-5B [53]. This encoder is held out from reward computation, RewardMemory, and checkpoint selection. It encodes 24 uniformly sampled frames per video and one text template for each of the eight Plutchik emotion rings. Let vfv_f be the frame-ff embedding and trt_r the text embedding for ring rr. The frame-averaged image–text cosine similarity for each of the eight rings is

sr=124∑f=124vfTtr∥vf∥2∥tr∥2.(36)s_r = \frac{1}{24}\sum_{f=1}^{24}\frac{v_f^{\mathsf{T}}t_r}{\lVert v_f\rVert_2\lVert t_r\rVert_2}. \tag*{(36)}

Let TT be the prompt’s set of target rings. DEA checks whether the largest srs_r in TT exceeds the largest score among the corresponding opposite rings after excluding opposites that are themselves targets. Of the 302 prompts, 261 have a usable opposite. TERS uses all eight ring scores. We sort them in descending order with ranks starting at one, using the fixed ring order (joy, trust, fear, surprise, sadness, disgust, anger, anticipation) for ties. The score for one video is

r∗=min⁡r∈Tranks⁡(r),TERS=8−r∗7.(37)r^* = \min_{r \in T} \operatorname{ranks}(r), \qquad\mathrm{TERS} = \frac{8-r^*}{7}. \tag*{(37)}

A score of one means a target ring ranks first. We compute TERS on all 302 prompts and report its mean and SEM across prompts. Table 6 compares the same configurations under LAION-2B ViT-H/14 and OpenAI ViT-L/14. For DEA, we report discordant algorithm–prompt pairs descriptively. The two-sided paired permutation test flips the base and guided labels jointly for all three RL algorithms within each prompt, accounting for prompt reuse.

CLIP ViT-H/14CLIP ViT-L/14
RL algorithmDEATERSDEATERS
Base+EmoRewardBase+EmoRewardBase+EmoRewardBase+EmoReward
DanceGRPO0.6050.6320.7640.7820.6250.6320.7630.770
FlowGRPO0.6210.6700.7860.8020.6250.6590.7820.802
SAGE-GRPO0.6170.6740.7880.8190.6130.6700.7740.808
Discordant wins90/145 (62.1%)–106/186 (57.0%)–

Table 6. Emotion metrics across frozen encoders. DEA uses 261 prompts and TERS uses all 302. The last two rows summarize discordant algorithm–prompt pairs and prompt-blocked permutation tests for DEA.

Visual metrics. For PickScore and CLIPScore, we divide the frame-level logits by 26 and 30, respectively, then average the scaled scores across frames.

Table 6 Emotion metrics across frozen encoders. DEA uses 261 prompts and TERS uses all 302. The last two rows summarize discordant algorithm–prompt pairs and prompt-blocked permutation tests for DEA.

Extended Evaluation and Ablations

Provider-pool construction. Figure 6 presents the main-text visual comparison; Table 7 quantifies the same four reward configurations used to construct the provider interface. Semantic emotion and motion cues improve some axes while exposing trade-offs on text alignment and preference. The configurations also differ in fusion weights and curricula; their comparison is not a single-factor ablation. In the semantic-cue configuration, the CLIP channel has zero fusion weight and cannot explain a training gain attributed to an active CLIP reward.

ConfigurationOverallVQMQTAImageReward
VideoAlign-1.614-0.505-0.313-0.797-0.866
+Semantic emotion-1.363-0.014-0.991-0.922
+Motion naturalness-1.453-0.5050.075-1.024-1.293
Balanced tool pool-1.178-0.2970.074-0.954-1.113

Table 7. Provider-pool configurations. Higher is better. TA denotes text alignment. The reward configurations use different weights and curricula; missing VQ is not imputed.

Table 7 Provider-pool configurations. Higher is better. TA denotes text alignment. The reward configurations use different weights and curricula; missing VQ is not imputed.

Ranking agreement. For each four-candidate group, let C={(i,j):i<j,yi≠yj}\mathcal{C} = \{(i,j): i < j, y_i \ne y_j\}. We use

Agr⁡(G)=1∣C∣∑(i,j)∈C[1{(Ri−Rj)(yi−yj)>0}+121{Ri=Rj}].(38)\operatorname{Agr}(\mathcal{G}) = \frac{1}{|\mathcal{C}|} \sum_{(i,j)\in\mathcal{C}} \left[\mathbb{1}\{(R_i-R_j)(y_i-y_j)>0\} + \frac{1}{2}\mathbb{1}\{R_i=R_j\}\right]. \tag*{(38)}

Groups without comparable pairs are excluded. Agreement is averaged over groups; reward ties receive half credit independently of sample ordering. The analysis contains 160 baseline groups for DanceGRPO and 168 groups for each other algorithm–reward configuration, totaling 1,504 groups and 6,016 videos. These training candidates are separate from the 30-prompt fixed-checkpoint evaluation sets.

Memory sensitivity. Across 27 temperature, neighborhood-size, and memory-window settings, the memory rule exceeds fixed weighting at the final window in 23 settings. None has a positive difference when averaged over all windows. These settings reuse the same traces and are not independent training seeds. The results motivate sensitivity analysis rather than a claim of monotonic improvement with accumulating memory.

RewardMemory: A Worked Rollout Example

Figure 8 connects recorded provider scores, a previously exported calibration table, and a reproducible allocation calculation. Appendix F explains the principles behind panels A–C. The clips contain 81 frames at 24 fps and have 624 × 352 resolution; we display frames 12, 40, and 72. The four videos are candidates for one prompt from a reference rollout group. Their scores and outcomes come directly from the accompanying reward records; the images are decoded full frames. The macro provider has positive within-group outcome correlation, while three channels return their neutral constant. These four examples illustrate local behavior, not the population estimate used to construct RewardMemory.

Separating observations from the replay. The exported table’s global entry differs from its context-specific entries. We use the global entry here to isolate the reweighting operation, without claiming that was the entry selected for these historical videos. The initial active allocation is (0.336,0.440,0.134,0.090)(0.336, 0.440, 0.134, 0.090) over macro, motion, CLIP-AUTT, and R1-Omni, with zero mass on the other three providers. These are illustrative starting weights, not reconstructed runtime gates. Multiplying by the exported global multipliers and renormalizing produces approximately (0.417,0.450,0.082,0.052)(0.417, 0.450, 0.082, 0.052). On rollout #2’s recorded scores, the weighted mean changes from 0.578 to 0.563; its actual logged emotion reward is 0.721. The distinction matters because the historical reward used its own training phase and configuration. An increase in a provider’s allocation need not increase the scalar fused score, and neither change alone establishes improved generated quality.

Prompt. The camera slowly pushes forward, a cold side light slanting through the window frame, carving a sharp beam across the dimly lit study and illuminating the man seated at his desk. He leans slightly forward, hands folded over the keyboard, fingers tapping the keys with slow, precise rhythm. Shot from a low angle behind and to the side, the frame captures his furrowed brow and narrowed eyes, his gaze fixed like a needle on the flickering stream of code on the screen. His lips are pressed into a thin line, the jawline taut with concentration, nostrils fluttering faintly with each breath, occasionally biting his lower lip as if swallowing an unspoken question. The light casts a soft shadow across his forehead, making the sharp glint in his eyes even more intense. In the background, scattered papers and empty coffee cups rest silently on the bookshelf, while the rhythmic tapping of the keyboard echoes in the stillness, like the pulse of a heartbeat. He suddenly pauses, raises his right hand to his temple, his eyes briefly drifting, then swiftly refocuses on the screen—a quiet resolve, a steadfast defiance against time.

Generalization to Image Generation

We evaluate Stable Diffusion 3.5 and FLUX with three RL algorithms and three reward configurations on 256 DrawBench prompts per checkpoint: 18 settings and 84 checkpoints. Internal Emo measures the emotion-stack reward; HPSv3 is a general preference objective also used to train the baseline. Neither independently measures perceived emotion.

SD3.5 baseline results average three checkpoints at 420–500 updates; calibrated variants average five at 2,780–3,000. FLUX baseline checkpoints span 2,760–3,000, and calibrated checkpoints 2,780–3,000. Standard errors describe checkpoint variation, not independent training seeds. Per-prompt arrays support the reported means; paired significance requires complete sample and generation-seed mappings and is not claimed.

Observed transfer pattern. With SD3.5–DanceGRPO, HPSv3 rises from 9.552 (baseline) to 10.096 (Reflect) and 10.302 (EmoReward). SD3.5–FlowGRPO declines on both HPSv3 and internal Emo; FLUX’s HPSv3-optimized baseline leads on HPSv3 across RL algorithms. Thus transfer is objective-dependent rather than uniformly beneficial.

Qualitative comparison protocol

Image plates use four curated prompts per displayed setting; video plates cover three RL algorithms. Figure 6 separately compares provider configurations. These examples illustrate behavior, not a randomly estimated win rate.

Images retain all original pixels. SD3.5 baseline outputs use 500 updates and calibrated outputs 3,000. The first two FLUX baseline cases use the only saved examples at 500 updates; all other FLUX rows use 3,000. Unequal budgets prevent attributing qualitative differences solely to calibration. Image seeds are unavailable, so only prompts are matched. The video rows in Appendix § 18 share prompts and recorded sampling seeds. Clips are 864×480 at 24 fps; full frames at 0.5, 2.5, and 4.5 seconds are decoded without interpolation or retouching.

The ordinal ratings in Appendix § 9 complement these proxy-based and qualitative comparisons; raw participant/prompt records are needed to quantify uncertainty. The depicted scenes are generated content.

Figure 9. (a) A stack of 3 plates. A blue plate is on the top, sitting on a blue plate. The blue plate is in the middle, sitting on a green plate. The green plate is on the bottom.

(b) A large keyboard musical instrument with a wooden case enclosing a soundboard and metal strings, which are struck by hammers when the keys are depressed. The strings' vibration is stopped by dampers when the keys are released and can be regulated for length and volume by two or three pedals. [Spelling normalized.]

(c) Artophagous.

(d) A stack of 3 books. A green book is on the top, sitting on a red book. The red book is in the middle, sitting on a blue book. The blue book is on the bottom.

Qualitative comparison of reward calibration. Columns show the baseline, the addition of Reflect, and the complete EmoReward configuration for SD3.5 with DanceGRPO. Prompt (b) is spelling-normalized for readability; the images retain the original perturbed input. Budgets and seed limitations appear in Appendix § 17.§ 1.

Figure 10. Image transfer across SD3.5 and FLUX. Five backbone–algorithm settings share four prompts; each panel compares HPSv3 RL, Reflective gating, and EmoReward. Rows show the same curated prompts as the preceding figure. The first two FLUX baseline rows use earlier saved outputs; see Appendix § 17.§ 1 for budgets and seed limitations.

Video Comparisons

Figure 11. Prompt: The camera slowly pushes in on a young woman standing at a street corner under the dim yellow glow of a streetlight, captured in a medium close-up. Side backlighting outlines her slender silhouette. Her hand unconsciously tightens around the backpack strap, knuckles turning slightly white, while a faint, forced smile tugs at her lips—never reaching her eyes. Her gaze drifts away from the front, settling on a puddle on the ground, where neon signs flicker and reflect like the instability of her emotions. She slightly turns her head, her lips twitching as if about to speak, but only a soft cough escapes. She lowers her head, her long hair falling to veil half her face. A breeze lifts her strands, and she instinctively reaches up to brush them aside, her fingers pausing in midair before dropping weakly. In the background, blurred traffic and pedestrians move past, their noise isolated beyond the frame, leaving her alone in the shifting light, trembling slightly, as if the entire world has gone silent except for the clear, pounding rhythm of her heartbeat in her ears. Figure 11 DanceGRPO. Street-corner hesitation: a strained smile and averted gaze.

Figure 12. Prompt: The slanting sunlight of dusk filters through the old window lattice, bathing the room in a warm, amber glow. An middle-aged couple sits on worn wicker chairs; she is bent over, mending an old sweater, her fingers deftly guiding the needle and thread. He watches her quietly, his fingers unconsciously tracing the rim of his cup, his gaze reflecting the swaying tree shadows outside. Suddenly, she looks up at him, a faint smile lifting the corners of her lips, fine lines crinkling at the edges of her eyes—gentle ripples of time’s quiet grace. His throat moves slightly, his lips parting as if to speak, but no sound comes. Instead, he reaches out, gently brushing a strand of her hair, his touch slow and reverent. A hint of shyness flickers in her eyes; she lowers her head with a soft laugh, her fingers trembling slightly as a thread slips free. He catches it with a quick, tender motion. They exchange a glance, and a silent understanding flows between them. The camera slowly pushes in, focusing on the delicate smile lines at the corner of her eye and the quiet glimmer deep in his gaze. Figure 12 DanceGRPO. A quiet exchange at dusk: the prompt couples a smile with a tentative hand gesture.

Figure 13. Prompt: The camera slowly pushes in on a young woman kneeling on ancient bluish stone steps, her back turned to the deepening twilight sky, hands folded gently on her knees, fingertips trembling faintly. Her face is bathed in warm gold by the slanting rays of the setting sun, her eyelashes lowered, her pupils reflecting the flickering oil lamp at the distant temple eaves—a silent mirror to some unspoken obsession. Her lips are slightly pursed, the line of her mouth taut, as if silently praying or suppressing a sigh that threatens to break free. She tilts her head upward, gazing into the night sky, a single tear slipping down her cheek, pausing at her chin, glimmering in the dim light. A breeze stirs her hair, her sleeve lifts slightly, her arm quivers faintly, as though awaiting a response that will never come. Behind her, a quiet courtyard lies still, wild grasses growing beside the stone steps, while in the distance, a bell tolls gently, echoing through the empty dusk. The entire scene is enveloped in a near-sacred sense of solitude. Figure 13 FlowGRPO. Unease on the stone steps at twilight.

Figure 14. Prompt: The camera slowly pushes in on a young woman sitting alone in the corner of an old-fashioned café, rain streaking diagonally across the window, blurring the street scene through water-traced glass. She holds a cup of coffee now cold, her fingertips trembling slightly, her gaze distant as she stares at the table, lips pressed into a thin line, as if chewing on some unspeakable contradiction. Light slants through the high window, bathing one side of her face in warm yellow while the other remains in shadow, creating a stark visual divide. Suddenly, she lifts her eyes, scanning the empty seats before returning her gaze to her phone screen, her brow furrowing slightly, her mouth twitching as if she’s heard a whisper she shouldn’t have. She wipes her eye with the back of her hand—restrained yet unable to hide her weariness—her fingers unconsciously picking at the rim of the cup, as if searching for some confirmation that has long since vanished. The air is thick with the scent of coffee and damp wood; time seems to have stopped, except for the flicker of doubt in her brow, quietly growing in the silence. Figure 14 FlowGRPO. A pause in a rain-streaked café.

Figure 15. Prompt: The camera slowly pushes in, revealing a middle-aged woman standing at the center of the stage, bathed in a spotlight that cascades from the upper side, softly outlining her silhouette with quiet strength. She tilts her head slightly upward, a subtle smile forming on her lips—a smile long held within, unassuming yet powerful enough to pierce the silence. Her gaze sweeps across the audience, her eyes glowing with a gentle warmth, like melting snow in winter, reflecting countless looks of reverence. Her fingers gently trace the emblem on her chest, her fingertips trembling slightly, as if touching a memory sealed by time. Slowly, she raises her hand and grasps the microphone, her lips parting slightly as her voice emerges—low, clear, each word like amber seeping from the depths of her soul. Her eyes never leave the audience, a look of calm acceptance and quiet response, as if saying: every step I’ve taken is worth being seen. Behind her, the applause blurs into soft waves, the melody gentle and hushed, while her face, stretched by light and shadow, resembles a still oil painting, frozen in her moment of radiant glory. Figure 15 SAGE-GRPO. Composure under the spotlight in a restrained, emotionally charged stage scene.

Figure 16. Prompt: The camera slowly pushes in, settling on a close-up of a middle-aged woman seated in an old wicker chair by the window, sunlight slanting across her slightly weary face, fine lines around her eyes trembling faintly in the light. She gazes down at an old letter in her hands, lips slightly parted, breath catching, as if reading a memory sealed by time. Suddenly, a breeze stirs, gently turning the page. She lifts her eyes, her gaze falling on a cherry blossom tree blooming outside, a flicker of recognition passing through her, a soft smile gradually forming, her brow relaxing into a long-forgotten tenderness. She raises a hand, fingertips brushing lightly over the letter, as if touching a distant moment; tears well in her eyes but do not fall, only a faint glimmer shining deep within her pupils. The scene slowly pulls back, revealing her sitting in a room bathed in shifting light and shadow, the flower shadows outside swaying gently, the entire space filled with a quiet, warm aura of bygone days. Figure 16 SAGE-GRPO. Recognition in an old letter in a close-up with gradually softening expression.

References

  1. [1]Shuai Bai, Yuxuan Cai, Ruizhe Chen, et al. Qwen3-VL technical report. arXiv preprint arXiv:2511.21631, 2025.arxiv.org/abs/2511.21631
  2. [2]Tadas Baltrušaitis, Amir Zadeh, Yao Chong Lim, and Louis-Philippe Morency. OpenFace 2.0: Facial Behavior Analysis Toolkit. In IEEE International Conference on Automatic Face and Gesture Recognition, pages 59–66, 2018.DOI
  3. [3]Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. In International Conference on Learning Representations, volume 2024, pages 4965–4987, 2024.arxiv.org/abs/2305.13301
  4. [4]Haodong Chen, Haojian Huang, Junhao Dong, Mingzhe Zheng, and Dian Shao. Fineclip: Multi-modal fine-grained clip for dynamic facial expression recognition with adapters. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 2301–2310, 2024.
  5. [5]Shuang Chen, Kaituo Feng, Hangting Chen, Wenxuan Huang, Dasen Dai, Quanxin Shou, Yunlong Lin, Xiangyu Yue, Shenghua Gao, and Tianyu Pang. OpenSearch-VL: An Open Recipe for Frontier Multimodal Search Agents. arXiv preprint arXiv:2605.05185, 2026.arxiv.org/abs/2605.05185
  6. [6]Sixiang Chen, Zhaohu Xing, Tian Ye, Xinyu Geng, Yunlong Lin, Jianyu Lai, Xuanhua He, Fuxiang Zhai, Jialin Gao, and Lei Zhu. GenEvolve: Self-Evolving Image Generation Agents via Tool-Orchestrated Visual Experience Distillation. arXiv preprint arXiv:2605.21605, 2026.
  7. [7]Wenshuo Chen, Kuimou Yu, Bowen Tian, Jianfei Song, Shaofeng Liang, Haozhe Jia, Kan Cheng, Haosen Li, Kaishen Yuan, Lei Wang, Jiemin Wu, Songning Lai, and Yutao Yue. MemoGen: Can Past Experience Improve Future Text-to-Image Generation? arXiv preprint arXiv:2606.03243, 2026.
  8. [8]Zebang Cheng, Zhi-Qi Cheng, Jun-Yan He, Jingdong Sun, Kai Wang, Yuxiang Lin, Zheng Lian, Xiaojiang Peng, and Alexander G Hauptmann. Emotion-llama: Multimodal emotion recognition and reasoning with instruction tuning. Advances in Neural Information Processing Systems, 37:110805–110853, 2024.
  9. [9]Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
  10. [10]Alan S. Cowen and Dacher Keltner. Self-report captures 27 distinct categories of emotion bridged by continuous gradients. Proceedings of the National Academy of Sciences, 114(38):E7900–E7909, 2017.
  11. [11]Alan S. Cowen and Dacher Keltner. What the face displays: Mapping 28 emotions conveyed by naturalistic expression. American Psychologist, 75(3):349–364, 2020.
  12. [12]Fangming Cui, Ruixiao Zhu, Cheng Fang, Sunan Li, and Jiahong Li. Rethinking agentic reinforcement learning in large language models. arXiv preprint arXiv:2604.27859, 2026.
  13. [13]Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. GoEmotions: A dataset of fine-grained emotions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4040–4054, 2020.
  14. [14]Paul Ekman and Wallace V. Friesen. Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto, CA, 1978.
  15. [15]Liaoyuan Fan, Yuhao Zheng, Dongxing Mao, Hangyu Ran, Siyuan Hu, Weijia Mao, Rui Zhao, Weijia Wu, Guian Fang, Yanzhe Chen, Zeyu Zhu, Yuxin Jiang, Yiren Song, Yuchao Gu, Junchao Yi, Qi Hu, Dan Kong, Alex Jinpeng Wang, Kevin Qinghong Lin, Mike Zheng Shou, and Wei Liu. Video Generation Meets Agent Harness: Agentic Video Generation, Agentic World Modeling and Recursive Self-Improvement. arXiv preprint arXiv:2610.40191, 2026.
  16. [16]Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models. Advances in neural information processing systems, 36:79858–79885, 2023.arxiv.org/abs/2305.16381
  17. [17]Zhen Fang, Wenxuan Huang, Yu Zeng, Yiming Zhao, Shuang Chen, Kaituо Feng, Yunlong Lin, Lin Chen, Zehui Chen, Shaosheng Cao, et al. Flow-OPD: On-policy distillation for flow matching models. arXiv preprint arXiv:2605.08063, 2026.arxiv.org/abs/2605.08063
  18. [18]Kaituo Feng, Manyuan Zhang, Shuang Chen, Yunlong Lin, Kaixuan Fan, Yilei Jiang, Hongyu Li, Dian Zheng, Chenyang Wang, and Xiangyu Yue. Gen-searcher: Reinforcing agentic search for image generation. arXiv preprint arXiv:2603.28767, 2026.arxiv.org/abs/2603.28767
  19. [19]Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Yunzhong He, Bing Liu, and Sean Hendryx. Rubrics as rewards: Reinforcement learning beyond verifiable domains. In International Conference on Learning Representations, volume 2026, pages 127924–127945, 2026.arxiv.org/abs/2507.17746
  20. [20]Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.DOI
  21. [21]Xuan He, Dongfu Jiang, Ge Zhang, Max Ku, Achint Soni, Sherman Siu, Haonan Chen, Abhranil Chandra, Ziyan Jiang, Aaron Arulraj, et al. Videoscore: Building automatic metrics to simulate fine-grained human feedback for video generation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2105–2123, 2024.arxiv.org/abs/2406.15252
  22. [22]Haobo Hu, Qi Mao, Yuanhang Li, and Libiao Jin. Camera Artist: A Multi-Agent Framework for Cinematic Language Storytelling Video Generation. arXiv preprint arXiv:2604.09195, 2026.arxiv.org/abs/2604.09195
  23. [23]Yinming Huang, Shuyuan Tu, Xi Yan, Zihan Yang, Jianhua Han, Hang Xu, Kaihang Pan, Yu-Gang Jiang, and Zuxuan Wu. Vaj-judger: Reward modeling from human preference feedback for joint video-audio generation. arXiv preprint arXiv:2608.18607, 2026.arxiv.org/abs/2608.18607
  24. [24]Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21807–21818. IEEE, 2024.arxiv.org/abs/2311.17982
  25. [25]Dengyang Jiang, Xin Jin, Dongyang Liu, Zanyi Wang, Mingzhe Zheng, Ruoyi Du, Xiangpeng Yang, Qilong Wu, Zhen Li, Peng Gao, et al. D-OPSD: On-Policy Self-Distillation for Continuously Tuning Step-Distilled Diffusion Models. arXiv preprint arXiv:2605.05204, 2026.arxiv.org/abs/2605.05204
  26. [26]Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbubland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. Advances in neural information processing systems, 36:36652–36663, 2023.arxiv.org/abs/2305.01569
  27. [27]Jinxiang Lai, Zexin Lu, Jiajun He, Rongwei Quan, Wenzhe Zhao, Qinyu Yang, Qi Chen, Qin Lin, Chuyue Li, Tao Gao, Yuhao Shan, Shuai Shao, Song Guo, and Qinglin Lu. VisionCreator: A Native Visual-Generation Agentic Model with Understanding, Thinking, Planning and Creation. arXiv preprint arXiv:2603.02681, 2026.arxiv.org/abs/2603.02681
  28. [28]Jiaqi Leng, Shuyuan Tu, Haidong Cao, Sicheng Xie, Daoguo Dong, Zuxuan Wu, and Yu-Gang Jiang. Preference score distillation: Leveraging 2d rewards to align text-to-3d generation with human preference. arXiv preprint arXiv:2603.01594, 2026.arxiv.org/abs/2603.01594
  29. [29]Yibo Li, Zijie Lin, Ailin Deng, Xuan Zhang, Yufei He, Shuo Ji, Tri Cao, and Bryan Hooi. Just-In-Time Reinforcement Learning: Continual Learning in LLM Agents Without Gradient Updates. arXiv preprint arXiv:2601.18510, 2026.arxiv.org/abs/2601.18510
  30. [30]Zheng Lian, Haoyu Chen, Lan Chen, Haiyang Sun, Licai Sun, Yong Ren, Zebang Cheng, Bin Liu, Rui Liu, Xiaojiang Peng, et al. Affectgpt: A new dataset, model, and benchmark for emotion understanding with multimodal large language models. arXiv preprint arXiv:2501.16566, 2025.arxiv.org/abs/2501.16566
  31. [31]Yunlong Lin, Zixu Lin, Kunjie Lin, Jinbin Bai, Panwang Pan, Chenxin Li, Haoyu Chen, Zhongdao Wang, Xinghao Ding, Wenbo Li, et al. JarvisArt: Liberating Human Artistic Creativity via an Intelligent Photo Retouching Agent. arXiv preprint arXiv:2506.17612, 2025.arxiv.org/abs/2506.17612
  32. [32]Yunlong Lin, Linqing Wang, Kunjie Lin, Zixu Lin, Kaixiang Gong, Wenbo Li, Bin Lin, Zhenxi Li, Shiyi Zhang, Yuyang Peng, et al. JarvisEvo: Towards a Self-Evolving Photo Editing Agent with Synergistic Editor-Evaluator Optimization. arXiv preprint arXiv:2511.23002, 2025.arxiv.org/abs/2511.23002
  33. [33]Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl. Advances in neural information processing systems, 38:40783–40818, 2026.arxiv.org/abs/2505.05470
  34. [34]Jie Liu, Gongye Liu, Jiajun Liang, Ziyang Yuan, Xiaokun Liu, Mingwu Zheng, Xiele Wu, Qiulin Wang, Menghan Xia, Xintao Wang, et al. Improving video generation with human feedback. Advances in Neural Information Processing Systems, 38:82155–82192, 2026.arxiv.org/abs/2501.13918
  35. [35]Runtao Liu, Haoyu Wu, Ziqiang Zheng, Chen Wei, Yingqing He, Renjie Pi, and Qifeng Chen. VideoDPO: Omni-Preference Alignment for Video Diffusion Generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.arxiv.org/abs/2412.14167
  36. [36]Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang, Haoxin Chen, Yang Liu, Tieyong Zeng, Raymond Chan, and Ying Shan. Evalcrafter: Benchmarking and evaluating large video generation models. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22139–22149. IEEE, 2024.arxiv.org/abs/2310.11440
  37. [37]Zeqian Long, Mingzhe Zheng, Kunyu Feng, Xinhua Zhang, Hongyu Liu, Harry Yang, Linfeng Zhang, Qifeng Chen, and Yue Ma. Follow-your-shape: Shape-aware image editing via trajectory-guided region control. International Conference on Learning Representations, 2026.arxiv.org/abs/2508.08134
  38. [38]Xuran Ma, Yexin Liu, Yaofu Liu, Xianfeng Wu, Mingzhe Zheng, Zihao Wang, Ser-Nam Lim, and Harry Yang. Model reveals what to cache: Profiling-based feature reuse for video diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17150–17159, 2025.
  39. [39]Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-Level Reward Design via Coding Large Language Models. In International Conference on Learning Representations, 2024.arxiv.org/abs/2310.12931
  40. [40]Yue Ma, Xinyu Wang, Qianli Ma, Qinghe Wang, Mingzhe Zheng, Xiangpeng Yang, Hao Li, Chongbo Zhao, Jixuan Ying, Harry Yang, et al. Group editing: Edit multiple images in one go. Proceedings of the IEEE/CVF Conference on Computer Vision & Pattern Recognition, 2026.arxiv.org/abs/2603.22883
  41. [41]Yue Ma, Zhikai Wang, Tianhao Ren, Mingzhe Zheng, Hongyu Liu, Jiayi Guo, Kunyu Feng, Yuxuan Xue, Zixiang Zhao, Konrad Schindler, et al. Fastvmt: Eliminating redundancy in video motion transfer. International Conference on Learning Representations, 2026.arxiv.org/abs/2602.05551
  42. [42]Yuhang Ma, Xiaoshi Wu, Keqiang Sun, and Hongsheng Li. Hpsv3: Towards wide-spectrum human preference score. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pages 15086–15095. IEEE, 2025.arxiv.org/abs/2508.03789
  43. [43]Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in neural information processing systems, 36:46534–46594, 2023.arxiv.org/abs/2303.17651
  44. [44]Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M. Waleed Kadous, and Ion Stoica. RouteLLM: Learning to Route LLMs with Preference Data. arXiv preprint arXiv:2406.18665, 2024.arxiv.org/abs/2406.18665
  45. [45]Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training Language Models to Follow Instructions with Human Feedback. In Advances in Neural Information Processing Systems, 2022.arxiv.org/abs/2203.02155
  46. [46]Yatian Pang, Bin Zhu, Bin Lin, Mingzhe Zheng, Francis EH Tay, Ser-Nam Lim, Harry Yang, and Li Yuan. Dreamdance: Animating human images by enriching 3d geometry cues from 2d poses. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14039–14050, 2025.arxiv.org/abs/2412.00397
  47. [47]W. Gerrod Parrott, editor. Emotions in Social Psychology: Essential Readings. Psychology Press, 2001.
  48. [48]Robert Plutchik. The nature of emotions. American Scientist, 89(4):344–350, 2001.
  49. [49]Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text-to-image diffusion models with reward backpropagation. 2023.arxiv.org/abs/2310.03739
  50. [50]Qwen Team. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.
  51. [51]Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728–53741, 2023.arxiv.org/abs/2305.18290
  52. [52]James A. Russell. A Circumplex Model of Affect. Journal of Personality and Social Psychology, 39(6):1161–1178, 1980.DOI
  53. [53]Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Arush Katto, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jena Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
  54. [54]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.arxiv.org/abs/2402.03300
  55. [55]Phillip R. Shaver, Judith Schwartz, Donald Kirson, and Cary O’Connor. Emotion knowledge: Further exploration of a prototype approach. Journal of Personality and Social Psychology, 52(6):1061–1086, 1987.pubmed.ncbi.nlm.nih.gov/3598857
  56. [56]Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in neural information processing systems, 36:8634–8652, 2023.arxiv.org/abs/2303.11366
  57. [57]Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008–3021, 2020.arxiv.org/abs/2009.01325
  58. [58]Kaiyue Sun, Kaiyi Huang, Xian Liu, Yue Wu, Zihan Xu, Zhenguo Li, and Xihui Liu. T2V-compbench: A comprehensive benchmark for compositional text-to-video generation. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8406–8416. IEEE, 2025.arxiv.org/abs/2407.14505
  59. [59]Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599, 2025.arxiv.org/abs/2501.12599
  60. [60]Ritesh Thawkar, Shravan Venkatraman, Omkar Thawakar, Abdelrahman Shaker, Fahad Khan, Hisham Cholakkal, Salman Khan, and Rao Muhammad Anwer. Ask, Solve, Generate: Self-Evolving Unified Multimodal Understanding and Generation via Self-Consistency Rewards. arXiv preprint arXiv:2606.27376, 2026.arxiv.org/abs/2606.27376
  61. [61]Shuyuan Tu, Qi Dai, Zuxuan Wu, Zhi-Qi Cheng, Han Hu, and Yu-Gang Jiang. Implicit temporal modeling with learnable alignment for video recognition. In ICCV, 2023.arxiv.org/abs/2304.10465
  62. [62]Shuyuan Tu, Qi Dai, Zhi-Qi Cheng, Han Hu, Xintong Han, Zuxuan Wu, and Yu-Gang Jiang. Motioneditor: Editing video motion via content-aware diffusion. In CVPR, 2024.arxiv.org/abs/2311.18830
  63. [63]Shuyuan Tu, Qi Dai, Zihao Zhang, Sicheng Xie, Zhi-Qi Cheng, Chong Luo, Xintong Han, Zuxuan Wu, and Yu-Gang Jiang. Motionfollower: Editing video motion via lightweight score-guided diffusion. arXiv preprint arXiv:2405.20325, 2024.arxiv.org/abs/2405.20325
  64. [64]Shuyuan Tu, Yueming Pan, Yinming Huang, Xintong Han, Zhen Xing, Qi Dai, Chong Luo, Zuxuan Wu, and Yu-Gang Jiang. Stableavatar: Infinite-length audio-driven avatar video generation. arXiv preprint arXiv:2508.08248, 2025.arxiv.org/abs/2508.08248
  65. [65]Shuyuan Tu, Zhen Xing, Xintong Han, Zhi-Qi Cheng, Qi Dai, Chong Luo, and Zuxuan Wu. Stableanimator: High-quality identity-preserving human image animation. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21096–21106. IEEE, 2025.arxiv.org/abs/2411.17697
  66. [66]Shuyuan Tu, Zhen Xing, Xintong Han, Zhi-Qi Cheng, Qi Dai, Chong Luo, Zuxuan Wu, and Yu-Gang Jiang. Stableanimator++: Overcoming pose misalignment and face distortion for human image animation. arXiv preprint arXiv:2507.15064, 2025.arxiv.org/abs/2507.15064
  67. [67]Shuyuan Tu, Yueming Pan, Yinming Huang, Xintong Han, Zhen Xing, Qi Dai, Kai Qiu, Chong Luo, and Zuxuan Wu. Flashportrait: 6x faster infinite portrait animation with adaptive latent prediction. In CVPR, 2026.arxiv.org/abs/2512.16900
  68. [68]Shuyuan Tu, Qi Tian, Zihan Yang, Yue Wu, Xintong Han, Weijie Kong, Jiangfeng Xiong, Jian-Wei Zhang, Zhao Zhong, Liefeng Bo, et al. Baton: Explicit Semantic Blueprints for Joint Video-Audio Generation. arXiv preprint arXiv:2605.25195, 2026.arxiv.org/abs/2605.25195
  69. [69]Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion Model Alignment Using Direct Preference Optimization. arXiv preprint arXiv:2311.12908, 2023.arxiv.org/abs/2311.12908
  70. [70]Huanzhen Wang, Ziheng Zhou, Jiaqі Song, Li He, Yunshi Lan, Yan Wang, and Wenqiang Zhang. ARGen: Affect-Reinforced Generative Augmentation towards Vision-based Dynamic Emotion Perception, 2026.arxiv.org/abs/2604.12255
  71. [71]Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou. Mixture-of-Agents Enhances Large Language Model Capabilities. In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2406.04692
  72. [72]Yibin Wang, Yuhang Zhang, Feng Han, Jiazi Bu, Yujie Zhou, Cheng Jin, and Jiaqi Wang. Unified personalized reward model for vision generation. arXiv preprint arXiv:2602.02380, 2026.arxiv.org/abs/2602.02380
  73. [73]Bing Wu, Chang Zou, Changlin Li, Doujun Huang, Fang Yang, Hao Tan, Jack Peng, Jianbing Wu, Jiangfeng Xiong, Jie Jiang, et al. Hunyuanvideo 1.5 technical report. arXiv preprint arXiv:2511.18870, 2025.arxiv.org/abs/2511.18870
  74. [74]Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. Autogen: Enabling next-gen llm applications via multi-agent conversation. arXiv preprint arXiv:2308.08155, 2023.arxiv.org/abs/2308.08155
  75. [75]Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.09341, 2023.arxiv.org/abs/2306.09341
  76. [76]Yang Wu, Stefano Petrangeli, Ishita Dasgupta, and Yu Shen. CamPilot: A Multi-Agent Cinematic Assistant for Camera-Controlled Movie Generation. arXiv preprint arXiv:2609.10943, 2026.
  77. [77]Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36:15903–15935, 2023.
  78. [78]Jiazheng Xu, Yu Huang, Jiale Cheng, Yuanming Yang, Jiajun Xu, Yuan Wang, Wenbo Duan, Shen Yang, Qunlin Jin, Shurun Li, et al. Visionreward: Fine-grained multi-dimensional human preference learning for image and video generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 11269–11277, 2026.
  79. [79]Ran Xu, Tianci Liu, Zihan Dong, Tony Yu, Ilgee Hong, Carl Yang, Linjun Zhang, Tao Zhao, and Haoyu Wang. Alternating reinforcement learning for rubric-based reward modeling in non-verifiable llm post-training. arXiv preprint arXiv:2602.01511, 2026.arxiv.org/abs/2602.01511
  80. [80]Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. A-MEM: Agentic Memory for LLM Agents. In Advances in Neural Information Processing Systems, volume 38, 2025.arxiv.org/abs/2502.12110
  81. [81]Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Linting Zhu, Mengzhao Chen, Zhiheng Liu, Wei Liu, Qiushan Guo, Weilin Huang, et al. Dancegrpo: Unleashing grpo on visual generation. arXiv preprint arXiv:2505.07818, 2025.arxiv.org/abs/2505.07818
  82. [82]Yu Yang, Yue Liao, Jianbiao Mei, Baisen Wang, Xuemeng Yang, Licheng Wen, Jiangning Zhang, Xiangtai Li, Liang Lv, Hanlin Chen, Botian Shi, Yong Liu, Shuicheng Yan, and Gim Hee Lee. SPIRAL: Self-Evolving Action-Conditioned Video Generation via Reflective Planning Agents, 2026.arxiv.org/abs/2603.08403
  83. [83]Zihan Yang, Shuyuan Tu, Licheng Zhang, Qi Dai, Yu-Gang Jiang, and Zuxuan Wu. ArcFlow: Unleashing 2-Step Text-to-Image Generation via High-Precision Non-Linear Flow Distillation. arXiv preprint arXiv:2602.09014, 2026.arxiv.org/abs/2602.09014
  84. [84]Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. Advances in Neural Information Processing Systems, 38:113222–113244, 2026.arxiv.org/abs/2503.14476
  85. [85]Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-Rewarding Language Models. In Proceedings of the 41st International Conference on Machine Learning, 2024.arxiv.org/abs/2401.10020
  86. [86]Muhammad Osama Zeeshan, Masoumeh Sharafi, Benoît Savary, Alessandro Lameiras Koerich, Marco Pedersoli, and Eric Granger. CLIP-AUTT: Test-time personalization with action unit prompting for fine-grained video emotion recognition. In European Conference on Computer Vision, pages 680–698, 2026.arxiv.org/abs/2603.27999
  87. [87]Guibin Zhang, Hejia Geng, Xiaohang Yu, Zhenfei Yin, Zaibin Zhang, Zelin Tan, Heng Zhou, Zhongzhi Li, Xiangyuan Xue, Yijiang Li, et al. The landscape of agentic reinforcement learning for llms: A survey. arXiv preprint arXiv:2509.02547, 2025.arxiv.org/abs/2509.02547
  88. [88]Rongzhi Zhang, Rui Feng, Zhihan Zhang, Jingfeng Yang, Qingyu Yin, Xin Liu, Zixuan Zhang, Priyanka Nigam, Bing Yin, Tuo Zhao, et al. Qubric: Co-designing queries and rubrics for rl beyond verifiable rewards. arXiv preprint arXiv:2606.03968, 2026.arxiv.org/abs/2606.03968
  89. [89]Jiaxing Zhao, Xihan Wei, and Liefeng Bo. R1-omni: Explainable omni-multimodal emotion recognition with reinforcement learning. arXiv preprint arXiv:2503.05379, 2025.arxiv.org/abs/2503.05379
  90. [90]Mingzhe Zheng, Dingjie Song, Guanyu Zhou, Jun You, Jiahao Zhan, Xuran Ma, Xinyuan Song, Ser-Nam Lim, Qifeng Chen, and Harry Yang. CML-Bench: A Framework for Evaluating and Enhancing LLM-Powered Movie Scripts Generation. arXiv preprint arXiv:2510.06231, 2025.arxiv.org/abs/2510.06231
  91. [91]Mingzhe Zheng, Yongqi Xu, Haojian Huang, Xuran Ma, Yexin Liu, Wenjie Shu, Yatian Pang, Feilong Tang, Qifeng Chen, Harry Yang, et al. VideoGen-of-Thought: Step-by-step generation multi-shot video with minimal manual intervention. NeurIPS NextVid Workshop, 2025.
  92. [92]Mingzhe Zheng, Weijie Kong, Yue Wu, Dengyang Jiang, Yue Ma, Xuanhua He, Bin Lin, Kaixiong Gong, Zhao Zhong, Liefeng Bo, et al. Manifold-aware exploration for reinforcement learning in video generation. arXiv preprint arXiv:2603.21872, 2026.arxiv.org/abs/2603.21872

Paper details

vlnc:W_sxas2mh2
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

OpenAI ChatGPT, Codex, Qwen3-VL, Claude

View statement
Looking for related papers…
Contents