WorksComputer Science
EmoReward: Self-Evolving Reward Calibration for Emotional Video Generation RL
Abstract
Group Relative Policy Optimization (GRPO) has advanced generative model post-training in verifiable domains such as code and mathematics, but evaluating video quality, especially emotional expression, often depends on subjective aesthetics and has no uniquely correct target. Existing approaches typically rely on fixed rewards for visual quality, motion quality, and text alignment that poorly capture emotional intent and can invite reward hacking, while RLHF is difficult to scale and subject to individual aesthetic differences that complicate the calibration of facial, body motion, and chromatic cues across narrative contexts. We introduce EmoReward, a new self-evolving training mechanism that distills evidence reliability from reward traces collected during an actual training run to adaptively guide multi-reward weight allocation in subsequent training by determining which emotional evidence to trust and how much to weight it. EmoReward first plans specific grounding cues from the emotional prompt to guide evaluation, then uses heterogeneous signal primitives to score the generated video during the act stage. Reflective Reward Gating calibrates their contributions through instance-level gates for phase, face confidence, conflict, and subtlety, then renormalizes weights over the active set to produce the sample's fused reward. RewardMemory then estimates each primitive's reliability from rollout scores and outcomes within each emotional regime in the finished training run, then fixes bounded multipliers that calibrate fusion weights in the next training run without updating reward models. Experiments on HunyuanVideo 1.5 show improvements across multiple RL algorithms and compare EmoReward with other agentic reward methods under SAGE-GRPO. Ablations and transfer experiments on SD 3.5 and FLUX further support adaptive reward calibration for visual generation.
*Work done during internship at Tencent Hunyuan
‡Project leader †Corresponding author
Introduction
Reinforcement learning (RL), especially Group Relative Policy Optimization (GRPO), has advanced model post-training in verifiable domains such as code and mathematics, where correctness provides a direct learning signal [20, 54, 59, 84]. In contrast, evaluating video quality, especially emotional expression, often depends on subjective aesthetics and has no uniquely correct target. Visual post-training adapts diffusion and flow-based generation models through RL and preference objectives [3, 16, 33, 49, 69, 81, 92]. Learned evaluators provide scores for image preference and video quality [21, 26, 34, 42, 77, 78], but a visually clear, temporally stable video can match its prompt without conveying the intended emotion. Facial analysis, dynamic expression recognition, and multimodal emotion models supply complementary evidence [2, 4, 8, 30, 89], whose reliability varies with context: facial cues require visible faces, strong motion can undermine restrained sadness, and saturated colors may reflect style rather than emotional intent. This raises a central question: Which emotional evidence should the reward system trust in the current video, and how can it adaptively determine how much weight to assign to that evidence?
Agentic systems coordinate interaction, refine outputs through feedback, and organize evaluation into rubrics [19, 43, 56, 71, 74]. These capabilities motivate adaptive evaluation, yet existing video GRPO methods typically rely on fixed rewards for visual quality, motion quality, and text alignment that inadequately capture emotional intent. Reinforcement learning from human feedback (RLHF) and preference optimization address subjective goals [9, 45, 51, 57], but human annotation is costly at scale, and aesthetic differences complicate the calibration of emotional cues across contexts. Fixed fusion can mix informative, missing, and saturated signals, allowing policy updates to exploit scoring shortcuts rather than improve emotional fidelity. This motivates reward calibration at two complementary levels: assessing which cues are valid within each video, and learning from a completed training run how much those cues should influence the reward in the next run.
We introduce EmoReward, a self-evolving training mechanism for emotional video-generation RL (Figure 1). It distills evidence reliability from reward–outcome traces collected during an actual, completed training run, then uses this experience to guide multi-reward weight allocation in the next run to decide which emotional evidence to trust and how much to weight it. Plan translates emotional intent into an EmoScript specifying affect targets, facial and body cues, and an expression window; Act scores generated videos with heterogeneous primitives, keeping criterion scores separate from evidence validity. Within each video, Reflective Reward Gating calibrates these cues using the training phase, face confidence, semantic conflict, and subtlety, then renormalizes weights over active providers. After a training run, RewardMemory estimates provider reliability for each emotional regime from traces. Providers with stronger agreement are given larger weight multipliers for the next run, with limits on the size of the adjustment. These multipliers and all reward models stay fixed throughout that run; for each new video, reflection further downweights or excludes missing or unsuitable cues before the active scores are combined into a weighted average. EmoReward integrates with several RL algorithms alongside general video rewards, using training experience to calibrate reward allocation without retraining the evaluators.
Figure 1. From fixed scoring to contextual reward calibration. The reward design progresses from a fixed evaluator to a heterogeneous provider pool and then to EmoReward. Reflective Reward Gating selects relevant evidence for the current video; RewardMemory calibrates provider trust from experience. All three stages ultimately return a scalar reward. The reward providers remain frozen while RL updates the video generation model.
We evaluate EmoReward on HunyuanVideo 1.5 [73] with DanceGRPO [81], FlowGRPO [33], and SAGE-GRPO [92], finding higher held-out emotion accuracy and rank scores across all three RL algorithms and higher VideoAlign Overall with FlowGRPO and SAGE-GRPO. Under a shared architecture and training budget, RewardMemory yields the highest reliability selectivity among the seven reward policies. Ablation studies and mechanism analyses examine the contributions of reflective gating, provider pools, and RewardMemory (§ 4.2). Transfer experiments on Stable Diffusion 3.5 and FLUX show gains in selected configurations alongside objective-dependent trade-offs, while user studies show that EmoReward receives the highest proportion of Good ratings and the lowest proportion of Bad ratings for emotional fidelity, temporal coherence, and visual quality.
Our contributions are threefold.
We introduce EmoReward, the first self-evolving training mechanism for emotional video generation RL that calibrates reward providers across training runs without updating reward model parameters.
We propose RewardMemory, the first memory designed for multi-reward systems in video generation. It distills evidence reliability into bounded fusion weights, enabling iterative calibration of RL reward signals across training runs.
We introduce a cross-run self-evolution paradigm for post-training multimodal foundation models: a completed training run informs reward allocation in the next, making training experience reusable at the run level. Extensive experiments across three video RL algorithms and two image backbones, with ablations and human evaluations, validate EmoReward’s effectiveness.
Related Work
From verifiable RL to agentic feedback for non-verifiable rewards. RLHF learns subjective preferences [9, 45], whereas verifiable RL exploits correctness checks [20, 84]. Agentic evaluators [12, 87] and adaptive rubrics [19, 79, 88] structure open-ended feedback. Related approaches route experts [44], refine responses [43], evolve rewards [39, 85], or organize experience [56, 80]. JitRL derives closed-form policy updates from retrieved action advantages [29]. EmoReward instead transfers reward–outcome evidence across completed training runs to calibrate frozen providers, combining sample-level gating with persistent fusion weights.
Agents for visual content creation. Agentic visual creation organizes generation into planning, tool use, and feedback-driven refinement. Video agents coordinate multi-shot narratives, script evaluation, audiovisual planning, and learned camera control [15, 22, 68, 76, 90, 91]. Image agents combine knowledge retrieval [5, 18], learned creation and editing policies [27, 31], and experience reuse [6, 7]. Other approaches jointly improve generation and evaluation [32, 60]. EmoReward naturally achieves a self-evolving mechanism in video generation RL training, rather than building a workflow or retraining reward models.
RLHF for visual generation and the emotion–reward gap. Visual reward models capture preferences and multidimensional quality [26, 34, 42, 72, 75, 77, 78]. Optimization using rewards and preferences extends from images [3, 16, 49, 69] to video and flow models [33, 35, 81, 92], with benchmarks assessing quality and compositional alignment [24, 36, 58]. Related 3D, audiovisual, and objectives for human animation address complementary aspects of visual generation [23, 28, 64, 65]. Motion editing and transfer [41, 62, 63], shape-aware and consistent image editing [37, 40], and pose- and identity-preserving animation [46, 66, 67] offer finer control over visual content. Distillation [17, 25, 83] and feature reuse [38] improve generation efficiency, complementing reward-based post-training. Affect-grounded augmentation [70] and reflective action-conditioned generation [82] further connect feedback with video synthesis. Temporal video modeling and dynamic facial-expression recognition provide additional perceptual foundations [4, 61]. Emotion understanding supplies multimodal evidence [8, 30, 89] and grounded affect representations [2, 14, 52]; EmoReward addresses how their reliability should govern reward fusion across scenes and training runs.
Method
Preliminaries
Multi-reward optimization with GRPO. Let be a prompt and a generated video from a policy ; a scalar reward is . For a prompt , GRPO samples a group and forms standardized group advantages
which enter the clipped, KL-regularized objective in Appendix § 13. The policy update depends on reward differences inside the group. A common additive constant cancels exactly; positive rescaling changes only the relative effect of . A coefficient of variation measures dispersion, not policy-learning signal quality; we also assess within-prompt ranking agreement and downstream quality.
Why static emotion fusion fails. Given reward primitives with normalized scores , a fixed fusion is with . This assumes a fixed trust allocation across scenes, although facial validity and the relevance of motion depend on the intended emotion. Writing each primitive as a constant plus a group-varying part and splitting into an active set and a dead/saturated set ,
the constant terms cancel from the centered GRPO numerator. Removing them alone need not change the standardized advantage. Calibration matters when provider validity, errors, or relative contributions vary across candidates, or when the emotion branch is mixed with another reward. EmoReward addresses these cases through gates and memory-conditioned fusion.
Overview
EmoReward grounds emotional intent, scores visual evidence, and calibrates provider trust:
Here is an EmoScript, denotes evidence validity, and is the emotional-context signature. Reward-Memory stores completed reference-run traces; Evolve produces and exports the corresponding regime multipliers (§ 3.6). Reflect combines these fixed multipliers with current-video gates. The resulting emotion reward is mixed with a general video reward:
where is the evaluation score supplied by the original reward system, for example a general video-quality evaluator [34]; control its mixture with the calibrated emotion reward, with . Figure 2 connects these stages through their shared fusion interface.

Figure 2. EmoReward calibrates emotional evidence within videos and across training runs. Plan translates intent into evaluation criteria, and Act obtains provider scores and evidence validity. Reflect combines current-video gates with memory-derived multipliers and normalizes over active providers. Evolve estimates contextual reliability from a completed reference run to construct the fixed multipliers used in subsequent training. The calibrated emotion reward complements the original video reward; GRPO updates the video generation model while the providers and memory table remain fixed.
Plan and Act: Emotional Grounding
Plan: prompt-conditioned criteria. The planner converts the prompt into an EmoScript independently of RewardMemory:
The implemented representation contains scalar valence and arousal targets, target facial AU intensities, body cues, and an expression window. The memory regime is inferred later from observed context. Continuous VAD trajectories and dominance targets are outside this implementation. The EmoScript separates intended affect from its observable cues. For example, restrained sadness need not imply large motion or an exaggerated face: target intensities, body cues, and the expression window define what to assess, while reliability is determined from the generated evidence and completed-run experience.
Act: observation and scoring. Act evaluates a video already sampled from ; it does not generate or revise the video. Each provider returns
where is provider ’s scoring operator, which aggregates frame-level measurements or evaluates the whole clip. The seven fusion providers comprise a VLM judge, facial AUs, body cues, temporal evidence, CLIP-AUTT, R1-Omni, and motion naturalness; chromatic atmosphere and kinetic intensity are auxiliary readouts, giving nine observation channels. Score measures criterion agreement, while marks usable evidence. A missing face channel’s neutral default is not evidence and is excluded through its validity mask. Auxiliary readouts do not enter the seven-provider calibration policy. Appendix § 10 specifies scoring interfaces; Figure 6 illustrates the toolkit studied in § 4.2.
Figure 6. Figure 6 Ablation of provider-pool construction. Three frames per clip compare four configurations on the same prompt. See Appendix § 15 for quantitative results.
Reflect: Instance-Level Calibration
Reflect computes provider-specific gates from the current evidence, prompt, and training step. The factorization is
with neutral factors for channels to which a condition does not apply. Phase denotes the training-step warmup, face gating controls face-dependent evidence, conflict gating attenuates disagreement between semantic providers, and subtlety gating restrains intensity-sensitive cues. Motion naturalness uses its observation confidence. These controls are computed, not learned (Appendix § 11).
Evolve: RewardMemory
Algorithm 1 EmoReward reward computation with experience-calibrated weights
Require: prompt , generated video , fixed table from a completed reference run, base weights
Build EmoScript
Compute primitive scores and validity ▷ Eq. (6)
Compute instance gates ▷ Eq. (7)
Form state signature
Look up regime in ▷ Appendix § 12
if a supported regime or global entry is available then
Read fixed multipliers , constructed by Eq. (10)
else
▷ cold start: fall back to base
end if
Form active weights
Fuse active primitives with renormalization ▷ Eq. (8)
Compute total reward ▷ Eq. (4)
Log the reward trace for inspection; keep fixed
return
Algorithm 2 RewardMemory construction from reflective reward traces
Require: Traces from a completed reference run, provider count , temperature , log-multiplier cap , minimum support
for each context and the global fallback do
Select matching traces; use the global set if the context has fewer than traces
for each provider do
; use zero for constant scores
end for
for each provider do
end for
Store the context multipliers
end for
Export once; freeze the table for a subsequent generator-training run
Evidence-dependent trust. The phase factor schedules reward evidence according to training progress, separately from the within-video expression window. Face gating limits channels requiring facial evidence; conflict and subtlety gating temper disagreeing semantic assessments and intensity-sensitive cues. Their provider-specific application changes relative influence; a shared positive attenuation would largely cancel under normalization.
Composition with experience. For a fixed base weight , define the effective contribution . Reflective Reward Gating uses ; EmoReward uses the fixed table produced by Evolve. With , active-provider normalization gives
Invalid evidence receives no mass. A common positive factor cancels except through the stabilizer; calibration acts through relative provider contributions and differences between videos. Regime selects a table entry; gate controls current evidence validity. A provider favored by memory still contributes nothing when its current evidence is invalid. With no memory entry, unit multipliers recover Reflect-only fusion; the scoring interfaces and the policy objective remain the same.
Evolve: RewardMemory
RewardMemory treats reward allocation as a decision over emotional context : the providers are actions, their fusion weights form a policy, and a stored outcome supplies a non-parametric reliability signal. Experience is collected during a completed reference training run with Reflective Reward Gating. Each trace records context, scores, gates, weights, and outcomes. Evolve groups these traces into a regime neighborhood , with a global fallback for insufficient support, and computes
Reliability uses raw provider scores; constant scores receive zero correlation. It measures agreement with the reference outcome, not independent emotional fidelity. The centered advantage tilts the base allocation through bounded multipliers:
This is the closed-form optimum of a KL-regularized allocation with clipped utilities (Appendix § 13). The exported compose with current instance gates in Eq. (8). The table is fixed throughout the next training run: gradients update the video generation model alone, and the table is read-only. Algorithms 1 and 2 separate reward computation from between-run memory construction. Appendix § 12 explains context grouping and table construction; Appendix § 16 provides a trace-based worked example.
Experiments
Experimental settings. We use HunyuanVideo 1.5 with DanceGRPO, FlowGRPO, and SAGE-GRPO, fusion , batch size 2, and group size . We use Qwen3-VL to create emotional prompts for training and use Claude to newly create 302 prompts in same style for evaluation (Appendix § 8). GT denotes the pretrained video generation model without RL. The controlled seven-policy comparison retains its separate evaluation protocol. VideoAlign supplies and the reference outcome . Memory calibration uses , , a log-multiplier cap , and minimum context support .
Main Experiments
Quantitative evaluation. We compute Directional Emotion Accuracy (DEA) and Target Emotion Rank Score (TERS) with a frozen CLIP ViT-H/14 encoder [53], which is different from the separate CLIP-AUTT model [86] used as the facial reward during training (details in Appendix § 14.§ 3). Table 2 compares EmoReward with each RL algorithm’s base reward and includes the pretrained video generation model (GT) as a reference. All three RL baselines score below GT on DEA and TERS. EmoReward improves both metrics under every RL algorithm, with TERS gains of 0.016–0.031. FlowGRPO and SAGE-GRPO also surpass GT on both metrics. The gains have the same direction with a second frozen CLIP encoder. FlowGRPO and SAGE-GRPO improve on all four VideoAlign measures. Both visual metrics increase with EmoReward under SAGE-GRPO, with mixed changes across the other algorithms. Figure 4 illustrates the changes in emotional expression across all three algorithms.
| Method | DEA | TERS | Overall | VQ | MQ | TA | PickScore | CLIPScore |
| GT (reference) | 0.655±0.029 | 0.7904±0.0145 | -1.1938±0.0520 | -0.5899±0.0255 | -0.4289±0.0220 | -0.1750±0.0363 | 0.7059±0.0018 | 0.5623±0.0037 |
| DanceGRPO | 0.605±0.030 | 0.7640±0.0153 | -1.2812±0.0513 | -0.6392±0.0230 | -0.3867±0.0235 | -0.2554±0.0357 | 0.7177±0.0018 | 0.5675±0.0036 |
| + EmoReward | 0.632±0.030 | 0.7824±0.0149 | -1.3930±0.0559 | -0.6279±0.0284 | -0.3833±0.0243 | -0.3818±0.0340 | 0.6986±0.0017 | 0.5599±0.0037 |
| FlowGRPO | 0.621±0.030 | 0.7862±0.0144 | -0.7719±0.0569 | -0.3567±0.0290 | -0.2753±0.0271 | -0.1398±0.0361 | 0.7061±0.0019 | 0.5634±0.0036 |
| + EmoReward | 0.670±0.029 | 0.8018±0.0142 | -0.5376±0.0556 | -0.3398±0.0291 | -0.2302±0.0275 | 0.0323±0.0356 | 0.7115±0.0018 | 0.5605±0.0037 |
| SAGE-GRPO | 0.617±0.030 | 0.7881±0.0143 | -0.8269±0.0532 | -0.4106±0.0261 | -0.3275±0.0263 | -0.0888±0.0360 | 0.7091±0.0019 | 0.5624±0.0036 |
| + EmoReward | 0.674±0.029 | 0.8193±0.0132 | -0.2624±0.0600 | -0.1710±0.0304 | -0.0926±0.0324 | 0.0012±0.0364 | 0.7107±0.0018 | 0.5663±0.0035 |
Table 2. EmoReward across video RL algorithms. DEA uses the 261 prompts with a usable opposite emotion. TERS ranks the target emotion on all 302 prompts. All metrics report mean ± SEM across prompts. GT is the pretrained no-RL reference. Bold marks an increase over its corresponding RL baseline.

Figure 4. Qualitative comparison across video RL algorithms. Each pair shares a prompt and seed. Red boxes mark the hands and facial cues. (Top) A bowed head and hands held over the face convey grief. (Middle) The reporter’s expression tightens as she reacts to the screen. (Bottom) A broad smile gives way to a fixed stare.

Figure 3. Emotional expression in generated videos. Eight clips are shown at three moments each. (Top) A performer looks up from a letter. (Middle rows) Gaze, gesture, and expression convey emotion in scenes such as a violin performance and a rooftop piano scene. (Bottom) Laughter fades into a quiet gaze.
Qualitative analysis. Figure 4 compares base and EmoReward results for each RL algorithm with matched prompts and seeds. For DanceGRPO, the man stays bowed with his face covered with better visual effects, and for FlowGRPO the reporter’s strained expression is clearer from a more frontal angle, while under SAGE-GRPO the musician’s smile fades into a fixed stare. These examples complement the clip-level emotion scores by showing the order in which the visible cues emerge.
Ablations and Mechanism Analysis
Emotion-category analysis. Figure 5 compares GT with SAGE-GRPO plus EmoReward across eight emotion categories. All four VideoAlign measures increase in every category, while PickScore and CLIPScore vary. This comparison captures the effect of post-training as a whole.
Figure 5. Figure 5 Emotion-category quality. GT (gray) versus SAGE-GRPO with EmoReward (red); mean SEM. Axes are truncated; categories overlap within the evaluated set.
Reward-policy ablation. We compare seven reward policies with a fixed backbone and training budget (Table 1). RewardMemory leads in Sel. (0.428), GD, and GR, while Reflect leads in EF, the ImageReward preference proxy. measures relative dispersion. The predefined core and emo scores are descriptive summaries, since min–max scaling can magnify small differences (Appendix § 14).
Table 1 Reward-policy ablation under SAGE-GRPO. Controlled policies share the backbone and training budget. Citations identify components. SNR is relative dispersion. Bold marks diagnostic and pillar maxima. Core and emo are shown for reference.
| Reward method | SNR | Sel. | EF | GD | GR | core | emo |
| VideoAlign [34] | 0.378 | 0.307 | 0.518 | 0.000 | 0.197 | 0.238 | 0.308 |
| Single provider [89] | 2.403 | 0.314 | 0.454 | 0.305 | 0.147 | 0.302 | 0.340 |
| Random routing [74] | 1.105 | 0.340 | 0.000 | 0.109 | 0.146 | 0.085 | 0.064 |
| Round-robin [74] | 1.217 | 0.376 | 0.291 | 0.126 | 0.381 | 0.266 | 0.272 |
| Fixed fusion [78] | 0.773 | 0.371 | 0.026 | 0.060 | 0.653 | 0.246 | 0.191 |
| EmoReward (w/o RewardMemory) | 0.451 | 0.293 | 1.000 | 0.011 | 0.000 | 0.337 | 0.503 |
| EmoReward (w/ RewardMemory) | 7.022 | 0.428 | 0.767 | 1.000 | 1.000 | 0.922 | 0.884 |
Table 1.
Reward-provider pool. Figure 6 compares VideoAlign, semantic emotion, motion naturalness, and a balanced pool. Weights and curricula co-vary with the pool (Appendix § 15). Seven channels enter memory calibration. Chromatic and kinetic cues supply auxiliary readouts.
Memory-conditioned trust. The largest global multipliers go to macro (1.833) and motion (1.510). Three constant channels receive 0.778. The gates still exclude invalid evidence. Macro shares an evaluator with the reference outcome, so its correlation reflects objective agreement. Appendices § 12 and § 16 describe and illustrate the allocation.
Generalization to image backbones
We test SD3.5 and FLUX across three RL algorithms with EmoReward. EmoReward improves HPSv3 for SD3.5, while FLUX’s baseline still leads on the same metric. We tested on several prompts and find EmoReward enhances the text following ability and reduce deformities. Appendix § 17 details the settings and qualitative comparisons.
Human evaluation
Eighteen participants rated three methods on 30 prompts for emotion fidelity, temporal coherence, and visual quality. They used the same Bad/Normal/Good scale for each criterion. SAGE-GRPO receives more Good and more Bad ratings than GT on all three criteria. EmoReward has the highest Good shares (60.0%, 54.3%, 57.2%), compared with 53.0%, 51.5%, and 45.0% for SAGE-GRPO, and the lowest Bad shares (Table 3). The same participants rated multiple prompts across methods, and Appendix § 9 details the protocol.
Table 3 Human evaluation. Rating percentages from 18 participants on 30 prompts. Each method and criterion block contains 540 judgments. Bold marks the highest Good share per criterion; percentages are rounded to one decimal place.
| Method | Emotion Fidelity | Temporal Coherence | Visual Quality | ||||||
| Bad | Normal | Good | Bad | Normal | Good | Bad | Normal | Good | |
| GT | 21.9% | 34.8% | 43.3% | 23.5% | 37.8% | 38.7% | 25.2% | 38.5% | 36.3% |
| SAGE-GRPO | 31.5% | 15.6% | 53.0% | 28.9% | 19.6% | 51.5% | 30.9% | 24.1% | 45.0% |
| SAGE-GRPO + EmoReward | 15.4% | 24.6% | 60.0% | 20.2% | 25.6% | 54.3% | 15.9% | 26.9% | 57.2% |
Table 3.
Conclusion
In this paper, we proposed EmoReward, a self-evolving reward calibration framework for emotional video generation reinforcement learning. EmoReward combines instance-level Reflective Reward Gating with RewardMemory, which transfers provider-reliability estimates from a completed training run to subsequent optimization without updating reward-model parameters. Experiments across video optimizers, complemented by ablations, image-transfer analyses, and human ratings, support the utility of adaptive calibration while identifying objective-dependent trade-offs. These findings suggest that completed-run experience can inform the allocation of trust among heterogeneous reward providers, providing a foundation for further research on reward-system evolution across training.
LLM Usage Statement
OpenAI ChatGPT and Codex assisted with manuscript editing and organization, checks of the mathematical exposition, and preparation of figure layouts and LaTeX source. We used Qwen3-VL to create emotional prompts for training and Claude to author a separate dataset of prompts for evaluation. The main comparison uses 302 prompts from this dataset (Appendix § 8). The vision–language and generative models used in the reward pipeline and experiments are described in the paper and supplement. Experimental comparison images and video frames are outputs of the evaluated models; no additional media were generated to illustrate experimental outcomes. The authors are responsible for the scientific claims, evidence, and final manuscript.
Limitations and Future Work
Beyond emotional video generation. EmoReward studies a training-adaptive reward system that transfers evidence about provider reliability between generator-training runs without updating reward-model parameters. The present evaluation focuses on emotion, where different scenes call for different combinations of facial, gestural, semantic, and motion evidence. A broader application would require a task-appropriate provider pool and a meaningful outcome for calibration. Action fidelity and audio–visual correspondence are natural visual extensions; language models and tool-use agents could similarly combine semantic judges, execution checks, and trajectory-level assessments. Whether the same calibration rule transfers effectively remains an empirical question, especially when providers share errors or outcomes favor only part of the task.
Iterative evolution across training runs. A completed training run can be treated as a unit of experience for its successor. Repeating this procedure could produce a sequence of generators and reward-allocation tables, with each generation contributing new traces. Our experiments evaluate a reference-to-calibrated transition, not sustained improvement across many generations. Such an extension must address distribution shift, stale evidence, and feedback loops that reinforce a proxy’s errors. Retaining diverse historical traces and evaluating held-out outcomes would be important tests of whether repeated calibration remains useful.
Model-level human feedback. The trace representation also suggests a way to carry human assessments of a completed model into subsequent training. For example, a reviewer who finds movement too weak could request greater emphasis on suitable motion providers in the affected contexts; excessive motion could motivate the opposite adjustment. This would complement sample-level preference feedback with structured assessments of a model’s final behavior, translated into bounded changes to the next run’s reward allocation. Mapping subjective feedback to providers and regimes, reconciling different raters, and preventing compensating failures remain open design questions. The ordinal study reported here evaluates outputs; it does not implement this proposed feedback mechanism.
Training and Evaluation Prompts Preparation
Training corpus. We use Qwen3-VL-30B-A3B [1, 50] to write emotional video prompts. Our 62-category taxonomy draws on Plutchik’s eight primary emotions and their derivatives, Cowen and Keltner’s 27–28-category semantic-space studies, Shaver’s three-level hierarchy as presented in Parrott’s collection, Ekman and Friesen’s Facial Action Coding System (FACS), Hume AI’s facial expression model, and GoEmotions’ 28 labels including neutral [10, 11, 13, 14, 47, 48, 55]. Each category has a Chinese and an English label and specifies valence, arousal, facial action units, body cues, and temporal dynamics. We assign a seed to each row, sample a category from that seed, and use the corresponding label to condition prompt generation. The resulting corpus contains 10,000 distinct prompts to diversify rollouts across training. Each prompt is used at most once in a complete online on-policy training run.
Example. For , we draw Love/Affection, with positive valence and medium arousal. Its required facial actions are AU6 (cheek raiser) and AU12 (lip corner puller). The body cues are intimate distance (0–45 cm), gentle stroking, and a soft, nonthreatening gaze. The target expression develops slowly and sustains a sense of warmth. The system prompt below is shown in English translation.
Write a visual description for text-to-video generation in Simplified Chinese. Output a single paragraph with no title, list, or quotation marks. Center the scene on people, with visible facial expressions, gaze, lip movements, and body language. Show subtle shifts in emotion and microexpressions. Convey love or affection through the scene and the performances and avoid repeating the emotion label. Write roughly 180–350 Chinese characters. Use cinematic language to describe one or two of framing, camera angle, lighting, and camera movement, and establish the atmosphere of the setting. Exclude prohibited content and excessively graphic violence. Output only the Chinese paragraph, with no explanatory preface or closing note.
Random seed for inspiration only (do not reproduce the number) 156744132
Showing the emotion through acting gives the facial and body reward providers visible cues to score. Varying the inspiration seed encourages different scenes when the same category is sampled again. An example rollout prompt is shown below in English. The slanting sunlight of dusk filters through the old window lattice, bathing the room in a warm orange glow. A middle-aged couple sits on worn wicker chairs; she is bent over, mending an old sweater, her fingers deftly guiding the needle and thread. He watches her quietly, his fingers absently tracing the rim of his cup. . . Suddenly, she looks up at him, a faint smile lifting the corners of her lips, fine lines crinkling at the edges of her eyes—gentle ripples of time-worn tenderness. . . he reaches out, gently brushing a strand of her hair, his touch slow and reverent. . . they share a glance, a silent understanding flowing between them. The camera slowly pushes in, focusing on the delicate smile lines at the corner of her eye and the quiet, shimmering light deep within his gaze.
Evaluation corpus. We used Claude to create a separate emotional prompt dataset for evaluation, following the training corpus’s emotion specifications and cinematic style. These prompts were held out from RL training. Table 2 uses the same 302-prompt subset for each RL algorithm and its matched base and EmoReward runs. Appendix § 14.§ 3 describes how the automatic metrics are aggregated, including DEA’s requirement for a usable opposite emotion.
Human Evaluation: Ordinal Ratings
Study and comparison groups. The study includes 18 participants and 30 prompts, with one video per prompt from each of three methods: the pretrained video generation model without RL (GT), SAGE-GRPO, and SAGE-GRPO with EmoReward. GT denotes the pretrained model baseline. Participants consented to take part in the study. Participants assessed each output separately on three criteria using Bad, Normal, or Good. The supplied aggregate results contain 540 judgments for each method–criterion combination, consistent with the full participant–prompt product. Across all three methods and criteria, there are 4,860 categorical judgments. These ordinal counts are separate from the earlier pairwise preference protocol; pairwise choices are not converted into categorical ratings.
Criteria. Emotion fidelity concerns how convincingly the video conveys the prompt’s intended affect, including whether expression and gesture fit its emotional tone. Temporal coherence concerns continuity of expression, identity, and movement through the clip. Visual quality concerns perceptual detail, anatomy, and visible artifacts. The categories distinguish poor, intermediate, and good quality on each axis. They are rated separately rather than collapsed into a single preference or an assumption that stronger emotional intensity is always better.
Aggregation and rounding. For method , criterion , and rating category , the percentage is:
Table 3 rounds each percentage independently to one decimal place and bolds only the highest Good percentage within each criterion. Rounding may make a three-category total differ from 100.0% by 0.1 percentage points. The raw integer counts are given in Table 4 and were not rounded before normalization. All nine aggregate blocks have the stated denominator; the totals alone do not establish participant-level completeness or the absence of duplicate records.
| Method | Emotion fidelity | Temporal coherence | Visual quality | ||||||
| Bad | Normal | Good | Bad | Normal | Good | Bad | Normal | Good | |
| GT | 118 | 188 | 234 | 127 | 204 | 209 | 136 | 208 | 196 |
| SAGE-GRPO | 170 | 84 | 286 | 156 | 106 | 278 | 167 | 130 | 243 |
| SAGE-GRPO + EmoReward | 83 | 133 | 324 | 109 | 138 | 293 | 86 | 145 | 309 |
Table 4. Table 4 Aggregate ordinal counts. Each method–criterion block contains 540 ratings. The counts reproduce the percentages in main-paper Table 3.
Reading the distributions. Relative to SAGE-GRPO, EmoReward’s Good share increases by 7.0, 2.8, and 12.2 percentage points for emotion fidelity, temporal coherence, and visual quality, respectively. These differences are computed from the unrounded counts. Its Bad share falls from 31.5% to 15.4% for emotion fidelity, from 28.9% to 20.2% for temporal coherence, and from 30.9% to 15.9% for visual quality. The complete distribution matters: compared with GT, SAGE-GRPO has more Good ratings but also more Bad ratings on every criterion, so a Good-only summary would omit part of the observed pattern.

Figure 7. Interface for ordinal human evaluation. The interface presents the prompt and generated video with independent ratings of emotional fidelity, temporal coherence, and visual quality, each using three ordered categories: Bad, Normal, and Good. This reconstruction illustrates the reported rating protocol rather than the original data-collection interface.
Statistical scope. The present report uses the supplied aggregated counts. It does not treat repeated judgments from the same participant or prompt as independent observations. Participant- and prompt-level records are needed for uncertainty estimates that respect both sources of dependence; no confidence interval or significance claim is inferred from the aggregate margins. The aggregates do not document recruitment, presentation order, blinding, exclusions, or prompt-selection provenance. Those collection details must be established from the study records rather than inferred from the rating totals.
Act: Provider Interface and Reference Scoring Rules
This appendix specifies the scoring operators used by the provider interface in Eq. (6). The six primitive families (facial expression; semantic emotion understanding; body and temporal evidence; chromatic atmosphere; kinetic intensity; motion naturalness) are instantiated by nine observation channels; Table 5 maps each family to its provider(s), the backing estimator, and the defining equation. Throughout, is the number of sampled frames, the face-detected subset, the EmoScript time weight (per-window kernels over ), and a bounded exponential score. Each provider returns a score and a validity signal .
| Family | Provider | Estimator / signal | Eq. |
| Facial | AU matching | Facial action-unit head | 12 |
| Facial | CLIP-AUTT | Temporal AU–text similarity | 13 |
| Semantic | R1-Omni | Valence–arousal estimation | 14 |
| Semantic | Macro | Alignment and AU compliance | 15 |
| Body | Body cues | Body-marker satisfaction | 16 |
| Temporal | Peak alignment | AU peak versus target window | 18 |
| Appearance | Chromatic | Saturation and warm hue | 19 |
| Appearance | Kinetic | Inter-frame differences | 20 |
| Motion | Naturalness | Optical-flow consistency | 21 |
Table 5. Table 5 Primitive family, provider, backing estimator, and defining equation. The two appearance channels (chromatic, kinetic) are always-on lightweight signals; the remaining seven form the memory support whose per-provider reliability is reported in the calibration analysis (Table 8).
| Provider | Macro | Motion | CLIP | R1-Omni | Micro | Body | Temporal |
| 0.429 | 0.332 | 0.073 | 0.047 | 0.000 | 0.000 | 0.000 | |
| 1.833 | 1.510 | 0.899 | 0.854 | 0.778 | 0.778 | 0.778 |
Table 8. Table 8 RewardMemory calibration. Provider reliability and unnormalized multipliers, estimated from 3,232 reflective reward traces (, ). Macro includes VideoAlign information, so its association is partly shared with the outcome.
Facial expression: AU matching and expression similarity
AU matching (micro). A face detector followed by a multi-task action-unit head emits eight AU intensities per detected face, for . Against the EmoScript target intensities we score agreement by a per-AU intersection-over-union overlap (rather than an kernel, so a target and an observation are rewarded for occupying the same activation interval), averaged over face frames:
with AU compliance and validity and . The provider is inactive when no frame yields a face with AU readings. In our implementation, .
Expression similarity (clip_autt). A temporal vision–language encoder embeds a -frame clip into and compares it to action-unit text-prompt embeddings by cosine similarity, mapping the mean similarity from to :
An auxiliary discrete head additionally yields a negative-affect probability , from which the scorer derives and ; the linear rescale in Eq. (13) (rather than clip) preserves the ordering on clips where . In our implementation, , , , and .
Semantic emotion understanding: Valence–arousal reading and the alignment seed
Valence–arousal reading (r1omni). A multimodal emotion-understanding model produces a free-text emotion description that is parsed to an emotion label and continuous valence–arousal estimates . Given EmoScript targets , the reward supports three modes (default: valence):
Text-alignment seed (macro). When a general video–text alignment model is available it blends its text-alignment head with AU compliance weighted by face rate ; otherwise a heuristic reads compliance directly:
In our implementation, , , and . The confidence belongs to the heuristic reflector. Because the alignment-seeded branch shares an evaluator with the reference outcome, its correlation cannot serve as independent emotion validation.
Body and temporal evidence (body, temporal)
The body provider counts each EmoScript body marker satisfied by at least one frame; the temporal provider locates the frame of peak expected-AU energy and scores its distance to the EmoScript window :
The window stabilizer is in our implementation. These scores require their corresponding observations; unavailable evidence receives the neutral fallback and validity zero. We use . In the analyzed traces, the micro, body, and temporal channels are constant at 0.5; this does not imply that every generated video lacks a face or body.
Chromatic atmosphere (chromatic)
A per-frame chromatic signal blends mean saturation with the fraction of warm-hue pixels on the standard hue scale:
The clip-level score aligns the frame mean to target valence , linearly mapped to :
In our implementation, and the warm-hue thresholds are , on the stated hue scale.
Kinetic intensity (kinetic)
A per-frame kinetic signal normalizes the mean absolute frame difference using saturation scale , and the clip-level score aligns the frame mean to the target arousal :
In our implementation, and the pixel range is . Appearance features describe potential cues; they are not part of the seven-provider memory update.
Motion naturalness (motion)
Dense optical flow on grayscale, short-side- frames yields per-frame magnitude and a global (dominant) motion vector . Four bounded sub-scores reward smooth flow, low camera jerk, low warp residual, and low compensated flicker:
where is flow-warping and a global-shift compensation. With base and activity , the arousal-conditioned score and confidence are
reverting to when no target arousal is supplied. The motion gate scales linearly with this observation confidence (Appendix § 11). The mixture coefficients obey . In our implementation, , , and . We use , , , , and .
Score normalization and validity
The evaluated fusion consumes the providers’ bounded scores directly. An optional running-statistic normalization is not an additional transformation applied to those scores. Availability and confidence are retained separately from score magnitude: unavailable evidence receives a neutral fallback but zero effective contribution. Motion confidence is incorporated once through its channel gate, while an activity mask handles unavailable motion observations. This convention avoids counting the same confidence twice. The micro, body, and temporal channels are constant in the analyzed reference traces; their neutral readings do not establish the absence of visible faces or bodies in the generated videos.
Reflect: Instance-Dependent Gate Construction
Reflect separates training progress, semantic consistency, observation availability, and the prompt’s intended intensity. A factor is neutral for providers to which it does not apply. Let denote the video generation model update step, the phase-transition endpoints, a phase floor, and the available face-evidence ratio:
The semantic-conflict factor attenuates disagreement above a threshold , with lower bound . Subtlety is detected from prompt cues or ; it selects restraint factors , , and for facial/body, temporal, and semantic channels, respectively. Otherwise these restraint factors equal one. Before applying observation-availability masks, the gates are
Here is the optical-flow observation confidence. Availability masking and continuous confidence together instantiate the validity term in Eq. (7); confidence is not multiplied a second time.
Implementation settings. In our implementation, the default warmup endpoints are and , with . Face gating uses ; semantic conflict uses and . For subtle prompts, detected using or prompt cues, ; the motion floor is . These are rule-based controls rather than learned reward-model parameters. Historical exported traces retain their recorded gates; the default specification alone does not reconstruct a past run’s full configuration.
Effect on normalized fusion. A factor shared by every active provider cannot by itself redistribute trust. For with , active normalization yields
Except for the stabilizer, the common scale cancels. Reflect changes the relative allocation through provider-specific factors, invalidation, and composition with the original reward branch. The face-activity proxy used to group exported traces in Appendix § 12 is distinct from the current video’s face-evidence ratio.
Evolve: RewardMemory Construction and Use
Record the reference run (Figure 8A). A completed run with Reflective Reward Gating supplies the experience used by RewardMemory. Each trace keeps the prompt–video context, raw provider scores, gates, applied weights, and VideoAlign outcome separate. The context describes target affect and available evidence; its face-activity proxy records whether selected scoring branches depart from their neutral defaults, rather than measuring visible-face coverage. The recorded scores and outcomes provide the basis for comparing providers.
Figure 8. Figure 8 From completed-run evidence to calibrated fusion. Recorded observations (A) supply provider scores and outcomes; RewardMemory (B) supplies fixed multipliers estimated from the complete reference bank, rather than the four displayed clips. A controlled replay (C) applies these multipliers to illustrative initial weights for rollout #2. The resulting 0.578→0.563 calculation demonstrates reallocation with fixed scores; it is distinct from the logged reward 0.721 and does not represent a second training run.
Build the calibration table (Figure 8B). The complete reference bank is grouped by target arousal and face-branch activity, with a global fallback for contexts with insufficient support. Within each group, reliability is the correlation between a provider’s raw scores and VideoAlign outcomes, before gating. Algorithm 2 compares each provider with the group’s average reliability and converts the difference into a capped weight multiplier (Eq. (10)). This calibrates reward allocation without fitting a value network or changing any reward model.
Use the memory in the next run (Figure 8C). Algorithm 1 looks up the matching multipliers, combines them with the base weights and current-video gates, and normalizes over active providers. Memory sets the allocation from prior experience; reflection can still suppress unavailable or unsuitable cues. The table remains fixed throughout training of the video generation model. Panel C illustrates this multiply-and-normalize step using recorded scores and illustrative starting weights: it is a controlled replay, not an additional training run. Reflect-only fusion omits the memory adjustment; Appendix § 16 explains the worked example.
Derivation of the Closed-Form KL Update
For positive base weights on the supported simplex, consider
The Lagrangian has stationary condition . Normalizing the resulting exponentials gives . Strict concavity yields a unique optimum. A shared additive baseline in the utilities cancels from this unclipped solution.
For , the same derivation applies to clipped utilities . Since , normalized weights satisfy
The cap bounds relative changes; it does not guarantee a one-hot distribution as . Zero base weights remain outside the support.
Scope of the bound and cold start. The interval bounds the exported, unnormalized multipliers; the normalized weight ratio has the wider bound above. Clipping the utilities is not equivalent to solving the original objective subject to a prescribed hard KL constraint. If all advantages are zero, the update returns . A constant provider has , but its centered advantage is negative only when the mean reliability is positive; constant scores therefore do not imply unconditional down-weighting. In the supplied traces, positive-reliability providers make this relative attenuation observable. None of these algebraic properties proves improved generalization or emotional fidelity.
Group-relative invariance. If with , then , , and
Translation cancels exactly. By contrast, is not translation invariant. Consequently, a reward offset can increase relative dispersion without improving the standardized group advantage. This is an algebraic property, not an experimental claim about learning efficiency.
RL Objective for Video Generation. For completeness, we state the clipped GRPO objective underlying the group advantages in Eq. (1) [54]. Let denote the generation state and its sampled transition in candidate trajectory . Define . With clipping radius and reference-policy regularization coefficient ,
Here is the number of sampled transitions, is the rollout policy, and is a fixed reference policy. The scalar reward of the completed video determines ; advantages and reward-provider outputs are treated as constants during the policy update. This expression states the common clipped objective; individual video RL algorithms specify their transition likelihoods, sampled timesteps, and regularization. EmoReward changes the scalar reward supplied to the RL algorithm and preserves those algorithm-specific choices. The policy KL penalty acts on the video generation model, while the reward-allocation KL term acts on the provider weights.
Reward-Gating Metrics and Reproducibility
Table 1 reports two reward diagnostics (SNR* and Sel.), three normalized pillars (EF, GD, GR), and two composites (core, emo). All seven policies use the same backbone and optimization budget. The exported evaluation contains 160 four-candidate groups for VideoAlign, single-provider, and random routing, and 152 groups for each remaining policy. Thus the available sample counts differ, even though the nominal training budget is shared. The references identify scoring models or routing mechanisms, not reproductions of complete agentic systems.
Reward Diagnostics
Let denote the fused training reward, the VideoAlign reference outcome, and provider ’s score. A group contains candidates for one prompt.
Reward–outcome agreement (ROA). ROA is an auxiliary input to GR rather than a displayed column in Table 1. It measures within-group pairwise ranking agreement. For outcome-nontied pairs , the intended statistic is
The reported agreement is ROA = . The supplied pairwise helper uses and returns when no outcome-nontied pairs exist. However, its treatment of reward ties differs from the symmetric half-credit convention above. The original aggregation script is unavailable; we therefore retain the archived ROA values as inputs to GR, without claiming to have recomputed their tie handling. VideoAlign-only obtains one by construction when its reward and reference ordering coincide. ROA measures agreement with that evaluator, not independent human or emotional quality.
Within-group relative dispersion (SNR).* With population standard deviation within each group,
The supplied diagnostic implementation uses . This is a coefficient of variation, not a signal-to-noise power ratio. A reward offset can increase it while leaving centered GRPO advantages unchanged; a high value neither establishes nor is necessary for better policy gradients. We report the archived per-policy reductions, separately from the aggregate-level checks below.
Reliability selectivity (Sel.). Let over the policy’s evaluated candidates, using the nine-channel diagnostic pool . For ,
The pool includes chromatic and kinetic readouts as well as the seven fusion providers. Sel. describes concentration of absolute provider–outcome correlations, not applied fusion weights. It discards correlation signs and does not establish causal provider usefulness. All seven reported policies have ; the auxiliary zero-sample row and its all-zero-profile convention are excluded. Recomputing Sel. from the exported correlation profiles agrees with the stored values within .
Normalized Pillars and Composite Scores
For the fixed comparison set of seven policies, define
The supplied reducer returns zero for a constant input across policies; none of the three reported inputs is constant. Normalization is specific to the compared set, so the resulting scores are not absolute measures transferable across benchmarks.
EF: preference-proxy pillar. , where is the archived mean ImageReward score [77]. EF retains the benchmark’s “Emotion Fidelity” abbreviation, but its input is a general visual preference model. We do not interpret it as an independent measurement of emotion fidelity. The raw means and normalization inputs are included in the supplement.
GD: relative-dispersion pillar. . The benchmark calls this Gating Discriminativeness; mathematically, it is the normalized coefficient of variation above. Its inclusion does not turn reward dispersion into a gradient-quality measurement.
GR: agreement-weighted selectivity pillar. . GR combines correlation-profile concentration with agreement with the reference evaluator. Both inputs are diagnostic summaries; the product does not establish an independent reliability ground truth.
core and emo: two specified aggregations. The equal-weight composite and preference-emphasized composite are
Here the nonnegative coefficients sum to one; our implementation uses . We report the predefined composites as descriptive summaries. Min–max normalization fixes the endpoints for this set of policies, so small raw differences can lead to large changes in a normalized pillar.
Emotion and visual metrics
Facial reward during training. The facial reward provider uses CLIP-AUTT [86] with a CLIP ViT-B/32 backbone and an action-unit (AU) text adapter and temporal classifier trained on BioVid. It produces AU similarity scores and probabilities for the BioVid pain and neutral classes.
Emotion metrics at evaluation. DEA and TERS use the standard frozen CLIP-ViT-H-14-laion2B-s32B-b79K checkpoint, trained on the LAION-2B subset of LAION-5B [53]. This encoder is held out from reward computation, RewardMemory, and checkpoint selection. It encodes 24 uniformly sampled frames per video and one text template for each of the eight Plutchik emotion rings. Let be the frame- embedding and the text embedding for ring . The frame-averaged image–text cosine similarity for each of the eight rings is
Let be the prompt’s set of target rings. DEA checks whether the largest in exceeds the largest score among the corresponding opposite rings after excluding opposites that are themselves targets. Of the 302 prompts, 261 have a usable opposite. TERS uses all eight ring scores. We sort them in descending order with ranks starting at one, using the fixed ring order (joy, trust, fear, surprise, sadness, disgust, anger, anticipation) for ties. The score for one video is
A score of one means a target ring ranks first. We compute TERS on all 302 prompts and report its mean and SEM across prompts. Table 6 compares the same configurations under LAION-2B ViT-H/14 and OpenAI ViT-L/14. For DEA, we report discordant algorithm–prompt pairs descriptively. The two-sided paired permutation test flips the base and guided labels jointly for all three RL algorithms within each prompt, accounting for prompt reuse.
| CLIP ViT-H/14 | CLIP ViT-L/14 | |||||||
| RL algorithm | DEA | TERS | DEA | TERS | ||||
| Base | +EmoReward | Base | +EmoReward | Base | +EmoReward | Base | +EmoReward | |
| DanceGRPO | 0.605 | 0.632 | 0.764 | 0.782 | 0.625 | 0.632 | 0.763 | 0.770 |
| FlowGRPO | 0.621 | 0.670 | 0.786 | 0.802 | 0.625 | 0.659 | 0.782 | 0.802 |
| SAGE-GRPO | 0.617 | 0.674 | 0.788 | 0.819 | 0.613 | 0.670 | 0.774 | 0.808 |
| Discordant wins | 90/145 (62.1%) | – | 106/186 (57.0%) | – |
Table 6. Emotion metrics across frozen encoders. DEA uses 261 prompts and TERS uses all 302. The last two rows summarize discordant algorithm–prompt pairs and prompt-blocked permutation tests for DEA.
Visual metrics. For PickScore and CLIPScore, we divide the frame-level logits by 26 and 30, respectively, then average the scaled scores across frames.
Table 6 Emotion metrics across frozen encoders. DEA uses 261 prompts and TERS uses all 302. The last two rows summarize discordant algorithm–prompt pairs and prompt-blocked permutation tests for DEA.
Extended Evaluation and Ablations
Provider-pool construction. Figure 6 presents the main-text visual comparison; Table 7 quantifies the same four reward configurations used to construct the provider interface. Semantic emotion and motion cues improve some axes while exposing trade-offs on text alignment and preference. The configurations also differ in fusion weights and curricula; their comparison is not a single-factor ablation. In the semantic-cue configuration, the CLIP channel has zero fusion weight and cannot explain a training gain attributed to an active CLIP reward.
| Configuration | Overall | VQ | MQ | TA | ImageReward |
| VideoAlign | -1.614 | -0.505 | -0.313 | -0.797 | -0.866 |
| +Semantic emotion | -1.363 | -0.014 | -0.991 | -0.922 | |
| +Motion naturalness | -1.453 | -0.505 | 0.075 | -1.024 | -1.293 |
| Balanced tool pool | -1.178 | -0.297 | 0.074 | -0.954 | -1.113 |
Table 7. Provider-pool configurations. Higher is better. TA denotes text alignment. The reward configurations use different weights and curricula; missing VQ is not imputed.
Table 7 Provider-pool configurations. Higher is better. TA denotes text alignment. The reward configurations use different weights and curricula; missing VQ is not imputed.
Ranking agreement. For each four-candidate group, let . We use
Groups without comparable pairs are excluded. Agreement is averaged over groups; reward ties receive half credit independently of sample ordering. The analysis contains 160 baseline groups for DanceGRPO and 168 groups for each other algorithm–reward configuration, totaling 1,504 groups and 6,016 videos. These training candidates are separate from the 30-prompt fixed-checkpoint evaluation sets.
Memory sensitivity. Across 27 temperature, neighborhood-size, and memory-window settings, the memory rule exceeds fixed weighting at the final window in 23 settings. None has a positive difference when averaged over all windows. These settings reuse the same traces and are not independent training seeds. The results motivate sensitivity analysis rather than a claim of monotonic improvement with accumulating memory.
RewardMemory: A Worked Rollout Example
Figure 8 connects recorded provider scores, a previously exported calibration table, and a reproducible allocation calculation. Appendix F explains the principles behind panels A–C. The clips contain 81 frames at 24 fps and have 624 × 352 resolution; we display frames 12, 40, and 72. The four videos are candidates for one prompt from a reference rollout group. Their scores and outcomes come directly from the accompanying reward records; the images are decoded full frames. The macro provider has positive within-group outcome correlation, while three channels return their neutral constant. These four examples illustrate local behavior, not the population estimate used to construct RewardMemory.
Separating observations from the replay. The exported table’s global entry differs from its context-specific entries. We use the global entry here to isolate the reweighting operation, without claiming that was the entry selected for these historical videos. The initial active allocation is over macro, motion, CLIP-AUTT, and R1-Omni, with zero mass on the other three providers. These are illustrative starting weights, not reconstructed runtime gates. Multiplying by the exported global multipliers and renormalizing produces approximately . On rollout #2’s recorded scores, the weighted mean changes from 0.578 to 0.563; its actual logged emotion reward is 0.721. The distinction matters because the historical reward used its own training phase and configuration. An increase in a provider’s allocation need not increase the scalar fused score, and neither change alone establishes improved generated quality.
Prompt. The camera slowly pushes forward, a cold side light slanting through the window frame, carving a sharp beam across the dimly lit study and illuminating the man seated at his desk. He leans slightly forward, hands folded over the keyboard, fingers tapping the keys with slow, precise rhythm. Shot from a low angle behind and to the side, the frame captures his furrowed brow and narrowed eyes, his gaze fixed like a needle on the flickering stream of code on the screen. His lips are pressed into a thin line, the jawline taut with concentration, nostrils fluttering faintly with each breath, occasionally biting his lower lip as if swallowing an unspoken question. The light casts a soft shadow across his forehead, making the sharp glint in his eyes even more intense. In the background, scattered papers and empty coffee cups rest silently on the bookshelf, while the rhythmic tapping of the keyboard echoes in the stillness, like the pulse of a heartbeat. He suddenly pauses, raises his right hand to his temple, his eyes briefly drifting, then swiftly refocuses on the screen—a quiet resolve, a steadfast defiance against time.
Generalization to Image Generation
We evaluate Stable Diffusion 3.5 and FLUX with three RL algorithms and three reward configurations on 256 DrawBench prompts per checkpoint: 18 settings and 84 checkpoints. Internal Emo measures the emotion-stack reward; HPSv3 is a general preference objective also used to train the baseline. Neither independently measures perceived emotion.
SD3.5 baseline results average three checkpoints at 420–500 updates; calibrated variants average five at 2,780–3,000. FLUX baseline checkpoints span 2,760–3,000, and calibrated checkpoints 2,780–3,000. Standard errors describe checkpoint variation, not independent training seeds. Per-prompt arrays support the reported means; paired significance requires complete sample and generation-seed mappings and is not claimed.
Observed transfer pattern. With SD3.5–DanceGRPO, HPSv3 rises from 9.552 (baseline) to 10.096 (Reflect) and 10.302 (EmoReward). SD3.5–FlowGRPO declines on both HPSv3 and internal Emo; FLUX’s HPSv3-optimized baseline leads on HPSv3 across RL algorithms. Thus transfer is objective-dependent rather than uniformly beneficial.
Qualitative comparison protocol
Image plates use four curated prompts per displayed setting; video plates cover three RL algorithms. Figure 6 separately compares provider configurations. These examples illustrate behavior, not a randomly estimated win rate.
Images retain all original pixels. SD3.5 baseline outputs use 500 updates and calibrated outputs 3,000. The first two FLUX baseline cases use the only saved examples at 500 updates; all other FLUX rows use 3,000. Unequal budgets prevent attributing qualitative differences solely to calibration. Image seeds are unavailable, so only prompts are matched. The video rows in Appendix § 18 share prompts and recorded sampling seeds. Clips are 864×480 at 24 fps; full frames at 0.5, 2.5, and 4.5 seconds are decoded without interpolation or retouching.
The ordinal ratings in Appendix § 9 complement these proxy-based and qualitative comparisons; raw participant/prompt records are needed to quantify uncertainty. The depicted scenes are generated content.
Figure 9. (a) A stack of 3 plates. A blue plate is on the top, sitting on a blue plate. The blue plate is in the middle, sitting on a green plate. The green plate is on the bottom.
(b) A large keyboard musical instrument with a wooden case enclosing a soundboard and metal strings, which are struck by hammers when the keys are depressed. The strings' vibration is stopped by dampers when the keys are released and can be regulated for length and volume by two or three pedals. [Spelling normalized.]
(c) Artophagous.
(d) A stack of 3 books. A green book is on the top, sitting on a red book. The red book is in the middle, sitting on a blue book. The blue book is on the bottom.
Qualitative comparison of reward calibration. Columns show the baseline, the addition of Reflect, and the complete EmoReward configuration for SD3.5 with DanceGRPO. Prompt (b) is spelling-normalized for readability; the images retain the original perturbed input. Budgets and seed limitations appear in Appendix § 17.§ 1.
Figure 10. Image transfer across SD3.5 and FLUX. Five backbone–algorithm settings share four prompts; each panel compares HPSv3 RL, Reflective gating, and EmoReward. Rows show the same curated prompts as the preceding figure. The first two FLUX baseline rows use earlier saved outputs; see Appendix § 17.§ 1 for budgets and seed limitations.
Video Comparisons
Figure 11. Prompt: The camera slowly pushes in on a young woman standing at a street corner under the dim yellow glow of a streetlight, captured in a medium close-up. Side backlighting outlines her slender silhouette. Her hand unconsciously tightens around the backpack strap, knuckles turning slightly white, while a faint, forced smile tugs at her lips—never reaching her eyes. Her gaze drifts away from the front, settling on a puddle on the ground, where neon signs flicker and reflect like the instability of her emotions. She slightly turns her head, her lips twitching as if about to speak, but only a soft cough escapes. She lowers her head, her long hair falling to veil half her face. A breeze lifts her strands, and she instinctively reaches up to brush them aside, her fingers pausing in midair before dropping weakly. In the background, blurred traffic and pedestrians move past, their noise isolated beyond the frame, leaving her alone in the shifting light, trembling slightly, as if the entire world has gone silent except for the clear, pounding rhythm of her heartbeat in her ears. Figure 11 DanceGRPO. Street-corner hesitation: a strained smile and averted gaze.
Figure 12. Prompt: The slanting sunlight of dusk filters through the old window lattice, bathing the room in a warm, amber glow. An middle-aged couple sits on worn wicker chairs; she is bent over, mending an old sweater, her fingers deftly guiding the needle and thread. He watches her quietly, his fingers unconsciously tracing the rim of his cup, his gaze reflecting the swaying tree shadows outside. Suddenly, she looks up at him, a faint smile lifting the corners of her lips, fine lines crinkling at the edges of her eyes—gentle ripples of time’s quiet grace. His throat moves slightly, his lips parting as if to speak, but no sound comes. Instead, he reaches out, gently brushing a strand of her hair, his touch slow and reverent. A hint of shyness flickers in her eyes; she lowers her head with a soft laugh, her fingers trembling slightly as a thread slips free. He catches it with a quick, tender motion. They exchange a glance, and a silent understanding flows between them. The camera slowly pushes in, focusing on the delicate smile lines at the corner of her eye and the quiet glimmer deep in his gaze. Figure 12 DanceGRPO. A quiet exchange at dusk: the prompt couples a smile with a tentative hand gesture.
Figure 13. Prompt: The camera slowly pushes in on a young woman kneeling on ancient bluish stone steps, her back turned to the deepening twilight sky, hands folded gently on her knees, fingertips trembling faintly. Her face is bathed in warm gold by the slanting rays of the setting sun, her eyelashes lowered, her pupils reflecting the flickering oil lamp at the distant temple eaves—a silent mirror to some unspoken obsession. Her lips are slightly pursed, the line of her mouth taut, as if silently praying or suppressing a sigh that threatens to break free. She tilts her head upward, gazing into the night sky, a single tear slipping down her cheek, pausing at her chin, glimmering in the dim light. A breeze stirs her hair, her sleeve lifts slightly, her arm quivers faintly, as though awaiting a response that will never come. Behind her, a quiet courtyard lies still, wild grasses growing beside the stone steps, while in the distance, a bell tolls gently, echoing through the empty dusk. The entire scene is enveloped in a near-sacred sense of solitude. Figure 13 FlowGRPO. Unease on the stone steps at twilight.
Figure 14. Prompt: The camera slowly pushes in on a young woman sitting alone in the corner of an old-fashioned café, rain streaking diagonally across the window, blurring the street scene through water-traced glass. She holds a cup of coffee now cold, her fingertips trembling slightly, her gaze distant as she stares at the table, lips pressed into a thin line, as if chewing on some unspeakable contradiction. Light slants through the high window, bathing one side of her face in warm yellow while the other remains in shadow, creating a stark visual divide. Suddenly, she lifts her eyes, scanning the empty seats before returning her gaze to her phone screen, her brow furrowing slightly, her mouth twitching as if she’s heard a whisper she shouldn’t have. She wipes her eye with the back of her hand—restrained yet unable to hide her weariness—her fingers unconsciously picking at the rim of the cup, as if searching for some confirmation that has long since vanished. The air is thick with the scent of coffee and damp wood; time seems to have stopped, except for the flicker of doubt in her brow, quietly growing in the silence. Figure 14 FlowGRPO. A pause in a rain-streaked café.
Figure 15. Prompt: The camera slowly pushes in, revealing a middle-aged woman standing at the center of the stage, bathed in a spotlight that cascades from the upper side, softly outlining her silhouette with quiet strength. She tilts her head slightly upward, a subtle smile forming on her lips—a smile long held within, unassuming yet powerful enough to pierce the silence. Her gaze sweeps across the audience, her eyes glowing with a gentle warmth, like melting snow in winter, reflecting countless looks of reverence. Her fingers gently trace the emblem on her chest, her fingertips trembling slightly, as if touching a memory sealed by time. Slowly, she raises her hand and grasps the microphone, her lips parting slightly as her voice emerges—low, clear, each word like amber seeping from the depths of her soul. Her eyes never leave the audience, a look of calm acceptance and quiet response, as if saying: every step I’ve taken is worth being seen. Behind her, the applause blurs into soft waves, the melody gentle and hushed, while her face, stretched by light and shadow, resembles a still oil painting, frozen in her moment of radiant glory. Figure 15 SAGE-GRPO. Composure under the spotlight in a restrained, emotionally charged stage scene.
Figure 16. Prompt: The camera slowly pushes in, settling on a close-up of a middle-aged woman seated in an old wicker chair by the window, sunlight slanting across her slightly weary face, fine lines around her eyes trembling faintly in the light. She gazes down at an old letter in her hands, lips slightly parted, breath catching, as if reading a memory sealed by time. Suddenly, a breeze stirs, gently turning the page. She lifts her eyes, her gaze falling on a cherry blossom tree blooming outside, a flicker of recognition passing through her, a soft smile gradually forming, her brow relaxing into a long-forgotten tenderness. She raises a hand, fingertips brushing lightly over the letter, as if touching a distant moment; tears well in her eyes but do not fall, only a faint glimmer shining deep within her pupils. The scene slowly pulls back, revealing her sitting in a room bathed in shifting light and shadow, the flower shadows outside swaying gently, the entire space filled with a quiet, warm aura of bygone days. Figure 16 SAGE-GRPO. Recognition in an old letter in a close-up with gradually softening expression.
References
- [1]Shuai Bai, Yuxuan Cai, Ruizhe Chen, et al. Qwen3-VL technical report. arXiv preprint arXiv:2511.21631, 2025.arxiv.org/abs/2511.21631
- [2]Tadas Baltrušaitis, Amir Zadeh, Yao Chong Lim, and Louis-Philippe Morency. OpenFace 2.0: Facial Behavior Analysis Toolkit. In IEEE International Conference on Automatic Face and Gesture Recognition, pages 59–66, 2018.DOI
- [3]Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. In International Conference on Learning Representations, volume 2024, pages 4965–4987, 2024.arxiv.org/abs/2305.13301
- [4]Haodong Chen, Haojian Huang, Junhao Dong, Mingzhe Zheng, and Dian Shao. Fineclip: Multi-modal fine-grained clip for dynamic facial expression recognition with adapters. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 2301–2310, 2024.
- [5]Shuang Chen, Kaituo Feng, Hangting Chen, Wenxuan Huang, Dasen Dai, Quanxin Shou, Yunlong Lin, Xiangyu Yue, Shenghua Gao, and Tianyu Pang. OpenSearch-VL: An Open Recipe for Frontier Multimodal Search Agents. arXiv preprint arXiv:2605.05185, 2026.arxiv.org/abs/2605.05185
- [6]Sixiang Chen, Zhaohu Xing, Tian Ye, Xinyu Geng, Yunlong Lin, Jianyu Lai, Xuanhua He, Fuxiang Zhai, Jialin Gao, and Lei Zhu. GenEvolve: Self-Evolving Image Generation Agents via Tool-Orchestrated Visual Experience Distillation. arXiv preprint arXiv:2605.21605, 2026.
- [7]Wenshuo Chen, Kuimou Yu, Bowen Tian, Jianfei Song, Shaofeng Liang, Haozhe Jia, Kan Cheng, Haosen Li, Kaishen Yuan, Lei Wang, Jiemin Wu, Songning Lai, and Yutao Yue. MemoGen: Can Past Experience Improve Future Text-to-Image Generation? arXiv preprint arXiv:2606.03243, 2026.
- [8]Zebang Cheng, Zhi-Qi Cheng, Jun-Yan He, Jingdong Sun, Kai Wang, Yuxiang Lin, Zheng Lian, Xiaojiang Peng, and Alexander G Hauptmann. Emotion-llama: Multimodal emotion recognition and reasoning with instruction tuning. Advances in Neural Information Processing Systems, 37:110805–110853, 2024.
- [9]Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
- [10]Alan S. Cowen and Dacher Keltner. Self-report captures 27 distinct categories of emotion bridged by continuous gradients. Proceedings of the National Academy of Sciences, 114(38):E7900–E7909, 2017.
- [11]Alan S. Cowen and Dacher Keltner. What the face displays: Mapping 28 emotions conveyed by naturalistic expression. American Psychologist, 75(3):349–364, 2020.
- [12]Fangming Cui, Ruixiao Zhu, Cheng Fang, Sunan Li, and Jiahong Li. Rethinking agentic reinforcement learning in large language models. arXiv preprint arXiv:2604.27859, 2026.
- [13]Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. GoEmotions: A dataset of fine-grained emotions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4040–4054, 2020.
- [14]Paul Ekman and Wallace V. Friesen. Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto, CA, 1978.
- [15]Liaoyuan Fan, Yuhao Zheng, Dongxing Mao, Hangyu Ran, Siyuan Hu, Weijia Mao, Rui Zhao, Weijia Wu, Guian Fang, Yanzhe Chen, Zeyu Zhu, Yuxin Jiang, Yiren Song, Yuchao Gu, Junchao Yi, Qi Hu, Dan Kong, Alex Jinpeng Wang, Kevin Qinghong Lin, Mike Zheng Shou, and Wei Liu. Video Generation Meets Agent Harness: Agentic Video Generation, Agentic World Modeling and Recursive Self-Improvement. arXiv preprint arXiv:2610.40191, 2026.
- [16]Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models. Advances in neural information processing systems, 36:79858–79885, 2023.arxiv.org/abs/2305.16381
- [17]Zhen Fang, Wenxuan Huang, Yu Zeng, Yiming Zhao, Shuang Chen, Kaituо Feng, Yunlong Lin, Lin Chen, Zehui Chen, Shaosheng Cao, et al. Flow-OPD: On-policy distillation for flow matching models. arXiv preprint arXiv:2605.08063, 2026.arxiv.org/abs/2605.08063
- [18]Kaituo Feng, Manyuan Zhang, Shuang Chen, Yunlong Lin, Kaixuan Fan, Yilei Jiang, Hongyu Li, Dian Zheng, Chenyang Wang, and Xiangyu Yue. Gen-searcher: Reinforcing agentic search for image generation. arXiv preprint arXiv:2603.28767, 2026.arxiv.org/abs/2603.28767
- [19]Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Yunzhong He, Bing Liu, and Sean Hendryx. Rubrics as rewards: Reinforcement learning beyond verifiable domains. In International Conference on Learning Representations, volume 2026, pages 127924–127945, 2026.arxiv.org/abs/2507.17746
- [20]Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.DOI
- [21]Xuan He, Dongfu Jiang, Ge Zhang, Max Ku, Achint Soni, Sherman Siu, Haonan Chen, Abhranil Chandra, Ziyan Jiang, Aaron Arulraj, et al. Videoscore: Building automatic metrics to simulate fine-grained human feedback for video generation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2105–2123, 2024.arxiv.org/abs/2406.15252
- [22]Haobo Hu, Qi Mao, Yuanhang Li, and Libiao Jin. Camera Artist: A Multi-Agent Framework for Cinematic Language Storytelling Video Generation. arXiv preprint arXiv:2604.09195, 2026.arxiv.org/abs/2604.09195
- [23]Yinming Huang, Shuyuan Tu, Xi Yan, Zihan Yang, Jianhua Han, Hang Xu, Kaihang Pan, Yu-Gang Jiang, and Zuxuan Wu. Vaj-judger: Reward modeling from human preference feedback for joint video-audio generation. arXiv preprint arXiv:2608.18607, 2026.arxiv.org/abs/2608.18607
- [24]Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21807–21818. IEEE, 2024.arxiv.org/abs/2311.17982
- [25]Dengyang Jiang, Xin Jin, Dongyang Liu, Zanyi Wang, Mingzhe Zheng, Ruoyi Du, Xiangpeng Yang, Qilong Wu, Zhen Li, Peng Gao, et al. D-OPSD: On-Policy Self-Distillation for Continuously Tuning Step-Distilled Diffusion Models. arXiv preprint arXiv:2605.05204, 2026.arxiv.org/abs/2605.05204
- [26]Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbubland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. Advances in neural information processing systems, 36:36652–36663, 2023.arxiv.org/abs/2305.01569
- [27]Jinxiang Lai, Zexin Lu, Jiajun He, Rongwei Quan, Wenzhe Zhao, Qinyu Yang, Qi Chen, Qin Lin, Chuyue Li, Tao Gao, Yuhao Shan, Shuai Shao, Song Guo, and Qinglin Lu. VisionCreator: A Native Visual-Generation Agentic Model with Understanding, Thinking, Planning and Creation. arXiv preprint arXiv:2603.02681, 2026.arxiv.org/abs/2603.02681
- [28]Jiaqi Leng, Shuyuan Tu, Haidong Cao, Sicheng Xie, Daoguo Dong, Zuxuan Wu, and Yu-Gang Jiang. Preference score distillation: Leveraging 2d rewards to align text-to-3d generation with human preference. arXiv preprint arXiv:2603.01594, 2026.arxiv.org/abs/2603.01594
- [29]Yibo Li, Zijie Lin, Ailin Deng, Xuan Zhang, Yufei He, Shuo Ji, Tri Cao, and Bryan Hooi. Just-In-Time Reinforcement Learning: Continual Learning in LLM Agents Without Gradient Updates. arXiv preprint arXiv:2601.18510, 2026.arxiv.org/abs/2601.18510
- [30]Zheng Lian, Haoyu Chen, Lan Chen, Haiyang Sun, Licai Sun, Yong Ren, Zebang Cheng, Bin Liu, Rui Liu, Xiaojiang Peng, et al. Affectgpt: A new dataset, model, and benchmark for emotion understanding with multimodal large language models. arXiv preprint arXiv:2501.16566, 2025.arxiv.org/abs/2501.16566
- [31]Yunlong Lin, Zixu Lin, Kunjie Lin, Jinbin Bai, Panwang Pan, Chenxin Li, Haoyu Chen, Zhongdao Wang, Xinghao Ding, Wenbo Li, et al. JarvisArt: Liberating Human Artistic Creativity via an Intelligent Photo Retouching Agent. arXiv preprint arXiv:2506.17612, 2025.arxiv.org/abs/2506.17612
- [32]Yunlong Lin, Linqing Wang, Kunjie Lin, Zixu Lin, Kaixiang Gong, Wenbo Li, Bin Lin, Zhenxi Li, Shiyi Zhang, Yuyang Peng, et al. JarvisEvo: Towards a Self-Evolving Photo Editing Agent with Synergistic Editor-Evaluator Optimization. arXiv preprint arXiv:2511.23002, 2025.arxiv.org/abs/2511.23002
- [33]Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl. Advances in neural information processing systems, 38:40783–40818, 2026.arxiv.org/abs/2505.05470
- [34]Jie Liu, Gongye Liu, Jiajun Liang, Ziyang Yuan, Xiaokun Liu, Mingwu Zheng, Xiele Wu, Qiulin Wang, Menghan Xia, Xintao Wang, et al. Improving video generation with human feedback. Advances in Neural Information Processing Systems, 38:82155–82192, 2026.arxiv.org/abs/2501.13918
- [35]Runtao Liu, Haoyu Wu, Ziqiang Zheng, Chen Wei, Yingqing He, Renjie Pi, and Qifeng Chen. VideoDPO: Omni-Preference Alignment for Video Diffusion Generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.arxiv.org/abs/2412.14167
- [36]Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang, Haoxin Chen, Yang Liu, Tieyong Zeng, Raymond Chan, and Ying Shan. Evalcrafter: Benchmarking and evaluating large video generation models. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22139–22149. IEEE, 2024.arxiv.org/abs/2310.11440
- [37]Zeqian Long, Mingzhe Zheng, Kunyu Feng, Xinhua Zhang, Hongyu Liu, Harry Yang, Linfeng Zhang, Qifeng Chen, and Yue Ma. Follow-your-shape: Shape-aware image editing via trajectory-guided region control. International Conference on Learning Representations, 2026.arxiv.org/abs/2508.08134
- [38]Xuran Ma, Yexin Liu, Yaofu Liu, Xianfeng Wu, Mingzhe Zheng, Zihao Wang, Ser-Nam Lim, and Harry Yang. Model reveals what to cache: Profiling-based feature reuse for video diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17150–17159, 2025.
- [39]Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-Level Reward Design via Coding Large Language Models. In International Conference on Learning Representations, 2024.arxiv.org/abs/2310.12931
- [40]Yue Ma, Xinyu Wang, Qianli Ma, Qinghe Wang, Mingzhe Zheng, Xiangpeng Yang, Hao Li, Chongbo Zhao, Jixuan Ying, Harry Yang, et al. Group editing: Edit multiple images in one go. Proceedings of the IEEE/CVF Conference on Computer Vision & Pattern Recognition, 2026.arxiv.org/abs/2603.22883
- [41]Yue Ma, Zhikai Wang, Tianhao Ren, Mingzhe Zheng, Hongyu Liu, Jiayi Guo, Kunyu Feng, Yuxuan Xue, Zixiang Zhao, Konrad Schindler, et al. Fastvmt: Eliminating redundancy in video motion transfer. International Conference on Learning Representations, 2026.arxiv.org/abs/2602.05551
- [42]Yuhang Ma, Xiaoshi Wu, Keqiang Sun, and Hongsheng Li. Hpsv3: Towards wide-spectrum human preference score. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pages 15086–15095. IEEE, 2025.arxiv.org/abs/2508.03789
- [43]Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in neural information processing systems, 36:46534–46594, 2023.arxiv.org/abs/2303.17651
- [44]Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M. Waleed Kadous, and Ion Stoica. RouteLLM: Learning to Route LLMs with Preference Data. arXiv preprint arXiv:2406.18665, 2024.arxiv.org/abs/2406.18665
- [45]Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training Language Models to Follow Instructions with Human Feedback. In Advances in Neural Information Processing Systems, 2022.arxiv.org/abs/2203.02155
- [46]Yatian Pang, Bin Zhu, Bin Lin, Mingzhe Zheng, Francis EH Tay, Ser-Nam Lim, Harry Yang, and Li Yuan. Dreamdance: Animating human images by enriching 3d geometry cues from 2d poses. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14039–14050, 2025.arxiv.org/abs/2412.00397
- [47]W. Gerrod Parrott, editor. Emotions in Social Psychology: Essential Readings. Psychology Press, 2001.
- [48]Robert Plutchik. The nature of emotions. American Scientist, 89(4):344–350, 2001.
- [49]Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text-to-image diffusion models with reward backpropagation. 2023.arxiv.org/abs/2310.03739
- [50]Qwen Team. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.
- [51]Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728–53741, 2023.arxiv.org/abs/2305.18290
- [52]James A. Russell. A Circumplex Model of Affect. Journal of Personality and Social Psychology, 39(6):1161–1178, 1980.DOI
- [53]Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Arush Katto, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jena Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
- [54]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.arxiv.org/abs/2402.03300
- [55]Phillip R. Shaver, Judith Schwartz, Donald Kirson, and Cary O’Connor. Emotion knowledge: Further exploration of a prototype approach. Journal of Personality and Social Psychology, 52(6):1061–1086, 1987.pubmed.ncbi.nlm.nih.gov/3598857
- [56]Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in neural information processing systems, 36:8634–8652, 2023.arxiv.org/abs/2303.11366
- [57]Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008–3021, 2020.arxiv.org/abs/2009.01325
- [58]Kaiyue Sun, Kaiyi Huang, Xian Liu, Yue Wu, Zihan Xu, Zhenguo Li, and Xihui Liu. T2V-compbench: A comprehensive benchmark for compositional text-to-video generation. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8406–8416. IEEE, 2025.arxiv.org/abs/2407.14505
- [59]Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599, 2025.arxiv.org/abs/2501.12599
- [60]Ritesh Thawkar, Shravan Venkatraman, Omkar Thawakar, Abdelrahman Shaker, Fahad Khan, Hisham Cholakkal, Salman Khan, and Rao Muhammad Anwer. Ask, Solve, Generate: Self-Evolving Unified Multimodal Understanding and Generation via Self-Consistency Rewards. arXiv preprint arXiv:2606.27376, 2026.arxiv.org/abs/2606.27376
- [61]Shuyuan Tu, Qi Dai, Zuxuan Wu, Zhi-Qi Cheng, Han Hu, and Yu-Gang Jiang. Implicit temporal modeling with learnable alignment for video recognition. In ICCV, 2023.arxiv.org/abs/2304.10465
- [62]Shuyuan Tu, Qi Dai, Zhi-Qi Cheng, Han Hu, Xintong Han, Zuxuan Wu, and Yu-Gang Jiang. Motioneditor: Editing video motion via content-aware diffusion. In CVPR, 2024.arxiv.org/abs/2311.18830
- [63]Shuyuan Tu, Qi Dai, Zihao Zhang, Sicheng Xie, Zhi-Qi Cheng, Chong Luo, Xintong Han, Zuxuan Wu, and Yu-Gang Jiang. Motionfollower: Editing video motion via lightweight score-guided diffusion. arXiv preprint arXiv:2405.20325, 2024.arxiv.org/abs/2405.20325
- [64]Shuyuan Tu, Yueming Pan, Yinming Huang, Xintong Han, Zhen Xing, Qi Dai, Chong Luo, Zuxuan Wu, and Yu-Gang Jiang. Stableavatar: Infinite-length audio-driven avatar video generation. arXiv preprint arXiv:2508.08248, 2025.arxiv.org/abs/2508.08248
- [65]Shuyuan Tu, Zhen Xing, Xintong Han, Zhi-Qi Cheng, Qi Dai, Chong Luo, and Zuxuan Wu. Stableanimator: High-quality identity-preserving human image animation. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21096–21106. IEEE, 2025.arxiv.org/abs/2411.17697
- [66]Shuyuan Tu, Zhen Xing, Xintong Han, Zhi-Qi Cheng, Qi Dai, Chong Luo, Zuxuan Wu, and Yu-Gang Jiang. Stableanimator++: Overcoming pose misalignment and face distortion for human image animation. arXiv preprint arXiv:2507.15064, 2025.arxiv.org/abs/2507.15064
- [67]Shuyuan Tu, Yueming Pan, Yinming Huang, Xintong Han, Zhen Xing, Qi Dai, Kai Qiu, Chong Luo, and Zuxuan Wu. Flashportrait: 6x faster infinite portrait animation with adaptive latent prediction. In CVPR, 2026.arxiv.org/abs/2512.16900
- [68]Shuyuan Tu, Qi Tian, Zihan Yang, Yue Wu, Xintong Han, Weijie Kong, Jiangfeng Xiong, Jian-Wei Zhang, Zhao Zhong, Liefeng Bo, et al. Baton: Explicit Semantic Blueprints for Joint Video-Audio Generation. arXiv preprint arXiv:2605.25195, 2026.arxiv.org/abs/2605.25195
- [69]Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion Model Alignment Using Direct Preference Optimization. arXiv preprint arXiv:2311.12908, 2023.arxiv.org/abs/2311.12908
- [70]Huanzhen Wang, Ziheng Zhou, Jiaqі Song, Li He, Yunshi Lan, Yan Wang, and Wenqiang Zhang. ARGen: Affect-Reinforced Generative Augmentation towards Vision-based Dynamic Emotion Perception, 2026.arxiv.org/abs/2604.12255
- [71]Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou. Mixture-of-Agents Enhances Large Language Model Capabilities. In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2406.04692
- [72]Yibin Wang, Yuhang Zhang, Feng Han, Jiazi Bu, Yujie Zhou, Cheng Jin, and Jiaqi Wang. Unified personalized reward model for vision generation. arXiv preprint arXiv:2602.02380, 2026.arxiv.org/abs/2602.02380
- [73]Bing Wu, Chang Zou, Changlin Li, Doujun Huang, Fang Yang, Hao Tan, Jack Peng, Jianbing Wu, Jiangfeng Xiong, Jie Jiang, et al. Hunyuanvideo 1.5 technical report. arXiv preprint arXiv:2511.18870, 2025.arxiv.org/abs/2511.18870
- [74]Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. Autogen: Enabling next-gen llm applications via multi-agent conversation. arXiv preprint arXiv:2308.08155, 2023.arxiv.org/abs/2308.08155
- [75]Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.09341, 2023.arxiv.org/abs/2306.09341
- [76]Yang Wu, Stefano Petrangeli, Ishita Dasgupta, and Yu Shen. CamPilot: A Multi-Agent Cinematic Assistant for Camera-Controlled Movie Generation. arXiv preprint arXiv:2609.10943, 2026.
- [77]Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36:15903–15935, 2023.
- [78]Jiazheng Xu, Yu Huang, Jiale Cheng, Yuanming Yang, Jiajun Xu, Yuan Wang, Wenbo Duan, Shen Yang, Qunlin Jin, Shurun Li, et al. Visionreward: Fine-grained multi-dimensional human preference learning for image and video generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 11269–11277, 2026.
- [79]Ran Xu, Tianci Liu, Zihan Dong, Tony Yu, Ilgee Hong, Carl Yang, Linjun Zhang, Tao Zhao, and Haoyu Wang. Alternating reinforcement learning for rubric-based reward modeling in non-verifiable llm post-training. arXiv preprint arXiv:2602.01511, 2026.arxiv.org/abs/2602.01511
- [80]Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. A-MEM: Agentic Memory for LLM Agents. In Advances in Neural Information Processing Systems, volume 38, 2025.arxiv.org/abs/2502.12110
- [81]Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Linting Zhu, Mengzhao Chen, Zhiheng Liu, Wei Liu, Qiushan Guo, Weilin Huang, et al. Dancegrpo: Unleashing grpo on visual generation. arXiv preprint arXiv:2505.07818, 2025.arxiv.org/abs/2505.07818
- [82]Yu Yang, Yue Liao, Jianbiao Mei, Baisen Wang, Xuemeng Yang, Licheng Wen, Jiangning Zhang, Xiangtai Li, Liang Lv, Hanlin Chen, Botian Shi, Yong Liu, Shuicheng Yan, and Gim Hee Lee. SPIRAL: Self-Evolving Action-Conditioned Video Generation via Reflective Planning Agents, 2026.arxiv.org/abs/2603.08403
- [83]Zihan Yang, Shuyuan Tu, Licheng Zhang, Qi Dai, Yu-Gang Jiang, and Zuxuan Wu. ArcFlow: Unleashing 2-Step Text-to-Image Generation via High-Precision Non-Linear Flow Distillation. arXiv preprint arXiv:2602.09014, 2026.arxiv.org/abs/2602.09014
- [84]Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. Advances in Neural Information Processing Systems, 38:113222–113244, 2026.arxiv.org/abs/2503.14476
- [85]Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-Rewarding Language Models. In Proceedings of the 41st International Conference on Machine Learning, 2024.arxiv.org/abs/2401.10020
- [86]Muhammad Osama Zeeshan, Masoumeh Sharafi, Benoît Savary, Alessandro Lameiras Koerich, Marco Pedersoli, and Eric Granger. CLIP-AUTT: Test-time personalization with action unit prompting for fine-grained video emotion recognition. In European Conference on Computer Vision, pages 680–698, 2026.arxiv.org/abs/2603.27999
- [87]Guibin Zhang, Hejia Geng, Xiaohang Yu, Zhenfei Yin, Zaibin Zhang, Zelin Tan, Heng Zhou, Zhongzhi Li, Xiangyuan Xue, Yijiang Li, et al. The landscape of agentic reinforcement learning for llms: A survey. arXiv preprint arXiv:2509.02547, 2025.arxiv.org/abs/2509.02547
- [88]Rongzhi Zhang, Rui Feng, Zhihan Zhang, Jingfeng Yang, Qingyu Yin, Xin Liu, Zixuan Zhang, Priyanka Nigam, Bing Yin, Tuo Zhao, et al. Qubric: Co-designing queries and rubrics for rl beyond verifiable rewards. arXiv preprint arXiv:2606.03968, 2026.arxiv.org/abs/2606.03968
- [89]Jiaxing Zhao, Xihan Wei, and Liefeng Bo. R1-omni: Explainable omni-multimodal emotion recognition with reinforcement learning. arXiv preprint arXiv:2503.05379, 2025.arxiv.org/abs/2503.05379
- [90]Mingzhe Zheng, Dingjie Song, Guanyu Zhou, Jun You, Jiahao Zhan, Xuran Ma, Xinyuan Song, Ser-Nam Lim, Qifeng Chen, and Harry Yang. CML-Bench: A Framework for Evaluating and Enhancing LLM-Powered Movie Scripts Generation. arXiv preprint arXiv:2510.06231, 2025.arxiv.org/abs/2510.06231
- [91]Mingzhe Zheng, Yongqi Xu, Haojian Huang, Xuran Ma, Yexin Liu, Wenjie Shu, Yatian Pang, Feilong Tang, Qifeng Chen, Harry Yang, et al. VideoGen-of-Thought: Step-by-step generation multi-shot video with minimal manual intervention. NeurIPS NextVid Workshop, 2025.
- [92]Mingzhe Zheng, Weijie Kong, Yue Wu, Dengyang Jiang, Yue Ma, Xuanhua He, Bin Lin, Kaixiong Gong, Zhao Zhong, Liefeng Bo, et al. Manifold-aware exploration for reinforcement learning in video generation. arXiv preprint arXiv:2603.21872, 2026.arxiv.org/abs/2603.21872











