Text to video models vary in expressing and controlling emotions accurately
EmoTrace-Bench: Diagnosing Emotional Expression in Text-to-Video Generation through Controlled Cues and Intensity
Computer Vision and Pattern Recognition
Summary
Generating videos from text that show specific emotions is tricky. The authors created a special test called EmoTrace-Bench to check how well these models show the right emotions, how strong those emotions appear, and which details in the text help. They used many prompts and videos across different emotions and intensity levels, finding that some ways of describing emotions work better than others. This test helps understand exactly where these video generation models do well or poorly without needing to change the models themselves.
What this means in practice
- •For video content creators: Diagnose and improve the emotional accuracy of AI-generated videos by testing prompt strategies using the EmoTrace-Bench framework.
- •For multimedia software developers: Incorporate detailed emotional evaluation tools from EmoTrace-Bench to validate emotion expression and intensity in text-to-video generation features.
Authors
Yeheng Wang, Liudan Shi, Yi Zheng, Jingke Li, Shiyuan Liu, Xin Tao, Yuanxing Zhang, Guoyu Sun
Abstract
Text-to-video (T2V) models are increasingly used to generate videos with targeted emotional expressions, yet existing evaluation methods mainly rely on holistic video-text alignment and provide limited insight into whether failures arise from target-emotion expression, intensity control, or specific affective cues. We introduce EmoTrace-Bench, a fine-grained diagnostic benchmark that controls emotion intensity and selected affective cues in prompts and evaluates generated videos at multiple levels. EmoTrace-Bench contains 448 controlled prompts covering eight emotions, two intensity levels, and diverse cue conditions. Using 12 T2V models, we generate 5,376 videos and evaluate target-emotion presence, perceived intensity, and ten audiovisual dimensions of emotional expression with an automated evaluator calibrated and validated against human annotations on a subset of the data. Results reveal substantial differences across models and prompt designs. Prompts that explicitly describe a character's emotional reaction are followed more reliably than prompts that describe only events or scenes, whereas adding more affective cues or using more explicit cinematic audiovisual language does not consistently improve emotional generation. These results illustrate how aggregate alignment scores may overlook localized differences in target-emotion expression, intensity control, and cue realization. EmoTrace-Bench provides a traceable and interpretable framework for diagnosing emotional video generation without modifying existing generation pipelines, and we release the prompts and the automated evaluation rules.