Image generation balances emotion and content through new reinforcement method
Balancing Emotional Alignment and Semantic Consistency in Image Generation via Reinforcement Learning with Valence-Arousal Anchoring
Computer Vision and Pattern Recognition
Summary
Generating pictures that match both what they show and the feelings they should convey is tricky because changing the emotion can unintentionally change the content. The authors developed a new approach that uses a kind of learning called reinforcement learning combined with an emotional map based on feelings called valence and arousal. This approach helps the system create images that better match the intended emotions while keeping the scene and objects consistent. They tested it on thousands of prompts and demonstrated it improves emotional accuracy without losing important picture details.
What this means in practice
- •For graphic designers: Generate images with precise emotional nuances without altering the original scene layout or objects described.
- •For advertising creatives: Create emotionally aligned visuals from text prompts for marketing campaigns, improving engagement through controlled affective content.$Commercial implications: Enables new marketing visuals with targeted emotions, increasing ad effectiveness and brand appeal.
Authors
Jisheng Dang, Zhenxuan Wang, Bin Li, Ronghao Lin, Bin Hu, Tat-Seng Chua
Abstract
Continuous emotion control in text-to-image generation requires a model to improve affective alignment without changing the objects, layout, or scene described by the prompt. Existing supervised emotion-injection methods often optimize feature-space proxies and may therefore exhibit emotion-semantic drift, in which stronger emotional conditioning is accompanied by unintended content changes. We address this problem with a flow-matching image-generation framework that combines continuous valence-arousal (VA) conditioning, Group Relative Policy Optimization (GRPO), and a neutral semantic anchor. The deterministic probability-flow ODE is converted into a marginal-preserving SDE, yielding non-degenerate transition densities for trajectory sampling and policy-ratio estimation. A frozen CLIP-based VA regressor supplies a terminal reward measuring the distance between the predicted and target VA coordinates, while an image generated from the same prompt under zero VA conditioning provides a feature-space reference for semantic preservation. A reduced denoising schedule is used for online RL sampling, whereas the original schedule is retained at inference. Experiments on 3,300 prompt-emotion combinations show substantially lower valence and arousal errors than the VA-conditioned baseline and an improved CLIPScore relative to EmotiCrafter, with a measurable trade-off in reference-free image quality. The results support anchor-regularized Flow-GRPO as a practical approach to balancing emotional alignment and semantic consistency in continuous-affect image synthesis.