Thinking reward model improves visual generation quality control
Think Before You Score: Thinking Reward Model for Visual Generation
Computer Vision and Pattern Recognition
Summary
Evaluating images made by computers can be tricky because it's not always clear what qualities to judge each image by. The authors propose a new way where the system first decides what matters most for each image before scoring it. Their method gives detailed scores that adapt to each case, helping models learn to generate better images. Tests show this approach matches or beats other top methods and helps improve image generation tools when used for learning.
What this means in practice
- •For visual content creators: Improve AI-based image generation tools by using detailed reward signals that adapt to each image's unique qualities.
- •For software engineers for creative ai: Enhance reinforcement learning processes in image editing software by integrating adaptive scoring models for better output quality control.
Authors
Xuehai Bai, Zhenchen Tang, Yang Shi, Dianyi Wang, Tengfei Liu, Wanshun Su, Xuanyu Zhu, Ruohui Wang, Haiwen Diao, Haotian Wang, Xiaoling Gu, Yuanxing Zhang
Abstract
Visual reward models are essential for evaluating and improving visual generation models, yet existing approaches typically map task conditions and candidate outputs directly to scalar rewards, leaving implicit what should be evaluated for each individual case. We introduce Think Before You Score, a paradigm that explicitly determines what matters for each case before judging how well the candidate performs. Following this principle, we propose the Thinking Reward Model (TRM), which formulates case-adaptive rubrics, performs rubric-guided assessment, and produces fine-grained pointwise rewards. We further observe that conventional pairwise preference optimization can induce score polarization, and introduce Pairwise Dual-Group Relative Policy Optimization (PD-GRPO), which leverages pairwise supervision to improve reward discrimination while preserving fine-grained pointwise scoring. Extensive experiments on image generation and editing reward-modeling benchmarks demonstrate that TRM achieves state-of-the-art performance among open-source reward models while remaining highly competitive with proprietary alternatives. Moreover, using TRM as a reward for reinforcement learning consistently improves diverse visual generation models, demonstrating that its fine-grained, case-adaptive rewards translate into effective optimization signals for visual generation.