Online learning method improves chatbots by teaching preference understanding
Beyond Reward Scoring: Online Multi-Agent Preference Learning for Multimodal Response Generation
Artificial Intelligence
Summary
Many AI chat systems learn to give better answers by trying to get higher preference scores, but they don't learn why humans prefer one response over another. The authors created a system called PrefMind with two AI agents that discuss and judge which answer better matches human preferences. This approach helps the AI both understand preference reasons and generate better responses. They tested this on different sized models and found improvements in judging and making preferred answers.
What this means in practice
- •For chatbot developers: Train conversational AI to explicitly understand and use human preferences to improve response quality through interactive learning.
- •For customer support teams: Enhance automated support agents’ response relevance by deploying systems that actively learn why certain replies are preferred by users.
Authors
Ziyun Zou, Jiwoo Hong, Yifan Lu, Shao Tang, Weiyi Wu, Aman Gupta, Jin Tian, Imran Razzak
Abstract
Multimodal preference alignment often follows a score-based RLHF pipeline: a reward model is first trained to learn human preferences, then used to score on-policy responses and optimize the response model's generation. This turns preference supervision into a scalar signal for generation: a model may learn to produce answers with higher preference scores, but is rarely trained explicitly to learn the reasons underlying human preferences. We therefore ask whether making preference understanding an internal capability of the response model can improve its generation. To investigate this question, we propose \textbf{PrefMind}, an online dual-agent reinforcement learning framework with two trainable agents: a response Generator and a preference Critic. Given a pair of candidate responses, the agents analyze and discuss which response better aligns with human preference. Human labels supervise their judgments and corrections. For pairs on which both the Generator and revised Critic correctly identify the human-preferred response, the Generator produces an improved answer, and the Critic compares it against the original preferred response to provide a generation reward. Both agents are jointly updated from interactions sampled with their current policies. Through this process, \textbf{PrefMind} trains the Generator to understand human preferences via explicit judgment and correction, and to translate this understanding into better responses. Experiments across three model scales show improvements in both standalone preference judgment and downstream response generation.