Multi reward learning improves language model alignment with better conflict use
GPDPO: Learning from Reward Conflicts via Group Pareto-Dominance Policy Optimization
Machine LearningComputation and Language
Summary
Large language models are trained using multiple rewards to make their outputs better, but sometimes these rewards conflict and confuse the training process. The authors propose a new method called GPDPO that finds ways to use the relationships between different responses, even when rewards conflict, to guide training more effectively. Their experiments show that GPDPO improves training stability and performance without needing extra data or reward calculations.
What this means in practice
- •For language model developers: Improve training methods for language models by better handling conflicting reward signals to align models with diverse objectives.
- •For ai product teams: Use enhanced multi-reward learning to produce more consistent and aligned AI outputs for user-facing applications without extra computational cost.
Authors
Zixuan Xu, Haoxuan Chen, Jiangxin Sun, Jian-Fang Hu
Abstract
Multi-reward reinforcement learning offers a promising framework for aligning large language models (LLMs) with diverse objectives, but conflicting reward signals can induce ambiguous update directions and hinder stable optimization. Existing approaches address such conflicts through reward reweighting, gradient coordination, or rollout filtering, yet they primarily operate on per-response or group-level optimization signals and underexploit relational information among sampled responses. As a result, conflict-filtered rollouts may still contain useful supervision that is discarded during training. We propose Group Pareto-Dominance Policy Optimization (GPDPO), a multi-reward policy optimization method that recovers pairwise supervision from conflict-filtered rollouts. GPDPO identifies Pareto-dominance relations among sampled responses and converts them into auxiliary preference signals, which are jointly optimized with the original policy objective. This enables more effective reuse of existing rollouts without additional sampling or reward evaluation. Experiments on API-Bank across multiple model scales and reward configurations show that GPDPO consistently outperforms competitive multi-reward optimization baselines. Further analysis demonstrates that GPDPO reduces both group-relative and pairwise reward conflicts, supporting the effectiveness of exploiting relational supervision for multi-reward alignment. Code will be released.