Multi-reward policy gradients improve helpfulness safety and math tasks
ORPG: Reconciling Multiple Reward Objectives through Objective-wise Policy Gradients
Machine LearningComputation and Language
Summary
Sometimes, computer programs need to learn from multiple goals or rewards that may not always agree. The authors developed a method called ORPG that treats each objective separately and then combines their learning signals smartly. When goals align, it mixes their directions to keep the combined learning strong; when they conflict, it prioritizes the more important goals. This approach improved performance on tasks like making AI responses helpful and safe, and solving math problems more accurately and efficiently.
What this means in practice
- •For chatbot developers: Improve AI assistants’ responses to balance helpfulness and safety with better learning methods for multiple goals.
- •For automated reasoning teams: Enhance mathematical problem-solving AI to produce more accurate and concise answers via refined multi-reward training.
Authors
Shicheng Fang, Yiwen Zhao, Wenbo Tian, Jiahao Lu, Yining Zheng, Yuxin Wang, Xipeng Qiu
Abstract
Multi-reward policy optimization requires a joint update that reflects both the learning signals and the intended relationships among objectives. We introduce Objective-wise Reconciled Policy Gradient (ORPG), which constructs a separate clipped policy objective for each reward and reconciles the resulting gradients into one policy update. For compatible gradients, a cosine-dependent interpolation coordinates their contributions through a partially normalized reference while preserving the norm of their sum. We characterize this update as the unique solution of a spherical directional compromise. For conflicting gradients, projection follows the task's priorities. We evaluate the same compatible rule in helpfulness--safety alignment and correctness--cost optimization for mathematical reasoning. ORPG substantially improves average Useful and Harmless scores over the strongest external baseline on each axis. In mathematics, it achieves the highest average full-budget accuracy and three-budget hypervolume among the compared methods, with more accurate and shorter responses than the initial policy. Component comparisons and training dynamics show the larger contribution of compatible coordination and a complementary benefit from conflict handling. These results support gradient reconciliation for objectives with equal standing and for objectives with an explicit priority.