Multi objective reinforcement learning improves real world results with better optimization
A Better Spur Should Start From Each Objective
Artificial IntelligenceMachine Learning
Summary
Multi-objective reinforcement learning faces problems like weak rewards and conflicts between goals, making it hard to get good results. The authors propose a new method called MMPO that handles these issues more carefully by fixing biases in the data, managing conflicting learning signals, and balancing the importance of different goals. Their approach helps training stay steady and achieves better performance on real-world tasks like e-commerce recommendations and coding assistants. This shows that fine-tuned techniques can make learning with multiple goals more practical and reliable.
Multi-objective reinforcement learningSparse rewardsReward conflictScalarizationGradient optimizationOrthogonal projectionExposure debiasingE-commerce datasetsTraining stabilityMulti-objective alignment
Authors
Shanwen Mao, Hao Zhang, Guangtao nie, Zhiheng Li, Huimu Wang, Sulong Xu, Gu Simiu
Abstract
Real-world Multi-Objective Reinforcement Learning (MORL) often suffers from sparse rewards, reward conflicts, and late-stage reward tug-of-war, causing traditional linear scalarization to experience severe metric oscillations. To address optimization conflicts among multiple objectives in real-world deployment scenarios, we propose Multi-Marginal Preference Optimization (MMPO), a fine-grained framework that intervenes at the data, gradient, and constraint levels rather than relying on coarse-grained global scalarization. Specifically, MMPO performs exposure debiasing to mitigate sparse and biased rewards, applies priority-aware orthogonal projection to decouple conflicting gradients, and introduces self-prompted gradient constraints to prevent dominant objectives from overwhelming weaker ones. Experiments on real-world e-commerce datasets show that MMPO improves training stability and consistently achieves better performance across conflicting metrics. Moreover, it generalizes robustly to broader tasks such as ToolRL and code generation, demonstrating its effectiveness as a practical paradigm for multi-objective alignment.