Multi-objective reinforcement learning method improves performance stability
A Better Spur Should Start From Each Objective
Artificial IntelligenceMachine Learning
Summary
Many real-world problems require balancing multiple goals at once, but this can cause conflicts that confuse traditional methods. The authors introduce a new approach called Multi-Marginal Preference Optimization (MMPO) that carefully manages these conflicts at different stages of learning. MMPO reduces bias from sparse feedback, separates conflicting updates, and avoids letting stronger goals overwhelm others. Their tests show that MMPO leads to more stable learning and better results across various tasks like e-commerce recommendations and code generation.
What this means in practice
- •For e-commerce engineers: Improve recommendation systems by managing conflicting goals like user engagement and sales through stable multi-objective training.
- •For software developers: Enhance AI models that generate code or use tools by balancing competing output objectives more reliably with MMPO techniques.
Authors
Shanwen Mao, Hao Zhang, Guangtao nie, Zhiheng Li, Huimu Wang, Sulong Xu, Gu Simiu
Abstract
Real-world Multi-Objective Reinforcement Learning (MORL) often suffers from sparse rewards, reward conflicts, and late-stage reward tug-of-war, causing traditional linear scalarization to experience severe metric oscillations. To address optimization conflicts among multiple objectives in real-world deployment scenarios, we propose Multi-Marginal Preference Optimization (MMPO), a fine-grained framework that intervenes at the data, gradient, and constraint levels rather than relying on coarse-grained global scalarization. Specifically, MMPO performs exposure debiasing to mitigate sparse and biased rewards, applies priority-aware orthogonal projection to decouple conflicting gradients, and introduces self-prompted gradient constraints to prevent dominant objectives from overwhelming weaker ones. Experiments on real-world e-commerce datasets show that MMPO improves training stability and consistently achieves better performance across conflicting metrics. Moreover, it generalizes robustly to broader tasks such as ToolRL and code generation, demonstrating its effectiveness as a practical paradigm for multi-objective alignment.