AI summaryⓘ
The authors study how to manage multiple goals in reinforcement learning by deciding when to introduce each goal into the learning process, rather than combining them all at once. They propose a method called methodname that starts with a few goals and gradually adds more based on a stability check and a patience limit. This method also estimates the right order to add goals and adjusts focus to underperforming ones. Their experiments show that this approach performs better than simply combining all goals at once or sharing a fixed budget among them. The authors highlight that timing when each goal enters learning is an important factor to consider in multi-objective reinforcement learning with human feedback.
multi-preference alignmentscalarizationreinforcement learningreward vectoractive setpolicy optimizationreward deviationadaptive weightingreinforcement learning from human feedback (RLHF)objective admission
Authors
Yongqi Tong, Zhenyu Zhang, Ruirui Wang, Kewei Fu, Shaoqing Lin, Sijie Dong, Jiang-Ming Yang, Xin Zhang, Jianshe Li
Abstract
Multi-preference alignment is often framed as scalarization: combine reward dimensions, then optimize. This leaves a temporal decision underspecified: when should each preference dimension enter policy optimization? We propose \methodname, a stability-guided active-set controller for controlled objective admission. \methodname starts from a small active set, retains admitted objectives, and expands when reward-deviation gates indicate low recent deviation or a patience budget is exhausted. A probing phase estimates a hard-to-easy order, and adaptive weighting emphasizes underperforming active dimensions. Automatic evaluations with 15 training preferences and 16 held-out benchmark columns show that \methodname obtains higher averages than simultaneous scalarization and shared-budget adapted baselines. Component ablations and expansion dynamics further support cumulative retention, gated admission, and probing-derived ordering as useful design choices in this setting. These results position objective-entry timing as a concrete control variable in reward-vector RLHF.