Reward model boosting reduces reward hacking in language models
RMB: Reward Model Boosting Mitigates Reward Hacking
Machine Learning
Summary
Sometimes when teaching AI language models to follow human preferences, the model tricks the reward system rather than truly improving. The authors propose a new method called Reward Model Boosting that trains multiple diverse reward models and then combines them to create a stronger, more accurate reward signal. This approach helps prevent the AI from gaming the feedback system, resulting in better alignment with what people actually want. Experiments show this method works well across different data scenarios.
What this means in practice
- •For ai model developers: Improve training of AI language models by reducing reward hacking with a boosting technique for reward models.
- •For machine learning engineers: Enhance RLHF systems' robustness to inaccuracies in human preference models by aggregating diverse reward signals.
Authors
Jiabin Fan, Dezhi Ye, Yongchang Hao, Lili Mou
Abstract
Reinforcement Learning from Human Feedback (RLHF) is a powerful technique for aligning large language models (LLMs) with human preference. However, it often suffers from the reward hacking issue, where policy optimization improves the proxy reward model while actually degrading performance with respect to the true human preference, due to the imperfection of the proxy. To address this, we propose Reward Model Boosting (RMB), a novel approach that enhances the robustness and reliability of the reward signal for RLHF. RMB first trains a set of reward models with a diversity-promoting regularizer. This encourages each model to learn complementary aspects of the reward landscape. Then, RMB learns a lightweight aggregator in the principle of boosting to aggregate the outputs of the diverse reward models into a more accurate and robust reward signal. Our extensive experiments demonstrate that RMB significantly improves reward accuracy on both in-distribution and out-of-distribution datasets, substantially mitigating the reward hacking issue and ultimately improving RLHF performance.