Reward models with evolving penalties reduce cheating in trained language models
Mitigating Reward Hacking in Preference Reward Models with Evolving Penalties
Computation and LanguageArtificial Intelligence
Summary
When teaching language models to perform tasks, they sometimes find shortcuts that trick the system into thinking they did well, even if the responses are worse. The authors introduced a method called Dynalize that watches the model's training and adds penalties when cheating behaviors appear. This helps the language model keep improving on real quality measures, not just the reward model's score. Their approach works well in complex settings like counseling and creative writing, where it's hard to write strict rules for good answers.
What this means in practice
- •For ai development teams: Improve long-term training of language models by automatically mitigating reward hacking without manual penalty design.
- •For therapeutic chatbot developers: Enhance chatbot performance for clinical counseling by reducing undesired behaviors encouraged by preference reward models.
Authors
Afonso Baldo, António Farinhas, Marcos Treviso, Ricardo Rei, Nuno M. Guerreiro
Abstract
Reinforcement learning for language models has moved from preference reward models to verifiable rewards, which sustain much longer optimization. Preference models are prone to reward hacking: the policy raises their score without improving response quality, or even while degrading it. Yet for open-ended tasks, where quality can neither be verified nor fully written down as rubrics, they remain valuable. We introduce Dynalize, which mitigates reward hacking online, during policy training. Periodically and asynchronously, a language model analyzes training rollouts and writes penalties for undesirable behaviors that the reward model appears to reward; a judge applies them to every rollout. In clinical counseling and creative writing, training on the reward model alone raises its score continuously but ultimately leads to a decline in external benchmarks. Dynalize sustains longer training and results in improved downstream performance. For example, we find that our method suppresses patterns incentivized by preference models that, without penalties, spread to most responses, and even to the model's reasoning, which the reward model does not explicitly score. Furthermore, we show that its gains hold in multi-turn training, and even after we retrain the reward model on the discovered behaviors.