Reward model bias reduction improves language model responses
BiasReducer: Adaptive Bias Mitigation for Reward Models
Machine Learning
Summary
Reward models help guide AI language systems to give answers people prefer, but sometimes they favor things like longer or more confident responses instead of correct ones. The authors propose BiasReducer, a method that adjusts these reward models to reduce such biases without expensive retraining. BiasReducer identifies which superficial factors influence the scores and tweaks the model accordingly for each new set of data. This approach makes language models less likely to give overly long or flattering responses while keeping quality high.
What this means in practice
- •For ai product developers: Adjust reward models in AI systems to reduce biases that cause verbose or overly confident outputs, improving user experience without retraining from scratch.
- •For chatbot platform engineers: Integrate adaptive bias correction to fine-tune response scoring models dynamically per dataset, enhancing chatbots’ relevance and conciseness.
Authors
Shuang Liu, Yongliang Miao, Yanguang Liu, Haoyi Xiong, Mengnan Du
Abstract
Reward models score responses from large language models (LLMs) and guide LLM training toward human preferences. However, reward models can favor superficial attributes such as length or confidence, leading LLMs to produce higher-scoring but not more correct responses. Existing mitigation methods either retrain the reward model or apply a fixed correction to one known bias, such as a preference for longer responses. Retraining requires additional data and computational resources, while existing editing methods require the target bias to be specified in advance and use a fixed edit for that bias. To this end, we propose BiasReducer, a lightweight framework that edits only the linear reward head and selects the relevant edits for each new dataset. First, BiasReducer uses a sparse autoencoder (SAE)-style encoder to learn which attributes (e.g., length and confidence) the reward model is sensitive to. Second, it learns how to reduce the reward model's dependence on each attribute by determining which direction to adjust the reward head and how much to adjust it. Third, for a new dataset, it ranks the attributes by their influence on reward scores, selects the relevant ones, and edits the reward model accordingly. BiasReducer consistently improves reward-model robustness to biases toward superficial response attributes. Across five reward models, BiasReducer-M improves the three benchmarks by 8.3, 18.0, and 6.9 percentage points on average, outperforming the two training-based baselines. The gains transfer downstream, reducing unnecessary verbosity and sycophancy while maintaining comparable judged quality.