Reward models score responses differently depending on prompts
What Pairwise Evaluation Misses: Reward Scores across Prompts
Computation and LanguageArtificial Intelligence
Summary
When AI systems rank or choose between responses, they use reward models that give scores. These scores should be comparable across different prompts, but the authors found that scores vary a lot depending on the prompt used. This variation is bigger than what humans show in their ratings, meaning the models might be biased by the prompt itself. The authors propose ways to test and measure this issue, suggesting that evaluations should separately check how scores change across prompts, how sensitive they are to wrong prompts, and how useful they are for selecting good responses.
What this means in practice
- •For machine learning engineers: Test and improve reward model evaluations by checking how scores vary across different prompts beyond pairwise comparisons.
- •For ai product testers: Use human-referenced audits to identify if reward scores are overly influenced by prompt wording when selecting AI-generated responses.
Authors
Mateusz Modrzejewski, Tomasz Radzikowski
Abstract
Reward models score responses for preference optimization, filtering and ranking, and pooling scores across prompts needs them to be comparable. Yet pairwise evaluation compares two responses to the same prompt, so any score component shared by a prompt's responses cancels and goes untested. We study this component by comparing between-prompt variation in model scores with the same variation in human ratings of the same responses. We propose a human-referenced audit, built from a between-prompt variance share, a sibling-rating regression and prompt substitutions, and apply it to eighteen released scorers and six matched training conditions. We find that scores cluster by prompt more than the ratings do: on HelpSteer2, six reward models have between-prompt shares of 0.34 to 0.64 against a human share near zero. Replacing the original prompt with a fixed instruction keeps substantial clustering but weakens agreement with prompt-level human ratings, so clustering alone does not show agreement with humans. Part of that variation is useful: subtracting prompt means lowers the helpfulness of globally selected responses, and scores stay sensitive to a wrong prompt even when training shuffles prompt means. We propose that reward evaluation check between-prompt variation, sensitivity to a wrong prompt and usefulness for selection separately.