Generative reward models scale to improve creative writing policy optimization
Scaling Laws for Generative Reward Models
Artificial Intelligence
Summary
This paper looks at how AI systems can learn to judge creative writing by training reward models of different sizes. The authors found that bigger reward models often make better judgments when tested on fixed examples, but smaller or differently trained models can lead to better AI writing performance after learning through feedback. This shows that being good at judging examples doesn’t always mean being better at guiding improvements in writing quality. The authors also explored how different training settings and model sizes impact this behavior.
What this means in practice
- •For ai developers: Use scaled generative reward models to improve feedback quality for AI systems that generate creative writing.
- •For chatbot designers: Select smaller or differently trained reward models for online learning to boost conversational AI’s creative outputs.
Authors
Angel Raychev, Nathan Lile, Rafael Rafailov
Abstract
We study generative reward models (GenRMs) as online preference providers for policy optimization in creative writing. Across Qwen3 reward-model sizes from 0.6B to 14B, Thinking judges trained with GRPO have higher static Gold-label accuracy than Answer-Only judges trained with SFT. This ranking often reverses when the judges train policies through online DPO: Answer-Only rewarders yield higher peak and top-five-average policy Elo in seven of eight matched settings under a fixed, learned 32B Gold evaluator. Intermediate Thinking-judge checkpoints can also outperform later, more statically accurate checkpoints as rewarders. Sweeps over rewarder size, policy size, training duration and DPO regularization show substantial benefits from reward-model capacity, including judges larger than the policy. We analyze trajectories with evaluator-specific Elo arenas, conditional match-resampling intervals and descriptive scaling fits. These results demonstrate an evaluator-rewarder gap in the studied pipelines: fixed-distribution judging accuracy does not determine downstream optimization quality. The comparison changes judge-training objective as well as response format, and the main ranking is conditional on its learned reference rather than a universal human-quality result. A limited follow-up using a different learned judge is inconclusive overall.