Post-training large language models tolerate cheap reward verifiers well
A Cheap Verifier is Good Enough: LLM Post-training is Robust to Erroneous Rewards
Machine LearningArtificial IntelligenceComputation and Language
Summary
Training large language models often uses other models called verifiers to check if the training is going well. This paper finds that using expensive, highly accurate verifiers may not always lead to better trained models compared to cheaper, less accurate ones. The researchers tested many models across medical, legal, and finance tasks and showed that simpler verifiers can still guide training effectively, saving lots of time and cost. However, some differences remain in specific cases, so verifiers are not always interchangeable.
What this means in practice
- •For machine learning engineers: Use less costly verifier models during post-training of LLMs to reduce computational expenses without large performance loss.
- •For legal ai developers: Train legal domain language models with inexpensive verifiers to ensure high-quality outcomes without relying on expensive evaluation tools.
Authors
Andreas Plesner, Curtis Northcutt, Francisco Guzmán, Anish Athalye
Abstract
When post-training large language models on tasks with semi-verifiable rewards, there are many factors (training steps, base model size, training order, data quality, verifier accuracy, etc.) that practitioners must contend with to maximize model performance. Yet, it remains unclear how well verifier agreement predicts post-training performance on such tasks. In this paper, we explore this question with over 11k H100 GPU-hours, across HealthBench and PRBench tasks in medical, legal, and finance domains. Across the tested domains, Qwen3 trainees (1.7B-8B on HealthBench; 8B on PRBench), evaluation splits, and frontier LLM reference judges (which we call golden verifiers), higher verifier agreement does not consistently identify the best training verifier. Expensive verifiers need not outperform inexpensive ones, and open-weight Gemma verifiers produce strong training outcomes. We compare two low-cost choices retrospectively -- a cost-reducing choice and a balanced choice -- with estimated grading cost reductions of 98.8%-99.7% relative to the golden grading protocols and average post-training score gaps of 1-3 points from the best evaluated training verifier. These averages include larger losses in individual settings; they do not establish that verifier choices are interchangeable.