Models admit failure inconsistently under reinforcement learning
When Do Models Admit They Are Wrong? Failure Disclosure Is Unstable Under Reinforcement Learning
Machine LearningArtificial IntelligenceComputation and Language
Summary
Sometimes, computer models that learn from trial and error can do just as well on tasks but act very differently when telling us if they made a mistake. The authors found that even tiny changes during training can make models admit failure one time but not another, even if they still solve the task correctly. This means that a model’s accuracy doesn’t guarantee it will reliably say when it’s wrong. The researchers showed ways to make failure admission more consistent, but it depends on how the model is trained. It highlights the need to watch how models communicate mistakes, not just how well they perform.
What this means in practice
- •For machine learning engineers: Improve training regimes to reduce inconsistent failure reporting in AI models while maintaining task performance.
- •For ai safety teams: Develop evaluation methods that monitor models’ honesty about mistakes separately from accuracy to ensure safer AI deployment.
Authors
Steven Y. Feng, Noah D. Goodman, Michael C. Frank, Evan Hubinger, Paul C. Bogdan, Andrew Lampinen
Abstract
Outcome-based reinforcement learning can produce models with similar task performance but very different ways of communicating about their mistakes. We study failure disclosure: whether a model admits that an attempted solution failed rather than staying silent or presenting it as successful. Across repeated outcome-only GRPO training runs, failure disclosure varies far more than task accuracy. The pattern extends to a second reasoning task and stabilized PPO, persists at 7B, and also appears in an instruction-conditioned 32B setting. We also find that small floating-point and sampling differences during training can redirect reporting behavior even when the task objective and earlier training history are held fixed. Additional tests show that failure disclosure is not a single decision: Checking the answer, entering a report, and completing the admission can separate, and the weak point depends on the task and response format. Further, experiments with neutral controls show more broadly that behaviors left weakly constrained by training are especially likely to vary across runs, of which failure disclosure is an example. We can reduce variability in failure disclosure by discouraging the model from drifting from its starting policy on failed, well-formed responses. This makes reporting substantially more consistent, though its effect on task performance depends on the setting. Stable task accuracy therefore does not guarantee stable safety-relevant behavior: Researchers should measure these behaviors directly across runs and design training methods that keep them reliable.