Policy learns from failures by rewriting mistakes to improve
Failure Is the Mother of Success: Recovering Supervision with Grounding Hindsight
Artificial Intelligence
Summary
When an AI tries to solve a problem but fails every time, it usually learns nothing useful from those failures. The authors developed a way for the AI to write hints to itself about what went wrong without seeing the answer first. The AI then uses these hints to try solving the problem again, and if it succeeds more often, it keeps the helpful hints. This method helps the AI improve even when it keeps failing initially.
What this means in practice
- •For machine learning engineers: Improve training of AI systems by letting them learn from all failures, not just successes.
- •For automated reasoning developers: Enhance problem-solving AI by enabling recovery of useful signals from unsuccessful attempts during training.
Authors
Sheldon Yu, Tong Yu, Nikki Lijing Kuang, Zhendong Chu, Zhenwei Tang, Sungchul Kim, Lina Yao, Julian McAuley, Junda Wu
Abstract
When every rollout for a problem fails, GRPO receives neither a successful exemplar nor a nonzero group-relative advantage, so it learns nothing from exactly the problems it most needs to learn. The failed reasoning can nevertheless suggest what to try next. We present Replay-Selected Hindsight Distillation (RSHD), which turns failed attempts into an additional, tested source of supervision during GRPO training. The policy writes a corrective note about each failed attempt without seeing a reference answer or solution, then re-solves the problem with the note in context. A note is admitted only if its replays succeed more often than the original group did, after screening for notes that state the answer, and an admitted replay is distilled into the plain-prompt policy. Because admission is decided by fresh successes rather than by the failed rollout itself, the procedure can recover supervision from groups in which every attempt failed, and it applies equally to failures in mixed groups. Only the policy is trained, the only judge is the outcome verifier, and deployment requires no notes or extra inference. Across model families and sizes and two training domains, RSHD consistently improves over GRPO on in-domain benchmarks, and the gains carry over to out-of-domain evaluation sets.