Self-explaining language models improve task solving without reinforcement learning
Shockingly Simple Self-retrospection Improves Agentic Models Without RL
Artificial IntelligenceComputation and Language
Summary
People learn not only by doing but also by thinking about and explaining their experiences, which helps them improve. The paper shows that a language model can train itself to do better on tasks just by explaining what it did, without needing rewards or teachers. This method, called ROFT, made the model solve more problems and even learn from tasks where it initially failed every attempt. The explanations help the model figure out which actions were good or bad, leading to better future behavior. This suggests teaching AI to explain could help it learn more effectively.
What this means in practice
- •For ai systems engineers: Improve AI agents that perform tasks by training them to learn from their own explanations without needing external reward signals.
- •For software automation teams: Increase effectiveness of code generation models by fine-tuning them on self-generated explanations to solve programming problems more reliably.
Authors
Jonathan Light, Christopher Zhang Cui, Jeonghye Kim, Roger Creus Castanyer, Emiliano Penaloza, Zhengyan Shi, Alessandro Sordoni, Marc-Alexandre Côté, Xingdi Yuan, Minseon Kim
Abstract
People learn not only by repeating successful actions, but also by recounting and explaining their experiences, revising their understanding to guide future behavior. Can a language-model agent improve its future actions by training only on explanations of its own experience? We investigate this question by studying Retrospection-Only Fine-Tuning (ROFT), a minimal online procedure designed to isolate the effect of explanation-only training on subsequent behavior. The agent attempts a task, observes available feedback, generates a retrospective explanation, and is fine-tuned with a next-token prediction loss on the explanation tokens alone. The procedure uses neither an external teacher nor a reward-based policy update. In software-engineering experiments with Qwen3.5-4B, ROFT is trained on problems with mixed successful and unsuccessful base-model attempts. On held-out SWE-bench Verified and Pro, it reaches 49.2% and 26.8% solve rates after 20 updates without using a verifier, compared with GRPO's 48.0% and 25.3% after 40 updates in the evaluated runs, and makes faster early progress in training time and sampled attempts. It also learns to solve individual tasks on which all 64 sampled base-model attempts failed, showing that learning can begin without any initially successful trajectories. Behavioral analyses find that ROFT indirectly assigns credit to actions, encouraging good actions and discouraging incorrect ones. Moreover, prompting retrospections to emphasize more direct solutions yields shorter subsequent attempts even without an explicit length penalty. Together, these findings show that learning to explain can also improve learning to do, establishing self-generated retrospections as useful training targets and motivating further study of explanation-to-action transfer.