Self retrospection improves learning from uniform rewards in AI agents
Self-Retrospection Distillation: Turning Post-hoc Experiences into Prior Foresight
Artificial IntelligenceComputation and LanguageInformation RetrievalMachine Learning
Summary
Sometimes AI agents get the same reward for different attempts, making it hard for them to learn what actions are better. The authors studied a way for agents to learn from looking back at their full experience after trying a task, so they can predict what good choices would look like before acting next time. This method, called Self-Retrospection Distillation, helps agents learn better even when they get no clear reward hints during practice. It showed big improvements in tasks where rewards were often uniform or zero.
What this means in practice
- •For ai system developers: Improve training of AI agents when clear reward signals are rare or absent by using retrospection to guide their predictions.
- •For robotics engineers: Enhance robots’ task learning in environments where success signals are sparse by integrating post-task experience into their learning process.
- •For game ai designers: Build game agents that learn better from uniform reward outcomes by distilling knowledge from entire gameplays into their pre-move predictions.$Commercial implications: Enables creation of game AI that adapts intelligently even when standard scoring feedback is limited or ambiguous.
Authors
Haoxiang Zhang, Qinglin Chen, Hiroaki Hayashi, Zhuofeng Li, Siming Zhang, Jiaxin Zhang, Jixuan Chen, Fang Wu, Pan Lu, Silvio Savarese, Julian McAuley, Chien-Sheng Wu
Abstract
Reinforcement learning with verifiable rewards (RLVR) turns agent experience into learning signals primarily through scalar outcome rewards after interaction. For group-relative objectives, however, this signal vanishes when all rollouts receive the same reward, even though their trajectories may reveal useful information about what the task requires and how the agent fails. We ask a complementary question: can hindsight teach an agent what it could have anticipated before acting? We introduce prospective learning, which uses post-hoc experience to supervise foresight predictions from the pre-interaction view, and instantiate it with Self-Retrospection Distillation (SRD). Intuitively, a completed trajectory reveals knowledge that would have been useful and pitfalls that should be avoided; SRD distills this privileged hindsight into trajectory-blind foresight of the same policy. Foresight serves only as a training target and need not be explicitly generated at inference time. Across 10 tool-integrated reasoning and long-horizon agentic tasks, SRD complements RLVR and self-distillation baselines with gains of up to $24.2$ pp. Its advantage is especially pronounced when reward contrast is scarce: when $37$--$98\%$ of rollout groups are reward-uniform across model scales, yet SRD can still exploit learning signal from sampled trajectories. In the 2B setting, where $98\%$ of groups are all-failure, the RLVR training ends up at $0.0\%$ success, while adding SRD reaches $60.6\%$ under the same rollout budget. Our results suggest that post-hoc agent experience is useful not only for evaluating or improving behavior, but also for shaping predictive representations before available interaction.