Reflection-guided learning improves long-term decision making in AI agents
ReflectPO: Reflection-Guided Policy Optimization for Long-Horizon LLM Agents
Computation and Language
Summary
Making good decisions over a long time is hard for AI agents because early mistakes can mess up everything that comes after. The researchers created a method called ReflectPO to help AI better recognize when they have gone off track and fix their mistakes along the way. ReflectPO uses smart ways to look back at previous actions and rewards to figure out which decisions led to errors and how to recover from them. Their tests show that agents using ReflectPO do better at completing long tasks with tools than other existing methods.
What this means in practice
- •For autonomous robotics teams: Improve robots' ability to recover from mistakes during complex multi-step tasks involving tool use or interaction with changing environments.
- •For virtual assistant developers: Enhance AI assistants to better handle long dialogs or workflows by detecting and correcting errors autonomously over extended interactions.
Authors
Rumei Li, Xunliang Cai
Abstract
With external tools, large language models (LLMs) can tackle complex tasks through sustained environmental interaction. LLM-based agents are therefore increasingly used to tackle complex long-horizon tasks in real-world environments. However, these tasks involve sequential, interdependent decisions and changes in environmental states. Early execution errors affect subsequent interaction histories and environmental states, and their effects accumulate along the decision chain. To address this challenge, we propose Reflection-Guided Policy Optimization (ReflectPO), a reinforcement learning framework for autonomous error recovery in long-horizon tasks. ReflectPO comprises two core components. The first is Reflection-Guided Sampling. We use branch differences and repeated visits in interaction trajectories to construct hints for error recovery. The second is Tree-Structured Credit Assignment. We organize sampled trajectories into trees and assign action-level credit using parent--child reward differences and recovery gains. Experiments over two long-horizon tool-use benchmarks show that ReflectPO outperforms multiple existing baselines in most domains. It also achieves the best overall performance on average. Further analyses show that ReflectPO effectively improves agents' ability to recognize their own deviations, recover from erroneous states, and continue making progress toward task completion.