Long horizon LLM agents learn coordination beyond local decisions
BLPO: Beyond-Local Policy Optimization for Long-Horizon LLM Agents
Artificial Intelligence
Summary
Solving tasks that require many linked steps is hard for AI models because actions affect later ones. The authors created a new training method called BLPO that helps AI agents learn by sharing experience across tasks, improving their ability to plan far ahead. This method summarizes what’s been done so far and groups similar actions, so learning is more effective and stable. Experiments show that BLPO leads to faster progress and better task completion without extra computing cost.
What this means in practice
- •For workflow automation teams: Train AI agents to complete complex multi-step workflows with improved success rates and efficiency.
- •For robotics developers: Improve robot planning over long sequences of actions by sharing learned experience across different tasks.
Authors
Xuanren Chen, Chongyang Tao, Wenpeng Hu, Yong Wang, Bo Deng, Xinlei Zheng, Shuai Ma
Abstract
Long-horizon tasks require large language model (LLM) agents to coordinate interdependent actions, a capability central to automating complex workflows. Stepwise group-based reinforcement learning provides finer credit than trajectory-level supervision, but reliable comparisons must account for differences in interaction history. Enforcing this consistency within the limited rollouts of each task can leave insufficient support for estimating action preferences. We propose Beyond-Local Policy Optimization (BLPO), which estimates task-progress advantages from experience shared across tasks and training updates. We summarize trajectory prefixes into task-progress states and abstract actions by their function, allowing corresponding decisions from different tasks to contribute to shared estimates. To control bias and noise from heterogeneous tasks, we accumulate context-relative returns in a global memory with balanced aggregation and reliability weighting. These global advantages complement local comparisons over normalized actions and trajectory-level credit without a learned critic or additional rollouts. Experiments on ALFWorld and WebShop with 1.5B and 7B policies show improved task success over the evaluated baselines under matched training budgets. Further analyses show faster learning and shorter trajectories with little computational overhead.