Pearl improves multi-turn agent learning by spotting recurring decision patterns
PEARL: Process-Experience Aware Reinforcement Learning for Multi-Turn Agents
Artificial Intelligence
Summary
Training agents to make decisions over multiple steps can be hard because it's unclear which actions along the way were good or bad. This paper introduces PEARL, a method that looks closely at where two possible decision paths first differ and identifies recurring patterns in successes and failures. By focusing learning on the critical points where things go wrong, PEARL helps agents improve faster without extra complex steps. The authors tested PEARL on different tasks and found it outperforms previous methods, especially on challenging problems.
What this means in practice
- •For conversational ai developers: Improve training of chatbot systems by better pinpointing which turns in dialogue need adjustment for more effective responses.
- •For robotics control engineers: Enhance multi-step robot task learning by focusing updates on recurring decision errors detected from past runs.
Authors
Jie Yang, Mianqiu Huang, Yiwei Zhao, Bo Wang, Yang Liu, Yuxin Wang
Abstract
Terminal rewards make multi-turn agents trainable without step-level annotations, but group-relative advantages provide little guidance about which intermediate decisions to adjust. Failed trajectories also contain reasonable actions, while unfavorable decision patterns can recur as training proceeds. We introduce PEARL (Process-Experience Aware Reinforcement Learning), which improves local credit assignment without an additional critic. Given two continuations from the same history with different outcomes, PEARL locates their first divergence and builds a feature representation of each branch's local decision. It clusters these decision features in recent failure and success memories to identify recurring local patterns. Using this pattern memory, PEARL selectively strengthens negative advantages at lower-reward decisions, while leaving positive advantages and the base policy optimizer unchanged. On AppWorld, PEARL improves over ARPO on Normal and Challenge tasks at both 4B and 8B scales. On ALFWorld, it achieves 97.9% mean success at 4B, 2.7 points above the strongest baseline, while remaining competitive at 8B. Ablations of shaping location, success-bank filtering, and memory window support the value of selective, recurring-pattern-based updates.