Hierarchical ranking improves learning for open-ended ai agents
ArenaFlow: From Trajectory Ranking to Hierarchical Credit Propagation for Open-Ended Agent RL
Computation and Language
Summary
Teaching AI agents to perform open-ended tasks is hard because the rewards they get can be unclear or inconsistent. The authors propose ArenaFlow, which uses competitions between different attempts to find what works best and breaks down rewards to specific important steps and reusable skills. This approach helps the AI learn better by focusing on key actions and improving strategies it can reuse later. Experiments show that ArenaFlow helps AI agents learn more effectively in complex, open-ended environments.
What this means in practice
- •For software engineers: Develop AI agents that learn complex open-ended tasks by identifying crucial steps and reusable strategies from comparative performance data.
- •For game developers: Create game AI that improves through competition-based feedback by recognizing key actions and skills to adapt gameplay strategies effectively.
Authors
Qiang Zhang, Ruixue Ding, Fanrui Zhang, Xi Chen, Boli Chen, Shihang Wang, Yinfeng Huang, Yi Zheng, Pengjun Xie, Kaipeng Zhang, Jiawei Liu, Zheng-Jun Zha
Abstract
Reinforcement learning has substantially improved large language model (LLM) agents in verifiable domains, but remains difficult to apply to open-ended agent tasks, where solutions are diverse and reliable scalar rewards are hard to obtain. Recent pairwise evaluation methods alleviate reward discrimination collapse by replacing pointwise scoring with relative preferences. However, they still compress rich comparative feedback into a single trajectory-level reward, obscuring decisive intermediate steps and preventing successful behaviors from being consolidated into reusable skills. We propose ArenaFlow, a hierarchical credit propagation framework for open-ended agent reinforcement learning. ArenaFlow leverages tournament-based relative ranking to derive trajectory-level reward signals. Each comparison is further equipped with structured reflective evaluation, which reveals three types of supervision: pivotal success steps, reusable strategy skills, and usage attribution of retrieved skills. At the step level, ArenaFlow propagates trajectory-level advantages to high-confidence pivotal steps according to tournament survival depth, enabling more targeted optimization of local reasoning behaviors. At the skill level, ArenaFlow estimates skill utility from group-level usage attribution and maintains a global skill memory through utility-aware updating, pruning, and retrieval. The resulting high-utility skills further serve as policy priors for future exploration. Extensive experiments validate ArenaFlow's effectiveness on open-ended agent tasks.