UnifiedPlayers improve reasoning in AI agents using tools
UnifiedPlayers: Enhance Tool-Integrated Reasoning in Agentic Reinforcement Learning
Artificial Intelligence
Summary
AI agents that use tools to solve problems need good training data, but getting humans to label this data is hard. The paper introduces UnifiedPlayers, a team of three cooperating AI components that plan tasks, execute them using Python tools, and evaluate the results together. This cooperation helps the AI learn better by constantly improving task plans, executions, and checks. Tests show this method enhances math and general reasoning skills and detects errors more accurately than previous methods.
What this means in practice
- •For software developers: Improve AI assistants that use programming tools by enabling them to generate and verify their own training tasks and solutions efficiently.
- •For automation engineers: Develop better adaptive automation systems that refine their reasoning and error checking collaboratively while performing complex scripted tasks.
Authors
Wenjie Liao, Liangjie Zhao, Zehong Cao
Abstract
Self-evolving methods reduce the need for human-annotated trajectories by allowing tool-using agents to generate their own training data. Yet existing methods typically separate trajectory generation from evaluation, relying on static verifiers that cannot adapt to emerging failure modes or self-consistency signals that may reinforce errors shared across trajectories. Jointly adapting planning, execution, and evaluation offers a promising alternative, but introduces a fundamental coordination challenge: each component continuously changes the data or feedback used to train the others. We address this challenge with \textbf{UnifiedPlayers}, a cooperative framework comprising a Planning Player that generates tasks, an Execution Player that produces multi-turn trajectories with Python tool calls, and an Evaluation Player that constructs executable verifiers. We design role-specific rewards that coordinate the three players toward a shared learning objective under GRPO. Across two model backbones and twelve reasoning benchmarks, UnifiedPlayers outperforms the strongest prior baseline by at least 3.5\% on mathematical reasoning and 3.9\% on general reasoning tasks. Moreover, the learned verifier achieves 84.2\% adversarial detection accuracy, while its reward signal exhibits 2.03$\times$ higher per-question variance than a self-consistency baseline, providing more discriminative verifications. These results highlight cooperation among specialized players as a promising path toward self-enhanced tool-integrated agents.