Recursive self-improving agent orchestration boosts complex task solving
R$^2$Orch: Recursive Self-Improvement for Agent Orchestration via Self-Play within a Skill-Internalization Loop
Artificial Intelligence
Summary
Solving complicated tasks with AI agents often requires guiding them carefully and dealing with unreliable signals to measure their skills. To improve this, the authors designed a system called R2Orch that teaches an AI orchestrator to get better by playing against itself on tasks it chooses. It gradually removes skills once they become unnecessary and learns from failures to improve further. This approach was tested on many types of tasks like answering questions and writing code, showing better results than older methods.
What this means in practice
- •For ai development teams: Build adaptive AI orchestrators that improve autonomously on complex tasks without manual curriculum design.
- •For interactive software developers: Create agents that self-optimize by withdrawing unnecessary skills and refining their task-solving strategies.
Authors
Mingda Zhang, Yun Huang, Qiang Huang, Yanjin Li, Rui Qian, Qichao Zhao, Tiesunlong Shen, Na Zhao, Jian Wang, Xiaoying Tang
Abstract
In recent years, agentic orchestration has been widely applied to solve complex human tasks. However, existing orchestration learning still faces key challenges, including hand-designed curricula, advantage collapse when a task's rollouts all reach the same outcome, and unreliable evaluation signals that judge each skill by one noisy score. To address these challenges, we propose R$^2$Orch, a new paradigm of Recursive Self-Play Orchestration: an inner loop of self-play that improves the orchestrator on self-chosen tasks, and an outer loop that improves how the inner loop improves. To support this paradigm, we introduce Recursive Learnability, a training objective that rewards a Proposer for choosing tasks whose rollouts keep reaching different outcomes as skills are withdrawn. Built on this objective, we further propose Skill Withdrawal, in which the outer loop withdraws each skill once a controlled evaluation within self-play shows that the orchestrator performs as well without it. Moreover, the two loops form one cycle: failures are distilled into skills, skills into parameters, and withdrawals lead the Proposer to new failures, so the orchestrator moves toward zero-skill deployment. Experimental results on twelve datasets show that R$^2$Orch consistently outperforms baselines across question answering, math, code, and interactive tasks.