EvoRouter cuts compute costs while improving multi-agent web routing accuracy
EvoRouter: Cost-Efficient Multi-Agent Routing via Insight-Guided On-Policy Distillation
Artificial IntelligenceMultiagent Systems
Summary
Handling many web queries with multiple AI agents is complicated because many settings affect performance and cost. The authors created EvoRouter, a system that learns to find the best settings by collecting insights from its own past decisions and using those insights to improve continuously. It balances how well tasks get done with how much computing power it uses. Evaluated on many datasets, EvoRouter performed better than other methods while using about half the computing resources.
What this means in practice
- •For cloud service operators: Optimize multi-agent query routing to improve accuracy while halving computational costs in large-scale web services.
- •For ai system engineers: Design multi-agent AI deployments that self-improve routing decisions by integrating execution insights and feedback for balanced cost and performance.
Authors
Yitao Xiao, Xu Zhang, Ye Cheng, Shaoyong Guo, Yan Sun, Ziyue Zhu, Sitao Xie, Guo Tong, Haoran Luo
Abstract
Serving web queries at scale with multi-agent systems requires jointly configuring large language models, collaboration modes, agent counts, interaction rounds, and skills. However, the resulting configuration space grows combinatorially, while sparse execution rewards provide limited fine-grained supervision for interdependent planning decisions. We propose EvoRouter, a self-evolving multi-agent routing framework driven by insight-guided on-policy distillation. A dedicated expert periodically synthesizes performance-cost feedback from configuration trajectories across tasks into reusable conditional routing insights about capability requirements, returns from resource scaling, and interactions among configuration choices. A privileged teacher combines these evolving insights with current execution feedback to provide dense token-level supervision for the router's own planning trajectories through on-policy distillation. Cost-aware reinforcement learning jointly optimizes task performance and computational cost. Together, these training signals internalize execution insights into the routing policy. The updated policy generates new trajectories that support subsequent insight extraction, forming an execution-driven self-improvement loop. We evaluate EvoRouter on 19 in-domain datasets spanning five domains and two out-of-domain benchmarks. EvoRouter achieves an average in-domain accuracy of 68.69%, outperforming the strongest routing baseline by 7.23 percentage points while reducing inference compute cost by 50.95%. Our code is open-sourced at https://github.com/CenYangtze/EvoRouter.