Search agents learn to find better answers with fewer steps
Heterogeneous Co-Evolution Learning for Tool-Efficient Search Agents with Zero Labels
Machine LearningArtificial IntelligenceInformation Retrieval
Summary
Search agents, which help answer questions by looking up information, usually need lots of labeled examples to learn. The authors introduce a method that allows these agents to improve themselves without any labeled training data. Their system creates new questions adapted to the agent's current skills and encourages it to find correct answers efficiently, avoiding wasted searches. Tests show the agents get better at answering and use tools more efficiently across several question-answering tasks.
What this means in practice
- •For chatbot developers: Train chatbots to retrieve and verify information efficiently without needing manually labeled data.
- •For digital assistant teams: Improve assistants to answer complex questions by dynamically generating challenging training tasks that match their skill level.
Authors
Qin-Wen Luo, Sheng Ren, Xiang Chen, Rui Liu, Jun Fang, Naiqiang Tan, Jun Wang, Hua Chai, Sheng-Jun Huang
Abstract
Deep search agents extend language models by interleaving reasoning with external retrieval. However, effective search-agent training requires addressing two challenges beyond final-answer optimization. First, the capability frontier of agents is largely determined by training questions, whereas fixed or labeled task collections cannot adapt as agents improve. Second, outcome-only reinforcement learning treats all successful trajectories equally, even when some correct answers rely on redundant searches, creating an inherent tension between answer accuracy and tool efficiency. To address these issues, we introduce \textsc{ZEST-Search}, a zero-label and tool-efficient framework for search-agent training via heterogeneous co-evolution. Starting from unlabeled reference questions, \textsc{ZEST-Search} constructs validated search tasks online through an answer-first and evidence-path-guided generation process. A heterogeneous proposer adaptively constructs training tasks around the solver's evolving capability boundary, while the solver learns search-augmented reasoning through reinforcement learning. To jointly optimize task difficulty and search efficiency, \textsc{ZEST-Search} employs an intermediate-difficulty reward for the proposer and a task-local tool-aware reward for the solver, encouraging informative tasks and correct answers with fewer redundant searches. Experiments on seven general-domain and multi-hop question answering benchmarks demonstrate that \textsc{ZEST-Search} consistently improves search-agent performance while achieving higher tool efficiency without labeled search-training data or expert search trajectories. Further analyses show that heterogeneous generation, adaptive difficulty control, and tool-aware optimization enable more effective search-agent training.