Search agents improve query training with environment feedback
IGSD: Environment-Verified Hindsight Self-Distillation for Search Agents
Artificial IntelligenceComputation and Language
Summary
Training search agents to find better answers is hard because they learn from their own attempts without someone else guiding them. The authors propose a method where the agent checks how useful a suggested search query is by actually running it and seeing if it helps find better information. This helps the agent learn which queries lead to better results, making its training more reliable. Their method works well on several question-answering tasks without needing extra checks when using the agent.
What this means in practice
- •For search engine developers: Improve training of search query generation to produce more effective retrieval queries by verifying proposals with actual environment feedback during learning.
- •For ai-powered customer support teams: Enhance automated question answering agents to better understand and refine queries leading to more accurate and precise responses in real-time.
Authors
Angqing Jiang, Gaoming Zhang, Chaoqun Zhang, Jianchun Song, Liyuan Kong, Kena Qi, Wei Lin, Defu Lian
Abstract
On-policy self-distillation densifies agent training without external teachers: a policy conditioned on privileged hindsight provides step-level guidance for its own unprivileged rollouts. For search agents, however, hindsight can make the teacher prefer a query that does not improve retrieval from the student's state. Existing methods either distill this preference directly or filter it with model-internal scores, but neither strategy verifies the query's executed retrieval consequence. We propose Information-Gain-Gated Self-Distillation (IGSD), which verifies on-policy token proposals with environment feedback before distilling them. Treating each query token as a micro-action, IGSD completes the teacher's token proposal and the student's sampled token into matched queries and executes both from the same failed state with the same retriever. Shared counterfactual controls account for query-conditioned shifts in answer likelihood, so their difference, the executed paired information gain, provides a relative utility contrast for the retrieved documents. IGSD uses this contrast as a positive-only soft weight for candidate-pair distillation, while leaving the GRPO objective unchanged and confining verification to training. Across seven single-hop and multi-hop QA benchmarks, IGSD reaches macro-average exact-match accuracies of 42.8% and 47.0% with 3B and 7B policies, respectively, without inference-time verification. These results support environment-verified hindsight as an effective approach to reliable action-level supervision for search agents.