Evidence guided credit assignment improves deep search agent training
EG-GRPO: Evidence-Guided Step-level Credit Assignment for Deep Search Agents
Machine LearningArtificial Intelligence
Summary
Training AI agents to search and reason over long tasks is hard because they usually only get feedback at the end, which doesn’t help with learning each step well. The authors created a new method called EG-GRPO that uses detailed evidence from synthetic data to give AI step-by-step guidance. This helps the AI learn better which actions contribute to good outcomes by tracking and rewarding useful evidence as it accumulates. Tests on complex search tasks show that this approach generally leads to better performance than previous methods.
What this means in practice
- •For software developers: Improve AI-powered search tools that handle complex, multi-step queries by training agents with fine-grained evidence-based feedback.
- •For knowledge management teams: Enhance systems that automate research or information gathering by better guiding AI agents through supporting evidence at each reasoning step.
Authors
Sijia Luo, Mengjie Gao, Yuxiao He, Mengshu Sun, Cuiping Li, Hong Chen, Jing Zhang
Abstract
LLM-based search agents demonstrate strong capabilities in long-horizon information seeking and reasoning, typically trained via reinforcement learning. However, existing approaches predominantly rely on trajectory-level outcome rewards, providing limited supervision for complex search and reasoning processes. Although step-level credit assignment offers a promising solution, obtaining reliable and task-grounded supervision for open-ended search remains challenging. In this paper, we propose Evidence-Guided Group Relative Policy Optimization (EG-GRPO), a framework that derives step-level supervision from the evidence annotations available in synthetic deep search data. EG-GRPO identifies the first supporting citation for each required evidence claim and uses the newly acquired evidence to construct step-level process rewards. By representing the accumulated evidence as an evolving knowledge state, EG-GRPO further performs state-conditioned relative advantage estimation and combines the resulting step-level signal with the standard trajectory-level outcome advantage. Experiments on five challenging deep search benchmarks show that EG-GRPO generally outperforms standard GRPO and existing reward-enhanced and step-level RL baselines across different model scales. These results demonstrate that task-grounded evidence annotations can provide effective process supervision for credit assignment in deep search RL.