Reinforcement learning improves search agents by tracking information sources
SOAR: Source-Aware Reinforcement Learning with Precise Credit Assignment for Search Agents
Computation and Language
Summary
Sometimes, computer programs that search for answers make up information instead of using real facts, leading to wrong results. This happens because these programs rely too much on their own guesses, which are hard to track. The authors created a method called SOAR that keeps track of where information comes from, separating facts found externally from what the program already knows. This helps the program learn better by rewarding it for using trustworthy sources, reducing made-up evidence by a lot and improving accuracy.
What this means in practice
- •For software developers: Enhance information retrieval systems to reliably distinguish and trust verified external data over internal guesses, improving answer accuracy.
- •For ai product teams: Build more truthful AI assistants that reduce fabricating false information by explicitly tracking and rewarding correct source usage.$Commercial implications: Enables development of AI products with stronger factual reliability, increasing user trust and market value.
Authors
Shengze Huang, Zihao Tang, Jinluan Yang, Xin Yu, Zhenghao Lin, Xiao Liu, Zengxuan Wen, Yeyun Gong, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang, Fei Wu, Kun Kuang
Abstract
Reinforcement learning (RL) for search agents often suffers from evidence fabrication, a failure mode where agents bypass external information ($E_{ext}$) and fabricate an incorrect answer. We argue that this fabrication arises from an over-reliance on unreliable internal knowledge ($E_{int}$) as shortcuts due to the inherent opacity of reasoning trajectories. To address this, we propose SOAR (SOurce-Aware Reinforcement learning), a paradigm that transforms monolithic reasoning into source-explicit trajectories. SOAR (i) disentangles $E_{ext}$ and $E_{int}$ into parallel, observable streams via a Source-Aware Workflow, and (ii) incorporates Source-Consistent Reward to guide the policy evolution. By making information-source attribution explicit, SOAR enables precise credit assignment between different knowledge streams. Evaluations show that SOAR outperforms baselines on 3B and 7B backbones by 7.3% and 5.4%, while reducing evidence fabrication by 74.8%. These confirm that SOAR significantly improves performance on knowledge-intensive benchmarks while effectively mitigating evidence fabrication, fostering more faithful and grounded reasoning with enhanced robustness.