Bridge improves retrieval and language model cooperation for better question answering
BRIDGE: Bilevel Retrieval-Credit-Aware Agentic Reinforcement Learning
Artificial IntelligenceMachine Learning
Summary
Answering tough questions often means a computer needs to search for information and then think carefully about it. The authors found that training the search part and the thinking part together works better if the search part is improved first. They created BRIDGE, a method that helps the search and thinking parts learn from each other more efficiently. This method made computers better at answering complicated questions across many tests, especially when the answers need multiple steps or when dealing with medical questions.
What this means in practice
- •For qa system builders: Create question answering systems that better combine searching documents and reasoning to improve answer accuracy on complex queries.
- •For medical ai developers: Develop medical question answering tools with enhanced accuracy and reasoning quality by jointly optimizing retrieval and language reasoning.
Authors
Quan Xiao, Mingda Liu, Gaowen Liu, Katsuki Fujisawa, Tianyi Chen
Abstract
Agentic reinforcement learning (ARL) with verifiable rewards improves the ability of large language models (LLMs) to tackle knowledge-intensive tasks by learning to interleave search and reasoning. However, most existing ARL methods optimize only LLM-generated tokens and treat retrieved evidence as environment observations. This creates an information-credit gap: failures caused by missing or misleading evidence are attributed to the LLM policy rather than to the retriever, which motivates training the LLM and the retriever jointly. In this paper, we show that retrieval and LLM policy learning are order-sensitive: adapting the retriever before optimizing the policy yields a larger reward gain than the reverse order. To preserve this hierarchy while allowing both components to co-adapt, we formulate retrieval-augmented agentic RL as a bilevel optimization problem. To solve it efficiently, we introduce BRIDGE, a memory-efficient first-order bilevel method motivated by a loss-landscape analysis of the RL and retrieval objectives. Across seven open-domain QA benchmarks, BRIDGE achieves the highest average accuracy with both 3B and 7B backbones, improving the multi-hop average over the strongest baseline by 9.6 and 3.4 EM points, respectively. It also achieves the best averaged answer accuracy and reasoning quality across medical QA benchmarks.