RANGER improves graph navigation with language model guided rewards
RANGER: Retrieval Agent for Navigating Graphs with Semantic Evaluation Rewards
Machine Learning
Summary
Finding answers often means jumping across many linked facts, like following a path in a big web of information. The authors created a system called RANGER that combines smart language tools with learning agents to better explore this web. The language model guides the agent by giving rewards based on how meaningful its chosen paths are, helping it learn which steps lead to good answers. This approach helps their system find information more accurately than older methods that didn’t use this kind of semantic guidance.
What this means in practice
- •For software engineers: Develop knowledge-based systems that more effectively find complex multi-step answers inside linked data structures.
- •For digital assistants developers: Improve virtual assistants’ ability to retrieve accurate answers by training agents that follow semantically guided paths in knowledge graphs.$Commercial implications: Enables more accurate retrieval in assistants, improving user experience and satisfaction through better information navigation.
Authors
Artem Mikhaylov, Pavel Konoplev, Nikolay Efanov, Artyom Sorokin, Evgeny Burnaev
Abstract
The integration of Large Language Models and Reinforcement Learning offers a promising direction for complex sequential decision-making tasks, such as multi-hop question answering over knowledge graphs. While LLMs provide strong semantic understanding and reasoning capabilities, they often struggle with precise, multi-step navigation in structured environments. Conversely, RL agents excel at learning navigation policies but are hindered by the difficulty of designing semantically meaningful reward functions. This paper introduces a novel hybrid approach in which an LLM serves as a mentor to an RL agent during training by providing dense, semantically informed reward signals. The RL agent, built on a Transformer-based policy and trained with Group Relative Policy Optimization, learns to navigate a knowledge graph to retrieve relevant information. The LLM evaluates the semantic relevance of the agent's traversed paths, generating reward signals that align the agent's behavior with the information needs expressed in a query. We integrate this Retrieval Agent for Navigating Graphs with Semantic Evaluation Rewards (RANGER) into a complete question-answering pipeline and evaluate it on the open question answering datasets. Experimental results show that the proposed approach outperforms both a baseline retrieval-augmented system with classical graph search and one using an RL agent trained with heuristic rewards, achieving a significant increase in NLP metrics and LLM-as-a-Judge scores. Our findings demonstrate that deep integration of LLMs into the RL training loop yields more effective and semantically aligned retrieval policies for complex reasoning tasks.