Attention guides faster multi-hop retrieval across linked documents
Read the Route: Attention-Steered Hypergraph Retrieval for Multi-hop RAG
Information Retrieval
Summary
Answering questions that need multiple steps often means searching through a network of connected information. The researchers showed that a special part of a language model’s 'attention' can figure out which parts of that network to look at next, without generating extra text each time. They built a system called HyperAddress that uses this insight to search more accurately and quickly through information networks. This method works better and faster than recent alternatives for answering complex questions that need multiple pieces of information.
What this means in practice
- •For search engine engineers: Improve multi-step question answering by guiding document retrieval using detected attention patterns, reducing processing time and enhancing accuracy.
- •For automated customer support teams: Enable faster and more precise retrieval of relevant information from complex knowledge bases to answer multi-part customer inquiries without costly inference.
Authors
Jiate Liu, Liuyi Chen, Xiaofei Yuan, Yifu Tang, Zihan Yang, Xu Zhou, Zhengyi Yang
Abstract
Graph-based retrieval-augmented generation (GraphRAG) answers multi-hop questions by organizing a corpus into a graph and retrieving along its structure. Existing retrieval methods face two limitations. Diffusion- and similarity-based methods expand retrieval along fixed edge weights or by similarity to the question, without resolving its implicit entity references, allowing relevance errors to accumulate across hops. In contrast, exploration-based methods let an LLM reason about the next hop, but require a separate generation at each hop, leading to high inference costs. We observe that an LLM can resolve implicit entity references from retrieved passages and a multi-hop question before generating any tokens. Further analysis identifies a small set of attention heads, termed addressing heads, that capture this capability across multiple hops in a single forward pass. Building on this finding, we propose HyperAddress, which runs Personalized PageRank on a description-labelled hypergraph, using addressing heads to determine the restart distribution and transition weights, thereby guiding multi-hop retrieval without any generation at query time. On MuSiQue, 2WikiMultiHopQA, and HotpotQA, HyperAddress outperforms the strongest baseline by 3.5 points of Recall@5 and 7.4 points of All@5 on average, with gains that grow with the number of hops. It answers more than twice as fast and builds its index at half the cost.