FuseRAG improves multi-hop question answering with faster graph-based retrieval
FuseRAG: Fusing Structure and Evidence into Inference for Cost-Efficient Graph-based RAG
Databases
Summary
Answering complex questions often means looking through connected pieces of information, which can be tricky and slow when telling a computer about their relationships. The authors of this paper created FuseRAG, a new way to organize these pieces inside a model so the computer understands connections without needing to repeat all details in text form. This helps the model reuse information more efficiently and keeps the structure of the knowledge intact. Tests show FuseRAG answers questions better and faster than earlier systems that try similar tasks.
What this means in practice
- •For ai developers: Build more efficient question-answering systems that handle complex related evidence faster and more accurately by integrating FuseRAG's graph-based inference method.
- •For knowledge management teams: Improve enterprise search systems to better connect and infer answers across multiple related documents using fused latent and explicit evidence representations.
Authors
Tingqi Wang, Yingshu Li, Zhipeng Cai
Abstract
Graph-based retrieval-augmented generation (RAG) exploits relationships among evidence to support multi-hop question answering. However, conveying retrieved graphs through non-unique linear textual contexts to Large Language Models (LLMs) leads to graph-structure degradation and inefficient KV-cache reuse. Meanwhile, graph-structure order optimization is already NP-hard, and the additional KV-cache reuse objective further complicates the search by introducing potentially conflicting orders. To tackle these issues, we propose FuseRAG, a cost-efficient graph-based RAG framework that shifts most retrieved evidence from explicit textual contexts to latent representations. Our main idea is to separate reusable evidence content from query-aware structural interactions in latent space, while retaining only a few explicit facts as semantic anchors. The resulting latent representations are injected into decoder self-attention, where they reshape attention over evidence and thereby influence hidden states and next-token probabilities. This design preserves graph structure through latent interactions, enables evidence reuse beyond exact token-prefix matching. Experiments on three multi-hop question-answering datasets with Llama-3-8B and Qwen3.5-9B show that FuseRAG improves answer quality while achieving up to 2.7x and 2.2x TTFT speedups over the fastest graph-based baseline, respectively.