ShiftRAG improves question answering by merging retrieval and generation
ShiftRAG: Bypassing the Textual Bottleneck via Decoupled Learning and Continuous Soft Tokens
Computation and LanguageArtificial Intelligence
Summary
Complex question answering systems often get confused because they only use text retrieved as input, missing other helpful signals. The authors found that mixing retrieval and generating answers inside one model creates conflicts and reduces quality. They designed ShiftRAG, which separates learning to find relevant information from generating answers, and connects these parts using special continuous tokens to share more precise clues. This approach helps the system use information better and gives more accurate answers without slowing it down.
What this means in practice
- •For customer support teams: Provide more accurate answers by integrating document retrieval clues with answer generation to reduce errors in automated support chatbots.
- •For legal document analysts: Enhance retrieval-based question answering systems to better separate and use relevant evidence for complex legal queries.
Authors
Youmin Ko, Jihong Jeong, Hyunjoon Kim
Abstract
Despite advances in retrieval-augmented generation (RAG), suppressing hallucinations during complex reasoning remains a persistent challenge. We identify that existing RAG pipelines suffer from a textual bottleneck: by passing only raw text to the generator, they discard discriminative signals of the retriever. Furthermore, recent attempts to unify retrieval and generation within a single LLM induce severe objective conflict and exacerbate the rank collapse of decoder-only architectures, degrading performance on both tasks. To address these limitations, we introduce ShiftRAG, a decoupled RAG framework that employs a single LLM backbone but isolates discriminative retrieval learning from autoregressive generation. To bridge these decoupled modes, we project retrieval alignments into continuous soft tokens, establishing a high-bandwidth interface that infuses the generator with retrieval-side signals. Concurrently, our soft orthogonality objective mitigates rank collapse, while layer-wise relevance dynamics preserve the fine-grained latent space required for highly discriminative evidence separation. Extensive evaluations demonstrate that ShiftRAG significantly improves retrieval and question answering performance over state-of-the-art baselines. ShiftRAG preserves foundational capabilities of the backbone while maintaining a favorable efficiency-performance trade-off on general text encoding and generative tasks.