Unified model improves evidence selection for long text and retrieval
UNREAL: Unifying Retrieval and Long-Context with a Single Model
Computation and LanguageInformation RetrievalMachine Learning
Summary
Finding relevant information for tasks can involve either searching a large collection of documents or examining very long pieces of text. The authors introduce a single approach called UNREAL that can pick out important evidence both from vast collections and from long text passages using one model. This method adds only a small number of trainable parameters and works inside a large language model without changing its core. Tests show that UNREAL finds more useful information and helps answer questions better than previous methods, while also speeding up processing for very long inputs.
What this means in practice
- •For natural language developers: Build question answering systems that more effectively retrieve and use relevant evidence from large text collections using a single model component.
- •For long-document analysis teams: Speed up and improve accuracy of inference over very long text by removing irrelevant information before generating responses.
Authors
Edan Kinderman, Elad Hoffer, Yochai Blau, Brian Chmiel, Ron Banner, Daniel Soudry, Boris Ginsburg
Abstract
Long-context inference and Retrieval-Augmented Generation (RAG) handle evidence selection at vastly different scales, from a single long prompt to an entire corpus. We ask whether a single model-internal mechanism can select evidence across this range. We introduce UNifying REtrieval And Long-Context with a Single Model (UNREAL), a model-native evidence selection framework to span corpus retrieval and long-context inference. UNREAL encodes chunks and derives retrieval queries directly from the frozen LLM's internal representations. It adds fewer than 500K trainable parameters and leaves the backbone unchanged. On a 3B-token, 21M-chunk Wikipedia index, all four dense and hybrid UNREAL backbones outperform state-of-the-art retriever-reranker systems. The best model raises recall from 49.1% to 73.2% on HotpotQA, from 31.7% to 60.1% on 2WikiMultiHopQA, and from 8.8% to 14.4% on MuSiQue. Applied to long-context tasks, the same selection mechanism removes distractors before generation, raising NoLiMa accuracy from 1.0% to 24.83% at its maximum context length of 128K tokens, and LV-Eval's F1 score from 49.97% to 54.66% at 256K. UNREAL also reduces FLOPs and time-to-first-token relative to full-context inference from roughly 32K tokens onward, with larger gains as context grows. Together, these results establish model-internal evidence selection as a common foundation for corpus retrieval and evidence-sparse long-context inference.