Papers for

long-document analysis teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Unified model improves evidence selection for long text and retrieval

UNREAL: Unifying Retrieval and Long-Context with a Single Model

Abstract: Long-context inference and Retrieval-Augmented Generation (RAG) handle evidence selection at vastly different scales, from a single long prompt to an entire corpus. We ask whether a single model-internal mechanism can select evidence across this range. We introduce UNifying REtrieval And Long-Context with a Single Model (UNREAL), a model-native evidence selection framework to span corpus retrieval and long-context inference. UNREAL encodes chunks and derives retrieval queries directly from the frozen LLM's internal representations. It adds fewer than 500K trainable parameters and leaves the backbone unchanged. On a 3B-token, 21M-chunk Wikipedia index, all four dense and hybrid UNREAL backbones outperform state-of-the-art retriever-reranker systems. The best model raises recall from 49.1% to 73.2% on HotpotQA, from 31.7% to 60.1% on 2WikiMultiHopQA, and from 8.8% to 14.4% on MuSiQue. Applied to long-context tasks, the same selection mechanism removes distractors before generation, raising NoLiMa accuracy from 1.0% to 24.83% at its maximum context length of 128K tokens, and LV-Eval's F1 score from 49.97% to 54.66% at 256K. UNREAL also reduces FLOPs and time-to-first-token relative to full-context inference from roughly 32K tokens onward, with larger gains as context grows. Together, these results establish model-internal evidence selection as a common foundation for corpus retrieval and evidence-sparse long-context inference.

Tue 6 OctComputation and LanguageInformation RetrievalMachine Learning
The gist
Finding relevant information for tasks can involve either searching a large collection of documents or examining very long pieces of text. The authors introduce a single approach called UNREAL that can pick out important evidence both from vast collections and from long text passages using one model. This method adds only a small number of trainable parameters and works inside a large language model without changing its core. Tests show that UNREAL finds more useful information and helps answer questions better than previous methods, while also speeding up processing for very long inputs.
Open → 2610.08463v1