Beyond Relevance: Utility-Centric Retrieval in the LLM Era
2026-04-10 • Information Retrieval
Information RetrievalArtificial IntelligenceComputation and LanguageMachine Learning
AI summaryⓘ
The authors explain that traditional search systems focus on finding documents that match a user's query, called topical relevance. However, with new tools like retrieval-augmented generation (RAG), the goal shifts to helping language models use these documents to produce better answers. Instead of just ranking documents by how relevant they are, the authors suggest measuring how much they improve the language model’s output. They offer a framework to understand this shift and guide designing systems that work well with large language models.
Information RetrievalTopical RelevanceUtilityRetrieval-Augmented GenerationLarge Language ModelsRanking MetricsContext-Dependent UtilityAgentic RAGGeneration Quality
Authors
Hengran Zhang, Minghao Tang, Keping Bi, Jiafeng Guo
Abstract
Information retrieval systems have traditionally optimized for topical relevance-the degree to which retrieved documents match a query. However, relevance only approximates a deeper goal: utility, namely, whether retrieved information helps accomplish a user's underlying task. The emergence of retrieval-augmented generation (RAG) fundamentally changes this paradigm. Retrieved documents are no longer consumed directly by users but instead serve as evidence for large language models (LLMs) that produce answers. As a result, retrieval effectiveness must be evaluated by its contribution to generation quality rather than by relevance-based ranking metrics alone. This tutorial argues that retrieval objectives are evolving from relevance-centric optimization toward LLM-centric utility. We present a unified framework covering LLM-agnostic versus LLM-specific utility, context-independent versus context-dependent utility, and the connection with LLM information needs and agentic RAG. By synthesizing recent advances, the tutorial provides conceptual foundations and practical guidance for designing retrieval systems aligned with the requirements of LLM-based information access.