Large language model ranking speeds up search with gist summaries
GistRank: Gist-Compressed Listwise Ranking with Large Language Models
Information Retrieval
Summary
Finding the most relevant information in a long list takes time because computers need to carefully read everything. The authors found that making short, meaningful summaries called gists for each piece of text can speed up this ranking process a lot. They created a new method called GistRank that trains a language model to make these summaries better at telling which bits are more important. This approach makes search results nearly as accurate but much faster, making it useful for real-world searches.
What this means in practice
- •For search engine engineers: Accelerate ranking of retrieved documents by compressing them into gist tokens without much loss in accuracy.
- •For web service platform teams: Deploy faster listwise ranking models for large-scale web search to improve response speed while maintaining result quality.
Authors
Guo Chen, Wenhan Liu, Reed Li, Qianwen Ma, Yu Lu, Haibo Shi, Bingbing Xu, Junjie Huang
Abstract
Large language models (LLMs) have become a powerful tool for listwise relevance ranking, which can improve web search by comparing retrieved passages jointly. However, processing every candidate verbatim and generating a permutation autoregressively can make inference costly, every candidate must be serialized verbatim into the prompt and the input therefore grows linearly with both the number and the length of the passages. While sliding-window ranking handles context limits through repeated re-encoding of overlapping blocks, hard compression techniques sacrifice the fine-grained details required for accurate discriminative judgment. To better understand and mitigate this trade-off, we conduct a comprehensive study of soft compression for LLM-based listwise ranking. Our findings reveal that a vanilla gist representation is ill-suited to ranking for two reasons: strictly causal encoding weakens how thoroughly each gist token absorbs its passage, and the representation is never trained to expose the relative-preference structure that separates closely competing candidates. To this end, we propose GistRank, a three-stage training framework that equips a large language model with gist-compressed passage representations by learning faithful gist tokens under a semi-bidirectional attention scheme, instilling discriminative comparison through pairwise training, and adapting the model to the listwise objective. GistRank collapses each passage into only $K{=}4$ gist tokens, this gives 14$\times$ speedups over one-shot full-text ranking while reaching 59.17 average nDCG@10 over TREC DL19/20 and five BEIR subsets within 0.89 of the strongest full-context baseline. Overall, GistRank provides an effective means to enhance efficiency for searches in real-world environments.