Semantic compression improves selection of important tokens in long texts
Selecting What Matters: Semantic Compression-Guided Selective Pooling for Long-Context Embeddings
Computation and Language
Summary
Long documents often have a lot of extra or less important information mixed in, which makes it harder for language models to understand the key points. The authors propose a method called SCSP that breaks documents into chunks, uses special prompts to identify the most important words, and then focuses on these words to create better summaries. This approach helps language models create more meaningful representations of long texts without needing extra training. The authors tested SCSP on multiple benchmarks and showed it improves performance when integrated with existing language models.
What this means in practice
- •For natural language processing engineers: Enhance long document embeddings by selecting semantically important tokens for better text understanding and retrieval.
- •For search engine developers: Improve document ranking by focusing on representative tokens identified through semantic compression, leading to more relevant search results.
Authors
Zifeng Cheng, Jie Zheng, Zhiwei Jiang, Shuwen Wang, Fei Shen, Shiping Ge, Qing Gu
Abstract
Large language models (LLMs) have shown strong potential as training-free text encoders for long-context embeddings. Existing approaches primarily improve information flow under causal attention and typically construct embeddings by uniformly averaging all token representations. However, for long documents, such mean pooling can dilute salient semantic information with abundant redundant or weakly informative content. To this end, we propose SCSP, a training-free framework that leverages semantic compression for informative token selection in long-context embedding. Specifically, SCSP first partitions a document into sentence-aware chunks and appends a semantic compression prompt to each chunk. A prompt-isolated attention mask preserves information flow among document tokens while restricting each prompt to its corresponding local context. We then use the attention patterns elicited by these prompts to estimate token importance, select informative tokens, and aggregate their intermediate-layer representations into the final embedding. Extensive experiments on long-context embedding benchmarks demonstrate that SCSP can be integrated into both zero-shot and fine-tuned models in a plug-and-play manner, consistently improving their performance.