SALT: Salience-Aware Lexical Trie for Long-Context Compression
2026-07-20 • Performance
PerformanceArtificial IntelligenceMachine Learning
AI summaryⓘ
The authors address the problem of large language models slowing down when handling very long inputs due to memory and computation limits. They point out that existing methods, which rank sentences individually for importance, often miss less obvious but important themes. To fix this, the authors propose SALT, a method that groups keywords from sentences into a trie structure based on how often themes appear, ensuring diverse themes are kept. This approach helps balance memory use, speeds up processing, and works well across multiple dialogue turns without needing to reprocess the whole document each time.
large language modelsKV-cacheprompt compressionsentence frequencytheme collapsetrie data structureextractive summarizationmulti-turn dialoguememory optimizationquery-based retrieval
Authors
Oteo Mamo, Hyunjin Yi, Joydhriti Choudhury, Shangqian Gao, Weikuan Yu
Abstract
As large language models (LLMs) process increasingly longer prompts, computation and KV-cache memory costs have emerged as major bottlenecks in inference systems. Existing input-level prompt compression methods address this, but rank each sentence by a scalar relevance score, treating the document as an unstructured pool of words and sentences. Under tight budgets, this causes theme collapse, where the dominant theme(s) of a document consumes the budget, discarding less-frequent yet task-relevant themes. Preserving thematic coverage instead requires allocating the budget across recurring themes rather than scoring sentences in isolation. To this end, we propose SALT, a model-agnostic extractive framework that organizes per-sentence keywords into a trie ordered by sentence frequency (SF), a lightweight, reusable proxy for document thematic structure. This trie-based organization smooths memory allocation and prevents dominant themes from monopolizing the budget. Multi-anchor retrieval activates trie nodes labeled by query keywords at any depth, and the trie persists across dialogue turns, supporting multi-turn use without re-encoding the document. By preserving document themes, SALT reduces the prefill computation and memory cost of long-context prompts while remaining composable with KV-cache methods that target decoding-time latency and memory.