Compressing long texts for faster large language model answers
Compressing Long Context into Answer-Aligned Memory Embeddings for LLM Inference
Computation and LanguageArtificial Intelligence
Summary
Large language models take more time, energy, and memory when reading very long texts. The authors created a new way to shrink these long texts into shorter memory summaries that still keep important information for answering questions. Their method works without changing the model that gives answers, and it smartly picks what parts of the memory to use based on the question. Tests show this approach makes answering faster and uses less memory while improving accuracy on several question-answer tasks.
What this means in practice
- •For natural language processing engineers: Reduce computation and memory needs during large language model inference over long documents while improving question answering accuracy.
- •For cloud ai service providers: Lower energy costs and latency when serving client queries on large text inputs by using compressed context memory.
Authors
Md Mostafizer Rahman, Md Faizul Ibne Amin, Md Shahajada Mia, Yutaka Watanobe, Fang Liu
Abstract
Large language model (LLM) inference is constrained by the quadratic scaling of self-attention and the linear scaling of the KV cache, increasing latency, energy consumption, and GPU memory demand as context length scales. Existing soft-compression methods either lack query-guided memory selection at inference time, train without answer-targeted supervision, or couple compression tightly to a specific decoder architecture. We propose a Context-to-Answer-Aligned Memory Compression (CMC) framework, which compresses long input contexts into compact Context Memory Embeddings (CMEs) aligned to any frozen decoder's embedding space, reducing inference costs without modifying decoder weights. CMC introduces a two-tier KV cache that combines question-guided CME selection with a local context window, and trains the compressor with answer-targeted distillation from a frozen LLM. Experiments across nine encoder-decoder combinations and four QA benchmarks show that CMC consistently outperforms the baseline, achieving up to 7.3 EM and 4.0 F1 point gains on SQuAD, while reducing inference time and energy consumption by up to 20% and peak reserved GPU memory by up to 50% at 3,000 generation tokens. Ablation studies confirm that each architectural component and training objective contributes to the performance.