Encoding cached within llms saves space and speeds up queries

Cache the Encoder Within:Compact, Reusable Memory across LLM Queries

Computation and Language

Summary

When asking a large language model (LLM) multiple questions about the same documents, the model often has to process the same information repeatedly, which wastes time and computing power. The authors introduce EncBank, a technique that stores a small, reusable memory of the first part of the model’s understanding of a document. This stored memory can be used later for new questions, making the process faster and using less storage space. They tested EncBank on different models and tasks, showing it keeps accuracy close to the original method while using less storage and speeding up some operations. However, how well it works depends on the tasks and how often the stored memory is reused.

What this means in practice

  • For ai infrastructure teams: Reduce GPU memory usage and speed up repeated LLM queries on shared documents by caching intermediate encoder outputs efficiently.
  • For cloud service operators: Offer more cost-effective LLM query services by storing compact reusable memory states to handle repeated user queries with less compute overhead.$Commercial implications: Enables scalable and cost-efficient deployment of multi-query LLM services by reducing persistent GPU storage needs without large accuracy loss.

Authors

Hanzuo Liu, Chunyu Liu, Chaofan Lin, Alex Lamb, Mingyu Gao

Abstract

Repeated queries over shared documents incur redundant encoding, while caching model states introduces persistent storage costs. Building on CoMem's intermediate-state interface, EncBank treats a pretrained LLM's lower layers as a reusable document encoder and compactly stores their outputs for an adapted upper-layer reader. A self-distilled suffix adapter is shared across storage precisions within each backbone, without quantization-specific retraining. Across five benchmark suites on three Qwen backbones spanning different sizes and full-attention and hybrid architectures, 4-bit storage keeps each reported benchmark aggregate within one score point of native-precision EncBank. In a fixed Qwen3-8B workload, it retains 28.1% of the native-precision persistent GPU store. Separate native-precision controls yield a 1.40x selected-pack prefill speedup over same-evidence, same-adapter text replay, at a 3.12-point RULER accuracy cost. A native-precision Qwen3.8-27B configuration also passes 70 of 89 Terminal-Bench 2.1 tasks. EncBank thus combines reusable computation with compact memory, while task fidelity and end-to-end benefits remain dependent on the workload, preparation costs, and reuse frequency.
View PDFOpen arXiv