Latent communication compresses context for faster large language model queries
LatComm: Reusable Context Compression for Latent Communication between Large Language Models
Computation and LanguageArtificial Intelligence
Summary
When large language models answer many questions using the same background information, they usually have to read all that information every time, which is slow. The authors created LatComm, a new way to squash that background down into a small, reusable package that another model can quickly read for each question. LatComm lets the models share context efficiently without repeating the work, making answering questions faster and more accurate on several tests.
What this means in practice
- •For llm system developers: Send compressed context packets between models to reduce repeated input processing and speed up multi-question workflows.
- •For knowledge base managers: Use LatComm to efficiently answer multiple questions from long documents without reprocessing the full text each time.
Authors
Huawei Lin, Jinting Liu, Tan Yu, Weijie Zhao
Abstract
Systems that combine several large language models (LLMs) often use one context for many questions. When the context is passed between models as text, the receiving model processes all of its tokens again for every question. Avoiding this cost requires a context representation that is compact, reusable across questions, and readable by a different LLM. Existing compression and latent communication methods each provide only some of these properties. We introduce LatComm, a Latent Communication interface that provides all three by separating encoding from reading: the context is encoded once without the question, and the receiver selects what to read for each question. A frozen context encoder LLM compresses the context into a fixed-size packet and sends it to a different frozen receiver LLM, which reads the packet through cross-attention, with gates computed from its prompt. On 12 multiple-choice question-answering datasets, LatComm improves average accuracy over single-vector baselines by up to 10.2 points. On QuALITY, it reduces the receiver's input by 57x and answers the questions of an article 3x faster.