Large language models share context to speed up inference by 6 to 21 times
ACCT: Asymmetric Cross-LLM Context Transfer
Artificial Intelligence
Summary
Computers that understand and generate language often use big models that take time to process information. The authors found a way to let one language model share its remembered information with another, so the second doesn’t have to start from scratch. This shared memory is smaller and faster to use but still keeps accuracy. Their method works even when shrinking memory by most of its size and helps different-sized models work together more efficiently.
What this means in practice
- •For ai engineering teams: Reduce latency and computational cost by transferring context memory across different model sizes during dynamic model invocation.
- •For cloud service operators: Speed up multi-model language inference workflows by reusing processed context to save resources without losing performance.
Authors
Yuzhe Liu, Haolin Tian, Tonghan Wang
Abstract
In real-world deployments, large language model (LLM) systems are increasingly evolving from single-model invocation toward heterogeneous deployments in which different models are dynamically invoked for inference. When the same context is reused across models, however, each newly selected target model typically has to process it again, repeatedly incurring substantial computational costs. Therefore, we propose ACCT (Asymmetric Cross-LLM Context Transfer), which targets KV cache transfer between models of different scales within the same model family. ACCT transforms the KV cache produced by a source model into sparse KV states that can be directly consumed by a target model, thereby enabling cross-model reuse of context computation and decoupling context computation from downstream inference. ACCT consists of three lightweight components: the Cartographer identifies query-relevant context tokens for each target layer and KV head; the Allocator determines how many context tokens each target layer and KV head retains under a shared total KV budget; and the Compiler transforms selected source KV into sparse target KV. The target model can therefore perform inference using the transferred KV without reprocessing the original context. We evaluate ACCT in two complementary settings comprising five general-capability benchmarks and six highly context-dependent tasks. Experiments show that ACCT matches or exceeds the target model's original inference performance even when reducing the KV cache size by 75%. At a context length of approximately 19K tokens, ACCT's KV conversion achieves a 6.5-21.2 times speedup over target context prefill. These results demonstrate the feasibility of transferring context computation across models under a limited KV budget.