Efficient key-value cache transfer speeds up communication between large language models
Prefill-Free Cross-Family KV Cache Transfer for Heterogeneous Multi-Agent LLMs
Artificial IntelligenceMachine LearningMultiagent Systems
Summary
When different large language models talk to each other, they often have to repeat work that was already done, which wastes time. The authors propose a method called HeteroFold that lets these models share memory directly, even if they are built differently, without reprocessing previous conversation. This makes the communication much faster and works well across several tests involving long conversations. Their method improves speed and efficiency compared to existing techniques, helping systems with multiple specialized AI agents work better together.
What this means in practice
- •For ai system engineers: Speed up communication between different large language models by reusing cached computation without repeating previous context processing.
- •For cloud service operators: Reduce computation time and resources when running multi-agent AI systems that combine different language models.
Authors
Vincent-Daniel Yun, Woosang Lim, Haneul Yoo, Sungjoo Yoo, Sai Praneeth Karimireddy, Murali Annavaram
Abstract
Recent multi-agent LLM systems increasingly combine heterogeneous models for specialized agent roles. However, text-based communication requires each receiver to prefill shared context already processed by the sender. Reusing the sender's key-value (KV) cache avoids this redundancy, but prefill-free transfer across model families must handle differences in tokenization, model depth, and KV representations. To address these issues, we propose \textit{HeteroFold}, a prefill-free cross-family KV cache transfer method that keeps both the sender and receiver frozen. HeteroFold aligns model structures, maps the sender cache into the receiver space, and calibrates it to preserve receiver behavior. Across six transfer directions, HeteroFold achieves the best cache-transfer performance on all four long-context benchmarks and most short-context settings. It also matches text-based communication on the multi-agent benchmark. At 32K context length, Llama-3.1-8B$\rightarrow$Ministral-3-14B transfer is $10.7\times$ faster than Native Prefill and $1.18$--$1.47\times$ faster than the state-of-the-art prefill-free baselines, Dense Latent and KV Ridge. These results show that HeteroFold enables efficient cross-family KV reuse without receiver prefill.