A Universal Context-Reuse Layer for Cross-Model KV Sharing
2026-08-31 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors explore a way for different large language models (LLMs) to share parts of their memory (KV states) when they process the same text, even if the models differ in size or design. Normally, each model does its own work from scratch, but the authors show that reusing KV states across models can save time and still keep accuracy high. They tested this method on various pairs of models and found it reduces computing time while maintaining or improving performance. This suggests that parts of the models' memory can be transferred between different LLMs to work more efficiently.
large language modelsKV cachecross-model sharingprefill computationcontext lengthinference latencymodel architecturetokenizerperplexitymulti-agent workflows
Authors
Yi Li, Dongming Jiang, Yi Zhao, Bingzhe Li
Abstract
Modern large language model (LLM) serving systems increasingly operate over repeated or shared context, yet each model typically performs its own prefill computation even when another model has already processed the same input. Existing KV-cache reuse mechanisms substantially reduce redundant computation within a single model, but generally assume that the producer and consumer of a cache are identical. We study \emph{cross-model KV sharing}, which translates the KV state produced by a source model into a representation that can be consumed by a different target model, including models that differ in scale, architecture, attention configuration, tokenizer, and model family. We evaluate the approach in both within-family and cross-family settings. For Qwen2.5-7B $\rightarrow$ Qwen2.5-1.5B, translated KV states improve LongBench2 accuracy from 27.59\% to 34.48\%, a gain of 6.89 percentage points over the native 1.5B baseline, while reducing handoff cost relative to native target prefill. For the cross-family Qwen2.5-1.5B $\rightarrow$ Gemma-2-2B setting, KV handoff reduces target-side prefill cost by up to 67.05\% at 4K context length while maintaining decoding perplexity close to native-model baselines. In a more heterogeneous Llama3.1-70B $\rightarrow$ Qwen2.5-7B setting, cross-family handoff achieves 44.0\% accuracy compared with 45.7\% for native Qwen2.5-7B inference, while reducing measured latency from 899ms to 138ms. These results provide initial evidence that KV states can serve as transferable computational representations rather than strictly model-local caches, and motivate \emph{context mobility} as a systems abstraction for reducing redundant prefill across heterogeneous LLM and multi-agent inference workflows.