Kv cache translation speeds up language model switching dramatically
KV-Lingo: Learning KV-Cache Translators with Distillation
Computation and LanguageMachine Learning
Summary
Large language models remember text context using what’s called a key-value cache. Different models can’t understand each other’s caches, so switching between models means reprocessing the same text again, which is slow. The authors propose KV-Lingo, a way to translate a cache from one model into another using simple math, so the new model can use the old cache directly. This makes switching models much faster without losing much accuracy in understanding the text.
What this means in practice
- •For language model engineers: Make model switching faster by using cache translation instead of full re-prefill, improving latency for applications needing multiple model architectures.
- •For cloud ai service teams: Reduce compute cost and delay when routing text contexts dynamically between different models, enabling more efficient multi-model inference pipelines.
Authors
Valérie Castin, Keitaro Sakamoto, Anastasiia Filippova, João Monteiro, Marco Cuturi, Pierre Ablin
Abstract
Large language models represent context with a key-value (KV) cache. Caches are model-specific: for the same text, models with different architectures or weights produce incompatible representations. This makes it costly to switch models over a shared context: although the context has already been processed by one model, the incoming model must process it again to build its own cache. We introduce KV-Lingo, a method for translating the KV cache of a source model into one that can be read by a target model. KV-Lingo consists of a collection of linear maps, typically one per layer of the target model, that are applied independently on all tokens' key and value representations. We train these maps using distillation, minimising the divergence between the target model's predictions from its native cache and those from the translated cache. We consider several model pairs spanning multiple sizes and architectures, training one translator per pair on a generic text corpus. The resulting translators preserve strong downstream performance in both small-to-large and large-to-small transfers. Since a switch then costs a linear map and a single decoding step instead of a prefill, replacing re-prefill with cache translation reduces the time to first token after a model switch by 9.6x already on a 64-token prompt for Qwen models on an Apple M3 Ultra, and by up to 29x at 32k context length on an H100. These gains make KV-Lingo particularly useful for dynamic model routing: a context can be processed by one model and handed off to another only when needed, without re-prefilling the shared prefix. We finally show that KV-Lingo can be used for seamless model switching, staying close to re-prefill across repeated switches in our multi-turn evaluations.