Cachelink improves cross-model key value cache transfer without reloading
CacheLink: Functional KV Transport for Zero-Prefill Cross-Model Context Handoff
Machine LearningArtificial Intelligence
Summary
When large AI models switch during a conversation, they usually need to reload all the context, which takes time. The authors look at how to transfer cached information smoothly between models without reloading the context. They found that just trying to copy the cached data accurately isn't enough, because the way the receiving model uses this data can vary a lot. They propose Cachelink, which learns to adjust the cached information based on how the receiving model behaves, making context handoff faster and more effective.
What this means in practice
- •For large language model developers: Enable efficient context switching between different models without costly context reloading delays.
- •For cloud inference service engineers: Reduce latency of serving long conversations by improving cache transfer in multi-model pipelines.
Authors
Haomiao Qiu, Yining Tang, Zijia Song, Ruizi Han, Ziyue Qiao, Miao Zhang
Abstract
KV caches make long-context serving efficient by reusing past computation, but model switching breaks this reuse and forces the receiver to prefill the context again. We study \emph{zero-prefill cross-model context handoff}: translating a sender cache into a receiver-compatible state so that the receiver can continue directly without prefilling same context. Existing affine transport methods show that receiver KV states can be substantially reconstructed from sender caches. However, through extensive analyses, we find that reconstruction fidelity is a poor proxy for receiver utility. Similar K/V reconstruction errors can lead to substantially different attention patterns, predictive distributions, and downstream accuracy, because conventional reconstruction objectives overlook the query-dependent functional importance of K and V in receiver attention. We thus introduce CACHELINK, which learns a coupled K/V correction through the receiver behavior directly. Across four independently trained sender--receiver pairs and seven evaluation panels, CACHELINK improves the equal-weight mean over all methods for every pair. In matched diagnostics, CACHELINK changes K/V squared error by less than 0.4% while substantially changing receiver attention and next-token behavior. At 32K context, it reduces handoff latency by up to $7.27\times$ relative to receiver re-prefill. Our findings verify that reconstruction fidelity alone is insufficient and that preserving the receiver computation induced by the transported cache is important for effective cross-model KV transport.