Hybrid attention model state handoff speeds up large language model usage

Beyond K/V Caches: Cross-Model State Handoff in Hybrid Linear-Attention LLMs

Computation and Language

Summary

Large language models (LLMs) often switch between different versions to balance quality and cost during long conversations, but switching requires repeating some setup work. The authors studied how to better transfer memory and state information between models with hybrid attention mechanisms, which keep multiple types of memory. They found interactions between these memory types make straightforward transfers less effective. To fix this, they designed a method called Hybrid State Transfer that carefully aligns and preserves these states, allowing faster and more accurate switching between models. Their tests show it keeps most of the task performance while speeding up the process over 20 times.

What this means in practice

  • For llm infrastructure engineers: Switch between different sized language models more quickly during conversations by efficiently transferring hybrid attention states.
  • For cloud service operators: Reduce compute costs and improve responsiveness by cascading large language models with faster cross-model state handoffs.
  • For chatbot platform developers: Enable smoother continuation of conversations when switching between models of different sizes using advanced state transfer techniques.$Commercial implications: This enables commercial chatbot products that combine speed and quality through dynamic model switching with high task retention.

Authors

Jiacheng Cai, Tao Zhang, Siqi Liang, Yun Dai, Ziqian Zeng

Abstract

Production deployments often use different models to balance cost and quality through cascading, routing, and switching models during long conversations. Context accumulates across turns. Each switch requires the target to repeat prefill over this context. Existing methods transfer full-attention (FA) K/V caches across models. Hybrid linear-attention models also retain recurrent states (REC) and convolution histories (Conv). The interaction between Conv and REC complicates cross-model state transfer. Conv perturbations can change keys and values in recurrent layers and alter REC updates. Conv histories are overwritten, but changes in REC can still affect later outputs. More accurate REC reconstruction need not improve continuation. Building on this, we propose Hybrid State Transfer (HST) for all three state types. It uses structured REC maps and preserves history order during Conv alignment. We calibrate the adapters using the frozen target's outputs during continuation. FA cache map updates are merged offline. Across five benchmarks, Qwen3.5 4B$\rightarrow$27B and 9B$\rightarrow$27B achieve mean task retentions of 89.88% and 94.11%, respectively. With source caches already available, separate timing on synthetic 16K inputs yields handoff body speedups of $21.66\times$ and $21.91\times$.
View PDFOpen arXiv