Kv cache refresh improves long multimodal generation accuracy

Drift Says When, Budget Says How Much: Rate-Controlled KV Re-selection for Long-Form Multimodal Generation

Machine Learning

Summary

Long multimodal AI models use a shortcut to remember past visual information by keeping only some image tokens. The authors found that picking these tokens once at the start doesn’t work well when generating detailed descriptions because the model’s focus changes over time. They propose a new method that watches when the model’s attention drifts and refreshes the selection at the right times and rates. This approach balances when to update with how often, improving description quality while keeping computing costs low. Their method works better than prior fixed or per-step refresh strategies on multiple tasks and models.

What this means in practice

  • For multimodal ai system engineers: Improve accuracy of long-form image and video descriptions by dynamically refreshing visual token selection based on attention drift.
  • For gpu performance engineers: Optimize GPU memory and data movement by applying rate-controlled refresh mechanisms for token caching in multimodal models.

Authors

Bin Hu, Yingbo Xiao

Abstract

KV cache compression for multimodal LLMs usually selects the visual tokens to keep once, at prefill, and reuses that selection for the whole generation. We show that this one-shot selection is severely limited on long-form generation: on detailed image description, even an oracle that selects with the attention of the uncompressed answer reaches only 0.44-0.52 ROUGE-L agreement with the uncompressed output, because the evidence the model reads moves as the description unfolds. Re-selecting whenever attention drifts out of the kept set is the natural remedy, but we find that a fixed drift threshold fails systematically: drift magnitudes are not comparable across samples, so low-drift samples are almost never refreshed although they benefit the most. We propose Rate-Controlled Refresh (RCR), which lets drift decide when and where to re-select while a per-sample controller regulates how often, combined with a low-rank drift detector, candidate-restricted re-selection, and a decision that stays on the GPU. With one setting for all models and tasks, RCR outperforms the per-step retrieval method InfiniGen by 1.6-3.8 ROUGE-L points on five of six model-task settings with no significant difference on the sixth, while staying within 1.7 points of exact per-step re-selection, with comparable estimated data movement per step and a 4x smaller GPU-resident detector. On DetailCaps, drift-chosen timing beats random timing at the same refresh rate and periodic timing even at a higher rate, by 2.0-5.0 points. On pre-registered video description, where drift is homogeneous across samples, RCR is on par with InfiniGen on Qwen2-VL and exceeds it on Qwen3-VL but no longer beats a fixed threshold at the nearest tested rates, consistent with the diagnosis; object-level evaluation shows that RCR achieves higher object F1 against the uncompressed output than InfiniGen on two of three models.