WnW: Waxing-and-Waning KV Cache for Long-Form Speech LLMs
2026-08-24 • Computation and Language
Computation and LanguageSound
AI summaryⓘ
The authors studied how speech language models handle very long audio inputs, focusing on a part called the KV cache that uses a lot of memory. They found that current methods, which discard some memory permanently, don’t work well because attention during decoding spreads out differently than during a first pass. To fix this, they designed a new method called WnW that smartly organizes memory into three types to keep important parts accessible. Their approach keeps most of the model's accuracy while reducing memory use and works well across different languages and tasks without adding much delay.
KV cachespeech language modelsattention mechanismprefilldecode-timememory compressionLibriSpeech-LongGPU memoryoffline calibrationtoken recall
Authors
Yiming Yao, Chenyang Lyu, Xuanfan Ni, Longyue Wang, Weihua Luo, Yazheng Yang, Jinsong Su
Abstract
Long-form audio inputs make the KV cache the dominant memory cost of speech LLMs. Prefill-only KV compression methods permanently discard audio KV positions once evicted, with no pathway to recover them during decoding. We show this is fragile on long-form audio: prefill attention concentrates near the audio start (an attention-sink effect), while decode-time attention distributes broadly, and the two rankings overlap weakly. We propose WnW (Waxing-and-Waning KV cache), which classifies KV-heads into anchor, tidal, and fixed roles via offline calibration. Anchor heads remain on GPU and serve as a decode-time importance observer; tidal heads keep a CPU-resident complement that is recalled chunk-by-chunk based on aggregated anchor-head scores; fixed heads keep only an on-GPU subset, with the rest permanently discarded. On LibriSpeech-Long with two 3B backbones (Voxtral-mini-3b and Qwen2.5-Omni-3B), WnW preserves near-Full-Cache accuracy while keeping only 20% of audio tokens on GPU, where prefill-only baselines fail to terminate. Results generalize across language, task, and domain shifts, and CPU-GPU recall adds little decode-time overhead in our measurements.