Kv cache quantization reduces memory use for omni-modal language models

OmniKVQuant: KV Cache Quantization for Omni-LLMs

Computer Vision and Pattern Recognition

Summary

Large language models that process text, audio, and video need to remember a lot of information, which uses a lot of memory. The authors studied a popular method to compress this memory, called TurboQuant, and found it didn’t work well for these multi-modal models because of two problems with how information changes over time and differs between types of data. They created a new method, OmniKVQuant, which fixes these issues by adjusting how the memory is compressed over short time windows and separately handling different data types. Their method keeps the model’s performance while cutting memory use dramatically, and they also built a tool to efficiently use this compressed memory during computations.

What this means in practice

Authors

Suho Yoo, Hyunjong Ok, Jongmin Choi, Jihoo Jung, Joon Son Chung

Abstract

As Omni-modal large language models (Omni-LLMs) take in audio, video and text together, their KV cache memory cost grows. KV cache quantization is the de facto approach in text-only LLMs, but its application to Omni-LLMs remains unexplored. In this paper, we analyze how TurboQuant, a representative rotation-based KV cache quantization method, behaves on multimodal caches and identify two critical issues: temporal key drift and heterogeneous value geometry. To address these, we propose OmniKVQuant, a training-free framework that (i) sets the key quantization range over each short window of the input stream; and (ii) rotates values separately per modality. On Qwen2.5-Omni and Qwen3-Omni, OmniKVQuant enables 2-bit KV caches while substantially preserving performance across seven audio-visual benchmarks. We further provide a fused Triton decode kernel that unpacks the 2-bit cache during attention, so no dense FP16 cache is ever built. Code: https://github.com/kaistmm/OmniKVQuant
View PDFOpen arXiv