E8 lattice vector quantization improves key compression for large language models
E8 Lattice Key Quantization for Low-Bit KV-Cache Compression
Machine LearningArtificial Intelligence
Summary
Large language models need to remember lots of information in memory, which can be expensive. Compressing this memory with fewer bits usually reduces quality or speed. The authors show that using a special math structure called the E8 lattice to group and compress memory keys in 2 bits keeps model quality high and runs just as fast. This method works better than older approaches that use 3 bits and results in roughly six times smaller memory for keys and values combined. Their approach works well across many language models and datasets without slowing down.
What this means in practice
- •For llm infrastructure engineers: Reduce memory footprint for serving long-context large language models without losing quality or speed.
- •For mobile device developers: Enable deployment of large language models with constrained memory by applying efficient key-value compression.
Authors
Sasho Nedelkoski, Alexander Acker, Soeren Becker, Dominik Scheinert, Odej Kao
Abstract
Compressing the key-value (KV) cache is essential for serving long-context large language models (LLMs) under memory constraints. While current scalar quantization methods achieve significant compression, they typically face a quality-fidelity trade-off, either suffering from performance instability at 2-bit precision or necessitating 3-bit keys to maintain model quality. In this work, we demonstrate that a uniform low-bit k2v2 configuration (2-bit keys and 2-bit values) can preserve near-baseline quality by using vector quantization (VQ) based on the $E_8$ lattice for KV keys. We show that structured VQ exploits correlations across adjacent dimensions to achieve signal-to-quantization-noise ratios (SQNR) higher than those of independent scalar quantization at the same key payload rate. To ensure practical utility, we adapt a table-based asymmetric scoring algorithm that avoids explicit codebook dequantization during attention computation and preserves decode-throughput parity with scalar baselines. Across five LLM families, our $E_8$ k2v2 approach yields \textasciitilde$6\times$ KV compression, roughly 20\% smaller than scalar k3v2 (3-bit keys and 2-bit values), while retaining near-baseline long-context perplexity. In our full-suite perplexity evaluation, $E_8$ improves over scalar k3v2 in 80\% of model--dataset--context settings. These results indicate that $E_8$ lattice VQ improves quality per byte while keeping throughput at parity, suggesting that structured key quantization is a useful primitive for KV-compression stacks. Overall, our work provides a low-overhead primitive for expanding the quality-memory frontier in long-context LLM inference.