Quantizing language model memories cuts storage by three quarters

N-Gram Memories at Two Bits: Quantizing Lookup Tables in Large Language Models

Machine LearningComputation and Language

Summary

Language models use large tables to remember context, but these tables can be huge and hard to store efficiently. The paper introduces a new way to compress these tables, storing only what’s needed for each input with fewer bits per entry. Their method uses a shared program to reconstruct specific parts, reducing storage size significantly without losing much accuracy. This lets bigger models run on devices like GPUs without needing as much memory.

What this means in practice

Authors

Donghoon Han, Minseop Jung, Dong Woo Kwak, SeungJae Lee, Junhoo Lee

Abstract

Conditional memory expands language-model capacity through selective lookup, but the full embedding table must remain available for every possible input. We introduce Conditional Program (CP), a compressed representation that reconstructs only the rows an input requests. CP represents each row by a short code and a local scale, with a reconstruction program shared across the table. We instantiate this representation with a Gaussian trellis, norm restoration and independently decodable row records. CP's two-bit code lowers output KL by 41--63\% relative to 2-bit round-to-nearest. Analysis across three trained n-gram memory families shows that Gaussian-like coordinate statistics coexist with learned relationships among rows; at matched storage budgets, the shared trellis outperforms the fitted codebooks and low-rank residual variants we test. For DeepSeek, it reduces table storage from 203GB to 51GB, leaving room for larger workloads when the table is kept on GPU.