Low bit n-gram embeddings cut memory use in language models
Low-Bit N-Gram Embeddings for Language Models
Computation and LanguageArtificial Intelligence
Summary
Language models often store information about common short word sequences to work better, but this requires a lot of memory. The authors studied ways to save memory by storing these short sequences at lower precision using fewer bits. They found that using four bits keeps almost all the benefits while saving about 75% of the memory. Even when using fewer bits, a bit of retraining helps recover lost accuracy. This means language models can be smaller and faster without much loss in quality.
What this means in practice
- •For machine learning engineers: Reduce memory use of n-gram embedding tables in language model deployments while maintaining nearly the same performance.
- •For cloud service operators: Lower hardware costs by running large language models with quantized n-gram embeddings that save memory without greatly hurting accuracy.
Authors
Stephen Chung, Wenyu Du
Abstract
Many recent open-weight language models use n-gram embedding tables to improve performance. These tables store learned representations of short token sequences, such as pairs or triples of consecutive tokens, which the model retrieves during processing. However, they can require substantial memory: DeepSeek-V4.1-Flash, for example, allocates 196 billion parameters to its Engram n-gram memory. We therefore investigate whether this memory cost can be reduced while preserving the performance improvement that the tables provide. We propose storing n-gram embeddings at low precision and systematically investigate different quantization methods and bit allocations across tables. Across nanochat-, Llama-, and GPT-2-family models trained from scratch, four-bit tables save approximately 75% of table memory while preserving more than 99% of the validation-loss improvement from adding the tables. Using fewer bits leads to a larger drop in performance, although we find that a short fine-tuning phase recovers a significant part of this loss. In addition, reinvesting the saved memory in larger tables can improve performance beyond that of the original full-precision tables while still using less memory. These table memory savings also extend to large existing models: on DeepSeek-V4.1-Flash and Qwen3.8-Flash-Next, low-bit tables save 47-55% of table memory, with perplexity increases of less than 1.2% and largely preserved downstream task performance. These findings establish low-bit precision as a practical way to reduce the memory requirements of n-gram embeddings and use the saved capacity to improve model performance.