Hybrid language models use fewer bits to store memory states
Low-Bit Recurrent States in Hybrid Language Models
Machine Learning
Summary
Language models use memory to understand and predict text, but storing this memory usually takes many bits, which uses a lot of space. The authors found a new way to store these memory states using fewer bits—down to four or six—with almost no loss in accuracy. They do this by cleverly deciding how many bits to use for different parts of the memory and by considering how quickly memory information fades. This method works without extra training or complicated adjustments.
What this means in practice
- •For mobile app developers: Reduce memory usage in language models embedded on devices by storing recurrent states with fewer bits while keeping accuracy high.
- •For cloud ai platform engineers: Lower storage and bandwidth costs by compressing recurrent states in hybrid language models without retraining or calibration.
Authors
Hongren Chen, Jiayang He
Abstract
Hybrid language models maintain fixed-size recurrent states, but existing quantizers typically use eight bits or more. Quantization errors persist according to channel decay rates. We derive distortion weights from the observability Gramian and combine them with normalized state ranges for mixed-precision bit allocation, without calibration data, rotation, or training. We also quantize decay rates logarithmically. With per-token state quantization, a four-bit mean payload reduces excess negative log-likelihood by factors of 3.3--27.9 relative to the best of seven baselines across three hybrid models; metadata costs vary. At six bits, negative log-likelihood differs from the FP32-state baseline by less than 0.005 nats. Ablations separate gains from variable bit widths, decay weighting, and range normalization. With less frequent write-backs, gains diminish and depend on the model and budget.