Golden ratio dithering improves memory efficiency in language models
Low-Discrepancy Dither for Quantized Recurrent State Caches
Machine Learning
Summary
Storing the memory of language models in low precision saves computer resources but causes tiny errors that add up over time. The authors found that using a special, predictable pattern based on the golden ratio to round numbers keeps these errors smaller than the usual random rounding methods. This approach makes the model's memory closer to high-precision versions without extra cost. They also show that the typical way of rounding to the nearest number can seem okay at first but gets worse over long use.
What this means in practice
- •For language model engineers: Improve low-precision recurrent memory updates to reduce error accumulation over long text generation runs.
- •For deep learning hardware designers: Design quantized memory systems that implement deterministic golden-ratio dithering to maintain model accuracy without random number generation overhead.
Authors
Snigdha Chandan Khilar
Abstract
Mamba-style and hybrid language models compress their past into a fixed-size recurrent state that is rewritten at every generated token. Storing this state in low precision saves memory bandwidth, but every rounding error is fed back into the next update and can accumulate over long generations. Production systems round the state stochastically; we ask which rounding rule such caches should use. We find that a deterministic golden-ratio Weyl dither, which needs no random numbers, consistently brings the quantized model closer to the full-precision one than stochastic rounding, across pure and hybrid models, storage formats, and long decoding horizons, at no extra cost. Round-to-nearest behaves differently: because it discards small updates, its error keeps growing, so it can look best in short evaluations yet falls far behind over long generations. A discrepancy analysis explains this ordering, and we document implementation pitfalls that silently remove the benefit.