Papers for

deep learning hardware designers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Golden ratio dithering improves memory efficiency in language models

Low-Discrepancy Dither for Quantized Recurrent State Caches

Abstract: Mamba-style and hybrid language models compress their past into a fixed-size recurrent state that is rewritten at every generated token. Storing this state in low precision saves memory bandwidth, but every rounding error is fed back into the next update and can accumulate over long generations. Production systems round the state stochastically; we ask which rounding rule such caches should use. We find that a deterministic golden-ratio Weyl dither, which needs no random numbers, consistently brings the quantized model closer to the full-precision one than stochastic rounding, across pure and hybrid models, storage formats, and long decoding horizons, at no extra cost. Round-to-nearest behaves differently: because it discards small updates, its error keeps growing, so it can look best in short evaluations yet falls far behind over long generations. A discrepancy analysis explains this ordering, and we document implementation pitfalls that silently remove the benefit.

Wed 30 SeptMachine Learning

The gist

Storing the memory of language models in low precision saves computer resources but causes tiny errors that add up over time. The authors found that using a special, predictable pattern based on the golden ratio to round numbers keeps these errors smaller than the usual random rounding methods. This approach makes the model's memory closer to high-precision versions without extra cost. They also show that the typical way of rounding to the nearest number can seem okay at first but gets worse over long use.

Open → 2609.39185v1