Looped language models save memory and computation on easy tokens
Enabling Dynamic Computation in Looped LMs
Artificial Intelligence
Summary
Language models that repeat calculations multiple times can be efficient but often end up doing all the work for every word anyway. The authors explain why previous models didn’t save much effort and suggest a new way to reuse stored information from earlier repeats, cutting down on memory and computation by up to 30%. They also improved how the model decides when to stop working on each word, so it only works as hard as needed. Their ideas were tested both on existing big models and smaller ones trained from scratch.
What this means in practice
- •For cloud ai infrastructure teams: Reduce computation costs and memory usage in language model serving by implementing better cache reuse strategies for dynamic inference.
- •For mobile app developers: Enable faster, more efficient on-device language understanding by integrating looped models that adaptively skip unnecessary computation for simple inputs.
Authors
Aayush Mishra, Arnau Padrés Masdemont, Victor Conchello Vendrell, Jordi Ros Giralt, Arash Behboodi, Fabio Valerio Massoli
Abstract
Looped LMs are parameter efficient and promise dynamic computation (saving memory and FLOPs on easy tokens). However, state-of-the-art open Looped LMs trained with this dynamic computation capability (Ouro models) do not realize it in practice as each loop iteration (depth) requires its own level of KV-cache, necessitating all loop computations. Moreover, Ouro's early-exit prior is enforced on each token equally, which results in static lower-depth like processing of all tokens regardless of difficulty. In this work, we propose a simple "best-available" KV caching strategy that works out-of-the-box, creating a new frontier in the performance vs depth space. Our approach enables up to 30% reduction in FLOPs and KV memory while retaining full-depth performance, showing the true flexibility of Looped LMs. Furthermore, training looped LMs with awareness about this KV caching strategy improves performance and efficiency. Finally, we apply a small but effective fix to the early-exit prior enforcement objective that makes tokens exit at truly heterogeneous depths based on effort. Our findings are validated on Ouro models as well as smaller looped LMs pre-trained from scratch.