Looped language models save memory and computation on easy tokens

Enabling Dynamic Computation in Looped LMs

Artificial Intelligence

Summary

Language models that repeat calculations multiple times can be efficient but often end up doing all the work for every word anyway. The authors explain why previous models didn’t save much effort and suggest a new way to reuse stored information from earlier repeats, cutting down on memory and computation by up to 30%. They also improved how the model decides when to stop working on each word, so it only works as hard as needed. Their ideas were tested both on existing big models and smaller ones trained from scratch.

What this means in practice

  • For cloud ai infrastructure teams: Reduce computation costs and memory usage in language model serving by implementing better cache reuse strategies for dynamic inference.
  • For mobile app developers: Enable faster, more efficient on-device language understanding by integrating looped models that adaptively skip unnecessary computation for simple inputs.

Authors

Aayush Mishra, Arnau Padrés Masdemont, Victor Conchello Vendrell, Jordi Ros Giralt, Arash Behboodi, Fabio Valerio Massoli

Abstract

Looped LMs are parameter efficient and promise dynamic computation (saving memory and FLOPs on easy tokens). However, state-of-the-art open Looped LMs trained with this dynamic computation capability (Ouro models) do not realize it in practice as each loop iteration (depth) requires its own level of KV-cache, necessitating all loop computations. Moreover, Ouro's early-exit prior is enforced on each token equally, which results in static lower-depth like processing of all tokens regardless of difficulty. In this work, we propose a simple "best-available" KV caching strategy that works out-of-the-box, creating a new frontier in the performance vs depth space. Our approach enables up to 30% reduction in FLOPs and KV memory while retaining full-depth performance, showing the true flexibility of Looped LMs. Furthermore, training looped LMs with awareness about this KV caching strategy improves performance and efficiency. Finally, we apply a small but effective fix to the early-exit prior enforcement objective that makes tokens exit at truly heterogeneous depths based on effort. Our findings are validated on Ouro models as well as smaller looped LMs pre-trained from scratch.