Looped model compression improves memory use without performance loss
A Tilted Bowl Is Not a Slippery Slope: Compressing Looped Models
Machine LearningComputation and Language
Summary
Some AI models repeat the same calculation many times to solve problems, which uses a lot of memory. The authors found that when these repeated steps stabilize, small rounding errors don’t build up and ruin results as people thought. Instead, these errors just shift the final point a bit, like tilting a bowl changes where a ball stops rolling. Using this insight, they created a way to compress these models safely and know when they might fail, which helps save memory traffic and keeps accuracy high.
What this means in practice
- •For machine learning engineers: Reduce memory bandwidth during looped model inference by applying adaptive stopping and low-bit finishing loops to maintain accuracy efficiently.
- •For embedded system developers: Deploy looped AI models on limited hardware with better efficiency by using compressed looping blocks and halting control for adaptive compute.
Authors
Steven Kolawole, Pearse Jim, Opegbemi M. Busoye, Glory Bagai, Virginia Smith
Abstract
Looped models reason by applying the same block of weights many times, so compressing that block saves memory traffic on every loop. Compressed looped models, however, often collapse, and the collapse is usually blamed on rounding error that accumulates from loop to loop. In this work we test that account on more than 30 models from five families and find, to our surprise, that it holds only for loops that never settle. When a loop settles, a fixed rounding error does not accumulate. It moves the point where the loop settles, much as tilting a bowl moves where a ball comes to rest, and the answer is lost only when the shift is larger than the readout tolerates. This picture lets us predict which models fail from a single label-free measurement, and it tells us why failed models recover: their loops still settle, so a few final loops with 8-bit weights bring the answer back. Motivated by these findings, we build a controller that stops when the model's halting head fires and then finishes with 8-bit loops. On Sudoku-Extreme and Maze-Hard it beats fixed-depth inference by up to 15 points under a third of the weight traffic.