Loop-native residuals enable transformers to handle tens of thousands of steps
Scaling to Tens of Thousands of Test-Time Iterations with Loop-Native Attention Residuals
Artificial Intelligence
Summary
Transformers that repeatedly process information can lose accuracy over many cycles because new, noisy updates overwrite earlier correct steps. The authors introduce InfiLoop, a new way to connect these cycles so that useful information sticks around while unreliable updates get ignored. This allows the system to improve reasoning steadily, even after tens of thousands of repetitions. Their method performs better than previous approaches on challenging tasks like extreme Sudoku puzzles.
What this means in practice
- •For ai model developers: Integrate loop-native residuals to build recursive Transformer models that maintain high accuracy across tens of thousands of inference steps.
- •For automated reasoning engineers: Use InfiLoop to improve long-step reasoning systems such as complex puzzle solvers and advanced question answering with sustained performance over many iterations.
Authors
Pengxiang Li, Dilxat Muhtar, Di He, Guinan Su, Lu Yin, Shiwei Liu
Abstract
In this paper, we argue that looped Transformers need their own residual connections to prevent performance degradation as the number of iterations grows. We observe that increasing loop iterations can reduce reasoning accuracy: noisy state updates overwrite correct intermediate deductions and even undo completed solutions. This leaves subsequent iterations to recover lost information from an already degraded representation: once an error arises in an earlier loop, often as a result of long-range propagation through the recurrence, later loops find it difficult to correct. In this paper, we introduce InfiLoop, a loop-native residual connection that learns which past computations to retain and how much to accept from each new update. InfiLoop combines content-based weighting with learned temporal decay to maintain a running summary of recurrent states. An exact streaming recurrence keeps its persistent aggregation memory constant as the loop count grows. The resulting adaptive update suppresses unreliable proposals and preserves useful intermediate states. Across extensive reasoning tasks, a 7M-parameter InfiLoop model outperforms existing recursive architectures, reaching 97.9% exact accuracy on Sudoku-Extreme, and 13.6% pass@2 on ARC-AGI-2. Notably, on Sudoku-Extreme, InfiLoop continues to improve with test-time looping beyond 20,000 effective steps, showing that added depth translates directly into stronger reasoning. Our code is available at https://github.com/pixeli99/InfiLoop.