Loop the Loopies!
Computation and LanguageArtificial Intelligence
Summary
The authors introduce Loopie, a new type of Transformer model that uses looping and Mixture-of-Experts techniques to be more efficient with computing power. They made two versions, one large and one smaller, and showed that Loopie does better than regular Transformers when given the same amount of training effort. Their tests, including comparisons with a big baseline model, demonstrate Loopie's improved performance. They also developed a special training step that helps Loopie solve hard reasoning problems, achieving top scores in major math and physics contests.
Authors
Zitian Gao, Yilong Chen, Yihao Xiao, Xinyu Yang, Ran Tao, Joey Zhou, Bryan Dai
Abstract
We present Loopie, the most powerful looped Transformer to date. The Loopie series consists of two Mixture-of-Experts (MoE) models: a 20B-parameter model with 2B active parameters and a 6Bparameter model with 0.6B active parameters. Looped Transformers have long faced a challenge: given an N-fold increase in pre-training compute, increasing the parameter count by a factor of N usually outperforms looping a model N times. Loopie addresses this challenge. Extensive ablation studies, including comparisons with a vanilla 30B-A3B model, show that Loopie substantially outperforms vanilla Transformer baselines trained with the same compute budget. Our novel post-training pipeline equips Loopie with strong reasoning abilities. At the 2025 IMO and IPhO, Loopie achieves gold-medal performance without tools.