Looped models improve language tasks with better structure and input use
Towards Looped Models Done Right, Part I: Topology, Input Injection, Recurrent-State Design
Machine Learning
Summary
Language models that reuse parts of themselves in a loop can be designed in many ways. The authors tested different choices about where to loop, how to feed in new information, and how to organize memory inside the model. They found a particular design, called Huginn, that works better than another popular design on tasks involving reasoning, knowledge, and coding. This design also works well in larger models that use mixtures of experts and can be more efficient.
What this means in practice
- •For ai engineers: Build more efficient language models that are smaller but perform better on reasoning and coding tasks by using the Huginn looped design.
- •For machine learning platform teams: Deploy larger mixture-of-experts models with more balanced expert use and improved performance under the Huginn recurrent design.
Authors
Benhao Huang, Chufan Shi, Junlin Chen, Shicheng Wen, Zhengzhong Liu, Eric Xing, Xuezhe Ma
Abstract
Looped language models reuse physical Transformer blocks across logical depth, but established architectures entangle recurrence placement, input injection, and recurrent state organization. We isolate these axes in models trained from scratch at matched parameter scale, logical depth, and token budgets, and then test whether the resulting design principles transfer to resident- and active-parameter-matched Mixture-of-Experts (MoE) models. The prelude-recurrent block-coda sandwich architecture consistently improves performance on context extraction and reasoning-intensive tasks. Input injection primarily enhances knowledge retrieval and context utilization, whereas random initialization and separate H/L recurrent states provide mixed or negative returns. Taken together, these results favor the Huginn-style design: at the 730M-parameter and 336B-token scale, the complete Huginn architecture outperforms Ouro across all ten benchmarks. The advantages of Huginn over Ouro transfer to MoE models. At the 8BA0.8B (32BA3B when unrolled) scale trained on 500B tokens, Huginn outperforms Ouro on nearly all benchmarks spanning knowledge, reasoning, and coding, while also exhibiting more balanced expert routing. Under matched training and inference FLOPs, the Huginn-style model uses 75% fewer resident parameters than a 112-layer feedforward MoE baseline. Despite its smaller physical parameters, it outperforms the feedforward baseline on DROP and GSM8K, matches it on MATH500, and remains competitive on the other benchmarks.