Upcycling pretrained large language models with efficient gated layer loops
UpLoop: Upcycling Large Language Models with Gated Loop
Machine LearningArtificial Intelligence
Summary
Large language models (LLMs) are typically very big and costly to run. The paper introduces UpLoop, a new technique that makes pretrained LLMs work more efficiently by looping parts of their layers multiple times with special gates, instead of retraining them from scratch. This method keeps the model’s structure intact and improves performance by refining outputs on each loop. The authors show that using three loops with UpLoop can boost accuracy by over 6% on many tests, using a smaller computational cost compared to training a bigger model.
What this means in practice
- •For natural language processing teams: Enhance performance of existing pretrained language models with limited retraining by adding gated looping layers for better accuracy cost-effectively.
- •For software engineers deploying ai models: Reduce computational resources needed for large language model inference by reusing pretrained models with iterative gated loops instead of increasing model size.
Authors
Tieyuan Chen, Huabin Liu, Yingtao Zhang, Shijie Li, Tengyu Pan, Weiyao Lin, Jianguo Li
Abstract
Looped large language models trade parameter storage for recurrent computation, but most are trained from scratch. In practice, pretraining with looped depth is expensive, making post-training upcycling of a pretrained non-looped LLM an attractive alternative. We introduce \textbf{UpLoop}, a post-training recipe based on \emph{layer-wise} recurrence. Instead of looping a whole model or a middle block, UpLoop loops each selected layer locally, preserving the pretrained depth hierarchy and avoiding feature-space mismatch. To make loop execution useful, we add hidden-wise gates on the attention and MLP residual branches, so later loops act as progressive refinements rather than redundant rewrites. Empirically, UpLoop breaks the commonly observed saturation at two loops ($R=2$) in prior looped models. Starting from Qwen3-4B Base and applying SFT, UpLoop improves the average accuracy across 20 benchmarks by 6.17\% at $R=3$. We use $R=3$ as the main setting for the best accuracy--cost trade-off, while both native and extrapolated $R=4$ yield further gains.