Small language model matches performance with fewer parameters
Looped GPT-BERT: Trading Parameters for Computation in Small Language Modeling
Computation and LanguageArtificial Intelligence
Summary
This paper explores how a language model with fewer parameters can perform as well as larger models by reusing the same layers multiple times. The authors combine two types of language modeling tasks and share parameters across layers, training on a relatively small English dataset. Their approach, called Looped GPT-BERT, achieves similar scores to bigger baseline models on standard language understanding tests. However, the model struggles on some tasks likely because reusing few layers limits its ability to represent complex information.
What this means in practice
- •For small ai development teams: Build efficient language models that save memory by reusing layers while maintaining good performance on common language tasks.
- •For natural language processing engineers: Optimize limited-resource language models used in speech recognition or chatbots by reducing parameter count without sacrificing effectiveness.
Authors
Tingshuo Fan, Hongtao Mu, Tianyu Zhou, Hansen Liu, Tao Ji
Abstract
When training data are limited, increasing parameter count is not the only way to improve language-model performance. A small parameter set, when repeatedly applied, can also deliver comparable performance. We study Looped GPT-BERT in the BabyLM 2026 Strict-small setting, combining GPT-BERT's masked next-token and causal language-modeling objectives with depth-wise parameter sharing. We train on a preprocessed 7.48M-word English corpus and compare objective ratios, non-looped and looped architectures, and loop counts. Our final $4\times12$ model uses four physical layers for twelve recurrent traversals and contains 12.18M parameters. The BabyLM 2026 leaderboard reports an Overall Average of 35.42 and an NLP Average of 48.48. Compared with public BabyLM 10M Strict-small GPT-2 and GPT-BERT baselines, it achieves comparable performance on selected linguistic and downstream metrics, including BLiMP and GLUE, with fewer parameters. The loop ablations show that additional recurrent computation can improve training and preserve strong performance on selected linguistic tasks, whereas poorer performance on other tasks may reveal an inherent limitation of the looped design: using only a few physical layers restricts the model's representational space.