Deeper and bigger language models improve performance with less training data
Depth and Scale in the Sub-150M Regime: JugnuLM-53M vs JugnuLM-110M
Artificial IntelligenceComputation and Language
Summary
Building bigger and deeper language models usually helps them understand and generate text better, but it’s not clear how well this works for smaller models. The authors compared two versions of a language model with about 53 million and 110 million parameters, using the same training method and data. They found the larger model performed better on language tasks even though it was trained on fewer examples. They also tested some tweaks but found that certain improvements helped some tasks while others didn’t, offering a baseline for future work in this size range.
What this means in practice
- •For ai engineers: Design more efficient small-scale language models that achieve better accuracy using deeper, narrow architectures rather than larger datasets.
- •For natural language processing teams: Improve language understanding benchmarks in limited data settings by choosing model depth and optimization techniques evaluated in this work.
Authors
Dushyant Rajput, Nirdesh Chauhan, Siddharth Kosaraju
Abstract
We scale our conventional sub-150M pretraining recipe from 53.5M to 109.7M parameters, holding the method fixed (Qwen3-style decoder with grouped-query attention, RoPE, SwiGLU, RMSNorm, QK-Norm, and a z-loss; FineWeb-Edu data) and changing only the geometry to a deep-and-thin 23-layer x 576-hidden design. The larger model improves across the board -- BLiMP 78.1 -> 81.3, ARC-Easy 51.4 -> 52.5, WikiText-2 byte-perplexity 2.04 -> 1.95 -- and its 81.3% BLiMP essentially matches GPT-X2-125M (81.28) at about 12% fewer parameters. Notably the 110M model achieves this on fewer training tokens (about 8B vs 12B), so the gain is attributable to capacity and depth, not more data. Both models are deliberately conventional; this report is a clean scaling control and the baseline rung (R0) of an ablation study of what further improves models in this regime. An ablation ladder follows: value residuals (R1) and the Muon optimizer (R2) lift ARC-Easy by a cumulative +3.6 (52.5 -> 56.1) at a near-flat BLiMP and are kept; a diverse data blend (R3) and two logit-distillation settings (R4a/R4b) are not kept -- honest negatives. R3 pins ARC-Easy to FineWeb-Edu's educational filtering rather than raw diversity; distillation from a 1.7B teacher can reach the class-leading ARC-Easy (56.99, matching GPT-X2-125M) but only at a perplexity cost that dialing KD down then erases -- so R2 remains the best kept stack.