Is More Data Worth the Cost? Dataset Scaling Laws in a Tiny Attention-Only Decoder
2026-04-10 • Machine Learning
Machine LearningComputation and Language
AI summaryⓘ
The authors studied how the size of the training dataset affects the performance of Transformer language models using a simpler version of the model. They found that using more data steadily improves results, but the gains get smaller as you add more data. Surprisingly, using about 30% of the full data can achieve around 90% of the accuracy reached with all the data. Their work helps understand how to choose the right amount of data when resources are limited.
Transformerlanguage modeldataset sizescaling lawsattention mechanismdecoder architecturetraining datavalidation accuracycomputational budget
Authors
Götz-Henrik Wiegand, Lorena Raichle, Rico Städeli, Tomas Hrycej, Bernhard Bermeitinger, Siegfried Handschuh
Abstract
Training Transformer language models is expensive, as performance typically improves with increasing dataset size and computational budget. Although scaling laws describe this trend at large scale, their implications in controlled, smaller-scale settings remain less explored. In this work, we isolate dataset-size effects using a strongly reduced attention-only decoder architecture. By training on progressively larger power-of-two subsets, we observe smooth performance improvements accompanied by clear diminishing returns, consistent with scaling-law behavior. Using only about 30% of the training data is sufficient to reach approximately 90% of the full-data validation token-level accuracy. These results provide actionable insights into dataset scaling in a controlled, component-isolated setting and offer practical guidance for balancing dataset size and computational cost in compute- and data-restricted environments, such as small research labs and exploratory model development.