Small language models struggle to generalize abstract reasoning tasks
A Systematic Study of Small Language Models on Abstract Reasoning Tasks
Machine LearningComputation and Language
Summary
This paper looks at how well small language models can solve puzzles that require abstract reasoning, like recognizing patterns in grids. The authors find that while these models can learn to solve tasks they've seen during training, they often fail when the tasks change slightly, even if the underlying rule stays the same. They also study different setups and model types to understand why this happens and how attention in the models behaves. Their results show that success depends a lot on the model's details, training methods, and how the problem is presented.
What this means in practice
- •For machine learning engineers: Improve training setups for small language models to better recognize abstract patterns under changing conditions.
- •For ai tool developers: Design language model-based tools that account for limited generalization in abstract reasoning tasks when adapting to new inputs.
Authors
Nur A Zarin Nishat, Jens Lehmann, Andrei Aioanei, Sahar Vahdati
Abstract
Endpoint accuracy on abstract-reasoning benchmarks does not reveal whether a language model has acquired a transferable rule or fit distribution-specific regularities. We study this distinction in small language models on the ARC-TGI benchmark, which organizes abstract grid transformations into controllable task families and supports resampling, spatial shifts, and cross-benchmark transfer. Across more than 1,000 runs, we profile decoder-only, encoder--decoder, and mixture-of-experts model families under supervised fine-tuning. We examine the efficiency and stability of skill acquisition, robustness beyond the training distribution, interactions with model family and task formulation, and layer-wise attention signatures that accompany behavioral differences. Substantial in-distribution accuracy is attainable, but acquisition is sensitive to optimization and unevenly distributed across task families. Performance deteriorates sharply outside the training distribution, including when the rule is retained but grid scale changes. Greater training-set depth and breadth yield uneven gains, while the effect of additional in-context examples depends on model family. Executable-rule induction also yields correct solutions not observed under direct grid generation. On selected tasks, attention diagnostics show distinct concentration and context-dependence profiles, but do not establish general causal mechanisms. Overall, abstract-reasoning scores are conditional on the model, adaptation regime, evaluation distribution, and response format.