Large language models can learn impossible languages better than small ones
Scaling Makes It Possible: How Large Models Master Impossible Languages
Computation and LanguageArtificial Intelligence
Summary
It is unclear if language models can learn languages that break normal rules humans use to understand language, called impossible languages. The authors introduce a way to measure meaning in languages, called semantic coherence, to study this question. They show that very large language models perform almost as well on impossible languages as on normal languages, while smaller models struggle more as the language meaning becomes more complex. Bigger models focus less on grammar details and more on the overall meaning, helping them handle strange languages better.
What this means in practice
- •For language technology teams: Develop language models that perform reliably across unconventional or artificially constructed languages by leveraging large-scale models.
- •For ai safety engineers: Design robust language systems less sensitive to minor grammatical tweaks by using larger models that emphasize semantic understanding.
Authors
Yaning Jia, Chunhui Zhang, Chiyu Ma, Zhongyu Ouyang, Ivory Yang, Soroush Vosoughi
Abstract
While language models (LMs) excel in natural language tasks, their ability to learn impossible languages--those that violate the structural, cognitive, or logical constraints that make human languages learnable and functional--remains debatable. This controversy is further fueled by a divide between linguistic theorists and empirical researchers working with small-scale LMs. To address this controversy, we introduce semantic coherence. This concept measures how effectively semantic information is expressed in language structures and examines the learning capabilities of LMs across multiple scales. By fine-tuning billion-parameter LMs on impossible language samples, we demonstrate that larger LMs achieve similar performance on impossible and possible languages, whereas million-parameter LMs exhibit significant degradation when trained on impossible languages. This performance gap widens as semantic complexity increases. We further show that scaling up the LM size fundamentally shifts the learning focus from grammatical rules to semantic understanding, with billion-scale LMs exhibiting a threefold reduction in sensitivity to low-level linguistic perturbations while maintaining stable performance under high-level semantic perturbations.