Stack based language models struggle with complex word orders in artificial languages
Typological Alignment of Stack-Based Language Models on Mildly Context-Sensitive Artificial Languages
Computation and Language
Summary
Some word orders in languages appear more often because they are easier for learners to pick up. The authors studied how certain types of language models, which use a stack-based memory to handle sentence structure, learn patterns in made-up languages that mimic complex grammar features found in real languages. They found that these models have trouble understanding complicated sentence patterns called cross-serial dependencies unless their memory is limited, which helps them generalize better. This suggests why some word orders might be more common in natural languages, based on how learners process them.
What this means in practice
- •For natural language processing engineers: Design more memory-efficient stack-based language models by controlling working memory limits to improve handling of complex sentence structures.
- •For computational linguists: Use insights about learning biases in stack-based models to model why some word orders are more common in human languages than others.
Authors
Nadine El-Naggar, Tatsuki Kuribayashi, Ted Briscoe
Abstract
Some properties of languages, e.g., subject-object-verb (SOV) word order, are more prevalent than others among the thousands of attested natural languages (NLs). Such typological commonality is often attributed to learning biases. Computational simulations, recently with language models (LMs), have facilitated the exploration of this theory. In this paper, we extend existing analyses of the relationship between LMs' learning biases and typological commonality on both data and model sides, focusing on: (i) cross-serial dependencies, the upper limit of attested syntactic complexity, and (ii) stack-based LMs (SLMs), potentially facilitating learning of hierarchical patterns. We first evaluate generalization of SLMs on cross-serial dependencies across diverse artificial languages and confirm that they struggle with such constructions. However, SLMs with limited working memory generalize better suggesting a possible basis for such inductive bias and thus the typological commonality of some word order configurations.