Type diversity helps transformers understand language structure better
Type Diversity Enables Transformers to Generalise Compositionally
Computation and Language
Summary
Transformers, a type of artificial intelligence, often struggle to understand new ways that words and sentence structures are combined. The authors found that this difficulty isn’t because of a built-in limitation, but rather because previous tests used datasets with less variety in sentence structures compared to word types. By creating new datasets with more balanced variety in both words and structures, they showed that transformers can actually learn new patterns equally well. They also questioned earlier ideas about what makes these tasks hard and looked into other factors that might affect learning.
What this means in practice
- •For natural language engineers: Improve language model training by balancing variety in both word and sentence structure types to enhance learning of novel expressions.
- •For programmers of semantic parsers: Develop semantic parsers that handle uncommon sentence patterns better by using datasets designed with high type diversity in both lexical and structural elements.
Authors
Anssi Moisio, Mathias Creutz, Mikko Kurimo
Abstract
Compositional generalisation has been divided into lexical and structural generalisation. Previous work has found that structural generalisation is harder than lexical for Transformers. We propose that this difference is not inherent to Transformers, but due to the high diversity of lexical types and low diversity of structural types in the specific datasets of these previous works. By type diversity we mean the number of different constructors of that type, instead of, for example, the specific word combinations that might populate the structure. To test this, we vary the amounts of type diversity of lexical and structural types in previously published datasets. We create linguistically diverse variants of the COGS and SLOG datasets using Grammatical Framework. We find that type diversity correlates with compositional generalisation equally in lexical and structural test cases, supporting our hypothesis. We note a contradiction with the proposition in previous work that compound divergence explains the difficulty in compositional generalisation tasks. We further investigate the effects of other dataset properties on compositional generalisation, such as the diversity of types other than the novel test structure, and surface properties of the logical semantics format.