Large language models struggle to identify semantic word relationships correctly
Evaluating Semantic Relation Knowledge through Bidirectional Relation Prediction from Gloss Pairs
Computation and Language
Summary
Understanding how words are related, like if one word is a kind of another or part of another, is important for language understanding. The authors created a tough test that asks AI language models to guess the relationship between word definitions presented in two different orders. They tested many large language models with definitions in several languages and found most models perform poorly, especially in recognizing certain relationships and being consistent when presented different ways. This shows current language models have limits in understanding and applying these word relationships accurately.
What this means in practice
- •For natural language processing engineers: Identify limitations in language models for semantic relation prediction to improve model design and testing strategies.
- •For multilingual chatbot developers: Refine chatbots’ understanding of word relationships across languages by addressing current model weaknesses in semantic consistency.
Authors
Zhihan Cao, Adam Nohejl, Hitomi Yanaka
Abstract
Semantic relations organise word senses and are fundamental to lexical semantics. In order to evaluate semantic relation knowledge of large language models (LLMs), we propose a challenging task in which an LLM predicts the relation between two glosses (definitions of senses) presented in two different orders. The predictions for the two orders are evaluated for consistency with the symmetry or asymmetry of semantic relations. Our task is designed for both monolingual and cross-lingual evaluation, allowing us to examine whether relation knowledge remains accessible when the same senses are verbalised through glosses in different languages. We construct a dataset of 3,069 sense pairs, with each sense verbalised in English, Greek, Romanian, Turkish, and Japanese. Experiments with 17 LLMs in zero-shot and five-shot settings reveal generally poor performance. In each setting, only eight LLMs achieve a macro-averaged F1 score significantly above a random baseline, and only one (zero-shot) and two (five-shot) significantly outperform a neural baseline. In particular, LLMs often fail to generate bidirectionally consistent predictions for hypernymy and to make correct predictions for holonymy in either direction. Our results suggest limitations in current LLMs' semantic relation knowledge, both in identifying relations and in respecting asymmetry.