Language models show varying sentiment direction consistency across languages

Concept Direction Reliability Across Languages with Different Tokenizer Fertility

Computation and Language

Summary

The authors studied how consistent the way language models represent sentiment directions is across different languages. They found that even when models accurately classify sentiment, the direction of the sentiment signals can vary a lot between language samples, especially for less common languages like Yoruba compared to English. This means that just because a model is good at judging sentiment doesn’t mean its internal signals are stable. They also discovered that some methods for combining word parts led to less consistent results, and factors like sentence length played a role. The authors stress that checking direction consistency is important on its own, beyond just looking at accuracy.

What this means in practice

Authors

Muhammad Abdullahi Said, Abass Oguntade, Elisha Komolafe, Babangida Sani, Fatima Muhammad Adam, Muhammad Sammani Sani

Abstract

Extracted sentiment directions can vary across samples even when downstream sentiment classification remains accurate. To evaluate direction reproducibility, we measure split-half agreement in English, Hausa, and Yoruba representations across four language models using both native and translated texts. We identify layers selected for agreement using ten topics and evaluate direction agreement across separate groups of fifteen topics. Using the final token, split-half agreement ranges from 0.737 to 0.870 for English, 0.589 to 0.762 for Hausa, and 0.101 to 0.399 for Yoruba, maintaining this language rank order across all 77 complete model comparisons. Classifiers trained on these same layers consistently predict sentiment above chance, demonstrating that predictive accuracy does not imply directional consistency. Furthermore, averaging token representations yields less consistent agreement, and high agreement can partially reflect sentence length. Ultimately, our findings highlight the need to measure vector direction reproducibility independently of classification performance, though they do not establish that tokenizer fertility which is the average number of tokens per whitespace separated word causes cross-lingual differences.
View PDFOpen arXiv