Multilingual large language models show uneven text similarity across languages
The Multilingual Artificial Hivemind: Do All LLMs Think Alike in Every Language?
Computation and Language
Summary
Large language models (LLMs) tend to produce text that is less diverse and more similar to each other than human writing, a phenomenon called the artificial hivemind effect. This study checked if this effect also happens in languages besides English by testing 23 models and 13 languages. The researchers found that all models showed some similarity in their outputs across languages, but the degree varied depending on the language and the model’s skill in that language. Better-performing languages saw more text similarity, revealing a tradeoff between how good a model is in a language and how diverse its generated text is.
What this means in practice
- •For multilingual ai developers: Design multilingual language models that balance output diversity with language proficiency to improve user experience across different languages.
- •For content moderation teams: Assess risks of homogenized automated content across multiple languages to better detect and manage repetitive or low-diversity outputs.
Authors
Xaver Raphael Davey, Shubhra Mishra, Anton Adelöw, Siyao Liu
Abstract
Text generated by Large Language Models (LLMs) often remains limited in diversity compared to that produced by humans. Recent advances demonstrate the presence of inter- and intra-model collapse in open-ended generation of English text, raising concerns about the long-term homogenization of LLM, and more importantly, human thought. However, our understanding of whether this homogenization extends to languages beyond English is limited. In this work, we investigate the role of language on output diversity. Specifically, we aim to understand whether multilingual LLM-generated text is also homogeneous, and whether this homogeneity matches what we observe in humans. Across 23 open-weight models and 13 languages, we find that the artificial hivemind effect is pervasive but uneven in its extent. The intra-model effect holds in three different embedding spaces and under a lexical metric that uses no embedding model, so it is not an artifact of how diversity is measured. The size of the homogenization gap is associated with a model's competence in a language, measured on a held-out translation benchmark, robustly across embedding spaces. Thus, we demonstrate a performance-diversity tradeoff in multilingual LLMs.