Automated dataset reveals large language model bias toward Anglophone culture
WikiOriginQA: Automated Construction of a QA Dataset with Knowledge Origins for Analyzing Cultural Bias
Computation and Language
Summary
Cultural knowledge in AI is often hard to study because making datasets for different cultures takes a lot of manual work. The authors created a new dataset, WikiOriginQA, by using the history of when Wikipedia pages appeared in different languages to automatically figure out where knowledge comes from. They found that large language models (LLMs) tend to perform better with information coming from English-speaking cultures. This shows the dataset can help test cultural bias in AI much more easily than before.
What this means in practice
- •For ai product developers: Evaluate how AI models handle cultural knowledge biases to improve fairness in multilingual applications.
- •For localization engineers: Test AI systems for cultural knowledge gaps to better adapt software for diverse language communities.
Authors
Kento Haneda, Yosuke Kishinami, Ryo Fujii, Keisuke Sakaguchi, Makoto Morishita
Abstract
Existing datasets on cultural knowledge have been typically created by human annotators familiar with each culture. However, the process requires considerable human effort, presenting a significant challenge to scalability. To address this challenge, we introduce WikiOriginQA, a QA dataset automatically constructed by objectively representing the origins of knowledge through the creation dates of different Wikipedia language editions. Experimental results with the created dataset confirm that LLM performance tends to be biased toward Anglophone cultures. Furthermore, the results suggest the importance of knowledge propagation into the Anglosphere. These findings demonstrate the validity of our automated approach, enabling cultural bias evaluations with superior scalability across diverse languages without extensive manual annotation.