Language models show gaps in Haitian Creole cultural understanding
Evaluating Cultural Awareness of LLMs for Haitian Creole
Computation and LanguageArtificial Intelligence
Summary
Large language models often struggle to understand and represent the culture of less common languages. The authors studied how well these models capture Haitian Creole culture and found they perform unevenly and often mix in French influences. They also noticed that stories about Haitian characters tend to focus on hardship and resilience, which could reinforce stereotypes even when positive. This work provides tools and examples to better evaluate cultural sensitivity in language technology for Haitian Creole speakers.
What this means in practice
- •For language technology developers: Improve Haitian Creole language model training by identifying cultural gaps and interference from French to produce more accurate culturally aware outputs.
- •For content creators for haitian audiences: Use evaluation tools to assess and reduce stereotypical narratives in AI-generated stories involving Haitian characters, ensuring diverse cultural representation.
Authors
Christelle Clervilsson, Yanzhu Guo
Abstract
Large language models (LLMs) exhibit substantial performance disparities between high- and low-resource languages. Beyond lower task performance, they often fail to capture the cultural norms and values of underrepresented communities. In this work, we present the first systematic evaluation of cultural awareness in LLMs for Haitian Creole, a language spoken by millions but severely underrepresented in digital resources. We assess cultural awareness along four complementary dimensions---specificity, bias, diversity, and variation---using a benchmark of culturally salient prompts curated by native speakers in a text infilling setting. Our results reveal a clear gap between cultural awareness in Haitian Creole and higher-resource French, with Haitian performance being more uneven across domains and more affected by French linguistic interference. Story generation further reveals recurring portrayals of Haitian characters through hardship and resilience, showing that even positive characterizations can encode stereotypical narratives. Our code, benchmark, and evaluation framework are publicly available.