Dataset captures diverse Nepali youth views on migration for AI analysis
NepPlural: A Human-Anchored Benchmark for Persona Annotation in Nepali Migration Discourse
Computation and Language
Summary
Language models often fail to represent different cultural views well, especially those outside Western societies. The paper introduces NepPlural, a collection of Nepali YouTube comments about youth migration, labeled by native speakers to capture different social viewpoints or 'personas.' The authors tested how well AI models can identify these personas and found that models do better when allowed to say 'no persona' applies. The new dataset and findings highlight challenges and methods for making AI understand diverse cultural perspectives in Nepali social issues.
What this means in practice
- •For natural language processing teams: Train AI systems to better recognize and represent varied social viewpoints within Nepali public discussions about migration.
- •For multilingual chatbot developers: Improve chatbot understanding and response to Nepali speakers by incorporating nuanced persona detection in mixed-script, multilingual comments.
Authors
Jinu Nyachhyon, Anish Khatiwada, Bimal Timilsina
Abstract
Pluralistic alignment asks that language models faithfully represent a society's diverse, competing values rather than collapse them into one dominant viewpoint, yet flagship pluralistic-alignment datasets remain heavily skewed toward Western respondents. We introduce NepPlural, a human-anchored benchmark for modelling competing socio-cultural personas in Nepali public discourse on youth migration and brain drain. We curate 2,148 code-mixed (Devanagari, Romanized Nepali, English) YouTube comments, annotate them along a four-axis persona taxonomy (Intent, Primary Driver, Value Orientation, Affect) plus an explicit No-Persona class, and build a 300-comment gold set labelled independently by three native Nepali speakers (mean Fleiss' $κ\approx 0.55$). Using this gold set to audit automatic annotation, we find that LLM annotation reliability is driven by protocol. Labels collected in batches through chat interfaces reach only near-chance inter-model agreement ($κ\approx 0.17$). Annotating one comment at a time raises agreement above the native-speaker level ($κ\approx 0.72$), but consistency is not validity: under a forced-choice schema, models agree on personas for the 39% of comments native speakers judge to express none, keeping macro-F1 against human gold near 0.50. Allowing abstention resolves this: models recover No-Persona comments with F1 of 0.79 to 0.87 and, against leave-one-annotator-out references, match or exceed individual native speakers (macro-F1 0.75 to 0.86 vs. 0.75). Encoders fine-tuned on the batched labels reach only 0.34. Finally, a human-judged steerability probe finds that three frontier models adopt single persona axes with 59 to 77% fidelity but compose full personas only 33% of the time, with no congruence gap detected at this scale between Western-congruent and Nepali-collectivist personas.