Language models show consistent moral judgments across languages with varied yes answers
Same Judgment, Different Yes: Measuring Multilingual Judgment Profiles in Language Models
Computation and LanguageArtificial Intelligence
Summary
When language models answer yes or no questions in different languages, their moral judgments stay mostly the same, even if they use different ways to say yes. The authors studied millions of answers from several models and found that how the model says yes or no can change without changing the actual judgment. This means scoring a simple yes or no word can be tricky because it reflects language habits as well as the model's opinion.
What this means in practice
- •For natural language processing teams: Improve multilingual language model evaluation by distinguishing true judgments from language-specific answer patterns in moral reasoning tasks.
- •For ai chatbot developers: Design chatbots that maintain consistent moral decisions across languages despite differences in their simple affirmative or negative responses.
Authors
Haonan Huang, Yuwei Lu, Chengkun Cai
Abstract
Multilingual evaluations often score a model's opening yes or no, a token that also carries model and language answering conventions. We ask when different ways of eliciting and reading an answer recover the same judgment, and when they measure an answer habit. We collect 1.1 million replies from eight models in six languages on up to 105 moral vignettes, varying the question's form, the answer constraint, and the reading of the reply, token or sentence, in both wordings (the FAR-6 battery). Story-level judgment profiles converge across the tested readouts and question forms, except for the yes/no token after a negative interrogative read under a preset mapping: on the same replies, token and sentence profiles agree at a median r = 0.67 across cells, while sentence profiles agree with free-sentence profiles at 0.94, a known linguistic variability. With each language's story consensus removed, a model's profile is nearest to its own in another language for 39 of 48 model-language cells under proposition labels and 15 under free sentences, against 5 expected by chance. Particle, position, and named-term preferences follow their own patterns. A model can keep its judgment and change its yes: scoring that token requires establishing which proposition it affirms.