Human-LLM Alignment in Language Attitudes Toward Non-Native Japanese
2026-08-03 • Computation and Language
Computation and Language
AI summaryⓘ
The authors studied how human raters and large language models (LLMs) judge emails written by native (L1) and non-native (L2) Japanese speakers based on fluency, status, and solidarity. Humans rated non-native texts lower on all dimensions, especially fluency. LLMs showed similar biases but underestimated the difference in solidarity and uniquely distinguished among L2 speakers' backgrounds. This shows LLMs imitate human language biases in a consistent but reduced way, and the authors suggest using language attitudes to evaluate such biases in many languages.
Large Language ModelsNative Speaker (L1)Non-native Speaker (L2)Language AttitudesFluencyStatusSolidarityBias in Language EvaluationJapanese LanguageHuman vs AI Evaluation
Authors
Naho Orita, Hayato Ogawa, Daisuke Kawahara
Abstract
Large language models (LLMs) increasingly evaluate human writing in high-stakes domains such as hiring and academic assessment, putting non-native speakers at particular risk. Drawing on the language attitudes framework, we compared human and LLM evaluations of parallel L1- and L2-written Japanese emails on three dimensions: fluency, status, and solidarity. Japanese raters rated L2 texts significantly lower on all three dimensions, with a fluency gap roughly twice the size of the status and solidarity gaps. Six LLM judges reproduced the direction of this bias, and five reproduced its ordering across dimensions. The models diverged from humans in two ways: all understated the solidarity gap, the most socially grounded dimension, and all differentiated among learner L1 backgrounds where humans did not. LLM judges thus reproduce native speakers' language attitudes in a structured yet attenuated form, and the language attitudes framework offers a ready-made yardstick for auditing them beyond English.