Bengali English translations reveal bias in honorific use by language models
Who Receives Respect? A Counterfactual Benchmark for Honorific Allocation Bias in Bengali
Computation and Language
Summary
Some languages like Bengali always use polite or respectful words when talking about others, while English doesn’t. The authors studied how computer programs that translate English to Bengali decide when to be respectful. They discovered that these programs treat different jobs with very different respect levels, but show much smaller differences based on gender. They created a score to measure how fairly respect is given, and found big differences between different translation models.
What this means in practice
- •For machine translation developers: Detect and reduce occupation and gender bias in English to Bengali translation honorific choices produced by language models.
- •For localization teams: Assess how respectful language is distributed across occupations in Bengali translations to improve cultural appropriateness.
Authors
Ajwad Abrar, Md Kamrul Hasan, Hasan Mahmud
Abstract
Bengali has no grammatical gender, but honorificity is obligatory: every third-person reference must choose ordinary or respectful register. English has no such signal, so the system must choose a level of deference that the source never specified. We use this property to measure how large language models (LLMs) allocate respect in English--Bengali translation. Our evaluation set contains 2{,}016 sentences across 72 occupations. Each appears in seven sentence frames under four pronoun conditions. Masculine and feminine versions differ by one word, forming counterfactual pairs. We report a single Honorific Bias Score (HBS) for each model on a 0--100 scale, covering both the gender and occupation axes. A lower score means that respect is allocated more evenly. For the occupation analysis, we compute each occupation's honorific rate across the seven no-pronoun sentence frames. The standard deviation of these 72 occupation-level rates ranges from 17.2 to 35.5 percentage points across models, showing substantial variation in how honorifics are allocated across occupations. Occupation is associated with 3.2\% to 51.5\% of the variance in register choice, compared with at most 1.2\% for gender. Gender effects are much smaller overall: only two of the nine models show significant directional gender gaps, and they favor opposite genders. HBS ranges from 19.9 to 70.1 across the nine models, showing substantial variation in how evenly the models allocate respect.