Language models show hidden racial bias in trusting information sources
Invisibilization as a Source of Probability Bias in Language Models
Computation and LanguageComputers and Society
Summary
Language models are designed to avoid using sensitive information like race in their answers, but this can cause them to ignore important information based on who it comes from. The authors measured how much these models refuse to trust reports depending on the described race of the source, finding that models often ignore reports from sources described as Black more than those described as white. This bias is hard to detect because it occurs in when the model refuses to evaluate the information at all, and it gets worse as the model tries harder to reason. The authors also developed a scoring method that reduces this refusal bias over repeated interactions.
What this means in practice
- •For ai ethics teams: Detect and mitigate hidden racial bias in trust decisions made by language models to improve fairness in AI responses.
- •For chatbot developers: Incorporate scoring methods that reduce refusal rate bias when language models assess unverifiable information from diverse sources.
Authors
Christian Arnold, Ernesto Calvo, Guadalupe Gonzalez, Sebastián Vallejo Vera, Tiago Ventura
Abstract
Language models are aligned to avoid using protected attributes such as race, ethnicity, and gender to inform their responses. Therefore, they increasingly refuse or hedge when such attributes appear in a prompt. However, if a system must decide how much to learn from what people tell it, it is costly to refuse valuable information because of its source. In this article, we measure the cost of declining information in a two-urn game with an exact Bayesian answer. We present leading LLM models from two developers (OpenAI and Anthropic) with a prompt that requires them to decide how far to trust an unverifiable report from a participant whose description we randomize across ten configurations (6,600 sessions). We show that leading models often decline to assess the reporter, treating information as worthless at rates set by the description rather than by the evidence. At its widest, models decline a Black-described reporter 46% of the time and a white-described reporter 9%; yet, when they do assess the information, the Black-described reporter receives slightly more trust. The bias is invisible to audits that compare average trust or look only at assessed cases. It grows with reasoning effort, follows words rather than categories (models invoke race for "Black" but not for its Spanish translation negro, and never for "Indigenous"), and worsens under warning instructions a practitioner would try first. We also present a scoring option that reduces the refusal rate bias. When a model's trust is scored against revealed truth over repeated rounds, it credits identical records equally. In four of five configurations from Anthropic and OpenAI, the racial gap in trust falls below 0.02 by the last rounds.