Large language model bias cannot be clearly tracked to specific attention heads

What Does a Bias Head Measure? Non-Identifiability in Head-Level Bias Attribution

Computation and LanguageMachine Learning

Summary

People try to find social bias in big language AI models by looking at parts called attention heads. The paper shows that measures used to find biased heads actually track other things like how long the AI's answers are or which question was asked. Because these factors mix together, it’s impossible to say that certain heads really measure bias. Also, removing those heads does not consistently reduce bias, so current methods to identify biased parts of the model are not reliable.

What this means in practice

  • For ai ethics teams: Avoid relying on current head-level bias metrics when auditing language models for fairness due to confounding factors.
  • For language model developers: Design better methods to identify and mitigate bias in models without mistakenly attributing bias to specific attention heads.

Authors

S. M Seefat Alam

Abstract

Methods that localise social bias in large language models to individual attention heads and remove it by ablation are validated by their effect on a fairness metric, never by evidence that the statistic selecting the heads measures bias. We audit this on Qwen-2.5-7B-Instruct and Llama-3.1-8B-Instruct. The head map correlates at $ρ= -0.950$ and $-0.768$ with a label-blind response-length map, apparently identifying a surface confound. An intervention that breaks the length-label coupling rejects this reading: with the prompt held fixed, response length explains 1% of the statistic's variance. What does explain it is prompt identity, at 64% of variance, in an ordering that is not monotonic in length. Within any fixed prompt the fairness label, response length and the presence of a protected-group token are collinear at $φ= 0.95$, partly by definition of the labelling rule, so no observational analysis can attribute the map to any of them, and no method in this family performs an experiment that could. The substantive claims fail independently: bias selectivity is not late-layer-concentrated on either model under either reference reading, and no head set we tested reduces the unfairness rate when ablated. Its complement, scored on the opposite label, reliably destroys refusal behaviour ($p = 2.7\times10^{-5}$), demonstrating that the pipeline detects effects at this scale.