A Mechanistic Understanding of Pronoun Fidelity in LLMs
2026-06-15 • Computation and Language
Computation and LanguageMachine Learning
AI summaryⓘ
The authors studied how large language models handle pronouns when different people are mentioned with different pronouns. They looked inside the models to find three main reasons why pronouns are used the way they are: grouping related words together, recent word use, and stereotypical associations. These three reasons work together and explain almost all of the models' behavior. They also found two ways the model copies pronoun references: one that connects occupations to pronouns and another that repeats recent words. Overall, pronoun usage depends on which of these internal mechanisms is stronger at a given time.
pronoun fidelitylarge language modelsgroup entity bindingrecency biasstereotypical biascausal subspacesBoundless Distributed Alignment Searchattention headsconcept-level routetoken-level route
Authors
Katharina Trinley, Jesujoba O. Alabi, Dietrich Klakow, Vagrant Gautam
Abstract
Faithful and robust pronoun use is important for fair and coherent generations, yet large language models largely fail when multiple referents use different pronouns. To study the interplay of reasoning, repetition, and bias in this task, prior work relies exclusively on behavioural approaches, which may not reflect a model's internal workings. Therefore, we provide a mechanistic, model-internal perspective on pronoun fidelity, testing whether three mechanisms -- group entity binding (G), recency bias (R), and stereotypical bias (S) -- are causally implemented across several SOTA language models. Using Boundless Distributed Alignment Search, we find all three coexist as causal subspaces distributed across network depth. No single mechanism fully explains model behaviour, but a combination of the three consistently accounts for 91-99.5%. An attention head analysis further reveals two competing copying routes; group binding and stereotype share a localized concept-level route that retrieves a bound occupation-pronoun unit, while recency uses a distributed token-level route that repeats surface forms. In sum, pronoun fidelity arises from competition between simultaneously active causal subspaces.