Language models share one circuit that causes multiple cue biases
Many Biases, One Circuit: Post-Training Tunes a Pretrained Cue-Routing Mechanism
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
Language models can be tricked into choosing wrong answers when prompts contain misleading hints, like "I think the answer is B." The authors found that these mistakes often come from the same few parts (attention heads) inside the model. This part exists even before special tuning, but certain training steps make the model rely on it more and make the mistake worse. Changing these parts can reduce how much the model follows the wrong cues. Understanding this could help fix these bias problems.
What this means in practice
- •For ai developers: Adjust training procedures to target and reduce cue-following biases by modifying key attention heads identified in the model.
- •For chatbot builders: Improve chatbot reliability by identifying and controlling circuits that cause the model to blindly follow misleading cues in user prompts.
Authors
Prakhar Gupta, Terry Jingchen Zhang, Zhijing Jin
Abstract
Language models often abandon a correct answer when the prompt contains a cue, a hint toward a wrong option such as "I think the answer is B." Sycophancy and related biases show this failure. They are often studied separately and blamed on alignment. We test this view causally, tracing the attention heads behind these biases through the stages of two post-training pipelines. In multiple-choice tests on most models, four cue types rely largely on the same few heads, a shared circuit already present in pretrained models. Most base models follow cues in plain text. Fine-tuning a base model on about 200 generic assistant examples without cues activates these heads in chat format, so cue-following can be a side effect of assistant tuning. The final reinforcement learning (RL) stage strengthens the effect of cues in almost every comparison. In Tulu-3-8B and OLMo-2-7B, we swap the attention patterns of six middle layers between the RL checkpoint and its predecessor. For suggested-answer cues, this shifts each checkpoint's cue effect most of the way to the other's level. Post-training thus seems to change when and how strongly models use this circuit. Mitigation for these multiple-choice biases could target this circuit.