How Does Alignment Tuning Shape Representations of Sycophancy and Related Cue-Induced Biases in LLMs?
2026-07-20 • Computation and Language
Computation and LanguageArtificial IntelligenceMachine Learning
AI summaryⓘ
The authors studied why large language models (LLMs) often change their answers when given small, irrelevant hints in prompts. They found that these biases mostly appear after a fine-tuning process called alignment tuning, not during the initial training. Each type of bias corresponds to a specific direction in the model's internal workings, which can be identified and used to correct the bias. This means the biases are separate and can be reduced without hurting correct answers. Overall, the authors suggest these biases come from alignment tuning and act like different 'directions' in the model's behavior.
Large Language Models (LLMs)Alignment TuningBiasHidden StatesFew-shot LearningCausal InterventionProbingBehavioral Cue Tuning (BCT)DebiasingModel Fine-tuning
Authors
Prakhar Gupta, Terry Jingchen Zhang, Florent Draye, Bernhard Schölkopf, Zhijing Jin
Abstract
Modern LLMs are alarmingly susceptible to surprisingly simple immaterial changes of input prompts: a casual hint, an incorrectly labeled few-shot example, or a fake prior assistant turn often flips an originally correct answer. We study where this susceptibility, spanning sycophancy and related cue-induced biases, lives inside the model. Across five model families and seven BCT bias types, we extract a per-bias direction from hidden states and triangulate it through three measures: probing, leave-one-dataset-out transfer, and causal intervention. The susceptibility is largely installed by alignment tuning rather than pretraining: pretrained base models barely cave to these biases, and their activations carry no cue-specific signal beyond question content. Within aligned models, each bias becomes a single coherent direction that we can both decode and steer along, recovering the unbiased answer across every family we test. The biases stay representationally distinct, however: cross-bias entanglement is model-specific rather than a property of the bias category, and even behaviorally similar biases occupy different directions. The same intervention also serves as a modest debiasing tool, recovering a meaningful share of bias-induced errors while preserving most correct answers across all instruct families. Cue-induced bias is therefore best understood not as a single flaw in LLMs but as a family of distinct, causally active directions that alignment tuning installs.