Large language models aligned with multiple human values via attention heads
Towards Multi-Value Alignment via Value Localization in LLMs
Artificial Intelligence
Summary
Aligning AI language models to follow different human values at the same time is difficult, especially when those values sometimes conflict. The authors found specific parts inside these models, called value-sensitive attention heads, that control how the model behaves according to different values like honesty or harmlessness. By studying and manipulating these parts, they can make the AI follow one or multiple values without retraining it. Their method makes the AI's decisions more understandable and easier to control.
What this means in practice
- •For ai system developers: Control language models to behave according to specific human values or style preferences at inference time without expensive retraining.
- •For content moderation teams: Use value-specific model interventions to reduce harmful or dishonest outputs while preserving overall model performance.
Authors
Xueqi Ma, Yanbei Jiang, Xingjun Ma, James Bailey, Sarah M. Erfani
Abstract
Aligning large language models (LLMs) with multiple, potentially conflicting human values remains challenging, as existing multi-objective alignment methods often require costly fine-tuning and provide limited insight into the internal mechanisms underlying value-related behaviors. In this work, we investigate multi-value alignment from a mechanistic interpretability perspective by identifying value-sensitive attention heads (value heads) in LLMs. Using a probing-based value localization framework, we discover sparse sets of attention heads associated with diverse dimensions, spanning both human values (e.g., harmlessness, honesty) and stylistic preferences (e.g., coherence, complexity). Analyses across multiple LLM families reveal functional specialization and structured interactions among value heads, including conflicts between certain values. Targeted head ablations further induce substantial, value-specific behavioral changes, providing causal evidence of their functional importance. Building on these findings, we propose an inference-time activation steering strategy that enables controllable alignment with individual values or combinations of values without updating the underlying model parameters, supported by theoretical analysis. Experiments across multiple value alignment benchmarks demonstrate effective single- and multi-value control while largely preserving non-target model performance. Our findings connect mechanistic interpretability with flexible multi-value alignment, offering a foundation for more interpretable and pluralistic LLM behavior. The code is available at https://github.com/sihuo-design/ValueHeads.