Model activations predict how value training changes AI behavior across contexts
Predicting Alignment Generalization with Value Representations
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
AI developers try to make large language models (LLMs) follow good values by fine-tuning them on examples. But teaching a model one value can unexpectedly affect how it behaves with other values later. The authors study how well we can predict these side effects using different ways to represent values. They find that looking at the model’s own internal reactions during fine-tuning works much better than just reading descriptions of the values. They also find hints of a shared way to describe values across models, which could help design safer AI.
What this means in practice
- •For ai alignment engineers: Predict how fine-tuning models on specific values affects their behavior on untrained values to improve safe AI development.
- •For ai safety auditors: Assess similarity of values within alignment targets to estimate model robustness against unexpected value shifts during deployment.
Authors
Andy Liu, Mehar Bhatia, Karolina Stanczak, Mona Diab, Vered Shwartz, Daniel Fried
Abstract
LLM developers post-train their models to exhibit prosocial values and behavioral traits, which are enumerated in an alignment target. However, while recent post-training developments have yielded models that score highly on alignment evaluations, training models on sets of narrow behaviors still influences their behavior across unseen contexts and environments in unexpected ways. In this paper, we establish the task of alignment generalization prediction, i.e., predicting how fine-tuning a model to follow a given value changes its behavior across a wide range of held-out values. We conduct a large-scale analysis of alignment generalization effects across 66 values found in modern alignment targets, and benchmark representational techniques on the alignment generalization prediction task. We find that representations based on model activations when applying values in context significantly outperform methods based on textual descriptions of the values. Specifically, the best activations-based methods achieve correlations of 0.45 with our generalization matrix, compared with 0.05 from description-based baselines. We then show the applicability of representations that predict alignment generalization toward downstream tasks by using them to measure how similar the values in a multi-value alignment target are, which we find is significantly correlated with model robustness. Finally, we show initial evidence towards a shared, model-independent value space, which we use to develop the first taxonomy of LLM values grounded in empirical generalization dynamics. Our work demonstrates the importance of studying value generalization in LLMs and its application toward the more empirical design and training of model behavior.