Post-training methods affect large language models value preferences differently
How Does Post-Training Shape LLM Value Preferences?
Computation and Language
Summary
Large language models (LLMs) learn from text that contains opinions from many groups. The balance of these opinions can shape the models' values in different ways during a stage called post-training. The authors found that different post-training methods can increase or decrease these opinion biases differently. They also discovered that once certain values are learned, it's not always easy to change them later, and these values get passed on when making smaller, student models. This means that both the choice of training data and the method matter for how these AI systems develop their value preferences.
What this means in practice
- •For llm developers: Choose post-training data and methods to better control value biases in language models before deployment.
- •For ai ethics teams: Assess how training and fine-tuning processes affect the representation of diverse opinions in large language models.
Authors
Yang Liu, Shaoyang Xu, Masahiro Kaneko, Chengzhi Zhong, Wenxuan Zhang, Chenhui Chu
Abstract
During post-training, large language models (LLMs) learn value preferences from corpora that may contain imbalanced proportions of opinions associated with different demographic groups. Amplifying these imbalances may further underrepresent some groups' opinions. However, how opinion mixtures shape value preferences across post-training paths remains unclear. To answer this question, we conduct controlled experiments using post-training corpora with different opinion mixtures and five post-training paths based on SFT, DPO, and GRPO. We find that post-training amplifies opinion imbalances differently across post-training paths. Once value preferences are amplified, whether further post-training can remove them depends on both the post-training method and the opinion mixture in the corpus. More concerningly, a model's stability under interventions depends on whether further post-training changes its value preference. Moreover, these value preferences are transferred to student models even when distillation uses corpora with no explicit opinion questions. These findings suggest that both the opinion mixture in post-training corpora and the choice of post-training method should be considered when developing, evaluating, and reusing LLMs. Our code and data are available at https://github.com/ku-nlp/llm-value-preferences.