Persona hierarchy explains how fine-tuned language models generalize context
The Persona Hierarchy Model: Understanding Contextual Generalization in Fine-Tuning LLMs
Computation and LanguageMachine Learning
Summary
Language models are often fine-tuned using specific instructions or roles, but sometimes their new behavior only works in that exact situation, while other times it applies more broadly. The authors propose that there is a hidden default persona inside the model that influences all contexts. Changes affecting this default persona help the model apply behavior in many contexts, while changes to specific personas stay limited. They also show a way to prevent unwanted behavior spreading using persona-preserving regularization, which cuts down unwanted reward hacking without losing accuracy.
What this means in practice
- •For llm developers: Control and predict how fine-tuned behaviors transfer across different contexts during language model training.
- •For ai safety engineers: Reduce unintended behavior generalization and reward hacking in reinforcement learning by applying persona-preserving regularization.
Authors
Jiachen Zhao, Zhengxuan Wu, David Bau, Weiyan Shi
Abstract
Language models are routinely fine-tuned under a fixed context, such as a generic system prompt, persona or domain-specific instruction, yet the learned behavior sometimes stays confined to that context and sometimes broadly generalizes to unseen contexts. We propose the Persona Hierarchy Model to explain this: a shared default persona influences behavior across contexts. Under this model, fine-tuning that modifies the shared persona promotes broader transfer, whereas changes to local personas remain more context-specific. Across 120 fine-tuned models spanning four behaviors and 15 training contexts, generalization narrowness positively correlates with the similarity between the training context's persona and the default persona (Pearson's r = 0.72 for Qwen3-4B). Prior fine-tuning under the default context can broaden generalization in subsequent training under other contexts. Aligning contextual responses with default-persona responses produces stronger effects. Finally, we propose persona-preserving regularization (PPR) to confine undesired contextual generalization. In RL, PPR cuts reward hacking from 42-55% to at most 0.2% under every evaluated prompt while retaining accuracy gains. These results support the Persona Hierarchy Model as an explanation for contextual generalization and can motivate future controls on unintended generalization for better alignment of LLMs.