Persona effects in language models predict but do not transfer well
How Far Do Persona Effects Generalize in Language Models?
Computation and LanguageArtificial Intelligence
Summary
Language models can be prompted to respond like certain kinds of people, called personas. The authors show that while these persona effects can be predicted for new questions within the same model, they do not transfer well across different language models or prompt styles. Rewording prompts helps, and tuning the models for target tasks outperforms simply reusing effects from other models. The findings highlight limits in generalizing persona behavior across models and the need for targeted calibration.
What this means in practice
- •For chatbot developers: Improve conversational agents by fine-tuning persona effects rather than relying on transferring them across different models or prompts.
- •For customer service platform engineers: Adjust chatbot persona behaviors specifically per task and context to better align with user expectations and reduce errors in persona consistency.
Authors
Yufan Zhou, Yuxuan Liu, Enze Ma, Lyumanshan Ye, Zhongqi Yue, Robin De Croon, Yucheng Jin, Katrien Verbert, Zhao Wang
Abstract
Persona prompts ask language models to answer as particular kinds of people. We test whether relationships learned from these effects predict responses to new questions and remain useful across models and prompts. Across 57 attributes, three behavioral domains, and seven pairs of open 7 to 9B checkpoints, persona effects can be predictable without being portable. Separate attribute and task gains improve prediction beyond shared scaling significantly in OLMo-3 and Qwen2.5, with the most robust evidence in OLMo-3. In that model, target refitting significantly outperforms gains borrowed from each of the other six pairs. Across model transfers, borrowed gains with one amplitude underperform shared scaling in most directions; allowing two target parameters removes the significant losses but yields no significant benefit over target shared scaling. After rewording, refitting significantly outperforms reuse with one amplitude in all six tested pairs, while changes of examples or country context often preserve reuse value. In the tested prompt transfers, regularized updates outperform both reuse strategies in median at 64 target questions per attribute. A separate survey comparison finds that selecting the more responsive checkpoint can worsen human fit; responsiveness is confounded with training status, and temperature calibration largely removes this cost but not errors in group ordering. Within the tested gain representation, apparent transfer can come from target calibration; source relationships must add predictive value beyond calibration and regularization. Code and data are available at https://github.com/thzva/persona-gain