Large language models show distinct psychological response profiles across languages
Measuring Behavioural Signatures of Large Language Models through Psychometric Profiling
Computation and LanguageArtificial Intelligence
Summary
Large language models (LLMs) such as AI chatbots are used to help people communicate and make decisions, but understanding their typical behaviors is hard. The authors tested nine different LLMs using psychological tests in English and Chinese to see patterns in their answers, including when they refuse to answer. They found that each model shows a unique behavioral pattern influenced by the language used and the model’s origin, with consistent results over repeated tests. This approach could help measure how these AI models behave in different situations.
What this means in practice
- •For ai deployment teams: Differentiate AI models based on behavioral profiles to choose suitable models for specific communication tasks and languages.
- •For content moderation teams: Identify where language models tend to refuse or avoid certain responses to improve content filtering and safety measures.
Authors
Yu Sha, Junqi Tao, Dixin Zhou, Yansheng Tu, Mingyang Chen, Xiang Fan, Yang Liu, Mengquan Yang, Jie Lin, Jiahui Fu, Hua Zheng, Benwei Zhang, Zhou Kai
Abstract
Large language models (LLMs) increasingly mediate human decisions and communication, yet their behavioural regularities remain difficult to characterize systematically. We develop a cross-linguistic psychometric profiling framework and evaluate nine LLMs using seven psychological instruments, with five repeated administrations per model and language in Chinese and English. Items unresolved after a prespecified retry procedure are retained as NA. Joint analysis of scored and NA responses captures response tendencies and boundaries of self-report applicability. LLMs exhibit structured, model-specific profiles despite a shared alignment-shaped pattern of higher prosocial and self-regulatory responses and lower dominance, disengagement and harmful-intent endorsement. NA responses are structured rather than uniformly distributed, indicating where outputs are treated as inapplicable, refused or cannot be mapped to valid response options. Language condition and provider origin are associated with profile configuration and answerability, whereas repeated administrations show high reproducibility and permit recovery of model identity. Human-reference and prompt-robustness analyses further indicate that these signatures are context dependent. Joint analysis of psychometric profiling and answerability offers a framework for quantifying deployment-level behavioural signatures.