How Do Large Language Models Judge Social Attraction? Evidence from Theory-Grounded Persona Ratings Across Multiple LLMs and Humans
2026-08-10 • Computation and Language
Computation and LanguageArtificial IntelligenceComputers and Society
AI summaryⓘ
The authors studied whether large language models (LLMs) can judge how socially attractive people seem based on detailed personality profiles. They found that LLMs are consistent and agree on ranking profiles into socially attractive, mixed, or unattractive groups, and that gender presentation did not affect LLM ratings. When compared with human judgments, both humans and LLMs agreed on the overall ranking of profiles, but LLMs gave stronger positive or negative ratings than humans. Neither LLMs nor humans showed bias based on gender presentation. This suggests LLMs can mimic human social judgments with some differences in rating intensity.
Large Language ModelsSocial AttractionPersona ProfilesPsychological ConstructsRelational ConstructsSubjective EvaluationGender PresentationHuman JudgmentRating StabilitySocial Perception
Authors
Hasan Mahmud, Khawaja Abaid Ullah, Mohammad Javad Khojasteh, Jamison Heard, Prabu David
Abstract
Large language models (LLMs) are increasingly used to perform subjective evaluations traditionally made by humans, yet their validity as social judges remains unclear. This paper examines whether LLMs can assess social attraction from theory-grounded persona profiles constructed from ten psychological and relational constructs and organized into three tiers: socially attractive, socially mixed, and socially unattractive. We examine LLM ratings in two studies and compare them with human judgments in a third study. In Study 1, 34 LLMs rated 12 profiles across three repeated runs. Although some models tended to give higher or lower ratings overall, they showed strong stability across runs, consistent three-tier ordering, and high agreement in relative profile ordering. Study 2 examined sensitivity to gender presentation using six matched name-and-pronoun profile pairs and a separate pronoun-only test with a gender-neutral name, finding no significant effects in either analysis. In Study 3, 198 human participants evaluated the six matched profiles from Study 2. Their ratings reproduced the three-tier structure and followed a profile ordering consistent with that of the LLMs. However, LLMs rated attractive profiles more positively and unattractive profiles more negatively than humans, while neither group showed a significant overall effect of gender presentation.