LLM models differ in truth bias from repeated social media claims
Illusory Truth or Mere Exposure? Model-Dependent Repetition Effects in LLM-Based Social Media Simulations
Artificial IntelligenceSocial and Information Networks
Summary
Repeated exposure to statements can make humans think those statements are more true, a bias called the Illusory Truth Effect. The authors tested whether large language models (LLMs) show this effect in social media-like settings. They found that some models behave like humans with this bias, some just show a general liking for repetition, and others show no such effect or even skepticism. This means not all LLMs simulate human belief patterns well when used to study misinformation.
What this means in practice
- •For social media platform designers: Identify which LLM best models human belief updating to improve misinformation simulation and detection methods.
- •For ai behavior modelers: Choose appropriate LLMs in agent-based models to replicate realistic human cognitive biases like the Illusory Truth Effect in social dynamics studies.
Authors
Azza Bouleimen, Nicolò Pagan, Anikó Hannák
Abstract
Generative agent-based models (GABMs) are increasingly used to simulate social media dynamics, including misinformation spread. For such social simulations to be valid proxies of human behavior, LLM agents should replicate established human cognitive biases, among them the Illusory Truth Effect (ITE), where repeated exposure to a claim increases its perceived truth value. We investigate whether and how the ITE manifests across four LLMs (Gemma-3-4b-it, Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, and GPT-5-nano) in a social media simulation context. We propose a two-phase within-context experimental design that embeds the repetition manipulation inside a realistic news feed interaction. Using this design, we collect 336,000 truth, importance, sentiment, and interest ratings across 100 statements, 10 feed variants, and 3 replications. The key comparison is between ratings assigned to repeated statements, seen throughout a simulation phase, and completely unseen ones, rated within the same experimental context window. We distinguish genuine ITE (truth-specific repetition boost) from mere exposure effects. We run an OLS regression followed by a Linear Mixed-Effect Model to account for differences across models and ratings. Our results reveal four qualitatively distinct patterns: Gemma-3 exhibits a genuine ITE; Qwen2.5 shows a mere exposure effect; GPT-5-nano displays no repetition effect on truth and mild skepticism toward repeated content; Llama-3.1 shows a small truth boost alongside decreases in evaluative dimensions. Crucially, temperature has no effect on these findings, and a variance decomposition highlights the high context-sensitivity of LLM rating behavior. Our findings caution against assuming uniform ITE replication across LLMs in social simulations, while suggesting that Gemma-3-4b-it may offer the most behaviorally realistic approximation for misinformation-related simulations.