LLMs struggle to match age differences in loneliness in European surveys
Do LLMs Reproduce the Age Gap in Loneliness? Evaluating Survey Simulation against European Survey Data
Computation and Language
Summary
People often use large language models (LLMs) to mimic answers from different groups in surveys, like how lonely different ages feel. This paper checks if LLMs can correctly show that younger adults tend to feel lonelier than older adults in four European countries. The researchers found that some LLMs got this backward, saying older adults were lonelier, while others underestimated loneliness levels but kept the right order. The study shows it's important to carefully check both group averages and differences if using LLMs to simulate survey data about age groups.
What this means in practice
- •For survey methodologists: Use detailed checks of both group averages and differences when simulating age-related survey responses with LLMs to improve accuracy.
- •For social data analysts: Assess how well LLM simulations reflect actual age-related loneliness patterns when modeling European adult populations.
Authors
Yuliya Li, Achim Rettinger, Dominik Beyer
Abstract
Large language models (LLMs) are often used to simulate survey respondents, but matching survey averages does not establish accurate demographic differences. We test whether LLM outputs reproduce within-country age gaps in loneliness using the 2022 EU Loneliness Survey, comparing adults aged 18-34 and 65 or older in France, Italy, Poland, and Sweden. We evaluate simulation with and without age information and direct distribution estimation against a probability-based sample, with a non-probability sensitivity reference. Both human samples show higher mean loneliness among younger adults. Under age-specified simulation with standard response-option order, Mistral Small 4 and GPT-5.6 Terra reverse this ordering under all three prompt framings in three and two countries, respectively, mainly by overestimating older-adult means. DeepSeek V4 Flash preserves the ordering everywhere but underestimates both group means, with errors largely cancelling in France and Italy. Against the probability-based reference, direct estimation has lower absolute age-gap error than simulation for Mistral and Terra in every country. For DeepSeek, simulation has lower gap error but direct estimation has lower group-mean error; these comparisons depend on low-validity older-adult simulation cells. Evaluating LLM outputs as proxies for age groups within countries therefore requires checking both group means and their contrast against human survey data.