Papers for

survey methodologists

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Model answer probabilities show limits in matching real survey patterns

Can Model Answer Probabilities Represent Survey Populations? Conditional Prediction and Joint Validity in Jev Survey Simulation

Abstract: Survey simulations require realistic combinations of answers, but predictive accuracy alone does not establish that a model can generate them. We evaluate the option probabilities returned by Jev, a TypeSafe decision model, using linked responses to 20 binary items from five archived survey waves. We compare population-only prediction, prediction given recorded answers, and sequential completion that uses generated answers to predict later responses. Joint comparisons hold the starting population information fixed, and two three-question cases are completed in both question orders. We evaluate predictive scores, answer frequencies, and associations against the survey records. Recorded answers can improve predictive scores while associations remain distorted. Large conditional errors also occur. For example, among men who described themselves as having traditional values, the model predicted that 16% would see men and women as similar in workplace abilities, compared with 56% in the survey. Sequential completion has mixed effects on joint validity, and reversing question order produces different joint distributions in both tested cases. Order checks and classical marginal bounds clarify what can be assessed before linked validation answers are available. The findings support validating the joint events a simulation uses and separating empirical constraints from model assumptions. The evidence concerns selected binary tasks and one model version.

Wed 11 NovArtificial IntelligenceComputers and Society
The gist
Survey simulations try to predict how people answer many questions together, but just giving accurate single answers isn’t enough to capture realistic response patterns. The authors studied a model called Jev that predicts answers to 20 yes/no questions and found that while it can predict individual answers well, it struggles to match how answers actually relate to each other in real people. For example, it underestimates how many men with traditional values think men and women are equally capable at work. The study shows it’s important to check if simulated answers match the joint patterns from real survey data, not just the overall accuracy.
Open → 2611.15576v1

Semantic priors improve questionnaire analysis with fewer respondents

When Respondents Are Scarce: LLM-Derived Semantic Priors for Bayesian Confirmatory Factor Analysis

Abstract: Across social and behavioral research, measuring latent constructs can be challenging when target populations are small or difficult to reach. Recruiting additional respondents may be costly or infeasible, leaving researchers with limited data for estimating questionnaire factor structures. Yet item wording contains structural expectations before data collection, and whether these can improve small-sample estimation without replacing empirical validation remains unclear. We introduce the Semantic-Empirical Updating Framework (SEUF), which converts large language model paraphrases and embeddings, aligned with hypothesized item-factor assignments, into prespecified priors for Bayesian confirmatory factor analysis. We evaluated SEUF using a 38-item survey of 617 industrial-park stakeholders and two public surveys, FEVS and RIASEC. Semantic structure persisted after wording controls across all three instruments. With 50 respondents, SEUF achieved loading precision comparable to conventional CFA with 70-82 respondents, equivalent to 29-39% fewer respondents at the specified precision benchmark. Results showed lower loading errors and improved credible-interval coverage relative to conventional CFA in both public surveys. Gains were greatest when respondent data were limited, highlighting the value of semantic priors under information scarcity. By harnessing structural information already embedded in questionnaires, SEUF advances a more resource-efficient approach to measurement and opens new directions for study design and empirical inference in hard-to-reach populations.

Mon 26 OctMachine Learning
The gist
Measuring hidden traits through surveys can be very hard when you don't have enough people answering. The authors found a way to use clever language models to understand the meaning behind survey questions and guide the analysis. This makes it possible to get accurate results with fewer responses. Their method, tested on real surveys, was better at estimating how questions relate to traits, especially when data was scarce.
Open → 2610.34843v1

Large language models generate survey answers that miss real human response patterns

Silicon Samples Match the Answers but Not the Respondents

Abstract: Silicon sampling generates survey respondents by having a large language model (LLM) answer a survey as a person with given demographic attributes, and the resulting synthetic populations are increasingly used in place of human ones. Existing evaluations focus on individual-answer accuracy, per-question answer distributions, and correlations across questions. None of these compares the respondents themselves, so a generator whose respondents have combinations of answers found in no human respondent can still score well if it reproduces every answer distribution. We propose the respondent coverage gap (RCG), which represents each respondent as a point defined by their complete answer profile and measures how far each human respondent is from the nearest generated one, relative to the distance between human samples. On two US social surveys, all six LLMs we evaluate leave human respondents uncovered, and under our main settings, they fall behind a baseline that samples each answer independently from the human marginals. Across the prompting and inference settings we test, every LLM has a higher RCG than a Gaussian copula fitted to the human data.

Thu 22 OctComputation and LanguageComputers and Society
The gist
The paper looks at how large language models (LLMs) create fake people to answer survey questions based on given demographics. These models can get individual answers right but often produce answer combinations that no real person actually gave. The authors introduce a new way to measure this problem, showing that LLM-generated respondents differ significantly from real ones. Even simple methods mixing answers independently from real survey data outperform LLMs at matching real human answer patterns.
Open → 2610.30900v1

Fixed size sampling improves variance without extra uncertainty

Fixed-Size Active Statistical Inference

Abstract: Variance reduction techniques improve the efficiency of drawing conclusions from data. Recent proposals for variance-reduction use two complementary techniques to reduce variance: incorporating machine learning predictions into estimators and weighted sampling. A popular weighted sampling algorithm is Poisson Sampling (PS), which independently flips a biased coin at each unit and includes all heads in the sample. A fundamental flaw with PS is that it has a random sample size, which presents practical challenges and introduces variance into the estimator. We show that a fixed-size alternative, known in survey sampling as Conditional Poisson Sampling (CPS), is a drop-in replacement for PS on general multivariate Z-estimation problems, which include convex M-estimation considered in prior work. Notably, we show that the asymptotic variance of CPS is no worse than that of PS, and can be substantially better in realistic scenarios where predictions are imperfect. We illustrate these advantages on both synthetic and real datasets, and release a Python package implementing CPS. In light of our contributions, there is no need to suffer a variable sample size.

Mon 19 OctMachine Learning
The gist
When gathering data to make decisions, choosing how many samples to collect can be tricky because sometimes the number of samples is random and causes extra uncertainty. The authors looked at a common sampling method called Poisson Sampling, which picks a random number of samples, and showed that another method called Conditional Poisson Sampling can pick a fixed number of samples instead. This fixed-size method has as good or better accuracy than the usual approach, especially when predictions used in the process aren’t perfect. They tested this on real and simulated data and provided code for others to use.
Open → 2610.27389v1

LLMs struggle to match age differences in loneliness in European surveys

Do LLMs Reproduce the Age Gap in Loneliness? Evaluating Survey Simulation against European Survey Data

Abstract: Large language models (LLMs) are often used to simulate survey respondents, but matching survey averages does not establish accurate demographic differences. We test whether LLM outputs reproduce within-country age gaps in loneliness using the 2022 EU Loneliness Survey, comparing adults aged 18-34 and 65 or older in France, Italy, Poland, and Sweden. We evaluate simulation with and without age information and direct distribution estimation against a probability-based sample, with a non-probability sensitivity reference. Both human samples show higher mean loneliness among younger adults. Under age-specified simulation with standard response-option order, Mistral Small 4 and GPT-5.6 Terra reverse this ordering under all three prompt framings in three and two countries, respectively, mainly by overestimating older-adult means. DeepSeek V4 Flash preserves the ordering everywhere but underestimates both group means, with errors largely cancelling in France and Italy. Against the probability-based reference, direct estimation has lower absolute age-gap error than simulation for Mistral and Terra in every country. For DeepSeek, simulation has lower gap error but direct estimation has lower group-mean error; these comparisons depend on low-validity older-adult simulation cells. Evaluating LLM outputs as proxies for age groups within countries therefore requires checking both group means and their contrast against human survey data.

Sun 18 OctComputation and Language
The gist
People often use large language models (LLMs) to mimic answers from different groups in surveys, like how lonely different ages feel. This paper checks if LLMs can correctly show that younger adults tend to feel lonelier than older adults in four European countries. The researchers found that some LLMs got this backward, saying older adults were lonelier, while others underestimated loneliness levels but kept the right order. The study shows it's important to carefully check both group averages and differences if using LLMs to simulate survey data about age groups.
Open → 2610.25158v1

Large language models generate coherent survey responses across questions

Simulating Respondents, Not Single Questions: Coherent Survey Generation with Large Language Models

Abstract: Large language models are increasingly used to simulate response distributions in social surveys. Prior work has achieved accurate population-level simulation for individual questions. Real questionnaires, however, ask each respondent a sequence of related questions. A simulated respondent should show coherent preferences across the whole questionnaire, not merely accurate distributions for isolated items. Existing single-item methods cannot accurately reproduce how the same person answers a complete survey. We propose FullRespondent-LLM (FR-LLM), which fine-tunes two specialized LLMs: a marginal model for each item's response distribution and a respondent-level autoregressive model for dependencies across answers. Marginal-Constrained Joint Projection (MCJP) then projects the autoregressive joint distribution onto the set satisfying the item-level marginals learned by the first model. This yields complete questionnaires with realistic cross-item relationships while retaining strong item-level accuracy. On two real-world social survey datasets, FR-LLM more accurately reproduces multi-question response patterns, maintains competitive single-item accuracy, and generalizes better to unseen populations and questions. In a small commercial-survey dataset, we use simulated responses to make pricing and stocking decisions; FR-LLM achieves the highest realized profit.

Mon 28 SeptArtificial Intelligence
The gist
Surveys often ask the same person many related questions, so their answers should make sense together. Existing methods simulate answers for single questions well but fail to keep answers consistent across an entire survey. The authors created a new method that trains two language models: one for each question's answer patterns and another for how answers relate across questions for the same person. Their method better mimics how real people respond to whole surveys and works well even on new populations or questions.
Open → 2609.34828v1