Model answer probabilities show limits in matching real survey patterns

Can Model Answer Probabilities Represent Survey Populations? Conditional Prediction and Joint Validity in Jev Survey Simulation

Artificial IntelligenceComputers and Society

Summary

Survey simulations try to predict how people answer many questions together, but just giving accurate single answers isn’t enough to capture realistic response patterns. The authors studied a model called Jev that predicts answers to 20 yes/no questions and found that while it can predict individual answers well, it struggles to match how answers actually relate to each other in real people. For example, it underestimates how many men with traditional values think men and women are equally capable at work. The study shows it’s important to check if simulated answers match the joint patterns from real survey data, not just the overall accuracy.

What this means in practice

  • For survey methodologists: Improve survey simulation tools by validating joint answer patterns to better replicate real respondent behavior across multiple questions.
  • For social science data teams: Use joint validity checks to assess if predictive models can generate realistic combined survey answers before applying them to population analysis.

Tested on one dataset.

Authors

Zengqing Wu, Chuan Xiao

Abstract

Survey simulations require realistic combinations of answers, but predictive accuracy alone does not establish that a model can generate them. We evaluate the option probabilities returned by Jev, a TypeSafe decision model, using linked responses to 20 binary items from five archived survey waves. We compare population-only prediction, prediction given recorded answers, and sequential completion that uses generated answers to predict later responses. Joint comparisons hold the starting population information fixed, and two three-question cases are completed in both question orders. We evaluate predictive scores, answer frequencies, and associations against the survey records. Recorded answers can improve predictive scores while associations remain distorted. Large conditional errors also occur. For example, among men who described themselves as having traditional values, the model predicted that 16% would see men and women as similar in workplace abilities, compared with 56% in the survey. Sequential completion has mixed effects on joint validity, and reversing question order produces different joint distributions in both tested cases. Order checks and classical marginal bounds clarify what can be assessed before linked validation answers are available. The findings support validating the joint events a simulation uses and separating empirical constraints from model assumptions. The evidence concerns selected binary tasks and one model version.