Semantic priors improve questionnaire analysis with fewer respondents
When Respondents Are Scarce: LLM-Derived Semantic Priors for Bayesian Confirmatory Factor Analysis
Machine Learning
Summary
Measuring hidden traits through surveys can be very hard when you don't have enough people answering. The authors found a way to use clever language models to understand the meaning behind survey questions and guide the analysis. This makes it possible to get accurate results with fewer responses. Their method, tested on real surveys, was better at estimating how questions relate to traits, especially when data was scarce.
What this means in practice
- •For survey methodologists: Estimate factor structures more precisely in small or costly-to-reach populations using semantic priors from language models.
- •For market research teams: Improve accuracy of survey-based insights with fewer respondents by incorporating semantic information from questionnaire items.
Authors
Shaohuai Feng, Mohd Wira Mohd Shafiei, Mohd Sukri Shafie, Siyang Tian
Abstract
Across social and behavioral research, measuring latent constructs can be challenging when target populations are small or difficult to reach. Recruiting additional respondents may be costly or infeasible, leaving researchers with limited data for estimating questionnaire factor structures. Yet item wording contains structural expectations before data collection, and whether these can improve small-sample estimation without replacing empirical validation remains unclear. We introduce the Semantic-Empirical Updating Framework (SEUF), which converts large language model paraphrases and embeddings, aligned with hypothesized item-factor assignments, into prespecified priors for Bayesian confirmatory factor analysis. We evaluated SEUF using a 38-item survey of 617 industrial-park stakeholders and two public surveys, FEVS and RIASEC. Semantic structure persisted after wording controls across all three instruments. With 50 respondents, SEUF achieved loading precision comparable to conventional CFA with 70-82 respondents, equivalent to 29-39% fewer respondents at the specified precision benchmark. Results showed lower loading errors and improved credible-interval coverage relative to conventional CFA in both public surveys. Gains were greatest when respondent data were limited, highlighting the value of semantic priors under information scarcity. By harnessing structural information already embedded in questionnaires, SEUF advances a more resource-efficient approach to measurement and opens new directions for study design and empirical inference in hard-to-reach populations.