Semanticfa factors meaning to analyze personality test items
semanticfa: An R Package for Response-Free Semantic Factor Analysis of Psychometric Scales
Computation and Language
Summary
Psychological tests have many questions that suggest a structure of personality traits. The authors created an R package called semanticfa that uses language models to turn test questions into numbers showing their meaning. This method groups similar questions and helps spot duplicates or make shorter versions of tests. They showed how semanticfa works by comparing its results on a famous personality test with traditional analysis done on lots of real answers. The new method provides a way to see how much the wording of questions matches how people actually respond.
What this means in practice
- •For psychometric test developers: Use semanticfa to measure and improve how well test items’ meanings align with personality factors without collecting response data.
- •For market research teams: Screen and shorten large survey question sets quickly by analyzing item meaning similarities with semanticfa instead of long surveys.
Authors
Devon Yanitski, Chris Westbury
Abstract
The items of a reflective psychometric scale can be read as a proposal about a construct's structure. Language models make that proposal measurable by converting each item into an embedding, a list of numbers, similar for similar meanings. We introduce semanticfa, an R package for semantic factor analysis, which factors item-embedding similarities. It also interprets factors, flags redundant items, builds short forms, and screens new items. A worked example analyzes the 50-item International Personality Item Pool Big-Five Factor Markers, embedded with Qwen3-Embedding-8B, and compares the results with a factor analysis of 874,434 respondents. Retention criteria tied between five and six factors, and the tie rule kept five. Three corresponded mainly to single domains, and two blended most Extraversion items with five Agreeableness items. Each factor's congruence with its best-matching human factor (1 = identical) ranged from .92 for Openness to .52 for Agreeableness. Item pairs more similar in meaning tended to be answered more alike (r = .46 with raw responses), but same-domain reverse-keyed opposites, answered oppositely, were about as similar as cross-domain pairs (.58 vs. .57). Sign-flipping reverse-keyed items' embeddings did not improve agreement with raw responses and worsened it with reverse-scored ones. With five factors, a 25-item short form matched the domains better than all 200 random five-per-domain sets, but parallel analysis kept only three. We propose semantic-behavioral coherence, the agreement between semantic and response structures, as a measurable property of a scale, given an embedding model. Semantic factor analysis complements, and does not replace, response-based factor analysis.