When Can Text Embeddings Replace Item Calibration? A Geometric Diagnostic for Semantic Loadings in Multidimensional Adaptive Testing
2026-08-10 • Computers and Society
Computers and Society
AI summaryⓘ
The authors explored if they could predict how test questions relate to different personality traits just by looking at the question text, instead of using large human response data. They used a personality test dataset and compared traditional item calibration with a method using sentence embeddings from text. Their results showed that the text-based method estimated trait scores almost as well as the traditional method but was much less certain about these estimates. They found that the shared language in personality items made it hard for the text method to distinguish traits well, and they suggested a way to check if test questions are suitable for this text-based approach before using it.
Multidimensional Item Response TheoryCalibrated Item ParametersSentence EmbeddingsComputerized Adaptive TestingPersonality AssessmentLatent TraitsItem Loading MatrixCondition NumberGraded Response ModelD-optimal Item Selection
Authors
Amirreza Mehrabi
Abstract
Multidimensional item response theory relies on calibrated item parameters, such as discrimination and category threshold values, which are usually estimated from large samples of human test responses. This study investigates whether the directional loadings of these parameters can be recovered directly from item text using pre-trained sentence embeddings, avoiding the need for initial item calibration. Using the open-source IPIP Big-Five dataset ($n=19{,}719$; 50 items), we built a multidimensional computerized adaptive testing (CAT) simulation using D-optimal item selection. We compared three item loading sources: fitted graded response model parameters, semantic text embeddings, and a lexical baseline. In simulation, semantic embeddings recovered latent trait profiles almost as accurately as fitted parameters (correlation $0.825$ vs. $0.857$), performing noticeably better than simple word overlap ($0.752$). However, the embedding-based model produced inflated posterior variance, showing nearly four times higher measurement uncertainty despite accurate point estimates. We attribute this to collinearity across dimensions, as embedding-derived loadings pointed in similar directions across traits (condition number $137$ vs. $1.0$; mean trait cosine $0.90$). This outcome reflects the shared vocabulary common in personality items. We propose a simple diagnostic metric based on the loading matrix condition number to evaluate whether an item bank is suitable for text-derived loadings prior to testing.