Do LLM Embedding Spaces Recover Expert Structure?
2026-06-22 • Computation and Language
Computation and Language
AI summaryⓘ
The authors looked at how well pretrained and fine-tuned language model embeddings capture the relationships between mental health symptoms, using data from Reddit communities. They found that pretrained embeddings somewhat reflect expert-defined symptom structures, but fine-tuning and using larger models improved this alignment, especially for more detailed categories. They also showed that this alignment is not just due to general language features like sentiment or style. Overall, the authors suggest that language model embeddings can map important expert knowledge, but this should be carefully checked beyond simple classification tests.
pretrained embeddingsfine-tuningmental health symptomsrepresentational similarity analysiscategory prototypeslarge language modelsconfound controlzero-shot learningReddit communitieslinguistic style features
Authors
Yixuan Zhu, Zhenke Duan, Fanghen Li
Abstract
Pretrained text embeddings are increasingly used as representational maps, yet high category separability does not imply that their geometry recovers expert-defined structure. We study this problem in mental-health-related language, where symptom relations provide an external reference and online communities introduce strong domain, affective, stylistic, and discourse confounds. Using 28 Reddit communities, we compare pretrained and supervised fine-tuned Qwen3 embedding spaces at two scales (0.6B and 4B). We construct category prototypes, evaluate their representational dissimilarity matrices against an expert symptom matrix with representational similarity analysis, and complement this global test with prototype-based typicality and multi-baseline confound controls. Pretrained embeddings show measurable alignment with expert structure within the mental-health subset; fine-tuning strengthens this alignment most at the finest category level; and larger scale improves both zero-shot alignment and supervision-induced gains. Residual alignment remains substantial after controlling for VAD, LIWC, lexical style, and topic-distribution structure. These results suggest that LLM embeddings can recover expert-relevant category geometry, but this recovery is level-dependent and should be tested against explicit confounds rather than inferred from classification alone.