When Can We Work in Embedding Space? What Text Embeddings Preserve
2026-08-31 • Computation and Language
Computation and Language
AI summaryⓘ
The authors explore when using text embeddings—compact numerical summaries of text—works well for analyzing documents. They explain this using a model where documents are mixtures of hidden topics, showing that grouping texts by their embeddings reflects grouping by these topics. They focus on two main tasks: clustering documents and controlling for text in statistical analyses. In a real-world example with economic descriptions of U.S. metro areas, the authors found that clustering based on embeddings created meaningful economic groups and better captured local employment trends than other methods.
text embeddingslatent topicsclusteringgenerative modelconfoundinglarge language modelseconomic archetypesempirical analysistext control
Authors
Simon Freyaldenhoven
Abstract
When do text embeddings work as inputs to empirical analysis? Their use rests on an assumption: that we can trade text for its low-dimensional embedding, and lose little in doing so. I make that assumption precise under a generative model in which documents are mixtures of latent topics. I study two uses---clustering units in embedding space and controlling for high-dimensional text. A cluster of embeddings is a set of documents with similar topic mixtures; controlling for the embedding is equivalent to controlling for the topic mixture, so validity reduces to whether that mixture captures the confounding. In an application to 363 U.S. metropolitan areas, embedding-based clusters of LLM-generated economic descriptions recover interpretable economic archetypes and separate local employment dynamics more sharply than clustering on model residuals, or on a curated set of industry and demographic covariates.