Modern text embedders hide sentiment in hard-to-find geometry
Semantically Rich, Metrically Affect-Blind: A Geometric Blind Spot in Modern Text Embedders
Computation and LanguageMachine Learning
Summary
Modern text embedding systems can store how positive or negative a sentence is, but this feeling isn't obvious when comparing their usual similarity scores. The authors show that while you can train a small tool to read sentiment from these embeddings with high accuracy, the typical way of measuring similarity misses this information almost entirely. They also created a fast method to inject this sentiment knowledge directly into embeddings, improving how well systems find and group sentences by feeling, though sometimes at the cost of other language tasks.
What this means in practice
- •For search engineers: Improve search and clustering quality by injecting sentiment signals directly into sentence embeddings using a lightweight supervised head.
- •For chatbot developers: Enhance retrieval-augmented generation systems by better capturing sentence sentiment in embedding metrics to improve relevant response selection.
Authors
Mahdi Naser Moghadasi
Abstract
We report a systematic dissociation in modern text embedders: sentiment is almost perfectly decodable from their embeddings, yet almost entirely absent from their metric. Across eleven frozen encoders spanning 2021 to 2026, including 2026-generation models and the 4B-parameter LLM embedder Qwen3-Embedding-4B, a linear probe recovers sentence polarity at up to 91.3% accuracy while cosine similarity separates polarity by a margin of only 0.003-0.064. Qwen3-Embedding-4B, the strongest probe in the panel, reaches 91.3% probe accuracy but only 64.3 retrieval MAP. Because retrieval, clustering, deduplication, and retrieval-augmented generation operate on the metric rather than on a trained probe, these systems are effectively blind to information their encoder demonstrably contains. A task-conditioning control suggests this reflects the pretraining objective rather than a prompting artifact: conditioning Jina-v5 on its classification task rather than its retrieval task recovers only about half of the MAP gain that injection achieves on the same weights (63.2 to 73.8, versus 85.1), and leaves the alignment gap essentially unchanged. Sentiment-Injected Sentence Embeddings (SISE) train a small (0.13M-0.69M-parameter) supervised-contrastive head on a frozen encoder in seconds on a CPU, using the labelled training set. The head improves every encoder released since 2022 on full-data probe accuracy (up to +5.8 points), retrieval MAP (up to +26.0) and polarity margin (7-61x). This is a specialization, not a free gain: the compact 128-d head loses 38-41 Spearman points on STS-B, while a fused variant that concatenates it with the base embedding loses 2.4-12.0. With the head, a 0.6B encoder at 128 dimensions matches a 4B encoder given the same head (91.8 vs. 90.3 MAP). We release a CPU-reproducible pipeline and a reusable diagnostic, the Attribute Alignment Gap.