Meaning identity emerges during joint model processing not from embeddings
Computation Over Geometry: Meaning Identity Is Computed, Not Shipped in the Embeddings
Computation and Language
Summary
This paper finds that knowing if two sentences mean the same thing is not about comparing their separate vector summaries, but happens when a model processes both sentences together. The researchers show that individual sentence encodings don’t reliably capture meaning similarity. Instead, the models compute this relationship during a joint pass over the sentence pair. This challenges common assumptions and shows that understanding meaning identity requires more than just fixed embeddings.
What this means in practice
- •For search engineers: Improve retrieval accuracy by designing systems that compute sentence similarity during joint processing rather than relying solely on fixed embeddings.
- •For chatbot developers: Enhance response retrieval in conversational AI by leveraging joint forward passes to better compute whether two sentences convey the same meaning.
Authors
Jiaqi Deng
Abstract
Meaning identity (whether two sentences say the same thing after wording changes) is treated in retrieval and RAG as a geometric fact about independently encoded sentence vectors. We show that, for frozen off-the-shelf encoders and language models, it is not: identity is computed when both sentences share one forward pass, and is not a property of the embedding geometry those systems ship. On overlap-matched PAWS-X, purpose-built encoders (BGE, E5, GTE, MiniLM, E5-Mistral-7B) reach English confirm AUC only 0.55-0.65 (dense peak 0.70). Independently encoded last-token states of Llama 3, Mistral, and Qwen do no better; late fusion of the two vectors stays near chance. The same probe on a joint forward pass reaches 0.90-0.96 from 1.5B to 32B, collapses under partner shuffle, is mid-depth, saturates near 0.94 by 3B, and appears more weakly in GPT-2 XL (0.76). The gap holds beyond Llama-style models on other causal LMs, bidirectional encoders (DeBERTa, RoBERTa), and encoder-decoders (Flan-T5, T5, BART). Fixed or linear readers over frozen independent encodings never unlock identity; nonlinear pair readers recover part of it only on the full 49k-pair PAWS train split (0.68-0.87). Off-the-shelf rerankers split: BGE-reranker-large reaches 0.94, while MS-MARCO and Jina stay at 0.55-0.64. Independently trained families compute the same relation and a 1.5B joint reader can distill it from unlabelled teacher scores, while no linear function of the teachers own independent vectors can. Bi-encoders can be fine-tuned to fit PAWS (0.87-0.93), but transfer and STS-B suffer. Cosine compares wording neighbourhoods; identity is a cheap computed operator, not a property of either sentence vector.