Exact conditions for perfect meaning readout in compressed word vectors
Exact semantic readout from compressed vector representations
Computation and LanguageMachine LearningLogic in Computer Science
Summary
Compressed word vectors are used to represent meanings of words in computers, but it is unclear when you can exactly recover the truth about word usage just by a simple fixed math operation. The authors found precise mathematical conditions under which such exact readouts exist and how many dimensions these vectors need for this to happen. They tested these findings on common pretrained word vectors and showed that most concepts can be separated, but exact perfect recovery isn’t possible without retraining. By carefully training, they achieved exact recovery while still closely preserving the original vector qualities.
What this means in practice
- •For natural language processing engineers: Design embedding systems with guaranteed exact semantic decoding based on dimension and independence conditions to improve interpretability and control.
- •For knowledge graph developers: Verify when relations like identity or order in entity vectors allow exact bilinear truth extraction for cleaner knowledge base construction.
Authors
Daniel Quigley
Abstract
We characterize when compressed vector representations admit exact linear or affine readouts of a finite lexicon's truth conditions: one fixed map per predicate, sending each entity vector to the corresponding truth vector. A necessary and sufficient row-space condition determines existence; the augmented truth matrix has rank r, giving minimum dimension r in the linear case, and r-1 in the affine. Exact readouts return values in a shared truth basis on which Boolean connectives act unchanged; separability alone requires an intervening threshold. For binary relations, exact bilinear readout of identity or strict total order requires linearly independent entity vectors. Experiments with GloVe and word2vec distinguish exact affine recovery, linear separability, and held-out prediction: most predicates are strictly separable, but none admits an exact affine readout from the pretrained embeddings. Supervised transductive training attains exact affine recovery to numerical precision at every tested dimension meeting the bound. At the embeddings' original dimension, geometries constrained to exact linear recovery retain 98-99 percent of the pretrained variance on the feature norms, and 80-83 percent on the WordNet lexicon.