All You Need Is Non-Commutative Words
Computation and Language
Summary
The authors represent words as special square matrices and combine them by multiplying in order, which naturally keeps track of word order without extra position markers. This mathematical setup allows efficient attention mechanisms and can combine text chunks while saving computational cost. They also introduce a method for reading the encoded information compactly and support continual learning without forgetting previous tasks. Their approach matches or beats simple bag-of-words models on text classification tasks using far fewer parameters than traditional large vocabularies.
Authors
Carla M. Quispe Flores, Stanley Salvatierra, Renan Cabrera
Abstract
We represent lexical tokens as unitary matrices and encode each sentence as their ordered product. The noncommutativity of matrix product captures word order without positional encodings (PEs). The same algebra yields several capabilities, including antisymmetric self-attention with no query, key, or value projections, and parallel composition of variable-length text chunks at a reduced attention cost. Furthermore, it provides a canonical-coset readout layer that encodes all true unitary degrees of freedom compactly, while supporting continual learning through nested group extensions that enlarge the operator space with each new task preserving prior representations exactly. Across standard text-classification benchmarks, the method matches or exceeds bag-of-words baselines. Achieving higher accuracy on IMDB and comparable performance on AG News. Notably, this is accomplished by replacing the conventional $\sim$30,000-dimensional vocabulary space with a dense, 64-parameter real-valued encoding, highlighting the expressive efficiency of our parameterization.