Summary
Attention mechanisms in AI work by comparing vectors, but these comparisons only make sense if both vectors share the same coordinate system. The authors study what happens when each entity has its own coordinate system that can rotate independently, which they call a symmetry. They show how this constraint changes what attention can learn about relationships between entities, and that some tasks require a minimum vector size to represent these relationships exactly. Their findings explain why certain attention models succeed or fail at modeling complex relations and provide ways to certify when a model correctly learns a relational structure.
What this means in practice
- •For machine learning engineers: Design and evaluate attention models that must represent relational structures exactly by selecting appropriate vector sizes and operator constraints based on symmetry principles.
- •For natural language processing developers: Improve multi-hop reasoning performance in language models by using per-entity operator-based composition instead of standard shared-frame attention.
Tested on simulated data.
Abstract
A dot product means something only when both vectors share a coordinate system. Content words have one; a fresh entity does not: rotating its coordinates alone changes nothing. We take this per-entity frame freedom as a symmetry and ask what it forces in attention. For orthogonal transports, invariant scores depend only on norms and the transported inner product $\langle x_i,T_{ij}x_j\rangle$; equivariant messages lie in the span of $x_i$ and $T_{ij}x_j$. Endpoint channel count bounds transport rank, so with one vector per entity relations must be supplied; standard attention is the flat sector of a shared frame. Representation theory attaches an integer to each mechanism; the readout decides which. With orthogonal transitions, a convex-cell readout composes a finite group exactly only at or above its smallest faithful real representation dimension. At this minimum, exactness forces a faithful representation. An even readout can decode a cover at smaller width, and the gap is unbounded. Below its threshold, the part of the group a mechanism cannot see sets an accuracy ceiling computable before training. 17/17 fitting seeds at minimal width passed tests through length 1000, versus 7/18 at twice the width; 12 models carry correctness certificates to explicit lengths, from the weights alone. On $C_3\times A_4$ (faithful width 5), descent at width 4 learned the 72-state double cover under sign-blind attention (6/48 seeds; monotone 0/48); on a frozen cover the cross-entropy-optimal affine head is uniform, a quadratic head exact. On parsed CLUTRR, transported composition beats a matched transformer by 41 points at 4-10 hops, but forcing isometries costs 65 points; on StepGame's commutative algebra, isometries help. Static embeddings gave each word one vector; contextual models gave each occurrence its own; per-entity frames make each relation an operator, the unit in which relations compose.