Rotating sparse connections match full attention with fewer links
Reaching Every Position Without Searching: Rotating Sparse Wiring on the Hypercube as a Substitute for Attention
Machine Learning
Summary
Attention mechanisms in AI usually check every possible connection, which takes a lot of work. The authors show that by thinking of positions in a sequence as points on a cube and rotating which neighbors are connected in each layer, information can flow efficiently without searching. Their method uses far fewer links but still lets information reach every position quickly. Tests on language data show this approach can be as good as or better than standard attention while using less computation and being faster.
What this means in practice
- •For natural language processing teams: Build faster and smaller language models by replacing full attention with rotated sparse connections to reduce computational cost without losing accuracy.$Commercial implications: Enables development of efficient, low-resource text understanding models for commercial AI products by reducing parameter count and runtime.
- •For software engineers in ai infrastructure: Implement rotated sparse wiring in sequence models to speed up training and inference on standard hardware with less memory usage.
Authors
Yoshiaki Takashita
Abstract
Attention pays, at every layer and for every input, the cost of searching for whom to connect. We ask how far one can get with wiring that is fixed, sparse, and simply rotated from layer to layer. Treating the $n$ positions of a sequence as the vertices of a $\log_2 n$-dimensional hypercube and connecting each position, at layer $\ell$, to its neighbour along dimension $\ell \bmod \log_2 n$, information from every position reaches every other in $\log_2 n$ layers with $2n$ links per layer instead of $n^2$. On a synthetic task that is unsolvable unless all positions are reached, this rotation matches all-to-all wiring at $1/32$ of the links, while the same sparse pattern held fixed across layers fails; what matters is that every dimension is touched, not the order. On character-level language modelling of a public corpus (the first $12$M characters of enwik8), a hybrid that keeps two attention layers among sixteen sparse ones reaches $0.06$ bits-per-character lower held-out loss than a fully attentive model of the same width at the same step budget (three seeds each, no overlap), with $1/7$ of the links, $42\%$ fewer parameters, and $2.4\times$ less wall-clock time; the purely rotated schedule is level with the hybrid. The same ordering holds on a second corpus of mixed Japanese, English and code, where the gap widens to $0.16$. The usable learning-rate window is four to eight times wider than attention's on both. We also report what did not work - learned coordinates, and a "dynamics" variant whose apparent gains turned out to be an artefact of a saturated kernel - and the measurement discipline (frozen corpus, full-coverage evaluation, seed spread as the bar for ranking) that we found necessary to say anything at all at this scale.