Local mixing layers help transformers understand word order over long texts
How Local Mixing Encodes Relative Position in Global NoPE Attention
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
Transformers are computer models that read and write language, but they usually need special clues about word order to make sense. The paper explains how mixing small chunks of words locally inside the model helps it remember the order even without those special clues. This creates a bias towards recent words that the model can use globally to understand position and meaning. Their findings help explain why some new transformer designs work well and might help design models that read longer texts better.
What this means in practice
- •For natural language engineers: Build more efficient language models that understand long texts without relying on explicit position codes by using local mixing layers.
- •For speech recognition developers: Improve speech-to-text accuracy over long conversations by integrating hybrid attention models that implicitly track word position using local context.
Authors
Cutter Dawes, Nick Alonso, Tom Figliolia, Beren Millidge
Abstract
The attention operation is naively position invariant. However, positional information is fundamental to natural language, and therefore a variety of explicit position encodings have been developed in transformer-based models, such as rotary position encoding (RoPE). Although explicit position encodings have long been assumed to be required, recent methods that interleave local mixing layers, such as sliding window attention (SWA) and gated linear attention, while not encoding position (NoPE) in global attention layers has recently been shown to be successful at scale. How and why this approach works is not well-understood. In this paper, we develop an explanation of how hybrid models of this sort can implicitly encode position at global NoPE layers. Supported by both theoretical and empirical evidence, our central argument is that SWA and gated linear attention induce a recency bias in the residual stream that propagates to, and is selected by, the global attention logits. Moreover, in contrast to the implicit position encodings found in models with only global NoPE attention, in which positional information arises solely from the causal mask, the recency bias in hybrid models can be maintained across long sequences. In addition to deepening our understanding of how hybrid models encode position, these findings may provide insights for how to encode position in a way that can extrapolate to longer sequence lengths indefinitely.