Self-attention in transformers focuses on few dominant directions over many tokens
Spectral Degeneration of Self-Attention under Isotropic Score Geometry
Machine LearningArtificial Intelligence
Summary
Transformer models use a mechanism called self-attention to decide which parts of input data to focus on. This paper studies the math behind how this attention works in detail. The researchers found that attention tends to concentrate on a small number of important directions while ignoring a large number of less important ones. They explain why this happens mathematically, based on properties of the attention scores and transformations applied. This insight may help understand and improve how transformers process information.
What this means in practice
- •For machine learning engineers: Design transformer architectures with attention mechanisms that maintain focus on key features while compressing less relevant data efficiently.
- •For nlp system builders: Optimize attention computations in language models by exploiting dominant attention directions to reduce computational complexity.
Tested on one dataset.
Authors
Mengda Li, Jianfeng Yao
Abstract
We study the spectral structure of the self-attention matrix in Transformer models through an idealized spectral model for the pre-softmax score matrix. Motivated by the rotational isotropy of query and key representations, our model replaces the random token-space orientation of the score matrix by its canonical singular-value representative with a logarithmic spike--bulk separation. Empirical diagnostics on pretrained vision and language transformers motivate this model: token-space singular matrices have nearly uniform eigen-angle distributions, and score spectra are dominated by a small number of effective directions. We then prove that the row-wise softmax map transforms this structure into a degenerate attention matrix: dominant modes are preserved, whereas the bulk spectral distributions concentrate at zero. Stable rank remains bounded for bidirectional attention but grows logarithmically for causal attention. Within the canonical model, these results explain how attention can preserve a small number of dominant directions while compressing the remaining bulk modes.