Transformer attention equality condition holds widely in popular AI models
An equality condition for the Dobrushin bound on attention rollout and how often it holds in trained transformers
Machine Learning
Summary
The authors studied a mathematical condition that can exactly describe how attention layers in transformer AI models behave when their effects are combined. They found that this condition isn’t always true, especially in random settings, but it does hold almost everywhere in many trained language and vision transformers. Some models used for images and text show this condition mostly holds through all layers, while others, including tabular data models, have lower rates. This finding helps understand how attention layers work in practice under different training scenarios.
What this means in practice
- •For transformer model developers: Use the equality condition to better analyze attention mechanisms layer by layer and diagnose attention behavior in trained transformer-based AI models.
- •For machine learning engineers: Improve interpretability tools for transformers by incorporating this exact equality condition to explain token attention dynamics across model depth.
Authors
Przemysław Rola
Abstract
The Dobrushin coefficient of each attention-rollout factor satisfies $κ(\frac12(I+A))\le\frac12(1+κ(A))$, and multiplying these inequalities over layers bounds the coefficient of the whole rollout. We characterise exactly when the layerwise bound is tight: equality holds if and only if some token pair attaining $κ(A)$ is mutually self-dominant - each of the two attends to itself at least as strongly as the other attends to it. The condition is far from automatic: uniformly random stochastic matrices satisfy it only 24-30% of the time. When tested on the head-averaged attention of each individual input and restricted to content tokens - image patches, words or tabular features, excluding cls, register and separator tokens - the condition holds for essentially every input at every layer of DINOv2 (three model sizes), RoBERTa and DistilBERT. In the supervised models DeiT-B and ViT-B/16 it holds for 91% and 64% of input-layer pairs respectively, with all failures occurring late in depth. In FT-Transformer trained on two standard tabular benchmarks it holds for only 11-44% of input-layer pairs. The special tokens account for almost all failures in DINOv2 and the language models: when they are included, the condition holds for only 82-97% of input-layer pairs.