Analogies between Transformer Layers and Power Method
2026-05-25 • Machine Learning
Machine Learning
AI summaryⓘ
The authors show that the math inside one layer of a transformer (ignoring some parts) is similar to a known math method called the power method. Because of this, when tokens pass through a layer, they tend to get closer to a special direction defined by a main vector from certain weight matrices. This effect is clearer when all layers share the same weights, and they back this up with math and experiments. Their idea also suggests a way to push the transformer’s output toward any chosen direction.
transformerlayer normalizationpower methodprincipal eigenvectorweight matricestoken representationshared weightsprojectioneigenvector alignment
Authors
Chenglong Li, Claudio Altafini
Abstract
In the paper we show that there is an analogy between the operations occurring in a layer of a transformer (projections and layer normalizations, disregarding the feedforward neural network) and a step in the power method. Coherently with this analogy, we show that passing through a layer the tokens tend to be tilted towards the principal eigenvector of a matrix which is the product of the output and value weight matrices of that layer. In the special case of a transformer with shared weights (i.e., in which all layers have identical weights) then the alignment with this principal eigenvector is particularly evident empirically, and can also be shown analytically. The analogy also suggests a method to steer the output of the transformer towards an arbitrary desired direction in token space.