Gaussian noise models multi-head self-attention spectrum accurately
Gaussian Equivalence for Multi-Head Self-Attention
Machine Learning
Summary
Understanding how multi-head self-attention works in neural networks is important for improving AI models. The authors show that by replacing the complex softmax attention step with simpler rescaled scores plus Gaussian noise, the overall behavior of the output remains the same in a mathematical sense. Their work helps separate how different parts of the attention mechanism contribute to the final output and clarifies when different ways of sharing information between attention heads keep the model’s properties stable.
What this means in practice
- •For neural network architects: Design attention mechanisms with predictable spectral behavior using Gaussian noise approximations to simplify analysis and optimization.
- •For signal processing engineers: Model complex attention outputs in communication systems by substituting softmax with Gaussian noise to understand system stability and performance.
A theory result. No direct application yet.
Authors
Tomohiro Hayase, Ryo Karakida
Abstract
A theoretical understanding of multi-head self-attention is fundamental to the study of modern neural networks. Using random matrix theory, we establish Gaussian equivalence for multi-head self-attention: replacing softmax attention with rescaled scores plus Gaussian noise preserves the limiting spectral law of the centered output. This equivalence also covers value and output projections that depend on the keys. The resulting laws separate the effects of head allocation and projection widths, and distinguish spectrum-preserving across-head sharing from within-head key--value dependence.