Symmetry limits model capacity and breaks rotation invariance
Symmetry Breaking Is the Price of Finite Capacity
Machine Learning
Summary
When a problem looks the same if you rotate it, you might expect the best model to treat all directions equally. This paper shows that when a model has limited size or capacity, it cannot maintain this perfect equality and must break the symmetry in a predictable way. The authors calculate exactly how much worse a symmetric model does compared to one that breaks symmetry, especially for models like attention heads in AI. They also provide methods to separate this effect from other reasons a model might break symmetry.
What this means in practice
- •For neural network engineers: Design attention mechanisms considering that limited capacity forces breaking of rotational symmetry at certain model sizes, improving understanding of model behavior.
- •For signal processing teams: Analyze Gaussian signal models with block symmetries, adjusting designs based on predictable performance loss from symmetry constraints under capacity limits.
Authors
Maggie Chlon, Leon Chlon
Abstract
If rotating a problem never changes its answer, one might expect its best model to be rotation-invariant; we show that this fails, predictably and by an exact amount, when capacity is limited. A symmetry groups a model's directions into blocks of equally strong directions, a symmetric model must use each block whole, and when the budget cuts a block every optimal model breaks the symmetry. For a Gaussian problem with a $k$-unit bilinear predictor, the log-loss price of symmetry is exactly $(U_k-I_k)/(2σ^2)$, the signal of the best $k$ directions minus that of the best whole blocks. Because an attention head compares tokens through a similarity of rank at most its head dimension, whatever computation precedes it, the same price governs heads fitted to a symmetric similarity by squared error, and heads trained this way on three-dimensional directions break symmetry at exactly the predicted head dimensions, with the predicted rotation sensitivity (RMSE 0.004). Under softmax attention, flatter attention restores the predicted symmetric head where tested, while with peaked attention broken heads beat every whole-shell symmetric head at 16 of 20 head dimensions. We give a procedure that separates capacity-driven breaking from other causes.