Equivariant networks learn better with blockwise normalization and tuned momentum
Equivariance Breaks the Learning Rate
Machine LearningArtificial Intelligence
Summary
Training special neural networks called equivariant networks is tricky because some standard methods treat parts of the network unevenly, which slows learning. The authors explain that inside certain layers, updates to parameters can get unbalanced due to how the optimizer rescales weights. They propose normalizing these updates separately for each block, which helps balance the learning process without adding extra settings to adjust. Testing this on controlled models and real molecular data shows that their approach, combined with tuning optimizer settings, improves performance and narrows the gap with more complex methods.
Equivariant networksAdam optimizerMuon optimizerIrreducible representationsMatrix factorizationGradient updatesSpectral step sizeBlock normalizationMomentum tuningInteratomic potential
Authors
Andrei Manolache, Mathias Niepert
Abstract
Equivariant networks are commonly trained with Adam, yet recent work reports that matrix-structured optimizers such as Muon can perform better on these architectures without explaining why. We identify one source of this difference inside equivariant linear layers. Each irrep block learns a channel-mixing matrix $W_l$ shared across its $2l+1$ components, giving the expanded map $W_l \otimes I_{2l+1}$. For a single application of the layer, the gradient of $W_l$ sums $2l+1$ outer product contributions and has rank at most $2l+1$. Adam rescales stored weights individually without using the irrep boundaries, so one learning rate can produce different spectral step sizes across blocks within a layer. We address this mismatch by normalizing each block update separately, without introducing a new hyperparameter. This changes only the scale of the update, leaving Adam's moment estimates and its direction within each block unchanged. We evaluate the mechanism in a controlled $\mathrm{SO}(3)$-equivariant model with a matched dense control and in an e3nn interatomic potential model trained on rMD17 and MD22. The toy setup isolates a mismatch that grows with width while the dense control shows no corresponding growth. In the interatomic potential model, block normalization and tuning Adam's momentum coefficients independently improve performance, but neither alone matches Muon. Combined, they make Adam competitive with Muon on all datasets, indicating that blockwise step control and momentum accumulation account for much of Muon's advantage.