Equivariant networks improve training with adaptive blockwise learning rates
Equivariance Breaks the Learning Rate
Machine LearningArtificial Intelligence
Summary
Training certain neural networks called equivariant networks can be tricky because updates to parts of the model don't scale well using common methods like Adam. The authors found that breaking down these networks into smaller blocks and adjusting the training steps for each block separately helps fix this problem. This adjustment improves how well the model learns without adding extra tuning. They tested this idea on simulated and real-world data, showing it helps Adam work almost as well as newer optimizers designed for these networks.
What this means in practice
- •For machine learning engineers: Optimize SO(3)-equivariant neural networks more effectively by applying blockwise normalization to improve training stability and performance.
- •For computational chemists: Train interatomic potential models with better accuracy by tuning optimizer updates per irrep block, enhancing molecular simulations.
Authors
Andrei Manolache, Mathias Niepert
Abstract
Equivariant networks are commonly trained with Adam, yet recent work reports that matrix-structured optimizers such as Muon can perform better on these architectures without explaining why. We identify one source of this difference inside equivariant linear layers. Each irrep block learns a channel-mixing matrix $W_l$ shared across its $2l+1$ components, giving the expanded map $W_l \otimes I_{2l+1}$. For a single application of the layer, the gradient of $W_l$ sums $2l+1$ outer product contributions and has rank at most $2l+1$. Adam rescales stored weights individually without using the irrep boundaries, so one learning rate can produce different spectral step sizes across blocks within a layer. We address this mismatch by normalizing each block update separately, without introducing a new hyperparameter. This changes only the scale of the update, leaving Adam's moment estimates and its direction within each block unchanged. We evaluate the mechanism in a controlled $\mathrm{SO}(3)$-equivariant model with a matched dense control and in an e3nn interatomic potential model trained on rMD17 and MD22. The toy setup isolates a mismatch that grows with width while the dense control shows no corresponding growth. In the interatomic potential model, block normalization and tuning Adam's momentum coefficients independently improve performance, but neither alone matches Muon. Combined, they make Adam competitive with Muon on all datasets, indicating that blockwise step control and momentum accumulation account for much of Muon's advantage.