AI summaryⓘ
The authors present EquiformerV3, an improved version of a special graph neural network designed to understand 3D atomic systems in a way that respects physical symmetries. They made the model faster by optimizing software, added new techniques like better normalization and smoother attention to improve learning, and introduced a new activation method called SwiGLU-S² to better capture complex interactions while keeping the model mathematically consistent. These changes help the model more accurately predict energy landscapes and their derivatives, which is important for simulating physical systems. When trained with a method to clean up noisy data, EquiformerV3 achieves top results in several benchmarks related to materials science.
SE(3)-equivariancegraph neural networksTransformerequivariant attentionfeedforward networkSwiGLU activationpotential energy surfaceenergy conservationdenoisingmaterials modeling
Authors
Yi-Lun Liao, Alexander J. Hoffman, Sabrina C. Shen, Alexandre Duval, Sam Walton Norwood, Tess Smidt
Abstract
As $SE(3)$-equivariant graph neural networks mature as a core tool for 3D atomistic modeling, improving their efficiency, expressivity, and physical consistency has become a central challenge for large-scale applications. In this work, we introduce EquiformerV3, the third generation of the $SE(3)$-equivariant graph attention Transformer, designed to advance all three dimensions: efficiency, expressivity, and generality. Building on EquiformerV2, we have the following three key advances. First, we optimize the software implementation, achieving $1.75\times$ speedup. Second, we introduce simple and effective modifications to EquiformerV2, including equivariant merged layer normalization, improved feedforward network hyper-parameters, and attention with smooth radius cutoff. Third, we propose SwiGLU-$S^2$ activations to incorporate many-body interactions for better theoretical expressivity and to preserve strict equivariance while reducing the complexity of sampling $S^2$ grids. Together, SwiGLU-$S^2$ activations and smooth-cutoff attention enable accurate modeling of smoothly varying potential energy surfaces (PES), generalizing EquiformerV3 to tasks requiring energy-conserving simulations and higher-order derivatives of PES. With these improvements, EquiformerV3 trained with the auxiliary task of denoising non-equilibrium structures (DeNS) achieves state-of-the-art results on OC20, OMat24, and Matbench Discovery.