Spectral disentangling improves neural operator efficiency and accuracy
SPEAR: A Spectral-Disentangled MoE Neural Operator with Knowledge-Guided Expert Aggregation for Large-Scale PDE Pretraining
Machine LearningArtificial Intelligence
Summary
Many physics problems are described by equations called PDEs, which can be very different and complex. Models that try to learn these equations sometimes get confused by mixing different patterns or wasting effort on similar solutions. The authors created SPEAR, a new AI approach that splits the problem into simple and complex parts and groups similar solution methods to work better together. This helps the model learn faster, be more accurate, and use fewer resources. Their tests on many different problems showed it works better than earlier methods.
What this means in practice
- •For computational physics teams: Improve simulation accuracy and efficiency by using SPEAR to model complex physical systems governed by PDEs.
- •For engineering simulation developers: Incorporate SPEAR into software to speed up fine-tuning and transfer learning for diverse simulation scenarios.
Authors
Dengdi Sun, Xiaoya Zhou, Xiao Wang, Wanli Lyu, Jin Tang, Bin Luo
Abstract
Large-scale pre-training has improved the generalization of neural operators across diverse PDEs. However, existing PDE foundation models still struggle with heterogeneous dynamics, where shared representations may cause knowledge interference, while mixture-of-experts (MoE) architectures suffer from increasing expert redundancy. We propose SPEAR, a spectral-disentangled MoE neural operator with knowledge-guided expert aggregation for large-scale PDE pre-training. SPEAR decouples latent features into low- and high-frequency components, enabling shared modeling of transferable dynamics and specialized learning of PDE-specific patterns. To address expert redundancy, we design a knowledge-guided expert aggregation strategy that measures expert similarity from dataset-specific learned knowledge and routing preferences, enabling the identification and consolidation of similar experts. Experiments on twelve PDE datasets and multiple downstream benchmarks demonstrate superior performance in pre-training, fine-tuning, and transfer learning. Furthermore, our aggregation strategy reduces the number of experts by 50\% while maintaining or improving prediction accuracy, achieving a balance between model efficiency and generalization for PDE foundation models.