HotPOT improves predictions for multiple physics equations at once
HotPOT: Operator Transformer with Heterogeneous Physics Experts for Large-Scale PDE Pre-training
Machine Learning
Summary
Predicting how physical systems change over time can be very tricky, especially when different systems follow different rules. The authors show that training one model for many different physical equations often fails because the systems interfere with each other. They develop HotPOT, which uses shared understanding but different expert components tailored to each kind of physics problem, improving accuracy a lot. Their approach works better than previous methods, even with smaller models, and can make good predictions for new situations it hasn't seen before.
What this means in practice
- •For engineering simulation teams: Build more accurate and scalable simulations for various physical systems by training a single model that adapts to different PDE families with specialized expert components.
- •For computational physics developers: Use HotPOT to extend predictive models to unseen physical conditions without retraining, enabling flexible testing of physical scenarios.
Authors
Jiayi Li, Guancheng Zhou, Yisi Luo, Ruihua Chen, Penghao Jiang, Deyu Meng, Flora D. Salim, Piotr Koniusz
Abstract
Large-scale PDE pre-training trains a single model on the data from multiple partial differential equations (PDEs). However, adding heterogeneous PDEs with different governing equations, physical variables, and variable interactions does not always improve predictive performance across tasks. Existing PDE mixture-of-experts models mitigate cross-task interference through learned routing, but typically use homogeneous expert architectures, making it difficult to directly reflect differences in state evolution across equations. We introduce HotPOT, an operator transformer with heterogeneous physics experts for large-scale PDE pre-training. HotPOT uses a shared encoder to capture information common across PDEs, while each PDE family uses a neural-numerical expert that pairs a learnable predictor with a fixed numerical update containing no trainable parameters. Each predictor outputs the quantities required by its paired numerical update, which then computes the next state. This design allows different PDEs to share a common encoder while retaining their own learnable predictors and numerical updates. Experiments across five PDE families and six settings show that HotPOT outperforms MoE-POT in all 24 matched-scale comparisons before and after fine-tuning, reducing rollout error by 50.7% on average. Notably, the 17M model also surpasses substantially larger MoE-POT models, and the performance gains extend to zero-shot prediction under unseen physical conditions. Code is available at https://github.com/cruiseresearchgroup/HotPOT-code.