Physics-informed networks train faster with adaptive gradient method
Gradient Surgery for Physics-Informed Neural Networks
Machine Learning
Summary
Physics-informed neural networks try to solve physics problems using patterns in data but often struggle to learn efficiently because different parts of the learning process interfere with each other. The authors studied these conflicts during training and found they happen in different phases with changing types of interference. They created a new method called PAM-GS that adjusts how the neural network learns based on these interference patterns. Their approach led to faster, more stable training and better balance between tasks on several physics problems.
What this means in practice
- •For simulation engineers: Improve solving complex physics simulations by training neural networks more effectively with adaptive gradient conflict handling.
- •For computational fluid dynamics teams: Enhance training stability for neural network models that approximate solutions to challenging flow equations with variable frequency behaviors.
Authors
Thomas Borsani, Giuseppe Di Fatta
Abstract
Physics-Informed Neural Networks (PINNs) are trained by optimising a composite objective that combines data fitting with physics-based constraints, typically resulting in a highly imbalanced multi-task optimisation problem. Under these conditions, existing optimisation strategies are affected by conflicting task gradients, leading to slow convergence and unstable training, particularly for stiff and high-frequency partial differential equations. We analyse gradient conflicts throughout training of PINNs with standard optimiser and investigate Multi-Task Deep Learning (MTDL) optimisation methods. In our analysis across four benchmark problems we observed that PINN optimisation exhibits three distinct phases in which angle- and magnitude-based gradient conflicts alternate, with only one present at a time. Building on these observations, we propose PAM-GS, a physics-aware gradient surgery method that adaptively mitigates task interference during training according to the observed conflict types. Experiments on four representative PDE benchmarks demonstrate that PAM-GS combines competitive solution accuracy with consistently strong task-balanced performance, outperforming existing methods on most problems.