SS-ESOAP: Self-Scaled Adaptive Preconditioning for Physics-Informed Learning
Machine LearningArtificial Intelligence
Summary
The authors address challenges in training physics-informed neural networks (PINNs) due to difficult optimization problems. They propose a new method called [1m[36mMethod[0m that improves an existing technique called SOAP by adding a special correction and an adaptive update process to make training more stable and efficient. Testing on several benchmark problems, their method achieves lower errors faster than common optimizers like Adam and some SOAP variants, especially for harder equations like Boussinesq. However, it is not always better on every problem, suggesting it is a useful option for specific cases needing high-accuracy solutions rather than a general replacement.
Physics-informed neural networks (PINNs)OptimizationQuasi-Newton methodsKronecker-factored preconditioningSOAPSecant correctionPartial differential equations (PDEs)ResidualBurgers equationBoussinesq equation
Authors
Guangyuan Wang, Mads Toftrup, Sebastian Loeschcke, Yixuan Wang, Anima Anandkumar
Abstract
Physics-informed neural networks (PINNs) often face ill-conditioned objectives that limit high-accuracy training. Dense quasi-Newton methods improve local conditioning but require expensive optimizer state, while Kronecker-factored methods such as SOAP scale to larger networks but rely on periodic basis updates. We introduce \method, which augments SOAP-style preconditioning with a scalar secant-energy correction adapted to Kronecker geometry and an adaptive basis update followed by variance-state downscaling. We characterize the directional secant matching induced by the scalar correction and give a bound on variance-state mismatch across basis changes. Across eight PDE benchmarks, \method attains the lowest final residual on six, including Burgers and Boussinesq, while SOAP-family baselines perform better on Gray-Scott and Ginzburg-Landau. On Boussinesq, \method reaches a residual of $10^{-5}$ in 4.1 hours with 9.2 GB peak VRAM, while Adam does not reach this target within 14 hours. Three-seed $L^2$ and $H^1$ errors on four representative PDEs support the link between lower residuals and improved solution accuracy. These results position \method as a scalable option for stiff, high-accuracy physics-informed training, rather than a uniform replacement for existing optimizers.