Variational Boosting for Physics-Informed Neural Networks
2026-07-27 • Machine Learning
Machine Learning
AI summaryⓘ
The authors address problems with traditional Physics-Informed Neural Networks (PINNs), which can be hard to train and sometimes unstable. They propose a new method that builds the solution step-by-step by adding small correction networks, each focused on improving the solution in a controlled way. This approach breaks a big hard problem into smaller, easier ones and allows for more stable and efficient training using advanced optimization techniques. Their method also offers a clear geometric view of how the learning process works. Overall, this makes solving nonlinear differential equations with neural networks more reliable.
Physics-Informed Neural NetworksDifferential EquationsVariational MethodsFunctional Gradient DescentNewton MethodConjugate GradientOptimizationSpectral BiasFunction ManifoldNonlinear Operator
Authors
Pavlos Protopapas, Kaylee Vo
Abstract
Physics-Informed Neural Networks (PINNs) solve differential equations by minimizing the residual of a nonlinear operator over a neural parameterization of the solution. However, monolithic PINNs often suffer from ill-conditioning, spectral bias, and optimization instability. We introduce a variational boosting framework in which solutions are constructed additively in function space. Each stage trains a weak learner whose converged correction satisfies a local orthogonality condition, equivalent to a projected functional gradient descent step onto the tangent space of the network's function manifold. Because each correction network is deliberately small, the restricted minimization admits full Newton or conjugate gradient updates, which are typically infeasible in large PINNs. The resulting method separates global nonlinear refinement into a sequence of well-conditioned subproblems while preserving the full variational structure of the operator. This framework provides a geometric interpretation of multi-stage PINNs as projected functional gradient descent and enables stable second-order optimization for nonlinear differential equations.