Per-Loss Adapters for Gradient Conflict in Physics-Informed Neural Networks
2026-05-11 • Machine Learning
Machine Learning
AI summaryⓘ
The authors study Physics-Informed Neural Networks (PINNs), which solve physics problems by combining multiple loss functions but often face issues because the gradients from these losses interfere. They find that these gradient conflicts come in different types, each needing a different fix: some need separate pathways for each loss, some benefit from adjusting loss weights, and others need no fix. To decide which fix to use, the authors propose a method that first checks the network's behavior before applying either scalar reweighting or a lightweight architectural change using adapters. Their approach works well across many physics problems, showing that different conflicts require different solutions.
Physics-Informed Neural Networksgradient conflictloss balancinggradient surgeryscalar reweightingparameter subspaceslow-rank adapterspartial differential equationsinverse problemsmulti-physics systems
Authors
Bum Jun Kim, Gnankan Landry Regis N'guessan
Abstract
Physics-informed neural networks (PINNs) train a single neural approximation by minimizing multiple physics- and data-derived losses, but the gradients of these losses often interfere and can stall optimization. Existing remedies typically treat this pathology either through scalar loss balancing or full-parameter-space gradient surgery, leaving it unclear which intervention is most appropriate. We show that PINN gradient conflict is not a uniform failure mode with one universal remedy. Instead, we identify distinct PINN gradient-conflict regimes, each associated with a different intervention class. Persistent directional conflict may require separate loss-indexed parameter subspaces, magnitude imbalance often favors scalar reweighting, and low or transient conflict may require no extra mitigation. To select between scalar reweighting and a lightweight architectural intervention, we propose a diagnostic-first framework. It profiles a 1000-step unmodified PINN run and, when intervention is warranted, uses one low-rank adapter per loss to create explicit loss-indexed parameter subspaces attached to a shared PINN trunk, providing each loss with a direct gradient pathway. Across more than 60 PDE configurations, including forward, inverse, multi-physics, parameter-varying, and high-dimensional problems up to 50D, persistent directional conflict dominates standard forward $K=3$ benchmarks and a natural $K=4$ thermoelastic system, where adapters combined with reweighting yield significant improvements. In contrast, $K=3$ inverse problems and natural $K=5$ and $K=6$ multi-physics systems are largely magnitude-dominated and often favor reweighting alone, while full-parameter-space gradient surgery can fail on heterogeneous parameter spaces.