Bregman approach improves training and accuracy of neural operators
A Bregman Proximal Viewpoint on Neural Operators
Machine Learning
Summary
Neural operators are tools that help computers learn how to perform complex mathematical tasks on functions. This paper shows a new way to understand parts of these tools using a mathematical concept called Bregman proximity operators, which helps explain how the neural operators work and allows for creating a new kind of neural operator. The authors introduce Bregman neural operators that can do the same jobs as traditional ones but may be better at learning deeper and more accurate models. Tests show these new operators can improve certain neural operator methods, especially when working with Fourier transforms.
What this means in practice
- •For machine learning engineers: Train deeper and more accurate neural operator models for scientific problems involving function approximation using the improved Bregman neural operators.
- •For signal processing developers: Enhance Fourier transform-based neural models to better handle complex function transformations with the Bregman approach.
Authors
Abdel-Rahim Mezidi, Jordan Patracone, Saverio Salzo, Amaury Habrard, Massimiliano Pontil, Remi Emonet, Marc Sebban
Abstract
We present several advances on neural operators by viewing the action of operator layers as the minimizers of Bregman regularized optimization problems over Banach function spaces. The proposed framework allows interpreting the activation operators as Bregman proximity operators from dual to primal space. This novel viewpoint is general enough to recover classical neural operators as well as a new variant, coined Bregman neural operators, which includes the inverse activation operator and features the same expressivity of standard neural operators. Numerical experiments support the added benefits of the Bregman variant of Fourier neural operators for training deeper and more accurate models.