Modest tuning greatly improves neural operator performance on large physics data
Performance of Neural Operator Baselines on 'The Well' Dataset
Machine Learning
Summary
Training machine learning models to predict complex physical systems often relies on benchmarks. The authors show that previous baseline models trained with strict time limits and little tuning are not the best measures of performance. By doing a small amount of extra tuning on a key model type, the authors greatly reduced prediction error on most tasks, even with stricter input conditions. This means earlier benchmarks should not be taken as the final word on model effectiveness.
What this means in practice
- •For simulation engineers: Improve surrogate models accuracy for complex physics simulations by modest hyperparameter tuning within tight compute budgets.
- •For software developers in scientific computing: Use tuned neural operators as more reliable baselines for developing machine learning models of physical systems.
Authors
Yash Pincha, Andy Dimnaku, Andrew Chiang, Adarsh Ganeshram, Valentin Duruisseaux, Anima Anandkumar
Abstract
The Well, a 15-terabyte dataset spanning 16 physical simulation domains, provides an extensive benchmark for developing and evaluating machine learning surrogate models for complex physical systems. The baselines released with the benchmark were presented by their authors as illustrative examples obtained at small computational cost, trained with zero to very limited hyperparameter tuning and under a highly constraining 12-hour per-run wall-clock limit. These results have subsequently been treated as fully tuned benchmarks, and in some cases effectively interpreted as unconstrained performance baselines. We perform a short non-exhaustive hyperparameter search for the Tucker-factorized Fourier Neural Operator (TFNO) under the same 12-hour per-run wall-clock limit, while using only a single input timestep rather than the history of up to four used by the published baselines. Despite this more restrictive setup, tuning alone reduces VRMSE below both published FNO and TFNO baselines on 15 of 17 tasks, sometimes by several orders of magnitude. This sensitivity to a modest tuning effort underscores that the released neural operator baselines should not be treated as architectural benchmarks.