Residual errors impact learning effectiveness in scientific machine learning
The Mechanics of Delta Learning: Target Design for Generalizable Scientific Machine Learning
Machine Learning
Summary
Models that learn from the difference between known physics and real data often assume smaller errors make learning easier. The authors show that just having small errors is not enough. Some errors, though small, can be harder for models to learn because of their complex patterns. They introduce a new way to measure how learnable these errors are and suggest combining different baseline methods to improve learning performance on scientific data.
What this means in practice
- •For computational chemists: Improve molecular energy prediction accuracy by choosing better baseline error targets for machine learning models.
- •For materials simulation teams: Enhance model generalization to new materials by analyzing residual target roughness before training.
Authors
Kareem M. Gameel, Ihor Neporozhnii, Sjoerd Hoogland, Oleksandr Voznyy
Abstract
In scientific machine learning, $Δ$-learning trains models on residual errors relative to physical baselines, assuming that more accurate baselines with smaller residual scales inherently improve downstream performance. Here, we demonstrate that residual scale alone is an insufficient heuristic for learnability. Evaluating molecular graph neural networks on total energy targets, we show that complex local descriptor baselines can yield small residual targets that are disproportionately rough within architecture-informed proxy spaces and harder to learn relative to their scale. Conversely, semi-empirical baseline reduces both scale and normalized roughness, improving in-domain and out-of-domain prediction. We introduce scale-normalized graph Dirichlet roughness ($D_{\text{IQR}}$) as a pre-training diagnostic for residual learnability and establish baseline complementarity as a core target-design principle, elevating target space formulation alongside model architecture as a key axis for scientific machine learning.