Transformer prediction errors decrease predictably with data size increase
Neural Scaling Laws of Transformer Operator Network
Machine Learning
Summary
Learning how physical systems behave is complicated, but transformer models have shown they get better when trained on more data and bigger models. The paper explains why this improvement happens by analyzing how transformers approximate and generalize from data, especially for certain smooth problems. The authors found that the error decreases following a power law that depends on the problem's complexity and the intrinsic dimensionality of the inputs. Their work helps understand why transformers perform well and how training data and model size impact accuracy.
What this means in practice
- •For machine learning engineers: Design training regimes for transformer models predicting physical system behaviors more efficiently based on intrinsic input dimensions.
- •For computational physicists: Understand prediction accuracy trends of transformer-based solvers to better plan data collection and model scaling for simulations.
A theory result. No direct application yet.
Authors
Haoran Yan, Zhongjie Shi, Yuanzhe Xi, Peng Chen, Wenjing Liao
Abstract
Transformers have emerged as powerful architectures for learning solution operators of physical systems. Empirically the prediction error has been observed to decrease when the data size and model size increase, suggesting neural scaling behavior. Yet a theoretical understanding of such scaling laws for transformer-based operator learning remains limited. In this work, we develop a theoretical framework for characterizing the approximation and generalization errors of transformer-based operator learning. Our analysis builds on a local-to-global approximation principle that is naturally aligned with the softmax attention mechanism and yields discretization-invariant output functions. On approximation theory, we derive a universal approximation error of transformer-based operator learning for Hölder-regular operators. On generalization theory, we establish a power scaling law between the prediction error and the training data size. The rate of convergence represented by the scaling exponent explicitly reflects the dimensions of the input and output domains, the regularity of the underlying functions and operators, and crucially, the intrinsic dimension of the input function class. By exploiting this intrinsic low-dimensional structure, our analysis yields a power-law generalization rate for operator learning, in contrast to the logarithmic-type power-law rates appearing in existing analyses of operator learning with feedforward neural networks. Numerical experiments validate the predicted power-law scaling and confirm that the convergence rate varies systematically with the intrinsic dimension of the input function class.