Linear models can predict better when complexity varies across settings
Why Linear Models Can Still Work in a Complex and Changing World
Machine Learning
Summary
Sometimes simple linear models predict better than complex ones, even in complicated situations. The authors explain this happens when the parts of the system that change a lot are nonlinear, causing complex models to struggle when faced with new environments. They show that if the unpredictable factors affect data differently in different places or times, dropping the nonlinear parts can make predictions more reliable. They suggest choosing models based on how stable they perform across different settings, not only how well they fit the training data.
What this means in practice
- •For hospital data teams: Choose prediction models that remain stable across different hospitals and time periods to improve patient outcome forecasting.
- •For financial risk analysts: Use linear models that omit unstable nonlinear effects to better predict risks under changing market conditions.
A theory result. No direct application yet.
Authors
Fangzhou Lin, Dengzhe Hou, Yinxing Li, Kazunori D Yamada
Abstract
Linear models can predict well in complex domains even when underlying mechanisms are nonlinear. We argue that this can occur because the complexity of the world exceeds the complexity that can be learned reliably from finite, partially observed data. Flexible models can exploit curvature, thresholds, and interactions that fit one hospital, period, population, policy setting, or measurement process but fail elsewhere. When this instability lies mainly in the nonlinear component, omitting it can give linear models lower prediction error in a new environment. We connect this claim to work on simple linear rules, bias and variance, regularization, best linear prediction, measurement error, distribution shift, invariance, Rashomon sets, and underspecification. Proposition 1 gives a simple squared-error case in which a nonlinear rule learned in the training environment raises target error because it transfers too weakly; finite-sample estimation error can strengthen this result. Partial observation alone is not enough: the link between observed variables and unobserved factors must vary across environments. Stable nonlinear structure favors flexible models, and linear models remain vulnerable to linear spurious relationships. Model complexity should therefore be chosen by cross-environment stability as well as training-data accuracy.