From Objectives to What Models Learn: A Landau Theory of Invariant Learning
2026-08-10 • Machine Learning
Machine Learning
AI summaryⓘ
The authors study how objectives in invariant learning change as regularization strength varies, which affects what models learn. They use ideas from physics (like magnetization and free energy) to create a framework that predicts different behaviors of learning objectives. Their theory explains phase transitions, mode selection, and stability as regularization changes, and their experiments confirm these patterns in simple models and deep ReLU networks. This work helps clarify how regularization shapes learned representations in invariant learning.
Invariant learningRegularization pathRepresentation learningLandau free energyPhase transitionMode eliminationReLU networksBilinear modelSpectral phase boundaryCritical strength
Authors
Pinli Wang, Yue He, Peng Cui
Abstract
Invariant learning seeks representations that remain predictive across environments, yet the behavior of its objectives along the regularization path is often opaque. We address this objective-behavior gap by viewing representation learning as multimode magnetization and deriving, from concrete invariant-learning objectives, a Landau-type effective free energy whose low-order coefficients form objective signatures and induce distinct regularization phenotypes. Effective quadratic corrections move the phase boundary and enable finite-strength mode elimination; quartic corrections regulate post-onset amplitude and typically leave residual loading at finite strength; higher-order structure governs non-monotone tails, instability, and collapse at large regularization. In a canonical bilinear model, the theory yields closed-form phase boundaries and steady-state loadings, as well as distinct critical strengths for shortcut and stable modes that define a selective-retention window. Controlled experiments confirm the predicted phase boundaries, loadings, and regularization phenotypes. In one- and two-hidden-layer ReLU networks, the same signatures remain predictive of qualitative regularization-path behavior despite depth-dependent shifts in scale. A matrix extension generalizes the framework to coupled collective modes and yields a spectral phase-boundary criterion. Together, the framework turns low-order objective signatures into predictions of regularization phenotypes and, ultimately, of what models learn as regularization varies.