Overparameterized model complexity linked to singularities and dimension
Singularity-Induced Local Complexity for High-Dimensional Learning Models
Machine Learning
Summary
Some machine learning models have more settings than needed, making their best solutions form complex shapes rather than single points. This means simple ways to measure their size or difficulty don’t work well. The authors study a special number called the local RLCT that captures this complexity more accurately. They show how this number relates to other ways of understanding the model’s complexity, helping explain why these models can generalize well despite having many parameters.
What this means in practice
- •For machine learning engineers: Improve model selection by using local RLCT to estimate effective parameter dimension and predict generalization capacity in complex models.
- •For data scientists: Refine complexity estimates for high-dimensional models to better understand model performance and avoid overfitting.
A theory result. No direct application yet.
Authors
Atsuyoshi Muta, Masaaki Imaizumi
Abstract
Many overparameterized models that have achieved remarkable success are singular models. That is, the set of optimal parameters consists of multiple points, or the Hessian of the generalization error at such parameters is degenerate, reflecting redundancy in the model parametrization. Therefore, understanding the effective dimension and intrinsic model complexity, beyond the apparent dimension of the parameter space, is essential for studying the generalization ability of such overparameterized models. In this work, we focus on the real log canonical threshold (RLCT) and its localized counterpart, the local RLCT, which are fundamental quantities for characterizing the generalization of singular models. We clarify the theoretical relationship between the local RLCT and existing two key metrics related to generalization under some localization assumption: (I) the rank of the Hessian matrix of the generalization error and (II) the covering number of a family of loss functions. This result provides novel insights into effective dimension and intrinsic complexity of overparameterized models.