Simplifying neural network complexity analysis using model symmetries
Symmetries and Singularities
Machine Learning
Summary
Deep learning models often have many parameters that can produce the same results, making it hard to understand how complex they really are. The authors focus on a way to measure this complexity called the local learning coefficient (LLC), which is usually expensive to calculate for big models. They find that using known symmetries in a specific type of model called a graph attention network can make estimating LLC easier and more practical. This helps better understand how complex these models are without heavy computation.
What this means in practice
- •For machine learning engineers: Evaluate model complexity more efficiently in graph attention networks to inform design and optimization decisions.
- •For data scientists: Use symmetry-based methods to speed up the assessment of neural network generalization without costly sampling.
A theory result. No direct application yet.
Authors
Vishnu Varadarajan, Mihir More, Aritra Das, Debayan Gupta
Abstract
Deep neural networks are highly over-parameterized, and different parameter values represent the same predictive function. This makes their effective complexity difficult to measure using only the number of parameters or the rank of the Hessian. Singular Learning Theory addresses this issue through the local learning coefficient (LLC), which characterizes the effective complexity of a model near a given solution. Existing methods for estimating the LLC often rely on posterior sampling, which can be computationally expensive for large neural networks. This makes accurate LLC estimation difficult at scale. In this work, we use known structures in the model to simplify the analysis and make LLC estimation more tractable. Specifically, we study the LLC of a graph attention model by exploiting symmetries in both the graph structure and the attention parameters. An analytic framework through a teacher--student setting, and explicit LLC estimates after considering the symmetry--induced degeneracies are developed.