Effects of width-dependent model hyperparameters and $\ell_2$-regularization on the loss landscape of two-layer ReLU networks
Machine Learning
Summary
The authors studied simple two-layer neural networks to better understand how certain settings affect learning. They found conditions where the best solutions make the network weights shrink to zero, meaning the network stops learning meaningful patterns. Their experiments showed that using the AdamW optimizer avoids this problem, unlike SGD, which helps explain why AdamW works well in practice. They also solved the problem exactly when the input has only one feature and found that regularization affects connections in the network differently depending on its width. Overall, the authors provide insights into how network size and hyperparameters shape the behavior of neural networks.
Authors
Haruka Eshima, Makoto Yamada
Abstract
Understanding deep neural networks remains a central challenge in machine learning. In particular, the theoretical properties of even two-layer ReLU networks, especially in the presence of weight decay, remain poorly understood. To this end, we derive a sufficient condition on the hyperparameter settings under which the global minima collapse to the zero solution. Interestingly, our experiments reveal that using AdamW as an optimizer prevents the collapse of the learned parameters, whereas using SGD does not, which may help explain the success of AdamW in deep learning training. In addition, when restricting the input dimension to one, we derive an analytical solution for the globally optimal parameter sets of two-layer ReLU networks and show that $\ell_2$-regularization has a width-invariant effect on connectivity, but its dimensionality-reducing effect becomes stronger as the network width increases. These results provide insight into how width-dependent hyperparameters influence the geometry of regularized loss landscapes.