Statistical complexity bounds for sparsely activated neural networks
Nearly Tight Rademacher Bounds for Sparsely Activated Neural Networks
Machine Learning
Summary
This paper studies how complicated neural networks are when only a few hidden neurons activate for each input, even if different inputs use many neurons overall. The authors provide tight mathematical bounds on how well these sparse neural networks can generalize from data, improving previous results by removing overly large dimension factors. They also show that the way inputs activate neurons affects learning difficulty and offer risk estimates based on network width, sparsity, and sample size. These findings clarify how sparse activations influence learning performance in shallow ReLU networks.
What this means in practice
- •For machine learning engineers: Understand how input-dependent neuron activation affects model generalization and inform network design choices under sparsity constraints.
- •For data scientists: Estimate sample sizes needed for reliable training of one-hidden-layer ReLU networks with sparse activations based on theoretical bounds.
A theory result. No direct application yet.
Authors
Xiaoyu Li, Zhizhou Sha, Jiaojiao Jiang, Junbin Gao, Andi Han
Abstract
An input may activate few hidden units even when different inputs collectively use an entire network. We study the statistical complexity of this input-dependent sparsity in the one-hidden-layer ReLU model of Awasthi et al. (COLT 2024). For width $s$, at most $k$ active units per input, and effective weight and bias bounds $W,B$, every size-$m$ sample in the class's fixed radius-$R$ input domain satisfies $\mathcal{R}(S)\le CWR\min\{k,\sqrt{sk/m}\log^{3/2}(2m)\}+kB/\sqrt m$. A support-preserving cover and a single normalized chaining argument remove the previous explicit dimension factor, up to logarithms. Lower bounds on appropriate i.i.d. marginals match up to those logarithms, showing how changing active units across inputs retains a width dependence. The input domain matters: zero-bias networks sparse on the entire ball have at most $2k$ nonzero units and complexity $O(kWR/\sqrt m)$, whereas bias bounds comparable to $WR$ restore the worst-case rate on that same domain in only logarithmic dimension. A spherical-cap construction proves the latter claim without assuming sparsity merely on the sampling support. For a specified normalized bounded loss and biases comparable to $WR$, we also obtain agnostic minimax excess-risk bounds of order $\min\{1,\sqrt{s/(km)}\}$ up to logarithms.