New way to measure neural network accuracy without size effects
Dimension-Free PAC-Bayes Generalization for Deep Neural Networks via Dirichlet Processes
Machine Learning
Summary
Traditional ways to predict how well deep neural networks will perform get worse when the networks become very large because they depend heavily on counting the number of parameters. The authors propose measuring generalization by focusing on the data distribution instead of the network parameters, using a mathematical tool called the Dirichlet Process. This approach removes dependence on network size and provides stable, meaningful guarantees even for networks with millions of parameters. Their method was tested on popular image recognition tasks and showed much tighter and practical bounds compared to older methods.
What this means in practice
- •For deep learning engineers: Provide reliable confidence estimates for large neural networks without being affected by their size or complexity.
- •For machine learning model auditors: Use stable and mathematically sound generalization certificates to assess model reliability across diverse architectures and datasets.
Authors
Sumit Vashishtha
Abstract
Classical PAC-Bayesian generalization theory places prior distributions over Euclidean parameter space $Θ\subseteq \mathbb{R}^D$, causing relative entropy to scale with network dimension $D$. For modern overparameterized neural networks where $D \gg n$, this parameter-counting dependency yields vacuous generalization bounds ($> 1{,}000\%$). In this work, we resolve this barrier by shifting prior specification from internal parameters to the non-parametric space of data distributions $\mathcal{M}_1(\mathcal{Z})$ via a \textbf{Dirichlet Process (DP)}, $\mathcal{DP}(α, F_0)$, inducing parameter distributions through push-forward measures under the learning algorithm. By the Data Processing Inequality, parameter-space relative entropy is strictly bounded by the data-space DP divergence, which we evaluate in exact closed form using Digamma functions. The resulting PAC-Bayesian certificate is strictly invariant to network parameter count $D$, width, and depth. On intrinsic $d^*$-dimensional submanifolds, optimal partition scaling achieves the minimax non-parametric rate $\mathcal{O}(n^{-1/(d^*+2)} \sqrt{\ln n})$. Across vision architectures scaling up to 4.2 million parameters ($D = 7{,}850$ to $D = 4{,}203{,}522$) on the full MNIST benchmark, while classical weight-space bounds scale past $940\%$ (becoming vacuous), our data-space DP-PAC-Bayes certificate remains stable, non-vacuous, and certified at \textbf{$8.8\%$--$9.4\%$}.