Learning from Noise: Effective-Rank Collapse and Out-of-Distribution Rejection in Restricted Boltzmann Machines
2026-07-11 • Machine Learning
Machine Learning
AI summaryⓘ
The authors studied a type of model called Restricted Boltzmann Machines (RBMs), which sometimes mistakenly classify unfamiliar data instead of rejecting it. They found that the way the model's internal connections spread out influences this problem. By training the RBM with extra random images labeled for rejection, the authors made the model focus on fewer important connection patterns. This change helped the model better reject unfamiliar images while still correctly identifying familiar ones like MNIST digits.
Restricted Boltzmann Machinesenergy landscapeout-of-distributionvisible-hidden weightseffective rankMarchenko-Pastur distributioneigenvaluesfree-energy landscaperandom binary imagesMNIST dataset
Authors
Oshada Rathnayake, Nikhil Shukla
Abstract
Restricted Boltzmann machines (RBMs) represent data by shaping an energy landscape over visible and hidden configurations, but their discriminative use is fragile under out-of-distribution (OOD) inputs: samples outside the training distribution can be absorbed into one of the learned class basins rather than rejected. Here, we analyze this failure mode through the spectrum of the induced visible--visible interaction $J=WW^{T}$, where \(W\) is the visible--hidden weight matrix. Relative to a Marchenko--Pastur random-matrix reference, conventional training spreads spectral weight into many weak, bulk-compatible directions, increasing the effective rank of $J$. When auxiliary random binary images are assigned to a rejection label during training, the learned interaction undergoes effective-rank collapse: weak bulk-like modes are depleted, spectral weight concentrates into fewer dominant eigendirections, and the effective rank of $J$ approaches that of the empirical data covariance matrix. The resulting RBM rejects structured OOD image datasets while preserving MNIST classification accuracy, showing that random auxiliary exposure can reshape both the interaction spectrum and the free-energy landscape of an energy-based classifier.