PYPM-GGD: Pitman-Yor Process Mixture with Generalized Gaussian Density using ADAM
2026-07-27 • Machine Learning
Machine Learning
AI summaryⓘ
The authors address a challenge in large-scale Bayesian learning where existing methods rely on simplifying assumptions that don't hold for all problems. They propose a new technique using adaptive step sizes in stochastic gradient ascent to handle more complex, non-conjugate posterior distributions without needing explicit formulas. Their method works well with deep neural network features on large datasets and performs competitively with recent deep clustering approaches. Overall, their approach extends scalable Bayesian learning to more complex models efficiently.
Bayesian nonparametricsStochastic Variational InferenceNon-conjugate posteriorStochastic gradient ascentAdaptive step sizesVariational posteriorMonte Carlo methodsDeep clusteringResNet featuresLarge-scale learning
Authors
Kart-Leong Lim
Abstract
Large scale Bayesian nonparametrics (BNP) learner such as Stochastic Variational Inference (SVI) can handle datasets with large class number and large training size at fractional cost. Like its predecessor, SVI rely on the assumption of conjugate variational posterior to approximate the true posterior. A more challenging problem is to consider large scale learning on non-conjugate posterior. Recent works in this direction are mostly associated with using Monte Carlo methods for approximating the learner. However, these works are usually demonstrated on non-BNP related task and less complex models such as logistic regression, due to higher computational complexity. In order to overcome the issue faced by SVI, we develop a novel approach based on the recently proposed constant stepsize stochastic gradient ascent to allow large scale learning on non-conjugate posterior. Unlike SVI, our new learner does not require closed- form expression for the variational posterior expectatations. Our only requirement is that the variational posterior is differentiable. In order to ensure convergence in stochastic settings, SVI rely on decaying step-sizes to slow its learning. Inspired by SVI and Adam, we propose the novel use of adaptive stepsizes in our method to significantly improve its learning. We show that our proposed methods is compatible with ResNet features when applied to large class number datasets such as MIT67 and SUN397. Finally, we compare our proposed learner with several recent works such as deep clustering algorithms and showed we were able to produce on-par or outperform the state-of-the-art methods in terms of clustering measures.