Compressed subspaces speed up uncertainty analysis in large models
Compressed Active Subspaces for Scalable Bayesian Inference
Machine LearningArtificial Intelligence
Summary
High-dimensional models often have many parameters, making it hard to understand which ones really affect the outcome. The authors propose a way to shrink these parameters into a smaller, compressed set without losing important information. This makes it easier and less memory-intensive to estimate uncertainty in predictions, especially for big models like neural networks. Their method still gives good predictions and reliable uncertainty estimates while being much more scalable.
What this means in practice
- •For machine learning engineers: Perform Bayesian uncertainty estimation on large neural networks where traditional methods are too memory-heavy.
- •For statistical modelers: Analyze models with many parameters more efficiently by working on compressed parameter spaces without losing key sensitivity directions.
Authors
Thomas Flynn, Sanket Jantre, Byung-Jun Yoon, Kibaek Kim
Abstract
Active subspace methods provide a framework for quantifying predictive uncertainty in high-dimensional models by identifying and performing inference along parameter directions that have the greatest influence on the model output. However, the construction of active subspaces requires storing many full-dimensional model gradients, which becomes prohibitive as model size increases. We address this limitation by proposing Compressed Active Subspaces (CAS), a scalable approach that first maps the model parameters to a compressed space using a structured isometric embedding and then constructs the active subspace within this reduced parameterization. Our approach substantially reduces the memory required for active subspace construction and enables Bayesian inference for large models where standard active subspace methods become impractical. We demonstrate the scalability of CAS on neural networks of increasing size while maintaining predictive performance and robust uncertainty estimates.