Bayesian inference improves by learning summaries and posteriors together
Two-way Guidance: Simultaneous Learning of Summary Statistics for Generative Bayes
Machine Learning
Summary
When using Bayesian methods to understand data, it can be hard to get accurate results as the amount of data grows. The authors show a way to learn key summary information about the data at the same time as learning the full picture, making the process better. They introduce a concept called two-way guidance, where the summary helps estimate the results, and the results help improve the summary. This approach makes the estimates more accurate when handling large data and comes with mathematical guarantees.
What this means in practice
- •For machine learning engineers: Improve probabilistic modeling pipelines by jointly learning data summaries and posterior estimates to handle large datasets more accurately.
- •For computational biologists: Use joint learning of summaries and posteriors to better infer complex biological models from simulations when direct likelihoods are unavailable.
Authors
Percy S. Zhai, Max Pachman, Peverel D. Shipley, Jungeum Kim
Abstract
Bayesian simulation-based inference (SBI) can struggle to recover increasingly concentrated posteriors as observation size grows. We consider learning sufficient summary statistics jointly with the posterior as a general principle for scalable inference in various SBI methods. A common measure taken in SBI is dimension reduction through predetermined summaries, which may partially discard posterior information. Our approach instead imposes structure on how the posterior learner depends on a learned summary (e.g., affinity), and learns the summary simultaneously with the posterior. We identify an important mechanism named two-way guidance: under a structural restriction on the summary, the posterior sufficiency of the learned summary simplifies posterior learning, whereas a suitable posterior objective guides the learned summary towards posterior sufficiency. This mechanism explains a counterintuitive finding that the structural restriction on the summary statistics improves posterior learning, rather than limit its performance. We establish theoretical guarantees on both posterior estimation error and the learned summary's departure from sufficiency, thus identifying conditions on the simulation budget for recovering increasingly concentrated posteriors as observation size grows.