Transport methods improve sampling from complex multimodal distributions

Transport along Local Annealing Paths: Wasserstein Geometry and Multimodal Sampling

Machine Learning

Summary

Sampling from complicated distributions that have multiple peaks can be hard because methods often get stuck on some peaks and miss others. The authors show that moving samples along a path between simple and complex distributions can be inefficient, especially if mass moves between distant modes. They propose a new method called componentwise annealed transport (CAT) that breaks down the problem by targeting each mode separately, which helps with learning the transport maps and speeds up sampling. Their experiments show that this approach improves sampling from challenging multimodal cases.

What this means in practice

  • For machine learning engineers: Improve sampling techniques for models with complicated multimodal distributions to enhance training and inference quality.
  • For bayesian data analysts: Generate more accurate samples from complex posterior distributions to improve statistical inference in Bayesian modeling.

Authors

Han Chen, Sifan Liu

Abstract

Variational approaches to learning transport maps for multimodal sampling can suffer from mode collapse. One way to mitigate this problem is to learn transport along a prescribed path of intermediate distributions defined by geometric annealing. We show that annealing can take a substantial detour in Wasserstein space: when mass moves between modes, the path can be much longer than the distance between its endpoints. We further link path geometry to transport learning, showing that long annealing paths can slow population optimization, while sufficiently regular paths permit rapid convergence. Motivated by these findings, we propose componentwise annealed transport (CAT) to reduce the need for mass transfer between modes. Starting from a mode-informed mixture reference, CAT constructs an exact mixture decomposition of the target and learns separate transport maps along componentwise annealing paths. We estimate the component weights separately and show that these estimates remain consistent even when the learned flows are inexact. We also develop adaptive annealing schedules guided by transport effort. Experiments on multimodal synthetic distributions and Bayesian inference problems demonstrate improvements in transport learning and sampling efficiency.