There and Back Again: Bidirectional Diffusion Bridges for Multimodality Translation
Machine Learning
Summary
The authors introduce BIT, a new method for translating between text and images that works both ways, unlike previous one-way approaches. BIT starts from text and gradually turns it into images while keeping track of the original source, allowing more flexible ways to generate content. It also works in reverse, converting images back into text, creating a unified system. Their method is based on mathematical tools called stochastic differential equations and performs well compared to existing techniques on various vision and language tests.
Authors
Gabe Guo, Elon Litman, Thanawat Sornwanee, Jose Blanchet, Stefano Ermon
Abstract
Multimodality translation (e.g., text-to-image) is a core generative AI task. However, existing approaches (1) follow generative paths that do not directly represent the source modality, limiting the flexibility of some sampling algorithms; and (2) are unidirectional, preventing inversion (e.g., image-to-text). We propose BIT: Bidirectional Image-Text Diffusion Bridges. In contrast to previous approaches, BIT starts directly from text and interpolates into images, providing (1) a source-aware generative path that enables diverse and flexible sampling algorithms; and (2) an endpoint-conditioned process that can be traversed from image to text, providing a unified, bidirectional generative framework. BIT is derived through stochastic calculus, yielding SDE forms amenable to simulation and tractable loss functions that scale to high dimensions. Our experiments show that BIT is competitive with denoising-diffusion and deterministic-flow baselines, and outperforms them on several vision--language and natural-science evaluations.