MUNITE enables flexible generation between any combination of modalities
MUNITE: Unified Multimodal Latent Inference for Any-to-Any Multimodal Generation
Machine Learning
Summary
Multimodal generation means creating outputs that mix different types of data like images, text, and audio. The new MUNITE method treats the process of understanding inputs and generating outputs as part of one system that works regardless of which data types are available. It uses a shared hidden space to keep consistent information across outputs, even if only some inputs are given. The authors tested MUNITE on various datasets and found it creates high-quality, coherent results across many tasks involving images, text, and sound.
What this means in practice
- •For multimedia app developers: Build apps that generate or convert between images, text, and audio flexibly from any available input types.$Commercial implications: Enables products offering seamless cross-modal content generation and transformation, enhancing user experience with flexible input handling.
- •For content creation teams: Create consistent multimedia content by generating coherent images, text, and audio from partial or mixed inputs.
Authors
Kyeongmin Yeo, Minhyuk Sung
Abstract
We introduce MUNITE, a latent-variable framework for flexible any-to-any multimodal generation that treats encoding and latent generation as the same inference problem under different amounts of observed evidence. Given any subset of modalities, MUNITE models the conditional distribution over the latent representation associated with the complete observation. Full observation recovers deterministic encoding, no observation recovers the latent marginal, and intermediate subsets define conditional latent inference, all within a single conditional flow model. A shared latent sample captures variation that must remain consistent across generated targets, while modality-specific generative decoders model the remaining uncertainty independently. To learn these conditional distributions from incomplete training examples, we extend conditional flow matching through self-distillation: predictions conditioned on richer available observations supervise the same model conditioned on smaller subsets at the same intermediate latent state. When the richer-evidence trajectory follows the exact conditional flow, this provides the same expected learning signal as full-target denoising. Across PolyMNIST-D-Q, FFHQ64, and image-text-audio, MUNITE achieves competitive or better generation quality and source-target alignment, with higher joint-generation coherence. In particular, it attains the highest coherence in all one-to-many and unconditional image-text-audio comparisons, showing the effectiveness of unified latent inference across diverse multimodal settings.