Modaliff enables flexible generation across text image and audio
Modaliff: A Unified Token Diffusion Model for Any-to-Any Generation
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Creating one model that can work with text, images, and audio at the same time is hard because these different types of information need special treatment. The researchers developed Modaliff, a new model that learns from pairs of two types of data without forcing them into one shared representation. It keeps each type of data in its own form and combines them into sequences that a single model can handle for many tasks. This way, Modaliff can generate new text, images, or sounds from various inputs efficiently while needing fewer resources.
What this means in practice
- •For multimedia application developers: Build apps that convert text, images, or audio into each other using a single efficient model without complex retraining for each modality pair.$Commercial implications: Allows creation of versatile media conversion products that handle multiple modalities with lower resource needs.
- •For creative content studios: Generate diverse content by combining text, images, and audio inputs flexibly without separate models for each conversion type.
Authors
Duc Kieu, Kien Do, Minh Khoa Le, Tung Kieu, Dang Nguyen, Thin Nguyen
Abstract
Any-to-any generation seeks a single model that translates flexibly among text, image, and audio, yet existing approaches either require fully aligned multimodal tuples or a shared representation that suppresses modality-specific structure. We introduce Modaliff, a novel token-based diffusion model that learns from widely available bimodal pairs while preserving each modality in its native latent sequence. Modaliff assembles conditions and targets into a unified token sequence, with per-token attributes to disambiguate token identities and role-based positional embedding to preserve each modality's internal structure across sequence layouts. This design enables a single denoiser to support diverse generation tasks without task-specific architectural changes or a shared cross-modal latent space. Beyond one-to-one generation, Modaliff supports multiple conditions either by combining single-condition predictions at inference or by lightweight finetuning on captions that jointly describe multiple inputs. Experiments across text, image, and audio show that Modaliff achieves the best fidelity among any-to-any models on most generation tasks while using substantially fewer parameters and training data.