Unified multimodal models speed up image and text tasks by 20 times
Omni-Diffusion-Distill: Few-Step Distillation of Unified Multimodal Diffusion Large Language Models
Computer Vision and Pattern RecognitionMachine Learning
Summary
Many advanced AI models that handle pictures and words work slowly because they need to process information step by step many times. The authors developed a method that teaches a smaller, faster model to do almost as well as the big one by learning to skip steps. Their new way lets the smaller model quickly generate images and understand text using fewer steps, making it much faster while still being accurate.
What this means in practice
- •For mobile app developers: Create apps that generate images and understand text faster by using compact multimodal models with fewer processing steps.$Commercial implications: Enables development of responsive AI-powered apps with faster image and text processing suitable for commercial deployment.
- •For cloud ai service providers: Reduce computational costs for offering multimodal image generation and text understanding services by deploying efficient distilled models.$Commercial implications: Cuts inference time and resource use, allowing more scalable and cost-effective AI services for customers.
Authors
Hong Huang, Chenhongyi Yang, Junzhe Sun, Animesh Sinha, Wuyang Chen, Yifan Jiang
Abstract
Unified multimodal diffusion large language models (dLLMs) offer a single architecture for both image generation and multimodal understanding, but their iterative decoding requires tens to hundreds of forward passes. Existing few-step distillation methods largely focus on either image generation or text generation, making it unclear how to compress a fully discrete multimodal dLLM into a single efficient student while preserving both generation and understanding. We introduce Omni-Diffusion-Distill, a unified two-stage distillation framework that retains strong generation and understanding capabilities while substantially reducing the inference cost of a unified multimodal dLLM. Omni-Diffusion-Distill aligns the distillation of both generation and understanding, for both images and text, in the discrete token space. In the first stage, the student is trained to skip decoding steps by replaying cached teacher trajectories, and in the second stage the student is refined on intermediate states along its own rollouts. We further remedy two sources of degradation in unified distillation with a pairwise collision penalty that reduces repetition under parallel text decoding, and entropy-matched guidance that prevents entropy collapse caused by fitting the sharpened teacher distribution in image generation. Omni-Diffusion-Distill achieves state-of-the-art trade-offs between decoding efficiency and generation and understanding performance for multimodal dLLMs, reducing image generation from 128 to 8 decoding steps and multimodal understanding from 512 to 64, giving 18.2x and 21.2x wall-clock speedups. Under these budgets, it scores 0.828 on GenEval and 83.0 on DPG-Bench for text-to-image generation, while reaching GPT judge scores of 20.0 on MM-Vet and 57.2 on COCO captioning (twice the teacher's 28.4 at the same steps) for multimodal understanding.