Nemotron Labs builds unified model for image and language tasks
Nemotron-Labs-Diffusion-Uni: Elastic Self-Refining Unified Diffusion LLM with Cross-Paradigm On-Policy Distillation
Computer Vision and Pattern Recognition
Summary
Handling both understanding and creating images usually requires different types of AI models. The authors developed Nemotron-Labs-Diffusion-Uni (NLD-Uni), which combines these tasks into one model by learning from existing models specialized in each area. They introduced a new training method that gradually refines predictions step by step, leading to better image quality and understanding. This single model can answer questions about images, reason across words and visuals, locate objects, create images from text, and edit pictures.
What this means in practice
- •For multimodal ai developers: Create one model that handles both visual understanding and image generation tasks for better software integration.
- •For content creation teams: Improve tools for text-guided image generation and editing that achieve higher visual quality via progressive refinement.$Commercial implications: Enables advanced image creation and editing products for designers and marketers by improving visual output quality and flexibility.
Authors
Shufan Li, Hanrong Ye, Greg Heinrich, Yonggan Fu, Haotian Zhang, Sifei Liu, Aditya Grover, Pavlo Molchanov
Abstract
We propose Nemotron-Labs-Diffusion-Uni (NLD-Uni), a unified discrete diffusion large language model (DLLM) for visual understanding and generation tasks. Compared with prior unified diffusion models, NLD-Uni introduces two major innovations. First, we propose a unified cross-paradigm on-policy distillation (OPD) framework that successfully transfers the visual understanding ability of autoregressive VLMs and visual generation ability of continuous flow models, to a unified DLLM. Second, NLD-Uni incorporates a novel progressive self-refining diffusion process that gradually corrupts each token across multiple steps during training, as opposed to adopting the conventional masked diffusion formulation that directly corrupts a clean image token to an uninformative mask token. At inference, this design allows NLD-Uni to make intermediate commit to a token, enabling progressive self-refinement for better visual quality. Through extensive experiments, we show that NLD-Uni achieves state-of-the-art performance among unified DLLMs on diverse tasks including visual question-answering, multi-modal reasoning, object grounding, text-to-image generation, and image editing.