UniData creates detailed multimodal instructions from simple requests
UniData: Universal Multimodal Instruction Generation Pipeline
Artificial IntelligenceComputation and Language
Summary
Creating detailed instructions for computers that understand text, images, and other types of data is hard and takes a lot of work. The authors developed UniData, a method that takes simple user ideas and turns them into rounds of detailed, multimodal instructions. To do this, UniData breaks down ideas into multiple events, uses a large model that works with many types of data, and fixes errors in the instructions it generates. They also created a big dataset called UniDataset to help train and test this method. Their tests show UniData makes better-quality instruction data and helps other models understand and create multimodal content better.
What this means in practice
- •For multimodal ai developers: Build AI systems that can follow complex, multi-step instructions involving different types of data like images and text more accurately using UniData-generated instructions.
- •For chatbot design teams: Create chatbots that handle multi-round conversations with multimodal inputs and outputs by training on richly generated instruction datasets from UniData.
Authors
Jiaqi Tang, Yi-Feng Wu, Yuting Zhang, Hao Lu, Bowen Fu, Qing-Guo Chen, Xiaogang Xu, Yuwei Hu, Shiyin Lu, Wei Wei, Lei Zhang, Zhao Xu, Weihua Luo, Qifeng Chen, Ying-Cong Chen
Abstract
Multimodal Large Language Models (MLLMs) are increasingly being applied in a wider range of real-world scenarios. However, due to the substantial labor cost, creating high-quality multimodal instruction datasets for MLLMs remains a significant challenge. Although some methods propose to generate instruction data, they often face limitations in modality support and struggle with generating multi-round instructions. To address these problems, we introduce UniData, a universal instruction generation pipeline, to transform simple user requirements into multi-round, multimodal instructions. Specifically, UniData first expands user requirements into multiple diverse events. Using these events, UniData then integrates an any-to-any large model for multimodal instruction generation. Finally, UniData enhances data quality by correcting irrelevant and redundant inference flow, leveraging correlations between instruction rounds. To train this pipeline, we also build UniDataset, a dataset comprising 20,000 entries across nine modalities for improved multimodal generation. Our experiments demonstrate that UniData achieves SOTA performance in data quality and can also enhance the understanding and generation capabilities of other multimodal models.