UniMore improves image and text understanding with fine-grained training
UniMore: Fine-Grained Supervision for Unified Understanding and Generation
MultimediaComputer Vision and Pattern Recognition
Summary
Unified models that do both image recognition and image creation often miss details either when describing a picture or making one from a description. The paper finds these models lose important specifics like the right color or unique details. The authors introduce UniMore, a training approach that teaches the model to pay attention to precise parts of images and their matching words. This improves both the quality of generated images and the accuracy of image captions without changing how the model works when it is actually used.
What this means in practice
- •For multimedia app developers: Create apps that generate images and captions with more precise detail from natural language input using fine-grained supervision.
- •For content moderation teams: Improve automated systems that verify if generated images and captions correctly match requested attributes and objects to reduce errors.
Authors
Haoyue Bai, Yinyi Luo, Yicheng He, Qingsong Wen, Sharon Li, Jindong Wang
Abstract
Unified multimodal models combine image understanding and generation within one system, creating an opportunity to improve both capabilities through paired image--text supervision. We examine the outputs of a pretrained unified model in both directions and observe a loss of specificity: generated images can contain the requested objects with incorrect attributes, while captions identify a scene without expressing its distinguishing details. These failures call for supervision that specifies both the visual content to realize and the descriptive differences to distinguish. We propose UniMore, a post-training method that derives these complementary learning tasks from the fine-grained content of image--text pairs. The generation route predicts grounded crops from their corresponding phrases, matching the scope of the condition and visual target while using the native visual objective. The understanding route combines positive-caption fitting with ranking against captions that omit or alter details, conditioned on the full image. The two routes are optimized independently, and their learned updates are combined into a single unified model without changing its inference procedure. Experiments show gains across multiple understanding and generation benchmarks.