IntReal dataset and benchmark improve text and image generation together
IntReal: A Real-World Dataset and Benchmark for Multimodal Interleaved Generation
Computer Vision and Pattern Recognition
Summary
People often read or view content where text and images are mixed together naturally, but computers struggle to make this kind of mixed content well. The authors created IntReal, including a big real-world dataset and a detailed way to test how good computer models are at making mixed text and images. They showed their tests agree well with human opinions and can spot where different models do better or worse. They also found that training models on just text or just images can help them get better at making mixed content, which could save effort when building these systems.
What this means in practice
- •For multimodal model developers: Improve and evaluate text-image generation models using a real-world dataset and detailed evaluation metrics that diagnose strengths and weaknesses.
- •For digital content creators: Enhance tools that automatically generate mixed text and images for multimedia content by leveraging models trained on IntReal-100K.$Commercial implications: This enables creation of advanced content generation software offering superior multimodal interleaved outputs for marketing and media companies.
Authors
Shaohan Yu, Jie Qin, Wende Tan, Haiwen Diao, Limeng Qiao, Yueming Lyu, Chenyang Si
Abstract
Humans routinely consume multimodal content, where textual and visual information is naturally interleaved. Although Unified Multimodal Models (UMMs) can jointly generate text and images, multimodal interleaved generation remains largely underexplored. Progress in this area is hindered by two key limitations: (1) the lack of well-defined interleaved generation tasks and corresponding training data, and (2) the absence of fine-grained evaluation protocols that can reliably diagnose model performance across different capability dimensions. To this end, we present IntReal, comprising two complementary components: IntReal-100K, a real-world dataset of multimodal interleaved samples spanning four major categories, and the IntReal Evaluation Suite, which evaluates model performance across seven complementary dimensions using specialized expert models, enabling coarse-to-fine diagnosis. Extensive experiments validate our evaluation suite through low inter-metric redundancy (max pairwise absolute Spearman's $ρ\leq 0.42$ ), high agreement with human judgments (83.3% accuracy), and strong consistency across different expert evaluators (Spearman's $ρ> 0.9$ ), supporting the complementarity and reliability of our metrics. By benchmarking four families of UMMs, we show that IntReal Evaluation Suite reveals distinct strengths and weaknesses across models. Post-training experiments demonstrate the utility of IntReal-100K for improving interleaved generation and further show that training unimodal generation can also benefit interleaved generation. These findings suggest that interleaved generation can benefit from modality-specific optimization, rather than requiring joint text-image training at every stage.