Benchmark reveals strengths and weaknesses of multi-image generation models
MDMR-Bench: A Multi-Dimensional Benchmark for Multi-Reference Image Generation
Computer Vision and Pattern Recognition
Summary
Generating new images from several examples and instructions is getting better, but current tests only look at a few simple parts of this task. The authors created MDMR-Bench, a new test set that checks many different challenges like mixing objects, transferring features, handling complex images, and reasoning visually. They also use detailed checklists to carefully measure how well models do in specific areas, instead of just giving one score. Testing many models showed that this approach helps find exactly where models succeed or fail.
What this means in practice
- •For graphic designers: Evaluate and compare multi-image generation tools that combine and modify visual elements for creative design work.
- •For digital content creators: Assess advanced image generation models' ability to fuse objects and attributes from multiple sources for enhanced visual storytelling.
Authors
Bowen Zheng, Jintao Lin, Changheng Yi, Haoran Yang, Binxiao Huang, Chenyang Lei, Rui Liu, Xihui Liu
Abstract
Image generation models have rapidly advanced, with many models now supporting generation conditioned on multiple reference images and textual instructions. Such models enable a wide range of applications, including entity fusion and attribute transfer across images. However, existing benchmarks for multi-reference image generation rely on limited evaluation dimensions and fail to capture the diverse challenges of the task. To address this limitation, we propose MDMR-Bench, a Multi-Dimensional benchmark for Multi-Reference image generation. MDMR-Bench is organized around representative capability settings and consists of multiple evaluation subsets spanning diverse dimensions of multi-reference image generation, including (1) common entity fusion and attribute transfer, (2) multi-element extraction from a single image, (3) infographic manipulation, and (4) visual reasoning-intensive generation. This design enables systematic analysis of model capabilities and failure modes across different challenge dimensions. Furthermore, to achieve more reliable and interpretable evaluation, we introduce a checklist-based evaluation protocol that decomposes generation quality into fine-grained evaluation aspects. Instead of relying solely on holistic scoring, our framework uses structured checklists composed of both global-level and atomic-level queries, enabling more diagnostic assessment of model performance. By benchmarking both open-source and proprietary models, we demonstrate that MDMR-Bench provides fine-grained insights into model performance across different dimensions. The dataset and code are publicly available.