TRACE-Bench: Decomposing and Diagnosing Multi-Reference Image Generation
2026-08-17 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
AI summaryⓘ
The authors noticed that current tests for multi-reference image generation models don’t cover all types of tasks well and are hard to analyze. They created a new way to break down these tasks into four basic steps: Anchor, Disentangle, Apply, and Compose, which lets them describe any task as a formula. Using this idea, they built TRACE-Bench, a large and diverse test set that helps evaluate how well models handle each step separately. Testing nine popular models showed that the hardest parts are distinguishing and correctly applying different attributes, not combining whole scenes. This approach gives clearer insights into where image generation models struggle.
multi-reference image generationmultimodal modelsbenchmarkatomic operationsdisentanglementattribute bindingcompositionevaluation protocolTRACE-Bench
Authors
Haoran Wang, Chaofan Ma, Ran Yi, Lizhuang Ma
Abstract
Despite recent advances in unified multimodal models for multi-reference image generation, existing benchmarks remain organized around predefined task types (e.g., "subject composition"), which are ill-suited to this combinatorial setting and lead to fragmented coverage, uncontrolled complexity, and little diagnostic value. Recognizing that diverse multi-reference tasks share a common set of atomic operations, we adopt a capability-oriented perspective and formalize four operators: Anchor ($f$), Disentangle ($g$), Apply ($\oplus$), and Compose ($C$). Any multi-reference prompt can then be represented as a compositional formula over these operators, whose structural complexity is quantified by the number of operator slots. Building on this formulation, we construct TRACE-Bench, comprising approximately 1,600 evaluation cases across slot counts 1--8, built from 631 formula templates and around 4,000 reference images spanning diverse artistic styles and real-world subjects. The formula structure directly drives an operator-aligned evaluation protocol for per-capability scoring and a diagnostic tree analysis for recursive failure localization. Evaluating 9 leading models reveals insights invisible to holistic scoring: the primary bottleneck lies in disentanglement ($g$) and attribute binding ($\oplus$) rather than scene-level composition ($C$), with even the best model scoring only 0.74 on attribute fidelity. Project page: https://amuseum-whr.github.io/TraceBench