Multimodal retrieval struggles to fully satisfy complex evolving user requests
TRIAD: Multimodal Retrieval across Abstraction, Constraints, and State
Computer Vision and Pattern Recognition
Summary
When people ask computers to find things that combine images and words, the requests can be very tricky. The paper by the authors shows that current search systems have trouble fully understanding complicated demands that change over time or include both what to include and exclude. They created a new test called TRIAD to measure how well these systems do with tasks involving different levels of detail, positive and negative criteria, and changing requirements in a conversation. Their tests show no system performs great on all parts, and handling negative or changing requests remains especially difficult.
What this means in practice
- •For ai assistant developers: Improve retrieval models that handle complex user requests evolving across conversation turns in multimodal assistants.
- •For digital content managers: Refine tools to filter and retrieve images and concepts considering positive and negative criteria to better satisfy user constraints.
Authors
Xiangyang Wu, Chen Ju, Xu Chen, Likang Peng, Jinsong Lan, Haihong Tang, Yang Hu
Abstract
Multimodal retrieval increasingly supports agentic systems, where user requests can involve cross-level relations, combine multiple constraints, and evolve across dialogue turns. In these settings, a highly similar result may still violate the user's requirements, while retrieved content may directly inform downstream actions without human screening. We introduce TRIAD, a multimodal retrieval benchmark for evaluating how well retrieval systems satisfy these complex, evolving requests. TRIAD comprises three tracks: abstraction, testing bidirectional retrieval between visual instances and concepts at different hierarchical levels; constraints, testing the joint satisfaction of positive and negative conditions; and state, testing the integration of requirements across dialogue turns and the updating of earlier conditions when users revise their requests. The benchmark contains six subsets, each with 1,000 queries and a fixed pool of 1,000 candidates per query, evaluated using a common multi-positive mean average precision (mAP) protocol. Simpler reference tasks and hard negatives grouped by the requirements they violate help characterize specific retrieval difficulties. We evaluate 20 configurations of 17 systems and find that no system leads across all six subsets. Performance declines with hierarchical distance, and negative-only queries generally underperform positive-only queries with the same number of constraints. Across paired tasks expressing the same final intent, multi-turn retrieval yields lower mAP in 39 of 40 configuration-domain comparisons. The benefits of explicit reasoning also vary across models and tasks. These findings highlight the gap between retrieving similar content and satisfying a complete request.