MULTI3IR: A Benchmark for Multi-perspective Multi-domain Multi-modal Information Retrieval
2026-08-31 • Information Retrieval
Information Retrieval
AI summaryⓘ
The authors created a new test called Multi³IR to see how well search systems find a variety of answers to questions that can have many viewpoints, using information from different topics and types like text and images. They found that current search tools tend to focus on just one perspective. To fix this, the authors made a method named SPIN that helps search systems understand and include more diverse viewpoints. Their method worked better on the Multi³IR test and also on other similar tests. They shared their data and code online for others to use.
Information RetrievalOpen-ended QueriesMultimodal RetrievalBenchmark DatasetStack ExchangeEmbeddingsNoise VectorsPerspective DiversitySemantic Directions
Authors
Seokwon Song, Sohyeon Kim, Gunhee Kim
Abstract
Information retrieval (IR) increasingly targets open-ended queries that admit diverse perspectives. Existing IR benchmarks, however, focus primarily on closed-ended queries, while even open-ended benchmarks largely consist of queries whose supporting documents span a single subject domain and modality. We introduce Multi$^3$IR, a benchmark that evaluates how well retrievers cover the multifaceted perspectives of open-ended queries across diverse domains and modalities. It comprises 104.9K Stack Exchange queries, each annotated with perspective descriptions that capture the query's implicit viewpoints. We further propose SPIN, a parameter- and label-efficient method that learns noise vectors to steer embeddings toward diverse yet meaningful semantic directions. Experiments show that existing multimodal retrievers suffer from single-perspective bias, while SPIN substantially improves perspective coverage on Multi$^3$IR and generalizes well to unseen open-ended IR benchmarks. The dataset and experimental code are available at https://github.com/seokwon99/Multi3IR.