SnapBench: Benchmarking Snap-and-Ask Multimodal Retrieval for Mobile Interactions
Computer Vision and Pattern RecognitionInformation Retrieval
Summary
The authors created SnapBench, a new testing method to check how well mobile AI systems work when users take blurry photos and type short or messy questions to find information. They tested 16 different AI models and found that blurry images make it much harder for the systems to find the right results, while mistakes in the text mostly affect text-only searches and not combined image-text searches. They also noticed that using only images sometimes works better than combining image and text when the inputs are noisy. To improve this, the authors proposed a new method called MOOR, which adjusts how the system trusts images and text to give better answers when inputs are unreliable.
Authors
Zirong Chen, Fuda Ye, Kuan Zhang, Enjun Du, Junfu Pu, Xinlei Wang, Xinyu Zuo, Lisheng Duan, Jin Ma, Yongqi Zhang
Abstract
Mobile AI acts as a visual oracle, empowering users to snap a picture of something and ask for information. Snap-and-ask retrieval is now one of the most common entry points for mobile AI, yet photos are often blurry, while text questions may be short or mistyped. Existing benchmarks only test on clean inputs or do not isolate paired robustness in snap-and-ask retrieval. Therefore, we introduce SnapBench, the first paired benchmark for robust snap-and-ask multimodal retrieval, spanning 1,145 queries, 9,085 gallery items under 53 controlled corruption conditions with human annotations. We evaluate 16 multimodal retrievers, covering dual-tower encoders and embedding-based VLMs. Results show that image corruptions substantially degrade retrieval, while text corruptions mainly affect text-only retrieval and have limited impact on joint retrieval. Clean image-only retrieval often outperforms joint retrieval, indicating the coarse-text drag and the lack of cross-modal fallback under noisy inputs. SnapBench provides a controlled testbed for evaluating robust retrieval in snap-and-ask scenarios. We further propose MOOR (Modality-anchored, Outlier-aware, Optimal Reweighting), a simple adaptive fusion approach, highlighting the need for reliability-aware modality calibration in snap-and-ask retrieval.