ProbeScout improves image searches with attribute guided ranking
ProbeScout: Visual Analytics for Attribute-Guided Image Search
Human-Computer Interaction
Summary
Finding images that meet several specific visual conditions, like a busy intersection at dusk, can be hard because some conditions may hide others or be hard to check quickly. The authors created ProbeScout, a tool that uses small checks for each condition combined smartly to rank images better and faster. This tool helps people quickly look through images, adjust their searches, and keep useful condition checks for later. It works better than older methods while needing to examine fewer images in detail.
What this means in practice
- •For computer vision teams: Curate and diagnose datasets by interactively searching images matching complex visual attribute combinations efficiently.
- •For media asset managers: Organize and find images tagged with multiple visual features by reusing condition checks for faster results.
Authors
Yifan Lv, Yiyun Chen, Daojun Ye, Haotian Yang, Weikai Yang
Abstract
Analysts often need to identify images that jointly satisfy multiple visual conditions, such as a crossroads with traffic lights at dusk, for model diagnosis, dataset curation, and targeted training. Embedding-based retrieval can rank the large gallery efficiently, but a visually dominant condition can obscure weaker conditions, and a single similarity score does not enforce the required conjunction. Visual question answering (VQA) can explicitly verify conditions, yet exhaustively applying it to the full gallery is costly, especially when analysts refine their query. These limitations motivate keeping humans in the loop at the attribute level, where analysts can quickly build evidence for each condition and reuse it when the request changes. We therefore present ProbeScout, a visual analytics system that supports this loop. It first builds composable attribute probes from sparse VQA labels and fuses them into a conjunction-aware initial ranking. Coordinated views support rapid screening, near-miss diagnosis, and on-the-fly subset construction by filtering and combining these probe outputs. Analysts provide lightweight attribute- and query-level feedback, which drives staged refinement of fusion weights while keeping the probes fixed. These verified attributes can be reused for future queries. We evaluate ProbeScout on 17 retrieval tasks across three datasets, showing improved retrieval over embedding baselines. A separate 10-task comparison achieves higher task-macro AP and F1 than exhaustive VQA while labeling at most 2% of the gallery images. Two case studies further demonstrate how ProbeScout supports interactive analysis and refinement in realistic workflows.