Visual feedback boosts interactive person search accuracy

Interactive Person Retrieval with On-the-Fly Visual Feedback

Computer Vision and Pattern RecognitionArtificial Intelligence

Summary

Finding people based on descriptions is often done by receiving a full description at once. The authors studied a method where the search happens in steps with questions and answers, improving the search with visual hints from previous results. They created a new dataset and a system that learns to use these visual hints to ask better questions and find people more accurately. Their approach consistently found the right people faster and better than earlier methods.

What this means in practice

  • For security operators: Improve suspect search by using visual feedback for interactive questioning during investigations.
  • For video surveillance teams: Enhance person re-identification in video streams by adapting retrieval based on ongoing visual evidence.

Authors

Lyucheng Qian, John Yuehan Zhang, Pingyu Wang

Abstract

Traditional text-based person retrieval (TPR) assumes that a witness provides a complete description at once, whereas interactive person retrieval (Inter-PR) refines a partial description through question-answer exchanges. In this setting, the questioner inspects retrieved candidates to decide what evidence to request next, while the answerer sees only the witness-side memory and cannot report candidate-specific visual differences. Existing methods mainly learn offline question orderings, leaving this visual-feedback loop underexplored. To address this limitation, we introduce visual-feedback interactive person retrieval (VFI-PR), a new interactive retrieval paradigm in which retrieved candidates produce visual feedback that becomes evidence for subsequent questioning and re-ranking. To support training and evaluation in this setting, we construct VFI-PEDES, a dataset and evaluation protocol for visual-feedback interaction, and propose VFI-ReID, a framework that learns to use visual feedback during interactive person retrieval. Through the Dynamic Accumulated-evidence Retriever (DAR) and retrieval-aware question optimization, VFI-ReID adapts retrieval and question generation to evolving evidence states, turning visual feedback into complementary evidence for subsequent retrieval. Experiments show that VFI-ReID consistently improves retrieval across VFI-PR interaction stages, demonstrating the utility of visual feedback for interactive person retrieval and outperforming existing Inter-PR baselines.