Prototype-based method improves visible-infrared person matching
Prototype Matters: Modality-unified Prototype Self-distillation for Unsupervised Visible-infrared Person Re-identification
Computer Vision and Pattern Recognition
Summary
Matching people captured in visible light with those in infrared is hard because the images look very different. The authors show that using shared group examples called prototypes helps computers learn better links between the two kinds of images. They also refine this process by letting the system teach itself, leading to better understanding inside and across both image types. Their combined method improves matching accuracy without needing labeled training data.
What this means in practice
- •For security system developers: Develop surveillance systems that accurately identify people across visible and infrared cameras without manual labeling.$Commercial implications: Enables building improved multispectrum identification software for security firms by enhancing accuracy without costly labeled data.
- •For smart city operators: Integrate reliable person tracking across different sensor modalities to enhance public safety monitoring during day and night.
Authors
Menglin Wang, Xiaojin Gong
Abstract
Estimating reliable cross-modality association is crucial to unsupervised visible-infrared person re-ID. While optimal transport is shown to be a practical solution for cross-modality association, it suffers from the rigidness of hard label assignment without considering the impact of cluster noise. Moreover, enforcing only cross-modality contrast is also suboptimal, as it fails to jointly optimize the similarity relation within and across modality. In this paper, we propose a novel framework for cross-modality learning by well exploitation of prototypes: First, instead of contrasting with cross-modality prototypes, we show that modality-unified prototypical contrast facilitates better modality invariance by jointly and simultaneously optimizing similarity relation within and across-modality. Taking self-prototype as a steady teacher, we further refine the instance-prototype online relation through prototype-guided self-distillation. The two components are optimized in a unified framework, leading to a simple yet effective model. On standard VI-ReID benchmarks, we perform extensive comparison and analysis, validating the effectiveness of our proposed method. Code is available at: https://github.com/Terminator8758/PoSeD.