Vision language model improves person re-identification accuracy significantly
VLM-as-a-Judge for Person Re-Identification
Computer Vision and Pattern Recognition
Summary
Identifying a person when their clothes change is very hard for computers. The authors show that a general AI model designed to understand images and text can be used as a judge to sort through candidate matches, making the identification more accurate. This model improves results even without being trained on the specific data it judges. Although using this model takes more computing power, the authors find a way to reduce that cost by only calling the model when two expert systems disagree. This approach leads to better identification performance than existing expert methods.
What this means in practice
- •For video surveillance teams: Improve long-term person tracking systems by integrating a vision-language model to better identify individuals despite changes in clothing.
- •For security system developers: Develop person re-identification software that uses a general-purpose vision-language model as a decision judge to increase accuracy across various environments.$Commercial implications: Enables creation of more reliable security products that better identify individuals regardless of clothing changes.
Authors
Bessie Dominguez-Dager, Miguel Cazorla, Vincent Lepetit
Abstract
Cloth-changing person re-identification (CC-ReID) is still very challenging, and no method clearly dominates: methods that lead on one benchmark collapse on the others, because they rest on different identity cues. We show that a general-purpose vision-language model (VLM), used zero-shot as a judge for reranking a short candidate list, beats the expert retrieval models it reranks by a wide and consistent margin. On the four scenarios of CHIRLA, a long-term CC-ReID benchmark that no publicly released Re-ID model is trained on, reranking the candidates of two frozen experts raises rank-1 on every scenario, by 12.9 points on average over the best expert and 9.4 over the best naive ensemble of the two, with no component fine-tuned on the target data. Merging both experts' shortlists matches or beats reranking the stronger expert's shortlist alone, at lower cost. Introducing a VLM does, however, come at a cost: almost all of the pipeline's FLOPs are spent on it. We therefore borrow a criterion from ensembling and invoke the VLM judge only where the two experts disagree at rank 1. This threshold-free gate removes 40% of the calls, and a comparable share of total compute, with no significant drop in rank-1, trading 1.8 to 3.8 points of mAP. Judging remains the dominant cost, but on a benchmark where dedicated experts stall, we find a frozen general-purpose model to be the strongest judge of identity. The code is available at https://github.com/bdager/JudgeReID.