Multi-modal object reidentification improved by handling noisy text and structure
Incentive Noise and Structural Prior Infusion for Multi-modal Object Re-Identification
Computer Vision and Pattern Recognition
Summary
Identifying objects across different types of images, like photos and text descriptions, can be tricky because text can be noisy or unclear. The authors developed a new method that introduces controlled noise to better understand mismatches between images and their descriptions. They also add tools that help machines focus on the correct spatial details to match objects accurately. Their approach improves performance on multiple tests for recognizing objects from different input types.
What this means in practice
- •For security system developers: Improve person or vehicle identification by combining images and text, managing noisy descriptions for more reliable cross-modal matching.
- •For retail inventory teams: Enhance product tracking using images and textual labels even when descriptions are ambiguous or inconsistent, ensuring better multi-modal identification accuracy.
Authors
Weixiang Zhou, Yuhao Wang, Xingguo Xu, Weizhen Zhou, Zhixun Su, Jinshan Pan, Cong Wang
Abstract
Multi-modal object Re-Identification (ReID) benefits from complementary information across heterogeneous imaging modalities. To further enrich semantic representation, text descriptions have recently been incorporated as an additional modality. However, recent vision-language approaches often treat text descriptions as clean, deterministic signals and overlook their inherent noise, including modality-mismatched phrases and semantically ambiguous expressions. Moreover, prevailing methods lack explicit mechanisms to reconcile fine-grained structural discrepancies between modalities, even after high-level semantic alignment. To address these challenges, we propose a novel framework centered on Positive-Incentive Noise (π-noise) and structured prompt modulation. First, the Semantic Cross-Modal Modulator harnesses task-aware π-noise, sampled from a distribution conditioned on both visual and text inputs, to perturb global tokens and enable semantics-guided cross-modal compensation. Second, the Structure-Aware Prompt Adapter injects learnable geometric priors via prompts to enhance spatial consistency. Third, the Context-Aware Sparse Fusion module distills structural context to guide adaptive fusion while shielding identity features from noisy local details. Experiments on three multi-modal ReID benchmarks demonstrate the effectiveness and robustness of our approach. The code is available at https://github.com/zw-absin/INSPI.