Textual semantics improve object matching across viewpoints in videos
Exploiting Textual Semantics for Robust Cross-View Object Correspondence
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Matching the same object seen from different angles in videos is hard because objects can look very different depending on the camera view and lighting. The paper introduces a method called TeSCo that uses descriptions of an object’s features like color and texture, written as text, to help match objects more reliably. These textual clues don’t change with viewpoint, so they add valuable information to the usual visual cues. TeSCo generates text descriptions from one view and then uses them with visual data to better find the object in another view, improving performance in tricky video scenarios.
What this means in practice
- •For video analytics developers: Improve object matching across-camera setups using text-augmented visual features to handle difficult viewpoint changes.
- •For augmented reality app developers: Achieve more stable object recognition under varying viewpoints by integrating textual descriptions with video input.
Authors
Bing Fan, Yunhe Feng, Yan Huang, Heng Fan
Abstract
Cross-view object correspondence (CVOC) aims to establish associations of the same target between query and search videos captured from different viewpoints. Existing methods often rely on visual cues for CVOC. Despite recent progress, these methods struggle in complicated scenarios with substantial viewpoint changes and appearance variations, due to lacking sufficient target information. Addressing this, we introduce TeSCo, a framework that exploits rich Textual Semantics of an object, in addition to its visual cues, for cross-view object Correspondence. Our key insight is, textual description of a target, by capturing diverse attributes such as category, texture, and color, provides viewpoint- and appearance-invariant semantics, which are complementary to visual cues and can hence enhance correspondence robustness in complicated scenes. Inspired by this, TeSCo first generates a textual expression of the target from the given query view using a vision-language model, and then extracts textual feature to enhance visual features for localization in the search view. To realize this, we introduce a text-conditioned cross-modal fusion (TCF) module that incorporates the textual feature into visual features with text-guided modulation, producing more robust multimodal target representation for object correspondence. Since not all textual cues are equally compatible with the target in current search-view frame, we propose an iterative textual feature refinement (ITFR) module that progressively adjusts textual feature using search-view information before the TCF module, enabling search-view-aware textual semantics to be fused into visual features and leading to better performance. In extensive experiments on Ego-Exo4D and HANDAL-X, TeSCo demonstrates state-of-the-art results and largely outperforms other models, validating its efficacy. Our code is released at https://github.com/VAIL-Fan/TeSCo.