Cross-view object recognition improves using 3d visibility aware training

3D-VITAL: Visibility-aware Identity Training from Artificial Liftings of 3D Reconstruction model for Cross-View Object Re-Identification

Computer Vision and Pattern Recognition

Summary

Matching the identity of objects seen from very different viewpoints, like from the ground and from the air, is hard. The authors use 3D models created from single images to generate new views and help computers learn better. They found that these 3D models sometimes contain mistakes or fake details, so they created methods to ignore bad parts and focus on trustworthy areas during training. This approach improves accuracy for recognizing objects across views.

What this means in practice

  • For surveillance operators: Enhance matching of object identities captured from aerial and ground cameras for security monitoring.
  • For autonomous vehicle teams: Improve vehicle detection and tracking accuracy between drone perspectives and street-level cameras for navigation.

Authors

William Grolleau, Astrid Sabourin, Guillaume Lapouge, Catherine Achard

Abstract

We address the Cross-View Object Re-Identification (ReID) task, aiming to match identities across drastically different viewpoints. We focus on Single-View Aerial-Ground ReID (SV AG-ReID), where models train on one viewpoint and must generalize to another at inference. To bridge this viewpoint gap, related work employs generative models to synthesize novel views from source images as additional training data. We leverage recent advances in large-scale single-view 3D mesh reconstruction to lift source images into textured meshes and to render them from new viewpoints. However, this pipeline produces unreliable data in two distinct ways. First, the reconstruction may fail and produce geometrically incorrect meshes, resulting in severe corruption in rendered novel views. Second, even when the geometry is correct, the rendered novel views contain hallucinated textures in regions not visible in the source image. The reconstructed mesh carries a per-pixel partition between regions visible from the source view and regions that the reconstruction model had to hallucinate, available as a deterministic byproduct of rendering. We introduce this per-pixel visibility map and exploit it in a two-stage design. Cross-View Consistency Filtering (CVCF) discards geometrically incorrect meshes by ensuring multi-view consistency between source and novel views over visible patches. Visibility-Aware Attention Supervision (VAAS) leverages the visibility map to guide the model's attention toward reliable regions of the rendered view during training. Combined, these two components enable our method, 3D-VITAL, to better leverage the synthetic signal from single-view 3D mesh reconstruction. 3D-VITAL sets a new state-of-the-art on three SV AG-ReID benchmarks. Code and data will be released at https://github.com/TurtleSmoke/3D-VITAL.