DINO features improved for partial 3D shape matching with self learning

Teaching DINOv3 About Partial 3D Geometry: A Self-Supervised Geometry-Aware Approach

Computer Vision and Pattern Recognition

Summary

Matching parts of 3D shapes is hard because often you only see bits of an object, and usual methods struggle to recognize these partial views. The paper shows how to teach a known image-based feature system called DINO to understand incomplete 3D shapes better through self-supervised learning. This improved DINO can create better shape descriptors that work well even when only part of the shape is visible. Their method beats older techniques on tasks like matching partial shapes and predicting left-right orientation.

What this means in practice

  • For 3d computer vision developers: Improve partial 3D shape matching in systems where objects are only partly observed, enhancing recognition and alignment accuracy.
  • For robotics engineers: Enhance robot perception to better understand and interact with partially visible 3D objects in real-world environments.

Authors

Viktoria Ehm, Dongliang Cao, Riccardo Marin, Daniel Scholz, Weikang Wang, Florian Bernard, Daniel Cremers

Abstract

Partial shape matching is a crucial yet underexplored problem in 3D vision, with significant relevance to real-world scenarios where shapes are often only partially observed. Existing feature descriptors face difficulties in this setting, as traditional representations either struggle with the boundaries of partial shapes or heavily depend on the shape's spatial position. While existing approaches have employed DINO features for partial shape matching, these features are not inherently suited for handling partial observations. In this work, we propose a method to refine DINO features using LoRA-based self-supervised learning, enabling the generation of feature descriptors that are robust to partiality. Our features substantially improve performance on partial shape matching compared to traditional or vision foundation features. Additionally, when integrated into existing partial shape matching pipelines, we achieve state-of-the-art results on partial shape matching and left-right prediction benchmarks.