Feed-forward 3D models improve image matching with new training approach
RoMa-$Ω$: What Feed-Forward 3D Models Know About Image Matching
Computer Vision and Pattern Recognition
Summary
Matching points between different images is important for many computer vision tasks. The authors studied whether models trained to directly predict 3D shapes and camera positions also understand how to match image details. They found that although these 3D models don't perform well at matching without extra help, their learned features are very useful when combined with additional training. Using this insight, they built a better image matcher that works well across various challenges.
What this means in practice
- •For computer vision engineers: Develop improved feature matching systems by integrating feed-forward 3D model representations for better robustness under viewpoint changes.
- •For augmented reality developers: Use enhanced image matching from retrained models to improve virtual object placement and tracking under different camera angles and lighting.
Authors
David Nordström, Xinyue Zhang, Thibaut Loiseau, Vincent Lepetit, Fredrik Kahl
Abstract
Learned image matching has experienced significant progress in recent years, culminating in robust and accurate matchers such as RoMa, whose robustness is often attributed to its use of frozen DINO features. In a parallel development, feed-forward reconstruction models, such as VGGT, have been trained on ever-growing datasets to accurately regress dense 3D point maps and camera poses. The distinction between matchers and feed-forward reconstruction models has become increasingly blurred with the introduction of matching losses in models such as MASt3R and VGGT-$Ω$. This raises a natural question: what do feed-forward 3D models know about image matching? In this work, we answer this question by analyzing three scenarios: (i) zero-shot matching of patch features, (ii) direct matching of 3D point predictions, and (iii) training a full matcher on top of the learned representations. We find that, despite performing poorly in zero-shot matching, especially in later layers, feed-forward reconstruction models provide strong representations for linear probing and full matching pipelines. We further show that, even without any training, their raw predictions alone enable competitive matching, albeit only under moderate viewpoint changes and modality gaps. Based on these insights, we retrain RoMa v2 by replacing its DINO backbone with VGGT-$Ω$. Our resulting model, \ours, outperforms state-of-the-art matchers on a wide range of benchmarks, e.g. +8.1 mAA compared to RoMa v2 on WxBS.