TeaMatch: Teachable Cross-Modal Representation Learning for 2D-3D Matching
2026-08-10 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors propose TeaMatch, a new way to improve how computers match points in 3D clouds to 2D images. They introduce the idea of 'teachability,' which means making sure the matching method works well even when the data is messy or incomplete. To do this, they build small helper models that try to copy the main model’s work under tough conditions, and then train the main model to help these helpers do better. Their approach fits easily into current matching systems and leads to stronger and more reliable results on hard test cases.
2D-3D matchingpoint cloudscross-modal representation learningteachabilityweak learnerscorrespondencegeometry-aware constraintscoarse-to-fine matchingrobustnessnoisy inputs
Authors
Chongjian Wang, Junjie Gao
Abstract
Learning reliable correspondences between images and point clouds is fundamental for 2D-3D matching. Despite recent progress in detection-free methods, existing approaches primarily optimize matching within a single model and often struggle to maintain reliable correspondences under challenging conditions such as noisy inputs, low overlap, and ambiguous structures. In this work, we propose TeaMatch, a novel framework that introduces teachability as a criterion for cross-modal representation learning. We define teachability as the ability of a representation to be effectively recovered by weak learners under degraded inputs, reflecting its structural consistency and robustness. To this end, we construct a set of task-specific weak students that simulate common failure modes and train them to imitate the teacher on a training split while evaluating their recoverability on a disjoint meta split. The teacher is then optimized to improve the students' ability to recover reliable correspondences, guided by correspondence-level and geometry-aware constraints. Our framework can be seamlessly integrated into existing coarse-to-fine matching pipelines without additional inference cost. Extensive experiments demonstrate that TeaMatch improves matching robustness and achieves state-of-the-art performance on challenging 2D-3D matching benchmarks.