Hyperbolic learning improves 3D shape and text alignment
Revisiting Triplet Contrastive Learning for 3D Shape Understanding
Artificial Intelligence
Summary
Understanding 3D shapes by comparing them to pictures can be tricky because pictures might be distorted or incomplete. This paper finds a better way by linking 3D shapes directly with words describing them instead of pictures. The authors use a kind of math (hyperbolic geometry) that naturally fits the way objects and their categories are organized, like a family tree. Their method helps computers recognize 3D shapes better and also find the right shapes when given text. They tested this approach on several benchmarks and showed it works better than previous methods.
What this means in practice
- •For 3d graphics developers: Improve 3D object recognition and retrieval by using language-based training that models hierarchical relationships among shapes.
- •For natural language processing teams: Enhance multimodal language models with hyperbolic embeddings to better capture hierarchical concepts linking text and 3D objects.
Authors
Chaoyang Zhu, Zhe Liao, Jiayi Zhang, Jie Li, Jiayi Ji, Liujuan Cao
Abstract
In multi-modal 3D pre-training, aligning 3D shapes, their 2D image counterparts, and language descriptions has made tremendous progress. However, we find that supervision from rendered images is unreliable, the semantics encoded in synthetic views are distorted due to domain gap, self-occlusion, and rendering artifacts. Instead, we learn transferable 3D representations from language only. We observe that, in real world, 3D shapes are organized in a tree-like structure, ranging from general concepts to fine-grained instances with specific attributes, while language typically denotes generic concepts and subsumes these various 3D instantiations. Hence, we propose to embed 3D representations in hyperbolic space, whose geometry is well-suited for modeling tree-like data without distortion. To capture the asymmetric hierarchy between language and 3D shapes, we further impose a partial-order relation that language entails 3D shapes. Our method, Hyperbolic Contrastive Learning (HyperCL), jointly optimizes a hyperbolic geodesic-based contrastive objective and an entailment loss to learn discriminative cross-modal representations while explicitly preserving this hierarchy. We also introduce a rigorously verified benchmark for comprehensively evaluating cross-modal 3D alignment. Across two widely used benchmarks and ours, we improve the performance up to 4.6% absolute percentage on 3D shape classification and 5.5% on cross-modal text-to-3D retrieval under the same parameter budget. The code and evaluation benchmark will be made publicly available.