Transformer model generates diverse 3D hand and body shapes
MHE-Former: Multi-Hypothesis Transformers via Entropy Maximization for 3D Mesh Recovery
Computer Vision and Pattern Recognition
Summary
3D hand and body shapes are hard to guess from a single photo because of hidden parts and unclear views. The authors present a new model that creates many possible shapes instead of one, helping to cover uncertainty. They also added a way to pick the best shape based on extra information or user questions, improving accuracy and usefulness. Their model works well on several tests and lets users choose the best guess more easily.
What this means in practice
- •For augmented reality developers: Create 3D hand and body models from single images that better handle hidden parts by producing multiple realistic shape guesses.
- •For human-computer interaction designers: Use context-aware hypothesis selection to improve accuracy and user control when interpreting 3D body poses from images.
Authors
Boshu Jia, Rongyu Chen, Linlin Yang, Zihao Liu, Yingjie Chen, Zhongqun Zhang, Zhulin Tao, Shaohui Lin, Xiaoyu Wu, Libiao Jin, Baochang Zhang, Angela Yao
Abstract
Monocular 3D hand and body mesh recovery often suffers from severe occlusion and ambiguity. Traditional deterministic methods typically regress a single optimal solution, leading to overconfident predictions. In this paper, we introduce an exploration--exploitation paradigm for ambiguous mesh recovery with multi-hypothesis learning and selection. Specifically, during exploration, based on our probabilistic formulation and entropy maximization, we propose a novel multi-hypothesis method referred to as MHE-Former. It is a Transformer-based multi-hypothesis framework, ensuring high training efficiency and label friendliness while generating plausible and diverse hypotheses. During exploitation, we propose Hypothesis Selection, a context-aware process for multiple predictions. Especially leveraging VLM's powerful visual understanding and reasoning capabilities, it allows users to choose the most plausible and desired estimate with additional evidence and natural language intent. Extensive experiments demonstrate that our framework achieves state-of-the-art performance in accuracy and diversity across multiple datasets. The user preference study further shows the practicality of our hypothesis selection process.