Vision transformer improves hand pose accuracy relative to camera
Estimating Accurate Hand Pose in Camera Space with Vision Transformer
Computer Vision and Pattern RecognitionArtificial IntelligenceGraphics
Summary
Estimating the exact position of a hand in 3D space using a single camera image is tricky because it’s hard to guess depth from just one view. The authors develop a new method using a type of AI model called a vision transformer that helps separate how the hand moves locally and where it is globally relative to the camera. They also introduce special training techniques to improve understanding of hand depth and the camera’s perspective. Their approach leads to better accuracy than previous methods on a standard hand pose dataset.
What this means in practice
- •For augmented reality developers: Create more accurate hand tracking in AR devices by better estimating hand positions directly in camera space.$Commercial implications: Enables development of improved AR hand tracking products by providing precise 3D hand poses from RGB cameras.
- •For robotic system builders: Enhance robot grasping and manipulation by supplying precise global and local hand pose estimates from monocular video input.
Authors
Kaiwen Ren, Yiran Jiang, Yongjing Ye, Shihong Xia
Abstract
Monocular RGB-based hand pose estimation has emerged as a critical research frontier in computer vision. The local hand pose estimation methods predict hand poses relative to the wrist, while global hand pose estimation also requires estimating the wrist's position in the camera coordinate system. However, this camera-space estimation confronts two fundamental challenges: (1) depth ambiguity in monocular settings, and (2) the coupling effect of hand local poses and global wrist positions in the perspective projections. In particular, this coupling reflects that the projections are jointly determined by local hand poses, wrist positions, and camera intrinsics. To overcome these challenges, our framework proposes two key innovations: Transformation-Isomorphism Supervision for hand-depth information extraction and Perspective Information Embedding for resolving above coupling effect of local pose and wrist position, both integrated within the mainstream encoder-decoder architecture. Besides, we propose a novel framerate-aware multi-dataset training strategy for sequential pose refinement. Our fully integrated approach achieves at most 37.1\% superiority in CS-MJE over SOTA on HO3D. Project page: https://github.com/Mine268/CS-ViT.