Person centered detection improves two hand and object interaction parsing
Single-Query Person-Centric Bimanual Hand-Object Interaction Detection
Computer Vision and Pattern Recognition
Summary
Understanding how people use both their hands to interact with objects is hard because it's tricky to know which hand belongs to which person and what each hand is doing. The authors created a new approach that looks at the whole person at once, including their body, hands, and what they might be holding or touching. Their method uses a special attention system to focus on important body parts and figures out hand interactions in one step. They also made a new dataset to test this and showed their method works better for recognizing these two-handed interactions.
What this means in practice
- •For human computer interface developers: Design systems that better understand dual-hand object use by detecting person-level hand ownership and interactions in scenes.
- •For augmented reality engineers: Create AR experiences that can track a person’s hands and their objects more reliably in multi-person real-world scenarios.
Authors
Jonghyun Kim, Junho Roh, Yubin Yoon, Hyotae Lee, Jongkuk Park, Taehwan Hwang, Jaechul Kim, Jungho Lee
Abstract
Understanding person-level bi-manual interactions requires not only detecting hands, but also identifying which two hands belong to the same person and what each hand interacts with. Existing hand--object interaction methods are mostly hand-centric: they treat each hand as an independent instance, which can lead to ambiguous ownership in multi-person scenes. We propose a person-centric formulation in which a single query predicts a structured output for one person, including the human box, body pose, hand boxes and states, and interaction targets. We introduce part-aware deformable attention to allocate attention across human, hand, and pose-specific reference regions, enabling one query to capture the full person structure. We further unify detection and interaction reasoning with a hand-to-query relationship matrix, where each hand selects its interaction target from the detected query set plus a learnable off token, directly recovering the target's box and class without separate object regression. We build a COCO-based dataset with person-centric bi-manual interaction annotations and define structured metrics for evaluating hand states and complete hand--object tuples. Experiments with a transformer-based detector show that our formulation improves person-level bi-manual interaction parsing and provides an effective unified framework for joint detection, pose estimation, and hand reasoning.