Uniffordinteractor enables more realistic human and object interactions in 3d scenes
UniffordInteractor: Reasoning the Unified Affordance for Human and Multi-Object Interaction
Graphics
Summary
When people move and interact with many objects at once, like picking up items or using tools together, computers find it hard to predict these actions realistically. The authors created a new method called UniffordInteractor that first imagines how objects can be used together and then simulates how a person would interact with them over time. This approach looks at both how humans touch objects and how objects relate to each other, improving the realism of these interactions. Their tests show it works better than previous tools at modeling complicated scenes with many objects.
What this means in practice
- •For virtual reality developers: Create more realistic avatar movements interacting with multiple objects naturally in immersive environments.$Commercial implications: Enables VR applications and games to simulate complex human-object interactions improving user immersion and experience.
- •For robotics engineers: Plan and simulate robots that handle multiple objects simultaneously by reasoning about object affordances and interactions.
Authors
Chen Su, Xingqun Qi, Xurui Hu, Bingkun Yang, Muyi Sun, Man Zhang, Zhenan Sun
Abstract
Deriving plausible and dynamic Human Object Interaction (HOI) has gained tremendous progress in Embodied AI and Immersive AR/VR. While existing HOI methods have effectively explored human interaction with individual or fixed objects, they still show limitations in complex multi-object interaction scenes. Meanwhile, previous approaches focus more on the contact information held by human-object relation, overlooking the significance of the contact situation of various entities (i.e., object-object). To address these limitations, we propose UniffordInteractor, a Reasoning-then-Interacting framework that leverages unified affordance for realistic and long-horizon human and multi-object interaction modeling. At the reasoning stage, we devise a Unified Affordance Reasoner (UAR) that employs a fine-tuned 3D Multimodal Large Language Model for unified object-centric affordance prediction. At the interacting stage, we propose Hierarchical Interaction Generator (HIG), which formulates the interaction generation as a hierarchical (e.g., root to body to hand to objects) autoregressive prediction, conditioned on the obtained unified affordance. Extensive experiments demonstrate that UniffordInteractor surpasses SOTA counterparts in human and multi-object interaction generation.