BridgeAfford improves 3D object interaction region detection on unseen items
BridgeAfford: Bridging Query-Target Gap for Generalizable 3D Affordance Grounding
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Finding where you can interact with parts of 3D objects based on language is hard, especially for objects the system has never seen before. The authors identify a problem they call the "query-target gap," where the system's understanding of the interaction request doesn't match the features of the object part it should find. To fix this, they created BridgeAfford, which helps the system better align similar interaction parts across different objects and uses 2D visual hints to focus better on the right areas. Their approach performs better than previous methods, particularly on new objects the system wasn't trained on.
What this means in practice
- •For robotic system builders: Locate functional parts on previously unseen 3D objects to enable intent-based robot manipulation and interaction.
- •For augmented reality developers: Identify interaction regions on diverse 3D objects from user language to improve AR object manipulation experiences.
Authors
Yijie Su, Yakun Zhu, Yi Bin, Pengpeng Zeng, Liang Chen, Guoqing Wang, Jingkuan Song, Heng Tao Shen
Abstract
Language-guided 3D affordance grounding aims to localize functional regions on 3D objects according to interaction intents. However, existing methods exhibit poor generalization to unseen objects. One challenge is the query-target gap: a semantic discrepancy between query features encoding interaction intents and target-region features, leading to inaccurate grounding. End-to-end training favors query representations tailored to seen objects, limiting their compatibility with shifted target-region features on unseen objects. To address this issue, we propose BridgeAfford, a framework that reduces the query-target semantic gap from two complementary perspectives. First, Interaction Region Alignment (IRA) promotes cross-object alignment among geometrically similar interaction regions during training. This enables structures on unseen objects that resemble previously observed ones to produce representations similar to those learned during training, facilitating target-region recognition. Second, Visual Evidence Aggregation (VEA) leverages a 2D affordance model to refine the attention distribution and guide the query toward the correct region, yielding more target-aware queries. Extensive experiments on the LASO and PIAD datasets demonstrate that BridgeAfford outperforms existing methods, particularly on unseen objects, highlighting its robust generalization ability.