Language guided robots retrieve objects by multiple attributes
FRAME: Factored Retrieval via Attribute Readouts for Object-Centric Scene Memory
Computer Vision and Pattern RecognitionRobotics
Summary
Robots that understand language need ways to remember and find objects in a scene, even after some time passes. The authors identify that objects are often described by several lasting features like size, color, or material. They introduce FRAME, a method that reads out these features from stored object information and ranks objects by how well they match a spoken description. FRAME improves upon earlier methods by efficiently combining multiple attributes to find the right object.
What this means in practice
- •For robotics engineers: Enhance robot memory systems to find objects based on several properties from spoken instructions.
- •For augmented reality developers: Improve object selection in AR scenes by querying objects with detailed attribute combinations.
Authors
Woosang Jeon, Sanghyeok Choi, Minwoo Kim, Taehyun Jung, Taehyeong Kim
Abstract
Language-guided robots need persistent scene memories to follow instructions, revisit objects, and resolve references to objects encountered over time. While much of language-guided scene-memory retrieval has emphasized spatial or relational references, many everyday object references specify objects by multiple persistent attributes, such as category, material, size, or surface appearance. We formalize this problem as attribute-compositional retrieval, where a fixed object-centric scene memory is queried with natural language to retrieve the object satisfying the requested attributes. To investigate this capability directly, we introduce a controlled evaluation protocol with fixed scene memories and attribute-defined targets, separating retrieval from perception and annotation ambiguities. We then propose FRAME, which turns language into query-relevant attribute weights, uses learned readouts to estimate per-attribute evidence from object embeddings, and ranks objects by aggregating this evidence according to the query. Across held-out scenes and object assets, FRAME outperforms representative scene-memory retrieval baselines while reducing post-decomposition object scoring to lightweight matrix-vector computation. These results position attribute-compositional retrieval as a complementary scene-memory capability for language-guided robots, showing that persistent object attributes can be exposed as composable evidence for accurate and efficient multi-attribute retrieval.