Language guided robots use attributes to find objects in scene memory
FRAME: Factored Retrieval via Attribute Readouts for Object-Centric Scene Memory
Computer Vision and Pattern RecognitionRobotics
Summary
Robots that understand language need to remember objects they've seen in a scene so they can follow instructions or find things later. The paper looks at how robots can find objects by combining different traits like shape, color, or material described in human language. The authors create a method called FRAME that turns language into signals for each trait, then ranks objects by matching those traits. Their approach works better than other methods and is efficient to compute. This helps robots remember and find objects more accurately by breaking down their features into parts.
language guided robotsscene memoryobject retrievalattributesobject embeddingsnatural language processingquerycompositional retrievalmatrix vector multiplication
Authors
Woosang Jeon, Sanghyeok Choi, Minwoo Kim, Taehyun Jung, Taehyeong Kim
Abstract
Language-guided robots need persistent scene memories to follow instructions, revisit objects, and resolve references to objects encountered over time. While much of language-guided scene-memory retrieval has emphasized spatial or relational references, many everyday object references specify objects by multiple persistent attributes, such as category, material, size, or surface appearance. We formalize this problem as attribute-compositional retrieval, where a fixed object-centric scene memory is queried with natural language to retrieve the object satisfying the requested attributes. To investigate this capability directly, we introduce a controlled evaluation protocol with fixed scene memories and attribute-defined targets, separating retrieval from perception and annotation ambiguities. We then propose FRAME, which turns language into query-relevant attribute weights, uses learned readouts to estimate per-attribute evidence from object embeddings, and ranks objects by aggregating this evidence according to the query. Across held-out scenes and object assets, FRAME outperforms representative scene-memory retrieval baselines while reducing post-decomposition object scoring to lightweight matrix-vector computation. These results position attribute-compositional retrieval as a complementary scene-memory capability for language-guided robots, showing that persistent object attributes can be exposed as composable evidence for accurate and efficient multi-attribute retrieval.