Robot system improves object search by learning spatial and semantic context

LOCUS: Landmark-Oriented Container Discrimination Using Spatial Graphs

Robotics

Summary

Finding objects hidden around a home is hard because many containers look and seem alike. The authors created a new method called LOCUS that helps robots figure out where items might be by considering both where containers are placed and what they usually hold. They use a special graph-based neural network to mix location and meaning information, making the robot smarter at guessing the right container. They tested it in simulations and on a real robot, showing better results than some popular methods.

What this means in practice

  • For robotics engineers: Enhance home robot object search by integrating spatial and semantic object relationships into scene understanding for better container choice.
  • For mobile robot developers: Improve autonomous navigation and item discovery in cluttered indoor environments by using spatial graph-augmented semantic embeddings.

Authors

Taylor Bergeron, Shibani Senthilbabu, Kevin Leahy

Abstract

As robots are increasingly deployed in unstructured, real-world environments, the ability to reason about complex spatial and semantic relationships among objects remains a fundamental challenge in enabling robust and generalizable manipulation and navigation. For example, deciding where to search for an object that is not in plain sight depends on where it is physically plausible as well as semantically likely. Popular methods such as cosine similarity with CLIP struggle to disambiguate between spatially and semantically similar objects that could contain a target object. We propose Landmark-Oriented Container Discrimination Using Spatial Graphs (LOCUS). To jointly reason about spatial information and semantics, we train a GNN to update CLIP embeddings on a full environment scene graph by passing embedding information between nodes based on proximity. CLIP embeddings, augmented by semantic knowledge from a fusion of household ontologies provide a robust semantic signal. We evaluate in simulation on a noiseless scene graph from simulator metadata, making the approach detection-agnostic. Our approach outperforms random, CLIP, Tidybot, and an LLM planner in the majority of room classes and configurations in simulation. To show our approach is robust to scene graph node placement and label noise, we demonstrate a physical mobile manipulator running in an exploration pipeline start-to-finish including scene graph labels from Detic.