Mobile robots improve tasks by remembering past experiences
MessyMem: Learning-from-Doing Memory for Mobile Manipulation
Robotics
Summary
Mobile robots that move around and handle objects usually start fresh each time they get a new task, forgetting what they learned before. This paper presents MessyMem, a system that helps robots remember things like which cabinets are locked or where items are located, using a 3D map combined with what it learned from interacting with the environment. The system can recall information from many past experiences to do better on future jobs. The authors tested it both in simulations and on a real robot, showing it significantly improved performance over other methods.
What this means in practice
- •For robotics engineers: Build robots that remember previous interactions to improve multi-room manipulation tasks.
- •For warehouse automation teams: Create mobile robots that recall object locations and task outcomes for more efficient item retrieval.$Commercial implications: Improves commercial mobile robot systems' efficiency by enabling knowledge reuse across tasks.
Authors
Anuva Banwasi, William Muckelroy, Priya Sundaresan, Linfeng Zhao, Jeannette Bohg, Cherie Ho
Abstract
Mobile manipulators deployed across many rooms and visits should improve with experience: after discovering that a cabinet is locked or finding an object in a drawer, the robot should reuse that knowledge rather than start each task from scratch. Yet today's robots often treat each task as new: compact scene representations omit interaction-derived knowledge, raw video histories are difficult to query, and VLM planners reason at inference time without persistently updating what the robot knows. We present MessyMem, a persistent memory system that enables mobile manipulators to learn from experience and reuse that knowledge across future tasks. It maintains a spatially grounded 3D scene graph of objects and locations, augments it with properties and outcomes learned through interaction, and links visual observations for fine-grained recall. We evaluate MessyMem in simulation and on a real mobile manipulator. In a continuous 25-task simulation spanning over 3 hours, MessyMem achieves 80.0% task progress, outperforming the strongest ablation by 14.8 percentage points and the strongest external baseline by 28.9 points, while retrieving task-relevant evidence from thousands of stored keyframes and over an hour into the past.