Memory types affect performance of robot agents indoors
What Should Embodied Agents Remember? Benchmarking and Evaluating Memory Modality for Embodied Agents
Robotics
Summary
Remembering past events helps robot agents answer questions or perform tasks better. The authors created a way to test what kinds of memory—like pictures, written descriptions, or structured maps—help these agents most in different indoor settings. They found that saving actual video frames or combining frames with scene graphs (which show how things relate in space) works best overall. Their tests also balance how well the robots do their jobs with how much computing power their memory uses.
What this means in practice
- •For robotics developers: Design robot memory systems that balance information quality and computation cost for indoor task performance.
- •For home automation engineers: Improve smart home robots by selecting memory types that enhance task success rates in complex indoor spaces.
Authors
Junxiang Wang, Yuchen Liu, Yi Wu, Guofei Chen, Nir Pechuk, Tao Chen, Zackory Erickson, Yuzhe Qin, Tianyu Li
Abstract
Memory allows large language model (LLM) agents to use past experience, often stored as textual records, when responding to requests. For embodied agents that control robots, visual and spatial observations introduce a broader choice of memory modalities, affecting both the information available for subsequent tasks and the model inference costs of constructing and using memory. We introduce MeMoREval, a benchmark for studying this problem through a controlled experience-request protocol. Agents first experience a sequence of events without knowing the subsequent tasks, and subsequently they use memory to respond to information and action requests without receiving new observations. The benchmark comprises 53 tasks across four simulated indoor environments and measures task success alongside the inference costs of memory construction and use. Our reference evaluations reveal that storing frames outperforms storing generated descriptions at lower write-time inference cost for the primary model, while storing frames with scene graphs achieves the highest aggregate task success for every tested model. Extensions to accumulated experience, selective frame retention, and visual input to memory writers demonstrate how the benchmark accommodates further memory-design investigations. Two physical-robot demonstrations establish the feasibility of applying its protocol beyond simulation.