Memory improves robot decisions when history matters most
Beyond Success Rate: Interventional Evaluation of Memory in Vision-Language-Action Systems
RoboticsMachine Learning
Summary
Robots often use memory to remember past information, but it’s hard to tell if the memory really helps them do better. The authors created a way to carefully test if robot memory actually changes important decisions, not just overall success rates. They found that when tasks require remembering past events, having the right memory almost doubles success. However, just measuring success rates isn’t enough; we need detailed tests to confirm memory is really used as intended.
What this means in practice
- •For robotics engineers: Design robot controllers that verify memory actually influences decisions in complex tasks requiring historical context.
- •For automation system developers: Evaluate and improve memory modules in vision-language-action systems for better task outcomes dependent on past information.
Authors
Keyuan Luo, Yi Xie, Wanyun Ling, Yuanzhe Li, Ziyue Li, Zhuoqi Zeng
Abstract
When we say that a robot uses memory, does that memory actually play a role? Success-rate ablations alone cannot establish the use of task-relevant facts: they change information and computation together, while aggregate rates can hide offsetting gains and losses. We propose an Interventional Memory Evaluation Protocol (IMEP) for vision-language-action (VLA) systems to address four questions: (1) Does memory provide a net benefit? (2) Under what conditions does memory help? (3) Do changes to memory alter task outcomes? (4) Are these effects specific to task-relevant facts? At the retrieval interface, we compare target-fact substitutions with fact-preserving replacements, minimal numerical perturbations, and exact replicas, while assessing decision correctness separately from motor execution success. We evaluate within-episode, cross-episode, and agentic memory. In cross-episode learned-bank experiments, a one-unit-in-the-last-place (ULP) intervention, including its installation procedure, changes 11.9% of seed-paired outcomes, while changing key facts produces no clear additional effect over fact-preserving replacements. To examine when memory helps, we introduce ChainBench, where correct decisions require historical information unavailable in current observations. The full memory configuration raises task success from 39.1% to 74.3%, matching the oracle-decision condition under shared recorded executions. These results show that correct history can support robot decisions, but establishing whether a memory system realizes this benefit requires fact-specific evidence beyond aggregate success rates.