Memory benchmark reveals limits in robotic manipulation recall
MEMOBench: A Process Level Memory Benchmark for Robotic Manipulation
RoboticsArtificial Intelligence
Summary
Robots that carry out tasks often need to remember information that they can't currently see. The authors noticed that current ways to test robot memory mostly check if a task is finished correctly, which doesn’t always show if the robot actually remembered well. They created MEMOBench, a new set of memory-related tasks and data to better measure how robots store, update, and compress memory during these tasks. Their tests show that current robot memory methods still struggle, proving this is a challenging problem worth further work.
What this means in practice
- •For robotics engineers: Assess and improve robot memory capabilities using MEMOBench’s detailed memory operation labels during manipulation tasks.
- •For machine learning developers: Train and evaluate memory modules in robotic perception models with MEMOBench’s semantic and contrastive memory alignment supervision.
Authors
Haiyang Sun, Haoxiao Wang, Junming Chen, Weicheng Fang, Zihao Su, Jingkun Yi, Wenyou Yi, Hao Chen, Zhou Zhao
Abstract
Robotic manipulation often requires acting on information that is no longer visible, yet Vision-Language-Action policies are usually evaluated when the current observation largely determines the next action. Existing robotic memory benchmarks expose this gap, but they still rely mainly on final task success and therefore conflate forgetting with manipulation failure. We present \textbf{MEMOBench}, a benchmark for process level memory evaluation in robotic manipulation. MEMOBench includes 30 history dependent tasks, 1{,}500 expert demonstrations, and 4{,}200 executable checkpoint instances from 84 templates. Each checkpoint pairs coarse to fine language with a simulator predicate and labels one memory operation: Storage, Update, or Compression. These annotations define Memory Storage Rate, Memory Update Rate, and Memory Compression Rate, which measure memory fidelity alongside task success. Across standard and memory augmented VLA policies, the strongest memory module baseline reaches only 31.9\% average success rate, and high storage often coexists with weak update and compression. Checkpoint language also supervises semantic, contrastive, and framewise memory alignment objectives, yielding modest gains across different memory operations. MEMOBench provides a diagnostic evaluation suite and training supervision for memory grounded robotic policies. The project page is available at https://github.com/Collab-Gen/MEMOBench.