Memory benchmark reveals challenges in robotic manipulation tasks
MEMOBench: A Process Level Memory Benchmark for Robotic Manipulation
RoboticsArtificial Intelligence
Summary
Robots that manipulate objects often need to remember things that are not visible anymore, but existing methods mostly check if the robot completed a task rather than how well it remembers important details. The authors created MEMOBench, a new benchmark designed to test how well robots store, update, and compress memory during manipulation tasks. They collected many examples and labeled specific memory operations to better understand robot memory performance. Their tests show that current robot memory systems struggle to balance storing information with updating and compressing it, and the new benchmark helps diagnose these memory shortcomings and guides improvements.
robotic manipulationvision-language-action policiesmemory benchmarkmemory storagememory updatememory compressiontask successmemory fidelityexpert demonstrationsmemory supervision
Authors
Haiyang Sun, Haoxiao Wang, Junming Chen, Weicheng Fang, Zihao Su, Jingkun Yi, Wenyou Yi, Hao Chen, Zhou Zhao
Abstract
Robotic manipulation often requires acting on information that is no longer visible, yet Vision-Language-Action policies are usually evaluated when the current observation largely determines the next action. Existing robotic memory benchmarks expose this gap, but they still rely mainly on final task success and therefore conflate forgetting with manipulation failure. We present \textbf{MEMOBench}, a benchmark for process level memory evaluation in robotic manipulation. MEMOBench includes 30 history dependent tasks, 1{,}500 expert demonstrations, and 4{,}200 executable checkpoint instances from 84 templates. Each checkpoint pairs coarse to fine language with a simulator predicate and labels one memory operation: Storage, Update, or Compression. These annotations define Memory Storage Rate, Memory Update Rate, and Memory Compression Rate, which measure memory fidelity alongside task success. Across standard and memory augmented VLA policies, the strongest memory module baseline reaches only 31.9\% average success rate, and high storage often coexists with weak update and compression. Checkpoint language also supervises semantic, contrastive, and framewise memory alignment objectives, yielding modest gains across different memory operations. MEMOBench provides a diagnostic evaluation suite and training supervision for memory grounded robotic policies. The project page is available at https://github.com/Collab-Gen/MEMOBench.