Learning fast memory boosts robot task success and speed
T$^2$Mem: Learning Test-Time Memory for Robotics
Robotics
Summary
Robots need to remember past events to do tasks well, but simply keeping history isn’t enough. The authors created a method called T2Mem that lets robots learn what to remember during tasks by updating their memory quickly at test time. This helps the robot focus on useful information without reprocessing everything again. Their approach improved success rates significantly on challenging robot tasks and was faster than existing memory methods.
What this means in practice
- •For robotics engineers: Enable robots to better remember and use past observations during tasks for improved manipulation success without heavy memory systems.
- •For machine learning practitioners: Speed up inference in sequential decision-making systems by encoding history into compact fast weights with test-time training.
Authors
Yize Liu, Huang Huang, Yining Hong, Zijian Du, Zhi Cao, Li Fei-Fei, Jiajun Wu
Abstract
Memory-dependent robotic manipulation requires policies to use information that is no longer available in the current observation. Retaining history alone is insufficient: memory must preserve information that supports future actions. One challenge is whether a memory-free foundation model can learn to retain and use historical information from action demonstrations alone, without external memory support. We introduce T$^2$Mem, a framework that develops this capability within a pretrained vision-language-action policy, without external reasoning models or memory-specific annotations. T$^2$Mem uses test-time training to encode observation history into compact fast weights through online self-supervised updates, avoiding repeated processing of the full history. An observation-grounded interface extracts vision-language information for memory formation and supplies retrieved context to the action expert. Action supervision shapes what the memory learns to retain and use, while alternating memory-policy learning gives each component a fixed counterpart during optimization. Across 16 RoboMME tasks, T$^2$Mem improves average success from 17.93% to 56.83% over the memory-free base policy and outperforms the recurrent-memory methods reported in the benchmark, while controlled profiling indicates at least 3x inference speedup over explicit methods. Project website: https://yzliu84.github.io/T2MEM-project/