Robots use lightweight memory to improve task performance
Workspace Models: Lightweight Robotic Memory via Saliency-Driven Supervision
RoboticsArtificial Intelligence
Summary
Robots often need to remember previous steps to complete complicated tasks well. The authors found that asking a big language model for help all the time slows things down and can cause mistakes. Instead, they trained the robot to create a small, easy-to-use memory called a workspace token by learning what past information matters most. This compact memory helps robots do their jobs better and faster without needing the big model during actual task execution.
What this means in practice
- •For robotics engineers: Enable robots to remember relevant past events efficiently to improve performance on complex tasks without slow external queries at runtime.
- •For industrial automation teams: Implement lightweight memory tokens to streamline robotic task sequences, reducing computational load and increasing reliability in real-world factories.
Authors
Nitish Dashora, Douglas Chen, Idan Shenfeld, John Marangola, Pulkit Agrawal, Max Simchowitz
Abstract
Complex robotic manipulation tasks frequently require a long-term memory of past events and actions. As conditioning on full histories renders policies prone to spurious correlations and degrades performance, many approaches to policy memory involve compressing historical information through expensive VLM queries in-the-loop to process only task-salient information. In this paper, we propose an alternative approach in which computationally intensive VLM queries are made during train-time to learn a lightweight latent memory that can be efficiently queried at deployment time. Our representation, which we call the \textbf{workspace token}, is trained by (1) using a VLM to identify current and historical information necessary for completing a task, then (2) distilling these into the workspace token using a set-reconstruction decoder loss. In both simulation and hardware, we show that the workspace token can be used as a drop-in replacement for observations during deployment, enabling policies to solve memory-intensive tasks without the need for VLM reasoning in-the-loop. Interestingly, we found that workspace tokens are not only more lightweight but also lead to better policy performance.