Latent memory improves robotic control by tracking unseen actions
LTM: Latent Transition Based In-Context Memory for Vision-Language-Action Models
RoboticsArtificial Intelligence
Summary
Robots often need to remember what happened before to perform tasks well, even if they can't currently see earlier actions. The authors present Latent Transition Memory (LTM), which stores a compact summary of changes between consecutive moments, helping the robot recall important past interactions. Their system, LTM-π, uses these summaries to improve tasks like manipulation without needing extra labels or complicated reasoning steps. Tests show it remembers better, works efficiently with limited memory, and even improves real robot performance.
What this means in practice
- •For robotic control engineers: Enhance robot manipulators' ability to remember past unseen interactions for more accurate and reliable task completion using compact latent memory.
- •For industrial automation teams: Improve efficiency of assembly line robots by enabling bounded memory use for long temporal action sequences without extra data labeling or complex reasoning modules.
Authors
Qixin Sun, WenLong Gao, Xiansheng Chen, Zhongyuan Wang, Si Liu, Pengwei Wang
Abstract
Robotic manipulation often requires remembering interactions that are no longer visible. We introduce Latent Transition Memory (LTM), which stores history as compact latent transitions between consecutive observations. Instantiated on $π_{0.5}$ as LTM-$π$, our method uses a frozen latent action encoder, a lightweight adapter, and causal KV caching for incremental inference without memory-specific semantic annotations or external reasoning-model calls. With DreamDojo transitions, LTM-$π$ achieves 58.54% success on RoboMME and 80.5% on LIBERO-Mem, performing best among baselines without external reasoning models or memory-specific annotations. Ablations support the contribution of transition content beyond temporal cues and the efficiency of delayed memory admission. Scaling RoboMME demonstrations tenfold improves success from 57.67% to 87.50%, while a Cup Swap diagnostic highlights the role of process supervision in temporal composition. Under a 200-token memory budget, success remains 84.00%, versus 87.50% without compression. Finally, real-robot counting success improves from 11.25% to 82.50%. Together, these results demonstrate the effectiveness, scalability, and efficiency of latent transition memory, highlighting its capacity for complex temporal information, bounded-context operation, and real-world robotic control.