Robot vision improves by condensing recent observations efficiently
CondenseVLA: Learnable History Condensation for Efficient Multi-Frame VLA
Computer Vision and Pattern RecognitionRobotics
Summary
Robots often need to remember brief but important visual details from recent moments to do tasks well. Simply looking at more past video frames is slow and has a lot of repeated or unhelpful information. The authors created a way for robots to smartly compress recent visual history into a small set of useful signals for making decisions. This method helps robots perform better on several testing tasks and real-world scenarios, showing it can pick out the right visual clues without extra wasted work. However, when tasks require longer-term memories, this short history approach hits limits.
What this means in practice
- •For robotics engineers: Improve robot action decisions by integrating recent visual history in a compact, learnable form.
- •For automation system designers: Design vision systems that efficiently use short-term observation data without high computational costs.
Authors
Feiyang Hong, Yujie Wei, Bo Zhao, Xiu Su, Hongxun Yao, Shuo Yang
Abstract
Robot manipulation often depends on short-term cues - such as brief contacts, temporary occlusions, and intermediate object states - that are ambiguous in a single frame and can only be resolved by integrating observations over time. However, injecting historical information solely by enlarging the history window is computationally costly and frequently yields limited gains due to substantial redundancy and task-agnostic, inflexible sampling of observations. We propose CondenseVLA, a multi-frame VLA policy with a learnable decoder-side history condensation module for using short-term visual context in action decoding. Given a sequence of cached per-frame features from the VLM backbone, CondenseVLA distills observation history into a fixed set of slot tokens using learnable queries and multi-head cross-attention. Simple auxiliary regularization penalizes overlap among slot attention maps and decorrelates slot embeddings, encouraging complementary attention and reducing slot redundancy. Across the evaluated tasks, CondenseVLA improves Simpler-WidowX success from 58.7% to 69.8% (+11.1 percentage points), achieves 97.2% on LIBERO, and reaches 52.8% on real-world tasks. It achieves 21.6% success on memory-dependent RMBench tasks. These results support selective condensation of bounded recent visual evidence for action decoding, while harder multi-observation tasks expose the limits of a finite history window. Project page: https://github.com/FeiyangHong/CondenseVLA