Vision language action models improve by accessing evolving visual data
From Pixels to Actions - Towards Native Vision-Language-Action Policies
Computer Vision and Pattern Recognition
Summary
Robots often rely on visual information to decide what actions to take, but most current models simplify this visual data in a way that loses important details. The authors propose a new approach called NEO-VLA, where visual information is processed together with language instructions throughout the model, rather than being fixed early on. This method helps the robot better understand tricky visual details, improving how well it can perform tasks, especially when conditions change. Their findings suggest that accessing visual data at multiple levels, not just the final summary, leads to better control in robots.
What this means in practice
- •For robotics engineers: Improve robot manipulation abilities by using visual representations that evolve with instructions, enhancing performance under varied conditions.
- •For industrial automation teams: Develop more robust automated systems by integrating vision and language inputs at multiple processing stages for better interaction with objects.
Authors
Weichen Fan, Haiwen Diao, Penghao Wu, Siyuan Hu, Ziqi Huang, Tianqi Liu, Mingze Sun, Ziyi Wang, Zhongang Cai, Hanming Deng, Quan Wang, Lei Yang, Lewei Lu, Fangzhou Hong, Zhaoxi Chen, Dahua Lin, Ziwei Liu
Abstract
Vision-language-action models are typically built on a simple premise: perception is first compressed by visual encoders, and action reasoning begins only afterwards. Yet manipulation often hinges on precisely the visual signals most vulnerable to this abstraction, e.g., local geometry, boundaries, depth, and subtle state changes. This raises a fundamental question: should action policies inherit a fixed visual endpoint, or directly access evolving visual computation? We introduce NEO-VLA, a structurally encoder-free vision-language-action framework. Images are tokenized by lightweight patch convolutional layers and evolve jointly with language throughout the backbone, allowing a flow-matching action expert to draw from instruction-conditioned visual representations at different depths. This simple change reveals a consistent pattern. Native architectures outperform encoder-based counterparts under controlled comparisons; full fine-tuning reduces but does not erase the gap, and additional scale alone cannot recover it. More strikingly, layer-wise probing shows that the action-critical signals often peak well before the final visual representation. Across diverse manipulation settings and distribution shifts, NEO-VLA consequently achieves stronger and more robust control. Together, these results suggest that embodied control benefits from access to visual representations across the computational hierarchy, rather than a fixed encoder output.