RawVLA improves robot vision by adapting raw camera data for tasks
RawVLA: Embodied Neural Image Signal Processor For Robotic Manipulation
RoboticsComputer Vision and Pattern Recognition
Summary
Robots often use pictures processed by fixed camera settings, which can limit their ability to see well in different lighting or conditions. The authors studied how changing these camera settings affects a robot's ability to act. They created RawVLA, a smart system that adjusts raw camera input to help robots see better for their tasks. This system keeps good performance in normal settings and works better when the camera image quality gets worse.
What this means in practice
- •For robotics engineers: Improve robot hand-eye coordination by adapting raw camera signals dynamically for manipulation tasks under varying visual conditions.
- •For drone operators: Enhance drone navigation and object interaction by adjusting raw imaging inputs to maintain vision quality in challenging environments.
Authors
Shuhong Liu, Heng Zhou, Lingfeng Qian, Yuhao Fang, Xianbao Hou, Qianyu Zhou, Lin Gu, Wei Sui, Jianfei Yang, Ziteng Cui
Abstract
Vision-language-action (VLA) models typically operate on RGB images produced by a fixed camera image signal processor (ISP), leaving the imaging pipeline outside the learning and evaluation loop. We systematically examine the consequences of this overlooked design choice across five fundamental ISP dimensions: gain, sensor noise, chromatic response, tonal response, and bit depth. Our analysis reveals that RAW-to-RGB processing materially shapes both action prediction and manipulation success, with different ISP dimensions exerting substantially different effects. Guided by these findings, we introduce RawVLA, a streaming neural ISP that adaptively renders RAW observations for frozen VLA policies while concentrating its capacity on the imaging factors relevant to embodied behavior. We further present RawVLA-Bench, a RAW-domain manipulation benchmark to expose image processing as an explicit evaluation variable across clean and adverse acquisition conditions. Experiments on RawVLA-Bench show that RawVLA preserves performance under standard conditions while substantially improving robustness under degraded imaging, establishing adaptive RAW processing as an effective interface between physical cameras and embodied policies.