FoveaVLA improves robot vision by replacing wrist cameras with foveation actions
FoveaVLA: Foveation Actions Replace the Wrist Camera in VLAs
RoboticsArtificial Intelligence
Summary
Robots often use a small camera near their hand to see details during tasks, but this limits their overall view. The authors created FoveaVLA, which replaces this wrist camera with smart eye movements called foveation actions to zoom in on important spots using external cameras. This way, the robot sees both the whole scene and detailed close-ups where needed. Their approach helps robots perform tasks more successfully, especially when objects are far or surrounded by distractions.
What this means in practice
- •For robotic system integrators: Build robot vision setups that use external cameras with foveation to improve manipulation accuracy without wrist cameras.
- •For warehousing automation teams: Enhance picking systems to better handle cluttered shelves and distant objects by leveraging foveation-guided observation regions.
Authors
Bingao Chen, Lihan Zha, C. Karen Liu, Guy Tevet
Abstract
We present FoveaVLA, a Vision-Language-Action (VLA) model that replaces the wrist camera with learned foveation actions. Wrist cameras provide close-up detail but restrict the view to the end-effector's surroundings. External cameras see the whole scene, but the policy receives them at low resolution, so the fine detail needed for manipulation is lost. FoveaVLA recovers this detail by predicting, alongside each action chunk, the interaction region to observe at the next policy query. The policy then receives crops of these regions, taken from the original-resolution images, together with the full views. We build FoveaVLA on $π_{0.5}$ in two configurations, Single and Dual, using one or two external stereo cameras with full and foveal RGB and depth inputs, and post-train both on DROID with RoboInter interaction annotations. Across four real-world task suites, FoveaVLA-Single achieves 70.8% success with one external RGB-D view, and FoveaVLA-Dual achieves 77.1%, versus 52.1% for the wrist-equipped $π_{0.5}$. Both match the baseline on plain pick-and-place and improve success with distractors, cloth manipulation, and far-away objects.