Green for Go, Red for No: Visual Grounding via Semantic Segmentation for VLA Navigation Policies
2026-07-06 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionRobotics
AI summaryⓘ
The authors studied how robots that navigate using both language and vision can better understand their surroundings to avoid confusion. They created a method that colors safe paths green and unsafe areas red in real-time images to help robots know where to go. Testing this on a dataset showed their method reduced navigation errors, especially for longer instructions, by making the robot's path more direct. However, it did not improve understanding of the scene per distance traveled and wasn't helpful when goal images were used or with instructions very different from the training examples.
Vision-language-action modelsRobot navigationVisual groundingSegmentationSegFormerWaypoint errorTrajectory regularizationNatural language instructionsOut-of-distribution generalizationGrand Tour dataset
Authors
Adrian Szvoren, Dimitrios Kanoulas, Nilufer Tuptuk
Abstract
Vision-language-action (VLA) models enable robot navigation from natural language and visual goals, but remain susceptible to perceptual distractions and ambiguous scene interpretations. This paper presents the first empirical evaluation of visual grounding for VLA navigation policies. We propose a real-time segmentation-based grounding method that highlights traversable areas in green and non-traversable areas in red using SegFormer. Two variants are evaluated: observation-only segmentation and joint observation-goal augmentation. Using OmniVLA on the Grand Tour dataset, we show that visual grounding reduces the mean waypoint error by 27-44% at the farthest waypoint, depending on the instruction length. The benefits are greater for long instructions than for short instructions, and grounding provides little improvement for image goals. Normalized error analysis indicates that grounding primarily acts as a trajectory length regularizer, reducing the predicted path length by 30% without improving per-unit-distance reasoning. Our results indicate that visual grounding offers a simple, computationally inexpensive method to improve VLA navigation without model retraining, although it cannot compensate for missing training signals in out-of-distribution instructions.