Vision language action models reveal shapes of task variables in robot control
A Phenomenology of Neural Geometry in Vision-Language-Action Models
RoboticsMachine Learning
Summary
Robots often need to understand complex tasks involving moving parts and targets, which have geometry like angles and distances. The authors studied how these geometric concepts are represented inside AI systems that control robots using vision and language. They found that these internal representations have shapes reflecting real-world constraints, like limited joint movement or detailed distance tracking. This means these AI models form rich, multi-dimensional shapes in their 'thoughts,' going beyond just recognizing if a concept is there.
What this means in practice
- •For robotic system engineers: Use geometric insights from model activations to guide robot policies more precisely along task-relevant concepts.
- •For autonomous vehicle developers: Incorporate understanding of multi-dimensional internal representations to improve sensor and control integration for navigation tasks.
Authors
Mozes Jacobs, Aiden Swann, Mathilde Papillon, Siddharth Boppana, Fenil R. Doshi, Usha Bhalla, Thomas Icard, Matt Feiszli, Thomas McGrath, Vasudev Shyam, Demba Ba, Leon Bergen, Nina Miolane, Owen Lewis, Jack Merullo, Ekdeep Singh Lubana, Atticus Geiger, Matthew Kowal, Thomas Fel
Abstract
Robotic manipulation is inherently geometric, but what shape does the world take inside the policy that performs it? Prior work on vision-language-action (VLA) policies establishes that task-relevant variables are linearly readable from activations, yet readability certifies only that a variable is present and says nothing about the shape its values trace, which may be curved and multi-dimensional. Using controlled sweeps, natural rollouts, linear probes, and activation interventions, we build a taxonomy of seven task-relevant concepts in VLA models. We find that realized geometry is predominantly multi-dimensional and is shaped by the constraints under which the policy acts. Physical joint limits, for example, leave wrist orientation an open arc rather than a circle, and distance to the target, a single scalar, is realized in two orthogonal subspaces, a coarse ruler for the approach and a fine ruler for the grasp. The same geometries reappear in two further VLA models on real-world robot data, indicating that they are neither model-specific nor artifacts of simulation. We demonstrate how the findings are actionable by steering along the geometry a concept realizes. Overall, this work lays out an initial phenomenology of realized geometry in VLA policies and demonstrates that a concept's shape carries as much importance as its readability.