Sensorimotor alignment predicts robot model success with no extra training
Sensorimotor Alignment: A Platonic Proxy for Embodied Policy Success
Computer Vision and Pattern RecognitionMachine LearningRobotics
Summary
Robots that learn to move and sense the world are usually tested by letting them act and seeing if they complete tasks, which can be slow and expensive. The authors propose a new way to guess how well a robot model will do by checking if its internal sensory understanding matches the natural patterns of real motion, without doing any action testing. They call this check Sensorimotor Alignment (S²) and show it is linked to how well robots perform tasks later. This method also helps pick good vision-language models before fully training them for robot actions.
What this means in practice
- •For robot developers: Estimate robot model capability offline using sensorimotor alignment metrics to prioritize models before costly real-world testing.
- •For machine learning engineers: Select vision-language model backbones more likely to succeed in robot tasks without expensive fine-tuning efforts.
Authors
Yunfei Xie, Howard H. Qian, Shiyi Lan, Kaiyu Hang, Chen Wei
Abstract
Robotic foundation models are typically evaluated through costly closed-loop rollouts, which motivates cheap offline proxies for model capability. The Platonic Representation Hypothesis (PRH) suggests that as representations improve, different modalities such as vision and language converge toward a shared statistical model of reality. We ask whether motion trajectories form another such modality: as embodied models become more capable, should their sensory representations converge toward the structure of physical motion? To examine this, we propose Sensorimotor Alignment ($S^2$), a training-free offline metric that measures how well a model's sensory representation aligns with the geometry of physical motion. We find that $S^2$ correlates positively with downstream robot task success across embodied model families, making it a low-cost offline proxy for prioritizing models before closed-loop evaluation. $S^2$ can also identify promising vision-language model (VLM) backbones even before costly fine-tuning into vision-language-action models (VLAs). Beyond this correlation, $S^2$ offers representation-level explanations for phenomena observed in VLAs and world action models (WAMs). With $S^2$, we extend PRH to embodied models and make sensorimotor alignment empirically measurable, offering insight into why an embodied model succeeds beyond its closed-loop success rate.