Robot reasoning with physical action words improves manipulation tasks
Physical Words: A Vocabulary for Structuring Latent Reasoning in Robot Manipulation
Robotics
Summary
Robots need to understand both the language and physical effects of their actions to perform tasks successfully. The authors created a system called PhysWord that uses simple action words as building blocks to represent what a robot will do and how the world will change next. This approach helps robots plan longer tasks better and work with both vision and touch information. PhysWord performed very well on different robot manipulation tests, including using two hands and delicate tools.
What this means in practice
- •For robotics engineers: Build robotic systems that better predict and organize physical actions for complex manipulation scenarios.
- •For automation system developers: Develop automated manufacturing solutions that require precise, long-sequence object handling using multimodal sensing.$Commercial implications: This paper enables improved robotic manipulation capabilities suitable for advanced automation products serving manufacturing industries.
Authors
Jiaming Liu, Xinghao Zhu, Yiming Zhang, Shanghang Zhang, Yinxi Wang, Yichu Yang, Zhuoyang Liu, Dantong Niu, Soha Pouya, Juekun Li, Yan Chang
Abstract
Multimodal reasoning provides robotic foundation models with predictive context needed for reliable action generation. However, language-based reasoning captures task semantics but struggles to express fine-grained physical dynamics, while latent reasoning models future state evolution but offers limited structure for organizing these dynamics into reusable reasoning primitives. To bridge this gap, we introduce PhysWord, a reasoning-before-acting foundation model built on Physical Words, which use atomic action verbs as shared semantic anchors for continuous representations of physical dynamics. Specifically, PhysWord predicts upcoming atomic action words and directly aligns their contextual hidden states with latent targets derived from the corresponding future observations. Building on these primitives, we construct a temporally structured PhysWord space with visual and optional tactile variants of each action word. Autoregressively predicting Physical Words across future timesteps captures transitions between execution stages, providing structured multimodal context for action generation. PhysWord achieves state-of-the-art success rates of 90.3\% on visuotactile UniVTAC and 99.6\% on vision-based LIBERO-Long, and outperforms competing methods on real-world tasks using bimanual grippers and dexterous hands. Further evaluations demonstrate that our structured latent reasoning framework generalizes to unseen scenarios and enables long-horizon manipulation.