Force aware vision and language model improves robot manipulation success
FR-VLA: Implicit Force Reasoning VLA for Contact-Rich Manipulation
Robotics
Summary
Robots need to understand when and how much to use touch or force information along with vision and language to better handle tasks that involve contact with objects. The authors propose a new model called FR-VLA that decides when force feedback is important based on what the robot sees and understands in language, and uses this knowledge to guide its actions. Their experiments show this approach improves performance on tasks requiring delicate or complex physical contact. The model dynamically adjusts how much it listens to force sensors depending on what phase the interaction is in.
What this means in practice
- •For robotics engineers: Build robots that dynamically weigh force feedback when manipulating objects in tasks needing careful contact control.
- •For industrial automation teams: Improve robotic assembly lines with smarter force-aware control improving handling of fragile or irregular parts.
Authors
Jiaju He, Zhiyuan Yang, Ruyang Han, Jiahong Tang, Zhiping Lin, Haiyue Zhu
Abstract
Integrating force awareness into Vision-Language-Action (VLA) models involves not merely encoding wrench measurements as an additional modality, but determining when and how strongly physical interaction should influence robot action generation. Existing VLAs that use force typically inject it through a fixed pathway, without explicitly determining when it is relevant to the current action. This calls for an intermediate force relevance mechanism inside the VLA, in which visual-language context estimates the relevance of physical feedback to the current task before force enters the action pathway. FR-VLA addresses this gap with a Vision-Language Force-Reasoning Model that uses learnable force query tokens to derive a force admission signal grounded in visual-language context, and uses this signal to modulate an external tokenized force pathway for action generation. In real-world experiments on five contact-rich tasks across two robotic platforms, FR-VLA achieves the highest mean success rate over six evaluated stages, and trajectory visualizations show that its force admission varies with the interaction phase.