Force aware vision language control improves humanoid robot manipulation
Opt2VLA: Force-Aware Vision-Language-Action for Contact-Rich Humanoid Whole-Body Manipulation
Robotics
Summary
Humanoid robots need to control how hard they push or pull when doing tasks, especially when touching things. The researchers created a system called Opt2VLA that helps robots plan actions not just by where to move but also by how much force to apply. Their system learns from simulated examples that include both motion and force, improving the robot’s ability to handle delicate and contact-heavy tasks in ways that just motion alone can't. They tested their approach in simulation and on real robots, showing better force control guided by spoken instructions.
What this means in practice
- •For robotic system integrators: Develop humanoid robots that perform precise force-sensitive tasks guided by natural language commands for better manipulation in complex environments.
- •For automation engineers: Create controllers that regulate both motion and contact forces to improve task reliability where robots interact physically with objects or humans.
- •For industrial robot programmers: Enhance robot programming for assembly or maintenance tasks requiring delicate force adjustments through vision and language interfaces.$Commercial implications: This enables more adaptable and precise humanoid robots for industrial markets demanding complex contact-rich manipulation.
Authors
Fukang Liu, Yipu Chen, Jaehwi Jang, Danfei Xu, Zsolt Kira, Ye Zhao
Abstract
Humanoid robots are expected to perform diverse human-level tasks in daily environments, many of which require precise regulation of interaction forces. While recent vision-language-action (VLA) models have shown promise for semantic planning and visuomotor control, existing humanoid systems primarily represent actions through geometric motion goals and rely on whole-body controllers focused on motion tracking, with limited explicit reasoning or control of interaction forces. This limitation is particularly relevant in contact-rich tasks, where geometrically similar motions may require different force regimes depending on the task context and where visual observations may become unreliable after contact. In this work, we present Opt2VLA, a force-aware VLA framework that introduces explicit force commands at the VLA-to-control interface for humanoid whole-body manipulation. A single multi-task VLA policy jointly predicts both geometric motion goals and continuous contact-force references, which are tracked by task-specific reinforcement learning (RL)-based whole-body controllers. To provide scalable and physically grounded supervision, we generate dynamically feasible and contact-consistent training data via whole-body trajectory optimization (TO) with explicit force references. We evaluate Opt2VLA on three contact-rich humanoid tasks and show that explicit force conditioning enables more accurate and consistent force regulation than motion-only control, while physically grounded torque supervision from TO further improves force tracking accuracy and stability. Closed-loop evaluations further demonstrate language-conditioned force modulation with Opt2VLA in simulation and on humanoid hardware.