AI summaryⓘ
The authors created HumanCLAW, a way to test if vision-language models (VLMs) can decide how a physical body should act, without confusing the model's choice with mistakes in movement execution. They separate decision-making from actual motion by letting the model give simple commands that are then turned into real physical movements, so they can focus on the model's smart choices. Using this setup, they made a big test called HumanCLAW-Bench with many tasks inside buildings, and found that current top VLMs struggle, with the best having under 17% success. The authors found that the main problem is not recognizing targets, but that the models can't keep track of their own body or know if they've reached a goal or hit something.
Vision-Language Model (VLM)Embodied AIAction Decision-MakingMotor ControlHumanCLAW FrameworkEgocentric NavigationPhysical ExecutionEmbodied Self-AwarenessBenchmarkingIndoor Scene Navigation
Authors
Siyao Li, Jiawei Gu, Shuai Liu, Kairui Hu, Zekun Li, Linjie Li, Chengcheng Tang, Po-Chen Wu, Ivan Shugurov, Lingni Ma, Michael Zollhoefer, Sizhe An, Abhay Mittal, Amy Zhao, Ranjay Krishna, Manling Li, Ziwei Liu, Chuan Guo
Abstract
Evaluating whether a vision-language model (VLM) can act through a physical body is challenging. The outcome of an action couples the VLM's decision with motor control. When a task fails, it is hard to tell whether the VLM made a bad choice or the motor controller simply failed to execute it, e.g., losing balance and falling. In this work, we introduce HumanCLAW, an evaluation framework that decouples action decision-making from low-level execution. At every step, a harnessed, off-the-shelf VLM issues an atomic skill command, and the command is translated into a sub-second chunk of continuous full-body motion with real physical consequences, including gravity and collisions. The body can therefore act freely in the physical world, while execution-side disturbances, balance and motor errors, are factored out. What remains measurable is the model's action intelligence: its moment-to-moment choice of what the body should execute next. Based on this framework, we build HumanCLAW-Bench: 1,218 long-horizon, egocentric find-navigate-interact episodes across 41 indoor scenes. We test nine state-of-the-art VLMs and find that none solves the benchmark; the best model reaches only a 16.8% success rate. Recognizing the target is not the bottleneck. What current VLMs lack is embodied self-awareness: they lose track of their own body, failing to tell where it is, whether it has reached the goal, or whether it has hit an obstacle.