Summary
This paper looks at smart systems that can think, plan, and act both on computers and in the real world as robots. The authors created a framework called PAPAV to describe five key abilities: seeing the current situation, guessing what actions will do, making plans, acting, and checking results. They show that when these systems move from computers to robots, physical limits like incomplete information and no easy way to undo mistakes make some abilities harder. Their framework helps focus on measuring all abilities, not just whether tasks succeed, guiding better design and evaluation of these systems.
What this means in practice
- •For robotics developers: Design robot systems with a comprehensive ability framework that balances perception, planning, action, and verification for reliable performance under real-world constraints.
- •For human-computer interface designers: Develop interfaces that better integrate reasoning, planning, and action verification to support embodied agents operating both on computers and robots.
A position paper. It proposes an approach and reports no results.
Authors
Yanzhe Chen, Ziyi Yang, Jifeng Zhu, Qiming Huang, Ruihe An, Peiyao Xu, Hesen Yang, Runda Liu, Chang Gong, Zhijun Cao, Zechen Bai, Wenzheng Zeng, Yiqi Lin, Guoqiang Liang, Kevin Yuchen Ma, Kevin Qinghong Lin, Mike Zheng Shou
Abstract
Multimodal agents (MMAs) sustain interaction through reasoning, memory, tools, and feedback, most visibly as computer-use agents, while robotic systems (RSs) couple sensing and actuation under physical dynamics. We define multimodal embodied agents (MMEAs) as goal-directed systems that couple multimodal task reasoning with physical action and revise decisions from the resulting feedback. This raises a central question: what changes when a multimodal agent moves from computer-use to robot-use? We introduce PAPAV, a unified capability-centric framework of five recurring functions: Perceive the current state, Anticipate action effects, Plan a feasible course, Act through an interface or body, and Verify the outcome. Defined by function rather than architecture, these capabilities let PAPAV identify what all three share and then compare MMEAs with MMAs and with RSs. Five physical constraints, from partial observability to unverifiable outcomes, leave fewer choices fixed in advance and less room to reverse errors. Across 64 benchmarks, explicit evaluation centers on Act; only 5 assess Anticipate and 3 assess Verify, leaving both largely hidden behind task success. These constraints also frame the open challenges we identify. PAPAV therefore provides a common basis for designing reliable physical agents and measuring progress across all five capabilities rather than task success alone.