Embodied world models evaluated with synchronized tri-view robot videos
TriWorldBench: A Tri-View Consistency Perspective on Embodied World Models
RoboticsArtificial Intelligence
Summary
Robots that use cameras on their head and wrists see different parts of a task while working, like watching their whole body and their hands closely. The authors created TriWorldBench, a test set that uses videos from these three camera views together to check if a robot’s prediction of actions is consistent and accurate from all views. This helps measure if the robot truly understands the task and the objects it interacts with, not just in isolated views but as a whole. Their benchmark includes many tasks and 19 ways to check these predictions, helping improve and compare robot learning models.
What this means in practice
- •For robotics engineers: Test and improve robot models that predict tasks by verifying their predictions across multiple synchronized camera views from head and wrists.
- •For robotics software developers: Develop and evaluate algorithms to ensure robot action predictions are consistent between overall and detailed hand views for bimanual tasks.
Authors
Xuanyi Liu, Haofeng Wang, Ruiqi Li, Danni Yu, Rui Wan, Ruixu Zhang, Siyu Tao, Xue Yang, Shaofeng Zhang, Zicheng Zhang, Jiaqi Zhang, Siwei Ma
Abstract
Embodied world models predict the outcomes of robot actions to support learning and planning. For robots equipped with head and wrist cameras, this requires complementary views: the head view captures the overall task, while wrist views reveal local gripper-object interactions. However, evaluating these views independently cannot determine whether they describe the same action and object state. We introduce TRIWORLDBENCH, a benchmark for evaluating embodied world models through synchronized head, left-wrist, and right-wrist videos. It contains 500 episodes across 50 bimanual manipulation tasks and uses 19 metrics to assess tri-view consistency, task alignment, physical and 3D coherence, motion quality, temporal consistency, and visual quality. By combining cross-view checks with measurements tailored to each camera, the benchmark evaluates whether plausible individual videos also form a consistent prediction of the intended task. We summarize overall performance with TWB-Score and retain per-view results to identify where predictions fail. This extends world-model evaluation beyond single-view visual quality. Code, data, and metric definitions are available at https://github.com/TriWorldBench/TriWorldBench.