PrimeSim simulates robot actions with consistent multiview long-term accuracy

PrimeSim: A Generalist World Model Simulator from Large-Scale Cross-Embodiment Demonstrations

Robotics

Summary

Simulating how robots act in different scenes is tricky because it requires following commands precisely, keeping all camera views consistent, and staying accurate over long periods. The authors created PrimeSim, a model that learns from a huge variety of robot videos across many types of machines and improves how it matches actions with visual outcomes. It also shares information between different camera angles to keep consistency and uses a special training method to reduce errors when predicting future frames. PrimeSim can run simulations fast, last a long time without losing accuracy, and quickly adapt to new robots it hasn't seen before.

What this means in practice

  • For robotics engineers: Simulate long sequences of robot actions accurately with consistent views to develop and test manipulation policies faster and more reliably.
  • For robot software developers: Fine-tune simulators quickly for new robot platforms using a pretrained model that adapts with minimal training steps.
  • For video game developers: Create realistic multi-camera robotic animations and interactions that stay consistent over time with less manual scripting.$Commercial implications: Enables advanced robot animation systems that can be sold as middleware or animation tools for games and VR.

Authors

Travis Davies, Wentao Ma, Lihao Guo, Tengfei Ma, Xiaowen Chang, Jiafeng Xu, Shilong Wang, Yiqi Deng, Shaowen He, Xiaoqi Li, Qian Zhu, Ran Cheng, Hao Dong

Abstract

A robot world model that serves as a simulator must do more than generate plausible videos: it must faithfully follow commanded actions, maintain consistency across camera views, and sustain coherent long-horizon rollouts. We present PrimeSim, an action-conditioned, multiview world model simulator that addresses these challenges through progressive midtraining and posttraining. Starting from a pretrained video generator, we midtrain on 30,110 hours of robot demonstrations spanning 22 platforms. To improve action following, we introduce a unified cross-embodiment action representation and frame-aligned dual-path conditioning to strengthen action-frame correspondence. To maintain multiview consistency, we develop view-aware interleaved attention with explicit camera identities, enabling cross-view information exchange while preserving view-specific representations. We further introduce a fast-video, slow-memory posttraining framework that transforms the bidirectional video generator into an efficient autoregressive simulator. By combining fast causal generation, self-rollout training, and hierarchical temporal memory, PrimeSim mitigates error accumulation from self-generated histories and supports coherent long-horizon simulation. PrimeSim runs at 10.1 FPS, remains stable over long-horizon rollouts on rigid and deformable manipulation tasks, and transfers to held-out platforms within 100 to 200 gradient steps where the pretrained backbone does not. Against five open-source action-conditioned world models it leads four of the five WorldSimProbe suites and posts the best overall EWMBench score, while fine-grained local calibration still trails the field; its rollouts are faithful enough to support GRPO-based policy optimization. We release the model weights, inference code, and training code.