Visual generators struggle with physical consistency during ego motion
Beyond Visual Quality: Evaluating Physical Consistency under Ego-Motion with EgoGenEval
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Generating clear images that stay realistic when the viewpoint changes is a tough problem. The authors created a new benchmark called EgoGenEval to test if image generators can keep scenes physically consistent when the camera moves. Their tests show most current models fail to match camera movements and keep the scene unchanged at the same time. They also explored training methods to improve this but found current approaches only help a little, pointing to the need for new techniques.
What this means in practice
- •For augmented reality developers: Improve AR systems by evaluating and enhancing how virtual scenes maintain physical consistency when users move their viewpoint.
- •For robotics software engineers: Develop visual perception modules that reliably reflect real-world scene changes during robot navigation and planning.
Authors
Yilin Long, Chenming Zhu, Zitang Gou, Jingli Lin, Tai Wang
Abstract
Recent visual generators produce high-fidelity images yet often violate physical consistency under ego-motion, limiting their use for spatial reasoning and embodied planning. Existing benchmarks largely focus on isolated images or single-step quality, leaving this challenge underexplored. We introduce EgoGenEval, a geometry-grounded, pose-free benchmark designed to evaluate the physical consistency of visual generators under ego-motion, and organize our study into two parts. (1) EgoGenEval contains 1,400 cases and 2,360 target views spanning single-step and multi-step ego-motion. It separately measures Camera Motion Grounding (CMG) and Scene State Preservation (SSP), with both metrics validated against blinded human judgments. Evaluating 16 pose-free generators together with two pose-conditioned references reveals that current models struggle to execute camera motion while maintaining scene state, and that no system performs well on both axes at once. (2) To examine whether benchmark-derived data can improve these capabilities, we build EgoGen-Train from the same geometry-grounded pipeline and run controlled SFT studies. These show that pairwise supervision does not reliably improve camera-motion grounding and scene-state preservation together: even at the full training pool and the longest budget, scene preservation gains a fraction of what camera motion does. This points to the pairwise teacher-forced objective itself as the binding constraint, motivating a trajectory-centric paradigm that couples self-conditioned rollouts with explicit pose and visibility supervision.