PhyMo recovers accurate continuous 3D motion from multi-view videos
PhyMo: Learning Physical Dynamics with Accurate and Continuous Motion from Multi-View Videos
Computer Vision and Pattern Recognition
Summary
Capturing how objects move in 3D across time using videos from multiple angles is very hard. Often, models guess motion incorrectly by changing colors or transparency to fool the system, which leads to bad predictions. The authors created PhyMo, a method that focuses on tracking true movement instead of appearance tricks. It also ensures smooth and consistent predictions over time, producing more realistic 3D motion from videos. Tests show PhyMo works better than previous methods on various dynamic scenes.
What this means in practice
- •For visual effects artists: Generate smooth, physically accurate 3D animations from multi-camera footage without manual motion capture setups.$Commercial implications: Enables production of realistic 3D animations for film and games by converting video directly into accurate motion data.
- •For robotics developers: Improve robot perception by reconstructing precise object motion from multi-view video streams for better interaction and control.
Authors
Shangjia Liu, Jinxi Li, Siyuan Zhou, Bo Yang, Bing Wang
Abstract
Recovering dynamic 3D scene geometry, appearance, and the physical motion states that govern scene evolution from multi-view videos is important yet challenging. A key difficulty is that appearance can be deceptive. Under rendering supervision alone, a model may fit the observed images by changing color or opacity to compensate for inaccurate deformation, rather than recovering the true evolving geometry and motion. This issue is especially severe in scenes with fast or locally complex motion, where inaccurate trajectories can still produce plausible images over the observed frames, but lead to unreliable reconstruction and discontinuous future prediction. In this paper, we propose PhyMo, a framework for learning evolving dynamics from multi-view videos. We introduce a Motion-Aware Trajectory Representation that drives the model to recover accurate motion from deformation, rather than relying on appearance shortcuts to explain observed motion. On top of this, we propose a Continuous Motion Evolution module with bidirectional kinematic continuity constraint to promote temporally consistent evolution of displacement, velocity, and acceleration. Together, these designs yield geometrically faithful interpolation and continuous motion extrapolation. Experiments on four dynamic scene benchmarks demonstrate state-of-the-art performance.