SM4RT improves 3D geometry and motion reconstruction from videos
SM4RT: Cascaded Feed-forward Model for 4D Reconstruction
Computer Vision and Pattern Recognition
Summary
Reconstructing moving 3D scenes from regular videos is challenging because camera motion and object movement can confuse algorithms. The authors present SM4RT, a step-by-step method that first builds the 3D shape and then estimates motion using that shape, making it easier to understand movements relative to the camera. They also use a special training strategy to handle missing motion data by relying on available position data. SM4RT performs very well on several tests involving dynamic scenes, improving motion estimation without sacrificing shape accuracy.
What this means in practice
- •For autonomous vehicle developers: Improve vehicle perception by accurately reconstructing moving environments from onboard cameras despite vehicle motion.
- •For augmented reality engineers: Enhance real-time dynamic scene understanding by combining geometry and motion estimates sequentially for better visual overlay stability.
Authors
Pang-Chi Lo, Jhao-Syun Lai, Tsung-Wei Ke
Abstract
We introduce SM4RT, a multi-task cascade feed-forward model for dynamic scene reconstruction from RGB videos. While existing feed-forward approaches typically address geometry reconstruction and motion estimation separately--and often predict motions in a single coordinate frame that must account for camera movement--SM4RT solves these tasks sequentially by first reconstructing 3D geometry and then bootstrapping motion estimation from the predicted structure. By expressing each pixel's 3D motion in its corresponding camera frame, SM4RT reduces ambiguity arising from camera motion. To address the sparsity of real-world motion labels, SM4RT further incorporates a weakly supervised loss that regularizes unlabeled point motions using labeled point positions at the other time step. We evaluate motion estimation on three dynamic-scene datasets--Kubric, Waymo, and Point Odyssey--and SM4RT achieves the best performance on most metrics compared with prior baselines. We additionally benchmark geometry reconstruction on five dynamic scene datasets and show that SM4RT achieves competitive performance with state-of-the-art methods. Overall, our unified framework demonstrates that combining geometry reconstruction and motion estimation in a cascade not only avoids performance trade-offs but also enhances motion estimation through improved geometric reasoning.