Monocular video enables detailed continuous 3d motion reconstruction

RayCo4D: Ray-Conditioned Continuous Motion Model for Monocular 4D Dynamic Reconstruction

Computer Vision and Pattern RecognitionArtificial Intelligence

Summary

Capturing moving 3D scenes from just one camera is hard because parts of the scene might be seen only briefly or unclearly. The authors created RayCo4D, a new method that keeps track of how objects move smoothly over time, even when the camera sees only fragments of them. It treats the camera rays and motion separately, linking them through shared motion points that change continuously. This approach helps create more consistent and detailed 3D videos from just one camera view.

What this means in practice

  • For game developers: Generate realistic 3D animations from single-camera footage to create dynamic virtual environments without multi-camera setups.$Commercial implications: Enables production of dynamic 3D content for games with reduced hardware costs and simpler capture workflows.
  • For film post-production teams: Improve 3D scene reconstruction for special effects by producing consistent, smoothly moving digital doubles from monocular video.

Authors

Yuanxiang Ni, Xianliang Huang, Ruihan Hou, Feng Xie, Hao Zhang

Abstract

Reconstructing dynamic 3D scenes from monocular video requires inferring motion from noisy and intermittently visible observations. Long-range tracks provide valuable geometric constraints, but integrating their fragmented temporal coverage into shared, continuous motion remains challenging. We introduce \textbf{RayCo4D}, a ray-conditioned continuous motion model for monocular 4D reconstruction that explicitly separates visual measurements from latent 3D motion states. Each track is associated with a reference-time 3D query constrained to its camera ray, and its image and depth observations constrain shared latent motion nodes through local many-to-many associations. Each node maintains an $\mathrm{SE}(3)$ pose and body-frame velocity at sparse support times. A white-noise-on-acceleration process prior couples neighboring states and defines continuous-time interpolation, allowing observations across visibility intervals to jointly constrain the same latent state sequence.The inferred shared motion is transferred to Gaussian primitives through dual-quaternion blending, while reference-anchored local $\mathrm{SE}(3)$ residuals capture fine-scale non-rigid deformation beyond the shared scaffold. Experiments on diverse monocular dynamic scenes show that RayCo4D achieves competitive novel-view synthesis quality, improves structural and temporal consistency under rapid motion and incomplete tracks, and supports continuous-time rendering.