Motion disentangling improves monocular 3D deformable scene reconstruction

DispFlow-GS: Displacement Flow Supervision with Motion Disentangling for Monocular Deformable 3D Gaussian Splatting

Computer Vision and Pattern Recognition

Summary

Creating accurate 3D models of moving scenes from a single camera is tough because motion is hard to capture precisely. The authors found that previous ways to teach computers about motion using Gaussian flow had limits due to differences from real optical flow. They propose a new method called Displacement Flow that better matches movement by handling 3D displacements directly and separating scene motion from camera motion. Their approach also introduces a new way to check how well the predicted motion matches improvements in the rendered images. This work helps produce more faithful dynamic 3D reconstructions without necessarily changing how the images look.

What this means in practice

  • For robot perception engineers: Improve dynamic scene understanding by providing more accurate and stable motion signals from monocular cameras in robots.
  • For visual effects artists: Refine 3D deformation and motion alignment for better dynamic scene models without degrading image quality.

Authors

Thai Duy Nguyen, Haitian Zhang, Addison Lin Wang

Abstract

Accurate dynamic scene reconstruction is important for robotic perception, where temporally consistent representations of dynamic environments are essential. Deformable 3D Gaussian Splatting (3DGS) models dynamic scenes through deformation fields, and recent methods incorporate motion supervision by aligning rendered Gaussian flow with optical flow. However, we find that such Gaussian-flow-based supervision provides only limited improvements in motion modeling. We identify a fundamental limitation of this supervision paradigm, namely a domain gap between rendered Gaussian flow and optical flow. To address this limitation, we propose a motion supervision framework built on Displacement Flow, which splats per-Gaussian 3D displacements onto the image plane to provide direct and stable optimization signals. We further disentangle scene motion from camera motion via intermediate-view rendering, enabling more reliable motion priors and targeted constraints on deformation and geometry. We also observe a discrepancy between motion fidelity and image-based evaluation, where improved motion awareness does not necessarily translate into better rendered image quality or higher image-based metric scores. Motivated by this mismatch, we introduce Deformation-Rendering Consistency (DRC), a motion-aware metric that measures the alignment between predicted deformation and rendering improvement. Experiments on dynamic scene benchmarks show substantial improvements in motion localization and motion--rendering consistency, reaching up to 39% and 6%, respectively, while image-based metrics change by only about 0.1%. These results confirm the observed mismatch between motion fidelity and image-based evaluation, demonstrating the significance of DRC for motion-aware evaluation.
View PDFOpen arXiv