Event-driven method reconstructs 3D scenes from low-frame video
EdMCGS: Event-Driven Markov Chain Gaussian Splatting for Extreme-Low-Frame-Rate Dynamic Scene Reconstruction
Computer Vision and Pattern Recognition
Summary
Capturing smooth 3D motion from just a few video frames is hard for computers because they miss what happens in between. This paper presents a new way to fill in those gaps by combining sparse video with special camera events that record quick changes. The authors use a mathematical model that directly moves 3D points using these events, creating more accurate and realistic 3D scenes at any point in time. Their method works better and faster than previous ones that only used video or events separately.
What this means in practice
- •For augmented reality developers: Create smooth 3D models of dynamic scenes using sparse video and event data to enable realistic real-time rendering at intermediate moments.
- •For robotics engineers: Improve visual understanding and 3D mapping in robots operating with low-frame-rate cameras by integrating event-driven motion estimates.
Authors
Yuzhong Wang, Wenmin Wang, Xinxing Yu
Abstract
We present EdMCGS (Event-driven Markov chain Gaussian Splatting), an end-to-end method for reconstructing dynamic 3D scenes from extreme-low-frame-rate RGB together with an event stream, which can then be rendered at any intermediate timestamp. Methods relying solely on RGB images generate numerous artifacts due to the lack of evidence from between consecutive frames. To supply this missing evidence, we model the scene motion as an event-driven Markov chain, in which the sparse RGB frames anchor the state at their own timestamps while the events recorded within an interval drive the transition across it. Since the transition reads the events of the current interval, it remains active at inference and produces the in-between motion of the 3D Gaussians directly from the events rather than by interpolation, which sets our method apart from prior work that uses events only as training-time supervision. The state is carried by a compact set of control points, each driven by the events sampled in the neighborhood of its own image projection, and a temporal local isometry term keeps the propagated motion locally rigid. Experiments on synthetic and real-world scenes show that EdMCGS outperforms both RGB-based and event-based baselines, while rendering in real time with far fewer Gaussians than the strongest event-based baseline. We release our source code and a new dataset at https://github.com/joseclipse/EdMCGS.