Event data helps improve 3D video from few blurry images
EdMCGS: Event-Driven Markov Chain Gaussian Splatting for Extreme-Low-Frame-Rate Dynamic Scene Reconstruction
Computer Vision and Pattern Recognition
Summary
Capturing smooth 3D videos from very few regular pictures is hard because important motion information can be missing. The authors created a method called EdMCGS that uses special event-based cameras along with sparse images to better understand how objects move between pictures. It treats the scene like a chain of steps driven by event data, which helps make smoother and more accurate 3D videos at times between the pictures. Their method works faster and with less data than other approaches while producing clearer results. They also shared their code and a new collection of data for others to test.
3D reconstructionevent camerasMarkov chainGaussian splattinglow-frame-rate videodynamic scenesmotion modelingreal-time renderingtemporal isometrycontrol points
Authors
Yuzhong Wang, Wenmin Wang, Xinxing Yu
Abstract
We present EdMCGS (Event-driven Markov chain Gaussian Splatting), an end-to-end method for reconstructing dynamic 3D scenes from extreme-low-frame-rate RGB together with an event stream, which can then be rendered at any intermediate timestamp. Methods relying solely on RGB images generate numerous artifacts due to the lack of evidence from between consecutive frames. To supply this missing evidence, we model the scene motion as an event-driven Markov chain, in which the sparse RGB frames anchor the state at their own timestamps while the events recorded within an interval drive the transition across it. Since the transition reads the events of the current interval, it remains active at inference and produces the in-between motion of the 3D Gaussians directly from the events rather than by interpolation, which sets our method apart from prior work that uses events only as training-time supervision. The state is carried by a compact set of control points, each driven by the events sampled in the neighborhood of its own image projection, and a temporal local isometry term keeps the propagated motion locally rigid. Experiments on synthetic and real-world scenes show that EdMCGS outperforms both RGB-based and event-based baselines, while rendering in real time with far fewer Gaussians than the strongest event-based baseline. We release our source code and a new dataset at https://github.com/joseclipse/EdMCGS.