Improving volumetric video by smart gaussian placement in dynamic scenes
Dusting the Phantom: Gaussian Allocation for Fading Dynamic Content
Computer Vision and Pattern Recognition
Summary
Volumetric videos create 3D videos by using many small 3D shapes called Gaussians to represent objects. Sometimes, fast-moving small objects or detailed textures get lost because the shapes are not placed in the right spots over time. The authors introduce two methods to detect these missing details and move or add Gaussians to better spots based on where the objects are headed. This helps improve the clarity and realism of fast-moving and complex scenes without needing more shapes overall.
What this means in practice
- •For augmented reality developers: Enhance real-time 3D scene reconstruction to preserve small fast-moving objects in AR experiences using improved Gaussian allocation.
- •For virtual event producers: Create more detailed and accurate live volumetric captures of dynamic performances to maintain fine textures and fast movements.
Authors
Minhyeok Roh, Minseo Lee, Jun Young Jeong, Byeonghyeon Lee, Lucas Yunkyu Lee, Jaesik Park, Eunbyung Park
Abstract
Dynamic Gaussian representations have made high-quality, real-time volumetric video increasingly practical, yet challenging motion can still break their reconstruction: small fast-moving objects may disappear, and fine textures in highly dynamic regions are often lost. In particular, simply training longer or using more Gaussians may still fail to recover small, fast-moving objects. We argue that the key bottleneck is not representational capacity itself, but where and when Gaussian primitives are allocated. Existing allocation strategies rely on current Gaussian locations and often overlook the spatiotemporal destinations of relocated Gaussians. To address these limitations, we introduce two complementary strategies. First, Seed Injection identifies under-reconstructed dynamic regions using temporal appearance changes and reconstruction errors, and initializes new Gaussians at geometrically plausible 3D locations through multi-view observations, independent of existing Gaussian locations. Second, Temporal Biased Relocation uses time-resolved optimization signals to guide the spatiotemporal placement of relocated Gaussians. Both strategies maintain a fixed Gaussian budget without modifying the underlying representation. Experiments across multiple dynamic-scene datasets demonstrate improved reconstruction quality, particularly in highly dynamic regions, recovering small, rapidly moving objects and fine-grained appearance details.