ClearGS improves 3D scene reconstruction from shaky phone videos
ClearGS: Reliability-Aware Gaussian Splatting from Handheld Videos
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Capturing 3D scenes from handheld videos is tricky because the video frames vary in quality and viewpoints. The authors developed ClearGS, a method that smartly weighs each frame based on how clear and useful it is, instead of just choosing or ignoring frames. It also uses clever restoration techniques guided by the 3D model to fix blurry or distorted parts without needing perfect clean images. This leads to better, more detailed 3D reconstructions from everyday videos.
What this means in practice
- •For mobile app developers: Enable smartphone applications to create high-quality 3D models from casual handheld video footage with uneven viewpoints and varying quality.
- •For animation studios: Improve 3D scene reconstructions from on-set handheld footage for more accurate environment modeling without costly reshoots.
Authors
Xuanzhi Liu, Xinyi Wu, Hang Pan, Wensi Huang, Zhenyao Wu, Ruize Han, Song Wang
Abstract
We present ClearGS for 3D Gaussian Splatting (3DGS) from handheld videos with uneven viewpoint coverage and mixed frame quality. Rather than selecting frames with binary decisions, ClearGS uses Reliability-aware View Allocation (RVA) to assign graded raw-supervision weights based on appearance reliability, degradation risk, and geometric utility, while weakly reactivating useful suppressed frames to maintain trajectory coverage. Since weighting cannot restore details lost to blur or distortion, ClearGS further introduces Render-Guided In-Video Restoration (RIVR). The current 3DGS render provides a pose-aligned structural candidate, a frozen no-reference restoration expert restores the corresponding raw video observation without any clean reference image, and no-reference perceptual scores select among the render, restored observation, and high-frequency fused candidate. ClearGS then applies Full-Trajectory Repair Consolidation to revisit accepted repairs and preserve details introduced early. On GS2E and GSOTM, ClearGS achieves state-of-the-art overall performance, with consistent CLIP-IQA and MUSIQ gains and LPIPS reductions in most degradation settings, without paired sharp supervision or matched clean references.