Federated learning speeds 3D scene reconstruction at wireless edge
Federated 3D Gaussian Splatting for Large-Scale Scene Reconstruction at Wireless Edge
Computer Vision and Pattern RecognitionDistributed, Parallel, and Cluster Computing
Summary
Building detailed 3D images of large scenes usually needs powerful computers and lots of data sharing. The authors found a way to train 3D scene models on less powerful edge devices with slow wireless connections, while keeping private information safe. They do this by smartly selecting parts of the 3D data to use and fixing differences between local models so they can be combined better. Their approach makes training faster and keeps the image quality high.
What this means in practice
- •For edge device developers: Design training workflows for 3D scene models that work efficiently with limited memory and compute on wireless edge devices.
- •For augmented reality platform teams: Build large-scale 3D reconstructions from edge-collected visual data while protecting user privacy and reducing training time.$Commercial implications: Enables commercial AR services that reconstruct environments on user devices without raw data sharing, improving privacy and performance.
Authors
Guanlin Wu, Chao Hu, Pu Chen, Juyong Zhang, Han Hu, Shuguang Cui, Jie Xu
Abstract
Three-dimensional (3D) Gaussian splatting (3D-GS) has emerged as a promising technique for large-scale scene reconstruction due to its high rendering efficiency and fidelity. However, the training of large-scale 3D-GS models at wireless edge faces various technical challenges including the limited communication, computation, and graphics processing unit (GPU) memory resources at edge devices, the structural inconsistency issue across local models hindering their effective aggregation, as well as privacy leakage risks associated with raw visual content and camera parameters. To address these challenges, this paper proposes a novel resource-efficient federated learning framework for efficiently training 3D-GS models of large scenes under severe resource constraints. First, we propose an on-device model lightweighting mechanism that adaptively selects and prunes Gaussian points to balance the rendering quality and training efficiency. In this mechanism, we quantitatively evaluate the importance of different Gaussian points at each device to facilitate the pruning, and use a novel importance-to-latency ratio criterion to determine the number of pruned Gaussian points under GPU memory and computation/communication latency constraints. Furthermore, we develop a 3D-GS model recovery mechanism that restores structural consistency across local 3D-GS models without accessing private camera parameters, enabling their effective aggregation towards a global model. Finally, extensive experiments show that our approach significantly accelerates convergence, maintains high rendering quality, and reduces training latency compared to state-of-the-art federated 3D-GS baselines.