Gpu system enables real-time textured 3d mesh reconstruction from multiple rgb-d cameras
GPU-Native Real-Time Textured Mesh Reconstruction with Seam-Aware Multi-View Radiometric Correction
Computer Vision and Pattern RecognitionGraphics
Summary
It is challenging to quickly create detailed 3D models with textures from multiple depth and color cameras without slowing down due to data moving between a computer's graphics card and processor. This work presents a method that keeps all the main steps, including mesh creation and texturing, entirely on the graphics card for fast, real-time processing. They also solve the problem of visible lighting differences between camera images by adjusting color smoothly across the model. The authors demonstrate much faster speeds with good visual quality compared to previous methods on a standard dataset.
What this means in practice
- •For augmented reality developers: Build real-time textured 3D models from multiple RGB-D cameras for immersive AR experiences without CPU bottlenecks.
- •For robotics engineers: Use fast GPU-native mesh reconstruction to create accurate environment maps from multiple sensors for real-time robot navigation.
Authors
Hiroyuki Sato
Abstract
We present a GPU-native volumetric-video system for real-time reconstruction of graphics-compatible textured triangle meshes from multiple RGB-D sensors. Existing TSDF systems accelerate fusion and mesh extraction, but extending them to per-frame textured meshing adds CPU mesh processing, CPU-side texturing work, or extra CPU-GPU transfers between stages. After input staging, our method keeps the reconstruction and texturing stages (TSDF fusion, Marching Cubes, connected-component noise removal, view-based texture mapping, inpainting, and radiometric seam correction) resident on the GPU; no intermediate mesh or image data returns to the host between these stages. A fixed-size-array representation of mesh topology gives constant-time neighbor access on the GPU and makes the downstream mesh-level operations data-parallel. Because we texture each face from a single camera, exposure seams that vary smoothly across each image remain; we estimate a per-camera low-frequency gain field and bake it into the atlas on the GPU. On the CMU Panoptic dataset, our method is faster than the baselines at every tested voxel resolution and number of sensors, achieves peak speedups of more than an order of magnitude over CUDA-accelerated nvblox and CPU-based Open3D, and approaches an offline textured-meshing baseline in perceptual re-rendering quality.