Unified 3d reconstruction combines rgb and x-ray for better object models
X-Splat: Unified Gaussian Splatting for Joint RGB and X-ray 3D Reconstruction
Computer Vision and Pattern RecognitionGraphics
Summary
Getting a full understanding of objects requires seeing their outside and inside together. The authors created a single 3D model that merges color images with X-ray scans, instead of making separate models for each. This helps keep the outside and inside views well aligned and improves the quality of both color and X-ray images. Their new way also uses less data and gives more stable combined results than previous methods.
What this means in practice
- •For medical imaging professionals: Create combined 3D visualizations that align external appearance and internal X-ray structures for better diagnostics.
- •For industrial inspection teams: Produce compact and well-aligned 3D models from RGB and X-ray views to detect defects in manufactured parts more effectively.
Authors
Wangze Xu, Chu Zhou, Mae Yamaguchi, Linrui Dai, I-Chao Shen, Takashi Ijiri, Imari Sato
Abstract
Jointly reconstructing RGB and X-ray provides new insights by connecting external appearance with internal structure, enabling a more complete understanding of real-world objects than single modality. Yet existing methods train separate models per modality and suffer from redundant geometry, alignment artifacts, and loose coupling that limits cross-modal insight discovery. We propose a unified representation in which all Gaussian primitives share a single set of geometric parameters, with only surface-proximal primitives additionally carrying appearance attributes. We perform cross-modal registration through clean multi-view stereo and X-ray geometric proxies instead of aligning two sets of optimized Gaussians, improving alignment stability. We then jointly optimize the unified representation via supervision from both modalities, complemented by a surface-depth consistency loss derived from X-ray geometry to guide RGB reconstruction. Experiments on the InsideOut dataset and our Flower & Insect dataset show that our framework achieves more robust cross-modal alignment, higher RGB rendering quality without compromising X-ray reconstruction, and a more compact representation than maintaining two independent geometries.