Model combines images and partial point clouds for accurate building reconstruction

Any2Building: Unifying Point Clouds and Images for Building Reconstruction via Shared Spatial Queries

Computer Vision and Pattern Recognition

Summary

Building detailed 3D models of buildings is hard when we only have partial data like incomplete 3D scans or overhead photos. The researchers present Any2Building, a system that cleverly combines these two types of data into one framework, improving building reconstruction accuracy. Their method uses shared spatial queries to connect visual and geometric cues in 3D space, helping to fill in missing parts. They also created a large dataset to train and test the system, showing it works better than existing methods.

What this means in practice

  • For urban planners: Generate detailed 3D city models from limited aerial photos and partial scans for planning and visualization projects.
  • For game developers: Create accurate low-poly building models combining aerial imagery and sparse 3D data for virtual environments.$Commercial implications: Enables production of realistic yet efficient building assets for games using combined image and point cloud data.

Authors

Hongye Hou, Zhuoxiao Li, Liu Xiangyu, Tengxi Wang, Jing Ou, Dongli Wu, Shuai Zhang, Tongyan Hua, Wufan Zhao

Abstract

Abstracting structured buildings from limited observations is essential for 3D urban modeling and digital twins. Partial point clouds and overhead images of the same building provide incomplete 3D surface measurements and visual cues to its footprint, roof geometry, and rooftop structures, respectively. This motivates a unified framework that aligns geometric and visual cues in 3D and reconstructs buildings from either or both modalities. Building on this connection, we present Any2Building, a unified model for Level of Detail 2 (LoD2) building reconstruction from partial point clouds, overhead images, or both, with optional coarse geometric priors. At its core is an occupancy-grounded shared spatial queries representation that organizes available observations within a common 3D grid. Supervised by complete building surface voxels, these queries connect observational evidence to target geometry, enabling a shared generator to learn structural priors across tasks and providing spatial guidance for recovering missing structure. A trained voxel conditional surface-generation and low-poly remeshing pipeline then converts the predicted structure into compact LoD2 building models. To support this framework, we introduce BuildingWorld-Image, comprising approximately one million paired samples of partial point clouds, overhead images, and LoD2 building models across 18 cities. Experiments on multiple benchmarks demonstrate that Any2Building outperforms existing methods in geometric reconstruction accuracy across unimodal and multimodal point-cloud completion, single-image building generation, and prior-assisted generation, while maintaining detailed and robust recovery when point cloud input modality is degraded.