Edge Prediction for Roof Wireframe Reconstruction with Transformers
2026-06-01 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors present a new method to create 3D models of house roofs using sparse 3D points and ground-level images showing what different parts of the scene are. Their method uses a specialized Transformer model that learns to connect points and semantic features to predict wireframe edges representing roof structures. They improve the input by focusing on important areas and combining different feature types through a pre-trained autoencoder. Tested on a large dataset, their approach outperformed other methods and achieved the second-best score in a competition.
3D reconstructionroof wireframesparse SfM point cloudssemantic segmentationdepth mapsTransformer encoder-decoderDETRGestalt featuresautoencodercross-attention
Authors
Gustav Hanning, Ludvig Dillén, Jonathan Astermark, Johanna Lidholm, Viktor Larsson
Abstract
This paper presents a competitive solution to the S23DR Challenge 2026, which aims to reconstruct 3D house roof wireframe models from sparse SfM point clouds and ground-level semantic segmentations and depth maps. Our proposed method utilizes an end-to-end Transformer encoder-decoder architecture inspired by DETR. To effectively process the geometric and semantic data, the sparse SfM point cloud input is dynamically subsampled based on semantic priority and augmented with Gestalt and ADE20k class features. To further increase segmentation context, we fuse the point features with additional Gestalt feature encodings which are obtained by projecting the points into latent feature maps produced by a frozen autoencoder. Learned query embeddings are then decoded directly into 3D wireframe edges via cross-attention mechanisms. Evaluated on the "HoHo 22k" dataset, our approach significantly outperforms both handcrafted and learned baselines, achieving a Hybrid Structure Score (HSS) of 0.6476 and securing the second-highest position on the challenge's private leaderboard.