Transformer reconstructs 3D editable curves from sparse images
CGGT: Curve-Grounded Geometry Transformer for 3D Parametric Curve Reconstruction
Computer Vision and Pattern Recognition
Summary
Turning flat images into editable 3D curves is tricky, especially from only a few photos. The authors created a new tool called CGGT that learns to find consistent curve shapes from several unaligned images. It predicts camera positions and depth while identifying curves, then refines these into neat 3D curve shapes. They also built a huge dataset of 3D models to help train CGGT, which works well even on real photos despite being trained just on computer-generated data.
What this means in practice
- •For cad designers: Generate editable 3D curve models directly from a small set of photos without requiring special camera setups or manual edge detection.$Commercial implications: Enables practical CAD wireframe generation tools from casual photos, reducing modeling time and expertise.
- •For computer graphics developers: Incorporate CGGT for efficient multi-view geometry reconstruction to improve 3D modeling pipelines from sparse image inputs.
Authors
Zhirui Gao, Renjiao Yi, Yunfan Ye, Ruizhen Hu, Chenyang Zhu, Wei Chen, Kai Xu
Abstract
Recovering editable 3D parametric curves from 2D images is a fundamental challenge in computer graphics, bridging pixel-based perception and vector-based CAD modeling. Existing NeRF- and 3DGS-based methods often rely on dense calibrated views, precomputed 2D edge maps, and costly per-scene optimization, limiting their applicability to casually captured real-world inputs. We propose CGGT, a Curve-Grounded Geometry Transformer that directly grounds 3D-consistent 2D curve instances in the image space from sparse, unposed multi-view images. CGGT combines a geometry-aware transformer encoder for multi-view feature learning with a curve-aware masked-attention decoder for cross-view instance association. In a single forward pass, it predicts camera parameters, dense depth maps, and instance-level 2D curve masks, which are then lifted into 3D and refined through a fast parametric optimization stage to recover compact, editable 3D curve primitives. To support structured curve learning, we introduce Wireframe-100K, a large-scale dataset comprising 100,000 CAD models with diverse topologies, realistic multi-view renderings, and accurate parametric curve annotations. Extensive experiments show that our framework achieves substantial improvements in both reconstruction accuracy and efficiency, particularly under challenging sparse-view settings and in separating persistent 3D structural edges from view-dependent image edges caused by silhouettes, textures, and appearance variations. Despite being trained solely on synthetic data, CGGT generalizes well to real-world images, demonstrating its potential for practical CAD-style wireframe reconstruction from unconstrained visual inputs.