Point cloud upsampling improves with distilled multi-view visual knowledge
ControlFlow3D: Distilling Multi-View Knowledge into Latent Flow Matching for Point Cloud Upsampling
Computer Vision and Pattern Recognition
Summary
Making 3D shapes look better when starting from sparse points is hard because there’s not enough detail. The authors created a system called ControlFlow3D that learns from multiple pictures of an object during training, then can improve 3D point clouds even without those pictures later. They use a clever way to connect image and 3D shape information through shared features, teaching a smaller network to fill in details efficiently. They also built a large dataset of objects to test their approach thoroughly.
What this means in practice
- •For 3d graphics developers: Improve the quality of 3D models generated from sparse scans without needing multiple images at runtime, enabling faster and more detailed reconstructions.
- •For autonomous vehicle engineers: Enhance sparse LiDAR point clouds by upsampling with learned image knowledge, helping perception systems recognize objects more accurately.
Authors
Yuang Liu, Zhi Zuo, Zhengkai Zhao, Lirui Zhang, Pan Gao, Hao Feng, Zhengzhe Liu
Abstract
Point cloud upsampling is essential for 3D applications but remains challenging due to sparse, under-constrained observations. Multi-view images can resolve geometric ambiguity, but existing multi-modal methods do not fully exploit visual cues. We propose ControlFlow3D, a framework that distills knowledge from pretrained foundation models into a lightweight generation network during training, enabling strong performance without multi-view images available at test time. Specifically, our method performs conditional flow matching in a structured latent space defined by a frozen 3D encoder. Multi-view image tokens and 3D latent tokens are jointly clustered into shared prototypes via unsupervised soft KMeans; the resulting prototypes guide the flow trajectory through velocity control and are aligned by a consistency loss that progressively transfers image knowledge into the base network. A Mamba-based geometry decoder then maps latent features to dense coordinates with linear complexity. We also construct ShapeNetPU, a 35K-object multi-modal benchmark that is 31 times larger than prior datasets. Extensive experiments across multiple benchmarks demonstrate the effectiveness of our method and its strong generalization capability.