Geometry-aware prompting improves 3D model adaptation with fewer parameters
GAPrompt++: Multi-Granular Geometry-Aware Point Cloud Prompt for 3D Vision Model
Computer Vision and Pattern Recognition
Summary
Adapting large 3D vision models to new tasks usually needs a lot of computing power and storage. The authors propose GAPrompt++, a method that adds special geometric clues at different detail levels to help models learn faster and use fewer adjustments. Their approach carefully highlights important shapes and points, spreading this information through the model to improve understanding of 3D scenes. Tests show this method works better than many others, even beating full retraining, while needing less than 2% of the parameters to be changed. They also created harder test sets for point cloud models to push future research further.
What this means in practice
- •For 3d computer vision engineers: Adapt pre-trained 3D models to new tasks using less computing and storage by injecting geometric prompts for faster training.
- •For augmented reality developers: Improve real-time 3D environment understanding by efficiently fine-tuning models with geometry-aware prompts on realistic point cloud data.
Authors
Zixiang Ai, Zhenyu Cui, Yufei Guo, Wenwen Qiang, Lei Chen, Jiwen Lu, Jiahuan Zhou
Abstract
Pre-trained 3D vision models have substantially advanced point cloud analysis, yet adapting them to downstream tasks via full fine-tuning is computationally expensive and storage-intensive. Parameter-Efficient Fine-Tuning (PEFT) offers a promising alternative by reducing both adaptation cost and storage burden. However, existing prompting-based approaches ignore the intrinsic geometric structures of point clouds, thereby limiting their adaptation capability. This limitation stems from their inability to encode both fine-grained geometric cues and coarse-grained structural semantics, as well as failing to propagate such information effectively through the model hierarchy. To address these challenges, we propose GAPrompt++, a multi-granular geometry-aware prompting method that provides richer geometric guidance for efficient 3D task adaptation. Specifically, we introduce a Point Shift Prompter that extracts multi-granular geometric features across different scales, enabling instance-specific geometric adjustments during adaptation. Next, a Keypoint Prompter adaptively generates point-level prompts to highlight local geometric saliency and fine-grained structural details. Furthermore, a Prompt Propagation mechanism injects these multi-granular geometric cues throughout the feature extraction hierarchy, strengthening the ability to capture essential geometric characteristics. Extensive experiments show that GAPrompt++ achieves state-of-the-art performance among prompting-based PEFT methods and even surpasses full fine-tuning across diverse benchmarks, while requiring less than 2\% trainable parameters. In addition, to address the saturation of existing evaluation datasets, we construct two more challenging benchmarks derived from 3D Gaussian Splatting and Multi-View Stereo reconstruction, offering diverse and realistic point cloud scenarios to promote future research.