Point cloud models recover performance from unseen rotations without gradients
Recovering Black-Box Point Cloud Models from Pose Shifts via Derivative-Free Optimization
Computer Vision and Pattern RecognitionArtificial IntelligenceMachine Learning
Summary
Point cloud models can struggle when objects are rotated in ways they weren't trained on, making them less accurate. The authors introduce CanDO, a method that fixes this by rotating the input again to find the version the model feels most confident about. This is done without needing to look inside the model or use its gradients. By searching carefully, CanDO can restore most of the lost accuracy caused by unfamiliar rotations after the model is deployed.
What this means in practice
- •For 3d software developers: Improve the robustness of 3D object recognition models by enabling rotation correction without accessing model internals or retraining.
- •For augmented reality developers: Enhance AR applications with point cloud models that maintain accuracy despite unexpected changes in object orientation.
Authors
Seonghwan Park, Min-Gyu Park
Abstract
Point cloud models are increasingly deployed through score-only black-box interfaces. Under this access model, test-time rotations outside the training distribution can substantially degrade performance. Although existing mitigations such as equivariant retraining and gradient-based adaptation are effective, they are infeasible without access to model weights or gradients. To address this limitation, we propose CanDO, a test-time canonicalization method that recovers lost performance using only forward queries. Specifically, CanDO searches for an input transformation that maximizes model confidence and uses derivative-free optimization to perform this search without gradients. We further identify three conditions for successful recovery: a training augmentation gap, an accuracy-aligned confidence surrogate, and global search over a multi-modal objective. Across multiple architectures and benchmarks, CanDO recovers 76% of the oracle gap under azimuthal misorientation and 51% under arbitrary SO(3) rotations for object part segmentation. These results show that frozen score-only point cloud models can recover from rotation-induced distribution shifts through post-deployment canonicalization alone.