Pretrained geometric features improve 3D object tracking accuracy
MAETrack: Unleashing the Potential of Pretrained Geometric Priors for 3D Single Object Tracking
Computer Vision and Pattern Recognition
Summary
Tracking a single object in 3D space is harder than in 2D, especially when trying to reuse knowledge from previous training. The authors found that early layers of a pretrained 3D model keep useful shape information that helps tracking, while deeper layers are too focused on a different training task. They designed a method to keep the good early layers and retrain the deeper parts to work better for tracking. Their approach, called MAETrack, boosts tracking performance without much extra computing.
What this means in practice
- •For autonomous vehicle teams: Improve the accuracy of vehicle systems that track objects around them using pretrained 3D features adapted for tracking.
- •For robotics engineers: Enhance robotic perception modules to better follow objects in 3D environments by selectively reusing pretrained model layers.
Authors
Sifan Zhou, Qiwei Wang, Linyue Tan, Ziyu Liu, Ziyu Zhao, Xiaobo Lu
Abstract
Large-scale pre-training has transformed representation learning in 2D vision, yet its transferability to 3D single object tracking (SOT) remains insufficiently understood. Directly fine-tuning self-supervised 3D encoders, such as masked autoencoders (MAE), often leads to sub-optimal adaptation because the reconstruction objective is not fully aligned with the spatial-temporal matching requirements of tracking. In this paper, we observe that this difficulty can be interpreted as a layer-wise transfer mismatch: shallow layers tend to preserve transferable geometric cues, while deeper layers become increasingly specialized to the reconstruction pretext task and are less suitable for downstream tracking. Based on this observation, we propose MAETrack, a lightweight adaptation framework for transferring pre-training MAE representations to 3D SOT. MAETrack includes Layer-Selective Initialization (LSI), which initializes only the shallow stages of the tracking backbone from pre-trained weights while re-initializing deeper stages, and Geometric Residual Gating (GRG), which reinforces structurally salient regions in the search BEV features before template-search fusion through residual spatial modulation. Extensive experiments on standard 3D SOT benchmarks show that MAETrack consistently improves upon vanilla fine-tuning baselines with limited computational overhead. More broadly, our results suggest that effective transfer from 3D reconstruction pre-training to 3D tracking is not merely a matter of partial fine-tuning, but depends on a tracking-oriented transfer principle that preserves shallow geometry while adapting deeper representations to the downstream objective.